CIRCA:Assessment Tools

From CIRCA

(Difference between revisions)
Jump to: navigation, search
(Statistical Analysis)
Line 50: Line 50:
*Link to the [[CIRCA:Initial questions to ask when starting group research projects with a practical game design component - |Grand Assessment Framework]] for which method may be useful for which section of the framework. For example, Interviewing can be linked to Stakeholders and Expectations as '1.0', under recommended use with framework.
*Link to the [[CIRCA:Initial questions to ask when starting group research projects with a practical game design component - |Grand Assessment Framework]] for which method may be useful for which section of the framework. For example, Interviewing can be linked to Stakeholders and Expectations as '1.0', under recommended use with framework.
-
==Developmental Evaluation==
+
==[[CIRCA: A/B Testing|AB Testing]]==
 +
Testing your current design against alternative ones to determine which ones produce positive results.
-
Developmental Evaluation makes use of a developmental evaluator to inquire into developments, track developments, facilitate interpretation of developments and their significance, and engages with multiple parties engaged in and affected by the product regarding making judgements about what is being developing, how long it is being developed, and the consequences and impacts of what has been developed, and what are the next stages of development.
+
Recommended Use with the Framework:
-
 
+
-
Recommended Use with Framework:
+
Associated Literature:
Associated Literature:
-
* Patton, Michael. Developmental Evaluation: Applying Complexity Concepts to Enhance Innovation and Use . New York: Guiford, 2011.
 
 +
==Developmental Evaluation==
-
==Play-Testing==
+
Developmental Evaluation makes use of a developmental evaluator to inquire into developments, track developments, facilitate interpretation of developments and their significance, and engages with multiple parties engaged in and affected by the product regarding making judgements about what is being developing, how long it is being developed, and the consequences and impacts of what has been developed, and what are the next stages of development.  
-
Play-testing focuses on measuring the product’s quality and functionality.  
+
Recommended Use with Framework:
Recommended Use with Framework:
Associated Literature:
Associated Literature:
-
* Warren, Scott, Greg Jones, and Line Lin. "Usability and Play Testing: The Often Missed Assessment." In Serious Educational Game Assessment: Practical Methods and Models for Educational Games, Simulations and Virtual Worlds . Rotterdam: Sense Publishers, 2011. 131-146.  
+
* Patton, Michael. Developmental Evaluation: Applying Complexity Concepts to Enhance Innovation and Use . New York: Guiford, 2011.
-
==Usability-Testing==
+
==Digitization==
-
Usability-testing focuses on measuring the capacity for the product to meet its intended purposes with the use of user-centered interaction.
+
-
Recommended Use with Framework:
+
Recommended Use with the Framework:
Associated Literature:
Associated Literature:
-
*[[CIRCA:Usability and Instructional Design Heuristics for E-Learning Evaluation - Benson, L., et al. | Perry, David., Rusel DeMaria, and Safari Books Online (Firm). Benson, L., D. Elliott, M. Grant, D. Holschuh, B. Kim, H. Kim, E. Lauber, S. Loh, and T.C. Reeves. "Usability and Instructional Design Heuristics for E-Learning Evaluation." Proceedings of World Conference on Educational Multimedia, Hypermedia and Telecommunications 2002 . Chesapeake: AACE, 2002. 1615-1621 ]] 
 
-
* [[CIRCA:David Perry on Game Design: A Brainstorming Toolbox |Perry, David., Rusel DeMaria, and Safari Books Online (Firm). David Perry On Game Design: A Brainstorming Toolbox. Boston: Charles River Media, 2009]]
 
-
* Warren, Scott, Greg Jones, and Line Lin. "Usability and Play Testing: The Often Missed Assessment. Serious Educational Game Assessment: Practical Methods and Models for Educational Games, Simulations and Virtual Worlds . Rotterdam: Sense Publishers, 2011. 131-146.
 
 +
==Focus Groups==
 +
Asking a group of people about their  perceptions, opinions, beliefs, and attitudes regarding a product, issue, service, or idea.
-
==Standardized Criteria==
+
Recommended Use with the Framework:
-
Including checklists and rubrics, standardized criteria sets expectations and a focus on measuring specific objectives.
+
-
 
+
-
Recommended Use with Framework:
+
Associated Literature:
Associated Literature:
-
* [[CIRCA:Game Analysis: Project Design Checklist - Hjartarson, Paul |Hjartarson, Paul. Note. Project Design Checklist.]] 
 
-
* [[CIRCA:Assessing Higher Order Thinking in Video Games - Rice, John W. |Rice, J. "Assessing Higher Order Thinking in Video Games."Journal of Technology and Teacher Education. 1.15 (2007): 87-100.]] 
 
-
* [[CIRCA:The Art of Game Design - Jesse Schell |Schell, Jesse. The art of game design: a book of lenses. San Francisco: Morgan Kaufmann Publishers Inc., 2008. ]] 
 
-
* [[CIRCA:Serious Educational Game Assessment - Leonard Annetta and Stephen Bronack (Eds.) |Annetta, Leonard A., Richard Lamb, and Marcus Stone. “Assessing Serious Educational Games: The Development of a Scoring Rubric”. In Serious Educational Game Assessment: Practical Methods and Models for Educational Games, Simulations and Virtual Worlds . Rotterdam: Sense Publishers, 2011. 75-94]] 
 
-
* Javier, Ricardo and Mena, Rademacher. “Game Assessment Using the E/E Grid”. In Serious Educational Game Assessment: Practical Methods and Models for Educational Games, Simulations and Virtual Worlds . Rotterdam: Sense Publishers, 2011. 95-118.
 
-
==Journaling==
+
==Historical Methods==
-
Keeping reflective reports during the progress of the project.
+
Recommended Use with the Framework:
Recommended Use with the Framework:
Associated Literature:
Associated Literature:
-
* Ortlipp, Michelle. "Keeping and Using Reflective Journals in the Qualitative Research Process". In The Qualitative Report. 14.4 (2008):695-705.
 
-
==Focus Groups==
+
==Interviewing==
-
Asking a group of people about their  perceptions, opinions, beliefs, and attitudes regarding a product, issue, service, or idea.
+
Recommended Use with the Framework:
Recommended Use with the Framework:
Line 106: Line 91:
Associated Literature:
Associated Literature:
-
==[[CIRCA: A/B Testing|AB Testing]]==
+
==Journaling==
-
Testing your current design against alternative ones to determine which ones produce positive results.
+
Keeping reflective reports during the progress of the project.
Recommended Use with the Framework:
Recommended Use with the Framework:
Associated Literature:
Associated Literature:
 +
* Ortlipp, Michelle. "Keeping and Using Reflective Journals in the Qualitative Research Process". In The Qualitative Report. 14.4 (2008):695-705.
-
==Interviewing==
+
==Play-Testing==
 +
Play-testing focuses on measuring the product’s quality and functionality.
-
Recommended Use with the Framework:
+
Recommended Use with Framework:
Associated Literature:
Associated Literature:
 +
* Warren, Scott, Greg Jones, and Line Lin. "Usability and Play Testing: The Often Missed Assessment." In Serious Educational Game Assessment: Practical Methods and Models for Educational Games, Simulations and Virtual Worlds . Rotterdam: Sense Publishers, 2011. 131-146.
-
==Talk Aloud==
+
==[[CIRCA: Basic Statistical Analysis|Statistical Analysis]]==
 +
The collection and interpretation of statistical data. Statistics are frequently used to find relationships between data sets, extracting important properties from data sets, and visualizations.
Recommended Use with the Framework:
Recommended Use with the Framework:
Line 125: Line 114:
Associated Literature:
Associated Literature:
-
==Text Methods==
+
==Standardized Criteria==
 +
Including checklists and rubrics, standardized criteria sets expectations and a focus on measuring specific objectives.
-
Recommended Use with the Framework:
+
Recommended Use with Framework:
Associated Literature:
Associated Literature:
 +
* [[CIRCA:Game Analysis: Project Design Checklist - Hjartarson, Paul |Hjartarson, Paul. Note. Project Design Checklist.]] 
 +
* [[CIRCA:Assessing Higher Order Thinking in Video Games - Rice, John W. |Rice, J. "Assessing Higher Order Thinking in Video Games."Journal of Technology and Teacher Education. 1.15 (2007): 87-100.]] 
 +
* [[CIRCA:The Art of Game Design - Jesse Schell |Schell, Jesse. The art of game design: a book of lenses. San Francisco: Morgan Kaufmann Publishers Inc., 2008. ]] 
 +
* [[CIRCA:Serious Educational Game Assessment - Leonard Annetta and Stephen Bronack (Eds.) |Annetta, Leonard A., Richard Lamb, and Marcus Stone. “Assessing Serious Educational Games: The Development of a Scoring Rubric”. In Serious Educational Game Assessment: Practical Methods and Models for Educational Games, Simulations and Virtual Worlds . Rotterdam: Sense Publishers, 2011. 75-94]] 
 +
* Javier, Ricardo and Mena, Rademacher. “Game Assessment Using the E/E Grid”. In Serious Educational Game Assessment: Practical Methods and Models for Educational Games, Simulations and Virtual Worlds . Rotterdam: Sense Publishers, 2011. 95-118.
-
==Digitization==
+
==Success/Failure Counters==
-
Recommended Use with the Framework:
+
Recommended Use with the Framework==
Associated Literature:
Associated Literature:
-
==[[CIRCA: Basic Statistical Analysis|Statistical Analysis]]==
+
==Talk Aloud==
-
The collection and interpretation of statistical data. Statistics are frequently used to find relationships between data sets, extracting important properties from data sets, and visualizations.
+
Recommended Use with the Framework:
Recommended Use with the Framework:
Line 144: Line 138:
Associated Literature:
Associated Literature:
-
==Historical Methods==
+
==Text Methods==
Recommended Use with the Framework:
Recommended Use with the Framework:
Line 150: Line 144:
Associated Literature:
Associated Literature:
-
==Success/Failure Counters==
+
==Usability-Testing==
 +
Usability-testing focuses on measuring the capacity for the product to meet its intended purposes with the use of user-centered interaction.
-
Recommended Use with the Framework==
+
Recommended Use with Framework:
Associated Literature:
Associated Literature:
 +
*[[CIRCA:Usability and Instructional Design Heuristics for E-Learning Evaluation - Benson, L., et al. | Perry, David., Rusel DeMaria, and Safari Books Online (Firm). Benson, L., D. Elliott, M. Grant, D. Holschuh, B. Kim, H. Kim, E. Lauber, S. Loh, and T.C. Reeves. "Usability and Instructional Design Heuristics for E-Learning Evaluation." Proceedings of World Conference on Educational Multimedia, Hypermedia and Telecommunications 2002 . Chesapeake: AACE, 2002. 1615-1621 ]] 
 +
* [[CIRCA:David Perry on Game Design: A Brainstorming Toolbox |Perry, David., Rusel DeMaria, and Safari Books Online (Firm). David Perry On Game Design: A Brainstorming Toolbox. Boston: Charles River Media, 2009]]
 +
* Warren, Scott, Greg Jones, and Line Lin. "Usability and Play Testing: The Often Missed Assessment. Serious Educational Game Assessment: Practical Methods and Models for Educational Games, Simulations and Virtual Worlds . Rotterdam: Sense Publishers, 2011. 131-146.

Revision as of 01:05, 16 October 2013

This page gathers together a variety of techniques for evaluating and assessing video games.

Contents

General List of Assessment Techniques

Note: See the Assessment Framework for more information regarding the assessment frameworking we're developing

Qualitative Assessment Techniques

Qualitative techniques are likely to dominate video game assessment, for the success of a video game experience is often predicated upon difficult to quantify human elements such as "fun" and "enjoyment".

Evaluation by Game Creators

Evaluation by Game Players

Game players may be asked to evaluate a game in a variety of ways.


Quantitative Assessment Techniques

Quantitative assessment of video games generally entails analyzing data collected about the progress of players through the game.

Quantitative assessment can involve the measurement of:

  • of tickets/tokens purchased
  • Success/failure rates
  • Journaling
  • Time in-game
  • Amount of content interacted with
  • Locations the game was accessed from
  • specific times the game was accessed


Specific approaches to projects

Draft Revised List For Methods/Techniques

Note: Below is a list of some methods that may be used to assess parts of game design and assessment. Each method includes a brief explanation why it may be relevant to design as well as a few associated sources from our literature review. Please feel free to add or the revise to the list.

Guidelines for Methods:

  • Add only relevant methods.
  • Use a brief description to define the method.
  • Link it to relevant literature in the Literature Review & Precises section when possible.
  • Use proper citation
  • Alphabetical order is preferred.
  • Link to the Grand Assessment Framework for which method may be useful for which section of the framework. For example, Interviewing can be linked to Stakeholders and Expectations as '1.0', under recommended use with framework.

AB Testing

Testing your current design against alternative ones to determine which ones produce positive results.

Recommended Use with the Framework:

Associated Literature:

Developmental Evaluation

Developmental Evaluation makes use of a developmental evaluator to inquire into developments, track developments, facilitate interpretation of developments and their significance, and engages with multiple parties engaged in and affected by the product regarding making judgements about what is being developing, how long it is being developed, and the consequences and impacts of what has been developed, and what are the next stages of development.

Recommended Use with Framework:

Associated Literature:

  • Patton, Michael. Developmental Evaluation: Applying Complexity Concepts to Enhance Innovation and Use . New York: Guiford, 2011.

Digitization

Recommended Use with the Framework:

Associated Literature:

Focus Groups

Asking a group of people about their perceptions, opinions, beliefs, and attitudes regarding a product, issue, service, or idea.

Recommended Use with the Framework:

Associated Literature:

Historical Methods

Recommended Use with the Framework:

Associated Literature:

Interviewing

Recommended Use with the Framework:

Associated Literature:

Journaling

Keeping reflective reports during the progress of the project.

Recommended Use with the Framework:

Associated Literature:

  • Ortlipp, Michelle. "Keeping and Using Reflective Journals in the Qualitative Research Process". In The Qualitative Report. 14.4 (2008):695-705.

Play-Testing

Play-testing focuses on measuring the product’s quality and functionality.

Recommended Use with Framework:

Associated Literature:

  • Warren, Scott, Greg Jones, and Line Lin. "Usability and Play Testing: The Often Missed Assessment." In Serious Educational Game Assessment: Practical Methods and Models for Educational Games, Simulations and Virtual Worlds . Rotterdam: Sense Publishers, 2011. 131-146.

Statistical Analysis

The collection and interpretation of statistical data. Statistics are frequently used to find relationships between data sets, extracting important properties from data sets, and visualizations.

Recommended Use with the Framework:

Associated Literature:

Standardized Criteria

Including checklists and rubrics, standardized criteria sets expectations and a focus on measuring specific objectives.

Recommended Use with Framework:

Associated Literature:

Success/Failure Counters

Recommended Use with the Framework==

Associated Literature:

Talk Aloud

Recommended Use with the Framework:

Associated Literature:

Text Methods

Recommended Use with the Framework:

Associated Literature:

Usability-Testing

Usability-testing focuses on measuring the capacity for the product to meet its intended purposes with the use of user-centered interaction.

Recommended Use with Framework:

Associated Literature:

Personal tools