CIRCA:Serious Educational Game Assessment - Leonard Annetta and Stephen Bronack (Eds.)


Revision as of 17:19, 3 March 2011 by CalenHenry (Talk | contribs)
(diff) ← Older revision | Current revision (diff) | Newer revision → (diff)
Jump to: navigation, search

Annetta, L., & Bronack, S. (2010). Serious Educational Game Assessment: Practical Methods and Models for Educational Games, Simulations and Virtual Worlds. Sense Publishers.

Précis – Calen Henry – February 2011

Serious Educational Games, Simulations and Virtual worlds are the 21st century. There is a push for technology use in schools, but assessment techniques remain archaic. The book tries to provide some accounts of how Serious Games pioneers are using virtual worlds and assessing their effectiveness.


1. Assessing Gaming, Computer and Scientific Inquiry Self-efficacy in a Virtual Environment – p. 1

Diane Jass Ketelhut

Ketelhut outlines the development, piloting and revision of a tool for measuring student self-efficacy, initiated because she noticed a decline in interest in studying science in high school student, as compared to elementary school students. Scientific inquiry has shown to increase interest in a scientific career and virtual environments have been investigated for scientific inquiry-based curricula. Multi-user virtual environments (MUVEs) have been used to teach scientific inquiry so self-efficacy assessment can start there.

Self-Efficacy in Technology and Science (SETS) tool developed. Surver Monkey was used to ask students self-efficacy questions.

SETS showed that once students perceive themselves as competent their effort and perseverance increases and they can be given greater challenges.

2. Self-Regulation within Game-based Learning Environments – p. 19

John Neitfeld and Lucy R. Shores

Digital games have proven capable of capturing the attention of youth for long periods of time and compelling them contend with steep learning curves in order to succeed. For this reason, Game Based Learning Environments (GBLEs) interest educators. The chapter investigates implementing Self-Regulated Learning (SRL) in GBLEs and goes through existing SRL research. SRL is described as beliefs, motives, strategies and reflective processes that allow learners to autonomously direct their own learning. It requires cognitive strategies and an environment conducive to learning and seeking help when appropriate.

3. The Wild West of Assessment: Measuring Aggression and Violence in Video Games – p. 43

Christopher J. Ferguson

The chapter reviews assessment in violent video game studies, acknowledging that much of it has been problematic and poorly researched. The study showed that using standardized and valid measures is a must and that the best outcome measures are developed by groups not evaluating the game as there is a tendency toward bias. They also found that the quality of assessment limits conclusions. Many violent video game researchers have been too ready to ignore the limits of aggression assessments when generalizing serious real-world examples of violence. By paying close attention to measurement and the mistakes of others serious games has much to offer for use of assessments in outcome research.

4. Diagrammatic Inquiry: Rosario is not Virtual and it’s not Reality – p. 57

Owen Kelly

Reports on the Marinetta Ombro project that created a synthetic culture on the island of Rosario. The project needed to be assessed but they were unsure how to suitably do so. Based upon the terms “cyberspace” and “virtual reality”, the Internet has come to be unduly viewed in spatial terms. They concluded that Second Life is best viewed as a tool, not as a geographical space and the best analysis is diagrammatic and not statistical. Diagrammetric thought is said to be using a set of understood relations to model a set of relations that are not understood.

5. Assessing Serious Educational Games: The Development of a Scoring Rubric – p. 75

Leonard A. Annetta, Richard Lamb and Marcus Stone

Bad educational games do not teach well, so a standard must be developed. Games must be able to show that goals have been specified and met. They developed an assessment rubric, taking into account the following types of learning: experiential, inquiry-based, self-efficacy, goal setting, continuous feedback and cooperation (team learning). A list of 15 elements all serious games should have was compiled:

  • Prologue
  • Tutorial/Practice Level
  • Interactive
  • Feedback
  • Identity
  • Immersion
  • Pleasurable frustration
  • Manipulation
  • Increasing Complexity
  • Rules
  • Informed Learning
  • Learning
  • Pedagogical Effectiveness
  • Reading Efficiency
  • Communication
  • Psychometrics

6. Game Assessment Using the E/E Grid – p. 95

Ricardo Javier Rademacher Mena

The author had previously presented the E/E Grid as a framework combining education and entertainment theories to study games. The grid uses Bartle’s player types: socializer, explorer, achiever, killer on one axis of the grid and Callois’ taxonomy of playstyles: competition, chance, mimicry, vertigo on the other. It then asks if there are game types that appease each combination of player type and gameplay style. It combines the latter with a grid placing Gardner’s intelligence types: physical, lingual, interpersonal, logical, spatial, musical on one axis and Anderson/Krathwohl’s knowledge domains: metacognitive, conceptual, procedural, factual on the other. The combined graph allows for entertainment and education analysis of games to be simultaneously displayed in one diagram.

7. Assessment Using After-action Review: Without Footage its Fiction - p. 119

Randy Brown

Chapter discusses options and usefulness for analyzing recorded game footage.

8. Usability and Play Testing: The Often Missed Assessment – p. 131

Scott Warren, Greg Jones and Lin Lin

Learning in games often takes longer than traditional face to face methods to achieve increases in formal learning outcomes. Many serious games are not play tested before being rolled out. They conclude that play testing is important as it helps to see parts of the game that interfere with learning objectives and could result in the gamer failing. It also helps to release high quality products that can achieve goals without wasting time with things that could have been fixed. Lastly, involving users in the development cycle helps developers see the game through the user’s eyes. Usability was tested through recording audio and videos of users playing games with usability testers, reading teacher and students in a classroom setting and closed usability lab.

9. Change is Constant; The Game is Serious: Managing Change with a Serious Game at One Automotive Company – p. 147

Kristen Cromer

Game is still a dirty word in many corporate environments. Because of this it’s important to reference things like play theory, flow theory, social learning theory and dynamic assessment when talking about games for training. Challenges in implementation included assessment in the affective domain, a heterogeneous target audience, and the personal level of the objectives. Assessment strategies used included self-assessment, game play as assessment, performance feedback and scoring, failure and replay and real world assessment and follow-through.

10. Learning in Virtual Worlds: Assessment Challenges and Opportunities – p. 159

Margaret Corbit, Jennifer Wofford and Suzanne Kolodziej

Many virtual worlds log user activity. Commercial game designers are experts at analyzing and using this data on the fly. Educators have limited access and knowledge in this area. The project discusses assessment methods used to assess Active Worlds. They found that students are motivated to build and create in virtual worlds. CyberCiv is presented as a refined structure for assessing building. The SciFair rubric adds self-assessment and chat and building log analysis can be used to develop tools for tracking real-time progress.

11. Mitigating the Hawthorne Effect Using Computer Simulations – p. 175

Shawn Y. Holmes

Presents methods to mitigate the Hawthorne effect, subject’s tendency to alter behaviour as they know they are being studied. Using digital environments allows every participant to experience the same environment and allows them to self-assess. Answers are then analyzed and catalogued depending on how often participants refer to being assessed.

12. Endogenous Learning in Multi User Virtual Environments – p. 189

Dave R. Dannenberg and Michael A. Evans

The chapter examines endogenous learning in World of Warcraft and Second Life. Endogenous games are games that are designed with gameplay tying to a specific curriculum or set of teaching goals. Good endogenous games should make distinguishing between learning and gameplay difficult. A player should have to master the content and context to master an endogenous game.

13. Art Education Avatars in Action: Learning, Teaching, and Assessing in 3D Virtual Worlds – p. 201

Lilly Lu

The chapter outlines how virtual worlds and avatars can benefit art education. The Second Life based VLE is collaborative, student-centered and activity-based. It treats SL as a study subject, a virtual world an emerging visual culture. Effective SL learning came from appropriate technical training. The duration of the course was only 8 weeks and the author wonders whether students kept up with SL.

14. Assessing Learning and Identity in Augmented Reality Science Games – p. 221

Matt Dunleavy and Brittney Simmons

This chapter describes assessment strategies for AR games for k-12.

Research Questions

How do students describe and understand:

  1. how playing the AR simulation aids or hinders their understanding of comprehension of the scientific inquiry process
  2. how playing AR simulations is different from a typical science class
  3. assuming different roles aids and/or hinders their motivation and learning
  4. understanding of complex systemic situations
  5. pedagogical practices and supports

AR games are promising for teaching collaborative problem solving. They can be viewed as mobile, physically intensive learning spaces with lowered real world consequences. AR simulations encourage risk free learning and capitalize on students’ ability to filter out relevant information.

15. Semi-virtual Embodied Learning-Real World STEM Assessment – p. 241

Mina C. Johnson-Glenberg, David Birchfield, Philippos Savvides and Colleen Megowan-Romanowicz

The chapter talks about SMALLab, a learning environment that allows users to move freely in a space while interacting with dynamic sonic and visual media. Game-like scenarios are presented using SMALLlab. 1 scenario is compared to a normal classroom study and the other is assesses embodiment and learning separately, compared to a desktop environment. The objective is to ascertain whether embodied learning is more effective that “mouse-driven” learning. They found the embodied environment to be conducive to learning, even complex concepts, but not more so than the desktop.

16. An Open-ended, Emergent Approach for Studying Serious Games – p. 259

Matthew J. Sharritt

The author tries to devise an in-depth, descriptive research design to provide a detailed account of the collaborative use of serious games. The approach is ethnomethodological and uses grounded theory and was found to be effective in conveying what was happening , in addition to findings from participants self-analysis.

Personal tools