CIRCA:Assessment Framework


Revision as of 23:08, 19 October 2012 by SimeonBlimke (Talk | contribs)
Jump to: navigation, search


(Tentative) Assessment Framework Questions

Use these question categories in the process of answering your own questions relevant to a specific project. Fill in category sub-questions as best you can, add your own or skip questions as needed.


  • Who are the legacy stakeholders for this project or this type of project?
    • sponsors
    • group members
    • community partners
    • audience demographics
  • Who are the existing stakeholders for this project?
    • administrative / legal
    • research
    • audience / customer
    • others
  • Who are new stakeholders, or what are recent changes in makeup/orientation/capacity?
  • What if any changes to this category can be expected during the course of this project?
  • Who are potential stakeholders
    • Is there a market or test group available, already existing or in demand?
    • What are potential inclusion or exclusion factors (Does working with these stakeholders require REB ethics clearance?)
  • Where can more information be found about a particular stakeholder, how can this information be used?
  • What is the criteria for acquiring and updating information about stakeholders?


  • What is the purpose of the project?
  • What is the requirement of the project at a minimum?
  • What ought to be the shared ambition of this project? (each stakeholder should describe this)
    • What is the agreed upon shared ambition for this project?
  • What deliverables are due to each stakeholder (what are they getting out of the project)?
  • What are the stakeholders' responsibilities to the group?
  • To what extent is the project contract to be formalized or kept informal?
  • What is the project timeline?
    • What are our deliverables' timelines
  • How will we conclude this project (what will the end-state look like?)


  • What are the deliverables?
  • What is the budget?
    • Who is responsible for all aspects of the budget?
  • What technology or tools are required to build, maintain, and play the game?
  • What is the length of time it would take to build and run the game?
    • Is there a deadline? Or can this project be delayed?
  • Is there previous work, either ours or another group’s, in this area?
  • How long should the project be able to last?
  • Are any outside stakeholders responsible for providing content/information/materials/funding/etc.?
  • Who is responsible for maintaining the project?
  • What does the end-state of the project look like?


  • Does the game work?
    • What doesn't work?
      • Is the problem technical, conceptual, or both?
    • Can the game be fixed or is this a future cautionary tale?
  • Who did the game work for (or not work for)?
    • How did the stakeholders react to the game?
  • Does the game advance our research goals?
  • Is this a successful research project?
  • Is this a successful game?
  • Are there ways to improve the game or the platform based on the results of this attempt?
  • Should the game be more intuitive or instructional?
    • How quickly will the target audience be able to learn the game?
  • If the project is repeatable, will we learn new things or benefit from running the game again?


  • Who and how do we want to give or receive feedback? (I think this needs to be broken down into several questions - SL)
    • How will we present findings to our stakeholders?
    • Who will we solicit feedback from?
      • What type of feedback is required? (qualitative? quantitative? game metrics?)
  • What would be the most efficient method - time and resource wise - of gathering assessment data?
  • Do the chosen assessment techniques require ethics clearance?
    • Do we need new ethics or can this fall under a previous project?
  • What tools will give us the most useful data?
  • What questions should the assessment tools pose?
    • Should there be multiple feedback tools available?

(I think this section needs to be much more detailed since this was the original focus of the framework - the assessment process) - SL

Assesment Schemas

Since everything is going into a report - it might be useful to try to order the types of activities you might end up having to report into some hierarchies.

For example, I am trying to make a scale, between nothing to report on the one extreme, and unexpected success (to the extent that reporting has a life outside the project) on the other.

    • 100 (a new academic area opens up discussing the major impact of this game)
    • (stuff in between)
  • 12 best practices with general applicability could be suggested from the project
  • 11 Revisions were put into practice and were demonstratively better, revealing best-practices for a specific area
  • 10 Comparisons led to good suggestions for revision
  • 9 Items implemented were evaluated in some meaningful comparison
  • 7 Several things were implemented in a way that achieved goals.
  • 7 Attempt to implement something was made that was well justified in terms of a concept/goal
  • 7 An attempt to implement with results that have some useful diagnostic value
  • 7 Any attempt to implement something was made
  • 6 Choices were made about or between concepts in a way that can be demonstrated to align with goals
  • 5 several implementation concepts were discussed in terms of goals, etc
  • 5 Goals, skills, abilities, values etc were connected to an implementation concept
  • 4 associations were categorized in relation to each other (e.g. creative writing for games is sometimes an ability, sometimes a skill that can be developed, and always contingent in determining whether to pursue a project where the gameplay is story-based)
  • 3 associations were categorized into general areas (e.g. that's a goal, that's a method, a challenge, an ability, a value)
  • 2 associations were categorized as in or out of bounds (e.g. a goal, value, or skill that is definitely or probably not relevant, relevant under some circumstances,or always relevant)
  • 1 associations were made between project and ideas that may or may not be applicable to the project
  • 0 nothing

  • Thematic Assessments
    • how well and in what ways did this project address the topic of gender?
    • What was the educational value of the project?

Misc. categories (fun, education, technology, etc...)

  • What are measurable variables that can be defined as '____'?
  • Can the game teach anything different than a textbook, class, or other resources? (Not all of our games are designed to be serious games - this is a better question for the assessment section for specific games, will not apply to all projects)
    • Can this game teach something that can be gained through other resources in a way that the targeted audience finds preferable? ( I think this is outside of the scope of our group, we are not really studying how best to teach a topic or curriculum, we are studying games.)
  • Is there technology involved? Or theory? (This question is very broad, I'm not sure what it means and how it is different from resources section - needs to be defined better or broken up into sub questions)
  • Do we need to advertise?
  • How will we launch the game and attract players?
    • Do we have a captive player group or is the game released in the wild?
  • Can we tweak the game during deployment? Is this a part of the plan?

Other things to put in

  • Intellectual Property: What sort of license does it use? Who owns the work? Who published it? What is the intellectual property?
  • Time and Money: How long did it take to make? How much did it cost to develop? What sort of organization developed it?
  • Comparison and Competition: What other games are similar? Who is the competition? How does it compare to other games?

Other Frameworks

External Links

has a long list of items - have a feeling that these are at different levels of importance so we should read through and sort.

Bits of advice that could be part of a framework content

  • flow in games (This is an awesome paper but please describe how it fits in with the design framework. I'm not sure why this is here)
Personal tools