CIRCA:Assessment Framework


Revision as of 20:39, 19 October 2012 by SimeonBlimke (Talk | contribs)
Jump to: navigation, search


(Tentative) Assessment Framework Questions

Use these question categories in the process of answering your own questions relevant to a specific project. Fill in category sub-questions as best you can, add your own or skip questions as needed.


  • Who are the legacy stakeholders for this project or this type of project?
    • sponsors
    • group members
    • community partners
    • audience demographics
  • Who are the existing stakeholders for this project?
    • administrative / legal
    • research
    • audience / customer
    • others
  • Who are new stakeholders, or what are recent changes in makeup/orientation/capacity?
  • What if any changes to this category can be expected during the course of this project?
  • Who are potential stakeholders
    • Is there a market or test group available, already existing or in demand?
    • What are potential inclusion or exclusion factors (Does working with these stakeholders require REB ethics clearance?)
  • Where can more information be found about a particular stakeholder, how can this information be used?
  • What is the criteria for acquiring and updating information about stakeholders?


  • What is the purpose of the project?
  • What is the requirement of the project at a minimum?
  • What ought to be the shared ambition of this project? (each stakeholder should describe this)
    • What is the agreed upon shared ambition for this project?
  • What deliverables are due to each stakeholder (what are they getting out of the project)?
  • What are the stakeholders' responsibilities to the group?
  • To what extent is the project contract to be formalized or kept informal?
  • What is the project timeline?
    • What are our deliverables' timelines
  • How will we conclude this project (what will the end-state look like?)


  • What are the deliverables?
  • What is the budget?
    • Who is responsible for all aspects of the budget?
  • What technology or tools are required to build, maintain, and play the game?
  • What is the length of time it would take to build and run the game?
    • Is there a deadline? Or can this project be delayed?
  • Is there previous work, either ours or another group’s, in this area?
  • How long should the project be able to last?
  • Are any outside stakeholders responsible for providing content/information/materials/funding/etc.?
  • Who is responsible for maintaining the project?
  • What does the end-state of the project look like?


  • How should the hierarchy of items we hope to report as completed activities (or intended activities) be sorted?

here's an example that attempts to be exhaustive, indeed too exhaustive, on a scale from nothing to minimum that you could do and still consider the project a project, to unexpected success)

    • (a new academic area opens up discussing the major impact of this game)
    • (stuff in between)

12 best practices with general applicability could be suggested 11 Revisions were demonstratively better, revealing best-practices for a specific area 10Comparisons led to good suggestions for revision 9Items implemented were compared to possible implementations and each other 7Several things were implemented in a way that achieved some goals. 7Attempt to implement something was made that was well justified in terms of a concept/goal 7An attempt to implement with results that can be assessed was made 7Any attempt to implement made 6Choices were made about or between concepts in a way that can be demonstrated to align with goals 5several implementation concepts were discussed in terms of goals, etc 5Goals, skills, abilities, values etc were connected to an implementation concept 4associations were categorized in relation to each other (e.g. creative writing for games is sometimes an ability, sometimes a skill that can be developed, and always contingent in determining whether to pursue a project where the gameplay is story-based) 3associations were categorized into general areas (e.g. that's a goal, that's a method, a challenge, an ability, a value) 2associations were categorized as in or out of bounds (e.g. a goal, value, or skill that is definitely or probably not relevant, relevant under some circumstances,or always relevant) 1associations were made between project and ideas that may or may not be applicable to the project 0 nothing

  • Does the game work?
    • What doesn't work?
      • Is the problem technical, conceptual, or both?
    • Can the game be fixed or is this a future cautionary tale?
  • Who did the game work for (or not work for)?
    • How did the stakeholders react to the game?
  • Does the game advance our research goals?
  • Is this a successful research project?
  • Is this a successful game?
  • Are there ways to improve the game or the platform based on the results of this attempt?
  • Should the game be more intuitive or instructional?
    • How quickly will the target audience be able to learn the game?
  • If the project is repeatable, will we learn new things or benefit from running the game again?


  • Who and how do we want to give or receive feedback? (I think this needs to be broken down into several questions - SL)
    • How will we present findings to our stakeholders?
    • Who will we solicit feedback from?
      • What type of feedback is required? (qualitative? quantitative? game metrics?)
  • What would be the most efficient method - time and resource wise - of gathering assessment data?
  • Do the chosen assessment techniques require ethics clearance?
    • Do we need new ethics or can this fall under a previous project?
  • What tools will give us the most useful data?
  • What questions should the assessment tools pose?
    • Should there be multiple feedback tools available?

(I think this section needs to be much more detailed since this was the original focus of the framework - the assessment process) - SL

Misc. categories (fun, education, technology, etc...)

  • What are measurable variables that can be defined as '____'?
  • Can the game teach anything different than a textbook, class, or other resources? (Not all of our games are designed to be serious games - this is a better question for the assessment section for specific games, will not apply to all projects)
    • Can this game teach something that can be gained through other resources in a way that the targeted audience finds preferable? ( I think this is outside of the scope of our group, we are not really studying how best to teach a topic or curriculum, we are studying games.)
  • Is there technology involved? Or theory? (This question is very broad, I'm not sure what it means and how it is different from resources section - needs to be defined better or broken up into sub questions)
  • Do we need to advertise?
  • How will we launch the game and attract players?
    • Do we have a captive player group or is the game released in the wild?
  • Can we tweak the game during deployment? Is this a part of the plan?

Other things to put in

  • Intellectual Property: What sort of license does it use? Who owns the work? Who published it? What is the intellectual property?
  • Time and Money: How long did it take to make? How much did it cost to develop? What sort of organization developed it?
  • Comparison and Competition: What other games are similar? Who is the competition? How does it compare to other games?

Other Frameworks

External Links

has a long list of items - have a feeling that these are at different levels of importance so we should read through and sort.

Bits of advice that could be part of a framework content

  • flow in games (This is an awesome paper but please describe how it fits in with the design framework. I'm not sure why this is here)
Personal tools