CIRCA:Assessment Framework
From CIRCA
Contents |
(Tentative) Assessment Framework Questions
Use these question categories in the process of answering your own questions relevant to a specific project. Fill in category sub-questions as best you can, add your own or skip questions as needed.
Stakeholders
- Who are the legacy stakeholders for this project or this type of project?
- sponsors
- group members
- community partners
- audience demographics
- Who are the existing stakeholders for this project?
- administrative
- ethical / legal
- research
- audience / customer
- others
- Who are new stakeholders, or what are recent changes in makeup/orientation/capacity?
- What if any changes to this category can be expected during the course of this project?
- Who are potential stakeholders
- Is there a market or test group available, already existing or in demand?
- What are potential inclusion or exclusion factors (Does working with these stakeholders require REB ethics clearance?)
- Where can more information be found about a particular stakeholder, how can this information be used?
- What is the criteria for acquiring and updating information about stakeholders?
Expectations
- What is the purpose of the project?
- What is the requirement of the project at a minimum?
- What ought to be the shared ambition of this project? (each stakeholder should describe this)
- What is the agreed upon shared ambition for this project?
- What deliverables are due to each stakeholder (what are they getting out of the project)?
- What are the stakeholders' responsibilities to the group?
- To what extent is the project contract to be formalized or kept informal?
- What is the project timeline?
- What are our deliverables' timelines
- How will we conclude this project (what will the end-state look like?)
Resources
- What are the deliverables?
- What is the budget?
- Who is responsible for all aspects of the budget?
- What technology or tools are required to build, maintain, and play the game?
- What is the length of time it would take to build and run the game?
- Is there a deadline? Or can this project be delayed?
- Is there previous work, either ours or another group’s, in this area?
- How long should the project be able to last?
- Are any outside stakeholders responsible for providing content/information/materials/funding/etc.?
- Who is responsible for maintaining the project?
- What does the end-state of the project look like?
Execution
- Does the game work?
- What doesn't work?
- Is the problem technical, conceptual, or both?
- Can the game be fixed or is this a future cautionary tale?
- What doesn't work?
- Who did the game work for (or not work for)?
- How did the stakeholders react to the game?
- Does the game advance our research goals?
- Is this a successful research project?
- Is this a successful game?
- Are there ways to improve the game or the platform based on the results of this attempt?
- Should the game be more intuitive or instructional?
- How quickly will the target audience be able to learn the game?
- If the project is repeatable, will we learn new things or benefit from running the game again?
Feedback
- Who and how do we want to give or receive feedback? (I think this needs to be broken down into several questions - SL)
- How will we present findings to our stakeholders?
- Who will we solicit feedback from?
- What type of feedback is required? (qualitative? quantitative? game metrics?)
- What would be the most efficient method - time and resource wise - of gathering assessment data?
- Do the chosen assessment techniques require ethics clearance?
- Do we need new ethics or can this fall under a previous project?
- What tools will give us the most useful data?
- What questions should the assessment tools pose?
- Should there be multiple feedback tools available?
(I think this section needs to be much more detailed since this was the original focus of the framework - the assessment process) - SL
Assesment Schemas
Since everything is going into a report - it might be useful to try to order the types of activities you might end up having to report into some hierarchies.
For example, I am trying to make a scale, between nothing to report on the one extreme, and unexpected success (to the extent that reporting has a life outside the project) on the other.
- 100 (a new academic area opens up discussing the major impact of this game)
- (stuff in between)
- 12 best practices with general applicability could be suggested from the project
- 11 Revisions were put into practice and were demonstratively better, revealing best-practices for a specific area
- 10 Comparisons led to good suggestions for revision
- 9 Items implemented were evaluated in some meaningful comparison
- 7 Several things were implemented in a way that achieved goals.
- 7 Attempt to implement something was made that was well justified in terms of a concept/goal
- 7 An attempt to implement with results that have some useful diagnostic value
- 7 Any attempt to implement something was made
- 6 Choices were made about or between concepts in a way that can be demonstrated to align with goals
- 5 several implementation concepts were discussed in terms of goals, etc
- 5 Goals, skills, abilities, values etc were connected to an implementation concept
- 4 associations were categorized in relation to each other (e.g. creative writing for games is sometimes an ability, sometimes a skill that can be developed, and always contingent in determining whether to pursue a project where the gameplay is story-based)
- 3 associations were categorized into general areas (e.g. that's a goal, that's a method, a challenge, an ability, a value)
- 2 associations were categorized as in or out of bounds (e.g. a goal, value, or skill that is definitely or probably not relevant, relevant under some circumstances,or always relevant)
- 1 associations were made between project and ideas that may or may not be applicable to the project
- 0 nothing
- Thematic Assessments
- how well and in what ways did this project address the topic of gender?
- What was the educational value of the project?
Misc. categories (fun, education, technology, etc...)
- What are measurable variables that can be defined as '____'?
- Can the game teach anything different than a textbook, class, or other resources? (Not all of our games are designed to be serious games - this is a better question for the assessment section for specific games, will not apply to all projects)
- Can this game teach something that can be gained through other resources in a way that the targeted audience finds preferable? ( I think this is outside of the scope of our group, we are not really studying how best to teach a topic or curriculum, we are studying games.)
- Is there technology involved? Or theory? (This question is very broad, I'm not sure what it means and how it is different from resources section - needs to be defined better or broken up into sub questions)
- Do we need to advertise?
- How will we launch the game and attract players?
- Do we have a captive player group or is the game released in the wild?
- Can we tweak the game during deployment? Is this a part of the plan?
Other things to put in
- Intellectual Property: What sort of license does it use? Who owns the work? Who published it? What is the intellectual property?
- Time and Money: How long did it take to make? How much did it cost to develop? What sort of organization developed it?
- Comparison and Competition: What other games are similar? Who is the competition? How does it compare to other games?
Other Frameworks
External Links
has a long list of items - have a feeling that these are at different levels of importance so we should read through and sort.
Bits of advice that could be part of a framework content
- assessment using pros and cons, example is pros and cons of using a wiki use for collaborative work (I think this document is supposed to be less a 'how to' guide and more of a check list/framework for doing the theoretical and technical research and implementation. Perhaps we can start a page of solid design 'how to' resources elsewhere on the wiki)
- flow in games (This is an awesome paper but please describe how it fits in with the design framework. I'm not sure why this is here)