CIRCA:Assessment Framework

From CIRCA

(Difference between revisions)
Jump to: navigation, search
(Execution)
(Execution)
Line 67: Line 67:
-
*How should the hierarchy of items we hope to report as completed activities (or intended activities) be sorted?
+
*Can the items we hope to report as completed activities (or intended activities) be sorted in a particular hierachy?
 +
 
 +
So, make a scale between nothing to unexpected success
-
here's an example that attempts to be exhaustive, indeed too exhaustive, on a scale from nothing to minimum that you could do and still consider the project a project, to unexpected success)
 
** (a new academic area opens up discussing the major impact of this game)
** (a new academic area opens up discussing the major impact of this game)
Line 77: Line 78:
*'''12''' best practices with general applicability could be suggested
*'''12''' best practices with general applicability could be suggested
*'''11''' Revisions were demonstratively better, revealing best-practices for a specific area
*'''11''' Revisions were demonstratively better, revealing best-practices for a specific area
-
*'''10'''Comparisons led to good suggestions for revision
+
*'''10''' Comparisons led to good suggestions for revision
-
*'''9'''Items implemented were compared to possible implementations and each other
+
*'''9''' Items implemented were compared to possible implementations and each other
-
*'''7'''Several things were implemented in a way that achieved some goals.
+
*'''7''' Several things were implemented in a way that achieved some goals.
-
*'''7'''Attempt to implement something was made that was well justified in terms of a concept/goal
+
*'''7''' Attempt to implement something was made that was well justified in terms of a concept/goal
-
*'''7'''An attempt to implement with results that can be assessed was made
+
*'''7''' An attempt to implement with results that can be assessed was made
-
*'''7'''Any attempt to implement somethingwas made
+
*'''7''' Any attempt to implement somethingwas made
-
*'''6'''Choices were made about or between concepts in a way that can be demonstrated to align with goals
+
*'''6''' Choices were made about or between concepts in a way that can be demonstrated to align with goals
-
*'''5'''several implementation concepts were discussed in terms of goals, etc
+
*'''5''' several implementation concepts were discussed in terms of goals, etc
-
*'''5'''Goals, skills, abilities, values etc were connected to an implementation concept
+
*'''5''' Goals, skills, abilities, values etc were connected to an implementation concept
-
*'''4'''associations were categorized in relation to each other (e.g. creative writing for games is sometimes an ability, sometimes a skill that can be developed, and always contingent in determining whether to pursue a project where the gameplay is story-based)
+
*'''4''' associations were categorized in relation to each other (e.g. creative writing for games is sometimes an ability, sometimes a skill that can be developed, and always contingent in determining whether to pursue a project where the gameplay is story-based)
-
*'''3'''associations were categorized into general areas (e.g. that's a goal, that's a method, a challenge, an ability, a value)
+
*'''3''' associations were categorized into general areas (e.g. that's a goal, that's a method, a challenge, an ability, a value)
-
*'''2'''associations were categorized as in or out of bounds (e.g. a goal, value, or skill that is definitely or probably not relevant, relevant under some circumstances,or always relevant)
+
*'''2''' associations were categorized as in or out of bounds (e.g. a goal, value, or skill that is definitely or probably not relevant, relevant under some circumstances,or always relevant)
-
*'''1'''associations were made between project and ideas that may or may not be applicable to the project
+
*'''1''' associations were made between project and ideas that may or may not be applicable to the project
*'''0''' nothing
*'''0''' nothing

Revision as of 20:42, 19 October 2012

Contents

(Tentative) Assessment Framework Questions

Use these question categories in the process of answering your own questions relevant to a specific project. Fill in category sub-questions as best you can, add your own or skip questions as needed.

Stakeholders

  • Who are the legacy stakeholders for this project or this type of project?
    • sponsors
    • group members
    • community partners
    • audience demographics
  • Who are the existing stakeholders for this project?
    • administrative / legal
    • research
    • audience / customer
    • others
  • Who are new stakeholders, or what are recent changes in makeup/orientation/capacity?
  • What if any changes to this category can be expected during the course of this project?
  • Who are potential stakeholders
    • Is there a market or test group available, already existing or in demand?
    • What are potential inclusion or exclusion factors (Does working with these stakeholders require REB ethics clearance?)
  • Where can more information be found about a particular stakeholder, how can this information be used?
  • What is the criteria for acquiring and updating information about stakeholders?

Expectations

  • What is the purpose of the project?
  • What is the requirement of the project at a minimum?
  • What ought to be the shared ambition of this project? (each stakeholder should describe this)
    • What is the agreed upon shared ambition for this project?
  • What deliverables are due to each stakeholder (what are they getting out of the project)?
  • What are the stakeholders' responsibilities to the group?
  • To what extent is the project contract to be formalized or kept informal?
  • What is the project timeline?
    • What are our deliverables' timelines
  • How will we conclude this project (what will the end-state look like?)

Resources

  • What are the deliverables?
  • What is the budget?
    • Who is responsible for all aspects of the budget?
  • What technology or tools are required to build, maintain, and play the game?
  • What is the length of time it would take to build and run the game?
    • Is there a deadline? Or can this project be delayed?
  • Is there previous work, either ours or another group’s, in this area?
  • How long should the project be able to last?
  • Are any outside stakeholders responsible for providing content/information/materials/funding/etc.?
  • Who is responsible for maintaining the project?
  • What does the end-state of the project look like?

Execution

  • Can the items we hope to report as completed activities (or intended activities) be sorted in a particular hierachy?

So, make a scale between nothing to unexpected success


    • (a new academic area opens up discussing the major impact of this game)
    • (stuff in between)
  • 12 best practices with general applicability could be suggested
  • 11 Revisions were demonstratively better, revealing best-practices for a specific area
  • 10 Comparisons led to good suggestions for revision
  • 9 Items implemented were compared to possible implementations and each other
  • 7 Several things were implemented in a way that achieved some goals.
  • 7 Attempt to implement something was made that was well justified in terms of a concept/goal
  • 7 An attempt to implement with results that can be assessed was made
  • 7 Any attempt to implement somethingwas made
  • 6 Choices were made about or between concepts in a way that can be demonstrated to align with goals
  • 5 several implementation concepts were discussed in terms of goals, etc
  • 5 Goals, skills, abilities, values etc were connected to an implementation concept
  • 4 associations were categorized in relation to each other (e.g. creative writing for games is sometimes an ability, sometimes a skill that can be developed, and always contingent in determining whether to pursue a project where the gameplay is story-based)
  • 3 associations were categorized into general areas (e.g. that's a goal, that's a method, a challenge, an ability, a value)
  • 2 associations were categorized as in or out of bounds (e.g. a goal, value, or skill that is definitely or probably not relevant, relevant under some circumstances,or always relevant)
  • 1 associations were made between project and ideas that may or may not be applicable to the project
  • 0 nothing


  • Does the game work?
    • What doesn't work?
      • Is the problem technical, conceptual, or both?
    • Can the game be fixed or is this a future cautionary tale?
  • Who did the game work for (or not work for)?
    • How did the stakeholders react to the game?
  • Does the game advance our research goals?
  • Is this a successful research project?
  • Is this a successful game?
  • Are there ways to improve the game or the platform based on the results of this attempt?
  • Should the game be more intuitive or instructional?
    • How quickly will the target audience be able to learn the game?
  • If the project is repeatable, will we learn new things or benefit from running the game again?

Feedback

  • Who and how do we want to give or receive feedback? (I think this needs to be broken down into several questions - SL)
    • How will we present findings to our stakeholders?
    • Who will we solicit feedback from?
      • What type of feedback is required? (qualitative? quantitative? game metrics?)
  • What would be the most efficient method - time and resource wise - of gathering assessment data?
  • Do the chosen assessment techniques require ethics clearance?
    • Do we need new ethics or can this fall under a previous project?
  • What tools will give us the most useful data?
  • What questions should the assessment tools pose?
    • Should there be multiple feedback tools available?


(I think this section needs to be much more detailed since this was the original focus of the framework - the assessment process) - SL

Misc. categories (fun, education, technology, etc...)

  • What are measurable variables that can be defined as '____'?
  • Can the game teach anything different than a textbook, class, or other resources? (Not all of our games are designed to be serious games - this is a better question for the assessment section for specific games, will not apply to all projects)
    • Can this game teach something that can be gained through other resources in a way that the targeted audience finds preferable? ( I think this is outside of the scope of our group, we are not really studying how best to teach a topic or curriculum, we are studying games.)
  • Is there technology involved? Or theory? (This question is very broad, I'm not sure what it means and how it is different from resources section - needs to be defined better or broken up into sub questions)
  • Do we need to advertise?
  • How will we launch the game and attract players?
    • Do we have a captive player group or is the game released in the wild?
  • Can we tweak the game during deployment? Is this a part of the plan?

Other things to put in

  • Intellectual Property: What sort of license does it use? Who owns the work? Who published it? What is the intellectual property?
  • Time and Money: How long did it take to make? How much did it cost to develop? What sort of organization developed it?
  • Comparison and Competition: What other games are similar? Who is the competition? How does it compare to other games?

Other Frameworks

External Links

has a long list of items - have a feeling that these are at different levels of importance so we should read through and sort.


Bits of advice that could be part of a framework content

  • flow in games (This is an awesome paper but please describe how it fits in with the design framework. I'm not sure why this is here)
Personal tools