CIRCA:The KEEP Project

From CIRCA

(Difference between revisions)
Jump to: navigation, search
m
Line 36: Line 36:
=== Virtual Machine ===
=== Virtual Machine ===
-
The main technology that the KEEP project uses is a virtual machine; a virtual machine is a type of software that runs programs instead of the hardware on a computer. Each of the emulations will represent a different iteration of retired operating system platforms. For example, one emulation would represent Windows x86, while another would represent BBC Micro, an operating system that runs on microcomputers developed in association with BBC television. By using a virtual machine to run each of the emulations the hardware of the computer in use will no longer interfere with the run requirements of a program. A different way to think of this is that older programs that no longer work on current computers, due to the updated drivers, will be able to be run through an emulator that mimics the original environment the program was developed and run in.  
+
The main technology that the KEEP project uses is a virtual machine; a virtual machine is a type of software that runs programs instead of the hardware on a computer (“Virtual Machine”). Each of the emulations will represent a different iteration of retired operating system platforms. For example, one emulation would represent Windows x86, while another would represent BBC Micro, an operating system that runs on microcomputers developed in association with BBC television. By using a virtual machine to run each of the emulations the hardware of the computer in use will no longer interfere with the run requirements of a program. A different way to think of this is that older programs that no longer work on current computers, due to the updated drivers, will be able to be run through an emulator that mimics the original environment the program was developed and run in.  
One issue with most current emulator systems is that they were only developed to work on one specific platform technology and have to be updated with every change. With the virtual machine that KEEP uses, the emulators do not have to be updated to continually work because they are separate from the technology restrictions and requirements of specific platforms. The use of the virtual machine to run the emulations is important to the KEEP project because it creates a more reliable environment for the emulations, and in extension the programs, to run. In today’s society, computer hardware is reconfigured every two years making older technology obsolete and every program that does not migrate (update) with the new technology is unable to run. Adobe Flash, for example, could still be run on older operating systems, such as Windows 8 through an emulation, with a previous version of the Flash software that can still be found on the internet, without needing to configure to current computer hardware that prevents it from playing.
One issue with most current emulator systems is that they were only developed to work on one specific platform technology and have to be updated with every change. With the virtual machine that KEEP uses, the emulators do not have to be updated to continually work because they are separate from the technology restrictions and requirements of specific platforms. The use of the virtual machine to run the emulations is important to the KEEP project because it creates a more reliable environment for the emulations, and in extension the programs, to run. In today’s society, computer hardware is reconfigured every two years making older technology obsolete and every program that does not migrate (update) with the new technology is unable to run. Adobe Flash, for example, could still be run on older operating systems, such as Windows 8 through an emulation, with a previous version of the Flash software that can still be found on the internet, without needing to configure to current computer hardware that prevents it from playing.
Line 107: Line 107:
=== The Internet Archive ===
=== The Internet Archive ===
-
One project that has some similarities to the KEEP project is The Internet Archive. The Internet Archive is an online database that works with thousands of partners to save digital resources, with a focus on books. The way the KEEP project and The Internet Archive are similar is they both have the same motive for creation, which is universal access to knowledge. How the two projects differ thou is through the way they are preserving. While The Internet Archive is fully web-based and uses an in-browser emulator, it still depends partially on migration. Migration is the constant updating of the software, or in this case, the emulator to run the digital objects. Every time a computer operating system or a web browser updates there is work that has to be done to maintain and sustain The Internet Archives emulator, with the added possibility of minor data loss. The KEEP project though uses a virtual machine to house all of the different emulators and digital objects so that they do not require any updates when technology updates occur and thus can be run as long as the virtual machine works without ever needing to touch the digital objects and risk any data lose.
+
One project that has some similarities to the KEEP project is The Internet Archive. The Internet Archive is an online database that works with thousands of partners to save digital resources, with a focus on books (“Flash Game Archive”). The way the KEEP project and The Internet Archive are similar is they both have the same motive for creation, which is universal access to knowledge. How the two projects differ thou is through the way they are preserving. While The Internet Archive is fully web-based and uses an in-browser emulator, it still depends partially on migration (“Flash Game Archive”). Migration is the constant updating of the software, or in this case, the emulator to run the digital objects. Every time a computer operating system or a web browser updates there is work that has to be done to maintain and sustain The Internet Archives emulator, with the added possibility of minor data loss (“Flash Game Archive”). The KEEP project though uses a virtual machine to house all of the different emulators and digital objects so that they do not require any updates when technology updates occur and thus can be run as long as the virtual machine works without ever needing to touch the digital objects and risk any data lose.
=== MAME ===
=== MAME ===
   
   
-
The other project that is similar to the KEEP project is MAME, which formerly meant Multi Arcade Machine Emulator. There are more similarities between KEEP and MAME than there are between KEEP and The Internet Archive, but once again the main similarity is the purpose of universal access to knowledge. In MAME’s case, the main focus is on preserving electronic entertainment systems, especially if they are considered vintage. One other way that the KEEP project and MAME are similar is the way they use a downloaded emulator to run the software.
+
The other project that is similar to the KEEP project is MAME, which formerly meant Multi Arcade Machine Emulator (“MAME Documentation”). There are more similarities between KEEP and MAME than there are between KEEP and The Internet Archive, but once again the main similarity is the purpose of universal access to knowledge. In MAME’s case, the main focus is on preserving electronic entertainment systems, especially if they are considered vintage. One other way that the KEEP project and MAME are similar is the way they use a downloaded emulator to run the software (“MAME Documentation”).
-
From there though there are many differences between MAME and the KEEP project. First off MAME uses C++ as its coding language, and as aforementioned, the KEEP project developers attempted to originally use C++, but it failed and crashed when different emulators were used. This lead to KEEP having to use the more sophisticated language of JAVA. From this it would seem if MAME tried to increase its complexity, to be on the same level as KEEP, it to would have to change the coding language.
+
From there though there are many differences between MAME and the KEEP project. First off MAME uses C++ as its coding language (“MAME Documentation”), and as aforementioned, the KEEP project developers attempted to originally use C++, but it failed and crashed when different emulators were used. This lead to KEEP having to use the more sophisticated language of JAVA. From this it would seem if MAME tried to increase its complexity, to be on the same level as KEEP, it to would have to change the coding language.
-
The second difference is that MAME is only an emulator and contains no internal database of preserved digital objects. To be able to use MAME “the emulator requires images of the original ROMs, CDs, hard disks or other media from the machines, which must be provided by the user. No portions of the original game code are included in the executable.” This means that if users want to access files they must already own a copy, which is not always plausible. With the KEEP project though there is an internal database that holds uploaded digital objects, so users can always have access to these uploaded resources and do not require original copies.
+
The second difference is that MAME is only an emulator and contains no internal database of preserved digital objects. To be able to use MAME “the emulator requires images of the original ROMs, CDs, hard disks or other media from the machines, which must be provided by the user. No portions of the original game code are included in the executable (“MAME Documentation”).” This means that if users want to access files they must already own a copy, which is not always plausible. With the KEEP project though there is an internal database that holds uploaded digital objects, so users can always have access to these uploaded resources and do not require original copies (“MAME Documentation”).
-
Finally, the last difference is similar to the one stated for The Internet Archive, MAME is only available on four operating systems that are fairly current and will need updates as technology progresses. With KEEP using a virtual machine, this problem is solved by just uploading another emulator not having to update a single emulator each time technology changes.
+
Finally, the last difference is similar to the one stated for The Internet Archive, MAME is only available on four operating systems that are fairly current and will need updates as technology progresses (“MAME Documentation”). With KEEP using a virtual machine, this problem is solved by just uploading another emulator not having to update a single emulator each time technology changes.
== References ==
== References ==

Revision as of 20:14, 16 October 2021

Contents

Project Summary

What is KEEP?

KEEP stands for Keeping Emulation Environments Portable and was a project created to facilitate continued access to cultural heritage by developing flexible tools to help access, manipulate and store digital objects. These digital objects would include accurate renderings of static and dynamic text, images, videos, sounds, websites, databases, games and other digital media. The way the digital objects would be accessed, manipulated, and stored would be done through an emulation that mimics the original hardware environment that the digital object was created on.

In addition to the emulation framework being developed, the KEEP project team would also design and develop a Virtual Machine that the emulation framework could be run on. This is so the emulations would operate independently from the computer’s actual software and hardware requirements.

The KEEP project is an important advancement within the field of technology preservation because the most common way to currently preserve media is through migration as technology is rapidly updating and growing. Migration is when a digital object is continually updated to work with current hardware and software developments when the digital object’s original environment becomes obsolete. This is an issue though because every time a digital object is migrated, pieces of information and metadata are lost because they are not compatible with the new environment. With the use of emulation and the virtual machine, accessing digital objects and materials is mediated by making the computer compatible with the materials, and not the other way around. This will allow digital objects to be accessed not in just the present time but also in the future.

Timeline

The KEEP project was a three-year project run between 2009 and 2012. For the first two years, the project team focused on researching the available tools and technology including transfer tools and emulation software. At the same time, with help from the National Libraries of France, Germany, and the Netherlands, the project team also did a user requirements analysis to help design a flexible, and user-friendly emulation service, media carrier, and media transfer tool.

During this research period, the project term also ended up doing an investigation into the potential legal issues that would arise from uploading information from the original data carrier, which later turned into a study. This study created a set of standards about the metadata needed for each of the transferred files and how the legal issues should be approached, using information from European laws.

Funding

Project KEEP was co-funded by the European Union's Seventh Framework Programme for research and technological development (FP7).

People Involved

During the development of the KEEP project, a consortium of organizations were involved internationally to represent a wide variety of stakeholders across Europe. This consortium included:

  • The national libraries of France, Germany and the Netherlands
  • Tessella, a company that provides services for sorting and preserving data in the United Kingdom and the Netherlands
  • Joguin SAS in France
  • Software developers specializing in preservation
  • Project consultants from CrossCzech in the Czech Republic
  • The Computerspielemuseum in Germany
  • The University of Portsmouth in the United Kingdom
  • The European Game Developers Federation

Technologies

Virtual Machine

The main technology that the KEEP project uses is a virtual machine; a virtual machine is a type of software that runs programs instead of the hardware on a computer (“Virtual Machine”). Each of the emulations will represent a different iteration of retired operating system platforms. For example, one emulation would represent Windows x86, while another would represent BBC Micro, an operating system that runs on microcomputers developed in association with BBC television. By using a virtual machine to run each of the emulations the hardware of the computer in use will no longer interfere with the run requirements of a program. A different way to think of this is that older programs that no longer work on current computers, due to the updated drivers, will be able to be run through an emulator that mimics the original environment the program was developed and run in.

One issue with most current emulator systems is that they were only developed to work on one specific platform technology and have to be updated with every change. With the virtual machine that KEEP uses, the emulators do not have to be updated to continually work because they are separate from the technology restrictions and requirements of specific platforms. The use of the virtual machine to run the emulations is important to the KEEP project because it creates a more reliable environment for the emulations, and in extension the programs, to run. In today’s society, computer hardware is reconfigured every two years making older technology obsolete and every program that does not migrate (update) with the new technology is unable to run. Adobe Flash, for example, could still be run on older operating systems, such as Windows 8 through an emulation, with a previous version of the Flash software that can still be found on the internet, without needing to configure to current computer hardware that prevents it from playing.

The virtual machine also allows a choice between multiple different emulators that each have different features. This will allow the computer to choose which emulator works best for the user in terms of ease of use and the digital object in terms of rendering. One way this is shown is by having different emulators for basic users, who just want the common functionality of using emulation to show digital files and advanced users with a more comprehensive interface that also allows access to all development files.

One other way the virtual machine is valuable to the KEEP project is because the virtual machine takes care of the technical requirements concerning digital objects that are uploaded. Instead of the person who is uploading the digital object having to either update the object to be usable or upload multiple different versions of the same object, the virtual machine will just help choose an emulator that works with the existing program. This is especially important for the three main libraries that provided the KEEP project digital objects to use because the librarians are not tasked with dealing with the technological requirements or restrictions.

Coding Language

During the design of the KEEP project, the coding language used for development created a significant learning opportunity. Originally the development team had the choice between two different languages to code with, Java and C/C++. At first glance, these coding languages were even in terms of difficulty with any difficult parts for development being equally hard in both languages. Eventually, the development team decided to run KEEP through C/C++ because it worked better with KEEP’s existing coding libraries for the visualization aspect of the emulations. When the first emulation test was run through, C/C++ crashed because it was not powerful enough to run the programs with the required features. Thus the development team had to reconfigure the scope of the project and develop KEEP using Java. Java did not work with the existing coding libraries the team had already developed so all coding has to be restarted. Luckily as the project progressed Java became easier to use as more technology and libraries were added and overall was a better fit for the KEEP project due to its flexibility in terms of features.

Templates

With KEEP wanting to use multiple emulators within the project, a lot of coding and development needed to be done. For each emulator, the team needed to write a script that was specific to each emulator and its function. Overall this takes a lot of time and effort, but the process was streamlined using templates. Originally the developer team wanted to use a process called Java classes for writing each of the scripts to run each of the different emulators. This would mean writing custom code for each emulator configuration and each version, as well as another specific script for importing the emulator into the framework. It was decided that this was too cumbersome for the team because writing all of these custom scripts from scratch would take too long and would not allow flexibility within the code in case something was needed to be changed. Instead, the developer team decided to use a template for writing each of these scripts because it cut down on time and effort. How the template is used is that there is one master template that each of the scripts will be based on for each emulator, that contains fillable fields. A developer will then write a custom script that will provide the information to fill in each field for each emulator. This custom script is much smaller and quicker to write because the only information needed is specifics about the emulators, not general run information. The custom code is then run which will fill in the fields on the template and export a completed copy thus creating a specific script to use for each emulator without a developer having to code everything from scratch. One way to think of this is when you are filling in your shipping information on an order from Amazon. The template is the form that you are filling out that asks your name, and address, while the code is represented by your specific information, especially if you have your information auto-complete. This way you don't have to type your address in every time and you save time at checkout.

Methodologies

The KEEP project uses two different databases to run; the first is the emulation framework and archive, and the second is the internal database. These databases work in tandem to accurately display the digital archives (text, sound, videos, games, etc.) in an emulator.

Emulation Framework and Archive

The emulation framework and archive is the database that contains all the different emulators and the metadata that allows them to run, including the certificates that state they run successfully. These emulators are the product of the different iterations of the template aforementioned, each representing a different operating system. With each of the emulations in the framework there is archived metadata that tells the emulator what it can run, how to run it, and the different services the emulator can provide to the user. These services include but are not limited to:

  • Take a blank input and returns a list of emulators,
  • Take a blank input and returns a list of supported hardware
  • Take an identifier of a hardware type as an input and returns a list of emulators
  • Take an identifier of an emulator as an input and returns a binary file containing the emulator and metadata
  • Take a blank input and returns a list of all languages

Internal Database

The internal databases contain all of the uploaded digital objects and their metadata. Most of the current database is populated by the materials from the three national libraries of France, Germany and the Netherlands. With each of the digital objects, there is also a list of corresponding metadata that partially matches with the metadata each of the libraries kept about each object, while also adding additional metadata to inform the emulation framework what emulations are needed. One thing that is unique about each of the objects in the internal database of the KEEP project is that all objects are required to work on two or more emulators. This will allows the computer to match the digital object to the best emulator for rendering. This unique feature though would not be possible without the virtual machine allowing the emulators to run independently from the specific technology of the computer in use.

Combining the Two Databases

Below are two similar diagrams of how the emulation framework and archive function when combined with the internal database. First, a digital object is chosen and identified by the user from the internal database, then a list of possible emulations is chosen from the framework with each one being tested to ensure it can successfully run. From there the computer chooses the emulator that is best to use, and the chosen emulator is configured. Finally, the digital object that the user wants is rendered in the emulator allowing the user to interact. caption caption

The Success of the Project

The KEEP project became a major success at its initial release in February of 2012 when the software was released to the public. At the time, version 1.0.0 contained:

  • 5 platforms: x86, C64, Amiga, BBC Micro, Amstrad
  • 6 emulators included: Dioscuri, Qemu, VICE, UAE, BeebEm, JavaCPC
  • 22 file formats: PDF, TXT, XML, JPG, TIFF, PNG, BMP, Quark, ARJ, EXE, disk/tape images and more

Development continued into April of 2012 and is the most current version of the software. This release is version 2.1.0 and contains:

  • 6 platforms: x86, C64, Amiga, BBC Micro, Amstrad, Thomson TO7
  • 7 emulators: Dioscuri, Qemu, VICE, UAE, BeebEm, JavaCPC, Thomson
  • 30+ file formats: PDF, TXT, XML, JPG, TIFF, PNG, BMP, Quark, ARJ, EXE, disk/tape images and more

This version of the software is available to download at http://emuframework.sourceforge.net/ and was a finalist for the prestigious Digital Preservation Award 2012 in Research & Innovation.

While not confirmed, there does seem to be some issues with the sustainability of the project due to no longer being able to access the secondary website that hosted the details of the project. If you would like to learn more about the current status of the KEEP project there though, there is a news mailing list that updates approximately twice a year.

At this point, while the project no longer seems to be getting public updates, Version 2.1.0 is still available with the preserved data initially used during the preservation project.

Example of How the Project Could Be Used?

At this moment the KEEP project still has a way to go in populating the archive with digital objects because it is currently dependent on existing archived materials. One of the major hopes of the project is that evaluating and enhancing the existing metadata will allow for maximum compatibility with games. An example of this is the preservation of Adobe Flash games that have become obsolete with the uninstallation of Adobe Flash across technology. While the KEEP project has not specifically stated that it is preserving Flash games, the project already contains some of the necessary emulation environments needed for Flash, because KEEP is already emulating older operating systems. Older versions of Flash, that correspond to these operating systems, can then be uploaded to the internal database from the free downloads on the Adobe site. From there, Adobe Flash games could then be archived to the internal database found within the virtual machine through multiple means. The first is the original creator could upload the files to the archive themselves, since a version of the project is available for public use, and make the game publicly accessible. The second way is a player could upload the game and either cite the original creators of the game or just upload it for personal and private use. The final way, which is probably the best way to do this, is for the KEEP project to work together with an existing archive of Flash games, such as the Flash Game Archive. The Flash Game Archive is an archive of thousands of Flash Games that have been built to be supported by a single emulator for a single platform. The KEEP project could easily upload the emulator and all of the games to the internal database, while also allowing further access through the KEEP project’s own emulators to expand the Flash Game Archives’ emulator to work on multiple platforms. Together the KEEP project and the Flash Game archive could preserve Flash games for generations because the archive has the digital objects and the KEEP project has the emulators to create a sustainable multiplatform environment to expand upon the archive’s existing emulator.

Comparison of the KEEP Project to Similar Projects

The Internet Archive

One project that has some similarities to the KEEP project is The Internet Archive. The Internet Archive is an online database that works with thousands of partners to save digital resources, with a focus on books (“Flash Game Archive”). The way the KEEP project and The Internet Archive are similar is they both have the same motive for creation, which is universal access to knowledge. How the two projects differ thou is through the way they are preserving. While The Internet Archive is fully web-based and uses an in-browser emulator, it still depends partially on migration (“Flash Game Archive”). Migration is the constant updating of the software, or in this case, the emulator to run the digital objects. Every time a computer operating system or a web browser updates there is work that has to be done to maintain and sustain The Internet Archives emulator, with the added possibility of minor data loss (“Flash Game Archive”). The KEEP project though uses a virtual machine to house all of the different emulators and digital objects so that they do not require any updates when technology updates occur and thus can be run as long as the virtual machine works without ever needing to touch the digital objects and risk any data lose.

MAME

The other project that is similar to the KEEP project is MAME, which formerly meant Multi Arcade Machine Emulator (“MAME Documentation”). There are more similarities between KEEP and MAME than there are between KEEP and The Internet Archive, but once again the main similarity is the purpose of universal access to knowledge. In MAME’s case, the main focus is on preserving electronic entertainment systems, especially if they are considered vintage. One other way that the KEEP project and MAME are similar is the way they use a downloaded emulator to run the software (“MAME Documentation”).

From there though there are many differences between MAME and the KEEP project. First off MAME uses C++ as its coding language (“MAME Documentation”), and as aforementioned, the KEEP project developers attempted to originally use C++, but it failed and crashed when different emulators were used. This lead to KEEP having to use the more sophisticated language of JAVA. From this it would seem if MAME tried to increase its complexity, to be on the same level as KEEP, it to would have to change the coding language. The second difference is that MAME is only an emulator and contains no internal database of preserved digital objects. To be able to use MAME “the emulator requires images of the original ROMs, CDs, hard disks or other media from the machines, which must be provided by the user. No portions of the original game code are included in the executable (“MAME Documentation”).” This means that if users want to access files they must already own a copy, which is not always plausible. With the KEEP project though there is an internal database that holds uploaded digital objects, so users can always have access to these uploaded resources and do not require original copies (“MAME Documentation”). Finally, the last difference is similar to the one stated for The Internet Archive, MAME is only available on four operating systems that are fairly current and will need updates as technology progresses (“MAME Documentation”). With KEEP using a virtual machine, this problem is solved by just uploading another emulator not having to update a single emulator each time technology changes.

References

  1. “About the Internet Archive.” Internet Archive, https://archive.org. Accessed October 16, 2021.
  2. “Flash Game Archive: Preserving Flash Gaming.” Flash Game Archive, http://www.flashgamearchive.com/. Accessed October 16, 2021
  3. Joguin, Vincent. “Keeping Emulation Environments Portable (KEEP).” epractice.eu: eGovernment, Aug. 25, 2011. Joinup, https://joinup.ec.europa.eu/collection/egovernment/document/keeping-emulation-environments-portable-keep
  4. KEEP. “Architectural Design Document Emulation Framework: Release 2.0.0 (February 2012).” SorceForge, Feb. 2012, http://emuframework.sourceforge.net/docs/Architectural-Design-Document_2.0.pdf .
  5. Lohman, Bram, et al. “KEEP: Emulation Framework [EF].” SorceForge, 2012, http://emuframework.sourceforge.net/
  6. “MAME Documentation.” MAME, https://docs.mamedev.org. Accessed October 16, 2021.
  7. Olson, Michael G. “KEEP - Keeping Emulation Environments Portable.” Digital Library Blog, Dec. 12, 2011, Stanford Libraries, https://library.stanford.edu/blogs/digital-library-blog/2011/12/keep-keeping-emulation-environments-portable. Accessed Sept. 25, 2021.
  8. Pinchbeck, Dan, et al. “Emulation as a Strategy for the Preservation of Games: The KEEP Project.” DiGRA '09 - Proceedings of the 2009 DiGRA International Conference: Breaking New Ground: Innovation in Games, Play, Practice and Theory, vol. 5, Sept. 2009, http://www.digra.org/digital-library/publications/emulation-as-a-strategy-for-the-preservation-of-games-the-keep-project/
  9. “Virtual Machine.” vm ware, https://www.vmware.com/topics/glossary/content/virtual-machine. Accessed Sept. 25, 2021.
Personal tools