Performance Tracing Report Assistant
The Performance Tracing Report Assistant (or PeTRA) is a real-time reporting tool for the Information Trails assessment framework (Loh, 2007, 2012). The purpose of PeTRA is to visualize players’ gameplay data (i.e., what their actions and behaviors are within the virtual environments) for communication. After tracing user-generated data in situ the virtual environment using Information Trails, we subjected the data for machine/statistical learning (exploratory study).
Loh and Sheng (2013, 2015) have examined the problem and devised several new metrics to address this problem. The most basic one, Similarity Index measures a learners’ performance against that of an established expert based on the navigational path chosen. The Maximum Similarity Index (MSI), on the other hand, measure the performance indices of novices against a group of experts, particularly when the expertise should not be averaged. (After all, averaged expertise is no expert at all.) The issue is of paramount importance because, in a situation such as medical diagnosis, surgeons may have opposing opinions on how to treat a condition. Military decision-making is another area where averaging may not make sense or will result in meaningless actions. A reporting tool can then be used to visualize the learners’ path graphically for communications with stakeholders (e.g., trainers, learners, and administrators) who are likely to have little statistical training. Since we already know the order of learning is important, we also need to know the order learners go about their learning. Information Trails can be used to trace the navigational path of learners within virtual environments to provide us with the data needed for analysis. Thus far, there are very few metrics or methods suitable for this kind of data analysis because no one has thought of using Virtual Environment for performance assessment. We need new metrics and methods for performance assessment in training with virtual environments.
In the image shown above, the yellow lines on the game map indicate movements and the dots depict game events. Since actions are a product of decision, the players’ actions in the game also infer his/her decisions. The learning events are tracked and translated into behavioral data, which can then be used for ad hoc (real-time, formative) or post hoc (after-action, summative, debriefing) reports for peer- or self-evaluation.
History of PeTRA
| Timeline | PeTRA Development |
| 11/2015 | PeTRA is now fully powered by R and RStudio Server. |
| 10/2015 | Unity3D Maze — first public game with Information Trails. Open for data collection. |
| 08/2015 | PeTRA now accepts data from Unity3D games. |
| 01/2015 | The Lost World — first Unity3D game with Information Trails. |
| 09/2014 | PeTRA being upgraded for Unity3D-driven assessment. |
| 11/2010 – 05/2014 | On-going game-based learning assessment research projects |
| 08/2010 | Information Trails tracer was renamed into “Performance Tracer Reporting Assistant” (PeTRA). |
| 05/2010 | Information Trails (version 2) was completed (developed using Adobe Flex). |
| 10/2009 – 07/2010 | Setup of the Virtual Environment Lab (V-LAB), funded through DURIP by the ARO. |
| 08/2009 | Porting game modding platform to Neverwinter Nights 2 |
| 01/2009 | Porting of Information Trails from a JAVA application (version 1) to Adobe Flex (version 2). |
| 07/2008 | We applied and received a provisional patent status for the Information Trails. |
| 05/2008 | The prototype of Information Trails (version 0.9) was completed, but we ran into problems with JAVA. |
| 12/2007 | We created our first serious game, Saving Adryanee, which won a Finalist Entry at the 2nd Serious Games Showcase & Challenge (sponsored by I/ITSEC). |
| 07/2007 | Dr. Loh received a $26,000 Seed Grant to create the first prototype of the assessment tool. |
| 10/2006 | The original concept of Information Trails was conceptualized and presented at the annual conference of AECT by Dr. Christian S. Loh. |
Reference:
- Loh, C. S., & Sheng, Y. (2014). Maximum Similarity Index (MSI): A metric to differentiate the performance of novices vs. multiple-experts in serious games. Computers in Human Behavior. 39: 322-330. DOI: 10.1016/j.chb.2014.07.022
- Loh, C. S., & Sheng, Y. (2015). Measuring the (dis-)similarity between expert and novice behaviors as Serious Games Analytics. Education and Information Technologies. 20(1): 5-19. DOI: 10.1007/s10639-013-9263-y
- Loh, C. S. (2012). Information Trails: In-process assessment for game-based learning. In D. Ifenthaler, D. Eseryel, & X. Ge (Eds). Assessment in game-based learning: Foundations, innovations, and perspectives. (pp.123-144) New York, NY: Springer. DOI: 10.1007/978-1-4614-3546-4_8
- Loh, C. S. (Jan 2007). Designing Online Games Assessment as “Information Trails.” In D. Gibson, C. Aldrich & M. Prensky (Eds.), Games and Simulation in Online Learning: Research and Development Frameworks. (pp. 323-348). Hershey, PA: Idea Group, Inc.