ADVENTURES in Online Testing (Volume 6, Issue 1)
by Chris Dalessandri, Managing Member, OWL Testing Software
Can you remember the first time you saw Raiders of the Lost Ark? If you are a fan, you’ve probably seen it multiple times, but I’m talking about the first time you saw the movie. I can still remember sitting on the proverbial (and actual) edge of my seat, watching Indy endure multiple tests of strength, intellect and of course will. What a rush it was to see him survive the tremendous adventure against such mighty enemies and take possession of the ultimate prize. That most powerful prize could lead armies, save lives and end WWII. And what happened to that Lost Ark anyway? Ultimately, it was packed away in some nondescript government warehouse -- lost forever in a seemingly hopeless sea of bureaucracy? To my mind, that was not only cynical, but also frustrating.
Well that is the way I am starting to feel about the state of online testing today. So much emphasis and effort is placed upon flashy presentation and delivery, the adventure. Little thought is given to the data collected, or what I see as the treasure. If collected and stored properly, the data gathered by a test management system is a powerful tool that improves learning and outcomes across an organization. Whether you are trying to improve a student’s ability to speak a language, learn a discipline, or if you are trying to find the right employee for your team or certify a candidate's knowledge; the data is critical to your success. Your ability to look at the data collected over the long term is the key to quantitatively improve your outcomes and reach your goals.
The Testing Adventure... There are a wealth of options when it comes to online testing: how students interface with the system, which question types are available, what types of multimedia can be integrated, how responses are recorded and scored, and so on. As a user or program administrator, you may be choosing an online test management system for many reasons:
Perhaps you would like to give your students or instructors access to hot new technology? Online testing does provide a vehicle for readily integrating multimedia to simulate real life situations.
Perhaps you wish to protect your back from lugging test booklets back and forth to work? I am sure most of us would appreciate a few less trips to the Chiropractor.
Perhaps you just want to see how students respond to a hot new song or movie clip? This is nice. It can even be a good motivator. However, this is probably more of a ‘nice to have’ perk than an actual system benefit.
All of these feature-based system benefits are good selection criteria. Testing company websites are cluttered with product-feature grids that map out all of their bells and whistles. While these are important elements that absolutely should be part of your online test management system, none of these perks capture the potential treasure of an on-line test management system. The true wealth of an online test management system is the data it captures. This data, if collected and stored carefully, can represent the ultimate treasure.
The Ultimate Treasure... Unfortunately, very little focus is given to what we will do with the 'treasure', in this case, all of the test response data. Reporting is the most important thing that an online testing system offers and too often it is little more than an afterthought. Administered properly, an online test management system gives you access to an incredible wealth of easily mined data at your fingertips.
Your testing data should be put into a warehouse, but a warehouse that is nothing like the one in Raiders of the Lost Ark. In this warehouse everything should be rapidly accessible, and ready to be analyzed at a moments notice. From this warehouse, data is easily converted into graphs and charts for a presentations or downloaded into a statistical analysis program for further analysis. This is how the OWL Test Management System carefully handles your precious data.
You ask students to complete practice activities, tutorials, practice tests and homework to help them learn. But why do you give assessments? There are two answers to this question. The first reason is to evaluate how well they have learned the material, idea or concept.
The second is to evaluate how well the instruction is being delivered. In either case, the real purpose of creating, delivering and scoring/rating tests is the end result, the data. No matter how rigorous your test design process is, or how much media you include within your tests; in the end all you really want to know is how well the test taker knows something. You also may want to know how students are able to learn something over time. Whether the validation of that knowledge is for acceptance, placement, grades, graduation, or certification; you want to know what they know, both as a group and as individuals.
By the time the test takers complete their final assessment, your opportunity to effect that individual learning has past. What makes the data useful to the program as a whole is your ability to identify trends and make improvement. Useful data allows you to compare frequency of practice with final scores, to analyze your test for question efficacy and, if you are involved in the instruction, to look at how instruction relates to results --the “wash back”. In this regard, it is critically important that data is not only collected, but also useful. In other words, it must be the “right” data.
The Value of the Treasure... For data to truly be a treasure worth mining, it must have several key qualities.
- Reliability - This seems obvious but it is more involved than you might think. For example, the testing system must accurately capture and maintain your test taker's original response. There should be no way for any outside party, including the rater, proctor or instructor, to modify the test taker’s responses. A rater’s comments and corrections can be shown with the response for ease of comprehension when displayed by the system, but the two data elements should never be commingled when they are stored.
- Completeness - Completeness also seems obvious but as with most “obvious” things it can be somewhat illusive. You may not even know what it means to have complete data until later when you try to mine the data to learn something specific about a given test. For this reason, it is very important to have access to a variety of data about the test experience. Some examples of testing data include: specific information about how the test was proctored; at exactly what point in the student's studies the test was given; how much time was given to complete the test, etc. This type of information is important both for test analysis and to close the wash-back loop.
- Access - It is important that you have continuous, rapid access to data as you need it. For the data to be useful to the instructor, administrator, certification body or admissions officer it must be readily available for ongoing analysis. As new needs present themselves, and changes are proposed, it is ideal to evaluate the potential impact of proposed changes using past data as a predictor.
- Portability - This means you should be able to easily move data from one format or platform to another. To achieve portability, data must be made uniform before it is stored in the data warehouse. With portable data, a program's results can be available for analysis regardless of the desired analysis tool. Data should be readily available to be analyzed using a simple spreadsheet or more complex psychometric tool.
- Compartmentalization - Hand-in-hand with portability is your ability to have the right people see specific parts of the data. For example, you may want the examinees to have access to their own scores as well as comments about their specific performance. But you probably would not want the examinees to have access to an item analysis report showing which options were chosen by each examinee on a given test.
In the end, a test management system is only as useful as the data it stores and the reporting it can produce. If the system's response and rating data is unreliable, incomplete, not available to anyone without an advanced computer programming degree, or if it is stuck in a proprietary format unable to be moved; then it may as well be packed away in some forgotten storage crate. When choosing a test management system for your enterprise, select one that is more than a shiny object. Choose a test management system that has been designed and engineered to properly capture, store and manage your valuable data over the long term.