ADMIN | BUILDING | TESTING | ASSESSMENT | REPORTING | TOOLS | ARCHIVES | SYSTEM

Assessments Menu Help Page

 

Start Blind Assessments

OWL Raters can select an activity at random from the Blind Rating Pool.

  • The Start Blind Assessments menu option allows the OWL Rater to select an activity at random from the Blind Pool.

  • This means the Rater can assess the response without seeing the examinee name.

  • The Rater must be placed on the Assignment's roster to receive an assessment request.

  • Assessment requests can be generated automatically when creating the Assignment or manually using the Assessment Monitor page

  • Commit -The system will not allow the rater to "Commit" a rating (and record a score) until all rubrics associated with the response have received a rating. This is to prevent raters from accidentally submitting a partial rating.

  • Save - A Rater can "Save" a rating at any time. This is intended to allow the rater to close their work and finish at a later time.

 

Rater Access to Assessments

This table applies to OWL Raters who are on the assignment’s rater roster for an assignment that has a blind rating count, or if the assignment owner or an administrator manually assigns the rater to evaluate the response.

 My Assessments

The Assessments List shows responses that a Rater has started or completed.
IMPORTANT: this page only displays assessments that have NOT been finalized.

OWL TMS my assessment list page

  1. To access the page go to the Assessment Menu >> My Assessments

  2. View - This control opens the rating module to view the completed assessment.
    The original assessment is left untouched.
    This option will only be available if the Rater has already committed the assessment in the OWL Rater Module

  3. Edit - This control opens the rating module of the specified assessment that is “In progress"
    This option is available when the rater has started and saved the Assessment, but not yet clicked the Commit button in the OWL Rater Module

Field Definitions

  • Assessment Tracking# - Every OWL assessment is given a unique tracking number

  • Assignment - the OWL Name of the of Assignment being evaluated

  • Assessment Started - Date and Time the Rater started the Assessment

  • Assessment Completed - Date and Time the Assignment was last completed

  • Points Scored - the points awarded on response thus far

  • Level - If available, this is the proficiency level the response has received from the Rater

  • Open - (Yes) Assessment has been started but not yet committed. (No) Rating is committed

  • Language - The target language of the Assignment

Interviews

The Interviews list page is used to by the OWL Interviewer user to Launch the interview for a student.

  • Interview Items are Assigned to a Test Taker, but must be launched by an OWL Interviewer.

  • You must add OWL Interviewers to the Assignment's Interviewer Roster.

  • Only ONE interview file (recording) can be uploaded for each Interview Item.

  • Prompts can be added to the Item for the Interviewer to view. (For example, a list of interviewing questions)

  • If you want an Interview to be part of a larger activity, such as a screening test, use the External Item Type

Setting up An OWL Interview Activity

  1. Create an OWL Test that includes an Interview Item. See below

  2. Create an OWL Assignment for the Test - Set applicable assignment properties such as: Processing Algorithm, # of Blind Ratings, % of Secondary Ratings

  3. Add to Assignment Rosters

    • Students (required)

    • Interviewers (required)

    • Raters

    • Test Admins

  4. The Interviewer launches an Interview from the Interview List Page

    • Perform the Interview

    • Upload file of the recorded interview

    • Select a Rating

  5. Raters Perform any required additional ratings

  6. Finalize the Interview assessments from the Assessment List Page

Using the Interview List Page

  1. Go to the Assessment menu >> Interviews

  2. Click on the Interview control for the student to begin the interview process

owl interview list page

Assessment List

A hierarchical listing of current and completed assignments, examinee responses to those assignments, and ratings/assessments of those responses.

  1. Go to the Assessment Menu and select Assessment List

  2. Use the expand controls on the left to access the tables

  3. Use the check boxes next to the assignments or the individual responses to use the top-level commands

Top Level Commands

  • Finalize/Unfinalize - When the response is finalized, OWL computes the examinee's final score for the assignment. One of several algorithms may be used to determine how a final rating/score is established. The finalization algorithm is determined when creating the assignment.

  • Publish/Unpublish - This makes a final rating available for the examinee to view in their Completed Assignments. An assignment must be finalized before it can be published.

  • Blind Rate - Adds completed assignments to a pool so that they may be rated in a blind fashion by anyone who is listed as a primary rater on the assignment’s roster.

    • Finalized Ratings will not be placed into the blind rating pool. If a rating is final, "unfinalize” then “blind rate”

    • An OWL rater will not receive the same response to rate more than once.

  • Auto Rate - Forces completed assignments to be rated by the system. This is not generally necessary as system rating happens automatically when an examinee finishes a test.

    • Example: If you have not included all correct answers (short answer) or mismarked an answer as correct (multiple choice, drop down).

    • This function does NOT work to adjust points on a Test after testing has started. Points must be correct before testing launches.

  • Print Response - If your subscription includes this feature, you can select the assignment (or individual responses) and create a .pdf file of the test taker’s response. Printing Responses Help Page.

 

Assignments with Auto Scored only Tests will finalize automatically.

Assignment Table

The Assignment Table is the top tier table of the Assessment list page. Each row represents an Assignment that is current or completed.

  1. To see the Responses for each assignment use the bcontrol.

  2. If you select the checkbox next to an assignment and

  3. Click on a command at the top of the list, all eligible* responses for the selected assignment will be affected.
    (* For example, a response must be completed before it can be auto rated, or finalized before it can be published)

  4. Please use care, if you select the box at the top of the list, you will be processing all of your assignments/responses.

Response Table

Second tier of the assessment list page. Each row in the table represents an examinee response to the above assignment.

 

  1. To view the response table use the expand control next to the parent Assignment.

  2. Select a specific response with it's check box.

  3. Perform any of the Top Level commands listed above.

  4. The available Response Table Commands (see below) are shown in dark blue.

 

Response Table Commands

View Final - View finalized assessment of the examinee's Response

Assess - Opens the rating module in a new window.

Input Data (premium feature) - Allows you to open the External Item interface.

Status Definitions

  • Processing - Displayed between two other statuses, indicates OWL is performing some function on the data.

  • Activity in progress - Examinee has started but not completed the assignment

  • Waiting for data - (premium feature) For tests with External Items

  • Ready to be rated - The Assignment has been completed by a examinee and auto-rating is complete, but no Assessments have commenced on the Response.

  • Rating in progress - The Response is currently being rated by a human rater.

  • Rating complete - The examinee's response has been rated, but not yet marked as final.

  • Re-rating in progress - The Response is currently being re-rated by a human rater.

  • Re-rating completed - The Response has been rated, and then re-rated. This Assessment is now complete, but the rating has not yet been marked as final.

  • Rating final - The Assessment is complete and marked as final. See top level commands.

Assessment Table

Third tier of the assessment list page. Each row in the table represents a rating of the above response.

  1. To view the Assessment Table expand the parent Response using the expand control.

  2. The available Assessment Table Commands (see below) are shown in dark blue.

Assessment Table Commands

View - view the Assessment the same way the examinee would see it.

Review - opens the rating module with a copy of the specified assessment. The original assessment is left untouched. The new copy is not saved unless you "commit" the new rating while you are in the OWL Rating Module. In this case, a new assessment will be generated (new row in the table) and the original will remain untouched.

Edit - Opens the specific assessment in the rating module. If you were not the originally the rater of the assessment, you will be set as the rater of record. Any changes made to the assessment will overwrite previous scores, comments, etc.

Include- Select the checkbox to include the Assessment in the finalization algorithm for the assignment. If an assessment is not eligible for inclusion, a "-" will appear in the box.

Finalize - Click this command to establish the particular assessment as the final rating (disregarding the finalization algorithm for the assignment)

Delete - Removes the assessment from the table

Assessments (Flat List)

Shows the completed assessments for the assignments the user has permission to view.

  1. Go to Assessments >> Admin menu >> Assessment (Flat List)

  2. The page filters can be used to monitor rater activity, by date or assignment.

  3. The commands listed below can be used from this page. (Available commands are dark blue.)

assessments flat list

Assessments (Flat List) Commands

View - view the Assessment the same way the examinee would see it.

Review - opens the rating module with a copy of the specified assessment. The original assessment is left untouched. The new copy is not saved unless you "commit" the new rating while you are in the OWL Rating Module. In this case, a new assessment will be generated (new row in the table) and the original will remain untouched.

Edit - Opens the specific assessment in the rating module. If you were not the originally the rater of the assessment, you will be set as the rater of record and your changes will overwrite the original.

Delete - Removes the assessment from the table

Responses

Displays all started testing sessions. Responses can be rated or reset directly from this page.

  1. Go to Assessment menu >> Admin >> Responses

  2. To reset a response, select the response using the checkbox for the individual response

    • Please use caution, if you use the checkbox in the header, you will select/delete all requests from the pool

  3. Click the Reset button at the top of the page.

  4. Use the Assess command to open the rating module in a new window.

  5. The command button will be displayed in light blue if the activity is in progress and therefore not available for rating.

Resetting an Assignment is PERMANENT. The examinee's response will be erased and none of their responses will be saved.

Proctoring Sessions

This list page allows users* with this feature to view AI Proctoring Sessions.

  1. Go to the Assessment Menu

  2. Select Proctoring Sessions

  3. View credibility metrics from the List Page

    • The credibility score is based on 100 points.

    • Deductions to 100 points are shown to the right in the list page.

    • Credibility can be reduced for the following reasons: focus changed, full screen disabled, webcam disabled, face not visible, multiple faces, face mismatch, no network connection, screen not shared, second display, volume low or muted, background noise.

  4. Click on the View Session icon (circle i) to see an individual's webcam footage and screen sharing

NOTE: This is a premium feature. Please contact support@owlts.com to find out about adding AI Proctoring to your OWL subscription.

*The following types of OWL Users can see this menu item:

  • System Administrators

  • Instructors who have created the assignment

Assessment Monitor

Shows the status of assessment requests and completed assessments for each assignment.
Page can be used to manually request a response be placed into the blind rating pool.

You must first select the assignment to view using the drop down menu at the top-right.

  1. Select the assignment you wish to view using the drop down menu at the top-right.

  2. To request a new blind assessment, select the response with the checkbox.

    NOTE: Only completed sessions will have an associated checkbox

  3. Click the button for a Primary or Secondary Assessment

  4. Select from the eligible raters that appear in the drop down, and click OK

More Info About How Blind Ratings are Allocated

  • Each rater will only get one opportunity to perform a blind rating on a given response.

  • For example, if a rater has completed a blind rating of a specific test response, if you use then the assessment monitor to request an assessment from [Any Rater] that rater will NOT get the blind assessment request

  • If you request that rater specifically (from the drop down as in step 4 above), then the WILL receive the response to rate a second time.

  • If you delete the assessment, the response will be placed back into the blind rating pool and it will be available to the same rater (or other raters) to rate again.

Assessment Requests

Displays all responses that have been completed by an examinee
but have not yet been rated.

  • For Assignments that use the Blind Rating Setting

  • There is one row for each request.

  • For example, if the Assignment’s Primary Blind Assessment Count = "3" each examinee’s response will be listed 3 times.

 

  1. Go to Assessments >> Admin menu >> Assessment Requests

  2. You can delete an assessment request from the pool by selecting it using the checkbox on the left

  3. Then click the Delete button

  4. Please use caution, if you use the checkbox in the header, you will select/delete all requests from the pool

  5. If you need to resubmit a request to the blind rating pool, go to the Assessment List Page or the Assessment Monitor

Assessment Requests Remaining

For users who need to manage the timeliness of their raters, this page displays the status of pending or open assessment requests.

1. Reset - This control will put the request back into the Blind Assessment pool, making it available to all assigned raters again. This control will only be active (darker blue) if a rating has been launched.

2. Response Age Hours - Shows how long ago the test taker completed the response.

Assessment Requests Remaining screen

3. Request Age Hours - Shows how long ago the request was put into the blind assessment pool.

4. Request In Use - Use this field to filter is an assessment has been launched or not.

5. Assessment Age Minutes - This shows how long ago the rater launched the assessment.

Assessment Requests Remaining screen extended
 

Unfinalized Responses

This page shows the status of responses that have yet to be finalized.

  1. Response Age - Shows how long ago the test taker completed the response.

  2. Status - Shows the responses place in the assessment process

 

Unrateable Assessments

This page displays assessments that have been marked as unrateable in someway.

  1. Additional information provided by the rater is shown here.

  2. These commands allow you to work with the specific assessment:

    • View as test taker.

    • Review in OWL Rater Module.

    • Edit the Rating and become the rater of record.

sample of OWL unrateable assessments list page

This page is visible to the following types of users:

  • System Administrators

  • Instructors who created the Assignment

  • Test Administrators on the Assignment’s Roster

 

Suspected Cheating

This page displays assessments that have been marked as suspected cheating by the rater.

  1. Additional information provided by the rater is shown here.

  2. These commands allow you to work with the specific assessment:

    • View as test taker.

    • Review in OWL Rater Module.

    • Edit the Rating and become the rater of record.

    • Delete the Assessment.

number screenshot of suspected cheating list page

This page is visible to the following types of users:

  • System Administrators

  • Instructors who created the Assignment

  • Test Administrators on the Assignment’s Roster

 

Results

This option is for subscriptions that include the Results Viewer user type.

  • Results Viewers need to be on the assignment roster.

  • Only finalized results are visible to Results Viewers.

  • A Results Report can be run for export to a csv file.

  • Results Viewers can receive an Automatic Notification triggered on the event “Rating Finalized”

Results Page Definitions

  • Assignment - Name of the Assignment

  • Examinee Name - First Name and Last Name of the Test Taker

  • Examinee Username - OWL username of the examinee, typically their email address

  • Language - Target Language the Assignment is evaluating

  • Score - Numerical Score of the examinee's response

  • Rating - The scale that was used to evaluate the response and the level that was assigned

  • Test Completed - The date and time the response was submitted

  • Rating Completed - The date and time the assessment was committed

  • Assignment Tracking # - The unique OWL tracking number of the assignment

  • Examinee Tracking # - The unique OWL tracking number of the test taker

External Scoring

This is a customization option for organizations who utilize an external or AI rating service.

  • This page allows you to view response details and their corresponding ratings.

  • Each line represents a rating for one question.

  • Additional information is provided about the rating service provided.

FAQs

How do I save a Test Taker’s audio response?

Option 1: Save the files directly from the OWL Rater Module

  1. Open the Student's Response/Assessment in Rater from the Assessment List Page

  2. Click on the words “Student Response” located on the header of the Oral Response
    This will open the file in a Web audio player.

  3. Right click on the Player (note you must click directly on the player)

  4. Select "Save As" from the menu to save the file to your computer.

Option 2: Run a Response Report. The .csv file will include links the test takers audio responses.

Option 3: Create an Examinee Portfolio

If your subscription includes this premium feature, an Instructor or an Administrator to create a .zip file of all recordings a student has produced within OWL with the Examinee Portfolios feature.

How do I manually add a response to the Blind Pool?

1. Go to the Assessment Menu

2. Select Admin >> Assessment Monitor

Assessment Monitor Help

Why isn't the "Commit" Button Active?

  • If the commit button is greyed out there is a rubric or a rubric row somewhere that does not have a radio button clicked.

  • A rater cannot use the "Commit" button to perform a rating until every Item or Section that requires a performance rating receives one.