Evaluating CRIs

Constructed response items are easier to develop but more difficult to grade. This is because they are open-ended questions that must be graded manually.

To avoid bias, a thorough process must be followed if you plan to include constructed response questions or items in your assessment. See the steps below for how to evaluate a CRI.

1. Create Your Assessment Panel
   Comprised of people from human resources, administrative offices, hiring managers, and support staff. Generally, the hiring manager is part of the assessment panel.
   While generally, 3 panel members are recommended for interviews, constructed response questions are typically double-marked (two graders), and at least 1 female should be on the panel.
   Panel members should be at or above the level of the position, and should have some knowledge of the expectations for the position being filled.
   Panel members should remain the same for all candidates being evaluated. Any variations or substitutions must be documented in inspira when drafting the transmittal memo.
   All panel members should be trained prior to serving in an assessment panel. Check with your recruiter for more information on this process.
   For more information about the assessment panel, read Section 7 of the Staff Selection System – ST/AI/2010/3(opens in a new tab).
2. Prepare Grading Documents
   Grading documents should have been prepared when drafting the constructed response questions. These documents include:
   Sample Responses
   The Mark Scheme
   Content Points
   Ensure that all panel members have reviewed the grading documents prior to evaluating candidate responses.
3. Standard Setting
   Once you have received all responses to the constructed response items, you must make sure the panel is “calibrated.” This means that they should all be in agreement about how to grade responses. This will set a standard that they will follow during the actual grading process.
   To set a standard:
   The panel picks 5 to 10 responses at random.
   The panel reviews these responses individually against the grading documents.
   The panel compares the individual mark allocation on the mark scheme for each response
   The panel evaluates differences in grades and discusses to determine the cause of the difference. They must then revisit the mark scheme and agree on a standard.
4. Grading
   Must take place under standardized conditions.
   To avoid bias, be sure to anonymize the responses. This means that responses should only show candidate ID# instead of names.
   Each response must only be evaluated against the grading documents, never against each other.
   Use the full range of the mark scheme.
   Try to give credit for the positive. (Focus on the good- you will notice the bad anyway)
   If responses include valid content points that were not considered in the original mark scheme, you may add these to the mark scheme as long as all responses are subsequently given consideration with this new content point.
   Grading should consist of two phases:
   First reading looks specifically for primary criterion
   Second reading looks for the remaining criteria.
5. Discrepancies/Consensus Process
   Just as with interviews, assessors’ scores should be within a certain range of one another. This helps to confirm that the marking scheme is being used consistently across the graders. There may be cases when there are discrepancies (significantly different scores from different graders.)
   A consensus rule should be decided in advance, for example: “more than a 20% discrepancy in the overall score will require re-evaluation.”
   Here are two examples of how this rule is applied.
   On a scale of 0 – 100, one grader scored 65 and one grader scored 80 [within consensus]
   On a scale of 0 – 50, one grader scored 25 and one grader scored 40 [discrepancy]
   Have both graders independently re-grade discrepant responses
   If discrepancy persists, either use a third grader, or two graders meet to discuss and resolve discrepancy.
6. Final Review
   Calculate overall average for each grader and overall score
   Calculate average for each grading criteria
   Calculate the min and max to determine the range of scores
   Look at overall score distribution
   Set the cut score / pass mark anonymously

)footnote¹