The LLM Rubric Grader

The LLM Rubric assessment requires instructor approval before feedback is displayed to students. It can be found in the Manually Graded section of the assessments menu.

If you wish to skip the approval process see the LLM Rubric Autograde assessment in the Auto-Graded section.

More information about adding assessments can be found in our assessment documentation.

There are three steps to the LLM Rubric process:

  1. Rubric generation

  2. LLM-based grading using the generated rubric

  3. Manual approval of LLM-generated grades by the instructor

Note

Step 3 is required - LLM feedback will not be released to the student until the instructor reviews and approves it.

Add an LLM Rubric assessment to your guide page and follow the steps below.

LLM Based Rubric Creation (Step 1)

Complete the sections below to set up your rubric grader. For more information on General, Metadata (optional) and Files (optional) see Assessments.

  1. Complete General.

  2. Click Grading in the navigation pane and complete the following information:

Generate a rubric

  • Add a solution file (1) if you wish the rubric creation process to consider your solution.

  • Click the Generate Rubrics (2) button to initiate the process.

The Rubric Creation Agent uses the following items to generate the rubric items:

  • The assessment name

  • Instructions provided in the General tab of the assessment

  • Content of the Guide Page where the assessment is being added

  • Contents of the provided solution file

  • The Course, Module, and Assignment name

  • Requirements specified in the Rubric creation tab

Note

If you do not specify rubric requirements, the system will generate rubric items using general code grading norms.

Add your requirements in the Rubric Requirements dialog (optional):

Area to add your rubric requirements

  • Once you are done, click Generate Using AI.

  • You can provide additional rubric items by clicking Add Rubric and entering information.

  • Once you have reviewed the rubric items and other settings, click Save to save the assessment.

LLM Grading Based on the Created Rubric (Step 2)

The grading occurs when the student clicks the Check It button. The LLM Grading agent uses the following to grade the student’s work:

  • Instructions provided in the General tab of the assessment

  • Contents of the Guide page where the assessment is located

  • Contents of the specified solution file

  • The student file

  • The rubric generated in the previous step to identify the grading criteria

Note

The student receives information about whether they have passed or failed each rubric item, but does not see the rest of the LLM-generated feedback until after the instructor conducts their review.

Sample feedback displayed to the student (before instructor approval):

Sample feedback displayed to the student before instructor approval

Manual Approval by Instructor (Step 3)

In the case of the manually graded version of this assessment, the instructor must open the student’s assignment and select the passing test cases as part of the grading process. The instructor can also edit the comments generated by the LLM. Once this process is complete, click on Apply Grade.

Warning

Once feedback is released to students, it cannot be modified.

Sample Instructor view of LLM-generated feedback:

Instructor view of LLM-generated feedback for approval

Rubric Requirements Example

(You can view another example on the LLM Rubric Autograde page.)

Use only the following criteria for evaluating the student code:

  • Program correctness

  • Proper and efficient usage of a dictionary data structure