At one business school, faculty spent part of an Assurance of Learning cycle disagreeing about what counts as a “tool” in a student presentation. The rubric asked raters to check whether students used one. Nobody had written down what the word meant. Every rater filled in the blank differently, and the scores told the committee more about the raters than about the students.
Stories like this one came up repeatedly on October 21, when we hosted the first module of the second cycle of our Roundtable Series. The session focused on rubric design, which we see as the foundation of an AoL portfolio of evidence.
In our research, most AoL time goes to operational work: compiling past reports, hunting down scattered spreadsheets, and coordinating faculty. We want AI to take on more of that load so faculty have time for academic and strategic decisions. We are piloting an internal artifact scorer, and this session was our first look at whether the same technology can help with the rubrics that sit on top of it.
In the workflow we are piloting, the AI drafts and flags, and nothing moves to the next step until faculty have approved it.
Why the rubric carries the whole report
AACSB Standard 5 asks schools to show, through a combination of direct and indirect measures, that students are meeting the competencies the program set out for them. Direct measures come from student work: exams, assignments, projects, scored against a rubric. Indirect measures come from third parties, such as surveys and employer feedback. The standard also expects schools to “close the loop,” which AACSB interprets as showing how the curriculum improved as a result of the AoL process, with a second round of data to judge whether the change worked.
That chain has a weak link. Every direct measure in the report depends on the rubric used to produce it. If two raters can read the same descriptor two ways, the scores will not agree, and a peer review team has no way to tell whether a change in results reflects student learning or rater drift.
The room summed up current practice candidly. One participant described a committee, a set of artifacts, a rubric, someone to assess them, and then the cycle is over. Checking that the rubric itself holds up rarely gets its own step.
On paper, rubric design has four steps. Most schools run all four. The difference is how much of each step is written down, and how much of it lives with whoever built the rubric.
What the research says about rubric quality
Rubrics have been studied for decades, and the findings are consistent on a few points that matter here.
Analytic, task-specific rubrics score more reliably. Jönsson and Svingby’s 2007 review of 75 studies found that rubrics improve the consistency of scoring, and that the gain is largest when the rubric is analytic (one judgment per criterion), tied to the specific task, and paired with exemplars or rater training. A later study by Jönsson and Balan (2018) put a number on the difference: assessors using an analytic model agreed with each other on 66% of grades, against 46% for a holistic model.
The criteria do the work. Susan Brookhart’s 2018 review of rubric studies in higher education argues that appropriate criteria are what make a rubric effective. She also flags a common shortcut: performance levels written in rating-scale language (”excellent,” “adequate”) or as counts of elements, rather than as descriptions of what the work looks like at each level. Descriptive language is what lets a rater, or a student, recognize where a piece of work sits.
Reliability is rarely checked. Reddy and Andrade’s 2010 review of rubric use in higher education found that validity and reliability can be established for rubrics, and that higher education applications often skip that step. The same review noted that some faculty resist rubrics, often because the purpose of the rubric was never made clear to them.
The gap is not small. Twenty points of agreement is the difference between a result a peer review team can rely on and one that mostly reflects who happened to do the grading.
Put together, the literature says a rubric is only as good as its criteria and descriptors, and most AoL committees do not have a step for testing either before scoring begins. That is where the rest of the session focused.
How one school rebuilt its rubrics
After its last peer review called for a more mature AoL system, one participant’s school paused AoL for a year and rebuilt it. All faculty brainstormed the rubrics at a summer workshop with an external consultant, an independent accreditation officer who also teaches for AACSB. The committee refined the drafts and the full faculty gave final approval. The process also taught faculty how AoL is supposed to work, which the participants described as a benefit in its own right.
The school came away with several lessons, and each one matches a pattern the research describes:
Length: The first rubrics were too long and too detailed, and many items were double or triple-barreled. The school has since cut both the wording and the number of rubrics.
Artifacts: Nobody checked that student artifacts existed to match each criterion before the rubrics went out.
Definitions: Some terms were never defined. The “tool” disagreement above came from this school.
Levels: There were no clear definitions of meets, does not meet and exceeds for each line item.
The school is now considering a new rubric to support its AACSB societal impact direction.
Where rubrics break, and what it costs
We presented three failure modes we see repeatedly. All three exist before AI enters the picture.
Vague criteria. Descriptors like “adequate understanding” mean different things to different raters, so the same artifact earns a different score depending on who grades it. The cost: results that are hard to defend in front of a peer review team.
Data holes that appear too late. Some criteria cannot be measured from the artifacts collected, and the gap is usually discovered during scoring. The cost: the course has ended, and the assessment cycle has to be repeated.
Outcome drift. Rubrics quietly fall out of alignment with the learning outcomes they are supposed to measure. The cost: scores stop telling you anything about the outcome.
The participants agreed the biggest cost is time. Fixing these problems delays closing the loop, and with it the improvements to student learning the loop exists to produce. Since AACSB expects curriculum changes to rest on data, a delayed loop means decisions made on best guesses in the meantime.
One piece of practical advice from the room: write down your decisions, mistakes and fixes, and include them in the AoL report. Peer reviewers want to see the process improving over time. They are not expecting a perfect one.
When a criterion has no matching evidence, participants listed four ways to close the gap:
Change the assignment or pick a different one.
Revise or reframe the rubric elements.
Find the evidence elsewhere, including through indirect measures.
Create a new assignment built to fit the rubric. One participant added an individual reflective “innovation” essay to her syllabus because group presentations can’t be assessed individually.
All of these depend on faculty buy-in. Asking colleagues to change their assignments takes trust, and how much of it exists varies a lot between schools.
Exercise: one descriptor, many readings
We gave the group a draft critical-thinking rubric with 3 primary traits and 9 sub-traits, scored 0, 1 or 2, and built several flaws into it on purpose. The question was where the room would differ and where it would agree. The group found most of the flaws, and added a few we had not planted.
Criteria that don’t match their descriptors: “Identifies the problem and the solution” asks for a solution, but the descriptors contrast positive and negative impacts.
Yes/no items on a three-level scale: Several sub-traits could only be present or absent, which leaves no room for a “meets” between “does not meet” and “exceeds.” Each item should ask how well the student does something.
Single words that change the meaning: “Shows the basis for any figure cited” effectively means every figure, which makes the item binary again. “Basis” could also mean a citation or a rationale, and two raters would read it two ways.
Thin descriptors: Phrases like “proposes some working plan” or “final recommendation reflecting risks” need more depth. They should say whether the plan is feasible and whether the recommendation creates value.
Two structural points came out of the discussion:
Assignment rubric vs. AoL rubric: An AoL rubric deliberately takes the reviewer out of the course context and measures only alignment with the learning outcome. An A paper can score poorly on it, and the reverse can happen too. Conflating the two leads to wrong conclusions.
The mapping challenge: A single assignment may cover only some traits or sub-traits. That forces a choice between a short rubric that many courses can assess and a fuller rubric whose pieces have to be gathered from different places.
Where AI can help
Before the session, we asked participants which parts of rubric design weigh on them most. They named drafting criteria, getting faculty involved, and catching problems late. AI can help with the first and the third. Faculty involvement still needs people in the room.
The research points to the same three levers for better rubrics: analytic, task-specific design; exemplars of student work at each level; and calibration sessions where raters score the same artifacts and reconcile their differences. Each is faculty work. The AI opportunity sits beside them: reviewing a draft rubric before the cycle starts and flagging what will cause trouble at scoring time.
We showed what a “rubric helper” can do with a general-purpose model such as ChatGPT or Claude, given the draft rubric, the program learning outcomes and the planned assignments. Here is what it flags, and what it suggests:
Each of these costs less to fix before data collection than after scoring. Faculty accept, edit or reject every suggestion, and each decision and its reason is logged for the AoL report, which is exactly the record of process improvement peer reviewers ask for.
Participants took the idea further than we had:
One suggested AI could manage the faculty-input process and check the curriculum for gaps or unclear terms, with faculty reviewing before and after drafting.
Another suggested AI could match a rubric against a set of assignments and recommend changes to both.
She also asked whether AI could test if the learning outcomes themselves fit a school’s mission and what sets it apart.
Our answer to that last question is to give the AI institutional context: the mission, vision, strategy documents, assignments and artifacts. With that, it can advise on how the rubric, the assignments and the outcomes fit together.
Our position is that AI works as a collaborative partner with a human in the loop. The helper suggests; faculty decide every change. In the exercise, the faculty’s suggestions went further than the AI’s flags, which is a good reminder of where the expertise sits.
Key takeaways
Rubric design and faculty calibration decide whether direct-measure results are consistent and defensible. The research has said so since 2007; the room confirmed it from experience.
Most rubric problems can be caught before data collection: vague descriptors, criteria that test two things at once, and criteria no artifact can show.
Analytic rubrics make scoring work better and build trust among the faculty who score. Artifact scoring earns that trust by running alongside faculty scoring first.
What’s next
Module 2 turns this into a hands-on workshop on designing an analytic rubric with AI. Module 3, planned for December, covers governance guardrails for using AI in AoL.
We are building this in the open alongside faculty, assessment directors and accreditation leaders. If you have a rubric you would like to stress-test, or you want to compare notes on how your committee handles rubric design, we would be glad to hear from you.
Sources
AACSB. 2020 Guiding Principles and Standards for Business Accreditation, Standard 5: Assurance of Learning, and the accompanying interpretive guidance on closing the loop.
Brookhart, S. M. (2018). Appropriate criteria: Key to effective rubrics. Frontiers in Education, 3(22).
Jönsson, A., & Balan, A. (2018). Analytic or holistic: A study of agreement between different grading models. Practical Assessment, Research & Evaluation, 23(12).
Jönsson, A., & Svingby, G. (2007). The use of scoring rubrics: Reliability, validity and educational consequences. Educational Research Review, 2(2), 130–144.
Reddy, Y. M., & Andrade, H. (2010). A review of rubric use in higher education. Assessment & Evaluation in Higher Education, 35(4), 435–448.