Two claims circulate in every conversation about graduates and work.
Programs say they develop critical thinkers, ethical technologists, people who can lead through change. Employers say the people they hire arrive without those things.
Both sides are sincere, both have anecdotes, and neither can be checked, because there is no instrument the two of them share.
The employer’s version of “critical thinking” lives in a competency framework or a job posting. The program’s version lives in a learning objective and a rubric.
The words match. Nothing underneath them does.
Nobody has agreed on what the construct is, what counts as evidence of it, or what “proficient” looks like in a piece of work. Without that, a program’s claim to develop a competency and an employer’s claim that graduates lack it are both unfalsifiable, and an argument between two unfalsifiable claims is the kind that lasts for decades.
This matters more now than it did ten years ago because the list of competencies employers ask for is moving. Generative AI literacy was not on anyone’s framework in 2021. A program that cannot measure whether it develops a competency cannot tell whether it is keeping up, and cannot show anyone that it is.
Why the usual fixes produce labels, not measurements
The standard response is a crosswalk. Take an employer-facing framework, NACE’s career readiness competencies or the OECD’s, and map each course to the competencies it addresses. The result is a matrix with a lot of checkmarks. It tells you that a course claims a competency. It does not tell you what the competency is in operational terms, how much of it the course develops, or whether two faculty would recognize the same student work as evidence of it. A crosswalk is a spreadsheet of assertions.
The newer response is to let AI read job postings and extract the skills in demand. This does capture that requirements are moving, and it is fast. But posting language is written by recruiters to attract applicants, not by anyone trying to define a construct. “Strong communicator” appears in postings for jobs that need entirely different things. And nothing in a job posting says what a student would have to produce to show they have it.
Both approaches stop at the label. The question that matters for a program is what sits underneath the label, and that question was answered some time ago in a literature most assessment offices never read.
The instruments already exist
Organizational and management research has spent decades building validated instruments for exactly the competencies employers now name. These were developed to measure working professionals, which is the population the employer complaint is about. A few of the ones we worked with:
Each instrument does three things a framework label cannot. It defines the construct precisely enough that two people mean the same thing by it. It decomposes the construct into dimensions concrete enough to look for in a piece of work: not “digital leadership” but cultivating digital culture, digital competitiveness intelligence. And it carries evidence that the construct is real and measurable in people, with factor structures and reliability coefficients that someone has already fought over in peer review.
That third property is the one that changes the conversation with employers. A program claiming to develop “digital leadership” is making an assertion. A program showing that its students’ work evidences four of Munsamy’s six digital leadership dimensions is reporting a measurement against the same construct an organization would use to assess its own managers. The employer and the program are, for the first time, talking about the same thing.
From a scale dimension to a sentence in a student paper
Here is how it works in a course. A graduate design course has a rubric competency called Strategic Problem Framing and Systems Thinking, with a top level that reads “frames a deep, strategically important organizational problem; demonstrates strong systems thinking linking stakeholders, context, and transformation opportunity.” Recognizable, and unmeasurable as written.
Working with the faculty, we traced it down.
To the program’s competency domain. Strategic Technology Leadership, with a critical thinking component.
To validated instruments. Munsamy’s Digital Leadership Competency, specifically its facilitating digital drive dimension (aligning digital initiatives to organizational strategy), cross-checked against the logical consistency and depth dimensions of Sumarni’s critical thinking assessment. The instructor confirmed this was what they had meant.
To behavioral sub-criteria, each a yes or no. The problem statement names organizational context, affected stakeholders and strategic impact. It explains why the problem is strategic rather than symptomatic. At least one concrete data point supports it. The design goal connects to at least two stakeholder groups and frames success as outcomes rather than outputs.
To the place in the artifact where the evidence should appear. Section 1 (Design Goal) and Section 7 (Success Criteria) of the assignment template.
The instructor’s phrase “strong systems thinking” is now a set of observable statements about a text, each tied to a dimension of a published instrument. When a student writes that recruiters at a 3,500-person company spend 60 to 70 percent of their time on manual screening and that this creates measurable bias downstream, that sentence is evidence for facilitating digital drive. Someone could disagree with the mapping. Nobody has to guess what was measured.
The instructor also came out of this with a rubric that says what it means, which is worth something on its own.
Where AI comes in
None of the above requires AI. What AI changes is scale. Checking twenty papers against twelve behavioral sub-criteria, consistently, with the supporting sentence recorded for each call, is work no faculty member has time for, which is why rubrics stayed vague in the first place.
In our study the evaluator is loaded with the full rubric and sub-criteria before it reads anything. It reads each submission whole, returns met or not met per sub-criterion, and writes a short narrative pointing to the text it relied on. It produces no score.
Where a sub-criterion cannot be settled from text (whether the research was actually conducted, how the oral presentation went), it says so and stops. The instructor reads the signals against the paper and assigns the level. The explainable-AI literature in education has been clear that an accurate model that cannot say why it decided is of limited use to the people who have to act on it; the narrative is what makes the output something a faculty member can check in the time it takes to read a paragraph.
The point for this post is narrower. The AI is reading against instruments that were validated before it existed. It inherits their precision. Hand it a vague rubric and it will be vague back, with confidence.
What a program can see that it couldn’t before
The real return is one level up from the course. A program’s learning objectives are usually assessed course by course, each with its own rubric in its own language, and the results are stitched together once a cycle into a report that nobody fully believes. The stitching is the problem.
“Critical thinking” in the first-year course and “critical thinking” in the capstone are measured with different instruments, so a change between them could be learning or could be the rubric.
When every course criterion is traced to the same validated dimensions, the program gets a common unit. The same dimension of the same instrument is observed in a first-semester memo and a final design document, and the comparison means something. A few things become visible:
The pattern-level findings are the ones that change decisions. In our test course, if six of eight teams failed evidence traceability, that is no longer six grading notes. It is a fact about how research methods are taught, and it is on the table before the next cohort. At program scale, a dimension that no course ever asks students to demonstrate is a curriculum finding. A dimension that courses ask for and students rarely show is a teaching finding. A dimension that employers have started naming and no validated instrument yet measures is a research finding, and an honest one to report.
This is also, as it happens, what assurance of learning processes have always asked for and rarely received: evidence that student work demonstrates the competencies the program claims, traced from the claim to the artifact.
The instruments make the trace possible. The AI makes it affordable.
What to be careful about
A scale validated on managers was not validated on students writing a design brief. Mapping a course criterion to Munsamy’s instrument borrows that instrument’s rigor. It does not inherit its validation. Whether the dimensions behave the same way in a classroom artifact is an empirical question, and treating it as settled would be the same mistake the crosswalks make, dressed better.
Explicit criteria can be written to. “At least one concrete data point” invites a data point dropped in for the checkbox. This is Goodhart’s law and it predates AI. Behaviorally anchored criteria at least make it visible when a box was ticked mechanically, because the supporting sentence is recorded.
Text-based assessment can reward fluency over the construct. Automated scoring has a documented tendency to track language proficiency, which disadvantages non-native writers. Criteria anchored in observable content (”names two stakeholder groups”) are less exposed than holistic ones (”writes compellingly”), and the communication dimensions need the closest watch.
Some competencies have no validated instrument yet. Employers are naming things faster than measurement research can catch up. Where that is true, the honest move is to say so rather than to borrow the nearest scale and hope.
The instrument was the missing piece
The argument between programs and employers has gone on as long as it has because both sides were making claims no instrument could test. The instruments turn out to exist. They were built by people measuring the same competencies in the same workplaces the employers are describing, and they have been sitting in management journals while assessment offices built crosswalks.
Tracing course criteria to those instruments gives a program a common unit across courses, a shared definition with the people who hire its graduates, and a way to see where a competency is developed, where it stalls, and where nobody asks for it. AI makes the reading affordable. The measurement discipline is what makes it mean anything.
For anyone who runs a program: pick one competency your graduates are supposed to have and one that employers in your field have started asking for. Then ask whether anyone in the building could say, with a citation, what either one is.
