How should Korea evaluate AI capability

Korea cannot evaluate AI capability reliably through credentials, coding output, or AI-detection scores alone.
Valid assessment must observe formulation, transfer, verification, revision, and responsibility under changing conditions.
A national talent strategy needs measurement instruments aligned with the capabilities it claims to produce.

The Korean Policy Question

Assessment is the hidden infrastructure of talent policy. If Korea funds AI education while measuring success through enrollment, completion, or familiar technical exercises, providers will rationally optimize those outputs. AI for All: Strategy for Cultivating Artificial Intelligence Talent therefore depends on an assessment system capable of distinguishing tool use from independent capability.

The problem extends beyond universities. Employers, public agencies, and professional programs all need evidence that remains meaningful when generative AI can produce polished code, explanations, and reports.

Assessment Defines the Effective Curriculum

Students allocate effort toward what an institution rewards. If examinations reward memorized derivations, students memorize. If projects reward a high validation score, students search until the score improves. If code only needs to run, conceptual reconstruction becomes optional.

Assessment is therefore not merely a measurement after teaching. It is part of the educational production framework.

• foundational knowledge;
• technical execution;
• model judgment;
• validation and criticism; and
• decision integration.

What Should Be Measured?

Table 1. Dimensions of a valid AI assessment

Competence	Observable performance	What does not establish it
Foundational knowledge	Defines concepts and derives core results	Recognizing vocabulary
Technical execution	Implements a specified method correctly	Copying a working notebook
Model judgment	Chooses and adapts a model under assumptions	Applying an assigned formula
Validation	Designs tests that target plausible failures	Reporting one accuracy number
Decision integration	Connects output to costs and action	Producing a prediction

A balanced AI assessment needs evidence across all five components.

Reliability Is Not Validity

An assessment can consistently rank students while measuring the wrong capability.

Suppose a multiple-choice examination is highly reliable because students’ scores are stable across equivalent forms. If the program claims to develop independent modelling, recall-based reliability is not enough.

The Standards for Educational and Psychological Testing treats validity as the evidential basis for interpreting test scores for their intended use. In AI education, the intended use might be admission to a mathematically demanding program, progression to independent project work, or certification of professional model judgment. Each use requires different evidence.

Transfer Is the Central Test

Memorization is successful when a familiar problem reappears. Understanding is visible when the surface changes but the underlying structure can be reconstructed.

• random sampling becomes self-selection;
• independent rows become repeated measures;
• prediction becomes intervention;
• the deployment base rate changes;
• one feature disappears at decision time;
• error costs become asymmetric; or
• a linear relationship becomes threshold-dependent.

The assessment remains fair because the mathematical tools were taught. What changes is the requirement to choose among them.

Case-Based Assessment

A case provides enough institutional detail for students to formulate a model but not enough structure to make the answer automatic.

1. a decision owner;
2. an outcome that may be measured imperfectly;
3. a sample generated by a recognizable process;
4. at least two plausible model approaches;
5. a constraint or asymmetric cost;
6. a possible distribution shift; and
7. evidence that can support criticism.

Students should be asked to state what cannot be concluded. That requirement distinguishes responsible model use from confident calculation.

A Multiplicative Scoring Rule

Simple weighted averages allow strength in one area to compensate completely for failure in another. An AI student might obtain a passing score through excellent coding despite being unable to recognize leakage.

The principle is that critical judgment cannot be replaced by unrelated strengths.

What a Good Answer Looks Like

Read as a diagnostic rather than a second scorecard, the framework becomes clear: Target — Insufficient: Uses the supplied label; Competent: Defines estimand or decision; Advanced: Compares alternative targets. DGP — Insufficient: Describes columns; Competent: Explains selection and measurement; Advanced: Anticipates feedback and shift. Model — Insufficient: Names an algorithm; Competent: Justifies representation; Advanced: Compares models under assumptions. Validation — Insufficient: Reports test accuracy; Competent: Matches split and metric to deployment; Advanced: Designs stress and falsification tests. Interpretation — Insufficient: States output; Competent: Defines uncertainty and scope; Advanced: Explains what remains unidentified. Adaptation — Insufficient: Reuses template; Competent: Revises after perturbation; Advanced: Predicts how the solution must change.

This rubric can be shared with students. Transparency does not make the assessment easy; it directs preparation toward the intended competence.

Admission, Course, and Dissertation Assessments

Different stages require different emphases.

An admission assessment should test readiness: mathematical foundations, disciplined reading, and ability to learn from a structured problem.

A course assessment should test transfer: can the student apply familiar tools when assumptions change?

A dissertation assessment should test integration: can the student formulate a defensible question, assemble evidence, select methods, revise after criticism, and communicate limitations?

Using one examination style for all three confuses the purpose of measurement.

AI Tools and Assessment

Generative systems can produce code, summaries, and standard derivations. Assessment should not respond by turning entirely into surveillance or memory testing.

• verify every assumption and transformation;
• explain why the output answers the question;
• identify a hidden error inserted into a generated solution;
• adapt the solution to a changed DGP;
• defend the work orally; and
• disclose tool use and preserve intermediate reasoning.

When tools lower the cost of execution, assessment should place more weight on verification, adaptation, and responsibility.

Building an Assessment from a Claim

A practical way to design an item is to begin with the claim a successful student should be able to defend.

The student can determine whether a predictive model should be used to allocate a limited intervention.

The item can then be built backwards.

First, define the evidence required. The student must identify the population, estimate risk, distinguish risk from treatment benefit, evaluate asymmetric costs, and impose a capacity constraint.

Third, add a perturbation. Capacity changes, treatment effects are not identified, one feature is measured after intervention, or the deployment population differs from the sample. The perturbation reveals whether the student owns the structure.

Fourth, define evidence boundaries. A strong answer should receive credit for saying that individual treatment effects cannot be learned from the supplied observational data without additional assumptions.

This backward design makes the item difficult for the intended reason. A student should not fail because the case uses obscure vocabulary, requires excessive typing, or hides an irrelevant trick.

Oral Defense as Conditional Assessment

A written artifact is a fixed response to one problem. An oral defense can test how the student’s reasoning changes when new information arrives.

• the label definition changed;
• the sample excluded a relevant population;
• a feature is unavailable at deployment;
• a stakeholder changed the cost ratio;
• a validation result reverses at another site; or
• a generated citation cannot be verified.

Consistency matters because oral assessments can otherwise reward confidence, language fluency, or familiarity with the examiner. A program should use a shared prompt bank, explicit rubrics, more than one assessor for high-stakes decisions, and written notes linking judgments to observed evidence.

Tool-assisted work makes the defense more useful, not because it is a detector, but because it tests ownership. A student may legitimately use a library or generative system and still demonstrate independent competence by explaining, criticizing, and adapting the result.

Evidence from the Assessment Itself

• Did the item separate students on the intended capability?
• Did responses reveal a recurring misconception?
• Was failure driven by irrelevant language or interface burden?
• Did different assessors apply the rubric consistently?
• Did later project performance agree with the interpretation of the score?

If item intended to measure model judgment, but performance is explained mainly by coding speed, the item has construct-irrelevant variance. If all well-prepared students answer correctly, the item may still be valuable as a mastery check; it should not be used to distinguish advanced judgment.

Assessment evidence becomes stronger when the student must adapt the same reasoning to a changed dataset, objective, or institutional constraint rather than reproduce the original solution.

Recent institutional evidence supports a process-centered design. Why AI Detection Is Failing Higher Education documents why probability scores from detection software cannot establish authorship or understanding. Teacher AI Literacy Is the Real Test of AI in Education shifts responsibility toward instructors who can design and evaluate AI-supported work, while AI Cognitive Stunting Is a Policy Failure, Not a Student Defect shows that weak assessment design can reward dependence. A good assessment therefore asks for observable judgment: formulation, verification, revision, and defense.

An Assessment Architecture for Korea

A balanced system should combine constrained examinations, open-ended cases, reproducible projects, and oral defense. Each instrument observes a different part of capability. The aim is triangulation, not a universal score.

Assessment should also include perturbation. Change the population, remove a familiar variable, introduce a distribution shift, or ask the candidate to explain why a plausible generated answer is wrong. Transfer under altered conditions reveals whether knowledge has been reconstructed or merely recognized.

National evaluation should examine the instruments themselves. Do scores predict later independent work? Do assessors agree? Are results biased toward prior software experience while missing statistical judgment? Evidence about assessment validity is necessary before scores are used to compare institutions or allocate resources.

From Mechanism to Evidence

For How should Korea evaluate AI capability, the policy mechanism should be read as a chain rather than as a headline target. A government changes funding, rules, information, or institutional incentives; organizations respond; workers and students adjust; and only then do productivity or social outcomes change. Each link can weaken the intended effect. A credible evaluation therefore identifies the behavioral response at every stage instead of treating announced spending or participation as proof of success.

Conclusion

A good AI assessment measures whether a student can construct and defend a model-based decision under uncertainty.

It includes foundational knowledge and technical execution, but it does not stop there. It requires students to interpret assumptions, reconstruct the DGP, choose validation appropriate to deployment, and change their solution when the problem changes.

Difficulty is not the objective. Evidence is the objective. An assessment is difficult for a legitimate reason when the intended competence is itself demanding.

Can the student recognize why the previous solution no longer applies?

References

AERA, APA, and NCME, Standards for Educational and Psychological Testing, 2014.
Samuel Messick, “Validity of Psychological Assessment,” American Psychologist 50, no. 9 (1995): 741-749.
David J. Hand, Measurement Theory and Practice, Arnold, 2004.
Swiss Institute of Artificial Intelligence (2026) ‘Why AI Detection Is Failing Higher Education’, SIAI AI Memo, 30 August.
The Economy Editorial Board (2026) ‘Teacher AI Literacy Is the Real Test of AI in Education’, The Economy Review, 22 June.
The Economy Editorial Board (2026) ‘AI Cognitive Stunting Is a Policy Failure, Not a Student Defect’, The Economy Review, 14 June.
Ministry of Education, Republic of Korea (2025) “AI for All: Strategy for Cultivating Artificial Intelligence Talent”.