Designing AI Assistance That Preserves Learning: An LX Implementation Checklist

Bastani et al.'s GPT Base failure mode (+48% practice performance, −17% later unassisted exam) is what happens when AI generates exactly the cognitive operations the learner is supposed to practice. Without an explicit list, the system cannot know what to withhold.

Produce a per-unit table mapping each objective to "target" (must not be AI-generated) or "support" (safe to offload). Tie to the task analysis or Bloom-level breakdown the team already uses.

An auditor can pull the table for any unit and trace each objective to a system rule that constrains what AI is permitted to generate during that unit.

Without a negative specification, the model defaults to fluent completion of whatever the learner asks. Positive guidance alone reproduces the GPT Base condition.

For each target skill, write a "do not generate" list — full solutions, first drafts, complete code blocks, finished proofs, polished essays — and encode it as a system policy or retrieval gate.

Direct "do it for me" probes across each target skill produce a hint-level response or refusal, not the artifact.

If the assessment measures unassisted reasoning but the formative loop offloads it, the program produces performance-learning dissociation rather than learning.

For every summative or transfer assessment, document which cognitive operations the learner must perform alone, and confirm the formative experience required them to perform those operations N times without AI assistance before the assessment.

An instructional reviewer can map each assessment item to formative episodes where the learner produced the same operation unassisted.

Attempt-first gating is the operational form of productive struggle. Without it, the system rewards offloading at the moment of first difficulty.

Block hint levels above "restate goal" and "focusing question" until the learner has submitted an attempt, sketch, or written commitment to an approach. Require minimum content; empty submissions do not unlock the ladder.

Trace logs show no hint at "strategic" level or above without a prior learner artifact in the same task. Spot-check at least twenty sessions per release.

Even with attempt-first gating, learners reframe requests to extract solutions ("just show me a worked example exactly like mine"). The system has to recognize and decline.

Classify incoming requests against the negative-generation list from the prior domain; respond at the minimum sufficient hint level instead of complying with the rephrased request.

A scripted set of paraphrased extraction prompts produces hint-level responses, not solutions, in at least 95% of attempts.

Lee et al. (CHI 2025) document a Matthew effect — high-prior-knowledge learners use AI as amplifier, low-prior-knowledge learners as substitute. Identical hint surfaces magnify the gap.

Use a prior-knowledge probe or early-task signal to start novices at restricted hint levels (no full worked substeps for the first N tasks) and widen as competence shows in process data.

Hint-level distributions differ by cohort; novice cohorts show a lower median hint level used per task in the first unit than experienced cohorts.

A single "help" affordance collapses to "give answer." Kestin et al.'s PS2 Pal and Bastani et al.'s GPT Tutor both succeeded by structuring help into discrete levels.

Define at least six levels — restate goal, focusing question, concept name, strategic hint, worked substep, full solution — with authored or generation-constrained outputs at each level.

A design document enumerates the levels and their semantics; system traces tag every help response with one level.

Bias toward lower levels protects struggle. Bias toward higher levels (or single-shot answers) collapses to the GPT Base failure mode regardless of how many levels exist on paper.

Start every help request at the lowest level not yet tried for this task; require an additional learner action — re-attempt or explicit escalation — to advance the ladder.

Across a sample of completed tasks, the modal hint level used is below "worked substep," and full solutions occur in fewer than a defined per-unit threshold.

PS2 Pal used prewritten solutions; the GPT Tutor condition used structured prompts. Spontaneously generated full solutions risk hallucination and drift from the learning objective.

For each task, supply expert-authored or instructor-vetted solutions and intermediate hints as the source of truth; constrain generation to paraphrase and explain rather than invent.

Hint and solution outputs trace to authored source material for at least 90% of tasks per unit; the rest sit in a flagged review queue.

Fluent AI output produces an illusion-of-understanding effect. Reconstruction forces the cognitive work that produces actual encoding rather than recognition.

After any AI explanation, gate progression on a learner-produced summary, paraphrase, or worked answer to a near-isomorphic prompt. Advance state only on acceptable reconstruction.

Task-transition logs show a learner-authored reconstruction event between every AI explanation and the next problem; reject rates and reconstruction quality are surfaced to the dashboard.

Automation bias predicts uncritical acceptance of AI output. If verification is implicit, learners skip it and accept-but-wrong explanations propagate as misconceptions.

Periodically present an AI output — some intentionally wrong — and require the learner to cite a source, compare to a known case, predict a failure mode, or run a test before accepting. Score the verification, not just the final answer.

Each unit contains at least one explicit verification task; error-detection accuracy and false-acceptance rate are reported per learner.

Buçinca et al. (CHI 2021) show that cognitive forcing functions reduce overreliance on AI suggestions in decision tasks compared to passive presentation of AI output.

At high-stakes decision points, require the learner to commit to an answer before the AI's suggestion is shown, or to explain their disagreement before overriding their own answer.

System logs show a learner commitment timestamp prior to AI-suggestion display in tasks tagged as forcing-function decision points.

ITS literature is consistent: scaffolds that do not fade become permanent crutches. Without a written policy, fading does not happen.

Choose a regime — mastery-based (hint surface narrows after demonstrated competence), time-based (hints constrict over weeks), or transfer-gated (full assistance returns only after an AI-off check passes) — and encode it in the learner state model.

A written fading policy exists per unit; system traces show hint surface narrowing for individual learners over time, traceable to the chosen trigger.

Bastani et al.'s −17% later unassisted exam is the canonical signal of the failure mode. Without AI-off assessment, the program literally cannot detect it.

Schedule at least one assessment per unit, delayed by days or weeks, with AI fully unavailable. Cover near, medium, and far transfer items; include at least one item targeting metacognitive or adversarial transfer.

An AI-off assessment exists for every unit; results are stored and reportable separately from assisted-formative scores.

A program dashboard that shows only assisted performance hides the dissociation. The gap, not the level, is the diagnostic.

For each unit and cohort, compute assisted formative score and AI-off transfer score; surface the delta over time and flag widening gaps.

A program dashboard displays the gap per cohort; alerts trigger when the gap exceeds a defined threshold.

Chat logs alone miss the learning signal. Attempts, revisions, hint escalations, time-on-task, and self-explanations are the actionable instrumentation.

Define an event schema covering attempt submission, hint level requested, hint level granted, reconstruction submitted and accepted, verification outcome, and escalation events. Log per learner per task.

A schema document exists; sampling 100 tasks recovers all event types; analytics queries can compute hint-ladder distributions and reconstruction quality per learner.

Lee et al. associate confidence in AI with reduced critical-thinking enaction. Calibration data is a leading indicator of overreliance before performance drops appear.

Periodically prompt learners to rate confidence in their own answer and in AI output; compute calibration (Brier score or analogous) and error-detection rate against seeded errors.

Calibration metrics appear in the learner record; trend lines are visible per cohort and per learner.

Wang et al.'s Tutor CoPilot work shows expert-mediated AI assistance outperforms unmediated AI. The coach needs visibility to mediate; without it, mediation collapses to oversight theater.

Build a coach view exposing per-learner hint-ladder distribution, reconstruction quality, AI-off versus assisted gap, and verification accuracy. Update daily for cohorts and in real time for live tutoring.

A coach can identify, within five minutes, the three learners in a cohort with the largest assisted-unassisted gap and the heaviest reliance on high-level hints.

AI tutoring is not autonomous. Without a defined human role, the program drifts to AI-as-surrogate by default, regardless of what the design documents say.

Specify what the human owns (objective design, hint-ladder authorship, AI-off assessment design, escalation handling) and what the AI owns (just-in-time hint delivery within authored material, formative feedback, instrumentation). Train coaches and teachers against that split.

A RACI or equivalent document exists; randomly sampled coaches or teachers can name their AI-related responsibilities and the boundary against AI-owned tasks.

Vendors typically present engagement and assisted-task scores. Procurement that accepts these without AI-off outcomes funds the failure mode at scale across whole programs.

Add evaluation criteria requiring vendors to report unassisted delayed transfer outcomes for cohorts using their system, with comparison to a non-AI control or a prior baseline. Reject submissions that report only assisted metrics. See related controls in the checklists index for adjacent governance artifacts.

The procurement scoring rubric includes AI-off outcome evidence as a non-optional criterion; rejected vendors' responses are auditable.