Tarek Halteh

Let’s chat

Why Most AI-in-Education Pilots Never Become Policy (and How to Design One That Does)

Every year, education systems around the world launch AI-in-education pilots. A tutoring chatbot in a handful of schools. An automated grading tool tested with a few hundred teachers. A lesson-planning assistant rolled out to one district.

Most of them end the same way. A launch event, a few photos, a final report, and then nothing. The tool quietly disappears when the funding cycle closes, and the ministry that hosted it is no closer to a decision than before.

The problem is rarely the technology. Many of these tools work reasonably well in the classroom. The problem is that the pilot was designed to prove that something could be done, not to give a decision-maker what they need to decide whether it should be done at scale.

After years working on donor-funded programs and the reporting that goes with them, I’ve come to believe the gap between a pilot and a policy is almost always built in on day one.

What the latest research says about AI in education

The frustrating part is that the evidence on AI in education has never been more promising. It has also never been clearer that how AI is designed and studied decides whether it matters.

The promise is real. A World Bank randomized controlled trial in Nigeria gave secondary students a six-week after-school English program built around a GPT-4 tutor. Students gained about 0.31 standard deviations, which the researchers estimate is equal to 1.5 to 2 years of typical schooling. That puts it among the most cost-effective education programs evaluated in low- and middle-income countries (World Bank, 2025).

At Harvard, a randomized trial with 194 physics students found that a custom AI tutor, built on active-learning principles, produced more than double the learning gains of a strong in-class lesson, in less time (Kestin et al., Scientific Reports, 2025).

The warning is just as real. In a field experiment with nearly 1,000 high school math students, unrestricted access to a standard ChatGPT-style tool helped students during practice. But once access was removed, they scored 17% worse than students who never had it. A version with learning guardrails largely avoided this harm (Bastani et al., PNAS, 2025).

The OECD’s Digital Education Outlook 2026 reaches the same conclusion at a global level. General-purpose AI tools can improve the work students produce without improving what they learn. Tools designed with a clear teaching purpose, and co-designed with teachers, tend to show lasting gains (OECD, 2026).

And the evidence base is thin. Stanford’s SCALE Initiative reviewed more than 800 papers on AI in K-12 education. Only 20 met the bar for strong causal evidence (Stanford SCALE, 2026). In other words, most of what is being tried is not producing evidence anyone can build policy on.

Notice one more detail. The World Bank team is careful to say its results reflect the whole package: structured prompts, curriculum alignment, teacher support and peer learning, not the chatbot alone. That is exactly the kind of finding a policymaker needs, because it says what must be funded at scale, not just which tool to buy.

The same technology produces opposite results depending on how it is designed and supported, which is why a pilot’s design matters as much as the tool it tests.

Why pilots stall

1. Nobody defined the question. “Test AI in classrooms” is not a research question. “Does a feedback tool cut teachers’ marking time by a third without lowering grading accuracy?” is. Without a sharp question, the pilot produces activity, not answers.

2. The audience for the evidence was never named. A teacher, a donor and a finance ministry need very different proof. Teachers want to know it saves time. Donors want outcomes they can report. Ministries want cost per student and what happens when outside funding stops. Most pilots collect evidence for the first group and present it to the third.

3. There was no baseline. If you didn’t measure where learners and teachers stood before the tool arrived, you can’t credibly claim it changed anything. Satisfaction surveys collected at the end are not a substitute.

4. The pilot ran on conditions that can’t scale. Extra staff, hand-picked schools, enthusiastic principals, free vendor licenses and reliable internet. Remove any one of these and the results may not hold. Decision-makers know this, so they discount the findings.

5. The timeline didn’t match the policy calendar. Budgets, curriculum reviews and strategy cycles happen on fixed dates. A pilot that reports six months after the budget is set has missed its window, however good the results.

What policy-ready pilots do differently

The pilots that do shape policy tend to share a few habits. None of them require more money. They require more discipline at the start.

  1. They begin with the decision, not the tool. Before choosing any technology, they ask: what decision will someone make at the end of this, and what would they need to see to make it?
  2. They write an evidence plan alongside the project plan. It answers three questions: what we must demonstrate, to whom, and by when. Every activity is checked against it.
  3. They test under realistic conditions. That means average schools, not showcase ones, normal staffing levels, and the connectivity most classrooms actually have. Weaker but honest results beat strong results nobody believes.
  4. They cost it from day one. A pilot that can’t say what it would cost per learner at national scale, after the donor leaves, is not giving a ministry enough to act on.
  5. They build governance in early. Who owns the data? What happens if the vendor raises prices or exits the market? How are teachers and parents informed? Answering these late is how promising pilots get shelved.
  6. They bring decision-makers in from the start. When officials help shape the questions, they trust the answers. When they only see the final report, they treat it as someone else’s project.
  7. They report what didn’t work. Honest findings, including failures, are what make the successes credible. Overselling a pilot is the fastest way to lose the next one.

Each stage of a policy-ready pilot ends with a gate question: would a minister ask this question, will results land before the budget, would results hold without extras, is it affordable after the donor leaves, and do officials trust the findings? A pilot that fails any of them usually ends as a one-off project, however well the technology performed.

Not sure your institution is ready? Before designing a pilot, take my free AI Readiness Assessment, a quick self-check of how prepared your organization is to adopt AI.

Prompt kit: use AI to design a policy-ready pilot

You can use any capable AI assistant (Claude, ChatGPT, Gemini) as a thinking partner for this work. Replace the text in [brackets] with your own details. Treat the output as a first draft to challenge, not a final answer.

1. Turn a vague idea into a decision question

I'm planning an AI-in-education pilot: [describe the tool and setting]. Rewrite this as 3 sharp research questions that a ministry of education could use to make a funding decision. For each, state the outcome to measure, the comparison group, and what result would justify scaling.

2. Map who needs what evidence

For a pilot of [tool] in [setting], list the key decision-makers (e.g., teachers, school leaders, ministry officials, finance, donors). For each, describe what evidence would convince them, in what format, and by what date in a typical budget or planning cycle.

3. Draft an evidence plan

Act as a monitoring and evaluation advisor. Draft a one-page evidence plan for [pilot]. Include: baseline measures, outcome indicators, data collection methods, sample size considerations, timeline, and the risks to credible findings. Flag anything that would make results hard to generalize.

4. Stress-test for scale

Here is my pilot design: [paste design]. List every condition in it that would not exist at national scale (staffing, connectivity, school selection, free licenses, extra support). For each, suggest how to test under more realistic conditions.

5. Estimate the cost at scale

Help me build a simple cost model for scaling [tool] to [number] students over [years]. Include licenses or API costs, devices, connectivity, teacher training, support staff and evaluation. Separate one-time from recurring costs, and show cost per learner per year. List the assumptions I need to verify.

6. Build a governance and vendor-risk checklist

Create a governance checklist for an AI tool used with [age group] students. Cover data ownership, privacy and consent, content safety, bias, teacher oversight, vendor lock-in, exit clauses, and what happens when donor funding ends.

7. Design learning guardrails

I want students to use [AI tool] for [subject] without using it as an answer machine. Propose design rules and a system prompt that make the AI guide students through reasoning, ask questions, and avoid giving direct answers. Include how teachers can monitor use.

8. Write the policy brief backward

Imagine this pilot has finished: [describe pilot]. Write the 150-word summary a minister would read to decide whether to scale it. Then list the evidence we must collect during the pilot so every sentence in that summary is supported.

Frequently asked questions

Does AI actually improve student learning?

It can. Randomized trials from the World Bank in Nigeria and from Harvard show strong gains when AI tutors are designed around good teaching. Unstructured, general-purpose chatbots can improve the work students hand in while weakening what they learn.

Why do AI-in-education pilots fail to scale?

Most are designed to show that a tool works, not to answer a decision-maker’s question. Common gaps are no baseline, unrealistic conditions, no cost model, weak governance and results that arrive after budgets are set.

What makes an AI pilot “policy-ready”?

A clear decision question, an evidence plan tied to specific audiences, testing under normal school conditions, a cost per learner at scale, and governance answers before launch.

How long should an AI pilot in education run?

Long enough to measure learning, not just usage, and timed so results land before the relevant budget or planning decision. Short trials can show early effects, but persistence over time needs longer follow-up.

What is the biggest risk of AI in schools?

Using AI as an answer machine. Research shows students can become dependent on it and perform worse once it is removed. Guardrails and teacher oversight reduce that risk.

From experiments to evidence

AI will reshape how students learn and how teachers work. That part is not in doubt. What is in doubt is whether education systems will adopt it through deliberate, evidence-based decisions, or through a patchwork of disconnected pilots and vendor pitches.

The difference comes down to how we design the early work. A pilot should not be a demonstration. It should be the first chapter of a policy argument, built so that whoever reads the final page can act on it.

If you’re designing or funding an AI initiative in education right now, try one test before you start: write the paragraph you hope a minister will read at the end. Then work backward and make sure everything you do will earn it.

Sources

Tarek Halteh is a communications and program professional with 14+ years of experience across technology, international development and the MENA region. He writes about AI adoption, evidence and the gap between innovation and institutions.

Leave a Reply