Why Medical School Is Such a Hard Test for AI Tutors

In early 2023, a study in PLOS Digital Health reported that ChatGPT performed at or near the passing threshold on all three steps of the US Medical Licensing Examination. It made headlines, and it raised an obvious question. If a chatbot can score near the pass mark, can it also tutor someone toward it?

Not automatically. Scoring well on an exam and teaching someone else to do the same are different jobs, and medical education is a good place to see why.

What makes medicine a hard market for AI tutoring?

Three things, mostly.

The first is the cost of a confident wrong answer. A tutor that misexplains a grammar rule wastes an evening. A tutor that misexplains a drug interaction can leave a student with an error they don’t know they’re carrying. Language models can still produce fluent mistakes, and a student who is still learning the material is not well placed to spot them.

The second is the curriculum. Medical schools don’t teach a generic body of knowledge in a generic order. In the US, the accreditation standards for MD programs give an institutional body, such as a faculty committee, responsibility for a coherent and coordinated curriculum. A tutor that ignores how a school has sequenced its courses can be accurate and still unhelpful, answering at the wrong depth or weeks ahead of the class.

The third is trust. Faculty decide what gets recommended to students. If they can’t see what a tool is telling students, it’s hard for them to stand behind it.

Where AI tutors could earn a place

None of this makes the category a dead end. There are jobs an AI tutor could do well, and they’re jobs faculty rarely have time for. Each is worth testing rather than assuming.

Re-explaining. A student stuck on renal physiology late at night doesn’t need another lecture. They need the same idea from a different angle, and then another. Software doesn’t run out of patience.

Practice on demand. Writing a fresh question on a weak topic, then explaining why each wrong option is wrong, takes a person real time. A model can draft one in seconds, though faculty should review both the question and the explanation before students rely on them.

Socratic follow-up. A well-designed tutor doesn’t just hand over the answer. It asks the student what they think happens next, which keeps the student doing the thinking.

Grounding is the product

The interesting split in this market isn’t between brands. It’s between two designs.

One is a general chatbot answering from whatever it learned in training. The other is a tutor built into a medical education platform that also carries video lectures, question banks and faculty analytics. That second design gives a curriculum committee something concrete to evaluate. It should still ask what content informs the tutor’s answers, how faculty can check them and what the reporting actually covers.

There’s a side benefit that rarely makes the sales deck. The questions students ask a tutor late at night can point to where teaching isn’t landing. If one topic keeps coming up week after week, a course director might rework a session, rewrite an exam item or give a concept a second pass in small groups. Repeated questions don’t prove a course has failed, but they’re a prompt worth reading. That only works if the tool shares what it sees with faculty, with sensible privacy rules for students.

The AAMC has published principles for the responsible use of AI in medical education. They put people at the center, ask for transparency about how tools work, call for equal access for learners and stress faculty development. A tool that faculty can see into fits those principles more easily than a black box.

What should schools ask before a pilot?

Vendors demo their best answers. The useful questions are about the worst ones.

• Where do the answers come from: reviewed content, the open web or a mix?
• What happens when the tutor is wrong, and how does a student or faculty member flag it?
• Can faculty see questions grouped by topic, so the tool tells them something about their own teaching?
• How does it line up with our curriculum map and assessment calendar?
• Does every student get the same version, or does the useful one cost extra?

Start with one course, one faculty owner and a defined evaluation plan before anyone talks about a campus-wide license. Pick a block with a known pain point, such as high remediation rates. Track how students use the tool, and compare exam results with the previous cohort as a rough check, keeping in mind everything else that changed between the two years.

Decide in advance what counts as success. Exam performance is the obvious measure, but it isn’t the only one. Faculty time spent on repeat explanations, the number of students seeking remediation and how often students flag a wrong answer are all worth tracking. Student enthusiasm alone shouldn’t decide whether the school renews.

The real exam

The 2023 study made a good headline. It showed performance at or near the passing threshold on those exams. Whether an AI tutor helps a student become a better doctor is a harder question, and benchmark scores won’t answer it.

Schools can find out course by course, by putting these tools in front of students with faculty in the loop and measuring what changes. That’s slower than the hype cycle would like. In medical education, caution is worth more than a strong headline.