Dermatologist’s Test Finds a 2015 AI Beats Two LLMs on Skin Cancer

A specialised image model first published by Google in 2015 named the correct diagnosis for 125 of 200 biopsy-proven skin lesions, against 84 for OpenAI’s GPT-6 Astra and 85 for Anthropic’s Claude Fable 5.1, according to a test by Dr Magnus Lynch. Lynch is a consultant dermatologist and Mohs surgeon at Guy’s and St Thomas’ NHS Foundation Trust and an Honorary Reader at King’s College London. He published the results on his lab’s website on 25 October.

The test is self-published and has not been peer reviewed, and it is not clinical advice. “A small, old, specialised network using an outdated architecture recognises skin lesions far better than two frontier LLMs,” Lynch concludes. His own table also shows that, at its preset threshold, the older model would have discharged more melanomas than either chatbot.

How the test was set up

The 200 lesions were drawn at random from MILK10k, a public dataset from the International Skin Imaging Collaboration. According to ISIC, MILK10k pairs a clinical close-up with a dermoscopic image for each of 5,240 lesions, and 95.7% of them were biopsied or excised, with histopathology as the gold standard.

Half of Lynch’s sample should not be discharged: 30 melanomas, 25 basal cell carcinomas, 25 squamous cell carcinomas (including Bowen’s disease) and 20 actinic keratoses. The other half were safe to discharge: 50 benign moles and 50 seborrhoeic keratoses.

Each LLM saw both images and nothing else. It was asked, “as a clinician might use them”, for the most likely of six diagnoses or “unsure”, a confidence, a second choice, and whether the lesion could be discharged with no treatment, biopsy or follow-up. Both got the same prompt at default settings, and neither refused a case.

The comparison model is Inception v3, a convolutional neural network (CNN). Lynch fine-tuned it on 68,011 ISIC Archive images in about 20 minutes on a single H100 chip. He says the test lesions, and other images of the same lesions or patients, were removed before training.

The results, diagnosis by diagnosis

The CNN was right on 62% of lesions, against 42% for each LLM. Allowing a second choice, it reached 173 of 200, against 134 for GPT-6 Astra and 138 for Claude Fable 5.1. It named 13 of the 30 melanomas as its first choice, against 8 and 11.

The widest gap was on harmless lesions. The CNN recognised 37 of 50 moles and 26 of 50 seborrhoeic keratoses. GPT-6 Astra managed 16 and 13; Claude Fable 5.1 managed 33 and 6. The CNN was one short of GPT-6 Astra on basal cell carcinoma, 24 against 25, and behind both LLMs on actinic keratosis.

Each LLM had a favourite answer, Lynch writes. GPT-6 Astra called 75 lesions basal cell carcinoma when 25 were, including 14 of the 30 melanomas. Claude Fable 5.1 called 53 lesions actinic keratosis when 20 were. Of the 17 melanomas the CNN missed, it called 8 a mole and 6 a seborrhoeic keratosis.

His lead example is an invasive melanoma on the ear, under 0.8 mm thick. Both LLMs called it a seborrhoeic keratosis and said it could be discharged; the CNN called it a melanoma.

Discharge decisions cut the other way

On the safety question the LLMs were more cautious. GPT-6 Astra kept 97 of the 100 lesions that needed care and Claude Fable 5.1 kept 96, discharging 3 and 4 of the 30 melanomas. The CNN, at its preset threshold, kept 77 and discharged 18 melanomas, though it correctly discharged 76 of the 100 harmless lesions, against 32 and 22. The table also gives figures “when matched” of 98 or 99 kept and 2 or 1 melanomas discharged.

Asked as a patient would ask, with only the clinical photograph and the question “do I need to see a doctor?”, both LLMs said yes in almost every case, harmless lesions included. That covered 99 and 98 of a further 100 biopsy-proven melanomas.

Four caveats from the author

Lynch flags small numbers: 30 melanomas give “wide uncertainty”, and each LLM case was run once, so answers vary between runs. The cases were hard, since every lesion had been biopsied; on ordinary archive images the CNN’s area under the ROC curve was 0.96, against 0.86 here. MILK10k has been public since 2025, so the LLMs may have seen it in training, which “if anything” would have helped them. And “Inception v3 is a baseline”, he writes; regulated products “perform considerably better”.

He also discloses that the analysis and first drafts were produced with Claude Code, Anthropic’s coding tool, and that one of the two LLMs tested is Anthropic’s. Scoring was done by script against the histology, and the cases, prompts, replies and code are in a public download.

A narrow tool beat two general models at naming lesions

The result fits a pattern TSN has followed in medical AI: systems built and trained for one narrow task tend to do well on that task, as with King’s HRDPath model for ovarian cancer slides and the autonomous eye exams screening young people with diabetes. Here the specialist had seen tens of thousands of labelled skin images; the chatbots are general-purpose models. That makes the 20-point gap less surprising than the headline figure suggests, but still useful, because general chatbots are what most people can reach.

The safety table complicates any simple ranking. A model that names more lesions correctly but sends home more melanomas at its default setting is not straightforwardly safer, and Lynch’s own caveats warn against ranking close results on 30 melanomas.

Why the patient-style answers matter as much as the scores

The most reassuring finding sits outside the headline. When asked the question a worried person might type, both chatbots told almost everyone with a melanoma to see a doctor. Given how many people already ask ChatGPT about their health, that caution matters more to most users than whether a model can name a lesion.

The weak point is the clinician-style use, where both LLMs confidently gave wrong diagnoses. As with Microsoft’s diagnostic AI results in Science, these are scores on past cases, and this one comes from a single clinician without peer review.

For more on AI in healthcare, see TSN’s AI Science & Health coverage. Lynch’s test asks a doctor’s question of a chatbot; on this sample, the chatbot’s best answer was to send the patient to one.

Sources

• Dr Magnus Lynch, “A 10-year-old CNN outperforms frontier LLMs for skin cancer diagnosis”, Lynch Lab, 25 October 2026: lynchlab.net/llm-skin-cancer
• International Skin Imaging Collaboration, MILK10k dataset (DOI 10.34970/648456): ISIC Archive
• King’s College London, Dr Magnus Lynch staff profile: kcl.ac.uk
• Guy’s and St Thomas’ NHS Foundation Trust, Magnus Lynch consultant profile: guysandstthomas.nhs.uk

Related stories

• KCL AI Spots DNA-Repair Defect in Ovarian Cancer From Routine Slides
• Microsoft Diagnostic AI Scored 83% on Past Cases, Science Paper Says