None of 6 AI models met a research group's staff standard on real tasks
Epoch AI, a research group, says it graded 6 AI models by hand on tasks from its own work. In its report, none reached the standard it expects of its employees, and it describes where they struggled.
Six AI models were given real tasks from a research group's own work, and none met the group's employee standard on any task, Epoch AI says in its report. They were Fable 5.2, GPT-6.2 Astra, Opus 5.5, Gemini 4 Argon, Fable 5.1 and GPT-6 Astra.
Epoch graded every output by hand, and 100% means the standard it expects of its employees. The tasks come from its benchmark, Epoch Automation Reports, a fixed set of test tasks.
GPT-6.2 Astra averaged 65% across the tasks. Epoch says that is 17 points below Fable 5.2, though the two roughly tie on its own index, the Epoch Capabilities Index.
The closest result came on a data centre task: adding a data centre to Epoch's AI Data Centers database. The work means researching news and regulatory documents, ordering satellite images and analysing them.
Fable 5.2 scored 97.5% and Opus 5.5 scored 87.5%. Epoch says its data centre analyst judged both outputs comparable to what contractors deliver, and contractors' work also usually needs some revision.
Epoch says a contractor typically spends about 16 hours per data centre. Opus 5.5 took just under 3 hours at a lower cost than a contractor. Fable 5.2 took around 6.5 hours at a higher cost, for a better result than Opus 5.5.
A second task is based on a chess test Epoch used to give job candidates. Each model had to write a research report on how well AI models cope with situations unlike anything in their training, using versions of chess with altered rules.
Epoch says every model used chess variants that already exist instead of creating its own, and these do not test what the task asked about. Fable 5.2 created a variant called Kingsgate, which Epoch says plays exactly like an existing variant, Midline Invasion.
Epoch says all the models mostly relied on their first attempt, rather than reviewing early results and trying again.
Asked to compare their own output with a human output that met Epoch's standard, without being told which was which, the models preferred their own in every case except one. Gemini 4 Argon preferred the human output on the hidden messages task.
Epoch names the limits of its test: each model ran once per task on a limited set of tasks, so scores are noisy, and many grading criteria involve subjective judgement.
The group says its qualitative findings are more informative than the exact numbers, and the results are not a general measure of what AI can or cannot do across all work tasks.
On its data centre work, Epoch says it is considering shifting contractors' time towards reviewing AI outputs and other parts of its analysis.