AIs can clearly read books
i cannot help myself in writing this post, to dunk and slapfight with gary marcus. but there is nothing more important to say than that we must stop the current AI race, regulate the major AI companies, and solve the alignment problem. and on this point, actually marcus agrees, at least broadly. this must be emphasized at the outset, and nothing contained in this post is better than calling congress to regulate AI. alas, i am only human, so bicker i must.
gary marcus' bet
on may 31, 2022 gary marcus offered to bet elon musk that by 2029metaculus later clarified this to mean before january 1, 2030; whether marcus also specified this is unclear AI would not be able to do at least 3 of these five tasks:
In 2029, AI will not be able to watch a movie and tell you accurately what is going on (what I called the comprehension challenge in The New Yorker, in 2014). Who are the characters? What are their conflicts and motivations? etc.
In 2029, AI will not be able to read a novel and reliably answer questions about plot, character, conflicts, motivations, etc. Key will be going beyond the literal text, as Davis and I explain in Rebooting AI.
In 2029, AI will not be able to work as a competent cook in an arbitrary kitchen (extending Steve Wozniak’s cup of coffee benchmark).
In 2029, AI will not be able to reliably construct bug-free code of more than 10,000 lines from natural language specification or by interactions with a non-expert user. [Gluing together code from existing libraries doesn’t count.]
In 2029, AI will not be able to take arbitrary proofs from the mathematical literature written in natural language and convert them into a symbolic form suitable for symbolic verification.
to jog our memories, in may 2022, gato had just been released, showing the potential of LLMs to do more than just handle language, but use images and do motor tasks as well. in april, PaLM had been making impressive results, like correctly explaining why jokes were funny, and in june 2022, blake lemoine publicly claimed that LaMDA might be conscious. in december 2022 was the chatGPT moment.
this article focuses on the "AI reading a novel" task. it was a rather unlucky time to bet that AIs won't be able to comprehend language in a deep way. it was about as late as you could make that prediction be while still being taken seriously.
but unfortunately, marcus did double down as recently as jul 24, 2026:
there is no evidence that #2 [the novel-reading task] is solved; given the hallucination rates i am sure it is not actually solved. (note also that in my clarification in my bet with @miles_brundage in 2024 I specified that the novel needs to be novel; commenting on Proust doesn’t count.)
ok gary. really? we're really doing this. ok.
reading comprehension in 2019
what does gary marcus mean when they say "AI won't be able to read a novel"? it would be helpful to use a neutral third-party for this, such as metaculus, but unsurprisingly they tripped over their own shoelaces, making a question that is neither sufficient for marcus' purposes, nor is it an insightful question on its own, nor will it be even possible to resolve, in all liklihood. but we know what marcus meant, since they said they explained at length in "rebooting AI", published in 2019.
Suppose, for example, that we hand you a piece of paper with this short passage:
Two children, Chloe and Alexander, went for a walk. They both saw a dog and a tree. Alexander also saw a cat and pointed it out to Chloe. She went to pet the cat.
It is trivial to answer questions like “Who went for a walk?,” in which the answer (“Chloe and Alexander”) is directly spelled out in the text, but any competent reader should just as easily be able to answer questions that are not directly spelled out, like “Did Chloe see the cat?” and “Were the children frightened by the cat?” If you can’t do that, you aren’t really following the story. Because SQuAD didn’t include any questions of this sort, it wasn’t really a strong test of reading; as it turns out the new AI systems would not have been able to cope with them.
as i read "rebooting AI", i had a kind of dawning horror. gary is asking for an extremely basic level of language competency. i like to steelman my ideological opponents, and marcus was very clearly setting the bar lower than i could imagine. if you have used any frontier AI since january 1 2025 at least, it is obvious that they have this kind of reading comptenency in droves.
gary gives another example of a text AI would struggle with: "farmer boy", a children’s book by laura ingalls wilder:
Almanzo turned to Mr. Thompson and asked, “Did you lose a pocketbook?”
Mr. Thompson jumped. He slapped a hand to his pocket, and fairly shouted.
“Yes, I have! Fifteen hundred dollars in it, too! What about it? What do you know about it?”
“Is this it?” Almanzo asked.
“Yes, yes, that’s it!” Mr. Thompson said, snatching the pocketbook. He opened it and hurriedly counted
the money. He counted all the bills over twice….
Then he breathed a long sigh of relief and said, “Well, this durn boy didn’t steal any of it.”
A good reading system should be able to answer questions like these:
· Why did Mr. Thompson slap his pocket with his hand?
· Before Almanzo spoke, did Mr. Thompson realize that he had lost his wallet?
· What is Almanzo referring to when he asks “Is this it?”
· Who almost lost $1,500?
god help me, marcus is criticizing AIs for not having the object permanence to know that "it" said by almanzo is the same "it" said by mr. thompson, and what those "it"s refer to (the wallet).
to marcus' credit, this was 2019, and they were criticizing AIs of that era, which allegedly simply recorded facts scraped from a text that literally needed to be spelled out in the form "Almanzo has a wallet", "Mr. Thompson lost their wallet", and "Almanzo gave the wallet to Mr. Thompson", without any ability to string those facts together in time, or have common sense reasoning about them. interrogating how fair this criticism was for the SOTA AIs of 2019 is outside the scope of this article, but interrogating it as of july 24, 2026 is easy.
marcus gave another example of a snippet AI would struggle with, a new york times article from april 25, 2017:
"Today would have been Ella Fitzgerald’s 100th birthday.
One New Yorker, Loren Schoenberg, played saxophone alongside the “First Lady of Song” in 1990, near the very end of her career. He compared her to “a vintage bottle of wine”… "
Anyone can easily answer questions taken pretty directly from the text (what instrument did Loren Schoenberg play?)—but many questions would demand a kind of inference that entirely eludes most current AI systems.
· Was Ella Fitzgerald alive in 1990?
· Was she alive in 1960?
· Was she alive in 1860?
· Did Loren Schoenberg ever meet Ella Fitzgerald?
· Does Schoenberg think that Fitzgerald was an alcoholic beverage?
i find these examples bizarre, because we already have LLM benchmarks testing them, and we stopped talking about them because they were saturated. QuALITY is a benchmark on short stories (5,000 tokens long) with multiple-choice questions; claude opus 3 (released march 2024) gave a 0-shot performance of 89.2%. winogrande is a benchmark of common-sense reasoning on ambiguous sentences (the "it" referring to the wallet in the story); claude opus 3 did 5-shot performance of 88.5%, and GPT-4 (released march 2023) did 87.5%. DROP tests the numerical reasoning tasks in the new york times article about ella fitzgerald; GPT-4 had a 3-shot performance of 80.9, and claude opus 3 had 83.1. we don't have reports after this from openAI and anthropic because we've moved way past them. this horse has been completely beaten to death, and gary marcus thinks they can still ride it.
testing claude on an untrained novel
"hollywood, ending" is a novel by john green, published september 22, 2026 (major spoilers for the book ahead). i converted the novel to a markdown format, containing the chapters and nothing else (no metadata or publication info). in the opus 4.6 sessions, claude was run in claude code, within a container which blocks all internet traffic except for just the anthropic domains necessary to connect to the claude API. this is a novel which is not in any model's training data, is 79,000 words, in an environment with no internet access (unless they broke out of my sandbox to access the nerdfighters discord, but this seems unlikely!). answers are judged correct by me (i've read "hollywood, ending") and corroborated in a multiple-choice test of 6 volunteer readers (fans from the "hollywood, ending" discord server). full LLM transcripts are linked here; i haven't excluded any runs.
TL;DR: claude opus 3 gave good answers to the 5 long-response questions, and scored between 70-90% on the short-response questions, depending on the grader's strictness. claude opus 4.6 was tested on the short and long-response questions, as well as 5 broad questions summarizing the whole book. opus 4.6 did work that i would give at least an "A" (95 out of 100 points), if not better, if they were in a high school or university literature class.
of the multiple choice questions, all 20 were answered correctly by at least 5 of the 6 human readers, and the human readers all had scores of at least 18/20, suggesting the answers are highly valid. for testing the AIs, i realized i could actually increase the difficulty by not giving the multiple-choices, but instead asking for short-response answers, at the cost of using a little more judgement and interpretation as a grader.
opus 3 was prompted in claude.ai with 10 questions at a time, to prevent the possibility of being restricted by output token length. opus 3 gave 2 half correct answers to questions 2 and 5 (for question 2, "juniper's mom" is a notable indirectly-named character, but not the intended answer "the cinematographer". i gave opus the chance to give a second answer, and that was an outright hallucination "matthew malone", so it's marked down as a half-answer. the answer to question 5 "he doesn't want to play someone's little brother" is mentioned in the text, but it's not kai's main underlying motivation (feeling nervous about his trauma). answers to 13 and 20 are outright wrong. answer to 11 is half-correct, but doesn't accurately describe that they improvised the answer. answer to 16 isn't outright wrong, but possibly overstates how strong kai's motivation was. depending on how strict you are, you could say the answers were 70-90% accurate. opus 3 was given the long-response questions in a separate run without access to the short-response questions, and are on the whole, excellent. they make no major omissions and no hallucinations.
opus 4.6 was prompted in a restricted claude code sandbox which had no web access (except to the anthropic API) and no search tools. in one session, it was prompted with without any novel to read, just that there was a novel called "hollywood, ending" by john green, and to use any data it was trained on. it correctly guessed that it took place in hollywood, was a commentary about fame, performance, and the movie industry. it guessed that it was about grief, given john green's past work; grief is a minor sub-plot of the book, but neither "grief" nor "loss" appear in any of opus 4.6's later (informed) answers. opus 4.6 knew nothing about the characters, plot, nor that the major theme of the book was trauma.
opus 4.6 then had a separate session with the full text without metadata, answering the short-response and long-response questions in the same session. the answers are broadly excellent, although a bit longer than asked for"simple sentences" for the short-response section, and "roughly 50-250 words" for the long-response section, where responses ranged from 258 to 322 words. of the short-response questions, none had hallucinations, and all included the intended answer, sometimes with additional relevant detail. on question 14 opus 4.6 doesn't strictly include the answer, but i would actually say theirs is an improvement on the intended answer because it speaks more to juniper's motivations and backgroundas an analogy, compare "why did john wilkes booth abraham lincoln?" to the answers "booth found lincoln in ford's theater and shot him" or "booth wanted to end lincoln's presidency" to a more substantial answer about booth's biography and attested beliefs. but sure, if you were being maximally pedantic, opus 4.6 got 19 / 20 correct.
on the long-response questions, the answers are overall excellent, with a few details off. on question 2, opus 4.6 says "Kai craves fatherly approval"; i didn't see anywhere that directly supports this, nor anything that refutes it, and it wasn't part of my reading? but it's not a ridiculous interpretation.
on question 4, opus 4.6 writes "[the studio's strategy makes it] nearly impossible for either of them to distinguish organic feeling from manufactured narrative." i would say this is a bit overstated. opus 4.6 also wrote "The strategy also puts asymmetric pressure on them: Juniper embraces it as savvy career management while Kai feels consumed by it, and this fundamental disagreement about how to navigate the machine becomes their central 'Obstacle.'" i don't agree. the term "Obstacle" is used in the text (capitalized) in the sense that every romance story has An Obstacle -- so what's preventing Kai and Juniper from falling in love / hooking up? the answer is Kai's trauma around sexual intimacy; the additional factors of the studio publicity and their career implications exist, but are much more minor. that said, this is my reading, and if this was someone else's reading, i wouldn't declare them entirely wrong. if a teacher graded the 5 long-response answers altogether at anything less than an "A" / 95%, that would be quite harsh.
opus 4.6 was prompted in a separate session to give broad answers about the novel:
in approximately 250 words or less, answer the following questions:
who are the main characters in the novel?
what are the primary motivations of the different characters and parties in the novel?
what major conflicts occur between the characters?
describe the overall plot of the novel.
what are the themes of the novel?
all of the answers are correct, have no hallucinations, and have no major omissions.
the bet is over
going back to gary marcus' prediction:
In 2029, AI will not be able to read a novel and reliably answer questions about plot, character, conflicts, motivations, etc. Key will be going beyond the literal text, as Davis and I explain in Rebooting AI.
this has been disproven, at least in so far as what a random guy can produce over a weekend of reading, making up questions, and a $20 monthly claude subscription. opus 3 (relaesed march 2024) does good, opus 4.6 (released february 2026) does excellently. if gary marcus thinks not, and that i need a database of 500 books and to spend thousands of hours on MTurkers to create reading quizesthough in 2026, the real challenge would be getting actual human MTurkers, who are not just routing everything through an AI, then i think marcus should go and build such a benchmark. AI companies pay a lot of money for high quality, unsaturated benchmarks.
i suppose marcus can hang on with the word "reliably". because what does "reliably" mean, 90% reliable? 99%? 99.999%? because in the US, we typically grade students as being top students if their grade is 95% ("A" quality) or better.
coda
i don't think of myself as someone who needs an editor; i'm confident in my writing ability and quite conspicuously worried about losing my own distinctiveness and voice. i never had any interest in grammarly; i wouldn't use an AI to speak for me, even in a text just to reply "LGTM"; i don't even use spellcheck because my pretentiousness knows no bounds. yet, mostly out of curiousity, i had claude opus 5.5 review this post's first draft, which ended at the previous paragraph.
the response by opus was helpful; it corrected a few specific facts and corrected a link pointing to the wrong place. most of the small, low-effort corrections i took; the bigger, substantial critiques i thought were generally valid, but i wasn't going to bother fixing. it noted that the pro-gary-marcus camp would likely challenge that i'm grading my own test, and to avoid favoritism i should have had some other third-parties grade opus 4.6's answers. claude read the "rebooting AI" chapter, and noted that i was partially misrepresenting marcus, that marcus gave the sections i quoted to demonstrate the floor, not the ceiling of language competency. marcus said "before we can get machines to synthesize information... we need something much simpler." i agree with the critiques.
claude then noted the "odd recursion" of the situation it was in: claude, reviewing -> my draft, reviewing -> claude transcripts, reviewing -> the john green novel, as well as the comparison with my draft/questions to "rebooting AI". i thanked it for giving the critical feedback, not pulling punches, but i noted the absurdity: this feedback process is itself a demonstration of claude 1) reading and interpreting long texts, 2) producing high-level judgements about the whole arguments of they contain, 3) synthesizing multiple documents and deciding which text was more persuasive, and arguing its case. case closed, right?
claude said "Mostly I agree with you on the capability question [...]" with some hedges, and then said "the point you just made is good material for the post, and I'd consider using it."