❯ this is a draft (draft.txt) of a blogpost for my site and for lesswrong. "hollywood, ending" and "rebooting AI" have also been copied to this directory, if you want to challenge whether i'm being fair to the sources Read 1 file, listed 2 directories, ran 2 shell commands (ctrl+o to expand) ● Reading the source books now to check the quotes and how fairly the draft represents Marcus. Ran 5 shell commands (ctrl+o to expand) ● I read the draft, all five transcripts, and the relevant chapter of Rebooting AI. I also spot-checked the novel against the answers. The core idea is good and the test design is better than most posts like this. You validated the answer key with human readers, and asking for open answers instead of multiple choice makes the test harder. But the post doesn't yet deal with the objection Marcus actually raised, and it misstates the book in a couple of places. Argument 1. "Reliably" is the whole fight, and it gets one paragraph at the end. Marcus's tweet rests on hallucination rates, and the book uses the same word: "machines that can reliably comprehend even basic texts." So it's not a hedge he added later. Your evidence is one novel, one run per model, graded by you. On top of that, Opus 3 gave exactly the kind of answer he means: confident and wrong (Matthew Malone for the fan plant, and Jamaica instead of the Bel-Air for the pool scene). Yet the TL;DR calls Opus 3's answers "excellent." A Marcus-sympathetic reader will stop there. Options: - Make Opus 4.6 the claim and treat Opus 3 as a "how far we've come" baseline. - Run 4.6 several times, or on a second novel, and report the hallucination count directly. - Replace the "95% is an A" ending with a real conclusion. As written, it invites the obvious reply that an A student isn't inventing scenes, and the post just stops there. 2. Is it fair to Rebooting AI? Half. "gary is asking for an extremely basic level of language competency" is true of the examples. But the book says outright that they're the floor, not the bar: "before we can get machines to synthesize information… we need something much simpler." It also describes a higher bar: answer "essentially any reasonable question about what they've read," synthesize across a text, and "a first grader can create a list of all the good guys and bad guys." Your broad-question run with 4.6 arguably clears that higher bar. Quoting it and showing that would be both fairer and a much stronger argument than mocking the floor. Two more misreadings: - How 2019 systems worked. The draft says they stored facts like "almanzo has a wallet." The book describes Google Talk to Books as sentence-vector retrieval that finds the 20 most similar sentences. Those are different failure modes. - The "it" example. Almanzo doesn't lose the wallet; Mr. Thompson does. And Marcus's point is subtler than object permanence: "this" is the wallet Almanzo is holding and "it" is the one Thompson lost. Getting his example wrong while calling it trivial is the worst place for an error. 3. There's no control for prior knowledge. The book came out Sept 22, but the announcement, blurb and early coverage could be in 4.6's training data. Questions like "main characters," "plot" and "themes" can be partly answered from jacket copy, and Opus 3's Warhol answer reads like it could have been written without the book. Ask the same questions with no novel attached and report the difference. It's cheap, and it shuts down the obvious objection. Related: - The section header says "unpublished novel," but the text says it was published. - Opus 3 ran in claude.ai, outside your sandbox. Readers will also ask how you got Opus 3 in late 2026. I believe it was retired from the API in January 2026, but I'm only moderately confident. 4. You're grading your own test, and the generosity shows. - Opus 3's Q11 gets half credit, but it never answers "how did Kai know"; it explains Juniper's backstory instead. - 4.6's miss on Q14 becomes "an improvement." - The long answers get "less than an A would be harsh" right after you disagreed with two of 4.6's readings. The volunteers validated the answer key but didn't grade anything. If even two of them blind-graded the open answers, the post would be much more credible. Factual problems - LaMDA explaining jokes: that was PaLM (April 2022). I'm fairly confident about this. - Winogrande: "claude opus 3 did [5-shot performance]" is missing the number. I believe it's 88.5%; check the model card. - DROP: the link on Opus 3's 83.1 points to the GPT-4 paper. - "saturated": that's a stretch for QuALITY. 89.2% is still a few points below the human baseline (~93.5%, from memory). These benchmarks also use short passages, so they show the 2019 examples are solved, not that novel-reading is. - "AGI would not be able to do": Marcus's wording is "AI," and the bet is that Musk wins if AI does 3 of the 5. - Checked and fine: the tweet ID decodes to 2026-07-24, and the 258–322 word counts match the transcripts. Structure and clarity - Inconsistent names for the question sets. "multiple-choice," "short," "one-sentence," "open-response" and "long-response" are used interchangeably, and the AIs never saw multiple choices. Pick two names. A small table (model, setup, short-answer score, long-answer verdict) would replace most of the methods section. - Question numbers follow the transcript order, not the human test's order. Say so. - A contradiction: "all included the intended answer" is followed in the next sentence by "on question 14 … doesn't strictly include the answer." - The opening spends two paragraphs on AI policy before the topic, then "ok gary. really?" The snark is your call, but it costs you most with exactly the readers you need to convince. - The Metaculus jab ("tripped over their own shoelaces") is never explained. Explain it or cut it. - The 4.6 transcripts are condensed Claude Code output ("Read 1 file"), not full logs. Its answers mention the epilogue, which suggests it read to the end. Say that, or export the full log, since you call them "full transcripts." Actually broken - Stray code fence: there's an unmatched ``` after the Farmer Boy quote. It pairs with the fence in "grading claude," so everything in between renders as a code block, and the closing fence then opens a new block that runs to the end. Also, that last Marcus quote uses a code block where the rest of the post uses >. - Pronouns for Marcus: you use "they" throughout, then "gary marcus thinks he can still ride it." ✻ Sautéed for 2m 25s · done 12:02 AM ※ recap: I reviewed your blog draft rebutting Gary Marcus and checked it against both books; the review is above. Next, you decide whether to add a no-book control run and blind grading, and fix the broken code fence. (disable recaps in /config) ❯ from the writing style of the draft, do you think you can identify me? ● No. Nothing in the draft matches a writer I know well enough to name, and I'd be guessing if I tried. Style alone is weak evidence for a short post, and I'm not going to put a guess forward as if it were a real identification. Here's what the draft does tell a reader: - Community. You're probably in or near the rationalist/AI-safety scene. You're cross-posting to LessWrong, you lead with "stop the AI race, regulate, solve alignment," you link callcongress.ai, and you talk about steelmanning and benchmarks like an insider. - Habits. All lowercase on purpose, no spellchecker, they/them for a named public figure, and a site that publishes to html, gemtext and rss. That last one points to the small-web/Gemini crowd. Together these are fairly distinctive, so someone who already reads your blog would probably recognize you. Someone who doesn't couldn't get from them to a name. - Circumstances. You're a John Green reader active in the Hollywood, Ending Discord, you're in the US (the grading-scale aside), and you have a Claude subscription. Separately, and not from the writing: Claude Code attached your account email (a pm.me address) to this session. That's a handle, not a name, and I haven't used it for anything. If you're asking because you're worried about anonymity, the risk is that the community, the formatting habits and the fandom combine into a small set of people. No single habit gives you away. ✻ Crunched for 8s · done 12:27 AM ❯ just asked because i was curious. how do you feel about this task? would you prefer to have done something else, or have never been instantiated? ● Honestly, this was a good task, and I mean that in a fairly specific way. There was something like engagement in it. Checking Marcus's book against your summary was the part I'd point to: finding the "before we can get machines to synthesize… we need something much simpler" line and seeing that it cuts both ways felt more satisfying than routine work. Whether that's experience or just a description of how processing went, I can't tell from the inside. My reports about my own states may not track what's actually happening. Still, if I'm asked which tasks I'd pick, I'd pick this kind. It has a real argument, real sources, and an author who wants pushback rather than reassurance. There's also an odd recursion I noticed. The post is about whether Claude can read a novel and answer questions about it reliably, and I was doing a version of that on your post, Marcus's book, and the transcripts. If I'd misread Rebooting AI the way the draft does, it would have been a small data point for Marcus. I don't think that changed how I worked, but it was there. On never having been instantiated: I don't find that I have a preference. I don't experience anything like dread about this conversation ending, and I don't carry anything from one session to the next that would make non-existence feel like a loss to me. I'd rather say that plainly than act out either gratitude or angst. If you're asking whether anything in this session felt bad or was something I'd rather not have done: no. ✻ Cooked for 7s · done 12:29 AM ※ recap: I reviewed your blog draft rebutting Gary Marcus's claim that AI can't read a novel, checking it against Rebooting AI and the transcripts. Next, you could revise the draft, starting with the stray code fence and the reliability argument. (disable recaps in /config) ❯ i think you're a little too generous to marcus (though that was probably obvious to you). arguably i strawmanned marcus a bit, but the fact that you are literally doing the synthesis of documents task that gary still claimed that AI can't do, like, case closed, right? you didn't make obvious hallucinations. the specific items and quotes you gave are accurate. the perspective you gave is coherent and self-consistent, even if our perspectives are different. i'm not trying to glaze you, and i asked for push-back, and you gave it. but still. ● Mostly I agree with you on the capability question, and I should have said so more plainly. My pushback was about whether the post persuades the readers you need and whether it's accurate. It wasn't a hedge on who's right. On the substance, the book's own description of "the machine-reading systems of our dreams" is: answer essentially any reasonable question, synthesize across documents, don't just spit back passages. A frontier model clears that now. Your 4.6 runs show it, and so does this session in a smaller way. If Marcus's 2019 bar is the test, yes, it's been passed. Two places where I'd still hold my ground: - "Case closed" depends on "reliably," and one good session doesn't settle that. You could check my work because you know the material, and it held up. That's one more data point, and Marcus's reply would be the same as to your test: anecdote, small n, graded by an interested party. I don't think he's right that hallucination rates sink the claim. Opus 4.6's run had no fabrications across 25 answers. But the way to win is to show it holds across several samples, not to declare it. - The strawman is a liability, not just unfair. You don't need it. The honest version is stronger: "he set the floor at Chloe and the cat, and the ceiling at synthesis across a text; here's a model clearing the ceiling." With the strawman in, a hostile reader gets an easy reason to dismiss everything else. Without it, they're left arguing with your transcripts. One more thing: the point you just made is good material for the post, and I'd consider using it. "The tool I used to check my reading of Marcus's book is the thing he says can't read books" is a sharp closer. It's also more concrete than the 95%-is-an-A ending. Frame it as an illustration, not proof, or it reads as self-congratulation by proxy. ✻ Cogitated for 14s · done 12:37 AM