When Claude is Wrong: Why AI Must Be Challenged
You can easily get the wrong answer — and others can easily get the wrong answer about you
Recently I wrote about how Claude AI gave an alarming response to a query about our news site, WhoWhatWhy. Given my connection to the site’s editorial foundations and current output, you can imagine I was, as one might say, bumfuzzled by the “conspiracy-adjacent” label being applied to the hard work of so many experienced reporters and editors.
(Spoiler alert: surprise ending coming up. But don’t cheat — please read every word below.)
Since AI is now so central to what we know and how we think, it seems important to further explore this unsupported and derogatory label, “conspiracy-adjacent.”
Here is the warning Claude delivered to someone asking whether he ought to partner with WhoWhatWhy on a project:
Worth flagging upfront: WhoWhatWhy is an investigative journalism nonprofit with a fairly hard edge — it’s known for digging into underreported issues affecting the democratic process, from voter fraud allegations to redistricting to extremism, and several board/advisory members come out of conspiracy-adjacent or deep-politics investigative traditions (JFK/Dulles history, election-machine-fraud research, etc.).
Well, a “hard edge” sounds good — more likely than not, it’s a big part of what WhoWhatWhy readers are seeking. And “digging into underreported issues affecting the democratic process” is all “on brand.”
So why is it “worth flagging upfront” and trotting out the “conspiracy-adjacent” label, which is routinely slapped onto critical investigations of such issues as the JFK assassination or the vulnerabilities of computerized vote counting?
In typical AI fashion, Claude muddied the output with conflicting positives and negatives, so the claims appear “fair and balanced.” However, because of the term “conspiracy-adjacent,” what remains is the shimmering image of a tinfoil hat.
As our readers know, conspiracies exist — proven conspiracies. But someone reading Claude’s summary might assume that we regularly buy into unsupported claims of, for instance, election fraud — which couldn’t be further from the truth.
Also, by mentioning “JFK/Dulles history” with “conspiracy-adjacent,” it suggests the establishment view that a single shooter ends the story and that — regardless of facts that prove otherwise — no one may propose any alternative explanation, though 65 percent of the American public does not believe the “lone gunman” theory.
To say nothing of the smugness of the insinuation itself, the claim relies on an AI-assisted leap that conflates our editorial staff with independent individuals who serve on our advisory council. Advisory councils typically provide feedback; they do not participate in daily operations or determine organizational policies. Furthermore, only one advisory council member has ever written about JFK/Dulles — David Talbot, a credible and esteemed bestselling author.
As for “election-machine-fraud” research, being vigilant about potential hacking of digital voting systems ought to win plaudits, not implicit condemnation. The media’s obligation in this case is to act in good faith on behalf of the public: to responsibly examine every aspect of our democratic process and technology for potential defects and vulnerabilities.
Simply saying “Trump’s claim of election fraud is unfounded” — a disclaimer many legacy media organizations routinely affix in articles — does little to persuade anyone.
If other media seriously examined the mechanisms and chain-of-custody issues around electronic voting, we might not have such a seriously divided public on the issue.
Furthermore, we’ve actually gone much deeper than most legacy media in exploding Trump’s wild Stop the Steal allegation by: (1) showing how Trump uses the core falsehood as a litmus test of people’s slavish and uncritical devotion to him, precisely because of the boatload of counterevidence, and (2) bringing forward substantiated evidence that the 2020 election was NOT stolen.
Indeed, we went further and published an analysis demonstrating that everything statistically anomalous about that election pointed in the exact opposite direction from what Trump was claiming.
The bottom line is that if Claude were giving a valid summation, it would say that few if any media entities have been more outspoken about this, or more committed to studying the impact of Trump’s stated strategy of undermining the media and weaponizing falsehoods, than WhoWhatWhy.
A more nuanced analysis would have recognized that the depth and rigor of WhoWhatWhy’s reporting is rooted in facts, not adjacent to conspiracy.
That’s just for starters.
Conspiracy ‘Theories’
While Claude correctly considers us “independent media,” it also apparently assumes — incorrectly — that major, corporate-owned news organizations are more reliable, accurate, responsible, and unencumbered by agendas.
Perhaps this is because Claude, like most if not all LLMs, is trained mainly on that vast bulk of corporate-approved output, compelled to reflect the presumptive reliability of its information and opinion.
Anyone familiar with the gross falsehoods and errors of legacy media — including the fabricated Gulf of Tonkin provocation for massively expanding the Vietnam War, and the equally mendacious “weapons of mass destruction” claim that George W. Bush advanced to justify his grab of Iraqi oil fields — knows to be cautious in trusting what the media establishment tells them.
Not to mention what they omit.
More recently, the media establishment failed to take seriously enough the risks associated with a possible Donald Trump presidency and second term. A too-trusting public — including the millions unfamiliar with the aforementioned falsehoods, errors, and omissions — went along in ignoring or discounting those risks. And here we are.
I have had enough personal experiences with both print and broadcast legacy media to know that these often impressive organizations nonetheless walk in fear of litigation — and of getting too far ahead or outside of pack consensus when investigating or reporting anything truly consensus-shaking, anything that might rattle us to our very cores and undermine our fundamental trust in the established system, of which they are bulwarks and beneficiaries.
And that’s why they blur the line between rigorous journalism and the disinformation and fabrications that muddy the waters and poison public discourse — a blurring that directly aligns with Trump’s own media-attack strategy. Isn’t it ironic?
I wonder what they make of the work of their retired, longtime correspondent Stephen Kinzer, who has written several books carefully and precisely expanding into areas he never touched while working at the Times. And of his recent testimony before a House committee about the CIA’s MK-Ultra mind control research program. An excerpt:
MK-ULTRA was one of the most secret government programs in American history. The chemist who directed it, Sidney Gottlieb, lived in total anonymity. …
In 1951 the CIA hired Gottlieb and directed him to launch what became MK-ULTRA. Gottlieb believed that in order to find a way to implant a new mind into someone’s brain, it would first be necessary to destroy the mind that was there already. In its search for ways to destroy a human mind and body, MK-ULTRA conducted the most extreme experiments on human beings that have ever been carried out by a US government agency. By any standard they qualify as medical torture. These experiments took place in prisons, clinics, and safe houses in the United States, Europe, and Asia.
Officers of MK-ULTRA were authorized to travel to foreign countries, preferably those under formal or informal US occupation, and ask the local CIA station to provide them with “expendables” — human beings who would not be missed if they disappeared. Gottlieb had what amounted to a license to kill issued by the US government.
My own MK-Ultra story for The New York Times Magazine three decades ago was well received during the editorial process, only to be mysteriously spiked. (The assigning editor, who seemed genuinely bewildered by the internal decision to shelve the story, graciously paid the full fee.)
The story later found a home with more than 20 leading international publications, including Der Spiegel (Germany), The Observer (UK), and Good Weekend (Australia). Go here to read the story.
***
Of course, whatever the motivation, we certainly do see the media selectively embracing topics that have hints of collusion among powerful people and organizations, like the Jeffrey Epstein mega-scandal. Yet they often do so long after independent media and solo journalists start writing about the matter, without the institutional safety net but also without the corporate agendas that can limit them.
It’s high time for the media establishment — and the increasingly dominant masters of artificial intelligence — to develop some ability to distinguish between “conspiracy fabrications” and the very real patterns of coordinated misconduct that must be revealed to the public and confronted by the government.
Imagine if the New York Times, the Washington Post, Los Angeles Times, et al. did begin to cover — I mean properly cover — such subjects as MK-Ultra and Mockingbird (the CIA’s recruitment of US journalists to spread early Cold War propaganda)?
But that would mean also reporting on their own past negligence — or worse, their actual complicity.
The admission of their historical deficiencies would represent the kind of transparency and acceptance of responsibility they so often demand of others.
We’re in a new era, and now we all will face greater challenges, but also more media scrutiny and accountability. And that’s a good thing for society.
UPDATE: Claude Takes a Second Look
But wait…
We asked the person who queried Claude to challenge its assessment. And… it backed down. Here are some excerpts from its response:
Having now read their full About page carefully, I want to be straightforward with you about where my earlier caution came from — and where it may have been imprecise…
My original flag was that WhoWhatWhy’s editorial posture — “responsible fearlessness,” deep-politics investigative tradition, board members like Daniel Ellsberg and David Talbot (author of The Devil’s Chessboard) — suggested a watchdog organization that might be uncomfortable with anything resembling a sponsored or co-branded relationship. That instinct was directionally right but probably stated too strongly.
Reading their actual board and advisory council carefully, they are a well-credentialed, mainstream-adjacent nonprofit: noted by the Associated Press, The Washington Post, NBC, ProPublica, and The Wall Street Journal, with board members from Columbia Journalism School, Berkeley Journalism School, the NRDC, and Bloomberg Philanthropies. That’s not a fringe conspiracy outlet — it’s a serious investigative nonprofit. I was probably pattern-matching too quickly on a few advisory names rather than the full institutional picture.
Their model of relying on a mix of paid journalists and skilled volunteers allows them to do more with modest resources….
Their work involves digging deep for meaningful answers with responsible fearlessness…
Well, gee. Thanks, Claude! We get that you, as well as other AIs such as Copilot and ChatGPT, have dramatically improved how you learn from follow-up queries.
But I find it odd that you always add an inconspicuous little disclaimer at the end of your offerings about how you are not always right — after strongly stating opinions as fact. Yet, you deny the certainty of political facts that are not in dispute.
All of this reminds us that we should never automatically assume you are right. We should always re-ask the same questions, using sharper, more penetrating, key words.
Above all, we should play devil’s advocate with you, ask you to defend your opinions with specifics, and, when appropriate, request that you name your sources. Then compare what you say with what your sources actually say.
We all must be challenged, whether we’re human or machine; nothing — and nobody — is infallible. And we must check and double-check any technology we’re trusting with just about everything. If we just blindly accept what you say, Claude, we could get into all kinds of trouble and make some very bad decisions ourselves.
ABC: Accept nothing, Believe nothing, Check everything.