A bank of white freezers hums away at a former watch factory in central London. Inside: dozens of small boxes with barcoded glass tubes, collectively containing one hundred billion genes. Just two boxes may hold 10 times more information than all the text on the internet, says Dr Oliver Vince, the facility’s co-founder. His firm, Basecamp Research, plans to collect as much DNA as possible to create an artificial intelligence that can speak the language of biology, in the same way that US and Chinese tech firms have hoovered up written material online to create Large Language Models (LLMs), the basis of current AI systems.
Vince, like many in business and academic research, believes AI will supercharge scientific research. But increasingly, it also comes with controversy. Last month, OpenAI, the creator of ChatGPT, said it had solved one of maths’ great unsolved problems, relating to the famous Navier–Stokes equations. The company said its AI agents had cracked it in 88 hours.
The body that runs a prize for whoever can solve the problem said it had “apparently been settled”, but that its review would take time. But hours before the announcement, Tristan Buckmaster, a mathematician at New York University, went public with concerns. He formed an independent team with Levent Alpöge, a mathematician employed by day by the rival company Anthropic. They had been working towards the same goal, and kept their drafts in OpenAI’s coding tool, Codex.
Buckmaster said that his co-researcher was tipped off that news of their progress had “reached” OpenAI, though he did not specify how. He asked whether his research, stored in Codex, could have fed into OpenAI’s own work, though he stressed he was “not accusing anyone of anything”. OpenAI said that nobody at the company, human or AI, saw the other teams’ work before it was published. Initially it said that while unlikely, it could not “rule out” that anonymised data from use of its products may have helped improve its models.
Days later, it said Buckmaster’s Codex prompts from the two months before the announcement “could not have influenced the system in any way”. It’s a complicated story; we still don’t fully know what happened. But it highlights a growing issue about the use of AI in science. Usually, those words make us conjure up images of robots building terrible genetically-engineered viruses, or chemical weapons, to wipe out humanity.
But there’s a more subtle problem that hasn’t attracted the same headline-grabbing attention: that AI could undermine faith in the scientific process, raising questions about the novelty and accuracy of research. And there’s another concern. It’s well known that AI chatbots like Claude or Gemini can sometimes hallucinate things that simply aren’t true. They will tell you about a film that never existed, or a quote that was never said.
Now there is evidence that this is increasingly happening in science. Eventually, it’s feared, it will become difficult to separate true scientific discovery from AI-generated fiction. And all of this could threaten to erode – if not explode – the rock on which scientific research rests: trust. One threat is to scientific journals, the guardians of scientific integrity, which are seen to be failing to deal with a tsunami of AI-generated distortions.
“Generative AI has basically disrupted the whole scientific process,” says Dalmeet Singh Chawla, a science journalist who has spent a decade reporting on research misconduct. He tells the BBC that publishers’ integrity staff – the people who screen submissions for accuracy and reliability – will need to “drastically increase [in number and effort] if they’re anywhere near the fight”. He says: “Right now, it seems like they’re not.” On the face of it, AI seems just the right tool to power scientific research. Scientists deal with huge amounts of information.
Telescopes and particle accelerators, for example, collect billions of data points – too large for the brain to comprehend. AI can pick the scientific highlights from the surrounding humdrum. The poster child of scientific AI is the UK-based firm Google DeepMind, which in 2020 solved one of biology’s biggest problems: how proteins in living things fold into their precise shapes. It may sound dull, but proteins are the tiny machines that make life work, and the shapes they fold into determine what they can do; from building cells and fighting infection to helping form the tissues and organs of the body.
Before AI, scientists had to conduct time-consuming experiments in the lab. Then, Google DeepMind developed an AI model called AlphaFold that accurately predicted 200 million protein folds in just one year. The equivalent experimental work would have taken hundreds of millions of years in laboratories. The firm gave the results away for free to researchers.
The head of DeepMind and the project’s lead scientist were both awarded the Nobel Prize for Chemistry. It was big hugs all round. Dr Pushmeet Kohli, who leads DeepMind’s science work, told me that AlphaFold could be seen as a “watershed moment”. But elsewhere, the use of AI in science has come with accusations of hype.
Some computer models are accused of exaggerating the significance of their findings, with the human researchers that oversee them then believing some of the claims without enough scepticism. Basking in its success over AlphaFold, for example, DeepMind turned its attention to predicting the structure of new materials. The work could lead to the development of better batteries, cheaper solar cells and electronic components that are not dependent on rare earth elements. (In theory, no-one would need to argue about mining rights ever again, and Greenland would be safe from invasion.) A DeepMind model came up with what the firm claimed were more than two million new crystal structures, announced as “nearly 800 years worth of knowledge” released in one go. Of those, some 380,000 were considered promising enough to make in a laboratory.
But then came a warning sign from two chemists at the University of California. Anthony Cheetham and Ram Seshadri went through a random sample of the supposed new crystals. Writing in the journal Chemistry of Materials, they concluded that they had “yet to find any strikingly novel compounds”, arguing that many were “trivial adaptations of known materials”. Their verdict was not that the method was wrong, but that the results need more scrutiny and validation from human chemists.
DeepMind rejects the charge, saying it tested each compound for novelty. Dismissing a new structure for looking familiar misses the point, the company argues: swapping a single atom is how many of today’s most useful materials arrived, from phone components to sunscreen. In those cases, at least, the science was mostly real and the issue was just AI potentially exaggerating its importance. Arguably of even greater concern is the growing risk that scientific journals will become polluted with so-called AI “slop” – where a computer, not a human scientist, is writing the research, leading to fake citations and phantom papers.
In 2024, computer scientist Prof Dmitry Kobak, of Ghent University, searched more than a million biomedical research papers for trigger words favoured by the LLMs that generate AI writing. His team found that by December last year, nearly 90% had AI fingerprints on them. AI detection technology is not perfect; some forms of the software have labelled the US Constitution, written in the 1780s, as “AI-generated”. But still, Kobak said he was shocked by the scale of apparent AI use.
There is a risk, he told Nature, that “whatever bias the LLM may have will just suddenly permeate the (scientific) literature”. Kobak is among many researchers who worries that AI is slowly killing scientific journals. And it matters: research papers are the lifeblood of scientific advance. When a discovery is published, another team in another part of the world might obtain funding to build on the results.
The process of peer review – in which research papers are checked by other scientists before they are published – is also being compromised. A team led by Weixin Liang and James Zou at Stanford University analysed referee reports submitted to four major computer science conferences, and found up to 17% of the text had been substantially modified by a language model. A critical part of a scientific paper’s integrity involves lead scientists listing earlier research in order to show what they have done is genuinely new. These are called citations.
A separate team at Cornell University found that researchers were using AI to compile citations, and that it was inventing bogus research, with around 147,000 pieces of phantom research cited last year. If AI continues to distort the integrity of the academic process in this way, then researchers will be sent up blind alleys, public money will be wasted, and the fruits of valid research – such as new drugs or green technologies – will be slowed or not realised at all. Some publishers of scientific journals, such as Elsevier, are taking steps to avert this growing crisis of confidence. Elsevier’s research integrity team has developed some 100 signals that flag a paper for closer human scrutiny.
Nature’s journals forbid AI-generated images and tell referees never to upload a manuscript into an AI system. The website arXiv, where scientists share their early-stage results, bans researchers for a year if they submit invented references. Prof Sarah Main, an Elsevier vice-president, is among those at the firm trying to work out how to responsibly use AI in research. “What we’re calling research-grade AI would focus on ensuring the researcher is in the driving seat,” she says.
As AI agents process the mass of data emerging from particle accelerators, the cosmos and the natural world, how do we know they are getting it right? And there’s another worrying question: how do we know that AI agents are not dismissing an important result as an anomaly? The discoveries of penicillin, radioactivity and weird stars called pulsars all came from odd results that could easily have been dismissed. Hunches by humans made them into some of the biggest scientific moments of the last century.
Can AI really replicate that sense of serendipity and human ingenuity? Kohli of DeepMind does not think the answer is to turn our backs on AI. That would be “like saying you should be computing without using calculators”, he told me. But he accepts there should be human guardrails: “The agents today are imperfect.
Somebody has to take accountability.” His test is a simple one: “You have to understand what the agent has done, and you have to take responsibility. You can’t just specify the problem and let the machine write the paper and submit it as true.” Why some experts increasingly fear AI will take over AI is creating a ‘human premium’ for art made by people. But will it hold? Tech billionaires seem to be doom prepping.
Should we all be worried? Prof Andrea Pauli of the Vienna BioCenter in Austria has followed that test. She spent almost 10 years on a question no-one has yet been able to answer: how a sperm gets into an egg. Responsible use of AI helped her toward an answer.
Her lab found a protein on the surface of zebrafish eggs which pointed to a lock-and-key mechanism for allowing the sperm in. Hers is a case for hope, in a sea of AI scientific slop. It shows that AI can help. This is also a story about humans.
Soon after the formation of Basecamp – the lab I visited in a former London watch factory – its co-founder, Oliver Vince, was ill in hospital with leukaemia (from which he has recovered). It motivated him to develop an AI system that stood a better chance of curing cancers. It might not work, he says, but it’s worth a shot. “We have a technically feasible path to curing cancer,” he says.
Not a promise from Vince, but a direction. The big question is whether AI carries the science there, or quietly erodes the bricks it is built on. BBC InDepth is the home on the website and app for the best analysis, with fresh perspectives that challenge assumptions and deep reporting on the biggest issues of the day. Emma Barnett and John Simpson bring their pick of the most thought-provoking deep reads and analysis, every Saturday.
Are you personally affected by the issues raised in this story?
