Consider the Source – Why AI Outputs Need Vetting in Academic Research

If you’ve been writing papers for very long, you’ve no doubt heard the phrase, “Consider the source.”

And with good reason!

You’d never cite a tabloid magazine or a retracted research study (as primary evidence) in an academic paper. If you want your research to hold weight, then the sources you're working from have to be equally reliable & trustworthy.

You may already know what makes a source credible – it’s work that’s authored by experts, vetted through peer review or reputable editing, and supported with clear evidence and transparent methods. AI, particularly some of the more commonly used LLMs (Large Language Models like ChatGPT), doesn’t always distinguish between a credible source and an unreliable one. There are also times when it makes up a source out of thin air. You may have heard this described as a “hallucination.”

For several years now, hundreds upon hundreds of papers containing “nonsense” have been found in journals and even conference proceedings, according to the work of Guillame Cabanac. Cabanac is a computer scientist who works on “decontaminating” scientific literature.

When using generative AI, like LLMs, they are trained on pre-existing data. Sometimes the data is input directly into the model; other times it is gathered from across the internet. What happens when AI hallucinates, generates false information, and then a future iteration of the LLM trains on the data generated by its predecessor? “It gets to a point where your model is practically meaningless,” Ilia Shumailov, a machine learning researcher at the University of Oxford, told Scientific American in 2023.

More recently, a joint study by the United Kingdom AI Security Institute, the Alan Turing Institute, and Anthropic (the company behind the Claude AI) suggested that even a small number of “poisoned” or hallucinatory documents can compromise an LLM’s training data, regardless of the LLM’s size.

Don’t get us wrong. We’re not saying, “Don’t use AI.” Even the APA has issued guidance on citing ChatGPT & other models.

When ranking our favorite AI's, Academic Integrity will always come first for PERRLA. We’re not convinced it comes first for LLMs trained on Reddit comment threads (or the internet at large, for that matter). After all, when was the last time someone told you that you should believe everything you read on the internet?

Now, perhaps more than ever before, consider the source. In all your work, AI should be used with the same kind of critical thinking and held to the same credibility standards you'd expect of the sources you've prepared to cite in your next paper.

As you work on your next paper, here are a few questions you may want to consider:

1. Who is the author and are they qualified?
2. Where was this published and does it have editorial/peer review?
3. Is the claim supported with data or citations I can check?
4. Is the publication date appropriate for my topic (i.e., recent enough)?
5. Are methods, funding, or conflicts disclosed?
6. Do other credible sources agree or contradict this finding?

If the answer is “no” or “I can’t tell” to more than one item you're considering, you may want to reconsider your source.