PERRLA/Blog/When AI Gets It Wrong: What Students and…
AI detection tools don't just miss AI-generated writing — they disproportionately flag students who wrote every word themselves. Here's what the research shows, and what it means for students and schools.
AI writing detectors don’t always get it right, and when they get it wrong, the consequences fall on students. Research shows these tools flag non-native English writers, Black students, and translated text at rates that have nothing to do with whether a student actually used AI.
Understanding how this happens matters whether you’re a student worried about a false accusation or an instructor deciding how to use these tools in your classroom.
Most AI writing detectors analyze something called text perplexity – essentially, how predictable the next word in a sentence is. The underlying assumption is that AI-generated text is more predictable than human writing because language models favor common, expected word sequences.
The problem is that human writing can also be highly predictable, particularly when the writer has a limited vocabulary range or is working in a second language. Liang et al. (2023) found that the average false-positive rate for Test of English as a Foreign Language (TOEFL) essays – real papers written by real students – was 61.3 percent. For U.S. student essays, the false-positive rate was near zero.
To test what was actually driving the misclassifications, researchers took the same TOEFL essays and used ChatGPT only to diversify the word choices, making the language sound more like a native speaker. That single change dropped the false-positive rate from 61.3 percent to 11.6 percent (Liang et al., 2023). The detectors weren’t identifying AI authorship. They were identifying underlying writing patterns associated with non-native speakers.
A separate study of 14 AI detection tools, including Turnitin, found that human writing that was machine-translated into English was classified correctly 20 percent less often than writing produced directly in English (Weber-Wulff et al., 2023). Translation introduces the same kind of predictable sentence patterns that trigger false positive.
The bias in these tools is not distributed evenly. Non-native English writers are at the highest risk, given the direct link between language predictability and false positives. But the disparity extends further.
A 2024 Common Sense Media survey found that 20 percent of Black teens reported having their work falsely flagged as AI-generated, compared to 7 percent of white teens (Klein, 2024). That’s nearly three times the rate, and it mirrors a broader pattern in AI systems that produce unequal outcomes for specific groups, often because of biases embedded in the data they were trained on.
Even students who successfully clear their name after a false accusation face lasting harm. As Liang et al. (2023) note, even if the accusation is eventually revoked, “the student’s reputation is already damaged.” For international students navigating an academic hearing in a second language, the stakes are especially high.
The accuracy issues are serious on their own. But how schools deploy these tools can make the harm significantly worse.
Weber-Wulff et al. (2023) point out a fundamental problem with using detector scores as evidence: most tools produce a score with no explanation of why AI generation was flagged. That means “a student accused of unauthorised content generation only on this basis would have no possibility for a defence.” Without a transparent explanation, there’s nothing to dispute.
Most detectors are also closed-source, so schools have no way to independently verify the accuracy claims vendors make or to test whether those claims hold for their own student population (Liang et al., 2023). When a school adopts a tool using the vendor’s default detection threshold without testing it on their own students first, they’re accepting risk they can’t fully see.
Automatic reporting systems remove the human judgment that might catch a false positive before it becomes a formal accusation. When a score triggers an academic integrity report with no human review, no conversation with the student, and no defined appeals process, a wrong answer becomes an official record.
The tools also fail in the other direction. Researchers were able to bypass AI detection almost entirely with a single rewording prompt (Liang et al., 2023). A system that punishes honest writers while letting dishonest ones through isn’t serving the purpose for which it was designed.
The research points toward several concrete steps schools can take before and after deploying AI detection tools.
Before rollout, any campus-wide tool should be tested on the school’s own students – including non-native English writers – not just on vendor-supplied benchmarks. Schools should require vendors to disclose error rates broken down by student group, and an ethical review board should assess both the tool’s accuracy and its detection thresholds before purchase (Weber-Wulff et al., 2023).
A detector score should never be the sole basis for an academic integrity report. A conversation with the student should always come first (Klein, 2024; Weber-Wulff et al., 2023). If human review is part of the process, that review should follow a defined structure. Reviewers carry their own biases, and an unstructured review can introduce new ones (Klein, 2024).
Schools should also establish a clear appeals process that feeds back to the review board, so incorrect flags can lead to tool recalibration or removal – not just individual case resolution.
After deployment, monitoring flag rates by student group on an ongoing basis is essential. Wisconsin’s own analysis found that its Dropout Early Warning System (DEWS) graduation prediction tool was flagging Black students at a 42-percentage-point higher false alarm than white students, and schools were not informed for nearly two years (Feathers, 2023). Monitoring without accountability produces no change.
Some researchers suggest rethinking what AI detectors are actually for. Rather than deploying them as enforcement tools against students, Liang et al. (2023) recommend making them available to students as self-check tools during the drafting process. A false positive in a student’s own self-review is a far better outcome than an incorrect accusation in front of an academic integrity board.
Weber-Wulff et al. (2023) recommend shifting focus from the final submitted paper to the entire writing process – drafts, revisions, research notes. Instructors who can see a student’s work develop over time have far better evidence of authorship than any detection tool score.
This is where PERRLA’s Writing Timeline becomes relevant. Because PERRLA tracks every version of a paper as it’s written and not just the final draft, instructors can see the actual writing process unfold. That kind of evidence doesn’t rely on a detection tool and can’t produce a false positive.
PERRLA supports the latest versions of APA, MLA, and Turabian. Try it free for 7 days, no credit card required.
Not reliably. Most AI detectors analyze text perplexity – how predictable the word choices are – and flag highly predictable writing as machine-generated. That approach produces significant false positives for non-native English writers, translated text, and other writing that happens to use common or expected word sequences. No current tool can definitively determine authorship.
Writers working in a second language often use a narrower vocabulary and simpler sentence structures, which produces the kind of predictable text patterns that AI detectors associate with machine-generated writing. Research found a 61.3 percent false-positive rate for TOEFL essays – real papers written by real students – compared to near-zero for native English writers (Liang et al., 2023).
Request a conversation with your instructor before any formal report is filed. Ask what specific evidence was used and whether a human reviewed the flag before it was escalated. If your school has an appeals process, use it. If you used PERRLA, your paper’s Writing Timeline shows the full revision history, which can serve as evidence of your own writing process.
Yes, and the differences are substantial. In a study of 14 tools, results varied widely: Turnitin produced no false positives in one test while GPT-Zero wrongly flagged half of the human-written text (Weber-Wulff et al., 2023). Accuracy also varies by student group, so a tool’s overall accuracy rate may not reflect how it performs for non-native writers or other specific populations.
The research doesn’t call for abandoning these tools, but it does call for using them differently. A detection score alone should never be the basis for an accusation. Schools that test tools on their own student populations, require vendor transparency about error rates, build in human review, and monitor outcomes over time can reduce the harm these tools cause when deployed prematurely.
Feathers, T. (2023, April 29). False alarm: How Wisconsin uses race and income to label students 'high risk'. Wisconsin Watch. https://wisconsinwatch.org/2023/04/wisconsin-race-income-label-high-risk-students/
Klein, A. (2024, September 18). Black students are more likely to be falsely accused of using AI to cheat. Education Week. https://www.edweek.org/technology/black-students-are-more-likely-to-be-falsely-accused-of-using-ai-to-cheat/2024/09
Liang, W., Yuksekgonul, M., Mao, Y., Wu, E., & Zou, J. (2023). GPT detectors are biased against non-native English writers. Patterns, 4(7), Article 100779. https://doi.org/10.1016/j.patter.2023.100779
Weber-Wulff, D., Anohina-Naumeca, A., Bjelobaba, S., Foltýnek, T., Guerrero-Dib, J., Popoola, O., Šigut, P., & Waddington, L. (2023). Testing of detection tools for AI-generated text. International Journal for Educational Integrity, 19(1), Article 26. https://doi.org/10.1007/s40979-023-00146-z
Once you see these four sentence-level patterns, you’ll start spotting them in every piece of academic writing you read. More importantly, you’ll be able to use them on purpose in the papers you write.
Learn the MLA 9 Works Cited format for books — one author, multiple authors, no author, editions, and translations — with in-text citation examples.
Five nursing programs ran full case studies with PERRLA, from a required rollout to one with no requirements at all. Here's what changed for faculty, students, and grades.
PERRLA formats the whole paper in APA 7, MLA 9, and Turabian 9, from the title page to the last reference.
Try PERRLA free for 7 days
