Using artificial intelligence (AI) to determine whether text has been generated by AI or written by a human is a wonderful concept, but today’s AI content checkers fail to deliver on their promise.
Theoretically, AI text checkers should be highly accurate, but because they are AI-based their accuracy depends on the context and the tool that’s used. These tools can quickly identify grammar errors, typos, and structural inconsistencies in text. They can also provide insights into tone, readability, and adherence to style guidelines.
However, while these tools are continually improving, they aren’t flawless. Their accuracy can vary, and they can misinterpret context, especially in nuanced or creative writing. They can over-correct or offer suggestions that, while technically correct, don’t align with the intended tone or style.
The Numbers Tell the Story
There have been a number of studies and articles discussing how AI content detectors don’t work. In a ZDNet study, only two out of nine AI content checkers correctly identified AI-generated content 100% of the time; it would have been three, but one bot failed a test it had passed previously. The remaining checkers had success rates of 40-80%.
In another study, some of the leading AI content checkers were shown to have low accuracy rates of 39.5%. This accuracy dropped to 17.4% when faced with manipulated content, such as when an automated paraphrasing tool is used to change content or errors are deliberately inserted into the text.
Forbes wrote about a copywriter running the Declaration of Independence through an AI content checker with the result that it was 98.51% AI generated, although it was written on parchment and adopted by the Continental Congress on July 4, 1776.
Axios wrote about how a team of researchers from the University of Pennsylvania, University College London, King’s College London, and Carnegie Mellon University created its own system for benchmarking AI content checker tools. They found that many of them did not live up to the claims and were, according to their study, too good to be true.
It’s also important to understand and be clear what the AI score means when using an AI content checker. A score of 60% should be interpreted to mean that the bot is 60% confident that the text is AI generated, not that 60% of the text is AI generated.
Needless to say, with these types of margins for error, AI content checkers can misidentify both human-written text and machine-generated text. Compounding the problem, different tools can disagree about the origins of the same piece of content.
Writing Styles & Contextual Complexity
AI content checkers are based on the flawed assumption that text generated by AI leaves consistent and identifiable markers that can be easily traced. But writing styles vary dramatically across different professional backgrounds, cultural contexts, and personal experiences. Every writer brings his or her own unique linguistic style shaped by their education, professional training, and personal creativity.
Contextual complexity adds another layer of difficulty to AI content detection. Different content types, such as academic papers, creative writing, technical documentation, and newspaper reporting, each possess their own unique characteristics. An academic paper in theoretical physics will read dramatically different than a technical report written by an engineer, a short story written by a novelist, a speech written for an executive, a personal blog post, or a marketing brochure.
These subtle distinctions create significant hurdles for detection algorithms. These detection tools risk perpetuating systemic biases by discriminating against unique writing styles and linguistic expressions.
The Danger of AI Content Checkers & “False Positives”
The consequences of overcoming misidentifying who or what has written something is not just a technical exercise. A single mistake by these bots can devastate professional reputations, particularly in industries with stringent communication standards. Businesses face substantial risks when over-relying on these imperfect technologies.
AI content checkers can sometimes misidentify human-written content as AI-generated, especially if the writing has a very consistent style. In academic settings, AI content checkers that generate “false positives” can lead to unjust accusations of academic misconduct with serious consequences for a college student’s career.
In one stance, according to a Bloomberg report, a student received a grade of zero on a written assignment when her professor used an AI detection tool that determined that her work was likely generated by AI. The student disputed the accusation and her grade was ultimately changed, but the ripple effect reached beyond a simple technological error that her work was likely AI generated.
Compliance presents another area of critical concern. Regulated industries cannot afford to rely on technologically immature detection tools that lack the nuanced understanding of their complex communication requirements.
AI content checkers today represent a fascinating, but fundamentally flawed technology. Their current limitations on accuracy demand a measured, intelligent approach that preserves the human expertise that’s irreplaceable in content oversight and verification. Expert human reviewers bring their informed judgments, contextual understanding, subject matter expertise, and editing experience to generative AI text. Their unique combination of art and science helps AI content editors ensure that the finished content is clear, concise, compelling and personalized to your customers and clients.
Aids Not Arbitrators
At the present time, investing significant resources in unreliable AI content detection methods diverts attention from more effective content quality control strategies, such as engaging subject matter experts and professional editors who can provide comprehensive content review and evaluation. AI content checkers need to be treated as aids to determining if content is human written or machine generated, not as the final arbitrator.
Interested in finding out how Writing For Humans™can humanize AI content to ensure the most clear, concise, and compelling content that will move your customers and client to act? Contact me at randy@writingforhumans.co for a free, no obligations consultation on leveraging this powerful technology to grow your business.