Most explanations about how AI Detectors work will mention Perplexity and Burstiness. This post explains what they are, how to calculate a basic version and why they result in false positives.

Perplexity

Perplexity is calculated by taking the probability assigned to each subsequent token by the language model. It gives an indication of how much of the text surprised the language model. Lower perplexity means the text was easier to predict. Higher perplexity means it was harder.

Language models will generally have low perplexities as they are sampling the most likely tokens. Human written texts will often have higher perplexities due to the variability of choice of words, tangents and idioms.

A basic example of this with GPT2 is:

import math import torch from transformers import GPT2LMHeadModel, GPT2TokenizerFast

tokenizer = GPT2TokenizerFast.from_pretrained("gpt2") model = GPT2LMHeadModel.from_pretrained("gpt2").eval()

def perplexity(text: str) -> float: enc = tokenizer(text, return_tensors="pt", truncation=True, max_length=1024) with torch.no_grad(): loss = model(enc, labels=enc["input_ids"]).loss return math.exp(loss.item())

This value will vary based on the model used. It won't give you a universal "AI score" but just an indication of how likely the text was to be AI based compared to the same model.

Burstiness

Humans usually write in a way that has a mixture of both short and long sentences. Language models often have a more even flow. This is what Burstiness is trying to measure. The easiest way to do this is by measuring how long your sentences are.

import re

def burstiness(text: str) -> float: sentences = [s for s in re.split(r"(?<=[.!?])\s+", text.strip()) if s] lengths = [len(s.split()) for s in sentences] if len(lengths) < 2: return 0.0 mean = sum(lengths)/len(lengths) variance = sum( (n - mean) 2 for n in lengths)/len(lengths) return (variance ** 0.5)/mean #Coefficient of variation

Higher values indicate more variable sentence lengths. This is a basic example of how you could calculate burstiness. True detectors use much more complex features.

Why they give false positives

These are both style based indicators and are therefore susceptible to:

Formal/technical documents often have low perplexities and even flow, making them appear like they were written by a machine.

Non-native English often has common words and simple and consistent sentences.

Short texts will not have enough of a signal and can get almost any score for a 2 sentence paragraph.

Text that has been heavily edited (by humans editing AI text or AI editing human text) will fall into the middle ground.

We have seen this ourselves. We tested 8 texts and a human written paragraph gave a 99% AI score. This does not mean they are useless but you shouldn't take them as proof.

What to look out for in a useful detector

When building or choosing a detector look out for:

Probability.

Sentence level information.

Clear limitations of what the detector can do. This should state what the minimum text length is, what languages it works on and that it is probabilistic.

No automatic actions.

Our RewriteAI AI Detector gives you a probability of how likely the text is AI and Sentence level information. Some other terminology you may see can be found in our AI Detection Glossary.

Key Points

Perplexity = How predictable the text is.

Burstiness = How varied the flow of the text is.

They are both style based so will cause issues with formal, non-native and short texts.

Never take the score as proof. Just take it as evidence.

If you need to prove who wrote the text keep your drafts, notes and version history. This will be a much better indicator than any score.

Let me know if you have had any surprising results from an AI Detector in the comments.