🔍 Read the full analysis: How AI Perplexity Relates To Intelligent Behavior on ThorstenMeyerAI.com
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
OpenAI has published internal measurements of its models’ pretraining perplexity and says the metric could help developers compare progress and discuss shared safety standards. Its examples connect lower loss with capabilities in earlier models, but do not establish that specific values guarantee particular abilities. Forecasts about future capabilities and a possible research acceleration remain uncertain.
OpenAI has published internal measurements of its models’ pretraining perplexity, arguing that the metric could help AI developers compare progress and inform shared safety standards. The release offers examples from OpenAI’s model history, but it does not show that a particular perplexity value reliably predicts a specific capability.
Perplexity measures predictive performance: it reflects how well a model predicts the next token in a held-out dataset, with lower values indicating better predictions. OpenAI says it has tracked the measure over time, including for models beginning with GPT-3.5, and is now making some internal results public. The company describes the measurements as a way to follow pretraining progress even as task-specific benchmarks change or become less useful.
In its account, OpenAI associates reported log-perplexity of about 0.99 for GPT-3.5 and about 0.91 for GPT-4 with stages of capability development. It says later models reached losses below 0.8, linking those stages to fluent conversation, productive use of chain-of-thought reasoning, and abilities in mathematics and cybersecurity. These are the company’s descriptions of its own models, not evidence that those numerical levels cause or guarantee the listed skills.
OpenAI also says a model at roughly 0.76 log-perplexity began producing results on open mathematics problems that the post characterizes as groundbreaking. It anticipates further pretraining research changes as soon as 2027 and new capabilities in the approximate 0.68–0.7 range. Those figures and dates are forecasts, not confirmed outcomes.
Perplexity and Intelligence A shared measure?
OpenAI published internal pretraining measurements and argues they could help developers compare progress and shape shared safety standards. The numbers offer a signal, not a guarantee of future abilities.
What the metric measures
Perplexity measures how well a model predicts the next token in a held-out dataset; lower values mean better predictions.
A model learns from a large dataset.
Perplexity tracks predictive performance on held-out data.
Prompting, fine-tuning, and reinforcement learning can elicit or shape what a model does.
OpenAI’s reported progression
The company associates lower log-perplexity with capabilities observed in its models; the pattern does not prove a value causes or guarantees an ability.
| Reported level | OpenAI’s account |
|---|---|
| ≈0.99 | GPT-3.5; fluent conversation |
| ≈0.91 | GPT-4; productive chain-of-thought reasoning |
| Below 0.8 | Later models; mathematics and cybersecurity capabilities |
| ≈0.68–0.7 | Range where OpenAI anticipates new capabilities |
Why a shared measure could matter
Task benchmarks can change or saturate as models improve. Comparable pretraining measurements could give researchers and policymakers a common reference.
- Track predictive performance over time
- Compare progress if developers use shared methods
- Inform discussion of common safety standards
- What a model can reliably do
- How safely it behaves
- How much oversight development requires
What remains unresolved
The supplied material does not include the underlying chart or enough detail to independently compare the measurements.
- No standardized reporting or audit process is specified.
- No formal industry standard, timetable, or commitments from other developers are announced.
- The possible 2027 research acceleration and future capabilities are projections, not confirmed outcomes.
A common measure could give researchers and policymakers a way to track one dimension of model progress across time, especially when task benchmarks are revised, saturated, or difficult to compare. OpenAI argues that comparable reporting could support discussion of shared safety standards as well as technical progress.
But perplexity alone cannot establish what a system can do in practice, how reliably or safely it behaves, or what oversight it needs. A comparable number would be most useful alongside capability evaluations, risk assessments, and information about training and testing methods. Without those, the metric could be mistaken for a complete account of a model’s abilities when it measures only a narrower aspect of predictive performance.
Products Worth Considering

Sonos Era 100 Wireless Smart Speaker with Alexa, Black

Jackery Solar Generator 1000 v2 with 200W Solar Panel, 1024Wh LiFePO4 Battery

Bose SoundLink Plus Portable Bluetooth Speaker, Black
As an affiliate, we earn on qualifying purchases.
From Pretraining to Model Abilities
OpenAI describes model development as starting with pretraining on a large dataset, followed by approaches such as prompting, fine-tuning, and reinforcement learning that can elicit or shape capabilities. The company argues that tracking pretraining is useful because downstream tests can change as systems improve, making comparisons across generations harder.
The post uses the history of GPT-2, GPT-3, GPT-3.5, and GPT-4 to explain why it sees value in monitoring predictive performance. It also says the capabilities associated with particular perplexity levels were not predictable in advance. That history is a rationale for continued measurement, not proof that future models will follow the same path.
OpenAI further raises the possibility that AI systems could contribute to research on pretraining algorithms. The post’s suggestion that advances in mathematics could make faster progress in deep-learning research conceivable by 2027 is a projection; it also notes that mathematics and deep learning are different fields.
Limits of the Published Data
The available account does not provide the underlying chart, full dataset details, or enough methodological information to independently compare OpenAI’s measurements with those from other developers. It also does not specify a standardized reporting process, independent audit arrangements, or how particular values should inform safety decisions.
It remains unproven that a given perplexity level maps reliably to a specific capability or risk. The timing and nature of future abilities, the possibility of AI-assisted advances in pretraining research, and the projected 2027 acceleration are all uncertain. OpenAI’s release does not announce commitments from other developers or adoption of a shared standard.
The next practical test is whether other AI developers publish comparable measurements and whether researchers can agree on shared datasets, evaluation conditions, and reporting methods. Those steps would help determine whether figures from different organizations can be meaningfully compared.
OpenAI has not announced a timetable for an industry standard or a set of safety actions tied to particular perplexity values. Until such methods and evidence exist, the metric is best treated as one measure of pretraining progress, rather than a standalone indicator of capability or safety.
Key Questions
What does AI perplexity measure?
Perplexity measures how well a model predicts the next token in a held-out dataset. Lower values indicate better predictive performance on that evaluation, but do not by themselves describe all of a model’s abilities.
What measurements did OpenAI publish?
OpenAI shared internal pretraining measurements, citing approximate log-perplexity values of 0.99 for GPT-3.5 and 0.91 for GPT-4, and saying later models reached losses below 0.8. The release does not provide enough detail in the supplied account for independent cross-company comparisons.
Does a lower perplexity value guarantee a new capability?
No. OpenAI associates lower values with capabilities observed in its models, but the material does not establish that a particular value reliably produces or guarantees a specific ability.
Is there already an AI industry standard for perplexity reporting?
No formal industry standard is announced in the source material. Comparable datasets, reporting methods, and independent scrutiny would be needed before measurements from different developers could be assessed consistently.
Will AI research accelerate by 2027?
OpenAI suggests that further pretraining research changes could come as soon as 2027, but this is a forecast rather than a confirmed outcome. The timing and nature of any acceleration remain unclear.
Primary source: OpenAI · via ThorstenMeyerAI.com
