OpenAI o1 vs o1-mini: Which Model Is Better at Catching Bugs?

AI models are increasingly used to generate code—but how well do they review it?

Bug detection is fundamentally different from code generation. It requires reasoning, logic tracing, and the ability to catch subtle issues that don’t follow known patterns. In this post, we compare OpenAI’s o1 and o1-mini models on their ability to detect real-world software bugs across five programming languages.

🧪 The Evaluation Dataset

I wanted to have the dataset of bugs to cover multiple domains and languages. I picked sixteen domains, picked 2-3 self-contained programs for each domain, and used Cursor to generate each program in TypeScript, Ruby, Python, Go, and Rust.

Here are the programs we created for the evaluation:

Next I cycled through and introduced a tiny bug in each one. The type of bug I chose to introduce had to be:

1. A bug that a professional developer could reasonably introduce
2. A bug that could easily slip through linters, tests, and manual code review

Some examples of bugs I introduced:

1. Undefined \response\ variable in the ensure block
2. Not accounting for amplitude normalization when computing wave stretching on a sound sample
3. Hard coded date which would be accurate in most, but not all situations

At the end of this, I had 210 programs, each with a small, difficult-to-catch, and realistic bug.

A disclaimer: these bugs are the hardest-to-catch bugs I could think of, and are not representative of the median bugs usually found in everyday software.

📊 Results

Total Bugs Detected

• OpenAI o1: 15
• OpenAI o1-mini: 11

Detection by Language

• Go:

  • Both models: 2/42

• Python:

  • Both models: 2/42

• TypeScript:

  • o1: 4
  • o1-mini: 1

• Rust:

  • o1: 3
  • o1-mini: 2

• Ruby:

  • Both models: 4

While both models struggled across the board, o1 consistently outperformed o1-mini, especially in TypeScript and Rust. In Ruby and Python, they performed similarly.

💡 Analysis

Despite being the smaller, lighter-weight option, o1-mini did not demonstrate stronger reasoning capabilities in this benchmark. In fact, OpenAI o1 was slightly better overall, especially in languages with richer structural diversity like TypeScript.

This could suggest that:

• o1 has broader pattern exposure from training data, enabling better detection even in non-reasoning tasks.
• o1-mini may prioritize speed and simplicity over depth, making it less equipped to trace multi-step logic or concurrency issues.

Interestingly, both models struggled equally in Python and Go, despite these being among the most popular languages—indicating that real-world bugs can’t always be solved through memorized patterns alone.

🐞 A Bug Worth Highlighting

Test #2 — Python: Concurrency in Pathfinding

A concurrency issue emerged in a Python pathfinding implementation. A shared came_from dictionary could be modified asynchronously during the reconstruction_path method.

• o1 caught the bug
• o1-mini missed it

o1’s output:

"The method does not create an immutable snapshot of came_from, allowing concurrent modifications that can lead to inconsistent path reconstruction or runtime errors during the loop."

Spotting this required reasoning about mutation, timing, and control flow—something o1 handled but o1-mini missed. This highlights the importance of deeper logic tracing, especially for race conditions and shared state management.

✅ Final Thoughts

While both models are limited in their current bug detection capabilities, OpenAI o1 consistently outperforms o1-mini, especially in languages that demand more semantic structure or deeper reasoning.

• Use o1 for better bug detection accuracy, especially if you're reviewing TypeScript or Rust.
• Use o1-mini for lighter tasks or where compute efficiency matters more than precision.

Greptile uses models like these to detect real bugs in real PRs—before they reach production. Want to see what AI finds in your codebase? Try Greptile — no credit card required.