While AI models are increasingly capable of generating code, their ability to detect real bugs is a much harder test — one that requires understanding logic, control flow, and intent. In this post, we benchmark two models from OpenAI — o1-mini and 4o — to evaluate how well they catch hard bugs across five programming languages.
🧪 The Evaluation Dataset
I wanted to have the dataset of bugs to cover multiple domains and languages. I picked sixteen domains, picked 2-3 self-contained programs for each domain, and used Cursor to generate each program in TypeScript, Ruby, Python, Go, and Rust.
Here are the programs we created for the evaluation:
Next I cycled through and introduced a tiny bug in each one. The type of bug I chose to introduce had to be:
- A bug that a professional developer could reasonably introduce
- A bug that could easily slip through linters, tests, and manual code review
Some examples of bugs I introduced:
- Undefined \response\ variable in the ensure block
- Not accounting for amplitude normalization when computing wave stretching on a sound sample
- Hard coded date which would be accurate in most, but not all situations
At the end of this, I had 210 programs, each with a small, difficult-to-catch, and realistic bug.
A disclaimer: these bugs are the hardest-to-catch bugs I could think of, and are not representative of the median bugs usually found in everyday software.
📊 Results
Overall Bugs Caught
- o1-mini: 11 bugs
- 4o: 20 bugs
That’s nearly a 2× improvement with 4o over o1-mini.
By Language
-
Go:
- o1-mini: 2
- 4o: 4
-
Python:
- o1-mini: 2
- 4o: 6
-
TypeScript:
- o1-mini: 1
- 4o: 4
-
Rust:
- o1-mini: 2
- 4o: 3
-
Ruby:
- o1-mini: 4
- 4o: 3
While 4o outperformed o1-mini in most languages, Ruby was the exception — where o1-mini actually performed better.
💡 Interpretation
The results reveal a clear trend: 4o’s reasoning capabilities give it an edge in detecting subtle bugs, especially in Python and TypeScript — languages where logic and context often matter more than syntax alone.
- o1-mini does better when pattern recognition alone is enough — or when it happens to have better representation in training data, as may be the case with Ruby.
- 4o appears better equipped for reasoning-first detection, catching bugs that require planning and inference rather than just matching syntax or familiar patterns.
That said, the Ruby result is interesting. o1-mini catching more bugs there may suggest:
- Better memorization of common Ruby bug patterns
- More favorable tuning for Ruby's syntax
- Or simply noise due to low absolute bug counts
In any case, 4o’s improvement across the board — even modestly in Rust and Go — suggests that reasoning gives it broader generalization.
🐞 A Bug Worth Highlighting
Test #1 — Ruby: Audio Processing Logic Bug
In the TimeStretchProcessor class of a Ruby audio library, a bug appeared in the normalize_gain calculation. The gain was fixed, rather than scaled based on the stretch_factor. This caused output amplitude to swing wildly depending on how much the sample was sped up or slowed down.
- o1-mini caught the bug
- 4o missed it
This was one of the rare cases where pattern recognition outperformed reasoning. o1-mini likely recognized the bug from similar training patterns involving formula-based gain miscalculations.
It’s a great reminder: reasoning helps generalize, but pattern familiarity still matters — especially for domain-specific or numerical errors.
✅ Final Thoughts
While both models have their strengths, OpenAI 4o clearly performs better overall for AI code review — especially for languages and bug types that benefit from logical deduction over rote memorization.
- Use 4o if you want better detection coverage across languages and logic-heavy bugs
- Use o1-mini if you want a lighter, faster model and can tolerate lower accuracy
Greptile uses models like 4o in production to review real pull requests and catch bugs before they hit production. Want to see how it works on your repo? Try Greptile — no credit card required.