LLM Bug Detection Comparison: OpenAI o3-mini vs DeepSeek R1

Bug detection is one of the hardest problems in software engineering — and in AI. Unlike code generation, which is pattern-heavy, bug detection demands inference, logic, and context awareness. The best models don’t just write — they reason.

In this post, we compare two such models: OpenAI’s o3-mini and DeepSeek’s R1. Both are compact, fast, and capable. But how well do they detect hard bugs across real codebases?

🧪 Evaluation Setup

We tested both models on a curated benchmark of 210 real-world-inspired programs, each with a subtle but critical bug. The languages: Python, TypeScript, Go, Rust, and Ruby.

Each model received the same prompts and context. The goal: detect the bug.

Here are the programs we created for the evaluation:

Next I cycled through and introduced a tiny bug in each one. The type of bug I chose to introduce had to be:

1. A bug that a professional developer could reasonably introduce
2. A bug that could easily slip through linters, tests, and manual code review

Some examples of bugs I introduced:

1. Undefined `response` variable in the ensure block
2. Not accounting for amplitude normalization when computing wave stretching on a sound sample
3. Hard coded date which would be accurate in most, but not all situations

At the end of this, I had 210 programs, each with a small, difficult-to-catch, and realistic bug.

A disclaimer: these bugs are the hardest-to-catch bugs I could think of, and are not representative of the median bugs usually found in everyday software.

📊 Results

Overall Bug Detection

• OpenAI o3-mini: 37 bugs
• DeepSeek R1: 23 bugs

By Language

• Python: o3-mini found 7, DeepSeek found 3
• TypeScript: o3-mini found 7, DeepSeek found 6
• Go: o3-mini found 7, DeepSeek found 3
• Rust: o3-mini found 9, DeepSeek found 7
• Ruby: o3-mini found 7, DeepSeek found 4

While o3-mini outperformed across the board, DeepSeek R1 was competitive in Rust and TypeScript — hinting at solid reasoning performance, especially in less conventional bug patterns.

🧠 Analysis

Why does o3-mini win overall?
It likely comes down to planning. o3-mini appears to follow a structured reasoning process before generating output. That makes it stronger in bugs that require deduction — especially concurrency issues or broken logic.

DeepSeek’s strengths show up in areas where training data is thinner (e.g., Ruby, Rust), suggesting its Llama-based foundation generalizes better when hardcoded bug patterns aren’t present.

But across Python, Go, and other common languages — where subtle bugs still benefit from extensive pattern memory — o3-mini takes the lead.

🧠 Reasoning Matters More in Some Languages

In languages like Ruby and Rust, where training data is more limited, models can’t rely on memorization. Here, reasoning becomes the difference-maker — and both o3-mini and DeepSeek show strength.

In Python and TypeScript, the game shifts toward pattern recognition. That’s where o3-mini shines — blending memory with structured logic.

The lesson? Model performance is highly language-dependent, and no single approach works universally.

🧩 A Bug Worth Highlighting

Test 2 — Go: Race Condition in ServiceRegistry

In this case, OpenAI o3-mini detected a critical concurrency bug: a shared map instances accessed across goroutines without synchronization.

• o3-mini's Output:

  "The most critical bug identifies a thread-safety issue in ServiceRegistry.instances accessed concurrently by multiple threads (Flask request handlers and async health checks) without proper synchronization, leading to race conditions and potential data corruption."

• DeepSeek R1 missed the issue entirely.

This wasn’t just a syntax error. It required recognizing async execution paths, shared mutable state, and missing locks — a textbook example where structured reasoning outperforms token prediction.

✅ Final Thoughts

This comparison makes one thing clear: OpenAI o3-mini is currently the stronger model for AI code review — especially in languages where pattern-rich bugs and logical reasoning intersect.

That said, DeepSeek R1 has promise, especially in lower-resource languages or scenarios requiring generalization. Its performance in Rust and TypeScript was solid, and continued improvements could make it a contender in reasoning-first applications.

Greptile uses models like o3-mini in production to automatically catch real bugs in PRs — concurrency issues, logic flaws, you name it. Want to see what it finds in your codebase? Try Greptile — no credit card required.