Using Git Bisect to Debug Production Issues Fast


Production broke. The app worked two weeks ago, it’s broken now. Somewhere in the 87 commits between then and now, someone introduced the bug. You could review each commit manually, or you could use git bisect and find the breaking commit in about seven tests through binary search.

I’ve used git bisect dozens of times over the years to track down mysterious bugs where the only information I have is “this worked before, it’s broken now, something between these two points caused it.” It’s one of those Git features that’s incredibly powerful but underused because people don’t know it exists.

How Binary Search Finds Bugs

Binary search is efficient. If you have 100 commits between a known good state and a known bad state, testing the middle commit tells you which half contains the bug. Then you test the middle of that half, narrowing it down again. Seven iterations finds the exact commit in a list of 100.

Git bisect automates this binary search process. You tell it which commit was good (the bug didn’t exist) and which commit is bad (the bug exists now). Git checks out the middle commit and asks you to test. You tell it whether this commit is good or bad, and it narrows down and repeats.

The basic workflow:

git bisect start
git bisect bad  # current commit is broken
git bisect good abc1234  # this old commit worked
# Git checks out middle commit
# You test and report result
git bisect good  # or 'git bisect bad'
# Git checks out next commit to test
# Repeat until Git identifies the breaking commit
git bisect reset  # return to original state

Each test eliminates half the remaining commits. The process finds the first bad commit with mathematical certainty in log2(n) tests.

Finding the Good Commit

The hardest part of using git bisect is identifying the known good commit. You need to find a point in history where the bug definitely didn’t exist.

Sometimes this is obvious. “It worked in production last week, so the commit from last week’s deployment must be good.” That’s your starting point.

Other times you need to investigate. Look at git log, try to remember when the feature was working, check deployment records or tags. Find any commit you’re confident didn’t have the bug and start there.

You don’t need to be precise about finding the earliest good commit. Any good commit works. If you know it worked a month ago but you’re not sure about two weeks ago, start with the month-old commit. The search will still work, it’ll just have more commits to search through.

Testing Each Commit

Git bisect checks out each commit for you to test. You need a reliable way to verify whether that commit is good or bad.

For bugs with automated tests, this is straightforward: run the test suite, check if the failing test passes or fails. The test is your oracle for good versus bad.

For bugs without tests (most production bugs in my experience), you need a manual test procedure. Reproduce the bug reliably, then execute that procedure on each bisect commit.

This requires making the bug reproducible. If it’s intermittent, bisect won’t work well because you can’t reliably classify commits as good or bad. You need deterministic reproduction.

Automating the Testing

If you can script the test, you can automate the entire bisect process:

git bisect start
git bisect bad HEAD
git bisect good abc1234
git bisect run ./test-script.sh

Git will binary search automatically, running your test script at each step, using the script’s exit code (0 for good, non-zero for bad) to navigate the search. When it finishes, you have the breaking commit identified without any manual intervention.

This works great for bugs with automated test coverage or when you can script reproduction. Write a script that reproduces the bug and exits with failure if the bug exists, success if it doesn’t. Let git bisect run handle the rest.

For my money, this is the cleanest way to use bisect when applicable. Set it up, go get coffee, come back to find the exact breaking commit identified.

Handling Untestable Commits

Sometimes you’ll hit a commit that doesn’t compile or won’t run for reasons unrelated to the bug you’re investigating. Maybe it’s a WIP commit, or it depends on environment setup that’s changed.

You can skip these commits:

git bisect skip

Git will choose a different commit to test and continue the search. Skipping a few commits is fine, but if you’re skipping many commits, the binary search efficiency breaks down and you might end up testing more commits than necessary.

If a range of commits is untestable (say, a series of refactoring commits that temporarily broke the build), you can give git bisect a range to skip:

git bisect skip abc1234..def5678

The Result: Identifying the Breaking Commit

When bisect finishes, it shows you the first bad commit—the exact commit that introduced the bug. This is incredibly valuable information.

You can see what changed in that commit, who wrote it, when it was committed. Often just seeing the diff immediately reveals what went wrong. Even if the root cause isn’t obvious, you’ve narrowed down from 87 commits to 1, which is a tractable debugging problem.

You can check out that commit, investigate thoroughly, understand what changed and why it broke. You can review the pull request, read the commit message, check if tests were modified or removed.

This is where the real debugging begins, but you’ve saved hours of searching by immediately identifying exactly where the bug was introduced.

Example: Debugging a Performance Regression

I used this recently for a performance regression. API response times had gone from 200ms to 800ms sometime in the past month. Profiling showed the slowdown was in a specific database query, but I couldn’t see what changed.

I started bisect with the current slow commit as bad and a month-old commit where I knew performance was good. I wrote a simple script that called the slow endpoint, measured response time, and exited with success if under 300ms, failure otherwise.

git bisect start
git bisect bad HEAD
git bisect good v2.3.0  # last known good release
git bisect run ./test-perf.sh

Bisect tested about eight commits and identified the breaking commit: a seemingly innocent change to how we eager-load associations in that query. The commit had added a join that caused a Cartesian product explosion for certain data patterns we didn’t test in development.

Total time from starting bisect to identifying the exact problem: about ten minutes. Manual review of 75 commits would have taken hours and I might have missed it.

When Bisect Won’t Help

Git bisect is powerful but not universally applicable. It works when:

  • The bug is reproducible deterministically
  • You can identify a good commit and a bad commit
  • Testing each commit is feasible (either automated or manual testing is reasonable)
  • The bug was introduced in a single commit (or at least can be pinpointed to one commit)

It doesn’t work well when:

  • The bug is intermittent or timing-dependent
  • You can’t reliably test commits (environment dependencies, long build times, complex setup)
  • The bug was introduced gradually across multiple commits
  • The regression is subtle and hard to detect mechanically

For these cases, other debugging approaches work better. But for the common scenario of “this worked, now it’s broken, something changed,” bisect is incredibly effective.

Tips for Using Bisect Effectively

Keep your commit history clean: Bisect is easier when each commit is self-contained and buildable. Avoid WIP commits in main branches, squash feature branches before merging, make sure each commit passes tests.

Write good commit messages: When bisect identifies the breaking commit, a clear commit message explaining what changed and why helps understand the bug faster.

Have automated tests: Even if tests didn’t catch the bug initially, having a test suite means you can often write a new test that fails with the bug, then use that test to bisect automatically.

Document your test procedure: When manually testing during bisect, write down exactly what you’re testing so you test consistently at each commit.

Use tags or releases as reference points: Knowing which released versions had the bug and which didn’t gives you good starting commits for bisect.

Making Bisect a Regular Tool

I’ve noticed that developers who know about git bisect use it regularly, and developers who don’t know about it never think to try binary search for debugging. It’s one of those tools that becomes obvious once you know it exists but isn’t discoverable through normal workflow.

Add it to your debugging toolkit. Next time you’re facing a regression where you know it worked before but don’t know what changed, try bisect before manually reviewing commits. You’ll probably save significant time.

The mathematical efficiency of binary search means bisect scales to large commit ranges. Even if you have 1000 commits between good and bad, bisect will find the breaking commit in about 10 tests. Manual review of 1000 commits is completely impractical.

Git bisect is one of those features that seems niche until you need it, then becomes indispensable. Add it to your workflow and you’ll wonder how you debugged regressions without it.