It’s 4:45 on a Thursday afternoon. The pipeline is green, every test has passed, and the team ships the release feeling good about it. Forty minutes later, support starts getting messages. Customers can’t check out, a report shows the wrong numbers, and someone in the team chat asks the question nobody wants to hear: “But QA passed it, right?”

If you work in software, you have probably lived a version of this story. It feels unfair, because the tests did their job. They ran, they passed, and they still missed the bug. So what really went wrong?

That is what this post is about. We will walk through the most common reasons a test passes in QA and fails in production, show you how to spot each one, and share how teams close the gap for good. You don’t need to be an engineer to follow along. If you build, test or manage software, you’ll find something useful here.

First, what does “passed in QA” really mean?

A passing test is a small promise. It says that, under the exact conditions we set up, the code did what we expected. That is useful, but it is a narrow promise. Production is a very different place from a test run, and the bugs that escape live in the space between the two.

Teams call a bug that gets past testing a test escape. Almost every escape fits one of three stories:

  • The test never existed, so nobody checked that scenario.
  • The test checked the wrong thing, so it passed for the wrong reason.
  • The test ran under different conditions, with data, settings or timing that did not match production.

The third story is the most common, and it is the one people overlook. When we say a test “passed,” we really mean it passed in QA’s version of the world. If that world is cleaner, smaller, faster and kinder than the real one, a green result tells you less than you think. The rest of this post is a tour of the ways those two worlds drift apart.

Tests pass in QA
↓
Production behaves differently
↓
Environment mismatch

Different settings, versions, feature flags or migrations.

Fix: Match configurations and infrastructure.
Unrealistic test data

Real records contain missing values, duplicates and edge cases.

Fix: Use realistic, anonymized data.
Mocked integrations

Real partners time out, return errors or change responses.

Fix: Run integration and contract tests.
API coverage gaps

Rate limits, expired tokens, bad inputs and retries are missed.

Fix: Test negative and failure paths.
↓
Test real behavior under production-like conditions

1. QA and production are not twins

Most teams test in a staging or QA environment that is meant to look like production. In practice, it rarely does. Maybe the database is a different version. Maybe a feature flag is switched on in one place and off in the other. Production might also run in another cloud region, use a bigger cache or sit behind tighter network rules.

Configuration is often the biggest culprit. Connection strings, API keys, timeout values, environment variables and security settings all live outside the code, so they are easy to forget. A single difference between a development settings file and the real production variables can change how the whole app behaves. When that happens, your test result was true, but it was true about a different system.

There is also the deployment itself. Build steps, database migrations and startup scripts can behave differently once they run on real infrastructure. A migration that finishes in a second on a tiny test table can lock a huge production table for minutes. Feature tests will never see that, because they only check the result, not the journey.

The fix is parity, which simply means making the two worlds match as closely as you can. Define your environments in code so they can be rebuilt the same way every time. Pin your versions, and compare settings between stages on a schedule. You won’t get a perfect copy, but every gap you close is one less surprise.

QA Passed. Production Still Broke?

Bring in experienced QA specialists to uncover hidden bugs, strengthen test coverage, and reduce release risks.

2. Your test data is too polite

Test databases are usually small, clean and tidy. Production data is old, huge and messy. It has empty fields, strange characters, duplicate records, giant entries and formats from three versions of your product ago. Real customers also have names with apostrophes, addresses in odd layouts and emoji in places you never expected.

This matters more than most people think. A query that runs instantly on 500 rows can crawl on 50 million. A form that works with plain text can break on a single apostrophe. And a report that looks perfect with ten demo accounts can show the wrong totals once real refunds, cancellations and edge cases pile up.

The answer is not to copy raw customer data into QA, because that creates privacy risks. Use anonymized samples that keep the shape of real data, including the ugly parts. Then add edge cases on purpose, like empty values, very long text, special characters and old records. If your data never surprises you, it is probably too clean to be useful.

3. Mocks and pretend partners

Mocks are stand-ins for the real thing, like a pretend payment provider or a pretend email service. They make tests fast and stable, which is why teams love them. The catch is that a mock only knows what you taught it. It always answers correctly, on time, and in exactly the format you expected.

Real partners are not that polite. They time out, return odd status codes, add new fields or change the type of an old one. Imagine a payment provider that quietly starts sending a total as text instead of a number. Every test that used a mock still passes, and the first real order breaks. Some people call this the isolation illusion: the false belief that green unit tests mean the whole system is safe.

Mocks are not bad. They belong in unit tests, where you want to check one piece of logic alone. The trouble starts when mocks replace every dependency, even in integration tests, and nobody ever checks the real connection. Then your suite is really proving that your code works with an imaginary partner. At least one test for each critical integration should talk to the real service, or to something that truly behaves like it.

The same thing happens inside your own company. One team changes a database or API schema, perhaps by making a field required, and another team’s service keeps sending it empty. Each team’s tests pass, because each team tested its own piece. The break only appears when the two pieces meet in production. Contract testing helps here, because it checks the agreement between services instead of trusting each side on its own.

4. API tests that only check the happy path

Modern apps run on APIs, both your own and other people’s. Many API tests check just one thing: send a good request, get a 200 back, move on. Production breaks in all the places that tests skip. Think of changed response fields, missing or expired login tokens, rate limits, retries, pagination edge cases, and older apps calling newer endpoints.

Rate limits are a sneaky example. A staging API key might have loose limits, so your load test sails through. Then the production key has a strict cap, and your batch job hits it twenty minutes after launch. Nothing in the code was wrong. The environment was just kinder than reality.

Good API testing covers bad inputs, missing permissions, error responses, timeouts and backward compatibility with older clients. For most products, solid API and integration testing gives the best return on effort. It runs faster than full user-journey tests and catches problems closer to where they start. If you only improve one layer this quarter, this is a strong choice.

5. Time, speed and the network

A bug appears only in real usage
↓
Timing and network

Slow responses, timeouts, retries and time-zone differences.

Check: Latency, network faults and time handling.
Traffic and concurrency

Race conditions, deadlocks and exhausted connections.

Check: Realistic load and simultaneous requests.
Real user behavior

Old browsers, slow phones, double taps and unusual journeys.

Check: Cross-browser, mobile and exploratory tests.
Weak or flaky tests

Coverage looks high, but assertions are weak or tests are unreliable.

Check: Mutation testing, assertions and flaky-test causes.
Late release changes

Hotfixes, configuration changes and rushed deployments.

Check: Release readiness and tested rollback plans.
AI and LLM variability

Outputs change across runs, prompts and connected tools.

Check: Repeated evaluations, safety and recovery paths.
↓
Validate under realistic conditions and unexpected behavior

On a developer’s laptop, everything answers in milliseconds. In production, a third-party service might take two seconds, and a busy database might take thirty. Timeouts that never fired in testing suddenly fire all the time. Retry logic that looked harmless starts piling requests on top of each other.

The network adds its own trouble. Real networks have delays, lost packets and limits on how many connections can be open at once. A service that sends many requests in parallel can work fine in a test, then hit a connection cap in production. Requests back up, everything slows down, and the system can collapse under its own weight.

Time itself is another quiet trap. Time zones, daylight saving changes and clock drift between servers cause bugs that only show on certain days or in certain places. A classic example is a test that builds a date without a time zone, while production expects a strict UTC timestamp. The test compares two dates that are both missing a zone, so they match and it passes. Real users then trigger the bug from a different time zone, often in the middle of the night.

6. One user in a test, thousands in real life

Most tests run alone, one request at a time. Production is a crowd. Some bugs only appear when many things happen at once, such as race conditions, deadlocks, cache stampedes, and used-up connection pools. These are hard to catch because the timing has to be just right, and a quiet test suite never creates that timing.

Picture two services updating the same record at nearly the same moment. In testing, one always finishes first, so the result looks fine. Under real traffic, they overlap, and one update quietly erases the other. Everything works for a single user, and then fifty people hit the same row and the story changes.

Load tests can fool you too. A test may pass because it used a simple traffic pattern that looks nothing like your real users. Real traffic has particular queries, long sessions and busy moments, like a big sale or a payroll run. Build your load tests from real usage patterns, and add concurrency checks to your most important flows. Proper performance and load testing is the best way to see these problems before your customers do.

7. Real people don’t follow scripts

Automated user-journey tests click through the app the way a developer imagines people will. Real users do not. They use slow phones, old browsers and weak connections. They tap a button twice, hit the back button in the middle of a payment and leave a form half filled for an hour.

Many production bugs are also plain visual problems. A button hides behind another element, a form fails on a small screen, or a layout breaks in one browser. Logic tests will pass all of these, because the logic is fine. Visual regression checks, accessibility scans and testing on real browsers and devices catch what code-only tests cannot. That is also why human-led web and mobile testing still matters, because a person notices when something just feels wrong.

Catch Bugs Before Your Customers Do

Strengthen software quality with expert functional, automation, integration, and performance testing.

8. The tests themselves have blind spots

Sometimes the problem is not the environment at all. The tests are just weaker than they look. A test can run a line of code without checking its result, so coverage goes up while safety does not. That is how a team can have very high coverage and still ship a bug. A better question than “how much did we cover?” is “how confident are we in these results?”

One way to check is mutation testing. A tool makes small, deliberate changes to your code that should break something. If your tests still pass, they are not really watching that part of the code. It is an honest way to find tests that look busy but guard nothing.

Flaky tests are another blind spot. A flaky test passes and fails on the same code. After a while, the team learns to rerun the build until it turns green, and real failures hide in the noise. Common causes include shared state, fixed waits, tests that depend on order, and uncontrolled time or network calls. Treat flakiness as a real defect with an owner, not as background noise.

Regression suites can quietly rot too. Teams add a test after every bug but rarely remove duplicates or review risky areas. Some tests get skipped, some check dead code, and some fake so much that the real behavior is never touched. A common example is a billing test that always passes because the billing call is faked to say “success.” A green dashboard only tells you that the tests that ran, passed.

9. Change shows up late

Many escapes happen because something changed right before release. A last-minute requirement touches several parts of the app at once, and the team only has time to retest one of them. A quick config tweak or a hotfix goes out with less checking than the original feature got. When a change spans several components, tests that look at each part alone cannot see the full chain.

Release readiness is also more than passing tests. An alert that was never set up will not fire. A rollback plan that was never tried may not work. Teams that ship because the calendar says so, instead of because a readiness check passed, often find these gaps while customers are watching.

A special case: AI and LLM features

If your product includes AI, everything above applies, and a few new problems appear. A traditional feature gives the same answer every time, so one passing test means something. An AI feature may answer differently each time, so a single pass can be luck. Real users also write messy, unclear and unexpected prompts that your clean test prompts never covered.

AI features also depend on tools, APIs and data that can fail or change, and the model’s behavior can drift over time. So testing should include repeated runs, messy inputs, tool failures and recovery paths. Specialized AI and LLM testing also looks at accuracy, hallucinations, bias, safety and prompt reliability, not just whether a reply came back. After launch, keep watching, because behavior can change as real usage grows.

Green Tests Don’t Guarantee Reliable AI

Go beyond basic pass-or-fail testing with expert LLM evaluations that reveal problems traditional QA may miss.

How to find out where your bug escaped

When a bug reaches production, don’t stop at the fix. A short review turns one bad day into a lasting improvement. Here is a simple routine you can follow after every serious incident. Keep it blameless, because the goal is to learn about the system, not to find a culprit.

  1. Reproduce it. Confirm the exact input, state and environment.
  2. Find the missing layer. Ask whether a unit, integration, API, user-journey or monitoring check should have caught it.
  3. Check for an existing test. If one existed, work out why it passed.
  4. Compare the environments. Look for differences in data, config, versions and traffic.
  5. Add the right test at the right layer. Don’t default to another slow end-to-end test.
  6. Record the cause. Tag it with a simple label, such as data, config or integration.

After a few months, those tags start to show a pattern. Maybe most of your escapes come from data, or config, or one fragile integration. That pattern tells you where to invest your time, which beats guessing.

How to build a testing strategy that catches real failures

A test strategy is a plan for what to test, at which level, with what data and how often. It is not a tool choice. Start with risk instead of coverage numbers by asking what would hurt most if it broke. Then match the type of test to that risk.

Think in layers. Have many fast unit tests at the bottom, a solid middle of integration, API and contract tests, a small set of user-journey tests at the top, and checks in production after release. The exact shape depends on your product, and some products need more integration testing than others. The goal is to catch each kind of problem at the cheapest layer that can see it.

Keep your user-journey tests small and sharp. Use them for the flows that make or lose money, like signup, login, checkout and your core workflows. Run them in a production-like environment, keep each one independent, avoid fixed waits and use the same data setup every time. Anything else is usually better handled lower down.

Treat regression testing as a living suite. Review it every quarter, remove duplicates, strengthen weak checks and add tests where incidents happened. Tag tests by risk so the critical set runs on every pull request and the full set runs overnight. For flaky tests, set a simple policy: quarantine on sight, give each one an owner, fix or delete it by a deadline, and track the flaky rate.

Shift right: add safety nets after release

No test suite covers everything, so strong teams also check quality after release. This is often called shifting right. The idea is to add safety nets exactly where testing ends. It does not replace pre-release testing, it backs it up.

A few practical tools do most of the work:

  • Canary releases send a small share of traffic to the new version while you watch error rates.
  • Feature flags let you turn a feature off in seconds without a new deployment.
  • Synthetic monitoring and observability run scheduled checks against production and use logs, metrics and traces to spot unusual behavior quickly.
  • Automatic rollback undoes a release when errors or slowness cross a limit you set.

With these in place, a missed bug stays small. Instead of a long outage, you get a short blip that few customers notice. Testing in production sounds scary, but done carefully it is one of the safest habits a team can build. The real goal is not zero bugs, but bugs that are found fast and cost little.

It’s a team habit, not just a QA job

Old-style QA sits at the end of the line. Testers receive a build and hunt for problems. Quality engineering spreads that work across the whole process. Developers own unit and integration tests, QA specialists design the strategy and build the frameworks, product managers write clear acceptance criteria before coding starts, and engineering and operations share responsibility for production health.

Last-minute surprises drop when everyone owns quality. Developers know how the code should behave, so they spot edge cases others miss. Testers bring an outside view and think like users. Both groups should share the work of integration testing, and the team should treat an escaped bug as a clue about the system, not a reason to blame a person.

Which numbers show it’s working?

Pick a few measures and watch the trend, not a single day:

  • Escaped defect rate compares bugs found by customers with bugs found before release.
  • Change failure rate shows how often a release needs a fix or a rollback.
  • Flaky test rate shows how much you can trust your suite.
  • Time to detect and restore shows how fast you notice and recover. Watch pipeline duration too, because slow feedback makes people skip steps.

Compare your numbers against your own past results. Another company’s numbers rarely fit your product. If your trend improves, you are closing the gap. If it doesn’t, your incident tags will tell you where to look next.

A simple 90-day plan

Production incident

Reproduce the failure and capture its conditions.

↓
Find the escape point

Identify missing tests, weak assertions and environment differences.

↓
Improve test coverage

Add the right unit, API, integration, contract or critical journey test.

↓
Protect the next release

Use canary releases, feature flags, monitoring and automatic rollback.

↓
Measure the outcome

Track escaped defects, change failures, flaky tests and recovery time.

↓
MONTH 1
Audit and stabilize

Review ten incidents, tag causes and assign flaky-test owners.

MONTH 2
Strengthen tests

Add API and contract checks and simplify journey tests.

MONTH 3
Improve release safety

Improve parity, synthetic checks, canaries and rollback triggers.

You don’t need to rebuild everything at once. Focus on a few improvements each month:

  • Month 1: Review your last ten production incidents, tag each by cause and missed layer, and quarantine your flakiest tests with owners.
  • Month 2: Add API and contract tests for your most critical services, and trim your user-journey tests down to the essentials.
  • Month 3: Improve environment parity and test data, then add canary releases, synthetic checks and automatic rollback triggers.

Review your metrics at the end and pick the next set of gaps.

Where GigaTester can help

Fixing all this while still shipping features is hard, and many teams don’t have spare hands. That is where an outside QA partner can help. GigaTester is a division of Riseup Labs, and we work with product teams through managed QA projects, dedicated QA teams and individual specialist roles. We pair experienced testers with AI-assisted test design, while human experts stay responsible for strategy, risk and the final quality call.

Not sure where to begin? Start with a conversation. Share your product, your release plans and your recent incidents, and we will recommend a testing approach, team structure and timeline. You can also explore our QA consulting for process reviews, or our AI-augmented automation for building a regression suite that stays healthy. Teams can launch in under a week, backed by a 7-day risk-free guarantee.

Subscribe to our Newsletter

Stay updated with our latest news and offers.
Thanks for signing up!

FAQs

Why do tests pass in QA but fail in production?

Tests run under controlled conditions, and production is not controlled. Differences in data, settings, traffic, timing and third-party behavior let bugs slip through. Weak checks, heavy mocking and coverage gaps make it worse.

How can we reduce production bugs without adding hundreds of tests?

Start with your past incidents. Add tests only where bugs escaped, and place each one at the fastest layer that can catch the problem. Improving environment parity and data quality often helps more than adding more tests.

Is 100 percent code coverage enough?

No. Coverage tells you which lines ran, not whether the results were checked. A suite can touch every line and still miss a bad input, a slow partner or a busy moment. Use coverage as a clue, and judge your suite by how confident it makes you.

How do we fix flaky tests?

Find the cause first, such as shared state, fixed waits, test order, time dependence or network calls. Quarantine the test, give it an owner and set a deadline to fix or remove it. Track the flaky rate so the problem stays visible.

Should we test in production?

Yes, carefully. Canary releases, feature flags, synthetic checks and good monitoring let you verify real behavior with limited risk. They work alongside pre-release testing and never replace it.

How much user-journey testing is enough?

Enough to cover the flows that earn or lose money, and not much more. Most teams do well with a small, stable set backed by strong unit, API and integration tests. If your journey suite is slow and flaky, that is a sign to move some checks to a lower layer.

Final thoughts

A test that passes in QA is telling the truth about QA. The problem is that production is a different place, with its own data, settings, speed, crowds and surprises. Closing that gap takes realistic environments and data, tests that check real behavior, a lean and healthy suite, and safety nets after release. Do those things, and a green pipeline will start to mean what everyone hopes it means.

Next time a bug slips through, resist the urge to say the tests failed. Ask what question the tests answered, and what question production asked instead. The difference between those two is where your next improvement lives. Start there, and keep going one escape at a time.

This page was last edited on 17 November 2026, at 7:17 am