Validate model performance.
Evaluate LLM response quality.
Uncover AI security risks.
Design agent-driven test automation.
Accelerate testing with AI.
Build frameworks and automated tests.
Define your AI testing approach.
Assess AI controls and evidence.
Extend your testing capacity.
Improve your testing strategy.
Validate AI model quality.
Test across browsers and devices.
Validate connected systems.
Assess performance under load.
Extend your QA capacity.
Delegate your QA delivery.
Written by Lina Rafi
Get experienced QA professionals to catch issues before your customers do.
It’s 4:45 on a Thursday afternoon. The pipeline is green, every test has passed, and the team ships the release feeling good about it. Forty minutes later, support starts getting messages. Customers can’t check out, a report shows the wrong numbers, and someone in the team chat asks the question nobody wants to hear: “But QA passed it, right?”
If you work in software, you have probably lived a version of this story. It feels unfair, because the tests did their job. They ran, they passed, and they still missed the bug. So what really went wrong?
That is what this post is about. We will walk through the most common reasons a test passes in QA and fails in production, show you how to spot each one, and share how teams close the gap for good. You don’t need to be an engineer to follow along. If you build, test or manage software, you’ll find something useful here.
A passing test is a small promise. It says that, under the exact conditions we set up, the code did what we expected. That is useful, but it is a narrow promise. Production is a very different place from a test run, and the bugs that escape live in the space between the two.
Teams call a bug that gets past testing a test escape. Almost every escape fits one of three stories:
The third story is the most common, and it is the one people overlook. When we say a test “passed,” we really mean it passed in QA’s version of the world. If that world is cleaner, smaller, faster and kinder than the real one, a green result tells you less than you think. The rest of this post is a tour of the ways those two worlds drift apart.
Different settings, versions, feature flags or migrations.
Real records contain missing values, duplicates and edge cases.
Real partners time out, return errors or change responses.
Rate limits, expired tokens, bad inputs and retries are missed.
Most teams test in a staging or QA environment that is meant to look like production. In practice, it rarely does. Maybe the database is a different version. Maybe a feature flag is switched on in one place and off in the other. Production might also run in another cloud region, use a bigger cache or sit behind tighter network rules.
Configuration is often the biggest culprit. Connection strings, API keys, timeout values, environment variables and security settings all live outside the code, so they are easy to forget. A single difference between a development settings file and the real production variables can change how the whole app behaves. When that happens, your test result was true, but it was true about a different system.
There is also the deployment itself. Build steps, database migrations and startup scripts can behave differently once they run on real infrastructure. A migration that finishes in a second on a tiny test table can lock a huge production table for minutes. Feature tests will never see that, because they only check the result, not the journey.
The fix is parity, which simply means making the two worlds match as closely as you can. Define your environments in code so they can be rebuilt the same way every time. Pin your versions, and compare settings between stages on a schedule. You won’t get a perfect copy, but every gap you close is one less surprise.
Bring in experienced QA specialists to uncover hidden bugs, strengthen test coverage, and reduce release risks.
Test databases are usually small, clean and tidy. Production data is old, huge and messy. It has empty fields, strange characters, duplicate records, giant entries and formats from three versions of your product ago. Real customers also have names with apostrophes, addresses in odd layouts and emoji in places you never expected.
This matters more than most people think. A query that runs instantly on 500 rows can crawl on 50 million. A form that works with plain text can break on a single apostrophe. And a report that looks perfect with ten demo accounts can show the wrong totals once real refunds, cancellations and edge cases pile up.
The answer is not to copy raw customer data into QA, because that creates privacy risks. Use anonymized samples that keep the shape of real data, including the ugly parts. Then add edge cases on purpose, like empty values, very long text, special characters and old records. If your data never surprises you, it is probably too clean to be useful.
Mocks are stand-ins for the real thing, like a pretend payment provider or a pretend email service. They make tests fast and stable, which is why teams love them. The catch is that a mock only knows what you taught it. It always answers correctly, on time, and in exactly the format you expected.
Real partners are not that polite. They time out, return odd status codes, add new fields or change the type of an old one. Imagine a payment provider that quietly starts sending a total as text instead of a number. Every test that used a mock still passes, and the first real order breaks. Some people call this the isolation illusion: the false belief that green unit tests mean the whole system is safe.
Mocks are not bad. They belong in unit tests, where you want to check one piece of logic alone. The trouble starts when mocks replace every dependency, even in integration tests, and nobody ever checks the real connection. Then your suite is really proving that your code works with an imaginary partner. At least one test for each critical integration should talk to the real service, or to something that truly behaves like it.
The same thing happens inside your own company. One team changes a database or API schema, perhaps by making a field required, and another team’s service keeps sending it empty. Each team’s tests pass, because each team tested its own piece. The break only appears when the two pieces meet in production. Contract testing helps here, because it checks the agreement between services instead of trusting each side on its own.
Modern apps run on APIs, both your own and other people’s. Many API tests check just one thing: send a good request, get a 200 back, move on. Production breaks in all the places that tests skip. Think of changed response fields, missing or expired login tokens, rate limits, retries, pagination edge cases, and older apps calling newer endpoints.
Rate limits are a sneaky example. A staging API key might have loose limits, so your load test sails through. Then the production key has a strict cap, and your batch job hits it twenty minutes after launch. Nothing in the code was wrong. The environment was just kinder than reality.
Good API testing covers bad inputs, missing permissions, error responses, timeouts and backward compatibility with older clients. For most products, solid API and integration testing gives the best return on effort. It runs faster than full user-journey tests and catches problems closer to where they start. If you only improve one layer this quarter, this is a strong choice.
Slow responses, timeouts, retries and time-zone differences.
Race conditions, deadlocks and exhausted connections.
Old browsers, slow phones, double taps and unusual journeys.
Coverage looks high, but assertions are weak or tests are unreliable.
Hotfixes, configuration changes and rushed deployments.
Outputs change across runs, prompts and connected tools.
On a developer’s laptop, everything answers in milliseconds. In production, a third-party service might take two seconds, and a busy database might take thirty. Timeouts that never fired in testing suddenly fire all the time. Retry logic that looked harmless starts piling requests on top of each other.
The network adds its own trouble. Real networks have delays, lost packets and limits on how many connections can be open at once. A service that sends many requests in parallel can work fine in a test, then hit a connection cap in production. Requests back up, everything slows down, and the system can collapse under its own weight.
Time itself is another quiet trap. Time zones, daylight saving changes and clock drift between servers cause bugs that only show on certain days or in certain places. A classic example is a test that builds a date without a time zone, while production expects a strict UTC timestamp. The test compares two dates that are both missing a zone, so they match and it passes. Real users then trigger the bug from a different time zone, often in the middle of the night.
Most tests run alone, one request at a time. Production is a crowd. Some bugs only appear when many things happen at once, such as race conditions, deadlocks, cache stampedes, and used-up connection pools. These are hard to catch because the timing has to be just right, and a quiet test suite never creates that timing.
Picture two services updating the same record at nearly the same moment. In testing, one always finishes first, so the result looks fine. Under real traffic, they overlap, and one update quietly erases the other. Everything works for a single user, and then fifty people hit the same row and the story changes.
Load tests can fool you too. A test may pass because it used a simple traffic pattern that looks nothing like your real users. Real traffic has particular queries, long sessions and busy moments, like a big sale or a payroll run. Build your load tests from real usage patterns, and add concurrency checks to your most important flows. Proper performance and load testing is the best way to see these problems before your customers do.
Automated user-journey tests click through the app the way a developer imagines people will. Real users do not. They use slow phones, old browsers and weak connections. They tap a button twice, hit the back button in the middle of a payment and leave a form half filled for an hour.
Many production bugs are also plain visual problems. A button hides behind another element, a form fails on a small screen, or a layout breaks in one browser. Logic tests will pass all of these, because the logic is fine. Visual regression checks, accessibility scans and testing on real browsers and devices catch what code-only tests cannot. That is also why human-led web and mobile testing still matters, because a person notices when something just feels wrong.
Strengthen software quality with expert functional, automation, integration, and performance testing.
Sometimes the problem is not the environment at all. The tests are just weaker than they look. A test can run a line of code without checking its result, so coverage goes up while safety does not. That is how a team can have very high coverage and still ship a bug. A better question than “how much did we cover?” is “how confident are we in these results?”
One way to check is mutation testing. A tool makes small, deliberate changes to your code that should break something. If your tests still pass, they are not really watching that part of the code. It is an honest way to find tests that look busy but guard nothing.
Flaky tests are another blind spot. A flaky test passes and fails on the same code. After a while, the team learns to rerun the build until it turns green, and real failures hide in the noise. Common causes include shared state, fixed waits, tests that depend on order, and uncontrolled time or network calls. Treat flakiness as a real defect with an owner, not as background noise.
Regression suites can quietly rot too. Teams add a test after every bug but rarely remove duplicates or review risky areas. Some tests get skipped, some check dead code, and some fake so much that the real behavior is never touched. A common example is a billing test that always passes because the billing call is faked to say “success.” A green dashboard only tells you that the tests that ran, passed.
Many escapes happen because something changed right before release. A last-minute requirement touches several parts of the app at once, and the team only has time to retest one of them. A quick config tweak or a hotfix goes out with less checking than the original feature got. When a change spans several components, tests that look at each part alone cannot see the full chain.
Release readiness is also more than passing tests. An alert that was never set up will not fire. A rollback plan that was never tried may not work. Teams that ship because the calendar says so, instead of because a readiness check passed, often find these gaps while customers are watching.
If your product includes AI, everything above applies, and a few new problems appear. A traditional feature gives the same answer every time, so one passing test means something. An AI feature may answer differently each time, so a single pass can be luck. Real users also write messy, unclear and unexpected prompts that your clean test prompts never covered.
AI features also depend on tools, APIs and data that can fail or change, and the model’s behavior can drift over time. So testing should include repeated runs, messy inputs, tool failures and recovery paths. Specialized AI and LLM testing also looks at accuracy, hallucinations, bias, safety and prompt reliability, not just whether a reply came back. After launch, keep watching, because behavior can change as real usage grows.
Go beyond basic pass-or-fail testing with expert LLM evaluations that reveal problems traditional QA may miss.
When a bug reaches production, don’t stop at the fix. A short review turns one bad day into a lasting improvement. Here is a simple routine you can follow after every serious incident. Keep it blameless, because the goal is to learn about the system, not to find a culprit.
After a few months, those tags start to show a pattern. Maybe most of your escapes come from data, or config, or one fragile integration. That pattern tells you where to invest your time, which beats guessing.
A test strategy is a plan for what to test, at which level, with what data and how often. It is not a tool choice. Start with risk instead of coverage numbers by asking what would hurt most if it broke. Then match the type of test to that risk.
Think in layers. Have many fast unit tests at the bottom, a solid middle of integration, API and contract tests, a small set of user-journey tests at the top, and checks in production after release. The exact shape depends on your product, and some products need more integration testing than others. The goal is to catch each kind of problem at the cheapest layer that can see it.
Keep your user-journey tests small and sharp. Use them for the flows that make or lose money, like signup, login, checkout and your core workflows. Run them in a production-like environment, keep each one independent, avoid fixed waits and use the same data setup every time. Anything else is usually better handled lower down.
Treat regression testing as a living suite. Review it every quarter, remove duplicates, strengthen weak checks and add tests where incidents happened. Tag tests by risk so the critical set runs on every pull request and the full set runs overnight. For flaky tests, set a simple policy: quarantine on sight, give each one an owner, fix or delete it by a deadline, and track the flaky rate.
No test suite covers everything, so strong teams also check quality after release. This is often called shifting right. The idea is to add safety nets exactly where testing ends. It does not replace pre-release testing, it backs it up.
A few practical tools do most of the work:
With these in place, a missed bug stays small. Instead of a long outage, you get a short blip that few customers notice. Testing in production sounds scary, but done carefully it is one of the safest habits a team can build. The real goal is not zero bugs, but bugs that are found fast and cost little.
Old-style QA sits at the end of the line. Testers receive a build and hunt for problems. Quality engineering spreads that work across the whole process. Developers own unit and integration tests, QA specialists design the strategy and build the frameworks, product managers write clear acceptance criteria before coding starts, and engineering and operations share responsibility for production health.
Last-minute surprises drop when everyone owns quality. Developers know how the code should behave, so they spot edge cases others miss. Testers bring an outside view and think like users. Both groups should share the work of integration testing, and the team should treat an escaped bug as a clue about the system, not a reason to blame a person.
Pick a few measures and watch the trend, not a single day:
Compare your numbers against your own past results. Another company’s numbers rarely fit your product. If your trend improves, you are closing the gap. If it doesn’t, your incident tags will tell you where to look next.
Reproduce the failure and capture its conditions.
Identify missing tests, weak assertions and environment differences.
Add the right unit, API, integration, contract or critical journey test.
Use canary releases, feature flags, monitoring and automatic rollback.
Track escaped defects, change failures, flaky tests and recovery time.
Review ten incidents, tag causes and assign flaky-test owners.
Add API and contract checks and simplify journey tests.
Improve parity, synthetic checks, canaries and rollback triggers.
You don’t need to rebuild everything at once. Focus on a few improvements each month:
Review your metrics at the end and pick the next set of gaps.
Fixing all this while still shipping features is hard, and many teams don’t have spare hands. That is where an outside QA partner can help. GigaTester is a division of Riseup Labs, and we work with product teams through managed QA projects, dedicated QA teams and individual specialist roles. We pair experienced testers with AI-assisted test design, while human experts stay responsible for strategy, risk and the final quality call.
Not sure where to begin? Start with a conversation. Share your product, your release plans and your recent incidents, and we will recommend a testing approach, team structure and timeline. You can also explore our QA consulting for process reviews, or our AI-augmented automation for building a regression suite that stays healthy. Teams can launch in under a week, backed by a 7-day risk-free guarantee.
Tests run under controlled conditions, and production is not controlled. Differences in data, settings, traffic, timing and third-party behavior let bugs slip through. Weak checks, heavy mocking and coverage gaps make it worse.
Start with your past incidents. Add tests only where bugs escaped, and place each one at the fastest layer that can catch the problem. Improving environment parity and data quality often helps more than adding more tests.
No. Coverage tells you which lines ran, not whether the results were checked. A suite can touch every line and still miss a bad input, a slow partner or a busy moment. Use coverage as a clue, and judge your suite by how confident it makes you.
Find the cause first, such as shared state, fixed waits, test order, time dependence or network calls. Quarantine the test, give it an owner and set a deadline to fix or remove it. Track the flaky rate so the problem stays visible.
Yes, carefully. Canary releases, feature flags, synthetic checks and good monitoring let you verify real behavior with limited risk. They work alongside pre-release testing and never replace it.
Enough to cover the flows that earn or lose money, and not much more. Most teams do well with a small, stable set backed by strong unit, API and integration tests. If your journey suite is slow and flaky, that is a sign to move some checks to a lower layer.
A test that passes in QA is telling the truth about QA. The problem is that production is a different place, with its own data, settings, speed, crowds and surprises. Closing that gap takes realistic environments and data, tests that check real behavior, a lean and healthy suite, and safety nets after release. Do those things, and a green pipeline will start to mean what everyone hopes it means.
Next time a bug slips through, resist the urge to say the tests failed. Ask what question the tests answered, and what question production asked instead. The difference between those two is where your next improvement lives. Start there, and keep going one escape at a time.
This page was last edited on 17 November 2026, at 7:17 am
Your email address will not be published. Required fields are marked *
Comment *
Name *
Email *
Website
Save my name, email, and website in this browser for the next time I comment.
Launch in less than a week - backed by our 7-day risk-free guarantee.
Welcome! My team and I personally ensure every project gets world-class attention, backed by experience you can trust.
By proceeding, you agree to our Privacy Policy
Thank you for filling out our contact form.A representative will contact you shortly.
You can also schedule a meeting with our team: