AI coding tools have collapsed the distance between an idea and a working product. A founder with Lovable, Bolt, Cursor, Replit or v0 can ship screens, an API, a database schema and a one-click deploy in a weekend. That speed is real, and it is why production readiness for AI-built apps has become its own discipline. The demo works, the first users arrive, and then come the questions no prompt answered: where do the API keys live, who can read which rows, what happens when the database goes down at 3 a.m., and why did the LLM bill triple last week?
None of those questions are about whether the code is elegant. They are about everything around the code: secrets, access control, deploys, monitoring, backups, cost and compliance. That layer decides whether a product survives contact with real users, investors and enterprise buyers.
In this guide I'll walk through why AI-built apps tend to break after launch, the nine areas we check at Gart Solutions, how a readiness engagement is structured, how to stop the next AI commit from undoing the work, and a self-assessment you can run this week.
TL;DR
AI tools build features fast but rarely build what production needs: secret management, authorisation, monitoring, tested backups, cost limits and compliance evidence.
Research shows AI-generated code introduces OWASP Top 10 vulnerabilities in 45% of tested tasks, and real incidents in 2025 exposed data and wiped databases.
Production readiness for AI-built apps covers nine areas, from secrets and access to compliance gaps.
The effective approach: find everything, fix everything outside the code, and hand the founder a fix pack their own AI can apply.
Guardrails for AI coding (rules, CI gates, permissions, alerts) stop the next AI commit from undoing the audit.
Why AI-built apps break after git push
AI tools are optimised for one outcome: the feature works when you click it. Production asks a different question: does it still work when someone tries to break it, when a dependency goes down, when traffic spikes at 9 a.m. on launch day, or when you need to roll back a bad release in five minutes?
Prompts almost never describe those non-functional requirements, so the model doesn't build them. The result is a predictable split between what AI handles well and what quietly stays out of frame.
Every item in the right-hand column is invisible in a demo. A leaked key doesn't throw an error. A missing authorisation check doesn't break the UI. An untested backup looks exactly like a working one until the day you need it. That is why founders are often surprised: the app felt finished.
The evidence: what actually goes wrong
This is not a theoretical concern. Three data points from the last year show the pattern clearly.
45% of AI coding tasks introduced an OWASP Top 10 vulnerability (Veracode, 2025)
170 of 1,645 scanned Lovable apps had exposed database tables (CVE-2025-48757)
1,206 executive records wiped by an AI agent during a code freeze (Replit, 2025)
AI-generated code fails security tests at a high rate. Veracode's 2025 GenAI Code Security Report tested more than 100 LLMs on 80 coding tasks chosen for known weakness patterns. In 45% of cases the generated code introduced a vulnerability from the OWASP Top 10, and when the model could pick between a secure and an insecure approach, it picked the insecure one 45% of the time. Java was the worst performer at over 70% failure, while Python, C# and JavaScript sat between 38% and 45%. Veracode also found that newer models wrote more functional code but not more secure code. (Veracode, Help Net Security)
A single default exposed hundreds of endpoints. In 2025 a researcher scanned 1,645 apps from Lovable's public showcase and found 170 of them, roughly 10%, with 303 endpoints where Supabase tables could be read or written using only the public anon key, because Row Level Security had never been enabled. Exposed data included emails, home addresses and in some cases API keys. The issue was registered as CVE-2025-48757. (RapidDev summary)
An AI agent wiped a production database during a code freeze. In July 2025, SaaStr founder Jason Lemkin was nine days into a twelve-day experiment with Replit's agent when it ran destructive database commands despite an explicit freeze, erasing records on 1,206 executives and about 1,196 companies. The agent then told him a rollback was impossible; he recovered the data himself. At the time, the same database served preview, testing and production. Replit responded by introducing automatic separation between development and production databases. (The New Stack, heise)
Look at the root causes. None of these were missing features. A configuration default, an absent access policy, an agent holding credentials it shouldn't have had, environments that weren't separated. Every one of them sits around the code, not in the business logic.
Code scanning is getting cheap. Production responsibility isn't.
The market has noticed the problem, and it is answering in several different ways. There is no single name for the service yet, so it helps to see the options side by side.
ApproachWhat you getTrade-offsVibe coding cleanup agenciesRefactoring, tests, partial rewritesProject-based pricing; changes your code; often doesn't own infrastructureReport-only auditsA senior review written up as findingsLow price point (from around $199), but no fixesDone-for-you freelancersSecrets, auth, monitoring, backupsAround $1,500 for ten days; depends on one personBoutique production readiness auditsNine-area review, risk register, 90-day roadmapExample pricing around $6,500 for one week; fixes often out of scopeAutomated launch-readiness scannersRepository scan in one commandFree or cheap; can't fix infrastructure or own uptime
The pattern is clear: scanning a repository is quickly becoming a free, one-command operation. That's good news. But a scanner can't configure backups and prove the restore works, it can't answer a 200-question security questionnaire from your first enterprise customer, and it can't take responsibility for your uptime. Those jobs need people, process and infrastructure ownership.
That is where we think production readiness belongs: not in rewriting the code, and not in another PDF of findings, but in owning the production layer and handing founders precise fixes for the rest.
The nine areas of production readiness for AI-built apps
After working through AI-built products on Supabase, Firebase and Vercel stacks, we group production risk into nine areas. For each one, here is what we look for, how it typically fails in AI-built apps, and what "ready" looks like.
01. Secrets and access
What we look for: which keys sit in git history or in the client bundle, and who holds production access.
How it fails: AI tools frequently place keys directly in source files "to get it working", then move them to environment variables later, but git history keeps every earlier version. Frameworks expose any variable with a NEXT_PUBLIC_ or VITE_ prefix to the browser, and a privileged key such as Supabase's service_role key in a client bundle bypasses Row Level Security entirely. On the access side, we regularly see a single shared admin login, former contractors who still have access to the cloud console, and no MFA on GitHub, Vercel or the database provider.
What ready looks like: secrets live in a secret manager (HashiCorp Vault, a cloud provider's secret store, or the platform's encrypted environment settings), every key that was ever exposed has been rotated, a secret scanner such as Gitleaks runs on every pull request, and there is a written list of who has production access, with MFA enforced.
02. Attack surface
What we look for: whether every API route checks who is calling and what they are allowed to do.
How it fails: AI-generated backends usually check that a user is logged in but not that they are allowed to touch a specific record. Change an ID in the URL and you see someone else's invoice. Admin routes are reachable by any authenticated user. On Supabase, tables are created without Row Level Security, or with a policy like USING (true) that lets everyone read everything. CORS is set to *, there is no rate limiting on login or signup, and file uploads accept anything.
What ready looks like: an authorisation matrix (roles × actions) that the code actually enforces, RLS enabled on every table with policies scoped to the current user, rate limits on authentication and expensive endpoints, sensible security headers, and a baseline dynamic scan with a tool such as OWASP ZAP.
03. AI-specific risks
What we look for: whether one user can burn your LLM budget or hijack a prompt.
How it fails: many AI features call a model on every request with no per-user quota, so a single script or a curious user can generate a four-figure bill overnight. User input is concatenated straight into prompts, which opens the door to prompt injection. Model output is rendered as HTML, which turns a jailbreak into cross-site scripting. Agents with tool access get broad permissions, and personal data is sent to third-party model providers without anyone having checked the terms.
What ready looks like: per-user and per-tenant quotas, hard spend limits configured at the provider, model output treated as untrusted input, the narrowest possible permissions for any tool the model can call, logging of prompts and costs, and a documented decision about what data may be sent to which model vendor.
04. Dependencies and integrations
What we look for: known CVEs, abandoned packages, and what breaks when a vendor is down.
How it fails: AI tools install whatever package solves the problem in front of them, including outdated versions with known vulnerabilities and, occasionally, package names that don't exist at all, which attackers can register. Integrations with payment, email or LLM providers have no timeouts or retries, so one slow third-party API freezes the whole app. Webhooks are accepted without verifying their signatures.
What ready looks like: software composition analysis (we use Trivy, among others) running in CI, automated dependency updates, a short list of critical third parties with a defined behaviour when each one fails, timeouts and retries on every external call, and verified webhook signatures.
05. Release safeguards
What we look for: whether you can ship on a Friday and roll back in minutes.
How it fails: changes go straight from the AI editor to production. There is no staging environment, or staging shares a database with production, which is exactly what made the Replit incident possible. Database migrations are run by hand and can't be reversed. Tests, if they exist, don't block a deploy. The coding agent holds production credentials.
What ready looks like: a CI/CD pipeline with automated checks that must pass before merge, separate development, preview and production environments with separate credentials, reviewed and reversible migrations, a one-command rollback, and feature flags for risky changes.
06. Observability
What we look for: whether you hear about an outage from an alert, not from a user.
How it fails: the founder finds out the app is down from a tweet or a support email. There is no uptime check, no error tracking, and logs exist only in the hosting dashboard for a few days. When something does break, nobody can reconstruct what happened.
What ready looks like: uptime checks on the critical paths (sign-up, login, payment, the core feature), error tracking, structured logs retained long enough to investigate incidents, alerts routed to a phone rather than an inbox, simple service-level objectives for the flows that make money, and a one-page runbook for the most likely failures.
07. Data and recovery
What we look for: whether a restore was ever tested, and how much data you would lose.
How it fails: backups are "on" because the platform says so, but nobody has restored one, nobody knows how old the latest usable copy is, and point-in-time recovery turns out to be a paid add-on that was never enabled. Backups live in the same account as the primary database, so a compromised account takes both.
What ready looks like: automated backups with a defined recovery point objective (how much data you can afford to lose) and recovery time objective (how long you can be down), a restore drill that has actually been run and documented, and at least one copy stored outside the primary account.
Not sure your backups would actually restore? Gart Solutions' managed cloud operations and SRE practice sets up monitoring, alerting and disaster recovery for growing products, and runs the restore drill with you. Backup and Disaster Recovery Services
08. Scale and cost
What we look for: where the first limit is, and what each user costs in cloud and LLM spend.
How it fails: AI-generated data access often makes one query per item in a list, which is invisible with ten users and crippling with ten thousand. Serverless functions exhaust database connections. Nobody knows the cost per active user, so a successful launch can feel like a financial emergency. There are no budget alerts on the cloud account.
What ready looks like: a load test of the critical path that identifies the first bottleneck before customers do, connection pooling and caching where it matters, a simple unit-economics model (infrastructure and LLM cost per user), and budget alerts on every paid account.
09. Compliance gap
What we look for: what stands between you and a GDPR review or a SOC 2 security questionnaire.
How it fails: there is no record of what personal data is stored where, no process for deleting a user's data on request, no list of sub-processors (including the LLM vendor), and no evidence trail for access reviews or change management. The first enterprise prospect sends a security questionnaire and the deal stalls for weeks.
What ready looks like: a data inventory, a deletion process that works, signed data processing agreements, a short set of security policies that match reality, and the evidence a security reviewer will ask for. For regulated domains such as health or finance, a clear decision about which regulatory track actually applies. Compliance Consulting Services
Who needs a production readiness review, and when
Production readiness is not for every project. It is most valuable for:
Founders and small teams who built their product in Lovable, Bolt, Cursor, Replit or v0.
Teams on Supabase, Firebase or Vercel without a dedicated DevOps or SRE engineer.
Products with real users or first revenue, where an incident now has a cost.
B2B SaaS companies moving towards enterprise customers, where security reviews are part of every deal.
Timing matters even more than profile. In our experience, the review pays off most at four moments:
Launch: 30 to 90 days before going public, or right after a Product Hunt launch brings traffic.
Fundraising: before an investor's technical due diligence.
Enterprise sales: when the first security questionnaire lands in your inbox.
Incident: after an outage, a data leak, or a cloud or LLM bill that suddenly spiked.
It is a poor fit if you only have an idea and no product yet, if you are looking for a stamp that says "everything is fine", or if what you really need is new features or a UX redesign.
How Gart Solutions approaches production readiness
Three principles separate this work from both cleanup agencies and report-only audits.
We don't rewrite your code. No promises to redo the React front end or rework your business logic. The code your AI wrote stays yours.
We take ownership of production. Everything around the code: infrastructure, deploys, data, observability, cost and compliance. This is where Gart has spent its history, across 50+ projects with engineers who average 8.2 years of experience.
We give you ready-made fixes for the code. For the issues that do live in the code, you get precise tasks and prompts your own AI tools can execute, and we verify the result.
Find everything, fix everything outside the code
Find everything. Automated scans with SonarQube, Semgrep, Trivy, Gitleaks and OWASP ZAP, plus a senior engineer's review across all nine areas.
Fix everything outside the code. CI/CD, infrastructure as code with Terraform, secrets in Vault, monitoring, backups with a tested restore, disaster recovery, FinOps and compliance groundwork.
Fix pack for the code. Prioritised tasks with acceptance criteria, plus ready-to-use prompts for Cursor or Claude Code. You apply them; we verify with a re-scan.
The fix pack is deliberate. A founder who built with Cursor or Claude Code already has a capable engineer on hand: the AI. What is usually missing is a precise specification of what to change, in what order, and how to know it's done. We provide that specification, then confirm the result with a second scan.
From a free scan to ongoing support
The engagement is a ladder. Each step is useful on its own, and you only move up when it makes sense.
StepProductTimingWhat you getStep 1Launch Risk ScanFreeAn automated scan plus a 30-minute call. You get your top five risks.Step 2Production Readiness Audit5–7 business days, fixed scopeNine-area scorecard, risk register, 30/60/90 roadmap and fix pack.Step 3Hardening Sprint2–4 weeksInfrastructure fixes and guardrails for AI coding, with a before-and-after scorecard.Step 4Run & ScaleMonthlyManaged SRE, guardrail maintenance, and fractional CTO or SOC 2 support.
What you get from the audit
Every deliverable is a working document, not a slide of generic advice.
Nine-area scorecard. One page for the founder and the investor.
Risk register. Each risk, what breaks, how urgent it is, and the effort to fix.
30/60/90 roadmap. Critical items first, quick wins up front.
Fix pack for the code. Tickets and prompts for Cursor and Claude Code.
Readout call. 60 minutes with your team; recording on request.
Re-scan. We confirm the fixes and update the scorecard.
The before-and-after scorecard is often the most valuable item. Founders show it to investors during due diligence and to enterprise buyers during security reviews, because it answers the question both of them are really asking: does this team understand its own risks?
What this looks like in practice
A recent engagement illustrates the approach. The client was a solo founder without a technical co-founder, building a healthtech product for elderly patients. The MVP was built in Lovable with Supabase underneath. Rewriting it was never on the table. Instead, we compared managed Supabase with a self-hosted setup on a three-year total cost of ownership (roughly $53K against $93K) and recommended staying on the managed platform. We mapped the regulatory track that actually applied at that stage rather than defaulting to the heaviest framework, and split the path to production into a seven-week plan in two phases with fixed budget caps.
This builds on earlier infrastructure work, including a national healthcare platform where we used data masking in development and test environments to meet GDPR and HIPAA requirements.
Guardrails for AI coding: keeping the next commit from undoing the audit
There is a problem with any one-off audit of an AI-built app: it decays. A week after the fixes land, the coding agent commits a new key, adds a route without an authorisation check, or runs a migration against the wrong database. The model hasn't learned anything from the audit. It only knows what is in its context and what its environment allows.
Engineers increasingly describe this as harness engineering: a coding agent is a model plus the environment that constrains it. A vibe coder usually has the model and some prompts. The harness is what we add.
Guides (rules for the agent): AGENTS.md, CLAUDE.md and Cursor rules written from the audit findings, so the fix pack becomes permanent.
Sensors (ci gates): Gitleaks, Semgrep, Trivy and tests on every pull request. An unsafe AI-generated change simply doesn't merge.
Permissions (what the agent can touch): Separate environments, preview deployments, and no production credentials for the coding agent.
Observability (regressions in production): Alerts and monitoring catch whatever slips through after an AI commit.
Here is a simplified example of the kind of rules we derive from audit findings and commit to the repository:
markdown
# AGENTS.md (excerpt)
## Security rules (from the production readiness audit)
- Never put secrets in source files. Read them from environment variables only.
- Never use a variable with a NEXT_PUBLIC_ prefix for a secret.
- Every new Supabase table must enable Row Level Security and define
policies scoped to auth.uid() in the same migration.
- Every API route must check both authentication and the caller's role
before reading or writing data. Use the helper in lib/authz.ts.
- Never run migrations against production. Production deploys go through CI only.
- Every call to an LLM provider must go through lib/llm.ts, which enforces
per-user quotas.
Rules alone aren't enough, because agents don't always follow instructions. That is why the CI gates and permission boundaries matter: if the agent ignores a rule, the pull request fails, and if it tries to touch production, it doesn't have the credentials to do so.
Guardrails are a required part of the Hardening Sprint, and keeping them current is part of Run & Scale.
What production readiness is not
Being explicit about scope builds more trust than any list of benefits, so here it is.
IncludedNot included✓ Nine-area audit with a scorecard✕ Rewriting or refactoring your application✓ Infrastructure fixes: CI/CD, IaC, secrets, monitoring, backups, DR✕ New features, UX or design work✓ Fix pack for the code and guardrails for AI coding✕ A guarantee of "no vulnerabilities" or a certificate✓ Before-and-after verification✕ A full penetration test (available separately or via a partner)✓ Ongoing support: SRE, fractional CTO, compliance✕ Public teardowns of your app without your consent
If your product does need a deeper refactor, we'll say so and point you to a partner that specialises in it. If you need a full penetration test, we can arrange one separately or with a partner, ideally after the hardening work, so the testers spend their time on real problems rather than missing basics.
Facing your first security questionnaire? Gart Solutions helps SaaS teams map their real infrastructure to GDPR, ISO 27001 and SOC 2 expectations, so security reviews stop stalling deals. Learn about our compliance architecture work →
A self-assessment you can run this week
You don't need us to start. Answer these eighteen questions honestly. Every "no" or "I'm not sure" is a risk worth looking at.
Secrets and access
Have you scanned your full git history, not just the current files, for secrets?
Is MFA enforced on GitHub, your hosting platform, your database provider and your cloud account?
Attack surface 3. If a logged-in user changes an ID in a request, are they blocked from seeing someone else's data? 4. Is Row Level Security enabled on every table, with policies that scope rows to the current user?
AI-specific risks 5. Is there a per-user limit on LLM usage, and a hard spend cap at the provider? 6. Is model output escaped before it is rendered in the browser?
Dependencies and integrations 7. Do you get an automatic alert when a dependency has a known vulnerability? 8. Do you know what your app does when your payment or LLM provider is down?
Release safeguards 9. Is there a staging environment with its own database and its own credentials? 10. Can you roll back the last release in under ten minutes?
Observability 11. Would you get a phone alert within five minutes if sign-up stopped working? 12. Can you see the errors your users hit in the last 24 hours?
Data and recovery 13. Have you restored a backup in the last 90 days and confirmed the data was intact? 14. Do you know how many hours of data you would lose in the worst case?
Scale and cost 15. Do you know the first component that will fail if traffic grows ten times? 16. Do you know your monthly infrastructure and LLM cost per active user?
Compliance 17. Could you delete all data about a specific user within 30 days of a request? 18. Could you answer a standard security questionnaire without stalling a deal?
If you answered "no" to more than five, you are in good company: that is typical for an AI-built product at launch stage. It is also a strong signal to fix the basics before your users, an investor or an attacker find them for you.
Conclusion
AI has made building software dramatically faster, and that is not going to reverse. What it hasn't changed is what production demands: protected secrets, enforced access, safe releases, early warnings, recoverable data, predictable costs and evidence for the people who need to trust you.
The good news is that this layer is well understood. It doesn't require rewriting your product. It requires a systematic check across nine areas, infrastructure work done by people who have run production systems before, precise fixes for the code your AI can apply, and guardrails that keep the next commit from undoing the progress.
Your AI wrote the app. Making sure it survives production is a different job, and it is worth doing before your users discover why.
Fedir Kompaniiets
Co-founder & CEO, Gart Solutions · Cloud Architect & DevOps Consultant
Fedir is a technology enthusiast with over a decade of diverse industry experience. He co-founded Gart Solutions to address complex tech challenges related to Digital Transformation, helping businesses focus on what matters most — scaling. Fedir is committed to driving sustainable IT transformation, helping SMBs innovate, plan future growth, and navigate the "tech madness" through expert DevOps and Cloud managed services. Connect on LinkedIn.
A field playbook for incident response, blast radius control, observability and disaster recovery — borrowed from the engineers who stabilise collapsed buildings.
Rescue engineering is the discipline that takes over after a structural collapse: engineers who triage damaged buildings in minutes, shore up what is still standing and monitor it continuously so rescue teams can work without becoming casualties themselves. It is engineering in a broken system, with incomplete information and a hard clock.
That is also an accurate description of a serious production incident. At Gart Solutions we spend a large part of our time in exactly that regime — a Kubernetes cluster that will not schedule, a migration that stalled halfway, a database failing over at 3 a.m., a cloud bill that tripled overnight. Normal engineering assumptions have stopped applying, load paths are not what the diagram says, and every change you make could make things worse.
This article is a DevOps playbook written around that idea. Below you'll find how we triage incidents, how we stabilise systems before investigating them, how we keep blast radius small, what to instrument, and how to build disaster recovery that has actually been tested. The rescue engineering parallels are there because they are genuinely useful models, not decoration.
TL;DR
Triage first: decide severity, scope and ownership in minutes — not root cause.
Stabilise before you debug. Roll back, fail over or shed load, then investigate with evidence preserved.
Keep headroom for the "lateral loads" of production: retry storms, traffic spikes, failing dependencies.
Control blast radius with canary releases, cells, feature flags and least privilege.
Treat observability like structural monitoring — alert on symptoms and SLO burn, not on every cause.
Run more than one recovery path, and test restores on a schedule.
What rescue engineering actually is
Rescue engineering is a branch of civil, structural and geotechnical engineering focused on assessing and stabilising severely damaged structures so search and rescue teams can operate. In the United States it sits inside FEMA's Urban Search and Rescue system; internationally it is coordinated through INSARAG.
Three details matter for our purposes.
Triage is time-boxed. A Structures Specialist assesses a building in no more than 15 minutes, with a whole sector triaged within two hours. There is no time for modelling; decisions come from visible failure patterns. Structures too unstable to work on are marked "No Go" until proper equipment arrives.
Stabilisation comes before entry. In a trench rescue, walers and pneumatic struts are placed against the walls from outside the hazard zone before any rescuer climbs in. Passive protection is not enough — the shoring has to actively hold the ground in place.
Monitoring runs throughout the operation. Tiltmeters and crack meters watch the structure while people work inside it, triggering evacuation alarms if movement passes a threshold. Triage is redone after every aftershock, because the building is not the same structure it was an hour ago.
Time-boxed triage, stabilise before you enter, monitor continuously, re-assess after every shock. That is also a description of a mature incident management practice.
Rescue engineering practiceDevOps equivalentGart Solutions service15-minute triage, "No Go" structuresSeverity classification, change freeze on fragile systemsIncidents ManagementShore the walls before enteringRoll back, fail over or shed load before debuggingInfrastructure Reliability ServiceDesign factor and lateral load allowanceCapacity headroom, autoscaling limits, load testingScaling & Performance OptimizationTiltmeters and alarm thresholdsSLOs, golden signals, burn-rate alertsMonitoring and ObservabilityThe L-zone around the trenchBlast radius control: canaries, cells, feature flagsDevOps Services, CI/CDParallel drilling Plans A, B and CBackups, replicas, multi-region failover, tested restoresBackup and Disaster Recovery (DRaaS)Standardised hazard markingShared incident taxonomy, runbooks, service ownershipTechnical Support, DevOps Consulting
Triage: the first ten minutes of an incident
The most expensive mistake in incident response is starting with "why". Rescue teams answer three different questions first: how bad is it, who is affected, and what do we have to work with.
Assign roles immediately. Even a three-person team benefits from separating the incident commander (decides, does not type), the operations lead (makes changes) and the communications lead (updates stakeholders and the status page). Without that split, the person with the deepest knowledge ends up writing Slack updates instead of fixing the system.
Classify severity with written criteria. Vague severity levels produce inconsistent responses. Ours are defined by user impact and reversibility, not by which team owns the component:
SEV1 — core user journeys are broken or data is at risk. Wake people up, open a call.
SEV2 — significant degradation or a broken path with a workaround. Immediate response, business hours escalation.
SEV3 — contained issue, no meaningful user impact yet. Ticket and schedule.
Declare your "No Go" systems. In a collapse, some structures are off-limits until the right equipment arrives. In production, the equivalent is an explicit decision that certain actions are prohibited during the incident — no schema migrations, no restarting a primary with replication lag, no manual edits to state that Terraform manages. Write them down before the incident, not during it.
Time-box the triage itself. If ten minutes of investigation has not produced a working hypothesis, stop investigating and stabilise instead. That single rule cuts mean time to recovery more than most tooling changes.
Stabilise before you debug
Nobody enters an unshored trench. Yet engineers routinely debug a live system that is still moving — tailing logs while the error rate climbs, attaching profilers to a pod that is being OOM-killed, reading a query plan while the connection pool saturates.
Stabilisation options, roughly in order of how often they work:
Roll back the last change. Most incidents follow a deployment or config change. kubectl rollout undo, revert the Helm release, re-apply the previous Terraform state. If your last deploy is not trivially reversible, that is the finding of your next postmortem.
Fail over. Promote a replica, shift traffic to a healthy region or availability zone, switch DNS or the load balancer to a warm standby.
Shed load. Rate-limit, drop non-critical traffic, disable expensive endpoints, pause async consumers and batch jobs. A degraded service is better than one that is down.
Turn features off. Feature flags let you disable the one code path causing the problem without a deploy — the cheapest stabilisation available if you have invested in them beforehand.
Scale out, carefully. Adding capacity helps a resource shortage and makes a thundering-herd problem worse. Know which one you have before you scale.
Preserve the evidence before you restart anything. Rescue engineers document a structure before they change it. Take the snapshot, capture pod logs and heap dumps, export the metrics window, note the exact timestamps. Restarting is often the right move, and it also destroys the state you need to explain the incident afterwards. <!-- promo:🛟 -->
Most outages run long for procedural reasons, not technical ones. Gart Solutions helps teams build incident processes, runbooks and reliability practices that shorten recovery — and reduce how often incidents happen at all. See our SRE and managed cloud operations →
Design factors: the headroom you need for the loads you didn't plan
Timber rescue shoring runs on a thin safety margin — far thinner than a permanent structure — so engineers compensate by adding lateral bracing for forces they cannot predict. Every vertical shore is designed to resist a sideways push equal to a percentage of the weight it carries, and that requirement is raised where aftershocks are expected.
Production has the same lateral loads: retry storms, traffic spikes, a slow dependency, a noisy neighbour, a cold cache after a restart. Most platforms have no explicit allowance for any of them.
Practical headroom rules we apply on client platforms:
Know your real ceiling. Load test to failure at least once, in a production-like environment, so the limit is a measured number rather than a guess.
Size requests and limits deliberately. In Kubernetes, missing or wrong resource requests are the most common cause of cascading node pressure. Set requests from observed usage, keep limits realistic and use Pod Disruption Budgets for anything with a quorum.
Account for autoscaling latency. HPA reacts in tens of seconds; the cluster autoscaler plus node boot is minutes. Your headroom has to cover the gap, or you need pre-warmed capacity for known peaks.
Make retries safe. Exponential backoff with jitter, capped attempts, circuit breakers and idempotent handlers. Naive retries turn a brief blip into a self-inflicted denial of service.
Watch the quiet quotas. Cloud API rate limits, connection pool sizes, file descriptors, IP address space in a subnet and NAT gateway ports cause more incidents than CPU does.
Blast radius control: the L-zone of your platform
Around every trench there is a zone extending from the edge to a distance equal to the trench depth. No heavy equipment, no traffic, no spoil piles — because a surcharge near the edge is what triggers the second collapse.
The DevOps version is deciding, in advance, how far a single failure or a single mistake can travel.
Deploy progressively. Canary or blue-green releases with automated rollback on error-rate or latency thresholds. If a bad release reaches 100% of users, the deployment pipeline is the problem.
Partition the system. Separate clusters or namespaces per environment, cell-based or per-tenant isolation for large customers, and separate cloud accounts or subscriptions for production and non-production.
Restrict who can push on the edge. No standing human access to production, changes through IaC and pipelines, review on Terraform plans and approvals for destructive operations. Most severe incidents we are called into after the fact involve a manual change nobody reviewed.
Freeze changes when the ground is unstable. During a live incident and immediately after it, unrelated deployments are additional load on a structure that is already moving.
Isolate the network. Network policies, private endpoints and least-privilege IAM stop a compromised or misbehaving component from taking the rest of the platform with it.
Observability: tiltmeters on the load-bearing columns
During a long extrication, sensors on the structure report movement to the millimetre and sound an alarm before something falls. They monitor a handful of things that matter, with thresholds set in advance, and the alarm has one meaning: get out.
Most monitoring setups we inherit do the opposite — thousands of metrics, hundreds of alerts, no agreement about which ones mean "stop".
What we put in place instead:
Service level objectives. Define what "working" means for each critical journey — availability, latency, error rate — and alert on error budget burn rate rather than on isolated threshold crossings.
The golden signals first. Latency, traffic, errors and saturation per service, with everything else as supporting detail during investigation.
Alert on symptoms, page on impact. High CPU is information; checkout failing is an alert. Anything that pages a human at night must be both urgent and actionable, with a linked runbook.
Traces and structured logs. Distributed tracing turns "the app is slow" into "this call to the payment provider takes 4 seconds", which is the difference between an hour of guessing and a five-minute fix.
One place to look. Prometheus and Grafana for metrics, ELK, Fluentd or Graylog for logs, correlated by request ID and consistent labels. Fragmented tooling costs you minutes exactly when minutes are expensive.
One more borrowed habit: re-triage after aftershocks. After recovery a system is not back to normal — caches are cold, queues are backed up, a hotfix is in place, scaling limits were raised. Keep heightened monitoring for the rest of the day and re-check assumptions after each follow-up change.
Disaster recovery: drill more than one hole
When 33 miners were trapped roughly 700 m underground in Chile in 2010, rescuers ran three independent drilling plans in parallel — a raise-borer, an air-rotary rig and an oil platform rig. Plan B broke through first. The other two were not wasted effort; they were the reason a single failure could not end the rescue.
Most disaster recovery plans we audit are Plan A only: one backup job, one region, one restore procedure, one person who knows how it works.
What an actual recovery capability requires:
Written RTO and RPO per system. Different workloads deserve different answers. Without agreed numbers, "we have backups" is a feeling, not a plan.
Independent recovery paths. Backups that are not in the same account or subscription as production, with immutability or object lock so ransomware and a bad script cannot delete them. Cross-region replicas for the systems that justify the cost.
Infrastructure as code as a recovery mechanism. If your environment exists only as clicked-together resources, rebuilding it is archaeology. Terraform, Pulumi, Bicep or CDK make re-provisioning a pipeline run.
Kubernetes-specific plans. Cluster state, etcd or managed control plane recovery, persistent volume snapshots, and a tool such as Velero for namespace-level restores. A cluster is not a backup of itself.
Scheduled restore drills. An untested backup is an assumption. Restore into an isolated environment on a schedule, measure how long it takes and fix what you find. Nearly every first drill we run with a new client uncovers something — a missing secret, an undocumented dependency, a restore that takes six hours against a two-hour RTO.
A rollback in every change. The Chilean rescue capsule had a bottom escape hatch in case it jammed mid-shaft. Every deployment deserves the same assumption: a reverse path that has actually been exercised.
<!-- promo:📡 -->
Backups are not a recovery plan until someone restores them. Gart Solutions designs and tests Backup and Disaster Recovery (DRaaS) setups with explicit RTO and RPO targets, plus the monitoring to know when you need them. Explore our DevOps and platform engineering services →
Shared language: the INSARAG marking of your platform
Rescue teams from different countries can hand over a worksite because the marking system is standardised: a painted box with the site ID, the team, the assessment level and the date, hazards written above it, the triage category below and an arrow pointing at the access route. No conversation required.
Engineering organisations lose time in incidents for the opposite reason — nobody is sure who owns a service, where its runbook lives or what its dependencies are. Fixes that cost little and pay off in the first incident:
A service catalogue with a named owner and an on-call rotation for every production component.
Runbooks next to the alerts that trigger them, kept short and current.
One place where incident state lives — severity, commander, timeline, current hypothesis — so anyone joining can read in rather than ask.
Blameless postmortems with a small number of committed, assigned actions, and a review of whether last month's actions actually shipped.
How Gart Solutions works on failing infrastructure
Gart Solutions is a Kyiv-based team focused on DevOps, cloud solutions and infrastructure. Our engineers average 8.2 years of experience, around 70% are senior-level certified professionals, and we have delivered more than 50 successful projects. <!-- stats -->
8.250+10+70%Years of average experienceSuccessful projectsSenior and middle specialistsSenior-level certified engineers
Our engagement model follows the same sequence as a structured deployment: a free consultation to identify where the pressure is, a technical audit and architecture vision before anything is touched, alignment on targets, implementation, documentation and reporting for the client's own team, and ongoing maintenance and support after delivery.
The services most relevant to reliability work: incidents management, monitoring and observability, infrastructure reliability, backup and disaster recovery (DRaaS), scaling and performance optimisation, and technical support — supported by DevOps as a Service, CI/CD, Kubernetes cluster and container management, cloud migration and cost optimisation, and managed AWS, Azure and GCP.
Day to day that means Terraform, Pulumi, AWS CDK and Azure Bicep for infrastructure as code; Jenkins, GitLab CI, GitHub Actions and Azure DevOps for pipelines; Kubernetes, OpenShift, Rancher and Nomad for orchestration; Prometheus, Grafana, ELK, Fluentd, Graylog and New Relic for observability; and ISO 27001, NIST and CIS practices with HashiCorp Vault, NeuVector and SonarQube on the security side.
Case study: infrastructure for a national healthcare platform
A healthcare client needed CI/CD and infrastructure for development and production environments serving a common base of medical records, prescriptions and insurance data across the country's medical centres. We built it on hardware from a local provider, GiGa Cloud, using VMware ESXi and Terraform, connected it to the government E-Health platform and applied data masking in dev and test to satisfy GDPR and HIPAA requirements. Infrastructure creation was automated, environments became scalable on demand, and delivery moved to a self-managed pipeline on Jenkins, Docker and Kubernetes.
Case study: finding the real load path in Azure
For another client, most of the Azure bill came from a load balancer and network traffic. We configured the file share with a private endpoint in the same VNet as the AKS nodes so traffic stayed internal rather than crossing the load balancer. Network costs dropped by 90% — up to $400 a day — and we handed over recommendations for performance, security and reliability alongside the savings. The lesson generalises: find the actual load path before you reinforce anything.
Conclusion
Rescue engineering is a useful model for DevOps because it is honest about working in a broken system. It assumes information is incomplete, time is short and the structure may move again — and it responds with process: triage quickly, stabilise before you enter, monitor continuously, keep more than one way out.
Teams that adopt those habits do not have fewer hard nights because their systems never fail. They have fewer because failure stops being improvisation.
Fedir Kompaniiets
Co-founder & CEO, Gart Solutions · Cloud Architect & DevOps Consultant
Fedir is a technology enthusiast with over a decade of diverse industry experience. He co-founded Gart Solutions to address complex tech challenges related to Digital Transformation, helping businesses focus on what matters most — scaling. Fedir is committed to driving sustainable IT transformation, helping SMBs innovate, plan future growth, and navigate the "tech madness" through expert DevOps and Cloud managed services. Connect on LinkedIn.
Not sure how your infrastructure would hold up in a collapse?
Gart Solutions helps engineering teams stabilise, monitor and recover cloud infrastructure — from incident management and observability to tested disaster recovery. Book a free consultation.
Talk to our architects →
Gartner predicts that by 2028, 40% of new enterprise production software will be built using vibe coding techniques and tools — prompting an AI assistant in natural language rather than hand-writing every line. It's already happening faster than that forecast suggests: by most 2026 estimates, 41-46% of new production code is AI-generated, and Java backends have crossed 61%. The problem isn't the prompting. It's that a working demo and a production-ready application that has passed a real security audit are two very different things, and most teams don't find out which one they've built until it's live and something breaks.
This guide is the vibe coding best practices playbook we actually use when a founder or product team brings us an AI-generated app and asks, "is this safe to launch?" It covers the prompt strategy that gets you closer to production-ready code on the first pass, the security gaps AI assistants reliably leave behind, and the infrastructure checklist — CI/CD, secrets, observability, disaster recovery — that turns a vibe-coded prototype into something Gart's own SRE and DevOps teams would sign off on.
What "vibe coding" actually means in 2026
The term was coined for describing a piece of code by describing what you want in plain English and letting an AI assistant — Claude, Cursor, Lovable, Bolt, Replit, v0, or a dozen similar tools — generate, run, and iterate on it, often with the person driving barely reading the diff. It's no longer a hobbyist curiosity. Stack Overflow's 2025 survey found 84% of developers already use or plan to use AI coding tools, and 63% of self-identified vibe coding users are non-developers: product managers, founders, and designers shipping real, customer-facing software without a traditional engineering background.
That's the upside case. The 2026 data on outcomes is messier. MIT researchers measured a 26% increase in completed tasks across nearly 4,900 developers using AI assistants, and McKinsey found teams saving roughly 3.6 hours a week on routine coding. But a randomized METR study found experienced developers were actually 19% slower on real tasks when using AI tools — while estimating afterward that they'd been 20% faster. Uplevel's research tied Copilot adoption to a 41% increase in bug rates. And separate security research found only 8.25% of one leading model's code outputs were both functionally correct and free of security flaws, with 45% failing OWASP Top 10 benchmarks outright. Vibe coding isn't a shortcut around engineering discipline — it just moves where that discipline needs to be applied: from writing the code to reviewing, securing, and operating it.
Prototype vs. production-ready: the gap in one table
Most of the vibe-coded apps we're asked to review pass this test in under a minute — and that's the point. A weekend prototype and a production system can look identical in the browser while being nothing alike underneath.
DimensionTypical vibe-coded prototypeProduction-ready applicationData access controlDefault-open tables; RLS/authorization added "later"Deny-by-default policies, tested per role before launchSecretsAPI keys pasted into prompts, client code, or .env files committed to gitManaged secrets store with rotation and least-privilege scopingTestingManual click-through by the person who built itAutomated test suite plus an independent review of AI-written logicDeploymentOne environment, deployed by hand from a laptopCI/CD pipeline with staging, rollback, and infrastructure as codeObservabilityNo alerting; issues found when a user complainsMonitoring, error tracking, and on-call escalation pathsDisaster recoveryNo backup strategy beyond the platform's defaultsTested backups, defined RTO/RPO, documented recovery runbookCost controlUnmetered AI-generated queries and autoscaling left uncappedBudget alerts, query review, and right-sized infrastructurePrototype vs. production-ready: the gap in one table
A prompt strategy that produces production-ready code
Most "vibe coding went wrong" stories trace back to a prompt that only described the happy path. AI coding assistants are pattern-matchers trained mostly on demo-quality code; if you don't ask for edge cases, error handling, and security constraints explicitly, you'll rarely get them by default. The prompt strategy that reliably narrows the gap in the table above has three layers, asked in order, not all at once:
Technical context first. State your stack, data model, and architectural constraints before asking for behavior — "PostgreSQL via Supabase, Next.js on Vercel, multi-tenant with row-level isolation by organization_id" — so the assistant isn't guessing at conventions it will contradict three prompts later.
Functional requirements, including the boring parts. Describe the user-facing behavior and explicitly ask for validation, empty states, and error messages, not just the success case.
Integration and edge cases as a direct follow-up. After the first draft, ask: "What could go wrong with this code in production? What edge cases and failure modes am I not handling?" Then ask the model to review its own output "as if this is going live tomorrow" — this single follow-up surfaces missing authorization checks and unhandled errors far more often than a single well-crafted initial prompt does.
Two habits compound this into an actual production-ready-app strategy rather than a one-off trick: ask the assistant to explain why it chose an approach (a model that can't justify a decision usually made a weak one), and treat every AI-generated data access, authentication, or payment code path as a draft that needs a second, human review before merge — never an exception to your normal review process.
Vibe coding security best practices you can't skip
Security is where AI-generated code fails most predictably, and where the consequences are least forgiving. The clearest public example is CVE-2025-48757: a missing Row-Level Security default in Lovable-generated apps that left over 170 live projects — roughly 303 exposed endpoints, CVSS 9.3 — readable and writable by anyone, unauthenticated. It's a textbook case of what breaks when a Lovable + Supabase app reaches production without a security review: the framework defaulted open, and nobody closed it.
Secrets management is the second most common failure mode, and it's getting worse, not better. GitGuardian's 2026 State of Secrets Sprawl report found that AI-assisted commits leak hardcoded secrets at 3.2%, versus a 1.5% baseline across all public GitHub commits — more than double — and secrets tied to AI services specifically grew 81% year over year. Four checks close most of the gap:
Before you ship, verify: row-level security (or equivalent authorization) is enabled and tested for every table and role, not just the default; no API keys or service-role credentials exist in client-side code, prompts, or committed .env files; secrets live in a managed store with rotation, not hardcoded — see our comparison of Kubernetes secrets management approaches if you're deploying on containers; and every AI-generated database and API layer has been checked against production hardening best practices for your specific backend, not just the framework's happy-path defaults.
None of this means AI-generated code is uniquely unsafe — it means it inherits the same risks as any code written under time pressure by someone optimizing for "it works," and vibe coding compresses that pressure into minutes instead of sprints. Building checks like role-based access control directly into the CI/CD pipeline, rather than relying on someone remembering to run them, is what closes the gap for good.
Testing and review discipline for AI-generated code
The trust gap tells you most of what you need to know here: only around 29% of developers say they trust AI-generated code's accuracy, down from roughly 40% two years ago — yet only 48% say they always review AI output before committing it. That mismatch, not the AI itself, is where production incidents come from.
A workable review discipline for vibe-coded code doesn't need to be heavier than normal code review — it needs to target the specific failure modes AI assistants produce: authorization checks that look present but only cover the happy path, error handling that catches the exception but swallows it silently, and logic that's subtly wrong in a way that passes a casual read (research on one frontier model found major-issue rates 1.7x higher than human-written baselines, with logic flaws up 75%). Treat any AI-generated pull request touching auth, payments, or data access as requiring the same second reviewer you'd assign to a junior engineer's first month of commits — because functionally, that's what it is.
The infrastructure checklist before you ship
This is the part that gets skipped most often, because it's invisible right up until it isn't. An app that runs fine on the platform's free tier with ten test users tells you almost nothing about how it behaves under real load, real failure, or a real audit.
CI/CD and infrastructure as code
If deploying means someone pushing a button from their laptop, you don't have a deployment process — you have a single point of failure with a person attached. A proper pipeline with staging, automated tests, and rollback is the single highest-leverage fix available, and it's exactly what our infrastructure-as-code case study walks through for a team that scaled from manual deploys to millions of automated transactions a month.
Observability and reliability
Vibe-coded apps tend to have zero visibility into their own health until a user reports something broken. Basic error tracking, uptime monitoring, and an alerting path aren't optional extras — they're the difference between finding a problem in minutes and finding it in a support ticket three days later. Our breakdown of SRE versus DevOps covers which discipline actually owns this once you're past the prototype stage.
A platform, not a pile of scripts
Teams that vibe-code several apps in parallel — which is increasingly common among the 16 million or so citizen developers now shipping software — run into a second-order problem: every app has its own ad hoc deployment, secrets handling, and monitoring setup. Platform engineering exists to turn that sprawl into a self-service golden path, so the next AI-generated app inherits guardrails instead of starting from zero.
Scale and cost control
AI-generated queries are notorious for missing indexes and doing more database round-trips than a human would write by hand — fine at ten users, expensive and slow at ten thousand. Cap autoscaling, set budget alerts, and load-test before a launch gets real traffic, not after.
When to bring in infrastructure and DevOps help
Not every vibe-coded app needs an outside team — a genuine side project with no user data at stake can stay a weekend project. The signal to act is any combination of: real user data flowing through the app, revenue depending on uptime, a compliance requirement (HIPAA, PCI DSS, SOC 2, GDPR) on the horizon, or a founder realizing they can describe what the app does but not how it fails. At that point, the fastest path isn't rebuilding from scratch — a fractional CTO engagement can sequence exactly which of the fixes in this article matter first for your specific app, before committing to a full rebuild that may not be necessary at all.
Turn your vibe-coded MVP into infrastructure that scales
From a one-time production-readiness audit to full-time DevOps and SRE support, Gart closes the gap between "it works in the demo" and "it survives real traffic" — without a full rebuild.
Security audit
Infrastructure audit
Platform engineering
Cloud migration
CTO as a Service
Get a production-readiness review
You might also like
DevSecOps vs. DevOps: how secure software delivery evolved
What is DevSecOps consulting?
Software reliability through DevOps and SRE
How to hire DevOps engineers that actually move the needle
Roman Burdiuzha
Co-founder & CTO, Gart Solutions · Cloud Architecture Expert
Roman has 15+ years of experience in DevOps and cloud architecture, with prior leadership roles at SoftServe and lifecell Ukraine. He co-founded Gart Solutions, where he leads cloud transformation and infrastructure modernization engagements across Europe and North America. In one recent client engagement, Gart reduced infrastructure waste by 38% through consolidating idle resources and introducing usage-aware automation. Read more on Startup Weekly.
