Failures are part of the game

What you think will happen vs what really happens

As a developer who joined the team recently, or someone who's still not that experienced, or just a guy with a pessimistic approach to work, you may often feel differently about what you think will happen than what really happens when you make a proper mistake.

How to prevent

Although errors are normal, they are always a problem. They stress people, they break the current work, they add to existing work (apology notes, status pages, etc.). Let's avoid them, really 🙂

“Prevention is mostly about stop assuming you're careful. The computer or agent doesn't care about your intentions, even if it tries.”

Use the most common, boring guards for error prevention. I won't cover even a fraction of troubleshooting ways here, but I'll share some simple things that can save you once or twice.

Preventing Mistakes

Things that are not code changes - wrong clicks, accidental server commands, too-fast deployments. Silly things usually. And not much smarter things can help to prevent these:

• Label your env with a badge. Use red for production 🙂 You will see it every time you work with staging / localhost / production
• Change SSH text color on live instances - so you do not run pnpm install there
• Take backups before risky changes - manual if needed (restoring them deserves a separate article)
• Use read-only access tokens with AI agents - they will tell you they will not mess up GitHub, then they will mess up terribly
• Demo on staging, not production - I deleted something on production once to show how it works (serious! lost a few hours)
• Be slow in risky places - database operations, SSH, Vercel. Rush is a close friend of mistakes; deadlines still exist

Preventing Bugs

Bugs are shipped in code, inevitable, and they love surfacing when you are off. Often less spectacular than one-off mistakes. More our day-to-day friends at work. Preventing bugs is generally boring, but bring satisfaction if you find out that your guard helped you 2nd time this year. Some hints below:

• Add tests - boring but works. Make sure AI-written tests make sense - I've seen enough
• Hide risky features behind a feature flag - if it fails, it might be just for you
• Add Bugbot or another AI code reviewer - annoying, but they check the right things usually
• Push small changes whenever possible (yeah, I know) - easier to review, especially by a human
• Add localhost guards on risky scripts - e.g. "Never run it on staging or production. Check if you are on localhost"
• Keep a book of fuckups - log bugs to a file, then ask AI agents to check against it at review
• Log early on risky changes - do not wait for issues. Add logs / Sentry with your regular deployment
• Stay online after risky deploys - rollback or fix fast if something obvious breaks

How to handle (while it's broken)

No technical recipe here, as every failure is different - some require just a quick rollback, some need customer support, some a long investigation, and the best ones - all of them. Less about commands - more about what to say and who to tell:

• Be transparent - tell what you did - even if it was stupid. Like a doctor visit: otherwise nobody knows what to do or how serious it is
• Flag the situation with the team
  • Someone else may have seen this before - and can solve it fast
  • Maybe only 2 users are affected and you don't need to care that much
  • Maybe only your judgment that it is critical - Maybe it can wait until tomorrow or after the weekend (I've panicked over that a few times)
• Try a quick workaround before the proper fix - that may bring peace quickly
• Stay cold under pressure - you can still make things worse (definitely!)
• Fix it well - people may remember the fix instead of the bug
• Offer help if it is not your bug (hahah, silly other dev) - some will need it, most won't, bonus points either way 🙂

What to do after the fix (yes, you should do something)

Ok, you made an error, you fixed it, Tuesday ends and the backlog awaits tackling the next day. However, it's really, really good to do some notes right after things are back to normal- otherwise, you can go to the "When it becomes a problem" section.

• We called it "post-mortem" once - not sure if it's still called that. It's simply a note describing what went wrong and what was done to bring it back, shared with team / client, whoever needs it
• To prevent it in the future (if you have such a stupid idea one more time) - add it to a checklist, which you or an AI agent can check before some action (deployment, release, publishing something). You can store it in the repo as an .md file
• Say thanks to all people that helped / offered help / or just left you alone without pointing fingers - including the manager who thought "oh come on, not him again," but didn't say it out loud
• Create a ticket, if you applied a workaround and the thing needs proper solving

When it becomes a real problem

Failures are very lightly treated in this article, like fun and daily work - however, there are situations when things become serious. Think of these:

• Your database was not deleted, but leaked - this may have a trust-related, law-ish, very critical impact (I haven't witnessed any and hopefully won't)
• You repeat the same mistake multiple times - that's a sign you are not learning from them (I know a guy who lost his job because he did not improve after several notices)
• You push bugs too often - support@ email becomes crowded, people need to switch context and fix your code
• You made a mistake (let's say, pushed an API key to the repo) - but didn't tell anyone and just removed it after an hour. Everyone is happy, but at some point it may become a problem and it's on you anyway. Revoking is not that costly a thing.

To wrap up - you know "Worse is better" phrase by Richard Gabriel. It's not about bugs 🙂, and this term was coined 40 years ago, so not so up to date, however you can always say it to yourself when you discover you shipped a nice bug or made a phenomenal mistake for next 3 generations.