Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

> I had already called out how sub-optimal this entire setup was before the incident occurred but it rang hollow from then on since it sounded like me just trying to cover for my mistake

That's what blameless post mortems are supposed to prevent. The only valid things to consider are questions like "is it a cost-effective prevention?", "what is the timeline of implementing this prevents / how should we prioritize?", and "does it meaningfully overlap with other changes that would prevent this?". Things like "well person A is new and that's why this happened" is not an acceptable position. Now of course, people can be not honest / not self-aware enough and are coming from that position without stating it anyway. That being said, that's why you try to focus on the objective evaluation of the recommendation and nothing else - that tends to make the bias irrelevant. Any EM/tech lead worth their salt would instantly say "is it actually important that staging and dev databases share creds"? Certainly prod should never. The other follow up to track down would be "if DNS flushing is so important when changing hosts, how have we automated it so that it happens automatically before every script that's run". I would also recommend to get rid of (ab)using the hosts file immediately and switch to explicit configuration files that you have to specify (perhaps automatically picking up "<username>.config" as a default so that scripts only ever run on your dev zone by default & locking down production.config so that only CI has the ability to touch that).



I absolutely concur with what you wrote but unfortunately not every team, company, or industry shares such a mindset. My teammates meant no offense and my mistake became a running joke but we never even considered doing an RCA for the problem. We did end up fixing the hosts file thing by moving to a different server-side architecture to rectify more pressing DX issues but the shared credentials issue persisted. One or two devs would actually lock the account by trying bad creds and block everybody's access - it took me diving into some RDBMS innards to figure out the responsible party and block them by hostname until they were able to correct the issue.


You don’t have to share the mindset but it means you’ll be out competed if another team or competitor does. So you should do it first




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: