Nightmare Infrastructures part 5

Hello again, dear horror lovers. If you have been following this series, you already know the ritual: once a year we open the crypt, dust off the incidents we collected during the year, and tell you what went wrong without pointing a finger at anyone. Context and time explain a lot of bad decisions. They do not make the bill smaller, though.

This is episode 5. If you missed the earlier ones, start from episode 1, episode 2, episode 3, and episode 4.

Sit back, keep a break-glass account where you can actually reach it, we're about to start!

There Is No Way Out

In Cube, people wake up inside a maze of rooms, where every door looks like an exit (but it's not). Our story starts the same way: an AWS Organization that has been locked down, with no way to access it from any door.

The company had done the modern and hassle-free thing: IAM Identity Center was federated with Microsoft Entra ID, allowing people to log in with their corporate identity and landing in the right permission sets. Nobody touched the root user nor the management account, and there was also a break-glass user with a hardware MFA token, for the day theory became practice.

Then the federation certificate expired, marking the day to put theory in practice!

The real issue was that nobody was watching the mailbox where the expiry notices landed, nobody had an alarm on the certificate lifetime, so one morning, federation simply stopped working. No one could assume a role through Identity Center. The "normal" door was gone.

So they reached for the management account root user. It turned out that that door had a second factor tied to a VoIP PBX, a phone system that had been dismissed earlier that year, so the codes that were supposed to arrive somewhere didn't arrive.

That left the break-glass user option: the hardware token existed, and credentials were stored in the company password manager. The only issue was that the person who held the token was on vacation, with the token at home, and a 10-hour expensive flight to get back home, with no one else able to get the token in their hands.

The result: they were locked out for about a week, business and services continued to work as normal, and luckily there were no incidents or downtime during this period of time.

Please, don't think "federation is bad": federation is fine, but you have to keep in mind that recovery path is part of the design. An email is not a monitoring system for expiring certificates, and you should not depend on a single person, or external systems for MFA. And, please, test the recovery path on purpose, the way you should test backups, because a successful restore is the only proof that everything will be fine when you need it.

In the film, Worth was right about one thing. If you did not design a way out, there is no way out of here.

It Won't Stop

You remember the story: the tape has to keep circulating; if you watch it, the only way to survive is to make a copy and give it to someone else.

We once looked at an integration between a software company and a third-party system; on paper it was a classic pattern: systems exchanged messages on a bus, and webhooks closed the loop when the elaboration was done. Everyone was happy that "async" was in place.

It turned out that almost all the work the third-party side did was this: receive a message, send the same message again to the webhook endpoint, write the same payload back on the same bus. No useful transformation and no durable side effect that meant "done". It was just another copy.

Like the tape, the act of receiving the message was the act of publishing it again. The integration looked busy. It was only keeping itself alive.

We discovered the loop during a debugging session, because sometimes there were dropped messages due to loops.

Watch out for loops: you can use key to track messages, put a TTL, or apply any other techniques can help in identifying duplicated messages and prevent loops. It can sound obvious, but keep a dead-letter queue you monitor (you monitor your DLQs, right?).

In The Ring, the curse goes on every time a person copies the VHS, the cloud version is quieter, but it means having scary bills!

A Hundred Tables

This one needs no opening quote. The bill is scary enough on its own! But you’ll see how an innocent operation can turn into a gremlin!

A product company wanted to modernize their legacy CRUD app: they wanted to go serverless to minimize work for operations, and they looked at DynamoDB with confidence and hope, because it is managed, scalable and with a pay as you go model, no fixed price for a database. The development team, knowing their product, kept the old relational model almost as-is, maybe biased by years of domain knowledge. This made a hundred tables appear: one per entity, plus leftovers from old joins. They maintained the same access pattern they knew, making it look like SQL, meaning a lot of Scan operations on the hot path, sometimes filtering data only after it was already read.

Everything was fine in the staging environment, but we know that production data grows, so issues and costs get amplified. Since every request read more data than it needed, at the end of the month DynamoDB had a billing impact of about $13,000.

We always tend to use new technology we love for everything, and the same happened to that team with DynamoDB: they started to use it for everything, without noticing that they used a key-value store as a relational engine, with all the cons this approach implies.

Since it's Halloween, we can find a similar story in Gremlins: there were rules for Mogwai, like keep him out of the light, do not get him wet, and never feed him after midnight. DynamoDB has different rules, but if you break them you'll find a gremlin in your bill.

DynamoDB is the best choice if you design tables for the queries you need, without adapting it to the ER diagram you already have. If you need to execute relational queries, it's better to use RDS or Aurora (and remember that you also have Aurora Serverless as an option). If you rely on scans to get data, your cute serverless pet will grow into a bill with teeth.

Put On the Glasses

John Carpenter's They Live is about putting on glasses and seeing that the official labels were a lie. Our version is about data.

A sales team needed industry classification codes for the companies in their CRM. The codes decided which leads got which pitch, so they bought firmographic data from a well-known business-data vendor. On paper that is a reasonable move: the vendor's whole product is "we know who these companies are". The feed became the source of truth, and nobody argued with a paid label.

Spot checks kept failing, like when a software company showed up as an industrial kitchen supplier or another looked fine until someone read the About page; the list was long, and most rows still were there as the vendor had delivered them.

To solve these issues, we thought about using AI, building a RAG system to check the official taxonomy and each company's public footprint (name, site, short description), to propose a code from the company's data, that was compared against the vendor's.

We ended up in finding out that for more than 80% of the companies in the list, the vendor code was wrong, and the one grounded in the company's public footprint was better. The supplier that should have known the truth had been wrong for a long time. The scary part is not that AI fixed it, but how long the wrong answers had been used as if they were facts.

What have we learned? Data quality matters, but a way to verify it matters more. In our case, the vendor feed looked like ground truth until we checked it against the reality, finding out that more than 80% of the codes were wrong.

Conclusion

"There are only 365 days left until next Halloween!"

The Mayor, The Nightmare Before Christmas

We still have stories for next year, and in these years we discovered a pattern: the issue is rarely a service, but the way it's used, monitored and operated.

If you have your own infrastructure nightmare (anonymized, please), tell us in the comments or reach out on LinkedIn. We are already taking notes for episode 6.

Stay safe and remember to not feed the Scan after midnight. See you next year!

About Proud2beCloud

Proud2beCloud è il blog di beSharp, APN Premier Consulting Partner italiano esperto nella progettazione, implementazione e gestione di infrastrutture Cloud complesse e servizi AWS avanzati. Prima di essere scrittori, siamo Solutions Architect che, dal 2007, lavorano quotidianamente con i servizi AWS. Siamo innovatori alla costante ricerca della soluzione più all'avanguardia per noi e per i nostri clienti. Su Proud2beCloud condividiamo regolarmente i nostri migliori spunti con chi come noi, per lavoro o per passione, lavora con il Cloud di AWS. Partecipa alla discussione!