The X-Files of IT: Real-life Unauthorized Change Sightings
About
This content is brought to you by Evolven. Evolven Change Analytics is a unique AIOps solution that tracks and analyzes all actual changes carried out in the enterprise cloud environment. Evolven helps leading enterprises cut the number of incidents, slash troubleshoot time, and eliminate unauthorized changes. Learn more
Welcome to the third blog in our series on unauthorized changes and misconfigurations. These unapproved and often unseen changes in the IT infrastructure pop up unexpectedly, like UFO sightings, causing bewildering disruptions.
Gather ‘round, and get ready for a rollercoaster ride! In the spirit of Halloween, we are breaking into our IT X-Files to recount some bone-chilling, real-life tales of Unauthorized Change encounters that sent shivers down IT teams' spines. We are altering the names of the companies to help protect the innocent.
Case Study 1: The Credit Card Fiasco
Our first tale takes us to the world of money and credit cards. A major financial institution called "CapitalCo" experienced an anomaly, or for the sake of this story a UCO (or Unauthorized Change Object) sighting.
One fateful Friday in 2019, their primary database went rogue, delivering faulty data and making decisions worthy of an ill-tempered Klingon from Star Trek. Upon investigation, they discovered an unauthorized change was made to their database configuration - a classic example of an unauthorized change encounter!
As the error snowballed, it affected millions of credit card accounts, sending the company’s customers into a frenzy - and their reputation into a freefall. It was a dramatic saga straight out of our best IT X-Files. Ultimately, CapitalCo managed to restore the changes, but the impact was expansive.
The moral of the story: Unauthorized changes - They do exist. And even the biggest and most sophisticated companies can be vulnerable to them. If your organization is ill-prepared to detect, investigate, and rectify these - and quickly - the impact can be destructive.
Case Study 2: The E-commerce Extraterrestrial Encounter
Next, we journey back to 2017 to the bustling world of e-commerce, visiting a global giant we’ll refer to as "Nile Web Services (NWS)". Like a UFO sighting on a calm night, in this X-File tale, an unsanctioned modification was made to the company’s customer-facing application. It’s just a small code update, nothing bad could happen, right?
Suddenly, the company’s previously user-friendly checkout process became more complicated than solving a Rubik's cube blindfolded. Sales plummeted and customers were lost into a black hole, never to be seen again.
NWS, however, proved to be a quick learner in recovering from unauthorized change encounters. They tracked down the unauthorized change (this time a fat finger mistake by a third-party vendor), fixed it - only taking 4 hours - and reinforced their monitoring protocols. However, not before their profit graph took a hit worthy of the Independence Day mothership. - Ouch!
The moral of this story: Failures and mistakes happen, and unauthorized changes can occur at any time, for any reason. However, those that survive continue to reinforce their armor. Like NWS - they learn from each attack, strategize, plan, and test - all to improve and reinforce their systems for better security, compliance, and overall resilience.
Case Study 3: The Social Media Supernova
Now to the realm of social media, to a platform loved by billions. This tale leads us to the titan "INFRA". In 2021, a change to backbone routers interrupted the flow of traffic in the company’s data centers around the world. This seemingly innocuous configuration change prevented users of the company’s three platforms from refreshing social feeds or sending messages - not to mention missed advertising revenue.
The outrage and frustration spread fast, with users demanding answers. It took six hours for the INFRA team to identify the misconfiguration, correct the issue, and get all platforms up and running again - however, not before losing a star's worth of user and market trust. Despite CEO apologies for the outage, INFRA took a 5% loss in market share that day. And, despite the encounter, INFRA has continued to experience unauthorized change issues, misconfigurations, and small outages throughout 2022 and 2023.
The moral of this story: Customer expectations are high. Your ability to prevent, detect, and quickly remediate issues that impact customer experience is paramount. Customer loyalty is not a given in today’s world of choice. Always knowing the configuration state of your end-to-end hybrid enterprise is like having a secret weapon to help you ensure availability, security, and compliance.
Case Study 4: The Data Cruncher’s Cosmic Episode
Our last tale takes us to where digital realms are governed by powerful software. In 2019 a shining star emerged called "DataPug”. However, the cosmos is unpredictable.
One Wednesday afternoon in early 2023, an unauthorized change phenomenon of mass complexity hit. The DataPug’s core system, the web application, went into an unforeseen eclipse. Users could not access the platform or various services via the browser and monitors were unavailable and not alerting.
As you can imagine, alarm bells rang across the digital space as the Pug’s engineers scrambled to investigate. Unfortunately, the blackout wasn’t just a glitch: and at the time it sent ripples across the financial universe, with shares taking a downward trajectory and causing worry about the potential revenue impact.
The engineers, akin to cosmic warriors, identified and mitigated the initial issue. But then within a few hours and got a few systems back up after approximately four. After a full investigation, it appears that a new system behavior introduced in Ubuntu 22.04 disconnected more than 60% of DataPug’s instances from their network, impacting multiple regions across distinct cloud providers, and also affecting the regions hosting CI/CD and automation tools.
The moral of this story: A stern reminder for all IT - software is vast and unpredictable. The more complex the system, the more interdependencies you will have - and the more potential for configuration risk. Advanced AI, risk-based technology, such as that provided by Evolven, helps better arm you for these events.
Evolven’s Configuration Risk Intelligence platform detects and prioritizes risks triggered by actual, granular changes in configuration, application, infrastructure, and data, to help prevent and rapidly resolve stability, compliance, and security issues. With Evolven, IT teams experience greater visibility into your environments resulting in greater productivity, fewer incidents, and faster MTTR.
These few tales are just a quick reminder that even the mightiest constellations can be vulnerable. Unauthorized changes and misconfigurations are invaders that can plague any IT organization. And unlike a scary alien movie, this won’t be over in just 2 hours. So until our next space odyssey, stay vigilant, and may the IT force be with you!
For more information about navigating and detecting unauthorized changes, we are gathering the troops together on Oct 31, 2023, Halloween, at 1:00pm. EST to reveal our best X-Files, on how the largest financial institutions use AI to battle unauthorized changes. Register today and join us.
