When it comes to service outages, I believe no one was surprised by the results, but it’s certainly interesting to note that the bar of expectations is continually rising, and the consequences of downtime are having an increasingly significant impact on earnings and user trust.<\/p>\n\n\n\n
The second bullet point, in my view, relates to using Cloud technology without fully understanding its fundamentals. As we mentioned, the Cloud is not magical; simply moving a workload to the Cloud doesn’t automatically increase uptime. <\/p>\n\n\n\n
This article aims to be an attempt to explain how the Cloud empowers us and what we need to do to achieve fault tolerance<\/strong>, improve uptime<\/strong>, and address some aspects of a structured business continuity<\/strong> plan.<\/p>\n\n\n\n If we were in school, we should start with the definition: <\/p>\n\n\n\n fault tolerance refers to the ability of a system or application to continue functioning reliably even when failures or malfunctions occur in one or more of its components. <\/em><\/p>\n\n\n\n Being able to be fault-tolerant is important for addressing many of the objectives defined in business continuity plans. A well-designed fault-tolerant infrastructure brings us benefits such as reduced downtime, improved reliability, risk reduction, and maintaining service quality.<\/p>\n\n\n\n It’s important to clarify the concept of fault tolerance and how it differs from concepts like HA (High Availability), redundancy, and DR (Disaster Recovery), often mistakenly used as interchangeable synonyms:<\/p>\n\n\n\n High Availability<\/strong> (HA) refers to a system’s ability to remain operational, without significant interruptions, for continuous periods of time. This goal can be achieved through continuous monitoring, redundancy, and other measures. A high-availability system is designed to avoid failures and ensure continuous service availability.<\/p>\n\n\n\n Redundancy<\/strong> is used to enhance a system’s reliability. Designing infrastructures that include redundant components means that if one component fails, another can take its place to prevent interruptions. Redundancy can be applied at both hardware and software levels.<\/p>\n\n\n\n Disaster Recovery<\/strong> (DR) is a plan aimed at restoring a system or application in the event of catastrophic events or large-scale failures. This usually involves data backup, business continuity planning, and specific procedures to restore the system at an alternate location or with alternative resources.<\/p>\n\n\n\n High availability, redundancy, and DR are all important components to achieve fault tolerance, but they alone may not be sufficient to guarantee it<\/strong>. Fault tolerance requires a thoughtful design that considers a wide range of scenarios and provides adequate measures to manage them without significant disruptions.<\/p>\n\n\n\n Fault tolerance is, therefore, a design and implementation strategy that significantly contributes to continued reliable operation, thus protecting a company’s reputation and maintaining customer trust.<\/p>\n\n\n\n Those approaching the Cloud should keep in mind that providers are not, do not want to be, and cannot be responsible for our applications in their entirety. Furthermore, depending on the service or building block we use for our infrastructure, the provider shifts the responsibility bar. Taking AWS as an example, as shown in the image, more IaaS services shift the responsibility towards the Customer, while managed services shift more of the responsibility towards the provider.<\/p>\n\n\n\n <\/p>\n\n\n https:\/\/aws.amazon.com\/blogs\/industries\/applying-the-aws-shared-responsibility-model-to-your-gxp-solution\/<\/a><\/p>\n\n\n\n <\/p>\n\n\n\n This means that despite the provider implementing every form of hardware and geographical redundancy, part of the responsibility still rests with the customer.<\/p>\n\n\n\n AWS highlights these two sides of the coin by distinguishing the provider, responsible for the resilience of<\/span><\/strong> the Cloud, and the customer, responsible for resilience in<\/span><\/strong> the cloud. <\/p>\n\n\n\n AWS is responsible for protecting the infrastructure on which all services offered in the AWS Cloud run. This infrastructure includes the hardware, software, networks, and facilities on which AWS Cloud services operate. The customer’s responsibility, instead, is determined by the AWS Cloud services they choose, based on the configurations they need to make and their management and design activities.<\/p>\n\n\n\n Let’s try to ground all the discussions on Fault Tolerance. <\/p>\n\n\n\n In the case of AWS, we need to start by understanding how the physical infrastructure of the data centers has been implemented. It is necessary to reiterate that AWS provides us with the tools to create resilient infrastructures, but it’s up to us to design and implement them correctly.<\/p>\n\n\n\n Let’s start with the Availability Zones (AZ), which are individual data centers or groups of them, logically grouped in Regions.<\/p>\n\n\n\n Redundancy<\/strong> is achieved by leveraging a single AZ but duplicating the instances of the service in use within it. Let’s provide two practical examples: I can create two EC2 instances within which I can install MySql, set them up in a cluster, active-standby, and be resilient to provider hardware failures.<\/p>\n\n\n\n High Availability<\/strong> is the ability to keep services running without interruptions. In a SaaS context, it means having redundant systems so that if one component fails, another immediately takes its place without causing downtime. HA is crucial to ensure that SaaS services are always accessible to users.<\/p>\n\n\n\n To achieve high availability, you can distribute your database across two different Availability Zones. It is essential to have designed, set up and configured the Cloud environment correctly. If you are not sure what I’m talking about, I must refer you to our series of articles on the Landing Zone<\/a>.<\/p>\n\n\n\n Disaster Recovery is the plan that a company implements to restore its operations after a major incident or catastrophe. In a SaaS context, this means having backup copies of critical data and services in a safe and easily accessible location. In case of an emergency, DR allows you to quickly restore services, minimizing downtime.<\/p>\n\n\n\n Disaster Recovery<\/strong> is achieved by utilizing two different AWS Regions worldwide. There are not many automatic mechanisms or services that allow us to achieve DR with low implementation effort. DR needs to be carefully thought out, and there are numerous aspects to keep in mind. Please check our series of articles on these aspects<\/a>.<\/p>\n\n\n\n So far, we have addressed the issue only from an infrastructure perspective, but from a software development standpoint, this topic also has an impact.<\/p>\n\n\n\n For example, it is a good practice to leverage the Cloud, which is a fully programmable environment, to create multiple development environments for testing changes before making them effective in production. The use of multiple environments for experimenting, validating, and testing workloads reduces the risk of human errors by creating an automated, controlled, and reversible process.<\/p>\n\n\n\n Even in the phases of releasing updates to our products or public portals, fault tolerance needs to be considered. For example, with Blue-Green Deployment<\/strong>, two identical environments (blue and green) are maintained in parallel. During an update, user traffic is redirected from the “blue” environment to the “green” one. This technique allows updates to be performed without downtime, and in case of issues, it is possible to quickly revert to the “blue” environment.<\/p>\n\n\n\n Blue-Green deployment can be complemented by practices like those defined in Canary Deployment<\/strong> or Rolling Deployment<\/strong>. With these techniques, the update is rolled out to a small subset of users only. These users serve as indicators to detect any issues, and if no problems are encountered, the update is gradually extended to a broader audience; otherwise, the update is halted by redirecting the few “tester” users to the old environment.<\/p>\n\n\n\n All of these are gradual and controlled deployment approaches that reduce the risk of production failures and ensure that changes are stable and reliable before being exposed to end-users.<\/p>\n\n\n\n (Check these articles for more on Blue-Green Deployment, Canary Deployment, and Rolling Deployment: How to setup a Continuous Deployment Pipeline on AWS for ECS Blue\/Green Deployments<\/a> – ECS deployment strategies: reduce downtime and risk with blue\/green deployment<\/a>).<\/em><\/p>\n\n\n\n At this point, the question that naturally arises is: how do I verify my fault tolerance?<\/p>\n\n\n\n Chaos Engineering<\/strong> is a practice designed to test the resilience and fault tolerance of a computer system in a controlled and planned manner. The main goal of chaos engineering is to identify weaknesses and vulnerabilities in the system before they can cause real production problems.<\/p>\n\n\n\n One of the examples of tools created to test the robustness of environments is Chaos Monkey developed by Netflix. You can check out the project here<\/a>. This tool has this peculiar name because it aims to simulate the unpredictable behaviors of a monkey left loose inside a data center. What could happen? Unplugged cables, pressed buttons, destroyed racks, and so on.<\/p>\n\n\n\n If the infrastructure holds up and there are no downtime incidents, then you can finally say that fault tolerance has been achieved.<\/p>\n\n\n\n Even redundancy, high availability, and disaster recovery are not enough to improve resilience. There are other principles that should be followed, such as horizontal scalability to accommodate the load curve, automatic recovery in case of failure (self-healing), automation, and verification of all recovery procedures.<\/p>\n\n\n\n I don’t want to sound clich\u00e9, but as Spiderman wisely reminds us: \u201cWith great power comes great responsibility\u201d. The Cloud gives us the power that we must learn to harness in the right way, through study and experience.<\/p>\n\n\n\n I hope this article helped in clarifying things around Fault Tolerance, Business Continuity and all the related concepts. <\/p>\n\n\n\n We will be back soon with more on this. Stay tuned!<\/p>\n\n\n\nWhat is Fault Tolerance?<\/h2>\n\n\n\n
Shared Responsibility Model<\/h2>\n\n\n\n
![\"\"](\"https:\/\/blog.besharp.it\/wp-content\/uploads\/2023\/11\/image1-1024x783.png\")
<\/figure><\/div>\n\n\nFault Tolerance on AWS<\/h2>\n\n\n\n
Fault Tolerance during deployments<\/h2>\n\n\n\n
Chaos Engineering<\/h2>\n\n\n\n
Conclusion<\/h2>\n\n\n\n
\n\n\n\nAbout Proud2beCloud<\/h4>\n\n\n\n