The complexities of going live with a Machine Learning workflow in the Cloud are called a deployment gap<\/a> and we will see together through this article how to tackle it by combining speed and agility in modeling and training with criteria of solidity, scalability, and resilience required by production environments.<\/p>\n\n\n\n The procedure we\u2019ll dive into is similar to what happened with the DevOps model for “traditional” software development, and the MLOps paradigm, this is how we call it, is commonly proposed as “an end-to-end process to design, create and manage Machine Learning applications in a reproducible, testable and evolutionary way<\/em><\/a>“.<\/p>\n\n\n\n So as we will guide you through the following paragraphs, we will dive deep into the reasons and principles behind the MLOps paradigm and how it easily relates to the AWS ecosystem and the best practices of the AWS Well-Architected Framework.<\/p>\n\n\n\n Let\u2019s start!<\/p>\n\n\n\n As said before, Machine Learning workloads can be essentially seen as complex pieces of software, so we can still apply “traditional” software practices. Nonetheless, due to its experimental nature, Machine Learning brings to the table some essential differences<\/strong>, which require a lifecycle management paradigm tailored to their needs. <\/p>\n\n\n\n These differences occur at all the various steps of a workload and contribute significantly to the deployment gap we talked about, so a description is obliged:<\/p>\n\n\n\n Managing code in Machine Learning appliances is a complex matter. Let\u2019s see why!<\/p>\n\n\n\n Collaboration on model experiments among data scientists<\/strong> is not as easy as sharing traditional code files: Jupyter Notebooks allow for writing and executing code, resulting in more difficult git chores to keep code synchronized between users, with frequent merge conflicts<\/strong>.<\/p>\n\n\n\n Developers must code on different sub-projects: ETL jobs<\/strong>, model logic<\/strong>, training and validation<\/strong>, inference logic<\/strong>, and Infrastructure-as-Code templates<\/strong>. All of these separate projects must be centrally managed and adequately versioned!<\/p>\n\n\n\n For modern software applications, there are many consolidated Version Control procedures<\/strong> like conventional commit<\/a>, feature branching, squash and rebase<\/a>, and continuous integration<\/a>. <\/p>\n\n\n\n These techniques however, are not always applicable to Jupyter Notebooks<\/strong> since, as stated before, they are not simple text files.<\/p>\n\n\n\n Data scientists need to try many combinations of datasets, features, modeling techniques, algorithms, and parameter configurations to find the solution which best extracts business value<\/strong>. <\/p>\n\n\n\n The key point is finding ways to track both succeeded<\/strong> and failed<\/strong> experiments while maintaining reproducibility<\/strong> and code reusability<\/strong>. Pursuing this goal means having instruments to allow for quick rollbacks and efficient monitoring of results, better if with visual tools.<\/p>\n\n\n\n Testing a Machine Learning workload is more complex<\/strong> than testing traditional software. <\/p>\n\n\n\n Dataset requires continuous validation<\/strong>. Models developed by data scientists require ongoing<\/strong> quality evaluation<\/strong>, training validation, and performance checks<\/strong>. <\/p>\n\n\n\n All these checks add to the typical unit and integration testing, defining the concept of Continuous Training, <\/strong>which is required to avoid model aging<\/strong> and concept drift<\/a>. <\/p>\n\n\n\n Unique to Machine Learning workflows, its purpose is to trigger retraining and serving the models automatically.<\/p>\n\n\n\n Deployment of Machine Learning models in the Cloud is a challenging task<\/strong>. It typically requires creating various multi-step pipelines<\/strong> which serve to retrain and deploy the models automatically. <\/p>\n\n\n\n This approach adds complexity to the solution and requires automating steps done manually by data scientists<\/strong> when training and validating new models in a project’s experimental phase. <\/p>\n\n\n\n It is crucial to create efficient retrain procedures!<\/p>\n\n\n\n Machine Learning models are prone to decay much faster than “traditional” software<\/strong>. They can have reduced performances due to suboptimal coding, incorrect hardware choices<\/strong> in training and inference phases, and evolving data sets.<\/p>\n\n\n\n A proper methodology must take this degradation into account; therefore, we need a tracking mechanism<\/strong> to summarize workload statistics<\/strong>, monitor performances<\/strong>, and send alarm notifications<\/strong>. <\/p>\n\n\n\n All of these procedures must be automated and are called Continuous Monitoring<\/strong>, which also has the added benefit of enabling Continuous Training by measuring meaningful thresholds.<\/p>\n\n\n\n We also want to apply rollbacks<\/strong> when a model inference deviates from selected scoring thresholds as quickly as possible to try new feature combinations.<\/p>\n\n\n\n Machine Learning shares similar approaches to standard CI\/CD pipelines of modern software applications: source control, unit testing, integration testing, continuous delivery of packages. <\/p>\n\n\n\n Nonetheless, models and data sets require particular interventions<\/strong>.<\/p>\n\n\n\n Continuous integration now also requires, as said before, testing and validating data, data schemas, and models.<\/p>\n\n\n\n In this context, continuous delivery must be designed as an ML training pipeline capable of automatically deploying the inference as a reachable service.<\/strong><\/p>\n\n\n\n As you can see, there is much on the table that makes structuring a Machine Learning project a very complex task. <\/p>\n\n\n\n Before introducing the reader to the MLOps methodology, which puts all these crucial aspects under its umbrella, we will see how a typical Machine Learning workflow is structured, keeping into account what we have said until now.<\/p>\n\n\n\n Let\u2019s go on!<\/p>\n\n\n\n <\/p>\n\n\n\n <\/p>\n\n\n\n A Machine Learning workflow is not meant to be linear, just like traditional software. It is mainly composed of three distinct layers: data<\/strong>, model<\/strong>, and code, <\/strong>and one will continuously give and retrieve feedback from others<\/strong>. <\/p>\n\n\n\n So while with traditional software, we can say that each step that composes a workflow can be atomic and somehow isolated, in Machine Learning, this is not entirely true as the layers are deeply intertwined<\/strong>. <\/p>\n\n\n\n A typical example is when changes to the data set require retraining or re-thinking a model. A different model also usually needs modifications to the code that runs it.<\/p>\n\n\n\n Let\u2019s see together what every Layer is composed of and how it works.<\/p>\n\n\n\n The Data layer comprises all the tasks needed to manipulate data and make it available for model design and training: data ingestion<\/strong>, data inspection, cleaning<\/strong>, and finally, data preprocessing<\/strong>.<\/p>\n\n\n\n Data for real-world problems can be in the numbers of GB or even TB, continuously increasing, so we need proper storage for handling massive data lakes. <\/p>\n\n\n\n The storage must be robust, allow efficient parallel processing, and integrate easily with tools for ETL<\/a> jobs.<\/p>\n\n\n\n This layer is the most crucial, representing 80% of the work done in a Machine Learning workflow<\/a>; two famous quotes state this fact: “garbage in, garbage out<\/em>” and “your model is only as good as your data.\u201d<\/em> <\/p>\n\n\n\n Most of these concepts are the prerogative of a Data Analytics practice<\/strong>, deeply entangled with Machine Learning, and we will analyze them in detail later on in this article.<\/p>\n\n\n\n The Model layer contains all the operations to design<\/strong>, experiment<\/strong>, train<\/strong>, and validate<\/strong> one or more Machine Learning models. ML practitioners conduct trials on data in this layer, try algorithms on different hardware<\/strong> solutions, and do Hyperparameters tuning<\/strong>.<\/p>\n\n\n\n This layer is typically subject to frequent changes due to updates on both Data and Code<\/strong>, necessary to avoid concept drift<\/strong>. To properly handle its lifecycle management at scale, we must define automatic procedures <\/strong>for retraining and validation.<\/p>\n\n\n\n The Model layer is also a stage where discussions occur, between data scientists and stakeholders, about model validation, conceptual soundness, and biases on expected results.<\/p>\n\n\n\n In the Code layer, we define a set of procedures to put a model in production<\/strong>, manage inferences requests<\/strong>, store a model’s metadata, analyze overall performances<\/strong>, monitor the<\/strong> workflow <\/strong>(debugging, logging, auditing), and orchestrate CI\/CD\/CT\/CM automatisms<\/strong>.<\/p>\n\n\n\n A good Code layer allows for a continuous feedback model<\/strong>, where the model evolves in time, taking into account the results of ongoing inferences.<\/p>\n\n\n\n All these three layers are managed by “sub-pipelines,” which add up to each other to form a “macro-pipeline” known as Machine Learning Pipeline<\/a>. <\/p>\n\n\n\n Automatically designing, building, and running this Pipeline while reducing the deployment gap in the process is the core of the MLOps paradigm. <\/p>\n\n\n\n MLOps aims to make developing and maintaining Machine Learning workflows seamless<\/strong> and efficient<\/strong>. The data science community generally agrees that it is not a single technical solution, yet a series of best practices and guiding principles around Machine Learning.<\/p>\n\n\n\n <\/p>\n\n\n\n <\/p>\n\n\n\n An MLOps approach involves operations, techniques, and tools, which we can group into four main pillars<\/strong>: Collaboration<\/strong>, Reproducibility<\/strong>, Continuity<\/strong>, and Monitoring<\/strong>. <\/p>\n\n\n\n We will now focus on each one, giving multiple practical examples that show how AWS, with many of its services, can be an invaluable tool<\/strong> to develop solutions that adhere to the paradigm\u2019s best practices.<\/p>\n\n\n\n A good Machine Learning workflow should be collaborative<\/strong>, and collaboration occurs on all the ML pipelines.<\/p>\n\n\n\n Starting from Data Layer, we need a shared infrastructure<\/strong>, which means a distributed data lake<\/strong>. AWS offers several different storage solutions for this purpose, like Amazon Redshift<\/strong>, which is best for Data Warehousing, or Amazon FSx for Lustre, <\/strong>perfect <\/strong>as a distributed file system. Still, the most common service used for data lake creation is Amazon S3<\/strong>. <\/p>\n\n\n\n To properly maintain a data lake, we need to regularly ingest data from different sources and manage shared access between collaborators, ensuring data is always up-to-date<\/strong>. <\/p>\n\n\n\n This is not an easy task, and for that, we can take advantage of S3 LakeFormation<\/strong>, a managed service that helps in creating and maintaining a data lake, by working as a wrapper around AWS Glue<\/strong> and Glue Studio<\/strong>, in particular simplifying Glue\u2019s Crawler set-up and maintenance.<\/p>\n\n\n\n S3 LakeFormation can also take care of data and collaborators’ permission rules by managing users and roles underneath AWS Glue Catalog<\/strong>. This feature is crucial as collaboration also means maintaining governance over the data lake<\/a>, avoiding unintended data manipulation by allowing or denying access to specific resources inside a catalog.<\/p>\n\n\n\n For the model layer, data scientists need a tool for collaborative design and coding of Machine Learning models<\/strong>. It must allow multiple users to work on the same experiment<\/strong>, quickly show the results of each collaborator, grant real-time pair programming<\/strong>, and avoid code regressions<\/strong> and merge conflicts<\/strong> as much as possible.<\/p>\n\n\n\n SageMaker<\/strong> is the all-in-one framework of choice for doing Machine Learning on AWS, and Amazon<\/strong> SageMaker Studio<\/strong> is a unique IDE explicitly developed for working with Jupyter Notebooks having collaboration in mind<\/a>.<\/p>\n\n\n\n SageMaker Studio allows sharing<\/strong> a<\/strong> dedicated EC2 instance<\/strong> between different registered users, in which it is possible to save all the experiments done while developing a Machine Learning model. This instance can host Jupyter Notebooks directly or receive results, attachments, and graphics via API from other Notebook instances. <\/p>\n\n\n\n SageMaker Studio is also directly integrated with SageMaker Experiments<\/strong> <\/sup>and SageMaker Feature Store<\/strong>.<\/p>\n\n\n\n The first one is a set of API that allows data scientists to record and archive a model trial, from tuning<\/strong> to validation<\/strong>, and report the results in the IDE console. The latter is a purpose-built managed store for sharing up-to-date parameters<\/strong> for different model trials.<\/p>\n\n\n\n SageMaker Feature Store represents a considerable step forward in maintaining governance over data parameters across different teams, mainly because it avoids a typical misused pattern of having different sets of parameters <\/strong>for training and inference. It is also a perfect solution to ensure that every data scientist working on a project has complete labeling visibility<\/strong>.<\/p>\n\n\n\n To be robust, fault-tolerant, and scale properly, just like “traditional” software applications, a Machine Learning workflow must be reproducible<\/strong>.<\/p>\n\n\n\n One crucial point we must address with care, as we said before, is Version Control<\/strong>: we must ensure code, data, model metadata, and features are appropriately versioned. <\/p>\n\n\n\n For Jupyter Notebooks, Git or AWS CodeCommit<\/strong> are natural choices, but managing the information of different trials, especially model metadata, requires some considerations.<\/p>\n\n\n\n We can use SageMaker Feature Store for metadata and features. It allows us to store data directly online in a managed store or integrate with AWS Glue <\/strong>(and S3 LakeFomation). It also enables data encryption using AWS KMS<\/strong> and can be controlled via API or inside SageMaker Studio.<\/p>\n\n\n\n When you want a workflow to be reproducible, you also mean experimenting on a larger scale<\/strong>, even in parallel, in a quick, predictable, and automatic way.<\/p>\n\n\n\n SageMaker offers different ways to mix and match different Machine Learning algorithms, and AWS allows for three possible approaches for executing a model.<\/p>\n\n\n\n Managed Algorithm<\/strong>: SageMaker offers up to 13 managed algorithms for common ML scenarios, and for each one, detailed documentation describes software and hardware specifications.<\/p>\n\n\n\n Bring your own algorithm<\/strong><\/a>: <\/strong>data scientists can quickly introduce custom logic on notebooks, as long as the model respects SageMaker fit() requirements.<\/strong><\/p>\n\n\n\n Bring your own Container<\/strong><\/a>: <\/strong>particular models such as <\/strong>DBScan<\/a> require custom Kernels for running the algorithm, so SageMaker allows registering a custom container with the special Kernel and the code for running the model.<\/p>\n\n\n\n Data Scientists can tackle all these approaches together. <\/p>\n\n\n\n SageMaker gives the possibility to define the hardware on which running a model training or validation by selecting the Instance Type<\/strong> and the Instance Size<\/strong> in the model properties, which is extremely important as different algorithms require CPU or GPU optimized machines. <\/p>\n\n\n\n To fine-tune a model, SageMaker can run different Hyperparameter Tuning Strategies<\/strong>: Random Search<\/strong> and Bayesian Search<\/strong>. These two strategies are entirely automatic, granting a way to test a more significant number of trial combinations in a fraction of time.<\/p>\n\n\n\n To enhancing the repeatability of experiments, we also need to manage different ways of doing data preprocessing (different data sets applied to the same model). For this, we have AWS Data Wrangler<\/strong>, which contains over 300 built-in data transformations<\/strong> to quickly normalize, transform, and combine features without having to write any code.<\/p>\n\n\n\n AWS Data Wrangler can be a good choice when the ML problem you\u2019re addressing is somehow standardized, but for most cases, the datasets are extremely diverse, which means tackling ETL jobs on your own. <\/p>\n\n\n\n For custom ETL jobs, AWS Glue<\/strong> is still the way to go, as it also allows saving Job Crawlers and Glue Catalogs (for repeatability). Along with AWS Glue and AWS Glue Studio, we have also tried AWS Glue Elastic Views<\/strong>, a new service to help to manage different data sources together<\/a>.<\/p>\n\n\n\n To make our Machine Learning workflow continuous<\/strong>, we must use pipeline automation<\/strong> as much as possible to manage its entire lifecycle.<\/p>\n\n\n\n <\/p>\n\n\n\n <\/p>\n\n\n\n We can break the entire ML workflow into three significant pipelines, one for each Machine Learning Layer.<\/p>\n\n\n\n The Data pipeline is composed of Ingestion<\/strong>, Exploration<\/strong>, Validation<\/strong>, Cleaning<\/strong>, and Splitting <\/strong>phases. <\/p>\n\n\n\n The Ingestion phase on AWS typically means bringing raw data to S3, using any available tool and technology: direct-API access, custom Lambda crawlers, S3<\/strong> LakeFormation<\/strong>, <\/strong>or Amazon<\/strong> Kinesis Firehose<\/strong>. <\/p>\n\n\n\n Then we have a preprocessing ETL phase<\/a>, which is always required<\/strong>! <\/p>\n\n\n\n AWS Glue<\/strong> is the most versatile among all the available tools for ETL, as it allows reading and aggregating information from all the previous services by using Glue Crawlers<\/strong>. These routines can poll from different data sources for new data.<\/p>\n\n\n\n We can manage Exploration, Validation, and Cleaning steps by creating custom scripts in a language of choice (e.g., Python) or using Jupyter Notebook, both orchestrated via AWS Step Functions<\/strong><\/a>. <\/p>\n\n\n\n AWS Data Wrangler<\/strong> represents another viable solution, as it can automatically take care of all the steps and connect directly to Amazon SageMaker Pipelines<\/strong>.<\/p>\n\n\n\n The Model pipeline consists of Training<\/strong>, Evaluation<\/strong>, Testing<\/strong>, and Packaging<\/strong> phases.<\/p>\n\n\n\n These phases can be managed directly from Jupyter Notebook files and integrated into a pipeline using AWS StepFunctions SageMaker SDK<\/strong>, which allows calling SageMaker functions inside a StepFunction script.<\/p>\n\n\n\n This exploit gives extreme flexibility as it allows to:<\/p>\n\n\n\n Having the logic of this Pipeline on Jupyter Notebooks has the added benefit of having everything versioned<\/strong> and easily testable<\/strong>.<\/p>\n\n\n\n Packaging can be managed through Why do we need MLOps?<\/h2>\n\n\n\n
Code<\/h3>\n\n\n\n
Development<\/h3>\n\n\n\n
Testing<\/h3>\n\n\n\n
Deployment<\/h3>\n\n\n\n
Monitoring in Production<\/h3>\n\n\n\n
Continuous Integration and Continuous Deployment<\/h3>\n\n\n\n
A typical Machine Learning workflow in the Cloud<\/h2>\n\n\n\n
![\"A](\"https:\/\/blog.besharp.it\/wp-content\/uploads\/2021\/09\/image2-1-1024x529.png\")
The Data layer<\/h3>\n\n\n\n
The Model layer<\/h3>\n\n\n\n
The Code layer<\/h3>\n\n\n\n
MLOps on AWS: the four pillars<\/h2>\n\n\n\n
![\"MLOps](\"https:\/\/blog.besharp.it\/wp-content\/uploads\/2021\/09\/image3-1-1024x630.png\")
Collaboration<\/h3>\n\n\n\n
Reproducibility<\/h3>\n\n\n\n
Continuity<\/h3>\n\n\n\n
![\"ML](\"https:\/\/blog.besharp.it\/wp-content\/uploads\/2021\/09\/image1-2-1024x768.png\")
Data engineering pipeline<\/h4>\n\n\n\n
Model pipeline<\/h4>\n\n\n\n