MLOps Stack for Scalable Machine Learning

Building a robust MLOps stack is crucial for scalable machine learning. This requires integrating various tools and technologies to streamline the entire machine learning lifecycle.

The MLOps stack typically includes a data pipeline for efficient data ingestion and processing. This is where data engineers play a crucial role in designing and implementing the pipeline.

Data quality and preprocessing are critical steps in the MLOps stack. A well-designed data preprocessing step can significantly improve model performance by handling missing values, outliers, and data normalization.

Model training and deployment are also key components of the MLOps stack. This involves selecting a suitable machine learning algorithm, training the model using a suitable dataset, and deploying it to a production environment.

MLOps is a set of best practices that focus on making it easier to deploy machine learning models in production.

It's a discipline that combines machine learning with IT operations, and its goal is to shorten the development lifecycle while improving model reliability and stability.

MLOps defines language, framework, platform, and infrastructure practices to design, develop, and maintain machine learning models.

As many as half of all organizations experience significant challenges with integrating their ML tooling, frameworks, and languages technology stacks.

The field of machine learning and related technologies are in a state of near-constant change, making it challenging to keep up with the pace of development.

The development of MLOps is also taking place alongside it at an equally rapid pace, creating the additional challenge of having to adopt infrastructure that itself is also evolving.

Automation is now the name of the game in ML lifecycle management, and successful ML cycle management requires the adoption of machine learning tools to achieve success.

The goal of MLOps is to shorten the development lifecycle while improving model reliability and stability by automating repeatable steps in ML workflows.

As many as half of all organizations experience significant challenges with integrating their ML tooling, frameworks, and languages technology stacks.

Automating repeatable steps in ML workflows is crucial for improving model reliability and stability.

Machine learning teams must adopt a sophisticated MLOps stack to achieve success, as manual processes are no longer sufficient.

To choose the right MLOps tools, ML teams must understand their organization's mission, long-term goals, its current data science environment, and the value that MLOps could deliver to it.

The field of machine learning and related technologies are in a state of near-constant change, making it challenging to keep up with the pace of development.

MLOps defines language, framework, platform, and infrastructure practices to design, develop, and maintain machine learning models.

Organizations will find that the pressure for adopting a sophisticated MLOps stack will only continue to increase alongside the proliferation of machine learning and the rapid growth of increasingly powerful tools.

An MLOps Stack uses Databricks Asset Bundles, a collection of source files that serves as the end-to-end definition of a project. These source files include information about how they are to be tested and deployed.

The files created for the default MLOps Stack include information about how they are to be tested and deployed. For details about the files included in the stack, see the documentation on the GitHub repository or Databricks Asset Bundles for MLOps Stacks.

The default MLOps Stack project includes an ML pipeline with CI/CD workflows to test and deploy automated model training and batch inference jobs across development, staging, and production Databricks workspaces.

Here's a breakdown of the files created for the default MLOps Stack:

These files make it easy to co-version changes and use software engineering best practices such as source control, code review, testing, and CI/CD.

The MLOps stack is a collection of tools that help with tasks that data scientists often don't like to do or worry about. These tools can be categorized into different areas, such as data labeling, model and data versioning, experiment tracking, scaled-out training, and model execution.

Assessing MLOps tools requires evaluating them based on your own requirements, which include flexibility, framework support, language support, multiuser support, maturity, and community support. These factors will help you determine which tool best fits your needs.

A typical MLOps stack includes a unified platform, such as Databricks, which uses a set of tools to support the development process. These tools include Databricks notebooks, MLflow, Feature engineering, Models in Unity Catalog, Mosaic AI Model Serving, Databricks Asset Bundles, Databricks Jobs, GitHub Actions, Azure DevOps, and Lakehouse monitoring.

Here are some key components of the MLOps stack:

Assessing tools for machine learning (ML) engineering and deployment can be a daunting task. With the vast number of existing ML tools and platforms, comparing them is not a trivial matter.

One key consideration is flexibility – can the tool be easily adopted in multiple situations, meeting the needs for different modeling techniques? This is crucial for organizations with diverse projects and requirements.

Flexibility is a top priority when evaluating ML tools, as it ensures that the tool can adapt to changing needs and support various modeling approaches.

Here are some key factors to consider when assessing ML tools:

By considering these factors, you can make an informed decision about which ML tool is best suited for your organization's needs.

Databricks offers a customizable stack for starting new ML projects that follow production best-practices out of the box.

The Databricks MLOps Stacks feature is currently in public preview and provides a quick way for data scientists to start iterating on ML code for new projects while ops engineers set up CI/CD and ML resources management.

This stack includes three modular components that make it easy to get started with ML projects.

Here are the three components of the Databricks MLOps Stacks:

Databricks asset bundles and Databricks asset bundle templates are also in public preview.

A feature store is an optional component for level 1 ML pipeline automation that helps standardize the definition, storage, and access of features for training and serving.

By using a feature store, data scientists can discover and reuse available feature sets for their entities, instead of re-creating the same or similar ones.

Having a feature store helps maintain features and their related metadata, which avoids having similar features with different definitions.

A feature store provides an API for both high-throughput batch serving and low-latency real-time serving for feature values.

This ensures that the features used for training are the same ones used during serving, avoiding training-serving skew.

Data scientists can serve up-to-date feature values from the feature store, making it easier to keep their models accurate and reliable.

Here are some benefits of using a feature store:

Pipeline automation and CI/CD are crucial components of a successful MLOps stack. They enable data scientists to rapidly iterate on ML code and file pull requests, triggering unit tests and integration tests in an isolated staging Databricks workspace.

Automating the ML pipeline is the goal of MLOps level 1, which involves continuous training of the model by automating the pipeline, introducing automated data and model validation steps, and pipeline triggers and metadata management.

A robust automated CI/CD system is necessary for a rapid and reliable update of the pipelines in production. This system lets data scientists rapidly explore new ideas and implement them, automatically building, testing, and deploying the new pipeline components to the target environment.

The CI/CD pipeline consists of several stages, including development and experimentation, pipeline continuous integration, pipeline continuous delivery, automated triggering, model continuous delivery, and monitoring. Each stage has specific outputs, such as pipeline components, deployed pipeline, trained model, and deployed model prediction service.

In the CI stage, the pipeline and its components are built, tested, and packaged when new code is committed or pushed to the source code repository. This stage includes various tests, such as unit testing, testing model training convergence, and testing for NaN values.

To ensure rapid and reliable continuous delivery of pipelines and models, it's essential to verify the compatibility of the model with the target infrastructure, test the prediction service, test prediction service performance, validate the data, and verify that models meet predictive performance targets before deployment.

Model quality is a crucial aspect of the MLOps stack. Unfortunately, most MLOps tools have focused on automating the logistics of operations, leaving model quality as a missing layer.

The lack of focus on model quality is due to the industry's limited thinking on what constitutes ML quality. Traditionally, predictive accuracy has been the primary focus, but a holistic approach is needed to cover factors such as transparency, justifiability, and data quality.

ML systems are characterized by greater uncertainty compared to traditional software, making it harder to ensure model quality. Models make predictions based on patterns learned from potentially incorrect, incomplete, or unrepresentative real-world data.

The ML lifecycle involves more iterations and interdependencies between stages, making it challenging to detect and address poor performance. For example, detecting poor performance in one segment may necessitate incremental data sourcing and/or re-sampling further back in the cycle.

A key reason for the lack of focus on model quality is that it's a more complex problem to solve compared to automating mechanical process steps. There are well-established precedents in traditional software engineering, and outcomes are easily measurable.

Here are the key factors that contribute to ML quality:

By focusing on these factors, organizations can build high-quality models that are more likely to survive messy, real-life situations during their lifetime.