Minimize Hosting Costs by Deploying Thousands of Model Ensembles on GPU with Amazon SageMaker Multi-Model Endpoints
Introduction:
Artificial intelligence (AI) adoption is rapidly increasing across various industries, leveraging deep learning (DL), large language models (LLMs), and generative AI. These advancements allow users to utilize advanced solutions with nearly human-like performance. To efficiently run DL applications, hardware acceleration is often required, and GPUs are highly suitable for these tasks due to their parallel processing capabilities. However, DL applications also involve preprocessing or postprocessing in an inference pipeline, such as image resizing or text tokenization. NVIDIA Triton is an open-source inference server that enables users to define inference pipelines using a Directed Acyclic Graph (DAG). Amazon SageMaker seamlessly supports Triton’s deployment, providing a managed, secure environment with MLOps tools integration, automatic model scaling, and cost optimization features like multi-model endpoints. This post demonstrates how to deploy multiple deep learning ensemble models on a GPU instance using SageMaker MME. With MMEs, a single container hosts multiple models, and SageMaker dynamically loads and caches models as they are invoked, optimizing memory utilization and reducing hosting costs. By utilizing Triton ensembles, users can create an inference pipeline with multiple models and define the connection between input and output tensors. SageMaker Deep Learning Containers (DLC) images are recommended for optimal performance, and this solution walkthrough deploys two different types of ensembles: one for image preprocessing and inference using DALI and TensorFlow Inception v3 models, and the other for text preprocessing and postprocessing using BERT models. This walkthrough also covers environment setup, preparing ensembles, and invoking the ensembles with image and text inputs. Get started with deploying powerful and efficient deep learning ensemble models on GPU instances using SageMaker MME and Triton inference server.
Full Article: Minimize Hosting Costs by Deploying Thousands of Model Ensembles on GPU with Amazon SageMaker Multi-Model Endpoints
AI Adoption Accelerates Across Industries and Use Cases: NVIDIA Triton and Amazon SageMaker Enable Efficient Deployment of Deep Learning Ensemble Models
The adoption of artificial intelligence (AI) is rapidly increasing across various industries and use cases. Recent advancements in deep learning (DL), large language models (LLMs), and generative AI have allowed customers to leverage state-of-the-art solutions with performance that rivals human capabilities. However, these complex models often require hardware acceleration to enable faster training and real-time inference using deep neural networks. Graphics processing units (GPUs) with their parallel processing cores are well-suited for such DL tasks.
NVIDIA Triton: An Open-Source Inference Server for DL Models
NVIDIA Triton is an open-source inference server that allows users to define complex inference pipelines as ensembles of DL models in the form of a Directed Acyclic Graph (DAG). It is designed to scale and run models on both CPU and GPU. By seamlessly integrating Triton with Amazon SageMaker, users can take advantage of Triton’s capabilities while benefiting from the managed and secured environment of SageMaker, which includes MLOps tools integration and automatic scaling of hosted models.
Cost Savings and Efficiency with SageMaker Multi-Model Endpoints (MMEs)
AWS understands the importance of helping customers achieve maximum cost savings. In addition to innovative pricing options and cost optimization services, AWS has launched features like multi-model endpoints (MMEs) to further enhance cost savings. MMEs provide a cost-effective solution for deploying numerous models using a shared serving container, reducing hosting costs by enabling multiple models to be deployed while paying for only a single inference environment. MMEs also simplify deployment by managing model loading in memory and scaling based on traffic patterns to the endpoint.
Running Multiple Deep Learning Ensemble Models with GPUs and SageMaker MMEs
In this post, we demonstrate how to run multiple deep learning ensemble models on a GPU instance using SageMaker MMEs. You can find the code for this example in the public SageMaker examples repository. The integration of MMEs with GPUs offers enhanced performance and enables efficient deployment of multiple models.
How SageMaker MMEs with GPU Work
SageMaker MMEs use a single container to host multiple models. SageMaker controls the lifecycle of the models by dynamically loading and caching them into the container’s memory as they are invoked. When an invocation request for a specific model is made, SageMaker routes the request to the endpoint instance. If the model has not been loaded, it downloads the model artifact from Amazon S3 to the instance’s Amazon Elastic Block Storage volume. Once loaded, the model is stored in the container’s memory on the GPU instance. If the model is already loaded, invocation is faster without the need for further steps. SageMaker automatically unloads unused models to free up memory when necessary, while keeping them stored on the instance’s EBS volume to avoid repeated downloads from S3. If the storage volume reaches capacity, unused models are deleted.
Dynamic Deployment and Cost Savings with MMEs and Triton Ensembles
SageMaker MMEs provide a cost-effective and efficient mechanism to deploy multiple models. When the MME receives numerous invocation requests, SageMaker routes requests to other instances in the inference cluster to accommodate high traffic. Adding or deleting models from an MME does not require updating the endpoint itself. To add a new model, upload it to the specified S3 bucket. To delete a model, stop sending requests and remove it from the bucket.
Triton Ensembles: Building Complex Inference Pipelines
Triton ensembles enable the creation of complex inference pipelines consisting of one or more models, data preprocessing, and postprocessing logic. A single inference request triggers the execution of the entire pipeline according to the ensemble scheduler. Triton accesses models from either local or remote locations, such as Amazon S3, through a model repository. Each model in the repository requires a model configuration file that specifies essential information about the model, such as the platform or backend, max batch size, and input/output tensors.
SageMaker Integration with Triton and Custom Code Deployment
SageMaker provides seamless integration with Triton server through managed Triton Inference Server Containers. These containers support various ML frameworks, including TensorFlow, ONNX, PyTorch, and custom model formats. Using SageMaker Deep Learning Containers (DLC) images is recommended for optimal performance and security. In this solution walkthrough, we deploy two different types of ensembles on a GPU instance using Triton and a single SageMaker endpoint. One ensemble consists of a DALI model for image preprocessing and a TensorFlow Inception v3 model for inference. The other ensemble transforms natural language sentences into embeddings using a preprocessing model, a BERT model for token embeddings extraction, and a postprocessing model for combining raw token embeddings into sentence embeddings.
Environment Setup and Example Invocations
Before deploying the ensembles, we set up the required environment and dependencies, including updating AWS libraries and installing Triton dependencies. We use the default SageMaker SDK execution role to enable access to S3 and the container registry. The post provides an example of invoking each ensemble on the SageMaker endpoint, specifying the target ensemble according to the input type.
Conclusion
The combination of NVIDIA Triton and Amazon SageMaker offers a powerful and efficient solution for deploying deep learning ensemble models. With the ability to scale models on both CPU and GPU, as well as features like MMEs and cost-saving options, users can leverage advanced AI capabilities while optimizing costs and achieving high performance. By following the provided solution walkthrough, users can easily deploy and run multiple deep learning ensemble models on a GPU instance using SageMaker MMEs and Triton.
Summary: Minimize Hosting Costs by Deploying Thousands of Model Ensembles on GPU with Amazon SageMaker Multi-Model Endpoints
Artificial intelligence (AI) adoption is rapidly increasing across various industries and applications, thanks to recent advancements in technologies like deep learning, large language models, and generative AI. These advanced models often require hardware acceleration, such as GPUs, to achieve high-performance results. NVIDIA Triton is an open-source inference server that enables users to define complex inference pipelines as a Directed Acyclic Graph (DAG) and run them at scale on both CPU and GPU. Amazon SageMaker seamlessly supports deploying Triton, providing a managed and secure environment with cost-saving features like multi-model endpoints. This post walks you through the process of running deep learning ensemble models on a GPU instance with SageMaker MME. By utilizing the powerful capabilities of MMEs, you can reduce hosting costs, optimize model deployment, and scale resources efficiently.
Frequently Asked Questions:
Q1: What is Artificial Intelligence (AI)?
A1: Artificial Intelligence, commonly known as AI, refers to the simulation of human intelligence in machines that are programmed to think and learn like humans. AI enables machines to analyze and interpret data, make decisions, solve problems, and perform tasks that usually require human intelligence.
Q2: How does Artificial Intelligence work?
A2: AI systems work through a combination of algorithms, data, and computing power. These systems use machine learning, deep learning, natural language processing, and other techniques to process and understand large amounts of data, recognize patterns, and make predictions or decisions based on this learning.
Q3: What are the main applications of Artificial Intelligence?
A3: Artificial Intelligence finds applications in various fields today. It powers virtual assistants like Siri and Alexa, enables autonomous vehicles, assists in medical diagnosis, improves customer service through chatbots, enhances cybersecurity, optimizes logistics and supply chain management, and even aids in personalized recommendations and targeted advertising.
Q4: What are the benefits of Artificial Intelligence?
A4: Artificial Intelligence offers a multitude of benefits. It can automate repetitive tasks, improve efficiency and accuracy, enhance decision-making abilities, provide real-time insights, boost productivity, enable faster data analysis, and enable systems to adapt and learn from new data or situations.
Q5: What are the ethical considerations surrounding Artificial Intelligence?
A5: As AI progresses, ethical concerns come to the forefront. Issues such as biased algorithms, privacy concerns, job displacement, transparency, accountability, and the potential misuse of AI require careful attention. Organizations and policymakers are actively working on establishing guidelines and regulations to address these concerns and ensure responsible AI deployment.
Remember, these questions and answers are designed to provide a general understanding of Artificial Intelligence and its related aspects, catering to individuals with basic knowledge on the topic. For more detailed or technical information, it is recommended to refer to specialized AI resources and expert opinions.
