Maximize AWS Inferentia Usage with FastAPI and PyTorch Models on Amazon EC2 Inf1 & Inf2 Instances: A Guide to Enhanced Performance
Introduction:
When deploying Deep Learning models at scale, it is essential to optimize performance and cost by efficiently utilizing hardware resources. This is especially crucial for production workloads with high throughput and low latency requirements. In this post, we will guide you through the process of deploying FastAPI model servers on AWS Inferentia devices, specifically on Amazon EC2 Inf1 and Inf2 instances.
FastAPI is an open-source web framework built for Python applications. With its asynchronous server gateway interface (ASGI), it offers faster processing of incoming requests compared to traditional frameworks like Flask and Django. This makes it an ideal choice for handling latency-sensitive requests.
By utilizing FastAPI, you can deploy a server on Inferentia instances that listens to client requests through a designated port. The objective is to achieve maximum performance at the lowest cost by fully utilizing the hardware capabilities. Each Inferentia device contains multiple NeuronCores, and the AWS Neuron SDK allows parallel utilization of these cores, enabling the loading and inference of multiple models simultaneously without compromising throughput.
FastAPI supports various Python web servers such as Gunicorn, Uvicorn, Hypercorn, and Daphne, providing an abstraction layer on top of the Machine Learning models. This abstraction allows clients to make requests without being aware of the specific model versions deployed. In contrast, framework-specific serving tools require clients to know and adapt to the changing endpoint names when models are updated.
To ensure availability and efficiency, an ASGI server spawns a specified number of workers to handle client requests and run the inference code. If a worker is terminated, the server automatically launches a new one. In this post, we utilize the Hypercorn server, a popular choice for Python web servers.
Our best practices guide will demonstrate how to deploy deep learning models with FastAPI on AWS Inferentia NeuronCores, showcasing the concurrent deployment of multiple models on separate NeuronCores. This setup maximizes throughput and NeuronCore utilization. You can find the code for this implementation on our GitHub repository.
AWS Inferentia NeuronCores, available in both Inf1 and Inf2 instances, offer different configurations and performance capabilities. Inf2 instances come with NeuronCores-v2, which provide 4 times higher throughput and 10 times lower latency compared to NeuronCores-v1 in Inf1 instances. The Neuron Runtime, responsible for running models on Neuron devices, can be configured using environment variables for optimized behavior. It allows you to reserve specific NeuronCores for your Python processes or span them across multiple devices.
Performance optimization requires careful consideration of available host vCPUs, memory, and NeuronCore utilization. By running tests and analyzing metrics like core utilization and memory usage with tools like Neuron Top, you can make informed decisions about configuration and instance selection.
To get started with AWS Neuron capabilities for PyTorch and try out the Neuron SDK features yourself, refer to the latest Neuron capabilities documentation.
In the upcoming sections of this blog, we provide a step-by-step system setup and detailed instructions on how to deploy and optimize deep learning models using FastAPI on AWS Inferentia NeuronCores.
Full Article: Maximize AWS Inferentia Usage with FastAPI and PyTorch Models on Amazon EC2 Inf1 & Inf2 Instances: A Guide to Enhanced Performance
Deploying FastAPI Model Servers on AWS Inferentia Devices for Maximum Hardware Utilization
Introduction
When deploying Deep Learning models at scale, it is crucial to effectively utilize the underlying hardware to maximize performance and cost benefits. In this article, we will walk you through the process of deploying FastAPI model servers on AWS Inferentia devices, specifically Inf1 and Inf2 instances. By utilizing FastAPI, Neuron SDK, and AWS Inferentia NeuronCores, you can achieve high throughput and low latency for your production workloads.
Solution Overview
FastAPI is an open-source web framework that allows for faster serving of Python applications compared to traditional frameworks like Flask and Django. It utilizes an Asynchronous Server Gateway Interface (ASGI) instead of the commonly used Web Server Gateway Interface (WSGI), enabling asynchronous processing of incoming requests. This makes FastAPI an ideal choice for handling latency-sensitive requests.
To achieve the highest performance at the lowest cost, we deploy FastAPI model servers on Inferentia instances, which are powered by AWS NeuronCores. Each Inferentia1 device contains four NeuronCores-v1, while each Inferentia2 device contains two NeuronCores-v2. The AWS Neuron SDK enables us to utilize each of the NeuronCores in parallel, allowing for the loading and inference of multiple models simultaneously without sacrificing throughput.
Deployment Architecture
The deployment architecture involves setting up a server that hosts an endpoint on an Inferentia instance, listening to client requests through a designated port. With FastAPI, you can choose from various Python web servers such as Gunicorn, Uvicorn, Hypercorn, and Daphne. These web servers provide an abstraction layer on top of the underlying Machine Learning (ML) model, making it easier for the requesting client to interact with the server without knowing the details of the hosted model.
Best Practices for Serving Models with FastAPI
By using a generic Python web server like FastAPI, you can serve multiple models concurrently on separate NeuronCores, increasing throughput and optimizing NeuronCore utilization. The endpoint name serves as a proxy to a function that loads and runs the model, eliminating the need for the client to know the specific model’s name or version. This is especially beneficial when continuously evolving the version models, such as in A/B testing scenarios.
Server Configuration and NeuronCores
An ASGI server, such as Hypercorn, is responsible for spawning a specified number of workers to process client requests and run the inference code. It ensures that the requested number of workers are active and available. The server and workers can be identified by their Unix process ID (PID).
AWS Inferentia NeuronCores, whether v1 or v2, offer high performance and lower latency compared to traditional instances. Their configuration can be controlled through environment variables like NEURON_RT_NUM_CORES and NEURON_RT_VISIBLE_CORES. These variables allow you to reserve a specific number or range of NeuronCores for your processes.
Choosing the Right Instance and Model Configuration
Depending on your application’s NeuronCore usage, vCPU, and memory usage, it is recommended to run tests to determine the most performant configuration. The Neuron Top tool can assist in visualizing core utilization and memory usage, helping you make informed decisions regarding instance selection and model deployment.
Conclusion
Deploying FastAPI model servers on AWS Inferentia devices, such as Inf1 and Inf2 instances, allows for maximum hardware utilization and optimal performance for production workloads. By leveraging the power of NeuronCores, FastAPI, and ASGI servers, you can achieve high throughput and low latency while efficiently utilizing your hardware resources.
Please note that this article only provides an overview of the deployment process and best practices. For detailed instructions and code examples, refer to the GitHub repository associated with this article.
Summary: Maximize AWS Inferentia Usage with FastAPI and PyTorch Models on Amazon EC2 Inf1 & Inf2 Instances: A Guide to Enhanced Performance
When deploying Deep Learning models at scale, it is important to optimize the hardware to maximize performance and cost benefits. This article discusses the process of deploying FastAPI model servers on AWS Inferentia devices, specifically on Amazon EC2 Inf1 and EC2 Inf2 instances. FastAPI is an efficient web framework for serving Python applications, ideal for handling latency-sensitive requests. By deploying multiple models on separate NeuronCores and utilizing the Neuron SDK, it is possible to achieve high throughput and optimal hardware utilization. The article also provides details on the NeuronCores available in different instance types and the configuration options for Neuron Runtime. Furthermore, it offers recommendations for choosing the right instance type based on your application’s requirements. A system setup and the steps to set up the solution are also discussed.
