Creating Powerful Generative AI Applications for Enterprise Search: Harnessing Haystack Pipelines and Amazon SageMaker JumpStart with LLMs
Introduction:
Enterprise search is a vital part of digital transformation and efficient knowledge management within organizations. It involves the storage, indexing, and retrieval of digital files, providing relevant results based on user queries. With the emergence of large language models (LLMs), conversational experiences have become possible in enterprise search. However, it is crucial to ensure that LLMs are limited to company data to avoid model hallucinations. In this blog post, we demonstrate how to build an end-to-end generative AI application for enterprise search using Retrieval Augmented Generation (RAG) with Haystack pipelines and the Falcon-40b-instruct model from Amazon SageMaker JumpStart and Amazon OpenSearch Service. We provide a step-by-step solution overview, along with the necessary prerequisites and resources to implement this solution.
Full Article: Creating Powerful Generative AI Applications for Enterprise Search: Harnessing Haystack Pipelines and Amazon SageMaker JumpStart with LLMs
Building an End-to-End Generative AI Application for Enterprise Search with RAG
Enterprise search plays a crucial role in streamlining organizational efficiency through document digitization and knowledge management. It involves storing and indexing digital files and providing relevant search results based on user queries. With the emergence of large language models (LLMs), conversational experiences can be integrated into enterprise search to enhance user interaction and improve search accuracy. However, it’s important to ensure that LLMs are restricted to company data to prevent model hallucinations. In this post, we will demonstrate how to create an end-to-end generative AI application for enterprise search using Retrieval Augmented Generation (RAG) with Haystack pipelines and the Falcon-40b-instruct model from Amazon SageMaker JumpStart and Amazon OpenSearch Service.
Retrieval Augmented Generation (RAG): Limiting Responses to Company Data
To ensure that the generative AI application only provides responses based on company data, we employ a technique called Retrieval Augmented Generation (RAG). This approach involves retrieving the most relevant information from an enterprise knowledge base or content based on the user’s request, bundling it as context, and sending it along with the user’s prompt to the LLM for generating a response. Since LLMs have input prompt limitations, the selection of the right passages from thousands or millions of documents directly impacts the accuracy of the LLM. RAG has gained significant importance in enterprise search due to its effectiveness in mitigating model hallucinations.
Workflow Overview: SageMaker JumpStart, Haystack, and OpenSearch
In this post, we will showcase a workflow that leverages SageMaker JumpStart to deploy the Falcon-40b-instruct model and uses Haystack to design and implement a retrieval augmented question answering pipeline. The workflow involves the following high-level steps:
1. Retrieval: The user query is used by a retriever component to perform a vector search in order to retrieve the most relevant context from the database.
2. Prompt Generation: The retrieved context is embedded into a prompt designed to instruct the LLM to generate an answer solely based on the provided context.
3. Response Generation: The LLM generates a response to the original query by considering only the context embedded in the prompt.
SageMaker JumpStart: A Model Hub and Solution Templates
SageMaker JumpStart serves as a model hub, offering a wide range of deep learning models for text, vision, audio, and embedding use cases. With over 500 models, it includes both public and proprietary models from AWS partners as well as Amazon’s own models. SageMaker JumpStart also provides solution templates to set up infrastructure for common use cases and example notebooks for machine learning with SageMaker. It allows fine-tuning capabilities with custom data and simplifies model deployment through its user-friendly interface.
Haystack: An Open-Source Framework for LLM Applications
Haystack is an open-source framework developed by deepset that enables developers to orchestrate LLM applications consisting of various components such as models, vector databases, and file converters. With a focus on state-of-the-art retrieval methods and solid evaluation metrics, Haystack provides pipelines and agents for designing LLM applications for different use cases. It supports search, question answering, and conversational AI applications and ensures reliability and scalability from prototype to production. It provides YAML serialization, REST API integration, and flexible workload scaling.
Amazon OpenSearch: A Managed Open-Source Search Service
Amazon OpenSearch Service is a fully managed solution that simplifies the deployment, scaling, and operation of OpenSearch in the AWS Cloud. OpenSearch is an open-source software suite designed for search, analytics, security monitoring, and observability applications. ML techniques, such as embedding models, have gained popularity in enhancing search capabilities. OpenSearch Service integrates vector databases for efficient vector similarity search, enabling semantic search, RAG with LLMs, recommendation engines, and search rich media functionalities.
Application Structure: Haystack Indexing and Query Pipelines
The final application structure comprises two main components: the Haystack Indexing Pipeline and the Haystack Query Pipeline.
1. Haystack Indexing Pipeline: This component manages document uploads, document indexing, and preprocessing. It uses OpenSearch as the DocumentStore and Haystack indexing pipeline to preprocess and index files to OpenSearch. Haystack FileConverters and PreProcessor enable data cleaning and preparation, ensuring compatibility with NLP pipelines and language models. The indexing pipeline incorporates sentence-transformers/all-MiniLM-L12-v2 to create document embeddings for efficient retrieval.
2. Haystack Query Pipeline: This component handles queries sent to the RAG pipeline. The EmbeddingRetriever component filters the top-k most relevant documents from the OpenSearch index. The chosen embedding model is used to embed both the query and the documents for retrieval. The retrieved documents are then embedded into the prompt for the Falcon-40b-instruct model, which generates a response based on the retrieved context.
Model Deployment with SageMaker JumpStart
For model deployment, we utilize SageMaker JumpStart, which streamlines the process with a simple push of a button. In this example, we have used the Falcon-40b-instruct model, but any Hugging Face model available on SageMaker can be used. The final solution is available in the haystack-sagemaker repository and features the OpenSearch website and documentation as example data for retrieval augmented question answering.
Prerequisites and Setup
Before using any AWS services, ensure that you have signed up for an AWS account and created an administrative user and group. This process is outlined in the Amazon SageMaker Prerequisites documentation. To use Haystack, you need to install the farm-haystack package and its dependencies. You can accomplish this by running “pip install requirements.txt” using the provided requirements.txt file in the GitHub repository. To index documents into OpenSearch, Haystack offers connectors called DocumentStores. For this workflow, we use the OpenSearchDocumentStore, and detailed instructions for indexing options are available in the README of the example repository. Additionally, the provided CloudFormation template can be used to set up an OpenSearch service on AWS, creating an empty OpenSearch instance ready for indexing.
Conclusion
Building an end-to-end generative AI application for enterprise search requires leveraging advanced techniques such as Retrieval Augmented Generation (RAG) and utilizing powerful frameworks like Haystack and managed services like Amazon OpenSearch and SageMaker JumpStart. By integrating these components, you can create a robust application that retrieves and generates context-sensitive responses based on company data, ensuring a more accurate and reliable enterprise search experience. The example provided in the haystack-sagemaker repository offers a starting point for implementing retrieval augmented question answering using open-source resources and AWS services.
Summary: Creating Powerful Generative AI Applications for Enterprise Search: Harnessing Haystack Pipelines and Amazon SageMaker JumpStart with LLMs
This blog post explores the use of Retrieval Augmented Generation (RAG) in building an end-to-end generative AI application for enterprise search. The post showcases how to implement RAG using Haystack pipelines and the Falcon-40b-instruct model from Amazon SageMaker JumpStart and Amazon OpenSearch Service. By leveraging the capabilities of RAG, the application retrieves contextually relevant information from the enterprise knowledge base and uses it to generate accurate responses. The post provides a step-by-step workflow and highlights the use of SageMaker JumpStart, Haystack, and OpenSearch in designing and deploying the application. The source code for the sample application is also available for reference on GitHub.
Frequently Asked Questions:
1) Question: What is artificial intelligence (AI)?
Answer: Artificial intelligence (AI) refers to the simulation of human intelligence in machines that are programmed to think and learn like humans. It involves the development of intelligent computer systems capable of performing tasks that would typically require human intelligence, such as speech recognition, problem-solving, decision-making, and more.
2) Question: How is artificial intelligence used in everyday life?
Answer: Artificial intelligence is integrated into various aspects of our daily lives, including virtual assistants like Siri and Alexa, personalized recommendations on streaming platforms, voice and facial recognition technologies, automated customer service chatbots, autonomous vehicles, fraud detection systems, and more. Its application continues to expand across industries, revolutionizing the way we work and interact with technology.
3) Question: What are the benefits of artificial intelligence?
Answer: Artificial intelligence offers numerous benefits, such as increased efficiency and productivity, improved accuracy and precision, enhanced decision-making capabilities, and the ability to handle large volumes of data. It can automate repetitive tasks, reduce human error, and provide valuable insights from complex datasets. AI also has the potential to support advancements in healthcare, finance, transportation, education, and various other fields.
4) Question: Are there any risks associated with artificial intelligence?
Answer: While artificial intelligence has huge potential, there are also certain risks to consider. One concern is the potential impact on employment, as AI automation may replace certain jobs. Ethical implications, such as privacy concerns and biases in machine learning algorithms, also need to be addressed. As AI becomes more autonomous, there are also concerns about its decision-making and accountability. However, these risks can be mitigated through proper regulation and guidelines.
5) Question: What is the future of artificial intelligence?
Answer: The future of artificial intelligence holds immense possibilities. AI technology is expected to advance further, enabling more complex tasks and human-like interactions. It will continue to transform industries and drive innovation. However, ethical considerations, transparency, and responsible development of AI will be critical factors in shaping its future. With ongoing research and development, AI has the potential to usher in a new era of progress and improve the quality of life for individuals and societies as a whole.
