Update HelloLlamaLocal.ipynb

demo_apps/HelloLlamaLocal.ipynb

@@ -11,6 +11,19 @@
     "* how to use LangChain to load a recent PDF doc - the Llama2 paper pdf - and ask questions about it. This is the well known RAG (Retrieval Augmented Generation) method to let LLM such as Llama2 be able to answer questions about the data not publicly available when Llama2 was trained, or about your own data. RAG is one way to prevent LLM's hallucination. "
    ]
   },
+  {
+   "cell_type": "markdown",
+   "id": "22450267",
+   "metadata": {},
+   "source": [
+    "We start by installing necessary requirements and import packages we will be using in this example.\n",
+    "- [llama-cpp-python](https://github.com/abetlen/llama-cpp-python) a simple Python bindings for [llama.cpp](https://github.com/ggerganov/llama.cpp) library\n",
+    "- pypdf gives us the ability to work with pdfs\n",
+    "- sentence-transformers for text embeddings\n",
+    "- chromadb gives us database capabilities \n",
+    "- langchain provides necessary RAG tools for this demo"
+   ]
+  },
   {
    "cell_type": "code",
    "execution_count": 1,
@@ -134,6 +147,14 @@
     "from langchain.prompts import PromptTemplate"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "id": "73df46d9",
+   "metadata": {},
+   "source": [
+    "Next, initialize the langchain CallBackManager. This handles callbacks from Langchain and for this example we will use token-wise streaming so the answer gets generated token by token when Llama is answering your question."
+   ]
+  },
   {
    "cell_type": "code",
    "execution_count": 3,
@@ -145,6 +166,25 @@
     "callback_manager = CallbackManager([StreamingStdOutCallbackHandler()])"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "id": "8536c352",
+   "metadata": {},
+   "source": [
+    "\n",
+    "Set up the Llama 2 model. \n",
+    "Replace `<path-to-llama-gguf-file>` with the path either to your downloaded quantized model file [here](https://drive.google.com/file/d/1afPv3HOy73BE2MoYCgYJvBDeQNa9rZbj/view?usp=sharing), or to the ggml-model-q4_0.gguf file built with the following commands:\n",
+    "\n",
+    "```bash\n",
+    "git clone https://github.com/ggerganov/llama.cpp\n",
+    "cd llama.cpp\n",
+    "python3 -m pip install -r requirements.txt\n",
+    "python convert.py <path_to_your_downloaded_llama-2-13b_model>\n",
+    "./quantize <path_to_your_downloaded_llama-2-13b_model>/ggml-model-f16.gguf <path_to_your_downloaded_llama-2-13b_model>/ggml-model-q4_0.gguf q4_0\n",
+    "```\n",
+    "For more info see https://python.langchain.com/docs/integrations/llms/llamacpp"
+   ]
+  },
   {
    "cell_type": "code",
    "execution_count": null,
@@ -152,7 +192,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "# create the Llama2 model - for more info see https://python.langchain.com/docs/integrations/llms/llamacpp\n",
+    "\n",
     "llm = LlamaCpp(\n",
     "    model_path=\"<path-to-llama-gguf-file>\"\n",
     "    temperature=0.0,\n",
@@ -163,6 +203,15 @@
     ")"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "id": "f2cae215",
+   "metadata": {},
+   "source": [
+    "With the model set up, you are now ready to ask some questions. \n",
+    "Here is an example of the simplest way to ask the model some general questions."
+   ]
+  },
   {
    "cell_type": "code",
    "execution_count": 5,
@@ -192,11 +241,20 @@
     }
    ],
    "source": [
-    "# the simplest way to ask Llama some general questions\n",
+    "\n",
     "question = \"who wrote the book Innovator's dilemma?\"\n",
     "answer = llm(question)"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "id": "545cb6aa",
+   "metadata": {},
+   "source": [
+    "Alternatively, you can sue LangChain's PromptTemplate for some flexibility in your prompts and questions.\n",
+    "For more information on LangChain's prompt template visit this [link](https://python.langchain.com/docs/modules/model_io/prompts/prompt_templates/)"
+   ]
+  },
   {
    "cell_type": "code",
    "execution_count": 6,
@@ -241,6 +299,15 @@
     "answer = chain.run(\"innovator's dilemma\")"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "id": "189de613",
+   "metadata": {},
+   "source": [
+    "Now, let's see how Llama2 hallucinates, because it did not have knowledge about Llama2 at the time it was trained. \n",
+    "By default it behaves like a know-it-all expert who will not say \"I don't know\"."
+   ]
+  },
   {
    "cell_type": "code",
    "execution_count": 7,
@@ -287,8 +354,7 @@
     }
    ],
    "source": [
-    "# let's see how Llama2 hallucinates, because it doesn't have the knowledge about Llama2 while the model was trained, \n",
-    "# but by default it behaves like a know-it-all expert who can't afford to say I don't know\n",
+    "\n",
     "prompt = PromptTemplate.from_template(\n",
     "    \"What is {what}?\"\n",
     ")\n",
@@ -296,6 +362,15 @@
     "answer = chain.run(\"llama2\")"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "id": "37f77909",
+   "metadata": {},
+   "source": [
+    "One way we can fix the hallucinations is to use RAG, to augment it with more recent or custom data that holds the info for it to answer correctly.\n",
+    "First we load the Llama2 paper using LangChain's [PDF loader](https://python.langchain.com/docs/modules/data_connection/document_loaders/pdf)"
+   ]
+  },
   {
    "cell_type": "code",
    "execution_count": 8,
@@ -303,8 +378,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "# to fix the LLM's hallucination, one way is to use RAG, to augment it with more recent or custom data that holds the info for it to answer correctly\n",
-    "# first load the Llama2 paper via the LangChain's PDF loader\n",
+    "\n",
     "from langchain.document_loaders import PyPDFLoader\n",
     "loader = PyPDFLoader(\"llama2.pdf\")\n",
     "documents = loader.load()"
@@ -333,6 +407,19 @@
     "print(len(documents), documents[0].page_content[0:300])"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "id": "8c4ede5b",
+   "metadata": {},
+   "source": [
+    "Next we will store our documents. \n",
+    "There are more than 30 vector stores (DBs) supported by LangChain. \n",
+    "For this example we will use [Chroma](https://python.langchain.com/docs/integrations/vectorstores/chroma) which is light-weight and in memory so it's easy to get started with.\n",
+    "For other vector stores especially if you need to store a large amount of data - see https://python.langchain.com/docs/integrations/vectorstores\n",
+    "\n",
+    "We will also import the HuggingFaceEmbeddings and RecursiveCharacterTextSplitter to assist in storing the documents."
+   ]
+  },
   {
    "cell_type": "code",
    "execution_count": 10,
@@ -340,8 +427,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "# there're more 30 vector stores (DBs) supported by LangChain. Chroma is light-weight and in memory so it's easy to get started with\n",
-    "# other vector stores can be used to store large amount of data - see https://python.langchain.com/docs/integrations/vectorstores\n",
+    "\n",
     "from langchain.vectorstores import Chroma\n",
     "\n",
     "# embeddings are numerical representations of the question and answer text\n",
@@ -351,6 +437,15 @@
     "from langchain.text_splitter import RecursiveCharacterTextSplitter"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "id": "0bfdacf7",
+   "metadata": {},
+   "source": [
+    "\n",
+    "To store the documents, we will need to split them into chunks using [`RecursiveCharacterTextSplitter`](https://python.langchain.com/docs/modules/data_connection/document_transformers/text_splitters/recursive_text_splitter) and create vector representations of these chunks using [`HuggingFaceEmbeddings`](https://www.google.com/search?q=langchain+hugging+face+embeddings&sca_esv=572890011&ei=ARUoZaH4LuumptQP48ah2Ac&oq=langchian+hugg&gs_lp=Egxnd3Mtd2l6LXNlcnAiDmxhbmdjaGlhbiBodWdnKgIIADIHEAAYgAQYCjIHEAAYgAQYCjIHEAAYgAQYCjIHEAAYgAQYCjIHEAAYgAQYCjIHEAAYgAQYCjIHEAAYgAQYCjIHEAAYgAQYCjIHEAAYgAQYCjIHEAAYgAQYCkjeHlC5Cli5D3ABeAGQAQCYAV6gAb4CqgEBNLgBAcgBAPgBAcICChAAGEcY1gQYsAPiAwQYACBBiAYBkAYI&sclient=gws-wiz-serp) to them before storing them into our vector database. \n"
+   ]
+  },
   {
    "cell_type": "code",
    "execution_count": 11,
@@ -370,6 +465,15 @@
     ")"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "id": "bddc38e8",
+   "metadata": {},
+   "source": [
+    "\n",
+    "We then use ` RetrievalQA` to retrieve the documents from the vector database and give the model more context on Llama 2, thereby increasing its knowledge."
+   ]
+  },
   {
    "cell_type": "code",
    "execution_count": 12,
@@ -411,14 +515,17 @@
     "    llm,\n",
     "    retriever=vectordb.as_retriever()\n",
     ")\n",
-    "\n",
-    "# for each question, LangChain performs a semantic similarity search of it in the vector db, then passes the search results as the context\n",
-    "# the Llama to answer question about the data stored in the verctor db\n",
-    "question = \"What is llama2?\"\n",
-    "result = qa_chain({\"query\": question})\n",
-    "# it takes close to 2 minutes to return the result (but using other vector store than Chroma such as FAISS can take longer), because \n",
-    "# Llama2 is running on a local Mac. To get much faster results, you can use a cloud service with GPU used for inference - see HelloLlamaCloud \n",
-    "# for a demo."
+    "\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "db71e5d7",
+   "metadata": {},
+   "source": [
+    "For each question, LangChain performs a semantic similarity search of it in the vector db, then passes the search results as the context to the model to answer the question.\n",
+    "It takes close to 2 minutes to return the result (but using other vector stores other than Chroma such as FAISS can take longer) because Llama2 is running on a local Mac. \n",
+    "To get much faster results, you can use a cloud service with GPU used for inference - see HelloLlamaCloud for a demo."
    ]
   },
   {
@@ -427,7 +534,11 @@
    "id": "dd2e62a4-6ea2-4ea7-b7ae-800185177e6c",
    "metadata": {},
    "outputs": [],
-   "source": []
+   "source": [
+    "\n",
+    "question = \"What is llama2?\"\n",
+    "result = qa_chain({\"query\": question})\n"
+   ]
   }
  ],
  "metadata": {