Update HelloLlamaLocal.ipynb

7089e316 · sekyonda · f0109b24 · 7089e316
Commit 7089e316 authored 1 year ago by sekyonda
--- a/demo_apps/HelloLlamaLocal.ipynb
+++ b/demo_apps/HelloLlamaLocal.ipynb
@@ -6,9 +6,9 @@
   "metadata": {},
   "source": [
    "## This demo app shows:\n",
-    "* how to run Llama2 locally on a Mac using llama-cpp-python and the llama-cpp's quantized Llama2 model;\n",
-    "* how to use LangChain to ask Llama general questions;\n",
-    "* how to use LangChain to load a recent PDF doc - the Llama2 paper pdf - and ask questions about it. This is the well known RAG (Retrieval Augmented Generation) method to let LLM such as Llama2 be able to answer questions about the data not publicly available when Llama2 was trained, or about your own data. RAG is one way to prevent LLM's hallucination. "
+    "* How to run Llama2 locally on a Mac using llama-cpp-python and the llama-cpp's quantized Llama2 model\n",
+    "* How to use LangChain to ask Llama general questions\n",
+    "* How to use LangChain to load a recent PDF doc - the Llama2 paper pdf - and ask questions about it. This is the well known RAG (Retrieval Augmented Generation) method to let LLM such as Llama2 be able to answer questions about the data not publicly available when Llama2 was trained, or about your own data. RAG is one way to prevent LLM's hallucination"
   ]
  },
  {

 %% Cell type:markdown id:b7cabb96-2715-492e-825a-8f4ff161dc3b tags:

 ## This demo app shows:
-* how to run Llama2 locally on a Mac using llama-cpp-python and the llama-cpp's quantized Llama2 model;
-* how to use LangChain to ask Llama general questions;
-* how to use LangChain to load a recent PDF doc - the Llama2 paper pdf - and ask questions about it. This is the well known RAG (Retrieval Augmented Generation) method to let LLM such as Llama2 be able to answer questions about the data not publicly available when Llama2 was trained, or about your own data. RAG is one way to prevent LLM's hallucination.
+* How to run Llama2 locally on a Mac using llama-cpp-python and the llama-cpp's quantized Llama2 model
+* How to use LangChain to ask Llama general questions
+* How to use LangChain to load a recent PDF doc - the Llama2 paper pdf - and ask questions about it. This is the well known RAG (Retrieval Augmented Generation) method to let LLM such as Llama2 be able to answer questions about the data not publicly available when Llama2 was trained, or about your own data. RAG is one way to prevent LLM's hallucination

 %% Cell type:markdown id:22450267 tags:

 We start by installing necessary requirements and import packages we will be using in this example.
 - [llama-cpp-python](https://github.com/abetlen/llama-cpp-python) a simple Python bindings for [llama.cpp](https://github.com/ggerganov/llama.cpp) library
 - pypdf gives us the ability to work with pdfs
 - sentence-transformers for text embeddings
 - chromadb gives us database capabilities
 - langchain provides necessary RAG tools for this demo

 %% Cell type:code id:2922732e-29e8-4ea7-8828-53364f5bf6fd tags:

 ``` python
 # install all the required packages for the demo
 !CMAKE_ARGS="-DLLAMA_METAL=on" FORCE_CMAKE=1 pip install llama-cpp-python
 !pip install pypdf sentence-transformers chromadb langchain
 ```

 %% Cell type:code id:26bc4912 tags:

 ``` python
 from langchain.llms import LlamaCpp
 from langchain.chains import LLMChain
 from langchain.callbacks.manager import CallbackManager
 from langchain.callbacks.streaming_stdout import StreamingStdOutCallbackHandler
 from langchain.prompts import PromptTemplate
 ```

 %% Cell type:markdown id:73df46d9 tags:

 Next, initialize the langchain `CallBackManager`. This handles callbacks from Langchain and for this example we will use token-wise streaming so the answer gets generated token by token when Llama is answering your question.

 %% Cell type:code id:01fe5b9c tags:

 ``` python
 # for token-wise streaming so you'll see the answer gets generated token by token when Llama is answering your question
 callback_manager = CallbackManager([StreamingStdOutCallbackHandler()])
 ```

 %% Cell type:markdown id:8536c352 tags:


 Set up the Llama 2 model.

 Replace `<path-to-llama-gguf-file>` with the path either to your downloaded quantized model file [here](https://drive.google.com/file/d/1afPv3HOy73BE2MoYCgYJvBDeQNa9rZbj/view?usp=sharing),

 or to the `ggml-model-q4_0.gguf` file built with the following commands:

 ```bash
 git clone https://github.com/ggerganov/llama.cpp
 cd llama.cpp
 python3 -m pip install -r requirements.txt
 python convert.py <path_to_your_downloaded_llama-2-13b_model>
 ./quantize <path_to_your_downloaded_llama-2-13b_model>/ggml-model-f16.gguf <path_to_your_downloaded_llama-2-13b_model>/ggml-model-q4_0.gguf q4_0

 ```
 For more info see https://python.langchain.com/docs/integrations/llms/llamacpp

 %% Cell type:code id:dff6aa6b tags:

 ``` python

 llm = LlamaCpp(
    model_path="<path-to-llama-gguf-file>"
    temperature=0.0,
    top_p=1,
    n_ctx=6000,
    callback_manager=callback_manager,
    verbose=True,
 )
 ```

 %% Cell type:markdown id:f2cae215 tags:

 With the model set up, you are now ready to ask some questions.

 Here is an example of the simplest way to ask the model some general questions.

 %% Cell type:code id:0e78549c-9c93-4bc2-b525-38d578a94fae tags:

 ``` python

 question = "who wrote the book Innovator's dilemma?"
 answer = llm(question)
 ```

 %% Cell type:markdown id:545cb6aa tags:

 Alternatively, you can sue LangChain's `PromptTemplate` for some flexibility in your prompts and questions.

 For more information on LangChain's prompt template visit this [link](https://python.langchain.com/docs/modules/model_io/prompts/prompt_templates/)

 %% Cell type:code id:f7305c5b-6f55-4664-9206-2d7467653498 tags:

 ``` python
 # a more flexible way to ask Llama general questions using LangChain's PromptTemplate and LLMChain
 prompt = PromptTemplate.from_template(
    "who wrote {book}?"
 )
 chain = LLMChain(llm=llm, prompt=prompt)
 answer = chain.run("innovator's dilemma")
 ```

 %% Cell type:markdown id:189de613 tags:

 Now, let's see how Llama2 hallucinates, because it did not have knowledge about Llama2 at the time it was trained.
 By default it behaves like a know-it-all expert who will not say "I don't know".

 %% Cell type:code id:8ba66a29-77e9-4149-9523-63a09545584e tags:

 ``` python

 prompt = PromptTemplate.from_template(
    "What is {what}?"
 )
 chain = LLMChain(llm=llm, prompt=prompt)
 answer = chain.run("llama2")
 ```

 %% Cell type:markdown id:37f77909 tags:

 One way we can fix the hallucinations is to use RAG, to augment it with more recent or custom data that holds the information for it to answer correctly.

 First we load the Llama2 paper using LangChain's [PDF loader](https://python.langchain.com/docs/modules/data_connection/document_loaders/pdf)

 %% Cell type:code id:f3ebc261 tags:

 ``` python

 from langchain.document_loaders import PyPDFLoader
 loader = PyPDFLoader("llama2.pdf")
 documents = loader.load()
 ```

 %% Cell type:code id:302eaa54 tags:

 ``` python
 # quick check on the loaded document for the correct pages etc
 print(len(documents), documents[0].page_content[0:300])
 ```

 %% Cell type:markdown id:8c4ede5b tags:

 Next we will store our documents.
 There are more than 30 vector stores (DBs) supported by LangChain.
 For this example we will use [Chroma](https://python.langchain.com/docs/integrations/vectorstores/chroma) which is light-weight and in memory so it's easy to get started with.
 For other vector stores especially if you need to store a large amount of data - see https://python.langchain.com/docs/integrations/vectorstores

 We will also import the `HuggingFaceEmbeddings` and `RecursiveCharacterTextSplitter` to assist in storing the documents.

 %% Cell type:code id:4f94f6f8 tags:

 ``` python

 from langchain.vectorstores import Chroma

 # embeddings are numerical representations of the question and answer text
 from langchain.embeddings import HuggingFaceEmbeddings

 # use a common text splitter to split text into chunks
 from langchain.text_splitter import RecursiveCharacterTextSplitter
 ```

 %% Cell type:markdown id:0bfdacf7 tags:


 To store the documents, we will need to split them into chunks using [`RecursiveCharacterTextSplitter`](https://python.langchain.com/docs/modules/data_connection/document_transformers/text_splitters/recursive_text_splitter) and create vector representations of these chunks using [`HuggingFaceEmbeddings`](https://www.google.com/search?q=langchain+hugging+face+embeddings&sca_esv=572890011&ei=ARUoZaH4LuumptQP48ah2Ac&oq=langchian+hugg&gs_lp=Egxnd3Mtd2l6LXNlcnAiDmxhbmdjaGlhbiBodWdnKgIIADIHEAAYgAQYCjIHEAAYgAQYCjIHEAAYgAQYCjIHEAAYgAQYCjIHEAAYgAQYCjIHEAAYgAQYCjIHEAAYgAQYCjIHEAAYgAQYCjIHEAAYgAQYCjIHEAAYgAQYCkjeHlC5Cli5D3ABeAGQAQCYAV6gAb4CqgEBNLgBAcgBAPgBAcICChAAGEcY1gQYsAPiAwQYACBBiAYBkAYI&sclient=gws-wiz-serp) on them before storing them into our vector database.

 %% Cell type:code id:2b101485 tags:

 ``` python
 # split the loaded documents into chunks
 text_splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=20)
 all_splits = text_splitter.split_documents(documents)

 # create the vector db to store all the split chunks as embeddings
 embeddings = HuggingFaceEmbeddings()
 vectordb = Chroma.from_documents(
    documents=all_splits,
    embedding=embeddings,
 )
 ```

 %% Cell type:markdown id:bddc38e8 tags:


 We then use ` RetrievalQA` to retrieve the documents from the vector database and give the model more context on Llama 2, thereby increasing its knowledge.

 %% Cell type:code id:1a2472c9 tags:

 ``` python
 # use another LangChain's chain, RetrievalQA, to associate Llama with the loaded documents stored in the vector db
 from langchain.chains import RetrievalQA

 qa_chain = RetrievalQA.from_chain_type(
    llm,
    retriever=vectordb.as_retriever()
 )

 ```

 %% Cell type:markdown id:db71e5d7 tags:

 For each question, LangChain performs a semantic similarity search of it in the vector db, then passes the search results as the context to the model to answer the question.

 It takes close to 2 minutes to return the result (but using other vector stores other than Chroma such as FAISS can take longer) because Llama2 is running on a local Mac.
 To get much faster results, you can use a cloud service with GPU used for inference - see HelloLlamaCloud for a demo.

 %% Cell type:code id:dd2e62a4-6ea2-4ea7-b7ae-800185177e6c tags:

 ``` python

 question = "What is llama2?"
 result = qa_chain({"query": question})
 ```