"The book \"The Innovator's Dilemma\" was written by Clayton Christensen, a professor at Harvard Business School. It was first published in 1997 and has since become a widely influential book on business strategy and innovation."
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\n",
"llama_print_timings: load time = 1202.24 ms\n",
"llama_print_timings: sample time = 46.44 ms / 58 runs ( 0.80 ms per token, 1249.03 tokens per second)\n",
"llama_print_timings: prompt eval time = 1815.15 ms / 15 tokens ( 121.01 ms per token, 8.26 tokens per second)\n",
"llama_print_timings: eval time = 5582.64 ms / 57 runs ( 97.94 ms per token, 10.21 tokens per second)\n",
"llama_print_timings: total time = 7545.78 ms\n"
]
}
],
"outputs": [],
"source": [
"\n",
"question = \"who wrote the book Innovator's dilemma?\"\n",
...
...
@@ -263,39 +144,10 @@
},
{
"cell_type": "code",
"execution_count": 6,
"execution_count": null,
"id": "f7305c5b-6f55-4664-9206-2d7467653498",
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"Llama.generate: prefix-match hit\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n",
"\n",
"Clayton Christensen is the author of \"The Innovator's Dilemma,\" which was first published in 1997. The book explores why successful companies often struggle to adapt to disruptive technologies and business models that ultimately lead to their downfall. Christensen argues that these companies are faced with a dilemma because they have invested so heavily in their existing products and processes that it is difficult for them to pivot and embrace new, disruptive technologies. He also introduces the concept of \"disruptive innovation,\" which he defines as a process by which a small company with limited resources is able to successfully challenge established industry leaders."
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\n",
"llama_print_timings: load time = 1202.24 ms\n",
"llama_print_timings: sample time = 116.69 ms / 147 runs ( 0.79 ms per token, 1259.79 tokens per second)\n",
"llama_print_timings: prompt eval time = 1180.31 ms / 8 tokens ( 147.54 ms per token, 6.78 tokens per second)\n",
"llama_print_timings: eval time = 13192.98 ms / 147 runs ( 89.75 ms per token, 11.14 tokens per second)\n",
"llama_print_timings: total time = 14746.13 ms\n"
]
}
],
"outputs": [],
"source": [
"# a more flexible way to ask Llama general questions using LangChain's PromptTemplate and LLMChain\n",
"prompt = PromptTemplate.from_template(\n",
...
...
@@ -316,49 +168,10 @@
},
{
"cell_type": "code",
"execution_count": 7,
"execution_count": null,
"id": "8ba66a29-77e9-4149-9523-63a09545584e",
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"Llama.generate: prefix-match hit\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n",
"\n",
"Llama2 is a free, open-source tool for generating high-quality, randomized test data for software applications. It is designed to be easy to use and to produce realistic, diverse test data that can help you identify and fix bugs in your application before it is released.\n",
"\n",
"Llama2 is the successor to the popular Llama tool, and it includes many new features and improvements over its predecessor. Some of the key features of Llama2 include:\n",
"\n",
"* Support for a wide range of data types, including strings, numbers, dates, and more\n",
"* The ability to generate random data based on user-defined rules and constraints\n",
"* A powerful and flexible API that allows you to customize and extend the tool to meet your specific needs\n",
"* Integration with popular testing frameworks and tools, such as JUnit and TestNG\n",
"* Support for a variety of programming languages, including Java, Python, C#, and more.\n",
"\n",
"Overall, Llama2 is a powerful and flexible tool that can help you improve the quality and reliability of your software applications by generating realistic and diverse test data."
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\n",
"llama_print_timings: load time = 1202.24 ms\n",
"llama_print_timings: sample time = 191.25 ms / 240 runs ( 0.80 ms per token, 1254.87 tokens per second)\n",
"llama_print_timings: prompt eval time = 480.79 ms / 6 tokens ( 80.13 ms per token, 12.48 tokens per second)\n",
"llama_print_timings: eval time = 22013.19 ms / 239 runs ( 92.11 ms per token, 10.86 tokens per second)\n",
"llama_print_timings: total time = 23111.55 ms\n"
]
}
],
"outputs": [],
"source": [
"\n",
"prompt = PromptTemplate.from_template(\n",
...
...
@@ -380,7 +193,7 @@
},
{
"cell_type": "code",
"execution_count": 8,
"execution_count": null,
"id": "f3ebc261",
"metadata": {},
"outputs": [],
...
...
@@ -393,22 +206,10 @@
},
{
"cell_type": "code",
"execution_count": 9,
"execution_count": null,
"id": "302eaa54",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"77 Llama 2 : Open Foundation and Fine-Tuned Chat Models\n",
"Hugo Touvron∗Louis Martin†Kevin Stone†\n",
"Peter Albert Amjad Almahairi Yasmine Babaei Nikolay Bashlykov Soumya Batra\n",
"Prajjwal Bhargava Shruti Bhosale Dan Bikel Lukas Blecher Cristian Canton Ferrer Moya Chen\n",
"Guillem Cucurull David Esiobu Jude Fernande\n"
]
}
],
"outputs": [],
"source": [
"# quick check on the loaded document for the correct pages etc\n",
" Llama 2 is a new language model developed by Meta AI that has been released openly to encourage responsible AI innovation. It is a fine-tuned version of the original Llama model and is optimized for dialogue use cases. The model has not covered all scenarios and may produce inaccurate or objectionable responses, so developers should perform safety testing and tuning before deploying any applications of Llama 2."
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\n",
"llama_print_timings: load time = 1202.24 ms\n",
"llama_print_timings: sample time = 76.83 ms / 97 runs ( 0.79 ms per token, 1262.48 tokens per second)\n",
"llama_print_timings: prompt eval time = 97067.98 ms / 1146 tokens ( 84.70 ms per token, 11.81 tokens per second)\n",
"llama_print_timings: eval time = 10431.81 ms / 96 runs ( 108.66 ms per token, 9.20 tokens per second)\n",
"llama_print_timings: total time = 107897.31 ms\n"
]
}
],
"outputs": [],
"source": [
"# use another LangChain's chain, RetrievalQA, to associate Llama with the loaded documents stored in the vector db\n",
* how to run Llama2 locally on a Mac using llama-cpp-python and the llama-cpp's quantized Llama2 model;
* how to use LangChain to ask Llama general questions;
* how to use LangChain to load a recent PDF doc - the Llama2 paper pdf - and ask questions about it. This is the well known RAG (Retrieval Augmented Generation) method to let LLM such as Llama2 be able to answer questions about the data not publicly available when Llama2 was trained, or about your own data. RAG is one way to prevent LLM's hallucination.
%% Cell type:markdown id:22450267 tags:
We start by installing necessary requirements and import packages we will be using in this example.
-[llama-cpp-python](https://github.com/abetlen/llama-cpp-python) a simple Python bindings for [llama.cpp](https://github.com/ggerganov/llama.cpp) library
- pypdf gives us the ability to work with pdfs
- sentence-transformers for text embeddings
- chromadb gives us database capabilities
- langchain provides necessary RAG tools for this demo
Next, initialize the langchain `CallBackManager`. This handles callbacks from Langchain and for this example we will use token-wise streaming so the answer gets generated token by token when Llama is answering your question.
%% Cell type:code id:01fe5b9c tags:
``` python
# for token-wise streaming so you'll see the answer gets generated token by token when Llama is answering your question
Replace `<path-to-llama-gguf-file>` with the path either to your downloaded quantized model file [here](https://drive.google.com/file/d/1afPv3HOy73BE2MoYCgYJvBDeQNa9rZbj/view?usp=sharing),
or to the `ggml-model-q4_0.gguf` file built with the following commands:
question="who wrote the book Innovator's dilemma?"
answer=llm(question)
```
%% Output
The book "The Innovator's Dilemma" was written by Clayton Christensen, a professor at Harvard Business School. It was first published in 1997 and has since become a widely influential book on business strategy and innovation.
llama_print_timings: load time = 1202.24 ms
llama_print_timings: sample time = 46.44 ms / 58 runs ( 0.80 ms per token, 1249.03 tokens per second)
llama_print_timings: prompt eval time = 1815.15 ms / 15 tokens ( 121.01 ms per token, 8.26 tokens per second)
llama_print_timings: eval time = 5582.64 ms / 57 runs ( 97.94 ms per token, 10.21 tokens per second)
llama_print_timings: total time = 7545.78 ms
%% Cell type:markdown id:545cb6aa tags:
Alternatively, you can sue LangChain's `PromptTemplate` for some flexibility in your prompts and questions.
For more information on LangChain's prompt template visit this [link](https://python.langchain.com/docs/modules/model_io/prompts/prompt_templates/)
# a more flexible way to ask Llama general questions using LangChain's PromptTemplate and LLMChain
prompt=PromptTemplate.from_template(
"who wrote {book}?"
)
chain=LLMChain(llm=llm,prompt=prompt)
answer=chain.run("innovator's dilemma")
```
%% Output
Llama.generate: prefix-match hit
Clayton Christensen is the author of "The Innovator's Dilemma," which was first published in 1997. The book explores why successful companies often struggle to adapt to disruptive technologies and business models that ultimately lead to their downfall. Christensen argues that these companies are faced with a dilemma because they have invested so heavily in their existing products and processes that it is difficult for them to pivot and embrace new, disruptive technologies. He also introduces the concept of "disruptive innovation," which he defines as a process by which a small company with limited resources is able to successfully challenge established industry leaders.
llama_print_timings: load time = 1202.24 ms
llama_print_timings: sample time = 116.69 ms / 147 runs ( 0.79 ms per token, 1259.79 tokens per second)
llama_print_timings: prompt eval time = 1180.31 ms / 8 tokens ( 147.54 ms per token, 6.78 tokens per second)
llama_print_timings: eval time = 13192.98 ms / 147 runs ( 89.75 ms per token, 11.14 tokens per second)
llama_print_timings: total time = 14746.13 ms
%% Cell type:markdown id:189de613 tags:
Now, let's see how Llama2 hallucinates, because it did not have knowledge about Llama2 at the time it was trained.
By default it behaves like a know-it-all expert who will not say "I don't know".
Llama2 is a free, open-source tool for generating high-quality, randomized test data for software applications. It is designed to be easy to use and to produce realistic, diverse test data that can help you identify and fix bugs in your application before it is released.
Llama2 is the successor to the popular Llama tool, and it includes many new features and improvements over its predecessor. Some of the key features of Llama2 include:
* Support for a wide range of data types, including strings, numbers, dates, and more
* The ability to generate random data based on user-defined rules and constraints
* A powerful and flexible API that allows you to customize and extend the tool to meet your specific needs
* Integration with popular testing frameworks and tools, such as JUnit and TestNG
* Support for a variety of programming languages, including Java, Python, C#, and more.
Overall, Llama2 is a powerful and flexible tool that can help you improve the quality and reliability of your software applications by generating realistic and diverse test data.
llama_print_timings: load time = 1202.24 ms
llama_print_timings: sample time = 191.25 ms / 240 runs ( 0.80 ms per token, 1254.87 tokens per second)
llama_print_timings: prompt eval time = 480.79 ms / 6 tokens ( 80.13 ms per token, 12.48 tokens per second)
llama_print_timings: eval time = 22013.19 ms / 239 runs ( 92.11 ms per token, 10.86 tokens per second)
llama_print_timings: total time = 23111.55 ms
%% Cell type:markdown id:37f77909 tags:
One way we can fix the hallucinations is to use RAG, to augment it with more recent or custom data that holds the information for it to answer correctly.
First we load the Llama2 paper using LangChain's [PDF loader](https://python.langchain.com/docs/modules/data_connection/document_loaders/pdf)
%% Cell type:code id:f3ebc261 tags:
``` python
fromlangchain.document_loadersimportPyPDFLoader
loader=PyPDFLoader("llama2.pdf")
documents=loader.load()
```
%% Cell type:code id:302eaa54 tags:
``` python
# quick check on the loaded document for the correct pages etc
77 Llama 2 : Open Foundation and Fine-Tuned Chat Models
Hugo Touvron∗Louis Martin†Kevin Stone†
Peter Albert Amjad Almahairi Yasmine Babaei Nikolay Bashlykov Soumya Batra
Prajjwal Bhargava Shruti Bhosale Dan Bikel Lukas Blecher Cristian Canton Ferrer Moya Chen
Guillem Cucurull David Esiobu Jude Fernande
%% Cell type:markdown id:8c4ede5b tags:
Next we will store our documents.
There are more than 30 vector stores (DBs) supported by LangChain.
For this example we will use [Chroma](https://python.langchain.com/docs/integrations/vectorstores/chroma) which is light-weight and in memory so it's easy to get started with.
For other vector stores especially if you need to store a large amount of data - see https://python.langchain.com/docs/integrations/vectorstores
We will also import the `HuggingFaceEmbeddings` and `RecursiveCharacterTextSplitter` to assist in storing the documents.
%% Cell type:code id:4f94f6f8 tags:
``` python
fromlangchain.vectorstoresimportChroma
# embeddings are numerical representations of the question and answer text
To store the documents, we will need to split them into chunks using [`RecursiveCharacterTextSplitter`](https://python.langchain.com/docs/modules/data_connection/document_transformers/text_splitters/recursive_text_splitter) and create vector representations of these chunks using [`HuggingFaceEmbeddings`](https://www.google.com/search?q=langchain+hugging+face+embeddings&sca_esv=572890011&ei=ARUoZaH4LuumptQP48ah2Ac&oq=langchian+hugg&gs_lp=Egxnd3Mtd2l6LXNlcnAiDmxhbmdjaGlhbiBodWdnKgIIADIHEAAYgAQYCjIHEAAYgAQYCjIHEAAYgAQYCjIHEAAYgAQYCjIHEAAYgAQYCjIHEAAYgAQYCjIHEAAYgAQYCjIHEAAYgAQYCjIHEAAYgAQYCjIHEAAYgAQYCkjeHlC5Cli5D3ABeAGQAQCYAV6gAb4CqgEBNLgBAcgBAPgBAcICChAAGEcY1gQYsAPiAwQYACBBiAYBkAYI&sclient=gws-wiz-serp) on them before storing them into our vector database.
# create the vector db to store all the split chunks as embeddings
embeddings=HuggingFaceEmbeddings()
vectordb=Chroma.from_documents(
documents=all_splits,
embedding=embeddings,
)
```
%% Cell type:markdown id:bddc38e8 tags:
We then use ` RetrievalQA` to retrieve the documents from the vector database and give the model more context on Llama 2, thereby increasing its knowledge.
%% Cell type:code id:1a2472c9 tags:
``` python
# use another LangChain's chain, RetrievalQA, to associate Llama with the loaded documents stored in the vector db
fromlangchain.chainsimportRetrievalQA
qa_chain=RetrievalQA.from_chain_type(
llm,
retriever=vectordb.as_retriever()
)
```
%% Output
Llama.generate: prefix-match hit
Llama 2 is a new language model developed by Meta AI that has been released openly to encourage responsible AI innovation. It is a fine-tuned version of the original Llama model and is optimized for dialogue use cases. The model has not covered all scenarios and may produce inaccurate or objectionable responses, so developers should perform safety testing and tuning before deploying any applications of Llama 2.
llama_print_timings: load time = 1202.24 ms
llama_print_timings: sample time = 76.83 ms / 97 runs ( 0.79 ms per token, 1262.48 tokens per second)
llama_print_timings: prompt eval time = 97067.98 ms / 1146 tokens ( 84.70 ms per token, 11.81 tokens per second)
llama_print_timings: eval time = 10431.81 ms / 96 runs ( 108.66 ms per token, 9.20 tokens per second)
llama_print_timings: total time = 107897.31 ms
%% Cell type:markdown id:db71e5d7 tags:
For each question, LangChain performs a semantic similarity search of it in the vector db, then passes the search results as the context to the model to answer the question.
It takes close to 2 minutes to return the result (but using other vector stores other than Chroma such as FAISS can take longer) because Llama2 is running on a local Mac.
To get much faster results, you can use a cloud service with GPU used for inference - see HelloLlamaCloud for a demo.
