diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
index 024b24be1818f09f3b9cd631d9a2a851df2b99ae..7905856b96d0b2a6207050e58d13cf2c9ab600fb 100644
--- a/CONTRIBUTING.md
+++ b/CONTRIBUTING.md
@@ -1,5 +1,8 @@
# 💡 Contributing to GPT Index
+> ⚠️ **NOTE**: We are rebranding GPT Index as LlamaIndex!
+> **2/19/2023**: We are still in the middle of the transition. If you are interested in contributing to LlamaIndex, make sure to follow the below steps. For testing, please do `import gpt_index` instead of `import llama_index`.
+
Interested in contributing to GPT Index? Here's how to get started!
## Contributions that we're looking for:
diff --git a/README.md b/README.md
index 09e9a0bf34a8c350bb3c9fcb10a8bb9f2ca3cede..35e5d5fdf9b9ccbb86e1be344052142e13c19181 100644
--- a/README.md
+++ b/README.md
@@ -2,14 +2,15 @@
> ⚠️ **NOTE**: We are rebranding GPT Index as LlamaIndex! We will carry out this transition gradually.
-> **2/27/2022**: We have a duplicate `llama-index` pip package. Simply replace all imports of `gpt_index` with `llama_index` if you choose to `pip install llama-index`.
+> **2/19/2023**: By default, our docs/notebooks/instructions now use the `llama-index` package. However the `gpt-index` package still exists as a duplicate!
+> **2/16/2023**: We have a duplicate `llama-index` pip package. Simply replace all imports of `gpt_index` with `llama_index` if you choose to `pip install llama-index`.
GPT Index (LlamaIndex) is a project consisting of a set of data structures designed to make it easier to
use large external knowledge bases with LLMs.
PyPi:
-- GPT Index: https://pypi.org/project/gpt-index/.
-- LlamaIndex (duplicate): https://pypi.org/project/llama-index/.
+- LlamaIndex: https://pypi.org/project/llama-index/.
+- GPT Index (duplicate): https://pypi.org/project/gpt-index/.
Documentation: https://gpt-index.readthedocs.io/en/latest/.
@@ -60,7 +61,7 @@ Please check it out for the most up-to-date tutorials, how-to guides, references
## 💻 Example Usage
```
-pip install gpt-index
+pip install llama-index
```
Examples are in the `examples` folder. Indices are in the `indices` folder (see list of indices below).
@@ -70,7 +71,7 @@ To build a simple vector store index:
import os
os.environ["OPENAI_API_KEY"] = 'YOUR_OPENAI_API_KEY'
-from gpt_index import GPTSimpleVectorIndex, SimpleDirectoryReader
+from llama_index import GPTSimpleVectorIndex, SimpleDirectoryReader
documents = SimpleDirectoryReader('data').load_data()
index = GPTSimpleVectorIndex(documents)
```
diff --git a/docs/getting_started/installation.md b/docs/getting_started/installation.md
index 446bf1e20cd85c84b324e7d926c58afacdfc1b8b..509512c16025c01879c9b691c3dffa487ce0249b 100644
--- a/docs/getting_started/installation.md
+++ b/docs/getting_started/installation.md
@@ -4,7 +4,7 @@
You can simply do:
```
-pip install gpt-index
+pip install llama-index
```
### Installation from Source
diff --git a/docs/getting_started/starter_example.md b/docs/getting_started/starter_example.md
index 105cd9d89f7c9a680a49dbd7402d40469531d92e..891f191c1f175c3e3971e51fc68625fef69fe621 100644
--- a/docs/getting_started/starter_example.md
+++ b/docs/getting_started/starter_example.md
@@ -35,7 +35,7 @@ This contains GPT Index examples around Paul Graham's essay, ["What I Worked On"
Create a new `.py` file with the following:
```python
-from gpt_index import GPTSimpleVectorIndex, SimpleDirectoryReader
+from llama_index import GPTSimpleVectorIndex, SimpleDirectoryReader
documents = SimpleDirectoryReader('data').load_data()
index = GPTSimpleVectorIndex(documents)
diff --git a/docs/guides/usage_pattern.md b/docs/guides/usage_pattern.md
index e4cb08eea2f092a3112e4743a088356ddd8b9bd8..84c5c35dc82b19c6425c9a481c193c31dde7acd9 100644
--- a/docs/guides/usage_pattern.md
+++ b/docs/guides/usage_pattern.md
@@ -13,7 +13,7 @@ We provide a variety of [data loaders](/how_to/data_connectors.md) which will lo
through the `load_data` function, e.g.:
```python
-from gpt_index import SimpleDirectoryReader
+from llama_index import SimpleDirectoryReader
documents = SimpleDirectoryReader('data').load_data()
@@ -22,7 +22,7 @@ documents = SimpleDirectoryReader('data').load_data()
You can also choose to construct documents manually. GPT Index exposes the `Document` struct.
```python
-from gpt_index import Document
+from llama_index import Document
text_list = [text1, text2, ...]
documents = [Document(t) for t in text_list]
@@ -33,7 +33,7 @@ documents = [Document(t) for t in text_list]
We can now build an index over these Document objects. The simplest is to load in the Document objects during index initialization.
```python
-from gpt_index import GPTSimpleVectorIndex
+from llama_index import GPTSimpleVectorIndex
index = GPTSimpleVectorIndex(documents)
@@ -47,7 +47,7 @@ You can also take advantage of the `insert` capability of indices to insert Docu
one at a time instead of during index construction.
```python
-from gpt_index import GPTSimpleVectorIndex
+from llama_index import GPTSimpleVectorIndex
index = GPTSimpleVectorIndex([])
for doc in documents:
@@ -63,7 +63,7 @@ By default, we use OpenAI's `text-davinci-003` model. You may choose to use anot
an index.
```python
-from gpt_index import LLMPredictor, GPTSimpleVectorIndex, PromptHelper
+from llama_index import LLMPredictor, GPTSimpleVectorIndex, PromptHelper
...
@@ -124,7 +124,7 @@ index = GPTSimpleVectorIndex.load_from_disk('index.json')
You can build indices on top of other indices!
```python
-from gpt_index import GPTSimpleVectorIndex, GPTListIndex
+from llama_index import GPTSimpleVectorIndex, GPTListIndex
index1 = GPTSimpleVectorIndex(documents1)
index2 = GPTSimpleVectorIndex(documents2)
diff --git a/docs/guides/use_cases.md b/docs/guides/use_cases.md
index 0be0c09d2ae51dc17da2b727bf26860ff7630088..7ed8bd6551f12a1b8192a45553b00ea193410b64 100644
--- a/docs/guides/use_cases.md
+++ b/docs/guides/use_cases.md
@@ -65,7 +65,7 @@ While a single vector store may implicitly do so (the top-k nearest neighbor tex
Assuming you've already defined "subindices" over each data source, you can define a higher-level list index on top of these subindices through [composability](/how_to/composability.md).
```python
-from gpt_index import GPTSimpleVectorIndex, GPTListIndex
+from llama_index import GPTSimpleVectorIndex, GPTListIndex
index1 = GPTSimpleVectorIndex(notion_docs)
index2 = GPTSimpleVectorIndex(slack_docs)
@@ -95,7 +95,7 @@ A `GPTKeywordTableIndex` uses keyword matching, and a `GPTVectorStoreIndex` uses
embedding cosine similarity.
```python
-from gpt_index import GPTTreeIndex, GPTSimpleVectorIndex
+from llama_index import GPTTreeIndex, GPTSimpleVectorIndex
...
@@ -137,7 +137,7 @@ You have a knowledge base that is organized in a hierarchy. For instance, you ma
You can do this by defining a subindex for each subsection, defining a *summary text* for that subindex, and [a higher order index](/how_to/composability.md) to combine the subindices. You can stack this as many times as you wish. By defining summary text for each subsection, the higher order index will *refine* the answer synthesized through a subindex with the summary.
```python
-from gpt_index import GPTTreeIndex, GPTSimpleVectorIndex
+from llama_index import GPTTreeIndex, GPTSimpleVectorIndex
index1 = GPTSimpleVectorIndex(chapter1)
diff --git a/docs/how_to/composability.md b/docs/how_to/composability.md
index c2c02b8c52e225da6918eb4674cf7a1699fcf97d..33ad4027aad43091beb956450d9ac0e56b2c3f71 100644
--- a/docs/how_to/composability.md
+++ b/docs/how_to/composability.md
@@ -69,7 +69,7 @@ This wrapper allows us to query, save, and load the graph to/from disk.
```python
-from gpt_index.composability import ComposableGraph
+from llama_index.composability import ComposableGraph
graph = ComposableGraph.build_from_index(list_index)
diff --git a/docs/how_to/cost_analysis.md b/docs/how_to/cost_analysis.md
index 47a31e1813fb5b66d4da2eb200cbd088bc2ff7cd..d2b829b1182fe3d74983a55fbbf5aefc4da96c43 100644
--- a/docs/how_to/cost_analysis.md
+++ b/docs/how_to/cost_analysis.md
@@ -52,7 +52,7 @@ any respective LLM calls are made.
To predict token usage of LLM calls, import and instantiate the MockLLMPredictor with the following:
```python
-from gpt_index import MockLLMPredictor
+from llama_index import MockLLMPredictor
llm_predictor = MockLLMPredictor(max_tokens=256)
```
@@ -61,7 +61,7 @@ You can then use this predictor during both index construction and querying. Exa
**Index Construction**
```python
-from gpt_index import GPTTreeIndex, MockLLMPredictor, SimpleDirectoryReader
+from llama_index import GPTTreeIndex, MockLLMPredictor, SimpleDirectoryReader
documents = SimpleDirectoryReader('../paul_graham_essay/data').load_data()
# the "mock" llm predictor is our token counter
@@ -88,7 +88,7 @@ You may also predict the token usage of embedding calls with `MockEmbedding`.
You can use it in tandem with `MockLLMPredictor`.
```python
-from gpt_index import (
+from llama_index import (
GPTSimpleVectorIndex,
MockLLMPredictor,
MockEmbedding,
diff --git a/docs/how_to/custom_llms.md b/docs/how_to/custom_llms.md
index 5f2c7f03de2dec4abfdf6083a44c7308860d5e04..ce871287dd8f1cef81054e710f3644455b22bd2d 100644
--- a/docs/how_to/custom_llms.md
+++ b/docs/how_to/custom_llms.md
@@ -34,7 +34,7 @@ you may plug in any LLM shown on Langchain's
```python
-from gpt_index import (
+from llama_index import (
GPTKeywordTableIndex,
SimpleDirectoryReader,
LLMPredictor,
@@ -66,7 +66,7 @@ For OpenAI, Cohere, AI21, you just need to set the `max_tokens` parameter
```python
-from gpt_index import (
+from llama_index import (
GPTKeywordTableIndex,
SimpleDirectoryReader,
LLMPredictor,
@@ -97,7 +97,7 @@ a custom PromptHelper class.
```python
-from gpt_index import (
+from llama_index import (
GPTKeywordTableIndex,
SimpleDirectoryReader,
LLMPredictor,
diff --git a/docs/how_to/custom_prompts.md b/docs/how_to/custom_prompts.md
index 29af0d1f3d88e3d88559e72248ca7cab1e8a908f..3ae8faeca3a8651218fc405e17598c66b2f922c5 100644
--- a/docs/how_to/custom_prompts.md
+++ b/docs/how_to/custom_prompts.md
@@ -15,7 +15,7 @@ contains optional prompts that the user may pass in.
### Example
-An example can be found in [this notebook](https://github.com/jerryjliu/gpt_index/blob/main/examples/paul_graham_essay/TestEssay.ipynb).
+An example can be found in [this notebook](https://github.com/jerryjliu/llama_index/blob/main/examples/paul_graham_essay/TestEssay.ipynb).
The corresponding snippet is below. We show how to define a custom Summarization Prompt that not only
contains a `text` field, but also `query_str` field during construction of `GPTTreeIndex`, so that
@@ -23,7 +23,7 @@ the answer to the query can be simply synthesized from the root nodes.
```python
-from gpt_index import SummaryPrompt, GPTTreeIndex, SimpleDirectoryReader
+from llama_index import SummaryPrompt, GPTTreeIndex, SimpleDirectoryReader
# load documents
documents = SimpleDirectoryReader('data').load_data()
diff --git a/docs/how_to/data_connectors.md b/docs/how_to/data_connectors.md
index e718a433eda8dc253b9ea431529538de351c32f2..ccf1d6d096a319a7823744e3d00959b99f915f57 100644
--- a/docs/how_to/data_connectors.md
+++ b/docs/how_to/data_connectors.md
@@ -20,7 +20,7 @@ downloads the loader file into a module that you can use within your application
Example usage:
```python
-from gpt_index import GPTSimpleVectorIndex, download_loader
+from llama_index import GPTSimpleVectorIndex, download_loader
GoogleDocsReader = download_loader('GoogleDocsReader')
diff --git a/docs/how_to/embeddings.md b/docs/how_to/embeddings.md
index 15e73c526b9282fca066fe7eed878f654d14f23b..13b5ef7961337bd2c115408ab511560bde2b59b5 100644
--- a/docs/how_to/embeddings.md
+++ b/docs/how_to/embeddings.md
@@ -65,9 +65,9 @@ We introduce a wrapper class,
An example snippet is shown below (to use Hugging Face embeddings) on the GPTListIndex:
```python
-from gpt_index import GPTListIndex, SimpleDirectoryReader
+from llama_index import GPTListIndex, SimpleDirectoryReader
from langchain.embeddings.huggingface import HuggingFaceEmbeddings
-from gpt_index import LangchainEmbedding
+from llama_index import LangchainEmbedding
# load in HF embedding model from langchain
embed_model = LangchainEmbedding(HuggingFaceEmbeddings())
@@ -88,9 +88,9 @@ print(response)
Another example snippet is shown for GPTSimpleVectorIndex.
```python
-from gpt_index import GPTSimpleVectorIndex, SimpleDirectoryReader
+from llama_index import GPTSimpleVectorIndex, SimpleDirectoryReader
from langchain.embeddings.huggingface import HuggingFaceEmbeddings
-from gpt_index import LangchainEmbedding
+from llama_index import LangchainEmbedding
# load in HF embedding model from langchain
embed_model = LangchainEmbedding(HuggingFaceEmbeddings())
diff --git a/docs/index.rst b/docs/index.rst
index 53be901cb7828d42f4f60606c61ec796d45afe27..2e92341d30f6f5782c4298bfab82ce7e5cb9c34d 100644
--- a/docs/index.rst
+++ b/docs/index.rst
@@ -11,10 +11,14 @@ use large external knowledge bases with LLMs.
⚠️ **NOTE**: We are rebranding GPT Index as LlamaIndex! We will carry out this transition gradually.
- **2/27/2022**: We have a duplicate `llama-index` pip package. Simply replace all imports of `gpt_index` with `llama_index` if you choose to `pip install llama-index`.
+ **2/19/2023**: By default, our docs/notebooks/instructions now use the `llama-index` package. However the `gpt-index` package still exists as a duplicate!
+
+ **2/16/2023**: We have a duplicate `llama-index` pip package. Simply replace all imports of `gpt_index` with `llama_index` if you choose to `pip install llama-index`.
- Github: https://github.com/jerryjliu/gpt_index
-- PyPi: https://pypi.org/project/gpt-index/
+- PyPi:
+ - LlamaIndex: https://pypi.org/project/llama-index/.
+ - GPT Index (duplicate): https://pypi.org/project/gpt-index/.
- Twitter: https://twitter.com/gpt_index
- Discord https://discord.gg/dGcwcsnxhU
diff --git a/examples/async/AsyncGPTTreeIndexDemo.ipynb b/examples/async/AsyncGPTTreeIndexDemo.ipynb
index 77c7fea1c3652e9e67a55dceaa9aff5400122a3a..f9f56a7ca26129ccbcdbe51c813412f8c899f79c 100644
--- a/examples/async/AsyncGPTTreeIndexDemo.ipynb
+++ b/examples/async/AsyncGPTTreeIndexDemo.ipynb
@@ -1,136 +1,136 @@
{
- "cells": [
- {
- "cell_type": "markdown",
- "id": "96b2b1e4",
- "metadata": {},
- "source": [
- "# Async GPTTreeIndex Demo"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "9331cfeb",
- "metadata": {},
- "outputs": [],
- "source": [
- "# NOTE: This is ONLY necessary in jupyter notebook.\n",
- "# Details: Jupyter runs an event-loop behind the scenes. \n",
- "# This results in nested event-loops when we start an event-loop to make async queries.\n",
- "# This is normally not allowed, we use nest_asyncio to allow it for convenience. \n",
- "import nest_asyncio\n",
- "nest_asyncio.apply()"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "8a1d2821",
- "metadata": {},
- "outputs": [],
- "source": [
- "import time\n",
- "from gpt_index import GPTTreeIndex, SimpleDirectoryReader"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "6948df36",
- "metadata": {},
- "outputs": [],
- "source": [
- "# load documents\n",
- "documents = SimpleDirectoryReader('../paul_graham_essay/data').load_data()"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "2d9115d1",
- "metadata": {},
- "source": [
- "#### By default, GPTTreeIndex makes blocking LLM calls"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "d9ef0fef",
- "metadata": {},
- "outputs": [],
- "source": [
- "start_time = time.perf_counter()\n",
- "index = GPTTreeIndex(documents)\n",
- "elapsed_time = time.perf_counter() - start_time"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "9392d573",
- "metadata": {},
- "source": [
- "It takes ~47s to finish building GPTTreeIndex from 5 text chunks."
- ]
- },
- {
- "cell_type": "markdown",
- "id": "474f82d1",
- "metadata": {},
- "source": [
- "#### Pass in `use_async=True` to enable asynchronous LLM calls"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "78a02987",
- "metadata": {
- "scrolled": true
- },
- "outputs": [],
- "source": [
- "start_time = time.perf_counter()\n",
- "index = GPTTreeIndex(documents, use_async=True)\n",
- "elapsed_time = time.perf_counter() - start_time"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "23469128",
- "metadata": {},
- "source": [
- "It takes ~12s to finish building the GPTTreeIndex from 5 text chunks."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "59c1c27e",
- "metadata": {},
- "outputs": [],
- "source": []
- }
- ],
- "metadata": {
- "kernelspec": {
- "display_name": "Python 3 (ipykernel)",
- "language": "python",
- "name": "python3"
- },
- "language_info": {
- "codemirror_mode": {
- "name": "ipython",
- "version": 3
- },
- "file_extension": ".py",
- "mimetype": "text/x-python",
- "name": "python",
- "nbconvert_exporter": "python",
- "pygments_lexer": "ipython3",
- "version": "3.10.9"
- }
- },
- "nbformat": 4,
- "nbformat_minor": 5
-}
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "id": "96b2b1e4",
+ "metadata": {},
+ "source": [
+ "# Async GPTTreeIndex Demo"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "9331cfeb",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# NOTE: This is ONLY necessary in jupyter notebook.\n",
+ "# Details: Jupyter runs an event-loop behind the scenes. \n",
+ "# This results in nested event-loops when we start an event-loop to make async queries.\n",
+ "# This is normally not allowed, we use nest_asyncio to allow it for convenience. \n",
+ "import nest_asyncio\n",
+ "nest_asyncio.apply()"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "8a1d2821",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import time\n",
+ "from llama_index import GPTTreeIndex, SimpleDirectoryReader"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "6948df36",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# load documents\n",
+ "documents = SimpleDirectoryReader('../paul_graham_essay/data').load_data()"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "2d9115d1",
+ "metadata": {},
+ "source": [
+ "#### By default, GPTTreeIndex makes blocking LLM calls"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "d9ef0fef",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "start_time = time.perf_counter()\n",
+ "index = GPTTreeIndex(documents)\n",
+ "elapsed_time = time.perf_counter() - start_time"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "9392d573",
+ "metadata": {},
+ "source": [
+ "It takes ~47s to finish building GPTTreeIndex from 5 text chunks."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "474f82d1",
+ "metadata": {},
+ "source": [
+ "#### Pass in `use_async=True` to enable asynchronous LLM calls"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "78a02987",
+ "metadata": {
+ "scrolled": true
+ },
+ "outputs": [],
+ "source": [
+ "start_time = time.perf_counter()\n",
+ "index = GPTTreeIndex(documents, use_async=True)\n",
+ "elapsed_time = time.perf_counter() - start_time"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "23469128",
+ "metadata": {},
+ "source": [
+ "It takes ~12s to finish building the GPTTreeIndex from 5 text chunks."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "59c1c27e",
+ "metadata": {},
+ "outputs": [],
+ "source": []
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3 (ipykernel)",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.10.9"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}
\ No newline at end of file
diff --git a/examples/async/AsyncLLMPredictorDemo.ipynb b/examples/async/AsyncLLMPredictorDemo.ipynb
index ca25420382af534c4dc065a5fcbd0ba6d88469fe..3c595cd59b3e05608923fdd8712b51efa5add7c0 100644
--- a/examples/async/AsyncLLMPredictorDemo.ipynb
+++ b/examples/async/AsyncLLMPredictorDemo.ipynb
@@ -1,129 +1,129 @@
{
- "cells": [
- {
- "cell_type": "markdown",
- "id": "43cea4f8",
- "metadata": {},
- "source": [
- "# Async LLMPredictor Demo"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "b6d46041",
- "metadata": {},
- "outputs": [],
- "source": [
- "from gpt_index.langchain_helpers.chain_wrapper import LLMPredictor\n",
- "from gpt_index.prompts.default_prompts import DEFAULT_SUMMARY_PROMPT\n",
- "import asyncio\n",
- "import time"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "19e0b5e0",
- "metadata": {},
- "outputs": [],
- "source": [
- "context_strs = [\n",
- " 'Paul Graham',\n",
- " 'Steve Jobs',\n",
- " 'Barack Obama',\n",
- "]\n",
- "\n",
- "prompt = DEFAULT_SUMMARY_PROMPT"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "884c9df3",
- "metadata": {},
- "outputs": [],
- "source": [
- "llm = LLMPredictor()"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "3c5b368b",
- "metadata": {},
- "source": [
- "#### By default, LLM calls are blocking (i.e. only one API request at a time)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "26ed839a",
- "metadata": {},
- "outputs": [],
- "source": [
- "start_time = time.perf_counter()\n",
- "outputs = [llm.predict(prompt, context_str=context_str) for context_str in context_strs]\n",
- "elapsed_time = time.perf_counter() - start_time"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "9bc08db7",
- "metadata": {},
- "source": [
- "It takes ~19s to finish all 3 calls."
- ]
- },
- {
- "cell_type": "markdown",
- "id": "dba14b97",
- "metadata": {},
- "source": [
- "#### We can enable asynchronous calls with the `apredict` function"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "fb007b19",
- "metadata": {},
- "outputs": [],
- "source": [
- "start_time = time.perf_counter()\n",
- "tasks = [llm.apredict(prompt, context_str=context_str) for context_str in context_strs]\n",
- "await asyncio.gather(*tasks)\n",
- "elapsed_time = time.perf_counter() - start_time"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "efb77aa3",
- "metadata": {},
- "source": [
- "It takes ~7s to finish all 3 calls."
- ]
- }
- ],
- "metadata": {
- "kernelspec": {
- "display_name": "Python 3 (ipykernel)",
- "language": "python",
- "name": "python3"
- },
- "language_info": {
- "codemirror_mode": {
- "name": "ipython",
- "version": 3
- },
- "file_extension": ".py",
- "mimetype": "text/x-python",
- "name": "python",
- "nbconvert_exporter": "python",
- "pygments_lexer": "ipython3",
- "version": "3.10.9"
- }
- },
- "nbformat": 4,
- "nbformat_minor": 5
-}
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "id": "43cea4f8",
+ "metadata": {},
+ "source": [
+ "# Async LLMPredictor Demo"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "b6d46041",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from llama_index.langchain_helpers.chain_wrapper import LLMPredictor\n",
+ "from llama_index.prompts.default_prompts import DEFAULT_SUMMARY_PROMPT\n",
+ "import asyncio\n",
+ "import time"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "19e0b5e0",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "context_strs = [\n",
+ " 'Paul Graham',\n",
+ " 'Steve Jobs',\n",
+ " 'Barack Obama',\n",
+ "]\n",
+ "\n",
+ "prompt = DEFAULT_SUMMARY_PROMPT"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "884c9df3",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "llm = LLMPredictor()"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "3c5b368b",
+ "metadata": {},
+ "source": [
+ "#### By default, LLM calls are blocking (i.e. only one API request at a time)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "26ed839a",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "start_time = time.perf_counter()\n",
+ "outputs = [llm.predict(prompt, context_str=context_str) for context_str in context_strs]\n",
+ "elapsed_time = time.perf_counter() - start_time"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "9bc08db7",
+ "metadata": {},
+ "source": [
+ "It takes ~19s to finish all 3 calls."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "dba14b97",
+ "metadata": {},
+ "source": [
+ "#### We can enable asynchronous calls with the `apredict` function"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "fb007b19",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "start_time = time.perf_counter()\n",
+ "tasks = [llm.apredict(prompt, context_str=context_str) for context_str in context_strs]\n",
+ "await asyncio.gather(*tasks)\n",
+ "elapsed_time = time.perf_counter() - start_time"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "efb77aa3",
+ "metadata": {},
+ "source": [
+ "It takes ~7s to finish all 3 calls."
+ ]
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3 (ipykernel)",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.10.9"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}
\ No newline at end of file
diff --git a/examples/async/AsyncTreeSummarizeQueryDemo.ipynb b/examples/async/AsyncTreeSummarizeQueryDemo.ipynb
index c86d6403e1c7c73820564d2beb2e5bf904106f8c..17fdddf54a1aeb5ce656cf1653916d073b1dc48a 100644
--- a/examples/async/AsyncTreeSummarizeQueryDemo.ipynb
+++ b/examples/async/AsyncTreeSummarizeQueryDemo.ipynb
@@ -1,148 +1,148 @@
{
- "cells": [
- {
- "cell_type": "markdown",
- "id": "96b2b1e4",
- "metadata": {},
- "source": [
- "# Async TreeSummarizeQuery Demo"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "9331cfeb",
- "metadata": {},
- "outputs": [],
- "source": [
- "# NOTE: This is ONLY necessary in jupyter notebook.\n",
- "# Details: Jupyter runs an event-loop behind the scenes. \n",
- "# This results in nested event-loops when we start an event-loop to make async queries.\n",
- "# This is normally not allowed, we use nest_asyncio to allow it for convenience. \n",
- "import nest_asyncio\n",
- "nest_asyncio.apply()"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "8a1d2821",
- "metadata": {},
- "outputs": [],
- "source": [
- "import time\n",
- "from gpt_index import GPTListIndex, SimpleDirectoryReader"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "4466dec2",
- "metadata": {},
- "outputs": [],
- "source": [
- "query_str = \"What is Paul Graham's biggest achievement?\""
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "6948df36",
- "metadata": {},
- "outputs": [],
- "source": [
- "# load documents\n",
- "documents = SimpleDirectoryReader('../paul_graham_essay/data').load_data()"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "466c3892",
- "metadata": {},
- "outputs": [],
- "source": [
- "index = GPTListIndex(documents)"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "2d9115d1",
- "metadata": {},
- "source": [
- "#### By default, generate a response through hierarchical tree summarization (i.e., `response_mode=tree_summarize`) makes blocking LLM calls"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "d9ef0fef",
- "metadata": {},
- "outputs": [],
- "source": [
- "start_time = time.perf_counter()\n",
- "index.query(query_str, response_mode='tree_summarize')\n",
- "elapsed_time = time.perf_counter() - start_time"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "9392d573",
- "metadata": {},
- "source": [
- "It takes ~26s to generate a response through hierarchical tree summarization (i.e., `response_mode=tree_summarize`)."
- ]
- },
- {
- "cell_type": "markdown",
- "id": "474f82d1",
- "metadata": {},
- "source": [
- "#### Pass in `use_async=True` to enable asynchronous LLM calls"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "78a02987",
- "metadata": {
- "scrolled": true
- },
- "outputs": [],
- "source": [
- "start_time = time.perf_counter()\n",
- "index.query(query_str, response_mode='tree_summarize', use_async=True)\n",
- "elapsed_time = time.perf_counter() - start_time"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "23469128",
- "metadata": {},
- "source": [
- "It takes ~8s to generate a response through hierarchical tree summarization (i.e., `response_mode=tree_summarize`)."
- ]
- }
- ],
- "metadata": {
- "kernelspec": {
- "display_name": "Python 3 (ipykernel)",
- "language": "python",
- "name": "python3"
- },
- "language_info": {
- "codemirror_mode": {
- "name": "ipython",
- "version": 3
- },
- "file_extension": ".py",
- "mimetype": "text/x-python",
- "name": "python",
- "nbconvert_exporter": "python",
- "pygments_lexer": "ipython3",
- "version": "3.10.9"
- }
- },
- "nbformat": 4,
- "nbformat_minor": 5
-}
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "id": "96b2b1e4",
+ "metadata": {},
+ "source": [
+ "# Async TreeSummarizeQuery Demo"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "9331cfeb",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# NOTE: This is ONLY necessary in jupyter notebook.\n",
+ "# Details: Jupyter runs an event-loop behind the scenes. \n",
+ "# This results in nested event-loops when we start an event-loop to make async queries.\n",
+ "# This is normally not allowed, we use nest_asyncio to allow it for convenience. \n",
+ "import nest_asyncio\n",
+ "nest_asyncio.apply()"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "8a1d2821",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import time\n",
+ "from llama_index import GPTListIndex, SimpleDirectoryReader"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "4466dec2",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "query_str = \"What is Paul Graham's biggest achievement?\""
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "6948df36",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# load documents\n",
+ "documents = SimpleDirectoryReader('../paul_graham_essay/data').load_data()"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "466c3892",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "index = GPTListIndex(documents)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "2d9115d1",
+ "metadata": {},
+ "source": [
+ "#### By default, generate a response through hierarchical tree summarization (i.e., `response_mode=tree_summarize`) makes blocking LLM calls"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "d9ef0fef",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "start_time = time.perf_counter()\n",
+ "index.query(query_str, response_mode='tree_summarize')\n",
+ "elapsed_time = time.perf_counter() - start_time"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "9392d573",
+ "metadata": {},
+ "source": [
+ "It takes ~26s to generate a response through hierarchical tree summarization (i.e., `response_mode=tree_summarize`)."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "474f82d1",
+ "metadata": {},
+ "source": [
+ "#### Pass in `use_async=True` to enable asynchronous LLM calls"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "78a02987",
+ "metadata": {
+ "scrolled": true
+ },
+ "outputs": [],
+ "source": [
+ "start_time = time.perf_counter()\n",
+ "index.query(query_str, response_mode='tree_summarize', use_async=True)\n",
+ "elapsed_time = time.perf_counter() - start_time"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "23469128",
+ "metadata": {},
+ "source": [
+ "It takes ~8s to generate a response through hierarchical tree summarization (i.e., `response_mode=tree_summarize`)."
+ ]
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3 (ipykernel)",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.10.9"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}
\ No newline at end of file
diff --git a/examples/azure_demo/AzureOpenAI.ipynb b/examples/azure_demo/AzureOpenAI.ipynb
index 923490b9ff03195d72a7f3e6f33a93645853a53a..4c7d198b989c3e730a81865b15599eece20611e7 100644
--- a/examples/azure_demo/AzureOpenAI.ipynb
+++ b/examples/azure_demo/AzureOpenAI.ipynb
@@ -1,253 +1,253 @@
{
- "cells": [
- {
- "cell_type": "markdown",
- "id": "ec51f656",
- "metadata": {},
- "source": [
- "# Using Azure OpenAI resource with GPT-Index"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "ef3d26db",
- "metadata": {},
- "source": [
- "Azure openAI resources unfortunately differ from standard openAI resources as you can't generate embeddings unless you use an embedding model. The regions where these models are available can be found here: https://learn.microsoft.com/en-us/azure/cognitive-services/openai/concepts/models#embeddings-models\n",
- "\n",
- "Furthermore the regions that support embedding models unfortunately don't support the latest versions (<*>-003) of openAI models, so we are forced to use one region for embeddings and another for the text generation."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 13,
- "id": "b05e71d5",
- "metadata": {},
- "outputs": [],
- "source": [
- "import os\n",
- "import json\n",
- "import openai\n",
- "from langchain.llms import AzureOpenAI\n",
- "from langchain.embeddings import OpenAIEmbeddings\n",
- "from gpt_index import LangchainEmbedding\n",
- "from gpt_index import (\n",
- " GPTSimpleVectorIndex,\n",
- " SimpleDirectoryReader, \n",
- " LLMPredictor,\n",
- " PromptHelper\n",
- ")"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "80a962f8",
- "metadata": {},
- "source": [
- "This is the API setup for the embedding model"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 10,
- "id": "fe013749",
- "metadata": {},
- "outputs": [],
- "source": [
- "openai.api_type = \"azure\"\n",
- "openai.api_base = \"<insert api base url from azure>\"\n",
- "openai.api_version = \"2022-12-01\"\n",
- "openai.api_key = \"<insert api key from azure>\""
- ]
- },
- {
- "cell_type": "markdown",
- "id": "fcaaafdf",
- "metadata": {},
- "source": [
- "And here you can see the setup for the text generation model, you should note we use deployment name instead of model name as an arguement. This name is the one chosen when deploying the model in Azure."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 6,
- "id": "e2569cb0",
- "metadata": {},
- "outputs": [],
- "source": [
- "llm = AzureOpenAI(deployment_name=\"<insert deployment name from azure>\", model_kwargs={\n",
- " \"api_key\": \"<insert api key from azure>\",\n",
- " \"api_base\": \"<insert api url from azure>\",\n",
- " \"api_type\": \"azure\",\n",
- " \"api_version\": \"2022-12-01\",\n",
- "})\n",
- "llm_predictor = LLMPredictor(llm=llm)\n",
- "\n",
- "embedding_llm = LangchainEmbedding(OpenAIEmbeddings(\n",
- " document_model_name=\"<insert deployment name from azure for embedding-doc model>\",\n",
- " query_model_name=\"<insert deployment name from azure for embedding-query model>\"\n",
- "))\n",
- "\n",
- "documents = SimpleDirectoryReader('../paul_graham_essay/data').load_data()"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 14,
- "id": "25d1c8be",
- "metadata": {},
- "outputs": [],
- "source": [
- "# max LLM token input size\n",
- "max_input_size = 500\n",
- "# set number of output tokens\n",
- "num_output = 48\n",
- "# set maximum chunk overlap\n",
- "max_chunk_overlap = 20\n",
- "\n",
- "prompt_helper = PromptHelper(max_input_size, num_output, max_chunk_overlap)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 15,
- "id": "1cf0e9c9",
- "metadata": {
- "scrolled": true
- },
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "> Adding chunk: \t\t\n",
- "\n",
- "What I Worked On\n",
- "\n",
- "February 2021\n",
- "\n",
- "Before col...\n",
- "> Adding chunk: interesting of that type. So I'm not surprised ...\n",
- "> Adding chunk: to be the study of the ultimate truths, compare...\n",
- "> Adding chunk: language called PL/I, and the situation was sim...\n",
- "> Adding chunk: or if there even was a specific moment, but dur...\n",
- "> Adding chunk: an uneasy alliance between two halves, theory a...\n",
- "> Adding chunk: were hundreds of years old.\n",
- "\n",
- "And moreover this ...\n",
- "> Adding chunk: that he'd found such a spectacular way to get o...\n",
- "> Adding chunk: the classes that everyone has to take in fundam...\n",
- "> Adding chunk: students wouldn't require the faculty to teach ...\n",
- "> Adding chunk: or you get merely photographic accuracy, and wh...\n",
- "> Adding chunk: But the Accademia wasn't teaching me anything e...\n",
- "> Adding chunk: In Florence, after paying my part of the rent, ...\n",
- "> Adding chunk: about a new thing called HTML, which was, as he...\n",
- "> Adding chunk: were plenty of earnest students too: kids who \"...\n",
- "> Adding chunk: Lisp hacking work was very rare, and I didn't w...\n",
- "> Adding chunk: had done for the popularity of microcomputers. ...\n",
- "> Adding chunk: shopping cart, and I wrote a new site generator...\n",
- "> Adding chunk: seed funding from Idelle's husband Julian. In r...\n",
- "> Adding chunk: for a month,\" he said, \"and it's still not done...\n",
- "> Adding chunk: fun to work on. If all I'd had to do was work o...\n",
- "> Adding chunk: the collar than a picture of the whole shirt. T...\n",
- "> Adding chunk: partly because that's what startups did during ...\n",
- "> Adding chunk: had given us a lot of options when they bought ...\n",
- "> Adding chunk: That's what I should have done, just gone off s...\n",
- "> Adding chunk: buy. Now I could actually choose what neighborh...\n",
- "> Adding chunk: trying to build what it's now clear is about tw...\n",
- "> Adding chunk: dream of building a new Lisp, partly because on...\n",
- "> Adding chunk: me several years to understand the implications...\n",
- "> Adding chunk: seems about as hip.\n",
- "\n",
- "It's not that unprestigiou...\n",
- "> Adding chunk: charge of marketing at a Boston investment bank...\n",
- "> Adding chunk: out \"But not me!\" and went on with the talk. Bu...\n",
- "> Adding chunk: And neither of them helped founders enough in t...\n",
- "> Adding chunk: fake investors, because they would in a similar...\n",
- "> Adding chunk: batch was so good. You had to be pretty bold to...\n",
- "> Adding chunk: had not originally intended YC to be a full-tim...\n",
- "> Adding chunk: internal software in Arc. But while I continued...\n",
- "> Adding chunk: double from a kidney stone, he suggested that i...\n",
- "> Adding chunk: we agreed to make it a complete changing of the...\n",
- "> Adding chunk: of 2014 painting. I'd never been able to work s...\n",
- "> Adding chunk: his grad student Steve Russell suggested it. Ru...\n",
- "> Adding chunk: defined goal, or it would have been hard to kee...\n",
- "> Adding chunk: pools. It felt like I was doing life right. I r...\n",
- "> Adding chunk: the more exciting.\n",
- "\n",
- "[2] Italian words for abstr...\n",
- "> Adding chunk: expensive.\n",
- "\n",
- "[7] Technically the apartment wasn'...\n",
- "> Adding chunk: online means you treat the online version as th...\n",
- "> Adding chunk: logo had been a white V on a red circle, so I m...\n",
- "> Adding chunk: YC was not working with Jessica anymore. We'd b...\n",
- "> [build_index_from_documents] Total LLM token usage: 0 tokens\n",
- "> [build_index_from_documents] Total embedding token usage: 17533 tokens\n"
- ]
- }
- ],
- "source": [
- "index = GPTSimpleVectorIndex(documents, embed_model=embedding_llm, llm_predictor=llm_predictor, prompt_helper=prompt_helper)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 17,
- "id": "98d9d3fd",
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "> [query] Total LLM token usage: 815 tokens\n",
- "> [query] Total embedding token usage: 8 tokens\n",
- "> Source (Doc id: ad03b507-8953-4201-b545-6195c5cfec49): me several years to understand the implications. It meant there would be a whole new generation o...\n",
- "query was: What is most interesting about this essay?\n",
- "answer was: \n",
- "\n",
- "The most interesting thing about this essay is the way the author reflects on the impact of online publishing on their life and career. They discuss how the opening up of the internet to allow for more diverse, and less prestigious, forms of writing allowed them to pursue the kind of writing they were interested in, which was something that had not been possible before. Furthermore, the author acknowledges that their work may not be seen as prestigious, such as Latin, but yet still has a great impact. They further reflect on how their life and career have been shaped by working on these types of projects.\n"
- ]
- }
- ],
- "source": [
- "query = 'What is most interesting about this essay?'\n",
- "answer = index.query(query)\n",
- "\n",
- "print(answer.get_formatted_sources())\n",
- "print('query was:', query)\n",
- "print('answer was:', answer)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "3e33f1eb",
- "metadata": {},
- "outputs": [],
- "source": []
- }
- ],
- "metadata": {
- "kernelspec": {
- "display_name": "Python 3",
- "language": "python",
- "name": "python3"
- },
- "language_info": {
- "codemirror_mode": {
- "name": "ipython",
- "version": 3
- },
- "file_extension": ".py",
- "mimetype": "text/x-python",
- "name": "python",
- "nbconvert_exporter": "python",
- "pygments_lexer": "ipython3",
- "version": "3.10.4"
- }
- },
- "nbformat": 4,
- "nbformat_minor": 5
-}
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "id": "ec51f656",
+ "metadata": {},
+ "source": [
+ "# Using Azure OpenAI resource with GPT-Index"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "ef3d26db",
+ "metadata": {},
+ "source": [
+ "Azure openAI resources unfortunately differ from standard openAI resources as you can't generate embeddings unless you use an embedding model. The regions where these models are available can be found here: https://learn.microsoft.com/en-us/azure/cognitive-services/openai/concepts/models#embeddings-models\n",
+ "\n",
+ "Furthermore the regions that support embedding models unfortunately don't support the latest versions (<*>-003) of openAI models, so we are forced to use one region for embeddings and another for the text generation."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 13,
+ "id": "b05e71d5",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import os\n",
+ "import json\n",
+ "import openai\n",
+ "from langchain.llms import AzureOpenAI\n",
+ "from langchain.embeddings import OpenAIEmbeddings\n",
+ "from llama_index import LangchainEmbedding\n",
+ "from llama_index import (\n",
+ " GPTSimpleVectorIndex,\n",
+ " SimpleDirectoryReader, \n",
+ " LLMPredictor,\n",
+ " PromptHelper\n",
+ ")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "80a962f8",
+ "metadata": {},
+ "source": [
+ "This is the API setup for the embedding model"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 10,
+ "id": "fe013749",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "openai.api_type = \"azure\"\n",
+ "openai.api_base = \"<insert api base url from azure>\"\n",
+ "openai.api_version = \"2022-12-01\"\n",
+ "openai.api_key = \"<insert api key from azure>\""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "fcaaafdf",
+ "metadata": {},
+ "source": [
+ "And here you can see the setup for the text generation model, you should note we use deployment name instead of model name as an arguement. This name is the one chosen when deploying the model in Azure."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 6,
+ "id": "e2569cb0",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "llm = AzureOpenAI(deployment_name=\"<insert deployment name from azure>\", model_kwargs={\n",
+ " \"api_key\": \"<insert api key from azure>\",\n",
+ " \"api_base\": \"<insert api url from azure>\",\n",
+ " \"api_type\": \"azure\",\n",
+ " \"api_version\": \"2022-12-01\",\n",
+ "})\n",
+ "llm_predictor = LLMPredictor(llm=llm)\n",
+ "\n",
+ "embedding_llm = LangchainEmbedding(OpenAIEmbeddings(\n",
+ " document_model_name=\"<insert deployment name from azure for embedding-doc model>\",\n",
+ " query_model_name=\"<insert deployment name from azure for embedding-query model>\"\n",
+ "))\n",
+ "\n",
+ "documents = SimpleDirectoryReader('../paul_graham_essay/data').load_data()"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 14,
+ "id": "25d1c8be",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# max LLM token input size\n",
+ "max_input_size = 500\n",
+ "# set number of output tokens\n",
+ "num_output = 48\n",
+ "# set maximum chunk overlap\n",
+ "max_chunk_overlap = 20\n",
+ "\n",
+ "prompt_helper = PromptHelper(max_input_size, num_output, max_chunk_overlap)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 15,
+ "id": "1cf0e9c9",
+ "metadata": {
+ "scrolled": true
+ },
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "> Adding chunk: \t\t\n",
+ "\n",
+ "What I Worked On\n",
+ "\n",
+ "February 2021\n",
+ "\n",
+ "Before col...\n",
+ "> Adding chunk: interesting of that type. So I'm not surprised ...\n",
+ "> Adding chunk: to be the study of the ultimate truths, compare...\n",
+ "> Adding chunk: language called PL/I, and the situation was sim...\n",
+ "> Adding chunk: or if there even was a specific moment, but dur...\n",
+ "> Adding chunk: an uneasy alliance between two halves, theory a...\n",
+ "> Adding chunk: were hundreds of years old.\n",
+ "\n",
+ "And moreover this ...\n",
+ "> Adding chunk: that he'd found such a spectacular way to get o...\n",
+ "> Adding chunk: the classes that everyone has to take in fundam...\n",
+ "> Adding chunk: students wouldn't require the faculty to teach ...\n",
+ "> Adding chunk: or you get merely photographic accuracy, and wh...\n",
+ "> Adding chunk: But the Accademia wasn't teaching me anything e...\n",
+ "> Adding chunk: In Florence, after paying my part of the rent, ...\n",
+ "> Adding chunk: about a new thing called HTML, which was, as he...\n",
+ "> Adding chunk: were plenty of earnest students too: kids who \"...\n",
+ "> Adding chunk: Lisp hacking work was very rare, and I didn't w...\n",
+ "> Adding chunk: had done for the popularity of microcomputers. ...\n",
+ "> Adding chunk: shopping cart, and I wrote a new site generator...\n",
+ "> Adding chunk: seed funding from Idelle's husband Julian. In r...\n",
+ "> Adding chunk: for a month,\" he said, \"and it's still not done...\n",
+ "> Adding chunk: fun to work on. If all I'd had to do was work o...\n",
+ "> Adding chunk: the collar than a picture of the whole shirt. T...\n",
+ "> Adding chunk: partly because that's what startups did during ...\n",
+ "> Adding chunk: had given us a lot of options when they bought ...\n",
+ "> Adding chunk: That's what I should have done, just gone off s...\n",
+ "> Adding chunk: buy. Now I could actually choose what neighborh...\n",
+ "> Adding chunk: trying to build what it's now clear is about tw...\n",
+ "> Adding chunk: dream of building a new Lisp, partly because on...\n",
+ "> Adding chunk: me several years to understand the implications...\n",
+ "> Adding chunk: seems about as hip.\n",
+ "\n",
+ "It's not that unprestigiou...\n",
+ "> Adding chunk: charge of marketing at a Boston investment bank...\n",
+ "> Adding chunk: out \"But not me!\" and went on with the talk. Bu...\n",
+ "> Adding chunk: And neither of them helped founders enough in t...\n",
+ "> Adding chunk: fake investors, because they would in a similar...\n",
+ "> Adding chunk: batch was so good. You had to be pretty bold to...\n",
+ "> Adding chunk: had not originally intended YC to be a full-tim...\n",
+ "> Adding chunk: internal software in Arc. But while I continued...\n",
+ "> Adding chunk: double from a kidney stone, he suggested that i...\n",
+ "> Adding chunk: we agreed to make it a complete changing of the...\n",
+ "> Adding chunk: of 2014 painting. I'd never been able to work s...\n",
+ "> Adding chunk: his grad student Steve Russell suggested it. Ru...\n",
+ "> Adding chunk: defined goal, or it would have been hard to kee...\n",
+ "> Adding chunk: pools. It felt like I was doing life right. I r...\n",
+ "> Adding chunk: the more exciting.\n",
+ "\n",
+ "[2] Italian words for abstr...\n",
+ "> Adding chunk: expensive.\n",
+ "\n",
+ "[7] Technically the apartment wasn'...\n",
+ "> Adding chunk: online means you treat the online version as th...\n",
+ "> Adding chunk: logo had been a white V on a red circle, so I m...\n",
+ "> Adding chunk: YC was not working with Jessica anymore. We'd b...\n",
+ "> [build_index_from_documents] Total LLM token usage: 0 tokens\n",
+ "> [build_index_from_documents] Total embedding token usage: 17533 tokens\n"
+ ]
+ }
+ ],
+ "source": [
+ "index = GPTSimpleVectorIndex(documents, embed_model=embedding_llm, llm_predictor=llm_predictor, prompt_helper=prompt_helper)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 17,
+ "id": "98d9d3fd",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "> [query] Total LLM token usage: 815 tokens\n",
+ "> [query] Total embedding token usage: 8 tokens\n",
+ "> Source (Doc id: ad03b507-8953-4201-b545-6195c5cfec49): me several years to understand the implications. It meant there would be a whole new generation o...\n",
+ "query was: What is most interesting about this essay?\n",
+ "answer was: \n",
+ "\n",
+ "The most interesting thing about this essay is the way the author reflects on the impact of online publishing on their life and career. They discuss how the opening up of the internet to allow for more diverse, and less prestigious, forms of writing allowed them to pursue the kind of writing they were interested in, which was something that had not been possible before. Furthermore, the author acknowledges that their work may not be seen as prestigious, such as Latin, but yet still has a great impact. They further reflect on how their life and career have been shaped by working on these types of projects.\n"
+ ]
+ }
+ ],
+ "source": [
+ "query = 'What is most interesting about this essay?'\n",
+ "answer = index.query(query)\n",
+ "\n",
+ "print(answer.get_formatted_sources())\n",
+ "print('query was:', query)\n",
+ "print('answer was:', answer)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "3e33f1eb",
+ "metadata": {},
+ "outputs": [],
+ "source": []
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.10.4"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}
\ No newline at end of file
diff --git a/examples/composable_indices/ComposableIndices.ipynb b/examples/composable_indices/ComposableIndices.ipynb
index ce2a30247667f62e09931e7fec802ef5b5a94d52..02a242bfb418ec6a8308659ac45566e30e8daa87 100644
--- a/examples/composable_indices/ComposableIndices.ipynb
+++ b/examples/composable_indices/ComposableIndices.ipynb
@@ -1,472 +1,472 @@
{
- "cells": [
- {
- "cell_type": "markdown",
- "id": "cfb64210-9c6b-47d7-81f4-67dbdab68e4c",
- "metadata": {
- "tags": []
- },
- "source": [
- "# Composable Indices Demo"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "fa0e62b6",
- "metadata": {},
- "outputs": [],
- "source": [
- "import logging\n",
- "import sys\n",
- "\n",
- "logging.basicConfig(stream=sys.stdout, level=logging.INFO)\n",
- "logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout))"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 1,
- "id": "e27b0473-4bda-47f0-b6ed-fd482eac1a13",
- "metadata": {},
- "outputs": [],
- "source": [
- "from gpt_index import (\n",
- " GPTSimpleVectorIndex, \n",
- " GPTSimpleKeywordTableIndex, \n",
- " GPTListIndex, \n",
- " SimpleDirectoryReader\n",
- ")"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "49e0d841-680f-4a0c-b455-788b54978ebf",
- "metadata": {},
- "source": [
- "#### Load Datasets\n",
- "\n",
- "Load both the NYC Wikipedia page as well as Paul Graham's \"What I Worked On\" essay"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 2,
- "id": "9ec16a8b-6aae-4bf7-9b83-b82087b4ea52",
- "metadata": {},
- "outputs": [],
- "source": [
- "# fetch \"New York City\" page from Wikipedia\n",
- "from pathlib import Path\n",
- "\n",
- "import requests\n",
- "response = requests.get(\n",
- " 'https://en.wikipedia.org/w/api.php',\n",
- " params={\n",
- " 'action': 'query',\n",
- " 'format': 'json',\n",
- " 'titles': 'New York City',\n",
- " 'prop': 'extracts',\n",
- " # 'exintro': True,\n",
- " 'explaintext': True,\n",
- " }\n",
- ").json()\n",
- "page = next(iter(response['query']['pages'].values()))\n",
- "nyc_text = page['extract']\n",
- "\n",
- "data_path = Path('data')\n",
- "if not data_path.exists():\n",
- " Path.mkdir(data_path)\n",
- "\n",
- "with open('../test_wiki/data/nyc_text.txt', 'w') as fp:\n",
- " fp.write(nyc_text)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 2,
- "id": "39c00aeb-adef-4ce3-8134-031de18e64ea",
- "metadata": {},
- "outputs": [],
- "source": [
- "# load NYC dataset\n",
- "nyc_documents = SimpleDirectoryReader('../test_wiki/data/').load_data()"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 3,
- "id": "ddff8f98-e002-40c5-93ac-93aa40dca5ca",
- "metadata": {},
- "outputs": [],
- "source": [
- "# load PG's essay\n",
- "essay_documents = SimpleDirectoryReader('../paul_graham_essay/data/').load_data()"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "f1782198-c0de-4679-8951-1297c21b8639",
- "metadata": {},
- "source": [
- "### Building the document indices\n",
- "Build a tree index for the NYC wiki page and PG essay"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "5431e83e-428b-4473-bad1-24b7a6c4db38",
- "metadata": {},
- "outputs": [],
- "source": [
- "# build NYC index\n",
- "nyc_index = GPTSimpleVectorIndex(nyc_documents)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 5,
- "id": "6ebb00ff-e9c0-4ec9-ac05-43d4ab7a0d0e",
- "metadata": {},
- "outputs": [],
- "source": [
- "nyc_index.save_to_disk('index_nyc.json')"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "8b5aad4a-49ef-4b24-962a-0793f4f09316",
- "metadata": {},
- "outputs": [],
- "source": [
- "# build essay index\n",
- "essay_index = GPTSimpleVectorIndex(essay_documents)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 7,
- "id": "de0d248b-bc87-4129-a12f-c96eda46f72e",
- "metadata": {},
- "outputs": [],
- "source": [
- "essay_index.save_to_disk('index_pg.json')"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "b8aaf556-df77-4fac-812b-0b6c6d1da0ef",
- "metadata": {},
- "source": [
- "### Loading the indices\n",
- "Build a tree indices for the NYC wiki page and PG essay"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 8,
- "id": "98068ef8-aead-46e7-8dac-0d05b5a86e6a",
- "metadata": {},
- "outputs": [],
- "source": [
- "# try loading\n",
- "nyc_index = GPTSimpleVectorIndex.load_from_disk('index_nyc.json')\n",
- "essay_index = GPTSimpleVectorIndex.load_from_disk('index_pg.json')"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "bdcb22d5-4df8-4d65-aa29-6493fc027fe2",
- "metadata": {
- "tags": []
- },
- "source": [
- "### Set summaries for the indices\n",
- "\n",
- "Add text summaries to indices, so we can compose other indices on top of it"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 9,
- "id": "4149cbbd-7d0b-48c4-8c47-7d67ae0c55f0",
- "metadata": {},
- "outputs": [],
- "source": [
- "nyc_index.set_text(\"\"\"\n",
- " New York, often called New York City or NYC, \n",
- " is the most populous city in the United States. \n",
- " With a 2020 population of 8,804,190 distributed over 300.46 square miles (778.2 km2), \n",
- " New York City is also the most densely populated major city in the United States, \n",
- " and is more than twice as populous as second-place Los Angeles. \n",
- " New York City lies at the southern tip of New York State, and \n",
- " constitutes the geographical and demographic center of both the \n",
- " Northeast megalopolis and the New York metropolitan area, the \n",
- " largest metropolitan area in the world by urban landmass.[8] With over \n",
- " 20.1 million people in its metropolitan statistical area and 23.5 million \n",
- " in its combined statistical area as of 2020, New York is one of the world's \n",
- " most populous megacities, and over 58 million people live within 250 mi (400 km) of \n",
- " the city. New York City is a global cultural, financial, and media center with \n",
- " a significant influence on commerce, health care and life sciences, entertainment, \n",
- " research, technology, education, politics, tourism, dining, art, fashion, and sports. \n",
- " Home to the headquarters of the United Nations, \n",
- " New York is an important center for international diplomacy,\n",
- " an established safe haven for global investors, and is sometimes described as the capital of the world.\n",
- "\"\"\") \n",
- "essay_index.set_text(\"\"\"\n",
- " Author: Paul Graham. \n",
- " The author grew up painting and writing essays. \n",
- " He wrote a book on Lisp and did freelance Lisp hacking work to support himself. \n",
- " He also became the de facto studio assistant for Idelle Weber, an early photorealist painter. \n",
- " He eventually had the idea to start a company to put art galleries online, but the idea was unsuccessful. \n",
- " He then had the idea to write software to build online stores, which became the basis for his successful company, Viaweb. \n",
- " After Viaweb was acquired by Yahoo!, the author returned to painting and started writing essays online. \n",
- " He wrote a book of essays, Hackers & Painters, and worked on spam filters. \n",
- " He also bought a building in Cambridge to use as an office. \n",
- " He then had the idea to start Y Combinator, an investment firm that would \n",
- " make a larger number of smaller investments and help founders remain as CEO. \n",
- " He and his partner Jessica Livingston ran Y Combinator and funded a batch of startups twice a year. \n",
- " He also continued to write essays, cook for groups of friends, and explore the concept of invented vs discovered in software. \n",
- "\n",
- "\"\"\")"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "d4d3cd8b-4134-4cfa-8002-e0a34694d2e1",
- "metadata": {
- "tags": []
- },
- "source": [
- "### Build Keyword Table Index on top of tree indices! \n",
- "\n",
- "We set summaries for each of the NYC and essay indices, and then compose a keyword index on top of it."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 10,
- "id": "76c251ca-b06b-42e9-ac99-aa0a0a5187d4",
- "metadata": {},
- "outputs": [],
- "source": [
- "# set query config\n",
- "query_configs = [\n",
- " {\n",
- " \"index_struct_type\": \"simple_dict\",\n",
- " \"query_mode\": \"default\",\n",
- " \"query_kwargs\": {\n",
- " \"similarity_top_k\": 1\n",
- " }\n",
- " },\n",
- " {\n",
- " \"index_struct_type\": \"keyword_table\",\n",
- " \"query_mode\": \"simple\",\n",
- " \"query_kwargs\": {}\n",
- " },\n",
- "]"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "f975514f-fddd-4737-91de-97bc61394ea9",
- "metadata": {},
- "outputs": [],
- "source": [
- "keyword_table = GPTSimpleKeywordTableIndex([nyc_index, essay_index], max_keywords_per_chunk=50)"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "eebbc448-1e0b-402c-b37e-f93bfcc0bf4f",
- "metadata": {},
- "source": [
- "### Define Graph"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 18,
- "id": "6d68750c-e5ae-481a-8b03-6173020c9bf3",
- "metadata": {},
- "outputs": [],
- "source": [
- "from gpt_index.composability import ComposableGraph"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 19,
- "id": "822ada9f-fb43-472e-95ce-0036d508e528",
- "metadata": {},
- "outputs": [],
- "source": [
- "graph = ComposableGraph.build_from_index(keyword_table)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 32,
- "id": "ae127943-afac-48b4-b22d-84a37e553e4b",
- "metadata": {},
- "outputs": [],
- "source": [
- "# [optional] save to disk\n",
- "graph.save_to_disk(\"index_graph.json\")"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 33,
- "id": "dca2b64b-9af1-456f-8dab-822bfdc5d0ac",
- "metadata": {},
- "outputs": [],
- "source": [
- "# [optional] load from disk\n",
- "graph = ComposableGraph.load_from_disk(\"index_graph.json\")"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "f3c4e58b-b153-4e43-bc02-274a85babbe8",
- "metadata": {},
- "outputs": [],
- "source": [
- "# set Logging to DEBUG for more detailed outputs\n",
- "# ask it a question about NYC \n",
- "response = graph.query(\n",
- " \"What is the climate of New York City like? How cold is it during the winter?\", \n",
- " query_configs=query_configs\n",
- ")"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 27,
- "id": "c0a43443-3e00-4e48-b3ab-f6369191d53a",
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "\n",
- "\n",
- "New York City has a humid subtropical climate (Cfa) under the Köppen climate classification. Winters are typically chilly and damp, with temperatures usually dropping to 10 °F (−12 °C) several times per winter, yet can also reach 60 °F (16 °C) for several days even in the coldest winter month. The daily mean temperature in January, the area's coldest month, is 33.3 °F (0.7 °C). The city receives an average of 46.9 inches (1,194 mm) of rainfall annually, with the wettest month being August 2011, with 18.95 inches (481 mm) of rainfall. The snowiest month on record is February 2010, with 36.9 inches (94 cm) of snowfall. The snowiest season (Jul–Jun) on record is 1995–1996, with 75.6 inches (192 cm) of snowfall.\n"
- ]
- }
- ],
- "source": [
- "print(str(response))"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 28,
- "id": "c78bc3da-6bad-4998-9a81-90a3fa9200a9",
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "> Source (Doc id: 4e8c9bbc-b42f-479f-8fb1-83d0b6198f1d): \n",
- " New York, often called New York City or NYC, \n",
- " is the most populous city in the United St...\n",
- "\n",
- "> Source (Doc id: 77f3b3ea-93ab-49c8-b938-2bd3c870a602): has been altered substantially by human intervention, with considerable land reclamation along th...\n"
- ]
- }
- ],
- "source": [
- "# Get source of response\n",
- "print(response.get_formatted_sources())"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "6b53e45e-93aa-4b49-a497-ab403f6254f9",
- "metadata": {},
- "outputs": [],
- "source": [
- "# ask it a question about PG's essay\n",
- "response = graph.query(\n",
- " \"What did the author do growing up, before his time at Y Combinator?\", \n",
- " query_configs=query_configs\n",
- ")"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 30,
- "id": "06dc71bb-882d-49f5-8566-69b0ea5019dd",
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "\n",
- "\n",
- "The author grew up in England and attended college in the United States. He studied computer science and art, and worked on a variety of projects, including writing essays, hacking, and working on a Lisp interpreter. He also worked on a startup called Viaweb, which was eventually acquired by Yahoo. He also worked on Interleaf, a high-end, special-purpose hardware and software company, and sought out signature styles at RISD. He also lived in a rent-stabilized apartment in New York, and worked on software projects that could be launched as soon as they were done.\n"
- ]
- }
- ],
- "source": [
- "print(str(response))"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 31,
- "id": "b0894565-2b2c-4987-a891-17ba44d775b5",
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "> Source (Doc id: ae92ab9a-c6ed-48c3-b333-d459908dec3f): \n",
- " Author: Paul Graham. \n",
- " The author grew up painting and writing essays. \n",
- " He wrote a bo...\n",
- "\n",
- "> Source (Doc id: a63ad7c4-87f2-42fd-a32b-f682a022af90): get their initial set of customers almost entirely from among their batchmates.\n",
- "\n",
- "I had not origin...\n"
- ]
- }
- ],
- "source": [
- "# Get source of response\n",
- "print(response.get_formatted_sources())"
- ]
- }
- ],
- "metadata": {
- "kernelspec": {
- "display_name": "Python 3 (ipykernel)",
- "language": "python",
- "name": "python3"
- },
- "language_info": {
- "codemirror_mode": {
- "name": "ipython",
- "version": 3
- },
- "file_extension": ".py",
- "mimetype": "text/x-python",
- "name": "python",
- "nbconvert_exporter": "python",
- "pygments_lexer": "ipython3",
- "version": "3.11.1"
- }
- },
- "nbformat": 4,
- "nbformat_minor": 5
-}
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "id": "cfb64210-9c6b-47d7-81f4-67dbdab68e4c",
+ "metadata": {
+ "tags": []
+ },
+ "source": [
+ "# Composable Indices Demo"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "fa0e62b6",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import logging\n",
+ "import sys\n",
+ "\n",
+ "logging.basicConfig(stream=sys.stdout, level=logging.INFO)\n",
+ "logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout))"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 1,
+ "id": "e27b0473-4bda-47f0-b6ed-fd482eac1a13",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from llama_index import (\n",
+ " GPTSimpleVectorIndex, \n",
+ " GPTSimpleKeywordTableIndex, \n",
+ " GPTListIndex, \n",
+ " SimpleDirectoryReader\n",
+ ")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "49e0d841-680f-4a0c-b455-788b54978ebf",
+ "metadata": {},
+ "source": [
+ "#### Load Datasets\n",
+ "\n",
+ "Load both the NYC Wikipedia page as well as Paul Graham's \"What I Worked On\" essay"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 2,
+ "id": "9ec16a8b-6aae-4bf7-9b83-b82087b4ea52",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# fetch \"New York City\" page from Wikipedia\n",
+ "from pathlib import Path\n",
+ "\n",
+ "import requests\n",
+ "response = requests.get(\n",
+ " 'https://en.wikipedia.org/w/api.php',\n",
+ " params={\n",
+ " 'action': 'query',\n",
+ " 'format': 'json',\n",
+ " 'titles': 'New York City',\n",
+ " 'prop': 'extracts',\n",
+ " # 'exintro': True,\n",
+ " 'explaintext': True,\n",
+ " }\n",
+ ").json()\n",
+ "page = next(iter(response['query']['pages'].values()))\n",
+ "nyc_text = page['extract']\n",
+ "\n",
+ "data_path = Path('data')\n",
+ "if not data_path.exists():\n",
+ " Path.mkdir(data_path)\n",
+ "\n",
+ "with open('../test_wiki/data/nyc_text.txt', 'w') as fp:\n",
+ " fp.write(nyc_text)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 2,
+ "id": "39c00aeb-adef-4ce3-8134-031de18e64ea",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# load NYC dataset\n",
+ "nyc_documents = SimpleDirectoryReader('../test_wiki/data/').load_data()"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 3,
+ "id": "ddff8f98-e002-40c5-93ac-93aa40dca5ca",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# load PG's essay\n",
+ "essay_documents = SimpleDirectoryReader('../paul_graham_essay/data/').load_data()"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "f1782198-c0de-4679-8951-1297c21b8639",
+ "metadata": {},
+ "source": [
+ "### Building the document indices\n",
+ "Build a tree index for the NYC wiki page and PG essay"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "5431e83e-428b-4473-bad1-24b7a6c4db38",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# build NYC index\n",
+ "nyc_index = GPTSimpleVectorIndex(nyc_documents)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 5,
+ "id": "6ebb00ff-e9c0-4ec9-ac05-43d4ab7a0d0e",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "nyc_index.save_to_disk('index_nyc.json')"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "8b5aad4a-49ef-4b24-962a-0793f4f09316",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# build essay index\n",
+ "essay_index = GPTSimpleVectorIndex(essay_documents)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 7,
+ "id": "de0d248b-bc87-4129-a12f-c96eda46f72e",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "essay_index.save_to_disk('index_pg.json')"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "b8aaf556-df77-4fac-812b-0b6c6d1da0ef",
+ "metadata": {},
+ "source": [
+ "### Loading the indices\n",
+ "Build a tree indices for the NYC wiki page and PG essay"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 8,
+ "id": "98068ef8-aead-46e7-8dac-0d05b5a86e6a",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# try loading\n",
+ "nyc_index = GPTSimpleVectorIndex.load_from_disk('index_nyc.json')\n",
+ "essay_index = GPTSimpleVectorIndex.load_from_disk('index_pg.json')"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "bdcb22d5-4df8-4d65-aa29-6493fc027fe2",
+ "metadata": {
+ "tags": []
+ },
+ "source": [
+ "### Set summaries for the indices\n",
+ "\n",
+ "Add text summaries to indices, so we can compose other indices on top of it"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 9,
+ "id": "4149cbbd-7d0b-48c4-8c47-7d67ae0c55f0",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "nyc_index.set_text(\"\"\"\n",
+ " New York, often called New York City or NYC, \n",
+ " is the most populous city in the United States. \n",
+ " With a 2020 population of 8,804,190 distributed over 300.46 square miles (778.2 km2), \n",
+ " New York City is also the most densely populated major city in the United States, \n",
+ " and is more than twice as populous as second-place Los Angeles. \n",
+ " New York City lies at the southern tip of New York State, and \n",
+ " constitutes the geographical and demographic center of both the \n",
+ " Northeast megalopolis and the New York metropolitan area, the \n",
+ " largest metropolitan area in the world by urban landmass.[8] With over \n",
+ " 20.1 million people in its metropolitan statistical area and 23.5 million \n",
+ " in its combined statistical area as of 2020, New York is one of the world's \n",
+ " most populous megacities, and over 58 million people live within 250 mi (400 km) of \n",
+ " the city. New York City is a global cultural, financial, and media center with \n",
+ " a significant influence on commerce, health care and life sciences, entertainment, \n",
+ " research, technology, education, politics, tourism, dining, art, fashion, and sports. \n",
+ " Home to the headquarters of the United Nations, \n",
+ " New York is an important center for international diplomacy,\n",
+ " an established safe haven for global investors, and is sometimes described as the capital of the world.\n",
+ "\"\"\") \n",
+ "essay_index.set_text(\"\"\"\n",
+ " Author: Paul Graham. \n",
+ " The author grew up painting and writing essays. \n",
+ " He wrote a book on Lisp and did freelance Lisp hacking work to support himself. \n",
+ " He also became the de facto studio assistant for Idelle Weber, an early photorealist painter. \n",
+ " He eventually had the idea to start a company to put art galleries online, but the idea was unsuccessful. \n",
+ " He then had the idea to write software to build online stores, which became the basis for his successful company, Viaweb. \n",
+ " After Viaweb was acquired by Yahoo!, the author returned to painting and started writing essays online. \n",
+ " He wrote a book of essays, Hackers & Painters, and worked on spam filters. \n",
+ " He also bought a building in Cambridge to use as an office. \n",
+ " He then had the idea to start Y Combinator, an investment firm that would \n",
+ " make a larger number of smaller investments and help founders remain as CEO. \n",
+ " He and his partner Jessica Livingston ran Y Combinator and funded a batch of startups twice a year. \n",
+ " He also continued to write essays, cook for groups of friends, and explore the concept of invented vs discovered in software. \n",
+ "\n",
+ "\"\"\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "d4d3cd8b-4134-4cfa-8002-e0a34694d2e1",
+ "metadata": {
+ "tags": []
+ },
+ "source": [
+ "### Build Keyword Table Index on top of tree indices! \n",
+ "\n",
+ "We set summaries for each of the NYC and essay indices, and then compose a keyword index on top of it."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 10,
+ "id": "76c251ca-b06b-42e9-ac99-aa0a0a5187d4",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# set query config\n",
+ "query_configs = [\n",
+ " {\n",
+ " \"index_struct_type\": \"simple_dict\",\n",
+ " \"query_mode\": \"default\",\n",
+ " \"query_kwargs\": {\n",
+ " \"similarity_top_k\": 1\n",
+ " }\n",
+ " },\n",
+ " {\n",
+ " \"index_struct_type\": \"keyword_table\",\n",
+ " \"query_mode\": \"simple\",\n",
+ " \"query_kwargs\": {}\n",
+ " },\n",
+ "]"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "f975514f-fddd-4737-91de-97bc61394ea9",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "keyword_table = GPTSimpleKeywordTableIndex([nyc_index, essay_index], max_keywords_per_chunk=50)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "eebbc448-1e0b-402c-b37e-f93bfcc0bf4f",
+ "metadata": {},
+ "source": [
+ "### Define Graph"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 18,
+ "id": "6d68750c-e5ae-481a-8b03-6173020c9bf3",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from llama_index.composability import ComposableGraph"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 19,
+ "id": "822ada9f-fb43-472e-95ce-0036d508e528",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "graph = ComposableGraph.build_from_index(keyword_table)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 32,
+ "id": "ae127943-afac-48b4-b22d-84a37e553e4b",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# [optional] save to disk\n",
+ "graph.save_to_disk(\"index_graph.json\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 33,
+ "id": "dca2b64b-9af1-456f-8dab-822bfdc5d0ac",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# [optional] load from disk\n",
+ "graph = ComposableGraph.load_from_disk(\"index_graph.json\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "f3c4e58b-b153-4e43-bc02-274a85babbe8",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# set Logging to DEBUG for more detailed outputs\n",
+ "# ask it a question about NYC \n",
+ "response = graph.query(\n",
+ " \"What is the climate of New York City like? How cold is it during the winter?\", \n",
+ " query_configs=query_configs\n",
+ ")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 27,
+ "id": "c0a43443-3e00-4e48-b3ab-f6369191d53a",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "\n",
+ "\n",
+ "New York City has a humid subtropical climate (Cfa) under the Köppen climate classification. Winters are typically chilly and damp, with temperatures usually dropping to 10 °F (−12 °C) several times per winter, yet can also reach 60 °F (16 °C) for several days even in the coldest winter month. The daily mean temperature in January, the area's coldest month, is 33.3 °F (0.7 °C). The city receives an average of 46.9 inches (1,194 mm) of rainfall annually, with the wettest month being August 2011, with 18.95 inches (481 mm) of rainfall. The snowiest month on record is February 2010, with 36.9 inches (94 cm) of snowfall. The snowiest season (Jul–Jun) on record is 1995–1996, with 75.6 inches (192 cm) of snowfall.\n"
+ ]
+ }
+ ],
+ "source": [
+ "print(str(response))"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 28,
+ "id": "c78bc3da-6bad-4998-9a81-90a3fa9200a9",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "> Source (Doc id: 4e8c9bbc-b42f-479f-8fb1-83d0b6198f1d): \n",
+ " New York, often called New York City or NYC, \n",
+ " is the most populous city in the United St...\n",
+ "\n",
+ "> Source (Doc id: 77f3b3ea-93ab-49c8-b938-2bd3c870a602): has been altered substantially by human intervention, with considerable land reclamation along th...\n"
+ ]
+ }
+ ],
+ "source": [
+ "# Get source of response\n",
+ "print(response.get_formatted_sources())"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "6b53e45e-93aa-4b49-a497-ab403f6254f9",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# ask it a question about PG's essay\n",
+ "response = graph.query(\n",
+ " \"What did the author do growing up, before his time at Y Combinator?\", \n",
+ " query_configs=query_configs\n",
+ ")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 30,
+ "id": "06dc71bb-882d-49f5-8566-69b0ea5019dd",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "\n",
+ "\n",
+ "The author grew up in England and attended college in the United States. He studied computer science and art, and worked on a variety of projects, including writing essays, hacking, and working on a Lisp interpreter. He also worked on a startup called Viaweb, which was eventually acquired by Yahoo. He also worked on Interleaf, a high-end, special-purpose hardware and software company, and sought out signature styles at RISD. He also lived in a rent-stabilized apartment in New York, and worked on software projects that could be launched as soon as they were done.\n"
+ ]
+ }
+ ],
+ "source": [
+ "print(str(response))"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 31,
+ "id": "b0894565-2b2c-4987-a891-17ba44d775b5",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "> Source (Doc id: ae92ab9a-c6ed-48c3-b333-d459908dec3f): \n",
+ " Author: Paul Graham. \n",
+ " The author grew up painting and writing essays. \n",
+ " He wrote a bo...\n",
+ "\n",
+ "> Source (Doc id: a63ad7c4-87f2-42fd-a32b-f682a022af90): get their initial set of customers almost entirely from among their batchmates.\n",
+ "\n",
+ "I had not origin...\n"
+ ]
+ }
+ ],
+ "source": [
+ "# Get source of response\n",
+ "print(response.get_formatted_sources())"
+ ]
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3 (ipykernel)",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.11.1"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}
\ No newline at end of file
diff --git a/examples/cost_analysis/TokenPredictor.ipynb b/examples/cost_analysis/TokenPredictor.ipynb
index c180f0f9f0cb6dfdfc50c22c1c7fd7d89bb07ad9..45a6593cb704f03d34ad69326e9e767e7e811fa2 100644
--- a/examples/cost_analysis/TokenPredictor.ipynb
+++ b/examples/cost_analysis/TokenPredictor.ipynb
@@ -1,352 +1,352 @@
{
- "cells": [
- {
- "cell_type": "markdown",
- "id": "df19606e-d67e-44d2-bed0-4b804e6fc6c3",
- "metadata": {},
- "source": [
- "# Using Token Predictors\n",
- "\n",
- "Using our token predictors, we can predict the token usage of an operation before actually performing it.\n",
- "\n",
- "We first show how to predict LLM token usage with the MockLLMPredictor class, see below.\n",
- "We then show how to also predict embedding token usage."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "f1a9eb90-335c-4214-8bb6-fd1edbe3ccbd",
- "metadata": {},
- "outputs": [],
- "source": [
- "# My OpenAI Key\n",
- "import os\n",
- "os.environ['OPENAI_API_KEY'] = \"INSERT OPENAI KEY\""
- ]
- },
- {
- "cell_type": "markdown",
- "id": "8a707fa6-d79e-4343-92fd-d0fadb25c466",
- "metadata": {},
- "source": [
- "## Using MockLLMPredictor"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "be3f7baa-1c0a-430b-981b-83ddca9e71f2",
- "metadata": {
- "tags": []
- },
- "source": [
- "#### Predicting Usage of GPT Tree Index\n",
- "\n",
- "Here we predict usage of GPTTreeIndex during index construction and querying, without making any LLM calls.\n",
- "\n",
- "NOTE: Predicting query usage before tree is built is only possible with GPTTreeIndex due to the nature of tree traversal. Results will be more accurate if GPTTreeIndex is actually built beforehand."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 6,
- "id": "c0ef16d1-45ef-43ec-9aad-4e44e9bb8578",
- "metadata": {},
- "outputs": [],
- "source": [
- "from gpt_index import GPTTreeIndex, MockLLMPredictor, SimpleDirectoryReader"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 7,
- "id": "b2ecdadc-1403-4bd4-a876-f80e4da911ef",
- "metadata": {},
- "outputs": [],
- "source": [
- "documents = SimpleDirectoryReader('../paul_graham_essay/data').load_data()"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 8,
- "id": "11056808-fd7f-4bc6-9348-0605fb4ee668",
- "metadata": {},
- "outputs": [],
- "source": [
- "llm_predictor = MockLLMPredictor(max_tokens=256)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "9ea4ba66-9a09-4478-b0a8-dee8645fa4e3",
- "metadata": {},
- "outputs": [],
- "source": [
- "index = GPTTreeIndex(documents, llm_predictor=llm_predictor)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 10,
- "id": "345433c2-5553-4645-a513-0186b771a21f",
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "19495\n"
- ]
- }
- ],
- "source": [
- "print(llm_predictor.last_token_usage)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "f43733ae-af35-46e6-99d9-8ba507acbb0d",
- "metadata": {},
- "outputs": [],
- "source": [
- "# default query\n",
- "response = index.query(\"What did the author do growing up?\", llm_predictor=llm_predictor)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 13,
- "id": "4ba19751-da2d-46af-9f8f-4f42871e65a0",
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "5493\n"
- ]
- }
- ],
- "source": [
- "print(llm_predictor.last_token_usage)"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "4324d85b-ae80-48ab-baf0-7dc160dfae46",
- "metadata": {},
- "source": [
- "#### Predicting Usage of GPT Keyword Table Index Query\n",
- "\n",
- "Here we build a real keyword table index over the data, but then predict query usage."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 15,
- "id": "10447805-38db-41b9-a2c6-b0c95437b276",
- "metadata": {},
- "outputs": [],
- "source": [
- "from gpt_index import GPTKeywordTableIndex, MockLLMPredictor, SimpleDirectoryReader"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 16,
- "id": "8ca76e72-5f43-47c1-a9a4-c5c5db4f0f21",
- "metadata": {},
- "outputs": [],
- "source": [
- "documents = SimpleDirectoryReader('../paul_graham_essay/data').load_data()\n",
- "index = GPTKeywordTableIndex.load_from_disk('../paul_graham_essay/index_table.json')"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 17,
- "id": "61f48870-65d2-4b23-b57e-79082ecb4ab2",
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "start token ct: 0\n",
- "> Starting query: What did the author do after his time at Y Combinator?\n",
- "query keywords: ['author', 'did', 'y', 'combinator', 'after', 'his', 'the', 'what', 'time', 'at', 'do']\n",
- "Extracted keywords: ['combinator']\n",
- "> Querying with idx: 3483810247393006047: of 2016 we moved to England. We wanted our kids...\n",
- "> Querying with idx: 7597483754542696814: people edit code on our server through the brow...\n",
- "> Querying with idx: 7572417251450701751: invited about 20 of the 225 groups to interview...\n",
- "end token ct: 11313\n",
- "> [query] Total token usage: 11313 tokens\n",
- "11313\n"
- ]
- }
- ],
- "source": [
- "llm_predictor = MockLLMPredictor(max_tokens=256)\n",
- "response = index.query(\"What did the author do after his time at Y Combinator?\", llm_predictor=llm_predictor)\n",
- "print(llm_predictor.last_token_usage)"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "0fee4405-05e0-46c2-87bb-64ec63a4c6c1",
- "metadata": {},
- "source": [
- "#### Predicting Usage of GPT List Index Query\n",
- "\n",
- "Here we build a real list index over the data, but then predict query usage."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 18,
- "id": "267f2213-67d1-4241-b73f-f1790661d06b",
- "metadata": {},
- "outputs": [],
- "source": [
- "from gpt_index import GPTListIndex, MockLLMPredictor, SimpleDirectoryReader"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 19,
- "id": "d553a8b1-7045-4756-9729-df84bd305279",
- "metadata": {},
- "outputs": [],
- "source": [
- "documents = SimpleDirectoryReader('../paul_graham_essay/data').load_data()\n",
- "index = GPTListIndex.load_from_disk('../paul_graham_essay/index_list.json')"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 20,
- "id": "69c99c68-6a23-48ed-aa41-e7af50fef2f3",
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "start token ct: 0\n",
- "> Starting query: What did the author do after his time at Y Combinator?\n",
- "end token ct: 23941\n",
- "> [query] Total token usage: 23941 tokens\n"
- ]
- }
- ],
- "source": [
- "llm_predictor = MockLLMPredictor(max_tokens=256)\n",
- "response = index.query(\"What did the author do after his time at Y Combinator?\", llm_predictor=llm_predictor)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 21,
- "id": "e8422c5c-af68-4138-a8dd-f6e8d7208c4c",
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "23941\n"
- ]
- }
- ],
- "source": [
- "print(llm_predictor.last_token_usage)"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "1e19cf61-6d6a-4dfa-af78-1ce184f41c6c",
- "metadata": {},
- "source": [
- "## Using MockEmbedding"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "106d86bf-7725-40bc-84ba-4f273493d3f6",
- "metadata": {},
- "source": [
- "#### Predicting Usage of GPT Simple Vector Index"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 3,
- "id": "9baf0fe7-2c11-4233-a930-4e593433ba84",
- "metadata": {},
- "outputs": [],
- "source": [
- "from gpt_index import GPTSimpleVectorIndex, MockLLMPredictor, MockEmbedding, SimpleDirectoryReader"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 5,
- "id": "97023361-fa47-4008-b8d7-e66d60c5b263",
- "metadata": {},
- "outputs": [],
- "source": [
- "documents = SimpleDirectoryReader('../paul_graham_essay/data').load_data()\n",
- "index = GPTSimpleVectorIndex.load_from_disk('../paul_graham_essay/index_simple_vec.json')"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 7,
- "id": "63ebe021-2b9c-4024-95f8-56cd9e7e7c47",
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "> [query] Total LLM token usage: 4374 tokens\n",
- "> [query] Total embedding token usage: 14 tokens\n"
- ]
- }
- ],
- "source": [
- "llm_predictor = MockLLMPredictor(max_tokens=256)\n",
- "embed_model = MockEmbedding(embed_dim=1536)\n",
- "response = index.query(\n",
- " \"What did the author do after his time at Y Combinator?\",\n",
- " llm_predictor=llm_predictor,\n",
- " embed_model=embed_model\n",
- ")"
- ]
- }
- ],
- "metadata": {
- "kernelspec": {
- "display_name": "gpt_retrieve_venv",
- "language": "python",
- "name": "gpt_retrieve_venv"
- },
- "language_info": {
- "codemirror_mode": {
- "name": "ipython",
- "version": 3
- },
- "file_extension": ".py",
- "mimetype": "text/x-python",
- "name": "python",
- "nbconvert_exporter": "python",
- "pygments_lexer": "ipython3",
- "version": "3.9.16"
- }
- },
- "nbformat": 4,
- "nbformat_minor": 5
-}
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "id": "df19606e-d67e-44d2-bed0-4b804e6fc6c3",
+ "metadata": {},
+ "source": [
+ "# Using Token Predictors\n",
+ "\n",
+ "Using our token predictors, we can predict the token usage of an operation before actually performing it.\n",
+ "\n",
+ "We first show how to predict LLM token usage with the MockLLMPredictor class, see below.\n",
+ "We then show how to also predict embedding token usage."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "f1a9eb90-335c-4214-8bb6-fd1edbe3ccbd",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# My OpenAI Key\n",
+ "import os\n",
+ "os.environ['OPENAI_API_KEY'] = \"INSERT OPENAI KEY\""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "8a707fa6-d79e-4343-92fd-d0fadb25c466",
+ "metadata": {},
+ "source": [
+ "## Using MockLLMPredictor"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "be3f7baa-1c0a-430b-981b-83ddca9e71f2",
+ "metadata": {
+ "tags": []
+ },
+ "source": [
+ "#### Predicting Usage of GPT Tree Index\n",
+ "\n",
+ "Here we predict usage of GPTTreeIndex during index construction and querying, without making any LLM calls.\n",
+ "\n",
+ "NOTE: Predicting query usage before tree is built is only possible with GPTTreeIndex due to the nature of tree traversal. Results will be more accurate if GPTTreeIndex is actually built beforehand."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 6,
+ "id": "c0ef16d1-45ef-43ec-9aad-4e44e9bb8578",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from llama_index import GPTTreeIndex, MockLLMPredictor, SimpleDirectoryReader"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 7,
+ "id": "b2ecdadc-1403-4bd4-a876-f80e4da911ef",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "documents = SimpleDirectoryReader('../paul_graham_essay/data').load_data()"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 8,
+ "id": "11056808-fd7f-4bc6-9348-0605fb4ee668",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "llm_predictor = MockLLMPredictor(max_tokens=256)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "9ea4ba66-9a09-4478-b0a8-dee8645fa4e3",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "index = GPTTreeIndex(documents, llm_predictor=llm_predictor)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 10,
+ "id": "345433c2-5553-4645-a513-0186b771a21f",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "19495\n"
+ ]
+ }
+ ],
+ "source": [
+ "print(llm_predictor.last_token_usage)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "f43733ae-af35-46e6-99d9-8ba507acbb0d",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# default query\n",
+ "response = index.query(\"What did the author do growing up?\", llm_predictor=llm_predictor)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 13,
+ "id": "4ba19751-da2d-46af-9f8f-4f42871e65a0",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "5493\n"
+ ]
+ }
+ ],
+ "source": [
+ "print(llm_predictor.last_token_usage)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "4324d85b-ae80-48ab-baf0-7dc160dfae46",
+ "metadata": {},
+ "source": [
+ "#### Predicting Usage of GPT Keyword Table Index Query\n",
+ "\n",
+ "Here we build a real keyword table index over the data, but then predict query usage."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 15,
+ "id": "10447805-38db-41b9-a2c6-b0c95437b276",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from llama_index import GPTKeywordTableIndex, MockLLMPredictor, SimpleDirectoryReader"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 16,
+ "id": "8ca76e72-5f43-47c1-a9a4-c5c5db4f0f21",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "documents = SimpleDirectoryReader('../paul_graham_essay/data').load_data()\n",
+ "index = GPTKeywordTableIndex.load_from_disk('../paul_graham_essay/index_table.json')"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 17,
+ "id": "61f48870-65d2-4b23-b57e-79082ecb4ab2",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "start token ct: 0\n",
+ "> Starting query: What did the author do after his time at Y Combinator?\n",
+ "query keywords: ['author', 'did', 'y', 'combinator', 'after', 'his', 'the', 'what', 'time', 'at', 'do']\n",
+ "Extracted keywords: ['combinator']\n",
+ "> Querying with idx: 3483810247393006047: of 2016 we moved to England. We wanted our kids...\n",
+ "> Querying with idx: 7597483754542696814: people edit code on our server through the brow...\n",
+ "> Querying with idx: 7572417251450701751: invited about 20 of the 225 groups to interview...\n",
+ "end token ct: 11313\n",
+ "> [query] Total token usage: 11313 tokens\n",
+ "11313\n"
+ ]
+ }
+ ],
+ "source": [
+ "llm_predictor = MockLLMPredictor(max_tokens=256)\n",
+ "response = index.query(\"What did the author do after his time at Y Combinator?\", llm_predictor=llm_predictor)\n",
+ "print(llm_predictor.last_token_usage)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "0fee4405-05e0-46c2-87bb-64ec63a4c6c1",
+ "metadata": {},
+ "source": [
+ "#### Predicting Usage of GPT List Index Query\n",
+ "\n",
+ "Here we build a real list index over the data, but then predict query usage."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 18,
+ "id": "267f2213-67d1-4241-b73f-f1790661d06b",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from llama_index import GPTListIndex, MockLLMPredictor, SimpleDirectoryReader"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 19,
+ "id": "d553a8b1-7045-4756-9729-df84bd305279",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "documents = SimpleDirectoryReader('../paul_graham_essay/data').load_data()\n",
+ "index = GPTListIndex.load_from_disk('../paul_graham_essay/index_list.json')"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 20,
+ "id": "69c99c68-6a23-48ed-aa41-e7af50fef2f3",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "start token ct: 0\n",
+ "> Starting query: What did the author do after his time at Y Combinator?\n",
+ "end token ct: 23941\n",
+ "> [query] Total token usage: 23941 tokens\n"
+ ]
+ }
+ ],
+ "source": [
+ "llm_predictor = MockLLMPredictor(max_tokens=256)\n",
+ "response = index.query(\"What did the author do after his time at Y Combinator?\", llm_predictor=llm_predictor)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 21,
+ "id": "e8422c5c-af68-4138-a8dd-f6e8d7208c4c",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "23941\n"
+ ]
+ }
+ ],
+ "source": [
+ "print(llm_predictor.last_token_usage)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "1e19cf61-6d6a-4dfa-af78-1ce184f41c6c",
+ "metadata": {},
+ "source": [
+ "## Using MockEmbedding"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "106d86bf-7725-40bc-84ba-4f273493d3f6",
+ "metadata": {},
+ "source": [
+ "#### Predicting Usage of GPT Simple Vector Index"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 3,
+ "id": "9baf0fe7-2c11-4233-a930-4e593433ba84",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from llama_index import GPTSimpleVectorIndex, MockLLMPredictor, MockEmbedding, SimpleDirectoryReader"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 5,
+ "id": "97023361-fa47-4008-b8d7-e66d60c5b263",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "documents = SimpleDirectoryReader('../paul_graham_essay/data').load_data()\n",
+ "index = GPTSimpleVectorIndex.load_from_disk('../paul_graham_essay/index_simple_vec.json')"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 7,
+ "id": "63ebe021-2b9c-4024-95f8-56cd9e7e7c47",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "> [query] Total LLM token usage: 4374 tokens\n",
+ "> [query] Total embedding token usage: 14 tokens\n"
+ ]
+ }
+ ],
+ "source": [
+ "llm_predictor = MockLLMPredictor(max_tokens=256)\n",
+ "embed_model = MockEmbedding(embed_dim=1536)\n",
+ "response = index.query(\n",
+ " \"What did the author do after his time at Y Combinator?\",\n",
+ " llm_predictor=llm_predictor,\n",
+ " embed_model=embed_model\n",
+ ")"
+ ]
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "gpt_retrieve_venv",
+ "language": "python",
+ "name": "gpt_retrieve_venv"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.9.16"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}
\ No newline at end of file
diff --git a/examples/data_connectors/DatabaseReaderDemo.ipynb b/examples/data_connectors/DatabaseReaderDemo.ipynb
index 75eba08ba227308e19782afc514bf86ffcd3f281..d3779a05eb981a284db9992ecce945c722461d7f 100644
--- a/examples/data_connectors/DatabaseReaderDemo.ipynb
+++ b/examples/data_connectors/DatabaseReaderDemo.ipynb
@@ -1,202 +1,202 @@
{
- "cells": [
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "import logging\n",
- "import sys\n",
- "\n",
- "logging.basicConfig(stream=sys.stdout, level=logging.INFO)\n",
- "logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout))"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "from __future__ import absolute_import\n",
- "\n",
- "# My OpenAI Key\n",
- "import os\n",
- "os.environ['OPENAI_API_KEY'] = \"\"\n",
- "\n",
- "from gpt_index.readers.database import DatabaseReader\n",
- "from gpt_index import GPTSimpleVectorIndex"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "# Initialize DatabaseReader object with the following parameters:\n",
- "\n",
- "db = DatabaseReader(\n",
- " scheme = \"postgresql\", # Database Scheme\n",
- " host = \"localhost\", # Database Host\n",
- " port = \"5432\", # Database Port\n",
- " user = \"postgres\", # Database User\n",
- " password = \"FakeExamplePassword\", # Database Password\n",
- " dbname = \"postgres\", # Database Name\n",
- ")"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "### DatabaseReader class ###\n",
- "# db is an instance of DatabaseReader:\n",
- "print(type(db))\n",
- "# DatabaseReader available method:\n",
- "print(type(db.load_data))\n",
- "\n",
- "### SQLDatabase class ###\n",
- "# db.sql is an instance of SQLDatabase:\n",
- "print(type(db.sql_database))\n",
- "# SQLDatabase available methods:\n",
- "print(type(db.sql_database.from_uri))\n",
- "print(type(db.sql_database.get_single_table_info))\n",
- "print(type(db.sql_database.get_table_columns))\n",
- "print(type(db.sql_database.get_table_info))\n",
- "print(type(db.sql_database.get_table_names))\n",
- "print(type(db.sql_database.insert_into_table))\n",
- "print(type(db.sql_database.run))\n",
- "print(type(db.sql_database.run_sql))\n",
- "# SQLDatabase available properties:\n",
- "print(type(db.sql_database.dialect))\n",
- "print(type(db.sql_database.engine))\n",
- "print(type(db.sql_database.table_info))"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "### Testing DatabaseReader\n",
- "### from SQLDatabase, SQLAlchemy engine and Database URI:\n",
- "\n",
- "# From SQLDatabase instance:\n",
- "print(type(db.sql_database))\n",
- "db_from_sql_database = DatabaseReader(sql_database = db.sql_database)\n",
- "print(type(db_from_sql_database))\n",
- "\n",
- "# From SQLAlchemy engine:\n",
- "print(type(db.sql_database.engine))\n",
- "db_from_engine = DatabaseReader(engine = db.sql_database.engine)\n",
- "print(type(db_from_engine))\n",
- "\n",
- "# From Database URI:\n",
- "print(type(db.uri))\n",
- "db_from_uri = DatabaseReader(uri = db.uri)\n",
- "print(type(db_from_uri))"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "# The below SQL Query example returns a list values of each row\n",
- "# with concatenated text from the name and age columns\n",
- "# from the users table where the age is greater than or equal to 18\n",
- "\n",
- "query = f\"\"\"\n",
- " SELECT\n",
- " CONCAT(name, ' is ', age, ' years old.') AS text\n",
- " FROM public.users\n",
- " WHERE age >= 18\n",
- " \"\"\""
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "# Please refer to gpt_index.langchain_helpers.sql_wrapper\n",
- "# SQLDatabase.run_sql method\n",
- "texts = db.sql_database.run_sql(command = query)\n",
- "\n",
- "# Display type(texts) and texts\n",
- "# type(texts) must return <class 'list'>\n",
- "print(type(texts))\n",
- "\n",
- "# Documents must return a list of Tuple objects\n",
- "print(texts)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "# Please refer to gpt_index.readers.database.DatabaseReader.load_data\n",
- "# DatabaseReader.load_data method\n",
- "documents = db.load_data(query = query)\n",
- "\n",
- "# Display type(documents) and documents\n",
- "# type(documents) must return <class 'list'>\n",
- "print(type(documents))\n",
- "\n",
- "# Documents must return a list of Document objects\n",
- "print(documents)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "try:\n",
- " # Try to load existing Index from disk\n",
- " index = GPTSimpleVectorIndex.load_from_disk('index.json')\n",
- "except:\n",
- " index = GPTSimpleVectorIndex(documents)\n",
- "\n",
- " # Save newly created Index to disk\n",
- " index.save_to_disk('index.json')"
- ]
- }
- ],
- "metadata": {
- "kernelspec": {
- "display_name": "Python 3 (ipykernel)",
- "language": "python",
- "name": "python3"
- },
- "language_info": {
- "codemirror_mode": {
- "name": "ipython",
- "version": 3
- },
- "file_extension": ".py",
- "mimetype": "text/x-python",
- "name": "python",
- "nbconvert_exporter": "python",
- "pygments_lexer": "ipython3",
- "version": "3.11.1"
- },
- "vscode": {
- "interpreter": {
- "hash": "bd5508c2ffc7f17f7d31cf4086cc872f89e96996a08987e995649e5fbe85a3a4"
- }
- }
- },
- "nbformat": 4,
- "nbformat_minor": 2
-}
+ "cells": [
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import logging\n",
+ "import sys\n",
+ "\n",
+ "logging.basicConfig(stream=sys.stdout, level=logging.INFO)\n",
+ "logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout))"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from __future__ import absolute_import\n",
+ "\n",
+ "# My OpenAI Key\n",
+ "import os\n",
+ "os.environ['OPENAI_API_KEY'] = \"\"\n",
+ "\n",
+ "from llama_index.readers.database import DatabaseReader\n",
+ "from llama_index import GPTSimpleVectorIndex"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Initialize DatabaseReader object with the following parameters:\n",
+ "\n",
+ "db = DatabaseReader(\n",
+ " scheme = \"postgresql\", # Database Scheme\n",
+ " host = \"localhost\", # Database Host\n",
+ " port = \"5432\", # Database Port\n",
+ " user = \"postgres\", # Database User\n",
+ " password = \"FakeExamplePassword\", # Database Password\n",
+ " dbname = \"postgres\", # Database Name\n",
+ ")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "### DatabaseReader class ###\n",
+ "# db is an instance of DatabaseReader:\n",
+ "print(type(db))\n",
+ "# DatabaseReader available method:\n",
+ "print(type(db.load_data))\n",
+ "\n",
+ "### SQLDatabase class ###\n",
+ "# db.sql is an instance of SQLDatabase:\n",
+ "print(type(db.sql_database))\n",
+ "# SQLDatabase available methods:\n",
+ "print(type(db.sql_database.from_uri))\n",
+ "print(type(db.sql_database.get_single_table_info))\n",
+ "print(type(db.sql_database.get_table_columns))\n",
+ "print(type(db.sql_database.get_table_info))\n",
+ "print(type(db.sql_database.get_table_names))\n",
+ "print(type(db.sql_database.insert_into_table))\n",
+ "print(type(db.sql_database.run))\n",
+ "print(type(db.sql_database.run_sql))\n",
+ "# SQLDatabase available properties:\n",
+ "print(type(db.sql_database.dialect))\n",
+ "print(type(db.sql_database.engine))\n",
+ "print(type(db.sql_database.table_info))"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "### Testing DatabaseReader\n",
+ "### from SQLDatabase, SQLAlchemy engine and Database URI:\n",
+ "\n",
+ "# From SQLDatabase instance:\n",
+ "print(type(db.sql_database))\n",
+ "db_from_sql_database = DatabaseReader(sql_database = db.sql_database)\n",
+ "print(type(db_from_sql_database))\n",
+ "\n",
+ "# From SQLAlchemy engine:\n",
+ "print(type(db.sql_database.engine))\n",
+ "db_from_engine = DatabaseReader(engine = db.sql_database.engine)\n",
+ "print(type(db_from_engine))\n",
+ "\n",
+ "# From Database URI:\n",
+ "print(type(db.uri))\n",
+ "db_from_uri = DatabaseReader(uri = db.uri)\n",
+ "print(type(db_from_uri))"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# The below SQL Query example returns a list values of each row\n",
+ "# with concatenated text from the name and age columns\n",
+ "# from the users table where the age is greater than or equal to 18\n",
+ "\n",
+ "query = f\"\"\"\n",
+ " SELECT\n",
+ " CONCAT(name, ' is ', age, ' years old.') AS text\n",
+ " FROM public.users\n",
+ " WHERE age >= 18\n",
+ " \"\"\""
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Please refer to llama_index.langchain_helpers.sql_wrapper\n",
+ "# SQLDatabase.run_sql method\n",
+ "texts = db.sql_database.run_sql(command = query)\n",
+ "\n",
+ "# Display type(texts) and texts\n",
+ "# type(texts) must return <class 'list'>\n",
+ "print(type(texts))\n",
+ "\n",
+ "# Documents must return a list of Tuple objects\n",
+ "print(texts)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Please refer to llama_index.readers.database.DatabaseReader.load_data\n",
+ "# DatabaseReader.load_data method\n",
+ "documents = db.load_data(query = query)\n",
+ "\n",
+ "# Display type(documents) and documents\n",
+ "# type(documents) must return <class 'list'>\n",
+ "print(type(documents))\n",
+ "\n",
+ "# Documents must return a list of Document objects\n",
+ "print(documents)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "try:\n",
+ " # Try to load existing Index from disk\n",
+ " index = GPTSimpleVectorIndex.load_from_disk('index.json')\n",
+ "except:\n",
+ " index = GPTSimpleVectorIndex(documents)\n",
+ "\n",
+ " # Save newly created Index to disk\n",
+ " index.save_to_disk('index.json')"
+ ]
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3 (ipykernel)",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.11.1"
+ },
+ "vscode": {
+ "interpreter": {
+ "hash": "bd5508c2ffc7f17f7d31cf4086cc872f89e96996a08987e995649e5fbe85a3a4"
+ }
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}
\ No newline at end of file
diff --git a/examples/data_connectors/DiscordDemo.ipynb b/examples/data_connectors/DiscordDemo.ipynb
index 8fe634bf7e16cc089b9e75e811782d1a1a90850b..0e402e36b1d684f6e9862461e8a7fceea4845bff 100644
--- a/examples/data_connectors/DiscordDemo.ipynb
+++ b/examples/data_connectors/DiscordDemo.ipynb
@@ -1,118 +1,118 @@
{
- "cells": [
- {
- "cell_type": "markdown",
- "id": "effeb5a7-8544-4ee4-8c11-bad0d8165394",
- "metadata": {},
- "source": [
- "# Discord Demo\n",
- "Demonstrates our Discord data connector"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "5fb15bc4",
- "metadata": {},
- "outputs": [],
- "source": [
- "import logging\n",
- "import sys\n",
- "\n",
- "logging.basicConfig(stream=sys.stdout, level=logging.INFO)\n",
- "logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout))"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "04edcd4a-5633-47ee-8a92-ff2f6abc2ec7",
- "metadata": {},
- "outputs": [],
- "source": [
- "# This is due to the fact that we use asyncio.loop_until_complete in\n",
- "# the DiscordReader. Since the Jupyter kernel itself runs on\n",
- "# an event loop, we need to add some help with nesting\n",
- "!pip install nest_asyncio\n",
- "import nest_asyncio\n",
- "nest_asyncio.apply()"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "6ea1f66d-10ed-4417-bdcb-f8a894836ea5",
- "metadata": {},
- "outputs": [],
- "source": [
- "from gpt_index import GPTListIndex, DiscordReader\n",
- "from IPython.display import Markdown, display\n",
- "import os"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "da90589a-fb44-4ec6-9706-753dba4fa968",
- "metadata": {},
- "outputs": [],
- "source": [
- "discord_token = os.getenv(\"DISCORD_TOKEN\")\n",
- "channel_ids = [1057178784895348746] # Replace with your channel_id\n",
- "documents = DiscordReader(discord_token=discord_token).load_data(channel_ids=channel_ids)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "341295df-2029-4728-ab3d-2ee178a7e6f1",
- "metadata": {},
- "outputs": [],
- "source": [
- "index = GPTListIndex(documents)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "01c26b9d-49ec-4a6e-9c61-5c06bb86bbb2",
- "metadata": {},
- "outputs": [],
- "source": [
- "# set Logging to DEBUG for more detailed outputs\n",
- "response = index.query(\"<query_text>\")"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "f160c678-2fb5-4d6d-b2bc-87abb61cfdec",
- "metadata": {},
- "outputs": [],
- "source": [
- "display(Markdown(f\"<b>{response}</b>\"))"
- ]
- }
- ],
- "metadata": {
- "kernelspec": {
- "display_name": "Python 3 (ipykernel)",
- "language": "python",
- "name": "python3"
- },
- "language_info": {
- "codemirror_mode": {
- "name": "ipython",
- "version": 3
- },
- "file_extension": ".py",
- "mimetype": "text/x-python",
- "name": "python",
- "nbconvert_exporter": "python",
- "pygments_lexer": "ipython3",
- "version": "3.11.1"
- }
- },
- "nbformat": 4,
- "nbformat_minor": 5
-}
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "id": "effeb5a7-8544-4ee4-8c11-bad0d8165394",
+ "metadata": {},
+ "source": [
+ "# Discord Demo\n",
+ "Demonstrates our Discord data connector"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "5fb15bc4",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import logging\n",
+ "import sys\n",
+ "\n",
+ "logging.basicConfig(stream=sys.stdout, level=logging.INFO)\n",
+ "logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout))"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "04edcd4a-5633-47ee-8a92-ff2f6abc2ec7",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# This is due to the fact that we use asyncio.loop_until_complete in\n",
+ "# the DiscordReader. Since the Jupyter kernel itself runs on\n",
+ "# an event loop, we need to add some help with nesting\n",
+ "!pip install nest_asyncio\n",
+ "import nest_asyncio\n",
+ "nest_asyncio.apply()"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "6ea1f66d-10ed-4417-bdcb-f8a894836ea5",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from llama_index import GPTListIndex, DiscordReader\n",
+ "from IPython.display import Markdown, display\n",
+ "import os"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "da90589a-fb44-4ec6-9706-753dba4fa968",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "discord_token = os.getenv(\"DISCORD_TOKEN\")\n",
+ "channel_ids = [1057178784895348746] # Replace with your channel_id\n",
+ "documents = DiscordReader(discord_token=discord_token).load_data(channel_ids=channel_ids)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "341295df-2029-4728-ab3d-2ee178a7e6f1",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "index = GPTListIndex(documents)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "01c26b9d-49ec-4a6e-9c61-5c06bb86bbb2",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# set Logging to DEBUG for more detailed outputs\n",
+ "response = index.query(\"<query_text>\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "f160c678-2fb5-4d6d-b2bc-87abb61cfdec",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "display(Markdown(f\"<b>{response}</b>\"))"
+ ]
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3 (ipykernel)",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.11.1"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}
\ No newline at end of file
diff --git a/examples/data_connectors/GithubRepositoryReaderDemo.ipynb b/examples/data_connectors/GithubRepositoryReaderDemo.ipynb
index 4f4e2759479bb2347220a6a9f2ee14269fe9762f..fa41dfb3f1823077be74b560a8e07970073066e9 100644
--- a/examples/data_connectors/GithubRepositoryReaderDemo.ipynb
+++ b/examples/data_connectors/GithubRepositoryReaderDemo.ipynb
@@ -1,110 +1,110 @@
{
- "cells": [
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "# This is due to the fact that we use asyncio.loop_until_complete in\n",
- "# the DiscordReader. Since the Jupyter kernel itself runs on\n",
- "# an event loop, we need to add some help with nesting\n",
- "!pip install nest_asyncio\n",
- "import nest_asyncio\n",
- "nest_asyncio.apply()"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "%env OPENAI_API_KEY=sk-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\n",
- "from gpt_index import GPTSimpleVectorIndex, GithubRepositoryReader\n",
- "from IPython.display import Markdown, display\n",
- "import os"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "%env GITHUB_TOKEN=github_pat_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\n",
- "github_token = os.environ.get(\"GITHUB_TOKEN\")\n",
- "owner = \"jerryjliu\"\n",
- "repo = \"gpt_index\"\n",
- "branch = \"main\"\n",
- "\n",
- "documents = GithubRepositoryReader(\n",
- " github_token=github_token,\n",
- " owner=owner,\n",
- " repo=repo,\n",
- " use_parser=False,\n",
- " verbose=False,\n",
- ").load_data(branch=branch)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "index = GPTSimpleVectorIndex(documents)\n",
- "index.save_to_disk(\"github_index.json\")"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "# import time\n",
- "# for document in documents:\n",
- "# print(document.extra_info)\n",
- "# time.sleep(.25) \n",
- "response = index.query(\"What is the difference between GPTSimpleVectorIndex and GPTListIndex?\", verbose=True)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "display(Markdown(f\"<b>{response}</b>\"))"
- ]
- }
- ],
- "metadata": {
- "kernelspec": {
- "display_name": "gpt_index-github-reader",
- "language": "python",
- "name": "python3"
- },
- "language_info": {
- "codemirror_mode": {
- "name": "ipython",
- "version": 3
- },
- "file_extension": ".py",
- "mimetype": "text/x-python",
- "name": "python",
- "nbconvert_exporter": "python",
- "pygments_lexer": "ipython3",
- "version": "3.11.0"
- },
- "orig_nbformat": 4,
- "vscode": {
- "interpreter": {
- "hash": "5bc2ab08ee48b6366504a28e3231c27a37c154a347ee8ac6184b716eff7bdbcd"
- }
- }
- },
- "nbformat": 4,
- "nbformat_minor": 2
-}
+ "cells": [
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# This is due to the fact that we use asyncio.loop_until_complete in\n",
+ "# the DiscordReader. Since the Jupyter kernel itself runs on\n",
+ "# an event loop, we need to add some help with nesting\n",
+ "!pip install nest_asyncio\n",
+ "import nest_asyncio\n",
+ "nest_asyncio.apply()"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "%env OPENAI_API_KEY=sk-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\n",
+ "from llama_index import GPTSimpleVectorIndex, GithubRepositoryReader\n",
+ "from IPython.display import Markdown, display\n",
+ "import os"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "%env GITHUB_TOKEN=github_pat_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\n",
+ "github_token = os.environ.get(\"GITHUB_TOKEN\")\n",
+ "owner = \"jerryjliu\"\n",
+ "repo = \"gpt_index\"\n",
+ "branch = \"main\"\n",
+ "\n",
+ "documents = GithubRepositoryReader(\n",
+ " github_token=github_token,\n",
+ " owner=owner,\n",
+ " repo=repo,\n",
+ " use_parser=False,\n",
+ " verbose=False,\n",
+ ").load_data(branch=branch)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "index = GPTSimpleVectorIndex(documents)\n",
+ "index.save_to_disk(\"github_index.json\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# import time\n",
+ "# for document in documents:\n",
+ "# print(document.extra_info)\n",
+ "# time.sleep(.25) \n",
+ "response = index.query(\"What is the difference between GPTSimpleVectorIndex and GPTListIndex?\", verbose=True)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "display(Markdown(f\"<b>{response}</b>\"))"
+ ]
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "gpt_index-github-reader",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.11.0"
+ },
+ "orig_nbformat": 4,
+ "vscode": {
+ "interpreter": {
+ "hash": "5bc2ab08ee48b6366504a28e3231c27a37c154a347ee8ac6184b716eff7bdbcd"
+ }
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}
\ No newline at end of file
diff --git a/examples/data_connectors/GoogleDocsDemo.ipynb b/examples/data_connectors/GoogleDocsDemo.ipynb
index fe04202d4a22fff8b680ad03d650f4b6ddf9a6c8..558da4344dc4b4733061e3bf03f5f942a4f15684 100644
--- a/examples/data_connectors/GoogleDocsDemo.ipynb
+++ b/examples/data_connectors/GoogleDocsDemo.ipynb
@@ -1,103 +1,103 @@
{
- "cells": [
- {
- "cell_type": "markdown",
- "id": "88141371-de4c-4c02-9e8f-10d2491b5a33",
- "metadata": {},
- "source": [
- "# Google Docs Demo\n",
- "Demonstrates our Google Docs data connector"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "f6b62adf",
- "metadata": {},
- "outputs": [],
- "source": [
- "import logging\n",
- "import sys\n",
- "\n",
- "logging.basicConfig(stream=sys.stdout, level=logging.INFO)\n",
- "logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout))"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "fafc701d-5a10-4503-9dae-22698fb1aee9",
- "metadata": {},
- "outputs": [],
- "source": [
- "from gpt_index import GPTListIndex, GoogleDocsReader\n",
- "from IPython.display import Markdown, display\n",
- "import os"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "b8f0a9f2-c6a9-4840-a38a-0b2f8e433063",
- "metadata": {},
- "outputs": [],
- "source": [
- "# make sure credentials.json file exists\n",
- "document_ids = [\"<document_id>\"]\n",
- "documents = GoogleDocsReader().load_data(document_ids=document_ids)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "89ef1fac-aa36-4a5f-b5cf-bc4dfa0bd332",
- "metadata": {},
- "outputs": [],
- "source": [
- "index = GPTListIndex(documents)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "c2c1573f-2e49-49e8-8daf-19e6f1777eaa",
- "metadata": {},
- "outputs": [],
- "source": [
- "# set Logging to DEBUG for more detailed outputs\n",
- "response = index.query(\"<query_text>\")"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "6d4533c9-9020-4f50-837c-316ec2c454f2",
- "metadata": {},
- "outputs": [],
- "source": [
- "display(Markdown(f\"<b>{response}</b>\"))"
- ]
- }
- ],
- "metadata": {
- "kernelspec": {
- "display_name": "Python 3 (ipykernel)",
- "language": "python",
- "name": "python3"
- },
- "language_info": {
- "codemirror_mode": {
- "name": "ipython",
- "version": 3
- },
- "file_extension": ".py",
- "mimetype": "text/x-python",
- "name": "python",
- "nbconvert_exporter": "python",
- "pygments_lexer": "ipython3",
- "version": "3.11.1"
- }
- },
- "nbformat": 4,
- "nbformat_minor": 5
-}
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "id": "88141371-de4c-4c02-9e8f-10d2491b5a33",
+ "metadata": {},
+ "source": [
+ "# Google Docs Demo\n",
+ "Demonstrates our Google Docs data connector"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "f6b62adf",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import logging\n",
+ "import sys\n",
+ "\n",
+ "logging.basicConfig(stream=sys.stdout, level=logging.INFO)\n",
+ "logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout))"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "fafc701d-5a10-4503-9dae-22698fb1aee9",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from llama_index import GPTListIndex, GoogleDocsReader\n",
+ "from IPython.display import Markdown, display\n",
+ "import os"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "b8f0a9f2-c6a9-4840-a38a-0b2f8e433063",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# make sure credentials.json file exists\n",
+ "document_ids = [\"<document_id>\"]\n",
+ "documents = GoogleDocsReader().load_data(document_ids=document_ids)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "89ef1fac-aa36-4a5f-b5cf-bc4dfa0bd332",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "index = GPTListIndex(documents)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "c2c1573f-2e49-49e8-8daf-19e6f1777eaa",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# set Logging to DEBUG for more detailed outputs\n",
+ "response = index.query(\"<query_text>\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "6d4533c9-9020-4f50-837c-316ec2c454f2",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "display(Markdown(f\"<b>{response}</b>\"))"
+ ]
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3 (ipykernel)",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.11.1"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}
\ No newline at end of file
diff --git a/examples/data_connectors/MakeDemo.ipynb b/examples/data_connectors/MakeDemo.ipynb
index 2178c30bebbc82a53fce209bef4e99b8c1bd15a9..b60cb53016c54c74a63e2de06f88083c62cc9d65 100644
--- a/examples/data_connectors/MakeDemo.ipynb
+++ b/examples/data_connectors/MakeDemo.ipynb
@@ -1,100 +1,100 @@
{
- "cells": [
- {
- "cell_type": "markdown",
- "id": "7fc13177-7d9d-4959-bbe9-fa26d60ea786",
- "metadata": {},
- "source": [
- "# Make Demo\n",
- "\n",
- "We show how GPT Index can fit with your Make.com workflow by sending the GPT Index response to a scenario webhook."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "d2289d27",
- "metadata": {},
- "outputs": [],
- "source": [
- "import logging\n",
- "import sys\n",
- "\n",
- "logging.basicConfig(stream=sys.stdout, level=logging.INFO)\n",
- "logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout))"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 2,
- "id": "f90c60a6-50b3-4b66-abf3-9723dac8a045",
- "metadata": {},
- "outputs": [],
- "source": [
- "from gpt_index import GPTSimpleVectorIndex\n",
- "from gpt_index.readers import MakeWrapper"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 5,
- "id": "dd8885c5-39e2-444b-9666-5032ab4cb50d",
- "metadata": {},
- "outputs": [],
- "source": [
- "# load index from disk\n",
- "index = GPTSimpleVectorIndex.load_from_disk('../vector_indices/index_simple.json')"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "e5f7d888-01ed-40f7-9216-6c7340b229bf",
- "metadata": {},
- "outputs": [],
- "source": [
- "# set Logging to DEBUG for more detailed outputs\n",
- "# query index\n",
- "query_str = \"What did the author do growing up?\"\n",
- "response = index.query(query_str)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 11,
- "id": "eaf06ad9-ba04-42fb-a7c8-daf7a5320b53",
- "metadata": {},
- "outputs": [],
- "source": [
- "# Send response to Make.com webhook\n",
- "wrapper = MakeWrapper()\n",
- "wrapper.pass_response_to_webhook(\n",
- " \"<webhook_url>,\n",
- " response,\n",
- " query_str\n",
- ")"
- ]
- }
- ],
- "metadata": {
- "kernelspec": {
- "display_name": "Python 3 (ipykernel)",
- "language": "python",
- "name": "python3"
- },
- "language_info": {
- "codemirror_mode": {
- "name": "ipython",
- "version": 3
- },
- "file_extension": ".py",
- "mimetype": "text/x-python",
- "name": "python",
- "nbconvert_exporter": "python",
- "pygments_lexer": "ipython3",
- "version": "3.11.1"
- }
- },
- "nbformat": 4,
- "nbformat_minor": 5
-}
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "id": "7fc13177-7d9d-4959-bbe9-fa26d60ea786",
+ "metadata": {},
+ "source": [
+ "# Make Demo\n",
+ "\n",
+ "We show how GPT Index can fit with your Make.com workflow by sending the GPT Index response to a scenario webhook."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "d2289d27",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import logging\n",
+ "import sys\n",
+ "\n",
+ "logging.basicConfig(stream=sys.stdout, level=logging.INFO)\n",
+ "logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout))"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 2,
+ "id": "f90c60a6-50b3-4b66-abf3-9723dac8a045",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from llama_index import GPTSimpleVectorIndex\n",
+ "from llama_index.readers import MakeWrapper"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 5,
+ "id": "dd8885c5-39e2-444b-9666-5032ab4cb50d",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# load index from disk\n",
+ "index = GPTSimpleVectorIndex.load_from_disk('../vector_indices/index_simple.json')"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "e5f7d888-01ed-40f7-9216-6c7340b229bf",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# set Logging to DEBUG for more detailed outputs\n",
+ "# query index\n",
+ "query_str = \"What did the author do growing up?\"\n",
+ "response = index.query(query_str)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 11,
+ "id": "eaf06ad9-ba04-42fb-a7c8-daf7a5320b53",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Send response to Make.com webhook\n",
+ "wrapper = MakeWrapper()\n",
+ "wrapper.pass_response_to_webhook(\n",
+ " \"<webhook_url>,\n",
+ " response,\n",
+ " query_str\n",
+ ")"
+ ]
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3 (ipykernel)",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.11.1"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}
\ No newline at end of file
diff --git a/examples/data_connectors/MboxReaderDemo.ipynb b/examples/data_connectors/MboxReaderDemo.ipynb
index ff7af8551430fa61f9095ebbc5b02b8db46aae3c..d72e949e86290d55ebd9e6ef3e5ef28d552cc2ef 100644
--- a/examples/data_connectors/MboxReaderDemo.ipynb
+++ b/examples/data_connectors/MboxReaderDemo.ipynb
@@ -1,102 +1,102 @@
{
- "cells": [
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "%env OPENAI_API_KEY=sk-************"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 2,
- "metadata": {},
- "outputs": [],
- "source": [
- "from gpt_index import MboxReader, GPTSimpleVectorIndex"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "documents = MboxReader().load_data('mbox_data_dir', max_count=1000) # Returns list of documents "
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "index = GPTSimpleVectorIndex(documents) # Initialize index with documents"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 4,
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "> [query] Total LLM token usage: 100 tokens\n",
- "> [query] Total embedding token usage: 10 tokens\n"
- ]
- }
- ],
- "source": [
- "res = index.query('When did i have that call with the London office?')"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "> There is a call scheduled with the London office at 12am GMT on the 10th of February."
- ]
- }
- ],
- "source": [
- "res.response"
- ]
- }
- ],
- "metadata": {
- "kernelspec": {
- "display_name": ".venv",
- "language": "python",
- "name": "python3"
- },
- "language_info": {
- "codemirror_mode": {
- "name": "ipython",
- "version": 3
- },
- "file_extension": ".py",
- "mimetype": "text/x-python",
- "name": "python",
- "nbconvert_exporter": "python",
- "pygments_lexer": "ipython3",
- "version": "3.10.8 (main, Oct 13 2022, 09:48:40) [Clang 14.0.0 (clang-1400.0.29.102)]"
- },
- "orig_nbformat": 4,
- "vscode": {
- "interpreter": {
- "hash": "7dd9b00487715d9ffc85f7f860a0013e7a0542b27fc53d2b1d33405d7679eac1"
- }
- }
- },
- "nbformat": 4,
- "nbformat_minor": 2
-}
+ "cells": [
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "%env OPENAI_API_KEY=sk-************"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 2,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from llama_index import MboxReader, GPTSimpleVectorIndex"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "documents = MboxReader().load_data('mbox_data_dir', max_count=1000) # Returns list of documents "
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "index = GPTSimpleVectorIndex(documents) # Initialize index with documents"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 4,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "> [query] Total LLM token usage: 100 tokens\n",
+ "> [query] Total embedding token usage: 10 tokens\n"
+ ]
+ }
+ ],
+ "source": [
+ "res = index.query('When did i have that call with the London office?')"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "> There is a call scheduled with the London office at 12am GMT on the 10th of February."
+ ]
+ }
+ ],
+ "source": [
+ "res.response"
+ ]
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": ".venv",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.10.8 (main, Oct 13 2022, 09:48:40) [Clang 14.0.0 (clang-1400.0.29.102)]"
+ },
+ "orig_nbformat": 4,
+ "vscode": {
+ "interpreter": {
+ "hash": "7dd9b00487715d9ffc85f7f860a0013e7a0542b27fc53d2b1d33405d7679eac1"
+ }
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}
\ No newline at end of file
diff --git a/examples/data_connectors/MongoDemo.ipynb b/examples/data_connectors/MongoDemo.ipynb
index 07079d4b65d77a9aa8fac0925c71e76f5e7177d6..6f1ddb507a9f893a03f3da67273df4aaeeaee64e 100644
--- a/examples/data_connectors/MongoDemo.ipynb
+++ b/examples/data_connectors/MongoDemo.ipynb
@@ -1,108 +1,108 @@
{
- "cells": [
- {
- "cell_type": "markdown",
- "id": "effeb5a7-8544-4ee4-8c11-bad0d8165394",
- "metadata": {},
- "source": [
- "# MongoDB Demo\n",
- "Demonstrates our MongoDB data connector"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "60355655",
- "metadata": {},
- "outputs": [],
- "source": [
- "import logging\n",
- "import sys\n",
- "\n",
- "logging.basicConfig(stream=sys.stdout, level=logging.INFO)\n",
- "logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout))"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "6ea1f66d-10ed-4417-bdcb-f8a894836ea5",
- "metadata": {},
- "outputs": [],
- "source": [
- "from gpt_index import GPTListIndex, SimpleMongoReader\n",
- "from IPython.display import Markdown, display\n",
- "import os"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "da90589a-fb44-4ec6-9706-753dba4fa968",
- "metadata": {},
- "outputs": [],
- "source": [
- "host = \"<host>\"\n",
- "port = \"<port>\"\n",
- "db_name = \"<db_name>\"\n",
- "collection_name = \"<collection_name>\"\n",
- "# query_dict is passed into db.collection.find()\n",
- "query_dict = {}\n",
- "reader = SimpleMongoReader(host, port)\n",
- "documents = reader.load_data(db_name, collection_name, query_dict=query_dict)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "341295df-2029-4728-ab3d-2ee178a7e6f1",
- "metadata": {},
- "outputs": [],
- "source": [
- "index = GPTListIndex(documents)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "01c26b9d-49ec-4a6e-9c61-5c06bb86bbb2",
- "metadata": {},
- "outputs": [],
- "source": [
- "# set Logging to DEBUG for more detailed outputs\n",
- "response = index.query(\"<query_text>\")"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "f160c678-2fb5-4d6d-b2bc-87abb61cfdec",
- "metadata": {},
- "outputs": [],
- "source": [
- "display(Markdown(f\"<b>{response}</b>\"))"
- ]
- }
- ],
- "metadata": {
- "kernelspec": {
- "display_name": "Python 3 (ipykernel)",
- "language": "python",
- "name": "python3"
- },
- "language_info": {
- "codemirror_mode": {
- "name": "ipython",
- "version": 3
- },
- "file_extension": ".py",
- "mimetype": "text/x-python",
- "name": "python",
- "nbconvert_exporter": "python",
- "pygments_lexer": "ipython3",
- "version": "3.11.1"
- }
- },
- "nbformat": 4,
- "nbformat_minor": 5
-}
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "id": "effeb5a7-8544-4ee4-8c11-bad0d8165394",
+ "metadata": {},
+ "source": [
+ "# MongoDB Demo\n",
+ "Demonstrates our MongoDB data connector"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "60355655",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import logging\n",
+ "import sys\n",
+ "\n",
+ "logging.basicConfig(stream=sys.stdout, level=logging.INFO)\n",
+ "logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout))"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "6ea1f66d-10ed-4417-bdcb-f8a894836ea5",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from llama_index import GPTListIndex, SimpleMongoReader\n",
+ "from IPython.display import Markdown, display\n",
+ "import os"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "da90589a-fb44-4ec6-9706-753dba4fa968",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "host = \"<host>\"\n",
+ "port = \"<port>\"\n",
+ "db_name = \"<db_name>\"\n",
+ "collection_name = \"<collection_name>\"\n",
+ "# query_dict is passed into db.collection.find()\n",
+ "query_dict = {}\n",
+ "reader = SimpleMongoReader(host, port)\n",
+ "documents = reader.load_data(db_name, collection_name, query_dict=query_dict)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "341295df-2029-4728-ab3d-2ee178a7e6f1",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "index = GPTListIndex(documents)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "01c26b9d-49ec-4a6e-9c61-5c06bb86bbb2",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# set Logging to DEBUG for more detailed outputs\n",
+ "response = index.query(\"<query_text>\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "f160c678-2fb5-4d6d-b2bc-87abb61cfdec",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "display(Markdown(f\"<b>{response}</b>\"))"
+ ]
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3 (ipykernel)",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.11.1"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}
\ No newline at end of file
diff --git a/examples/data_connectors/NotionDemo.ipynb b/examples/data_connectors/NotionDemo.ipynb
index ca6c130ac6da0c11fb998a7904840ccf28757d86..3f96dc5713bd59ed7e54e61d9e1ac05c71f85a12 100644
--- a/examples/data_connectors/NotionDemo.ipynb
+++ b/examples/data_connectors/NotionDemo.ipynb
@@ -1,145 +1,145 @@
{
- "cells": [
- {
- "cell_type": "markdown",
- "id": "effeb5a7-8544-4ee4-8c11-bad0d8165394",
- "metadata": {},
- "source": [
- "# Notion Demo\n",
- "Demonstrates our Notion data connector"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "995afc19",
- "metadata": {},
- "outputs": [],
- "source": [
- "import logging\n",
- "import sys\n",
- "\n",
- "logging.basicConfig(stream=sys.stdout, level=logging.INFO)\n",
- "logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout))"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 1,
- "id": "6ea1f66d-10ed-4417-bdcb-f8a894836ea5",
- "metadata": {},
- "outputs": [],
- "source": [
- "from gpt_index import GPTListIndex, NotionPageReader\n",
- "from IPython.display import Markdown, display\n",
- "import os"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "da90589a-fb44-4ec6-9706-753dba4fa968",
- "metadata": {},
- "outputs": [],
- "source": [
- "integration_token = os.getenv(\"NOTION_INTEGRATION_TOKEN\")\n",
- "page_ids = [\"<page_id>\"]\n",
- "documents = NotionPageReader(integration_token=integration_token).load_data(page_ids=page_ids)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "341295df-2029-4728-ab3d-2ee178a7e6f1",
- "metadata": {},
- "outputs": [],
- "source": [
- "index = GPTListIndex(documents)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "01c26b9d-49ec-4a6e-9c61-5c06bb86bbb2",
- "metadata": {},
- "outputs": [],
- "source": [
- "# set Logging to DEBUG for more detailed outputs\n",
- "response = index.query(\"<query_text>\")"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "f160c678-2fb5-4d6d-b2bc-87abb61cfdec",
- "metadata": {},
- "outputs": [],
- "source": [
- "display(Markdown(f\"<b>{response}</b>\"))"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "8e8e1b13",
- "metadata": {},
- "source": [
- "You can also pass the id of a database to index all the pages in that database:"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 4,
- "id": "087431a2-b04c-441c-820f-6d6d3cdf831c",
- "metadata": {},
- "outputs": [],
- "source": [
- "database_id = \"<database-id>\"\n",
- "\n",
- "# https://developers.notion.com/docs/working-with-databases for how to find your database id\n",
- "\n",
- "documents = NotionPageReader(integration_token=integration_token).load_data(database_id=database_id)\n",
- "\n",
- "print(documents)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "6464025d-0c5a-4e2d-8a90-91c29ece9884",
- "metadata": {},
- "outputs": [],
- "source": [
- "# set Logging to DEBUG for more detailed outputs\n",
- "index = GPTListIndex(documents)\n",
- "response = index.query(\"<query_text>\")\n",
- "display(Markdown(f\"<b>{response}</b>\"))"
- ]
- }
- ],
- "metadata": {
- "kernelspec": {
- "display_name": "Python 3 (ipykernel)",
- "language": "python",
- "name": "python3"
- },
- "language_info": {
- "codemirror_mode": {
- "name": "ipython",
- "version": 3
- },
- "file_extension": ".py",
- "mimetype": "text/x-python",
- "name": "python",
- "nbconvert_exporter": "python",
- "pygments_lexer": "ipython3",
- "version": "3.11.1"
- },
- "vscode": {
- "interpreter": {
- "hash": "c32397a35d2e76e766f80c3872b208f0c0029e8a6a9b8e2a8fe7b1641cfa009b"
- }
- }
- },
- "nbformat": 4,
- "nbformat_minor": 5
-}
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "id": "effeb5a7-8544-4ee4-8c11-bad0d8165394",
+ "metadata": {},
+ "source": [
+ "# Notion Demo\n",
+ "Demonstrates our Notion data connector"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "995afc19",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import logging\n",
+ "import sys\n",
+ "\n",
+ "logging.basicConfig(stream=sys.stdout, level=logging.INFO)\n",
+ "logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout))"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 1,
+ "id": "6ea1f66d-10ed-4417-bdcb-f8a894836ea5",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from llama_index import GPTListIndex, NotionPageReader\n",
+ "from IPython.display import Markdown, display\n",
+ "import os"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "da90589a-fb44-4ec6-9706-753dba4fa968",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "integration_token = os.getenv(\"NOTION_INTEGRATION_TOKEN\")\n",
+ "page_ids = [\"<page_id>\"]\n",
+ "documents = NotionPageReader(integration_token=integration_token).load_data(page_ids=page_ids)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "341295df-2029-4728-ab3d-2ee178a7e6f1",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "index = GPTListIndex(documents)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "01c26b9d-49ec-4a6e-9c61-5c06bb86bbb2",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# set Logging to DEBUG for more detailed outputs\n",
+ "response = index.query(\"<query_text>\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "f160c678-2fb5-4d6d-b2bc-87abb61cfdec",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "display(Markdown(f\"<b>{response}</b>\"))"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "8e8e1b13",
+ "metadata": {},
+ "source": [
+ "You can also pass the id of a database to index all the pages in that database:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 4,
+ "id": "087431a2-b04c-441c-820f-6d6d3cdf831c",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "database_id = \"<database-id>\"\n",
+ "\n",
+ "# https://developers.notion.com/docs/working-with-databases for how to find your database id\n",
+ "\n",
+ "documents = NotionPageReader(integration_token=integration_token).load_data(database_id=database_id)\n",
+ "\n",
+ "print(documents)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "6464025d-0c5a-4e2d-8a90-91c29ece9884",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# set Logging to DEBUG for more detailed outputs\n",
+ "index = GPTListIndex(documents)\n",
+ "response = index.query(\"<query_text>\")\n",
+ "display(Markdown(f\"<b>{response}</b>\"))"
+ ]
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3 (ipykernel)",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.11.1"
+ },
+ "vscode": {
+ "interpreter": {
+ "hash": "c32397a35d2e76e766f80c3872b208f0c0029e8a6a9b8e2a8fe7b1641cfa009b"
+ }
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}
\ No newline at end of file
diff --git a/examples/data_connectors/ObsidianReaderDemo.ipynb b/examples/data_connectors/ObsidianReaderDemo.ipynb
index d69e94051d61b09dcafdd9cf4339a73f47673ec1..c5be6d41acb3e7a6b2b3df14203bf9d3c92a3f7b 100644
--- a/examples/data_connectors/ObsidianReaderDemo.ipynb
+++ b/examples/data_connectors/ObsidianReaderDemo.ipynb
@@ -1,135 +1,135 @@
{
- "cells": [
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "%env OPENAI_API_KEY=sk-************"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "import logging\n",
- "import sys\n",
- "\n",
- "logging.basicConfig(stream=sys.stdout, level=logging.INFO)\n",
- "logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout))"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 2,
- "metadata": {},
- "outputs": [],
- "source": [
- "from gpt_index import ObsidianReader, GPTSimpleVectorIndex"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "documents = ObsidianReader('/Users/hursh/vault').load_data() # Returns list of documents "
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "index = GPTSimpleVectorIndex(documents) # Initialize index with documents"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 3,
- "metadata": {},
- "outputs": [],
- "source": [
- "# index.save_to_disk('index.json')\n",
- "index = GPTSimpleVectorIndex.load_from_disk('index.json')"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 4,
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "> [query] Total LLM token usage: 920 tokens\n",
- "> [query] Total embedding token usage: 7 tokens\n"
- ]
- }
- ],
- "source": [
- "# set Logging to DEBUG for more detailed outputs\n",
- "res = index.query('What is the meaning of life?')"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 6,
- "metadata": {},
- "outputs": [
- {
- "data": {
- "text/plain": [
- "'\\nThe meaning of life is subjective and can vary from person to person. It is ultimately up to each individual to decide what they believe is the purpose and value of life. Some may find meaning in their faith, while others may find it in their relationships, work, or hobbies. Ultimately, it is up to each individual to decide what brings them joy and fulfillment and to pursue that path.'"
- ]
- },
- "execution_count": 6,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "res.response"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": []
- }
- ],
- "metadata": {
- "kernelspec": {
- "display_name": "Python 3 (ipykernel)",
- "language": "python",
- "name": "python3"
- },
- "language_info": {
- "codemirror_mode": {
- "name": "ipython",
- "version": 3
- },
- "file_extension": ".py",
- "mimetype": "text/x-python",
- "name": "python",
- "nbconvert_exporter": "python",
- "pygments_lexer": "ipython3",
- "version": "3.11.1"
- },
- "vscode": {
- "interpreter": {
- "hash": "31f2aee4e71d21fbe5cf8b01ff0e069b9275f58929596ceb00d14d90e3e16cd6"
- }
- }
- },
- "nbformat": 4,
- "nbformat_minor": 2
-}
+ "cells": [
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "%env OPENAI_API_KEY=sk-************"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import logging\n",
+ "import sys\n",
+ "\n",
+ "logging.basicConfig(stream=sys.stdout, level=logging.INFO)\n",
+ "logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout))"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 2,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from llama_index import ObsidianReader, GPTSimpleVectorIndex"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "documents = ObsidianReader('/Users/hursh/vault').load_data() # Returns list of documents "
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "index = GPTSimpleVectorIndex(documents) # Initialize index with documents"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 3,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# index.save_to_disk('index.json')\n",
+ "index = GPTSimpleVectorIndex.load_from_disk('index.json')"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 4,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "> [query] Total LLM token usage: 920 tokens\n",
+ "> [query] Total embedding token usage: 7 tokens\n"
+ ]
+ }
+ ],
+ "source": [
+ "# set Logging to DEBUG for more detailed outputs\n",
+ "res = index.query('What is the meaning of life?')"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 6,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "'\\nThe meaning of life is subjective and can vary from person to person. It is ultimately up to each individual to decide what they believe is the purpose and value of life. Some may find meaning in their faith, while others may find it in their relationships, work, or hobbies. Ultimately, it is up to each individual to decide what brings them joy and fulfillment and to pursue that path.'"
+ ]
+ },
+ "execution_count": 6,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "res.response"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": []
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3 (ipykernel)",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.11.1"
+ },
+ "vscode": {
+ "interpreter": {
+ "hash": "31f2aee4e71d21fbe5cf8b01ff0e069b9275f58929596ceb00d14d90e3e16cd6"
+ }
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}
\ No newline at end of file
diff --git a/examples/data_connectors/PineconeDemo.ipynb b/examples/data_connectors/PineconeDemo.ipynb
index 3f32c809ed08a1a8b3cb584c60c87af1e57ef418..4b52b7ddad9aa7234637d84378993e265cee0206 100644
--- a/examples/data_connectors/PineconeDemo.ipynb
+++ b/examples/data_connectors/PineconeDemo.ipynb
@@ -1,151 +1,151 @@
{
- "cells": [
- {
- "cell_type": "markdown",
- "id": "f3ca56f0-6ef1-426f-bac5-fd7c374d0f51",
- "metadata": {},
- "source": [
- "# Pinecone Demo"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "b2bd3c59",
- "metadata": {},
- "outputs": [],
- "source": [
- "import logging\n",
- "import sys\n",
- "\n",
- "logging.basicConfig(stream=sys.stdout, level=logging.INFO)\n",
- "logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout))"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 1,
- "id": "e2f49003-b952-4b9b-b935-2941f9303773",
- "metadata": {},
- "outputs": [],
- "source": [
- "api_key = \"<api_key>\""
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 2,
- "id": "262f990a-79c8-413a-9f3c-cd9a3c191307",
- "metadata": {},
- "outputs": [],
- "source": [
- "from gpt_index.readers.pinecone import PineconeReader"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 3,
- "id": "252f8163-7297-44b6-a838-709e9662f3d6",
- "metadata": {},
- "outputs": [],
- "source": [
- "reader = PineconeReader(api_key=api_key, environment=\"us-west1-gcp\")"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 4,
- "id": "53b49187-8477-436c-9718-5d2f8cc6fad0",
- "metadata": {},
- "outputs": [],
- "source": [
- "# the id_to_text_map specifies a mapping from the ID specified in Pinecone to your text. \n",
- "id_to_text_map = {\n",
- " \"id1\": \"text blob 1\",\n",
- " \"id2\": \"text blob 2\",\n",
- "}\n",
- "\n",
- "# the query_vector is an embedding representation of your query_vector\n",
- "# Example query vector:\n",
- "# query_vector=[0.3, 0.3, 0.3, 0.3, 0.3, 0.3, 0.3, 0.3]\n",
- "\n",
- "query_vector=[n1, n2, n3, ...]"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "a88be1c4-603f-48b9-ac64-10a219af4951",
- "metadata": {},
- "outputs": [],
- "source": [
- "# NOTE: Required args are index_name, id_to_text_map, vector.\n",
- "# In addition, we pass-through all kwargs that can be passed into the the `Query` operation in Pinecone.\n",
- "# See the API reference: https://docs.pinecone.io/reference/query\n",
- "# and also the Python client: https://github.com/pinecone-io/pinecone-python-client\n",
- "# for more details. \n",
- "documents = reader.load_data(index_name='quickstart', id_to_text_map=id_to_text_map, top_k=3, vector=query_vector, separate_documents=True)"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "a4baf59e-fc97-4a1e-947f-354a6438ffa6",
- "metadata": {},
- "source": [
- "### Create index "
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "109d083e-f3b4-420b-886b-087c8cf3f98b",
- "metadata": {},
- "outputs": [],
- "source": [
- "index = GPTListIndex(documents)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "e15b9177-9e94-4e4e-9a2e-cd3a288a7faf",
- "metadata": {},
- "outputs": [],
- "source": [
- "# set Logging to DEBUG for more detailed outputs\n",
- "response = index.query(\"<query_text>\")"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "67b50613-a589-4acf-ba16-10571b415268",
- "metadata": {},
- "outputs": [],
- "source": [
- "display(Markdown(f\"<b>{response}</b>\"))"
- ]
- }
- ],
- "metadata": {
- "kernelspec": {
- "display_name": "Python 3 (ipykernel)",
- "language": "python",
- "name": "python3"
- },
- "language_info": {
- "codemirror_mode": {
- "name": "ipython",
- "version": 3
- },
- "file_extension": ".py",
- "mimetype": "text/x-python",
- "name": "python",
- "nbconvert_exporter": "python",
- "pygments_lexer": "ipython3",
- "version": "3.11.1"
- }
- },
- "nbformat": 4,
- "nbformat_minor": 5
-}
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "id": "f3ca56f0-6ef1-426f-bac5-fd7c374d0f51",
+ "metadata": {},
+ "source": [
+ "# Pinecone Demo"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "b2bd3c59",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import logging\n",
+ "import sys\n",
+ "\n",
+ "logging.basicConfig(stream=sys.stdout, level=logging.INFO)\n",
+ "logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout))"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 1,
+ "id": "e2f49003-b952-4b9b-b935-2941f9303773",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "api_key = \"<api_key>\""
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 2,
+ "id": "262f990a-79c8-413a-9f3c-cd9a3c191307",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from llama_index.readers.pinecone import PineconeReader"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 3,
+ "id": "252f8163-7297-44b6-a838-709e9662f3d6",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "reader = PineconeReader(api_key=api_key, environment=\"us-west1-gcp\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 4,
+ "id": "53b49187-8477-436c-9718-5d2f8cc6fad0",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# the id_to_text_map specifies a mapping from the ID specified in Pinecone to your text. \n",
+ "id_to_text_map = {\n",
+ " \"id1\": \"text blob 1\",\n",
+ " \"id2\": \"text blob 2\",\n",
+ "}\n",
+ "\n",
+ "# the query_vector is an embedding representation of your query_vector\n",
+ "# Example query vector:\n",
+ "# query_vector=[0.3, 0.3, 0.3, 0.3, 0.3, 0.3, 0.3, 0.3]\n",
+ "\n",
+ "query_vector=[n1, n2, n3, ...]"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "a88be1c4-603f-48b9-ac64-10a219af4951",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# NOTE: Required args are index_name, id_to_text_map, vector.\n",
+ "# In addition, we pass-through all kwargs that can be passed into the the `Query` operation in Pinecone.\n",
+ "# See the API reference: https://docs.pinecone.io/reference/query\n",
+ "# and also the Python client: https://github.com/pinecone-io/pinecone-python-client\n",
+ "# for more details. \n",
+ "documents = reader.load_data(index_name='quickstart', id_to_text_map=id_to_text_map, top_k=3, vector=query_vector, separate_documents=True)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "a4baf59e-fc97-4a1e-947f-354a6438ffa6",
+ "metadata": {},
+ "source": [
+ "### Create index "
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "109d083e-f3b4-420b-886b-087c8cf3f98b",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "index = GPTListIndex(documents)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "e15b9177-9e94-4e4e-9a2e-cd3a288a7faf",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# set Logging to DEBUG for more detailed outputs\n",
+ "response = index.query(\"<query_text>\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "67b50613-a589-4acf-ba16-10571b415268",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "display(Markdown(f\"<b>{response}</b>\"))"
+ ]
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3 (ipykernel)",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.11.1"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}
\ No newline at end of file
diff --git a/examples/data_connectors/QdrantDemo.ipynb b/examples/data_connectors/QdrantDemo.ipynb
index 06ce1adc8f8b11fceccb9b2a0d6d9f65e88ea41d..68f74cbb9c9534d2a323040597bd3d72f2f4e1a0 100644
--- a/examples/data_connectors/QdrantDemo.ipynb
+++ b/examples/data_connectors/QdrantDemo.ipynb
@@ -1,133 +1,133 @@
{
- "cells": [
- {
- "cell_type": "markdown",
- "id": "f3ca56f0-6ef1-426f-bac5-fd7c374d0f51",
- "metadata": {},
- "source": [
- "# Qdrant Demo"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "778ee662",
- "metadata": {},
- "outputs": [],
- "source": [
- "import logging\n",
- "import sys\n",
- "\n",
- "logging.basicConfig(stream=sys.stdout, level=logging.INFO)\n",
- "logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout))"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "262f990a-79c8-413a-9f3c-cd9a3c191307",
- "metadata": {},
- "outputs": [],
- "source": [
- "from gpt_index.readers.qdrant import QdrantReader"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "252f8163-7297-44b6-a838-709e9662f3d6",
- "metadata": {},
- "outputs": [],
- "source": [
- "reader = QdrantReader(host=\"localhost\")"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "53b49187-8477-436c-9718-5d2f8cc6fad0",
- "metadata": {},
- "outputs": [],
- "source": [
- "# the query_vector is an embedding representation of your query_vector\n",
- "# Example query vector:\n",
- "# query_vector=[0.3, 0.3, 0.3, 0.3, 0.3, 0.3, 0.3, 0.3]\n",
- "\n",
- "query_vector=[n1, n2, n3, ...]"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "a88be1c4-603f-48b9-ac64-10a219af4951",
- "metadata": {},
- "outputs": [],
- "source": [
- "# NOTE: Required args are collection_name, query_vector.\n",
- "# See the Python client: https://github.com/qdrant/qdrant_client\n",
- "# for more details. \n",
- "documents = reader.load_data(collection_name=\"demo\", query_vector=query_vector, limit=5)"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "169b4273-eb20-4d06-9ffe-71320f4570f6",
- "metadata": {},
- "source": [
- "### Create index"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "ac4563a1",
- "metadata": {},
- "outputs": [],
- "source": [
- "index = GPTListIndex(documents)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "f06b02db",
- "metadata": {},
- "outputs": [],
- "source": [
- "# set Logging to DEBUG for more detailed outputs\n",
- "response = index.query(\"<query_text>\")"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "97d1ae80",
- "metadata": {},
- "outputs": [],
- "source": [
- "display(Markdown(f\"<b>{response}</b>\"))"
- ]
- }
- ],
- "metadata": {
- "kernelspec": {
- "display_name": "Python 3 (ipykernel)",
- "language": "python",
- "name": "python3"
- },
- "language_info": {
- "codemirror_mode": {
- "name": "ipython",
- "version": 3
- },
- "file_extension": ".py",
- "mimetype": "text/x-python",
- "name": "python",
- "nbconvert_exporter": "python",
- "pygments_lexer": "ipython3",
- "version": "3.11.1"
- }
- },
- "nbformat": 4,
- "nbformat_minor": 5
-}
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "id": "f3ca56f0-6ef1-426f-bac5-fd7c374d0f51",
+ "metadata": {},
+ "source": [
+ "# Qdrant Demo"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "778ee662",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import logging\n",
+ "import sys\n",
+ "\n",
+ "logging.basicConfig(stream=sys.stdout, level=logging.INFO)\n",
+ "logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout))"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "262f990a-79c8-413a-9f3c-cd9a3c191307",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from llama_index.readers.qdrant import QdrantReader"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "252f8163-7297-44b6-a838-709e9662f3d6",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "reader = QdrantReader(host=\"localhost\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "53b49187-8477-436c-9718-5d2f8cc6fad0",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# the query_vector is an embedding representation of your query_vector\n",
+ "# Example query vector:\n",
+ "# query_vector=[0.3, 0.3, 0.3, 0.3, 0.3, 0.3, 0.3, 0.3]\n",
+ "\n",
+ "query_vector=[n1, n2, n3, ...]"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "a88be1c4-603f-48b9-ac64-10a219af4951",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# NOTE: Required args are collection_name, query_vector.\n",
+ "# See the Python client: https://github.com/qdrant/qdrant_client\n",
+ "# for more details. \n",
+ "documents = reader.load_data(collection_name=\"demo\", query_vector=query_vector, limit=5)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "169b4273-eb20-4d06-9ffe-71320f4570f6",
+ "metadata": {},
+ "source": [
+ "### Create index"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "ac4563a1",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "index = GPTListIndex(documents)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "f06b02db",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# set Logging to DEBUG for more detailed outputs\n",
+ "response = index.query(\"<query_text>\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "97d1ae80",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "display(Markdown(f\"<b>{response}</b>\"))"
+ ]
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3 (ipykernel)",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.11.1"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}
\ No newline at end of file
diff --git a/examples/data_connectors/SlackDemo.ipynb b/examples/data_connectors/SlackDemo.ipynb
index 986fdb51dfa7445f49d956f7d6a45aab1e12ac82..1e82af60ffe42f5a273e95cf2e6d4bc4f95a778e 100644
--- a/examples/data_connectors/SlackDemo.ipynb
+++ b/examples/data_connectors/SlackDemo.ipynb
@@ -1,103 +1,103 @@
{
- "cells": [
- {
- "cell_type": "markdown",
- "id": "effeb5a7-8544-4ee4-8c11-bad0d8165394",
- "metadata": {},
- "source": [
- "# Slack Demo\n",
- "Demonstrates our Slack data connector"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "dc664882",
- "metadata": {},
- "outputs": [],
- "source": [
- "import logging\n",
- "import sys\n",
- "\n",
- "logging.basicConfig(stream=sys.stdout, level=logging.INFO)\n",
- "logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout))"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "6ea1f66d-10ed-4417-bdcb-f8a894836ea5",
- "metadata": {},
- "outputs": [],
- "source": [
- "from gpt_index import GPTListIndex, SlackReader\n",
- "from IPython.display import Markdown, display\n",
- "import os"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "da90589a-fb44-4ec6-9706-753dba4fa968",
- "metadata": {},
- "outputs": [],
- "source": [
- "slack_token = os.getenv(\"SLACK_BOT_TOKEN\")\n",
- "channel_ids = [\"<channel_id>\"]\n",
- "documents = SlackReader(slack_token=slack_token).load_data(channel_ids=channel_ids)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "341295df-2029-4728-ab3d-2ee178a7e6f1",
- "metadata": {},
- "outputs": [],
- "source": [
- "index = GPTListIndex(documents)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "01c26b9d-49ec-4a6e-9c61-5c06bb86bbb2",
- "metadata": {},
- "outputs": [],
- "source": [
- "# set Logging to DEBUG for more detailed outputs\n",
- "response = index.query(\"<query_text>\")"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "f160c678-2fb5-4d6d-b2bc-87abb61cfdec",
- "metadata": {},
- "outputs": [],
- "source": [
- "display(Markdown(f\"<b>{response}</b>\"))"
- ]
- }
- ],
- "metadata": {
- "kernelspec": {
- "display_name": "Python 3 (ipykernel)",
- "language": "python",
- "name": "python3"
- },
- "language_info": {
- "codemirror_mode": {
- "name": "ipython",
- "version": 3
- },
- "file_extension": ".py",
- "mimetype": "text/x-python",
- "name": "python",
- "nbconvert_exporter": "python",
- "pygments_lexer": "ipython3",
- "version": "3.11.1"
- }
- },
- "nbformat": 4,
- "nbformat_minor": 5
-}
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "id": "effeb5a7-8544-4ee4-8c11-bad0d8165394",
+ "metadata": {},
+ "source": [
+ "# Slack Demo\n",
+ "Demonstrates our Slack data connector"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "dc664882",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import logging\n",
+ "import sys\n",
+ "\n",
+ "logging.basicConfig(stream=sys.stdout, level=logging.INFO)\n",
+ "logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout))"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "6ea1f66d-10ed-4417-bdcb-f8a894836ea5",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from llama_index import GPTListIndex, SlackReader\n",
+ "from IPython.display import Markdown, display\n",
+ "import os"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "da90589a-fb44-4ec6-9706-753dba4fa968",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "slack_token = os.getenv(\"SLACK_BOT_TOKEN\")\n",
+ "channel_ids = [\"<channel_id>\"]\n",
+ "documents = SlackReader(slack_token=slack_token).load_data(channel_ids=channel_ids)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "341295df-2029-4728-ab3d-2ee178a7e6f1",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "index = GPTListIndex(documents)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "01c26b9d-49ec-4a6e-9c61-5c06bb86bbb2",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# set Logging to DEBUG for more detailed outputs\n",
+ "response = index.query(\"<query_text>\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "f160c678-2fb5-4d6d-b2bc-87abb61cfdec",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "display(Markdown(f\"<b>{response}</b>\"))"
+ ]
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3 (ipykernel)",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.11.1"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}
\ No newline at end of file
diff --git a/examples/data_connectors/TwitterDemo.ipynb b/examples/data_connectors/TwitterDemo.ipynb
index a39b66b8256b0e7cf5cb6acd5c05eadaf075d2d3..46448a25b564894e10f080dc21f3dfcabb969f2b 100644
--- a/examples/data_connectors/TwitterDemo.ipynb
+++ b/examples/data_connectors/TwitterDemo.ipynb
@@ -1,105 +1,105 @@
{
- "cells": [
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "367a6eae",
- "metadata": {},
- "outputs": [],
- "source": [
- "import logging\n",
- "import sys\n",
- "\n",
- "logging.basicConfig(stream=sys.stdout, level=logging.INFO)\n",
- "logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout))"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "21d03e9b-8a47-45b2-ab27-295b7397ecad",
- "metadata": {},
- "outputs": [],
- "source": [
- "from gpt_index import GPTSimpleVectorIndex, TwitterTweetReader\n",
- "from IPython.display import Markdown, display\n",
- "import os"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "ef5d2334-9661-4648-a823-a335ea277826",
- "metadata": {},
- "outputs": [],
- "source": [
- "# create an app in https://developer.twitter.com/en/apps\n",
- "BEARER_TOKEN = \"<bearer_token>\""
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "1d6a1153-1383-4aaf-b39d-72c1fc9cc428",
- "metadata": {},
- "outputs": [],
- "source": [
- "# create reader, specify twitter handles\n",
- "reader = TwitterTweetReader(BEARER_TOKEN)\n",
- "documents = reader.load_data([\"@twitter_handle1\"])"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "ca319024-88e7-424f-b1d8-4daa06c6bc6a",
- "metadata": {},
- "outputs": [],
- "source": [
- "index = GPTSimpleVectorIndex(documents)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "78680a17-9088-419e-97cf-ac3d5783a709",
- "metadata": {},
- "outputs": [],
- "source": [
- "# set Logging to DEBUG for more detailed outputs\n",
- "response = index.query(\"<query_text>\")"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "2f0f92a7-cdd9-478f-9765-0a122d6e8508",
- "metadata": {},
- "outputs": [],
- "source": [
- "display(Markdown(f\"<b>{response}</b>\"))"
- ]
- }
- ],
- "metadata": {
- "kernelspec": {
- "display_name": "Python 3 (ipykernel)",
- "language": "python",
- "name": "python3"
- },
- "language_info": {
- "codemirror_mode": {
- "name": "ipython",
- "version": 3
- },
- "file_extension": ".py",
- "mimetype": "text/x-python",
- "name": "python",
- "nbconvert_exporter": "python",
- "pygments_lexer": "ipython3",
- "version": "3.11.1"
- }
- },
- "nbformat": 4,
- "nbformat_minor": 5
-}
+ "cells": [
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "367a6eae",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import logging\n",
+ "import sys\n",
+ "\n",
+ "logging.basicConfig(stream=sys.stdout, level=logging.INFO)\n",
+ "logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout))"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "21d03e9b-8a47-45b2-ab27-295b7397ecad",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from llama_index import GPTSimpleVectorIndex, TwitterTweetReader\n",
+ "from IPython.display import Markdown, display\n",
+ "import os"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "ef5d2334-9661-4648-a823-a335ea277826",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# create an app in https://developer.twitter.com/en/apps\n",
+ "BEARER_TOKEN = \"<bearer_token>\""
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "1d6a1153-1383-4aaf-b39d-72c1fc9cc428",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# create reader, specify twitter handles\n",
+ "reader = TwitterTweetReader(BEARER_TOKEN)\n",
+ "documents = reader.load_data([\"@twitter_handle1\"])"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "ca319024-88e7-424f-b1d8-4daa06c6bc6a",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "index = GPTSimpleVectorIndex(documents)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "78680a17-9088-419e-97cf-ac3d5783a709",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# set Logging to DEBUG for more detailed outputs\n",
+ "response = index.query(\"<query_text>\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "2f0f92a7-cdd9-478f-9765-0a122d6e8508",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "display(Markdown(f\"<b>{response}</b>\"))"
+ ]
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3 (ipykernel)",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.11.1"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}
\ No newline at end of file
diff --git a/examples/data_connectors/WeaviateDemo.ipynb b/examples/data_connectors/WeaviateDemo.ipynb
index 7a46df05fcf9094b3717a26a8765789fc0b680f8..751c1df2e01745e9e4e71a086c64028b0ea4b9ed 100644
--- a/examples/data_connectors/WeaviateDemo.ipynb
+++ b/examples/data_connectors/WeaviateDemo.ipynb
@@ -1,177 +1,177 @@
{
- "cells": [
- {
- "cell_type": "markdown",
- "id": "36e7bb96-0c27-47e9-a525-c11f40be3b86",
- "metadata": {},
- "source": [
- "# Weaviate"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "38ca1434",
- "metadata": {},
- "outputs": [],
- "source": [
- "import logging\n",
- "import sys\n",
- "\n",
- "logging.basicConfig(stream=sys.stdout, level=logging.INFO)\n",
- "logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout))"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 3,
- "id": "d99bc57b-85df-46ac-8262-2409344af428",
- "metadata": {},
- "outputs": [],
- "source": [
- "import weaviate\n",
- "from gpt_index.readers.weaviate import WeaviateReader"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "fec36c7a-3766-4167-890e-b93adb831a64",
- "metadata": {},
- "outputs": [],
- "source": [
- "# See https://weaviate.io/developers/weaviate/current/client-libraries/python.html\n",
- "# for more details on authentication\n",
- "resource_owner_config = weaviate.AuthClientPassword(\n",
- " username = \"<username>\", \n",
- " password = \"<password>\", \n",
- ")\n",
- "\n",
- "# initialize reader\n",
- "reader = WeaviateReader(\"https://<cluster-id>.semi.network/\", auth_client_secret=resource_owner_config)"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "ce9f299c-4f0a-4bca-bc90-79848f02b381",
- "metadata": {},
- "source": [
- "You have two options for the Weaviate reader: 1) directly specify the class_name and properties, or 2) input the raw graphql_query. Examples are shown below."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 4,
- "id": "b92d69a1-d39f-45cf-a136-cb9c2f2f5cdf",
- "metadata": {},
- "outputs": [],
- "source": [
- "# 1) load data using class_name and properties\n",
- "# docs = reader.load_data(\n",
- "# class_name=\"Author\", properties=[\"name\", \"description\"], separate_documents=True\n",
- "# )\n",
- "\n",
- "documents = reader.load_data(\n",
- " class_name=\"<class_name>\", \n",
- " properties=[\"property1\", \"property2\", \"...\"], \n",
- " separate_documents=True\n",
- ")"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 4,
- "id": "722b5d47-9897-4c54-9734-259ab0c1634c",
- "metadata": {},
- "outputs": [],
- "source": [
- "# 2) example GraphQL query\n",
- "# query = \"\"\"\n",
- "# {\n",
- "# Get {\n",
- "# Author {\n",
- "# name\n",
- "# description\n",
- "# }\n",
- "# }\n",
- "# }\n",
- "# \"\"\"\n",
- "# docs = reader.load_data(graphql_query=query, separate_documents=True)\n",
- "\n",
- "query = \"\"\"\n",
- "{\n",
- " Get {\n",
- " <class_name> {\n",
- " <property1>\n",
- " <property2>\n",
- " ...\n",
- " }\n",
- " }\n",
- "}\n",
- "\"\"\"\n",
- "\n",
- "documents = reader.load_data(graphql_query=query, separate_documents=True)"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "169b4273-eb20-4d06-9ffe-71320f4570f6",
- "metadata": {},
- "source": [
- "### Create index"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "92599a0a-93ba-4c93-80f1-9acae0663c34",
- "metadata": {},
- "outputs": [],
- "source": [
- "index = GPTListIndex(documents)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "52d93c3f-a08d-4637-98bc-0c3cc693c563",
- "metadata": {},
- "outputs": [],
- "source": [
- "# set Logging to DEBUG for more detailed outputs\n",
- "response = index.query(\"<query_text>\")"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "771b42be-4108-43a0-a1b4-b259a7819936",
- "metadata": {},
- "outputs": [],
- "source": [
- "display(Markdown(f\"<b>{response}</b>\"))"
- ]
- }
- ],
- "metadata": {
- "kernelspec": {
- "display_name": "Python 3 (ipykernel)",
- "language": "python",
- "name": "python3"
- },
- "language_info": {
- "codemirror_mode": {
- "name": "ipython",
- "version": 3
- },
- "file_extension": ".py",
- "mimetype": "text/x-python",
- "name": "python",
- "nbconvert_exporter": "python",
- "pygments_lexer": "ipython3",
- "version": "3.11.1"
- }
- },
- "nbformat": 4,
- "nbformat_minor": 5
-}
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "id": "36e7bb96-0c27-47e9-a525-c11f40be3b86",
+ "metadata": {},
+ "source": [
+ "# Weaviate"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "38ca1434",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import logging\n",
+ "import sys\n",
+ "\n",
+ "logging.basicConfig(stream=sys.stdout, level=logging.INFO)\n",
+ "logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout))"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 3,
+ "id": "d99bc57b-85df-46ac-8262-2409344af428",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import weaviate\n",
+ "from llama_index.readers.weaviate import WeaviateReader"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "fec36c7a-3766-4167-890e-b93adb831a64",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# See https://weaviate.io/developers/weaviate/current/client-libraries/python.html\n",
+ "# for more details on authentication\n",
+ "resource_owner_config = weaviate.AuthClientPassword(\n",
+ " username = \"<username>\", \n",
+ " password = \"<password>\", \n",
+ ")\n",
+ "\n",
+ "# initialize reader\n",
+ "reader = WeaviateReader(\"https://<cluster-id>.semi.network/\", auth_client_secret=resource_owner_config)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "ce9f299c-4f0a-4bca-bc90-79848f02b381",
+ "metadata": {},
+ "source": [
+ "You have two options for the Weaviate reader: 1) directly specify the class_name and properties, or 2) input the raw graphql_query. Examples are shown below."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 4,
+ "id": "b92d69a1-d39f-45cf-a136-cb9c2f2f5cdf",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# 1) load data using class_name and properties\n",
+ "# docs = reader.load_data(\n",
+ "# class_name=\"Author\", properties=[\"name\", \"description\"], separate_documents=True\n",
+ "# )\n",
+ "\n",
+ "documents = reader.load_data(\n",
+ " class_name=\"<class_name>\", \n",
+ " properties=[\"property1\", \"property2\", \"...\"], \n",
+ " separate_documents=True\n",
+ ")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 4,
+ "id": "722b5d47-9897-4c54-9734-259ab0c1634c",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# 2) example GraphQL query\n",
+ "# query = \"\"\"\n",
+ "# {\n",
+ "# Get {\n",
+ "# Author {\n",
+ "# name\n",
+ "# description\n",
+ "# }\n",
+ "# }\n",
+ "# }\n",
+ "# \"\"\"\n",
+ "# docs = reader.load_data(graphql_query=query, separate_documents=True)\n",
+ "\n",
+ "query = \"\"\"\n",
+ "{\n",
+ " Get {\n",
+ " <class_name> {\n",
+ " <property1>\n",
+ " <property2>\n",
+ " ...\n",
+ " }\n",
+ " }\n",
+ "}\n",
+ "\"\"\"\n",
+ "\n",
+ "documents = reader.load_data(graphql_query=query, separate_documents=True)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "169b4273-eb20-4d06-9ffe-71320f4570f6",
+ "metadata": {},
+ "source": [
+ "### Create index"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "92599a0a-93ba-4c93-80f1-9acae0663c34",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "index = GPTListIndex(documents)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "52d93c3f-a08d-4637-98bc-0c3cc693c563",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# set Logging to DEBUG for more detailed outputs\n",
+ "response = index.query(\"<query_text>\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "771b42be-4108-43a0-a1b4-b259a7819936",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "display(Markdown(f\"<b>{response}</b>\"))"
+ ]
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3 (ipykernel)",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.11.1"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}
\ No newline at end of file
diff --git a/examples/data_connectors/WebPageDemo.ipynb b/examples/data_connectors/WebPageDemo.ipynb
index a503928ce3d374bddfb5beaf1ee645f505a02dec..ba456ee22466134d49ec7178878f669c5abeeead 100644
--- a/examples/data_connectors/WebPageDemo.ipynb
+++ b/examples/data_connectors/WebPageDemo.ipynb
@@ -1,221 +1,221 @@
{
- "cells": [
- {
- "cell_type": "markdown",
- "id": "30146ad2-f165-4f4b-ae07-fe6597a2964f",
- "metadata": {},
- "source": [
- "# Web Page Demo\n",
- "\n",
- "Demonstrates our web page reader."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "3c39063b",
- "metadata": {},
- "outputs": [],
- "source": [
- "import logging\n",
- "import sys\n",
- "\n",
- "logging.basicConfig(stream=sys.stdout, level=logging.INFO)\n",
- "logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout))"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "2315a154-f72d-4447-b1eb-cde9b66868cb",
- "metadata": {},
- "source": [
- "#### Using SimpleWebPageReader"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "87bf7ecd-50cd-47da-9f0e-bc48d7ae45d8",
- "metadata": {},
- "outputs": [],
- "source": [
- "from gpt_index import GPTListIndex, SimpleWebPageReader\n",
- "from IPython.display import Markdown, display\n",
- "import os"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "b6de3929-51eb-4064-b4b6-c203bb6debc4",
- "metadata": {},
- "outputs": [],
- "source": [
- "# NOTE: the html_to_text=True option requires html2text to be installed"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "663403de-2e6e-4340-ab8f-8ee681bc06aa",
- "metadata": {},
- "outputs": [],
- "source": [
- "documents = SimpleWebPageReader(html_to_text=True).load_data([\"http://paulgraham.com/worked.html\"])"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "b8cd183a-2423-4a3e-ad92-dfe89ed5454e",
- "metadata": {},
- "outputs": [],
- "source": [
- "documents[0]"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "26854cc3-af61-4910-ab6b-3bed6acfb447",
- "metadata": {},
- "outputs": [],
- "source": [
- "index = GPTListIndex(documents)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "5cfdf87a-97cb-481f-ad51-be5bf8b5217f",
- "metadata": {},
- "outputs": [],
- "source": [
- "# set Logging to DEBUG for more detailed outputs\n",
- "response = index.query(\"What did the author do growing up?\")"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "7278d033-cae3-4ddf-96bd-75ea570ca53f",
- "metadata": {},
- "outputs": [],
- "source": [
- "display(Markdown(f\"<b>{response}</b>\"))"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "2708dc99-0e4d-4c7e-b180-8392286d87c2",
- "metadata": {},
- "source": [
- "#### Using TrafilaturaWebReader"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "aa2d54c6-c694-4852-a743-165e4777bd56",
- "metadata": {},
- "outputs": [],
- "source": [
- "from gpt_index import TrafilaturaWebReader"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "46854f2f-426e-40a3-a87f-5fb51f90e14c",
- "metadata": {},
- "outputs": [],
- "source": [
- "documents = TrafilaturaWebReader().load_data([\"http://paulgraham.com/worked.html\"])"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "80752ad3-1ed8-4695-9247-22efbe475746",
- "metadata": {},
- "outputs": [],
- "source": [
- "index = GPTListIndex(documents)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "8cc9b154-1dcf-479b-b49b-251874aea506",
- "metadata": {},
- "outputs": [],
- "source": [
- "# set Logging to DEBUG for more detailed outputs\n",
- "response = index.query(\"What did the author do growing up?\")"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "971b6415-8bcd-4d8b-a1de-9b7ada3cd392",
- "metadata": {},
- "outputs": [],
- "source": [
- "display(Markdown(f\"<b>{response}</b>\"))"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "b2b6d07c",
- "metadata": {},
- "source": [
- "### Using RssReader"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "a5ad5ca8",
- "metadata": {},
- "outputs": [],
- "source": [
- "from gpt_index import GPTListIndex, RssReader\n",
- "\n",
- "documents = RssReader().load_data([\n",
- " \"https://rss.nytimes.com/services/xml/rss/nyt/HomePage.xml\"\n",
- " ])\n",
- "\n",
- "index = GPTListIndex(documents)\n",
- "\n",
- "# set Logging to DEBUG for more detailed outputs\n",
- "response = index.query(\"What happened in the news today?\")"
- ]
- }
- ],
- "metadata": {
- "kernelspec": {
- "display_name": "Python 3 (ipykernel)",
- "language": "python",
- "name": "python3"
- },
- "language_info": {
- "codemirror_mode": {
- "name": "ipython",
- "version": 3
- },
- "file_extension": ".py",
- "mimetype": "text/x-python",
- "name": "python",
- "nbconvert_exporter": "python",
- "pygments_lexer": "ipython3",
- "version": "3.11.1"
- },
- "vscode": {
- "interpreter": {
- "hash": "c32397a35d2e76e766f80c3872b208f0c0029e8a6a9b8e2a8fe7b1641cfa009b"
- }
- }
- },
- "nbformat": 4,
- "nbformat_minor": 5
-}
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "id": "30146ad2-f165-4f4b-ae07-fe6597a2964f",
+ "metadata": {},
+ "source": [
+ "# Web Page Demo\n",
+ "\n",
+ "Demonstrates our web page reader."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "3c39063b",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import logging\n",
+ "import sys\n",
+ "\n",
+ "logging.basicConfig(stream=sys.stdout, level=logging.INFO)\n",
+ "logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout))"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "2315a154-f72d-4447-b1eb-cde9b66868cb",
+ "metadata": {},
+ "source": [
+ "#### Using SimpleWebPageReader"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "87bf7ecd-50cd-47da-9f0e-bc48d7ae45d8",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from llama_index import GPTListIndex, SimpleWebPageReader\n",
+ "from IPython.display import Markdown, display\n",
+ "import os"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "b6de3929-51eb-4064-b4b6-c203bb6debc4",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# NOTE: the html_to_text=True option requires html2text to be installed"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "663403de-2e6e-4340-ab8f-8ee681bc06aa",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "documents = SimpleWebPageReader(html_to_text=True).load_data([\"http://paulgraham.com/worked.html\"])"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "b8cd183a-2423-4a3e-ad92-dfe89ed5454e",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "documents[0]"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "26854cc3-af61-4910-ab6b-3bed6acfb447",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "index = GPTListIndex(documents)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "5cfdf87a-97cb-481f-ad51-be5bf8b5217f",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# set Logging to DEBUG for more detailed outputs\n",
+ "response = index.query(\"What did the author do growing up?\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "7278d033-cae3-4ddf-96bd-75ea570ca53f",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "display(Markdown(f\"<b>{response}</b>\"))"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "2708dc99-0e4d-4c7e-b180-8392286d87c2",
+ "metadata": {},
+ "source": [
+ "#### Using TrafilaturaWebReader"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "aa2d54c6-c694-4852-a743-165e4777bd56",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from llama_index import TrafilaturaWebReader"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "46854f2f-426e-40a3-a87f-5fb51f90e14c",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "documents = TrafilaturaWebReader().load_data([\"http://paulgraham.com/worked.html\"])"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "80752ad3-1ed8-4695-9247-22efbe475746",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "index = GPTListIndex(documents)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "8cc9b154-1dcf-479b-b49b-251874aea506",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# set Logging to DEBUG for more detailed outputs\n",
+ "response = index.query(\"What did the author do growing up?\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "971b6415-8bcd-4d8b-a1de-9b7ada3cd392",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "display(Markdown(f\"<b>{response}</b>\"))"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "b2b6d07c",
+ "metadata": {},
+ "source": [
+ "### Using RssReader"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "a5ad5ca8",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from llama_index import GPTListIndex, RssReader\n",
+ "\n",
+ "documents = RssReader().load_data([\n",
+ " \"https://rss.nytimes.com/services/xml/rss/nyt/HomePage.xml\"\n",
+ " ])\n",
+ "\n",
+ "index = GPTListIndex(documents)\n",
+ "\n",
+ "# set Logging to DEBUG for more detailed outputs\n",
+ "response = index.query(\"What happened in the news today?\")"
+ ]
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3 (ipykernel)",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.11.1"
+ },
+ "vscode": {
+ "interpreter": {
+ "hash": "c32397a35d2e76e766f80c3872b208f0c0029e8a6a9b8e2a8fe7b1641cfa009b"
+ }
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}
\ No newline at end of file
diff --git a/examples/gatsby/TestGatsby.ipynb b/examples/gatsby/TestGatsby.ipynb
index d4b8d0c5eb7a3db8575c0f6af17a3581359be484..0d8b6f88d8ebf8e0d41264ef4865ae8dd73dc453 100644
--- a/examples/gatsby/TestGatsby.ipynb
+++ b/examples/gatsby/TestGatsby.ipynb
@@ -1,184 +1,184 @@
{
- "cells": [
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "ffeb4eee",
- "metadata": {},
- "outputs": [],
- "source": [
- "import logging\n",
- "import sys\n",
- "\n",
- "logging.basicConfig(stream=sys.stdout, level=logging.INFO)\n",
- "logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout))"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "f1a9eb90-335c-4214-8bb6-fd1edbe3ccbd",
- "metadata": {},
- "outputs": [],
- "source": [
- "# My OpenAI Key\n",
- "import os\n",
- "os.environ['OPENAI_API_KEY'] = \"INSERT OPENAI KEY\""
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 6,
- "id": "8d0b2364-4806-4656-81e7-3f6e4b910b5b",
- "metadata": {},
- "outputs": [],
- "source": [
- "from gpt_index import GPTTreeIndex, SimpleDirectoryReader\n",
- "from IPython.display import Markdown, display"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 2,
- "id": "1298bbb4-c99e-431e-93ef-eb32c0a2fc2a",
- "metadata": {
- "tags": []
- },
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "> Building index from nodes: 9 chunks\n",
- "0/95\n",
- "10/95\n",
- "20/95\n",
- "30/95\n",
- "40/95\n",
- "50/95\n",
- "60/95\n",
- "70/95\n",
- "80/95\n",
- "90/95\n",
- "> [build_index_from_documents] Total token usage: 34226 tokens\n"
- ]
- }
- ],
- "source": [
- "documents = SimpleDirectoryReader('data').load_data()\n",
- "index = GPTTreeIndex(documents)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 3,
- "id": "0b4fe9b6-5762-4e86-b51e-aac45d3ecdb1",
- "metadata": {},
- "outputs": [],
- "source": [
- "index.save_to_disk('index_gatsby.json')"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 4,
- "id": "5eec265d-211b-4f26-b05b-5b4e7072bc6e",
- "metadata": {},
- "outputs": [],
- "source": [
- "# try loading\n",
- "new_index = GPTTreeIndex.load_from_disk('index_gatsby.json')"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 7,
- "id": "68c9ebfe-b1b6-4f4e-9278-174346de8c90",
- "metadata": {
- "tags": []
- },
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "> Starting query: What did the narrator do after getting back to Chicago?\n",
- ">[Level 0] Selected node: [8]/[8]\n",
- ">[Level 1] Selected node: [8]/[8]\n",
- "> [query] Total token usage: 6058 tokens\n"
- ]
- }
- ],
- "source": [
- "# set Logging to DEBUG for more detailed outputs\n",
- "\n",
- "response = new_index.query(\"What did the narrator do after getting back to Chicago?\")"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 8,
- "id": "91581e60-6051-40ae-bba6-8fa08ffbb728",
- "metadata": {},
- "outputs": [
- {
- "data": {
- "text/markdown": [
- "<b>The narrator returned to his home in Chicago and began calling people to inform them of Gatsby's funeral. He was worried that the funeral would draw a sightseeing crowd and wanted to keep it private. He was relieved when Klipspringer called and promised to tell anyone who might be interested about the funeral. He then asked Klipspringer to commit to attending the funeral, but Klipspringer hesitated.</b>"
- ],
- "text/plain": [
- "<IPython.core.display.Markdown object>"
- ]
- },
- "metadata": {},
- "output_type": "display_data"
- }
- ],
- "source": [
- "display(Markdown(f\"<b>{response}</b>\"))"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "ca10a9c1-9dff-476d-b218-3208a1b8e7f6",
- "metadata": {},
- "outputs": [],
- "source": [
- "# GPT is confused by the text evidence\n",
- "response = new_index.query(\"What did Gatsby do before he met Daisy?\")"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "4fc3f18a-0ef9-453c-acf8-7aedd784cdcf",
- "metadata": {},
- "outputs": [],
- "source": [
- "display(Markdown(f\"<b>{response}</b>\"))"
- ]
- }
- ],
- "metadata": {
- "kernelspec": {
- "display_name": "Python 3 (ipykernel)",
- "language": "python",
- "name": "python3"
- },
- "language_info": {
- "codemirror_mode": {
- "name": "ipython",
- "version": 3
- },
- "file_extension": ".py",
- "mimetype": "text/x-python",
- "name": "python",
- "nbconvert_exporter": "python",
- "pygments_lexer": "ipython3",
- "version": "3.11.1"
- }
- },
- "nbformat": 4,
- "nbformat_minor": 5
-}
+ "cells": [
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "ffeb4eee",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import logging\n",
+ "import sys\n",
+ "\n",
+ "logging.basicConfig(stream=sys.stdout, level=logging.INFO)\n",
+ "logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout))"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "f1a9eb90-335c-4214-8bb6-fd1edbe3ccbd",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# My OpenAI Key\n",
+ "import os\n",
+ "os.environ['OPENAI_API_KEY'] = \"INSERT OPENAI KEY\""
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 6,
+ "id": "8d0b2364-4806-4656-81e7-3f6e4b910b5b",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from llama_index import GPTTreeIndex, SimpleDirectoryReader\n",
+ "from IPython.display import Markdown, display"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 2,
+ "id": "1298bbb4-c99e-431e-93ef-eb32c0a2fc2a",
+ "metadata": {
+ "tags": []
+ },
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "> Building index from nodes: 9 chunks\n",
+ "0/95\n",
+ "10/95\n",
+ "20/95\n",
+ "30/95\n",
+ "40/95\n",
+ "50/95\n",
+ "60/95\n",
+ "70/95\n",
+ "80/95\n",
+ "90/95\n",
+ "> [build_index_from_documents] Total token usage: 34226 tokens\n"
+ ]
+ }
+ ],
+ "source": [
+ "documents = SimpleDirectoryReader('data').load_data()\n",
+ "index = GPTTreeIndex(documents)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 3,
+ "id": "0b4fe9b6-5762-4e86-b51e-aac45d3ecdb1",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "index.save_to_disk('index_gatsby.json')"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 4,
+ "id": "5eec265d-211b-4f26-b05b-5b4e7072bc6e",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# try loading\n",
+ "new_index = GPTTreeIndex.load_from_disk('index_gatsby.json')"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 7,
+ "id": "68c9ebfe-b1b6-4f4e-9278-174346de8c90",
+ "metadata": {
+ "tags": []
+ },
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "> Starting query: What did the narrator do after getting back to Chicago?\n",
+ ">[Level 0] Selected node: [8]/[8]\n",
+ ">[Level 1] Selected node: [8]/[8]\n",
+ "> [query] Total token usage: 6058 tokens\n"
+ ]
+ }
+ ],
+ "source": [
+ "# set Logging to DEBUG for more detailed outputs\n",
+ "\n",
+ "response = new_index.query(\"What did the narrator do after getting back to Chicago?\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 8,
+ "id": "91581e60-6051-40ae-bba6-8fa08ffbb728",
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/markdown": [
+ "<b>The narrator returned to his home in Chicago and began calling people to inform them of Gatsby's funeral. He was worried that the funeral would draw a sightseeing crowd and wanted to keep it private. He was relieved when Klipspringer called and promised to tell anyone who might be interested about the funeral. He then asked Klipspringer to commit to attending the funeral, but Klipspringer hesitated.</b>"
+ ],
+ "text/plain": [
+ "<IPython.core.display.Markdown object>"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ }
+ ],
+ "source": [
+ "display(Markdown(f\"<b>{response}</b>\"))"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "ca10a9c1-9dff-476d-b218-3208a1b8e7f6",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# GPT is confused by the text evidence\n",
+ "response = new_index.query(\"What did Gatsby do before he met Daisy?\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "4fc3f18a-0ef9-453c-acf8-7aedd784cdcf",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "display(Markdown(f\"<b>{response}</b>\"))"
+ ]
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3 (ipykernel)",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.11.1"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}
\ No newline at end of file
diff --git a/examples/knowledge_graph/KnowledgeGraphDemo.ipynb b/examples/knowledge_graph/KnowledgeGraphDemo.ipynb
index 0986c246c94adf50dbcf120d0950075dbc58d86c..b79a4ff6bed584e4feb660d0f926693250670217 100644
--- a/examples/knowledge_graph/KnowledgeGraphDemo.ipynb
+++ b/examples/knowledge_graph/KnowledgeGraphDemo.ipynb
@@ -1,316 +1,316 @@
{
- "cells": [
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "f1a9eb90-335c-4214-8bb6-fd1edbe3ccbd",
- "metadata": {},
- "outputs": [],
- "source": [
- "# My OpenAI Key\n",
- "import os\n",
- "os.environ['OPENAI_API_KEY'] = \"INSERT OPENAI KEY\""
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 1,
- "id": "88a9f2e3-c729-455a-a338-2f83776c1d4c",
- "metadata": {},
- "outputs": [],
- "source": [
- "import logging\n",
- "import sys\n",
- "\n",
- "logging.basicConfig(stream=sys.stdout, level=logging.INFO)"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "be3f7baa-1c0a-430b-981b-83ddca9e71f2",
- "metadata": {
- "tags": []
- },
- "source": [
- "## Using Knowledge Graph"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "75f1d565-04e8-41bc-9165-166dc89b6b47",
- "metadata": {},
- "source": [
- "#### Building the Knowledge Graph"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 2,
- "id": "8d0b2364-4806-4656-81e7-3f6e4b910b5b",
- "metadata": {},
- "outputs": [],
- "source": [
- "from gpt_index import SimpleDirectoryReader, LLMPredictor\n",
- "from gpt_index.indices.knowledge_graph.base import GPTKnowledgeGraphIndex\n",
- "from langchain import OpenAI\n",
- "from IPython.display import Markdown, display"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 21,
- "id": "1c297fd3-3424-41d8-9d0d-25fe6310ab62",
- "metadata": {
- "tags": []
- },
- "outputs": [],
- "source": [
- "documents = SimpleDirectoryReader('../paul_graham_essay/data').load_data()"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 4,
- "id": "61679142-7595-492b-8792-26cbc439caf8",
- "metadata": {},
- "outputs": [],
- "source": [
- "# define LLM\n",
- "# NOTE: at the time of demo, text-davinci-002 did not have rate-limit errors\n",
- "llm_predictor = LLMPredictor(llm=OpenAI(temperature=0, model_name=\"text-davinci-002\"))"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "370fd08f-56ff-4c24-b0c4-c93116a6d482",
- "metadata": {
- "tags": []
- },
- "outputs": [],
- "source": [
- "# NOTE: can take a while! \n",
- "index = GPTKnowledgeGraphIndex(\n",
- " documents, \n",
- " chunk_size_limit=512, \n",
- " max_triplets_per_chunk=2,\n",
- " llm_predictor=llm_predictor\n",
- ")"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 24,
- "id": "0b4fe9b6-5762-4e86-b51e-aac45d3ecdb1",
- "metadata": {},
- "outputs": [],
- "source": [
- "index.save_to_disk('index_kg.json')"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "5eec265d-211b-4f26-b05b-5b4e7072bc6e",
- "metadata": {},
- "outputs": [],
- "source": [
- "# try loading\n",
- "new_index = GPTKnowledgeGraphIndex.load_from_disk('index_kg.json', llm_predictor=llm_predictor)"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "c39a0eeb-ef16-4982-8ba8-b37c2c5f4437",
- "metadata": {},
- "source": [
- "#### Querying the Knowledge Graph"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 17,
- "id": "670300d8-d0a8-4201-bbcd-4a74b199fcdd",
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "INFO:root:> Starting query: Tell me more about Interleaf\n",
- "INFO:root:> Query keywords: ['company', 'Interleaf', 'history', 'software']\n",
- "INFO:root:> Extracted relationships: The following are knowledge triplets in the form of (subset, predicate, object):\n",
- "('Interleaf', 'made software for', 'creating documents')\n",
- "('Interleaf', 'added', 'scripting language')\n",
- "('software', 'generate', 'web sites')\n",
- "INFO:root:> Building index from nodes: 0 chunks\n",
- "INFO:root:> [query] Total LLM token usage: 312 tokens\n",
- "INFO:root:> [query] Total embedding token usage: 0 tokens\n"
- ]
- }
- ],
- "source": [
- "response = new_index.query(\n",
- " \"Tell me more about Interleaf\", \n",
- " include_text=False, \n",
- " response_mode=\"tree_summarize\"\n",
- ")"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 18,
- "id": "eecf2d57-3efa-4b0d-941a-95438d42893c",
- "metadata": {},
- "outputs": [
- {
- "data": {
- "text/markdown": [
- "<b>\n",
- "Interleaf was a software company that made software for creating documents. They later added a scripting language to their software, which allowed users to generate web sites.</b>"
- ],
- "text/plain": [
- "<IPython.core.display.Markdown object>"
- ]
- },
- "metadata": {},
- "output_type": "display_data"
- }
- ],
- "source": [
- "display(Markdown(f\"<b>{response}</b>\"))"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 15,
- "id": "bd14686d-1c53-4637-9340-3745f2121ae2",
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "INFO:root:> Starting query: Tell me more about what the author worked on at Interleaf\n",
- "INFO:root:> Query keywords: ['work', 'author', 'Interleaf']\n",
- "INFO:root:> Querying with idx: ed39a830-a116-41b9-a551-bdd348dba61d: life, we aren't consciously aware of much we're seeing. Most visual perceptio...\n",
- "INFO:root:> Querying with idx: fa1cfbb9-782b-4352-b610-cdae080b8f4f: painting that looks like a certain kind of cartoon, you know it's by Roy Lich...\n",
- "INFO:root:> Extracted relationships: The following are knowledge triplets in the form of (subset, predicate, object):\n",
- "('Interleaf', 'made software for', 'creating documents')\n",
- "('Interleaf', 'added', 'scripting language')\n",
- "INFO:root:> Building index from nodes: 0 chunks\n",
- "INFO:root:> [query] Total LLM token usage: 1254 tokens\n",
- "INFO:root:> [query] Total embedding token usage: 0 tokens\n"
- ]
- }
- ],
- "source": [
- "response = new_index.query(\n",
- " \"Tell me more about what the author worked on at Interleaf\", \n",
- " include_text=True, \n",
- " response_mode=\"tree_summarize\"\n",
- ")"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 16,
- "id": "b4c87d14-d2d8-4d80-89f6-1e5972973528",
- "metadata": {},
- "outputs": [
- {
- "data": {
- "text/markdown": [
- "<b>\n",
- "The author worked on software that allowed users to create documents, similar to Microsoft Word. The software also had a scripting language that was based on Lisp.</b>"
- ],
- "text/plain": [
- "<IPython.core.display.Markdown object>"
- ]
- },
- "metadata": {},
- "output_type": "display_data"
- }
- ],
- "source": [
- "display(Markdown(f\"<b>{response}</b>\"))"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "cd582500-584c-409a-9963-921738f1beb8",
- "metadata": {},
- "source": [
- "#### Visualizing the Graph"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 27,
- "id": "b9fe3d26-4f9a-4651-b83f-0018672a34e4",
- "metadata": {},
- "outputs": [
- {
- "data": {
- "text/html": [
- "\n",
- " <iframe\n",
- " width=\"100%\"\n",
- " height=\"600px\"\n",
- " src=\"example.html\"\n",
- " frameborder=\"0\"\n",
- " allowfullscreen\n",
- " \n",
- " ></iframe>\n",
- " "
- ],
- "text/plain": [
- "<IPython.lib.display.IFrame at 0x127e30c70>"
- ]
- },
- "execution_count": 27,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "## create graph\n",
- "from pyvis.network import Network\n",
- "\n",
- "g = new_index.get_networkx_graph()\n",
- "net = Network(notebook=True, cdn_resources=\"in_line\", directed=True)\n",
- "net.from_nx(g)\n",
- "net.show(\"example.html\")"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "7c2d09ee-22eb-475f-9aed-60f85a2b572f",
- "metadata": {},
- "outputs": [],
- "source": []
- }
- ],
- "metadata": {
- "kernelspec": {
- "display_name": "gpt_index_hinthornw",
- "language": "python",
- "name": "gpt_index_hinthornw"
- },
- "language_info": {
- "codemirror_mode": {
- "name": "ipython",
- "version": 3
- },
- "file_extension": ".py",
- "mimetype": "text/x-python",
- "name": "python",
- "nbconvert_exporter": "python",
- "pygments_lexer": "ipython3",
- "version": "3.8.4"
- }
- },
- "nbformat": 4,
- "nbformat_minor": 5
-}
+ "cells": [
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "f1a9eb90-335c-4214-8bb6-fd1edbe3ccbd",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# My OpenAI Key\n",
+ "import os\n",
+ "os.environ['OPENAI_API_KEY'] = \"INSERT OPENAI KEY\""
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 1,
+ "id": "88a9f2e3-c729-455a-a338-2f83776c1d4c",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import logging\n",
+ "import sys\n",
+ "\n",
+ "logging.basicConfig(stream=sys.stdout, level=logging.INFO)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "be3f7baa-1c0a-430b-981b-83ddca9e71f2",
+ "metadata": {
+ "tags": []
+ },
+ "source": [
+ "## Using Knowledge Graph"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "75f1d565-04e8-41bc-9165-166dc89b6b47",
+ "metadata": {},
+ "source": [
+ "#### Building the Knowledge Graph"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 2,
+ "id": "8d0b2364-4806-4656-81e7-3f6e4b910b5b",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from llama_index import SimpleDirectoryReader, LLMPredictor\n",
+ "from llama_index.indices.knowledge_graph.base import GPTKnowledgeGraphIndex\n",
+ "from langchain import OpenAI\n",
+ "from IPython.display import Markdown, display"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 21,
+ "id": "1c297fd3-3424-41d8-9d0d-25fe6310ab62",
+ "metadata": {
+ "tags": []
+ },
+ "outputs": [],
+ "source": [
+ "documents = SimpleDirectoryReader('../paul_graham_essay/data').load_data()"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 4,
+ "id": "61679142-7595-492b-8792-26cbc439caf8",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# define LLM\n",
+ "# NOTE: at the time of demo, text-davinci-002 did not have rate-limit errors\n",
+ "llm_predictor = LLMPredictor(llm=OpenAI(temperature=0, model_name=\"text-davinci-002\"))"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "370fd08f-56ff-4c24-b0c4-c93116a6d482",
+ "metadata": {
+ "tags": []
+ },
+ "outputs": [],
+ "source": [
+ "# NOTE: can take a while! \n",
+ "index = GPTKnowledgeGraphIndex(\n",
+ " documents, \n",
+ " chunk_size_limit=512, \n",
+ " max_triplets_per_chunk=2,\n",
+ " llm_predictor=llm_predictor\n",
+ ")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 24,
+ "id": "0b4fe9b6-5762-4e86-b51e-aac45d3ecdb1",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "index.save_to_disk('index_kg.json')"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "5eec265d-211b-4f26-b05b-5b4e7072bc6e",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# try loading\n",
+ "new_index = GPTKnowledgeGraphIndex.load_from_disk('index_kg.json', llm_predictor=llm_predictor)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "c39a0eeb-ef16-4982-8ba8-b37c2c5f4437",
+ "metadata": {},
+ "source": [
+ "#### Querying the Knowledge Graph"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 17,
+ "id": "670300d8-d0a8-4201-bbcd-4a74b199fcdd",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:root:> Starting query: Tell me more about Interleaf\n",
+ "INFO:root:> Query keywords: ['company', 'Interleaf', 'history', 'software']\n",
+ "INFO:root:> Extracted relationships: The following are knowledge triplets in the form of (subset, predicate, object):\n",
+ "('Interleaf', 'made software for', 'creating documents')\n",
+ "('Interleaf', 'added', 'scripting language')\n",
+ "('software', 'generate', 'web sites')\n",
+ "INFO:root:> Building index from nodes: 0 chunks\n",
+ "INFO:root:> [query] Total LLM token usage: 312 tokens\n",
+ "INFO:root:> [query] Total embedding token usage: 0 tokens\n"
+ ]
+ }
+ ],
+ "source": [
+ "response = new_index.query(\n",
+ " \"Tell me more about Interleaf\", \n",
+ " include_text=False, \n",
+ " response_mode=\"tree_summarize\"\n",
+ ")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 18,
+ "id": "eecf2d57-3efa-4b0d-941a-95438d42893c",
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/markdown": [
+ "<b>\n",
+ "Interleaf was a software company that made software for creating documents. They later added a scripting language to their software, which allowed users to generate web sites.</b>"
+ ],
+ "text/plain": [
+ "<IPython.core.display.Markdown object>"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ }
+ ],
+ "source": [
+ "display(Markdown(f\"<b>{response}</b>\"))"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 15,
+ "id": "bd14686d-1c53-4637-9340-3745f2121ae2",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:root:> Starting query: Tell me more about what the author worked on at Interleaf\n",
+ "INFO:root:> Query keywords: ['work', 'author', 'Interleaf']\n",
+ "INFO:root:> Querying with idx: ed39a830-a116-41b9-a551-bdd348dba61d: life, we aren't consciously aware of much we're seeing. Most visual perceptio...\n",
+ "INFO:root:> Querying with idx: fa1cfbb9-782b-4352-b610-cdae080b8f4f: painting that looks like a certain kind of cartoon, you know it's by Roy Lich...\n",
+ "INFO:root:> Extracted relationships: The following are knowledge triplets in the form of (subset, predicate, object):\n",
+ "('Interleaf', 'made software for', 'creating documents')\n",
+ "('Interleaf', 'added', 'scripting language')\n",
+ "INFO:root:> Building index from nodes: 0 chunks\n",
+ "INFO:root:> [query] Total LLM token usage: 1254 tokens\n",
+ "INFO:root:> [query] Total embedding token usage: 0 tokens\n"
+ ]
+ }
+ ],
+ "source": [
+ "response = new_index.query(\n",
+ " \"Tell me more about what the author worked on at Interleaf\", \n",
+ " include_text=True, \n",
+ " response_mode=\"tree_summarize\"\n",
+ ")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 16,
+ "id": "b4c87d14-d2d8-4d80-89f6-1e5972973528",
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/markdown": [
+ "<b>\n",
+ "The author worked on software that allowed users to create documents, similar to Microsoft Word. The software also had a scripting language that was based on Lisp.</b>"
+ ],
+ "text/plain": [
+ "<IPython.core.display.Markdown object>"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ }
+ ],
+ "source": [
+ "display(Markdown(f\"<b>{response}</b>\"))"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "cd582500-584c-409a-9963-921738f1beb8",
+ "metadata": {},
+ "source": [
+ "#### Visualizing the Graph"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 27,
+ "id": "b9fe3d26-4f9a-4651-b83f-0018672a34e4",
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/html": [
+ "\n",
+ " <iframe\n",
+ " width=\"100%\"\n",
+ " height=\"600px\"\n",
+ " src=\"example.html\"\n",
+ " frameborder=\"0\"\n",
+ " allowfullscreen\n",
+ " \n",
+ " ></iframe>\n",
+ " "
+ ],
+ "text/plain": [
+ "<IPython.lib.display.IFrame at 0x127e30c70>"
+ ]
+ },
+ "execution_count": 27,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "## create graph\n",
+ "from pyvis.network import Network\n",
+ "\n",
+ "g = new_index.get_networkx_graph()\n",
+ "net = Network(notebook=True, cdn_resources=\"in_line\", directed=True)\n",
+ "net.from_nx(g)\n",
+ "net.show(\"example.html\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "7c2d09ee-22eb-475f-9aed-60f85a2b572f",
+ "metadata": {},
+ "outputs": [],
+ "source": []
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "gpt_index_hinthornw",
+ "language": "python",
+ "name": "gpt_index_hinthornw"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.8.4"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}
\ No newline at end of file
diff --git a/examples/langchain_demo/LangchainDemo.ipynb b/examples/langchain_demo/LangchainDemo.ipynb
index 4004ef4a53d468f39e61f7ce5d0d1971c40650eb..04db2f3b0b44484555caee76a21dc44f83cf370e 100644
--- a/examples/langchain_demo/LangchainDemo.ipynb
+++ b/examples/langchain_demo/LangchainDemo.ipynb
@@ -1,337 +1,337 @@
{
- "cells": [
- {
- "cell_type": "markdown",
- "id": "3bf01a75-a01b-472e-bc0c-9fe97658eb46",
- "metadata": {},
- "source": [
- "## GPT Index <> Langchain Integrations\n",
- "\n",
- "This demo notebook shows how you can provide integrations between GPT Index and Langchain. It provides the following examples:\n",
- "- Using GPT Index as a callable tool with a Langchain agent\n",
- "- Using GPT Index as a memory module; this allows you to insert arbitrary amounts of conversation history with a Langchain chatbot!"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "c1568569",
- "metadata": {},
- "outputs": [],
- "source": [
- "import logging\n",
- "import sys\n",
- "\n",
- "logging.basicConfig(stream=sys.stdout, level=logging.INFO)\n",
- "logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout))"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "bb9177a4-9cb7-4211-b463-121d850b5917",
- "metadata": {},
- "source": [
- "#### Using GPT Index as a Callable Tool"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 2,
- "id": "12af1b0e-983f-4fc1-b5b4-2edeb2e8f07e",
- "metadata": {},
- "outputs": [],
- "source": [
- "from langchain.agents import Tool\n",
- "from langchain.chains.conversation.memory import ConversationBufferMemory\n",
- "from langchain import OpenAI\n",
- "from langchain.agents import initialize_agent\n",
- "\n",
- "from gpt_index import GPTSimpleVectorIndex"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 3,
- "id": "1226132b-2c5f-4073-bfdb-0e36c681c12f",
- "metadata": {},
- "outputs": [],
- "source": [
- "index = GPTSimpleVectorIndex.load_from_disk('../vector_indices/index_simple.json')"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 4,
- "id": "8c9b3567-c95c-473d-afc0-516b5f35e197",
- "metadata": {},
- "outputs": [],
- "source": [
- "tools = [\n",
- " Tool(\n",
- " name = \"GPT Index\",\n",
- " func=lambda q: str(index.query(q)),\n",
- " description=\"useful for when you want to answer questions about the author. The input to this tool should be a complete english sentence.\",\n",
- " return_direct=True\n",
- " ),\n",
- "]"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 5,
- "id": "374e9f69-1f75-4a62-afdc-22f748d4bddd",
- "metadata": {},
- "outputs": [],
- "source": [
- "# set Logging to DEBUG for more detailed outputs\n",
- "memory = ConversationBufferMemory(memory_key=\"chat_history\")\n",
- "llm=OpenAI(temperature=0)\n",
- "agent_chain = initialize_agent(tools, llm, agent=\"conversational-react-description\", memory=memory)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 6,
- "id": "579fbc9f-9f13-416c-bde4-7e56fb899727",
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "\n",
- "\n",
- "\u001b[1m> Entering new AgentExecutor chain...\u001b[0m\n",
- "\u001b[32;1m\u001b[1;3m\n",
- "Thought: Do I need to use a tool? No\n",
- "AI: Hi Bob, nice to meet you! How can I help you today?\u001b[0m\n",
- "\n",
- "\u001b[1m> Finished chain.\u001b[0m\n"
- ]
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "id": "3bf01a75-a01b-472e-bc0c-9fe97658eb46",
+ "metadata": {},
+ "source": [
+ "## GPT Index <> Langchain Integrations\n",
+ "\n",
+ "This demo notebook shows how you can provide integrations between GPT Index and Langchain. It provides the following examples:\n",
+ "- Using GPT Index as a callable tool with a Langchain agent\n",
+ "- Using GPT Index as a memory module; this allows you to insert arbitrary amounts of conversation history with a Langchain chatbot!"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "c1568569",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import logging\n",
+ "import sys\n",
+ "\n",
+ "logging.basicConfig(stream=sys.stdout, level=logging.INFO)\n",
+ "logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout))"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "bb9177a4-9cb7-4211-b463-121d850b5917",
+ "metadata": {},
+ "source": [
+ "#### Using GPT Index as a Callable Tool"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 2,
+ "id": "12af1b0e-983f-4fc1-b5b4-2edeb2e8f07e",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from langchain.agents import Tool\n",
+ "from langchain.chains.conversation.memory import ConversationBufferMemory\n",
+ "from langchain import OpenAI\n",
+ "from langchain.agents import initialize_agent\n",
+ "\n",
+ "from llama_index import GPTSimpleVectorIndex"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 3,
+ "id": "1226132b-2c5f-4073-bfdb-0e36c681c12f",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "index = GPTSimpleVectorIndex.load_from_disk('../vector_indices/index_simple.json')"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 4,
+ "id": "8c9b3567-c95c-473d-afc0-516b5f35e197",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "tools = [\n",
+ " Tool(\n",
+ " name = \"GPT Index\",\n",
+ " func=lambda q: str(index.query(q)),\n",
+ " description=\"useful for when you want to answer questions about the author. The input to this tool should be a complete english sentence.\",\n",
+ " return_direct=True\n",
+ " ),\n",
+ "]"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 5,
+ "id": "374e9f69-1f75-4a62-afdc-22f748d4bddd",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# set Logging to DEBUG for more detailed outputs\n",
+ "memory = ConversationBufferMemory(memory_key=\"chat_history\")\n",
+ "llm=OpenAI(temperature=0)\n",
+ "agent_chain = initialize_agent(tools, llm, agent=\"conversational-react-description\", memory=memory)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 6,
+ "id": "579fbc9f-9f13-416c-bde4-7e56fb899727",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "\n",
+ "\n",
+ "\u001b[1m> Entering new AgentExecutor chain...\u001b[0m\n",
+ "\u001b[32;1m\u001b[1;3m\n",
+ "Thought: Do I need to use a tool? No\n",
+ "AI: Hi Bob, nice to meet you! How can I help you today?\u001b[0m\n",
+ "\n",
+ "\u001b[1m> Finished chain.\u001b[0m\n"
+ ]
+ },
+ {
+ "data": {
+ "text/plain": [
+ "'Hi Bob, nice to meet you! How can I help you today?'"
+ ]
+ },
+ "execution_count": 6,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "agent_chain.run(input=\"hi, i am bob\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 7,
+ "id": "a9841c8e-f90b-4e40-a2f9-ad1e98bb9eef",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "\n",
+ "\n",
+ "\u001b[1m> Entering new AgentExecutor chain...\u001b[0m\n",
+ "\u001b[32;1m\u001b[1;3m\n",
+ "Thought: Do I need to use a tool? Yes\n",
+ "Action: GPT Index\n",
+ "Action Input: What did the author do growing up?\u001b[0m> [query] Total LLM token usage: 3841 tokens\n",
+ "> [query] Total embedding token usage: 8 tokens\n",
+ "\n",
+ "Observation: \u001b[36;1m\u001b[1;3m\n",
+ "\n",
+ "The author grew up writing short stories, programming on an IBM 1401, and building a computer kit with a friend. He also wrote simple games, a program to predict how high his model rockets would fly, and a word processor. He studied philosophy in college, but switched to AI and taught himself Lisp. He wrote a book about Lisp hacking and reverse-engineered SHRDLU. He also took art classes at Harvard and applied to art schools, but was disappointed by the lack of teaching and learning in the painting department at the Accademia. He also had experience with 19th century studio painting conventions, such as having a little stove fed with kindling and a nude model sitting as close to it as possible.\u001b[0m\n",
+ "\u001b[32;1m\u001b[1;3m\u001b[0m\n",
+ "\n",
+ "\u001b[1m> Finished chain.\u001b[0m\n"
+ ]
+ },
+ {
+ "data": {
+ "text/plain": [
+ "'\\n\\nThe author grew up writing short stories, programming on an IBM 1401, and building a computer kit with a friend. He also wrote simple games, a program to predict how high his model rockets would fly, and a word processor. He studied philosophy in college, but switched to AI and taught himself Lisp. He wrote a book about Lisp hacking and reverse-engineered SHRDLU. He also took art classes at Harvard and applied to art schools, but was disappointed by the lack of teaching and learning in the painting department at the Accademia. He also had experience with 19th century studio painting conventions, such as having a little stove fed with kindling and a nude model sitting as close to it as possible.'"
+ ]
+ },
+ "execution_count": 7,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "agent_chain.run(input=\"What did the author do growing up?\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "7c02eb88-5a4a-4694-9b77-cd46adc691f5",
+ "metadata": {},
+ "source": [
+ "#### Using GPT Index as a memory module"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 12,
+ "id": "e06a04c1-c5fa-482c-b4d7-9b3fa0f904af",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# try using GPT List Index!\n",
+ "from langchain import OpenAI\n",
+ "from langchain.agents import initialize_agent\n",
+ "\n",
+ "from llama_index import GPTListIndex, GPTIndexMemory"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 13,
+ "id": "00e6694b-25fc-4fbc-8223-9a5605dc641f",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "> [build_index_from_documents] Total LLM token usage: 0 tokens\n",
+ "> [build_index_from_documents] Total embedding token usage: 0 tokens\n"
+ ]
+ }
+ ],
+ "source": [
+ "index = GPTListIndex([])"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 14,
+ "id": "25c0b10c-bca4-49f1-9353-646a182050cf",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# set Logging to DEBUG for more detailed outputs\n",
+ "# NOTE: you can also use a conversational chain\n",
+ "memory = GPTIndexMemory(index=index, memory_key=\"chat_history\", query_kwargs={\"response_mode\": \"compact\"})\n",
+ "llm=OpenAI(temperature=0)\n",
+ "agent_chain = initialize_agent([], llm, agent=\"conversational-react-description\", memory=memory)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 15,
+ "id": "76193275-c47c-426c-b7e4-c54d31fda92d",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "> [query] Total LLM token usage: 0 tokens\n",
+ "> [query] Total embedding token usage: 0 tokens\n",
+ "\n",
+ "\n",
+ "\u001b[1m> Entering new AgentExecutor chain...\u001b[0m\n",
+ "\u001b[32;1m\u001b[1;3m\n",
+ "Thought: Do I need to use a tool? No\n",
+ "AI: Hi Bob, nice to meet you! How can I help you today?\u001b[0m\n",
+ "\n",
+ "\u001b[1m> Finished chain.\u001b[0m\n",
+ "> Adding chunk: Human: hi, i am bob\n",
+ "AI: Hi Bob, nice to meet yo...\n",
+ "> [insert] Total LLM token usage: 0 tokens\n",
+ "> [insert] Total embedding token usage: 0 tokens\n"
+ ]
+ },
+ {
+ "data": {
+ "text/plain": [
+ "'Hi Bob, nice to meet you! How can I help you today?'"
+ ]
+ },
+ "execution_count": 15,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "agent_chain.run(input=\"hi, i am bob\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 16,
+ "id": "d426239a-a38b-4ae9-838b-b6fab43970e0",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "> [query] Total LLM token usage: 64 tokens\n",
+ "> [query] Total embedding token usage: 0 tokens\n",
+ "\n",
+ "\n",
+ "\u001b[1m> Entering new AgentExecutor chain...\u001b[0m\n",
+ "\u001b[32;1m\u001b[1;3m\n",
+ "Thought: Do I need to use a tool? No\n",
+ "AI: Your name is Bob.\u001b[0m\n",
+ "\n",
+ "\u001b[1m> Finished chain.\u001b[0m\n",
+ "> Adding chunk: Human: what's my name?\n",
+ "AI: Your name is Bob....\n",
+ "> [insert] Total LLM token usage: 0 tokens\n",
+ "> [insert] Total embedding token usage: 0 tokens\n"
+ ]
+ },
+ {
+ "data": {
+ "text/plain": [
+ "'Your name is Bob.'"
+ ]
+ },
+ "execution_count": 16,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "# NOTE: the query now calls the GPTListIndex memory module. \n",
+ "agent_chain.run(input=\"what's my name?\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "ddb06074-917e-4c54-acc6-d74ffae23766",
+ "metadata": {},
+ "outputs": [],
+ "source": []
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3 (ipykernel)",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.11.1"
+ }
},
- {
- "data": {
- "text/plain": [
- "'Hi Bob, nice to meet you! How can I help you today?'"
- ]
- },
- "execution_count": 6,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "agent_chain.run(input=\"hi, i am bob\")"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 7,
- "id": "a9841c8e-f90b-4e40-a2f9-ad1e98bb9eef",
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "\n",
- "\n",
- "\u001b[1m> Entering new AgentExecutor chain...\u001b[0m\n",
- "\u001b[32;1m\u001b[1;3m\n",
- "Thought: Do I need to use a tool? Yes\n",
- "Action: GPT Index\n",
- "Action Input: What did the author do growing up?\u001b[0m> [query] Total LLM token usage: 3841 tokens\n",
- "> [query] Total embedding token usage: 8 tokens\n",
- "\n",
- "Observation: \u001b[36;1m\u001b[1;3m\n",
- "\n",
- "The author grew up writing short stories, programming on an IBM 1401, and building a computer kit with a friend. He also wrote simple games, a program to predict how high his model rockets would fly, and a word processor. He studied philosophy in college, but switched to AI and taught himself Lisp. He wrote a book about Lisp hacking and reverse-engineered SHRDLU. He also took art classes at Harvard and applied to art schools, but was disappointed by the lack of teaching and learning in the painting department at the Accademia. He also had experience with 19th century studio painting conventions, such as having a little stove fed with kindling and a nude model sitting as close to it as possible.\u001b[0m\n",
- "\u001b[32;1m\u001b[1;3m\u001b[0m\n",
- "\n",
- "\u001b[1m> Finished chain.\u001b[0m\n"
- ]
- },
- {
- "data": {
- "text/plain": [
- "'\\n\\nThe author grew up writing short stories, programming on an IBM 1401, and building a computer kit with a friend. He also wrote simple games, a program to predict how high his model rockets would fly, and a word processor. He studied philosophy in college, but switched to AI and taught himself Lisp. He wrote a book about Lisp hacking and reverse-engineered SHRDLU. He also took art classes at Harvard and applied to art schools, but was disappointed by the lack of teaching and learning in the painting department at the Accademia. He also had experience with 19th century studio painting conventions, such as having a little stove fed with kindling and a nude model sitting as close to it as possible.'"
- ]
- },
- "execution_count": 7,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "agent_chain.run(input=\"What did the author do growing up?\")"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "7c02eb88-5a4a-4694-9b77-cd46adc691f5",
- "metadata": {},
- "source": [
- "#### Using GPT Index as a memory module"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 12,
- "id": "e06a04c1-c5fa-482c-b4d7-9b3fa0f904af",
- "metadata": {},
- "outputs": [],
- "source": [
- "# try using GPT List Index!\n",
- "from langchain import OpenAI\n",
- "from langchain.agents import initialize_agent\n",
- "\n",
- "from gpt_index import GPTListIndex, GPTIndexMemory"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 13,
- "id": "00e6694b-25fc-4fbc-8223-9a5605dc641f",
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "> [build_index_from_documents] Total LLM token usage: 0 tokens\n",
- "> [build_index_from_documents] Total embedding token usage: 0 tokens\n"
- ]
- }
- ],
- "source": [
- "index = GPTListIndex([])"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 14,
- "id": "25c0b10c-bca4-49f1-9353-646a182050cf",
- "metadata": {},
- "outputs": [],
- "source": [
- "# set Logging to DEBUG for more detailed outputs\n",
- "# NOTE: you can also use a conversational chain\n",
- "memory = GPTIndexMemory(index=index, memory_key=\"chat_history\", query_kwargs={\"response_mode\": \"compact\"})\n",
- "llm=OpenAI(temperature=0)\n",
- "agent_chain = initialize_agent([], llm, agent=\"conversational-react-description\", memory=memory)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 15,
- "id": "76193275-c47c-426c-b7e4-c54d31fda92d",
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "> [query] Total LLM token usage: 0 tokens\n",
- "> [query] Total embedding token usage: 0 tokens\n",
- "\n",
- "\n",
- "\u001b[1m> Entering new AgentExecutor chain...\u001b[0m\n",
- "\u001b[32;1m\u001b[1;3m\n",
- "Thought: Do I need to use a tool? No\n",
- "AI: Hi Bob, nice to meet you! How can I help you today?\u001b[0m\n",
- "\n",
- "\u001b[1m> Finished chain.\u001b[0m\n",
- "> Adding chunk: Human: hi, i am bob\n",
- "AI: Hi Bob, nice to meet yo...\n",
- "> [insert] Total LLM token usage: 0 tokens\n",
- "> [insert] Total embedding token usage: 0 tokens\n"
- ]
- },
- {
- "data": {
- "text/plain": [
- "'Hi Bob, nice to meet you! How can I help you today?'"
- ]
- },
- "execution_count": 15,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "agent_chain.run(input=\"hi, i am bob\")"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 16,
- "id": "d426239a-a38b-4ae9-838b-b6fab43970e0",
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "> [query] Total LLM token usage: 64 tokens\n",
- "> [query] Total embedding token usage: 0 tokens\n",
- "\n",
- "\n",
- "\u001b[1m> Entering new AgentExecutor chain...\u001b[0m\n",
- "\u001b[32;1m\u001b[1;3m\n",
- "Thought: Do I need to use a tool? No\n",
- "AI: Your name is Bob.\u001b[0m\n",
- "\n",
- "\u001b[1m> Finished chain.\u001b[0m\n",
- "> Adding chunk: Human: what's my name?\n",
- "AI: Your name is Bob....\n",
- "> [insert] Total LLM token usage: 0 tokens\n",
- "> [insert] Total embedding token usage: 0 tokens\n"
- ]
- },
- {
- "data": {
- "text/plain": [
- "'Your name is Bob.'"
- ]
- },
- "execution_count": 16,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "# NOTE: the query now calls the GPTListIndex memory module. \n",
- "agent_chain.run(input=\"what's my name?\")"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "ddb06074-917e-4c54-acc6-d74ffae23766",
- "metadata": {},
- "outputs": [],
- "source": []
- }
- ],
- "metadata": {
- "kernelspec": {
- "display_name": "Python 3 (ipykernel)",
- "language": "python",
- "name": "python3"
- },
- "language_info": {
- "codemirror_mode": {
- "name": "ipython",
- "version": 3
- },
- "file_extension": ".py",
- "mimetype": "text/x-python",
- "name": "python",
- "nbconvert_exporter": "python",
- "pygments_lexer": "ipython3",
- "version": "3.11.1"
- }
- },
- "nbformat": 4,
- "nbformat_minor": 5
-}
+ "nbformat": 4,
+ "nbformat_minor": 5
+}
\ No newline at end of file
diff --git a/examples/paul_graham_essay/DavinciComparison.ipynb b/examples/paul_graham_essay/DavinciComparison.ipynb
index 3402183a5d42eb62b97d2a6cb5e48a3683d4640a..2567534a533e53189ff68829d3895f006d1b40a3 100644
--- a/examples/paul_graham_essay/DavinciComparison.ipynb
+++ b/examples/paul_graham_essay/DavinciComparison.ipynb
@@ -1,215 +1,215 @@
{
- "cells": [
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "df7cb4c1-a1b8-4e80-ad88-8878008bde89",
- "metadata": {},
- "outputs": [],
- "source": [
- "# My OpenAI Key\n",
- "import os\n",
- "os.environ['OPENAI_API_KEY'] = \"INSERT OPENAI KEY\""
- ]
- },
- {
- "cell_type": "markdown",
- "id": "7096589b-daaf-440a-b89d-b4956f2db4b2",
- "metadata": {
- "tags": []
- },
- "source": [
- "# Comparing text-davinci-002 vs. text-davinci-003\n",
- "\n",
- "Does text-davinci-003 do better?"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "d8cfbe6f-4c50-4c4f-90f9-03bb91201ef5",
- "metadata": {},
- "source": [
- "#### text-davinci-002"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 1,
- "id": "b48c6d7a-7a38-440b-8ecb-f43f9050ee54",
- "metadata": {},
- "outputs": [
- {
- "name": "stderr",
- "output_type": "stream",
- "text": [
- "None of PyTorch, TensorFlow >= 2.0, or Flax have been found. Models won't be available and only tokenizers, configuration and file/data utilities can be used.\n",
- "[nltk_data] Downloading package stopwords to\n",
- "[nltk_data] /Users/jerryliu/nltk_data...\n",
- "[nltk_data] Package stopwords is already up-to-date!\n"
- ]
- }
- ],
- "source": [
- "from gpt_index import GPTKeywordTableIndex, SimpleDirectoryReader, LLMPredictor\n",
- "from langchain import OpenAI\n",
- "from IPython.display import Markdown, display"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 2,
- "id": "708b323e-d314-4b83-864b-22a1ead60de9",
- "metadata": {},
- "outputs": [],
- "source": [
- "# load index\n",
- "llm_predictor = LLMPredictor(llm=OpenAI(temperature=0, model_name=\"text-davinci-002\"))\n",
- "index = GPTKeywordTableIndex.load_from_disk('index_table.json', llm_predictor=llm_predictor)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "2c2de13b-133f-404e-9661-2acafcdf2573",
- "metadata": {},
- "outputs": [],
- "source": [
- "response = index.query(\"What did the author do after his time at Y Combinator?\")"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 4,
- "id": "b27a05df-7c62-49e3-aa8e-f917a67f2bc1",
- "metadata": {},
- "outputs": [
- {
- "data": {
- "text/markdown": [
- "<b>\n",
- "\n",
- "After a few years, the author decided to step away from Y Combinator to focus on other projects, such as painting and writing essays. In 2013, he handed over control of Y Combinator to Sam Altman. The author's mother passed away in 2014, and after taking some time to grieve, he returned to writing essays and working on Lisp. He continued working on Lisp until 2019, when he finally completed the project.\n",
- "\n",
- "In 2015, the author decided to move to England with his family. They originally intended to only stay for a year, but ended up liking it so much that they remained there. The author wrote Bel while living in England. In 2019, he finally finished the project. After completing Bel, the author wrote a number of essays on various topics. In 2020, he started to think about what his next project should be.</b>"
- ],
- "text/plain": [
- "<IPython.core.display.Markdown object>"
- ]
- },
- "metadata": {},
- "output_type": "display_data"
- }
- ],
- "source": [
- "display(Markdown(f\"<b>{response}</b>\"))"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "5ad95ede-b0ef-46e9-b6d7-cd397d54afc3",
- "metadata": {},
- "source": [
- "#### text-davinci-003"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 1,
- "id": "be70b923-c562-430b-91ce-30b4a052463e",
- "metadata": {},
- "outputs": [
- {
- "name": "stderr",
- "output_type": "stream",
- "text": [
- "None of PyTorch, TensorFlow >= 2.0, or Flax have been found. Models won't be available and only tokenizers, configuration and file/data utilities can be used.\n",
- "[nltk_data] Downloading package stopwords to\n",
- "[nltk_data] /Users/jerryliu/nltk_data...\n",
- "[nltk_data] Package stopwords is already up-to-date!\n"
- ]
- }
- ],
- "source": [
- "from gpt_index import GPTKeywordTableIndex, SimpleDirectoryReader, LLMPredictor\n",
- "from langchain import OpenAI\n",
- "from IPython.display import Markdown, display"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 4,
- "id": "66ccf04f-0c19-4e5e-8441-837199aaac75",
- "metadata": {},
- "outputs": [],
- "source": [
- "# load index\n",
- "llm_predictor = LLMPredictor(llm=OpenAI(temperature=0, model_name=\"text-davinci-003\"))\n",
- "index = GPTKeywordTableIndex.load_from_disk('index_table.json', llm_predictor=llm_predictor)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "b610773d-879c-45c2-a503-5fb691e2b7ac",
- "metadata": {},
- "outputs": [],
- "source": [
- "response = index.query(\"What did the author do after his time at Y Combinator?\")"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 6,
- "id": "40a4ca0a-2013-44b8-a937-13fa2e9c60db",
- "metadata": {},
- "outputs": [
- {
- "data": {
- "text/markdown": [
- "<b>\n",
- "\n",
- "After his time at Y Combinator, the author returned to painting and writing essays. He moved to New York City and rented a rent-controlled apartment in Yorkville. He wrote a book on Lisp and did freelance Lisp hacking work to support himself. He also became the de facto studio assistant for Idelle Weber, an early photorealist painter. He eventually had the idea to start a company to put art galleries online, but the idea was unsuccessful. He then had the idea to write software to build online stores, which became the basis for his successful company, Viaweb. After Viaweb was acquired by Yahoo!, the author returned to painting and started writing essays online. He wrote a book of essays, Hackers & Painters, and worked on spam filters. He also bought a building in Cambridge to use as an office. He then had the idea to start Y Combinator, an investment firm that would make a larger number of smaller investments and help founders remain as CEO. He and his partner Jessica Livingston ran Y Combinator and funded a batch of startups twice a year. He also continued to write essays, cook for groups of friends, and explore the concept of invented vs discovered in software. He created the Summer Founders Program, which invited under</b>"
- ],
- "text/plain": [
- "<IPython.core.display.Markdown object>"
- ]
- },
- "metadata": {},
- "output_type": "display_data"
- }
- ],
- "source": [
- "display(Markdown(f\"<b>{response}</b>\"))"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "2298afe3-2c27-4da6-9a31-ff3478f07741",
- "metadata": {},
- "outputs": [],
- "source": []
- }
- ],
- "metadata": {
- "kernelspec": {
- "display_name": "gpt_retrieve_venv",
- "language": "python",
- "name": "gpt_retrieve_venv"
- },
- "language_info": {
- "codemirror_mode": {
- "name": "ipython",
- "version": 3
- },
- "file_extension": ".py",
- "mimetype": "text/x-python",
- "name": "python",
- "nbconvert_exporter": "python",
- "pygments_lexer": "ipython3",
- "version": "3.8.4"
- }
- },
- "nbformat": 4,
- "nbformat_minor": 5
-}
+ "cells": [
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "df7cb4c1-a1b8-4e80-ad88-8878008bde89",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# My OpenAI Key\n",
+ "import os\n",
+ "os.environ['OPENAI_API_KEY'] = \"INSERT OPENAI KEY\""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "7096589b-daaf-440a-b89d-b4956f2db4b2",
+ "metadata": {
+ "tags": []
+ },
+ "source": [
+ "# Comparing text-davinci-002 vs. text-davinci-003\n",
+ "\n",
+ "Does text-davinci-003 do better?"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "d8cfbe6f-4c50-4c4f-90f9-03bb91201ef5",
+ "metadata": {},
+ "source": [
+ "#### text-davinci-002"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 1,
+ "id": "b48c6d7a-7a38-440b-8ecb-f43f9050ee54",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "None of PyTorch, TensorFlow >= 2.0, or Flax have been found. Models won't be available and only tokenizers, configuration and file/data utilities can be used.\n",
+ "[nltk_data] Downloading package stopwords to\n",
+ "[nltk_data] /Users/jerryliu/nltk_data...\n",
+ "[nltk_data] Package stopwords is already up-to-date!\n"
+ ]
+ }
+ ],
+ "source": [
+ "from llama_index import GPTKeywordTableIndex, SimpleDirectoryReader, LLMPredictor\n",
+ "from langchain import OpenAI\n",
+ "from IPython.display import Markdown, display"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 2,
+ "id": "708b323e-d314-4b83-864b-22a1ead60de9",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# load index\n",
+ "llm_predictor = LLMPredictor(llm=OpenAI(temperature=0, model_name=\"text-davinci-002\"))\n",
+ "index = GPTKeywordTableIndex.load_from_disk('index_table.json', llm_predictor=llm_predictor)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "2c2de13b-133f-404e-9661-2acafcdf2573",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "response = index.query(\"What did the author do after his time at Y Combinator?\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 4,
+ "id": "b27a05df-7c62-49e3-aa8e-f917a67f2bc1",
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/markdown": [
+ "<b>\n",
+ "\n",
+ "After a few years, the author decided to step away from Y Combinator to focus on other projects, such as painting and writing essays. In 2013, he handed over control of Y Combinator to Sam Altman. The author's mother passed away in 2014, and after taking some time to grieve, he returned to writing essays and working on Lisp. He continued working on Lisp until 2019, when he finally completed the project.\n",
+ "\n",
+ "In 2015, the author decided to move to England with his family. They originally intended to only stay for a year, but ended up liking it so much that they remained there. The author wrote Bel while living in England. In 2019, he finally finished the project. After completing Bel, the author wrote a number of essays on various topics. In 2020, he started to think about what his next project should be.</b>"
+ ],
+ "text/plain": [
+ "<IPython.core.display.Markdown object>"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ }
+ ],
+ "source": [
+ "display(Markdown(f\"<b>{response}</b>\"))"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "5ad95ede-b0ef-46e9-b6d7-cd397d54afc3",
+ "metadata": {},
+ "source": [
+ "#### text-davinci-003"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 1,
+ "id": "be70b923-c562-430b-91ce-30b4a052463e",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "None of PyTorch, TensorFlow >= 2.0, or Flax have been found. Models won't be available and only tokenizers, configuration and file/data utilities can be used.\n",
+ "[nltk_data] Downloading package stopwords to\n",
+ "[nltk_data] /Users/jerryliu/nltk_data...\n",
+ "[nltk_data] Package stopwords is already up-to-date!\n"
+ ]
+ }
+ ],
+ "source": [
+ "from llama_index import GPTKeywordTableIndex, SimpleDirectoryReader, LLMPredictor\n",
+ "from langchain import OpenAI\n",
+ "from IPython.display import Markdown, display"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 4,
+ "id": "66ccf04f-0c19-4e5e-8441-837199aaac75",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# load index\n",
+ "llm_predictor = LLMPredictor(llm=OpenAI(temperature=0, model_name=\"text-davinci-003\"))\n",
+ "index = GPTKeywordTableIndex.load_from_disk('index_table.json', llm_predictor=llm_predictor)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "b610773d-879c-45c2-a503-5fb691e2b7ac",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "response = index.query(\"What did the author do after his time at Y Combinator?\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 6,
+ "id": "40a4ca0a-2013-44b8-a937-13fa2e9c60db",
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/markdown": [
+ "<b>\n",
+ "\n",
+ "After his time at Y Combinator, the author returned to painting and writing essays. He moved to New York City and rented a rent-controlled apartment in Yorkville. He wrote a book on Lisp and did freelance Lisp hacking work to support himself. He also became the de facto studio assistant for Idelle Weber, an early photorealist painter. He eventually had the idea to start a company to put art galleries online, but the idea was unsuccessful. He then had the idea to write software to build online stores, which became the basis for his successful company, Viaweb. After Viaweb was acquired by Yahoo!, the author returned to painting and started writing essays online. He wrote a book of essays, Hackers & Painters, and worked on spam filters. He also bought a building in Cambridge to use as an office. He then had the idea to start Y Combinator, an investment firm that would make a larger number of smaller investments and help founders remain as CEO. He and his partner Jessica Livingston ran Y Combinator and funded a batch of startups twice a year. He also continued to write essays, cook for groups of friends, and explore the concept of invented vs discovered in software. He created the Summer Founders Program, which invited under</b>"
+ ],
+ "text/plain": [
+ "<IPython.core.display.Markdown object>"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ }
+ ],
+ "source": [
+ "display(Markdown(f\"<b>{response}</b>\"))"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "2298afe3-2c27-4da6-9a31-ff3478f07741",
+ "metadata": {},
+ "outputs": [],
+ "source": []
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "gpt_retrieve_venv",
+ "language": "python",
+ "name": "gpt_retrieve_venv"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.8.4"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}
\ No newline at end of file
diff --git a/examples/paul_graham_essay/InsertDemo.ipynb b/examples/paul_graham_essay/InsertDemo.ipynb
index 6231b3634e306b84ebcb183b6d5d8caa8c837d98..860e56e951152083b4fc3a24d8bfc0758e4d5812 100644
--- a/examples/paul_graham_essay/InsertDemo.ipynb
+++ b/examples/paul_graham_essay/InsertDemo.ipynb
@@ -1,329 +1,329 @@
{
- "cells": [
- {
- "cell_type": "markdown",
- "id": "46e5110c-ed35-463e-a9f6-cff9cda6221b",
- "metadata": {},
- "source": [
- "This notebook showcases the insert capabilities of different GPT Index data structures.\n",
- "\n",
- "To see how to build the index during initialization, see `TestEssay.ipynb` instead."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "6ef7a7a6-dc10-4d94-8bdd-22b4954d365a",
- "metadata": {},
- "outputs": [],
- "source": [
- "# My OpenAI Key\n",
- "import os\n",
- "os.environ['OPENAI_API_KEY'] = \"INSERT OPENAI KEY\""
- ]
- },
- {
- "cell_type": "markdown",
- "id": "708da88f-d1b0-41d2-ad71-773300bf3ec5",
- "metadata": {},
- "source": [
- "## GPT List Insert"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "5ff6b625-fb53-4885-8fbb-0fa8dcf6ef57",
- "metadata": {
- "tags": []
- },
- "source": [
- "#### Data Prep\n",
- "Chunk up the data into sub documents that we can insert"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "a191e411-f07f-4895-9943-6a8b030abd66",
- "metadata": {},
- "outputs": [],
- "source": [
- "from gpt_index.langchain_helpers.text_splitter import TokenTextSplitter\n",
- "from gpt_index import SimpleDirectoryReader, Document"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 15,
- "id": "0a23c1a8-71ea-4b6d-ae42-5c1cf4014dff",
- "metadata": {},
- "outputs": [],
- "source": [
- "document = SimpleDirectoryReader('data').load_data()[0]\n",
- "text_splitter = TokenTextSplitter(separator=\" \", chunk_size=2048, chunk_overlap=20)\n",
- "text_chunks = text_splitter.split_text(document.text)\n",
- "doc_chunks = [Document(t) for t in text_chunks]"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "bbdd1b84-d8e4-421a-972c-389a5b0160c6",
- "metadata": {},
- "source": [
- "#### Insert into Index and Query"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 16,
- "id": "d6dcb65d-1a10-471a-8b80-d1bedf7437dc",
- "metadata": {},
- "outputs": [],
- "source": [
- "from gpt_index import GPTListIndex, SimpleDirectoryReader\n",
- "from IPython.display import Markdown, display"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 17,
- "id": "52c3839a-aa57-412b-b5cf-78c71d6dae3c",
- "metadata": {},
- "outputs": [],
- "source": [
- "# initialize blank list index\n",
- "index = GPTListIndex([])"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "3abf2203-54b6-44c3-ac98-97503b18d3ef",
- "metadata": {},
- "outputs": [],
- "source": [
- "# insert new document chunks\n",
- "for doc_chunk in doc_chunks:\n",
- " index.insert(doc_chunk)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 19,
- "id": "228ccfb8-00c5-49d1-ba6c-61a757dd76eb",
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "> Starting query: What did the author do growing up?\n"
- ]
- }
- ],
- "source": [
- "# query\n",
- "response = index.query(\"What did the author do growing up?\")"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 21,
- "id": "2580025c-7a5a-419e-84f0-b2ad7b62b6c2",
- "metadata": {},
- "outputs": [
- {
- "data": {
- "text/markdown": [
- "<b>\n",
- "\n",
- "The author worked on writing and programming and also took classes at Harvard and RISD. The author also worked at a company called Interleaf and wrote a book on Lisp. The author decided to write another book on Lisp and also started a company called Viaweb with the goal of putting art galleries online. The company eventually pivoted to focus on creating software to build online stores. The author also worked on making ecommerce software in the second half of the 90s.\n",
- "\n",
- "The author also worked on building the infrastructure of the web and wrote essays that were published online. The author also worked on spam filters and bought a building in Cambridge to use as an office. The author also had a dinner party every Thursday night.\n",
- "\n",
- "The author also worked on marketing at a Boston VC firm, writing essays, and building the infrastructure of the web. The author also started the Y Combinator program to help fund startups.</b>"
- ],
- "text/plain": [
- "<IPython.core.display.Markdown object>"
- ]
- },
- "metadata": {},
- "output_type": "display_data"
- }
- ],
- "source": [
- "display(Markdown(f\"<b>{response}</b>\"))"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "dd41721b-0388-4300-b1cd-75c1013795f4",
- "metadata": {},
- "source": [
- "## GPT Tree Insert"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "7d5ecfd2-5913-4484-9619-8825b252406b",
- "metadata": {},
- "source": [
- "#### Data Prep\n",
- "Chunk up the data into sub documents that we can insert"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 2,
- "id": "4d55ffef-a309-40ca-bbcf-231e418b6d4c",
- "metadata": {},
- "outputs": [],
- "source": [
- "from gpt_index.langchain_helpers.text_splitter import TokenTextSplitter\n",
- "from gpt_index import SimpleDirectoryReader, Document"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 2,
- "id": "2d850fe2-78e3-46de-a78d-4ef12848e41d",
- "metadata": {},
- "outputs": [],
- "source": [
- "# NOTE: we truncate to the first 30 nodes to save on cost\n",
- "document = SimpleDirectoryReader('data').load_data()[0]\n",
- "text_splitter = TokenTextSplitter(separator=\" \", chunk_size=256, chunk_overlap=20)\n",
- "text_chunks = text_splitter.split_text(document.get_text())\n",
- "doc_chunks = [Document(t) for t in text_chunks]\n",
- "\n",
- "doc_chunks = doc_chunks[:30]"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "d4f13125-5627-48f8-9035-e734849dc9da",
- "metadata": {},
- "source": [
- "#### Insert into Index and Query"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 3,
- "id": "4d631ba9-e5fe-46d1-ae10-936a4704c95e",
- "metadata": {},
- "outputs": [],
- "source": [
- "from gpt_index import GPTTreeIndex, SimpleDirectoryReader\n",
- "from IPython.display import Markdown, display"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 4,
- "id": "54454e62-4df1-482d-978b-807d4a802033",
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "> Building index from nodes: 0 chunks\n"
- ]
- }
- ],
- "source": [
- "# initialize blank tree index\n",
- "tree_index = GPTTreeIndex([])"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "422dde38-5228-499c-a9d3-49b40fd14d34",
- "metadata": {},
- "outputs": [],
- "source": [
- "# insert new document chunks\n",
- "for i, doc_chunk in enumerate(doc_chunks):\n",
- " print(f\"Inserting {i}/{len(doc_chunks)}\")\n",
- " tree_index.insert(doc_chunk)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 6,
- "id": "ecf96fc3-e30c-4ff5-93a5-132e82236806",
- "metadata": {},
- "outputs": [],
- "source": [
- "tree_index.save_to_disk('index_tree_insert.json')"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 7,
- "id": "398ed7bd-5bfd-45f4-bdb3-32512582f5fb",
- "metadata": {},
- "outputs": [],
- "source": [
- "tree_index = GPTTreeIndex.load_from_disk('index_tree_insert.json')"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "f528cded-7a43-4854-82ce-361a6610bc34",
- "metadata": {},
- "outputs": [],
- "source": [
- "# query\n",
- "response_tree = tree_index.query(\"What did the author do growing up?\")"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 9,
- "id": "c753016d-b21e-47f3-b9f3-a70943f407c5",
- "metadata": {},
- "outputs": [
- {
- "data": {
- "text/markdown": [
- "<b>The author wrote stories and tried to program computers.</b>"
- ],
- "text/plain": [
- "<IPython.core.display.Markdown object>"
- ]
- },
- "metadata": {},
- "output_type": "display_data"
- }
- ],
- "source": [
- "display(Markdown(f\"<b>{response_tree}</b>\"))"
- ]
- }
- ],
- "metadata": {
- "kernelspec": {
- "display_name": "gpt_retrieve_venv",
- "language": "python",
- "name": "gpt_retrieve_venv"
- },
- "language_info": {
- "codemirror_mode": {
- "name": "ipython",
- "version": 3
- },
- "file_extension": ".py",
- "mimetype": "text/x-python",
- "name": "python",
- "nbconvert_exporter": "python",
- "pygments_lexer": "ipython3",
- "version": "3.9.16"
- }
- },
- "nbformat": 4,
- "nbformat_minor": 5
-}
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "id": "46e5110c-ed35-463e-a9f6-cff9cda6221b",
+ "metadata": {},
+ "source": [
+ "This notebook showcases the insert capabilities of different GPT Index data structures.\n",
+ "\n",
+ "To see how to build the index during initialization, see `TestEssay.ipynb` instead."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "6ef7a7a6-dc10-4d94-8bdd-22b4954d365a",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# My OpenAI Key\n",
+ "import os\n",
+ "os.environ['OPENAI_API_KEY'] = \"INSERT OPENAI KEY\""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "708da88f-d1b0-41d2-ad71-773300bf3ec5",
+ "metadata": {},
+ "source": [
+ "## GPT List Insert"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "5ff6b625-fb53-4885-8fbb-0fa8dcf6ef57",
+ "metadata": {
+ "tags": []
+ },
+ "source": [
+ "#### Data Prep\n",
+ "Chunk up the data into sub documents that we can insert"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "a191e411-f07f-4895-9943-6a8b030abd66",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from llama_index.langchain_helpers.text_splitter import TokenTextSplitter\n",
+ "from llama_index import SimpleDirectoryReader, Document"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 15,
+ "id": "0a23c1a8-71ea-4b6d-ae42-5c1cf4014dff",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "document = SimpleDirectoryReader('data').load_data()[0]\n",
+ "text_splitter = TokenTextSplitter(separator=\" \", chunk_size=2048, chunk_overlap=20)\n",
+ "text_chunks = text_splitter.split_text(document.text)\n",
+ "doc_chunks = [Document(t) for t in text_chunks]"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "bbdd1b84-d8e4-421a-972c-389a5b0160c6",
+ "metadata": {},
+ "source": [
+ "#### Insert into Index and Query"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 16,
+ "id": "d6dcb65d-1a10-471a-8b80-d1bedf7437dc",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from llama_index import GPTListIndex, SimpleDirectoryReader\n",
+ "from IPython.display import Markdown, display"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 17,
+ "id": "52c3839a-aa57-412b-b5cf-78c71d6dae3c",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# initialize blank list index\n",
+ "index = GPTListIndex([])"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "3abf2203-54b6-44c3-ac98-97503b18d3ef",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# insert new document chunks\n",
+ "for doc_chunk in doc_chunks:\n",
+ " index.insert(doc_chunk)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 19,
+ "id": "228ccfb8-00c5-49d1-ba6c-61a757dd76eb",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "> Starting query: What did the author do growing up?\n"
+ ]
+ }
+ ],
+ "source": [
+ "# query\n",
+ "response = index.query(\"What did the author do growing up?\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 21,
+ "id": "2580025c-7a5a-419e-84f0-b2ad7b62b6c2",
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/markdown": [
+ "<b>\n",
+ "\n",
+ "The author worked on writing and programming and also took classes at Harvard and RISD. The author also worked at a company called Interleaf and wrote a book on Lisp. The author decided to write another book on Lisp and also started a company called Viaweb with the goal of putting art galleries online. The company eventually pivoted to focus on creating software to build online stores. The author also worked on making ecommerce software in the second half of the 90s.\n",
+ "\n",
+ "The author also worked on building the infrastructure of the web and wrote essays that were published online. The author also worked on spam filters and bought a building in Cambridge to use as an office. The author also had a dinner party every Thursday night.\n",
+ "\n",
+ "The author also worked on marketing at a Boston VC firm, writing essays, and building the infrastructure of the web. The author also started the Y Combinator program to help fund startups.</b>"
+ ],
+ "text/plain": [
+ "<IPython.core.display.Markdown object>"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ }
+ ],
+ "source": [
+ "display(Markdown(f\"<b>{response}</b>\"))"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "dd41721b-0388-4300-b1cd-75c1013795f4",
+ "metadata": {},
+ "source": [
+ "## GPT Tree Insert"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "7d5ecfd2-5913-4484-9619-8825b252406b",
+ "metadata": {},
+ "source": [
+ "#### Data Prep\n",
+ "Chunk up the data into sub documents that we can insert"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 2,
+ "id": "4d55ffef-a309-40ca-bbcf-231e418b6d4c",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from llama_index.langchain_helpers.text_splitter import TokenTextSplitter\n",
+ "from llama_index import SimpleDirectoryReader, Document"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 2,
+ "id": "2d850fe2-78e3-46de-a78d-4ef12848e41d",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# NOTE: we truncate to the first 30 nodes to save on cost\n",
+ "document = SimpleDirectoryReader('data').load_data()[0]\n",
+ "text_splitter = TokenTextSplitter(separator=\" \", chunk_size=256, chunk_overlap=20)\n",
+ "text_chunks = text_splitter.split_text(document.get_text())\n",
+ "doc_chunks = [Document(t) for t in text_chunks]\n",
+ "\n",
+ "doc_chunks = doc_chunks[:30]"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "d4f13125-5627-48f8-9035-e734849dc9da",
+ "metadata": {},
+ "source": [
+ "#### Insert into Index and Query"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 3,
+ "id": "4d631ba9-e5fe-46d1-ae10-936a4704c95e",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from llama_index import GPTTreeIndex, SimpleDirectoryReader\n",
+ "from IPython.display import Markdown, display"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 4,
+ "id": "54454e62-4df1-482d-978b-807d4a802033",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "> Building index from nodes: 0 chunks\n"
+ ]
+ }
+ ],
+ "source": [
+ "# initialize blank tree index\n",
+ "tree_index = GPTTreeIndex([])"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "422dde38-5228-499c-a9d3-49b40fd14d34",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# insert new document chunks\n",
+ "for i, doc_chunk in enumerate(doc_chunks):\n",
+ " print(f\"Inserting {i}/{len(doc_chunks)}\")\n",
+ " tree_index.insert(doc_chunk)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 6,
+ "id": "ecf96fc3-e30c-4ff5-93a5-132e82236806",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "tree_index.save_to_disk('index_tree_insert.json')"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 7,
+ "id": "398ed7bd-5bfd-45f4-bdb3-32512582f5fb",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "tree_index = GPTTreeIndex.load_from_disk('index_tree_insert.json')"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "f528cded-7a43-4854-82ce-361a6610bc34",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# query\n",
+ "response_tree = tree_index.query(\"What did the author do growing up?\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 9,
+ "id": "c753016d-b21e-47f3-b9f3-a70943f407c5",
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/markdown": [
+ "<b>The author wrote stories and tried to program computers.</b>"
+ ],
+ "text/plain": [
+ "<IPython.core.display.Markdown object>"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ }
+ ],
+ "source": [
+ "display(Markdown(f\"<b>{response_tree}</b>\"))"
+ ]
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "gpt_retrieve_venv",
+ "language": "python",
+ "name": "gpt_retrieve_venv"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.9.16"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}
\ No newline at end of file
diff --git a/examples/paul_graham_essay/KeywordTableComparison.ipynb b/examples/paul_graham_essay/KeywordTableComparison.ipynb
index f9d6889e19c937a70923a5dd91b2e356742668bb..b4087978194c4f3329dd2872b55835a4253753ae 100644
--- a/examples/paul_graham_essay/KeywordTableComparison.ipynb
+++ b/examples/paul_graham_essay/KeywordTableComparison.ipynb
@@ -1,428 +1,428 @@
{
- "cells": [
- {
- "cell_type": "markdown",
- "id": "a6457769-dfaf-4241-ab32-dcf901dde902",
- "metadata": {
- "tags": []
- },
- "source": [
- "## GPT Keyword Table Index Comparisons\n",
- "\n",
- "Comparing GPTSimpleKeywordTableIndex, GPTRAKEKeywordTableIndex, GPTKeywordTableIndex.\n",
- "\n",
- "- GPTSimpleKeywordTableIndex - uses simple regex to extract keywords.\n",
- "- GPTRAKEKeywordTableIndex - uses RAKE to extract keywords.\n",
- "- GPTKeywordTableIndex - uses GPT to extract keywords."
- ]
- },
- {
- "cell_type": "markdown",
- "id": "075080e5-c255-4a5c-9330-9da11532e1c8",
- "metadata": {
- "tags": []
- },
- "source": [
- "#### GPTSimpleKeywordTableIndex"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 1,
- "id": "b367b7ef-6a7d-4aee-b174-dba6ec4d2e21",
- "metadata": {},
- "outputs": [
- {
- "name": "stderr",
- "output_type": "stream",
- "text": [
- "[nltk_data] Downloading package stopwords to /home/jerry/nltk_data...\n",
- "[nltk_data] Package stopwords is already up-to-date!\n"
- ]
- }
- ],
- "source": [
- "from gpt_index import GPTSimpleKeywordTableIndex, SimpleDirectoryReader\n",
- "from IPython.display import Markdown, display"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "1f8248fa-e0bd-494a-ad68-8192ccc87696",
- "metadata": {},
- "outputs": [],
- "source": [
- "# build keyword index\n",
- "documents = SimpleDirectoryReader('data').load_data()\n",
- "index = GPTSimpleKeywordTableIndex(documents)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "53833655-0296-4bcb-b501-259b043d68b3",
- "metadata": {},
- "outputs": [],
- "source": [
- "response = index.query(\"What did the author do after his time at YC?\")"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 5,
- "id": "62bcca18-b644-4393-ad29-6c5f0424fb22",
- "metadata": {},
- "outputs": [
- {
- "data": {
- "text/markdown": [
- "<b>\n",
- "\n",
- "The author went on to write essays and work on other projects, including a new version of the Arc programming language and Hacker News. He also started painting, but stopped after a few months. In 2015, he started working on a new Lisp programming language, which he finished in 2019. The author then moved to England in 2016 with his family and continued writing essays. In 2019, he finished Bel and wrote a bunch of essays on various topics.\n",
- "\n",
- "The author also worked on building online stores in 1995 after finishing ANSI Common Lisp. He ran the software on servers and let users control it by clicking on links, which was a new concept at the time. In 1996, he co-founded Viaweb with Robert Morris, which was later acquired by Yahoo in 1998. After leaving Yahoo, the author moved back to New York and started painting again. In 2000, he had the idea for a web application that would let people edit code on a server and host the resulting applications, which later became known as \"Reddit\".</b>"
- ],
- "text/plain": [
- "<IPython.core.display.Markdown object>"
- ]
- },
- "metadata": {},
- "output_type": "display_data"
- }
- ],
- "source": [
- "display(Markdown(f\"<b>{response}</b>\"))"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "d24f9a20-48a6-4131-91b9-b01448c6ecb5",
- "metadata": {},
- "source": [
- "#### GPTRAKEKeywordTableIndex"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 1,
- "id": "c4d3f293-e608-4b90-86aa-9bce666dbcd5",
- "metadata": {},
- "outputs": [
- {
- "name": "stderr",
- "output_type": "stream",
- "text": [
- "[nltk_data] Downloading package stopwords to /home/jerry/nltk_data...\n",
- "[nltk_data] Package stopwords is already up-to-date!\n"
- ]
- }
- ],
- "source": [
- "from gpt_index import GPTRAKEKeywordTableIndex, SimpleDirectoryReader\n",
- "from IPython.display import Markdown, display"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "66b1da3b-8231-4da9-8026-4f95481c79df",
- "metadata": {
- "scrolled": true,
- "tags": []
- },
- "outputs": [],
- "source": [
- "# build keyword index\n",
- "documents = SimpleDirectoryReader('data').load_data()\n",
- "index = GPTRAKEKeywordTableIndex(documents)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 10,
- "id": "f13e5543-c6cb-4651-986c-ecde0f4bf789",
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "> Starting query: What did the author do after his time at YC?\n",
- "Extracted keywords: []\n"
- ]
- }
- ],
- "source": [
- "response = index.query(\"What did the author do after his time at YC?\")"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 11,
- "id": "5ae01ac3-55fa-43a3-9b24-f733072d5f8d",
- "metadata": {},
- "outputs": [
- {
- "data": {
- "text/markdown": [
- "<b>Empty response</b>"
- ],
- "text/plain": [
- "<IPython.core.display.Markdown object>"
- ]
- },
- "metadata": {},
- "output_type": "display_data"
- }
- ],
- "source": [
- "display(Markdown(f\"<b>{response}</b>\"))"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "59cee6cf-92df-40d8-8dad-a40b792de96f",
- "metadata": {},
- "source": [
- "#### GPTKeywordTableIndex"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 7,
- "id": "78d59ef6-70b0-47bb-818d-7237a3b7de75",
- "metadata": {},
- "outputs": [],
- "source": [
- "from gpt_index import GPTKeywordTableIndex, SimpleDirectoryReader\n",
- "from IPython.display import Markdown, display"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "5a3f1c67-6d73-4f37-afcf-9e637002fcff",
- "metadata": {},
- "outputs": [],
- "source": [
- "# build keyword index\n",
- "documents = SimpleDirectoryReader('data').load_data()\n",
- "index = GPTKeywordTableIndex(documents)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "69d4f686-6825-49cf-a113-d2fdd484de77",
- "metadata": {},
- "outputs": [],
- "source": [
- "response = index.query(\"What did the author do after his time at Y Combinator?\")"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 10,
- "id": "a483514d-4ab5-489d-8b99-7250df491ce3",
- "metadata": {},
- "outputs": [
- {
- "data": {
- "text/markdown": [
- "<b>\n",
- "\n",
- "After a few years, the author decided to step away from Y Combinator to focus on other projects, such as painting and writing essays. In 2013, he handed over control of Y Combinator to Sam Altman. The author's mother passed away in 2014, and after taking some time to grieve, he returned to writing essays and working on Lisp. He continued working on Lisp until 2019, when he finally completed the project.\n",
- "\n",
- "In 2015, the author decided to move to England with his family. They originally intended to only stay for a year, but ended up liking it so much that they remained there. The author wrote Bel while living in England. In 2019, he finally finished the project. After completing Bel, the author wrote a number of essays on various topics. He continued writing essays through 2020, but also started thinking about other things he could work on.</b>"
- ],
- "text/plain": [
- "<IPython.core.display.Markdown object>"
- ]
- },
- "metadata": {},
- "output_type": "display_data"
- }
- ],
- "source": [
- "display(Markdown(f\"<b>{response}</b>\"))"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "112e21ee-587c-4d8b-871e-cb99b94e3778",
- "metadata": {},
- "source": [
- "## GPT Keyword Table Query Comparisons\n",
- "Compare mode={\"default\", \"simple\", \"rake\"}"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "3029961a-ec22-42a1-90d6-f5892eb81e34",
- "metadata": {},
- "outputs": [],
- "source": [
- "# build table with default GPTKeywordTableIndex\n",
- "from gpt_index import GPTKeywordTableIndex, SimpleDirectoryReader\n",
- "from IPython.display import Markdown, display\n",
- "\n",
- "documents = SimpleDirectoryReader('data').load_data()\n",
- "index = GPTKeywordTableIndex(documents)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 3,
- "id": "d75b31da-4788-4295-8642-07ac5c4f11a5",
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "> Starting query: What did the author do after his time at Y Combinator?\n",
- "Extracted keywords: ['y combinator', 'combinator']\n",
- "> Querying with idx: 235042210695008001: of excluding them, because there were so many s...\n",
- "> Querying with idx: 7029274505691774319: it was like living in another country, and sinc...\n",
- "> Querying with idx: 1773317813360405038: browser, and then host the resulting applicatio...\n",
- "> Querying with idx: 3866067077574405334: person, and from those we picked 8 to fund. The...\n"
- ]
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "id": "a6457769-dfaf-4241-ab32-dcf901dde902",
+ "metadata": {
+ "tags": []
+ },
+ "source": [
+ "## GPT Keyword Table Index Comparisons\n",
+ "\n",
+ "Comparing GPTSimpleKeywordTableIndex, GPTRAKEKeywordTableIndex, GPTKeywordTableIndex.\n",
+ "\n",
+ "- GPTSimpleKeywordTableIndex - uses simple regex to extract keywords.\n",
+ "- GPTRAKEKeywordTableIndex - uses RAKE to extract keywords.\n",
+ "- GPTKeywordTableIndex - uses GPT to extract keywords."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "075080e5-c255-4a5c-9330-9da11532e1c8",
+ "metadata": {
+ "tags": []
+ },
+ "source": [
+ "#### GPTSimpleKeywordTableIndex"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 1,
+ "id": "b367b7ef-6a7d-4aee-b174-dba6ec4d2e21",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "[nltk_data] Downloading package stopwords to /home/jerry/nltk_data...\n",
+ "[nltk_data] Package stopwords is already up-to-date!\n"
+ ]
+ }
+ ],
+ "source": [
+ "from llama_index import GPTSimpleKeywordTableIndex, SimpleDirectoryReader\n",
+ "from IPython.display import Markdown, display"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "1f8248fa-e0bd-494a-ad68-8192ccc87696",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# build keyword index\n",
+ "documents = SimpleDirectoryReader('data').load_data()\n",
+ "index = GPTSimpleKeywordTableIndex(documents)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "53833655-0296-4bcb-b501-259b043d68b3",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "response = index.query(\"What did the author do after his time at YC?\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 5,
+ "id": "62bcca18-b644-4393-ad29-6c5f0424fb22",
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/markdown": [
+ "<b>\n",
+ "\n",
+ "The author went on to write essays and work on other projects, including a new version of the Arc programming language and Hacker News. He also started painting, but stopped after a few months. In 2015, he started working on a new Lisp programming language, which he finished in 2019. The author then moved to England in 2016 with his family and continued writing essays. In 2019, he finished Bel and wrote a bunch of essays on various topics.\n",
+ "\n",
+ "The author also worked on building online stores in 1995 after finishing ANSI Common Lisp. He ran the software on servers and let users control it by clicking on links, which was a new concept at the time. In 1996, he co-founded Viaweb with Robert Morris, which was later acquired by Yahoo in 1998. After leaving Yahoo, the author moved back to New York and started painting again. In 2000, he had the idea for a web application that would let people edit code on a server and host the resulting applications, which later became known as \"Reddit\".</b>"
+ ],
+ "text/plain": [
+ "<IPython.core.display.Markdown object>"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ }
+ ],
+ "source": [
+ "display(Markdown(f\"<b>{response}</b>\"))"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "d24f9a20-48a6-4131-91b9-b01448c6ecb5",
+ "metadata": {},
+ "source": [
+ "#### GPTRAKEKeywordTableIndex"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 1,
+ "id": "c4d3f293-e608-4b90-86aa-9bce666dbcd5",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "[nltk_data] Downloading package stopwords to /home/jerry/nltk_data...\n",
+ "[nltk_data] Package stopwords is already up-to-date!\n"
+ ]
+ }
+ ],
+ "source": [
+ "from llama_index import GPTRAKEKeywordTableIndex, SimpleDirectoryReader\n",
+ "from IPython.display import Markdown, display"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "66b1da3b-8231-4da9-8026-4f95481c79df",
+ "metadata": {
+ "scrolled": true,
+ "tags": []
+ },
+ "outputs": [],
+ "source": [
+ "# build keyword index\n",
+ "documents = SimpleDirectoryReader('data').load_data()\n",
+ "index = GPTRAKEKeywordTableIndex(documents)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 10,
+ "id": "f13e5543-c6cb-4651-986c-ecde0f4bf789",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "> Starting query: What did the author do after his time at YC?\n",
+ "Extracted keywords: []\n"
+ ]
+ }
+ ],
+ "source": [
+ "response = index.query(\"What did the author do after his time at YC?\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 11,
+ "id": "5ae01ac3-55fa-43a3-9b24-f733072d5f8d",
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/markdown": [
+ "<b>Empty response</b>"
+ ],
+ "text/plain": [
+ "<IPython.core.display.Markdown object>"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ }
+ ],
+ "source": [
+ "display(Markdown(f\"<b>{response}</b>\"))"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "59cee6cf-92df-40d8-8dad-a40b792de96f",
+ "metadata": {},
+ "source": [
+ "#### GPTKeywordTableIndex"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 7,
+ "id": "78d59ef6-70b0-47bb-818d-7237a3b7de75",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from llama_index import GPTKeywordTableIndex, SimpleDirectoryReader\n",
+ "from IPython.display import Markdown, display"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "5a3f1c67-6d73-4f37-afcf-9e637002fcff",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# build keyword index\n",
+ "documents = SimpleDirectoryReader('data').load_data()\n",
+ "index = GPTKeywordTableIndex(documents)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "69d4f686-6825-49cf-a113-d2fdd484de77",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "response = index.query(\"What did the author do after his time at Y Combinator?\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 10,
+ "id": "a483514d-4ab5-489d-8b99-7250df491ce3",
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/markdown": [
+ "<b>\n",
+ "\n",
+ "After a few years, the author decided to step away from Y Combinator to focus on other projects, such as painting and writing essays. In 2013, he handed over control of Y Combinator to Sam Altman. The author's mother passed away in 2014, and after taking some time to grieve, he returned to writing essays and working on Lisp. He continued working on Lisp until 2019, when he finally completed the project.\n",
+ "\n",
+ "In 2015, the author decided to move to England with his family. They originally intended to only stay for a year, but ended up liking it so much that they remained there. The author wrote Bel while living in England. In 2019, he finally finished the project. After completing Bel, the author wrote a number of essays on various topics. He continued writing essays through 2020, but also started thinking about other things he could work on.</b>"
+ ],
+ "text/plain": [
+ "<IPython.core.display.Markdown object>"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ }
+ ],
+ "source": [
+ "display(Markdown(f\"<b>{response}</b>\"))"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "112e21ee-587c-4d8b-871e-cb99b94e3778",
+ "metadata": {},
+ "source": [
+ "## GPT Keyword Table Query Comparisons\n",
+ "Compare mode={\"default\", \"simple\", \"rake\"}"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "3029961a-ec22-42a1-90d6-f5892eb81e34",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# build table with default GPTKeywordTableIndex\n",
+ "from llama_index import GPTKeywordTableIndex, SimpleDirectoryReader\n",
+ "from IPython.display import Markdown, display\n",
+ "\n",
+ "documents = SimpleDirectoryReader('data').load_data()\n",
+ "index = GPTKeywordTableIndex(documents)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 3,
+ "id": "d75b31da-4788-4295-8642-07ac5c4f11a5",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "> Starting query: What did the author do after his time at Y Combinator?\n",
+ "Extracted keywords: ['y combinator', 'combinator']\n",
+ "> Querying with idx: 235042210695008001: of excluding them, because there were so many s...\n",
+ "> Querying with idx: 7029274505691774319: it was like living in another country, and sinc...\n",
+ "> Querying with idx: 1773317813360405038: browser, and then host the resulting applicatio...\n",
+ "> Querying with idx: 3866067077574405334: person, and from those we picked 8 to fund. The...\n"
+ ]
+ },
+ {
+ "data": {
+ "text/markdown": [
+ "<b>\n",
+ "\n",
+ "The author went on to write a book about his experiences at Y Combinator, and then moved to England. He started writing essays again and also began working on a new Lisp programming language. He also wrote an essay about how he chooses what to work on.</b>"
+ ],
+ "text/plain": [
+ "<IPython.core.display.Markdown object>"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ }
+ ],
+ "source": [
+ "# default\n",
+ "response = index.query(\"What did the author do after his time at Y Combinator?\", mode=\"default\")\n",
+ "display(Markdown(f\"<b>{response}</b>\"))"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 4,
+ "id": "07b713f4-adfc-46f7-a795-5b333e33d49d",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "> Starting query: What did the author do after his time at Y Combinator?\n",
+ "Extracted keywords: ['combinator']\n",
+ "> Querying with idx: 235042210695008001: of excluding them, because there were so many s...\n",
+ "> Querying with idx: 7029274505691774319: it was like living in another country, and sinc...\n",
+ "> Querying with idx: 1773317813360405038: browser, and then host the resulting applicatio...\n",
+ "> Querying with idx: 3866067077574405334: person, and from those we picked 8 to fund. The...\n"
+ ]
+ },
+ {
+ "data": {
+ "text/markdown": [
+ "<b>\n",
+ "\n",
+ "The author went on to write a book about his experiences at Y Combinator, and then moved to England. He started writing essays again and also began working on a new Lisp programming language. He also wrote an essay about how he chooses what to work on.</b>"
+ ],
+ "text/plain": [
+ "<IPython.core.display.Markdown object>"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ }
+ ],
+ "source": [
+ "# simple\n",
+ "response = index.query(\"What did the author do after his time at Y Combinator?\", mode=\"simple\")\n",
+ "display(Markdown(f\"<b>{response}</b>\"))"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 5,
+ "id": "d2e19ad9-3190-45e5-a28d-235c28296d70",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "> Starting query: What did the author do after his time at Y Combinator?\n",
+ "Extracted keywords: ['combinator']\n",
+ "> Querying with idx: 235042210695008001: of excluding them, because there were so many s...\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "[nltk_data] Downloading package punkt to /home/jerry/nltk_data...\n",
+ "[nltk_data] Package punkt is already up-to-date!\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "> Querying with idx: 7029274505691774319: it was like living in another country, and sinc...\n",
+ "> Querying with idx: 1773317813360405038: browser, and then host the resulting applicatio...\n",
+ "> Querying with idx: 3866067077574405334: person, and from those we picked 8 to fund. The...\n"
+ ]
+ },
+ {
+ "data": {
+ "text/markdown": [
+ "<b>\n",
+ "\n",
+ "The author went on to write a book about his experiences at Y Combinator, and then moved to England. He started writing essays again and also began working on a new Lisp programming language. He also wrote an essay about how he chooses what to work on.</b>"
+ ],
+ "text/plain": [
+ "<IPython.core.display.Markdown object>"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ }
+ ],
+ "source": [
+ "# rake\n",
+ "response = index.query(\"What did the author do after his time at Y Combinator?\", mode=\"rake\")\n",
+ "display(Markdown(f\"<b>{response}</b>\"))"
+ ]
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "myvenv",
+ "language": "python",
+ "name": "myvenv"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.9.0"
+ }
},
- {
- "data": {
- "text/markdown": [
- "<b>\n",
- "\n",
- "The author went on to write a book about his experiences at Y Combinator, and then moved to England. He started writing essays again and also began working on a new Lisp programming language. He also wrote an essay about how he chooses what to work on.</b>"
- ],
- "text/plain": [
- "<IPython.core.display.Markdown object>"
- ]
- },
- "metadata": {},
- "output_type": "display_data"
- }
- ],
- "source": [
- "# default\n",
- "response = index.query(\"What did the author do after his time at Y Combinator?\", mode=\"default\")\n",
- "display(Markdown(f\"<b>{response}</b>\"))"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 4,
- "id": "07b713f4-adfc-46f7-a795-5b333e33d49d",
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "> Starting query: What did the author do after his time at Y Combinator?\n",
- "Extracted keywords: ['combinator']\n",
- "> Querying with idx: 235042210695008001: of excluding them, because there were so many s...\n",
- "> Querying with idx: 7029274505691774319: it was like living in another country, and sinc...\n",
- "> Querying with idx: 1773317813360405038: browser, and then host the resulting applicatio...\n",
- "> Querying with idx: 3866067077574405334: person, and from those we picked 8 to fund. The...\n"
- ]
- },
- {
- "data": {
- "text/markdown": [
- "<b>\n",
- "\n",
- "The author went on to write a book about his experiences at Y Combinator, and then moved to England. He started writing essays again and also began working on a new Lisp programming language. He also wrote an essay about how he chooses what to work on.</b>"
- ],
- "text/plain": [
- "<IPython.core.display.Markdown object>"
- ]
- },
- "metadata": {},
- "output_type": "display_data"
- }
- ],
- "source": [
- "# simple\n",
- "response = index.query(\"What did the author do after his time at Y Combinator?\", mode=\"simple\")\n",
- "display(Markdown(f\"<b>{response}</b>\"))"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 5,
- "id": "d2e19ad9-3190-45e5-a28d-235c28296d70",
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "> Starting query: What did the author do after his time at Y Combinator?\n",
- "Extracted keywords: ['combinator']\n",
- "> Querying with idx: 235042210695008001: of excluding them, because there were so many s...\n"
- ]
- },
- {
- "name": "stderr",
- "output_type": "stream",
- "text": [
- "[nltk_data] Downloading package punkt to /home/jerry/nltk_data...\n",
- "[nltk_data] Package punkt is already up-to-date!\n"
- ]
- },
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "> Querying with idx: 7029274505691774319: it was like living in another country, and sinc...\n",
- "> Querying with idx: 1773317813360405038: browser, and then host the resulting applicatio...\n",
- "> Querying with idx: 3866067077574405334: person, and from those we picked 8 to fund. The...\n"
- ]
- },
- {
- "data": {
- "text/markdown": [
- "<b>\n",
- "\n",
- "The author went on to write a book about his experiences at Y Combinator, and then moved to England. He started writing essays again and also began working on a new Lisp programming language. He also wrote an essay about how he chooses what to work on.</b>"
- ],
- "text/plain": [
- "<IPython.core.display.Markdown object>"
- ]
- },
- "metadata": {},
- "output_type": "display_data"
- }
- ],
- "source": [
- "# rake\n",
- "response = index.query(\"What did the author do after his time at Y Combinator?\", mode=\"rake\")\n",
- "display(Markdown(f\"<b>{response}</b>\"))"
- ]
- }
- ],
- "metadata": {
- "kernelspec": {
- "display_name": "myvenv",
- "language": "python",
- "name": "myvenv"
- },
- "language_info": {
- "codemirror_mode": {
- "name": "ipython",
- "version": 3
- },
- "file_extension": ".py",
- "mimetype": "text/x-python",
- "name": "python",
- "nbconvert_exporter": "python",
- "pygments_lexer": "ipython3",
- "version": "3.9.0"
- }
- },
- "nbformat": 4,
- "nbformat_minor": 5
-}
+ "nbformat": 4,
+ "nbformat_minor": 5
+}
\ No newline at end of file
diff --git a/examples/paul_graham_essay/TestEssay.ipynb b/examples/paul_graham_essay/TestEssay.ipynb
index 8bbd1ba24be201d14ad6dcb3821b786c59593878..e7bbe8ec3d08d6abe6dc307b3868722b72f55de5 100644
--- a/examples/paul_graham_essay/TestEssay.ipynb
+++ b/examples/paul_graham_essay/TestEssay.ipynb
@@ -1,660 +1,660 @@
{
- "cells": [
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "f1a9eb90-335c-4214-8bb6-fd1edbe3ccbd",
- "metadata": {},
- "outputs": [],
- "source": [
- "# My OpenAI Key\n",
- "import os\n",
- "os.environ['OPENAI_API_KEY'] = \"INSERT OPENAI KEY\""
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "6a712b56",
- "metadata": {},
- "outputs": [],
- "source": [
- "import logging\n",
- "import sys\n",
- "\n",
- "logging.basicConfig(stream=sys.stdout, level=logging.INFO)\n",
- "logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout))"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "be3f7baa-1c0a-430b-981b-83ddca9e71f2",
- "metadata": {
- "tags": []
- },
- "source": [
- "## Using GPT Tree Index"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "0881f151-279e-4910-95c7-f49d3d6a4c69",
- "metadata": {},
- "source": [
- "#### [Demo] Default leaf traversal "
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "8d0b2364-4806-4656-81e7-3f6e4b910b5b",
- "metadata": {},
- "outputs": [],
- "source": [
- "from gpt_index import GPTTreeIndex, SimpleDirectoryReader\n",
- "from IPython.display import Markdown, display"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 2,
- "id": "1c297fd3-3424-41d8-9d0d-25fe6310ab62",
- "metadata": {
- "tags": []
- },
- "outputs": [],
- "source": [
- "documents = SimpleDirectoryReader('data').load_data()"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "370fd08f-56ff-4c24-b0c4-c93116a6d482",
- "metadata": {
- "tags": []
- },
- "outputs": [],
- "source": [
- "index = GPTTreeIndex(documents)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "0b4fe9b6-5762-4e86-b51e-aac45d3ecdb1",
- "metadata": {},
- "outputs": [],
- "source": [
- "index.save_to_disk('index.json')"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 2,
- "id": "5eec265d-211b-4f26-b05b-5b4e7072bc6e",
- "metadata": {},
- "outputs": [],
- "source": [
- "# try loading\n",
- "new_index = GPTTreeIndex.load_from_disk('index.json')"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "bd14686d-1c53-4637-9340-3745f2121ae2",
- "metadata": {},
- "outputs": [],
- "source": [
- "# set Logging to DEBUG for more detailed outputs\n",
- "response = new_index.query(\"What did the author do growing up?\")"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 4,
- "id": "b4c87d14-d2d8-4d80-89f6-1e5972973528",
- "metadata": {},
- "outputs": [
- {
- "data": {
- "text/markdown": [
- "<b>The author wrote short stories and tried to program on an IBM 1401.</b>"
- ],
- "text/plain": [
- "<IPython.core.display.Markdown object>"
- ]
- },
- "metadata": {},
- "output_type": "display_data"
- }
- ],
- "source": [
- "display(Markdown(f\"<b>{response}</b>\"))"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "68c9ebfe-b1b6-4f4e-9278-174346de8c90",
- "metadata": {
- "tags": []
- },
- "outputs": [],
- "source": [
- "# set Logging to DEBUG for more detailed outputs\n",
- "response = new_index.query(\"What did the author do after his time at Y Combinator?\")"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 6,
- "id": "a5ab5943-7c84-4c2b-ac99-ec4b5fc67e64",
- "metadata": {},
- "outputs": [
- {
- "data": {
- "text/markdown": [
- "<b>The author went on to start his own company.</b>"
- ],
- "text/plain": [
- "<IPython.core.display.Markdown object>"
- ]
- },
- "metadata": {},
- "output_type": "display_data"
- }
- ],
- "source": [
- "display(Markdown(f\"<b>{response}</b>\"))"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "85c62ec3-c3cf-467e-ab0f-88ffb9f990be",
- "metadata": {},
- "source": [
- "#### [Demo] Leaf traversal with child_branch_factor=2"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "46714db4-9592-4c55-9ca7-916758f2ce68",
- "metadata": {
- "tags": []
- },
- "outputs": [],
- "source": [
- "# try using branching factor 2\n",
- "response = new_index.query(\"What did the author do growing up?\", child_branch_factor=2)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 6,
- "id": "1ea7f891-b7e1-497a-a965-14201b220404",
- "metadata": {},
- "outputs": [
- {
- "data": {
- "text/markdown": [
- "<b>The author grew up writing simple programs on a TRS-80 computer, as well as trying to program on an IBM 1401. In the early 1990s, the author was a student at the Rhode Island School of Design (RISD) and then the Accademia di Belle Arti in Florence, Italy. They eventually dropped out of RISD and moved to New York City, where they got a job at Interleaf, a software company. While working there, they learned about a new markup language called HTML, which would later become a big part of their life. He also wrote a book on Lisp programming.</b>"
- ],
- "text/plain": [
- "<IPython.core.display.Markdown object>"
- ]
- },
- "metadata": {},
- "output_type": "display_data"
- }
- ],
- "source": [
- "display(Markdown(f\"<b>{response}</b>\"))"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "3c572726-bb95-49c3-a762-d966de59ee5f",
- "metadata": {},
- "source": [
- "#### [Demo] Build Tree Index during Query-Time"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 3,
- "id": "255fb052-1ff6-4f27-881f-28d4790e9520",
- "metadata": {},
- "outputs": [],
- "source": [
- "documents = SimpleDirectoryReader('data').load_data()"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 4,
- "id": "85371256-292c-473e-9485-7de5c1997a59",
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "> [build_index_from_documents] Total token usage: 0 tokens\n"
- ]
- }
- ],
- "source": [
- "index_light = GPTTreeIndex(documents, build_tree=False)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 7,
- "id": "77b0acb3-5593-4f00-8eef-315a031fedc2",
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "> Starting query: What did the author do after his time at Y Combinator?\n",
- "> Building index from nodes: 5 chunks\n",
- "0/57\n",
- "10/57\n",
- "20/57\n",
- "30/57\n",
- "40/57\n",
- "50/57\n",
- "> [query] Total token usage: 18200 tokens\n"
- ]
+ "cells": [
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "f1a9eb90-335c-4214-8bb6-fd1edbe3ccbd",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# My OpenAI Key\n",
+ "import os\n",
+ "os.environ['OPENAI_API_KEY'] = \"INSERT OPENAI KEY\""
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "6a712b56",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import logging\n",
+ "import sys\n",
+ "\n",
+ "logging.basicConfig(stream=sys.stdout, level=logging.INFO)\n",
+ "logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout))"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "be3f7baa-1c0a-430b-981b-83ddca9e71f2",
+ "metadata": {
+ "tags": []
+ },
+ "source": [
+ "## Using GPT Tree Index"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "0881f151-279e-4910-95c7-f49d3d6a4c69",
+ "metadata": {},
+ "source": [
+ "#### [Demo] Default leaf traversal "
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "8d0b2364-4806-4656-81e7-3f6e4b910b5b",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from llama_index import GPTTreeIndex, SimpleDirectoryReader\n",
+ "from IPython.display import Markdown, display"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 2,
+ "id": "1c297fd3-3424-41d8-9d0d-25fe6310ab62",
+ "metadata": {
+ "tags": []
+ },
+ "outputs": [],
+ "source": [
+ "documents = SimpleDirectoryReader('data').load_data()"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "370fd08f-56ff-4c24-b0c4-c93116a6d482",
+ "metadata": {
+ "tags": []
+ },
+ "outputs": [],
+ "source": [
+ "index = GPTTreeIndex(documents)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "0b4fe9b6-5762-4e86-b51e-aac45d3ecdb1",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "index.save_to_disk('index.json')"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 2,
+ "id": "5eec265d-211b-4f26-b05b-5b4e7072bc6e",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# try loading\n",
+ "new_index = GPTTreeIndex.load_from_disk('index.json')"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "bd14686d-1c53-4637-9340-3745f2121ae2",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# set Logging to DEBUG for more detailed outputs\n",
+ "response = new_index.query(\"What did the author do growing up?\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 4,
+ "id": "b4c87d14-d2d8-4d80-89f6-1e5972973528",
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/markdown": [
+ "<b>The author wrote short stories and tried to program on an IBM 1401.</b>"
+ ],
+ "text/plain": [
+ "<IPython.core.display.Markdown object>"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ }
+ ],
+ "source": [
+ "display(Markdown(f\"<b>{response}</b>\"))"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "68c9ebfe-b1b6-4f4e-9278-174346de8c90",
+ "metadata": {
+ "tags": []
+ },
+ "outputs": [],
+ "source": [
+ "# set Logging to DEBUG for more detailed outputs\n",
+ "response = new_index.query(\"What did the author do after his time at Y Combinator?\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 6,
+ "id": "a5ab5943-7c84-4c2b-ac99-ec4b5fc67e64",
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/markdown": [
+ "<b>The author went on to start his own company.</b>"
+ ],
+ "text/plain": [
+ "<IPython.core.display.Markdown object>"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ }
+ ],
+ "source": [
+ "display(Markdown(f\"<b>{response}</b>\"))"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "85c62ec3-c3cf-467e-ab0f-88ffb9f990be",
+ "metadata": {},
+ "source": [
+ "#### [Demo] Leaf traversal with child_branch_factor=2"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "46714db4-9592-4c55-9ca7-916758f2ce68",
+ "metadata": {
+ "tags": []
+ },
+ "outputs": [],
+ "source": [
+ "# try using branching factor 2\n",
+ "response = new_index.query(\"What did the author do growing up?\", child_branch_factor=2)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 6,
+ "id": "1ea7f891-b7e1-497a-a965-14201b220404",
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/markdown": [
+ "<b>The author grew up writing simple programs on a TRS-80 computer, as well as trying to program on an IBM 1401. In the early 1990s, the author was a student at the Rhode Island School of Design (RISD) and then the Accademia di Belle Arti in Florence, Italy. They eventually dropped out of RISD and moved to New York City, where they got a job at Interleaf, a software company. While working there, they learned about a new markup language called HTML, which would later become a big part of their life. He also wrote a book on Lisp programming.</b>"
+ ],
+ "text/plain": [
+ "<IPython.core.display.Markdown object>"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ }
+ ],
+ "source": [
+ "display(Markdown(f\"<b>{response}</b>\"))"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "3c572726-bb95-49c3-a762-d966de59ee5f",
+ "metadata": {},
+ "source": [
+ "#### [Demo] Build Tree Index during Query-Time"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 3,
+ "id": "255fb052-1ff6-4f27-881f-28d4790e9520",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "documents = SimpleDirectoryReader('data').load_data()"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 4,
+ "id": "85371256-292c-473e-9485-7de5c1997a59",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "> [build_index_from_documents] Total token usage: 0 tokens\n"
+ ]
+ }
+ ],
+ "source": [
+ "index_light = GPTTreeIndex(documents, build_tree=False)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 7,
+ "id": "77b0acb3-5593-4f00-8eef-315a031fedc2",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "> Starting query: What did the author do after his time at Y Combinator?\n",
+ "> Building index from nodes: 5 chunks\n",
+ "0/57\n",
+ "10/57\n",
+ "20/57\n",
+ "30/57\n",
+ "40/57\n",
+ "50/57\n",
+ "> [query] Total token usage: 18200 tokens\n"
+ ]
+ },
+ {
+ "data": {
+ "text/plain": [
+ "'\\nThe author went back to painting.'"
+ ]
+ },
+ "execution_count": 7,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "index_light.query(\"What did the author do after his time at Y Combinator?\", mode=\"summarize\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "f9773497-9aa6-4a16-884a-cd882e63d012",
+ "metadata": {},
+ "source": [
+ "#### [Demo] Build Tree Index with a custom Summary Prompt, directly retrieve answer from root node"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 20,
+ "id": "8ab6d3ad-95e1-477a-a0dc-2ce4763ff2c4",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from llama_index import SummaryPrompt"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 23,
+ "id": "5a91a445-6ab2-457c-850e-79c5386129db",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "> Building index from nodes: 5 chunks\n",
+ "0/57\n",
+ "10/57\n",
+ "20/57\n",
+ "30/57\n",
+ "40/57\n",
+ "50/57\n",
+ "> [build_index_from_documents] Total token usage: 18031 tokens\n"
+ ]
+ }
+ ],
+ "source": [
+ "documents = SimpleDirectoryReader('data').load_data()\n",
+ "\n",
+ "query_str = \"What did the author do growing up?\"\n",
+ "SUMMARY_PROMPT_TMPL = (\n",
+ " \"Context information is below. \\n\"\n",
+ " \"---------------------\\n\"\n",
+ " \"{context_str}\"\n",
+ " \"\\n---------------------\\n\"\n",
+ " \"Given the context information and not prior knowledge, \"\n",
+ " f\"answer the question: {query_str}\\n\"\n",
+ ")\n",
+ "SUMMARY_PROMPT = SummaryPrompt(SUMMARY_PROMPT_TMPL)\n",
+ "index_with_query = GPTTreeIndex(documents, summary_template=SUMMARY_PROMPT)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 6,
+ "id": "985dad0c-1ede-4576-a4c9-c077b815edd8",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "index_with_query.save_to_disk(\"index_with_query.json\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 7,
+ "id": "de04fce5-88f9-41b7-87d9-dcde8f84a872",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "index_with_query = GPTTreeIndex.load_from_disk(\"index_with_query.json\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "9223ffa8-d49d-4de3-821a-701b2a0352d4",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# directly retrieve response from root nodes instead of traversing tree\n",
+ "response = index_with_query.query(query_str, mode=\"retrieve\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 10,
+ "id": "fdca6970-2f3f-4741-ae98-555db8d3d9a0",
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/markdown": [
+ "<b>\n",
+ "The author was homeschooled and then attended a prestigious art school. The author grew up writing essays and thinking about other things he could work on.</b>"
+ ],
+ "text/plain": [
+ "<IPython.core.display.Markdown object>"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ }
+ ],
+ "source": [
+ "display(Markdown(f\"<b>{response}</b>\"))"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "a6457769-dfaf-4241-ab32-dcf901dde902",
+ "metadata": {},
+ "source": [
+ "## Using GPT Keyword Table Index"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 7,
+ "id": "78d59ef6-70b0-47bb-818d-7237a3b7de75",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from llama_index import GPTKeywordTableIndex, SimpleDirectoryReader\n",
+ "from IPython.display import Markdown, display"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 2,
+ "id": "5a3f1c67-6d73-4f37-afcf-9e637002fcff",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "> Processing chunk 0 of 6: \t\t\n",
+ "\n",
+ "What I Worked On\n",
+ "\n",
+ "February 2021\n",
+ "\n",
+ "Before col...\n",
+ "> Keywords: ['painting', 'computers', 'programming', 'lisp', 'ai', 'college', 'graduate school', 'graduate', 'school', 'writing']\n",
+ "> Processing chunk 1 of 6: of excluding them, because there were so many s...\n",
+ "> Keywords: ['school', 'students', 'painting', 'florence', 'risd', 'accademia', 'still lives', 'still', 'lives', 'color', 'new york', 'new', 'york', 'yorkville', 'idelle weber', 'idelle', 'weber', 'harvard', 'world wide web', 'world', 'wide', 'web', 'y combinator', 'combinator', 'software', 'lisp']\n",
+ "> Processing chunk 2 of 6: an alarming prospect, because neither of us kne...\n",
+ "> Keywords: ['windows', 'unix', 'lisp', 'web app', 'web', 'app', 'browser', 'store builder', 'store', 'builder', 'ecommerce', 'startup', 'painting']\n",
+ "> Processing chunk 3 of 6: browser, and then host the resulting applicatio...\n",
+ "> Keywords: ['y combinator', 'combinator', 'investment', 'summer founders program', 'summer', 'founders', 'program', 'microsoft', 'goldman sachs', 'goldman', 'sachs']\n",
+ "> Processing chunk 4 of 6: person, and from those we picked 8 to fund. The...\n",
+ "> Keywords: ['y combinator', 'combinator', 'yc', 'lisp', 'bel', 'essays', 'writing', 'software', 'programming', 'arc']\n",
+ "> Processing chunk 5 of 6: it was like living in another country, and sinc...\n",
+ "> Keywords: ['software', 'technology', 'y combinator', 'combinator', 'essays', 'online publishing', 'online', 'publishing', 'venture capital', 'venture', 'capital', 'startups', 'space aliens', 'space', 'aliens', 'lisp']\n"
+ ]
+ }
+ ],
+ "source": [
+ "# build keyword index\n",
+ "documents = SimpleDirectoryReader('data').load_data()\n",
+ "index = GPTKeywordTableIndex(documents)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 4,
+ "id": "7ec97988-0190-4df7-b19a-e3130122298f",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# save index\n",
+ "index.save_to_disk('index_table.json')"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 8,
+ "id": "d94d0fe0-43c1-41cd-901b-0d748d30f1c7",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# reload index\n",
+ "index = GPTKeywordTableIndex.load_from_disk('index_table.json')"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 9,
+ "id": "69d4f686-6825-49cf-a113-d2fdd484de77",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "> Starting query: What did the author do after his time at Y Combinator?\n",
+ "Extracted keywords: ['y combinator', 'combinator']\n",
+ "> Querying with idx: 7143669651211954504: of excluding them, because there were so many s...\n",
+ "> Querying with idx: 4978118451876167434: browser, and then host the resulting applicatio...\n",
+ "> Querying with idx: 7378313280237489139: person, and from those we picked 8 to fund. The...\n",
+ "> Querying with idx: 2670584622494666310: it was like living in another country, and sinc...\n"
+ ]
+ }
+ ],
+ "source": [
+ "# set Logging to DEBUG for more detailed outputs\n",
+ "response = index.query(\"What did the author do after his time at Y Combinator?\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 10,
+ "id": "a483514d-4ab5-489d-8b99-7250df491ce3",
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/markdown": [
+ "<b>\n",
+ "\n",
+ "After a few years, the author decided to step away from Y Combinator to focus on other projects, such as painting and writing essays. In 2013, he handed over control of Y Combinator to Sam Altman. The author's mother passed away in 2014, and after taking some time to grieve, he returned to writing essays and working on Lisp. He continued working on Lisp until 2019, when he finally completed the project.\n",
+ "\n",
+ "In 2015, the author decided to move to England with his family. They originally intended to only stay for a year, but ended up liking it so much that they remained there. The author wrote Bel while living in England. In 2019, he finally finished the project. After completing Bel, the author wrote a number of essays on various topics. He continued writing essays through 2020, but also started thinking about other things he could work on.</b>"
+ ],
+ "text/plain": [
+ "<IPython.core.display.Markdown object>"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ }
+ ],
+ "source": [
+ "display(Markdown(f\"<b>{response}</b>\"))"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "aae1bea9-b534-430a-a52b-1f4414957ac9",
+ "metadata": {},
+ "source": [
+ "## Using GPT List Index"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "1aa8c8c1-7fce-4737-9141-d14fd37a779c",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from llama_index import GPTListIndex, SimpleDirectoryReader\n",
+ "from IPython.display import Markdown, display"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 2,
+ "id": "191caa65-a77f-4d8c-b095-4aed61300ea5",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "> Adding chunk: \t\t\n",
+ "\n",
+ "What I Worked On\n",
+ "\n",
+ "February 2021\n",
+ "\n",
+ "Before col...\n",
+ "> Adding chunk: only up to age 25 and already there are such co...\n",
+ "> Adding chunk: clear that it was even possible. To find out, w...\n",
+ "> Adding chunk: a name for the kind of company Viaweb was, an \"...\n",
+ "> Adding chunk: get their initial set of customers almost entir...\n",
+ "> Adding chunk: had smart people and built impressive technolog...\n",
+ "> [build_index_from_documents] Total token usage: 0 tokens\n"
+ ]
+ }
+ ],
+ "source": [
+ "# build linked list index\n",
+ "documents = SimpleDirectoryReader('data').load_data()\n",
+ "index = GPTListIndex(documents)\n",
+ "# save index\n",
+ "index.save_to_disk('index_list.json')"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 14,
+ "id": "af2d049d-518d-4ec4-b84f-1fab8aece04f",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# load index from disk\n",
+ "index = GPTListIndex.load_from_disk('index_list.json')"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "1b3d4bd8-7540-4c6f-8616-ab2d8c6ae2b2",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# set Logging to DEBUG for more detailed outputs\n",
+ "response = index.query(\"What did the author do after his time at Y Combinator?\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 16,
+ "id": "5101b979-175f-490e-9b32-27689fe4b789",
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/markdown": [
+ "<b>\n",
+ "\n",
+ "After his time at Y Combinator, the author moved back to Providence to continue at RISD. However, he found that art school was not what he expected it to be and dropped out. He then moved to New York City and started writing a book on Lisp. When that didn't work out, he started a company to put art galleries online. However, that also failed. He then had the idea to start a company to build online stores, which became a success.\n",
+ "\n",
+ "The author then left his position at Yahoo to pursue painting full-time. However, he found it difficult to get back into the painting mindset and eventually returned to New York City. It was there that he had the idea to create a web application that would allow users to create and host their own web applications.</b>"
+ ],
+ "text/plain": [
+ "<IPython.core.display.Markdown object>"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ }
+ ],
+ "source": [
+ "display(Markdown(f\"<b>{response}</b>\"))"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "65cfce56-853e-431b-888e-946771c3b07e",
+ "metadata": {},
+ "outputs": [],
+ "source": []
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3 (ipykernel)",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.9.16"
+ }
},
- {
- "data": {
- "text/plain": [
- "'\\nThe author went back to painting.'"
- ]
- },
- "execution_count": 7,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "index_light.query(\"What did the author do after his time at Y Combinator?\", mode=\"summarize\")"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "f9773497-9aa6-4a16-884a-cd882e63d012",
- "metadata": {},
- "source": [
- "#### [Demo] Build Tree Index with a custom Summary Prompt, directly retrieve answer from root node"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 20,
- "id": "8ab6d3ad-95e1-477a-a0dc-2ce4763ff2c4",
- "metadata": {},
- "outputs": [],
- "source": [
- "from gpt_index import SummaryPrompt"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 23,
- "id": "5a91a445-6ab2-457c-850e-79c5386129db",
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "> Building index from nodes: 5 chunks\n",
- "0/57\n",
- "10/57\n",
- "20/57\n",
- "30/57\n",
- "40/57\n",
- "50/57\n",
- "> [build_index_from_documents] Total token usage: 18031 tokens\n"
- ]
- }
- ],
- "source": [
- "documents = SimpleDirectoryReader('data').load_data()\n",
- "\n",
- "query_str = \"What did the author do growing up?\"\n",
- "SUMMARY_PROMPT_TMPL = (\n",
- " \"Context information is below. \\n\"\n",
- " \"---------------------\\n\"\n",
- " \"{context_str}\"\n",
- " \"\\n---------------------\\n\"\n",
- " \"Given the context information and not prior knowledge, \"\n",
- " f\"answer the question: {query_str}\\n\"\n",
- ")\n",
- "SUMMARY_PROMPT = SummaryPrompt(SUMMARY_PROMPT_TMPL)\n",
- "index_with_query = GPTTreeIndex(documents, summary_template=SUMMARY_PROMPT)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 6,
- "id": "985dad0c-1ede-4576-a4c9-c077b815edd8",
- "metadata": {},
- "outputs": [],
- "source": [
- "index_with_query.save_to_disk(\"index_with_query.json\")"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 7,
- "id": "de04fce5-88f9-41b7-87d9-dcde8f84a872",
- "metadata": {},
- "outputs": [],
- "source": [
- "index_with_query = GPTTreeIndex.load_from_disk(\"index_with_query.json\")"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "9223ffa8-d49d-4de3-821a-701b2a0352d4",
- "metadata": {},
- "outputs": [],
- "source": [
- "# directly retrieve response from root nodes instead of traversing tree\n",
- "response = index_with_query.query(query_str, mode=\"retrieve\")"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 10,
- "id": "fdca6970-2f3f-4741-ae98-555db8d3d9a0",
- "metadata": {},
- "outputs": [
- {
- "data": {
- "text/markdown": [
- "<b>\n",
- "The author was homeschooled and then attended a prestigious art school. The author grew up writing essays and thinking about other things he could work on.</b>"
- ],
- "text/plain": [
- "<IPython.core.display.Markdown object>"
- ]
- },
- "metadata": {},
- "output_type": "display_data"
- }
- ],
- "source": [
- "display(Markdown(f\"<b>{response}</b>\"))"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "a6457769-dfaf-4241-ab32-dcf901dde902",
- "metadata": {},
- "source": [
- "## Using GPT Keyword Table Index"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 7,
- "id": "78d59ef6-70b0-47bb-818d-7237a3b7de75",
- "metadata": {},
- "outputs": [],
- "source": [
- "from gpt_index import GPTKeywordTableIndex, SimpleDirectoryReader\n",
- "from IPython.display import Markdown, display"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 2,
- "id": "5a3f1c67-6d73-4f37-afcf-9e637002fcff",
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "> Processing chunk 0 of 6: \t\t\n",
- "\n",
- "What I Worked On\n",
- "\n",
- "February 2021\n",
- "\n",
- "Before col...\n",
- "> Keywords: ['painting', 'computers', 'programming', 'lisp', 'ai', 'college', 'graduate school', 'graduate', 'school', 'writing']\n",
- "> Processing chunk 1 of 6: of excluding them, because there were so many s...\n",
- "> Keywords: ['school', 'students', 'painting', 'florence', 'risd', 'accademia', 'still lives', 'still', 'lives', 'color', 'new york', 'new', 'york', 'yorkville', 'idelle weber', 'idelle', 'weber', 'harvard', 'world wide web', 'world', 'wide', 'web', 'y combinator', 'combinator', 'software', 'lisp']\n",
- "> Processing chunk 2 of 6: an alarming prospect, because neither of us kne...\n",
- "> Keywords: ['windows', 'unix', 'lisp', 'web app', 'web', 'app', 'browser', 'store builder', 'store', 'builder', 'ecommerce', 'startup', 'painting']\n",
- "> Processing chunk 3 of 6: browser, and then host the resulting applicatio...\n",
- "> Keywords: ['y combinator', 'combinator', 'investment', 'summer founders program', 'summer', 'founders', 'program', 'microsoft', 'goldman sachs', 'goldman', 'sachs']\n",
- "> Processing chunk 4 of 6: person, and from those we picked 8 to fund. The...\n",
- "> Keywords: ['y combinator', 'combinator', 'yc', 'lisp', 'bel', 'essays', 'writing', 'software', 'programming', 'arc']\n",
- "> Processing chunk 5 of 6: it was like living in another country, and sinc...\n",
- "> Keywords: ['software', 'technology', 'y combinator', 'combinator', 'essays', 'online publishing', 'online', 'publishing', 'venture capital', 'venture', 'capital', 'startups', 'space aliens', 'space', 'aliens', 'lisp']\n"
- ]
- }
- ],
- "source": [
- "# build keyword index\n",
- "documents = SimpleDirectoryReader('data').load_data()\n",
- "index = GPTKeywordTableIndex(documents)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 4,
- "id": "7ec97988-0190-4df7-b19a-e3130122298f",
- "metadata": {},
- "outputs": [],
- "source": [
- "# save index\n",
- "index.save_to_disk('index_table.json')"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 8,
- "id": "d94d0fe0-43c1-41cd-901b-0d748d30f1c7",
- "metadata": {},
- "outputs": [],
- "source": [
- "# reload index\n",
- "index = GPTKeywordTableIndex.load_from_disk('index_table.json')"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 9,
- "id": "69d4f686-6825-49cf-a113-d2fdd484de77",
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "> Starting query: What did the author do after his time at Y Combinator?\n",
- "Extracted keywords: ['y combinator', 'combinator']\n",
- "> Querying with idx: 7143669651211954504: of excluding them, because there were so many s...\n",
- "> Querying with idx: 4978118451876167434: browser, and then host the resulting applicatio...\n",
- "> Querying with idx: 7378313280237489139: person, and from those we picked 8 to fund. The...\n",
- "> Querying with idx: 2670584622494666310: it was like living in another country, and sinc...\n"
- ]
- }
- ],
- "source": [
- "# set Logging to DEBUG for more detailed outputs\n",
- "response = index.query(\"What did the author do after his time at Y Combinator?\")"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 10,
- "id": "a483514d-4ab5-489d-8b99-7250df491ce3",
- "metadata": {},
- "outputs": [
- {
- "data": {
- "text/markdown": [
- "<b>\n",
- "\n",
- "After a few years, the author decided to step away from Y Combinator to focus on other projects, such as painting and writing essays. In 2013, he handed over control of Y Combinator to Sam Altman. The author's mother passed away in 2014, and after taking some time to grieve, he returned to writing essays and working on Lisp. He continued working on Lisp until 2019, when he finally completed the project.\n",
- "\n",
- "In 2015, the author decided to move to England with his family. They originally intended to only stay for a year, but ended up liking it so much that they remained there. The author wrote Bel while living in England. In 2019, he finally finished the project. After completing Bel, the author wrote a number of essays on various topics. He continued writing essays through 2020, but also started thinking about other things he could work on.</b>"
- ],
- "text/plain": [
- "<IPython.core.display.Markdown object>"
- ]
- },
- "metadata": {},
- "output_type": "display_data"
- }
- ],
- "source": [
- "display(Markdown(f\"<b>{response}</b>\"))"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "aae1bea9-b534-430a-a52b-1f4414957ac9",
- "metadata": {},
- "source": [
- "## Using GPT List Index"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "1aa8c8c1-7fce-4737-9141-d14fd37a779c",
- "metadata": {},
- "outputs": [],
- "source": [
- "from gpt_index import GPTListIndex, SimpleDirectoryReader\n",
- "from IPython.display import Markdown, display"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 2,
- "id": "191caa65-a77f-4d8c-b095-4aed61300ea5",
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "> Adding chunk: \t\t\n",
- "\n",
- "What I Worked On\n",
- "\n",
- "February 2021\n",
- "\n",
- "Before col...\n",
- "> Adding chunk: only up to age 25 and already there are such co...\n",
- "> Adding chunk: clear that it was even possible. To find out, w...\n",
- "> Adding chunk: a name for the kind of company Viaweb was, an \"...\n",
- "> Adding chunk: get their initial set of customers almost entir...\n",
- "> Adding chunk: had smart people and built impressive technolog...\n",
- "> [build_index_from_documents] Total token usage: 0 tokens\n"
- ]
- }
- ],
- "source": [
- "# build linked list index\n",
- "documents = SimpleDirectoryReader('data').load_data()\n",
- "index = GPTListIndex(documents)\n",
- "# save index\n",
- "index.save_to_disk('index_list.json')"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 14,
- "id": "af2d049d-518d-4ec4-b84f-1fab8aece04f",
- "metadata": {},
- "outputs": [],
- "source": [
- "# load index from disk\n",
- "index = GPTListIndex.load_from_disk('index_list.json')"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "1b3d4bd8-7540-4c6f-8616-ab2d8c6ae2b2",
- "metadata": {},
- "outputs": [],
- "source": [
- "# set Logging to DEBUG for more detailed outputs\n",
- "response = index.query(\"What did the author do after his time at Y Combinator?\")"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 16,
- "id": "5101b979-175f-490e-9b32-27689fe4b789",
- "metadata": {},
- "outputs": [
- {
- "data": {
- "text/markdown": [
- "<b>\n",
- "\n",
- "After his time at Y Combinator, the author moved back to Providence to continue at RISD. However, he found that art school was not what he expected it to be and dropped out. He then moved to New York City and started writing a book on Lisp. When that didn't work out, he started a company to put art galleries online. However, that also failed. He then had the idea to start a company to build online stores, which became a success.\n",
- "\n",
- "The author then left his position at Yahoo to pursue painting full-time. However, he found it difficult to get back into the painting mindset and eventually returned to New York City. It was there that he had the idea to create a web application that would allow users to create and host their own web applications.</b>"
- ],
- "text/plain": [
- "<IPython.core.display.Markdown object>"
- ]
- },
- "metadata": {},
- "output_type": "display_data"
- }
- ],
- "source": [
- "display(Markdown(f\"<b>{response}</b>\"))"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "65cfce56-853e-431b-888e-946771c3b07e",
- "metadata": {},
- "outputs": [],
- "source": []
- }
- ],
- "metadata": {
- "kernelspec": {
- "display_name": "Python 3 (ipykernel)",
- "language": "python",
- "name": "python3"
- },
- "language_info": {
- "codemirror_mode": {
- "name": "ipython",
- "version": 3
- },
- "file_extension": ".py",
- "mimetype": "text/x-python",
- "name": "python",
- "nbconvert_exporter": "python",
- "pygments_lexer": "ipython3",
- "version": "3.9.16"
- }
- },
- "nbformat": 4,
- "nbformat_minor": 5
-}
+ "nbformat": 4,
+ "nbformat_minor": 5
+}
\ No newline at end of file
diff --git a/examples/query_transformations/HyDEQueryTransformDemo.ipynb b/examples/query_transformations/HyDEQueryTransformDemo.ipynb
index a96ea282218e6333e50bb97dd58cf52159bafc9d..711980609562e23005e8e39e9e80c794159dca48 100644
--- a/examples/query_transformations/HyDEQueryTransformDemo.ipynb
+++ b/examples/query_transformations/HyDEQueryTransformDemo.ipynb
@@ -1,397 +1,397 @@
{
- "cells": [
- {
- "cell_type": "markdown",
- "id": "f20bdf25",
- "metadata": {},
- "source": [
- "# HyDE Query Transform Demo"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "a1e7e17f",
- "metadata": {},
- "source": [
- "#### Load documents, build the GPTSimpleVectorIndex"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "f7476659",
- "metadata": {},
- "outputs": [],
- "source": [
- "import logging\n",
- "import sys\n",
- "\n",
- "logging.basicConfig(stream=sys.stdout, level=logging.INFO)\n",
- "logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout))\n",
- "\n",
- "from gpt_index import GPTSimpleVectorIndex, SimpleDirectoryReader\n",
- "from gpt_index.indices.query.query_transform import HyDEQueryTransform\n",
- "from IPython.display import Markdown, display"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "16ff354a",
- "metadata": {},
- "outputs": [],
- "source": [
- "# load documents\n",
- "documents = SimpleDirectoryReader('../paul_graham_essay/data').load_data()"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "7d6a1b5e",
- "metadata": {},
- "outputs": [],
- "source": [
- "index = GPTSimpleVectorIndex(documents)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "85665424",
- "metadata": {},
- "outputs": [],
- "source": [
- "# save index to disk\n",
- "index.save_to_disk('index_simple.json')"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "78ce3cec",
- "metadata": {},
- "outputs": [],
- "source": [
- "# load index from disk\n",
- "index = GPTSimpleVectorIndex.load_from_disk('index_simple.json')"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "1cec25b6",
- "metadata": {},
- "source": [
- "## Example: HyDE improves specific temporal queries"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "f3c39ddd",
- "metadata": {},
- "outputs": [],
- "source": [
- "query_str = \"what did paul graham do after going to RISD\""
- ]
- },
- {
- "cell_type": "markdown",
- "id": "8b022d86",
- "metadata": {},
- "source": [
- "#### First, we query *without* transformation: The same query string is used for embedding lookup and also summarization."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "a5b93d35",
- "metadata": {},
- "outputs": [],
- "source": [
- "response = index.query(query_str)\n",
- "display(Markdown(f\"<b>{response}</b>\"))"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "d8b9a9db",
- "metadata": {},
- "source": [
- "> After going to RISD, Paul Graham continued to pursue his passion for painting and art. He took classes in the painting department at the Accademia di Belli Arti in Florence, and he also took the entrance exam for the school. He also continued to work on his book On Lisp, and he took on consulting work to make money. At the school, Paul Graham and the other students had an arrangement where the faculty wouldn't require the students to learn anything, and in return the students wouldn't require the faculty to teach anything. Paul Graham was one of the few students who actually painted the nude model that was provided, while the rest of the students spent their time chatting or occasionally trying to imitate things they'd seen in American art magazines. The model turned out to live just down the street from Paul Graham, and she made a living from a combination of modelling and making fakes for a local antique dealer."
- ]
- },
- {
- "cell_type": "markdown",
- "id": "1b117ead",
- "metadata": {},
- "source": [
- "#### Now, we use `HyDEQueryTransform` to generate a hypothetical document and use it for embedding lookup. "
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "8d43ee7a",
- "metadata": {},
- "outputs": [],
- "source": [
- "hyde = HyDEQueryTransform(include_original=True)\n",
- "response = index.query(query_str, query_transform=hyde)\n",
- "display(Markdown(f\"<b>{response}</b>\"))"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "660c2286",
- "metadata": {},
- "source": [
- "> After going to RISD, Paul Graham worked as a consultant for Interleaf and then co-founded Viaweb with Robert Morris. They created a software that allowed users to build websites via the web and received $10,000 in seed funding from Idelle's husband Julian. They gave Julian 10% of the company in return for the initial legal work and business advice. Paul Graham had a negative net worth due to taxes he owed, so the seed funding was necessary for him to live on. They opened for business in January 1996 with 6 stores.\n",
- "\n",
- "> Paul Graham then left Yahoo after his options vested and went back to New York. He resumed his old life, but now he was rich. He tried to paint, but he didn't have much energy or ambition. He eventually moved back to Cambridge and started working on a web app for making web apps. He recruited Dan Giffin and two undergrads to help him, but he eventually realized he didn't want to run a company and decided to build a subset of the project as an open source project. He and Dan worked on a new dialect of Lisp, which he called Arc, in a house he bought in Cambridge. The subset he built as an open source project was the new Lisp, whose"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "84f4ba19",
- "metadata": {},
- "source": [
- "#### In this example, `HyDE` improves output quality significantly, by hallucinating accurately what Paul Graham did after RISD (see below), and thus improving the embedding quality, and final output."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "a733f80d",
- "metadata": {},
- "outputs": [],
- "source": [
- "query_bundle = hyde(query_str)\n",
- "hyde_doc = query_bundle.embedding_strs[0]"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "30e11fc3",
- "metadata": {},
- "outputs": [],
- "source": [
- "hyde_doc"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "d66a4296",
- "metadata": {},
- "source": [
- "> After graduating from the Rhode Island School of Design (RISD) in 1985, Paul Graham went on to pursue a career in computer programming. He worked as a software developer for several companies, including Viaweb, which he co-founded in 1995. Viaweb was eventually acquired by Yahoo in 1998, and Graham used the proceeds to become a venture capitalist. He founded Y Combinator in 2005, a startup accelerator that has helped launch over 2,000 companies, including Dropbox, Airbnb, and Reddit. Graham has also written several books on programming and startups, and he continues to be an active investor in the tech industry."
- ]
- },
- {
- "cell_type": "markdown",
- "id": "e5099208",
- "metadata": {},
- "source": [
- "## Failure case 1: HyDE may mislead when query can be mis-interpreted without context."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "2e2c0f87",
- "metadata": {},
- "outputs": [],
- "source": [
- "query_str = \"What is Bel?\""
- ]
- },
- {
- "cell_type": "markdown",
- "id": "4bed7470",
- "metadata": {},
- "source": [
- "### Querying without transformation yields reasonable answer"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "da43432f",
- "metadata": {},
- "outputs": [],
- "source": [
- "response = index.query(query_str)\n",
- "display(Markdown(f\"<b>{response}</b>\"))"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "1a7ad6e2",
- "metadata": {},
- "source": [
- "> Bel is a programming language that was written in Arc by Paul Graham over the course of four years (March 26, 2015 to October 12, 2019). It is based on John McCarthy's original Lisp, but with additional features added. It is a spec expressed as code, and is meant to be a formal model of computation, an alternative to the Turing machine."
- ]
- },
- {
- "cell_type": "markdown",
- "id": "9fa43d10",
- "metadata": {},
- "source": [
- "#### Querying with `HyDEQueryTransform` results in nonsense"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "b2f7be02",
- "metadata": {},
- "outputs": [],
- "source": [
- "hyde = HyDEQueryTransform(include_original=False)\n",
- "response = index.query(query_str, query_transform=hyde)\n",
- "display(Markdown(f\"<b>{response}</b>\"))"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "9d910f39",
- "metadata": {},
- "source": [
- "> Bel is the pseudonym of Paul Graham, the author of the context information who was in need of seed funding to live on and was part of a deal that became the model for Y Combinator's."
- ]
- },
- {
- "cell_type": "markdown",
- "id": "59d82d63",
- "metadata": {},
- "source": [
- "#### In this example, `HyDE` mis-interprets Bel without document context (see below), resulting in a completely unrelated embedding string and poor retrieval outcome."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "3e771a56",
- "metadata": {},
- "outputs": [],
- "source": [
- "query_bundle = hyde(query_str)\n",
- "hyde_doc = query_bundle.embedding_strs[0]"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "8f5ca4c7",
- "metadata": {},
- "outputs": [],
- "source": [
- "hyde_doc"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "b879b64b",
- "metadata": {},
- "source": [
- "> Bel is an ancient Semitic god, originating from the Middle East. He is often associated with the sun and is sometimes referred to as the \"Lord of Heaven\". Bel is also known as the god of fertility, abundance, and prosperity. He is often depicted as a bull or a man with a bull\\'s head. In some cultures, Bel is seen as a creator god, responsible for the creation of the universe. He is also associated with the underworld and is sometimes seen as a god of death. Bel is also associated with justice and is often seen as a protector of the innocent. Bel is an important figure in many religions, including Judaism, Christianity, and Islam."
- ]
- },
- {
- "cell_type": "markdown",
- "id": "7b8df34d",
- "metadata": {},
- "source": [
- "## Failure case 2: HyDE may bias open-ended queries"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "3942dc8d",
- "metadata": {},
- "outputs": [],
- "source": [
- "query_str = \"What would the author say about art vs. engineering?\""
- ]
- },
- {
- "cell_type": "markdown",
- "id": "decc9107",
- "metadata": {},
- "source": [
- "#### Querying without transformation yields a reasonable answer"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "168fab08",
- "metadata": {},
- "outputs": [],
- "source": [
- "response = index.query(query_str)\n",
- "display(Markdown(f\"<b>{response}</b>\"))"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "c1e4939b",
- "metadata": {},
- "source": [
- "> The author would likely say that art and engineering are two different disciplines that require different skills and approaches. Art is more focused on expression and creativity, while engineering is more focused on problem-solving and technical knowledge. The author also suggests that art school does not always provide the same level of rigor as engineering school, and that painting students are often encouraged to develop a signature style rather than learn the fundamentals of painting. Furthermore, the author would likely point out that engineering can provide more financial stability than art, as evidenced by the author's own experience of needing seed funding to live on while launching a company."
- ]
- },
- {
- "cell_type": "markdown",
- "id": "d01a0ed4",
- "metadata": {},
- "source": [
- "#### Querying with `HyDEQueryTransform` results in a more biased output"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "0229cf6f",
- "metadata": {},
- "outputs": [],
- "source": [
- "hyde = HyDEQueryTransform(include_original=False)\n",
- "response = index.query(query_str, query_transform=hyde)\n",
- "display(Markdown(f\"<b>{response}</b>\"))"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "16c8f053",
- "metadata": {},
- "source": [
- "> The author would likely say that art is a more lasting and independent form of work than engineering. They mention that software written today will be obsolete in a couple decades, and that systems work does not last. In contrast, they note that paintings can last hundreds of years and that it is possible to make a living as an artist. They also mention that as an artist, you can be truly independent and don't need to have a boss or research funding. Furthermore, they note that art can be a source of income for people who may not have access to traditional forms of employment, such as the model in the example who was able to make a living from modelling and making fakes for a local antique dealer."
- ]
- }
- ],
- "metadata": {
- "kernelspec": {
- "display_name": "Python 3 (ipykernel)",
- "language": "python",
- "name": "python3"
- },
- "language_info": {
- "codemirror_mode": {
- "name": "ipython",
- "version": 3
- },
- "file_extension": ".py",
- "mimetype": "text/x-python",
- "name": "python",
- "nbconvert_exporter": "python",
- "pygments_lexer": "ipython3",
- "version": "3.9.16"
- }
- },
- "nbformat": 4,
- "nbformat_minor": 5
-}
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "id": "f20bdf25",
+ "metadata": {},
+ "source": [
+ "# HyDE Query Transform Demo"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "a1e7e17f",
+ "metadata": {},
+ "source": [
+ "#### Load documents, build the GPTSimpleVectorIndex"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "f7476659",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import logging\n",
+ "import sys\n",
+ "\n",
+ "logging.basicConfig(stream=sys.stdout, level=logging.INFO)\n",
+ "logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout))\n",
+ "\n",
+ "from llama_index import GPTSimpleVectorIndex, SimpleDirectoryReader\n",
+ "from llama_index.indices.query.query_transform import HyDEQueryTransform\n",
+ "from IPython.display import Markdown, display"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "16ff354a",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# load documents\n",
+ "documents = SimpleDirectoryReader('../paul_graham_essay/data').load_data()"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "7d6a1b5e",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "index = GPTSimpleVectorIndex(documents)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "85665424",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# save index to disk\n",
+ "index.save_to_disk('index_simple.json')"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "78ce3cec",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# load index from disk\n",
+ "index = GPTSimpleVectorIndex.load_from_disk('index_simple.json')"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "1cec25b6",
+ "metadata": {},
+ "source": [
+ "## Example: HyDE improves specific temporal queries"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "f3c39ddd",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "query_str = \"what did paul graham do after going to RISD\""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "8b022d86",
+ "metadata": {},
+ "source": [
+ "#### First, we query *without* transformation: The same query string is used for embedding lookup and also summarization."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "a5b93d35",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "response = index.query(query_str)\n",
+ "display(Markdown(f\"<b>{response}</b>\"))"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "d8b9a9db",
+ "metadata": {},
+ "source": [
+ "> After going to RISD, Paul Graham continued to pursue his passion for painting and art. He took classes in the painting department at the Accademia di Belli Arti in Florence, and he also took the entrance exam for the school. He also continued to work on his book On Lisp, and he took on consulting work to make money. At the school, Paul Graham and the other students had an arrangement where the faculty wouldn't require the students to learn anything, and in return the students wouldn't require the faculty to teach anything. Paul Graham was one of the few students who actually painted the nude model that was provided, while the rest of the students spent their time chatting or occasionally trying to imitate things they'd seen in American art magazines. The model turned out to live just down the street from Paul Graham, and she made a living from a combination of modelling and making fakes for a local antique dealer."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "1b117ead",
+ "metadata": {},
+ "source": [
+ "#### Now, we use `HyDEQueryTransform` to generate a hypothetical document and use it for embedding lookup. "
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "8d43ee7a",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "hyde = HyDEQueryTransform(include_original=True)\n",
+ "response = index.query(query_str, query_transform=hyde)\n",
+ "display(Markdown(f\"<b>{response}</b>\"))"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "660c2286",
+ "metadata": {},
+ "source": [
+ "> After going to RISD, Paul Graham worked as a consultant for Interleaf and then co-founded Viaweb with Robert Morris. They created a software that allowed users to build websites via the web and received $10,000 in seed funding from Idelle's husband Julian. They gave Julian 10% of the company in return for the initial legal work and business advice. Paul Graham had a negative net worth due to taxes he owed, so the seed funding was necessary for him to live on. They opened for business in January 1996 with 6 stores.\n",
+ "\n",
+ "> Paul Graham then left Yahoo after his options vested and went back to New York. He resumed his old life, but now he was rich. He tried to paint, but he didn't have much energy or ambition. He eventually moved back to Cambridge and started working on a web app for making web apps. He recruited Dan Giffin and two undergrads to help him, but he eventually realized he didn't want to run a company and decided to build a subset of the project as an open source project. He and Dan worked on a new dialect of Lisp, which he called Arc, in a house he bought in Cambridge. The subset he built as an open source project was the new Lisp, whose"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "84f4ba19",
+ "metadata": {},
+ "source": [
+ "#### In this example, `HyDE` improves output quality significantly, by hallucinating accurately what Paul Graham did after RISD (see below), and thus improving the embedding quality, and final output."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "a733f80d",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "query_bundle = hyde(query_str)\n",
+ "hyde_doc = query_bundle.embedding_strs[0]"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "30e11fc3",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "hyde_doc"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "d66a4296",
+ "metadata": {},
+ "source": [
+ "> After graduating from the Rhode Island School of Design (RISD) in 1985, Paul Graham went on to pursue a career in computer programming. He worked as a software developer for several companies, including Viaweb, which he co-founded in 1995. Viaweb was eventually acquired by Yahoo in 1998, and Graham used the proceeds to become a venture capitalist. He founded Y Combinator in 2005, a startup accelerator that has helped launch over 2,000 companies, including Dropbox, Airbnb, and Reddit. Graham has also written several books on programming and startups, and he continues to be an active investor in the tech industry."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "e5099208",
+ "metadata": {},
+ "source": [
+ "## Failure case 1: HyDE may mislead when query can be mis-interpreted without context."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "2e2c0f87",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "query_str = \"What is Bel?\""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "4bed7470",
+ "metadata": {},
+ "source": [
+ "### Querying without transformation yields reasonable answer"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "da43432f",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "response = index.query(query_str)\n",
+ "display(Markdown(f\"<b>{response}</b>\"))"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "1a7ad6e2",
+ "metadata": {},
+ "source": [
+ "> Bel is a programming language that was written in Arc by Paul Graham over the course of four years (March 26, 2015 to October 12, 2019). It is based on John McCarthy's original Lisp, but with additional features added. It is a spec expressed as code, and is meant to be a formal model of computation, an alternative to the Turing machine."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "9fa43d10",
+ "metadata": {},
+ "source": [
+ "#### Querying with `HyDEQueryTransform` results in nonsense"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "b2f7be02",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "hyde = HyDEQueryTransform(include_original=False)\n",
+ "response = index.query(query_str, query_transform=hyde)\n",
+ "display(Markdown(f\"<b>{response}</b>\"))"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "9d910f39",
+ "metadata": {},
+ "source": [
+ "> Bel is the pseudonym of Paul Graham, the author of the context information who was in need of seed funding to live on and was part of a deal that became the model for Y Combinator's."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "59d82d63",
+ "metadata": {},
+ "source": [
+ "#### In this example, `HyDE` mis-interprets Bel without document context (see below), resulting in a completely unrelated embedding string and poor retrieval outcome."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "3e771a56",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "query_bundle = hyde(query_str)\n",
+ "hyde_doc = query_bundle.embedding_strs[0]"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "8f5ca4c7",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "hyde_doc"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "b879b64b",
+ "metadata": {},
+ "source": [
+ "> Bel is an ancient Semitic god, originating from the Middle East. He is often associated with the sun and is sometimes referred to as the \"Lord of Heaven\". Bel is also known as the god of fertility, abundance, and prosperity. He is often depicted as a bull or a man with a bull\\'s head. In some cultures, Bel is seen as a creator god, responsible for the creation of the universe. He is also associated with the underworld and is sometimes seen as a god of death. Bel is also associated with justice and is often seen as a protector of the innocent. Bel is an important figure in many religions, including Judaism, Christianity, and Islam."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "7b8df34d",
+ "metadata": {},
+ "source": [
+ "## Failure case 2: HyDE may bias open-ended queries"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "3942dc8d",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "query_str = \"What would the author say about art vs. engineering?\""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "decc9107",
+ "metadata": {},
+ "source": [
+ "#### Querying without transformation yields a reasonable answer"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "168fab08",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "response = index.query(query_str)\n",
+ "display(Markdown(f\"<b>{response}</b>\"))"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "c1e4939b",
+ "metadata": {},
+ "source": [
+ "> The author would likely say that art and engineering are two different disciplines that require different skills and approaches. Art is more focused on expression and creativity, while engineering is more focused on problem-solving and technical knowledge. The author also suggests that art school does not always provide the same level of rigor as engineering school, and that painting students are often encouraged to develop a signature style rather than learn the fundamentals of painting. Furthermore, the author would likely point out that engineering can provide more financial stability than art, as evidenced by the author's own experience of needing seed funding to live on while launching a company."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "d01a0ed4",
+ "metadata": {},
+ "source": [
+ "#### Querying with `HyDEQueryTransform` results in a more biased output"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "0229cf6f",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "hyde = HyDEQueryTransform(include_original=False)\n",
+ "response = index.query(query_str, query_transform=hyde)\n",
+ "display(Markdown(f\"<b>{response}</b>\"))"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "16c8f053",
+ "metadata": {},
+ "source": [
+ "> The author would likely say that art is a more lasting and independent form of work than engineering. They mention that software written today will be obsolete in a couple decades, and that systems work does not last. In contrast, they note that paintings can last hundreds of years and that it is possible to make a living as an artist. They also mention that as an artist, you can be truly independent and don't need to have a boss or research funding. Furthermore, they note that art can be a source of income for people who may not have access to traditional forms of employment, such as the model in the example who was able to make a living from modelling and making fakes for a local antique dealer."
+ ]
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3 (ipykernel)",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.9.16"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}
\ No newline at end of file
diff --git a/examples/struct_indices/SQLIndexDemo.ipynb b/examples/struct_indices/SQLIndexDemo.ipynb
index 84b9dbb9c68a857116959a8ec1d785e403897342..751a83143e2b1742b2dee14f8663ec5ebd12ac91 100644
--- a/examples/struct_indices/SQLIndexDemo.ipynb
+++ b/examples/struct_indices/SQLIndexDemo.ipynb
@@ -1,459 +1,459 @@
{
- "cells": [
- {
- "cell_type": "markdown",
- "id": "e45f9b60-cd6b-4c15-958f-1feca5438128",
- "metadata": {},
- "source": [
- "# SQL Index Demo"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "119eb42b",
- "metadata": {},
- "outputs": [],
- "source": [
- "import logging\n",
- "import sys\n",
- "\n",
- "logging.basicConfig(stream=sys.stdout, level=logging.INFO)\n",
- "logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout))"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 19,
- "id": "107396a9-4aa7-49b3-9f0f-a755726c19ba",
- "metadata": {},
- "outputs": [],
- "source": [
- "from gpt_index import SimpleDirectoryReader, WikipediaReader\n",
- "from IPython.display import Markdown, display"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "77ac8d94-cd61-4869-a32b-0b2e7d18b83f",
- "metadata": {},
- "source": [
- "### Load Wikipedia Data"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "93301fcf-a52b-430c-98a3-5360e6c8fc4a",
- "metadata": {},
- "outputs": [],
- "source": [
- "# install wikipedia python package\n",
- "!pip install wikipedia"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 2,
- "id": "ba3f7e5e-cdc4-4529-bba9-db45d8457dba",
- "metadata": {},
- "outputs": [],
- "source": [
- "wiki_docs = WikipediaReader().load_data(pages=['Toronto', 'Berlin', 'Tokyo'])"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "461438c8-302d-45c5-8e69-16ad604686d1",
- "metadata": {},
- "source": [
- "### Create Database Schema"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 3,
- "id": "a370b266-66f5-4624-bbf9-2ad57f0511f8",
- "metadata": {},
- "outputs": [],
- "source": [
- "from sqlalchemy import create_engine, MetaData, Table, Column, String, Integer, select, column"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 4,
- "id": "ea24f794-f10b-42e6-922d-9258b7167405",
- "metadata": {},
- "outputs": [],
- "source": [
- "engine = create_engine(\"sqlite:///:memory:\")\n",
- "metadata_obj = MetaData(bind=engine)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 5,
- "id": "b4154b29-7e23-4c26-a507-370a66186ae7",
- "metadata": {},
- "outputs": [],
- "source": [
- "# create city SQL table\n",
- "table_name = \"city_stats\"\n",
- "city_stats_table = Table(\n",
- " table_name,\n",
- " metadata_obj,\n",
- " Column(\"city_name\", String(16), primary_key=True),\n",
- " Column(\"population\", Integer),\n",
- " Column(\"country\", String(16), nullable=False),\n",
- ")\n",
- "metadata_obj.create_all()"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "1c09089a-6bcd-48db-8120-a84c8da3f82e",
- "metadata": {
- "tags": []
- },
- "source": [
- "### Build Index"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 20,
- "id": "768d1581-b482-4c73-9963-5ffd68a2aafb",
- "metadata": {},
- "outputs": [],
- "source": [
- "from gpt_index import GPTSQLStructStoreIndex, SQLDatabase\n",
- "from langchain import OpenAI\n",
- "from gpt_index import LLMPredictor"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 21,
- "id": "bffabba0-8e54-4f24-ad14-2c8979c582a5",
- "metadata": {},
- "outputs": [],
- "source": [
- "llm_predictor = LLMPredictor(llm=OpenAI(temperature=0, model_name=\"text-davinci-002\"))"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 9,
- "id": "9432787b-a8f0-4fc3-8323-e2cd9497df73",
- "metadata": {},
- "outputs": [],
- "source": [
- "sql_database = SQLDatabase(engine, include_tables=[\"city_stats\"])"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 10,
- "id": "84d4ee54-9f00-40fd-bab0-36e5e579dc9f",
- "metadata": {},
- "outputs": [
- {
- "data": {
- "text/plain": [
- "\"Table 'city_stats' has columns: city_name (VARCHAR(16)), population (INTEGER), country (VARCHAR(16)).\""
- ]
- },
- "execution_count": 10,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "sql_database.table_info"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "95043e10-6cdf-4f66-96bd-ce307ea7df3e",
- "metadata": {},
- "outputs": [],
- "source": [
- "# NOTE: the table_name specified here is the table that you\n",
- "# want to extract into from unstructured documents.\n",
- "index = GPTSQLStructStoreIndex(\n",
- " wiki_docs, \n",
- " sql_database=sql_database, \n",
- " table_name=\"city_stats\",\n",
- " llm_predictor=llm_predictor\n",
- ")"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 12,
- "id": "b315b8ff-7dd7-4e7d-ac47-8c5a0c3e7ae9",
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "[('Toronto', 2731571, 'Canada'), ('Berlin', 600000, 'Germany'), ('Tokyo', 13929286, 'Japan')]\n"
- ]
- }
- ],
- "source": [
- "# view current table\n",
- "stmt = select(\n",
- " [column(\"city_name\"), column(\"population\"), column(\"country\")]\n",
- ").select_from(city_stats_table)\n",
- "\n",
- "with engine.connect() as connection:\n",
- " results = connection.execute(stmt).fetchall()\n",
- " print(results)\n"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "051a171f-8c97-40ed-ae17-4e3fa3785487",
- "metadata": {},
- "source": [
- "### Query Index"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "f6a2303f-3bae-4fa2-8750-03f9af747848",
- "metadata": {},
- "source": [
- "We first show a raw SQL query, which directly executes over the table"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 15,
- "id": "eddd3608-31ff-4591-a02a-90987e312669",
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "> [query] Total LLM token usage: 0 tokens\n",
- "> [query] Total embedding token usage: 0 tokens\n"
- ]
- }
- ],
- "source": [
- "response = index.query(\"SELECT city_name from city_stats\", mode=\"sql\")"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 14,
- "id": "4253d4c3-f3e5-4779-bcd1-2e6e2818305f",
- "metadata": {},
- "outputs": [
- {
- "data": {
- "text/markdown": [
- "<b>[('Berlin',), ('Tokyo',), ('Toronto',)]</b>"
- ],
- "text/plain": [
- "<IPython.core.display.Markdown object>"
- ]
- },
- "metadata": {},
- "output_type": "display_data"
- }
- ],
- "source": [
- "display(Markdown(f\"<b>{response}</b>\"))"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "abff69de-80c4-4fe6-afa1-b3d7208a5c4c",
- "metadata": {},
- "source": [
- "Here we show a natural language query, which is translated to a SQL query under the hood"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 16,
- "id": "d71045c0-7a96-4e86-b38c-c378b7759aa4",
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "> Predicted SQL query: SELECT city_name, population\n",
- "FROM city_stats\n",
- "ORDER BY population DESC\n",
- "LIMIT 1\n",
- "> [query] Total LLM token usage: 144 tokens\n",
- "> [query] Total embedding token usage: 0 tokens\n"
- ]
- }
- ],
- "source": [
- "# set Logging to DEBUG for more detailed outputs\n",
- "response = index.query(\"Which city has the highest population?\", mode=\"default\")"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 23,
- "id": "e713d73e-73ed-4748-8673-f476899fac8e",
- "metadata": {},
- "outputs": [
- {
- "data": {
- "text/markdown": [
- "<b>[('Tokyo', 13929286)]</b>"
- ],
- "text/plain": [
- "<IPython.core.display.Markdown object>"
- ]
- },
- "metadata": {},
- "output_type": "display_data"
- }
- ],
- "source": [
- "display(Markdown(f\"<b>{response}</b>\"))"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 18,
- "id": "54a99cb0-578a-40ec-a3eb-1666ac18fbed",
- "metadata": {},
- "outputs": [
- {
- "data": {
- "text/plain": [
- "[('Tokyo', 13929286)]"
- ]
- },
- "execution_count": 18,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "# you can also fetch the raw result from SQLAlchemy! \n",
- "response.extra_info[\"result\"]"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "2a567566-9828-4aa4-afde-b253c01eda69",
- "metadata": {},
- "source": [
- "### Using Langchain for Querying\n",
- "\n",
- "Since our SQLDatabase inherits from langchain, you can also use langchain itself for querying purposes."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 22,
- "id": "8e0acde4-ca61-42e9-97f8-c9cf11502157",
- "metadata": {},
- "outputs": [],
- "source": [
- "from langchain import OpenAI, SQLDatabase, SQLDatabaseChain"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 24,
- "id": "30860e8b-9ad0-418c-b266-753242c1f208",
- "metadata": {},
- "outputs": [],
- "source": [
- "llm = OpenAI(temperature=0)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 26,
- "id": "07068a3a-30a4-4473-ba82-ab6e93e3437c",
- "metadata": {},
- "outputs": [],
- "source": [
- "# set Logging to DEBUG for more detailed outputs\n",
- "db_chain = SQLDatabaseChain(llm=llm, database=sql_database)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 27,
- "id": "a04c0a1d-f6a8-4a4a-9181-4123b09ec614",
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "\n",
- "\n",
- "\u001b[1m> Entering new SQLDatabaseChain chain...\u001b[0m\n",
- "Which city has the highest population? \n",
- "SQLQuery:\u001b[32;1m\u001b[1;3m SELECT city_name FROM city_stats ORDER BY population DESC LIMIT 1;\u001b[0m\n",
- "SQLResult: \u001b[33;1m\u001b[1;3m[('Tokyo',)]\u001b[0m\n",
- "Answer:\u001b[32;1m\u001b[1;3m Tokyo has the highest population.\u001b[0m\n",
- "\u001b[1m> Finished chain.\u001b[0m\n"
- ]
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "id": "e45f9b60-cd6b-4c15-958f-1feca5438128",
+ "metadata": {},
+ "source": [
+ "# SQL Index Demo"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "119eb42b",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import logging\n",
+ "import sys\n",
+ "\n",
+ "logging.basicConfig(stream=sys.stdout, level=logging.INFO)\n",
+ "logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout))"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 19,
+ "id": "107396a9-4aa7-49b3-9f0f-a755726c19ba",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from llama_index import SimpleDirectoryReader, WikipediaReader\n",
+ "from IPython.display import Markdown, display"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "77ac8d94-cd61-4869-a32b-0b2e7d18b83f",
+ "metadata": {},
+ "source": [
+ "### Load Wikipedia Data"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "93301fcf-a52b-430c-98a3-5360e6c8fc4a",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# install wikipedia python package\n",
+ "!pip install wikipedia"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 2,
+ "id": "ba3f7e5e-cdc4-4529-bba9-db45d8457dba",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "wiki_docs = WikipediaReader().load_data(pages=['Toronto', 'Berlin', 'Tokyo'])"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "461438c8-302d-45c5-8e69-16ad604686d1",
+ "metadata": {},
+ "source": [
+ "### Create Database Schema"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 3,
+ "id": "a370b266-66f5-4624-bbf9-2ad57f0511f8",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from sqlalchemy import create_engine, MetaData, Table, Column, String, Integer, select, column"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 4,
+ "id": "ea24f794-f10b-42e6-922d-9258b7167405",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "engine = create_engine(\"sqlite:///:memory:\")\n",
+ "metadata_obj = MetaData(bind=engine)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 5,
+ "id": "b4154b29-7e23-4c26-a507-370a66186ae7",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# create city SQL table\n",
+ "table_name = \"city_stats\"\n",
+ "city_stats_table = Table(\n",
+ " table_name,\n",
+ " metadata_obj,\n",
+ " Column(\"city_name\", String(16), primary_key=True),\n",
+ " Column(\"population\", Integer),\n",
+ " Column(\"country\", String(16), nullable=False),\n",
+ ")\n",
+ "metadata_obj.create_all()"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "1c09089a-6bcd-48db-8120-a84c8da3f82e",
+ "metadata": {
+ "tags": []
+ },
+ "source": [
+ "### Build Index"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 20,
+ "id": "768d1581-b482-4c73-9963-5ffd68a2aafb",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from llama_index import GPTSQLStructStoreIndex, SQLDatabase\n",
+ "from langchain import OpenAI\n",
+ "from llama_index import LLMPredictor"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 21,
+ "id": "bffabba0-8e54-4f24-ad14-2c8979c582a5",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "llm_predictor = LLMPredictor(llm=OpenAI(temperature=0, model_name=\"text-davinci-002\"))"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 9,
+ "id": "9432787b-a8f0-4fc3-8323-e2cd9497df73",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "sql_database = SQLDatabase(engine, include_tables=[\"city_stats\"])"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 10,
+ "id": "84d4ee54-9f00-40fd-bab0-36e5e579dc9f",
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "\"Table 'city_stats' has columns: city_name (VARCHAR(16)), population (INTEGER), country (VARCHAR(16)).\""
+ ]
+ },
+ "execution_count": 10,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "sql_database.table_info"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "95043e10-6cdf-4f66-96bd-ce307ea7df3e",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# NOTE: the table_name specified here is the table that you\n",
+ "# want to extract into from unstructured documents.\n",
+ "index = GPTSQLStructStoreIndex(\n",
+ " wiki_docs, \n",
+ " sql_database=sql_database, \n",
+ " table_name=\"city_stats\",\n",
+ " llm_predictor=llm_predictor\n",
+ ")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 12,
+ "id": "b315b8ff-7dd7-4e7d-ac47-8c5a0c3e7ae9",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "[('Toronto', 2731571, 'Canada'), ('Berlin', 600000, 'Germany'), ('Tokyo', 13929286, 'Japan')]\n"
+ ]
+ }
+ ],
+ "source": [
+ "# view current table\n",
+ "stmt = select(\n",
+ " [column(\"city_name\"), column(\"population\"), column(\"country\")]\n",
+ ").select_from(city_stats_table)\n",
+ "\n",
+ "with engine.connect() as connection:\n",
+ " results = connection.execute(stmt).fetchall()\n",
+ " print(results)\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "051a171f-8c97-40ed-ae17-4e3fa3785487",
+ "metadata": {},
+ "source": [
+ "### Query Index"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "f6a2303f-3bae-4fa2-8750-03f9af747848",
+ "metadata": {},
+ "source": [
+ "We first show a raw SQL query, which directly executes over the table"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 15,
+ "id": "eddd3608-31ff-4591-a02a-90987e312669",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "> [query] Total LLM token usage: 0 tokens\n",
+ "> [query] Total embedding token usage: 0 tokens\n"
+ ]
+ }
+ ],
+ "source": [
+ "response = index.query(\"SELECT city_name from city_stats\", mode=\"sql\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 14,
+ "id": "4253d4c3-f3e5-4779-bcd1-2e6e2818305f",
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/markdown": [
+ "<b>[('Berlin',), ('Tokyo',), ('Toronto',)]</b>"
+ ],
+ "text/plain": [
+ "<IPython.core.display.Markdown object>"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ }
+ ],
+ "source": [
+ "display(Markdown(f\"<b>{response}</b>\"))"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "abff69de-80c4-4fe6-afa1-b3d7208a5c4c",
+ "metadata": {},
+ "source": [
+ "Here we show a natural language query, which is translated to a SQL query under the hood"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 16,
+ "id": "d71045c0-7a96-4e86-b38c-c378b7759aa4",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "> Predicted SQL query: SELECT city_name, population\n",
+ "FROM city_stats\n",
+ "ORDER BY population DESC\n",
+ "LIMIT 1\n",
+ "> [query] Total LLM token usage: 144 tokens\n",
+ "> [query] Total embedding token usage: 0 tokens\n"
+ ]
+ }
+ ],
+ "source": [
+ "# set Logging to DEBUG for more detailed outputs\n",
+ "response = index.query(\"Which city has the highest population?\", mode=\"default\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 23,
+ "id": "e713d73e-73ed-4748-8673-f476899fac8e",
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/markdown": [
+ "<b>[('Tokyo', 13929286)]</b>"
+ ],
+ "text/plain": [
+ "<IPython.core.display.Markdown object>"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ }
+ ],
+ "source": [
+ "display(Markdown(f\"<b>{response}</b>\"))"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 18,
+ "id": "54a99cb0-578a-40ec-a3eb-1666ac18fbed",
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "[('Tokyo', 13929286)]"
+ ]
+ },
+ "execution_count": 18,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "# you can also fetch the raw result from SQLAlchemy! \n",
+ "response.extra_info[\"result\"]"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "2a567566-9828-4aa4-afde-b253c01eda69",
+ "metadata": {},
+ "source": [
+ "### Using Langchain for Querying\n",
+ "\n",
+ "Since our SQLDatabase inherits from langchain, you can also use langchain itself for querying purposes."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 22,
+ "id": "8e0acde4-ca61-42e9-97f8-c9cf11502157",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from langchain import OpenAI, SQLDatabase, SQLDatabaseChain"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 24,
+ "id": "30860e8b-9ad0-418c-b266-753242c1f208",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "llm = OpenAI(temperature=0)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 26,
+ "id": "07068a3a-30a4-4473-ba82-ab6e93e3437c",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# set Logging to DEBUG for more detailed outputs\n",
+ "db_chain = SQLDatabaseChain(llm=llm, database=sql_database)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 27,
+ "id": "a04c0a1d-f6a8-4a4a-9181-4123b09ec614",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "\n",
+ "\n",
+ "\u001b[1m> Entering new SQLDatabaseChain chain...\u001b[0m\n",
+ "Which city has the highest population? \n",
+ "SQLQuery:\u001b[32;1m\u001b[1;3m SELECT city_name FROM city_stats ORDER BY population DESC LIMIT 1;\u001b[0m\n",
+ "SQLResult: \u001b[33;1m\u001b[1;3m[('Tokyo',)]\u001b[0m\n",
+ "Answer:\u001b[32;1m\u001b[1;3m Tokyo has the highest population.\u001b[0m\n",
+ "\u001b[1m> Finished chain.\u001b[0m\n"
+ ]
+ },
+ {
+ "data": {
+ "text/plain": [
+ "' Tokyo has the highest population.'"
+ ]
+ },
+ "execution_count": 27,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "db_chain.run(\"Which city has the highest population?\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "7eeb4af4-d18e-4cf4-b1cf-1347420e24fb",
+ "metadata": {},
+ "outputs": [],
+ "source": []
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3 (ipykernel)",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.11.1"
+ }
},
- {
- "data": {
- "text/plain": [
- "' Tokyo has the highest population.'"
- ]
- },
- "execution_count": 27,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "db_chain.run(\"Which city has the highest population?\")"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "7eeb4af4-d18e-4cf4-b1cf-1347420e24fb",
- "metadata": {},
- "outputs": [],
- "source": []
- }
- ],
- "metadata": {
- "kernelspec": {
- "display_name": "Python 3 (ipykernel)",
- "language": "python",
- "name": "python3"
- },
- "language_info": {
- "codemirror_mode": {
- "name": "ipython",
- "version": 3
- },
- "file_extension": ".py",
- "mimetype": "text/x-python",
- "name": "python",
- "nbconvert_exporter": "python",
- "pygments_lexer": "ipython3",
- "version": "3.11.1"
- }
- },
- "nbformat": 4,
- "nbformat_minor": 5
-}
+ "nbformat": 4,
+ "nbformat_minor": 5
+}
\ No newline at end of file
diff --git a/examples/test_wiki/TestNYC.ipynb b/examples/test_wiki/TestNYC.ipynb
index 9e1d008eeb7496f723ee497ddcb074b141312a2c..ae55432aa453f9a6b2408266645b7be3dc4dc31b 100644
--- a/examples/test_wiki/TestNYC.ipynb
+++ b/examples/test_wiki/TestNYC.ipynb
@@ -1,180 +1,180 @@
{
- "cells": [
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "9080b39e",
- "metadata": {},
- "outputs": [],
- "source": [
- "import logging\n",
- "import sys\n",
- "\n",
- "logging.basicConfig(stream=sys.stdout, level=logging.INFO)\n",
- "logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout))"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "b4b4387b-413e-4016-ba1e-88b3d9410a38",
- "metadata": {},
- "outputs": [],
- "source": [
- "# fetch \"New York City\" page from Wikipedia\n",
- "from pathlib import Path\n",
- "\n",
- "import requests\n",
- "response = requests.get(\n",
- " 'https://en.wikipedia.org/w/api.php',\n",
- " params={\n",
- " 'action': 'query',\n",
- " 'format': 'json',\n",
- " 'titles': 'New York City',\n",
- " 'prop': 'extracts',\n",
- " # 'exintro': True,\n",
- " 'explaintext': True,\n",
- " }\n",
- ").json()\n",
- "page = next(iter(response['query']['pages'].values()))\n",
- "nyc_text = page['extract']\n",
- "\n",
- "data_path = Path('data')\n",
- "if not data_path.exists():\n",
- " Path.mkdir(data_path)\n",
- "\n",
- "with open('data/nyc_text.txt', 'w') as fp:\n",
- " fp.write(nyc_text)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "f1a9eb90-335c-4214-8bb6-fd1edbe3ccbd",
- "metadata": {},
- "outputs": [],
- "source": [
- "# My OpenAI Key\n",
- "import os\n",
- "os.environ['OPENAI_API_KEY'] = \"INSERT OPENAI KEY\""
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "8d0b2364-4806-4656-81e7-3f6e4b910b5b",
- "metadata": {},
- "outputs": [],
- "source": [
- "from gpt_index import GPTTreeIndex, SimpleDirectoryReader"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "1298bbb4-c99e-431e-93ef-eb32c0a2fc2a",
- "metadata": {
- "tags": []
- },
- "outputs": [],
- "source": [
- "documents = SimpleDirectoryReader('data').load_data()\n",
- "index = GPTTreeIndex(documents)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "0b4fe9b6-5762-4e86-b51e-aac45d3ecdb1",
- "metadata": {},
- "outputs": [],
- "source": [
- "index.save_to_disk('index.json')"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "5eec265d-211b-4f26-b05b-5b4e7072bc6e",
- "metadata": {},
- "outputs": [],
- "source": [
- "# try loading\n",
- "new_index = GPTTreeIndex.load_from_disk('index.json')"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "68c9ebfe-b1b6-4f4e-9278-174346de8c90",
- "metadata": {
- "tags": []
- },
- "outputs": [],
- "source": [
- "# GPT doesn't find the corresponding evidence in the leaf node, but still gives the correct answer\n",
- "# set Logging to DEBUG for more detailed outputs\n",
- "\n",
- "new_index.query(\"What is the name of the professional women's basketball team in New York City?\")"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "4fc3f18a-0ef9-453c-acf8-7aedd784cdcf",
- "metadata": {},
- "outputs": [],
- "source": [
- "# GPT doesn't find the corresponding evidence in the leaf node, but still gives the correct answer\n",
- "# set Logging to DEBUG for more detailed outputs\n",
- "\n",
- "new_index.query(\"What battles took place in New York City in the American Revolution?\")"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "97f3ddf1-8dc2-4fb8-831f-2c06649e0955",
- "metadata": {},
- "outputs": [],
- "source": [
- "# GPT doesn't find the corresponding evidence in the leaf node, but still gives the correct answer\n",
- "# set Logging to DEBUG for more detailed outputs\n",
- "\n",
- "new_index.query(\"What are the airports in New York City?\")"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "53265fd4-da98-4cf9-abfb-3f76105fd2ff",
- "metadata": {},
- "outputs": [],
- "source": [
- "# Try using embedding query\n",
- "new_index.query(\"What are the airports in New York City?\", mode=\"embedding\")"
- ]
- }
- ],
- "metadata": {
- "kernelspec": {
- "display_name": "Python 3 (ipykernel)",
- "language": "python",
- "name": "python3"
- },
- "language_info": {
- "codemirror_mode": {
- "name": "ipython",
- "version": 3
- },
- "file_extension": ".py",
- "mimetype": "text/x-python",
- "name": "python",
- "nbconvert_exporter": "python",
- "pygments_lexer": "ipython3",
- "version": "3.11.1"
- }
- },
- "nbformat": 4,
- "nbformat_minor": 5
-}
+ "cells": [
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "9080b39e",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import logging\n",
+ "import sys\n",
+ "\n",
+ "logging.basicConfig(stream=sys.stdout, level=logging.INFO)\n",
+ "logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout))"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "b4b4387b-413e-4016-ba1e-88b3d9410a38",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# fetch \"New York City\" page from Wikipedia\n",
+ "from pathlib import Path\n",
+ "\n",
+ "import requests\n",
+ "response = requests.get(\n",
+ " 'https://en.wikipedia.org/w/api.php',\n",
+ " params={\n",
+ " 'action': 'query',\n",
+ " 'format': 'json',\n",
+ " 'titles': 'New York City',\n",
+ " 'prop': 'extracts',\n",
+ " # 'exintro': True,\n",
+ " 'explaintext': True,\n",
+ " }\n",
+ ").json()\n",
+ "page = next(iter(response['query']['pages'].values()))\n",
+ "nyc_text = page['extract']\n",
+ "\n",
+ "data_path = Path('data')\n",
+ "if not data_path.exists():\n",
+ " Path.mkdir(data_path)\n",
+ "\n",
+ "with open('data/nyc_text.txt', 'w') as fp:\n",
+ " fp.write(nyc_text)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "f1a9eb90-335c-4214-8bb6-fd1edbe3ccbd",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# My OpenAI Key\n",
+ "import os\n",
+ "os.environ['OPENAI_API_KEY'] = \"INSERT OPENAI KEY\""
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "8d0b2364-4806-4656-81e7-3f6e4b910b5b",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from llama_index import GPTTreeIndex, SimpleDirectoryReader"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "1298bbb4-c99e-431e-93ef-eb32c0a2fc2a",
+ "metadata": {
+ "tags": []
+ },
+ "outputs": [],
+ "source": [
+ "documents = SimpleDirectoryReader('data').load_data()\n",
+ "index = GPTTreeIndex(documents)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "0b4fe9b6-5762-4e86-b51e-aac45d3ecdb1",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "index.save_to_disk('index.json')"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "5eec265d-211b-4f26-b05b-5b4e7072bc6e",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# try loading\n",
+ "new_index = GPTTreeIndex.load_from_disk('index.json')"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "68c9ebfe-b1b6-4f4e-9278-174346de8c90",
+ "metadata": {
+ "tags": []
+ },
+ "outputs": [],
+ "source": [
+ "# GPT doesn't find the corresponding evidence in the leaf node, but still gives the correct answer\n",
+ "# set Logging to DEBUG for more detailed outputs\n",
+ "\n",
+ "new_index.query(\"What is the name of the professional women's basketball team in New York City?\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "4fc3f18a-0ef9-453c-acf8-7aedd784cdcf",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# GPT doesn't find the corresponding evidence in the leaf node, but still gives the correct answer\n",
+ "# set Logging to DEBUG for more detailed outputs\n",
+ "\n",
+ "new_index.query(\"What battles took place in New York City in the American Revolution?\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "97f3ddf1-8dc2-4fb8-831f-2c06649e0955",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# GPT doesn't find the corresponding evidence in the leaf node, but still gives the correct answer\n",
+ "# set Logging to DEBUG for more detailed outputs\n",
+ "\n",
+ "new_index.query(\"What are the airports in New York City?\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "53265fd4-da98-4cf9-abfb-3f76105fd2ff",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Try using embedding query\n",
+ "new_index.query(\"What are the airports in New York City?\", mode=\"embedding\")"
+ ]
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3 (ipykernel)",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.11.1"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}
\ No newline at end of file
diff --git a/examples/test_wiki/TestWikiReader.ipynb b/examples/test_wiki/TestWikiReader.ipynb
index 3c4d30f58f8e165eba2b77525d2aaa07a0ff4504..ba33da5ce0ad4e0ad497b76967a770b60c42d5b7 100644
--- a/examples/test_wiki/TestWikiReader.ipynb
+++ b/examples/test_wiki/TestWikiReader.ipynb
@@ -1,289 +1,289 @@
{
- "cells": [
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "52295407",
- "metadata": {},
- "outputs": [],
- "source": [
- "import logging\n",
- "import sys\n",
- "\n",
- "logging.basicConfig(stream=sys.stdout, level=logging.INFO)\n",
- "logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout))"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "c5d167a5-81f8-4d2c-b42f-0a190577132f",
- "metadata": {},
- "outputs": [],
- "source": [
- "# My OpenAI Key\n",
- "import os\n",
- "os.environ['OPENAI_API_KEY'] = \"INSERT OPENAI KEY\""
- ]
- },
- {
- "cell_type": "markdown",
- "id": "575750cc-479f-4b1f-b93f-4b00ed756d52",
- "metadata": {},
- "source": [
- "## Wikipedia Reader + Keyword Table"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 36,
- "id": "5f60348e-731d-4a95-bae2-426e184a914e",
- "metadata": {},
- "outputs": [],
- "source": [
- "from gpt_index import GPTKeywordTableIndex, WikipediaReader"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 37,
- "id": "952c4659-7fbb-447e-8caf-06916412cc37",
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "page: Covid-19\n"
- ]
- }
- ],
- "source": [
- "wiki_docs = WikipediaReader().load_data(pages=['Covid-19'])"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "3be202db-a4c7-41d2-ba7d-446d1f934830",
- "metadata": {},
- "outputs": [],
- "source": [
- "index = GPTKeywordTableIndex(wiki_docs)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 42,
- "id": "7f5667a9-6758-447b-9af2-5e5a4d008a29",
- "metadata": {},
- "outputs": [],
- "source": [
- "# save index to docs\n",
- "index.save_to_disk('index_covid.json')"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 45,
- "id": "77340460-8319-474f-91eb-545ea5790127",
- "metadata": {},
- "outputs": [],
- "source": [
- "new_index = GPTKeywordTableIndex.load_from_disk('index_covid.json')"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 46,
- "id": "28d7163e-f26f-4ad8-89d5-9cb7662c4d9c",
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "> Starting query: Which country included tocilizumab in treatment for covid-19?\n",
- "Extracted keywords: ['tocilizumab', 'treatment', 'covid-19', 'covid', '19']\n",
- "> Querying with idx: 1105763466456338724: of age or older weighing at least 40 kilograms ...\n",
- "> Querying with idx: 2820318727532393752: Coronavirus disease 2019 (COVID-19) is a contag...\n",
- "> Querying with idx: 897499143815831368: if the mask includes an exhalation valve, a wea...\n",
- "> Querying with idx: 8628144746434065339: pulmonary fibrosis, cystic fibrosis. Evidence s...\n"
- ]
+ "cells": [
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "52295407",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import logging\n",
+ "import sys\n",
+ "\n",
+ "logging.basicConfig(stream=sys.stdout, level=logging.INFO)\n",
+ "logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout))"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "c5d167a5-81f8-4d2c-b42f-0a190577132f",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# My OpenAI Key\n",
+ "import os\n",
+ "os.environ['OPENAI_API_KEY'] = \"INSERT OPENAI KEY\""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "575750cc-479f-4b1f-b93f-4b00ed756d52",
+ "metadata": {},
+ "source": [
+ "## Wikipedia Reader + Keyword Table"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 36,
+ "id": "5f60348e-731d-4a95-bae2-426e184a914e",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from llama_index import GPTKeywordTableIndex, WikipediaReader"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 37,
+ "id": "952c4659-7fbb-447e-8caf-06916412cc37",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "page: Covid-19\n"
+ ]
+ }
+ ],
+ "source": [
+ "wiki_docs = WikipediaReader().load_data(pages=['Covid-19'])"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "3be202db-a4c7-41d2-ba7d-446d1f934830",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "index = GPTKeywordTableIndex(wiki_docs)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 42,
+ "id": "7f5667a9-6758-447b-9af2-5e5a4d008a29",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# save index to docs\n",
+ "index.save_to_disk('index_covid.json')"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 45,
+ "id": "77340460-8319-474f-91eb-545ea5790127",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "new_index = GPTKeywordTableIndex.load_from_disk('index_covid.json')"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 46,
+ "id": "28d7163e-f26f-4ad8-89d5-9cb7662c4d9c",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "> Starting query: Which country included tocilizumab in treatment for covid-19?\n",
+ "Extracted keywords: ['tocilizumab', 'treatment', 'covid-19', 'covid', '19']\n",
+ "> Querying with idx: 1105763466456338724: of age or older weighing at least 40 kilograms ...\n",
+ "> Querying with idx: 2820318727532393752: Coronavirus disease 2019 (COVID-19) is a contag...\n",
+ "> Querying with idx: 897499143815831368: if the mask includes an exhalation valve, a wea...\n",
+ "> Querying with idx: 8628144746434065339: pulmonary fibrosis, cystic fibrosis. Evidence s...\n"
+ ]
+ },
+ {
+ "data": {
+ "text/plain": [
+ "'\\n\\nChina'"
+ ]
+ },
+ "execution_count": 46,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "# GPT doesn't find the corresponding evidence in the leaf node, but still gives the correct answer\n",
+ "# set Logging to DEBUG for more detailed outputs\n",
+ "new_index.query(\"Which country included tocilizumab in treatment for covid-19?\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "addb0c4d-f1ae-40c1-8b69-5a989609672f",
+ "metadata": {},
+ "source": [
+ "## Wikipedia Reader + List"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "a0fc24e1-eca5-4267-a962-f7fe0fc5c7df",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from llama_index import GPTListIndex, WikipediaReader"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 3,
+ "id": "872a651a-ca4a-43e2-8b29-e4f667f9d3c5",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "wiki_docs = WikipediaReader().load_data(pages=['Covid-19'])"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 4,
+ "id": "37e85af0-b1c3-4c18-b239-6e32a7acf8d6",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "> Adding chunk: Coronavirus disease 2019 (COVID-19) is a contag...\n",
+ "> Adding chunk: people with COVID‑19 and acute respiratory dist...\n",
+ "> Adding chunk: encourage or mandate the use of face masks or c...\n",
+ "> Adding chunk: have elevated liver enzymes, reflecting liver i...\n",
+ "> Adding chunk: insofar as their drug use may have caused lung ...\n",
+ "> Adding chunk: treatment of mild-to-moderate COVID‑19 in adult...\n"
+ ]
+ }
+ ],
+ "source": [
+ "index = GPTListIndex(wiki_docs)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 13,
+ "id": "ec0119ef-786e-40ea-89af-f1ca0ad26de6",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "> Starting query: Which country included tocilizumab in treatment for covid-19?\n"
+ ]
+ }
+ ],
+ "source": [
+ "# set Logging to DEBUG for more detailed outputs\n",
+ "# with keyword lookup\n",
+ "response = index.query(\n",
+ " \"Which country included tocilizumab in treatment for covid-19?\", \n",
+ " required_keywords=[\"tocilizumab\"]\n",
+ ")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 18,
+ "id": "b4087a84-0939-444f-93f2-a1a7aa32db3f",
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "'China'"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ }
+ ],
+ "source": [
+ "display(response.strip())"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 19,
+ "id": "fb155bc7-cb50-47b6-b92b-895852c2d8f4",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "> Starting query: Which country included tocilizumab in treatment for covid-19?\n"
+ ]
+ }
+ ],
+ "source": [
+ "# set Logging to DEBUG for more detailed outputs\n",
+ "# without keyword lookup\n",
+ "response = index.query(\n",
+ " \"Which country included tocilizumab in treatment for covid-19?\"\n",
+ ")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 20,
+ "id": "5b45c07a-4e76-4a45-86b6-6b2df1ef4f7b",
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "'There is no definite answer to this question as different countries have different treatment methods for covid-19. However, according to the context information, it is known that the virus SARS-CoV-2 can cause severe damage to various organs in the human body by inducing systemic inflammation. Therefore, it is possible that tocilizumab, which is a drug that inhibits the virus, may be included in treatment for covid-19 in some countries in order to prevent or reduce the severity of a cytokine storm. Additionally, passive antibodies may be used to treat people with active COVID-19 in order to help them recover.'"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ }
+ ],
+ "source": [
+ "display(response.strip())"
+ ]
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3 (ipykernel)",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.11.1"
+ }
},
- {
- "data": {
- "text/plain": [
- "'\\n\\nChina'"
- ]
- },
- "execution_count": 46,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "# GPT doesn't find the corresponding evidence in the leaf node, but still gives the correct answer\n",
- "# set Logging to DEBUG for more detailed outputs\n",
- "new_index.query(\"Which country included tocilizumab in treatment for covid-19?\")"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "addb0c4d-f1ae-40c1-8b69-5a989609672f",
- "metadata": {},
- "source": [
- "## Wikipedia Reader + List"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "a0fc24e1-eca5-4267-a962-f7fe0fc5c7df",
- "metadata": {},
- "outputs": [],
- "source": [
- "from gpt_index import GPTListIndex, WikipediaReader"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 3,
- "id": "872a651a-ca4a-43e2-8b29-e4f667f9d3c5",
- "metadata": {},
- "outputs": [],
- "source": [
- "wiki_docs = WikipediaReader().load_data(pages=['Covid-19'])"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 4,
- "id": "37e85af0-b1c3-4c18-b239-6e32a7acf8d6",
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "> Adding chunk: Coronavirus disease 2019 (COVID-19) is a contag...\n",
- "> Adding chunk: people with COVID‑19 and acute respiratory dist...\n",
- "> Adding chunk: encourage or mandate the use of face masks or c...\n",
- "> Adding chunk: have elevated liver enzymes, reflecting liver i...\n",
- "> Adding chunk: insofar as their drug use may have caused lung ...\n",
- "> Adding chunk: treatment of mild-to-moderate COVID‑19 in adult...\n"
- ]
- }
- ],
- "source": [
- "index = GPTListIndex(wiki_docs)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 13,
- "id": "ec0119ef-786e-40ea-89af-f1ca0ad26de6",
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "> Starting query: Which country included tocilizumab in treatment for covid-19?\n"
- ]
- }
- ],
- "source": [
- "# set Logging to DEBUG for more detailed outputs\n",
- "# with keyword lookup\n",
- "response = index.query(\n",
- " \"Which country included tocilizumab in treatment for covid-19?\", \n",
- " required_keywords=[\"tocilizumab\"]\n",
- ")"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 18,
- "id": "b4087a84-0939-444f-93f2-a1a7aa32db3f",
- "metadata": {},
- "outputs": [
- {
- "data": {
- "text/plain": [
- "'China'"
- ]
- },
- "metadata": {},
- "output_type": "display_data"
- }
- ],
- "source": [
- "display(response.strip())"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 19,
- "id": "fb155bc7-cb50-47b6-b92b-895852c2d8f4",
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "> Starting query: Which country included tocilizumab in treatment for covid-19?\n"
- ]
- }
- ],
- "source": [
- "# set Logging to DEBUG for more detailed outputs\n",
- "# without keyword lookup\n",
- "response = index.query(\n",
- " \"Which country included tocilizumab in treatment for covid-19?\"\n",
- ")"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 20,
- "id": "5b45c07a-4e76-4a45-86b6-6b2df1ef4f7b",
- "metadata": {},
- "outputs": [
- {
- "data": {
- "text/plain": [
- "'There is no definite answer to this question as different countries have different treatment methods for covid-19. However, according to the context information, it is known that the virus SARS-CoV-2 can cause severe damage to various organs in the human body by inducing systemic inflammation. Therefore, it is possible that tocilizumab, which is a drug that inhibits the virus, may be included in treatment for covid-19 in some countries in order to prevent or reduce the severity of a cytokine storm. Additionally, passive antibodies may be used to treat people with active COVID-19 in order to help them recover.'"
- ]
- },
- "metadata": {},
- "output_type": "display_data"
- }
- ],
- "source": [
- "display(response.strip())"
- ]
- }
- ],
- "metadata": {
- "kernelspec": {
- "display_name": "Python 3 (ipykernel)",
- "language": "python",
- "name": "python3"
- },
- "language_info": {
- "codemirror_mode": {
- "name": "ipython",
- "version": 3
- },
- "file_extension": ".py",
- "mimetype": "text/x-python",
- "name": "python",
- "nbconvert_exporter": "python",
- "pygments_lexer": "ipython3",
- "version": "3.11.1"
- }
- },
- "nbformat": 4,
- "nbformat_minor": 5
-}
+ "nbformat": 4,
+ "nbformat_minor": 5
+}
\ No newline at end of file
diff --git a/experimental/classifier/TitanicModel.ipynb b/experimental/classifier/TitanicModel.ipynb
index 09e8676ed9c2cd06b8836e56e3473f5b54e05398..394f2e8b7b6c2dcc238e11d03edfbd452d8da283 100644
--- a/experimental/classifier/TitanicModel.ipynb
+++ b/experimental/classifier/TitanicModel.ipynb
@@ -1,552 +1,552 @@
{
- "cells": [
- {
- "cell_type": "code",
- "execution_count": 1,
- "id": "f445c1d1-acb9-431e-a7ff-50c41f064359",
- "metadata": {},
- "outputs": [
- {
- "name": "stderr",
- "output_type": "stream",
- "text": [
- "None of PyTorch, TensorFlow >= 2.0, or Flax have been found. Models won't be available and only tokenizers, configuration and file/data utilities can be used.\n",
- "[nltk_data] Downloading package stopwords to\n",
- "[nltk_data] /Users/jerryliu/nltk_data...\n",
- "[nltk_data] Package stopwords is already up-to-date!\n"
- ]
- }
- ],
- "source": [
- "from utils import (\n",
- " get_train_str,\n",
- " get_train_and_eval_data,\n",
- " get_eval_preds,\n",
- " train_prompt\n",
- ")\n",
- "\n",
- "import warnings\n",
- "warnings.filterwarnings('ignore')\n",
- "warnings.simplefilter('ignore')"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 2,
- "id": "cf3cbd90-d5e1-4c30-a3bc-8b39fbd85d70",
- "metadata": {},
- "outputs": [],
- "source": [
- "# load up the titanic data\n",
- "train_df, train_labels, eval_df, eval_labels = get_train_and_eval_data('data/train.csv')"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "fa2634f9-cb33-4f1e-81f9-3a3b285e2580",
- "metadata": {
- "tags": []
- },
- "source": [
- "## Few-shot Prompting with GPT-3 for Titanic Dataset\n",
- "In this section, we can show how we can prompt GPT-3 on its own (without using GPT Index) to attain ~80% accuracy on Titanic! \n",
- "\n",
- "We can do this by simply providing a few example inputs. Or we can simply provide no example inputs at all (zero-shot). Both achieve the same results."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 3,
- "id": "d0698fd2-1361-49ae-8c17-8124e9b932a4",
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "The following structured data is provided in \"Feature Name\":\"Feature Value\" format.\n",
- "Each datapoint describes a passenger on the Titanic.\n",
- "The task is to decide whether the passenger survived.\n",
- "Some example datapoints are given below: \n",
- "-------------------\n",
- "{train_str}\n",
- "-------------------\n",
- "Given this, predict whether the following passenger survived. Return answer as a number between 0 or 1. \n",
- "{eval_str}\n",
- "Survived: \n"
- ]
- }
- ],
- "source": [
- "# first demonstrate the prompt template\n",
- "print(train_prompt.template)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 4,
- "id": "4b39e2e7-be07-42f8-a27a-3419e84cfb2c",
- "metadata": {
- "scrolled": true,
- "tags": []
- },
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "Example datapoints in `train_str`: \n",
- "This is the Data:\n",
- "Age:28.0\n",
- "Embarked:S\n",
- "Fare:7.8958\n",
- "Parch:0\n",
- "Pclass:3\n",
- "Sex:male\n",
- "SibSp:0\n",
- "This is the correct answer:\n",
- "Survived: 0\n",
- "\n",
- "This is the Data:\n",
- "Age:17.0\n",
- "Embarked:S\n",
- "Fare:7.925\n",
- "Parch:2\n",
- "Pclass:3\n",
- "Sex:female\n",
- "SibSp:4\n",
- "This is the correct answer:\n",
- "Survived: 1\n",
- "\n",
- "This is the Data:\n",
- "Age:30.0\n",
- "Embarked:S\n",
- "Fare:16.1\n",
- "Parch:0\n",
- "Pclass:3\n",
- "Sex:male\n",
- "SibSp:1\n",
- "This is the correct answer:\n",
- "Survived: 0\n",
- "\n",
- "This is the Data:\n",
- "Age:22.0\n",
- "Embarked:S\n",
- "Fare:7.25\n",
- "Parch:0\n",
- "Pclass:3\n",
- "Sex:male\n",
- "SibSp:0\n",
- "This is the correct answer:\n",
- "Survived: 0\n",
- "\n",
- "This is the Data:\n",
- "Age:45.0\n",
- "Embarked:S\n",
- "Fare:13.5\n",
- "Parch:0\n",
- "Pclass:2\n",
- "Sex:female\n",
- "SibSp:0\n",
- "This is the correct answer:\n",
- "Survived: 1\n",
- "\n",
- "This is the Data:\n",
- "Age:25.0\n",
- "Embarked:S\n",
- "Fare:0.0\n",
- "Parch:0\n",
- "Pclass:3\n",
- "Sex:male\n",
- "SibSp:0\n",
- "This is the correct answer:\n",
- "Survived: 1\n",
- "\n",
- "This is the Data:\n",
- "Age:18.0\n",
- "Embarked:S\n",
- "Fare:20.2125\n",
- "Parch:1\n",
- "Pclass:3\n",
- "Sex:male\n",
- "SibSp:1\n",
- "This is the correct answer:\n",
- "Survived: 0\n",
- "\n",
- "This is the Data:\n",
- "Age:33.0\n",
- "Embarked:S\n",
- "Fare:9.5\n",
- "Parch:0\n",
- "Pclass:3\n",
- "Sex:male\n",
- "SibSp:0\n",
- "This is the correct answer:\n",
- "Survived: 0\n",
- "\n",
- "This is the Data:\n",
- "Age:24.0\n",
- "Embarked:S\n",
- "Fare:65.0\n",
- "Parch:2\n",
- "Pclass:2\n",
- "Sex:female\n",
- "SibSp:1\n",
- "This is the correct answer:\n",
- "Survived: 1\n",
- "\n",
- "This is the Data:\n",
- "Age:26.0\n",
- "Embarked:S\n",
- "Fare:7.925\n",
- "Parch:0\n",
- "Pclass:3\n",
- "Sex:female\n",
- "SibSp:0\n",
- "This is the correct answer:\n",
- "Survived: 1\n"
- ]
- }
- ],
- "source": [
- "# Get \"training\" prompt string \n",
- "train_n = 10\n",
- "eval_n = 40\n",
- "train_str = get_train_str(train_df, train_labels, train_n=train_n)\n",
- "print(f\"Example datapoints in `train_str`: \\n{train_str}\")"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "819a06f7-3171-4edb-b90c-0a3eae308a04",
- "metadata": {},
- "source": [
- "#### Do evaluation with the training prompt string"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "4a7f2202-518c-41a3-80ab-1e98bbcca903",
- "metadata": {},
- "outputs": [],
- "source": [
- "from sklearn.metrics import accuracy_score\n",
- "import numpy as np\n",
- "\n",
- "eval_preds = get_eval_preds(train_prompt, train_str, eval_df, n=eval_n)\n",
- "eval_label_chunk = eval_labels[:eval_n]"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 7,
- "id": "64323a4d-6eea-4e40-9eac-b2deed60192b",
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "ACCURACY: 0.8\n"
- ]
- }
- ],
- "source": [
- "acc = accuracy_score(eval_label_chunk, np.array(eval_preds).round())\n",
- "print(f'ACCURACY: {acc}')"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "11790d28-8f34-42dd-b11f-6aad21fd5f46",
- "metadata": {},
- "source": [
- "#### Do evaluation with no training prompt string! "
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "aaf993e5-c363-4f18-a28f-09761e49cb6d",
- "metadata": {},
- "outputs": [],
- "source": [
- "from sklearn.metrics import accuracy_score\n",
- "import numpy as np\n",
- "\n",
- "eval_preds_null = get_eval_preds(train_prompt, \"\", eval_df, n=eval_n)\n",
- "eval_label_chunk = eval_labels[:eval_n]"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 11,
- "id": "c3b8bcd5-5972-4ce5-9aa1-57460cdde199",
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "ACCURACY: 0.8\n"
- ]
- }
- ],
- "source": [
- "acc_null = accuracy_score(eval_label_chunk, np.array(eval_preds_null).round())\n",
- "print(f'ACCURACY: {acc_null}')"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "8f0a5e4b-e627-4b47-a807-939813596594",
- "metadata": {
- "tags": []
- },
- "source": [
- "## Extending with GPT List Index"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "42a1ca28-96e9-4cd2-bd48-0673917ad057",
- "metadata": {},
- "source": [
- "#### Build Index"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 12,
- "id": "6c59b030-855d-4e27-89c3-74c972d1bf19",
- "metadata": {},
- "outputs": [],
- "source": [
- "from gpt_index import GPTListIndex\n",
- "from gpt_index.readers.schema.base import Document"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 13,
- "id": "8f9556de-e323-4318-bb71-cff75bf8c3c1",
- "metadata": {},
- "outputs": [],
- "source": [
- "index = GPTListIndex([])"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "e27720fc-af36-40fd-8c55-41485248aa9f",
- "metadata": {},
- "outputs": [],
- "source": [
- "# insertion into index \n",
- "batch_size = 40\n",
- "num_train_chunks = 5\n",
- "\n",
- "for i in range(num_train_chunks):\n",
- " print(f\"Inserting chunk: {i}/{num_train_chunks}\")\n",
- " start_idx = i*batch_size\n",
- " end_idx = (i+1)*batch_size\n",
- " train_batch = train_df.iloc[start_idx:end_idx+batch_size]\n",
- " labels_batch = train_labels.iloc[start_idx:end_idx+batch_size]\n",
- " all_train_str = get_train_str(train_batch, labels_batch, train_n=batch_size)\n",
- " index.insert(Document(all_train_str))"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "e78db088-6649-44db-b52a-766316713b96",
- "metadata": {},
- "source": [
- "#### Query Index"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 15,
- "id": "9cb90564-1de2-412f-8318-d5280855004e",
- "metadata": {},
- "outputs": [],
- "source": [
- "from utils import query_str, qa_data_prompt, refine_prompt"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 16,
- "id": "77c1ae36-e0af-47bc-a656-4971af699755",
- "metadata": {},
- "outputs": [
- {
- "data": {
- "text/plain": [
- "'Which is the relationship between these features and predicting survival?'"
- ]
- },
- "execution_count": 16,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "query_str"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 17,
- "id": "c403710f-d4b3-4287-94f5-e275ea19b476",
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "> Starting query: Which is the relationship between these features and predicting survival?\n"
- ]
- }
- ],
- "source": [
- "response = index.query(\n",
- " query_str, \n",
- " text_qa_template=qa_data_prompt, \n",
- " refine_template=refine_prompt, \n",
- ")"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 19,
- "id": "d2545ab1-980a-4fbd-8add-7ef957801644",
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "\n",
- "\n",
- "There is no definitive answer to this question, as the relationship between the features and predicting survival will vary depending on the data. However, some possible relationships include: age (younger passengers are more likely to survive), sex (females are more likely to survive), fare (passengers who paid more for their ticket are more likely to survive), and pclass (passengers in first or second class are more likely to survive).\n"
- ]
- }
- ],
- "source": [
- "print(response)"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "d0d7d260-2283-49f6-ac40-35c7071cc54d",
- "metadata": {},
- "source": [
- "#### Get Predictions and Evaluate"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 26,
- "id": "e7b98057-957c-48ef-be85-59ff9813d201",
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "The following structured data is provided in \"Feature Name\":\"Feature Value\" format.\n",
- "Each datapoint describes a passenger on the Titanic.\n",
- "The task is to decide whether the passenger survived.\n",
- "We discovered the following relationship between features and survival:\n",
- "-------------------\n",
- "{train_str}\n",
- "-------------------\n",
- "Given this, predict whether the following passenger survived. \n",
- "Return answer as a number between 0 or 1. \n",
- "{eval_str}\n",
- "Survived: \n",
- "\n",
- "\n",
- "`train_str`: \n",
- "\n",
- "There is no definitive answer to this question, as the relationship between the features and predicting survival will vary depending on the data. However, some possible relationships include: age (younger passengers are more likely to survive), sex (females are more likely to survive), fare (passengers who paid more for their ticket are more likely to survive), and pclass (passengers in first or second class are more likely to survive).\n"
- ]
- }
- ],
- "source": [
- "# get eval preds\n",
- "from utils import train_prompt_with_context\n",
- "\n",
- "train_str = response\n",
- "print(train_prompt_with_context.template)\n",
- "print(f'\\n\\n`train_str`: {train_str}')"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "659c6a3f-1c5d-4314-87dc-908e76d50e4a",
- "metadata": {},
- "outputs": [],
- "source": [
- "# do evaluation\n",
- "from sklearn.metrics import accuracy_score\n",
- "import numpy as np\n",
- "eval_n = 40\n",
- "eval_preds = get_eval_preds(train_prompt_with_context, train_str, eval_df, n=eval_n)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 28,
- "id": "7424e7d3-2576-42bc-b626-cf8088265004",
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "ACCURACY: 0.85\n"
- ]
- }
- ],
- "source": [
- "eval_label_chunk = eval_labels[:eval_n]\n",
- "acc = accuracy_score(eval_label_chunk, np.array(eval_preds).round())\n",
- "print(f'ACCURACY: {acc}')"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "e010b497-eeed-4142-a8ac-f5545e85fcc2",
- "metadata": {},
- "outputs": [],
- "source": []
- }
- ],
- "metadata": {
- "kernelspec": {
- "display_name": "gpt_retrieve_venv",
- "language": "python",
- "name": "gpt_retrieve_venv"
- },
- "language_info": {
- "codemirror_mode": {
- "name": "ipython",
- "version": 3
- },
- "file_extension": ".py",
- "mimetype": "text/x-python",
- "name": "python",
- "nbconvert_exporter": "python",
- "pygments_lexer": "ipython3",
- "version": "3.8.4"
- }
- },
- "nbformat": 4,
- "nbformat_minor": 5
-}
+ "cells": [
+ {
+ "cell_type": "code",
+ "execution_count": 1,
+ "id": "f445c1d1-acb9-431e-a7ff-50c41f064359",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "None of PyTorch, TensorFlow >= 2.0, or Flax have been found. Models won't be available and only tokenizers, configuration and file/data utilities can be used.\n",
+ "[nltk_data] Downloading package stopwords to\n",
+ "[nltk_data] /Users/jerryliu/nltk_data...\n",
+ "[nltk_data] Package stopwords is already up-to-date!\n"
+ ]
+ }
+ ],
+ "source": [
+ "from utils import (\n",
+ " get_train_str,\n",
+ " get_train_and_eval_data,\n",
+ " get_eval_preds,\n",
+ " train_prompt\n",
+ ")\n",
+ "\n",
+ "import warnings\n",
+ "warnings.filterwarnings('ignore')\n",
+ "warnings.simplefilter('ignore')"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 2,
+ "id": "cf3cbd90-d5e1-4c30-a3bc-8b39fbd85d70",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# load up the titanic data\n",
+ "train_df, train_labels, eval_df, eval_labels = get_train_and_eval_data('data/train.csv')"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "fa2634f9-cb33-4f1e-81f9-3a3b285e2580",
+ "metadata": {
+ "tags": []
+ },
+ "source": [
+ "## Few-shot Prompting with GPT-3 for Titanic Dataset\n",
+ "In this section, we can show how we can prompt GPT-3 on its own (without using GPT Index) to attain ~80% accuracy on Titanic! \n",
+ "\n",
+ "We can do this by simply providing a few example inputs. Or we can simply provide no example inputs at all (zero-shot). Both achieve the same results."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 3,
+ "id": "d0698fd2-1361-49ae-8c17-8124e9b932a4",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "The following structured data is provided in \"Feature Name\":\"Feature Value\" format.\n",
+ "Each datapoint describes a passenger on the Titanic.\n",
+ "The task is to decide whether the passenger survived.\n",
+ "Some example datapoints are given below: \n",
+ "-------------------\n",
+ "{train_str}\n",
+ "-------------------\n",
+ "Given this, predict whether the following passenger survived. Return answer as a number between 0 or 1. \n",
+ "{eval_str}\n",
+ "Survived: \n"
+ ]
+ }
+ ],
+ "source": [
+ "# first demonstrate the prompt template\n",
+ "print(train_prompt.template)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 4,
+ "id": "4b39e2e7-be07-42f8-a27a-3419e84cfb2c",
+ "metadata": {
+ "scrolled": true,
+ "tags": []
+ },
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Example datapoints in `train_str`: \n",
+ "This is the Data:\n",
+ "Age:28.0\n",
+ "Embarked:S\n",
+ "Fare:7.8958\n",
+ "Parch:0\n",
+ "Pclass:3\n",
+ "Sex:male\n",
+ "SibSp:0\n",
+ "This is the correct answer:\n",
+ "Survived: 0\n",
+ "\n",
+ "This is the Data:\n",
+ "Age:17.0\n",
+ "Embarked:S\n",
+ "Fare:7.925\n",
+ "Parch:2\n",
+ "Pclass:3\n",
+ "Sex:female\n",
+ "SibSp:4\n",
+ "This is the correct answer:\n",
+ "Survived: 1\n",
+ "\n",
+ "This is the Data:\n",
+ "Age:30.0\n",
+ "Embarked:S\n",
+ "Fare:16.1\n",
+ "Parch:0\n",
+ "Pclass:3\n",
+ "Sex:male\n",
+ "SibSp:1\n",
+ "This is the correct answer:\n",
+ "Survived: 0\n",
+ "\n",
+ "This is the Data:\n",
+ "Age:22.0\n",
+ "Embarked:S\n",
+ "Fare:7.25\n",
+ "Parch:0\n",
+ "Pclass:3\n",
+ "Sex:male\n",
+ "SibSp:0\n",
+ "This is the correct answer:\n",
+ "Survived: 0\n",
+ "\n",
+ "This is the Data:\n",
+ "Age:45.0\n",
+ "Embarked:S\n",
+ "Fare:13.5\n",
+ "Parch:0\n",
+ "Pclass:2\n",
+ "Sex:female\n",
+ "SibSp:0\n",
+ "This is the correct answer:\n",
+ "Survived: 1\n",
+ "\n",
+ "This is the Data:\n",
+ "Age:25.0\n",
+ "Embarked:S\n",
+ "Fare:0.0\n",
+ "Parch:0\n",
+ "Pclass:3\n",
+ "Sex:male\n",
+ "SibSp:0\n",
+ "This is the correct answer:\n",
+ "Survived: 1\n",
+ "\n",
+ "This is the Data:\n",
+ "Age:18.0\n",
+ "Embarked:S\n",
+ "Fare:20.2125\n",
+ "Parch:1\n",
+ "Pclass:3\n",
+ "Sex:male\n",
+ "SibSp:1\n",
+ "This is the correct answer:\n",
+ "Survived: 0\n",
+ "\n",
+ "This is the Data:\n",
+ "Age:33.0\n",
+ "Embarked:S\n",
+ "Fare:9.5\n",
+ "Parch:0\n",
+ "Pclass:3\n",
+ "Sex:male\n",
+ "SibSp:0\n",
+ "This is the correct answer:\n",
+ "Survived: 0\n",
+ "\n",
+ "This is the Data:\n",
+ "Age:24.0\n",
+ "Embarked:S\n",
+ "Fare:65.0\n",
+ "Parch:2\n",
+ "Pclass:2\n",
+ "Sex:female\n",
+ "SibSp:1\n",
+ "This is the correct answer:\n",
+ "Survived: 1\n",
+ "\n",
+ "This is the Data:\n",
+ "Age:26.0\n",
+ "Embarked:S\n",
+ "Fare:7.925\n",
+ "Parch:0\n",
+ "Pclass:3\n",
+ "Sex:female\n",
+ "SibSp:0\n",
+ "This is the correct answer:\n",
+ "Survived: 1\n"
+ ]
+ }
+ ],
+ "source": [
+ "# Get \"training\" prompt string \n",
+ "train_n = 10\n",
+ "eval_n = 40\n",
+ "train_str = get_train_str(train_df, train_labels, train_n=train_n)\n",
+ "print(f\"Example datapoints in `train_str`: \\n{train_str}\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "819a06f7-3171-4edb-b90c-0a3eae308a04",
+ "metadata": {},
+ "source": [
+ "#### Do evaluation with the training prompt string"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "4a7f2202-518c-41a3-80ab-1e98bbcca903",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from sklearn.metrics import accuracy_score\n",
+ "import numpy as np\n",
+ "\n",
+ "eval_preds = get_eval_preds(train_prompt, train_str, eval_df, n=eval_n)\n",
+ "eval_label_chunk = eval_labels[:eval_n]"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 7,
+ "id": "64323a4d-6eea-4e40-9eac-b2deed60192b",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "ACCURACY: 0.8\n"
+ ]
+ }
+ ],
+ "source": [
+ "acc = accuracy_score(eval_label_chunk, np.array(eval_preds).round())\n",
+ "print(f'ACCURACY: {acc}')"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "11790d28-8f34-42dd-b11f-6aad21fd5f46",
+ "metadata": {},
+ "source": [
+ "#### Do evaluation with no training prompt string! "
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "aaf993e5-c363-4f18-a28f-09761e49cb6d",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from sklearn.metrics import accuracy_score\n",
+ "import numpy as np\n",
+ "\n",
+ "eval_preds_null = get_eval_preds(train_prompt, \"\", eval_df, n=eval_n)\n",
+ "eval_label_chunk = eval_labels[:eval_n]"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 11,
+ "id": "c3b8bcd5-5972-4ce5-9aa1-57460cdde199",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "ACCURACY: 0.8\n"
+ ]
+ }
+ ],
+ "source": [
+ "acc_null = accuracy_score(eval_label_chunk, np.array(eval_preds_null).round())\n",
+ "print(f'ACCURACY: {acc_null}')"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "8f0a5e4b-e627-4b47-a807-939813596594",
+ "metadata": {
+ "tags": []
+ },
+ "source": [
+ "## Extending with GPT List Index"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "42a1ca28-96e9-4cd2-bd48-0673917ad057",
+ "metadata": {},
+ "source": [
+ "#### Build Index"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 12,
+ "id": "6c59b030-855d-4e27-89c3-74c972d1bf19",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from llama_index import GPTListIndex\n",
+ "from llama_index.readers.schema.base import Document"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 13,
+ "id": "8f9556de-e323-4318-bb71-cff75bf8c3c1",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "index = GPTListIndex([])"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "e27720fc-af36-40fd-8c55-41485248aa9f",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# insertion into index \n",
+ "batch_size = 40\n",
+ "num_train_chunks = 5\n",
+ "\n",
+ "for i in range(num_train_chunks):\n",
+ " print(f\"Inserting chunk: {i}/{num_train_chunks}\")\n",
+ " start_idx = i*batch_size\n",
+ " end_idx = (i+1)*batch_size\n",
+ " train_batch = train_df.iloc[start_idx:end_idx+batch_size]\n",
+ " labels_batch = train_labels.iloc[start_idx:end_idx+batch_size]\n",
+ " all_train_str = get_train_str(train_batch, labels_batch, train_n=batch_size)\n",
+ " index.insert(Document(all_train_str))"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "e78db088-6649-44db-b52a-766316713b96",
+ "metadata": {},
+ "source": [
+ "#### Query Index"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 15,
+ "id": "9cb90564-1de2-412f-8318-d5280855004e",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from utils import query_str, qa_data_prompt, refine_prompt"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 16,
+ "id": "77c1ae36-e0af-47bc-a656-4971af699755",
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "'Which is the relationship between these features and predicting survival?'"
+ ]
+ },
+ "execution_count": 16,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "query_str"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 17,
+ "id": "c403710f-d4b3-4287-94f5-e275ea19b476",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "> Starting query: Which is the relationship between these features and predicting survival?\n"
+ ]
+ }
+ ],
+ "source": [
+ "response = index.query(\n",
+ " query_str, \n",
+ " text_qa_template=qa_data_prompt, \n",
+ " refine_template=refine_prompt, \n",
+ ")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 19,
+ "id": "d2545ab1-980a-4fbd-8add-7ef957801644",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "\n",
+ "\n",
+ "There is no definitive answer to this question, as the relationship between the features and predicting survival will vary depending on the data. However, some possible relationships include: age (younger passengers are more likely to survive), sex (females are more likely to survive), fare (passengers who paid more for their ticket are more likely to survive), and pclass (passengers in first or second class are more likely to survive).\n"
+ ]
+ }
+ ],
+ "source": [
+ "print(response)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "d0d7d260-2283-49f6-ac40-35c7071cc54d",
+ "metadata": {},
+ "source": [
+ "#### Get Predictions and Evaluate"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 26,
+ "id": "e7b98057-957c-48ef-be85-59ff9813d201",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "The following structured data is provided in \"Feature Name\":\"Feature Value\" format.\n",
+ "Each datapoint describes a passenger on the Titanic.\n",
+ "The task is to decide whether the passenger survived.\n",
+ "We discovered the following relationship between features and survival:\n",
+ "-------------------\n",
+ "{train_str}\n",
+ "-------------------\n",
+ "Given this, predict whether the following passenger survived. \n",
+ "Return answer as a number between 0 or 1. \n",
+ "{eval_str}\n",
+ "Survived: \n",
+ "\n",
+ "\n",
+ "`train_str`: \n",
+ "\n",
+ "There is no definitive answer to this question, as the relationship between the features and predicting survival will vary depending on the data. However, some possible relationships include: age (younger passengers are more likely to survive), sex (females are more likely to survive), fare (passengers who paid more for their ticket are more likely to survive), and pclass (passengers in first or second class are more likely to survive).\n"
+ ]
+ }
+ ],
+ "source": [
+ "# get eval preds\n",
+ "from utils import train_prompt_with_context\n",
+ "\n",
+ "train_str = response\n",
+ "print(train_prompt_with_context.template)\n",
+ "print(f'\\n\\n`train_str`: {train_str}')"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "659c6a3f-1c5d-4314-87dc-908e76d50e4a",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# do evaluation\n",
+ "from sklearn.metrics import accuracy_score\n",
+ "import numpy as np\n",
+ "eval_n = 40\n",
+ "eval_preds = get_eval_preds(train_prompt_with_context, train_str, eval_df, n=eval_n)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 28,
+ "id": "7424e7d3-2576-42bc-b626-cf8088265004",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "ACCURACY: 0.85\n"
+ ]
+ }
+ ],
+ "source": [
+ "eval_label_chunk = eval_labels[:eval_n]\n",
+ "acc = accuracy_score(eval_label_chunk, np.array(eval_preds).round())\n",
+ "print(f'ACCURACY: {acc}')"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "e010b497-eeed-4142-a8ac-f5545e85fcc2",
+ "metadata": {},
+ "outputs": [],
+ "source": []
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "gpt_retrieve_venv",
+ "language": "python",
+ "name": "gpt_retrieve_venv"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.8.4"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}
\ No newline at end of file