diff --git a/docs/examples/retrievers/bm25_retriever.ipynb b/docs/examples/retrievers/bm25_retriever.ipynb
index 71904b024cae295af58f1c4c9863a2924e3dcec4..550e777f91333a28b164d71ebb1454f4ccf4cfc9 100644
--- a/docs/examples/retrievers/bm25_retriever.ipynb
+++ b/docs/examples/retrievers/bm25_retriever.ipynb
@@ -42,7 +42,8 @@
"metadata": {},
"outputs": [],
"source": [
- "%pip install llama-index-llms-openai"
+ "%pip install llama-index-llms-openai\n",
+ "%pip install llama-index-retrievers-bm25"
]
},
{
@@ -104,7 +105,7 @@
" StorageContext,\n",
" VectorStoreIndex,\n",
")\n",
- "from llama_index.core.retrievers import BM25Retriever\n",
+ "from llama_index.retrievers.bm25 import BM25Retriever\n",
"from llama_index.core.retrievers import VectorIndexRetriever\n",
"from llama_index.core.node_parser import SentenceSplitter\n",
"from llama_index.llms.openai import OpenAI"
@@ -181,7 +182,15 @@
"execution_count": null,
"id": "0d6162df-9da7-4aad-a2ca-eb318f67daec",
"metadata": {},
- "outputs": [],
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "HTTP Request: POST https://api.openai.com/v1/embeddings \"HTTP/1.1 200 OK\"\n"
+ ]
+ }
+ ],
"source": [
"index = VectorStoreIndex(\n",
" nodes=nodes,\n",
@@ -199,16 +208,6 @@
"We will search document with bm25 retriever."
]
},
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "5e94711d",
- "metadata": {},
- "outputs": [],
- "source": [
- "# !pip install rank_bm25"
- ]
- },
{
"cell_type": "code",
"execution_count": null,
@@ -229,7 +228,7 @@
{
"data": {
"text/markdown": [
- "**Node ID:** d95537b4-b398-4b47-94ff-da86f05a27f7<br>**Similarity:** 5.171801938898801<br>**Text:** I wanted to go back to RISD, but I was now broke and RISD was very expensive, so I decided to get...<br>"
+ "**Node ID:** 9afb8cb9-42f3-4160-807c-1cd6685fa774<br>**Similarity:** 1.6781810094192822<br>**Text:** Now that I could write essays again, I wrote a bunch about topics I'd had stacked up. I kept writ...<br>"
],
"text/plain": [
"<IPython.core.display.Markdown object>"
@@ -241,9 +240,7 @@
{
"data": {
"text/markdown": [
- "**Node ID:** 6f84e2a5-1ab1-4389-8799-b7713e085931<br>**Similarity:** 4.838241203957084<br>**Text:** All you had to do was teach SHRDLU more words.\n",
- "\n",
- "There weren't any classes in AI at Cornell then, ...<br>"
+ "**Node ID:** 71de4371-10ff-4d3a-8a31-14b2d4ddec83<br>**Similarity:** 1.546534805781164<br>**Text:** I couldn't have put this into words when I was 18. All I knew at the time was that I kept taking ...<br>"
],
"text/plain": [
"<IPython.core.display.Markdown object>"
@@ -271,7 +268,7 @@
{
"data": {
"text/markdown": [
- "**Node ID:** a4fd0b29-4138-4741-9e27-9f65d6968eb4<br>**Similarity:** 8.090884087344435<br>**Text:** Not so much because it was badly written as because the problem is so convoluted. When you're wor...<br>"
+ "**Node ID:** 42c88008-dd05-4590-8fd1-df85567579ea<br>**Similarity:** 5.389397745654172<br>**Text:** It was missing a lot of things you'd want in a programming language. So these had to be added, an...<br>"
],
"text/plain": [
"<IPython.core.display.Markdown object>"
@@ -283,7 +280,7 @@
{
"data": {
"text/markdown": [
- "**Node ID:** d95537b4-b398-4b47-94ff-da86f05a27f7<br>**Similarity:** 5.830874349482576<br>**Text:** I wanted to go back to RISD, but I was now broke and RISD was very expensive, so I decided to get...<br>"
+ "**Node ID:** 4d98c2ad-60cc-4cc0-abe2-b95c0c4be87b<br>**Similarity:** 1.141523170922594<br>**Text:** Painting students were supposed to express themselves, which to the more worldly ones meant to tr...<br>"
],
"text/plain": [
"<IPython.core.display.Markdown object>"
@@ -359,15 +356,15 @@
"name": "stdout",
"output_type": "stream",
"text": [
- "Selecting retriever 0: The author's life context is a broad topic, which may require a comprehensive approach that is useful in most cases..\n"
+ "HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n",
+ "Selecting retriever 0: The author's life context is a broad topic and can be useful in most cases..\n",
+ "HTTP Request: POST https://api.openai.com/v1/embeddings \"HTTP/1.1 200 OK\"\n"
]
},
{
"data": {
"text/markdown": [
- "**Node ID:** fcd399c1-3544-4df3-80a9-0a7d3fd41f1f<br>**Similarity:** 0.7942753162501964<br>**Text:** [10]\n",
- "\n",
- "Wow, I thought, there's an audience. If I write something and put it on the web, anyone can...<br>"
+ "**Node ID:** 9afb8cb9-42f3-4160-807c-1cd6685fa774<br>**Similarity:** 0.7961776744788842<br>**Text:** Now that I could write essays again, I wrote a bunch about topics I'd had stacked up. I kept writ...<br>"
],
"text/plain": [
"<IPython.core.display.Markdown object>"
@@ -379,9 +376,7 @@
{
"data": {
"text/markdown": [
- "**Node ID:** b203e140-d549-4284-99f4-b1b5bcd996ea<br>**Similarity:** 0.7788031317604815<br>**Text:** Now all I had to do was learn Italian.\n",
- "\n",
- "Only stranieri (foreigners) had to take this entrance exa...<br>"
+ "**Node ID:** b1ffb78d-dc4c-439b-906a-cca73b713940<br>**Similarity:** 0.7924813308773564<br>**Text:** We actually had one of those little stoves, fed with kindling, that you see in 19th century studi...<br>"
],
"text/plain": [
"<IPython.core.display.Markdown object>"
@@ -427,17 +422,7 @@
"execution_count": null,
"id": "5ebd3ddf",
"metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- " % Total % Received % Xferd Average Speed Time Time Time Current\n",
- " Dload Upload Total Spent Left Speed\n",
- "100 20.7M 100 20.7M 0 0 361k 0 0:00:58 0:00:58 --:--:-- 422k\n"
- ]
- }
- ],
+ "outputs": [],
"source": [
"!curl https://www.ipcc.ch/report/ar6/wg2/downloads/report/IPCC_AR6_WGII_Chapter03.pdf --output IPCC_AR6_WGII_Chapter03.pdf"
]
@@ -455,7 +440,7 @@
{
"cell_type": "code",
"execution_count": null,
- "id": "e46f93b3",
+ "id": "48bd32ac-6d99-4bb5-8559-45b61d528525",
"metadata": {},
"outputs": [],
"source": [
@@ -463,6 +448,7 @@
" VectorStoreIndex,\n",
" StorageContext,\n",
" SimpleDirectoryReader,\n",
+ " Document,\n",
")\n",
"from llama_index.core.node_parser import SentenceSplitter\n",
"from llama_index.llms.openai import OpenAI\n",
@@ -470,14 +456,33 @@
"# load documents\n",
"documents = SimpleDirectoryReader(\n",
" input_files=[\"IPCC_AR6_WGII_Chapter03.pdf\"]\n",
- ").load_data()\n",
- "\n",
+ ").load_data()"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "7c771a80-92e3-4178-b4a3-1f03c489b1a2",
+ "metadata": {},
+ "outputs": [],
+ "source": [
"# initialize llm + node parser\n",
"# -- here, we set a smaller chunk size, to allow for more effective re-ranking\n",
"llm = OpenAI(model=\"gpt-3.5-turbo\")\n",
"splitter = SentenceSplitter(chunk_size=256)\n",
- "nodes = splitter.get_nodes_from_documents(documents)\n",
- "\n",
+ "# limit to a smaller section\n",
+ "nodes = splitter.get_nodes_from_documents(\n",
+ " [Document(text=documents[0].get_content()[:1000000])]\n",
+ ")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "52f4243d-fd8f-4193-bf6a-73c07d661bcd",
+ "metadata": {},
+ "outputs": [],
+ "source": [
"# initialize storage context (by default it's in-memory)\n",
"storage_context = StorageContext.from_defaults()\n",
"storage_context.docstore.add_documents(nodes)"
@@ -500,7 +505,7 @@
"metadata": {},
"outputs": [],
"source": [
- "from llama_index.core.retrievers import BM25Retriever\n",
+ "from llama_index.retrievers.bm25 import BM25Retriever\n",
"\n",
"# retireve the top 10 most similar nodes using embeddings\n",
"vector_retriever = index.as_retriever(similarity_top_k=10)\n",
@@ -574,7 +579,7 @@
"metadata": {},
"outputs": [],
"source": [
- "# !pip install sentence_transformers"
+ "!pip install sentence-transformers"
]
},
{
@@ -582,26 +587,7 @@
"execution_count": null,
"id": "638e7786",
"metadata": {},
- "outputs": [
- {
- "name": "stderr",
- "output_type": "stream",
- "text": [
- "Downloading (…)lve/main/config.json: 100%|██████████| 799/799 [00:00<00:00, 3.86MB/s]\n",
- "Downloading pytorch_model.bin: 100%|██████████| 1.11G/1.11G [00:32<00:00, 34.4MB/s]\n",
- "Downloading (…)okenizer_config.json: 100%|██████████| 443/443 [00:00<00:00, 2.19MB/s]\n",
- "Downloading (…)tencepiece.bpe.model: 100%|██████████| 5.07M/5.07M [00:00<00:00, 14.1MB/s]\n",
- "Downloading (…)cial_tokens_map.json: 100%|██████████| 279/279 [00:00<00:00, 1.48MB/s]\n"
- ]
- },
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "Use pytorch device: cpu\n"
- ]
- }
- ],
+ "outputs": [],
"source": [
"from llama_index.core.postprocessor import SentenceTransformerRerank\n",
"\n",
@@ -621,34 +607,11 @@
"execution_count": null,
"id": "9235f79d",
"metadata": {},
- "outputs": [
- {
- "name": "stderr",
- "output_type": "stream",
- "text": [
- "Batches: 100%|██████████| 1/1 [00:05<00:00, 5.61s/it]"
- ]
- },
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "Initial retrieval: 19 nodes\n",
- "Re-ranked retrieval: 4 nodes\n"
- ]
- },
- {
- "name": "stderr",
- "output_type": "stream",
- "text": [
- "\n"
- ]
- }
- ],
+ "outputs": [],
"source": [
"from llama_index.core import QueryBundle\n",
"\n",
- "nodes = hybrid_retriever.retrieve(\n",
+ "retrieved_nodes = hybrid_retriever.retrieve(\n",
" \"What is the impact of climate change on the ocean?\"\n",
")\n",
"reranked_nodes = reranker.postprocess_nodes(\n",
@@ -658,7 +621,7 @@
" ),\n",
")\n",
"\n",
- "print(\"Initial retrieval: \", len(nodes), \" nodes\")\n",
+ "print(\"Initial retrieval: \", len(retrieved_nodes), \" nodes\")\n",
"print(\"Re-ranked retrieval: \", len(reranked_nodes), \" nodes\")"
]
},
@@ -671,10 +634,10 @@
{
"data": {
"text/markdown": [
- "**Node ID:** 74b12b7b-f4b9-490a-9342-b640211468dd<br>**Similarity:** 0.998129665851593<br>**Text:** 3\n",
- "469Oceans and Coastal Ecosystems and Their Services Chapter 3\n",
- "Frequently Asked Questions\n",
- "FAQ 3...<br>"
+ "**Node ID:** 735c563d-e68c-42e7-bbeb-d360a2918d22<br>**Similarity:** 0.6910567283630371<br>**Text:** lb88\\8a|hiac2:,sg\u001c",
+ "\u001a`HfvyȔs\u001fkz4\u000b",
+ "4+)t$EAs9\b<\"L\u0013䴺-iai]\u000f}?R)J\u0003\u0002\u0018v8Q0V#9>f\u001b=3`\bב&WTFK\u001bNӅ9GN8v`4g\f",
+ "tz...<br>"
],
"text/plain": [
"<IPython.core.display.Markdown object>"
@@ -686,9 +649,11 @@
{
"data": {
"text/markdown": [
- "**Node ID:** 2b35824c-2e96-47b7-8dfb-da25c4eefb7d<br>**Similarity:** 0.996731162071228<br>**Text:** {Box 3.2, 3.2.2.1, 3.4.2.5, 3.4.2.10, 3.4.3.3, Cross-Chapter \n",
- "Box PALEO in Chapter 1}\n",
- "Climate imp...<br>"
+ "**Node ID:** 07a796db-8307-47a0-83a0-9b5f953b86b0<br>**Similarity:** 0.609274685382843<br>**Text:** {_d3<,f\u0001CȀ\u001e",
+ "0K0~n\u001e",
+ "(\u0018A\t\b̙$saxt\f",
+ " :\u001e",
+ "xY3\u000fz/ߕXAW\u000fwpTeHY0\u0015\u0012H\u001bZHe\b̚kÇ>.\u001882%ϖ\u0017^.rCS.2^\u001a12C?<br>"
],
"text/plain": [
"<IPython.core.display.Markdown object>"
@@ -700,7 +665,9 @@
{
"data": {
"text/markdown": [
- "**Node ID:** 01ef2a9e-0dd0-4bce-ab60-e6a3f6456f7b<br>**Similarity:** 0.9954373240470886<br>**Text:** These ecosystems are also influenced by non-climate drivers, especially fisheries, oil and gas ex...<br>"
+ "**Node ID:** 0ee98fbd-d015-4cd4-aba0-812423880915<br>**Similarity:** 0.5765283107757568<br>**Text:** X+V/Z\u001d",
+ "\u000b",
+ "6\u000e;\u0005O8/%Z3EC\\a\u0010sU1f(\u0016xLͼ]X\\\u0014\u0018q\"1}.66OKGSi\u0001\u000f9ǧ\u0015'(?Iʲi=4ޮ^m,Mp1~\u0013lBxP\u00153]\u0011S<?C)3WM;\u0016WGQt@\bl...<br>"
],
"text/plain": [
"<IPython.core.display.Markdown object>"
@@ -712,8 +679,9 @@
{
"data": {
"text/markdown": [
- "**Node ID:** 8a23b728-0352-4b01-a5c0-42765669855d<br>**Similarity:** 0.9872682690620422<br>**Text:** Additionally, climate-change-driven oxygen loss (Section 3.2.3.2; Luna et al., 2012; \n",
- "Belley et...<br>"
+ "**Node ID:** 2562bbfb-fda2-432d-9674-5ba77ff4b767<br>**Similarity:** 0.3882860541343689<br>**Text:** P\bxS0s\u0010.MAK|+MOUÊ\u0016^;tc0\n",
+ "\u001a$\u0002XH\u0006mL\u0004<ҁ;Q@nUd&\u0013ի\n",
+ "2`\u0018B0\u00041i^yOw:rsM冫S*wu4a{\u0001-.Խ=j鷩nQ\u0006D_AA%~T>~\u0018'OxU\tgk...<br>"
],
"text/plain": [
"<IPython.core.display.Markdown object>"
@@ -743,15 +711,7 @@
"execution_count": null,
"id": "549b9a96",
"metadata": {},
- "outputs": [
- {
- "name": "stderr",
- "output_type": "stream",
- "text": [
- "Batches: 100%|██████████| 1/1 [00:05<00:00, 5.74s/it]\n"
- ]
- }
- ],
+ "outputs": [],
"source": [
"from llama_index.core.query_engine import RetrieverQueryEngine\n",
"\n",
@@ -775,7 +735,7 @@
{
"data": {
"text/markdown": [
- "**`Final Response:`** Climate change has significant impacts on the ocean. It is degrading ocean health and altering stocks of marine resources. This, combined with over-harvesting, is threatening the sustenance provided to Indigenous Peoples, the livelihoods of artisanal fisheries, and marine-based industries such as tourism, shipping, and transportation. Climate change can also influence human activities and employment by altering resource availability, spreading pathogens, flooding shorelines, and degrading ocean ecosystems. Additionally, increases in intensity, reoccurrence, and duration of marine heatwaves due to climate change can lead to species extirpation, habitat collapse, and surpassing ecological tipping points. Some habitat-forming coastal ecosystems, including coral reefs, kelp forests, and seagrass meadows, are at high risk of irreversible phase shifts due to marine heatwaves. Non-climate drivers such as fisheries, oil and gas extraction, cable laying, and mineral resource exploration also influence ocean ecosystems."
+ "**`Final Response:`** Climate change can have a significant impact on the ocean. Rising temperatures can lead to the melting of polar ice caps, resulting in sea-level rise and coastal flooding. It can also disrupt ocean currents and affect marine ecosystems, including coral reefs and fish populations. Additionally, climate change can lead to ocean acidification, which can harm marine life and impact the overall health of the ocean."
],
"text/plain": [
"<IPython.core.display.Markdown object>"
@@ -794,9 +754,9 @@
],
"metadata": {
"kernelspec": {
- "display_name": "Python 3 (ipykernel)",
+ "display_name": "llama_index_v3",
"language": "python",
- "name": "python3"
+ "name": "llama_index_v3"
},
"language_info": {
"codemirror_mode": {