Skip to content
Snippets Groups Projects
Unverified Commit 53e9f523 authored by Shorthills AI's avatar Shorthills AI Committed by GitHub
Browse files

Fixed some minor gramatical issues (#11530)

* Fixed some gramatical mistakes (#82)

* Update discover_llamaindex.md

* Update installation.md

* Update reading.md

* Update starter_example.md

* Update starter_example_local.md

* Update v0_10_0_migration.md

* Update 2024-02-28-rag-bootcamp-vector-institute.ipynb

* Update multimodal.md

* Update chatbots.md
parent 65e96de2
No related branches found
No related tags found
No related merge requests found
%% Cell type:markdown id:5ab60f84-39b3-4bdd-ae83-6527acb315e5 tags:
# RAG Bootcamp ◦ February 2024 ◦ Vector Institute
%% Cell type:code id:c6b62610 tags:
``` python
##################################################################
# Venue: RAG Bootcamp - Vector Institute Canada
# Talk: RAG Bootcamp: Intro to RAG with the LlamaIndexFramework
# Speaker: Andrei Fajardo
##################################################################
```
%% Cell type:markdown id:3bee89b8-a04d-4326-9392-b9e7e1bcb8af tags:
![Title Image](https://d3ddy8balm3goa.cloudfront.net/rag-bootcamp-vector/title.excalidraw.svg)
%% Cell type:markdown id:e4d38b38-ea48-4012-81ae-84e1d1f40a69 tags:
![Title Image](https://d3ddy8balm3goa.cloudfront.net/rag-bootcamp-vector/framework.excalidraw.svg)
%% Cell type:markdown id:34d1f8e7-f978-4f19-bdfb-37c0d235b5bf tags:
#### Notebook Setup & Dependency Installation
%% Cell type:code id:f227a52a-a147-4e8f-b7d3-e03f983fd5f1 tags:
``` python
%pip install llama-index llama-index-vector-stores-qdrant -q
```
%% Cell type:code id:7bc383fc-19b2-47b5-af61-e83210ea9c37 tags:
``` python
import nest_asyncio
nest_asyncio.apply()
```
%% Cell type:code id:4e4c84ad tags:
``` python
!mkdir data
!wget "https://arxiv.org/pdf/2402.09353.pdf" -O "./data/dorav1.pdf"
```
%% Cell type:markdown id:275c00f1-e358-498a-88c3-8e810a5a2546 tags:
## Motivation
![Motivation Image](https://d3ddy8balm3goa.cloudfront.net/rag-bootcamp-vector/motivation.excalidraw.svg)
%% Cell type:code id:25d4ce76-8eea-44cb-aa99-94844dfed9c7 tags:
``` python
# query an LLM and ask it about DoRA
from llama_index.llms.openai import OpenAI
llm = OpenAI(model="gpt-4")
response = llm.complete("What is DoRA?")
```
%% Cell type:code id:c3f18489-4f25-40ce-86e9-697ddea7d6c6 tags:
``` python
print(response.text)
```
%% Output
Without specific context, it's hard to determine what DoRA refers to as it could mean different things in different fields. However, in general, DoRA could refer to:
1. Division of Research and Analysis: In some organizations, this is a department responsible for conducting research and analyzing data.
2. Department of Regulatory Agencies: In some U.S. states, this is a government agency responsible for consumer protection and regulation of businesses.
3. Declaration of Research Assessment: In academia, this could refer to a statement or policy regarding how research is evaluated.
4. Digital on-Ramp's Assessment: In the field of digital technology, this could refer to an assessment tool used by the Digital On-Ramps program.
4. Digital On-Ramp's Assessment: In the field of digital technology, this could refer to an assessment tool used by the Digital On-Ramps program.
Please provide more context for a more accurate definition.
%% Cell type:markdown id:04a0ef8d-d55c-4b64-887b-18d343503a76 tags:
## Basic RAG in 3 Steps
![Divider Image](https://d3ddy8balm3goa.cloudfront.net/rag-bootcamp-vector/subheading.excalidraw.svg)
1. Build external knowledge (i.e., updated data sources)
2. Retrieve
3. Augment and Generate
%% Cell type:markdown id:598a5257-20ae-468e-85d6-d4e8c46b8cb5 tags:
## 1. Build External Knowledge
![Divider Image](https://d3ddy8balm3goa.cloudfront.net/rag-bootcamp-vector/step1.excalidraw.svg)
%% Cell type:code id:a2963f90-9da5-4a0d-8dbe-f16fcb8627a3 tags:
``` python
"""Load the data.
With llama-index, before any transformations are applied,
data is loaded in the `Document` abstraction, which is
a container that holds the text of the document.
"""
from llama_index.core import SimpleDirectoryReader
loader = SimpleDirectoryReader(input_dir="./data")
documents = loader.load_data()
```
%% Cell type:code id:da321e2c-8428-4c04-abf2-b204416e816f tags:
``` python
# if you want to see what the text looks like
# documents[0].text
```
%% Cell type:code id:4801e74a-8c52-45c4-967d-7a1a94f54ad3 tags:
``` python
"""Chunk, Encode, and Store into a Vector Store.
To streamline the process, we can make use of the IngestionPipeline
class that will apply your specified transformations to the
Document's.
"""
from llama_index.core.ingestion import IngestionPipeline
from llama_index.core.node_parser import SentenceSplitter
from llama_index.embeddings.openai import OpenAIEmbedding
from llama_index.vector_stores.qdrant import QdrantVectorStore
import qdrant_client
client = qdrant_client.QdrantClient(location=":memory:")
vector_store = QdrantVectorStore(client=client, collection_name="test_store")
pipeline = IngestionPipeline(
transformations=[
SentenceSplitter(),
OpenAIEmbedding(),
],
vector_store=vector_store,
)
_nodes = pipeline.run(documents=documents, num_workers=4)
```
%% Cell type:code id:02afea25-098b-49c7-a965-21c7576757af tags:
``` python
# if you want to see the nodes
# len(_nodes)
# _nodes[0].text
```
%% Cell type:code id:44cd8a86-089d-4329-9484-35b98b3a26f9 tags:
``` python
"""Create a llama-index... wait for it... Index.
After uploading your encoded documents into your vector
store of choice, you can connect to it with a VectorStoreIndex
which then gives you access to all of the llama-index functionality.
"""
from llama_index.core import VectorStoreIndex
index = VectorStoreIndex.from_vector_store(vector_store=vector_store)
```
%% Cell type:markdown id:286b1827-7547-49c6-aba3-82f08d6d86b8 tags:
## 2. Retrieve Against A Query
![Step2 Image](https://d3ddy8balm3goa.cloudfront.net/rag-bootcamp-vector/step2.excalidraw.svg)
%% Cell type:code id:49f86af1-db08-4641-89ad-d60abd04e6b3 tags:
``` python
"""Retrieve relevant documents against a query.
With our Index ready, we can now query it to
retrieve the most relevant document chunks.
"""
retriever = index.as_retriever(similarity_top_k=2)
retrieved_nodes = retriever.retrieve("What is DoRA?")
```
%% Cell type:code id:05f9ce3b-a4e3-4862-b58c-2d9fba1f9abc tags:
``` python
# to view the retrieved node
# print(retrieved_nodes[0].text)
```
%% Cell type:markdown id:978ae2c5-8c2a-41c7-a2eb-85a5562f2db5 tags:
## 3. Generate Final Response
![Step3 Image](https://d3ddy8balm3goa.cloudfront.net/rag-bootcamp-vector/step3.excalidraw.svg)
%% Cell type:code id:ef33c349-eed4-4e35-9b5d-9473adf2ce01 tags:
``` python
"""Context-Augemented Generation.
With our Index ready, we can create a QueryEngine
that handles the retrieval and context augmentation
in order to get the final response.
"""
query_engine = index.as_query_engine()
```
%% Cell type:code id:4139c48a-ece8-4244-b4eb-7cff74cb1325 tags:
``` python
# to inspect the default prompt being used
print(
query_engine.get_prompts()[
"response_synthesizer:text_qa_template"
].default_template.template
)
```
%% Output
Context information is below.
---------------------
{context_str}
---------------------
Given the context information and not prior knowledge, answer the query.
Query: {query_str}
Answer:
%% Cell type:code id:6179639d-af96-4a09-b440-b47ad599a26f tags:
``` python
response = query_engine.query("What is DoRA?")
print(response)
```
%% Output
DoRA is a method that introduces incremental directional updates in a model by replacing them with alternative LoRA variants. It is compatible with other LoRA variants such as VeRA, which suggests freezing a unique pair of random low-rank matrices shared across all layers and employing minimal layer-specific trainable scaling vectors to capture each layer's incremental updates. DoRA effectively reduces the number of trainable parameters significantly while maintaining accuracy, showcasing improvements over other variants like VeRA and LoRA.
%% Cell type:markdown id:dae63946-be38-4807-af2a-8113661a806b tags:
## In Summary
- LLMs as powerful as they are, don't perform too well with knowledge-intensive tasks (domain specific, updated data, long-tail)
- LLMs as powerful as they are, don't perform too well with knowledge-intensive tasks (domain-specific, updated data, long-tail)
- Context augmentation has been shown (in a few studies) to outperform LLMs without augmentation
- In this notebook, we showed one such example that follows that pattern.
%% Cell type:markdown id:fc857227-3fed-4bb6-a062-99ea3c55e294 tags:
# LlamaIndex Has More To Offer
- Data infrastructure that enables production-grade, advanced RAG systems
- Agentic solutions
- Newly released: `llama-index-networks`
- Enterprise offerings (alpha):
- LlamaParse (proprietary complex PDF parser) and
- LlamaCloud
%% Cell type:markdown id:17c1c027-be8b-48f4-87ee-06f3e2c71797 tags:
### Useful links
[website](https://www.llamaindex.ai/)[llamahub](https://llamahub.ai)[github](https://github.com/run-llama/llama_index)[medium](https://medium.com/@llama_index)[rag-bootcamp-poster](https://d3ddy8balm3goa.cloudfront.net/rag-bootcamp-vector/final_poster.excalidraw.svg)
......
......@@ -10,7 +10,7 @@ Here are some relevant resources:
- [create-llama](https://blog.llamaindex.ai/create-llama-a-command-line-tool-to-generate-llamaindex-apps-8f7683021191), a command line tool that generates a full-stack chatbot application for you
- [SECinsights.ai](https://www.secinsights.ai/), an open-source application that uses LlamaIndex to build a chatbot that answers questions about SEC filings
- [RAGs](https://blog.llamaindex.ai/introducing-rags-your-personalized-chatgpt-experience-over-your-data-2b9d140769b1), a project inspired by OpenAI's GPTs that lets you build a low-code chatbot over your data using Streamlit
- Our [OpenAI agents](/module_guides/deploying/agents/modules.md) are all chat bots in nature
- Our [OpenAI agents](/module_guides/deploying/agents/modules.md) are all chatbots in nature
## External sources
......
# Multi-modal
LlamaIndex offers capabilities to not only build language-based applications, but also **multi-modal** applications - combining language and images.
LlamaIndex offers capabilities to not only build language-based applications but also **multi-modal** applications - combining language and images.
## Types of Multi-modal Use Cases
This space is actively being explored right now, but there are some fascinating use cases popping up.
This space is actively being explored right now, but some fascinating use cases are popping up.
### RAG (Retrieval Augmented Generation)
......@@ -73,7 +73,7 @@ maxdepth: 1
These sections show comparisons between different multi-modal models for different use cases.
### LLaVa-13, Fuyu-8B and MiniGPT-4 Multi-Modal LLM Models Comparison for Image Reasoning
### LLaVa-13, Fuyu-8B, and MiniGPT-4 Multi-Modal LLM Models Comparison for Image Reasoning
These notebooks show how to use different Multi-Modal LLM models for image understanding/reasoning. The various model inferences are supported by Replicate or OpenAI GPT4-V API. We compared several popular Multi-Modal LLMs:
......@@ -97,7 +97,7 @@ GPT4-V: </examples/multi_modal/openai_multi_modal.ipynb>
### Simple Evaluation of Multi-Modal RAG
In this notebook guide, we'll demonstrate how to evaluate a Multi-Modal RAG system. As in the text-only case, we will consider the evaluation of Retrievers and Generators separately. As we alluded in our blog on the topic of Evaluating Multi-Modal RAGs, our approach here involves the application of adapted versions of the usual techniques for evaluating both Retriever and Generator (used for the text-only case). These adapted versions are part of the llama-index library (i.e., evaluation module), and this notebook will walk you through how you can apply them to your evaluation use-cases.
In this notebook guide, we'll demonstrate how to evaluate a Multi-Modal RAG system. As in the text-only case, we will consider the evaluation of Retrievers and Generators separately. As we alluded to in our blog on the topic of Evaluating Multi-Modal RAGs, our approach here involves the application of adapted versions of the usual techniques for evaluating both Retriever and Generator (used for the text-only case). These adapted versions are part of the llama-index library (i.e., evaluation module), and this notebook will walk you through how you can apply them to your evaluation use cases.
```{toctree}
---
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment