[doc] update jaguar vector store documentation (#10179)

* jaguar vector store * fixed embedding None non-iterable issue * Fixed test_jaguar.py with mock client and added documentation. * Fixed format and added separate sections and outputs in JaguarIndexDemo.ipynb * added an example of query filter to jaguar vector store * Feature #10056: add feature "text_tag" to add_text() jaguar vector store * update jaguar vector store documentation --------- Co-authored-by: JY <jyjy@jaguardb>

[doc] update jaguar vector store documentation (#10179)
3dd55db6 · JaguarDB · GitHub · 9f9d048d · 3dd55db6
Unverified Commit 3dd55db6 authored 1 year ago by JaguarDB Committed by GitHub 1 year ago
--- a/docs/examples/vector_stores/JaguarIndexDemo.ipynb
+++ b/docs/examples/vector_stores/JaguarIndexDemo.ipynb
@@ -28,6 +28,8 @@
    "\n",
    "You must install packages llama-index and jaguardb-http-client.\n",
    "\n",
+    "    docker pull jaguardb/jaguardb_with_http\n",
+    "    docker run -d -p 8888:8888 -p 8080:8080 --name jaguardb_with_http jaguardb/jaguardb_with_http\n",
    "    pip install -U llama-index\n",
    "    pip install -U jaguardb-http-client\n",
    "\n",
@@ -233,6 +235,10 @@
    "    documents, storage_context=storage_context, service_context=service_context\n",
    ")\n",
    "\n",
+    "### You could add more documents to the vector store:\n",
+    "# jaguarstore.add_documents(some_docs)\n",
+    "# jaguarstore.add_documents(more_docs, text_tag=\"tag to these documents\")\n",
+    "\n",
    "### print number of documents in jaguar vector store\n",
    "num = jaguarstore.count()\n",
    "print(f\"There are {num} vectors in jaguar vector store\")"

 %% Cell type:markdown id:41a004a3-d197-4a59-b63d-886faaf83ab8 tags:

 # Jaguar Vector Store

 This document demonstrates llama_index working with Jaguar vector store.

 - It is a distributed vector database that can store large number of vectors.
 - The ZeroMove feature enables instant horizontal scaling.
 - It supports embeddings, text, images, videos, PDFs, audio, time series, and spatial data.
 - The all-master architecture allows both parallel reads and writes.
 - Its anomaly detection capabilities can distinguish outliers in the dataset.
 - The RAG support can combine LLMs and proprietary and real-time data.
 - Sharing of metadata across multiple vector indexes improves data consistency.
 - Distance metrics include Euclidean, Cosine, InnerProduct, Manhatten, Chebyshev, Hamming, Jeccard, and Minkowski.
 - Similarity search can be performed with time cutoff and time decay effects.

 ## Prerequisites

 There are two requirements for running the examples in this file.

 You must install and set up the JaguarDB server and its HTTP gateway server.
 Please follow the instructions in [Jaguar Setup](http://www.jaguardb.com/docsetup.html) as a reference.

 You must install packages llama-index and jaguardb-http-client.

+    docker pull jaguardb/jaguardb_with_http
+    docker run -d -p 8888:8888 -p 8080:8080 --name jaguardb_with_http jaguardb/jaguardb_with_http
    pip install -U llama-index
    pip install -U jaguardb-http-client



 %% Cell type:code id:a1c49fef-48b3-4319-8d3c-afdfeff1f832 tags:

 ``` python
 !pip install -U jaguardb-http-client
 ```

 %% Output

    Collecting jaguardb-http-client
      Using cached jaguardb_http_client-3.4.1-py2.py3-none-any.whl (15 kB)
    Installing collected packages: jaguardb-http-client
    Successfully installed jaguardb-http-client-3.4.1

 %% Cell type:markdown id:3e347858-487b-4776-ae5a-6ee52097a14a tags:

 ## Imports
 The following packages should be imported. We use the OpenAIEmbedding as an example. You could choose other embedding models in your application.

 %% Cell type:code id:873e27a8-7807-4f21-b505-84aff5299262 tags:

 ``` python
 from llama_index import (
    VectorStoreIndex,
    ServiceContext,
    SimpleDirectoryReader,
 )
 from llama_index.storage.storage_context import StorageContext
 from llama_index.embeddings import OpenAIEmbedding
 from llama_index.vector_stores.jaguar import JaguarVectorStore
 from jaguardb_http_client.JaguarHttpClient import JaguarHttpClient
 ```

 %% Cell type:markdown id:0ae84951-c6cb-41c1-b02b-643128c639c8 tags:

 ## Client Object
 We now instantiate a jaguar vector store client object. The url is the http endpoint of the gateway server. The url should be replaced with your environment settings. The pod is the Pod (or database) name. The store is the name of the vector store. A pod may have multiple stores. The vector_index is the name of the vector index in the store. A store may have multiple vector indexes. The store client object is, however, bound to one vector index only. The vector_type specifies the attributes of the vector index. In the string "cosine_fraction_short", cosine means that the distance between two vectors is computed with the cosine distance. Fraction means the vector components are fractional numbers. Short means the storage format of the vector components is a short integer of signed 16-bits integers. Storage format could be float of 32-bit floating point numbers. It can also be a byte of 8-bit signed integers. The vector_dimension is the dimension of the vector generated by the provided embedding model.

 %% Cell type:code id:7e2fc699-d38d-47df-9d73-0adb7b2331b5 tags:

 ``` python
 url = "http://127.0.0.1:8080/fwww/"
 pod = "vdb"
 store = "llamaindex_jaguar_store"
 vector_index = "v"
 vector_type = "cosine_fraction_float"
 # vector_type = "cosine_fraction_short"  # half of memory usage compared to float
 # vector_type = "cosine_fraction_byte" # quarter of memory usage compared to float
 vector_dimension = 1536  # per OpenAIEmbedding model
 jaguarstore = JaguarVectorStore(
    pod,
    store,
    vector_index,
    vector_type,
    vector_dimension,
    url,
 )
 ```

 %% Cell type:markdown id:b2c6991c-5223-459d-9247-32eb643a60cd tags:

 ## Authentication
 The client must login or connect to back-end jaguar server for system security and user authentication. Environment variable JAGUAR_API_KEY or file $HOME/.jagrc file must  contain the jaguar api ke issued by your system administrator. The login() method returns True or False. If it returns False, then it may mean that your jaguar api key is invalid, or the http gateway server is not running, or the jaguar server is not running properly.

 %% Cell type:code id:54e667f0-934b-4d9a-9056-5c7a0cc4f9e6 tags:

 ``` python
 true_or_false = jaguarstore.login()
 print(f"login result is {true_or_false}")
 ```

 %% Output

    login result is True

 %% Cell type:markdown id:d305ef91-2de1-4754-af22-dca1e96ec4c2 tags:

 ## Create Vector Store
 We now create a vector store with a field 'v:text' of size 1024 bytes
 to hold text, and two additional metadata fields 'author' and 'category'.

 %% Cell type:code id:179a90fc-47a0-45c7-b116-3a5fe3b14adc tags:

 ``` python
 metadata_str = "author char(32), category char(16)"
 text_size = 1024
 jaguarstore.create(metadata_str, text_size)
 ```

 %% Cell type:markdown id:558a5fbb-5cee-4782-8934-cefb0af22822 tags:

 ## Load Documents
 The following code opens the example Paul Gram documents and read them into memory

 %% Cell type:code id:851a2625-0e5f-4b9d-aa09-77eed9f46efc tags:

 ``` python
 documents = SimpleDirectoryReader("../data/paul_graham/").load_data()
 print(f"loading {len(documents)} doument(s)")
 ```

 %% Output

    loading 1 doument(s)

 %% Cell type:markdown id:4d072cc6-6e93-42aa-9b73-be3025a8190a tags:

 ## Make Index
 Prepare storage context, service context, and make an index object. After the call of from_documents(), there will be 22 vectors saved in the vector store.

 %% Cell type:code id:542cd51b-2aa1-4ff6-b4d5-35b2ac67de61 tags:

 ``` python
 ### make a storage context using our vector store
 storage_context = StorageContext.from_defaults(vector_store=jaguarstore)

 ### have a service context using the openai embedding model
 embed_model = OpenAIEmbedding()
 service_context = ServiceContext.from_defaults(embed_model=embed_model)

 ### clear all vectors in the vector store
 jaguarstore.clear()

 ### make an index with the documents,storage context, and service context
 index = VectorStoreIndex.from_documents(
    documents, storage_context=storage_context, service_context=service_context
 )

+### You could add more documents to the vector store:
+# jaguarstore.add_documents(some_docs)
+# jaguarstore.add_documents(more_docs, text_tag="tag to these documents")
+
 ### print number of documents in jaguar vector store
 num = jaguarstore.count()
 print(f"There are {num} vectors in jaguar vector store")
 ```

 %% Output

    There are 22 vectors in jaguar vector store

 %% Cell type:markdown id:afebb7c0-47f5-4df7-a571-b23628176fb7 tags:

 ## Ask Questions
 We get a query engine and ask some questions to the engine.

 %% Cell type:code id:a37cf316-01c0-4ade-8fbc-e8e5f7ab76c1 tags:

 ``` python
 query_engine = index.as_query_engine()
 q = "What did the author do growing up?"
 print(f"Question: {q}")
 response = query_engine.query(q)
 print(f"Answer: {str(response)}\n")

 q = "What did the author do after his time at Viaweb?"
 print(f"Question: {q}")
 response = query_engine.query(q)
 print(f"Answer: {str(response)}")
 ```

 %% Output

    Question: What did the author do growing up?
    Answer: The author mentioned that growing up, they worked on two main things outside of school: writing and programming. They wrote short stories and tried writing programs on an IBM 1401 computer.
    
    Question: What did the author do after his time at Viaweb?
    Answer: After his time at Viaweb, the author started a company to put art galleries online. However, this idea did not turn out to be successful as art galleries did not want to be online.

 %% Cell type:markdown id:32d0fb08-1860-494a-bac8-f3ead4d0af3a tags:

 ## Pass Query Options
 We can pass extra arguments to the query engine to select only a subset of data from the jaguar vector store. This can be achieved by using the `vector_store_kwargs` argument. Parameter day_cutoff is number of days beyond which text will be ignored. day_decay_rate is rate of daily decay for similarity scores.

 %% Cell type:code id:e5003795-40cf-44e6-a366-f45d98acee22 tags:

 ``` python
 qkwargs = {
    "args": "day_cutoff=365,day_decay_rate=0.01",
    "where": "category='startup' or category=''",
 }
 query_engine_filter = index.as_query_engine(vector_store_kwargs=qkwargs)
 q = "What was the author's life style?"
 print(f"Question: {q}")
 response = query_engine_filter.query(q)
 print(f"Answer: {str(response)}")
 ```

 %% Output

    Question: What was the author's life style?
    Answer: The author's lifestyle involved attending the Accademia as a student and painting still lives in their bedroom at night. They also wrote essays and had a messy life, which they thought would be interesting and encouraging to others.

 %% Cell type:markdown id:13233170-ad42-48e7-9de6-9ba52a6070f1 tags:

 ## Cleanup and Logout
 All vectors and related data in the vector store can be deleted and the vector store can be removed completely to finish the test. Logout call makes sure resources used by the client are released.

 %% Cell type:code id:0d162991-b128-4451-94a9-8ca363cbf4c3 tags:

 ``` python
 ### remove all the data in the vector store if you want
 jaguarstore.clear()

 ### delete the whole vector in the database if you want
 jaguarstore.drop()

 ### disconnect from jaguar server and cleanup resources
 jaguarstore.logout()
 ```