* wip

* wip

* wip

* wip

* wip

* add notebook

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wi

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* rip out token counter

* wip

* wip

* wip

* remove formatted prompt logging

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wipg

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* better typing

* fix bug

* wip

* wip

* wip

* simplify and rename

* wip

* wip

* wip

* wup

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* allow tiktoken for python38

* wip

* wip

* address comments

* fix lint

* add note

* fix merge

* wip

* wip

* wiup

* add to agent and programs

* wip

* update notebook

* wip

* wip

* wiup

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* remove unused imports

* update change log

* add back missing file

* clean

* wip

* update notebook

* update notebook

* wip

* expose message history

* wip

* wip

* revert chat engine changes

* wip

* lint

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* revert ordering

* wip

* wiup

* wip

* wip

* wip

* revert settings change

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wi

* wip

* wip

* use llama_index

* replace in docs

* replace all tests

* replace other packages

* replace more docs

* more docs

* update others

* update setup

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* fix

* wip

* wip

* wip

* wip

* wip

* mwip

* wip

* wip

* wip

* wrong import

* wip

* Update docs and notebooks for index refactor (#1302)

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* fix dependency

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* fix merge

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* no aquery

* wip

* wip

* wip

* wip storage refactor

* wip

* remove

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* add updated reference docs  (#1316)

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* fix faiss

* wip

* wip

* wip

* wip

* wip

* wip

* remove common patch

* cleanup data structs

* wip

* remove simple vector store index

* Revert "remove simple vector store index"

This reverts commit 2acacd4fc80aa666c3276894ccac068ce60a3d7a.

* Revert "Revert "remove simple vector store index""

This reverts commit 6d4af899d889d49836310a1042b183f46830e0ef.

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* remove extra files

* add retriever module

* wip

* add docstrings, and reference section, to storage refactor  (#1342)

* cr

* cr

* wip

* lint

* wip

* remove migration script

* fix test

* wip

* wip

* remove mention from notebooks

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* add storage docs (#1344)

* wip

* part 1: fix notebooks (#1346)

* wip

* wip

* notebook

* wip

* wip

* wip

* add path to `persist` for all storage classes (#1347)

* cr

* cr

* wip

* wip

* wip

* wip

* wi

* wip

* wip

* wip

---------

Co-authored-by: Simon Suo <simonsdsuo@gmail.com>

* wip

* wip

* remove from_dict and config_dict from vector stores (#1349)

* fix vector store docs (#1355)

* set index id  (#1350)

* wip

* wip

* wip

* wi

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* add image

* wip

* wip

* wip

* wip

* remove outdated docs

* wip

---------

Co-authored-by: Jerry Liu <jerryjliu98@gmail.com>

Co-authored-by: Jerry Liu <jerry@robustintelligence.com>

Co-authored-by: Jerry Liu <jerry@robustintelligence.com>

Co-authored-by: Jerry Liu <jerry@robustintelligence.com>