support for gemini-1.0-pro model and fixes to prompt window limit

implement bing search engine for agents

* Patch handling of end chunk stream events for OpenAI endpoints

* update LiteLLM to use generic handler

* update for empty choices

* Improve RAG responses via source backfilling

* Hide irrelevant citations from UI

* add support for gemini-1.5-flash-latest

* update comment in gemini LLM provider

resolves #1491

* Update CHROMA_SETUP.md

* Update ASTRA_SETUP.md

* Allow setting of safety thresholds for Gemini

* linting

* add support for voyageai embedder

* remove unneeded import

* linting

* Add ENV examples
Update how chunks are processed for Voyage
use correct langchain import
Add data handling

---------

Co-authored-by: Timothy Carambat <rambat1010@gmail.com>

resolves #1439

* Patch WSS upgrade for manual HTTPS certs

* update comment

* refactor

* litellm LLM provider support

* fix lint error

* change import orders
fix issue with model retrieval

---------

Co-authored-by: Timothy Carambat <rambat1010@gmail.com>

* Support SQL Agent skill

* add MSSQL agent connector

* Add frontend to agent skills
remove FAKE_DB mock
reset skills to pickup child-skill dynamically

* add prompt examples for tools on untooled

* add better logging on SQL agents

* Wipe toolruns on each chat relay so tools can be used within the same session

* update comments

* Add Speech-to-text and Text-to-speech providers

* add files and update comment

* update comments

* patch: bad playerRef check

* add api key support for oobabooga web ui

* dont expose API Key for TextWebGenUi

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>

Make LanceDB the vector database default provider in backend to prevent issues where somehow this key is not set by the user resulting in a Pinecone error even though they never said they wanted Pinecone to be their vector db

validate messages schema for gemini provider

* prevent accidental lockout from restrict chars in single pass mode

* update error message

* add max tokens field to generic openai llm connector

* add max_tokens property to generic openai agent provider

* WIP slash presets

* WIP slash command customization CRUD + validations complete

* backend slash command support

* fix permission setting on new slash commands
rework form submit and pattern on frontend

* Add field updates for hooks,
required=true to field
add user<>command constraint to keep them unique
enforce uniquness via teritary uid field on table for multi and non-multi user

* reset migration

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>

* add OpenRouter support

* add mistral agents
add perplexity agents
add textwebgenui agents

* add LMStudio agent support (generic) support
"work" with non-tool callable LLMs, highly dependent on system specs

* add comments

* enable few-shot prompting per function for OSS models

* Add Agent support for Ollama models

* azure, groq, koboldcpp agent support complete + WIP togetherai

* WIP gemini agent support

* WIP gemini blocked and will not fix for now

* azure fix

* merge fix

* add localai agent support

* azure untooled agent support

* merge fix

* refactor implementation of several agent provideers

* update bad merge comment

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>

* add text gen web ui LLM provider support

* update README

* README typo

* update TextWebUI display name
patch workspace<>model support for provider

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>

* add LMStudio agent support (generic) support
"work" with non-tool callable LLMs, highly dependent on system specs

* add comments

* enable few-shot prompting per function for OSS models

* Add Agent support for Ollama models

* improve json parsing for ollama text responses

* add LMStudio agent support (generic) support
"work" with non-tool callable LLMs, highly dependent on system specs

* add comments

* enable few-shot prompting per function for OSS models

* koboldcpp LLM support

* update .env.examples for koboldcpp support

* update LLM preference order
update koboldcpp comments

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>

* getChatCompletion working WIP streaming

* WIP

* working streaming WIP abort stream

* implement cohere embedder support

* remove inputType option from cohere embedder

* fix cohere LLM from not aborting stream when canceled by user

* Patch Cohere implemention

* add cohere to onboarding

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>

* if document is pinned, do not give queryRefusalResponse message

* forgot embed.js patch

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>