* Enable `num_ctx` to match defined chunk length in ollama embedder

* remove console

connect #3023
Note: depends on user naming the deployment correctly.

* Add tokenizer improvments via Singleton class
linting

* dev build

* Estimation fallback when string exceeds a fixed byte size

* Add notice to tiktoken on backend

* Add ability to disable default agent skills

* debug build

* remove native llm

* remove node-llama-cpp from dockerfile

* remove unneeded items from dockerfile

---------

Co-authored-by: Timothy Carambat <rambat1010@gmail.com>

* implement dynamic fetching of togetherai models

* implement caching for togetherai models

* update gitignore for togetherai model caching

* Remove models.json from git tracking

* Remove .cached_at from git tracking

* lint

* revert unneeded change

---------

Co-authored-by: Timothy Carambat <rambat1010@gmail.com>

* bump perplexity models

---------

Co-authored-by: Timothy Carambat <rambat1010@gmail.com>

* wip agent ui animation

* WIP agent ui revision

* linting

* simplify css

* memoize agent responses

---------

Co-authored-by: shatfield4 <seanhatfield5@gmail.com>

* fix similarity score bug in lance/chroma dbs

* batch lower bound case

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>

* Support historical message image inputs/attachments for n+1 queries

* patch gemini

* OpenRouter vision support cleanup

* xai vision history support

* Mistral logging

---------

Co-authored-by: shatfield4 <seanhatfield5@gmail.com>

resolves #2956

resolves #2972

rename file typo

* feat: add support for voyage-3-large and voyage-code-3 embedding models

- Add voyage-3-large and voyage-code-3 to VoyageAiOptions dropdown
- Update getMaxEmbeddingLength to support 32k context for new models
- Update .env.example with new model options

* unset env example

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>

update example env files

* Reranker WIP

* add cacheing and singleton loading

* Add field to workspaces for vectorSearchMode
Add UI for lancedb to change mode
update all search endpoints to pass in reranker prop if provider can use it

* update hint text

* When reranking, swap score to rerank score

* update optchain

* make `similarityResponse` object arguments and not positional

* reuse client for qdrant

floor ms value

* Add support for Google Generative AI (Gemini) embedder

* Add missing example in docker
Fix UI key elements in options
Add Gemini to data handling section
Patch issues with chunk handling during embedding

* remove dupe in env

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>

* prevent non admin users from changing llm settings via custom-models endpoint

* permission middleware to JSDOC

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>

* patch unauthorized viewing of other user's pfps

* inline return responses

---------

Co-authored-by: Timothy Carambat <rambat1010@gmail.com>

* normalize paths on files uploaded to prevent arbitrary file writes

* force normalize path in string parse

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>

fix typo in frontend for SearXNG agent setup

fixed wrong metadata assignment

remove fine-tuning flow