* Reranker WIP

* add cacheing and singleton loading

* Add field to workspaces for vectorSearchMode
Add UI for lancedb to change mode
update all search endpoints to pass in reranker prop if provider can use it

* update hint text

* When reranking, swap score to rerank score

* update optchain

* make `similarityResponse` object arguments and not positional

* reuse client for qdrant

floor ms value

* Add support for Google Generative AI (Gemini) embedder

* Add missing example in docker
Fix UI key elements in options
Add Gemini to data handling section
Patch issues with chunk handling during embedding

* remove dupe in env

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>

* prevent non admin users from changing llm settings via custom-models endpoint

* permission middleware to JSDOC

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>

* normalize paths on files uploaded to prevent arbitrary file writes

* force normalize path in string parse

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>

fixed wrong metadata assignment

* Add support for gemini authenticated models endpoint
add customModels entry
add un-authed fallback to default listing
separate models by expiermental status
resolves #2866

* add back improved logic for apiVersion decision making

* WIP performance metric tracking

* fix: patch UI trying to .toFixed() null metric
Anthropic tracking migraiton
cleanup logs

* Apipie implmentation, not tested

* Cleanup Anthropic notes, Add support for AzureOpenAI tracking

* bedrock token metric tracking

* Cohere support

* feat: improve default stream handler to track for provider who are actually OpenAI compliant in usage reporting
add deepseek support

* feat: Add FireworksAI tracking reporting
fix: improve handler when usage:null is reported (why?)

* Add token reporting for GenericOpenAI

* token reporting for koboldcpp + lmstudio

* lint

* support Groq token tracking

* HF token tracking

* token tracking for togetherai

* LiteLLM token tracking

* linting + Mitral token tracking support

* XAI token metric reporting

* native provider runner

* LocalAI token tracking

* Novita token tracking

* OpenRouter token tracking

* Apipie stream metrics

* textwebgenui token tracking

* perplexity token reporting

* ollama token reporting

* lint

* put back comment

* Rip out LC ollama wrapper and use official library

* patch images with new ollama lib

* improve ollama offline message

* fix image handling in ollama llm provider

* lint

* NVIDIA NIM token tracking

* update openai compatbility responses

* UI/UX show/hide metrics on click for user preference

* update bedrock client

---------

Co-authored-by: shatfield4 <seanhatfield5@gmail.com>

* wip remove all docs clear vector db on embedder/vector db change

* purge all cached docs and remove docs from workspaces on vectordb/embedder change

* lint

* remove unneeded console log

* remove reset vector stores endpoint and move to server side updateENV with postUpdate check

* reset embed module

* remove unused import

* simplify deletion process
rescoped document deletion to be more general for speed, everything needs to be reset anyway
fixed issue where unembedded docs not in any workspaces, but cached, were not removed

* add back missing readme file
update warning text modals

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>

* added attachments to genericopenai prompt

* add devnote

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>

* fix apipie streaming/sort by chat models

* lint

* linting

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>

connect #2788

* Add Support for NVIDIA NIM

* update README

* linting

* wip hub connection page fe + backend

* lint

* implement backend for local hub items + placeholder endpoints to fetch hub app data

* fix hebrew translations

* revamp community integration flow

* change sidebar

* Auto import if id in URL param
remove preview in card screen and instead go to import flow

* get user's items + team items from hub + ui improvements to hub settings

* lint

* fix merge conflict

* refresh hook for community items

* add fallback for user items

* Disable bundle items by default on all instances

* remove translations (will complete later)

* loading skeleton

* Make community hub endpoints admin only
show visibility on items
combine import/apply for items to they are event logged for review

* improve middleware and import flow

* community hub ui updates

* Adjust importing process

* community hub to dev

* Add webscraper preload into imported plugins

* add runtime property to plugins

* Fix button status on imported skill change
show alert on skill change
Update markdown type and theme on import of agent skill

* update documentaion paths

* remove unused import

* linting

* review loading state

---------

Co-authored-by: Timothy Carambat <rambat1010@gmail.com>

* exposes `maxConcurrentChunks` parameter for the generic openai embedder through configuration. This allows setting a batch size for endpoints which don't support the default of 500

* Update new field to new UI
make getting to ensure proper type and format

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>

* add mistral embedding engine support

* remove console log + fix data handling onboarding

* update data handling description

---------

Co-authored-by: Timothy Carambat <rambat1010@gmail.com>

resolves #2657

* togetherai llama 3.2 vision models support

* remove console log

* fix listing to reflect what is on the chart

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>

* Refactor workspace sidebar component styles (#2380)

rely on css for conditional styles

* New sidebar colors (#2381)

new sidebar colors

* Main container color update (#2382)

* Dark mode setup themes (#2411)

* setup generic tailwind theme + ability to add new themes

* add theme context

* use correct colors from design for sidebar + fix padding

* Settings sidebar UI updates (#2416)

settings sidebar ui updates

* fix sidebar resizing/truncate issue on hover

* Dark mode chat window (#2443)

* Support XLSX files (#2403)

* support xlsx files

* lint

* create seperate docs for each xlsx sheet

* lint

* use node-xlsx pkg for parsing xslx files

* lint

* update error handling

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>

* wip chat window

* ux+ux improvements and update new colors

* chat window dark mode

* remove comment

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>

* Dark mode welcome page (#2444)

* dark mode welcome page styles + refactor

* remove AI_BACKGROUND_COLOR and USER_BACKGROUND_COLOR constants

* Dark mode UI for admin and tools pages + mobile view improvements (#2454)

* dark mode ui for admin and tools pages + mobile view improvements

* lint

* ai provider pages + options darkmode ui

* placeholder generic class

* appearance settings styles

* ai providers mobile margin

* dark mode styles for agent skills + experimental features

* mobile styles on security settings

* fine tune flow ui dark mode

* workspace settings page

* lint

* Dark mode onboarding (#2461)

dark mode onboarding

* update all modals + normalize styles (#2471)

* lint

* Dark mode privacy & experimental pages (#2479)

* document watch + privacy pages ui + mobile modal darkmode

* lint

* Dark mode login screens (#2483)

* multi-user auth screen ui update

* dark mode login screen + recovery key modals

* remove unneeded import

* Workspace preset commands modals dark mode (#2484)

update workspace preset dark mode modal

* Document pinning modal ui update (#2490)

document pinning modal ui update

* Experimental agreement modal dark mode (#2491)

experimental agreement modal dark mode

* Serp options dark mode (#2492)

serp options dark mode

* field fixes

* attempt light mode wip

* setting sidebar

* Toasts and threads

* main page content and privacy page

* force rewrite for light

* add border for light mode rightside content

* more fixes

* wip

* wip

* wip light mode implementation

* wip dark light mode file picker

* document picker light mode ui

* slight ui tweaks

* light mode fine tuning flow

* light mode tweaks + qa fixes

* fix md rendering of light mode + tooltip fixes

* lint

* qa bug fixes

* Add developer hook for theme
move provider to outmost layer

* qa light mode bug fixes

* Linting and hotfixes for UI

* Light mode to dev

* accept invite light mode ui fix

* Fix onboarding inputs in dark mode

* fix close icons
last minute items

* patch z-index on tooltips

* patch light mode citations

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>

* bump together ai models

* Run post-bump command

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>

* patch bad models endpoint path in lm studio embedding engine

* convert to OpenAI wrapper compatibility

* add URL force parser/validation for LMStudio connections

* remove comment

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>

Fix the problem of web-searching can not complete searching sessions when using bing search. (#2611)

Fix the problem of web-searching can not complete searching sessions.

resolves #2593

* duckduckgo web search agent skill support

* move ddg to first option in dropdown menu

* lint

* add duckduckgo option stating no config required

* Add header static class for metadata assembly

* update comments

* patch header parsing for links

* feat: add new model provider: Novita AI

* feat: finished novita AI

* fix: code lint

* remove unneeded logging

* add back log for novita stream not self closing

* Clarify ENV vars for LLM/embedder seperation for future
Patch ENV check for workspace/agent provider

---------

Co-authored-by: Jason <ggbbddjm@gmail.com>
Co-authored-by: shatfield4 <seanhatfield5@gmail.com>