Skip to content
Snippets Groups Projects
Unverified Commit f1a4fca0 authored by Andrei Fajardo's avatar Andrei Fajardo Committed by GitHub
Browse files

Adds abstractions for `LlamaDatasets` and downloader (#9165)

* init

* add rag dataset

* reshape Dataset

* use reference context and answers

* add RagDatasetGenerator

* add serialization

* add prediction abstratction and predict method

* separate predictions from examples

* add download dataset from github logic

* remove train test splits

* add notebooks

* add async for predictions

* clewan up base

* add model name attr

* mark QRD and DatasetGenerator as deprecated

* update batch_runner to load asycio_module from async_utils

* add kwargs is_dataset flag in download_utils

* use pydantic BaseModel instead of dataclass

* use pydantic BaseModel for CreatedBy

* update nb

* fix sync

* update nb

* point to run-llama llama_datasets repo

* update nbs

* remove unit test

* lint

* relax download utils return type

* lint

* removed data file

* add cli command

* point to run-llama

* make entry to CHANGELOG; add open in colab button
parent 2e99331a
No related branches found
No related tags found
Loading
Showing
with 2781 additions and 54 deletions
Loading
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Please register or to comment