From bcdc39e32b270e319eff81c69dee626ccb6ec4fb Mon Sep 17 00:00:00 2001 From: Matthias Reso <13337103+mreso@users.noreply.github.com> Date: Tue, 3 Oct 2023 05:37:20 -0700 Subject: [PATCH] Added documentation for th batching strategy --- docs/Dataset.md | 16 ++++++++++++++-- 1 file changed, 14 insertions(+), 2 deletions(-) diff --git a/docs/Dataset.md b/docs/Dataset.md index 100dd3ab..034b00a5 100644 --- a/docs/Dataset.md +++ b/docs/Dataset.md @@ -7,6 +7,18 @@ The provided fine tuning script allows you to select between three datasets by p * [samsum_dataset](https://huggingface.co/datasets/samsum) contains about 16k messenger-like conversations with summaries. * [OpenAssistant/oasst1](https://huggingface.co/datasets/OpenAssistant/oasst1/) contains about 88k messages from assistant-style conversations. +## Batching Strategies +Llama-recipes support two strategies to batch requests together. +The default setting is `packing` which concatenates the tokenized samples into long sequences filling up the context length of the model. +This is the most compute efficient variant as it avoids any padding and all sequences have the same langth. +Samples at the boundary of the context length are truncated and the remainder of the cut sequence it used as the start of the next long sequence. + +If the amount of training data is small this proceedure might introduce a lot of noise into the training data which can hurt the prediction performance of the fine-tune model. +Therefore, we also support a `padding` strategy which does not introduce the addition noise due to truncated sequences. +The strategy tries to minimize the efficiency loss by batching samples of similar length together so only minimal padding is necessary. + +The batching strategy can be selected though the command line parameter `--batching_strategy [packing]/[padding]`. + ## Using custom datasets The list of available datasets in llama-recipes is supposed to give users a quick start on training their Llama model. @@ -25,7 +37,7 @@ The `dataset_config` in the above signature will be an instance of llama_recipes The split signals wether to return the training or validation dataset. The default function name is `get_custom_dataset` but this can be changes as described below. -In order to start a training with the custom dataset we need to set the `--dataset` as well as the `--custom_dataset.file` parameter. +In order to start a training with the custom dataset we need to set the `--dataset` as well as the `--custom_dataset.file` parameter. ``` python -m llama_recipes.finetuning --dataset "custom_dataset" --custom_dataset.file "examples/custom_dataset.py" [TRAINING PARAMETERS] ``` @@ -35,7 +47,7 @@ python -m llama_recipes.finetuning --dataset "custom_dataset" --custom_dataset.f ``` This will call the function `get_foo` instead of `get_custom_dataset` when retrieving the dataset. -### Adding new dataset +### Adding new dataset Each dataset has a corresponding configuration (dataclass) in [configs/datasets.py](../src/llama_recipes/configs/datasets.py) which contains the dataset name, training/validation split names, as well as optional parameters like datafiles etc. Additionally, there is a preprocessing function for each dataset in the [datasets](../src/llama_recipes/datasets) folder. -- GitLab