Skip to content
Snippets Groups Projects
Unverified Commit 71cbaa29 authored by Suraj Subramanian's avatar Suraj Subramanian Committed by GitHub
Browse files

add clarity on EOS token

parent 9944493d
No related branches found
No related tags found
No related merge requests found
...@@ -7,8 +7,9 @@ The 'llama-recipes' repository is a companion to the [Llama 2 model](https://git ...@@ -7,8 +7,9 @@ The 'llama-recipes' repository is a companion to the [Llama 2 model](https://git
> | Token | Description | > | Token | Description |
> |---|---| > |---|---|
> `<\|begin_of_text\|>` | This is equivalent to the BOS token. | > `<\|begin_of_text\|>` | This is equivalent to the BOS token. |
> `<\|eot_id\|>` | This signifies the end of the message in a turn. This is equivalent to the EOS token. | > `<\|eot_id\|>` | This signifies the end of the message in a turn. |
> `<\|start_header_id\|>{role}<\|end_header_id\|>` | These tokens enclose the role for a particular message. The possible roles can be: system, user, assistant. | > `<\|start_header_id\|>{role}<\|end_header_id\|>` | These tokens enclose the role for a particular message. The possible roles can be: system, user, assistant. |
> `<\|end_of_text\|>` | This is equivalent to the EOS token. On generating this token, Llama 3 will cease to generate more tokens |
> >
> A multiturn-conversation with Llama 3 follows this prompt template: > A multiturn-conversation with Llama 3 follows this prompt template:
> ``` > ```
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment