Skip to content
Snippets Groups Projects
Commit ed4dcafb authored by sekyonda's avatar sekyonda
Browse files

Update inference.md

parent 18ea0a62
No related branches found
No related tags found
No related merge requests found
...@@ -43,4 +43,10 @@ Alternate inference options include: ...@@ -43,4 +43,10 @@ Alternate inference options include:
To use vLLM you will need to install it using the instructions [here](https://vllm.readthedocs.io/en/latest/getting_started/installation.html#installation). To use vLLM you will need to install it using the instructions [here](https://vllm.readthedocs.io/en/latest/getting_started/installation.html#installation).
Once installed, you can use the vLLM_ineference.py script provided [here](vLLM_inference.py). Once installed, you can use the vLLM_ineference.py script provided [here](vLLM_inference.py).
[**TGI**](https://github.com/huggingface/text-generation-inference): Text Generation Inference (TGI) is another inference option available to you. For more information on how to set up and use TGI see [here](https://github.com/huggingface/text-generation-inference). Below is an example of how to run the vLLM_inference.py script found within the inference folder.
``` bash
python vLLM_inference.py --model_name <PATH/TO/LLAMA/7B>
```
[**TGI**](https://github.com/huggingface/text-generation-inference): Text Generation Inference (TGI) is another inference option available to you. For more information on how to set up and use TGI see [here](hf-text-generation-inference/README.md).
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment