89 lines
2.3 KiB
Markdown
89 lines
2.3 KiB
Markdown
|
# TGI LLM
|
||
|
|
|
||
|
|
In this example, we're running an infernet node along with a TGI service.
|
||
|
|
|
||
|
|
## Deploying TGI Service
|
||
|
|
|
||
|
|
If you have your own TGI service running, feel free to skip this part. Otherwise,
|
||
|
|
you can deploy the TGI service using the following command.
|
||
|
|
|
||
|
|
Make sure you have a machine with proper GPU support. Clone this repository &
|
||
|
|
run the following command:
|
||
|
|
|
||
|
|
```bash
|
||
|
|
make run-service project=tgi-llm service=tgi
|
||
|
|
```
|
||
|
|
|
||
|
|
## Deploying Infernet Node Locally
|
||
|
|
|
||
|
|
Running an infernet node involves a simple configuration step & running step.
|
||
|
|
|
||
|
|
### Configuration
|
||
|
|
|
||
|
|
Copy our [sample config file](./config.sample.json) into a new file
|
||
|
|
called `config.json`.
|
||
|
|
|
||
|
|
```bash
|
||
|
|
cp config.sample.json config.json
|
||
|
|
```
|
||
|
|
|
||
|
|
Then provide the `"env"` field of the `"containers"` section of the file to point to the
|
||
|
|
TGI Service you just deployed.
|
||
|
|
|
||
|
|
```json
|
||
|
|
{
|
||
|
|
// etc.
|
||
|
|
"containers": [
|
||
|
|
{
|
||
|
|
"id": "tgi-llm",
|
||
|
|
"image": "ritualnetwork/llm_inference_service:latest",
|
||
|
|
"external": true,
|
||
|
|
"port": "3000",
|
||
|
|
"allowed_delegate_addresses": [],
|
||
|
|
"allowed_addresses": [],
|
||
|
|
"allowed_ips": [],
|
||
|
|
"command": "--bind=0.0.0.0:3000 --workers=2",
|
||
|
|
"env": {
|
||
|
|
"TGI_SERVICE_URL": "http://{your-service-ip}:{your-service-port}" // <- Change this to the TGI service you deployed
|
||
|
|
}
|
||
|
|
}
|
||
|
|
]
|
||
|
|
}
|
||
|
|
```
|
||
|
|
|
||
|
|
### Running the Infernet Node Locally
|
||
|
|
|
||
|
|
With that out of the way, you can now run the infernet node using the following command
|
||
|
|
at the top-level directory of this repo:
|
||
|
|
|
||
|
|
```
|
||
|
|
make deploy-container project=tgi-llm
|
||
|
|
```
|
||
|
|
|
||
|
|
## Testing the Infernet Node
|
||
|
|
|
||
|
|
You can test the infernet node by posting a job in the node's REST api.
|
||
|
|
|
||
|
|
```bash
|
||
|
|
curl -X POST "http://127.0.0.1:4000/api/jobs" \
|
||
|
|
-H "Content-Type: application/json" \
|
||
|
|
-d '{"containers":["tgi-llm"], "data": {"prompt": "can shrimp actually fry rice?"}}'
|
||
|
|
```
|
||
|
|
|
||
|
|
You can expect a response similar to the following:
|
||
|
|
|
||
|
|
```json
|
||
|
|
{
|
||
|
|
"id": "f026c7c2-7027-4c2d-b662-2b48c9433a12"
|
||
|
|
}
|
||
|
|
```
|
||
|
|
|
||
|
|
You can then check the status of the job using the following command:
|
||
|
|
|
||
|
|
```bash
|
||
|
|
curl -X GET http://127.0.0.1:4000/api/jobs\?id\=f026c7c2-7027-4c2d-b662-2b48c9433a12
|
||
|
|
[{"id":"f026c7c2-7027-4c2d-b662-2b48c9433a12","result":{"container":"tgi-llm","output":{"output":"\n\nI\u2019m not sure if this is a real question or not, but I\u2019m"}},"status":"success"}]
|
||
|
|
```
|
||
|
|
|
||
|
|
Congratulations! You've successfully ran an infernet node with a TGI service.
|