zc/compose-rkllm_chat
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Bring your own rkllm_server for use on updated kernels.

Browse source
This commit is contained in:
Zhengyi Chen 2025-01-25 23:41:07 +08:00
parent c86afd3590
commit 29d061c89d
5 changed files with 870 additions and 2 deletions
Download patch file Download diff file Expand all files Collapse all files

5
docker-compose.yml
View file

@ -7,8 +7,9 @@ services:
devices:
- /dev:/dev
volumes:
- ./model:/rkllm_server/model:ro
- ./rkllm_server/:/rkllm_server/ # bring-your-own server
- ./model/:/rkllm_server/model/:ro
ports:
- "8080:8080"
command: >
sh -c "python3 gradio_server.py --target_platform rk3588 --rkllm_model_path /rkllm_server/model/Qwen2.5-3B.rkllm"
sh -c "python3 gradio_server.py --target_platform rk3588 --rkllm_model_path /rkllm_server/model/Phi-3.5-mini-instruct-w8a8.rkllm"