news
244
A new dependency-free C inference engine called waste allows users to run the massive 2.78-trillion-parameter Kimi K3 model using only 29 GB of RAM. It achieves this feat by streaming activated weights directly from NVMe storage rather than loading the entire model into memory. While the inference speed is a slow 0.50 tokens per second, it demonstrates a novel approach to running extremely large AI models on limited hardware.
Comments
0Discuss on Hacker NewsOnly full members can comment
Log in to express your opinion.
No comments yet.