news
244

A new dependency-free C inference engine called waste allows users to run the massive 2.78-trillion-parameter Kimi K3 model using only 29 GB of RAM. It achieves this feat by streaming activated weights directly from NVMe storage rather than loading the entire model into memory. While the inference speed is a slow 0.50 tokens per second, it demonstrates a novel approach to running extremely large AI models on limited hardware.

Only full members can comment

Log in to express your opinion.

No comments yet.