news
295

A new project called Janas allows users to run large language models that exceed their system memory by streaming expert weights directly from an SSD. The pure C tool supports both CPU and GPU acceleration and significantly outperforms llama.cpp in benchmark tests. Notably, it achieved 37 tokens per second on an 80-billion parameter model using a standard 32GB laptop.

Only full members can comment

Log in to express your opinion.

No comments yet.