news
295
A new project called Janas allows users to run large language models that exceed their system memory by streaming expert weights directly from an SSD. The pure C tool supports both CPU and GPU acceleration and significantly outperforms llama.cpp in benchmark tests. Notably, it achieved 37 tokens per second on an 80-billion parameter model using a standard 32GB laptop.
Comments
0Discuss on Hacker NewsOnly full members can comment
Log in to express your opinion.
No comments yet.