news
312
Delta Inference Engine, a Windows port of the Linux engine Xolotl, runs all 552B parameters of DeepSeek-V4.1-Flash-1017 with 2-bit experts on a single consumer PC. It serves an OpenAI- and Anthropic-compatible API with up to 1M tokens of context. It needs an NVIDIA RTX card with 12 GB+ VRAM, 128 GB+ RAM and about 385 GB on a fast NVMe SSD. On an RTX 3090 with 128 GB RAM it writes about 33 tokens per second.
Comments
0Only full members can comment
Log in to express your opinion.
No comments yet.