Helldez/BigMoeOnEdge
Streaming inference engine for oversized MoE models
- Chart
- Not on a chart right now
- Stars
- 1k
What it is
A llama.cpp-based inference engine that runs Mixture-of-Experts models larger than device memory by streaming needed experts from flash storage. It includes desktop and mobile apps alongside an OpenAI- and Anthropic-compatible local server.
Who it is for
Developers who want to run large MoE models locally on phones, laptops, or consumer GPUs.
Why it is trending
It gained 98 stars today and stands out by supporting any GGUF MoE model and quantization without per-model code across Android, iOS, desktop, and multiple accelerators.
Main features
- Lossless MoE streaming across tiered VRAM, RAM, and flash caches
- Automatic hardware and model profiling with `--auto` planning
- OpenAI- and Anthropic-compatible local server for coding agents
- Vision support via `mmproj` across CPU, GPU, and NPU backends
Summarized on November 17, 2026 from the README and repository data. It can be wrong; check the source before relying on it. Report a wrong summary