Back to the chart

Neroued/ninfer

Native C++/CUDA inference engine for RTX 5090 and DGX Spark

Chart
Not on a chart right now
Stars
3.4k

Open on GitHub

What it is

NInfer is a native C++/CUDA inference engine for selected models such as Qwen3.6, Qwen3.8, and GLM-5.3-Flash on RTX 5090 and DGX Spark. It runs text and image/video generation through a local CLI or OpenAI- and Anthropic-compatible HTTP APIs.

Who it is for

Developers and engineers deploying high-performance local or dual-node inference servers on RTX 5090 or DGX Spark hardware would use this.

Why it is trending

With 3,376 total stars and 9 gained today, it stands out for its native C++/CUDA implementation featuring CUDA Graph decode, speculative decoding, and shared-prefix caching tailored to RTX 5090 and DGX Spark.

Main features

  • Local CLI and OpenAI/Anthropic-compatible HTTP APIs
  • CUDA Graph decode, speculative decoding (MTP/DFlash), and shared-prefix caching
  • Single-device execution and dual DGX Spark model-parallel execution via RDMA or NCCL
  • Support for text, vision, tool calling, constrained output, and CPU expert offload

Summarized on November 18, 2026 from the README and repository data. It can be wrong; check the source before relying on it. Report a wrong summary