NVIDIA/flare-local-inference

Flare is a local inference server for NVIDIA SM120/SM121 GPUs, built for long conversations, coding agents, and repeated tool use. Its goal is to make the memory, bandwidth, and compute in workstations and compact systems such as DGX Spark go further, while keeping generation responsive. This is a research prototype, and is made available via open-source to enable technical collaboration and collect early feedback. The design rests on a few principles: Ignition uses PyTorch, Transformers, and FlashInfer/b12x to load models, tune kernels, compile forwards, and capture CUDA graphs into a runtime package. Helios, the Rust runtime, serves that package without Python. Compilation and graph capture finish before serving. wherever their dataflow fits, and specialize the parts that need different layouts or execution. Model development keeps the tools and libraries that make the Python ecosystem productive. ahead of time. Treat conversation state—including recurrent state alongside KV caches—as a whole, so reuse,…