ekzhang/openjev-sglang

A server implementing the TypeSafe/Jev HTTP API with Qwen3.6-35B-A3B on SGLang. Each container one B200 with SGLang 0.5.19's Rust frontend, radix caching, and breakable prefill CUDA graphs. A separate Python API process uses FastAPI, uvloop, the Rust-backed HF tokenizer, and pooled asynchronous HTTP connections to SGLang on localhost. CUDA dependencies stay in SGLang's container; uv sync on your laptop installs only the API, deployment tools, and tests. The deployment prints a https://...us-west.modal.direct URL. It uses a Modal Server, unauthenticated=True, routingregion="us-west", and computeregion=["us-west", "us-central", "us"]. Autoscaling has no explicit container cap and scales to zero after five idle minutes. Set mincontainers=1 in modalapp.py to keep a B200 warm. If SGLang exits unexpectedly, the API exits too. The Modal launcher watches the API and exits the container so Modal can replace it, rather than leaving a live HTTP process with a dead inference backend. Normal shutdown disarms both watchers. Cache warmups also…