The Large Model Systems Organization develops large models and systems that are open, accessible, and scalable.
Latest Blog
See all posts
Chasing the Batch-1 Floor: Ling-3.0-flash Speculative Decode on Blackwell
Batch-1 decode keeps getting more important. Xiaomi MiMo, for example, announced MiMo-V2.5-Pro UltraSpeed in June, claiming 1,000 tok/s decode on a one-trillion-parameter MoE model. Batch 1 gives an ...
Fast Engine Recovery: Sub-Second Engine Restart for SGLang via Weight Cache Daemon
Nowadays, State-of-the-Art (SOTA) models are getting much bigger and reloading the model service after a crash is very expensive. Therefore, we are introducing the Weight Cache Daemon, a persistent GP...
Mooncake for Miles: From Fragmented Rollout Data to Efficient Bulk I/O
Reinforcement learning for large language models combines two very different workloads: rollout generation and model training. During rollout, inference workers run the current policy on a set of pro...
Projects
View all projects
Our Sponsors & Partners
Backed by leading companies and institutions advancing AI research.
Voltage Park, NVIDIA, Nebius, Google Cloud, AtlasCloud, a16z, AMD, InnoMatrix, Laude Institute, Hyperbolic, NovitaAI, Verda Cloud, Sky9, Kaggle, MBZUAI, Together, RunPod, Anyscale, HuggingFace