Skip to main content

GoodServe: Towards High-Goodput Serving of Agentic LLM Inferences over Heterogeneous Resources

Boxiao Du, Boning Huangfu, Yizhou Luo, Chen Chen, Zijun Li, Minchen Yu, Xiaoyi Fan, Minyi Guo · arXiv preprint arXiv:2605.16867 , 2026

HTML DOI arXiv PDF

Abstract

GoodServe is a goodput-optimized serving system for agentic LLM inference over heterogeneous GPU resources. It combines predict-and-rectify routing, just-enough instance selection, SLO-risk monitoring, and request migration to improve goodput for latency-sensitive agentic workloads.

Citation

@article{goodserve2026,
  title = {GoodServe: Towards High-Goodput Serving of Agentic LLM Inferences over Heterogeneous Resources},
  author = {Du, Boxiao and Huangfu, Boning and Luo, Yizhou and Chen, Chen and Li, Zijun and Yu, Minchen and Fan, Xiaoyi and Guo, Minyi},
  journal = {arXiv preprint arXiv:2605.16867},
  year = {2026},
  arxiv = {2605.16867},
  doi = {10.48550/arXiv.2605.16867}
}