RouteLLM
A framework for serving and evaluating LLM routers - save LLM costs without compromising quality
About this project
RouteLLM RouteLLM is a framework for serving and evaluating LLM routers. [Blog ] [Paper ] Our core features include: — Drop-in replacement for OpenAI's client (or launch an OpenAI-compatible server) to route simpler queries to cheaper models. — Trained routers are provided out of the box, which we have shown to reduce costs by up to 85% while maintaining 95% GPT-4 performance on widely-used benchmarks like MT Bench. — Benchmarks also demonstrate that these routers achieve the same performance as commercial offerings while being 40% cheaper. — Easily extend the framework to include new routers and compare the performance of routers across multiple benchmarks. Installation From…
Technologies
Project health
GitHub
Reviews
Built by
Maintain lm-sys/RouteLLM? Claiming verifies admin access through your GitHub account and gives you control of this listing.
