Access Top-Tier Models at 1/10 the Cost
A trusted, compliant, and private gateway that mixes and matches 100+ LLMs into more reliable, cost-efficient AI capabilities.
from gatellm import Client
client = Client(api_key="your-key")
# Compose: cheap models handle bulk work, top-tier for critical steps
workflow = client.compose([
{
"model": "deepseek-v3",
"task": "analyze",
"prompt": "Analyze feasibility of this requirement"
},
{
"model": "claude-4-opus",
"task": "implement",
"prompt": "Implement core architecture"
}
])Integrated with 100+ LLM Providers
"Under extremely stringent compliance requirements, GateLLM not only met our security standards—its cost control capabilities far exceeded expectations. Cost optimization was meant to be a side benefit, but the results were stunning: over 90% reduction in monthly API spend."— Compliance Lead, Top-Tier Investment Firm
One Interface, Mix and Match All Models
From simple model routing to multi-model orchestration, GateLLM provides comprehensive enterprise-grade capabilities.
Multi-Model Routing
One interface to call all models. Intelligent fallback, load balancing, automatic retry.
- 100+ LLM providers unified API
- Intelligent fallback routing
- Load balancing and rate limiting
Composition
Different models for different steps in the same workflow. Claude for planning, DeepSeek for coding, small models for batch classification.
- Seamless integration, no glue code
- Capability-level compatibility normalization
- Best model for each subtask
Ensemble
Fan out to N models per call, aggregate outputs via voting/judging/merging. Achieve quality and reliability no single model can deliver.
- Multi-model voting / judging / best-of-N
- Cross-vendor redundancy, single model failures don't affect overall
- High-risk/high-value call scenarios
Cost Control
Budget management, rate limiting, team cost attribution. Never markup token fees, BYOK with your own keys.
- Cost attribution by project/feature
- Soft budget alerts
- BYOK: pay providers at standard rates
Model Mix-and-Match: Quality No Single Model Can Deliver
Composition mixes multiple models in one workflow; ensemble aggregates multiple models in one call. Both rely on the underlying **capability normalization layer**.
Composition: Multiple Models in One Workflow
Different models for different steps in the same Agent/workflow. The gateway handles capability-level compatibility normalization (tool calling, structured output, vision, caching, reasoning tokens), so composition has no integration tax—no glue code rewrite for each model.
- ✓Best model for each subtask
- ✓No glue code, unified developer experience
- ✓No vendor lock-in, free to switch
# Composition: plan → code → review, each with the best model
workflow = client.compose([
{
"model": "claude-4-opus",
"task": "plan",
"prompt": "Analyze this requirement, output implementation plan"
},
{
"model": "deepseek-v3",
"task": "implement",
"prompt": "Implement code according to plan"
},
{
"model": "gpt-5",
"task": "review",
"prompt": "Review code quality and security"
}
])Ensemble: Aggregate Multiple Models Per Call
One logical call fans out to N models; outputs aggregated via voting/judging/merging/best-of-N. Ensemble effect exceeds any single model, cross-vendor redundancy ensures production stability.
- ✓Accuracy boost for high-risk/high-value calls
- ✓Resilience to single model fluctuations
- ✓Clearest ROI in fraud detection, medical assistance, compliance judgment scenarios
Deploy GateLLM in 5 Minutes
Upgrade from single-model to multi-model orchestration in three steps, no business code rewrite required.
Deploy Gateway
One-click Docker launch, or Kubernetes Helm deployment to private environment. Supports MLPS Level 3 and Xinchuang environments.
Configure Keys
BYOK mode: fill in your existing provider API keys, data stays on-prem, never persisted.
Replace Endpoint
Point OpenAI SDK's base_url to GateLLM, call all models with unified interface.
# Just one line change: point base_url to GateLLM
from openai import OpenAI
client = OpenAI(
base_url="https://your-gateway.gatellm.io/v1",
api_key="your-gatellm-api-key"
)
# Existing code unchanged, call any model
response = client.chat.completions.create(
model="claude-4-opus", # or gpt-5, deepseek-v3, qwen-max...
messages=[{"role": "user", "content": "Hello"}]
)Capability Normalization: Making Orchestration Possible
Both composition and ensemble rely on the underlying capability normalization layer. We solve the famous "seven coupling points" problem—LiteLLM only translates API envelopes, we guarantee capability parity.
Tool Calling Normalization
OpenAI / Anthropic / Gemini / Chinese models all have different tool calling schemas. We unify them at the foundation, write code once, call all models.
Structured Output Compatibility
Each provider uses different protocols (strict JSON schema, json_mode, tool_use). We unify to a consistent interface.
Prompt Caching Unification
Anthropic cache_control, OpenAI auto-caching, Gemini context—each has different mechanisms. We provide unified caching strategy.
Reasoning Token Adaptation
o1 / Claude thinking / DeepSeek R1 output formats differ. We adapt to each provider's reasoning token protocol.
Multimodal Format Support
vision / audio / file upload protocols differ across providers. We unify multimodal input formats.
Billing Model Abstraction
Each provider's pricing structure differs. We abstract to a unified cost tracking interface.
Data Flow: Stays On-Prem, Never Persisted
In BYOK mode, API keys are stored only in gateway memory, never persisted or transmitted. Requests route through gateway directly to provider official APIs, data never leaves your private network.
Why 1/10 the Cost?
Taking an enterprise user with 1B daily tokens as example, see how GateLLM compresses API spend from "top-tier all the way" to "mix-and-match".
| Plan | Monthly Tokens | Unit Price Source | Monthly Cost | vs Baseline |
|---|---|---|---|---|
Top-Tier Models Only (e.g. Claude Opus / GPT-4 class) | ~30B | Provider standard $15 / 1M input tokens | ~$450,000 | 100% (baseline) |
GateLLM Mix-and-Match + Cache Hits + BYOK No Markup | ~30B | Cheap models handle 80% bulk work, top-tier only for critical 20% + Prompt cache hits | ~$45,000 | ~10% (1/10) |
Three Sources of Savings:
- 1.BYOK no token markup—you pay providers at standard rates, GateLLM takes no middle cut;
- 2.Composition—cheap models handle bulk work (classification, extraction, translation), top-tier only for critical steps (planning, complex reasoning);
- 3.Prompt caching—high-frequency calls hit cache, unit price drops 50%–90%.
Figures are illustrative models based on public pricing, actual savings depend on business scenarios and mix ratios. Pro license fee of $750/instance/month is included in GateLLM plan.
GateLLM vs LiteLLM
LiteLLM focuses on API envelope translation; GateLLM guarantees **capability parity** on top of that, with enterprise-grade controls.
| Capability | GateLLM | LiteLLM |
|---|---|---|
| API Routing / Fallback / Load Balancing | ✓ | ✓ |
| BYOK No Token Markup | ✓ | ✓ |
| Capability Normalization (7 coupling points solved) | ✓ Full | Partial |
| Cross-Model Tool Calling Unification | ✓ | ✗ |
| Structured Output (JSON Schema) Unification | ✓ | ✗ |
| Cross-Model Prompt Caching Unification | ✓ | ✗ |
| Reasoning Token Adaptation (o1 / thinking) | ✓ | ✗ |
| Multi-Model Composition (Workflows) | ✓ | ✗ |
| Multi-Model Ensemble (Voting/Judging/best-of-N) | ✓ | ✗ |
| Fully Self-Hosted / Data Stays On-Prem | ✓ | ✓ |
| MLPS Level 3 / Xinchuang Ready | ✓ | ✗ |
| SSO / SCIM (SAML / OIDC / Lark) | ✓ | ✗ |
| Enterprise Support (SLA / Priority Response) | ✓ | Community |
Comparison based on each product's public documentation as of July 2026. LiteLLM column is illustrative, please refer to LiteLLM official sources.
Fully Self-Hosted · Per-Instance Licensing · Never Markup Token Fees
BYOK (bring your own keys), you pay providers at standard rates. We license by "instance count", not by seats or token markup.
Free
Fully self-hosted · Single license
- ✓100+ LLM routing · Fallback · Load balancing
- ✓Ensemble (voting · judging · best-of-N)
- ✓Prompt caching
- ✓BYOK (bring your own keys, no markup)
- ✓512MB storage · SQLite · Single license deployment
- ✓No log retention
Capability
| Capability | Free | Pro | Enterprise |
|---|---|---|---|
| Multi-Model Routing · Fallback · Load Balancing | ✓ | ✓ | ✓ |
| Ensemble (Voting · Judging · best-of-N) | ✓ | ✓ | ✓ |
| Prompt Caching | ✓ | ✓ | ✓ |
| BYOK (Bring Your Own Keys, No Markup) | ✓ | ✓ | ✓ |
| Storage / Database | 512MB · SQLite | Scalable | Scalable |
| Deployment Scale | 1 license | Multi-license tiered discounts | Multi-Node / Cluster |
| Log Retention | None | Retained | Extended Retention |
| SSO / SCIM | — | ✓ | ✓ |
| Compliance (MLPS Level 3 / Xinchuang / Audit / DLP) | — | — | ✓ |
| Support Response | Community | Email / Ticket | Priority Response |