Apache APISIX · ai-proxy-multi

Let a System One model pick the LLM

system_one is a load-balancing algorithm for APISIX's ai-proxy-multi. For every request, a fast discriminative model served over the System One API, such as TypeSafe's Jev, ranks your LLM instances by plain-language routing roles, and the gateway sends the request to the best one that is up.

Get started APISIX branch Project on GitHub

How it works

A fourth balancer.algorithm, next to roundrobin, chash and semantic.

Describe the instances

Each ai-proxy-multi instance gets a description: one sentence saying which requests it should get.

Ask once

The prompt goes to the System One API as the state of one choice question whose options are your instances.

Rank

The model returns a probability for every instance. Instances below threshold, unhealthy or rate limited are skipped.

Route and retry

The top instance answers. On a 429 or 5xx, the next one in the ranking does, without asking the model again.

Why use it

Readable policy

One sentence per instance instead of an example corpus. No embedding model to run or keep in sync.

Calibrated

Probabilities, not similarity scores: a threshold of 0.3 means what it says.

Health aware

Uses ai-proxy-multi's active health checks and rate-limit state, which semantic routing ignores.

Ranked retries

A failing pick falls through to the next best instance, or to your fallback when nothing else fits.

Fails open

If the System One API is slow or down, requests go to the fallback instance, bounded by timeout.

Built in, not bolted on

A change to ai-proxy-multi itself, following the conventions of semantic. Runs on APISIX 3.19 today as an overlay.

Configure a route

Pick the algorithm, point it at a System One API, and describe each instance.

ai-proxy-multi:
  balancer:
    algorithm: system_one
  system_one_opts:
    endpoint: https://api.blablador.fz-juelich.de/v1/systemone
    model: alias-clef
    auth:
      header:
        Authorization: "Bearer ${{API_KEY}}"
    threshold: 0.3
    fallback: alias-large
  fallback_strategy: [http_429, http_5xx]
  instances:
    - name: alias-code
      description: Debugging existing code and reviewing patches.
      provider: openai-compatible
      options: { model: alias-code }
      ...
    - name: alias-mimo-v2.6-pro
      description: Rigorous proofs and deep multi-step reasoning.
      ...
    - name: alias-eve
      description: Earth observation and remote sensing.
      ...
    - name: alias-large
      description: Broad synthesis with no dominant specialist.
      ...

Routed live

The bundled Blablador example: 14 instances ranked by alias-clef, against the live API.

Prompt	Picked instance	p
Python traceback: KeyError in a loop	`alias-code`	0.81
Implement an LRU cache in Rust	`alias-glm-5.3-flash`	0.74
Prove √2 is irrational	`alias-mimo-v2.6-pro`	0.94
Sentinel-2 deforestation pipeline	`alias-eve`	0.95
German press release	`alias-apertus`	0.91
400 PDFs of meeting minutes	`alias-kimi-k3-1m`	0.93
Capital of Australia	`alias-fast`	0.94
Multi-agent deployment system	`alias-huge` → 503, retried on `alias-large`	0.92

cd examples/blablador
export BLABLADOR_API_KEY=...
docker compose up -d
curl -i localhost:9080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"auto","messages":[{"role":"user","content":"Prove that sqrt(2) is irrational"}]}'