Routing Rules

A routing rule sends matching requests through the Netzilo AI Gateway to a provider or model other than the one the caller asked for. A rule matches requests by model, provider, header, query parameter or request type, spreads them over one or more targets by weight, and tries fallbacks in order when the targets fail.

Use routing rules to:

  • Cut cost. Send simple chat traffic to a cheaper model, or part of it to a model you host yourself.
  • Keep data where it belongs. Send requests marked as EU data to a deployment in the EU.
  • Stay up. Fall back to another provider when one fails, is overloaded, or is slow to start answering.
  • Move coding agents. Run a coding agent such as Claude Code on another model without touching any developer's machine.

You manage routing rules under Edge > Governance > Routing Rules. Only administrators and owners see this page.

How it works

  1. You define a rule: the requests it matches (Conditions), where they go (Targets), what to try when the targets fail (Fallbacks), and its Priority.
  2. You add the rule to one or more gateway filters on their Routing tab.
  3. For each request, the AI Gateway takes the enabled rules that the user's gateway filters carry and evaluates them from the lowest Priority number up. The first rule that matches routes the request.
  4. The request goes to one of the rule's targets, chosen by weight. When that target fails, for example with an error such as 500, 503 or 429, or when a streamed answer does not start within the target's First token timeout, the gateway tries the fallbacks in order.
  5. The gateway records the call with the rule that routed it.

The rules to keep in mind:

  • Gateway filters only. A rule nobody carries is never evaluated. A disabled rule, or a rule carried only by disabled filters, is not applied.
  • Several filters. A user in several gateway filters gets every rule those filters carry, evaluated together by priority. A rule matches only users of the filters that carry it. A caller cannot claim another filter's rules: the gateway decides which filters apply.
  • First match wins. Equal priorities are evaluated oldest rule first. Only a rule with Chain Rule on lets the rules be evaluated again, see Targets.
  • Filters must grant the targets. Every filter that carries a rule must grant every target and fallback of that rule, see The grants a rule needs.
  • Guardrails win. When a guardrail sends a request to another model, routing rules and their fallbacks do not apply to that request.
  • About a minute. Changes to rules and filters reach the AI Gateways within about a minute. A gateway that restarts keeps applying the rules.
  • No conditions, every request. A rule without conditions applies to every request of the filter's users.

The Routing Rules list

The Governance page under Edge on the 3 Routing Rules tab, with the Add Routing Rule button and three rules: EU data stays in the EU, priority 0, condition Header x-data-region = eu, target gpt-5.1 on Azure OpenAI (EU), 1 fallback, routed 10, used by 2 Filters; Cheaper model for simple chats, priority 10, conditions Model = gpt-5.1 AND Request Type in Chat Completion, targets gpt-5.1-mini on OpenAI at 80 % and on Self-hosted vLLM at 20 %, 1 fallback, routed 40, used by 1 Filter; Claude Code on Haiku, priority 20, conditions Model starts with claude- AND Header user-agent contains, target claude-haiku-5-5 on Anthropic, routed 6, used by 1 Filter

ColumnWhat it shows
Priority · NameThe rule's priority, name and description. Lower numbers are evaluated first.
ActiveWhether the rule is enabled. Turn the switch off to stop a rule without deleting it.
ConditionsThe conditions in short form, for example Header x-data-region = eu, or All requests for a rule without conditions.
TargetsThe target models with their providers and weights, and the number of fallbacks.
Routed (7 days)How many requests the rule routed in the last seven days.
Used ByThe filters that carry the rule, or No filters.

Search with Search by name.... The edit button on a row opens the rule; the delete button deletes it, see Delete a routing rule.

Add a routing rule

Click Add Routing Rule. The dialog Create Routing Rule opens with the tabs Conditions, Targets, Fallbacks and Name. Continue walks through them in that order; on the Name tab, click Add Routing Rule to save. When you edit a rule, the button is Save Changes.

Conditions

Conditions decide which requests the rule applies to. Choose Builder to put them together from fields, or CEL to write the expression yourself. The Enable Rule switch at the bottom turns the rule on or off; matching requests are routed only while it is on.

The Edit Routing Rule dialog on the Conditions tab in Builder mode: an AND group with the conditions Model equals gpt-5.1 and Request Type is in list Chat Completion and Chat Completion (Streaming), the Add Condition and Add Group buttons, the generated CEL Expression model == 'gpt-5.1' && request_type in ['chat_completion', 'chat_completion_stream'] below, and Enable Rule on

In the builder, a group joins its conditions with AND (all must match) or OR (any may match). Add Condition adds a condition to the group, Add Group adds a nested group. Groups nest at most 3 levels deep, and a rule has at most 50 conditions. The CEL the builder produces is shown under CEL Expression.

FieldOperatorsValue
Modelequals, does not equal, is in list, is not in list, contains, begins with, ends with, matches (regex)A model name, typed or picked from your providers' catalogues, for example gpt-5.1.
Providerequals, does not equal, is in list, is not in list, matches (regex), does not exist, existsOne or more of your providers, picked by name.
Request Typeequals, does not equal, is in list, is not in list, matches (regex)The kind of call, for example Chat Completion, Responses, Text Completion, Embeddings, Image Generation, Speech, Transcription, Count Tokens, Rerank or Video Generation. A streamed call is its own value, for example Chat Completion (Streaming).
Headerequals, does not equal, contains, begins with, ends with, matches (regex), does not exist, existsA header name and a value, for example x-data-region and eu.
Query Parameterthe same as HeaderA query parameter name and a value.
  • Header and parameter names use lowercase letters, digits, dots, dashes and underscores. Names starting with x-netzilo- or x-bf- are reserved.
  • Values are case-sensitive: eu does not match EU.
  • Regular expressions do not support lookarounds or backreferences.
  • A list holds at most 100 values.

The dialog marks an incomplete or invalid condition where it is and does not save, or switch to CEL, until you fix it.

CEL

In CEL mode you write the condition as a CEL expression, for example to match Claude Code by its user agent:

The Conditions tab in CEL mode with the expression model.startsWith('claude-') && headers['user-agent'].contains('claude-cli') and Enable Rule on

The expression can use these variables:

VariableWhat it holds
modelThe model the request asks for.
providerThe provider the request asks for.
request_typeThe kind of call, for example chat_completion or chat_completion_stream.
headersThe request headers, by lowercase name.
paramsThe query parameters.

Read a header by a fixed name, such as headers["x-team"] == "data" or "x-team" in headers. An expression is at most 4096 characters. The server refuses:

  • headers whose names start with x-netzilo- or x-bf-;
  • complexity_tier: complexity-based routing is not available yet;
  • identity variables such as user_id and those starting with team_, virtual_key_ or customer_: the filters that carry a rule decide whom it applies to;
  • usage percentages such as budget_used, tokens_used and request.

Switching from CEL back to Builder replaces the expression with the builder's conditions; the dialog asks first.

Targets

Targets are where matching requests go. A rule has 1 to 10 targets, and requests are spread over them by weight.

The Targets tab headed 2 Targets, with gpt-5.1-mini on OpenAI at weight 80 % and gpt-5.1-mini on Self-hosted vLLM at weight 20 %, First token timeout Off for both, Total 100 %, the Browse Models button, and the Chain Rule switch off

Click Browse Models and pick one or more models. The picker lists the enabled models of your enabled providers, and a Requested model row for each provider: the model the request names, sent to that provider. Use Requested model to move traffic to another provider without changing the model, for example to an EU deployment that serves the same models.

The model picker with a Search providers and models field and an Apply (0) button, listing for each provider a Requested model row, The model the request names, sent to this provider, followed by its models: Anthropic with claude-sonnet-5-5 and claude-haiku-5-5, Azure OpenAI (EU) with gpt-5.1 and gpt-5.1-mini, OpenAI with gpt-5.1, and Self-hosted vLLM with gpt-4.1

ColumnWhat to enter
ModelThe target model and its provider.
WeightThe share of the traffic, as a whole percent. The weights of all targets add up to 100 %. Targets you add are split evenly; adjust them as you need.
First token timeoutFor streamed requests, in milliseconds, at most 300000. When no first token arrives in this time, the request moves on to the fallbacks. Empty or 0 is Off.

Chain Rule: when it is on and this rule matches, the rules are evaluated again with the routed provider and model, so a second rule can refine the result. When it is off, this rule's routing is final. A rule with Chain Rule on is marked in the list.

Fallbacks

Fallbacks are tried in order when the targets fail, for example with an error such as 500, 503 or 429, or a first token timeout. A rule has 0 to 10 fallbacks.

The Fallbacks tab headed 1 Fallback, Tried in order when the targets fail, listing gpt-5.1 on OpenAI with up and down arrows and a remove button, and the Browse Models button

Click Browse Models to add fallbacks, and use the arrows to put them in order. A fallback names a model: Requested model cannot be a fallback.

Name

The Name tab with Name of the Routing Rule Cheaper model for simple chats, an empty Description, and Priority 10

FieldWhat to enter
Name of the Routing RuleRequired. Name the rule so you can find it in filters, for example Keep EU requests in Azure EU.
DescriptionOptional.
PriorityRequired, from 0 to 9999. Rules are evaluated from the lowest number up; the first match routes the request. Leave gaps, such as 0, 10, 20, so that you can add rules in between later.

Click Add Routing Rule. Creating, updating and deleting a rule are recorded in Activity > Events.

Apply the rule to a filter

  1. Go to Edge > Governance > Filters and open the gateway filter for the users whose requests you want to route.
  2. On the Routing tab, click Browse Routing Rules and pick the rules, or Add Routing Rule to create one from there.
  3. Save the filter.

See Filters.

The grants a rule needs

A filter's users may only be routed to models the filter grants them. So the server checks, when you save a rule or a filter, that every filter carrying the rule grants every target and every fallback of the rule on its Models tab:

  • A target or fallback with a model needs that model granted from that provider, or All models of the provider.
  • A Requested model target needs All models of its provider, since the caller can ask for any model.

If a grant is missing, the save is refused with a message naming the filter, the model and the rule. The same check stops you from removing a grant from a filter while one of its rules needs it. Targets and fallbacks must also be enabled models of enabled providers.

What a rule in use protects

While a rule uses a model or a provider, the server refuses changes that would break it:

  • removing the model from its provider's catalogue, or disabling the model;
  • deleting the provider;
  • deleting the rule while a filter carries it (the delete button is disabled; remove the rule from those filters first).

Disabling a whole provider is allowed, as an off-switch in an incident. The rules that use it then stop applying within about a minute: matching requests go to the model the caller asked for, and nothing fails. Enable the provider again and the rules apply again.

See what a rule routed

  • Routed (7 days) on the list counts the requests each rule routed in the last seven days.
  • In an AI session, a model call's Source tab shows Model (the model that answered), Asked for (the model the caller named) and Routed by (the rule's name, or A deleted routing rule), next to Provider. See AI Sessions.
  • Activity > Events records Routing rule created, Routing rule updated and Routing rule deleted, with the administrator who made the change.

Examples

The three rules in the screenshots above, carried by the same gateway filter:

EU data stays in the EU (priority 0). Requests whose callers mark them with the header x-data-region: eu go to gpt-5.1 on the Azure OpenAI provider deployed in the EU, with one fallback. The filter grants gpt-5.1 of that provider and the fallback model.

TabSetting
ConditionsHeader x-data-region equals eu
Targetsgpt-5.1, Azure OpenAI (EU), weight 100 %
Fallbacksone model, granted by the filter

Cheaper model for simple chats (priority 10). Chat calls that ask for gpt-5.1 go to gpt-5.1-mini: 80 % to OpenAI and 20 % to a self-hosted vLLM server, with gpt-5.1 on OpenAI as the fallback when both fail.

TabSetting
ConditionsModel equals gpt-5.1 AND Request Type is in list Chat Completion, Chat Completion (Streaming)
Targetsgpt-5.1-mini, OpenAI, 80 %; gpt-5.1-mini, Self-hosted vLLM, 20 %
Fallbacksgpt-5.1, OpenAI

Claude Code on Haiku (priority 20). Claude Code sessions run on claude-haiku-5-5 whatever Claude model they ask for. Claude Code names itself claude-cli in its user agent.

TabSetting
ConditionsModel begins with claude- AND Header user-agent contains claude-cli, or in CEL: model.startsWith("claude-") && headers["user-agent"].contains("claude-cli")
Targetsclaude-haiku-5-5, Anthropic, 100 %

A request for gpt-5.1 with x-data-region: eu matches the first rule and goes to the EU deployment; the second rule is never reached for it.

To send traffic to a model in your own network, add a provider for your vLLM, Ollama or SGLang server under Providers and pick its models as targets.

Use the API

Routing rules are part of the public API. Use an administrator's personal access token.

Method and pathWhat it does
GET /api/edge/routing-rulesLists the rules.
POST /api/edge/routing-rulesCreates a rule.
GET /api/edge/routing-rules/{ruleId}Reads a rule.
PUT /api/edge/routing-rules/{ruleId}Replaces a rule.
DELETE /api/edge/routing-rules/{ruleId}Deletes a rule that no filter carries.
GET /api/edge/routing-rules/stats?days=7Requests routed per rule over the last 1 to 90 days, as [{"rule_id": "…", "requests": 10}].

A provider has two identifiers in GET /api/edge/providers, and each field takes one of them:

IdentifierLooks likeUsed in
slug (the ID in the provider dialog)anthropic-productiona Provider condition's value, provider == "…" in CEL, and the model name clients send (<slug>/<model>)
iddb4scbta5gic73bpm1p0targets[].provider_id, fallbacks (<id>/<model>) and the filter's model grants

A Provider condition written with the id is refused with no provider is named "…"; a target or fallback written with the slug is refused with provider … does not exist.

Create a rule. provider_id is the provider's id from GET /api/edge/providers; a fallback is <provider id>/<model>. A model of "" in a target is Requested model. Omitting enabled creates the rule disabled.

curl -X POST -H "Authorization: Token $NZ_TOKEN" -H 'Content-Type: application/json' \
  https://<api-host>/api/edge/routing-rules -d '{
    "name": "EU data stays in the EU",
    "enabled": true,
    "priority": 0,
    "chain_rule": false,
    "condition_mode": "builder",
    "condition": {"id": "root", "combinator": "and", "rules": [
      {"id": "c1", "field": "headers", "operator": "=", "key": "x-data-region", "value": "eu"}]},
    "targets": [{"provider_id": "<azure-eu-provider-id>", "model": "gpt-5.1", "weight": 100, "ttft_timeout_ms": 4000}],
    "fallbacks": ["<openai-provider-id>/gpt-5.1"]
  }'

For a CEL rule, send "condition_mode": "cel" and the expression in cel_expression instead of condition.

Attach rules to a gateway filter with its routing_rules field, a list of rule ids, in PUT /api/edge/filters/{filterId}. Leaving the field out keeps the filter's rules.

{ "routing_rules": ["<rule-id>"] }

Send the filter's other fields as you read them from GET /api/edge/filters/{filterId}.

StatusMeaning
422The rule is invalid: a missing or disabled target, weights that do not add up to 100, a refused CEL variable, or a filter that does not grant a target or fallback. The message says which.
412The change would break a rule in use: deleting a rule a filter carries (the message names the filters), or removing or disabling a model, or deleting a provider, that a rule uses (the message names the rule). Also returned when the account already has 200 rules.

Delete a routing rule

A rule cannot be deleted while a filter carries it: the delete button is disabled and its tooltip says how many filters use it. Remove the rule from those filters first. Once deleted, the requests it routed go to the model they ask for. This cannot be undone.

Troubleshooting

To check a rule, send a test request as a user of the filter, matching the rule's conditions, then open the session under AI Sessions and look at the model call's Source tab: Routed by names the rule, Asked for the model the request named, and Model and Provider where it went. No Routed by means no rule routed the call.

SymptomCause and fix
The rule never routes anything; Used By says No filtersNo gateway filter carries it. Add it on the filter's Routing tab.
The rule never routes for some usersThose users are in none of the groups of the filters that carry the rule. Check the filters' Targets.
The rule stopped routingThe rule or its filter was disabled (the Active switch), or a provider it uses was disabled. Matching requests then go to the model the caller asked for. Turn them on again.
Another rule routed the requestA rule with a lower Priority number matched first. Give the rule a lower number, or narrow the other rule's conditions.
The rule matches nothingA condition does not fit the requests: header values are case-sensitive, the request type of a streamed call is the (Streaming) value, and the model name is the one the caller sends. Compare with Asked for in the session.
The change has no effect yetChanges reach the gateways within about a minute. Send the request again after a minute.
The rule or filter does not save, naming a filter and a modelThat filter does not grant a target or fallback of the rule. Grant the model, or All models of the provider for a Requested model target, on the filter's Models tab.
A model cannot be removed or disabled, or a provider cannot be deletedA routing rule uses it. Change or delete the rule first.
Requests fail instead of falling backThe rule has no fallbacks, or the fallbacks failed too. Add fallbacks on another provider. A request a guardrail or a missing grant refuses is never retried on a fallback.

Not available yet

  • Complexity-based routing, which sends a request to a model by how hard the request is.
  • Adaptive routing, which shifts traffic by the providers' observed performance.
  • Filters - carry routing rules to the users of a group
  • Providers - the providers and models rules route to
  • Limits - cap spend, tokens and requests per user
  • Guardrails - guardrail routes take precedence over routing rules
  • AI Gateway - the gateway that applies the rules
  • AI Sessions - see which rule routed a call
  • Events - routing rule changes in the activity log