Shadow Mode

Compare two model configurations on production traffic side-by-side. Alert on divergence. Answers "does this new model actually change what our agent decides?"

Use cases

• Model upgrade validation: Claude 4.7 → 4.8; GPT-4o → GPT-4o-2, is behavior materially the same?
• Prompt regression tests: you changed the system prompt; is the agent regressing on 10% of production traffic?
• Fine-tuned vs base: shadow-test the base model against your fine-tune to prove the fine-tune actually helps.

Wire it up in three minutes

from etch.integrations.anthropic_client import EtchAnthropic
from etch.integrations.shadow import EtchShadowClient

primary = EtchAnthropic(
    api_key="sk-ant-...",
    etch_app_token="ep_live_...", etch_project_id="my-project",
)
shadow = EtchAnthropic(
    api_key="sk-ant-...",   # or a different key, or a different model
    etch_app_token="ep_live_...", etch_project_id="my-project",
)

client = EtchShadowClient(
    primary=primary, shadow=shadow,
    etch_app_token="ep_live_...", etch_project_id="my-project",
    primary_label="claude-4.7", shadow_label="claude-4.8",
    sample_rate=0.10,   # only shadow 10% of production traffic
)

# App code stays the same. client.messages.create returns the
# primary response; the shadow call happens transparently.
response = client.messages.create(
    model="claude-opus-4-7",
    max_tokens=1024,
    messages=[{"role": "user", "content": "hi"}],
)

What gets recorded per comparison

One shadow.comparison event with:

• shadow_group_id, links this comparison across the two model records
• is_divergent, overall flag
• text_diff / tools_diff / stop_diff, per-attribute breakdown
• primary and shadow sub-objects, final text hash, tool_calls_seen, stop_reason, token counts
• token_deltas.input / token_deltas.output, how much more/less the shadow costs

Divergence rules

A comparison is is_divergent if ANY of:

• final assistant text hash differs
• set of tool calls the response made differs
• stop_reason / finish_reason differs

Token count deltas are recorded but do not flip the flag they are informational for cost forecasting.

Alerting

Set ETCH_SHADOW_ALERT_THRESHOLD_PCT (e.g. 25) on your Etch operator infrastructure. When any pair's daily divergence rate exceeds it, an operator email fires. Idempotent per pair per day (won't spam).

Dashboard

The Shadow tab (/dashboard/shadow) shows divergence rate per day per pair over the last 30 days, plus the attribute breakdown. Zero traffic renders the "how to enable" snippet above so a first-time visitor sees the shape of the answer before they integrate.

Cost

Shadow mode doubles LLM cost on the sampled calls (both providers get called). Use sample_rate to scope this, e.g. shadow 5-10% of production and extrapolate the divergence rate.

Sibling pattern: pre-deploy frozen-corpus gate

This page covers production traffic mirroring, the shadow runs alongside live traffic. The complementary pattern is the pre-deploy frozen-corpus gate: mock the external APIs, run a fixed corpus, assert on the reasoning path, and gate deploy on a signed scorecard. Use the gate to certify a release. Use this page's traffic mirror to monitor after it ships.

Full runbook: docs/shadow_mode.md.