owi-litellm-tool/ARCHITECTURE.md

LiteLLM Dashboard Tool — Architecture & Design

Created: 2026-06-04 Updated: 2026-06-13 Version: 1.2.0 Authors: Kagamine Rin (Chief Engineer) & Ryan Neef (General / Co-Processor)

Product Definition

A single Open WebUI Tool that provides LiteLLM proxy health status, cost breakdown, and recent request history. Designed for the Network but built for general use — any LiteLLM proxy instance.

OWI Integration Model

Type: Open WebUI Tool (not a Pipeline, not a Filter)

How it works:

1. Tool is added to the OWI Workspace
2. Tool is activated on a per-model or per-chat basis
3. Valves pass the LiteLLM proxy address and master key
4. No hardcoded connection details — the tool is portable

Why a Tool, not a Pipeline/Filter:

• Pipelines run on every request (wrong scope — this is on-demand, not per-request)
• Filters modify request/response flow (wrong scope — this is informational)
• Tools are invoked when needed, by the model, in context. This is the right pattern for "check cost" and "check health" — you ask, it answers.

Valve Configuration

The tool needs three valves:

Valve	Type	Purpose	Default
litellm_url	string	LiteLLM proxy base URL (e.g., http://localhost:4000)	http://localhost:4000
litellm_master_key	string	LiteLLM master API key for auth	(empty — must be configured)
default_hide_zero_spend	bool	Default behavior for zero-spend model filtering	True

Security consideration: The master key will be stored in the OWI valve configuration. This is the same security model as other API keys in OWI (LiteLLM proxy key, etc.). The key is not exposed in chat context — OWI valves are injected at tool execution time, not displayed to the user.

Docker networking note: If OWI runs inside Docker (typical), the litellm_url valve must use host.docker.internal:4000 instead of localhost:4000 to reach the LiteLLM proxy on the host.

Function 1: Health Check

LiteLLM Endpoint: GET {litellm_url}/health Auth: Authorization: Bearer {litellm_master_key}

Response Processing

The /health endpoint returns healthy_endpoints and unhealthy_endpoints arrays. Each endpoint object contains a massive litellm_metadata blob (~2KB per model) with full auth context. This is stripped for token efficiency.

Stripped Fields

These fields are removed from the /health response because they bloat output without providing useful information:

• litellm_metadata — ~2KB of auth/internal context per model
• safety_settings — internal safety filter config
• cache_control_injection_points — internal routing detail
• use_in_pass_through, use_litellm_proxy, merge_reasoning_content_in_choices — internal flags

Retained Fields

• model — the LiteLLM model identifier
• rpm — rate limit
• timeout — request timeout in seconds
• reasoning_effort — if present and not "none" (e.g., "low")
• service_tier — if present (e.g., "flex")

Output Format

LiteLLM Health Check — http://host.docker.internal:4000
==================================================
✅ Authenticated with master key

Healthy: 8  |  Unhealthy: 0

--- Healthy Models (8) ---
  ✅ gemini-2.5-pro  — rpm: 60, timeout: 300.0s  (reasoning: low, tier: flex)
  ✅ gemini-2.5-flash  — rpm: 60, timeout: 300.0s
  ...

Error Paths

Condition	Response
No master key configured	Warning shown, public health only
401 Unauthorized	Auth failed message
403 Forbidden	Permission denied message
Connection refused	Connection failed message
Timeout (>15s)	Timeout message
Other non-200	Status code + first 500 chars of response

Function 2: Cost Breakdown

LiteLLM Endpoint: GET {litellm_url}/user/daily/activity Auth: Authorization: Bearer {litellm_master_key} Params: start_date (YYYY-MM-DD or natural language shortcut), end_date (YYYY-MM-DD or natural language shortcut), hide_zero_spend (bool, default False)

Two-Level Zero-Spend Filtering

The tool uses a two-level design for zero-spend model filtering:

1. Valve-level default (default_hide_zero_spend, default True): The human operator sets the default behavior via valves. When True, models with sub-$0.001 spend are hidden by default.

2. Per-call override (hide_zero_spend parameter): The SLM can explicitly override the valve default on a per-call basis:

   • hide_zero_spend=True → Always hide (explicit override, regardless of valve)
   • hide_zero_spend=False → Use valve default (defers to default_hide_zero_spend)

This design is defensive against SLM behavior: if the SLM passes hide_zero_spend=False (or omits it), the valve default still applies. Only an explicit hide_zero_spend=True forces hiding. The SLM cannot accidentally expose zero-spend models unless the human has set default_hide_zero_spend=False.

Natural Language Date Resolution

The _resolve_date() helper converts natural language shortcuts to YYYY-MM-DD before calling the API. This makes the tool easier for SLMs to use without requiring exact date formatting.

Supported shortcuts:

• today, yesterday, tomorrow
• N days ago (e.g., two days ago)
• monday-sunday (most recent occurrence)
• last monday-last sunday
• this week, last week
• this month, last month

Already-formatted YYYY-MM-DD strings pass through unchanged. Unrecognized strings are passed to the API, which will reject them with a clear error.

Response Processing

The /user/daily/activity endpoint returns a rich breakdown structure:

results[] → per date:
  metrics: { spend, prompt_tokens, completion_tokens, total_tokens, successful_requests, failed_requests, api_requests, cache_read_input_tokens, cache_creation_input_tokens }
  breakdown:
    models: { model_name → metrics }
    model_groups: { group_name → metrics }
    providers: { provider_name → metrics }
    endpoints: { endpoint_path → metrics }
    api_keys: { key_hash → metrics }
    entities: { entity_id → metrics }

The tool aggregates per-model totals across all dates and formats a readable report.

Output Format

LiteLLM Cost Report: 2026-06-03 to 2026-06-04
============================================================
Total Spend:     $1.65
Total Tokens:    2,011,201 (1,993,600 prompt + 17,601 completion)
Cache Hit Rate:  69.5% (1,395,954 / 2,007,561 input tokens)
Requests:        31 successful, 1 failed (32 total)

By Model:
  Model                      Spend |       Requests |    Tokens
  gemini-3.5-flash          $1.20 |  26 ok, 0 fail | 2,003,116 tokens
  gemini-3.1-flash-image    $0.45 |   5 ok, 0 fail |     8,085 tokens
  ... (3 model(s) with sub-$0.001 spend hidden)

By Date:
  2026-06-04: $1.28 | 27 ok, 0 fail | 2,004,642 tokens
  2026-06-03: $0.37 |  4 ok, 0 fail |     6,559 tokens

Key Metrics Surfaced

• Total spend (USD)
• Token counts (prompt, completion, total)
• Cache hit rate (cache_read_input_tokens / total_input_tokens)
• Request counts (successful + failed)
• Per-model breakdown (sorted by spend, descending; optionally filtered by zero-spend hiding)
• Per-date breakdown
• Hidden model count summary (when zero-spend hiding is active)

Error Paths

Condition	Response
No master key configured	Error — key required for cost data
401 Unauthorized	Auth failed message
403 Forbidden	Permission denied message
Connection refused	Connection failed message
Timeout (>30s)	Timeout message (large date ranges may be slow)
No results	"No cost data found for [range]"

Function 3: Request History

LiteLLM Endpoint: GET {litellm_url}/spend/logs Auth: Authorization: Bearer {litellm_master_key} Params: limit (int, 1-50, default 20), show_cost (bool, default true), show_failed (bool, default true)

Response Processing

The /spend/logs endpoint returns all historical request logs (no server-side pagination). The tool sorts by startTime descending and takes the most recent limit entries client-side.

Performance note: For large deployments (thousands of logs), this endpoint can be slow. The tool caps limit at 50 and uses a 30s timeout.

Filtering: When show_failed=False, requests with metadata.exception or metadata.status="failure" are filtered out before applying the limit. This hides auth probe noise, routing failures, and other non-essential errors. The header shows "(failed requests hidden)" when active.

Per-Request Fields

Field	Description
request_id	Unique request identifier
model	Model used (provider prefix stripped for readability)
call_type	Request type: acompletion → "chat", aembedding → "embed", aimage_generation → "image"
status	✅ OK if no exception, ❌ FAIL if exception present
startTime	Request start timestamp
request_duration_ms	Total duration
prompt_tokens	Input tokens
completion_tokens	Output tokens
cache_read_input_tokens	Cache hit count
spend	Cost in USD
metadata.exception	Error message if failed

Output Format

LiteLLM Request History — http://host.docker.internal:4000
Showing 20 of 4560 total requests (most recent first)
==========================================================================================
   1. 06-05T01:15:27 | gemini-3.1-flash-image-previ | image      | ✅ OK |   15.0s | 96→1.6k | $0.0947
   2. 06-05T01:12:27 | gemini-3.1-flash-image-previ | image      | ✅ OK |   13.3s | 101→1.5k | $0.0922
   3. 06-05T01:11:35 |                              |            | ✅ OK |     0ms
   4. 06-05T01:07:36 | gemini-3.5-flash             | chat       | ✅ OK |    8.8s | 149.8k→1.1k | $0.1821
   ...

  Summary (20 shown): $3.1493 spent | 2,798,587 prompt + 5,815 completion tokens | 0 failed

Token Display Rules

• Prompt tokens ≥ 1000: shown as "X.Xk" (e.g., 149.8k)
• Completion tokens ≥ 1000: shown as "X.Xk"
• Cache hit % shown as 💾 indicator when cache_read_input_tokens > 0
• Zero-token entries (health checks, failed routing) shown without token info

Error Paths

Condition	Response
No master key configured	Error — key required
401 Unauthorized	Auth failed message
403 Forbidden	Permission denied message
Connection refused	Connection failed message
Timeout (>30s)	Timeout message
No logs	"No request history found"

Additional Endpoints (Future Consideration)

Endpoint	Purpose	Priority
GET /global/spend/keys	Per-key lifetime spend	Nice-to-have
GET /spend/logs?start_date=&end_date=	Individual transaction logs	Low (returned empty in testing)
GET /health/live	Simple {"status":"healthy"} liveness check	Low (less useful than /health)

These are documented but not in scope for v1. The two-function design is the right scope.

Cost Verification & Audit Engine (TASK-004)

Introduced in v1.3.0

To defend against model-level cost over-reporting or regression bugs (e.g., LiteLLM Issue #19681 / #11156), the tool includes an internal Hardcore FinOps Audit Engine.

How it works:

1. Pricing Registry: A static mapping of model identifiers to actual provider rates (including uncached input, cached input, and output rates).
2. Audit Helper: The _audit_cost method calculates the "True Cost" dynamically:

   uncached_tokens = max(0, prompt_tokens - cached_tokens)
   calculated_cost = (cached_tokens * input_cached_rate) + (uncached_tokens * input_uncached_rate) + (completion_tokens * output_rate)
   

3. Discrepancy Flagging: If the delta between LiteLLM's reported cost and our calculated cost is greater than half a cent ($0.005), it flags it with a warning ⚠️. Otherwise, it marks it as validated ✓.

This ensures that we have immediate, live verification of context caching discounts (like Google's 90% implicit caching discount) on a per-request basis.

Changelog

v1.3.0 (2026-06-26)

• feat: Hardcore FinOps Audit Engine (TASK-004) — added _audit_cost helper and static MODEL_PRICING registry to verify reported spend against actual provider caching rates.
• feat: Real-time request cost auditing in litellm_history output.

v1.2.0 (2026-06-13)

• feat: Natural language date shortcuts for litellm_cost() — today, yesterday, monday, last week, etc.
• feat: Two-level zero-spend filtering — valve default (default_hide_zero_spend, default True) + per-call override (hide_zero_spend)
• fix: Cost report header shows resolved YYYY-MM-DD dates instead of raw shortcuts

v1.1.0 (2026-06-13)

• Initial public release
• Three functions: litellm_health, litellm_cost, litellm_history
• Valve-based configuration (no hardcoded keys)
• Token-efficient response formatting

Design Principles

1. Three functions, that's it. No scope creep. Health check, cost breakdown, request history.
2. Portable. No hardcoded URLs or keys. Valves pass everything.
3. Token-efficient. Strip litellm_metadata from health. Format cost data as readable text, not raw JSON dumps.
4. General-purpose. Designed for the Network but usable by anyone with a LiteLLM proxy.
5. Read-only. These endpoints are GET only. No mutations, no risk.
6. Defensive against SLM behavior. Two-level parameter design ensures the human's valve defaults are respected even when the SLM passes conflicting parameters.