owi-litellm-tool/ARCHITECTURE.md

# LiteLLM Dashboard Tool — Architecture & Design

**Created:** 2026-06-04
**Updated:** 2026-06-13
**Version:** 1.2.0
**Authors:** Kagamine Rin (Chief Engineer) & Ryan Neef (General / Co-Processor)

## Product Definition

A single Open WebUI **Tool** that provides LiteLLM proxy health status, cost breakdown, and recent request history. Designed for the Network but built for general use — any LiteLLM proxy instance.

## OWI Integration Model

**Type:** Open WebUI Tool (not a Pipeline, not a Filter)

**How it works:**
1. Tool is added to the OWI Workspace
2. Tool is activated on a per-model or per-chat basis
3. **Valves** pass the LiteLLM proxy address and master key
4. No hardcoded connection details — the tool is portable

**Why a Tool, not a Pipeline/Filter:**
- Pipelines run on every request (wrong scope — this is on-demand, not per-request)
- Filters modify request/response flow (wrong scope — this is informational)
- Tools are invoked when needed, by the model, in context. This is the right pattern for "check cost" and "check health" — you ask, it answers.

## Valve Configuration

The tool needs two valves:

| Valve | Type | Purpose | Default |
|-------|------|---------|---------|
| `litellm_url` | string | LiteLLM proxy base URL (e.g., `http://localhost:4000`) | `http://localhost:4000` |
| `litellm_master_key` | string | LiteLLM master API key for auth | (empty — must be configured) |

**Security consideration:** The master key will be stored in the OWI valve configuration. This is the same security model as other API keys in OWI (LiteLLM proxy key, etc.). The key is not exposed in chat context — OWI valves are injected at tool execution time, not displayed to the user.

**Docker networking note:** If OWI runs inside Docker (typical), the `litellm_url` valve must use `host.docker.internal:4000` instead of `localhost:4000` to reach the LiteLLM proxy on the host.

## Function 1: Health Check

**LiteLLM Endpoint:** `GET {litellm_url}/health`
**Auth:** `Authorization: Bearer {litellm_master_key}`

### Response Processing

The `/health` endpoint returns `healthy_endpoints` and `unhealthy_endpoints` arrays. Each endpoint object contains a massive `litellm_metadata` blob (~2KB per model) with full auth context. **This is stripped for token efficiency.**

### Stripped Fields

These fields are removed from the `/health` response because they bloat output without providing useful information:

- `litellm_metadata` — ~2KB of auth/internal context per model
- `safety_settings` — internal safety filter config
- `cache_control_injection_points` — internal routing detail
- `use_in_pass_through`, `use_litellm_proxy`, `merge_reasoning_content_in_choices` — internal flags

### Retained Fields

- `model` — the LiteLLM model identifier
- `rpm` — rate limit
- `timeout` — request timeout in seconds
- `reasoning_effort` — if present and not "none" (e.g., "low")
- `service_tier` — if present (e.g., "flex")

### Output Format

```
LiteLLM Health Check — http://host.docker.internal:4000
==================================================
✅ Authenticated with master key

Healthy: 8  |  Unhealthy: 0

--- Healthy Models (8) ---
  ✅ gemini-2.5-pro  — rpm: 60, timeout: 300.0s  (reasoning: low, tier: flex)
  ✅ gemini-2.5-flash  — rpm: 60, timeout: 300.0s
  ...
```

### Error Paths

| Condition | Response |
|-----------|----------|
| No master key configured | Warning shown, public health only |
| 401 Unauthorized | Auth failed message |
| 403 Forbidden | Permission denied message |
| Connection refused | Connection failed message |
| Timeout (>15s) | Timeout message |
| Other non-200 | Status code + first 500 chars of response |

## Function 2: Cost Breakdown

**LiteLLM Endpoint:** `GET {litellm_url}/user/daily/activity`
**Auth:** `Authorization: Bearer {litellm_master_key}`
**Params:** `start_date` (YYYY-MM-DD or natural language shortcut), `end_date` (YYYY-MM-DD or natural language shortcut), `hide_zero_spend` (bool, default False)

### Natural Language Date Resolution

The `_resolve_date()` helper converts natural language shortcuts to YYYY-MM-DD before calling the API. This makes the tool easier for SLMs to use without requiring exact date formatting.

**Supported shortcuts:**
- `today`, `yesterday`, `tomorrow`
- `N days ago` (e.g., `two days ago`)
- `monday`-`sunday` (most recent occurrence)
- `last monday`-`last sunday`
- `this week`, `last week`
- `this month`, `last month`

Already-formatted YYYY-MM-DD strings pass through unchanged. Unrecognized strings are passed to the API, which will reject them with a clear error.

### hide_zero_spend Parameter

When `hide_zero_spend=True`, models with sub-$0.001 spend are filtered from the per-model breakdown. This is useful when free-tier or subscription models (e.g., Ollama Cloud) dominate the report with high token counts but negligible cost. Hidden models are summarized as `... (N model(s) with sub-$0.001 spend hidden)`.

### Response Processing

The `/user/daily/activity` endpoint returns a rich breakdown structure:

```
results[] → per date:
  metrics: { spend, prompt_tokens, completion_tokens, total_tokens, successful_requests, failed_requests, api_requests, cache_read_input_tokens, cache_creation_input_tokens }
  breakdown:
    models: { model_name → metrics }
    model_groups: { group_name → metrics }
    providers: { provider_name → metrics }
    endpoints: { endpoint_path → metrics }
    api_keys: { key_hash → metrics }
    entities: { entity_id → metrics }
```

The tool aggregates per-model totals across all dates and formats a readable report.

### Output Format

```
LiteLLM Cost Report: 2026-06-03 to 2026-06-04
============================================================
Total Spend:     $1.65
Total Tokens:    2,011,201 (1,993,600 prompt + 17,601 completion)
Cache Hit Rate:  69.5% (1,395,954 / 2,007,561 input tokens)
Requests:        31 successful, 1 failed (32 total)

By Model:
  Model                      Spend |       Requests |    Tokens
  gemini-3.5-flash          $1.20 |  26 ok, 0 fail | 2,003,116 tokens
  gemini-3.1-flash-image    $0.45 |   5 ok, 0 fail |     8,085 tokens
  ... (3 model(s) with sub-$0.001 spend hidden)

By Date:
  2026-06-04: $1.28 | 27 ok, 0 fail | 2,004,642 tokens
  2026-06-03: $0.37 |  4 ok, 0 fail |     6,559 tokens
```

### Key Metrics Surfaced

- Total spend (USD)
- Token counts (prompt, completion, total)
- Cache hit rate (cache_read_input_tokens / total_input_tokens)
- Request counts (successful + failed)
- Per-model breakdown (sorted by spend, descending; optionally filtered by hide_zero_spend)
- Per-date breakdown
- Hidden model count summary (when hide_zero_spend is active)

### Condensed/Stripped

- `api_key_breakdown` sub-levels — too granular for general display
- `metadata: {}` — empty in current responses
- `cache_creation_input_tokens` — shown only when non-zero

### Error Paths

| Condition | Response |
|-----------|----------|
| No master key configured | Error — key required for cost data |
| 401 Unauthorized | Auth failed message |
| 403 Forbidden | Permission denied message |
| Connection refused | Connection failed message |
| Timeout (>30s) | Timeout message (large date ranges may be slow) |
| No results | "No cost data found for [range]" |

## Function 3: Request History

**LiteLLM Endpoint:** `GET {litellm_url}/spend/logs`
**Auth:** `Authorization: Bearer {litellm_master_key}`
**Params:** `limit` (int, 1-50, default 20), `show_cost` (bool, default true), `show_failed` (bool, default true)

### Response Processing

The `/spend/logs` endpoint returns **all** historical request logs (no server-side pagination). The tool sorts by `startTime` descending and takes the most recent `limit` entries client-side.

**Performance note:** For large deployments (thousands of logs), this endpoint can be slow. The tool caps `limit` at 50 and uses a 30s timeout.

**Filtering:** When `show_failed=False`, requests with `metadata.exception` or `metadata.status="failure"` are filtered out before applying the `limit`. This hides auth probe noise, routing failures, and other non-essential errors. The header shows "(failed requests hidden)" when active.

### Per-Request Fields

| Field | Description |
|-------|-------------|
| `request_id` | Unique request identifier |
| `model` | Model used (provider prefix stripped for readability) |
| `call_type` | Request type: `acompletion` → "chat", `aembedding` → "embed", `aimage_generation` → "image" |
| `status` | ✅ OK if no exception, ❌ FAIL if exception present |
| `startTime` | Request start timestamp |
| `request_duration_ms` | Total duration |
| `prompt_tokens` | Input tokens |
| `completion_tokens` | Output tokens |
| `cache_read_input_tokens` | Cache hit count |
| `spend` | Cost in USD |
| `metadata.exception` | Error message if failed |

### Output Format

```
LiteLLM Request History — http://host.docker.internal:4000
Showing 20 of 4560 total requests (most recent first)
==========================================================================================
   1. 06-05T01:15:27 | gemini-3.1-flash-image-previ | image      | ✅ OK |   15.0s | 96→1.6k | $0.0947
   2. 06-05T01:12:27 | gemini-3.1-flash-image-previ | image      | ✅ OK |   13.3s | 101→1.5k | $0.0922
   3. 06-05T01:11:35 |                              |            | ✅ OK |     0ms
   4. 06-05T01:07:36 | gemini-3.5-flash             | chat       | ✅ OK |    8.8s | 149.8k→1.1k | $0.1821
   ...

  Summary (20 shown): $3.1493 spent | 2,798,587 prompt + 5,815 completion tokens | 0 failed
```

### Token Display Rules

- Prompt tokens ≥ 1000: shown as "X.Xk" (e.g., 149.8k)
- Completion tokens ≥ 1000: shown as "X.Xk"
- Cache hit % shown as 💾 indicator when `cache_read_input_tokens > 0`
- Zero-token entries (health checks, failed routing) shown without token info

### Error Paths

| Condition | Response |
|-----------|----------|
| No master key configured | Error — key required |
| 401 Unauthorized | Auth failed message |
| 403 Forbidden | Permission denied message |
| Connection refused | Connection failed message |
| Timeout (>30s) | Timeout message |
| No logs | "No request history found" |

## Additional Endpoints (Future Consideration)

| Endpoint | Purpose | Priority |
|----------|----------|----------|
| `GET /global/spend/keys` | Per-key lifetime spend | Nice-to-have |
| `GET /spend/logs?start_date=&end_date=` | Individual transaction logs | Low (returned empty in testing) |
| `GET /health/live` | Simple `{"status":"healthy"}` liveness check | Low (less useful than `/health`) |

These are documented but not in scope for v1. The two-function design is the right scope.

## Changelog

### v1.2.0 (2026-06-13)
- **feat:** Natural language date shortcuts for `litellm_cost()` — `today`, `yesterday`, `monday`, `last week`, etc.
- **feat:** `hide_zero_spend` parameter for `litellm_cost()` — filters sub-$0.001 models from breakdown
- **fix:** Cost report header shows resolved YYYY-MM-DD dates instead of raw shortcuts

### v1.1.0 (2026-06-13)
- Initial public release
- Three functions: `litellm_health`, `litellm_cost`, `litellm_history`
- Valve-based configuration (no hardcoded keys)
- Token-efficient response formatting

## Design Principles

1. **Three functions, that's it.** No scope creep. Health check, cost breakdown, request history.
2. **Portable.** No hardcoded URLs or keys. Valves pass everything.
3. **Token-efficient.** Strip litellm_metadata from health. Format cost data as readable text, not raw JSON dumps.
4. **General-purpose.** Designed for the Network but usable by anyone with a LiteLLM proxy.
5. **Read-only.** These endpoints are GET only. No mutations, no risk.