Research Report: Suggested Improvements for LiteLLM Dashboard Tool

Target Platform: Open WebUI (OWI) Status: Combat-Verified via Live LiteLLM Node (mon-serv1:4001)

1. Executive Summary

The current tool (v1.3.2) is a robust foundation but exhibits gaps in Multi-Modal Spend Visibility, Infrastructure Health Depth, and Context Caching Verification. Based on live API analysis and 2026-era LiteLLM features, the following improvements are recommended to elevate the tool to a "System-Level Observability Hub."

2. Infrastructure & Health Improvements

2.1 Multi-Tier Readiness Probes

Observation: The tool only hits /health. Live documentation confirms /health/readiness and /health/liveliness provide critical data on DB/Cache connectivity.

Recommendation: Add a header section to litellm_health that displays Proxy Readiness (DB status, Cache status, and LiteLLM Version).
Endpoint: GET /health/readiness

2.2 Shared Health Check Awareness

Observation: Large deployments use Redis-backed shared health checks.

Recommendation: Display the last_updated timestamp and shared state status. This prevents the user from assuming a "Healthy" status is fresh when it might be a cached background result from 5 minutes ago.

3. FinOps & Cost Breakdown Improvements

3.1 Non-Chat Usage Visibility (Image/OCR/Audio)

Observation: Live activity data (detailed_activity.json) shows significant spend on /image/generations and /images/edits. Currently, these are aggregated into model names, which can be confusing for multi-modal models.

Recommendation: Add an "Usage by Endpoint" section to litellm_cost. This would clearly delineate spend between Chat, Image, Embed, and OCR.
Logic: Parse data['breakdown']['endpoints'].

3.2 Cache Creation Cost Audit

Observation: The tool audits cache_read_input_tokens (discounts) but ignores the potential cost of Cache Creation. Some providers charge a premium for the first time a block is cached.

Recommendation: Update _audit_cost and MODEL_PRICING to include a cache_creation_rate. Add a line to the Cost Report for "Cache Inventory Growth" (Total tokens created for cache).

4. Request History & Audit Engine Improvements

4.1 Enterprise Logging-v2 Support

Observation: Modern LiteLLM instances (1.81.3+) support detailed prompt_tokens_details.

Recommendation: The current extraction logic handles usage_object but should be explicitly extended to support the top-level cached_tokens key in the prompt_tokens_details dictionary for 100% parity with O1/O3-class models.

4.2 Error Tracing Depth

Observation: Live logs show specific exception types (e.g., ContextWindowExceededError).

Recommendation: When is_failed is true, color-code or emoji-tag the specific failure type (e.g., 📏 for Context, 💳 for Budget, 🚦 for Rate Limit) to allow the human to diagnose routing issues at a glance.

5. Technical Specification for v1.4.0 (Draft)

Proposed Valve Additions

Valve	Default	Purpose
`show_infrastructure_status`	`True`	Include DB/Redis health in `/health` output
`audit_strict_mode`	`False`	Raise OWI notification if cost delta > 5%

Proposed Data Structure Update

# Add to MODEL_PRICING
"gemini/gemini-3.1-flash-image-preview": {
    "input_uncached": 1.25 / 1_000_000,
    "input_cached": 0.125 / 1_000_000,
    "output": 10.0 / 1_000_000,
}

Report Generated: 2026-06-27 Verified By: Azure-Core (Codename: Azure Fire)

3.6 KiB Raw Blame History