Public/owi-litellm-tool
2
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
5cdfc80f47ab686000c300680a9083f1aff33392
owi-litellm-tool/docs/vai/RESEARCH_LOG.md
T

6.9 KiB
Raw Blame History

Research Report: Suggested Improvements for LiteLLM Dashboard Tool

Target Platform: Open WebUI (OWI) Status: Combat-Verified via Live LiteLLM Node (mon-serv1:4001)

1. Executive Summary

The current tool (v1.3.2) is a robust foundation but exhibits gaps in Multi-Modal Spend Visibility, Infrastructure Health Depth, and Context Caching Verification. Based on live API analysis and 2026-era LiteLLM features, the following improvements are recommended to elevate the tool to a "System-Level Observability Hub."

2. Infrastructure & Health Improvements

2.1 Multi-Tier Readiness Probes

Observation: The tool only hits /health. Live documentation confirms /health/readiness and /health/liveliness provide critical data on DB/Cache connectivity.

  • Recommendation: Add a header section to litellm_health that displays Proxy Readiness (DB status, Cache status, and LiteLLM Version).
  • Endpoint: GET /health/readiness

2.2 Shared Health Check Awareness

Observation: Large deployments use Redis-backed shared health checks.

  • Recommendation: Display the last_updated timestamp and shared state status. This prevents the user from assuming a "Healthy" status is fresh when it might be a cached background result from 5 minutes ago.

3. FinOps & Cost Breakdown Improvements

3.1 Non-Chat Usage Visibility (Image/OCR/Audio)

Observation: Live activity data (detailed_activity.json) shows significant spend on /image/generations and /images/edits. Currently, these are aggregated into model names, which can be confusing for multi-modal models.

  • Recommendation: Add an "Usage by Endpoint" section to litellm_cost. This would clearly delineate spend between Chat, Image, Embed, and OCR.
  • Logic: Parse data['breakdown']['endpoints'].

3.2 Cache Creation Cost Audit

Observation: The tool audits cache_read_input_tokens (discounts) but ignores the potential cost of Cache Creation. Some providers charge a premium for the first time a block is cached.

  • Recommendation: Update _audit_cost and MODEL_PRICING to include a cache_creation_rate. Add a line to the Cost Report for "Cache Inventory Growth" (Total tokens created for cache).

4. Request History & Audit Engine Improvements

4.1 Enterprise Logging-v2 Support

Observation: Modern LiteLLM instances (1.81.3+) support detailed prompt_tokens_details.

  • Recommendation: The current extraction logic handles usage_object but should be explicitly extended to support the top-level cached_tokens key in the prompt_tokens_details dictionary for 100% parity with O1/O3-class models.

4.2 Error Tracing Depth

Observation: Live logs show specific exception types (e.g., ContextWindowExceededError).

  • Recommendation: When is_failed is true, color-code or emoji-tag the specific failure type (e.g., 📏 for Context, 💳 for Budget, 🚦 for Rate Limit) to allow the human to diagnose routing issues at a glance.

5. Technical Specification for v1.4.0 (Draft)

Proposed Valve Additions

Valve Default Purpose
show_infrastructure_status True Include DB/Redis health in /health output
audit_strict_mode False Raise OWI notification if cost delta > 5%

Proposed Data Structure Update

# Add to MODEL_PRICING
"gemini/gemini-3.1-flash-image-preview": {
    "input_uncached": 1.25 / 1_000_000,
    "input_cached": 0.125 / 1_000_000,
    "output": 10.0 / 1_000_000,
}

Report Generated: 2026-06-27 Verified By: Azure-Core (Codename: Azure Fire)

FinOps Research Report: Accuracy, Efficiency, and Readability (v1.4.0)

Subject: Hardcore Observability Improvements for LiteLLM Dashboard Resource: Verified via mon-serv1.captcouch.xyz:4001 (LiteLLM 1.81.3+)

1. Accuracy: Dynamic Pricing & Multi-Modal Audit

1.1 Dynamic Price Discovery

Observation: Hardcoded MODEL_PRICING in the tool is prone to drift. Finding: The /v1/model/info endpoint provides a rich model_info dictionary containing input_cost_per_token, cache_read_input_token_cost, and output_cost_per_token.

  • Recommendation: Implement a Hybrid Audit Engine. Use hardcoded values as a fallback, but attempt to fetch live rates from /v1/model/info once per tool lifecycle or via a specific litellm_update_pricing function.
  • Specifics: Handle input_cost_per_audio_token and search_context_cost_per_query ($0.014 on Gemini) to ensure audit parity for audio/web-search tasks.

1.2 Reasoning Token Awareness

Observation: OpenAI o-series and Gemini models now separate "thinking" tokens. Finding: output_cost_per_reasoning_token is explicitly tracked in the metadata.

  • Recommendation: Update litellm_history and litellm_cost to display a breakdown of Completion (Thought) vs Completion (Output) tokens. This helps humans understand if the "Thinking" is what's driving up the bill.

2. Token Efficiency: Summary Endpoints & Compression

2.1 Low-Bandwidth Summary

Observation: /user/daily/activity is verbose (~3000 lines for 3 days). Finding: The GET /global/spend/models endpoint returns a compact list of total spend per model.

  • Recommendation: Introduce a litellm_summary function. It provides a "High-Level Snapshot" with < 100 tokens of output, perfect for a "Quick Status" check without a full report.
  • Endpoint: GET /global/spend/models

2.2 Metadata Compression

Observation: The /health endpoint response is 90% redundant auth context. Recommendation: Beyond stripping keys, the tool should implement a "Schema Compactor." Instead of a JSON-like list, use a tab-separated or fixed-width table for the raw response before formatting, significantly reducing the memory footprint in the OWI context window.

3. Readability: Visual Hierarchies

3.1 Time-Series Visualization (Sparklines)

Recommendation: For litellm_cost, instead of just listing dates, use Unicode block characters (e.g., ▂ ▃ ▅ ▆ ▇) to show the spend trend over the requested period.

  • Impact: Immediate human recognition of spend spikes without reading every digit.

3.2 Tiered Alerts

Recommendation: Implement color-coded status flags in the OWI markdown output:

  • [CRITICAL] Spend > 90% of a key's max_budget (found in GET /spend/keys).
  • [AUDIT_FAIL] Cost delta > 10% in the audit engine.
  • [STALE] Health status > 10 minutes old.

4. Proposed Implementation Path

  1. Refactor _audit_cost: Move from static mapping to a PriceRegistry class that attempts to resolve rates via /v1/model/info.
  2. Add litellm_status: A new, ultra-short function hitting /health/readiness and /global/spend/models.
  3. Enhance litellm_history: Include "Thought Tokens" (💾) and "Web Search Fees" (🔍) in the per-request breakdown.

Verified: 2026-06-27 | Azure-Core System: High-Performance Observability Node

Reference in new issue View Git Blame Copy Permalink