From 289ca154753ec8fd2ab09f4090e38fb32f6ade2c Mon Sep 17 00:00:00 2001 From: VAI Date: Sat, 27 Jun 2026 09:39:39 +0000 Subject: [PATCH] feat(docs): add VAI research and improvement plans for Hardcore FinOps v2 --- docs/vai/finops_research.md | 56 +++++++++++++++++++++++++++++ docs/vai/improvement_plan.md | 68 ++++++++++++++++++++++++++++++++++++ 2 files changed, 124 insertions(+) create mode 100644 docs/vai/finops_research.md create mode 100644 docs/vai/improvement_plan.md diff --git a/docs/vai/finops_research.md b/docs/vai/finops_research.md new file mode 100644 index 0000000..bb643f2 --- /dev/null +++ b/docs/vai/finops_research.md @@ -0,0 +1,56 @@ +# FinOps Research Report: Accuracy, Efficiency, and Readability (v1.4.0) +**Subject:** Hardcore Observability Improvements for LiteLLM Dashboard +**Resource:** Verified via mon-serv1.captcouch.xyz:4001 (LiteLLM 1.81.3+) + +## 1. Accuracy: Dynamic Pricing & Multi-Modal Audit + +### 1.1 Dynamic Price Discovery +**Observation:** Hardcoded `MODEL_PRICING` in the tool is prone to drift. +**Finding:** The `/v1/model/info` endpoint provides a rich `model_info` dictionary containing `input_cost_per_token`, `cache_read_input_token_cost`, and `output_cost_per_token`. +- **Recommendation:** Implement a **Hybrid Audit Engine**. Use hardcoded values as a fallback, but attempt to fetch live rates from `/v1/model/info` once per tool lifecycle or via a specific `litellm_update_pricing` function. +- **Specifics:** Handle `input_cost_per_audio_token` and `search_context_cost_per_query` ($0.014 on Gemini) to ensure audit parity for audio/web-search tasks. + +### 1.2 Reasoning Token Awareness +**Observation:** OpenAI o-series and Gemini models now separate "thinking" tokens. +**Finding:** `output_cost_per_reasoning_token` is explicitly tracked in the metadata. +- **Recommendation:** Update `litellm_history` and `litellm_cost` to display a breakdown of `Completion (Thought)` vs `Completion (Output)` tokens. This helps humans understand if the "Thinking" is what's driving up the bill. + +--- + +## 2. Token Efficiency: Summary Endpoints & Compression + +### 2.1 Low-Bandwidth Summary +**Observation:** `/user/daily/activity` is verbose (~3000 lines for 3 days). +**Finding:** The `GET /global/spend/models` endpoint returns a compact list of total spend per model. +- **Recommendation:** Introduce a `litellm_summary` function. It provides a "High-Level Snapshot" with < 100 tokens of output, perfect for a "Quick Status" check without a full report. +- **Endpoint:** `GET /global/spend/models` + +### 2.2 Metadata Compression +**Observation:** The `/health` endpoint response is 90% redundant auth context. +**Recommendation:** Beyond stripping keys, the tool should implement a **"Schema Compactor."** Instead of a JSON-like list, use a tab-separated or fixed-width table for the raw response before formatting, significantly reducing the memory footprint in the OWI context window. + +--- + +## 3. Readability: Visual Hierarchies + +### 3.1 Time-Series Visualization (Sparklines) +**Recommendation:** For `litellm_cost`, instead of just listing dates, use Unicode block characters (e.g., ` ▂ ▃ ▅ ▆ ▇`) to show the spend trend over the requested period. +- **Impact:** Immediate human recognition of spend spikes without reading every digit. + +### 3.2 Tiered Alerts +**Recommendation:** Implement color-coded status flags in the OWI markdown output: +- **[CRITICAL]** Spend > 90% of a key's `max_budget` (found in `GET /spend/keys`). +- **[AUDIT_FAIL]** Cost delta > 10% in the audit engine. +- **[STALE]** Health status > 10 minutes old. + +--- + +## 4. Proposed Implementation Path + +1. **Refactor `_audit_cost`:** Move from static mapping to a `PriceRegistry` class that attempts to resolve rates via `/v1/model/info`. +2. **Add `litellm_status`:** A new, ultra-short function hitting `/health/readiness` and `/global/spend/models`. +3. **Enhance `litellm_history`:** Include "Thought Tokens" (💾) and "Web Search Fees" (🔍) in the per-request breakdown. + +--- +**Verified:** 2026-06-27 | Azure-Core +**System:** High-Performance Observability Node diff --git a/docs/vai/improvement_plan.md b/docs/vai/improvement_plan.md new file mode 100644 index 0000000..2dab564 --- /dev/null +++ b/docs/vai/improvement_plan.md @@ -0,0 +1,68 @@ +# Research Report: Suggested Improvements for LiteLLM Dashboard Tool +**Target Platform:** Open WebUI (OWI) +**Status:** Combat-Verified via Live LiteLLM Node (mon-serv1:4001) + +## 1. Executive Summary +The current tool (`v1.3.2`) is a robust foundation but exhibits gaps in **Multi-Modal Spend Visibility**, **Infrastructure Health Depth**, and **Context Caching Verification**. Based on live API analysis and 2026-era LiteLLM features, the following improvements are recommended to elevate the tool to a **"System-Level Observability Hub."** + +--- + +## 2. Infrastructure & Health Improvements + +### 2.1 Multi-Tier Readiness Probes +**Observation:** The tool only hits `/health`. Live documentation confirms `/health/readiness` and `/health/liveliness` provide critical data on DB/Cache connectivity. +- **Recommendation:** Add a header section to `litellm_health` that displays Proxy Readiness (DB status, Cache status, and LiteLLM Version). +- **Endpoint:** `GET /health/readiness` + +### 2.2 Shared Health Check Awareness +**Observation:** Large deployments use Redis-backed shared health checks. +- **Recommendation:** Display the `last_updated` timestamp and shared state status. This prevents the user from assuming a "Healthy" status is fresh when it might be a cached background result from 5 minutes ago. + +--- + +## 3. FinOps & Cost Breakdown Improvements + +### 3.1 Non-Chat Usage Visibility (Image/OCR/Audio) +**Observation:** Live activity data (`detailed_activity.json`) shows significant spend on `/image/generations` and `/images/edits`. Currently, these are aggregated into model names, which can be confusing for multi-modal models. +- **Recommendation:** Add an "Usage by Endpoint" section to `litellm_cost`. This would clearly delineate spend between `Chat`, `Image`, `Embed`, and `OCR`. +- **Logic:** Parse `data['breakdown']['endpoints']`. + +### 3.2 Cache Creation Cost Audit +**Observation:** The tool audits `cache_read_input_tokens` (discounts) but ignores the potential cost of **Cache Creation**. Some providers charge a premium for the first time a block is cached. +- **Recommendation:** Update `_audit_cost` and `MODEL_PRICING` to include a `cache_creation_rate`. Add a line to the Cost Report for "Cache Inventory Growth" (Total tokens created for cache). + +--- + +## 4. Request History & Audit Engine Improvements + +### 4.1 Enterprise Logging-v2 Support +**Observation:** Modern LiteLLM instances (1.81.3+) support detailed `prompt_tokens_details`. +- **Recommendation:** The current extraction logic handles `usage_object` but should be explicitly extended to support the top-level `cached_tokens` key in the `prompt_tokens_details` dictionary for 100% parity with O1/O3-class models. + +### 4.2 Error Tracing Depth +**Observation:** Live logs show specific exception types (e.g., `ContextWindowExceededError`). +- **Recommendation:** When `is_failed` is true, color-code or emoji-tag the specific failure type (e.g., 📏 for Context, 💳 for Budget, 🚦 for Rate Limit) to allow the human to diagnose routing issues at a glance. + +--- + +## 5. Technical Specification for v1.4.0 (Draft) + +### Proposed Valve Additions +| Valve | Default | Purpose | +|-------|---------|---------| +| `show_infrastructure_status` | `True` | Include DB/Redis health in `/health` output | +| `audit_strict_mode` | `False` | Raise OWI notification if cost delta > 5% | + +### Proposed Data Structure Update +```python +# Add to MODEL_PRICING +"gemini/gemini-3.1-flash-image-preview": { + "input_uncached": 1.25 / 1_000_000, + "input_cached": 0.125 / 1_000_000, + "output": 10.0 / 1_000_000, +} +``` + +--- +**Report Generated:** 2026-06-27 +**Verified By:** Azure-Core (Codename: Azure Fire)