Skip to content

Fix S3 Folder Listing & Add LLM Usage Tracking with LiteLLM and Langfuse

zakariae yahya a demandé de fusionner feature/accounts-folders vers develop

Summary

Implements provider-specific folder listing for cloud integrations and adds comprehensive LLM usage tracking with LiteLLM client integration.

This PR fixes the HTTP 422 error in Amazon S3 folder listing and establishes a complete monitoring and usage tracking infrastructure for chat operations.

Changes Overview

1. FolderService Provider-Specific Payloads

Commit: e27f75c0 File: Integration/services/folder_service.py

Problem: Generic payload sent to all provider webhooks caused HTTP 422 errors in Amazon S3.

Solution: Implement conditional payload generation per provider:

  • Amazon S3: bucket_name + prefix parameters
  • SharePoint/OneDrive: site_url + library_name + account_id + parent_folder_id
  • Google Drive/Dropbox: account_id + parent_folder_id
  • All providers: Include credential_uid, user_uuid, extra_metadata

2. Chat Enhancement: Summary Prompt & LiteLLM Client

Commit: d6e471a4 Files: Integration/services/message_service.py, Integration/monitoring/litellm_client.py

Changes:

  • SUMMARY_PROMPT: Stricter summarization rules with clear example
  • LangChain Memory: Renamed max_tokens → max_completion_tokens
  • LiteLLMClient: New client for direct LiteLLM API interaction
    • Supports streaming and non-streaming chat completions
    • Direct endpoint calls (bypass n8n when needed)

3. User Usage Tracking & Logging

Commit: 45fe84b9
Files:

  • Integration/models/usage.py (new)
  • Integration/services/usage_service.py (new)
  • Integration/services/message_service.py (updated)

Features:

  • UserUsage Model: Track cumulative costs per user
  • UsageLog Model: Detailed logs of each LLM call
  • UsageService: Record usage and update statistics
  • MessageService Integration: Automatically records costs during message processing
    • Tracks tokens (prompt + completion)
    • Calculates costs (GPT-5: $0.005/$0.015 per 1K tokens)

4. Langfuse v3 Docker Setup & Monitoring

Commits: 12f0f12f
Files:

  • docker-compose.yml (updated)
  • Integration/monitoring/langfuse_client.py (new)
  • .env (example)

Langfuse Services:

  • langfuse-web (UI on port 3001)
  • langfuse-worker (async processing)
  • langfuse-db (PostgreSQL)
  • langfuse-clickhouse (analytics)
  • langfuse-redis (caching)
  • langfuse-minio (S3 storage)

LLM Observability:

  • Trace all chatbot-rag calls
  • Track token usage and costs
  • Monitor failures and timeouts
  • User and session-level tracing

Technical Details

Data Flow

User Message ↓ MessageService.send_message() ↓ [Validate conversation ownership] ↓ [Save user message to DB] ↓ call_n8n_chatbot() with Langfuse tracing ↓ [Record usage: tokens, cost, model, trace_id] ↓ [Save agent response to DB] ↓ Return formatted response with metadata

Cost Calculation

GPT-5 Pricing (per 1K tokens):
- Input:  $0.005/1K
- Output: $0.015/1K

Example: 1000 prompt tokens + 500 completion tokens
- Input cost:  (1000/1000) * 0.005 = $0.005
- Output cost: (500/1000) * 0.015 = $0.0075
- Total: $0.0125

---
Testing Checklist

- Amazon S3 folder listing: GET /integrations/folders/amazon_s3/{account_uid} → 200 OK
- SharePoint folder listing: GET /integrations/folders/sharepoint/{account_uid} → 200 OK
- Google Drive folder listing: GET /integrations/folders/google_drive/{account_uid} → 200 OK
- Chat message with usage tracking: POST /chat/conversations/{id}/messages records usage
- Langfuse traces: All chatbot calls appear in Langfuse UI
- Usage statistics: Dashboard shows cumulative costs per user
- Error handling: Failed calls are traced with error details

---
Environment Requirements

# Langfuse (optional, defaults to fail-silent)
LANGFUSE_ENABLED=true
LANGFUSE_HOST=http://localhost:3001
LANGFUSE_PUBLIC_KEY=pk-lf-xxx
LANGFUSE_SECRET_KEY=sk-lf-xxx

# LiteLLM (for direct API calls)
LITELLM_API_BASE=https://api.openai.com/v1
LITELLM_API_KEY=sk-xxx

# N8N Chatbot
N8N_WEBHOOK_BASE_URL=https://ai.izemx.com
N8N_CHATBOT_TIMEOUT=60.0

---
Dependencies Added

langfuse>=3.0.0      # LLM observability platform
langchain>=0.1.0     # Memory management
langchain-community>=0.0.1  # LLM integrations

---
Breaking Changes

❌ None - fully backward compatible

---
Related Issues

- Fixes: HTTP 422 errors in Amazon S3 folder listing
- Implements: Complete LLM usage tracking and monitoring
- Enables: Cost analytics and performance monitoring

---

Rapports de requête de fusion