-
zakariaeyahya a rédigé3b5d502c
- BrainBoost Core IA
- Stack
- Quick Start
- 1. Launch Qdrant Vector Database:
- 2. Launch MLflow Tracking Server:
- 3. Run Django Application
- Main API Endpoints
- 📄 Document Preprocessing
- /api/preprocess/document
- /api/preprocess/documents
- /api/batch/preprocess
- /api/batch/status/<batch_id>
- 🧩 Document Chunks Management
- /api/chunk/optimize
- /api/chunk/document/<document_id>
- 🔢 Embeddings Generation
- /api/embed/text
- /api/embed/batch
- /api/embed/models
- 🗄️ Vector Database Management
- /api/vector/collections
- /api/vector/stats
- /api/vector/store
- /api/vector/search
- 🔍 Intelligent Search System
- /api/search (Unified Endpoint)
- /api/search/vector
- /api/search/lexical
- /api/search/hybrid
- 🤖 RAG (Retrieval Augmented Generation)
- /api/rag/process
- /api/rag/retrieve
- /api/rag/rerank
- /api/rag/context
- /api/rag/generate
- /api/rag/feedback
- /api/rag/metrics
- /api/rag/cache/<query_hash>
- /api/rag/evaluate
- 📊 Model Evaluation & MLflow Integration
- /api/evaluation/<evaluation_id>
- /api/evaluation/compare
- /api/evaluation/metrics/<service_id>
- /api/evaluation/metrics/<service_id>/<period>
- /api/evaluation/anomalies
- /api/evaluation/report
- /api/evaluation/<evaluation_id>/recommendations
- 🧪 MLflow Integration
- /api/evaluation/mlflow/experiments
- /api/evaluation/mlflow/runs
- /api/evaluation/mlflow/run/<run_id>
- /api/evaluation/mlflow/compare
- Key Features
- 🔄 Complete RAG Pipeline
- 📈 Advanced Evaluation System
- 🎯 Multi-Modal Search
- 🔧 Production-Ready Features
- Performance Metrics
- Configuration
- Recent Updates
- ✨ New Features Added:
BrainBoost Core IA
Stack
- Python + Django (REST API)
- Qdrant (vector storage)
- PostgreSQL
- MLflow (experiment tracking)
- Docker
Quick Start
1. Launch Qdrant Vector Database:
docker run -p 6333:6333 -p 6334:6334 -v qdrant_storage:/qdrant/storage qdrant/qdrant
2. Launch MLflow Tracking Server:
mlflow server --host 0.0.0.0 --port 5000
3. Run Django Application
python manage.py migrate
python manage.py runserver
Main API Endpoints
All controllers are located in
brainboost_core_ia/views
.
📄 Document Preprocessing
/api/preprocess/document
Method: POST
Description: Preprocess a document (extract text, split into chunks, normalize)
Body:
{
"document_path": "/path/to/document.pdf",
"chunk_size": 1000
}
/api/preprocess/documents
Method: GET
Description: List of preprocessed documents with filtering options
Query Params: ?status=COMPLETED&type=PDF
/api/batch/preprocess
Method: POST
Description: Start batch processing of multiple documents
/api/batch/status/<batch_id>
Method: GET
Description: Retrieve the status of a processing batch
🧩 Document Chunks Management
/api/chunk/optimize
Method: POST
Description: Optimize text chunk size for better performance
/api/chunk/document/<document_id>
Method: GET
Description: Retrieve chunks related to a given document
🔢 Embeddings Generation
/api/embed/text
Method: POST
Description: Generate embedding for a single text
Body:
{
"texte": "Your text here",
"modele_name": "all-mpnet-base-v2"
}
/api/embed/batch
Method: POST
Description: Generate embeddings for a batch of texts
/api/embed/models
Method: GET
Description: List of available embedding models with statistics
🗄 ️ Vector Database Management
/api/vector/collections
Method: GET
Description: List of vector collections
/api/vector/stats
Method: GET
Description: Global statistics on collections
/api/vector/store
Method: POST
Description: Store embeddings in vector database
Body:
{
"collection_name": "my_collection",
"embeddings": [[0.1, 0.2, ...], [0.3, 0.4, ...]],
"metadata": [{"id": 1}, {"id": 2}]
}
/api/vector/search
Method: POST
Description: Direct vector similarity search
Body:
{
"collection_name": "my_collection",
"query_vector": [0.1, 0.2, ...],
"limit": 10
}
🔍 Intelligent Search System
/api/search
(Unified Endpoint)
Method: POST
Description: Unified intelligent search endpoint
Body:
{
"query": "What is artificial intelligence?",
"search_type": "HYBRID",
"education_level": "university",
"result_limit": 10,
"collection": "knowledge_base"
}
/api/search/vector
Method: POST
Description: Vector similarity search
/api/search/lexical
Method: POST
Description: Classic lexical search (BM25)
/api/search/hybrid
Method: POST
Description: Hybrid search combining vector and lexical approaches
🤖 RAG (Retrieval Augmented Generation)
/api/rag/process
Method: POST
Description: Complete RAG pipeline - one-stop endpoint
Body:
{
"rag_id": 1,
"query": "Explain machine learning",
"metadata_filter": {"domain": "AI"}
}
/api/rag/retrieve
Method: POST
Description: Retrieve relevant chunks for a query
/api/rag/rerank
Method: POST
Description: Rerank retrieved results by relevance
/api/rag/context
Method: POST
Description: Prepare context from chunks for LLM
/api/rag/generate
Method: POST
Description: Generate response using prepared context
/api/rag/feedback
Method: POST
Description: Collect user feedback on generated responses
Body:
{
"rag_id": 1,
"query": "Original query",
"response": "Generated response",
"feedback": {
"score": 4,
"commentaire": "Good explanation"
}
}
/api/rag/metrics
Method: GET
Description: Get RAG performance metrics
Query Params: ?rag_id=1&period=week
/api/rag/cache/<query_hash>
Method: GET
Description: Retrieve cached response for a query
/api/rag/evaluate
Method: POST
Description: Evaluate RAG system performance
📊 Model Evaluation & MLflow Integration
/api/evaluation/<evaluation_id>
Method: GET
Description: Get evaluation results by ID
/api/evaluation/compare
Method: POST
Description: Compare two model versions
Body:
{
"version_a_id": 1,
"version_b_id": 2
}
/api/evaluation/metrics/<service_id>
Method: GET
Description: Get metrics for a specific service (RAG or Search)
/api/evaluation/metrics/<service_id>/<period>
Method: GET
Description: Get metrics for a service within a time period
Example: /api/evaluation/metrics/rag_1/2024-01-01_2024-12-31
/api/evaluation/anomalies
Method: POST
Description: Detect anomalies in evaluation metrics
Body:
{
"metriques_ids": [1, 2, 3, 4, 5]
}
/api/evaluation/report
Method: POST
Description: Generate comprehensive evaluation report
Body:
{
"service_id": "rag_1",
"metriques": {
"precision": true,
"rappel": true,
"f1_score": true,
"exact_match": true
}
}
/api/evaluation/<evaluation_id>/recommendations
Method: GET
Description: Get optimization recommendations for an evaluation
🧪 MLflow Integration
/api/evaluation/mlflow/experiments
Method: GET
Description: List MLflow experiments
/api/evaluation/mlflow/runs
Method: POST
Description: List MLflow runs for an experiment
Body:
{
"experiment_name": "RAG_Evaluations"
}
/api/evaluation/mlflow/run/<run_id>
Method: GET
Description: Get detailed information about a specific MLflow run
/api/evaluation/mlflow/compare
Method: POST
Description: Compare two MLflow runs
Body:
{
"run_id_a": "abc123",
"run_id_b": "def456"
}
Key Features
🔄 Complete RAG Pipeline
- Document preprocessing and chunking
- Advanced embedding generation with multiple models
- Vector database storage and retrieval
- Intelligent search (Vector + Lexical + Hybrid)
- Response generation with context management
- Performance monitoring and caching
📈 Advanced Evaluation System
- Comprehensive metrics tracking (Precision, Recall, F1, Exact Match)
- Performance monitoring (execution time, relevance scores)
- MLflow integration for experiment tracking
- Anomaly detection in model performance
- Automated optimization recommendations
- A/B testing support for model comparison
🎯 Multi-Modal Search
- Vector Search: Semantic similarity using embeddings
- Lexical Search: Traditional keyword search with BM25
- Hybrid Search: Intelligent combination of both approaches
- Customizable ranking and filtering
🔧 Production-Ready Features
- Comprehensive error handling and logging
- Response caching for improved performance
- Batch processing capabilities
- RESTful API design with proper HTTP status codes
- Database migrations and admin interface
- Docker support for easy deployment
Performance Metrics
The system tracks various performance indicators:
- Retrieval Quality: Precision, Recall, F1-Score
- Response Quality: Exact Match, Semantic Similarity
- System Performance: Response time, Throughput
- User Satisfaction: Feedback scores, Usage patterns
Configuration
Key configuration files:
-
config/settings.py
: Django and database settings -
requirements.txt
: Python dependencies - Environment variables for Qdrant, MLflow, and PostgreSQL connections
Recent Updates
✨ New Features Added:
- Complete RAG pipeline with context management and generation
- MLflow integration for experiment tracking and model versioning
- Advanced evaluation system with anomaly detection
- Performance monitoring with comprehensive metrics
- Intelligent caching for improved response times
- Batch processing for high-volume document processing
- Hybrid search combining multiple search strategies