Improve backend
This commit is contained in:
@@ -748,4 +748,159 @@ Bakery-IA Orchestrator provides:
|
||||
|
||||
---
|
||||
|
||||
## 🆕 Forecast Validation Integration (NEW)
|
||||
|
||||
### Overview
|
||||
The orchestrator now integrates with the Forecasting Service's validation system to automatically validate forecast accuracy and trigger model improvements.
|
||||
|
||||
### Daily Workflow Integration
|
||||
|
||||
The daily workflow now includes a **Step 5: Validate Previous Forecasts** after generating new forecasts:
|
||||
|
||||
```python
|
||||
# Step 5: Validate previous day's forecasts
|
||||
await log_step(execution_id, "validate_forecasts", tenant.id, "Validating forecasts")
|
||||
validation_result = await forecast_client.validate_forecasts(
|
||||
tenant_id=tenant.id,
|
||||
orchestration_run_id=execution_id
|
||||
)
|
||||
await log_step(
|
||||
execution_id,
|
||||
"validate_forecasts",
|
||||
tenant.id,
|
||||
f"Validation complete: MAPE={validation_result.get('overall_mape', 'N/A')}%"
|
||||
)
|
||||
execution.steps_completed += 1
|
||||
```
|
||||
|
||||
### What Gets Validated
|
||||
|
||||
Every morning at 8:00 AM, the orchestrator:
|
||||
|
||||
1. **Generates today's forecasts** (Steps 1-4)
|
||||
2. **Validates yesterday's forecasts** (Step 5) by:
|
||||
- Fetching yesterday's forecast predictions
|
||||
- Fetching yesterday's actual sales from Sales Service
|
||||
- Calculating accuracy metrics (MAE, MAPE, RMSE, R², Accuracy %)
|
||||
- Storing validation results in `validation_runs` table
|
||||
- Identifying poor-performing products/locations
|
||||
|
||||
### Benefits
|
||||
|
||||
**For Bakery Owners:**
|
||||
- **Daily Accuracy Tracking**: See how accurate yesterday's forecast was
|
||||
- **Product-Level Insights**: Know which products have reliable forecasts
|
||||
- **Continuous Improvement**: Models automatically retrain when accuracy drops
|
||||
- **Trust & Confidence**: Validated accuracy metrics build trust in forecasts
|
||||
|
||||
**For Platform Operations:**
|
||||
- **Automated Quality Control**: No manual validation needed
|
||||
- **Early Problem Detection**: Performance degradation identified within 24 hours
|
||||
- **Model Health Monitoring**: Track accuracy trends over time
|
||||
- **Automatic Retraining**: Models improve automatically when needed
|
||||
|
||||
### Validation Metrics
|
||||
|
||||
Each validation run tracks:
|
||||
- **Overall Metrics**: MAPE, MAE, RMSE, R², Accuracy %
|
||||
- **Coverage**: % of forecasts with actual sales data
|
||||
- **Product Performance**: Top/bottom performers by MAPE
|
||||
- **Location Performance**: Accuracy by location/POS
|
||||
- **Trend Analysis**: Week-over-week accuracy changes
|
||||
|
||||
### Historical Data Handling
|
||||
|
||||
When late sales data arrives (e.g., from CSV imports or delayed POS sync):
|
||||
- **Webhook Integration**: Sales Service notifies Forecasting Service
|
||||
- **Gap Detection**: System identifies dates with forecasts but no validation
|
||||
- **Automatic Backfill**: Validates historical forecasts retroactively
|
||||
- **Complete Coverage**: Ensures 100% of forecasts eventually get validated
|
||||
|
||||
### Performance Monitoring & Retraining
|
||||
|
||||
**Weekly Evaluation** (runs Sunday night):
|
||||
```python
|
||||
# Analyze 30-day performance
|
||||
await retraining_service.evaluate_and_trigger_retraining(
|
||||
tenant_id=tenant.id,
|
||||
auto_trigger=True # Automatically retrain poor performers
|
||||
)
|
||||
```
|
||||
|
||||
**Retraining Triggers:**
|
||||
- MAPE > 30% (critical threshold)
|
||||
- MAPE increased > 5% in 30 days
|
||||
- Model age > 30 days
|
||||
- Manual trigger via API
|
||||
|
||||
**Automatic Actions:**
|
||||
- Identifies products with MAPE > 30%
|
||||
- Triggers retraining via Training Service
|
||||
- Tracks retraining job status
|
||||
- Validates improved accuracy after retraining
|
||||
|
||||
### Integration Flow
|
||||
|
||||
```
|
||||
Daily Orchestrator (8:00 AM)
|
||||
↓
|
||||
Step 1-4: Generate forecasts, production, procurement
|
||||
↓
|
||||
Step 5: Validate yesterday's forecasts
|
||||
↓
|
||||
Forecasting Service validates vs Sales Service
|
||||
↓
|
||||
Store validation results in validation_runs table
|
||||
↓
|
||||
If poor performance detected → Queue for retraining
|
||||
↓
|
||||
Weekly Retraining Job (Sunday night)
|
||||
↓
|
||||
Trigger Training Service for poor performers
|
||||
↓
|
||||
Models improve over time
|
||||
```
|
||||
|
||||
### Expected Results
|
||||
|
||||
**After 1 month:**
|
||||
- 100% validation coverage (all forecasts validated)
|
||||
- Baseline accuracy metrics established
|
||||
- Poor performers identified for retraining
|
||||
|
||||
**After 3 months:**
|
||||
- 10-15% accuracy improvement from automatic retraining
|
||||
- Reduced MAPE from 25% → 15% average
|
||||
- Better inventory decisions from trusted forecasts
|
||||
- Reduced waste from more accurate predictions
|
||||
|
||||
**After 6 months:**
|
||||
- Continuous model improvement cycle established
|
||||
- Optimal accuracy for each product category
|
||||
- Predictable performance metrics
|
||||
- Trust in forecast-driven decisions
|
||||
|
||||
### Monitoring Dashboard Additions
|
||||
|
||||
New metrics available for dashboards:
|
||||
|
||||
1. **Validation Status Card**
|
||||
- Last validation: timestamp, status
|
||||
- Overall MAPE: % with trend arrow
|
||||
- Validation coverage: %
|
||||
- Health status: healthy/warning/critical
|
||||
|
||||
2. **Accuracy Trends Graph**
|
||||
- 30-day MAPE trend line
|
||||
- Target threshold lines (20%, 30%)
|
||||
- Product performance distribution
|
||||
|
||||
3. **Retraining Activity**
|
||||
- Models retrained this week
|
||||
- Retraining success rate
|
||||
- Products pending retraining
|
||||
- Next scheduled retraining
|
||||
|
||||
---
|
||||
|
||||
**Copyright © 2025 Bakery-IA. All rights reserved.**
|
||||
|
||||
Reference in New Issue
Block a user