bakery-admin/bakery-ia
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

New alert service

Browse Source
This commit is contained in:
Urtzi Alfaro
2025-12-05 20:07:01 +01:00
parent 1fe3a73549
commit 667e6e0404
393 changed files with 26002 additions and 61033 deletions
Download Patch File Download Diff File Expand all files Collapse all files

314
services/orchestrator/README.md
View File

@@ -925,318 +925,4 @@ New metrics available for dashboards:
---
## Delivery Tracking Service
### Overview
The Delivery Tracking Service provides **proactive monitoring** of expected deliveries with time-based alert generation. Unlike reactive event-driven alerts, this service periodically checks delivery windows against current time to generate predictive and overdue notifications.
**Key Capabilities**:
- Proactive "arriving soon" alerts (T-2 hours before delivery)
- Overdue delivery detection (30 min after window)
- Incomplete receipt reminders (2 hours after window)
- Integration with Procurement Service for PO delivery schedules
- Automatic alert resolution when deliveries are received
### Cronjob Configuration
```yaml
# infrastructure/kubernetes/base/cronjobs/delivery-tracking-cronjob.yaml
apiVersion: batch/v1
kind: CronJob
metadata:
name: delivery-tracking-cronjob
spec:
schedule: "30 * * * *" # Hourly at minute 30
concurrencyPolicy: Forbid
successfulJobsHistoryLimit: 3
failedJobsHistoryLimit: 3
jobTemplate:
spec:
activeDeadlineSeconds: 1800 # 30 minutes timeout
template:
spec:
containers:
- name: delivery-tracking
image: orchestrator-service:latest
command: ["python3", "-m", "app.services.delivery_tracking_service"]
resources:
requests:
memory: "128Mi"
cpu: "50m"
limits:
memory: "256Mi"
cpu: "100m"
```
**Schedule Rationale**: Hourly checks provide timely alerts without excessive polling. The :30 offset avoids collision with priority recalculation cronjob (:15).
### Delivery Alert Lifecycle
```
Purchase Order Approved (t=0)
↓
System publishes DELIVERY_SCHEDULED (informational event)
↓
[Time passes - no alerts]
↓
T-2 hours before expected delivery time
↓
CronJob detects: now >= (expected_delivery - 2 hours)
↓
Generate DELIVERY_ARRIVING_SOON alert
- Priority: 70 (important)
- Class: action_needed
- Action Queue: Yes
- Smart Action: Open StockReceiptModal in create mode
↓
[Delivery window arrives]
↓
Expected delivery time + 30 minutes (grace period)
↓
CronJob detects: now >= (delivery_window_end + 30 min)
↓
Generate DELIVERY_OVERDUE alert
- Priority: 95 (critical)
- Class: critical
- Escalation: Time-sensitive
- Smart Action: Contact supplier + Open receipt modal
↓
Expected delivery time + 2 hours
↓
CronJob detects: still no stock receipt
↓
Generate STOCK_RECEIPT_INCOMPLETE alert
- Priority: 80 (important)
- Class: action_needed
- Smart Action: Open existing receipt in edit mode
```
**Auto-Resolution**: All delivery alerts are automatically resolved when:
- Stock receipt is confirmed (`onConfirm` in StockReceiptModal)
- Event `delivery.received` is published
- Alert Processor marks alerts as `resolved` with reason: "Delivery received"
### Service Methods
#### `check_expected_deliveries()` - Main Entry Point
```python
async def check_expected_deliveries(tenant_id: str) -> None:
"""
Hourly job to check all purchase orders with expected deliveries.
Queries Procurement Service for POs with:
- status: approved or sent
- expected_delivery_date: within next 48 hours or past due
For each PO, checks:
1. Arriving soon? (T-2h) → _send_arriving_soon_alert()
2. Overdue? (T+30m) → _send_overdue_alert()
3. Receipt incomplete? (T+2h) → _send_receipt_incomplete_alert()
"""
```
#### `_send_arriving_soon_alert(po: PurchaseOrder)` - Proactive Warning
```python
async def _send_arriving_soon_alert(po: PurchaseOrder) -> None:
"""
Generates alert 2 hours before expected delivery.
Alert Details:
- event_type: DELIVERY_ARRIVING_SOON
- priority_score: 70 (important)
- alert_class: action_needed
- domain: supply_chain
- smart_action: open_stock_receipt_modal (create mode)
Context Enrichment:
- PO ID, supplier name, expected items count
- Delivery window (start/end times)
- Preparation checklist (clear receiving area, verify items)
"""
```
#### `_send_overdue_alert(po: PurchaseOrder)` - Critical Escalation
```python
async def _send_overdue_alert(po: PurchaseOrder) -> None:
"""
Generates critical alert 30 minutes after delivery window.
Alert Details:
- event_type: DELIVERY_OVERDUE
- priority_score: 95 (critical)
- alert_class: critical
- domain: supply_chain
- smart_actions: [contact_supplier, open_receipt_modal]
Business Impact:
- Production delays if ingredients missing
- Spoilage risk if perishables delayed
- Customer order fulfillment risk
Suggested Actions:
1. Contact supplier immediately
2. Check for delivery rescheduling
3. Activate backup supplier if needed
4. Adjust production plan if ingredients critical
"""
```
#### `_send_receipt_incomplete_alert(po: PurchaseOrder)` - Reminder
```python
async def _send_receipt_incomplete_alert(po: PurchaseOrder) -> None:
"""
Generates reminder 2 hours after delivery window if no receipt.
Alert Details:
- event_type: STOCK_RECEIPT_INCOMPLETE
- priority_score: 80 (important)
- alert_class: action_needed
- domain: inventory
- smart_action: open_stock_receipt_modal (edit mode if draft exists)
Checks:
- Stock receipts table for PO ID
- If draft exists → Edit mode with pre-filled data
- If no draft → Create mode
HACCP Compliance Note:
- Food safety requires timely receipt documentation
- Expiration date tracking depends on receipt
- Incomplete receipts block lot tracking
"""
```
### Integration with Alert System
**Publishing Flow**:
```python
# services/orchestrator/app/services/delivery_tracking_service.py
from shared.clients.alerts_client import AlertsClient
alerts_client = AlertsClient(service_name="orchestrator")
await alerts_client.publish_alert(
tenant_id=tenant_id,
event_type="DELIVERY_OVERDUE",
entity_type="purchase_order",
entity_id=po.id,
severity="critical",
priority_score=95,
context={
"po_number": po.po_number,
"supplier_name": po.supplier.name,
"expected_delivery": po.expected_delivery_date.isoformat(),
"delay_minutes": delay_in_minutes,
"items_count": len(po.line_items)
}
)
```
**Alert Processing**:
1. Delivery Tracking Service → RabbitMQ (supply_chain.alerts exchange)
2. Alert Processor consumes message
3. Full enrichment pipeline (Tier 1 - ALERTS)
4. Smart action handler assigned (open_stock_receipt_modal)
5. Store in PostgreSQL with priority_score
6. Publish to Redis Pub/Sub → Gateway SSE
7. Frontend `useSupplyChainNotifications()` hook receives alert
8. UnifiedActionQueueCard displays in "Urgent" section
9. User clicks → StockReceiptModal opens with PO context
### Architecture Decision: Why CronJob Over Event System?
**Question**: Could we replace this cronjob with scheduled events?
**Answer**: ❌ No - CronJob is the right tool for this job.
#### Comparison Matrix
| Feature | Event System | CronJob | Best Choice |
|---------|--------------|---------|-------------|
| Time-based alerts | ❌ Requires complex scheduling | ✅ Natural fit | **CronJob** |
| Predictive alerts | ❌ Must schedule at PO creation | ✅ Dynamic checks | **CronJob** |
| Delivery window changes | ❌ Need to reschedule events | ✅ Adapts automatically | **CronJob** |
| System restarts | ❌ Lose scheduled events | ✅ Persistent schedule | **CronJob** |
| Complexity | ❌ High (event scheduler needed) | ✅ Low (periodic check) | **CronJob** |
| Maintenance | ❌ Many scheduled events | ✅ Single job | **CronJob** |
**Event System Challenges**:
- Would need to schedule 3 events per PO at approval time:
1. "arriving_soon" event at (delivery_time - 2h)
2. "overdue" event at (delivery_time + 30m)
3. "incomplete" event at (delivery_time + 2h)
- Requires persistent event scheduler (like Celery Beat)
- Rescheduling when delivery dates change is complex
- System restarts would lose in-memory scheduled events
- Essentially rebuilding cron functionality
**CronJob Advantages**:
- ✅ Simple periodic check against current time
- ✅ Adapts to delivery date changes automatically
- ✅ No state management for scheduled events
- ✅ Easy to adjust alert timing thresholds
- ✅ Built-in Kubernetes scheduling and monitoring
- ✅ Resource-efficient (runs 1 minute every hour)
**Verdict**: Periodic polling is more maintainable than scheduled events for time-based conditions.
### Monitoring & Observability
**Metrics Tracked**:
- `delivery_tracking_job_duration_seconds` - Execution time
- `delivery_alerts_generated_total{type}` - Counter by alert type
- `deliveries_checked_total` - Total POs scanned
- `delivery_tracking_errors_total` - Failure rate
**Logs**:
```
[2025-11-26 14:30:02] INFO: Delivery tracking job started for tenant abc123
[2025-11-26 14:30:03] INFO: Found 12 purchase orders with upcoming deliveries
[2025-11-26 14:30:03] INFO: Generated DELIVERY_ARRIVING_SOON for PO-2025-043 (delivery in 1h 45m)
[2025-11-26 14:30:03] WARNING: Generated DELIVERY_OVERDUE for PO-2025-041 (45 minutes late)
[2025-11-26 14:30:04] INFO: Delivery tracking job completed in 2.3s
```
**Alerting** (for Ops team):
- Job fails 3 times consecutively → Page on-call engineer
- Job duration > 5 minutes → Warning (performance degradation)
- Zero deliveries checked for 24 hours → Warning (data issue)
### Testing
**Unit Tests**:
```python
# tests/services/test_delivery_tracking_service.py
async def test_arriving_soon_alert_generated():
# Given: PO with delivery in 1 hour 55 minutes
po = create_test_po(expected_delivery=now() + timedelta(hours=1, minutes=55))
# When: Check deliveries
await delivery_tracking_service.check_expected_deliveries(tenant_id)
# Then: DELIVERY_ARRIVING_SOON alert generated
assert_alert_published("DELIVERY_ARRIVING_SOON", po.id)
```
**Integration Tests**:
- Test full flow from cronjob → alert → frontend SSE
- Verify alert auto-resolution on stock receipt confirmation
- Test grace period boundaries (exactly 30 minutes)
### Performance Characteristics
**Typical Execution**:
- Query Procurement Service: 50-100ms
- Filter POs by time windows: 5-10ms
- Generate alerts (avg 3 per run): 150-300ms
- Total: **200-400ms per tenant**
**Scaling**:
- Single-tenant deployment: Trivial (<1s per hour)
- Multi-tenant (100 tenants): ~40s per run (well under 30min timeout)
- Multi-tenant (1000+ tenants): Consider tenant sharding across multiple cronjobs
---
**Copyright © 2025 Bakery-IA. All rights reserved.**

3
services/orchestrator/app/api/__init__.py
View File

@@ -1,4 +1,3 @@
from .orchestration import router as orchestration_router
from .dashboard import router as dashboard_router
__all__ = ["orchestration_router", "dashboard_router"]
__all__ = ["orchestration_router"]

800
services/orchestrator/app/api/dashboard.py
View File

@@ -1,800 +0,0 @@
# ================================================================
# services/orchestrator/app/api/dashboard.py
# ================================================================
"""
Dashboard API endpoints for JTBD-aligned bakery dashboard
"""
from fastapi import APIRouter, Depends, HTTPException, Query
from sqlalchemy.ext.asyncio import AsyncSession
from typing import Dict, Any, List, Optional
from pydantic import BaseModel, Field
from datetime import datetime
import structlog
import asyncio
from app.core.database import get_db
from app.core.config import settings
from ..services.dashboard_service import DashboardService
from ..utils.cache import get_cached, set_cached, delete_pattern
from shared.clients import (
get_inventory_client,
get_production_client,
get_alerts_client,
ProductionServiceClient,
InventoryServiceClient,
AlertsServiceClient
)
from shared.clients.procurement_client import ProcurementServiceClient
logger = structlog.get_logger()
# Initialize service clients
inventory_client = get_inventory_client(settings, "orchestrator")
production_client = get_production_client(settings, "orchestrator")
procurement_client = ProcurementServiceClient(settings)
alerts_client = get_alerts_client(settings, "orchestrator")
router = APIRouter(prefix="/api/v1/tenants/{tenant_id}/dashboard", tags=["dashboard"])
# ============================================================
# Response Models
# ============================================================
class I18nData(BaseModel):
"""i18n translation data"""
key: str = Field(..., description="i18n translation key")
params: Optional[Dict[str, Any]] = Field(default_factory=dict, description="Parameters for translation")
class HeadlineData(BaseModel):
"""i18n-ready headline data"""
key: str = Field(..., description="i18n translation key")
params: Dict[str, Any] = Field(default_factory=dict, description="Parameters for translation")
class HealthChecklistItem(BaseModel):
"""Individual item in tri-state health checklist"""
icon: str = Field(..., description="Icon name: check, warning, alert, ai_handled")
text: Optional[str] = Field(None, description="Deprecated: Use textKey instead")
textKey: Optional[str] = Field(None, description="i18n translation key")
textParams: Optional[Dict[str, Any]] = Field(None, description="Parameters for i18n translation")
actionRequired: bool = Field(..., description="Whether action is required")
status: str = Field(..., description="Tri-state status: good, ai_handled, needs_you")
actionPath: Optional[str] = Field(None, description="Path to navigate for action")
class BakeryHealthStatusResponse(BaseModel):
"""Overall bakery health status with tri-state checklist"""
status: str = Field(..., description="Health status: green, yellow, red")
headline: HeadlineData = Field(..., description="i18n-ready status headline")
lastOrchestrationRun: Optional[str] = Field(None, description="ISO timestamp of last orchestration")
nextScheduledRun: str = Field(..., description="ISO timestamp of next scheduled run")
checklistItems: List[HealthChecklistItem] = Field(..., description="Tri-state status checklist")
criticalIssues: int = Field(..., description="Count of critical issues")
pendingActions: int = Field(..., description="Count of pending actions")
aiPreventedIssues: int = Field(0, description="Count of issues AI prevented")
class ReasoningInputs(BaseModel):
"""Inputs used by orchestrator for decision making"""
customerOrders: int = Field(..., description="Number of customer orders analyzed")
historicalDemand: bool = Field(..., description="Whether historical data was used")
inventoryLevels: bool = Field(..., description="Whether inventory levels were considered")
aiInsights: bool = Field(..., description="Whether AI insights were used")
class PurchaseOrderSummary(BaseModel):
"""Summary of a purchase order for dashboard"""
supplierName: str
itemCategories: List[str]
totalAmount: float
class ProductionBatchSummary(BaseModel):
"""Summary of a production batch for dashboard"""
productName: str
quantity: float
readyByTime: str
class OrchestrationSummaryResponse(BaseModel):
"""What the orchestrator did for the user"""
runTimestamp: Optional[str] = Field(None, description="When the orchestration ran")
runNumber: Optional[str] = Field(None, description="Run number identifier")
status: str = Field(..., description="Run status")
purchaseOrdersCreated: int = Field(..., description="Number of POs created")
purchaseOrdersSummary: List[PurchaseOrderSummary] = Field(default_factory=list)
productionBatchesCreated: int = Field(..., description="Number of batches created")
productionBatchesSummary: List[ProductionBatchSummary] = Field(default_factory=list)
reasoningInputs: ReasoningInputs
userActionsRequired: int = Field(..., description="Number of actions needing approval")
durationSeconds: Optional[int] = Field(None, description="How long orchestration took")
aiAssisted: bool = Field(False, description="Whether AI insights were used")
message_i18n: Optional[I18nData] = Field(None, description="i18n data for message")
class ActionButton(BaseModel):
"""Action button configuration"""
label_i18n: I18nData = Field(..., description="i18n data for button label")
type: str = Field(..., description="Button type: primary, secondary, tertiary")
action: str = Field(..., description="Action identifier")
class ActionItem(BaseModel):
"""Individual action requiring user attention"""
id: str
type: str = Field(..., description="Action type")
urgency: str = Field(..., description="Urgency: critical, important, normal")
title: Optional[str] = Field(None, description="Legacy field for alerts")
title_i18n: Optional[I18nData] = Field(None, description="i18n data for title")
subtitle: Optional[str] = Field(None, description="Legacy field for alerts")
subtitle_i18n: Optional[I18nData] = Field(None, description="i18n data for subtitle")
reasoning: Optional[str] = Field(None, description="Legacy field for alerts")
reasoning_i18n: Optional[I18nData] = Field(None, description="i18n data for reasoning")
consequence_i18n: I18nData = Field(..., description="i18n data for consequence")
reasoning_data: Optional[Dict[str, Any]] = Field(None, description="Structured reasoning data")
amount: Optional[float] = Field(None, description="Amount for financial actions")
currency: Optional[str] = Field(None, description="Currency code")
actions: List[ActionButton]
estimatedTimeMinutes: int
class ActionQueueResponse(BaseModel):
"""Prioritized queue of actions"""
actions: List[ActionItem]
totalActions: int
criticalCount: int
importantCount: int
class ProductionTimelineItem(BaseModel):
"""Individual production batch in timeline"""
id: str
batchNumber: str
productName: str
quantity: float
unit: str
plannedStartTime: Optional[str]
plannedEndTime: Optional[str]
actualStartTime: Optional[str]
status: str
statusIcon: str
statusText: str
progress: int = Field(..., ge=0, le=100, description="Progress percentage")
readyBy: Optional[str]
priority: str
reasoning_data: Optional[Dict[str, Any]] = Field(None, description="Structured reasoning data")
reasoning_i18n: Optional[I18nData] = Field(None, description="i18n data for reasoning")
status_i18n: Optional[I18nData] = Field(None, description="i18n data for status")
class ProductionTimelineResponse(BaseModel):
"""Today's production timeline"""
timeline: List[ProductionTimelineItem]
totalBatches: int
completedBatches: int
inProgressBatches: int
pendingBatches: int
class InsightCardI18n(BaseModel):
"""i18n data for insight card"""
label: I18nData = Field(..., description="i18n data for label")
value: I18nData = Field(..., description="i18n data for value")
detail: Optional[I18nData] = Field(None, description="i18n data for detail")
class InsightCard(BaseModel):
"""Individual insight card"""
color: str = Field(..., description="Color: green, amber, red")
i18n: InsightCardI18n = Field(..., description="i18n translation data")
class InsightsResponse(BaseModel):
"""Key insights grid"""
savings: InsightCard
inventory: InsightCard
waste: InsightCard
deliveries: InsightCard
# ============================================================
# API Endpoints
# ============================================================
@router.get("/health-status", response_model=BakeryHealthStatusResponse)
async def get_bakery_health_status(
tenant_id: str,
db: AsyncSession = Depends(get_db)
) -> BakeryHealthStatusResponse:
"""
Get overall bakery health status with tri-state checklist
This is the top-level indicator showing if the bakery is running smoothly
or if there are issues requiring attention. Includes AI-prevented issues.
"""
try:
# Try to get from cache
if settings.CACHE_ENABLED:
cache_key = f"dashboard:health:{tenant_id}"
cached = await get_cached(cache_key)
if cached:
return BakeryHealthStatusResponse(**cached)
dashboard_service = DashboardService(db)
# Gather metrics from various services in parallel
# Use asyncio.gather to make all HTTP calls concurrently
async def fetch_alerts():
try:
alerts_data = await alerts_client.get_alerts(tenant_id, limit=100) or {}
alerts_list = alerts_data.get("alerts", [])
# Count critical alerts
critical_count = sum(1 for a in alerts_list if a.get('priority_level') == 'CRITICAL')
# Count AI prevented issues
prevented_count = sum(1 for a in alerts_list if a.get('type_class') == 'prevented_issue')
return critical_count, prevented_count, alerts_list
except Exception as e:
logger.warning(f"Failed to fetch alerts: {e}")
return 0, 0, []
async def fetch_pending_pos():
try:
po_data = await procurement_client.get_pending_purchase_orders(tenant_id, limit=100) or []
return len(po_data) if isinstance(po_data, list) else 0
except Exception as e:
logger.warning(f"Failed to fetch POs: {e}")
return 0
async def fetch_production_delays():
try:
prod_data = await production_client.get_production_batches_by_status(
tenant_id, status="ON_HOLD", limit=100
) or {}
return len(prod_data.get("batches", []))
except Exception as e:
logger.warning(f"Failed to fetch production batches: {e}")
return 0
async def fetch_inventory():
try:
inv_data = await inventory_client.get_inventory_dashboard(tenant_id) or {}
return inv_data.get("out_of_stock_count", 0)
except Exception as e:
logger.warning(f"Failed to fetch inventory: {e}")
return 0
# Execute all fetches in parallel
alerts_result, pending_approvals, production_delays, out_of_stock_count = await asyncio.gather(
fetch_alerts(),
fetch_pending_pos(),
fetch_production_delays(),
fetch_inventory()
)
critical_alerts, ai_prevented_count, all_alerts = alerts_result
# System errors (would come from monitoring system)
system_errors = 0
# Calculate health status with tri-state checklist
health_status = await dashboard_service.get_bakery_health_status(
tenant_id=tenant_id,
critical_alerts=critical_alerts,
pending_approvals=pending_approvals,
production_delays=production_delays,
out_of_stock_count=out_of_stock_count,
system_errors=system_errors,
ai_prevented_count=ai_prevented_count,
action_needed_alerts=all_alerts
)
# Cache the result
if settings.CACHE_ENABLED:
cache_key = f"dashboard:health:{tenant_id}"
await set_cached(cache_key, health_status, ttl=settings.CACHE_TTL_HEALTH)
return BakeryHealthStatusResponse(**health_status)
except Exception as e:
logger.error(f"Error getting health status: {e}", exc_info=True)
raise HTTPException(status_code=500, detail=str(e))
@router.get("/orchestration-summary", response_model=OrchestrationSummaryResponse)
async def get_orchestration_summary(
tenant_id: str,
run_id: Optional[str] = Query(None, description="Specific run ID, or latest if not provided"),
db: AsyncSession = Depends(get_db)
) -> OrchestrationSummaryResponse:
"""
Get narrative summary of what the orchestrator did
This provides transparency into the automation, showing what was planned
and why, helping build user trust in the system.
"""
try:
# Try to get from cache (only if no specific run_id is provided)
if settings.CACHE_ENABLED and run_id is None:
cache_key = f"dashboard:summary:{tenant_id}"
cached = await get_cached(cache_key)
if cached:
return OrchestrationSummaryResponse(**cached)
dashboard_service = DashboardService(db)
# Get orchestration summary
summary = await dashboard_service.get_orchestration_summary(
tenant_id=tenant_id,
last_run_id=run_id
)
# Enhance with detailed PO and batch summaries
if summary["purchaseOrdersCreated"] > 0:
try:
po_data = await procurement_client.get_pending_purchase_orders(tenant_id, limit=10)
if po_data and isinstance(po_data, list):
# Override stale orchestration count with actual real-time PO count
summary["purchaseOrdersCreated"] = len(po_data)
summary["userActionsRequired"] = len(po_data) # Update actions required to match actual pending POs
summary["purchaseOrdersSummary"] = [
PurchaseOrderSummary(
supplierName=po.get("supplier_name", "Unknown"),
itemCategories=[item.get("ingredient_name", "Item") for item in po.get("items", [])[:3]],
totalAmount=float(po.get("total_amount", 0))
)
for po in po_data[:5] # Show top 5
]
except Exception as e:
logger.warning(f"Failed to fetch PO details: {e}")
if summary["productionBatchesCreated"] > 0:
try:
batch_data = await production_client.get_todays_batches(tenant_id)
if batch_data:
batches = batch_data.get("batches", [])
# Override stale orchestration count with actual real-time batch count
summary["productionBatchesCreated"] = len(batches)
summary["productionBatchesSummary"] = [
ProductionBatchSummary(
productName=batch.get("product_name", "Unknown"),
quantity=batch.get("planned_quantity", 0),
readyByTime=batch.get("planned_end_time", "")
)
for batch in batches[:5] # Show top 5
]
except Exception as e:
logger.warning(f"Failed to fetch batch details: {e}")
# Cache the result (only if no specific run_id)
if settings.CACHE_ENABLED and run_id is None:
cache_key = f"dashboard:summary:{tenant_id}"
await set_cached(cache_key, summary, ttl=settings.CACHE_TTL_SUMMARY)
return OrchestrationSummaryResponse(**summary)
except Exception as e:
logger.error(f"Error getting orchestration summary: {e}", exc_info=True)
raise HTTPException(status_code=500, detail=str(e))
@router.get("/action-queue", response_model=ActionQueueResponse)
async def get_action_queue(
tenant_id: str,
db: AsyncSession = Depends(get_db)
) -> ActionQueueResponse:
"""
Get prioritized queue of actions requiring user attention
This is the core of the JTBD dashboard - showing exactly what the user
needs to do right now, prioritized by urgency and impact.
"""
try:
dashboard_service = DashboardService(db)
# Fetch data from various services in parallel
async def fetch_pending_pos():
try:
po_data = await procurement_client.get_pending_purchase_orders(tenant_id, limit=20)
if po_data and isinstance(po_data, list):
return po_data
return []
except Exception as e:
logger.warning(f"Failed to fetch pending POs: {e}")
return []
async def fetch_critical_alerts():
try:
alerts_data = await alerts_client.get_critical_alerts(tenant_id, limit=20)
if alerts_data:
return alerts_data.get("alerts", [])
return []
except Exception as e:
logger.warning(f"Failed to fetch alerts: {e}")
return []
async def fetch_onboarding():
try:
onboarding_data = await procurement_client.get(
"/procurement/auth/onboarding-progress",
tenant_id=tenant_id
)
if onboarding_data:
return {
"incomplete": not onboarding_data.get("completed", True),
"steps": onboarding_data.get("steps", [])
}
return {"incomplete": False, "steps": []}
except Exception as e:
logger.warning(f"Failed to fetch onboarding status: {e}")
return {"incomplete": False, "steps": []}
# Execute all fetches in parallel
pending_pos, critical_alerts, onboarding = await asyncio.gather(
fetch_pending_pos(),
fetch_critical_alerts(),
fetch_onboarding()
)
onboarding_incomplete = onboarding["incomplete"]
onboarding_steps = onboarding["steps"]
# Build action queue
actions = await dashboard_service.get_action_queue(
tenant_id=tenant_id,
pending_pos=pending_pos,
critical_alerts=critical_alerts,
onboarding_incomplete=onboarding_incomplete,
onboarding_steps=onboarding_steps
)
# Count by urgency
critical_count = sum(1 for a in actions if a["urgency"] == "critical")
important_count = sum(1 for a in actions if a["urgency"] == "important")
return ActionQueueResponse(
actions=[ActionItem(**action) for action in actions[:10]], # Show top 10
totalActions=len(actions),
criticalCount=critical_count,
importantCount=important_count
)
except Exception as e:
logger.error(f"Error getting action queue: {e}", exc_info=True)
raise HTTPException(status_code=500, detail=str(e))
@router.get("/production-timeline", response_model=ProductionTimelineResponse)
async def get_production_timeline(
tenant_id: str,
db: AsyncSession = Depends(get_db)
) -> ProductionTimelineResponse:
"""
Get today's production timeline
Shows what's being made today in chronological order with status and progress.
"""
try:
dashboard_service = DashboardService(db)
# Fetch today's production batches
batches = []
try:
batch_data = await production_client.get_todays_batches(tenant_id)
if batch_data:
batches = batch_data.get("batches", [])
except Exception as e:
logger.warning(f"Failed to fetch production batches: {e}")
# Transform to timeline format
timeline = await dashboard_service.get_production_timeline(
tenant_id=tenant_id,
batches=batches
)
# Count by status
completed = sum(1 for item in timeline if item["status"] == "COMPLETED")
in_progress = sum(1 for item in timeline if item["status"] == "IN_PROGRESS")
pending = sum(1 for item in timeline if item["status"] == "PENDING")
return ProductionTimelineResponse(
timeline=[ProductionTimelineItem(**item) for item in timeline],
totalBatches=len(timeline),
completedBatches=completed,
inProgressBatches=in_progress,
pendingBatches=pending
)
except Exception as e:
logger.error(f"Error getting production timeline: {e}", exc_info=True)
raise HTTPException(status_code=500, detail=str(e))
@router.get("/unified-action-queue")
async def get_unified_action_queue(
tenant_id: str,
db: AsyncSession = Depends(get_db)
) -> Dict[str, Any]:
"""
Get unified action queue with time-based grouping
Combines all alerts (PO approvals, delivery tracking, production, etc.)
into URGENT (<6h), TODAY (<24h), and THIS WEEK (<7d) sections.
"""
try:
dashboard_service = DashboardService(db)
# Fetch all alerts from alert processor
alerts_data = await alerts_client.get_alerts(tenant_id, limit=100) or {}
alerts = alerts_data.get("alerts", [])
# Build unified queue
action_queue = await dashboard_service.get_unified_action_queue(
tenant_id=tenant_id,
alerts=alerts
)
return action_queue
except Exception as e:
logger.error(f"Error getting unified action queue: {e}", exc_info=True)
raise HTTPException(status_code=500, detail=str(e))
@router.get("/execution-progress")
async def get_execution_progress(
tenant_id: str,
db: AsyncSession = Depends(get_db)
) -> Dict[str, Any]:
"""
Get execution progress for today's plan
Shows plan vs actual for production batches, deliveries, and approvals
"""
try:
dashboard_service = DashboardService(db)
# Fetch today's data in parallel
async def fetch_todays_batches():
try:
batch_data = await production_client.get_todays_batches(tenant_id)
if batch_data:
return batch_data.get("batches", [])
return []
except Exception as e:
logger.warning(f"Failed to fetch today's batches: {e}")
return []
async def fetch_expected_deliveries():
try:
# Get POs with expected deliveries today
from datetime import datetime, timedelta, timezone
pos_result = await procurement_client.get_pending_purchase_orders(tenant_id, limit=100)
if pos_result and isinstance(pos_result, list):
today_start = datetime.now(timezone.utc).replace(hour=0, minute=0, second=0, microsecond=0)
today_end = today_start.replace(hour=23, minute=59, second=59)
deliveries_today = []
for po in pos_result:
expected_date = po.get("expected_delivery_date")
if expected_date:
if isinstance(expected_date, str):
expected_date = datetime.fromisoformat(expected_date.replace('Z', '+00:00'))
if today_start <= expected_date <= today_end:
deliveries_today.append(po)
return deliveries_today
return []
except Exception as e:
logger.warning(f"Failed to fetch expected deliveries: {e}")
return []
async def fetch_pending_approvals():
try:
po_data = await procurement_client.get_pending_purchase_orders(tenant_id, limit=100)
if po_data is None:
logger.error(
"Procurement client returned None for pending POs",
tenant_id=tenant_id,
context="likely HTTP 404 error - check URL construction"
)
return 0
if not isinstance(po_data, list):
logger.error(
"Unexpected response format from procurement client",
tenant_id=tenant_id,
response_type=type(po_data).__name__,
response_value=str(po_data)[:200]
)
return 0
logger.info(
"Successfully fetched pending purchase orders",
tenant_id=tenant_id,
count=len(po_data)
)
return len(po_data)
except Exception as e:
logger.error(
"Exception while fetching pending approvals",
tenant_id=tenant_id,
error=str(e),
exc_info=True
)
return 0
# Execute in parallel
todays_batches, expected_deliveries, pending_approvals = await asyncio.gather(
fetch_todays_batches(),
fetch_expected_deliveries(),
fetch_pending_approvals()
)
# Calculate progress
progress = await dashboard_service.get_execution_progress(
tenant_id=tenant_id,
todays_batches=todays_batches,
expected_deliveries=expected_deliveries,
pending_approvals=pending_approvals
)
return progress
except Exception as e:
logger.error(f"Error getting execution progress: {e}", exc_info=True)
raise HTTPException(status_code=500, detail=str(e))
@router.get("/insights", response_model=InsightsResponse)
async def get_insights(
tenant_id: str,
db: AsyncSession = Depends(get_db)
) -> InsightsResponse:
"""
Get key insights for dashboard grid
Provides glanceable metrics on savings, inventory, waste, and deliveries.
"""
try:
# Try to get from cache
if settings.CACHE_ENABLED:
cache_key = f"dashboard:insights:{tenant_id}"
cached = await get_cached(cache_key)
if cached:
return InsightsResponse(**cached)
dashboard_service = DashboardService(db)
# Fetch data from various services in parallel
from datetime import datetime, timedelta, timezone
async def fetch_sustainability():
try:
return await inventory_client.get_sustainability_widget(tenant_id) or {}
except Exception as e:
logger.warning(f"Failed to fetch sustainability data: {e}")
return {}
async def fetch_inventory():
try:
raw_inventory_data = await inventory_client.get_stock_status(tenant_id)
# Handle case where API returns a list instead of dict
if isinstance(raw_inventory_data, dict):
return raw_inventory_data
elif isinstance(raw_inventory_data, list):
# If it's a list, aggregate the data
return {
"low_stock_count": sum(1 for item in raw_inventory_data if item.get("status") == "low_stock"),
"out_of_stock_count": sum(1 for item in raw_inventory_data if item.get("status") == "out_of_stock"),
"total_items": len(raw_inventory_data)
}
return {}
except Exception as e:
logger.warning(f"Failed to fetch inventory data: {e}")
return {}
async def fetch_deliveries():
try:
# Get recent POs with pending deliveries
pos_result = await procurement_client.get_pending_purchase_orders(tenant_id, limit=100)
if pos_result and isinstance(pos_result, list):
# Count deliveries expected today
today_start = datetime.now(timezone.utc).replace(hour=0, minute=0, second=0, microsecond=0)
today_end = today_start.replace(hour=23, minute=59, second=59)
deliveries_today = 0
for po in pos_result:
expected_date = po.get("expected_delivery_date")
if expected_date:
if isinstance(expected_date, str):
expected_date = datetime.fromisoformat(expected_date.replace('Z', '+00:00'))
if today_start <= expected_date <= today_end:
deliveries_today += 1
return {"deliveries_today": deliveries_today}
return {}
except Exception as e:
logger.warning(f"Failed to fetch delivery data: {e}")
return {}
async def fetch_savings():
try:
# Get prevented issue savings from alert analytics
analytics = await alerts_client.get_dashboard_analytics(tenant_id, days=7)
if analytics:
weekly_savings = analytics.get('estimated_savings_eur', 0)
prevented_count = analytics.get('prevented_issues_count', 0)
# Calculate trend from period comparison
period_comparison = analytics.get('period_comparison', {})
current_prevented = period_comparison.get('current_prevented', 0)
previous_prevented = period_comparison.get('previous_prevented', 0)
trend_percentage = 0
if previous_prevented > 0:
trend_percentage = ((current_prevented - previous_prevented) / previous_prevented) * 100
return {
"weekly_savings": round(weekly_savings, 2),
"trend_percentage": round(trend_percentage, 1),
"prevented_count": prevented_count
}
return {"weekly_savings": 0, "trend_percentage": 0, "prevented_count": 0}
except Exception as e:
logger.warning(f"Failed to calculate savings data: {e}")
return {"weekly_savings": 0, "trend_percentage": 0, "prevented_count": 0}
# Execute all fetches in parallel
sustainability_data, inventory_data, delivery_data, savings_data = await asyncio.gather(
fetch_sustainability(),
fetch_inventory(),
fetch_deliveries(),
fetch_savings()
)
# Merge delivery data into inventory data
inventory_data.update(delivery_data)
# Calculate insights
insights = await dashboard_service.calculate_insights(
tenant_id=tenant_id,
sustainability_data=sustainability_data,
inventory_data=inventory_data,
savings_data=savings_data
)
# Prepare response
response_data = {
"savings": insights["savings"],
"inventory": insights["inventory"],
"waste": insights["waste"],
"deliveries": insights["deliveries"]
}
# Cache the result
if settings.CACHE_ENABLED:
cache_key = f"dashboard:insights:{tenant_id}"
await set_cached(cache_key, response_data, ttl=settings.CACHE_TTL_INSIGHTS)
return InsightsResponse(
savings=InsightCard(**insights["savings"]),
inventory=InsightCard(**insights["inventory"]),
waste=InsightCard(**insights["waste"]),
deliveries=InsightCard(**insights["deliveries"])
)
except Exception as e:
logger.error(f"Error getting insights: {e}", exc_info=True)
raise HTTPException(status_code=500, detail=str(e))

200
services/orchestrator/app/api/enterprise_dashboard.py
View File

@@ -1,200 +0,0 @@
"""
Enterprise Dashboard API Endpoints for Orchestrator Service
"""
from fastapi import APIRouter, Depends, HTTPException
from typing import List, Optional, Dict, Any
from datetime import date
import structlog
from app.services.enterprise_dashboard_service import EnterpriseDashboardService
from shared.auth.tenant_access import verify_tenant_access_dep
from shared.clients.tenant_client import TenantServiceClient
from shared.clients.forecast_client import ForecastServiceClient
from shared.clients.production_client import ProductionServiceClient
from shared.clients.sales_client import SalesServiceClient
from shared.clients.inventory_client import InventoryServiceClient
from shared.clients.distribution_client import DistributionServiceClient
logger = structlog.get_logger()
router = APIRouter(prefix="/api/v1/tenants/{tenant_id}/enterprise", tags=["enterprise"])
# Add dependency injection function
from app.services.enterprise_dashboard_service import EnterpriseDashboardService
from shared.clients import (
get_tenant_client,
get_forecast_client,
get_production_client,
get_sales_client,
get_inventory_client,
get_procurement_client,
get_distribution_client
)
def get_enterprise_dashboard_service() -> EnterpriseDashboardService:
from app.core.config import settings
tenant_client = get_tenant_client(settings)
forecast_client = get_forecast_client(settings)
production_client = get_production_client(settings)
sales_client = get_sales_client(settings)
inventory_client = get_inventory_client(settings)
distribution_client = get_distribution_client(settings)
procurement_client = get_procurement_client(settings)
return EnterpriseDashboardService(
tenant_client=tenant_client,
forecast_client=forecast_client,
production_client=production_client,
sales_client=sales_client,
inventory_client=inventory_client,
distribution_client=distribution_client,
procurement_client=procurement_client
)
@router.get("/network-summary")
async def get_network_summary(
tenant_id: str,
enterprise_service: EnterpriseDashboardService = Depends(get_enterprise_dashboard_service),
verified_tenant: str = Depends(verify_tenant_access_dep)
):
"""
Get network summary metrics for enterprise dashboard
"""
try:
# Verify user has network access
tenant_info = await enterprise_service.tenant_client.get_tenant(tenant_id)
if not tenant_info:
raise HTTPException(status_code=404, detail="Tenant not found")
if tenant_info.get('tenant_type') != 'parent':
raise HTTPException(status_code=403, detail="Only parent tenants can access enterprise dashboard")
result = await enterprise_service.get_network_summary(parent_tenant_id=tenant_id)
return result
except Exception as e:
logger.error(f"Error getting network summary: {e}", exc_info=True)
raise HTTPException(status_code=500, detail="Failed to get network summary")
@router.get("/children-performance")
async def get_children_performance(
tenant_id: str,
metric: str = "sales",
period_days: int = 30,
enterprise_service: EnterpriseDashboardService = Depends(get_enterprise_dashboard_service),
verified_tenant: str = Depends(verify_tenant_access_dep)
):
"""
Get anonymized performance ranking of child tenants
"""
try:
# Verify user has network access
tenant_info = await enterprise_service.tenant_client.get_tenant(tenant_id)
if not tenant_info:
raise HTTPException(status_code=404, detail="Tenant not found")
if tenant_info.get('tenant_type') != 'parent':
raise HTTPException(status_code=403, detail="Only parent tenants can access enterprise dashboard")
result = await enterprise_service.get_children_performance(
parent_tenant_id=tenant_id,
metric=metric,
period_days=period_days
)
return result
except Exception as e:
logger.error(f"Error getting children performance: {e}", exc_info=True)
raise HTTPException(status_code=500, detail="Failed to get children performance")
@router.get("/distribution-overview")
async def get_distribution_overview(
tenant_id: str,
target_date: Optional[date] = None,
enterprise_service: EnterpriseDashboardService = Depends(get_enterprise_dashboard_service),
verified_tenant: str = Depends(verify_tenant_access_dep)
):
"""
Get distribution overview for enterprise dashboard
"""
try:
# Verify user has network access
tenant_info = await enterprise_service.tenant_client.get_tenant(tenant_id)
if not tenant_info:
raise HTTPException(status_code=404, detail="Tenant not found")
if tenant_info.get('tenant_type') != 'parent':
raise HTTPException(status_code=403, detail="Only parent tenants can access enterprise dashboard")
if target_date is None:
target_date = date.today()
result = await enterprise_service.get_distribution_overview(
parent_tenant_id=tenant_id,
target_date=target_date
)
return result
except Exception as e:
logger.error(f"Error getting distribution overview: {e}", exc_info=True)
raise HTTPException(status_code=500, detail="Failed to get distribution overview")
@router.get("/forecast-summary")
async def get_enterprise_forecast_summary(
tenant_id: str,
days_ahead: int = 7,
enterprise_service: EnterpriseDashboardService = Depends(get_enterprise_dashboard_service),
verified_tenant: str = Depends(verify_tenant_access_dep)
):
"""
Get aggregated forecast summary for the enterprise network
"""
try:
# Verify user has network access
tenant_info = await enterprise_service.tenant_client.get_tenant(tenant_id)
if not tenant_info:
raise HTTPException(status_code=404, detail="Tenant not found")
if tenant_info.get('tenant_type') != 'parent':
raise HTTPException(status_code=403, detail="Only parent tenants can access enterprise dashboard")
result = await enterprise_service.get_enterprise_forecast_summary(
parent_tenant_id=tenant_id,
days_ahead=days_ahead
)
return result
except Exception as e:
logger.error(f"Error getting enterprise forecast summary: {e}", exc_info=True)
raise HTTPException(status_code=500, detail="Failed to get enterprise forecast summary")
@router.get("/network-performance")
async def get_network_performance_metrics(
tenant_id: str,
start_date: Optional[date] = None,
end_date: Optional[date] = None,
enterprise_service: EnterpriseDashboardService = Depends(get_enterprise_dashboard_service),
verified_tenant: str = Depends(verify_tenant_access_dep)
):
"""
Get aggregated performance metrics across the tenant network
"""
try:
# Verify user has network access
tenant_info = await enterprise_service.tenant_client.get_tenant(tenant_id)
if not tenant_info:
raise HTTPException(status_code=404, detail="Tenant not found")
if tenant_info.get('tenant_type') != 'parent':
raise HTTPException(status_code=403, detail="Only parent tenants can access enterprise dashboard")
if not start_date:
start_date = date.today()
if not end_date:
end_date = date.today()
result = await enterprise_service.get_network_performance_metrics(
parent_tenant_id=tenant_id,
start_date=start_date,
end_date=end_date
)
return result
except Exception as e:
logger.error(f"Error getting network performance metrics: {e}", exc_info=True)
raise HTTPException(status_code=500, detail="Failed to get network performance metrics")

41
services/orchestrator/app/api/orchestration.py
View File

@@ -303,3 +303,44 @@ async def list_orchestration_runs(
tenant_id=tenant_id,
error=str(e))
raise HTTPException(status_code=500, detail=str(e))
@router.get("/last-run")
async def get_last_orchestration_run(
tenant_id: str,
db: AsyncSession = Depends(get_db)
):
"""
Get timestamp of last orchestration run
Lightweight endpoint for health status frontend migration (Phase 4).
Returns only timestamp and run number for the most recent completed run.
Args:
tenant_id: Tenant ID
Returns:
Dict with timestamp and runNumber (or None if no runs)
"""
try:
tenant_uuid = uuid.UUID(tenant_id)
repo = OrchestrationRunRepository(db)
# Get most recent completed run
latest_run = await repo.get_latest_run_for_tenant(tenant_uuid)
if not latest_run:
return {"timestamp": None, "runNumber": None}
return {
"timestamp": latest_run.started_at.isoformat() if latest_run.started_at else None,
"runNumber": latest_run.run_number
}
except ValueError as e:
raise HTTPException(status_code=400, detail=f"Invalid tenant ID: {str(e)}")
except Exception as e:
logger.error("Error getting last orchestration run",
tenant_id=tenant_id,
error=str(e))
raise HTTPException(status_code=500, detail=str(e))

7
services/orchestrator/app/core/config.py
View File

@@ -114,6 +114,13 @@ class OrchestratorSettings(BaseServiceSettings):
CACHE_TTL_INSIGHTS: int = int(os.getenv("CACHE_TTL_INSIGHTS", "60")) # 1 minute (reduced for faster metrics updates)
CACHE_TTL_SUMMARY: int = int(os.getenv("CACHE_TTL_SUMMARY", "60")) # 1 minute
# Enterprise dashboard cache TTLs
CACHE_TTL_ENTERPRISE_SUMMARY: int = int(os.getenv("CACHE_TTL_ENTERPRISE_SUMMARY", "60")) # 1 minute
CACHE_TTL_ENTERPRISE_PERFORMANCE: int = int(os.getenv("CACHE_TTL_ENTERPRISE_PERFORMANCE", "60")) # 1 minute
CACHE_TTL_ENTERPRISE_DISTRIBUTION: int = int(os.getenv("CACHE_TTL_ENTERPRISE_DISTRIBUTION", "30")) # 30 seconds
CACHE_TTL_ENTERPRISE_FORECAST: int = int(os.getenv("CACHE_TTL_ENTERPRISE_FORECAST", "120")) # 2 minutes
CACHE_TTL_ENTERPRISE_NETWORK: int = int(os.getenv("CACHE_TTL_ENTERPRISE_NETWORK", "60")) # 1 minute
# Global settings instance
settings = OrchestratorSettings()

78
services/orchestrator/app/main.py
View File

@@ -16,7 +16,7 @@ from shared.service_base import StandardFastAPIService
class OrchestratorService(StandardFastAPIService):
"""Orchestrator Service with standardized setup"""
expected_migration_version = "00001"
expected_migration_version = "001_initial_schema"
async def verify_migrations(self):
"""Verify database schema matches the latest migrations"""
@@ -38,6 +38,9 @@ class OrchestratorService(StandardFastAPIService):
'orchestration_runs'
]
self.rabbitmq_client = None
self.event_publisher = None
super().__init__(
service_name="orchestrator-service",
app_name=settings.APP_NAME,
@@ -45,20 +48,51 @@ class OrchestratorService(StandardFastAPIService):
version=settings.VERSION,
api_prefix="", # Empty because RouteBuilder already includes /api/v1
database_manager=database_manager,
expected_tables=orchestrator_expected_tables
expected_tables=orchestrator_expected_tables,
enable_messaging=True # Enable RabbitMQ for event publishing
)
async def _setup_messaging(self):
"""Setup messaging for orchestrator service"""
from shared.messaging import UnifiedEventPublisher, RabbitMQClient
try:
self.rabbitmq_client = RabbitMQClient(settings.RABBITMQ_URL, service_name="orchestrator-service")
await self.rabbitmq_client.connect()
# Create event publisher
self.event_publisher = UnifiedEventPublisher(self.rabbitmq_client, "orchestrator-service")
self.logger.info("Orchestrator service messaging setup completed")
except Exception as e:
self.logger.error("Failed to setup orchestrator messaging", error=str(e))
raise
async def _cleanup_messaging(self):
"""Cleanup messaging for orchestrator service"""
try:
if self.rabbitmq_client:
await self.rabbitmq_client.disconnect()
self.logger.info("Orchestrator service messaging cleanup completed")
except Exception as e:
self.logger.error("Error during orchestrator messaging cleanup", error=str(e))
async def on_startup(self, app: FastAPI):
"""Custom startup logic for orchestrator service"""
# Verify migrations first
await self.verify_migrations()
# Call parent startup (includes database, messaging, etc.)
await super().on_startup(app)
self.logger.info("Orchestrator Service starting up...")
# Initialize orchestrator scheduler service
# Initialize orchestrator scheduler service with EventPublisher
from app.services.orchestrator_service import OrchestratorSchedulerService
scheduler_service = OrchestratorSchedulerService(settings)
scheduler_service = OrchestratorSchedulerService(self.event_publisher, settings)
await scheduler_service.start()
app.state.scheduler_service = scheduler_service
self.logger.info("Orchestrator scheduler service started")
# REMOVED: Delivery tracking service - moved to procurement service (domain ownership)
async def on_shutdown(self, app: FastAPI):
"""Custom shutdown logic for orchestrator service"""
self.logger.info("Orchestrator Service shutting down...")
@@ -68,6 +102,7 @@ class OrchestratorService(StandardFastAPIService):
await app.state.scheduler_service.stop()
self.logger.info("Orchestrator scheduler service stopped")
def get_service_features(self):
"""Return orchestrator-specific features"""
return [
@@ -94,45 +129,10 @@ service.setup_standard_endpoints()
# Include routers
# BUSINESS: Orchestration operations
from app.api.orchestration import router as orchestration_router
from app.api.dashboard import router as dashboard_router
from app.api.enterprise_dashboard import router as enterprise_dashboard_router
from app.api.internal import router as internal_router
service.add_router(orchestration_router)
service.add_router(dashboard_router)
service.add_router(enterprise_dashboard_router)
service.add_router(internal_router)
# Add enterprise dashboard service to dependencies
from app.services.enterprise_dashboard_service import EnterpriseDashboardService
from shared.clients import (
get_tenant_client,
get_forecast_client,
get_production_client,
get_sales_client,
get_inventory_client,
get_procurement_client,
get_distribution_client
)
def get_enterprise_dashboard_service() -> EnterpriseDashboardService:
tenant_client = get_tenant_client(settings)
forecast_client = get_forecast_client(settings)
production_client = get_production_client(settings)
sales_client = get_sales_client(settings)
inventory_client = get_inventory_client(settings)
distribution_client = get_distribution_client(settings)
procurement_client = get_procurement_client(settings)
return EnterpriseDashboardService(
tenant_client=tenant_client,
forecast_client=forecast_client,
production_client=production_client,
sales_client=sales_client,
inventory_client=inventory_client,
distribution_client=distribution_client,
procurement_client=procurement_client
)
# INTERNAL: Service-to-service endpoints
from app.api import internal_demo
service.add_router(internal_demo.router)

1090
services/orchestrator/app/services/dashboard_service.py
View File

File diff suppressed because it is too large Load Diff

420
services/orchestrator/app/services/delivery_tracking_service.py
View File

@@ -1,420 +0,0 @@
"""
Delivery Tracking Service
Tracks purchase order deliveries and generates appropriate alerts:
- DELIVERY_SCHEDULED: When PO is approved and delivery date is set
- DELIVERY_ARRIVING_SOON: 2 hours before delivery window
- DELIVERY_OVERDUE: 30 minutes after expected delivery time
- STOCK_RECEIPT_INCOMPLETE: If delivery not marked as received
Integrates with procurement service to get PO details and expected delivery windows.
"""
import structlog
from datetime import datetime, timedelta, timezone
from typing import Dict, Any, Optional, List
from uuid import UUID
import httpx
from shared.schemas.alert_types import AlertTypeConstants
from shared.alerts.base_service import BaseAlertService
logger = structlog.get_logger()
class DeliveryTrackingService:
"""Tracks deliveries and generates lifecycle alerts"""
def __init__(self, config, db_manager, redis_client, rabbitmq_client):
self.config = config
self.db_manager = db_manager
self.redis = redis_client
self.rabbitmq = rabbitmq_client
self.alert_service = BaseAlertService(config)
self.http_client = httpx.AsyncClient(
timeout=30.0,
follow_redirects=True
)
async def check_expected_deliveries(self, tenant_id: UUID) -> Dict[str, int]:
"""
Check all expected deliveries for a tenant and generate appropriate alerts.
Called by scheduled job (runs every hour).
Returns:
Dict with counts: {
'arriving_soon': int,
'overdue': int,
'receipt_incomplete': int
}
"""
logger.info("Checking expected deliveries", tenant_id=str(tenant_id))
counts = {
'arriving_soon': 0,
'overdue': 0,
'receipt_incomplete': 0
}
try:
# Get expected deliveries from procurement service
deliveries = await self._get_expected_deliveries(tenant_id)
now = datetime.now(timezone.utc)
for delivery in deliveries:
po_id = delivery.get('po_id')
po_number = delivery.get('po_number')
expected_date = delivery.get('expected_delivery_date')
delivery_window_hours = delivery.get('delivery_window_hours', 4) # Default 4h window
status = delivery.get('status')
if not expected_date:
continue
# Parse expected date
if isinstance(expected_date, str):
expected_date = datetime.fromisoformat(expected_date)
# Make timezone-aware
if expected_date.tzinfo is None:
expected_date = expected_date.replace(tzinfo=timezone.utc)
# Calculate delivery window
window_start = expected_date
window_end = expected_date + timedelta(hours=delivery_window_hours)
# Check if arriving soon (2 hours before window)
arriving_soon_time = window_start - timedelta(hours=2)
if arriving_soon_time <= now < window_start and status == 'approved':
if await self._send_arriving_soon_alert(tenant_id, delivery):
counts['arriving_soon'] += 1
# Check if overdue (30 min after window end)
overdue_time = window_end + timedelta(minutes=30)
if now >= overdue_time and status == 'approved':
if await self._send_overdue_alert(tenant_id, delivery):
counts['overdue'] += 1
# Check if receipt incomplete (delivery window passed, not marked received)
if now > window_end and status == 'approved':
if await self._send_receipt_incomplete_alert(tenant_id, delivery):
counts['receipt_incomplete'] += 1
logger.info(
"Delivery check completed",
tenant_id=str(tenant_id),
**counts
)
except Exception as e:
logger.error(
"Error checking deliveries",
tenant_id=str(tenant_id),
error=str(e)
)
return counts
async def _get_expected_deliveries(self, tenant_id: UUID) -> List[Dict[str, Any]]:
"""
Query procurement service for expected deliveries.
Returns:
List of delivery dicts with:
- po_id, po_number, expected_delivery_date
- supplier_id, supplier_name
- line_items (product list)
- status (approved, in_transit, received)
"""
try:
procurement_url = self.config.PROCUREMENT_SERVICE_URL
response = await self.http_client.get(
f"{procurement_url}/api/internal/expected-deliveries",
params={
"tenant_id": str(tenant_id),
"days_ahead": 1, # Check today + tomorrow
"include_overdue": True
},
headers={"X-Internal-Service": "orchestrator"}
)
if response.status_code == 200:
data = response.json()
return data.get('deliveries', [])
else:
logger.warning(
"Failed to get expected deliveries",
status_code=response.status_code,
tenant_id=str(tenant_id)
)
return []
except Exception as e:
logger.error(
"Error fetching expected deliveries",
tenant_id=str(tenant_id),
error=str(e)
)
return []
async def _send_arriving_soon_alert(
self,
tenant_id: UUID,
delivery: Dict[str, Any]
) -> bool:
"""
Send DELIVERY_ARRIVING_SOON alert (2h before delivery window).
This appears in the action queue with "Mark as Received" action.
"""
# Check if already sent
cache_key = f"delivery_alert:arriving:{tenant_id}:{delivery['po_id']}"
if await self.redis.exists(cache_key):
return False
po_number = delivery.get('po_number', 'N/A')
supplier_name = delivery.get('supplier_name', 'Supplier')
expected_date = delivery.get('expected_delivery_date')
line_items = delivery.get('line_items', [])
# Format product list
products = [item['product_name'] for item in line_items[:3]]
product_list = ", ".join(products)
if len(line_items) > 3:
product_list += f" (+{len(line_items) - 3} more)"
# Calculate time until arrival
if isinstance(expected_date, str):
expected_date = datetime.fromisoformat(expected_date)
if expected_date.tzinfo is None:
expected_date = expected_date.replace(tzinfo=timezone.utc)
hours_until = (expected_date - datetime.now(timezone.utc)).total_seconds() / 3600
alert_data = {
"tenant_id": str(tenant_id),
"alert_type": AlertTypeConstants.DELIVERY_ARRIVING_SOON,
"title": f"Delivery arriving soon: {supplier_name}",
"message": f"Purchase order {po_number} expected in ~{hours_until:.1f} hours. Products: {product_list}",
"service": "orchestrator",
"actions": ["mark_delivery_received", "call_supplier"],
"alert_metadata": {
"po_id": delivery['po_id'],
"po_number": po_number,
"supplier_id": delivery.get('supplier_id'),
"supplier_name": supplier_name,
"supplier_phone": delivery.get('supplier_phone'),
"expected_delivery_date": expected_date.isoformat(),
"line_items": line_items,
"hours_until_arrival": hours_until,
"confidence_score": 0.9
}
}
success = await self.alert_service.send_alert(alert_data)
if success:
# Cache for 24 hours to avoid duplicate alerts
await self.redis.set(cache_key, "1", ex=86400)
logger.info(
"Sent arriving soon alert",
po_number=po_number,
supplier=supplier_name
)
return success
async def _send_overdue_alert(
self,
tenant_id: UUID,
delivery: Dict[str, Any]
) -> bool:
"""
Send DELIVERY_OVERDUE alert (30min after expected window).
Critical priority - needs immediate action (call supplier).
"""
# Check if already sent
cache_key = f"delivery_alert:overdue:{tenant_id}:{delivery['po_id']}"
if await self.redis.exists(cache_key):
return False
po_number = delivery.get('po_number', 'N/A')
supplier_name = delivery.get('supplier_name', 'Supplier')
expected_date = delivery.get('expected_delivery_date')
# Calculate how late
if isinstance(expected_date, str):
expected_date = datetime.fromisoformat(expected_date)
if expected_date.tzinfo is None:
expected_date = expected_date.replace(tzinfo=timezone.utc)
hours_late = (datetime.now(timezone.utc) - expected_date).total_seconds() / 3600
alert_data = {
"tenant_id": str(tenant_id),
"alert_type": AlertTypeConstants.DELIVERY_OVERDUE,
"title": f"Delivery overdue: {supplier_name}",
"message": f"Purchase order {po_number} was expected {hours_late:.1f} hours ago. Contact supplier immediately.",
"service": "orchestrator",
"actions": ["call_supplier", "snooze", "report_issue"],
"alert_metadata": {
"po_id": delivery['po_id'],
"po_number": po_number,
"supplier_id": delivery.get('supplier_id'),
"supplier_name": supplier_name,
"supplier_phone": delivery.get('supplier_phone'),
"expected_delivery_date": expected_date.isoformat(),
"hours_late": hours_late,
"financial_impact": delivery.get('total_amount', 0), # Blocked capital
"affected_orders": len(delivery.get('affected_production_batches', [])),
"confidence_score": 1.0
}
}
success = await self.alert_service.send_alert(alert_data)
if success:
# Cache for 48 hours
await self.redis.set(cache_key, "1", ex=172800)
logger.warning(
"Sent overdue delivery alert",
po_number=po_number,
supplier=supplier_name,
hours_late=hours_late
)
return success
async def _send_receipt_incomplete_alert(
self,
tenant_id: UUID,
delivery: Dict[str, Any]
) -> bool:
"""
Send STOCK_RECEIPT_INCOMPLETE alert.
Delivery window has passed but stock not marked as received.
"""
# Check if already sent
cache_key = f"delivery_alert:receipt:{tenant_id}:{delivery['po_id']}"
if await self.redis.exists(cache_key):
return False
po_number = delivery.get('po_number', 'N/A')
supplier_name = delivery.get('supplier_name', 'Supplier')
alert_data = {
"tenant_id": str(tenant_id),
"alert_type": AlertTypeConstants.STOCK_RECEIPT_INCOMPLETE,
"title": f"Confirm stock receipt: {po_number}",
"message": f"Delivery from {supplier_name} should have arrived. Please confirm receipt and log lot details.",
"service": "orchestrator",
"actions": ["complete_stock_receipt", "report_missing"],
"alert_metadata": {
"po_id": delivery['po_id'],
"po_number": po_number,
"supplier_id": delivery.get('supplier_id'),
"supplier_name": supplier_name,
"expected_delivery_date": delivery.get('expected_delivery_date'),
"confidence_score": 0.8
}
}
success = await self.alert_service.send_alert(alert_data)
if success:
# Cache for 7 days
await self.redis.set(cache_key, "1", ex=604800)
logger.info(
"Sent receipt incomplete alert",
po_number=po_number
)
return success
async def mark_delivery_received(
self,
tenant_id: UUID,
po_id: UUID,
received_by_user_id: UUID
) -> Dict[str, Any]:
"""
Mark delivery as received and trigger stock receipt workflow.
This is called when user clicks "Mark as Received" action button.
Returns:
Dict with receipt_id and status
"""
try:
# Call inventory service to create draft stock receipt
inventory_url = self.config.INVENTORY_SERVICE_URL
response = await self.http_client.post(
f"{inventory_url}/api/inventory/stock-receipts",
json={
"tenant_id": str(tenant_id),
"po_id": str(po_id),
"received_by_user_id": str(received_by_user_id)
},
headers={"X-Internal-Service": "orchestrator"}
)
if response.status_code in [200, 201]:
receipt_data = response.json()
# Clear delivery alerts
await self._clear_delivery_alerts(tenant_id, po_id)
logger.info(
"Delivery marked as received",
po_id=str(po_id),
receipt_id=receipt_data.get('id')
)
return {
"status": "success",
"receipt_id": receipt_data.get('id'),
"message": "Stock receipt created. Please complete lot details."
}
else:
logger.error(
"Failed to create stock receipt",
status_code=response.status_code,
po_id=str(po_id)
)
return {
"status": "error",
"message": "Failed to create stock receipt"
}
except Exception as e:
logger.error(
"Error marking delivery received",
po_id=str(po_id),
error=str(e)
)
return {
"status": "error",
"message": str(e)
}
async def _clear_delivery_alerts(self, tenant_id: UUID, po_id: UUID):
"""Clear all delivery-related alerts for a PO once received"""
alert_types = [
"arriving",
"overdue",
"receipt"
]
for alert_type in alert_types:
cache_key = f"delivery_alert:{alert_type}:{tenant_id}:{po_id}"
await self.redis.delete(cache_key)
logger.debug("Cleared delivery alerts", po_id=str(po_id))
async def close(self):
"""Close HTTP client on shutdown"""
await self.http_client.aclose()

648
services/orchestrator/app/services/enterprise_dashboard_service.py
View File

@@ -1,648 +0,0 @@
"""
Enterprise Dashboard Service for Orchestrator
Handles aggregated metrics and data for enterprise tier parent tenants
"""
import asyncio
from typing import Dict, Any, List
from datetime import date, datetime, timedelta
import structlog
from decimal import Decimal
# Import clients
from shared.clients.tenant_client import TenantServiceClient
from shared.clients.forecast_client import ForecastServiceClient
from shared.clients.production_client import ProductionServiceClient
from shared.clients.sales_client import SalesServiceClient
from shared.clients.inventory_client import InventoryServiceClient
from shared.clients.distribution_client import DistributionServiceClient
from shared.clients.procurement_client import ProcurementServiceClient
logger = structlog.get_logger()
class EnterpriseDashboardService:
"""
Service for providing enterprise dashboard data for parent tenants
"""
def __init__(
self,
tenant_client: TenantServiceClient,
forecast_client: ForecastServiceClient,
production_client: ProductionServiceClient,
sales_client: SalesServiceClient,
inventory_client: InventoryServiceClient,
distribution_client: DistributionServiceClient,
procurement_client: ProcurementServiceClient
):
self.tenant_client = tenant_client
self.forecast_client = forecast_client
self.production_client = production_client
self.sales_client = sales_client
self.inventory_client = inventory_client
self.distribution_client = distribution_client
self.procurement_client = procurement_client
async def get_network_summary(
self,
parent_tenant_id: str
) -> Dict[str, Any]:
"""
Get network summary metrics for enterprise dashboard
Args:
parent_tenant_id: Parent tenant ID
Returns:
Dict with aggregated network metrics
"""
logger.info("Getting network summary for parent tenant", parent_tenant_id=parent_tenant_id)
# Get child tenants
child_tenants = await self.tenant_client.get_child_tenants(parent_tenant_id)
child_tenant_ids = [child['id'] for child in (child_tenants or [])]
# Fetch metrics in parallel
tasks = [
self._get_child_count(parent_tenant_id),
self._get_network_sales(parent_tenant_id, child_tenant_ids),
self._get_production_volume(parent_tenant_id),
self._get_pending_internal_transfers(parent_tenant_id),
self._get_active_shipments(parent_tenant_id)
]
results = await asyncio.gather(*tasks, return_exceptions=True)
# Handle results and errors
child_count = results[0] if not isinstance(results[0], Exception) else 0
network_sales = results[1] if not isinstance(results[1], Exception) else 0
production_volume = results[2] if not isinstance(results[2], Exception) else 0
pending_transfers = results[3] if not isinstance(results[3], Exception) else 0
active_shipments = results[4] if not isinstance(results[4], Exception) else 0
return {
'parent_tenant_id': parent_tenant_id,
'child_tenant_count': child_count,
'network_sales_30d': float(network_sales),
'production_volume_30d': float(production_volume),
'pending_internal_transfers_count': pending_transfers,
'active_shipments_count': active_shipments,
'last_updated': datetime.utcnow().isoformat()
}
async def _get_child_count(self, parent_tenant_id: str) -> int:
"""Get count of child tenants"""
try:
child_tenants = await self.tenant_client.get_child_tenants(parent_tenant_id)
return len(child_tenants)
except Exception as e:
logger.warning(f"Could not get child count for parent tenant {parent_tenant_id}: {e}")
return 0
async def _get_network_sales(self, parent_tenant_id: str, child_tenant_ids: List[str]) -> float:
"""Get total network sales for the last 30 days"""
try:
total_sales = Decimal("0.00")
start_date = date.today() - timedelta(days=30)
end_date = date.today()
# Include parent tenant sales
try:
parent_sales = await self.sales_client.get_sales_summary(
tenant_id=parent_tenant_id,
start_date=start_date,
end_date=end_date
)
total_sales += Decimal(str(parent_sales.get('total_revenue', 0)))
except Exception as e:
logger.warning(f"Could not get sales for parent tenant {parent_tenant_id}: {e}")
# Add child tenant sales
for child_id in child_tenant_ids:
try:
child_sales = await self.sales_client.get_sales_summary(
tenant_id=child_id,
start_date=start_date,
end_date=end_date
)
total_sales += Decimal(str(child_sales.get('total_revenue', 0)))
except Exception as e:
logger.warning(f"Could not get sales for child tenant {child_id}: {e}")
return float(total_sales)
except Exception as e:
logger.error(f"Error getting network sales: {e}")
return 0.0
async def _get_production_volume(self, parent_tenant_id: str) -> float:
"""Get total production volume for the parent tenant (central production)"""
try:
production_summary = await self.production_client.get_dashboard_summary(
tenant_id=parent_tenant_id
)
# Return total production value
return float(production_summary.get('total_value', 0))
except Exception as e:
logger.warning(f"Could not get production volume for parent tenant {parent_tenant_id}: {e}")
return 0.0
async def _get_pending_internal_transfers(self, parent_tenant_id: str) -> int:
"""Get count of pending internal transfer orders from parent to children"""
try:
# Get pending internal purchase orders for parent tenant
pending_pos = await self.procurement_client.get_approved_internal_purchase_orders(
parent_tenant_id=parent_tenant_id,
status="pending" # or whatever status indicates pending delivery
)
return len(pending_pos) if pending_pos else 0
except Exception as e:
logger.warning(f"Could not get pending internal transfers for parent tenant {parent_tenant_id}: {e}")
return 0
async def _get_active_shipments(self, parent_tenant_id: str) -> int:
"""Get count of active shipments for today"""
try:
today = date.today()
shipments = await self.distribution_client.get_shipments_for_date(
parent_tenant_id,
today
)
# Filter for active shipments (not delivered/cancelled)
active_statuses = ['pending', 'in_transit', 'packed']
active_shipments = [s for s in shipments if s.get('status') in active_statuses]
return len(active_shipments)
except Exception as e:
logger.warning(f"Could not get active shipments for parent tenant {parent_tenant_id}: {e}")
return 0
async def get_children_performance(
self,
parent_tenant_id: str,
metric: str = "sales",
period_days: int = 30
) -> Dict[str, Any]:
"""
Get anonymized performance ranking of child tenants
Args:
parent_tenant_id: Parent tenant ID
metric: Metric to rank by ('sales', 'inventory_value', 'order_frequency')
period_days: Number of days to look back
Returns:
Dict with anonymized ranking data
"""
logger.info("Getting children performance",
parent_tenant_id=parent_tenant_id,
metric=metric,
period_days=period_days)
child_tenants = await self.tenant_client.get_child_tenants(parent_tenant_id)
# Gather performance data for each child
performance_data = []
for child in (child_tenants or []):
child_id = child['id']
child_name = child['name']
metric_value = 0
try:
if metric == 'sales':
start_date = date.today() - timedelta(days=period_days)
end_date = date.today()
sales_summary = await self.sales_client.get_sales_summary(
tenant_id=child_id,
start_date=start_date,
end_date=end_date
)
metric_value = float(sales_summary.get('total_revenue', 0))
elif metric == 'inventory_value':
inventory_summary = await self.inventory_client.get_inventory_summary(
tenant_id=child_id
)
metric_value = float(inventory_summary.get('total_value', 0))
elif metric == 'order_frequency':
# Count orders placed in the period
orders = await self.sales_client.get_sales_orders(
tenant_id=child_id,
start_date=start_date,
end_date=end_date
)
metric_value = len(orders) if orders else 0
except Exception as e:
logger.warning(f"Could not get performance data for child {child_id}: {e}")
continue
performance_data.append({
'tenant_id': child_id,
'original_name': child_name,
'metric_value': metric_value
})
# Sort by metric value and anonymize
performance_data.sort(key=lambda x: x['metric_value'], reverse=True)
# Anonymize data (no tenant names, just ranks)
anonymized_data = []
for rank, data in enumerate(performance_data, 1):
anonymized_data.append({
'rank': rank,
'tenant_id': data['tenant_id'],
'anonymized_name': f"Outlet {rank}",
'metric_value': data['metric_value']
})
return {
'parent_tenant_id': parent_tenant_id,
'metric': metric,
'period_days': period_days,
'rankings': anonymized_data,
'total_children': len(performance_data),
'last_updated': datetime.utcnow().isoformat()
}
async def get_distribution_overview(
self,
parent_tenant_id: str,
target_date: date = None
) -> Dict[str, Any]:
"""
Get distribution overview for enterprise dashboard
Args:
parent_tenant_id: Parent tenant ID
target_date: Date to get distribution data for (default: today)
Returns:
Dict with distribution metrics and route information
"""
if target_date is None:
target_date = date.today()
logger.info("Getting distribution overview",
parent_tenant_id=parent_tenant_id,
date=target_date)
try:
# Get all routes for the target date
routes = await self.distribution_client.get_delivery_routes(
parent_tenant_id=parent_tenant_id,
date_from=target_date,
date_to=target_date
)
# Get all shipments for the target date
shipments = await self.distribution_client.get_shipments_for_date(
parent_tenant_id,
target_date
)
# Aggregate by status
status_counts = {}
for shipment in shipments:
status = shipment.get('status', 'unknown')
status_counts[status] = status_counts.get(status, 0) + 1
# Prepare route sequences for map visualization
route_sequences = []
for route in routes:
route_sequences.append({
'route_id': route.get('id'),
'route_number': route.get('route_number'),
'status': route.get('status', 'unknown'),
'total_distance_km': route.get('total_distance_km', 0),
'stops': route.get('route_sequence', []),
'estimated_duration_minutes': route.get('estimated_duration_minutes', 0)
})
return {
'parent_tenant_id': parent_tenant_id,
'target_date': target_date.isoformat(),
'route_count': len(routes),
'shipment_count': len(shipments),
'status_counts': status_counts,
'route_sequences': route_sequences,
'last_updated': datetime.utcnow().isoformat()
}
except Exception as e:
logger.error(f"Error getting distribution overview: {e}", exc_info=True)
return {
'parent_tenant_id': parent_tenant_id,
'target_date': target_date.isoformat(),
'route_count': 0,
'shipment_count': 0,
'status_counts': {},
'route_sequences': [],
'last_updated': datetime.utcnow().isoformat(),
'error': str(e)
}
async def get_enterprise_forecast_summary(
self,
parent_tenant_id: str,
days_ahead: int = 7
) -> Dict[str, Any]:
"""
Get aggregated forecast summary for the enterprise network
Args:
parent_tenant_id: Parent tenant ID
days_ahead: Number of days ahead to forecast
Returns:
Dict with aggregated forecast data
"""
try:
end_date = date.today() + timedelta(days=days_ahead)
start_date = date.today()
# Get aggregated forecast from the forecasting service
forecast_data = await self.forecast_client.get_aggregated_forecast(
parent_tenant_id=parent_tenant_id,
start_date=start_date,
end_date=end_date
)
# Aggregate the forecast data for the summary
total_demand = 0
daily_summary = {}
if not forecast_data:
logger.warning("No forecast data returned", parent_tenant_id=parent_tenant_id)
return {
'parent_tenant_id': parent_tenant_id,
'days_forecast': days_ahead,
'total_predicted_demand': 0,
'daily_summary': {},
'last_updated': datetime.utcnow().isoformat()
}
for forecast_date_str, products in forecast_data.get('aggregated_forecasts', {}).items():
day_total = sum(item.get('predicted_demand', 0) for item in products.values())
total_demand += day_total
daily_summary[forecast_date_str] = {
'predicted_demand': day_total,
'product_count': len(products)
}
return {
'parent_tenant_id': parent_tenant_id,
'days_forecast': days_ahead,
'total_predicted_demand': total_demand,
'daily_summary': daily_summary,
'last_updated': datetime.utcnow().isoformat()
}
except Exception as e:
logger.error(f"Error getting enterprise forecast summary: {e}", exc_info=True)
return {
'parent_tenant_id': parent_tenant_id,
'days_forecast': days_ahead,
'total_predicted_demand': 0,
'daily_summary': {},
'last_updated': datetime.utcnow().isoformat(),
'error': str(e)
}
async def get_network_performance_metrics(
self,
parent_tenant_id: str,
start_date: date,
end_date: date
) -> Dict[str, Any]:
"""
Get aggregated performance metrics across the enterprise network
Args:
parent_tenant_id: Parent tenant ID
start_date: Start date for metrics
end_date: End date for metrics
Returns:
Dict with aggregated network metrics
"""
try:
# Get all child tenants
child_tenants = await self.tenant_client.get_child_tenants(parent_tenant_id)
child_tenant_ids = [child['id'] for child in (child_tenants or [])]
# Include parent in tenant list for complete network metrics
all_tenant_ids = [parent_tenant_id] + child_tenant_ids
# Parallel fetch of metrics for all tenants
tasks = []
for tenant_id in all_tenant_ids:
# Create individual tasks for each metric
sales_task = self._get_tenant_sales(tenant_id, start_date, end_date)
production_task = self._get_tenant_production(tenant_id, start_date, end_date)
inventory_task = self._get_tenant_inventory(tenant_id)
# Gather all tasks for this tenant
tenant_tasks = asyncio.gather(sales_task, production_task, inventory_task, return_exceptions=True)
tasks.append(tenant_tasks)
results = await asyncio.gather(*tasks, return_exceptions=True)
# Aggregate metrics
total_network_sales = Decimal("0.00")
total_network_production = Decimal("0.00")
total_network_inventory_value = Decimal("0.00")
metrics_error_count = 0
for i, result in enumerate(results):
if isinstance(result, Exception):
logger.error(f"Error getting metrics for tenant {all_tenant_ids[i]}: {result}")
metrics_error_count += 1
continue
if isinstance(result, list) and len(result) == 3:
sales, production, inventory = result
total_network_sales += Decimal(str(sales or 0))
total_network_production += Decimal(str(production or 0))
total_network_inventory_value += Decimal(str(inventory or 0))
return {
'parent_tenant_id': parent_tenant_id,
'start_date': start_date.isoformat(),
'end_date': end_date.isoformat(),
'total_network_sales': float(total_network_sales),
'total_network_production': float(total_network_production),
'total_network_inventory_value': float(total_network_inventory_value),
'included_tenant_count': len(all_tenant_ids),
'child_tenant_count': len(child_tenant_ids),
'metrics_error_count': metrics_error_count,
'coverage_percentage': (
(len(all_tenant_ids) - metrics_error_count) / len(all_tenant_ids) * 100
if all_tenant_ids else 0
)
}
except Exception as e:
logger.error(f"Error getting network performance metrics: {e}", exc_info=True)
raise
async def _get_tenant_sales(self, tenant_id: str, start_date: date, end_date: date) -> float:
"""Helper to get sales for a specific tenant"""
try:
sales_data = await self.sales_client.get_sales_summary(
tenant_id=tenant_id,
start_date=start_date,
end_date=end_date
)
return float(sales_data.get('total_revenue', 0))
except Exception as e:
logger.warning(f"Could not get sales for tenant {tenant_id}: {e}")
return 0
async def _get_tenant_production(self, tenant_id: str, start_date: date, end_date: date) -> float:
"""Helper to get production for a specific tenant"""
try:
production_data = await self.production_client.get_dashboard_summary(
tenant_id=tenant_id
)
return float(production_data.get('total_value', 0))
except Exception as e:
logger.warning(f"Could not get production for tenant {tenant_id}: {e}")
return 0
async def _get_tenant_inventory(self, tenant_id: str) -> float:
"""Helper to get inventory value for a specific tenant"""
try:
inventory_data = await self.inventory_client.get_inventory_summary(tenant_id=tenant_id)
return float(inventory_data.get('total_value', 0))
except Exception as e:
logger.warning(f"Could not get inventory for tenant {tenant_id}: {e}")
return 0
async def initialize_enterprise_demo(
self,
parent_tenant_id: str,
child_tenant_ids: List[str],
session_id: str
) -> Dict[str, Any]:
"""
Initialize enterprise demo data including parent-child relationships and distribution setup
Args:
parent_tenant_id: Parent tenant ID
child_tenant_ids: List of child tenant IDs
session_id: Demo session ID
Returns:
Dict with initialization results
"""
logger.info("Initializing enterprise demo",
parent_tenant_id=parent_tenant_id,
child_tenant_ids=child_tenant_ids)
try:
# Step 1: Set up parent-child tenant relationships
await self._setup_parent_child_relationships(
parent_tenant_id=parent_tenant_id,
child_tenant_ids=child_tenant_ids
)
# Step 2: Initialize distribution for the parent
await self._setup_distribution_for_enterprise(
parent_tenant_id=parent_tenant_id,
child_tenant_ids=child_tenant_ids
)
# Step 3: Generate initial internal transfer orders
await self._generate_initial_internal_transfers(
parent_tenant_id=parent_tenant_id,
child_tenant_ids=child_tenant_ids
)
logger.info("Enterprise demo initialized successfully",
parent_tenant_id=parent_tenant_id)
return {
'status': 'success',
'parent_tenant_id': parent_tenant_id,
'child_tenant_count': len(child_tenant_ids),
'session_id': session_id,
'initialized_at': datetime.utcnow().isoformat()
}
except Exception as e:
logger.error(f"Error initializing enterprise demo: {e}", exc_info=True)
raise
async def _setup_parent_child_relationships(
self,
parent_tenant_id: str,
child_tenant_ids: List[str]
):
"""Set up parent-child tenant relationships"""
try:
for child_id in child_tenant_ids:
# Update child tenant to have parent reference
await self.tenant_client.update_tenant(
tenant_id=child_id,
updates={
'parent_tenant_id': parent_tenant_id,
'tenant_type': 'child',
'hierarchy_path': f"{parent_tenant_id}.{child_id}"
}
)
# Update parent tenant
await self.tenant_client.update_tenant(
tenant_id=parent_tenant_id,
updates={
'tenant_type': 'parent',
'hierarchy_path': parent_tenant_id # Root path
}
)
logger.info("Parent-child relationships established",
parent_tenant_id=parent_tenant_id,
child_count=len(child_tenant_ids))
except Exception as e:
logger.error(f"Error setting up parent-child relationships: {e}", exc_info=True)
raise
async def _setup_distribution_for_enterprise(
self,
parent_tenant_id: str,
child_tenant_ids: List[str]
):
"""Set up distribution routes and schedules for the enterprise network"""
try:
# In a real implementation, this would call the distribution service
# to set up default delivery routes and schedules between parent and children
logger.info("Setting up distribution for enterprise network",
parent_tenant_id=parent_tenant_id,
child_count=len(child_tenant_ids))
except Exception as e:
logger.error(f"Error setting up distribution: {e}", exc_info=True)
raise
async def _generate_initial_internal_transfers(
self,
parent_tenant_id: str,
child_tenant_ids: List[str]
):
"""Generate initial internal transfer orders for demo"""
try:
for child_id in child_tenant_ids:
# Generate initial internal purchase orders from parent to child
# This would typically be done through the procurement service
logger.info("Generated initial internal transfer order",
parent_tenant_id=parent_tenant_id,
child_tenant_id=child_id)
except Exception as e:
logger.error(f"Error generating initial internal transfers: {e}", exc_info=True)
raise

305
services/orchestrator/app/services/orchestration_notification_service.py
View File

@@ -1,88 +1,60 @@
"""
Orchestration Notification Service
Orchestration Notification Service - Simplified
Emits informational notifications for orchestration events:
- orchestration_run_started: When an orchestration run begins
- orchestration_run_completed: When an orchestration run finishes successfully
- action_created: When the orchestrator creates an action (PO, batch, adjustment)
These are NOTIFICATIONS (not alerts) - informational state changes that don't require user action.
Emits minimal events using EventPublisher.
All enrichment handled by alert_processor.
"""
import logging
from datetime import datetime, timezone
from typing import Optional, Dict, Any, List
from sqlalchemy.orm import Session
from typing import Optional, Dict, Any
from uuid import UUID
import structlog
from shared.schemas.event_classification import RawEvent, EventClass, EventDomain
from shared.alerts.base_service import BaseAlertService
from shared.messaging import UnifiedEventPublisher
logger = structlog.get_logger()
logger = logging.getLogger(__name__)
class OrchestrationNotificationService(BaseAlertService):
class OrchestrationNotificationService:
"""
Service for emitting orchestration notifications (informational state changes).
Service for emitting orchestration notifications using EventPublisher.
"""
def __init__(self, rabbitmq_url: str = None):
super().__init__(service_name="orchestrator", rabbitmq_url=rabbitmq_url)
def __init__(self, event_publisher: UnifiedEventPublisher):
self.publisher = event_publisher
async def emit_orchestration_run_started_notification(
self,
db: Session,
tenant_id: str,
tenant_id: UUID,
run_id: str,
run_type: str, # 'scheduled', 'manual', 'triggered'
scope: str, # 'full', 'inventory_only', 'production_only'
) -> None:
"""
Emit notification when an orchestration run starts.
Args:
db: Database session
tenant_id: Tenant ID
run_id: Orchestration run ID
run_type: Type of run
scope: Scope of run
"""
try:
event = RawEvent(
tenant_id=tenant_id,
event_class=EventClass.NOTIFICATION,
event_domain=EventDomain.OPERATIONS,
event_type="orchestration_run_started",
title="Orchestration Started",
message=f"AI orchestration run started ({run_type}, scope: {scope})",
service="orchestrator",
event_metadata={
"run_id": run_id,
"run_type": run_type,
"scope": scope,
"started_at": datetime.now(timezone.utc).isoformat(),
},
timestamp=datetime.now(timezone.utc),
)
metadata = {
"run_id": run_id,
"run_type": run_type,
"scope": scope,
"started_at": datetime.now(timezone.utc).isoformat(),
}
await self.publish_item(tenant_id, event.dict(), item_type="notification")
await self.publisher.publish_notification(
event_type="operations.orchestration_run_started",
tenant_id=tenant_id,
data=metadata
)
logger.info(
f"Orchestration run started notification emitted: {run_id}",
extra={"tenant_id": tenant_id, "run_id": run_id}
)
except Exception as e:
logger.error(
f"Failed to emit orchestration run started notification: {e}",
extra={"tenant_id": tenant_id, "run_id": run_id},
exc_info=True,
)
logger.info(
"orchestration_run_started_notification_emitted",
tenant_id=str(tenant_id),
run_id=run_id
)
async def emit_orchestration_run_completed_notification(
self,
db: Session,
tenant_id: str,
tenant_id: UUID,
run_id: str,
duration_seconds: float,
actions_created: int,
@@ -91,63 +63,39 @@ class OrchestrationNotificationService(BaseAlertService):
) -> None:
"""
Emit notification when an orchestration run completes.
Args:
db: Database session
tenant_id: Tenant ID
run_id: Orchestration run ID
duration_seconds: Run duration
actions_created: Total actions created
actions_by_type: Breakdown of actions by type
status: Run status (success, partial, failed)
"""
try:
# Build message with action summary
if actions_created == 0:
message = "No actions needed"
else:
action_summary = ", ".join([f"{count} {action_type}" for action_type, count in actions_by_type.items()])
message = f"Created {actions_created} actions: {action_summary}"
# Build message with action summary
if actions_created == 0:
action_summary = "No actions needed"
else:
action_summary = ", ".join([f"{count} {action_type}" for action_type, count in actions_by_type.items()])
message += f" ({duration_seconds:.1f}s)"
metadata = {
"run_id": run_id,
"status": status,
"duration_seconds": float(duration_seconds),
"actions_created": actions_created,
"actions_by_type": actions_by_type,
"action_summary": action_summary,
"completed_at": datetime.now(timezone.utc).isoformat(),
}
event = RawEvent(
tenant_id=tenant_id,
event_class=EventClass.NOTIFICATION,
event_domain=EventDomain.OPERATIONS,
event_type="orchestration_run_completed",
title=f"Orchestration Completed: {status.title()}",
message=message,
service="orchestrator",
event_metadata={
"run_id": run_id,
"status": status,
"duration_seconds": duration_seconds,
"actions_created": actions_created,
"actions_by_type": actions_by_type,
"completed_at": datetime.now(timezone.utc).isoformat(),
},
timestamp=datetime.now(timezone.utc),
)
await self.publisher.publish_notification(
event_type="operations.orchestration_run_completed",
tenant_id=tenant_id,
data=metadata
)
await self.publish_item(tenant_id, event.dict(), item_type="notification")
logger.info(
f"Orchestration run completed notification emitted: {run_id} ({actions_created} actions)",
extra={"tenant_id": tenant_id, "run_id": run_id}
)
except Exception as e:
logger.error(
f"Failed to emit orchestration run completed notification: {e}",
extra={"tenant_id": tenant_id, "run_id": run_id},
exc_info=True,
)
logger.info(
"orchestration_run_completed_notification_emitted",
tenant_id=str(tenant_id),
run_id=run_id,
actions_created=actions_created
)
async def emit_action_created_notification(
self,
db: Session,
tenant_id: str,
tenant_id: UUID,
run_id: str,
action_id: str,
action_type: str, # 'purchase_order', 'production_batch', 'inventory_adjustment'
@@ -157,70 +105,33 @@ class OrchestrationNotificationService(BaseAlertService):
) -> None:
"""
Emit notification when the orchestrator creates an action.
Args:
db: Database session
tenant_id: Tenant ID
run_id: Orchestration run ID
action_id: Created action ID
action_type: Type of action
action_details: Action-specific details
reason: Reason for creating action
estimated_impact: Estimated impact (optional)
"""
try:
# Build title and message based on action type
if action_type == "purchase_order":
title = f"Purchase Order Created: {action_details.get('supplier_name', 'Unknown')}"
message = f"Ordered {action_details.get('items_count', 0)} items - {reason}"
elif action_type == "production_batch":
title = f"Production Batch Scheduled: {action_details.get('product_name', 'Unknown')}"
message = f"Scheduled {action_details.get('quantity', 0)} {action_details.get('unit', 'units')} - {reason}"
elif action_type == "inventory_adjustment":
title = f"Inventory Adjustment: {action_details.get('ingredient_name', 'Unknown')}"
message = f"Adjusted by {action_details.get('quantity', 0)} {action_details.get('unit', 'units')} - {reason}"
else:
title = f"Action Created: {action_type}"
message = reason
metadata = {
"run_id": run_id,
"action_id": action_id,
"action_type": action_type,
"action_details": action_details,
"reason": reason,
"estimated_impact": estimated_impact,
"created_at": datetime.now(timezone.utc).isoformat(),
}
event = RawEvent(
tenant_id=tenant_id,
event_class=EventClass.NOTIFICATION,
event_domain=EventDomain.OPERATIONS,
event_type="action_created",
title=title,
message=message,
service="orchestrator",
event_metadata={
"run_id": run_id,
"action_id": action_id,
"action_type": action_type,
"action_details": action_details,
"reason": reason,
"estimated_impact": estimated_impact,
"created_at": datetime.now(timezone.utc).isoformat(),
},
timestamp=datetime.now(timezone.utc),
)
await self.publisher.publish_notification(
event_type="operations.action_created",
tenant_id=tenant_id,
data=metadata
)
await self.publish_item(tenant_id, event.dict(), item_type="notification")
logger.info(
f"Action created notification emitted: {action_type} - {action_id}",
extra={"tenant_id": tenant_id, "action_id": action_id}
)
except Exception as e:
logger.error(
f"Failed to emit action created notification: {e}",
extra={"tenant_id": tenant_id, "action_id": action_id},
exc_info=True,
)
logger.info(
"action_created_notification_emitted",
tenant_id=str(tenant_id),
action_id=action_id,
action_type=action_type
)
async def emit_action_completed_notification(
self,
db: Session,
tenant_id: str,
tenant_id: UUID,
action_id: str,
action_type: str,
action_status: str, # 'approved', 'completed', 'rejected', 'cancelled'
@@ -228,48 +139,24 @@ class OrchestrationNotificationService(BaseAlertService):
) -> None:
"""
Emit notification when an orchestrator action is completed/resolved.
Args:
db: Database session
tenant_id: Tenant ID
action_id: Action ID
action_type: Type of action
action_status: Final status
completed_by: Who completed it (optional)
"""
try:
message = f"{action_type.replace('_', ' ').title()}: {action_status}"
if completed_by:
message += f" by {completed_by}"
metadata = {
"action_id": action_id,
"action_type": action_type,
"action_status": action_status,
"completed_by": completed_by,
"completed_at": datetime.now(timezone.utc).isoformat(),
}
event = RawEvent(
tenant_id=tenant_id,
event_class=EventClass.NOTIFICATION,
event_domain=EventDomain.OPERATIONS,
event_type="action_completed",
title=f"Action {action_status.title()}",
message=message,
service="orchestrator",
event_metadata={
"action_id": action_id,
"action_type": action_type,
"action_status": action_status,
"completed_by": completed_by,
"completed_at": datetime.now(timezone.utc).isoformat(),
},
timestamp=datetime.now(timezone.utc),
)
await self.publisher.publish_notification(
event_type="operations.action_completed",
tenant_id=tenant_id,
data=metadata
)
await self.publish_item(tenant_id, event.dict(), item_type="notification")
logger.info(
f"Action completed notification emitted: {action_id} ({action_status})",
extra={"tenant_id": tenant_id, "action_id": action_id}
)
except Exception as e:
logger.error(
f"Failed to emit action completed notification: {e}",
extra={"tenant_id": tenant_id, "action_id": action_id},
exc_info=True,
)
logger.info(
"action_completed_notification_emitted",
tenant_id=str(tenant_id),
action_id=action_id,
action_status=action_status
)

225
services/orchestrator/app/services/orchestrator_service.py
View File

@@ -1,9 +1,9 @@
"""
Orchestrator Scheduler Service - REFACTORED
Coordinates daily auto-generation workflow: Forecasting → Production → Procurement → Notifications
Coordinates daily auto-generation workflow: Forecasting → Production → Procurement
CHANGES FROM ORIGINAL:
- Removed all TODO/stub code
- Updated to use new EventPublisher pattern for all messaging
- Integrated OrchestrationSaga for error handling and compensation
- Added circuit breakers for all service calls
- Implemented real Forecasting Service integration
@@ -21,7 +21,8 @@ from typing import List, Dict, Any, Optional
import structlog
from apscheduler.triggers.cron import CronTrigger
from shared.alerts.base_service import BaseAlertService
# Updated imports - removed old alert system
from shared.messaging import UnifiedEventPublisher
from shared.clients.forecast_client import ForecastServiceClient
from shared.clients.production_client import ProductionServiceClient
from shared.clients.procurement_client import ProcurementServiceClient
@@ -40,14 +41,15 @@ from app.services.orchestration_saga import OrchestrationSaga
logger = structlog.get_logger()
class OrchestratorSchedulerService(BaseAlertService):
class OrchestratorSchedulerService:
"""
Orchestrator Service extending BaseAlertService
Orchestrator Service using EventPublisher for messaging
Handles automated daily orchestration of forecasting, production, and procurement
"""
def __init__(self, config):
super().__init__(config)
def __init__(self, event_publisher: UnifiedEventPublisher, config):
self.publisher = event_publisher
self.config = config
# Service clients
self.forecast_client = ForecastServiceClient(config, "orchestrator-service")
@@ -98,47 +100,149 @@ class OrchestratorSchedulerService(BaseAlertService):
success_threshold=2
)
def setup_scheduled_checks(self):
async def emit_orchestration_run_started(
self,
tenant_id: uuid.UUID,
run_id: str,
run_type: str, # 'scheduled', 'manual', 'triggered'
scope: str, # 'full', 'inventory_only', 'production_only'
):
"""
Configure scheduled orchestration jobs
Runs daily at 5:30 AM (configured via ORCHESTRATION_SCHEDULE)
Emit notification when an orchestration run starts.
"""
# Parse cron schedule from config (default: "30 5 * * *" = 5:30 AM daily)
cron_parts = settings.ORCHESTRATION_SCHEDULE.split()
if len(cron_parts) == 5:
minute, hour, day, month, day_of_week = cron_parts
else:
# Fallback to default
minute, hour, day, month, day_of_week = "30", "5", "*", "*", "*"
metadata = {
"run_id": run_id,
"run_type": run_type,
"scope": scope,
"started_at": datetime.now(timezone.utc).isoformat(),
}
# Schedule daily orchestration
self.scheduler.add_job(
func=self.run_daily_orchestration,
trigger=CronTrigger(
minute=minute,
hour=hour,
day=day,
month=month,
day_of_week=day_of_week
),
id="daily_orchestration",
name="Daily Orchestration (Forecasting → Production → Procurement)",
misfire_grace_time=300, # 5 minutes grace period
max_instances=1 # Only one instance running at a time
await self.publisher.publish_notification(
event_type="operations.orchestration_run_started",
tenant_id=tenant_id,
data=metadata
)
logger.info("Orchestrator scheduler configured",
schedule=settings.ORCHESTRATION_SCHEDULE)
logger.info(
"orchestration_run_started_notification_emitted",
tenant_id=str(tenant_id),
run_id=run_id
)
async def emit_orchestration_run_completed(
self,
tenant_id: uuid.UUID,
run_id: str,
duration_seconds: float,
actions_created: int,
actions_by_type: Dict[str, int], # e.g., {'purchase_order': 2, 'production_batch': 3}
status: str = "success",
):
"""
Emit notification when an orchestration run completes.
"""
# Build message with action summary
if actions_created == 0:
action_summary = "No actions needed"
else:
action_summary = ", ".join([f"{count} {action_type}" for action_type, count in actions_by_type.items()])
metadata = {
"run_id": run_id,
"status": status,
"duration_seconds": float(duration_seconds),
"actions_created": actions_created,
"actions_by_type": actions_by_type,
"action_summary": action_summary,
"completed_at": datetime.now(timezone.utc).isoformat(),
}
await self.publisher.publish_notification(
event_type="operations.orchestration_run_completed",
tenant_id=tenant_id,
data=metadata
)
logger.info(
"orchestration_run_completed_notification_emitted",
tenant_id=str(tenant_id),
run_id=run_id,
actions_created=actions_created
)
async def emit_action_created_notification(
self,
tenant_id: uuid.UUID,
run_id: str,
action_id: str,
action_type: str, # 'purchase_order', 'production_batch', 'inventory_adjustment'
action_details: Dict[str, Any], # Type-specific details
reason: str,
estimated_impact: Optional[Dict[str, Any]] = None,
):
"""
Emit notification when the orchestrator creates an action.
"""
metadata = {
"run_id": run_id,
"action_id": action_id,
"action_type": action_type,
"action_details": action_details,
"reason": reason,
"estimated_impact": estimated_impact,
"created_at": datetime.now(timezone.utc).isoformat(),
}
await self.publisher.publish_notification(
event_type="operations.action_created",
tenant_id=tenant_id,
data=metadata
)
logger.info(
"action_created_notification_emitted",
tenant_id=str(tenant_id),
action_id=action_id,
action_type=action_type
)
async def emit_action_completed_notification(
self,
tenant_id: uuid.UUID,
action_id: str,
action_type: str,
action_status: str, # 'approved', 'completed', 'rejected', 'cancelled'
completed_by: Optional[str] = None,
):
"""
Emit notification when an orchestrator action is completed/resolved.
"""
metadata = {
"action_id": action_id,
"action_type": action_type,
"action_status": action_status,
"completed_by": completed_by,
"completed_at": datetime.now(timezone.utc).isoformat(),
}
await self.publisher.publish_notification(
event_type="operations.action_completed",
tenant_id=tenant_id,
data=metadata
)
logger.info(
"action_completed_notification_emitted",
tenant_id=str(tenant_id),
action_id=action_id,
action_status=action_status
)
async def run_daily_orchestration(self):
"""
Main orchestration workflow - runs daily
Executes for all active tenants in parallel (with limits)
"""
if not self.is_leader:
logger.debug("Not leader, skipping orchestration")
return
if not settings.ORCHESTRATION_ENABLED:
logger.info("Orchestration disabled via config")
return
@@ -188,7 +292,7 @@ class OrchestratorSchedulerService(BaseAlertService):
logger.info("Starting orchestration for tenant", tenant_id=str(tenant_id))
# Create orchestration run record
async with self.db_manager.get_session() as session:
async with self.config.database_manager.get_session() as session:
repo = OrchestrationRunRepository(session)
run_number = await repo.generate_run_number()
@@ -204,6 +308,14 @@ class OrchestratorSchedulerService(BaseAlertService):
run_id = run.id
try:
# Emit orchestration started event
await self.emit_orchestration_run_started(
tenant_id=tenant_id,
run_id=str(run_id),
run_type='scheduled',
scope='full'
)
# Set timeout for entire tenant orchestration
async with asyncio.timeout(settings.TENANT_TIMEOUT_SECONDS):
# Execute orchestration using Saga pattern
@@ -241,6 +353,16 @@ class OrchestratorSchedulerService(BaseAlertService):
result
)
# Emit orchestration completed event
await self.emit_orchestration_run_completed(
tenant_id=tenant_id,
run_id=str(run_id),
duration_seconds=result.get('duration_seconds', 0),
actions_created=result.get('total_actions', 0),
actions_by_type=result.get('actions_by_type', {}),
status='success'
)
logger.info("Tenant orchestration completed successfully",
tenant_id=str(tenant_id), run_id=str(run_id))
return True
@@ -250,6 +372,17 @@ class OrchestratorSchedulerService(BaseAlertService):
run_id,
result.get('error', 'Saga execution failed')
)
# Emit orchestration failed event
await self.emit_orchestration_run_completed(
tenant_id=tenant_id,
run_id=str(run_id),
duration_seconds=result.get('duration_seconds', 0),
actions_created=0,
actions_by_type={},
status='failed'
)
return False
except asyncio.TimeoutError:
@@ -318,7 +451,7 @@ class OrchestratorSchedulerService(BaseAlertService):
run_id: Orchestration run ID
saga_result: Result from saga execution
"""
async with self.db_manager.get_session() as session:
async with self.config.database_manager.get_session() as session:
repo = OrchestrationRunRepository(session)
run = await repo.get_run_by_id(run_id)
@@ -489,7 +622,7 @@ class OrchestratorSchedulerService(BaseAlertService):
async def _mark_orchestration_failed(self, run_id: uuid.UUID, error_message: str):
"""Mark orchestration run as failed"""
async with self.db_manager.get_session() as session:
async with self.config.database_manager.get_session() as session:
repo = OrchestrationRunRepository(session)
run = await repo.get_run_by_id(run_id)
@@ -535,6 +668,16 @@ class OrchestratorSchedulerService(BaseAlertService):
'message': 'Orchestration completed' if success else 'Orchestration failed'
}
async def start(self):
"""Start the orchestrator scheduler service"""
logger.info("OrchestratorSchedulerService started")
# Add any initialization logic here if needed
async def stop(self):
"""Stop the orchestrator scheduler service"""
logger.info("OrchestratorSchedulerService stopped")
# Add any cleanup logic here if needed
def get_circuit_breaker_stats(self) -> Dict[str, Any]:
"""Get circuit breaker statistics for monitoring"""
return {
@@ -545,4 +688,4 @@ class OrchestratorSchedulerService(BaseAlertService):
'inventory_service': self.inventory_breaker.get_stats(),
'suppliers_service': self.suppliers_breaker.get_stats(),
'recipes_service': self.recipes_breaker.get_stats()
}
}

32
services/orchestrator/scripts/demo/seed_demo_orchestration_runs.py
View File

@@ -172,11 +172,24 @@ def generate_reasoning_metadata(
This creates structured reasoning data that the alert processor can use to provide
context when showing AI reasoning to users.
"""
# Calculate aggregate metrics for dashboard display
# Dashboard expects these fields at the top level of the 'reasoning' object
critical_items_count = random.randint(1, 3) if purchase_orders_created > 0 else 0
financial_impact_eur = random.randint(200, 1500) if critical_items_count > 0 else 0
min_depletion_hours = random.uniform(6.0, 48.0) if critical_items_count > 0 else 0
reasoning_metadata = {
'reasoning': {
'type': 'daily_orchestration_summary',
'timestamp': datetime.now(timezone.utc).isoformat(),
# TOP-LEVEL FIELDS - Dashboard reads these directly (dashboard_service.py:411-413)
'critical_items_count': critical_items_count,
'financial_impact_eur': round(financial_impact_eur, 2),
'min_depletion_hours': round(min_depletion_hours, 1),
'time_until_consequence_hours': round(min_depletion_hours, 1),
'affected_orders': random.randint(0, 5) if critical_items_count > 0 else 0,
'summary': 'Daily orchestration run completed successfully',
# Keep existing details structure for backward compatibility
'details': {
'forecasting': {
'forecasts_created': forecasts_generated,
@@ -419,11 +432,20 @@ async def generate_orchestration_for_tenant(
notification_error = error_scenario["message"]
# Generate results summary
forecasts_generated = random.randint(5, 15)
production_batches_created = random.randint(3, 8)
procurement_plans_created = random.randint(2, 6)
purchase_orders_created = random.randint(1, 4)
notifications_sent = random.randint(10, 25)
# For professional tenant, use realistic fixed counts that match PO seed data
if tenant_id == DEMO_TENANT_PROFESSIONAL:
forecasts_generated = 12 # Realistic daily forecast count
production_batches_created = 6 # Realistic batch count
procurement_plans_created = 3 # 3 procurement plans
purchase_orders_created = 18 # Total POs including 9 delivery POs (PO #11-18)
notifications_sent = 24 # Realistic notification count
else:
# Enterprise tenant can keep random values
forecasts_generated = random.randint(5, 15)
production_batches_created = random.randint(3, 8)
procurement_plans_created = random.randint(2, 6)
purchase_orders_created = random.randint(1, 4)
notifications_sent = random.randint(10, 25)
# Generate performance metrics for completed runs
fulfillment_rate = None