docs: Add delivery tracking service documentation to orchestrator README
This commit is contained in:
Urtzi Alfaro
@@ -903,4 +903,318 @@ New metrics available for dashboards:
|
||||
|
||||
---
|
||||
|
||||
## Delivery Tracking Service
|
||||
|
||||
### Overview
|
||||
|
||||
The Delivery Tracking Service provides **proactive monitoring** of expected deliveries with time-based alert generation. Unlike reactive event-driven alerts, this service periodically checks delivery windows against current time to generate predictive and overdue notifications.
|
||||
|
||||
**Key Capabilities**:
|
||||
- Proactive "arriving soon" alerts (T-2 hours before delivery)
|
||||
- Overdue delivery detection (30 min after window)
|
||||
- Incomplete receipt reminders (2 hours after window)
|
||||
- Integration with Procurement Service for PO delivery schedules
|
||||
- Automatic alert resolution when deliveries are received
|
||||
|
||||
### Cronjob Configuration
|
||||
|
||||
```yaml
|
||||
# infrastructure/kubernetes/base/cronjobs/delivery-tracking-cronjob.yaml
|
||||
apiVersion: batch/v1
|
||||
kind: CronJob
|
||||
metadata:
|
||||
name: delivery-tracking-cronjob
|
||||
spec:
|
||||
schedule: "30 * * * *" # Hourly at minute 30
|
||||
concurrencyPolicy: Forbid
|
||||
successfulJobsHistoryLimit: 3
|
||||
failedJobsHistoryLimit: 3
|
||||
jobTemplate:
|
||||
spec:
|
||||
activeDeadlineSeconds: 1800 # 30 minutes timeout
|
||||
template:
|
||||
spec:
|
||||
containers:
|
||||
- name: delivery-tracking
|
||||
image: orchestrator-service:latest
|
||||
command: ["python3", "-m", "app.services.delivery_tracking_service"]
|
||||
resources:
|
||||
requests:
|
||||
memory: "128Mi"
|
||||
cpu: "50m"
|
||||
limits:
|
||||
memory: "256Mi"
|
||||
cpu: "100m"
|
||||
```
|
||||
|
||||
**Schedule Rationale**: Hourly checks provide timely alerts without excessive polling. The :30 offset avoids collision with priority recalculation cronjob (:15).
|
||||
|
||||
### Delivery Alert Lifecycle
|
||||
|
||||
```
|
||||
Purchase Order Approved (t=0)
|
||||
↓
|
||||
System publishes DELIVERY_SCHEDULED (informational event)
|
||||
↓
|
||||
[Time passes - no alerts]
|
||||
↓
|
||||
T-2 hours before expected delivery time
|
||||
↓
|
||||
CronJob detects: now >= (expected_delivery - 2 hours)
|
||||
↓
|
||||
Generate DELIVERY_ARRIVING_SOON alert
|
||||
- Priority: 70 (important)
|
||||
- Class: action_needed
|
||||
- Action Queue: Yes
|
||||
- Smart Action: Open StockReceiptModal in create mode
|
||||
↓
|
||||
[Delivery window arrives]
|
||||
↓
|
||||
Expected delivery time + 30 minutes (grace period)
|
||||
↓
|
||||
CronJob detects: now >= (delivery_window_end + 30 min)
|
||||
↓
|
||||
Generate DELIVERY_OVERDUE alert
|
||||
- Priority: 95 (critical)
|
||||
- Class: critical
|
||||
- Escalation: Time-sensitive
|
||||
- Smart Action: Contact supplier + Open receipt modal
|
||||
↓
|
||||
Expected delivery time + 2 hours
|
||||
↓
|
||||
CronJob detects: still no stock receipt
|
||||
↓
|
||||
Generate STOCK_RECEIPT_INCOMPLETE alert
|
||||
- Priority: 80 (important)
|
||||
- Class: action_needed
|
||||
- Smart Action: Open existing receipt in edit mode
|
||||
```
|
||||
|
||||
**Auto-Resolution**: All delivery alerts are automatically resolved when:
|
||||
- Stock receipt is confirmed (`onConfirm` in StockReceiptModal)
|
||||
- Event `delivery.received` is published
|
||||
- Alert Processor marks alerts as `resolved` with reason: "Delivery received"
|
||||
|
||||
### Service Methods
|
||||
|
||||
#### `check_expected_deliveries()` - Main Entry Point
|
||||
```python
|
||||
async def check_expected_deliveries(tenant_id: str) -> None:
|
||||
"""
|
||||
Hourly job to check all purchase orders with expected deliveries.
|
||||
|
||||
Queries Procurement Service for POs with:
|
||||
- status: approved or sent
|
||||
- expected_delivery_date: within next 48 hours or past due
|
||||
|
||||
For each PO, checks:
|
||||
1. Arriving soon? (T-2h) → _send_arriving_soon_alert()
|
||||
2. Overdue? (T+30m) → _send_overdue_alert()
|
||||
3. Receipt incomplete? (T+2h) → _send_receipt_incomplete_alert()
|
||||
"""
|
||||
```
|
||||
|
||||
#### `_send_arriving_soon_alert(po: PurchaseOrder)` - Proactive Warning
|
||||
```python
|
||||
async def _send_arriving_soon_alert(po: PurchaseOrder) -> None:
|
||||
"""
|
||||
Generates alert 2 hours before expected delivery.
|
||||
|
||||
Alert Details:
|
||||
- event_type: DELIVERY_ARRIVING_SOON
|
||||
- priority_score: 70 (important)
|
||||
- alert_class: action_needed
|
||||
- domain: supply_chain
|
||||
- smart_action: open_stock_receipt_modal (create mode)
|
||||
|
||||
Context Enrichment:
|
||||
- PO ID, supplier name, expected items count
|
||||
- Delivery window (start/end times)
|
||||
- Preparation checklist (clear receiving area, verify items)
|
||||
"""
|
||||
```
|
||||
|
||||
#### `_send_overdue_alert(po: PurchaseOrder)` - Critical Escalation
|
||||
```python
|
||||
async def _send_overdue_alert(po: PurchaseOrder) -> None:
|
||||
"""
|
||||
Generates critical alert 30 minutes after delivery window.
|
||||
|
||||
Alert Details:
|
||||
- event_type: DELIVERY_OVERDUE
|
||||
- priority_score: 95 (critical)
|
||||
- alert_class: critical
|
||||
- domain: supply_chain
|
||||
- smart_actions: [contact_supplier, open_receipt_modal]
|
||||
|
||||
Business Impact:
|
||||
- Production delays if ingredients missing
|
||||
- Spoilage risk if perishables delayed
|
||||
- Customer order fulfillment risk
|
||||
|
||||
Suggested Actions:
|
||||
1. Contact supplier immediately
|
||||
2. Check for delivery rescheduling
|
||||
3. Activate backup supplier if needed
|
||||
4. Adjust production plan if ingredients critical
|
||||
"""
|
||||
```
|
||||
|
||||
#### `_send_receipt_incomplete_alert(po: PurchaseOrder)` - Reminder
|
||||
```python
|
||||
async def _send_receipt_incomplete_alert(po: PurchaseOrder) -> None:
|
||||
"""
|
||||
Generates reminder 2 hours after delivery window if no receipt.
|
||||
|
||||
Alert Details:
|
||||
- event_type: STOCK_RECEIPT_INCOMPLETE
|
||||
- priority_score: 80 (important)
|
||||
- alert_class: action_needed
|
||||
- domain: inventory
|
||||
- smart_action: open_stock_receipt_modal (edit mode if draft exists)
|
||||
|
||||
Checks:
|
||||
- Stock receipts table for PO ID
|
||||
- If draft exists → Edit mode with pre-filled data
|
||||
- If no draft → Create mode
|
||||
|
||||
HACCP Compliance Note:
|
||||
- Food safety requires timely receipt documentation
|
||||
- Expiration date tracking depends on receipt
|
||||
- Incomplete receipts block lot tracking
|
||||
"""
|
||||
```
|
||||
|
||||
### Integration with Alert System
|
||||
|
||||
**Publishing Flow**:
|
||||
```python
|
||||
# services/orchestrator/app/services/delivery_tracking_service.py
|
||||
from shared.clients.alerts_client import AlertsClient
|
||||
|
||||
alerts_client = AlertsClient(service_name="orchestrator")
|
||||
|
||||
await alerts_client.publish_alert(
|
||||
tenant_id=tenant_id,
|
||||
event_type="DELIVERY_OVERDUE",
|
||||
entity_type="purchase_order",
|
||||
entity_id=po.id,
|
||||
severity="critical",
|
||||
priority_score=95,
|
||||
context={
|
||||
"po_number": po.po_number,
|
||||
"supplier_name": po.supplier.name,
|
||||
"expected_delivery": po.expected_delivery_date.isoformat(),
|
||||
"delay_minutes": delay_in_minutes,
|
||||
"items_count": len(po.line_items)
|
||||
}
|
||||
)
|
||||
```
|
||||
|
||||
**Alert Processing**:
|
||||
1. Delivery Tracking Service → RabbitMQ (supply_chain.alerts exchange)
|
||||
2. Alert Processor consumes message
|
||||
3. Full enrichment pipeline (Tier 1 - ALERTS)
|
||||
4. Smart action handler assigned (open_stock_receipt_modal)
|
||||
5. Store in PostgreSQL with priority_score
|
||||
6. Publish to Redis Pub/Sub → Gateway SSE
|
||||
7. Frontend `useSupplyChainNotifications()` hook receives alert
|
||||
8. UnifiedActionQueueCard displays in "Urgent" section
|
||||
9. User clicks → StockReceiptModal opens with PO context
|
||||
|
||||
### Architecture Decision: Why CronJob Over Event System?
|
||||
|
||||
**Question**: Could we replace this cronjob with scheduled events?
|
||||
|
||||
**Answer**: ❌ No - CronJob is the right tool for this job.
|
||||
|
||||
#### Comparison Matrix
|
||||
|
||||
| Feature | Event System | CronJob | Best Choice |
|
||||
|---------|--------------|---------|-------------|
|
||||
| Time-based alerts | ❌ Requires complex scheduling | ✅ Natural fit | **CronJob** |
|
||||
| Predictive alerts | ❌ Must schedule at PO creation | ✅ Dynamic checks | **CronJob** |
|
||||
| Delivery window changes | ❌ Need to reschedule events | ✅ Adapts automatically | **CronJob** |
|
||||
| System restarts | ❌ Lose scheduled events | ✅ Persistent schedule | **CronJob** |
|
||||
| Complexity | ❌ High (event scheduler needed) | ✅ Low (periodic check) | **CronJob** |
|
||||
| Maintenance | ❌ Many scheduled events | ✅ Single job | **CronJob** |
|
||||
|
||||
**Event System Challenges**:
|
||||
- Would need to schedule 3 events per PO at approval time:
|
||||
1. "arriving_soon" event at (delivery_time - 2h)
|
||||
2. "overdue" event at (delivery_time + 30m)
|
||||
3. "incomplete" event at (delivery_time + 2h)
|
||||
- Requires persistent event scheduler (like Celery Beat)
|
||||
- Rescheduling when delivery dates change is complex
|
||||
- System restarts would lose in-memory scheduled events
|
||||
- Essentially rebuilding cron functionality
|
||||
|
||||
**CronJob Advantages**:
|
||||
- ✅ Simple periodic check against current time
|
||||
- ✅ Adapts to delivery date changes automatically
|
||||
- ✅ No state management for scheduled events
|
||||
- ✅ Easy to adjust alert timing thresholds
|
||||
- ✅ Built-in Kubernetes scheduling and monitoring
|
||||
- ✅ Resource-efficient (runs 1 minute every hour)
|
||||
|
||||
**Verdict**: Periodic polling is more maintainable than scheduled events for time-based conditions.
|
||||
|
||||
### Monitoring & Observability
|
||||
|
||||
**Metrics Tracked**:
|
||||
- `delivery_tracking_job_duration_seconds` - Execution time
|
||||
- `delivery_alerts_generated_total{type}` - Counter by alert type
|
||||
- `deliveries_checked_total` - Total POs scanned
|
||||
- `delivery_tracking_errors_total` - Failure rate
|
||||
|
||||
**Logs**:
|
||||
```
|
||||
[2025-11-26 14:30:02] INFO: Delivery tracking job started for tenant abc123
|
||||
[2025-11-26 14:30:03] INFO: Found 12 purchase orders with upcoming deliveries
|
||||
[2025-11-26 14:30:03] INFO: Generated DELIVERY_ARRIVING_SOON for PO-2025-043 (delivery in 1h 45m)
|
||||
[2025-11-26 14:30:03] WARNING: Generated DELIVERY_OVERDUE for PO-2025-041 (45 minutes late)
|
||||
[2025-11-26 14:30:04] INFO: Delivery tracking job completed in 2.3s
|
||||
```
|
||||
|
||||
**Alerting** (for Ops team):
|
||||
- Job fails 3 times consecutively → Page on-call engineer
|
||||
- Job duration > 5 minutes → Warning (performance degradation)
|
||||
- Zero deliveries checked for 24 hours → Warning (data issue)
|
||||
|
||||
### Testing
|
||||
|
||||
**Unit Tests**:
|
||||
```python
|
||||
# tests/services/test_delivery_tracking_service.py
|
||||
async def test_arriving_soon_alert_generated():
|
||||
# Given: PO with delivery in 1 hour 55 minutes
|
||||
po = create_test_po(expected_delivery=now() + timedelta(hours=1, minutes=55))
|
||||
|
||||
# When: Check deliveries
|
||||
await delivery_tracking_service.check_expected_deliveries(tenant_id)
|
||||
|
||||
# Then: DELIVERY_ARRIVING_SOON alert generated
|
||||
assert_alert_published("DELIVERY_ARRIVING_SOON", po.id)
|
||||
```
|
||||
|
||||
**Integration Tests**:
|
||||
- Test full flow from cronjob → alert → frontend SSE
|
||||
- Verify alert auto-resolution on stock receipt confirmation
|
||||
- Test grace period boundaries (exactly 30 minutes)
|
||||
|
||||
### Performance Characteristics
|
||||
|
||||
**Typical Execution**:
|
||||
- Query Procurement Service: 50-100ms
|
||||
- Filter POs by time windows: 5-10ms
|
||||
- Generate alerts (avg 3 per run): 150-300ms
|
||||
- Total: **200-400ms per tenant**
|
||||
|
||||
**Scaling**:
|
||||
- Single-tenant deployment: Trivial (<1s per hour)
|
||||
- Multi-tenant (100 tenants): ~40s per run (well under 30min timeout)
|
||||
- Multi-tenant (1000+ tenants): Consider tenant sharding across multiple cronjobs
|
||||
|
||||
---
|
||||
|
||||
**Copyright © 2025 Bakery-IA. All rights reserved.**
|
||||
|
||||
Reference in New Issue
Block a user