536 lines
16 KiB
Markdown
536 lines
16 KiB
Markdown
# 📊 Bakery-ia Monitoring System Documentation
|
|
|
|
## 🎯 Overview
|
|
|
|
The bakery-ia platform features a comprehensive, modern monitoring system built on **OpenTelemetry** and **SigNoz**. This documentation provides a complete guide to the monitoring architecture, setup, and usage.
|
|
|
|
## 🚀 Monitoring Architecture
|
|
|
|
### Core Components
|
|
|
|
```mermaid
|
|
graph TD
|
|
A[Microservices] -->|OTLP| B[OpenTelemetry Collector]
|
|
B -->|gRPC| C[SigNoz]
|
|
C --> D[Traces Dashboard]
|
|
C --> E[Metrics Dashboard]
|
|
C --> F[Logs Dashboard]
|
|
C --> G[Alerts]
|
|
```
|
|
|
|
### Technology Stack
|
|
|
|
- **Instrumentation**: OpenTelemetry Python SDK
|
|
- **Protocol**: OTLP (OpenTelemetry Protocol) over gRPC
|
|
- **Backend**: SigNoz (open-source observability platform)
|
|
- **Metrics**: Prometheus-compatible metrics via OTLP
|
|
- **Traces**: Jaeger-compatible tracing via OTLP
|
|
- **Logs**: Structured logging with trace correlation
|
|
|
|
## 📋 Monitoring Coverage
|
|
|
|
### Service Coverage (100%)
|
|
|
|
| Service Category | Services | Monitoring Type | Status |
|
|
|-----------------|----------|----------------|--------|
|
|
| **Critical Services** | auth, orders, sales, external | Base Class | ✅ Monitored |
|
|
| **AI Services** | ai-insights, training | Direct | ✅ Monitored |
|
|
| **Data Services** | inventory, procurement, production, forecasting | Base Class | ✅ Monitored |
|
|
| **Operational Services** | tenant, notification, distribution | Base Class | ✅ Monitored |
|
|
| **Specialized Services** | suppliers, pos, recipes, orchestrator | Base Class | ✅ Monitored |
|
|
| **Infrastructure** | gateway, alert-processor, demo-session | Direct | ✅ Monitored |
|
|
|
|
**Total: 20 services with 100% monitoring coverage**
|
|
|
|
## 🔧 Monitoring Implementation
|
|
|
|
### Implementation Patterns
|
|
|
|
#### 1. Base Class Pattern (16 services)
|
|
|
|
Services using `StandardFastAPIService` inherit comprehensive monitoring:
|
|
|
|
```python
|
|
from shared.service_base import StandardFastAPIService
|
|
|
|
class MyService(StandardFastAPIService):
|
|
def __init__(self):
|
|
super().__init__(
|
|
service_name="my-service",
|
|
app_name="My Service",
|
|
description="Service description",
|
|
version="1.0.0",
|
|
# Monitoring enabled by default
|
|
enable_metrics=True, # ✅ Metrics collection
|
|
enable_tracing=True, # ✅ Distributed tracing
|
|
enable_health_checks=True # ✅ Health endpoints
|
|
)
|
|
```
|
|
|
|
#### 2. Direct Pattern (4 services)
|
|
|
|
Critical services with custom monitoring needs:
|
|
|
|
```python
|
|
# services/ai_insights/app/main.py
|
|
from shared.monitoring.metrics import MetricsCollector, add_metrics_middleware
|
|
from shared.monitoring.system_metrics import SystemMetricsCollector
|
|
|
|
# Initialize metrics collectors
|
|
metrics_collector = MetricsCollector("ai-insights")
|
|
system_metrics = SystemMetricsCollector("ai-insights")
|
|
|
|
# Add middleware
|
|
add_metrics_middleware(app, metrics_collector)
|
|
```
|
|
|
|
### Monitoring Components
|
|
|
|
#### OpenTelemetry Instrumentation
|
|
|
|
```python
|
|
# Automatic instrumentation in base class
|
|
FastAPIInstrumentor.instrument_app(app) # HTTP requests
|
|
HTTPXClientInstrumentor().instrument() # Outgoing HTTP
|
|
RedisInstrumentor().instrument() # Redis operations
|
|
SQLAlchemyInstrumentor().instrument() # Database queries
|
|
```
|
|
|
|
#### Metrics Collection
|
|
|
|
```python
|
|
# Standard metrics automatically collected
|
|
metrics_collector.register_counter("http_requests_total", "Total HTTP requests")
|
|
metrics_collector.register_histogram("http_request_duration", "Request duration")
|
|
metrics_collector.register_gauge("active_requests", "Active requests")
|
|
|
|
# System metrics automatically collected
|
|
system_metrics = SystemMetricsCollector("service-name")
|
|
# → CPU, Memory, Disk I/O, Network I/O, Threads, File Descriptors
|
|
```
|
|
|
|
#### Health Checks
|
|
|
|
```python
|
|
# Automatic health check endpoints
|
|
GET /health # Overall service health
|
|
GET /health/detailed # Detailed health with dependencies
|
|
GET /health/ready # Readiness probe
|
|
GET /health/live # Liveness probe
|
|
```
|
|
|
|
## 📊 Metrics Reference
|
|
|
|
### Standard Metrics (All Services)
|
|
|
|
| Metric Type | Metric Name | Description | Labels |
|
|
|-------------|------------|-------------|--------|
|
|
| **HTTP Metrics** | `{service}_http_requests_total` | Total HTTP requests | method, endpoint, status_code |
|
|
| **HTTP Metrics** | `{service}_http_request_duration_seconds` | Request duration histogram | method, endpoint, status_code |
|
|
| **HTTP Metrics** | `{service}_active_requests` | Currently active requests | - |
|
|
| **System Metrics** | `process.cpu.utilization` | Process CPU usage | - |
|
|
| **System Metrics** | `process.memory.usage` | Process memory usage | - |
|
|
| **System Metrics** | `system.cpu.utilization` | System CPU usage | - |
|
|
| **System Metrics** | `system.memory.usage` | System memory usage | - |
|
|
| **Database Metrics** | `db.query.duration` | Database query duration | operation, table |
|
|
| **Cache Metrics** | `cache.operation.duration` | Cache operation duration | operation, key |
|
|
|
|
### Custom Metrics (Service-Specific)
|
|
|
|
Examples of service-specific metrics:
|
|
|
|
**Auth Service:**
|
|
- `auth_registration_total` (by status)
|
|
- `auth_login_success_total`
|
|
- `auth_login_failure_total` (by reason)
|
|
- `auth_registration_duration_seconds`
|
|
|
|
**Orders Service:**
|
|
- `orders_created_total`
|
|
- `orders_processed_total` (by status)
|
|
- `orders_processing_duration_seconds`
|
|
|
|
**AI Insights Service:**
|
|
- `ai_insights_generated_total`
|
|
- `ai_model_inference_duration_seconds`
|
|
- `ai_feedback_received_total`
|
|
|
|
## 🔍 Tracing Guide
|
|
|
|
### Trace Propagation
|
|
|
|
Traces automatically flow across service boundaries:
|
|
|
|
```mermaid
|
|
sequenceDiagram
|
|
participant Client
|
|
participant Gateway
|
|
participant Auth
|
|
participant Orders
|
|
|
|
Client->>Gateway: HTTP Request (trace_id: abc123)
|
|
Gateway->>Auth: Auth Check (trace_id: abc123)
|
|
Auth-->>Gateway: Auth Response (trace_id: abc123)
|
|
Gateway->>Orders: Create Order (trace_id: abc123)
|
|
Orders-->>Gateway: Order Created (trace_id: abc123)
|
|
Gateway-->>Client: Final Response (trace_id: abc123)
|
|
```
|
|
|
|
### Trace Context in Logs
|
|
|
|
All logs include trace correlation:
|
|
|
|
```json
|
|
{
|
|
"level": "info",
|
|
"message": "Processing order",
|
|
"service": "orders-service",
|
|
"trace_id": "abc123def456",
|
|
"span_id": "789ghi",
|
|
"order_id": "12345",
|
|
"timestamp": "2024-01-08T19:00:00Z"
|
|
}
|
|
```
|
|
|
|
### Manual Trace Enhancement
|
|
|
|
Add custom trace attributes:
|
|
|
|
```python
|
|
from shared.monitoring.tracing import add_trace_attributes, add_trace_event
|
|
|
|
# Add custom attributes
|
|
add_trace_attributes(
|
|
user_id="123",
|
|
tenant_id="abc",
|
|
operation="order_creation"
|
|
)
|
|
|
|
# Add trace events
|
|
add_trace_event("order_validation_started")
|
|
# ... validation logic ...
|
|
add_trace_event("order_validation_completed", status="success")
|
|
```
|
|
|
|
## 🚨 Alerting Guide
|
|
|
|
### Standard Alerts (Recommended)
|
|
|
|
| Alert Name | Condition | Severity | Notification |
|
|
|------------|-----------|----------|--------------|
|
|
| **High Error Rate** | `error_rate > 5%` for 5m | High | PagerDuty + Slack |
|
|
| **High Latency** | `p99_latency > 2s` for 5m | High | PagerDuty + Slack |
|
|
| **Service Unavailable** | `up == 0` for 1m | Critical | PagerDuty + Slack + Email |
|
|
| **High Memory Usage** | `memory_usage > 80%` for 10m | Medium | Slack |
|
|
| **High CPU Usage** | `cpu_usage > 90%` for 5m | Medium | Slack |
|
|
| **Database Connection Issues** | `db_connections < minimum_pool_size` | High | PagerDuty + Slack |
|
|
| **Cache Hit Ratio Low** | `cache_hit_ratio < 70%` for 15m | Low | Slack |
|
|
|
|
### Creating Alerts in SigNoz
|
|
|
|
1. **Navigate to Alerts**: SigNoz UI → Alerts → Create Alert
|
|
2. **Select Metric**: Choose from available metrics
|
|
3. **Set Condition**: Define threshold and duration
|
|
4. **Configure Notifications**: Add notification channels
|
|
5. **Set Severity**: Critical, High, Medium, Low
|
|
6. **Add Description**: Explain alert purpose and resolution steps
|
|
|
|
### Example Alert Configuration (YAML)
|
|
|
|
```yaml
|
|
# Example for Terraform/Kubernetes
|
|
apiVersion: monitoring.coreos.com/v1
|
|
kind: PrometheusRule
|
|
metadata:
|
|
name: bakery-ia-alerts
|
|
namespace: monitoring
|
|
spec:
|
|
groups:
|
|
- name: service-health
|
|
rules:
|
|
- alert: ServiceDown
|
|
expr: up{service!~"signoz.*"} == 0
|
|
for: 1m
|
|
labels:
|
|
severity: critical
|
|
annotations:
|
|
summary: "Service {{ $labels.service }} is down"
|
|
description: "{{ $labels.service }} has been down for more than 1 minute"
|
|
runbook: "https://github.com/yourorg/bakery-ia/blob/main/RUNBOOKS.md#service-down"
|
|
|
|
- alert: HighErrorRate
|
|
expr: rate(http_requests_total{status_code=~"5.."}[5m]) / rate(http_requests_total[5m]) > 0.05
|
|
for: 5m
|
|
labels:
|
|
severity: high
|
|
annotations:
|
|
summary: "High error rate in {{ $labels.service }}"
|
|
description: "Error rate is {{ $value }}% (threshold: 5%)"
|
|
runbook: "https://github.com/yourorg/bakery-ia/blob/main/RUNBOOKS.md#high-error-rate"
|
|
```
|
|
|
|
## 📈 Dashboard Guide
|
|
|
|
### Recommended Dashboards
|
|
|
|
#### 1. Service Overview Dashboard
|
|
- HTTP Request Rate
|
|
- Error Rate
|
|
- Latency Percentiles (p50, p90, p99)
|
|
- Active Requests
|
|
- System Resource Usage
|
|
|
|
#### 2. Performance Dashboard
|
|
- Request Duration Histogram
|
|
- Database Query Performance
|
|
- Cache Performance
|
|
- External API Call Performance
|
|
|
|
#### 3. System Health Dashboard
|
|
- CPU Usage (Process & System)
|
|
- Memory Usage (Process & System)
|
|
- Disk I/O
|
|
- Network I/O
|
|
- File Descriptors
|
|
- Thread Count
|
|
|
|
#### 4. Business Metrics Dashboard
|
|
- User Registrations
|
|
- Order Volume
|
|
- AI Insights Generated
|
|
- API Usage by Tenant
|
|
|
|
### Creating Dashboards in SigNoz
|
|
|
|
1. **Navigate to Dashboards**: SigNoz UI → Dashboards → Create Dashboard
|
|
2. **Add Panels**: Click "Add Panel" and select metric
|
|
3. **Configure Visualization**: Choose chart type and settings
|
|
4. **Set Time Range**: Default to last 1h, 6h, 24h, 7d
|
|
5. **Add Variables**: For dynamic filtering (service, environment)
|
|
6. **Save Dashboard**: Give it a descriptive name
|
|
|
|
## 🛠️ Troubleshooting Guide
|
|
|
|
### Common Issues & Solutions
|
|
|
|
#### Issue: No Metrics Appearing in SigNoz
|
|
|
|
**Checklist:**
|
|
- ✅ OpenTelemetry Collector running? `kubectl get pods -n bakery-ia -l app.kubernetes.io/instance=signoz`
|
|
- ✅ Service can reach collector? `telnet signoz-otel-collector.bakery-ia 4318`
|
|
- ✅ OTLP endpoint configured correctly? Check `OTEL_EXPORTER_OTLP_ENDPOINT`
|
|
- ✅ Service logs show OTLP export? Look for "Exporting metrics"
|
|
- ✅ No network policies blocking? Check Kubernetes network policies
|
|
|
|
**Debugging:**
|
|
```bash
|
|
# Check OpenTelemetry Collector logs
|
|
kubectl logs -n bakery-ia -l app=otel-collector
|
|
|
|
# Check service logs for OTLP errors
|
|
kubectl logs -l app=auth-service | grep -i otel
|
|
|
|
# Test OTLP connectivity from service pod
|
|
kubectl exec -it auth-service-pod -- curl -v http://signoz-otel-collector.bakery-ia:4318
|
|
```
|
|
|
|
#### Issue: High Latency in Specific Service
|
|
|
|
**Checklist:**
|
|
- ✅ Database queries slow? Check `db.query.duration` metrics
|
|
- ✅ External API calls slow? Check trace waterfall
|
|
- ✅ High CPU usage? Check system metrics
|
|
- ✅ Memory pressure? Check memory metrics
|
|
- ✅ Too many active requests? Check concurrency
|
|
|
|
**Debugging:**
|
|
```python
|
|
# Add detailed tracing to suspicious code
|
|
from shared.monitoring.tracing import add_trace_event
|
|
|
|
add_trace_event("database_query_started", table="users")
|
|
# ... database query ...
|
|
add_trace_event("database_query_completed", duration_ms=45)
|
|
```
|
|
|
|
#### Issue: High Error Rate
|
|
|
|
**Checklist:**
|
|
- ✅ Database connection issues? Check health endpoints
|
|
- ✅ External API failures? Check dependency metrics
|
|
- ✅ Authentication failures? Check auth service logs
|
|
- ✅ Validation errors? Check application logs
|
|
- ✅ Rate limiting? Check gateway metrics
|
|
|
|
**Debugging:**
|
|
```bash
|
|
# Check error logs with trace correlation
|
|
kubectl logs -l app=auth-service | grep -i error | grep -i trace
|
|
|
|
# Filter traces by error status
|
|
# In SigNoz: Add filter http.status_code >= 400
|
|
```
|
|
|
|
## 📚 Runbook Reference
|
|
|
|
See [RUNBOOKS.md](RUNBOOKS.md) for detailed troubleshooting procedures.
|
|
|
|
## 🔧 Development Guide
|
|
|
|
### Adding Custom Metrics
|
|
|
|
```python
|
|
# In any service using direct monitoring
|
|
self.metrics_collector.register_counter(
|
|
"custom_metric_name",
|
|
"Description of what this metric tracks",
|
|
labels=["label1", "label2"] # Optional labels
|
|
)
|
|
|
|
# Increment the counter
|
|
self.metrics_collector.increment_counter(
|
|
"custom_metric_name",
|
|
value=1,
|
|
labels={"label1": "value1", "label2": "value2"}
|
|
)
|
|
```
|
|
|
|
### Adding Custom Trace Attributes
|
|
|
|
```python
|
|
# Add context to current span
|
|
from shared.monitoring.tracing import add_trace_attributes
|
|
|
|
add_trace_attributes(
|
|
user_id=user.id,
|
|
tenant_id=tenant.id,
|
|
operation="premium_feature_access",
|
|
feature_name="advanced_forecasting"
|
|
)
|
|
```
|
|
|
|
### Service-Specific Monitoring Setup
|
|
|
|
For services needing custom monitoring beyond the base class:
|
|
|
|
```python
|
|
# In your service's __init__ method
|
|
from shared.monitoring.system_metrics import SystemMetricsCollector
|
|
from shared.monitoring.metrics import MetricsCollector
|
|
|
|
class MyService(StandardFastAPIService):
|
|
def __init__(self):
|
|
# Call parent constructor first
|
|
super().__init__(...)
|
|
|
|
# Add custom metrics collector
|
|
self.custom_metrics = MetricsCollector("my-service")
|
|
|
|
# Register custom metrics
|
|
self.custom_metrics.register_counter(
|
|
"business_specific_events",
|
|
"Custom business event counter"
|
|
)
|
|
|
|
# Add system metrics if not using base class defaults
|
|
self.system_metrics = SystemMetricsCollector("my-service")
|
|
```
|
|
|
|
## 📊 SigNoz Configuration
|
|
|
|
### Environment Variables
|
|
|
|
```env
|
|
# OpenTelemetry Collector endpoint
|
|
OTEL_EXPORTER_OTLP_ENDPOINT=http://signoz-otel-collector.bakery-ia:4318
|
|
|
|
# Service-specific configuration
|
|
OTEL_SERVICE_NAME=auth-service
|
|
OTEL_RESOURCE_ATTRIBUTES=deployment.environment=production,k8s.namespace=bakery-ia
|
|
|
|
# Metrics export interval (default: 60000ms = 60s)
|
|
OTEL_METRIC_EXPORT_INTERVAL=60000
|
|
|
|
# Batch span processor configuration
|
|
OTEL_BSP_SCHEDULE_DELAY=5000
|
|
OTEL_BSP_MAX_QUEUE_SIZE=2048
|
|
OTEL_BSP_MAX_EXPORT_BATCH_SIZE=512
|
|
```
|
|
|
|
### Kubernetes Configuration
|
|
|
|
```yaml
|
|
# Example deployment with monitoring sidecar
|
|
apiVersion: apps/v1
|
|
kind: Deployment
|
|
metadata:
|
|
name: auth-service
|
|
spec:
|
|
template:
|
|
spec:
|
|
containers:
|
|
- name: auth-service
|
|
image: auth-service:latest
|
|
env:
|
|
- name: OTEL_EXPORTER_OTLP_ENDPOINT
|
|
value: "http://signoz-otel-collector.bakery-ia:4318"
|
|
- name: OTEL_SERVICE_NAME
|
|
value: "auth-service"
|
|
- name: ENVIRONMENT
|
|
value: "production"
|
|
resources:
|
|
limits:
|
|
cpu: "1"
|
|
memory: "512Mi"
|
|
requests:
|
|
cpu: "200m"
|
|
memory: "256Mi"
|
|
```
|
|
|
|
## 🎯 Best Practices
|
|
|
|
### Monitoring Best Practices
|
|
|
|
1. **Use Consistent Naming**: Follow OpenTelemetry semantic conventions
|
|
2. **Add Context to Traces**: Include user/tenant IDs in trace attributes
|
|
3. **Monitor Dependencies**: Track external API and database performance
|
|
4. **Set Appropriate Alerts**: Avoid alert fatigue with meaningful thresholds
|
|
5. **Document Metrics**: Keep metrics documentation up to date
|
|
6. **Review Regularly**: Update dashboards as services evolve
|
|
7. **Test Alerts**: Ensure alerts fire correctly before production
|
|
|
|
### Performance Best Practices
|
|
|
|
1. **Batch Metrics Export**: Use default 60s interval for most services
|
|
2. **Sample Traces**: Consider sampling for high-volume services
|
|
3. **Limit Custom Metrics**: Only track metrics that provide value
|
|
4. **Use Histograms Wisely**: Histograms can be resource-intensive
|
|
5. **Monitor Monitoring**: Track OTLP export success/failure rates
|
|
|
|
## 📞 Support
|
|
|
|
### Getting Help
|
|
|
|
1. **Check Documentation**: This file and RUNBOOKS.md
|
|
2. **Review SigNoz Docs**: https://signoz.io/docs/
|
|
3. **OpenTelemetry Docs**: https://opentelemetry.io/docs/
|
|
4. **Team Channel**: #monitoring in Slack
|
|
5. **GitHub Issues**: https://github.com/yourorg/bakery-ia/issues
|
|
|
|
### Escalation Path
|
|
|
|
1. **First Line**: Development team (service owners)
|
|
2. **Second Line**: DevOps team (monitoring specialists)
|
|
3. **Third Line**: SigNoz support (vendor support)
|
|
|
|
## 🎉 Summary
|
|
|
|
The bakery-ia monitoring system provides:
|
|
|
|
- **📊 100% Service Coverage**: All 20 services monitored
|
|
- **🚀 Modern Architecture**: OpenTelemetry + SigNoz
|
|
- **🔧 Comprehensive Metrics**: System, HTTP, database, cache
|
|
- **🔍 Full Observability**: Traces, metrics, logs integrated
|
|
- **✅ Production Ready**: Battle-tested and scalable
|
|
|
|
**All services are fully instrumented and ready for production monitoring!** 🎉 |