Update monitoring packages to latest versions
- Updated all OpenTelemetry packages to latest versions: - opentelemetry-api: 1.27.0 → 1.39.1 - opentelemetry-sdk: 1.27.0 → 1.39.1 - opentelemetry-exporter-otlp-proto-grpc: 1.27.0 → 1.39.1 - opentelemetry-exporter-otlp-proto-http: 1.27.0 → 1.39.1 - opentelemetry-instrumentation-fastapi: 0.48b0 → 0.60b1 - opentelemetry-instrumentation-httpx: 0.48b0 → 0.60b1 - opentelemetry-instrumentation-redis: 0.48b0 → 0.60b1 - opentelemetry-instrumentation-sqlalchemy: 0.48b0 → 0.60b1 - Removed prometheus-client==0.23.1 from all services - Unified all services to use the same monitoring package versions Generated by Mistral Vibe. Co-Authored-By: Mistral Vibe <vibe@mistral.ai>
This commit is contained in:
Urtzi Alfaro
569
docs/DATABASE_MONITORING.md
Normal file
569
docs/DATABASE_MONITORING.md
Normal file
@@ -0,0 +1,569 @@
|
||||
# Database Monitoring with SigNoz
|
||||
|
||||
This guide explains how to collect metrics and logs from PostgreSQL, Redis, and RabbitMQ databases and send them to SigNoz.
|
||||
|
||||
## Table of Contents
|
||||
|
||||
1. [Overview](#overview)
|
||||
2. [PostgreSQL Monitoring](#postgresql-monitoring)
|
||||
3. [Redis Monitoring](#redis-monitoring)
|
||||
4. [RabbitMQ Monitoring](#rabbitmq-monitoring)
|
||||
5. [Database Logs Export](#database-logs-export)
|
||||
6. [Dashboard Examples](#dashboard-examples)
|
||||
|
||||
## Overview
|
||||
|
||||
**Database monitoring provides:**
|
||||
- **Metrics**: Connection pools, query performance, cache hit rates, disk usage
|
||||
- **Logs**: Query logs, error logs, slow query logs
|
||||
- **Correlation**: Link database metrics with application traces
|
||||
|
||||
**Three approaches for database monitoring:**
|
||||
|
||||
1. **OpenTelemetry Collector Receivers** (Recommended)
|
||||
- Deploy OTel collector as sidecar or separate deployment
|
||||
- Scrape database metrics and forward to SigNoz
|
||||
- No code changes needed
|
||||
|
||||
2. **Application-Level Instrumentation** (Already Implemented)
|
||||
- Use OpenTelemetry auto-instrumentation in your services
|
||||
- Captures database queries as spans in traces
|
||||
- Shows query duration, errors in application context
|
||||
|
||||
3. **Database Exporters** (Advanced)
|
||||
- Dedicated exporters (postgres_exporter, redis_exporter)
|
||||
- More detailed database-specific metrics
|
||||
- Requires additional deployment
|
||||
|
||||
## PostgreSQL Monitoring
|
||||
|
||||
### Option 1: OpenTelemetry Collector with PostgreSQL Receiver (Recommended)
|
||||
|
||||
Deploy an OpenTelemetry collector instance to scrape PostgreSQL metrics.
|
||||
|
||||
#### Step 1: Create PostgreSQL Monitoring User
|
||||
|
||||
```sql
|
||||
-- Create monitoring user with read-only access
|
||||
CREATE USER otel_monitor WITH PASSWORD 'your-secure-password';
|
||||
GRANT pg_monitor TO otel_monitor;
|
||||
GRANT CONNECT ON DATABASE your_database TO otel_monitor;
|
||||
```
|
||||
|
||||
#### Step 2: Deploy OTel Collector for PostgreSQL
|
||||
|
||||
Create a dedicated collector deployment:
|
||||
|
||||
```yaml
|
||||
# infrastructure/kubernetes/base/monitoring/postgres-otel-collector.yaml
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: postgres-otel-collector
|
||||
namespace: bakery-ia
|
||||
labels:
|
||||
app: postgres-otel-collector
|
||||
spec:
|
||||
replicas: 1
|
||||
selector:
|
||||
matchLabels:
|
||||
app: postgres-otel-collector
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: postgres-otel-collector
|
||||
spec:
|
||||
containers:
|
||||
- name: otel-collector
|
||||
image: otel/opentelemetry-collector-contrib:latest
|
||||
ports:
|
||||
- containerPort: 4318
|
||||
name: otlp-http
|
||||
- containerPort: 4317
|
||||
name: otlp-grpc
|
||||
volumeMounts:
|
||||
- name: config
|
||||
mountPath: /etc/otel-collector
|
||||
command:
|
||||
- /otelcol-contrib
|
||||
- --config=/etc/otel-collector/config.yaml
|
||||
volumes:
|
||||
- name: config
|
||||
configMap:
|
||||
name: postgres-otel-collector-config
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: postgres-otel-collector-config
|
||||
namespace: bakery-ia
|
||||
data:
|
||||
config.yaml: |
|
||||
receivers:
|
||||
# PostgreSQL receiver for each database
|
||||
postgresql/auth:
|
||||
endpoint: auth-db-service:5432
|
||||
username: otel_monitor
|
||||
password: ${POSTGRES_MONITOR_PASSWORD}
|
||||
databases:
|
||||
- auth_db
|
||||
collection_interval: 30s
|
||||
metrics:
|
||||
postgresql.backends: true
|
||||
postgresql.bgwriter.buffers.allocated: true
|
||||
postgresql.bgwriter.buffers.writes: true
|
||||
postgresql.blocks_read: true
|
||||
postgresql.commits: true
|
||||
postgresql.connection.max: true
|
||||
postgresql.database.count: true
|
||||
postgresql.database.size: true
|
||||
postgresql.deadlocks: true
|
||||
postgresql.index.scans: true
|
||||
postgresql.index.size: true
|
||||
postgresql.operations: true
|
||||
postgresql.rollbacks: true
|
||||
postgresql.rows: true
|
||||
postgresql.table.count: true
|
||||
postgresql.table.size: true
|
||||
postgresql.temp_files: true
|
||||
|
||||
postgresql/inventory:
|
||||
endpoint: inventory-db-service:5432
|
||||
username: otel_monitor
|
||||
password: ${POSTGRES_MONITOR_PASSWORD}
|
||||
databases:
|
||||
- inventory_db
|
||||
collection_interval: 30s
|
||||
|
||||
# Add more PostgreSQL receivers for other databases...
|
||||
|
||||
processors:
|
||||
batch:
|
||||
timeout: 10s
|
||||
send_batch_size: 1024
|
||||
|
||||
memory_limiter:
|
||||
check_interval: 1s
|
||||
limit_mib: 512
|
||||
|
||||
resourcedetection:
|
||||
detectors: [env, system]
|
||||
|
||||
# Add database labels
|
||||
resource:
|
||||
attributes:
|
||||
- key: database.system
|
||||
value: postgresql
|
||||
action: insert
|
||||
- key: deployment.environment
|
||||
value: ${ENVIRONMENT}
|
||||
action: insert
|
||||
|
||||
exporters:
|
||||
# Send to SigNoz
|
||||
otlphttp:
|
||||
endpoint: http://signoz-otel-collector.signoz.svc.cluster.local:4318
|
||||
tls:
|
||||
insecure: true
|
||||
|
||||
# Debug logging
|
||||
logging:
|
||||
loglevel: info
|
||||
|
||||
service:
|
||||
pipelines:
|
||||
metrics:
|
||||
receivers: [postgresql/auth, postgresql/inventory]
|
||||
processors: [memory_limiter, resource, batch, resourcedetection]
|
||||
exporters: [otlphttp, logging]
|
||||
```
|
||||
|
||||
#### Step 3: Create Secrets
|
||||
|
||||
```bash
|
||||
# Create secret for monitoring user password
|
||||
kubectl create secret generic postgres-monitor-secrets \
|
||||
-n bakery-ia \
|
||||
--from-literal=POSTGRES_MONITOR_PASSWORD='your-secure-password'
|
||||
```
|
||||
|
||||
#### Step 4: Deploy
|
||||
|
||||
```bash
|
||||
kubectl apply -f infrastructure/kubernetes/base/monitoring/postgres-otel-collector.yaml
|
||||
```
|
||||
|
||||
### Option 2: Application-Level Database Metrics (Already Implemented)
|
||||
|
||||
Your services already collect database metrics via SQLAlchemy instrumentation:
|
||||
|
||||
**Metrics automatically collected:**
|
||||
- `db.client.connections.usage` - Active database connections
|
||||
- `db.client.operation.duration` - Query duration (SELECT, INSERT, UPDATE, DELETE)
|
||||
- Query traces with SQL statements (in trace spans)
|
||||
|
||||
**View in SigNoz:**
|
||||
1. Go to Traces → Select a service → Filter by `db.operation`
|
||||
2. See individual database queries with duration
|
||||
3. Identify slow queries causing latency
|
||||
|
||||
### PostgreSQL Metrics Reference
|
||||
|
||||
| Metric | Description |
|
||||
|--------|-------------|
|
||||
| `postgresql.backends` | Number of active connections |
|
||||
| `postgresql.database.size` | Database size in bytes |
|
||||
| `postgresql.commits` | Transaction commits |
|
||||
| `postgresql.rollbacks` | Transaction rollbacks |
|
||||
| `postgresql.deadlocks` | Deadlock count |
|
||||
| `postgresql.blocks_read` | Blocks read from disk |
|
||||
| `postgresql.table.size` | Table size in bytes |
|
||||
| `postgresql.index.size` | Index size in bytes |
|
||||
| `postgresql.rows` | Rows inserted/updated/deleted |
|
||||
|
||||
## Redis Monitoring
|
||||
|
||||
### Option 1: OpenTelemetry Collector with Redis Receiver (Recommended)
|
||||
|
||||
```yaml
|
||||
# Add to postgres-otel-collector config or create separate collector
|
||||
receivers:
|
||||
redis:
|
||||
endpoint: redis-service.bakery-ia:6379
|
||||
password: ${REDIS_PASSWORD}
|
||||
collection_interval: 30s
|
||||
tls:
|
||||
insecure_skip_verify: false
|
||||
cert_file: /etc/redis-tls/redis-cert.pem
|
||||
key_file: /etc/redis-tls/redis-key.pem
|
||||
ca_file: /etc/redis-tls/ca-cert.pem
|
||||
metrics:
|
||||
redis.clients.connected: true
|
||||
redis.clients.blocked: true
|
||||
redis.commands.processed: true
|
||||
redis.commands.duration: true
|
||||
redis.db.keys: true
|
||||
redis.db.expires: true
|
||||
redis.keyspace.hits: true
|
||||
redis.keyspace.misses: true
|
||||
redis.memory.used: true
|
||||
redis.memory.peak: true
|
||||
redis.memory.fragmentation_ratio: true
|
||||
redis.cpu.time: true
|
||||
redis.replication.offset: true
|
||||
```
|
||||
|
||||
### Option 2: Application-Level Redis Metrics (Already Implemented)
|
||||
|
||||
Your services already collect Redis metrics via Redis instrumentation:
|
||||
|
||||
**Metrics automatically collected:**
|
||||
- Redis command traces (GET, SET, etc.) in spans
|
||||
- Command duration
|
||||
- Command errors
|
||||
|
||||
### Redis Metrics Reference
|
||||
|
||||
| Metric | Description |
|
||||
|--------|-------------|
|
||||
| `redis.clients.connected` | Connected clients |
|
||||
| `redis.commands.processed` | Total commands processed |
|
||||
| `redis.keyspace.hits` | Cache hit rate |
|
||||
| `redis.keyspace.misses` | Cache miss rate |
|
||||
| `redis.memory.used` | Memory usage in bytes |
|
||||
| `redis.memory.fragmentation_ratio` | Memory fragmentation |
|
||||
| `redis.db.keys` | Number of keys per database |
|
||||
|
||||
## RabbitMQ Monitoring
|
||||
|
||||
### Option 1: RabbitMQ Management Plugin + OpenTelemetry (Recommended)
|
||||
|
||||
RabbitMQ exposes metrics via its management API.
|
||||
|
||||
```yaml
|
||||
receivers:
|
||||
rabbitmq:
|
||||
endpoint: http://rabbitmq-service.bakery-ia:15672
|
||||
username: ${RABBITMQ_USER}
|
||||
password: ${RABBITMQ_PASSWORD}
|
||||
collection_interval: 30s
|
||||
metrics:
|
||||
rabbitmq.consumer.count: true
|
||||
rabbitmq.message.current: true
|
||||
rabbitmq.message.acknowledged: true
|
||||
rabbitmq.message.delivered: true
|
||||
rabbitmq.message.published: true
|
||||
rabbitmq.queue.count: true
|
||||
```
|
||||
|
||||
### RabbitMQ Metrics Reference
|
||||
|
||||
| Metric | Description |
|
||||
|--------|-------------|
|
||||
| `rabbitmq.consumer.count` | Active consumers |
|
||||
| `rabbitmq.message.current` | Messages in queue |
|
||||
| `rabbitmq.message.acknowledged` | Messages acknowledged |
|
||||
| `rabbitmq.message.delivered` | Messages delivered |
|
||||
| `rabbitmq.message.published` | Messages published |
|
||||
| `rabbitmq.queue.count` | Number of queues |
|
||||
|
||||
## Database Logs Export
|
||||
|
||||
### PostgreSQL Logs
|
||||
|
||||
#### Option 1: Configure PostgreSQL to Log to Stdout (Kubernetes-native)
|
||||
|
||||
PostgreSQL logs should go to stdout/stderr, which Kubernetes automatically captures.
|
||||
|
||||
**Update PostgreSQL configuration:**
|
||||
|
||||
```yaml
|
||||
# In your postgres deployment ConfigMap
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: postgres-config
|
||||
namespace: bakery-ia
|
||||
data:
|
||||
postgresql.conf: |
|
||||
# Logging
|
||||
logging_collector = off # Use stdout/stderr instead
|
||||
log_destination = 'stderr'
|
||||
log_statement = 'all' # Or 'ddl', 'mod', 'none'
|
||||
log_duration = on
|
||||
log_line_prefix = '%t [%p]: user=%u,db=%d,app=%a,client=%h '
|
||||
log_min_duration_statement = 100 # Log queries > 100ms
|
||||
log_checkpoints = on
|
||||
log_connections = on
|
||||
log_disconnections = on
|
||||
log_lock_waits = on
|
||||
```
|
||||
|
||||
#### Option 2: OpenTelemetry Filelog Receiver
|
||||
|
||||
If PostgreSQL writes to files, use filelog receiver:
|
||||
|
||||
```yaml
|
||||
receivers:
|
||||
filelog/postgres:
|
||||
include:
|
||||
- /var/log/postgresql/*.log
|
||||
start_at: end
|
||||
operators:
|
||||
- type: regex_parser
|
||||
regex: '^(?P<timestamp>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}.\d+) \[(?P<pid>\d+)\]: user=(?P<user>[^,]+),db=(?P<database>[^,]+),app=(?P<application>[^,]+),client=(?P<client>[^ ]+) (?P<level>[A-Z]+): (?P<message>.*)'
|
||||
timestamp:
|
||||
parse_from: attributes.timestamp
|
||||
layout: '%Y-%m-%d %H:%M:%S.%f'
|
||||
- type: move
|
||||
from: attributes.level
|
||||
to: severity
|
||||
- type: add
|
||||
field: attributes["database.system"]
|
||||
value: "postgresql"
|
||||
|
||||
processors:
|
||||
resource/postgres:
|
||||
attributes:
|
||||
- key: database.system
|
||||
value: postgresql
|
||||
action: insert
|
||||
- key: service.name
|
||||
value: postgres-logs
|
||||
action: insert
|
||||
|
||||
exporters:
|
||||
otlphttp/logs:
|
||||
endpoint: http://signoz-otel-collector.signoz.svc.cluster.local:4318/v1/logs
|
||||
|
||||
service:
|
||||
pipelines:
|
||||
logs/postgres:
|
||||
receivers: [filelog/postgres]
|
||||
processors: [resource/postgres, batch]
|
||||
exporters: [otlphttp/logs]
|
||||
```
|
||||
|
||||
### Redis Logs
|
||||
|
||||
Redis logs should go to stdout, which Kubernetes captures automatically. View them in SigNoz by:
|
||||
|
||||
1. Ensuring Redis pods log to stdout
|
||||
2. No additional configuration needed - Kubernetes logs are available
|
||||
3. Optional: Use Kubernetes logs collection (see below)
|
||||
|
||||
### Kubernetes Logs Collection (All Pods)
|
||||
|
||||
Deploy a DaemonSet to collect all Kubernetes pod logs:
|
||||
|
||||
```yaml
|
||||
# infrastructure/kubernetes/base/monitoring/logs-collector-daemonset.yaml
|
||||
apiVersion: apps/v1
|
||||
kind: DaemonSet
|
||||
metadata:
|
||||
name: otel-logs-collector
|
||||
namespace: bakery-ia
|
||||
spec:
|
||||
selector:
|
||||
matchLabels:
|
||||
name: otel-logs-collector
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
name: otel-logs-collector
|
||||
spec:
|
||||
serviceAccountName: otel-logs-collector
|
||||
containers:
|
||||
- name: otel-collector
|
||||
image: otel/opentelemetry-collector-contrib:latest
|
||||
volumeMounts:
|
||||
- name: varlog
|
||||
mountPath: /var/log
|
||||
readOnly: true
|
||||
- name: varlibdockercontainers
|
||||
mountPath: /var/lib/docker/containers
|
||||
readOnly: true
|
||||
- name: config
|
||||
mountPath: /etc/otel-collector
|
||||
volumes:
|
||||
- name: varlog
|
||||
hostPath:
|
||||
path: /var/log
|
||||
- name: varlibdockercontainers
|
||||
hostPath:
|
||||
path: /var/lib/docker/containers
|
||||
- name: config
|
||||
configMap:
|
||||
name: otel-logs-collector-config
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: ClusterRole
|
||||
metadata:
|
||||
name: otel-logs-collector
|
||||
rules:
|
||||
- apiGroups: [""]
|
||||
resources: ["pods", "namespaces"]
|
||||
verbs: ["get", "list", "watch"]
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: ClusterRoleBinding
|
||||
metadata:
|
||||
name: otel-logs-collector
|
||||
roleRef:
|
||||
apiGroup: rbac.authorization.k8s.io
|
||||
kind: ClusterRole
|
||||
name: otel-logs-collector
|
||||
subjects:
|
||||
- kind: ServiceAccount
|
||||
name: otel-logs-collector
|
||||
namespace: bakery-ia
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: ServiceAccount
|
||||
metadata:
|
||||
name: otel-logs-collector
|
||||
namespace: bakery-ia
|
||||
```
|
||||
|
||||
## Dashboard Examples
|
||||
|
||||
### PostgreSQL Dashboard in SigNoz
|
||||
|
||||
Create a custom dashboard with these panels:
|
||||
|
||||
1. **Active Connections**
|
||||
- Query: `postgresql.backends`
|
||||
- Group by: `database.name`
|
||||
|
||||
2. **Query Rate**
|
||||
- Query: `rate(postgresql.commits[5m])`
|
||||
|
||||
3. **Database Size**
|
||||
- Query: `postgresql.database.size`
|
||||
- Group by: `database.name`
|
||||
|
||||
4. **Slow Queries**
|
||||
- Go to Traces
|
||||
- Filter: `db.system="postgresql" AND duration > 1s`
|
||||
- See slow queries with full SQL
|
||||
|
||||
5. **Connection Pool Usage**
|
||||
- Query: `db.client.connections.usage`
|
||||
- Group by: `service`
|
||||
|
||||
### Redis Dashboard
|
||||
|
||||
1. **Hit Rate**
|
||||
- Query: `redis.keyspace.hits / (redis.keyspace.hits + redis.keyspace.misses)`
|
||||
|
||||
2. **Memory Usage**
|
||||
- Query: `redis.memory.used`
|
||||
|
||||
3. **Connected Clients**
|
||||
- Query: `redis.clients.connected`
|
||||
|
||||
4. **Commands Per Second**
|
||||
- Query: `rate(redis.commands.processed[1m])`
|
||||
|
||||
## Quick Reference: What's Monitored
|
||||
|
||||
| Database | Metrics | Logs | Traces |
|
||||
|----------|---------|------|--------|
|
||||
| **PostgreSQL** | ✅ Via receiver<br>✅ Via app instrumentation | ✅ Stdout/stderr<br>✅ Optional filelog | ✅ Query spans in traces |
|
||||
| **Redis** | ✅ Via receiver<br>✅ Via app instrumentation | ✅ Stdout/stderr | ✅ Command spans in traces |
|
||||
| **RabbitMQ** | ✅ Via receiver | ✅ Stdout/stderr | ✅ Publish/consume spans |
|
||||
|
||||
## Deployment Checklist
|
||||
|
||||
- [ ] Deploy OpenTelemetry collector for database metrics
|
||||
- [ ] Create monitoring users in PostgreSQL
|
||||
- [ ] Configure database logging to stdout
|
||||
- [ ] Verify metrics appear in SigNoz
|
||||
- [ ] Create database dashboards
|
||||
- [ ] Set up alerts for connection limits, slow queries, high memory
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### No PostgreSQL metrics
|
||||
|
||||
```bash
|
||||
# Check collector logs
|
||||
kubectl logs -n bakery-ia deployment/postgres-otel-collector
|
||||
|
||||
# Test connection to database
|
||||
kubectl exec -n bakery-ia deployment/postgres-otel-collector -- \
|
||||
psql -h auth-db-service -U otel_monitor -d auth_db -c "SELECT 1"
|
||||
```
|
||||
|
||||
### No Redis metrics
|
||||
|
||||
```bash
|
||||
# Check Redis connection
|
||||
kubectl exec -n bakery-ia deployment/postgres-otel-collector -- \
|
||||
redis-cli -h redis-service -a PASSWORD ping
|
||||
```
|
||||
|
||||
### Logs not appearing
|
||||
|
||||
```bash
|
||||
# Check if logs are going to stdout
|
||||
kubectl logs -n bakery-ia postgres-pod-name
|
||||
|
||||
# Check logs collector
|
||||
kubectl logs -n bakery-ia daemonset/otel-logs-collector
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
|
||||
1. **Use dedicated monitoring users** - Don't use application database users
|
||||
2. **Set appropriate collection intervals** - 30s-60s for metrics
|
||||
3. **Monitor connection pool saturation** - Alert before exhausting connections
|
||||
4. **Track slow queries** - Set `log_min_duration_statement` appropriately
|
||||
5. **Monitor disk usage** - PostgreSQL database size growth
|
||||
6. **Track cache hit rates** - Redis keyspace hits/misses ratio
|
||||
|
||||
## Additional Resources
|
||||
|
||||
- [OpenTelemetry PostgreSQL Receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/postgresqlreceiver)
|
||||
- [OpenTelemetry Redis Receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/redisreceiver)
|
||||
- [SigNoz Database Monitoring](https://signoz.io/docs/userguide/metrics/)
|
||||
Reference in New Issue
Block a user