Improve teh securty of teh DB
This commit is contained in:
Urtzi Alfaro
627
docs/DEVELOPMENT_WITH_SECURITY.md
Normal file
627
docs/DEVELOPMENT_WITH_SECURITY.md
Normal file
@@ -0,0 +1,627 @@
|
||||
# Development with Database Security Enabled
|
||||
|
||||
**Author:** Claude Security Implementation
|
||||
**Date:** October 18, 2025
|
||||
**Status:** Ready for Use
|
||||
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
This guide explains how to develop with the new secure database infrastructure that includes TLS encryption, strong passwords, persistent storage, and audit logging.
|
||||
|
||||
---
|
||||
|
||||
## 🚀 Quick Start
|
||||
|
||||
### Option 1: Using Tilt (Recommended)
|
||||
|
||||
**Secure Development Mode:**
|
||||
```bash
|
||||
# Use the secure Tiltfile
|
||||
tilt up -f Tiltfile.secure
|
||||
|
||||
# Or rename it to be default
|
||||
mv Tiltfile Tiltfile.old
|
||||
mv Tiltfile.secure Tiltfile
|
||||
tilt up
|
||||
```
|
||||
|
||||
**Features:**
|
||||
- ✅ Automatic security setup on startup
|
||||
- ✅ TLS certificates applied before databases start
|
||||
- ✅ Live code updates with hot reload
|
||||
- ✅ Built-in TLS and PVC verification
|
||||
- ✅ Visual dashboard at http://localhost:10350
|
||||
|
||||
### Option 2: Using Skaffold
|
||||
|
||||
**Secure Development Mode:**
|
||||
```bash
|
||||
# Use the secure Skaffold config
|
||||
skaffold dev -f skaffold-secure.yaml
|
||||
|
||||
# Or rename it to be default
|
||||
mv skaffold.yaml skaffold.old.yaml
|
||||
mv skaffold-secure.yaml skaffold.yaml
|
||||
skaffold dev
|
||||
```
|
||||
|
||||
**Features:**
|
||||
- ✅ Pre-deployment hooks apply security configs
|
||||
- ✅ Post-deployment verification messages
|
||||
- ✅ Automatic rebuilds on code changes
|
||||
|
||||
### Option 3: Manual Deployment
|
||||
|
||||
**For full control:**
|
||||
```bash
|
||||
# Apply security configurations
|
||||
./scripts/apply-security-changes.sh
|
||||
|
||||
# Deploy with kubectl
|
||||
kubectl apply -k infrastructure/kubernetes/overlays/dev
|
||||
|
||||
# Verify
|
||||
kubectl get pods -n bakery-ia
|
||||
kubectl get pvc -n bakery-ia
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🔐 What Changed?
|
||||
|
||||
### Database Connections
|
||||
|
||||
**Before (Insecure):**
|
||||
```python
|
||||
# Old connection string
|
||||
DATABASE_URL = "postgresql+asyncpg://user:password@host:5432/db"
|
||||
```
|
||||
|
||||
**After (Secure):**
|
||||
```python
|
||||
# New connection string (automatic)
|
||||
DATABASE_URL = "postgresql+asyncpg://user:strong_password@host:5432/db?ssl=require&sslmode=require"
|
||||
```
|
||||
|
||||
**Key Changes:**
|
||||
- `ssl=require` - Enforces TLS encryption
|
||||
- `sslmode=require` - Rejects unencrypted connections
|
||||
- Strong 32-character passwords
|
||||
- Automatic SSL parameter addition in `shared/database/base.py`
|
||||
|
||||
### Redis Connections
|
||||
|
||||
**Before (Insecure):**
|
||||
```python
|
||||
REDIS_URL = "redis://password@host:6379"
|
||||
```
|
||||
|
||||
**After (Secure):**
|
||||
```python
|
||||
REDIS_URL = "rediss://password@host:6379?ssl_cert_reqs=required"
|
||||
```
|
||||
|
||||
**Key Changes:**
|
||||
- `rediss://` protocol - Uses TLS
|
||||
- `ssl_cert_reqs=required` - Enforces certificate validation
|
||||
- Automatic in `shared/config/base.py`
|
||||
|
||||
### Environment Variables
|
||||
|
||||
**New Environment Variables:**
|
||||
```bash
|
||||
# Optional: Disable TLS for local testing (NOT recommended)
|
||||
REDIS_TLS_ENABLED=false # Default: true
|
||||
|
||||
# Database URLs now include SSL parameters automatically
|
||||
# No changes needed to your service code!
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📁 File Structure Changes
|
||||
|
||||
### New Files Created
|
||||
|
||||
```
|
||||
infrastructure/
|
||||
├── tls/ # TLS certificates
|
||||
│ ├── ca/
|
||||
│ │ ├── ca-cert.pem # Certificate Authority
|
||||
│ │ └── ca-key.pem # CA private key
|
||||
│ ├── postgres/
|
||||
│ │ ├── server-cert.pem # PostgreSQL server cert
|
||||
│ │ ├── server-key.pem # PostgreSQL private key
|
||||
│ │ └── ca-cert.pem # CA for clients
|
||||
│ ├── redis/
|
||||
│ │ ├── redis-cert.pem # Redis server cert
|
||||
│ │ ├── redis-key.pem # Redis private key
|
||||
│ │ └── ca-cert.pem # CA for clients
|
||||
│ └── generate-certificates.sh # Regeneration script
|
||||
│
|
||||
└── kubernetes/
|
||||
├── base/
|
||||
│ ├── secrets/
|
||||
│ │ ├── postgres-tls-secret.yaml # PostgreSQL TLS secret
|
||||
│ │ └── redis-tls-secret.yaml # Redis TLS secret
|
||||
│ └── configmaps/
|
||||
│ └── postgres-logging-config.yaml # Audit logging
|
||||
└── encryption/
|
||||
└── encryption-config.yaml # Secrets encryption
|
||||
|
||||
scripts/
|
||||
├── encrypted-backup.sh # Create encrypted backups
|
||||
├── apply-security-changes.sh # Deploy security changes
|
||||
└── ... (other security scripts)
|
||||
|
||||
docs/
|
||||
├── SECURITY_IMPLEMENTATION_COMPLETE.md # Full implementation guide
|
||||
├── DATABASE_SECURITY_ANALYSIS_REPORT.md # Security analysis
|
||||
└── DEVELOPMENT_WITH_SECURITY.md # This file
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🔧 Development Workflow
|
||||
|
||||
### Starting Development
|
||||
|
||||
**With Tilt (Recommended):**
|
||||
```bash
|
||||
# Start all services with security
|
||||
tilt up -f Tiltfile.secure
|
||||
|
||||
# Watch the Tilt dashboard
|
||||
open http://localhost:10350
|
||||
```
|
||||
|
||||
**With Skaffold:**
|
||||
```bash
|
||||
# Start development mode
|
||||
skaffold dev -f skaffold-secure.yaml
|
||||
|
||||
# Or with debug ports
|
||||
skaffold dev -f skaffold-secure.yaml -p debug
|
||||
```
|
||||
|
||||
### Making Code Changes
|
||||
|
||||
**No changes needed!** Your code works the same way:
|
||||
|
||||
```python
|
||||
# Your existing code (unchanged)
|
||||
from shared.database import DatabaseManager
|
||||
|
||||
db_manager = DatabaseManager(
|
||||
database_url=settings.DATABASE_URL,
|
||||
service_name="my-service"
|
||||
)
|
||||
|
||||
# TLS is automatically added to the connection!
|
||||
```
|
||||
|
||||
**Hot Reload:**
|
||||
- Python services: Changes detected automatically, uvicorn reloads
|
||||
- Frontend: Requires rebuild (nginx static files)
|
||||
- Shared libraries: All services reload when changed
|
||||
|
||||
### Testing Database Connections
|
||||
|
||||
**Verify TLS is Working:**
|
||||
```bash
|
||||
# Test PostgreSQL with TLS
|
||||
kubectl exec -n bakery-ia <auth-db-pod> -- \
|
||||
psql "postgresql://auth_user@localhost:5432/auth_db?sslmode=require" -c "SELECT version();"
|
||||
|
||||
# Test Redis with TLS
|
||||
kubectl exec -n bakery-ia <redis-pod> -- \
|
||||
redis-cli --tls \
|
||||
--cert /tls/redis-cert.pem \
|
||||
--key /tls/redis-key.pem \
|
||||
--cacert /tls/ca-cert.pem \
|
||||
PING
|
||||
|
||||
# Check if TLS certs are mounted
|
||||
kubectl exec -n bakery-ia <db-pod> -- ls -la /tls/
|
||||
```
|
||||
|
||||
**Verify from Service:**
|
||||
```python
|
||||
# In your service code
|
||||
import asyncpg
|
||||
import ssl
|
||||
|
||||
# This is what happens automatically now:
|
||||
ssl_context = ssl.create_default_context()
|
||||
conn = await asyncpg.connect(
|
||||
"postgresql://user:pass@host:5432/db",
|
||||
ssl=ssl_context
|
||||
)
|
||||
```
|
||||
|
||||
### Viewing Logs
|
||||
|
||||
**Database Logs (with audit trail):**
|
||||
```bash
|
||||
# View PostgreSQL logs
|
||||
kubectl logs -n bakery-ia <db-pod>
|
||||
|
||||
# Filter for connections
|
||||
kubectl logs -n bakery-ia <db-pod> | grep "connection"
|
||||
|
||||
# Filter for queries
|
||||
kubectl logs -n bakery-ia <db-pod> | grep "statement"
|
||||
|
||||
# View Redis logs
|
||||
kubectl logs -n bakery-ia <redis-pod>
|
||||
```
|
||||
|
||||
**Service Logs:**
|
||||
```bash
|
||||
# View service logs
|
||||
kubectl logs -n bakery-ia <service-pod>
|
||||
|
||||
# Follow logs in real-time
|
||||
kubectl logs -f -n bakery-ia <service-pod>
|
||||
|
||||
# View logs in Tilt dashboard
|
||||
# Click on service in Tilt UI
|
||||
```
|
||||
|
||||
### Debugging Connection Issues
|
||||
|
||||
**Common Issues:**
|
||||
|
||||
1. **"SSL not supported" Error**
|
||||
|
||||
```bash
|
||||
# Check if TLS certs are mounted
|
||||
kubectl exec -n bakery-ia <db-pod> -- ls /tls/
|
||||
|
||||
# Restart the pod
|
||||
kubectl delete pod <db-pod> -n bakery-ia
|
||||
|
||||
# Check secret exists
|
||||
kubectl get secret postgres-tls -n bakery-ia
|
||||
```
|
||||
|
||||
2. **"Connection refused" Error**
|
||||
|
||||
```bash
|
||||
# Check if database is running
|
||||
kubectl get pods -n bakery-ia -l app.kubernetes.io/component=database
|
||||
|
||||
# Check database logs
|
||||
kubectl logs -n bakery-ia <db-pod>
|
||||
|
||||
# Verify service is reachable
|
||||
kubectl exec -n bakery-ia <service-pod> -- nc -zv <db-service> 5432
|
||||
```
|
||||
|
||||
3. **"Authentication failed" Error**
|
||||
|
||||
```bash
|
||||
# Verify password is updated
|
||||
kubectl get secret database-secrets -n bakery-ia -o jsonpath='{.data.AUTH_DB_PASSWORD}' | base64 -d
|
||||
|
||||
# Check .env file has matching password
|
||||
grep AUTH_DB_PASSWORD .env
|
||||
|
||||
# Restart services to pick up new passwords
|
||||
kubectl rollout restart deployment -n bakery-ia --selector='app.kubernetes.io/component=service'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📊 Monitoring & Observability
|
||||
|
||||
### Checking PVC Usage
|
||||
|
||||
```bash
|
||||
# List all PVCs
|
||||
kubectl get pvc -n bakery-ia
|
||||
|
||||
# Check PVC details
|
||||
kubectl describe pvc <pvc-name> -n bakery-ia
|
||||
|
||||
# Check disk usage in pod
|
||||
kubectl exec -n bakery-ia <db-pod> -- df -h /var/lib/postgresql/data
|
||||
```
|
||||
|
||||
### Monitoring Database Connections
|
||||
|
||||
```bash
|
||||
# Check active connections (PostgreSQL)
|
||||
kubectl exec -n bakery-ia <db-pod> -- \
|
||||
psql -U <user> -d <db> -c "SELECT count(*) FROM pg_stat_activity;"
|
||||
|
||||
# Check Redis info
|
||||
kubectl exec -n bakery-ia <redis-pod> -- \
|
||||
redis-cli -a <password> --tls \
|
||||
--cert /tls/redis-cert.pem \
|
||||
--key /tls/redis-key.pem \
|
||||
--cacert /tls/ca-cert.pem \
|
||||
INFO clients
|
||||
```
|
||||
|
||||
### Security Audit
|
||||
|
||||
```bash
|
||||
# Verify TLS certificates
|
||||
kubectl exec -n bakery-ia <db-pod> -- \
|
||||
openssl x509 -in /tls/server-cert.pem -noout -text
|
||||
|
||||
# Check certificate expiry
|
||||
kubectl exec -n bakery-ia <db-pod> -- \
|
||||
openssl x509 -in /tls/server-cert.pem -noout -dates
|
||||
|
||||
# Verify pgcrypto extension
|
||||
kubectl exec -n bakery-ia <db-pod> -- \
|
||||
psql -U <user> -d <db> -c "SELECT * FROM pg_extension WHERE extname='pgcrypto';"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🔄 Common Tasks
|
||||
|
||||
### Rotating Passwords
|
||||
|
||||
**Manual Rotation:**
|
||||
```bash
|
||||
# Generate new passwords
|
||||
./scripts/generate-passwords.sh > new-passwords.txt
|
||||
|
||||
# Update .env
|
||||
./scripts/update-env-passwords.sh
|
||||
|
||||
# Update Kubernetes secrets
|
||||
./scripts/update-k8s-secrets.sh
|
||||
|
||||
# Apply new secrets
|
||||
kubectl apply -f infrastructure/kubernetes/base/secrets.yaml
|
||||
|
||||
# Restart databases
|
||||
kubectl rollout restart deployment -n bakery-ia --selector='app.kubernetes.io/component=database'
|
||||
|
||||
# Restart services
|
||||
kubectl rollout restart deployment -n bakery-ia --selector='app.kubernetes.io/component=service'
|
||||
```
|
||||
|
||||
### Regenerating TLS Certificates
|
||||
|
||||
**When to Regenerate:**
|
||||
- Certificates expired (October 17, 2028)
|
||||
- Adding new database hosts
|
||||
- Security incident
|
||||
|
||||
**How to Regenerate:**
|
||||
```bash
|
||||
# Regenerate all certificates
|
||||
cd infrastructure/tls && ./generate-certificates.sh
|
||||
|
||||
# Update Kubernetes secrets
|
||||
./scripts/create-tls-secrets.sh
|
||||
|
||||
# Apply new secrets
|
||||
kubectl apply -f infrastructure/kubernetes/base/secrets/postgres-tls-secret.yaml
|
||||
kubectl apply -f infrastructure/kubernetes/base/secrets/redis-tls-secret.yaml
|
||||
|
||||
# Restart databases
|
||||
kubectl rollout restart deployment -n bakery-ia --selector='app.kubernetes.io/component=database'
|
||||
```
|
||||
|
||||
### Creating Backups
|
||||
|
||||
**Manual Backup:**
|
||||
```bash
|
||||
# Create encrypted backup of all databases
|
||||
./scripts/encrypted-backup.sh
|
||||
|
||||
# Backups saved to: /backups/<db>_<timestamp>.sql.gz.gpg
|
||||
```
|
||||
|
||||
**Restore from Backup:**
|
||||
```bash
|
||||
# Decrypt and restore
|
||||
gpg --decrypt backup_file.sql.gz.gpg | gunzip | \
|
||||
kubectl exec -i -n bakery-ia <db-pod> -- \
|
||||
psql -U <user> -d <db>
|
||||
```
|
||||
|
||||
### Adding a New Database
|
||||
|
||||
**Steps:**
|
||||
1. Create database YAML (copy from existing)
|
||||
2. Add PVC to the YAML
|
||||
3. Add TLS volume mount and environment variables
|
||||
4. Update Tiltfile or Skaffold config
|
||||
5. Deploy
|
||||
|
||||
**Example:**
|
||||
```yaml
|
||||
# new-db.yaml
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: new-db
|
||||
namespace: bakery-ia
|
||||
spec:
|
||||
# ... (same structure as other databases)
|
||||
volumes:
|
||||
- name: postgres-data
|
||||
persistentVolumeClaim:
|
||||
claimName: new-db-pvc
|
||||
- name: tls-certs
|
||||
secret:
|
||||
secretName: postgres-tls
|
||||
defaultMode: 0600
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
metadata:
|
||||
name: new-db-pvc
|
||||
namespace: bakery-ia
|
||||
spec:
|
||||
accessModes:
|
||||
- ReadWriteOnce
|
||||
resources:
|
||||
requests:
|
||||
storage: 2Gi
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🎯 Best Practices
|
||||
|
||||
### Security
|
||||
|
||||
1. **Never commit certificates or keys to git**
|
||||
- `.gitignore` already excludes `*.pem` and `*.key`
|
||||
- TLS certificates are generated locally
|
||||
|
||||
2. **Rotate passwords regularly**
|
||||
- Recommended: Every 90 days
|
||||
- Use the password rotation scripts
|
||||
|
||||
3. **Monitor audit logs**
|
||||
- Check PostgreSQL logs daily
|
||||
- Look for failed authentication attempts
|
||||
- Review long-running queries
|
||||
|
||||
4. **Keep certificates up to date**
|
||||
- Current certificates expire: October 17, 2028
|
||||
- Set a calendar reminder for renewal
|
||||
|
||||
### Performance
|
||||
|
||||
1. **TLS has minimal overhead**
|
||||
- ~5-10ms additional latency
|
||||
- Worth the security benefit
|
||||
|
||||
2. **Connection pooling still works**
|
||||
- No changes needed to connection pool settings
|
||||
- TLS connections are reused efficiently
|
||||
|
||||
3. **PVCs don't impact performance**
|
||||
- Same performance as before
|
||||
- Better reliability (no data loss)
|
||||
|
||||
### Development
|
||||
|
||||
1. **Use Tilt for fastest iteration**
|
||||
- Live updates without rebuilds
|
||||
- Visual dashboard for monitoring
|
||||
|
||||
2. **Test locally before pushing**
|
||||
- Verify TLS connections work
|
||||
- Check service logs for SSL errors
|
||||
|
||||
3. **Keep shared code in sync**
|
||||
- Changes to `shared/` affect all services
|
||||
- Test affected services after changes
|
||||
|
||||
---
|
||||
|
||||
## 🆘 Troubleshooting
|
||||
|
||||
### Tilt Issues
|
||||
|
||||
**Problem:** "security-setup" resource fails
|
||||
|
||||
**Solution:**
|
||||
```bash
|
||||
# Check if secrets exist
|
||||
kubectl get secrets -n bakery-ia
|
||||
|
||||
# Manually apply security configs
|
||||
kubectl apply -f infrastructure/kubernetes/base/secrets.yaml
|
||||
kubectl apply -f infrastructure/kubernetes/base/secrets/postgres-tls-secret.yaml
|
||||
kubectl apply -f infrastructure/kubernetes/base/secrets/redis-tls-secret.yaml
|
||||
|
||||
# Restart Tilt
|
||||
tilt down && tilt up -f Tiltfile.secure
|
||||
```
|
||||
|
||||
### Skaffold Issues
|
||||
|
||||
**Problem:** Deployment hooks fail
|
||||
|
||||
**Solution:**
|
||||
```bash
|
||||
# Apply hooks manually
|
||||
kubectl apply -f infrastructure/kubernetes/base/secrets.yaml
|
||||
kubectl apply -f infrastructure/kubernetes/base/secrets/postgres-tls-secret.yaml
|
||||
kubectl apply -f infrastructure/kubernetes/base/secrets/redis-tls-secret.yaml
|
||||
|
||||
# Run skaffold without hooks
|
||||
skaffold dev -f skaffold-secure.yaml --skip-deploy-hooks
|
||||
```
|
||||
|
||||
### Database Won't Start
|
||||
|
||||
**Problem:** Database pod in CrashLoopBackOff
|
||||
|
||||
**Solution:**
|
||||
```bash
|
||||
# Check pod events
|
||||
kubectl describe pod <db-pod> -n bakery-ia
|
||||
|
||||
# Check logs
|
||||
kubectl logs <db-pod> -n bakery-ia
|
||||
|
||||
# Common causes:
|
||||
# 1. TLS certs not mounted - check secret exists
|
||||
# 2. PVC not binding - check storage class
|
||||
# 3. Wrong password - check secrets match .env
|
||||
```
|
||||
|
||||
### Services Can't Connect
|
||||
|
||||
**Problem:** Services show database connection errors
|
||||
|
||||
**Solution:**
|
||||
```bash
|
||||
# 1. Verify database is running
|
||||
kubectl get pods -n bakery-ia -l app.kubernetes.io/component=database
|
||||
|
||||
# 2. Test connection from service pod
|
||||
kubectl exec -n bakery-ia <service-pod> -- nc -zv <db-service> 5432
|
||||
|
||||
# 3. Check if TLS is the issue
|
||||
kubectl logs -n bakery-ia <service-pod> | grep -i ssl
|
||||
|
||||
# 4. Restart service
|
||||
kubectl rollout restart deployment/<service> -n bakery-ia
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📚 Additional Resources
|
||||
|
||||
- **Full Implementation Guide:** [SECURITY_IMPLEMENTATION_COMPLETE.md](SECURITY_IMPLEMENTATION_COMPLETE.md)
|
||||
- **Security Analysis:** [DATABASE_SECURITY_ANALYSIS_REPORT.md](DATABASE_SECURITY_ANALYSIS_REPORT.md)
|
||||
- **Deployment Script:** `scripts/apply-security-changes.sh`
|
||||
- **Backup Script:** `scripts/encrypted-backup.sh`
|
||||
|
||||
---
|
||||
|
||||
## 🎓 Learning Resources
|
||||
|
||||
### TLS/SSL Concepts
|
||||
- PostgreSQL SSL: https://www.postgresql.org/docs/17/ssl-tcp.html
|
||||
- Redis TLS: https://redis.io/docs/management/security/encryption/
|
||||
|
||||
### Kubernetes Security
|
||||
- Secrets: https://kubernetes.io/docs/concepts/configuration/secret/
|
||||
- PVCs: https://kubernetes.io/docs/concepts/storage/persistent-volumes/
|
||||
|
||||
### Python Database Libraries
|
||||
- asyncpg: https://magicstack.github.io/asyncpg/current/
|
||||
- redis-py: https://redis-py.readthedocs.io/
|
||||
|
||||
---
|
||||
|
||||
**Last Updated:** October 18, 2025
|
||||
**Maintained By:** Bakery IA Development Team
|
||||
Reference in New Issue
Block a user