bakery-admin/bakery-ia

Fork 0

Files

Urtzi Alfaro 05da20357d Improve teh securty of teh DB

2025-10-19 19:22:37 +02:00

15 KiB

Raw Blame History

Development with Database Security Enabled

Author: Claude Security Implementation Date: October 18, 2025 Status: Ready for Use

Overview

This guide explains how to develop with the new secure database infrastructure that includes TLS encryption, strong passwords, persistent storage, and audit logging.

🚀 Quick Start

Option 1: Using Tilt (Recommended)

Secure Development Mode:

# Use the secure Tiltfile
tilt up -f Tiltfile.secure

# Or rename it to be default
mv Tiltfile Tiltfile.old
mv Tiltfile.secure Tiltfile
tilt up

Features:

✅ Automatic security setup on startup
✅ TLS certificates applied before databases start
✅ Live code updates with hot reload
✅ Built-in TLS and PVC verification
✅ Visual dashboard at http://localhost:10350

Option 2: Using Skaffold

Secure Development Mode:

# Use the secure Skaffold config
skaffold dev -f skaffold-secure.yaml

# Or rename it to be default
mv skaffold.yaml skaffold.old.yaml
mv skaffold-secure.yaml skaffold.yaml
skaffold dev

Features:

✅ Pre-deployment hooks apply security configs
✅ Post-deployment verification messages
✅ Automatic rebuilds on code changes

Option 3: Manual Deployment

For full control:

# Apply security configurations
./scripts/apply-security-changes.sh

# Deploy with kubectl
kubectl apply -k infrastructure/kubernetes/overlays/dev

# Verify
kubectl get pods -n bakery-ia
kubectl get pvc -n bakery-ia

🔐 What Changed?

Database Connections

Before (Insecure):

# Old connection string
DATABASE_URL = "postgresql+asyncpg://user:password@host:5432/db"

After (Secure):

# New connection string (automatic)
DATABASE_URL = "postgresql+asyncpg://user:strong_password@host:5432/db?ssl=require&sslmode=require"

Key Changes:

ssl=require - Enforces TLS encryption
sslmode=require - Rejects unencrypted connections
Strong 32-character passwords
Automatic SSL parameter addition in shared/database/base.py

Redis Connections

Before (Insecure):

REDIS_URL = "redis://password@host:6379"

After (Secure):

REDIS_URL = "rediss://password@host:6379?ssl_cert_reqs=required"

Key Changes:

rediss:// protocol - Uses TLS
ssl_cert_reqs=required - Enforces certificate validation
Automatic in shared/config/base.py

Environment Variables

New Environment Variables:

# Optional: Disable TLS for local testing (NOT recommended)
REDIS_TLS_ENABLED=false  # Default: true

# Database URLs now include SSL parameters automatically
# No changes needed to your service code!

📁 File Structure Changes

New Files Created

infrastructure/
├── tls/                                    # TLS certificates
│   ├── ca/
│   │   ├── ca-cert.pem                    # Certificate Authority
│   │   └── ca-key.pem                     # CA private key
│   ├── postgres/
│   │   ├── server-cert.pem                # PostgreSQL server cert
│   │   ├── server-key.pem                 # PostgreSQL private key
│   │   └── ca-cert.pem                    # CA for clients
│   ├── redis/
│   │   ├── redis-cert.pem                 # Redis server cert
│   │   ├── redis-key.pem                  # Redis private key
│   │   └── ca-cert.pem                    # CA for clients
│   └── generate-certificates.sh           # Regeneration script
│
└── kubernetes/
    ├── base/
    │   ├── secrets/
    │   │   ├── postgres-tls-secret.yaml   # PostgreSQL TLS secret
    │   │   └── redis-tls-secret.yaml      # Redis TLS secret
    │   └── configmaps/
    │       └── postgres-logging-config.yaml # Audit logging
    └── encryption/
        └── encryption-config.yaml          # Secrets encryption

scripts/
├── encrypted-backup.sh                     # Create encrypted backups
├── apply-security-changes.sh               # Deploy security changes
└── ... (other security scripts)

docs/
├── SECURITY_IMPLEMENTATION_COMPLETE.md    # Full implementation guide
├── DATABASE_SECURITY_ANALYSIS_REPORT.md   # Security analysis
└── DEVELOPMENT_WITH_SECURITY.md           # This file

🔧 Development Workflow

Starting Development

With Tilt (Recommended):

# Start all services with security
tilt up -f Tiltfile.secure

# Watch the Tilt dashboard
open http://localhost:10350

With Skaffold:

# Start development mode
skaffold dev -f skaffold-secure.yaml

# Or with debug ports
skaffold dev -f skaffold-secure.yaml -p debug

Making Code Changes

No changes needed! Your code works the same way:

# Your existing code (unchanged)
from shared.database import DatabaseManager

db_manager = DatabaseManager(
    database_url=settings.DATABASE_URL,
    service_name="my-service"
)

# TLS is automatically added to the connection!

Hot Reload:

Python services: Changes detected automatically, uvicorn reloads
Frontend: Requires rebuild (nginx static files)
Shared libraries: All services reload when changed

Testing Database Connections

Verify TLS is Working:

# Test PostgreSQL with TLS
kubectl exec -n bakery-ia <auth-db-pod> -- \
  psql "postgresql://auth_user@localhost:5432/auth_db?sslmode=require" -c "SELECT version();"

# Test Redis with TLS
kubectl exec -n bakery-ia <redis-pod> -- \
  redis-cli --tls \
    --cert /tls/redis-cert.pem \
    --key /tls/redis-key.pem \
    --cacert /tls/ca-cert.pem \
    PING

# Check if TLS certs are mounted
kubectl exec -n bakery-ia <db-pod> -- ls -la /tls/

Verify from Service:

# In your service code
import asyncpg
import ssl

# This is what happens automatically now:
ssl_context = ssl.create_default_context()
conn = await asyncpg.connect(
    "postgresql://user:pass@host:5432/db",
    ssl=ssl_context
)

Viewing Logs

Database Logs (with audit trail):

# View PostgreSQL logs
kubectl logs -n bakery-ia <db-pod>

# Filter for connections
kubectl logs -n bakery-ia <db-pod> | grep "connection"

# Filter for queries
kubectl logs -n bakery-ia <db-pod> | grep "statement"

# View Redis logs
kubectl logs -n bakery-ia <redis-pod>

Service Logs:

# View service logs
kubectl logs -n bakery-ia <service-pod>

# Follow logs in real-time
kubectl logs -f -n bakery-ia <service-pod>

# View logs in Tilt dashboard
# Click on service in Tilt UI

Debugging Connection Issues

Common Issues:

"SSL not supported" Error

# Check if TLS certs are mounted
kubectl exec -n bakery-ia <db-pod> -- ls /tls/

# Restart the pod
kubectl delete pod <db-pod> -n bakery-ia

# Check secret exists
kubectl get secret postgres-tls -n bakery-ia

"Connection refused" Error

# Check if database is running
kubectl get pods -n bakery-ia -l app.kubernetes.io/component=database

# Check database logs
kubectl logs -n bakery-ia <db-pod>

# Verify service is reachable
kubectl exec -n bakery-ia <service-pod> -- nc -zv <db-service> 5432

"Authentication failed" Error

# Verify password is updated
kubectl get secret database-secrets -n bakery-ia -o jsonpath='{.data.AUTH_DB_PASSWORD}' | base64 -d

# Check .env file has matching password
grep AUTH_DB_PASSWORD .env

# Restart services to pick up new passwords
kubectl rollout restart deployment -n bakery-ia --selector='app.kubernetes.io/component=service'

📊 Monitoring & Observability

Checking PVC Usage

# List all PVCs
kubectl get pvc -n bakery-ia

# Check PVC details
kubectl describe pvc <pvc-name> -n bakery-ia

# Check disk usage in pod
kubectl exec -n bakery-ia <db-pod> -- df -h /var/lib/postgresql/data

Monitoring Database Connections

# Check active connections (PostgreSQL)
kubectl exec -n bakery-ia <db-pod> -- \
  psql -U <user> -d <db> -c "SELECT count(*) FROM pg_stat_activity;"

# Check Redis info
kubectl exec -n bakery-ia <redis-pod> -- \
  redis-cli -a <password> --tls \
    --cert /tls/redis-cert.pem \
    --key /tls/redis-key.pem \
    --cacert /tls/ca-cert.pem \
    INFO clients

Security Audit

# Verify TLS certificates
kubectl exec -n bakery-ia <db-pod> -- \
  openssl x509 -in /tls/server-cert.pem -noout -text

# Check certificate expiry
kubectl exec -n bakery-ia <db-pod> -- \
  openssl x509 -in /tls/server-cert.pem -noout -dates

# Verify pgcrypto extension
kubectl exec -n bakery-ia <db-pod> -- \
  psql -U <user> -d <db> -c "SELECT * FROM pg_extension WHERE extname='pgcrypto';"

🔄 Common Tasks

Rotating Passwords

Manual Rotation:

# Generate new passwords
./scripts/generate-passwords.sh > new-passwords.txt

# Update .env
./scripts/update-env-passwords.sh

# Update Kubernetes secrets
./scripts/update-k8s-secrets.sh

# Apply new secrets
kubectl apply -f infrastructure/kubernetes/base/secrets.yaml

# Restart databases
kubectl rollout restart deployment -n bakery-ia --selector='app.kubernetes.io/component=database'

# Restart services
kubectl rollout restart deployment -n bakery-ia --selector='app.kubernetes.io/component=service'

Regenerating TLS Certificates

When to Regenerate:

Certificates expired (October 17, 2028)
Adding new database hosts
Security incident

How to Regenerate:

# Regenerate all certificates
cd infrastructure/tls && ./generate-certificates.sh

# Update Kubernetes secrets
./scripts/create-tls-secrets.sh

# Apply new secrets
kubectl apply -f infrastructure/kubernetes/base/secrets/postgres-tls-secret.yaml
kubectl apply -f infrastructure/kubernetes/base/secrets/redis-tls-secret.yaml

# Restart databases
kubectl rollout restart deployment -n bakery-ia --selector='app.kubernetes.io/component=database'

Creating Backups

Manual Backup:

# Create encrypted backup of all databases
./scripts/encrypted-backup.sh

# Backups saved to: /backups/<db>_<timestamp>.sql.gz.gpg

Restore from Backup:

# Decrypt and restore
gpg --decrypt backup_file.sql.gz.gpg | gunzip | \
  kubectl exec -i -n bakery-ia <db-pod> -- \
    psql -U <user> -d <db>

Adding a New Database

Steps:

Create database YAML (copy from existing)
Add PVC to the YAML
Add TLS volume mount and environment variables
Update Tiltfile or Skaffold config
Deploy

Example:

# new-db.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: new-db
  namespace: bakery-ia
spec:
  # ... (same structure as other databases)
  volumes:
    - name: postgres-data
      persistentVolumeClaim:
        claimName: new-db-pvc
    - name: tls-certs
      secret:
        secretName: postgres-tls
        defaultMode: 0600
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: new-db-pvc
  namespace: bakery-ia
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 2Gi

🎯 Best Practices

Security

Never commit certificates or keys to git
- .gitignore already excludes *.pem and *.key
- TLS certificates are generated locally
Rotate passwords regularly
- Recommended: Every 90 days
- Use the password rotation scripts
Monitor audit logs
- Check PostgreSQL logs daily
- Look for failed authentication attempts
- Review long-running queries
Keep certificates up to date
- Current certificates expire: October 17, 2028
- Set a calendar reminder for renewal

Performance

TLS has minimal overhead
- ~5-10ms additional latency
- Worth the security benefit
Connection pooling still works
- No changes needed to connection pool settings
- TLS connections are reused efficiently
PVCs don't impact performance
- Same performance as before
- Better reliability (no data loss)

Development

Use Tilt for fastest iteration
- Live updates without rebuilds
- Visual dashboard for monitoring
Test locally before pushing
- Verify TLS connections work
- Check service logs for SSL errors
Keep shared code in sync
- Changes to shared/ affect all services
- Test affected services after changes

🆘 Troubleshooting

Tilt Issues

Problem: "security-setup" resource fails

Solution:

# Check if secrets exist
kubectl get secrets -n bakery-ia

# Manually apply security configs
kubectl apply -f infrastructure/kubernetes/base/secrets.yaml
kubectl apply -f infrastructure/kubernetes/base/secrets/postgres-tls-secret.yaml
kubectl apply -f infrastructure/kubernetes/base/secrets/redis-tls-secret.yaml

# Restart Tilt
tilt down && tilt up -f Tiltfile.secure

Skaffold Issues

Problem: Deployment hooks fail

Solution:

# Apply hooks manually
kubectl apply -f infrastructure/kubernetes/base/secrets.yaml
kubectl apply -f infrastructure/kubernetes/base/secrets/postgres-tls-secret.yaml
kubectl apply -f infrastructure/kubernetes/base/secrets/redis-tls-secret.yaml

# Run skaffold without hooks
skaffold dev -f skaffold-secure.yaml --skip-deploy-hooks

Database Won't Start

Problem: Database pod in CrashLoopBackOff

Solution:

# Check pod events
kubectl describe pod <db-pod> -n bakery-ia

# Check logs
kubectl logs <db-pod> -n bakery-ia

# Common causes:
# 1. TLS certs not mounted - check secret exists
# 2. PVC not binding - check storage class
# 3. Wrong password - check secrets match .env

Services Can't Connect

Problem: Services show database connection errors

Solution:

# 1. Verify database is running
kubectl get pods -n bakery-ia -l app.kubernetes.io/component=database

# 2. Test connection from service pod
kubectl exec -n bakery-ia <service-pod> -- nc -zv <db-service> 5432

# 3. Check if TLS is the issue
kubectl logs -n bakery-ia <service-pod> | grep -i ssl

# 4. Restart service
kubectl rollout restart deployment/<service> -n bakery-ia

📚 Additional Resources

Full Implementation Guide: SECURITY_IMPLEMENTATION_COMPLETE.md
Security Analysis: DATABASE_SECURITY_ANALYSIS_REPORT.md
Deployment Script: scripts/apply-security-changes.sh
Backup Script: scripts/encrypted-backup.sh

🎓 Learning Resources

TLS/SSL Concepts

PostgreSQL SSL: https://www.postgresql.org/docs/17/ssl-tcp.html
Redis TLS: https://redis.io/docs/management/security/encryption/

Kubernetes Security

Python Database Libraries

asyncpg: https://magicstack.github.io/asyncpg/current/
redis-py: https://redis-py.readthedocs.io/

Last Updated: October 18, 2025 Maintained By: Bakery IA Development Team

15 KiB Raw Blame History

Development with Database Security Enabled

Overview

🚀 Quick Start

Option 1: Using Tilt (Recommended)

Option 2: Using Skaffold

Option 3: Manual Deployment

🔐 What Changed?

Database Connections

Redis Connections

Environment Variables

📁 File Structure Changes

New Files Created

🔧 Development Workflow

Starting Development

Making Code Changes

Testing Database Connections

Viewing Logs

Debugging Connection Issues

📊 Monitoring & Observability

Checking PVC Usage

Monitoring Database Connections

Security Audit

🔄 Common Tasks

Rotating Passwords

Regenerating TLS Certificates

Creating Backups

Adding a New Database

🎯 Best Practices

Security

Performance

Development

🆘 Troubleshooting

Tilt Issues

Skaffold Issues

Database Won't Start

Services Can't Connect

📚 Additional Resources

🎓 Learning Resources

TLS/SSL Concepts

Kubernetes Security

Python Database Libraries

15 KiB

Raw Blame History