bakery-ia/docs/DEVELOPMENT_WITH_SECURITY.md

# Development with Database Security Enabled

**Author:** Claude Security Implementation
**Date:** October 18, 2025
**Status:** Ready for Use

---

## Overview

This guide explains how to develop with the new secure database infrastructure that includes TLS encryption, strong passwords, persistent storage, and audit logging.

---

## 🚀 Quick Start

### Option 1: Using Tilt (Recommended)

**Secure Development Mode:**
```bash
# Use the secure Tiltfile
tilt up -f Tiltfile.secure

# Or rename it to be default
mv Tiltfile Tiltfile.old
mv Tiltfile.secure Tiltfile
tilt up
```

**Features:**
- ✅ Automatic security setup on startup
- ✅ TLS certificates applied before databases start
- ✅ Live code updates with hot reload
- ✅ Built-in TLS and PVC verification
- ✅ Visual dashboard at http://localhost:10350

### Option 2: Using Skaffold

**Secure Development Mode:**
```bash
# Use the secure Skaffold config
skaffold dev -f skaffold-secure.yaml

# Or rename it to be default
mv skaffold.yaml skaffold.old.yaml
mv skaffold-secure.yaml skaffold.yaml
skaffold dev
```

**Features:**
- ✅ Pre-deployment hooks apply security configs
- ✅ Post-deployment verification messages
- ✅ Automatic rebuilds on code changes

### Option 3: Manual Deployment

**For full control:**
```bash
# Apply security configurations
./scripts/apply-security-changes.sh

# Deploy with kubectl
kubectl apply -k infrastructure/kubernetes/overlays/dev

# Verify
kubectl get pods -n bakery-ia
kubectl get pvc -n bakery-ia
```

---

## 🔐 What Changed?

### Database Connections

**Before (Insecure):**
```python
# Old connection string
DATABASE_URL = "postgresql+asyncpg://user:password@host:5432/db"
```

**After (Secure):**
```python
# New connection string (automatic)
DATABASE_URL = "postgresql+asyncpg://user:strong_password@host:5432/db?ssl=require&sslmode=require"
```

**Key Changes:**
- `ssl=require` - Enforces TLS encryption
- `sslmode=require` - Rejects unencrypted connections
- Strong 32-character passwords
- Automatic SSL parameter addition in `shared/database/base.py`

### Redis Connections

**Before (Insecure):**
```python
REDIS_URL = "redis://password@host:6379"
```

**After (Secure):**
```python
REDIS_URL = "rediss://password@host:6379?ssl_cert_reqs=required"
```

**Key Changes:**
- `rediss://` protocol - Uses TLS
- `ssl_cert_reqs=required` - Enforces certificate validation
- Automatic in `shared/config/base.py`

### Environment Variables

**New Environment Variables:**
```bash
# Optional: Disable TLS for local testing (NOT recommended)
REDIS_TLS_ENABLED=false  # Default: true

# Database URLs now include SSL parameters automatically
# No changes needed to your service code!
```

---

## 📁 File Structure Changes

### New Files Created

```
infrastructure/
├── tls/                                    # TLS certificates
│   ├── ca/
│   │   ├── ca-cert.pem                    # Certificate Authority
│   │   └── ca-key.pem                     # CA private key
│   ├── postgres/
│   │   ├── server-cert.pem                # PostgreSQL server cert
│   │   ├── server-key.pem                 # PostgreSQL private key
│   │   └── ca-cert.pem                    # CA for clients
│   ├── redis/
│   │   ├── redis-cert.pem                 # Redis server cert
│   │   ├── redis-key.pem                  # Redis private key
│   │   └── ca-cert.pem                    # CA for clients
│   └── generate-certificates.sh           # Regeneration script
│
└── kubernetes/
    ├── base/
    │   ├── secrets/
    │   │   ├── postgres-tls-secret.yaml   # PostgreSQL TLS secret
    │   │   └── redis-tls-secret.yaml      # Redis TLS secret
    │   └── configmaps/
    │       └── postgres-logging-config.yaml # Audit logging
    └── encryption/
        └── encryption-config.yaml          # Secrets encryption

scripts/
├── encrypted-backup.sh                     # Create encrypted backups
├── apply-security-changes.sh               # Deploy security changes
└── ... (other security scripts)

docs/
├── SECURITY_IMPLEMENTATION_COMPLETE.md    # Full implementation guide
├── DATABASE_SECURITY_ANALYSIS_REPORT.md   # Security analysis
└── DEVELOPMENT_WITH_SECURITY.md           # This file
```

---

## 🔧 Development Workflow

### Starting Development

**With Tilt (Recommended):**
```bash
# Start all services with security
tilt up -f Tiltfile.secure

# Watch the Tilt dashboard
open http://localhost:10350
```

**With Skaffold:**
```bash
# Start development mode
skaffold dev -f skaffold-secure.yaml

# Or with debug ports
skaffold dev -f skaffold-secure.yaml -p debug
```

### Making Code Changes

**No changes needed!** Your code works the same way:

```python
# Your existing code (unchanged)
from shared.database import DatabaseManager

db_manager = DatabaseManager(
    database_url=settings.DATABASE_URL,
    service_name="my-service"
)

# TLS is automatically added to the connection!
```

**Hot Reload:**
- Python services: Changes detected automatically, uvicorn reloads
- Frontend: Requires rebuild (nginx static files)
- Shared libraries: All services reload when changed

### Testing Database Connections

**Verify TLS is Working:**
```bash
# Test PostgreSQL with TLS
kubectl exec -n bakery-ia <auth-db-pod> -- \
  psql "postgresql://auth_user@localhost:5432/auth_db?sslmode=require" -c "SELECT version();"

# Test Redis with TLS
kubectl exec -n bakery-ia <redis-pod> -- \
  redis-cli --tls \
    --cert /tls/redis-cert.pem \
    --key /tls/redis-key.pem \
    --cacert /tls/ca-cert.pem \
    PING

# Check if TLS certs are mounted
kubectl exec -n bakery-ia <db-pod> -- ls -la /tls/
```

**Verify from Service:**
```python
# In your service code
import asyncpg
import ssl

# This is what happens automatically now:
ssl_context = ssl.create_default_context()
conn = await asyncpg.connect(
    "postgresql://user:pass@host:5432/db",
    ssl=ssl_context
)
```

### Viewing Logs

**Database Logs (with audit trail):**
```bash
# View PostgreSQL logs
kubectl logs -n bakery-ia <db-pod>

# Filter for connections
kubectl logs -n bakery-ia <db-pod> | grep "connection"

# Filter for queries
kubectl logs -n bakery-ia <db-pod> | grep "statement"

# View Redis logs
kubectl logs -n bakery-ia <redis-pod>
```

**Service Logs:**
```bash
# View service logs
kubectl logs -n bakery-ia <service-pod>

# Follow logs in real-time
kubectl logs -f -n bakery-ia <service-pod>

# View logs in Tilt dashboard
# Click on service in Tilt UI
```

### Debugging Connection Issues

**Common Issues:**

1. **"SSL not supported" Error**

```bash
# Check if TLS certs are mounted
kubectl exec -n bakery-ia <db-pod> -- ls /tls/

# Restart the pod
kubectl delete pod <db-pod> -n bakery-ia

# Check secret exists
kubectl get secret postgres-tls -n bakery-ia
```

2. **"Connection refused" Error**

```bash
# Check if database is running
kubectl get pods -n bakery-ia -l app.kubernetes.io/component=database

# Check database logs
kubectl logs -n bakery-ia <db-pod>

# Verify service is reachable
kubectl exec -n bakery-ia <service-pod> -- nc -zv <db-service> 5432
```

3. **"Authentication failed" Error**

```bash
# Verify password is updated
kubectl get secret database-secrets -n bakery-ia -o jsonpath='{.data.AUTH_DB_PASSWORD}' | base64 -d

# Check .env file has matching password
grep AUTH_DB_PASSWORD .env

# Restart services to pick up new passwords
kubectl rollout restart deployment -n bakery-ia --selector='app.kubernetes.io/component=service'
```

---

## 📊 Monitoring & Observability

### Checking PVC Usage

```bash
# List all PVCs
kubectl get pvc -n bakery-ia

# Check PVC details
kubectl describe pvc <pvc-name> -n bakery-ia

# Check disk usage in pod
kubectl exec -n bakery-ia <db-pod> -- df -h /var/lib/postgresql/data
```

### Monitoring Database Connections

```bash
# Check active connections (PostgreSQL)
kubectl exec -n bakery-ia <db-pod> -- \
  psql -U <user> -d <db> -c "SELECT count(*) FROM pg_stat_activity;"

# Check Redis info
kubectl exec -n bakery-ia <redis-pod> -- \
  redis-cli -a <password> --tls \
    --cert /tls/redis-cert.pem \
    --key /tls/redis-key.pem \
    --cacert /tls/ca-cert.pem \
    INFO clients
```

### Security Audit

```bash
# Verify TLS certificates
kubectl exec -n bakery-ia <db-pod> -- \
  openssl x509 -in /tls/server-cert.pem -noout -text

# Check certificate expiry
kubectl exec -n bakery-ia <db-pod> -- \
  openssl x509 -in /tls/server-cert.pem -noout -dates

# Verify pgcrypto extension
kubectl exec -n bakery-ia <db-pod> -- \
  psql -U <user> -d <db> -c "SELECT * FROM pg_extension WHERE extname='pgcrypto';"
```

---

## 🔄 Common Tasks

### Rotating Passwords

**Manual Rotation:**
```bash
# Generate new passwords
./scripts/generate-passwords.sh > new-passwords.txt

# Update .env
./scripts/update-env-passwords.sh

# Update Kubernetes secrets
./scripts/update-k8s-secrets.sh

# Apply new secrets
kubectl apply -f infrastructure/kubernetes/base/secrets.yaml

# Restart databases
kubectl rollout restart deployment -n bakery-ia --selector='app.kubernetes.io/component=database'

# Restart services
kubectl rollout restart deployment -n bakery-ia --selector='app.kubernetes.io/component=service'
```

### Regenerating TLS Certificates

**When to Regenerate:**
- Certificates expired (October 17, 2028)
- Adding new database hosts
- Security incident

**How to Regenerate:**
```bash
# Regenerate all certificates
cd infrastructure/tls && ./generate-certificates.sh

# Update Kubernetes secrets
./scripts/create-tls-secrets.sh

# Apply new secrets
kubectl apply -f infrastructure/kubernetes/base/secrets/postgres-tls-secret.yaml
kubectl apply -f infrastructure/kubernetes/base/secrets/redis-tls-secret.yaml

# Restart databases
kubectl rollout restart deployment -n bakery-ia --selector='app.kubernetes.io/component=database'
```

### Creating Backups

**Manual Backup:**
```bash
# Create encrypted backup of all databases
./scripts/encrypted-backup.sh

# Backups saved to: /backups/<db>_<timestamp>.sql.gz.gpg
```

**Restore from Backup:**
```bash
# Decrypt and restore
gpg --decrypt backup_file.sql.gz.gpg | gunzip | \
  kubectl exec -i -n bakery-ia <db-pod> -- \
    psql -U <user> -d <db>
```

### Adding a New Database

**Steps:**
1. Create database YAML (copy from existing)
2. Add PVC to the YAML
3. Add TLS volume mount and environment variables
4. Update Tiltfile or Skaffold config
5. Deploy

**Example:**
```yaml
# new-db.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: new-db
  namespace: bakery-ia
spec:
  # ... (same structure as other databases)
  volumes:
    - name: postgres-data
      persistentVolumeClaim:
        claimName: new-db-pvc
    - name: tls-certs
      secret:
        secretName: postgres-tls
        defaultMode: 0600
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: new-db-pvc
  namespace: bakery-ia
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 2Gi
```

---

## 🎯 Best Practices

### Security

1. **Never commit certificates or keys to git**
   - `.gitignore` already excludes `*.pem` and `*.key`
   - TLS certificates are generated locally

2. **Rotate passwords regularly**
   - Recommended: Every 90 days
   - Use the password rotation scripts

3. **Monitor audit logs**
   - Check PostgreSQL logs daily
   - Look for failed authentication attempts
   - Review long-running queries

4. **Keep certificates up to date**
   - Current certificates expire: October 17, 2028
   - Set a calendar reminder for renewal

### Performance

1. **TLS has minimal overhead**
   - ~5-10ms additional latency
   - Worth the security benefit

2. **Connection pooling still works**
   - No changes needed to connection pool settings
   - TLS connections are reused efficiently

3. **PVCs don't impact performance**
   - Same performance as before
   - Better reliability (no data loss)

### Development

1. **Use Tilt for fastest iteration**
   - Live updates without rebuilds
   - Visual dashboard for monitoring

2. **Test locally before pushing**
   - Verify TLS connections work
   - Check service logs for SSL errors

3. **Keep shared code in sync**
   - Changes to `shared/` affect all services
   - Test affected services after changes

---

## 🆘 Troubleshooting

### Tilt Issues

**Problem:** "security-setup" resource fails

**Solution:**
```bash
# Check if secrets exist
kubectl get secrets -n bakery-ia

# Manually apply security configs
kubectl apply -f infrastructure/kubernetes/base/secrets.yaml
kubectl apply -f infrastructure/kubernetes/base/secrets/postgres-tls-secret.yaml
kubectl apply -f infrastructure/kubernetes/base/secrets/redis-tls-secret.yaml

# Restart Tilt
tilt down && tilt up -f Tiltfile.secure
```

### Skaffold Issues

**Problem:** Deployment hooks fail

**Solution:**
```bash
# Apply hooks manually
kubectl apply -f infrastructure/kubernetes/base/secrets.yaml
kubectl apply -f infrastructure/kubernetes/base/secrets/postgres-tls-secret.yaml
kubectl apply -f infrastructure/kubernetes/base/secrets/redis-tls-secret.yaml

# Run skaffold without hooks
skaffold dev -f skaffold-secure.yaml --skip-deploy-hooks
```

### Database Won't Start

**Problem:** Database pod in CrashLoopBackOff

**Solution:**
```bash
# Check pod events
kubectl describe pod <db-pod> -n bakery-ia

# Check logs
kubectl logs <db-pod> -n bakery-ia

# Common causes:
# 1. TLS certs not mounted - check secret exists
# 2. PVC not binding - check storage class
# 3. Wrong password - check secrets match .env
```

### Services Can't Connect

**Problem:** Services show database connection errors

**Solution:**
```bash
# 1. Verify database is running
kubectl get pods -n bakery-ia -l app.kubernetes.io/component=database

# 2. Test connection from service pod
kubectl exec -n bakery-ia <service-pod> -- nc -zv <db-service> 5432

# 3. Check if TLS is the issue
kubectl logs -n bakery-ia <service-pod> | grep -i ssl

# 4. Restart service
kubectl rollout restart deployment/<service> -n bakery-ia
```

---

## 📚 Additional Resources

- **Full Implementation Guide:** [SECURITY_IMPLEMENTATION_COMPLETE.md](SECURITY_IMPLEMENTATION_COMPLETE.md)
- **Security Analysis:** [DATABASE_SECURITY_ANALYSIS_REPORT.md](DATABASE_SECURITY_ANALYSIS_REPORT.md)
- **Deployment Script:** `scripts/apply-security-changes.sh`
- **Backup Script:** `scripts/encrypted-backup.sh`

---

## 🎓 Learning Resources

### TLS/SSL Concepts
- PostgreSQL SSL: https://www.postgresql.org/docs/17/ssl-tcp.html
- Redis TLS: https://redis.io/docs/management/security/encryption/

### Kubernetes Security
- Secrets: https://kubernetes.io/docs/concepts/configuration/secret/
- PVCs: https://kubernetes.io/docs/concepts/storage/persistent-volumes/

### Python Database Libraries
- asyncpg: https://magicstack.github.io/asyncpg/current/
- redis-py: https://redis-py.readthedocs.io/

---

**Last Updated:** October 18, 2025
**Maintained By:** Bakery IA Development Team