Fix startup issues

DEPLOYMENT_COMMANDS.md

@@ -0,0 +1,322 @@
+# Deployment Commands - Quick Reference
+
+## Implementation Complete ✅
+
+All changes are implemented. Services now only verify database readiness - they never run migrations.
+
+---
+
+## Deploy the New Architecture
+
+### Option 1: Skaffold (Recommended)
+
+```bash
+# Development mode (auto-rebuild on changes)
+skaffold dev
+
+# Production deployment
+skaffold run
+```
+
+### Option 2: Manual Deployment
+
+```bash
+# 1. Build all service images
+for service in auth orders inventory external pos sales recipes \
+               training suppliers tenant notification forecasting \
+               production alert-processor; do
+  docker build -t bakery/${service}-service:latest services/${service}/
+done
+
+# 2. Apply Kubernetes manifests
+kubectl apply -f infrastructure/kubernetes/base/
+
+# 3. Wait for rollout
+kubectl rollout status deployment --all -n bakery-ia
+```
+
+---
+
+## Verification Commands
+
+### Check Services Are Using New Code:
+
+```bash
+# Check external service logs for verification (not migration)
+kubectl logs -n bakery-ia deployment/external-service | grep -i "verification"
+
+# Expected output:
+# [info] Database verification mode - checking database is ready
+# [info] Database verification successful
+
+# Should NOT see (old behavior):
+# [info] Running pending migrations
+```
+
+### Check All Services:
+
+```bash
+# Check all service logs
+for service in auth orders inventory external pos sales recipes \
+               training suppliers tenant notification forecasting \
+               production alert-processor; do
+  echo "=== Checking $service-service ==="
+  kubectl logs -n bakery-ia deployment/${service}-service --tail=20 | grep -E "(verification|migration)" || echo "No logs yet"
+done
+```
+
+### Check Startup Times:
+
+```bash
+# Watch pod startup times
+kubectl get events -n bakery-ia --sort-by='.lastTimestamp' --watch
+
+# Or check specific service
+kubectl describe pod -n bakery-ia -l app.kubernetes.io/name=external-service | grep -A 5 "Events:"
+```
+
+---
+
+## Troubleshooting
+
+### Service Won't Start - "Database is empty"
+
+```bash
+# 1. Check migration job status
+kubectl get jobs -n bakery-ia | grep migration
+
+# 2. Check specific migration job
+kubectl logs -n bakery-ia job/external-migration
+
+# 3. Re-run migration job if needed
+kubectl delete job external-migration -n bakery-ia
+kubectl apply -f infrastructure/kubernetes/base/migrations/external-migration.yaml
+```
+
+### Service Won't Start - "No migration files found"
+
+```bash
+# 1. Check if migrations exist in image
+kubectl exec -n bakery-ia deployment/external-service -- ls -la /app/migrations/versions/
+
+# 2. If missing, regenerate and rebuild
+./regenerate_migrations_k8s.sh --verbose
+skaffold build
+kubectl rollout restart deployment/external-service -n bakery-ia
+```
+
+### Check Migration Job Logs:
+
+```bash
+# List all migration jobs
+kubectl get jobs -n bakery-ia | grep migration
+
+# Check specific job logs
+kubectl logs -n bakery-ia job/<service>-migration
+
+# Example:
+kubectl logs -n bakery-ia job/auth-migration
+```
+
+---
+
+## Performance Testing
+
+### Measure Startup Time Improvement:
+
+```bash
+# 1. Record current startup times
+kubectl get events -n bakery-ia --sort-by='.lastTimestamp' | grep "Started container" > before.txt
+
+# 2. Deploy new code
+skaffold run
+
+# 3. Restart services to measure
+kubectl rollout restart deployment --all -n bakery-ia
+
+# 4. Record new startup times
+kubectl get events -n bakery-ia --sort-by='.lastTimestamp' | grep "Started container" > after.txt
+
+# 5. Compare (should be 50-80% faster)
+diff before.txt after.txt
+```
+
+### Monitor Database Load:
+
+```bash
+# Check database connections during startup
+kubectl exec -n bakery-ia external-db-<pod> -- \
+  psql -U external_user -d external_db -c \
+  "SELECT count(*) FROM pg_stat_activity WHERE datname='external_db';"
+```
+
+---
+
+## Rollback (If Needed)
+
+### Rollback Deployments:
+
+```bash
+# Rollback specific service
+kubectl rollout undo deployment/external-service -n bakery-ia
+
+# Rollback all services
+kubectl rollout undo deployment --all -n bakery-ia
+
+# Check rollout status
+kubectl rollout status deployment --all -n bakery-ia
+```
+
+### Rollback to Specific Revision:
+
+```bash
+# List revisions
+kubectl rollout history deployment/external-service -n bakery-ia
+
+# Rollback to specific revision
+kubectl rollout undo deployment/external-service --to-revision=2 -n bakery-ia
+```
+
+---
+
+## Clean Deployment
+
+### If You Want Fresh Start:
+
+```bash
+# 1. Delete everything
+kubectl delete namespace bakery-ia
+
+# 2. Recreate namespace
+kubectl create namespace bakery-ia
+
+# 3. Apply all manifests
+kubectl apply -f infrastructure/kubernetes/base/
+
+# 4. Wait for all to be ready
+kubectl wait --for=condition=ready pod --all -n bakery-ia --timeout=300s
+```
+
+---
+
+## Health Checks
+
+### Check All Pods:
+
+```bash
+kubectl get pods -n bakery-ia
+```
+
+### Check Services Are Ready:
+
+```bash
+# Check all services
+kubectl get deployments -n bakery-ia
+
+# Check specific service health
+kubectl exec -n bakery-ia deployment/external-service -- \
+  curl -s http://localhost:8000/health/live
+```
+
+### Check Migration Jobs Completed:
+
+```bash
+# Should all show "Complete"
+kubectl get jobs -n bakery-ia | grep migration
+```
+
+---
+
+## Useful Aliases
+
+Add to your `~/.bashrc` or `~/.zshrc`:
+
+```bash
+# Kubernetes bakery-ia shortcuts
+alias k='kubectl'
+alias kn='kubectl -n bakery-ia'
+alias kp='kubectl get pods -n bakery-ia'
+alias kd='kubectl get deployments -n bakery-ia'
+alias kj='kubectl get jobs -n bakery-ia'
+alias kl='kubectl logs -n bakery-ia'
+alias kdesc='kubectl describe -n bakery-ia'
+
+# Quick log checks
+alias klogs='kubectl logs -n bakery-ia deployment/'
+
+# Example usage:
+# klogs external-service | grep verification
+```
+
+---
+
+## Expected Output Examples
+
+### Migration Job (Successful):
+
+```
+[info] Migration job starting service=external
+[info] Migration mode - running database migrations
+[info] Running pending migrations
+INFO [alembic.runtime.migration] Context impl PostgresqlImpl.
+INFO [alembic.runtime.migration] Will assume transactional DDL.
+[info] Migrations applied successfully
+[info] Migration job completed successfully
+```
+
+### Service Startup (New Behavior):
+
+```
+[info] Starting external-service version=1.0.0
+[info] Database connection established
+[info] Database verification mode - checking database is ready
+[info] Database state checked
+[info] Database verification successful
+       migration_count=1 current_revision=374752db316e table_count=6
+[info] Database verification completed
+[info] external-service started successfully
+```
+
+---
+
+## CI/CD Integration
+
+### GitHub Actions Example:
+
+```yaml
+name: Deploy to Kubernetes
+on:
+  push:
+    branches: [main]
+
+jobs:
+  deploy:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v2
+
+      - name: Build and push images
+        run: skaffold build
+
+      - name: Deploy to cluster
+        run: skaffold run
+
+      - name: Verify deployment
+        run: |
+          kubectl rollout status deployment --all -n bakery-ia
+          kubectl get pods -n bakery-ia
+```
+
+---
+
+## Summary
+
+**To Deploy**: Just run `skaffold dev` or `skaffold run`
+
+**To Verify**: Check logs show "verification" not "migration"
+
+**To Troubleshoot**: Check migration job logs first
+
+**Expected Result**: Services start 50-80% faster, no redundant migration execution
+
+**Status**: ✅ Ready to deploy!