323 lines
7.1 KiB
Markdown
323 lines
7.1 KiB
Markdown
# Deployment Commands - Quick Reference
|
|
|
|
## Implementation Complete ✅
|
|
|
|
All changes are implemented. Services now only verify database readiness - they never run migrations.
|
|
|
|
---
|
|
|
|
## Deploy the New Architecture
|
|
|
|
### Option 1: Skaffold (Recommended)
|
|
|
|
```bash
|
|
# Development mode (auto-rebuild on changes)
|
|
skaffold dev
|
|
|
|
# Production deployment
|
|
skaffold run
|
|
```
|
|
|
|
### Option 2: Manual Deployment
|
|
|
|
```bash
|
|
# 1. Build all service images
|
|
for service in auth orders inventory external pos sales recipes \
|
|
training suppliers tenant notification forecasting \
|
|
production alert-processor; do
|
|
docker build -t bakery/${service}-service:latest services/${service}/
|
|
done
|
|
|
|
# 2. Apply Kubernetes manifests
|
|
kubectl apply -f infrastructure/kubernetes/base/
|
|
|
|
# 3. Wait for rollout
|
|
kubectl rollout status deployment --all -n bakery-ia
|
|
```
|
|
|
|
---
|
|
|
|
## Verification Commands
|
|
|
|
### Check Services Are Using New Code:
|
|
|
|
```bash
|
|
# Check external service logs for verification (not migration)
|
|
kubectl logs -n bakery-ia deployment/external-service | grep -i "verification"
|
|
|
|
# Expected output:
|
|
# [info] Database verification mode - checking database is ready
|
|
# [info] Database verification successful
|
|
|
|
# Should NOT see (old behavior):
|
|
# [info] Running pending migrations
|
|
```
|
|
|
|
### Check All Services:
|
|
|
|
```bash
|
|
# Check all service logs
|
|
for service in auth orders inventory external pos sales recipes \
|
|
training suppliers tenant notification forecasting \
|
|
production alert-processor; do
|
|
echo "=== Checking $service-service ==="
|
|
kubectl logs -n bakery-ia deployment/${service}-service --tail=20 | grep -E "(verification|migration)" || echo "No logs yet"
|
|
done
|
|
```
|
|
|
|
### Check Startup Times:
|
|
|
|
```bash
|
|
# Watch pod startup times
|
|
kubectl get events -n bakery-ia --sort-by='.lastTimestamp' --watch
|
|
|
|
# Or check specific service
|
|
kubectl describe pod -n bakery-ia -l app.kubernetes.io/name=external-service | grep -A 5 "Events:"
|
|
```
|
|
|
|
---
|
|
|
|
## Troubleshooting
|
|
|
|
### Service Won't Start - "Database is empty"
|
|
|
|
```bash
|
|
# 1. Check migration job status
|
|
kubectl get jobs -n bakery-ia | grep migration
|
|
|
|
# 2. Check specific migration job
|
|
kubectl logs -n bakery-ia job/external-migration
|
|
|
|
# 3. Re-run migration job if needed
|
|
kubectl delete job external-migration -n bakery-ia
|
|
kubectl apply -f infrastructure/kubernetes/base/migrations/external-migration.yaml
|
|
```
|
|
|
|
### Service Won't Start - "No migration files found"
|
|
|
|
```bash
|
|
# 1. Check if migrations exist in image
|
|
kubectl exec -n bakery-ia deployment/external-service -- ls -la /app/migrations/versions/
|
|
|
|
# 2. If missing, regenerate and rebuild
|
|
./regenerate_migrations_k8s.sh --verbose
|
|
skaffold build
|
|
kubectl rollout restart deployment/external-service -n bakery-ia
|
|
```
|
|
|
|
### Check Migration Job Logs:
|
|
|
|
```bash
|
|
# List all migration jobs
|
|
kubectl get jobs -n bakery-ia | grep migration
|
|
|
|
# Check specific job logs
|
|
kubectl logs -n bakery-ia job/<service>-migration
|
|
|
|
# Example:
|
|
kubectl logs -n bakery-ia job/auth-migration
|
|
```
|
|
|
|
---
|
|
|
|
## Performance Testing
|
|
|
|
### Measure Startup Time Improvement:
|
|
|
|
```bash
|
|
# 1. Record current startup times
|
|
kubectl get events -n bakery-ia --sort-by='.lastTimestamp' | grep "Started container" > before.txt
|
|
|
|
# 2. Deploy new code
|
|
skaffold run
|
|
|
|
# 3. Restart services to measure
|
|
kubectl rollout restart deployment --all -n bakery-ia
|
|
|
|
# 4. Record new startup times
|
|
kubectl get events -n bakery-ia --sort-by='.lastTimestamp' | grep "Started container" > after.txt
|
|
|
|
# 5. Compare (should be 50-80% faster)
|
|
diff before.txt after.txt
|
|
```
|
|
|
|
### Monitor Database Load:
|
|
|
|
```bash
|
|
# Check database connections during startup
|
|
kubectl exec -n bakery-ia external-db-<pod> -- \
|
|
psql -U external_user -d external_db -c \
|
|
"SELECT count(*) FROM pg_stat_activity WHERE datname='external_db';"
|
|
```
|
|
|
|
---
|
|
|
|
## Rollback (If Needed)
|
|
|
|
### Rollback Deployments:
|
|
|
|
```bash
|
|
# Rollback specific service
|
|
kubectl rollout undo deployment/external-service -n bakery-ia
|
|
|
|
# Rollback all services
|
|
kubectl rollout undo deployment --all -n bakery-ia
|
|
|
|
# Check rollout status
|
|
kubectl rollout status deployment --all -n bakery-ia
|
|
```
|
|
|
|
### Rollback to Specific Revision:
|
|
|
|
```bash
|
|
# List revisions
|
|
kubectl rollout history deployment/external-service -n bakery-ia
|
|
|
|
# Rollback to specific revision
|
|
kubectl rollout undo deployment/external-service --to-revision=2 -n bakery-ia
|
|
```
|
|
|
|
---
|
|
|
|
## Clean Deployment
|
|
|
|
### If You Want Fresh Start:
|
|
|
|
```bash
|
|
# 1. Delete everything
|
|
kubectl delete namespace bakery-ia
|
|
|
|
# 2. Recreate namespace
|
|
kubectl create namespace bakery-ia
|
|
|
|
# 3. Apply all manifests
|
|
kubectl apply -f infrastructure/kubernetes/base/
|
|
|
|
# 4. Wait for all to be ready
|
|
kubectl wait --for=condition=ready pod --all -n bakery-ia --timeout=300s
|
|
```
|
|
|
|
---
|
|
|
|
## Health Checks
|
|
|
|
### Check All Pods:
|
|
|
|
```bash
|
|
kubectl get pods -n bakery-ia
|
|
```
|
|
|
|
### Check Services Are Ready:
|
|
|
|
```bash
|
|
# Check all services
|
|
kubectl get deployments -n bakery-ia
|
|
|
|
# Check specific service health
|
|
kubectl exec -n bakery-ia deployment/external-service -- \
|
|
curl -s http://localhost:8000/health/live
|
|
```
|
|
|
|
### Check Migration Jobs Completed:
|
|
|
|
```bash
|
|
# Should all show "Complete"
|
|
kubectl get jobs -n bakery-ia | grep migration
|
|
```
|
|
|
|
---
|
|
|
|
## Useful Aliases
|
|
|
|
Add to your `~/.bashrc` or `~/.zshrc`:
|
|
|
|
```bash
|
|
# Kubernetes bakery-ia shortcuts
|
|
alias k='kubectl'
|
|
alias kn='kubectl -n bakery-ia'
|
|
alias kp='kubectl get pods -n bakery-ia'
|
|
alias kd='kubectl get deployments -n bakery-ia'
|
|
alias kj='kubectl get jobs -n bakery-ia'
|
|
alias kl='kubectl logs -n bakery-ia'
|
|
alias kdesc='kubectl describe -n bakery-ia'
|
|
|
|
# Quick log checks
|
|
alias klogs='kubectl logs -n bakery-ia deployment/'
|
|
|
|
# Example usage:
|
|
# klogs external-service | grep verification
|
|
```
|
|
|
|
---
|
|
|
|
## Expected Output Examples
|
|
|
|
### Migration Job (Successful):
|
|
|
|
```
|
|
[info] Migration job starting service=external
|
|
[info] Migration mode - running database migrations
|
|
[info] Running pending migrations
|
|
INFO [alembic.runtime.migration] Context impl PostgresqlImpl.
|
|
INFO [alembic.runtime.migration] Will assume transactional DDL.
|
|
[info] Migrations applied successfully
|
|
[info] Migration job completed successfully
|
|
```
|
|
|
|
### Service Startup (New Behavior):
|
|
|
|
```
|
|
[info] Starting external-service version=1.0.0
|
|
[info] Database connection established
|
|
[info] Database verification mode - checking database is ready
|
|
[info] Database state checked
|
|
[info] Database verification successful
|
|
migration_count=1 current_revision=374752db316e table_count=6
|
|
[info] Database verification completed
|
|
[info] external-service started successfully
|
|
```
|
|
|
|
---
|
|
|
|
## CI/CD Integration
|
|
|
|
### GitHub Actions Example:
|
|
|
|
```yaml
|
|
name: Deploy to Kubernetes
|
|
on:
|
|
push:
|
|
branches: [main]
|
|
|
|
jobs:
|
|
deploy:
|
|
runs-on: ubuntu-latest
|
|
steps:
|
|
- uses: actions/checkout@v2
|
|
|
|
- name: Build and push images
|
|
run: skaffold build
|
|
|
|
- name: Deploy to cluster
|
|
run: skaffold run
|
|
|
|
- name: Verify deployment
|
|
run: |
|
|
kubectl rollout status deployment --all -n bakery-ia
|
|
kubectl get pods -n bakery-ia
|
|
```
|
|
|
|
---
|
|
|
|
## Summary
|
|
|
|
**To Deploy**: Just run `skaffold dev` or `skaffold run`
|
|
|
|
**To Verify**: Check logs show "verification" not "migration"
|
|
|
|
**To Troubleshoot**: Check migration job logs first
|
|
|
|
**Expected Result**: Services start 50-80% faster, no redundant migration execution
|
|
|
|
**Status**: ✅ Ready to deploy!
|