Files
bakery-ia/infrastructure/kubernetes
Claude 71759dc67a Fix Tilt hanging on frontend deployment
Fixed the issue where Tilt would get stuck after building the frontend image:

1. Changed imagePullPolicy from 'Always' to 'IfNotPresent'
   - Prevents Kubernetes from trying to pull from a remote registry
   - Uses locally built images for development

2. Removed ':latest' tag from image reference
   - Now uses 'bakery/dashboard' without explicit tag
   - Allows Skaffold/Tilt to inject the correct tag ('dev')

3. Optimized health check configuration:
   - Use dedicated /health endpoint instead of /
   - Reduced readiness probe initialDelaySeconds from 20s to 5s
   - Reduced liveness probe initialDelaySeconds from 60s to 15s
   - Faster pod startup and readiness detection

These changes ensure Kubernetes can find and use the locally built image,
and the pod becomes ready much faster, preventing Tilt from hanging during deployment.
2025-11-06 18:43:23 +00:00
..
2025-10-19 19:22:37 +02:00
2025-11-06 11:04:50 +01:00
2025-11-06 11:04:50 +01:00

Bakery IA Kubernetes Configuration

This directory contains Kubernetes manifests for deploying the Bakery IA platform in local development and production environments with HTTPS support using cert-manager and NGINX ingress.

Quick Start

Deploy the entire platform with these 5 commands:

# 1. Start Colima with adequate resources
colima start --cpu 6 --memory 12 --disk 120 --runtime docker --profile k8s-local

# 2. Create Kind cluster with permanent localhost access
kind create cluster --config kind-config.yaml

# 3. Install NGINX Ingress Controller
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/deploy/static/provider/kind/deploy.yaml
kubectl wait --namespace ingress-nginx --for=condition=ready pod --selector=app.kubernetes.io/component=controller --timeout=300s

# 4. Configure permanent localhost access
kubectl patch svc ingress-nginx-controller -n ingress-nginx -p '{"spec":{"type":"NodePort","ports":[{"name":"http","port":80,"targetPort":"http","nodePort":30080},{"name":"https","port":443,"targetPort":"https","nodePort":30443}]}}'

# 5. Deploy with Skaffold
skaffold dev --profile=dev

# 🎉 Access at: https://localhost

Prerequisites

Install the following tools on macOS:

# Install via Homebrew
brew install colima kind kubectl skaffold

# Verify installations
colima version && kind version && kubectl version --client && skaffold version

Directory Structure

infrastructure/kubernetes/
├── base/                           # Base Kubernetes resources
│   ├── namespace.yaml             # Namespace definition
│   ├── configmap.yaml             # Shared configuration
│   ├── secrets.yaml               # Base64 encoded secrets
│   ├── ingress-https.yaml         # HTTPS ingress rules
│   ├── kustomization.yaml         # Base kustomization
│   └── components/                # Individual component manifests
│       ├── cert-manager/          # Certificate management
│       ├── auth/                  # Authentication service
│       ├── tenant/                # Tenant management
│       ├── training/              # ML training service
│       ├── forecasting/           # Demand forecasting
│       ├── sales/                 # Sales management
│       ├── external/              # External API service
│       ├── notification/          # Notification service
│       ├── inventory/             # Inventory management
│       ├── recipes/               # Recipe management
│       ├── suppliers/             # Supplier management
│       ├── pos/                   # Point of sale
│       ├── orders/                # Order management
│       ├── production/            # Production planning
│       ├── alert-processor/       # Alert processing
│       ├── frontend/              # React frontend
│       ├── databases/             # Database deployments
│       └── infrastructure/        # Gateway & monitoring
└── overlays/
    └── dev/                       # Development environment
        ├── kustomization.yaml     # Dev-specific configuration
        └── dev-patches.yaml       # Development patches

Access URLs

Primary Access (Standard Web Ports)

Named Host Access (Optional)

Add to /etc/hosts for named access:

echo "127.0.0.1 bakery-ia.local" | sudo tee -a /etc/hosts
echo "127.0.0.1 api.bakery-ia.local" | sudo tee -a /etc/hosts
echo "127.0.0.1 monitoring.bakery-ia.local" | sudo tee -a /etc/hosts

Then access via:

Direct Service Access (Development)

Development Workflow

Start Development Environment

# Start development mode with hot-reload
skaffold dev --profile=dev

# Or one-time deployment
skaffold run --profile=dev

Key Features

  • Hot-reload development - Automatic rebuilds on code changes
  • Permanent localhost access - No port forwarding needed
  • HTTPS by default - Local CA certificates for secure development
  • Microservices architecture - All services deployed together
  • Database management - PostgreSQL, Redis, and RabbitMQ included

Monitor and Debug

# Check all resources
kubectl get all -n bakery-ia

# View logs
kubectl logs -n bakery-ia deployment/auth-service -f

# Check ingress status
kubectl get ingress -n bakery-ia

# Debug certificate issues
kubectl describe certificate bakery-ia-tls-cert -n bakery-ia

Certificate Management

The platform uses cert-manager for automatic HTTPS certificate generation:

  • Local CA: For development (default)
  • Let's Encrypt Staging: For testing
  • Let's Encrypt Production: For production deployments

Trust Local Certificates

# Export CA certificate
kubectl get secret local-ca-key-pair -n cert-manager -o jsonpath='{.data.tls\.crt}' | base64 -d > bakery-ia-ca.crt

# Trust in macOS
open bakery-ia-ca.crt
# In Keychain Access, set "bakery-ia-local-ca" to "Always Trust"

Configuration Management

Secrets

Base64-encoded secrets are stored in base/secrets.yaml. For production:

  • Use external secret management (HashiCorp Vault, AWS Secrets Manager)
  • Never commit real secrets to version control
# Encode secrets
echo -n "your-secret-value" | base64

# Decode secrets
echo "eW91ci1zZWNyZXQtdmFsdWU=" | base64 -d

Environment Configuration

Development-specific settings are in overlays/dev/:

  • Resource limits: Reduced for local development
  • Image pull policy: Never (for local images)
  • Debug settings: Enabled
  • CORS: Configured for localhost

Scaling and Resource Management

Scale Services

# Scale individual service
kubectl scale -n bakery-ia deployment/auth-service --replicas=3

# Or update kustomization.yaml replicas section

Resource Configuration

Development environment uses minimal resources:

  • Databases: 64Mi-256Mi memory, 25m-200m CPU
  • Services: 64Mi-256Mi memory, 25m-200m CPU
  • Training Service: 256Mi-1Gi memory (ML workloads)

Troubleshooting

Common Issues

  1. Images not found

    # Build images with Skaffold
    skaffold build --profile=dev
    
  2. Database corruption after restart

    # Delete corrupted PVC and restart
    kubectl delete pod -n bakery-ia -l app.kubernetes.io/name=inventory-db
    kubectl delete pvc -n bakery-ia inventory-db-pvc
    
  3. HTTPS certificate not issued

    # Check cert-manager logs
    kubectl logs -n cert-manager deployment/cert-manager
    kubectl describe certificate bakery-ia-tls-cert -n bakery-ia
    
  4. Port conflicts

    # Check what's using ports 80/443
    sudo lsof -i :80 -i :443
    

Debug Commands

# Get cluster events
kubectl get events -n bakery-ia --sort-by='.firstTimestamp'

# Resource usage
kubectl top pods -n bakery-ia
kubectl top nodes

# Execute in pod
kubectl exec -n bakery-ia -it <pod-name> -- bash

Cleanup

Quick Cleanup

# Stop Skaffold (Ctrl+C or)
skaffold delete --profile=dev

Complete Cleanup

# Delete everything
kubectl delete namespace bakery-ia
kind delete cluster --name bakery-ia-local
colima stop --profile k8s-local

Restart Sequence

# Post-restart startup
colima start --cpu 6 --memory 12 --disk 120 --runtime docker --profile k8s-local
kind create cluster --config kind-config.yaml
skaffold dev --profile=dev

Production Considerations

For production deployment:

  • Security: Implement RBAC, network policies, pod security standards
  • Monitoring: Deploy Prometheus, Grafana, and alerting
  • Backup: Database backup strategies
  • High Availability: Multi-replica deployments with anti-affinity
  • External Secrets: Use managed secret services
  • TLS: Production Let's Encrypt certificates
  • CI/CD: Automated deployment pipelines

Next Steps

  1. Add comprehensive monitoring and logging
  2. Implement automated testing
  3. Set up CI/CD pipelines
  4. Add health checks and metrics endpoints
  5. Implement proper backup strategies