bakery-ia/infrastructure/kubernetes/README.md

# Bakery IA Kubernetes Configuration

This directory contains Kubernetes manifests for deploying the Bakery IA forecasting platform in a local development environment with **permanent localhost access** and **FREE HTTPS support** using cert-manager and NGINX ingress.

## ⚡ Quick Start (5 Commands)

```bash
# 1. Start Colima
colima start --cpu 4 --memory 8 --disk 50 --runtime docker --profile k8s-local

# 2. Create Kind cluster with permanent localhost access
kind create cluster --config kind-config.yaml

# 3. Install NGINX Ingress Controller
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/deploy/static/provider/kind/deploy.yaml && kubectl wait --namespace ingress-nginx --for=condition=ready pod --selector=app.kubernetes.io/component=controller --timeout=300s

# 4. Configure permanent localhost access
kubectl patch svc ingress-nginx-controller -n ingress-nginx -p '{"spec":{"type":"NodePort","ports":[{"name":"http","port":80,"targetPort":"http","nodePort":30080},{"name":"https","port":443,"targetPort":"https","nodePort":30443}]}}'

# 5. Deploy your application
skaffold dev --profile=dev

# 🎉 Done! Access at: http://localhost
```

## Prerequisites (macOS Local Development)

1. **Colima**: Docker runtime for macOS
2. **Kind**: Kubernetes in Docker for local clusters
3. **kubectl**: Kubernetes command-line tool
4. **Skaffold**: For building and deploying applications
5. **NGINX Ingress Controller**: For routing traffic (installed automatically)
6. **cert-manager**: For automatic TLS certificate management (installed automatically)

### Install Prerequisites (macOS)

```bash
# Install Homebrew (if not already installed)
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

# Install required tools
brew install colima kind kubectl skaffold

# Verify installations
colima version
kind version
kubectl version --client
skaffold version
```

## 🔒 HTTPS Setup Options

### Option 1: Automated HTTPS Setup (Recommended)
```bash
# Run the automated HTTPS setup script
./setup-https.sh
```

### Option 2: HTTP Only (Basic Setup)
```bash
# Deploy without HTTPS
kubectl apply -k infrastructure/kubernetes/overlays/dev/
```

## Kind Configuration for Permanent Localhost Access

The `kind-config.yaml` file in the root directory provides permanent localhost access without port forwarding:

```yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
name: bakery-ia-local
nodes:
- role: control-plane
  kubeadmConfigPatches:
  - |
    kind: InitConfiguration
    nodeRegistration:
      kubeletExtraArgs:
        node-labels: "ingress-ready=true"
  extraPortMappings:
  # HTTP ingress
  - containerPort: 30080
    hostPort: 80
    protocol: TCP
  # HTTPS ingress
  - containerPort: 30443
    hostPort: 443
    protocol: TCP
  # Direct frontend access (backup)
  - containerPort: 30300
    hostPort: 3000
    protocol: TCP
  # Direct gateway access (backup)
  - containerPort: 30800
    hostPort: 8000
    protocol: TCP
```

This configuration maps:
- Port 80 → localhost:80 (HTTP)
- Port 443 → localhost:443 (HTTPS)
- Port 3000 → localhost:3000 (Direct frontend)
- Port 8000 → localhost:8000 (Direct gateway)

## Directory Structure

```
infrastructure/kubernetes/
├── kind-config.yaml                # Kind cluster configuration with port mapping
├── base/                           # Base Kubernetes resources
│   ├── namespace.yaml             # Namespace definition
│   ├── configmap.yaml             # Shared configuration
│   ├── secrets.yaml               # Secrets (base64 encoded)
│   ├── ingress.yaml               # HTTP ingress rules
│   ├── ingress-https.yaml         # HTTPS ingress rules
│   └── kustomization.yaml         # Base kustomization
├── components/                     # Individual component manifests
│   ├── cert-manager/              # Certificate management
│   │   ├── cluster-issuer-staging.yaml    # Let's Encrypt staging
│   │   ├── cluster-issuer-production.yaml # Let's Encrypt production
│   │   └── local-ca-issuer.yaml           # Local CA for development
│   ├── auth/                      # Auth service
│   ├── tenant/                    # Tenant service
│   ├── training/                  # Training service
│   ├── forecasting/               # Forecasting service
│   ├── sales/                     # Sales service
│   ├── external/                  # External service
│   ├── notification/              # Notification service
│   ├── inventory/                 # Inventory service
│   ├── recipes/                   # Recipes service
│   ├── suppliers/                 # Suppliers service
│   ├── pos/                       # POS service
│   ├── orders/                    # Orders service
│   ├── production/                # Production service
│   ├── alert-processor/           # Alert processor
│   ├── frontend/                  # Frontend application
│   ├── databases/                 # Database deployments
│   └── infrastructure/            # Infrastructure components (gateway, etc.)
└── overlays/
    └── dev/                       # Development environment overlay
        ├── kustomization.yaml     # Dev-specific kustomization
        ├── https-kustomization.yaml # HTTPS-specific kustomization
        ├── dev-patches.yaml       # Development patches
        └── ingress-https-patch.yaml # HTTPS ingress patch
```

## 🚀 Quick Start (macOS with Kind + Colima)

### 1. Start Colima and Create Kind Cluster with Permanent Localhost Access

```bash
# Start Colima with proper resources for development
colima start --cpu 4 --memory 8 --disk 100 --runtime docker --profile k8s-local

# Create Kind cluster with permanent port mapping for localhost access
kind create cluster --config kind-config.yaml

# Verify cluster is running and port mappings
kubectl cluster-info
docker port bakery-ia-local-control-plane
```

The `kind-config.yaml` configuration provides permanent localhost access on ports 80 and 443 without requiring port forwarding!

### 2. Install NGINX Ingress Controller for Kind

```bash
# Install NGINX Ingress Controller (Kind-specific with permanent localhost access)
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/deploy/static/provider/kind/deploy.yaml

# Wait for ingress controller to be ready
kubectl wait --namespace ingress-nginx \
  --for=condition=ready pod \
  --selector=app.kubernetes.io/component=controller \
  --timeout=300s

# Configure ingress controller for permanent localhost access
kubectl patch svc ingress-nginx-controller -n ingress-nginx -p '{"spec":{"type":"NodePort","ports":[{"name":"http","port":80,"targetPort":"http","nodePort":30080},{"name":"https","port":443,"targetPort":"https","nodePort":30443}]}}'
```

### 3. Deploy with Skaffold (No Port Forwarding Required!)

```bash
# Option A: Development mode with auto-rebuild (Recommended)
skaffold dev --profile=dev

# Option B: One-time deployment
skaffold run --profile=dev

# Option C: Debug mode (still includes port forwarding for individual services)
skaffold debug --profile=debug

# Check deployment status
kubectl get pods -n bakery-ia
kubectl get services -n bakery-ia
kubectl get ingress -n bakery-ia
```

**Note**: With the new configuration, skaffold no longer needs port forwarding for frontend access since localhost:80 and localhost:443 are permanently mapped!

### 4. Access the Application - Permanent Localhost Access! 🎉

**No /etc/hosts modification needed!** The application is now accessible directly via standard localhost URLs:

**Primary Access (Recommended):**
- **Frontend**: http://localhost or https://localhost
- **API Gateway**: http://localhost/api or https://localhost/api

**Named Host Access (Optional):**
If you prefer named hosts, add to your `/etc/hosts` file:
```bash
echo "127.0.0.1 bakery-ia.local" | sudo tee -a /etc/hosts
echo "127.0.0.1 api.bakery-ia.local" | sudo tee -a /etc/hosts
echo "127.0.0.1 monitoring.bakery-ia.local" | sudo tee -a /etc/hosts
```

Then access via:
- Frontend: http://bakery-ia.local or https://bakery-ia.local
- API Gateway: http://api.bakery-ia.local or https://api.bakery-ia.local
- Monitoring: http://monitoring.bakery-ia.local or https://monitoring.bakery-ia.local

## 🔒 HTTPS Configuration (FREE with Let's Encrypt)

### Automated HTTPS Setup

The quickest way to enable HTTPS is using the automated setup script:

```bash
# Run the automated HTTPS setup script
./setup-https.sh
```

This script will:
- ✅ Install cert-manager (FREE Let's Encrypt client)
- ✅ Install NGINX Ingress Controller
- ✅ Set up cluster issuers (staging, production, and local CA)
- ✅ Deploy your application with HTTPS support
- ✅ Generate and configure TLS certificates
- ✅ Export CA certificate for browser trust

### Manual HTTPS Setup

If you prefer manual setup:

#### 1. Install cert-manager
```bash
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.13.2/cert-manager.yaml
kubectl wait --for=condition=ready pod -l app.kubernetes.io/instance=cert-manager -n cert-manager --timeout=300s
```

#### 2. Install NGINX Ingress Controller for Kind
```bash
kubectl apply -f https://kind.sigs.k8s.io/examples/ingress/deploy-ingress-nginx.yaml
kubectl wait --namespace ingress-nginx --for=condition=ready pod --selector=app.kubernetes.io/component=controller --timeout=300s
```

#### 3. Apply Certificate Issuers
```bash
kubectl apply -f infrastructure/kubernetes/base/components/cert-manager/cluster-issuer-staging.yaml
kubectl apply -f infrastructure/kubernetes/base/components/cert-manager/local-ca-issuer.yaml
kubectl apply -f infrastructure/kubernetes/base/components/cert-manager/cluster-issuer-production.yaml
```

#### 4. Deploy with HTTPS
```bash
kubectl apply -k infrastructure/kubernetes/overlays/dev/
kubectl patch ingress bakery-ingress -n bakery-ia --patch-file infrastructure/kubernetes/overlays/dev/ingress-https-patch.yaml
```

#### 5. Export CA Certificate for Browser Trust
```bash
kubectl get secret local-ca-key-pair -n cert-manager -o jsonpath='{.data.tls\.crt}' | base64 -d > bakery-ia-ca.crt
```

### Access HTTPS Application

After HTTPS setup:
- **🔐 Frontend:** https://bakery-ia.local
- **🔐 API Gateway:** https://api.bakery-ia.local
- **🔐 Monitoring:** https://monitoring.bakery-ia.local

### Trust the CA Certificate

**For macOS:**
```bash
open bakery-ia-ca.crt
# In Keychain Access, find "bakery-ia-local-ca" and set to "Always Trust"
```

**For Linux:**
```bash
sudo cp bakery-ia-ca.crt /usr/local/share/ca-certificates/
sudo update-ca-certificates
```

### Certificate Management Commands

```bash
# Check certificate status
kubectl get certificates -n bakery-ia

# Check certificate details
kubectl describe certificate bakery-ia-tls-cert -n bakery-ia

# Check cluster issuers
kubectl get clusterissuers

# Check TLS secret
kubectl get secret bakery-ia-tls-cert -n bakery-ia
```

### Switching to Production Let's Encrypt

To use real Let's Encrypt certificates (requires public domain):

1. Update the cluster issuer in `ingress-https-patch.yaml`:
```yaml
cert-manager.io/cluster-issuer: "letsencrypt-production"  # Change from local-ca-issuer
```

2. Update email in cluster issuers to your real email
3. Ensure your domain points to your cluster's external IP

### Cleanup HTTPS Setup

```bash
# Run cleanup script
./cleanup-https.sh

# Or manually clean up
kubectl delete -k infrastructure/kubernetes/overlays/dev/
kubectl delete -f https://github.com/cert-manager/cert-manager/releases/download/v1.13.2/cert-manager.yaml
rm -f bakery-ia-ca.crt
```

## Port Forwarding for Direct Access

If you prefer to access services directly without ingress:

```bash
# Frontend
kubectl port-forward -n bakery-ia svc/frontend-service 3000:3000

# Gateway
kubectl port-forward -n bakery-ia svc/gateway-service 8000:8000

# Auth Service
kubectl port-forward -n bakery-ia svc/auth-service 8001:8000

# Redis
kubectl port-forward -n bakery-ia svc/redis-service 6379:6379

# Database example (auth-db)
kubectl port-forward -n bakery-ia svc/auth-db-service 5432:5432
```

## Managing the Deployment

### Check Status

```bash
# Check all resources
kubectl get all -n bakery-ia

# Check specific resource types
kubectl get pods -n bakery-ia
kubectl get services -n bakery-ia
kubectl get deployments -n bakery-ia
kubectl get pvc -n bakery-ia

# Check logs
kubectl logs -n bakery-ia deployment/auth-service
kubectl logs -n bakery-ia deployment/frontend -f  # Follow logs
```

### Update Deployments

```bash
# After making changes to manifests
kubectl apply -k infrastructure/kubernetes/overlays/dev/

# Force restart a deployment
kubectl rollout restart -n bakery-ia deployment/auth-service

# Check rollout status
kubectl rollout status -n bakery-ia deployment/auth-service
```

### Scaling Services

```bash
# Scale a service
kubectl scale -n bakery-ia deployment/auth-service --replicas=3

# Or edit the kustomization.yaml replicas section and reapply
```

### Clean Up (macOS + Kind + Colima + Skaffold)

```bash
# Option 1: Quick cleanup (development session)
skaffold delete --profile=dev

# Option 2: Clean up HTTPS setup
./cleanup-https.sh

# Option 3: Complete cleanup (everything)
./complete-cleanup.sh

# Option 4: Manual cleanup steps
kubectl delete namespace bakery-ia
kind delete cluster --name bakery-ia-local
colima stop --profile k8s-local
```

**📖 For detailed cleanup options, see [CLEANUP-GUIDE.md](../../CLEANUP-GUIDE.md)**

## Configuration

### Secrets

The `secrets.yaml` file contains base64-encoded secrets. For production, these should be:
1. Generated securely
2. Managed through external secret management systems
3. Not committed to version control

To encode/decode secrets:
```bash
# Encode
echo -n "your-secret-value" | base64

# Decode
echo "eW91ci1zZWNyZXQtdmFsdWU=" | base64 -d
```

### Environment-Specific Configuration

Modify the `overlays/dev/` files to customize the development environment:
- `kustomization.yaml`: Image tags, replicas, resource references
- `dev-patches.yaml`: Environment-specific configuration overrides

### Adding New Services

1. Create a new directory under `components/`
2. Add the service YAML manifest
3. Update `base/kustomization.yaml` to include the new resource
4. Update configuration maps and secrets as needed

## Troubleshooting

### Common Issues

1. **Images not found**: Ensure images are built and available to the cluster
2. **Pending pods**: Check resource requests and cluster capacity
3. **CrashLoopBackOff**: Check logs and environment variables
4. **Service not accessible**: Verify ingress controller is running and localhost ports are mapped
5. **Database corruption**: If PostgreSQL databases show "could not locate a valid checkpoint record", delete the PVC and restart the pod to get fresh storage
6. **Port conflicts**: If localhost:80 or localhost:443 are already in use, stop other services or change the Kind configuration
7. **HTTPS certificate not issued**: Check cert-manager logs and cluster issuer status
8. **Browser security warnings**: Import and trust the CA certificate (`bakery-ia-ca.crt`)
9. **Certificate pending**: Wait for cert-manager to issue the certificate (usually takes 30-60 seconds)
10. **Kustomize deprecation warnings**: Fixed - using modern `patches` syntax instead of deprecated `patchesStrategicMerge` and `patchesJson6902`

### Database Recovery Commands

If you encounter database corruption (common after improper cluster shutdown):

```bash
# Check which databases are failing
kubectl get pods -n bakery-ia | grep -E "(db|CrashLoopBackOff)"

# For each corrupted database (example with inventory-db):
kubectl delete pod -n bakery-ia -l app.kubernetes.io/name=inventory-db
kubectl delete pvc -n bakery-ia inventory-db-pvc

# The deployment will automatically recreate with fresh storage
# Repeat for pos-db-pvc and training-db-pvc if needed
```

### Debugging Commands

```bash
# Describe resources for detailed information
kubectl describe pod -n bakery-ia <pod-name>
kubectl describe deployment -n bakery-ia <deployment-name>

# Get events
kubectl get events -n bakery-ia --sort-by='.firstTimestamp'

# Execute commands in pods
kubectl exec -n bakery-ia -it <pod-name> -- bash
kubectl exec -n bakery-ia -it <pod-name> -- env

# Check resource usage
kubectl top pods -n bakery-ia
kubectl top nodes

# HTTPS/Certificate debugging
kubectl logs -n cert-manager deployment/cert-manager
kubectl describe clusterissuer letsencrypt-staging
kubectl describe certificate bakery-ia-tls-cert -n bakery-ia
kubectl get challenges -n bakery-ia
kubectl get certificaterequests -n bakery-ia
```

## Production Considerations

For production deployment, consider:

1. **Resource Limits**: Set appropriate CPU and memory limits
2. **Persistent Volumes**: Use proper storage classes for databases
3. **Secrets Management**: Use external secret management (HashiCorp Vault, AWS Secrets Manager, etc.)
4. **Monitoring**: Deploy Prometheus and Grafana
5. **Backup**: Implement database backup strategies
6. **High Availability**: Use multiple replicas and anti-affinity rules
7. **Security**: Network policies, RBAC, pod security policies
8. **TLS/HTTPS**: Use production Let's Encrypt certificates for public domains
9. **CI/CD**: Integrate with your deployment pipeline

## Next Steps

1. Add monitoring with Prometheus and Grafana
2. Implement proper logging with ELK stack or similar
3. Add health checks and metrics endpoints
4. Implement automated testing
5. Set up CI/CD pipelines for automated deployments

## 🚀 Complete Setup Guide (macOS + Kind + Colima) - New Permanent Solution!

### Method 1: Permanent Localhost Access (Recommended - No Port Forwarding!)

```bash
# 1. Start Colima
colima start --cpu 4 --memory 8 --disk 50 --runtime docker --profile k8s-local

# 2. Create Kind cluster with permanent port mapping
kind create cluster --config kind-config.yaml

# 3. Install NGINX Ingress Controller with NodePort configuration
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/deploy/static/provider/kind/deploy.yaml
kubectl wait --namespace ingress-nginx --for=condition=ready pod --selector=app.kubernetes.io/component=controller --timeout=300s

# 4. Configure ingress for permanent localhost access
kubectl patch svc ingress-nginx-controller -n ingress-nginx -p '{"spec":{"type":"NodePort","ports":[{"name":"http","port":80,"targetPort":"http","nodePort":30080},{"name":"https","port":443,"targetPort":"https","nodePort":30443}]}}'

# 5. Deploy with Skaffold
skaffold dev --profile=dev

# 6. Access your application - NO /etc/hosts needed!
# Frontend: http://localhost
# API: http://localhost/api
# HTTPS: https://localhost (with browser security warnings)
```

### Method 2: Legacy Setup with HTTPS and Named Hosts

```bash
# 1. Start Colima
colima start --cpu 4 --memory 8 --disk 50 --runtime docker --profile k8s-local

# 2. Create standard Kind cluster
kind create cluster --name bakery-ia-local

# 3. Run automated HTTPS setup (includes cert-manager and ingress)
./setup-https.sh

# 4. Deploy with Skaffold
skaffold dev --profile=dev

# 5. Add hosts entries for named hosts
sudo tee -a /etc/hosts << EOF
127.0.0.1 bakery-ia.local
127.0.0.1 api.bakery-ia.local
127.0.0.1 monitoring.bakery-ia.local
EOF

# 6. Trust CA certificate (for HTTPS)
open bakery-ia-ca.crt
# In Keychain Access, set "bakery-ia-local-ca" to "Always Trust"
```

## 🚀 Skaffold Development Workflow

### Development Mode (Recommended)
```bash
# Start continuous development mode
skaffold dev --profile=dev
```

This will:
- ✅ **Build all Docker images** automatically
- ✅ **Deploy to your Kind cluster**
- ✅ **Watch for file changes** in real-time
- ✅ **Automatically rebuild and redeploy** when you save files
- ✅ **Stream logs** from all services in one terminal

### Other Skaffold Commands

```bash
# One-time deployment (no file watching)
skaffold run --profile=dev

# Debug mode with port forwarding
skaffold debug --profile=debug

# Force rebuild and deploy
skaffold build --file-output=build.json
skaffold deploy --build-artifacts=build.json

# Clean up deployed resources
skaffold delete
```

### Stopping Skaffold

```bash
# Stop Skaffold (press Ctrl+C in the terminal running skaffold dev)
# Or run:
skaffold delete

# Complete cleanup
kind delete cluster --name bakery-ia-local
colima stop --profile k8s-local
```

### 🎯 Key Skaffold Benefits

1. **🔄 Automated builds**: No manual Docker image building
2. **👀 File watching**: Instant rebuilds on code changes
3. **📊 Log streaming**: All service logs in one place
4. **🔗 Port forwarding**: Easy access to services during development
5. **⚡ One command deployment**: `skaffold dev` does everything

### 💡 Pro Tips

- Use `skaffold dev --profile=dev` for daily development
- Code changes trigger automatic rebuilds and deployments
- Logs are automatically streamed to your terminal
- Press `Ctrl+C` to stop and clean up everything

## 🎉 Summary: What You Get

### 🚀 NEW: Permanent Localhost Access (No Port Forwarding!)
- ✅ **Direct localhost access** at http://localhost and https://localhost
- ✅ **Standard web ports** 80 and 443 work directly
- ✅ **No /etc/hosts modifications** required for basic access
- ✅ **No port forwarding commands** needed during development
- ✅ **Bookmark-friendly URLs** like any standard web application
- ✅ **Kind cluster configuration** with permanent port mapping

### Development Environment
- ✅ **One-command deployment** with `skaffold dev --profile=dev`
- ✅ **Hot-reload development** with automatic rebuilds
- ✅ **Complete observability** with streaming logs and metrics
- ✅ **Easy cleanup** with `skaffold delete` or cleanup scripts
- ✅ **Database corruption protection** with proper PVC management

### FREE HTTPS with Let's Encrypt (Optional)
- ✅ **Automated certificate management** with cert-manager
- ✅ **Local development certificates** for offline work
- ✅ **Production-ready** Let's Encrypt integration
- ✅ **Auto-renewal** of certificates before expiration
- ✅ **Browser-trusted certificates** with CA import

### Security Features
- ✅ **TLS 1.3 encryption** for all traffic (when HTTPS is configured)
- ✅ **HTTPS redirects** from HTTP (configurable)
- ✅ **Secure headers** via NGINX Ingress
- ✅ **Certificate transparency** compliance

### Access URLs - Choose Your Style!

**🌟 Primary Access (New Permanent Solution):**
- **Frontend:** http://localhost or https://localhost
- **API Gateway:** http://localhost/api or https://localhost/api

**🏷️ Named Host Access (Optional with /etc/hosts):**
- **Frontend:** http://bakery-ia.local or https://bakery-ia.local
- **API:** http://api.bakery-ia.local or https://api.bakery-ia.local
- **Monitoring:** http://monitoring.bakery-ia.local or https://monitoring.bakery-ia.local

**🔧 Direct Service Access (Backup):**
- **Frontend Direct:** http://localhost:3000
- **Gateway Direct:** http://localhost:8000

This setup provides production-like development experience with the convenience of standard localhost URLs! 🚀


  Pre-Restart Shutdown Sequence:

  1. Stop Skaffold:
  # If running interactively: Ctrl+C
  # If running in background:
  pkill -f skaffold

  2. Delete Kind cluster:
  kind delete cluster --name bakery-ia-local

  3. Stop Colima:
  colima stop

  Post-Restart Startup Sequence:

  1. Start Colima:
  colima start

  2. Create Kind cluster:
  kind create cluster --config kind-config.yaml --name bakery-ia-local

  3. Start Skaffold with dev profile:
  skaffold dev -p dev

  What Skaffold Will Do:

  - Check existing Docker images (tagged as :dev)
  - Skip rebuilds if source code unchanged
  - Load images to new Kind cluster
  - Deploy using infrastructure/kubernetes/overlays/dev
  - Watch for changes and hot-reload

  The -p dev profile ensures consistent tagging and deployment configuration
   as defined in your skaffold.yaml profiles section.