bakery-ia/docs/K8S-MIGRATION-GUIDE.md

# Kubernetes Migration Guide: Local Dev to Production (MicroK8s)

## Overview

This guide covers migrating the Bakery IA platform from local development environment to production on a Clouding.io VPS.

**Current Setup (Local Development):**
- macOS with Colima
- Kind (Kubernetes in Docker)
- NGINX Ingress Controller
- Local storage
- Development domains (localhost, bakery-ia.local)

**Target Setup (Production):**
- Ubuntu VPS (Clouding.io)
- MicroK8s
- MicroK8s NGINX Ingress
- Persistent storage
- Production domains (your actual domain)

---

## Key Differences & Required Adaptations

### 1. **Ingress Controller**
- **Local:** Custom NGINX installed via manifest
- **Production:** MicroK8s ingress addon
- **Action Required:** Enable MicroK8s ingress addon

### 2. **Storage**
- **Local:** Kind uses `standard` storage class (hostPath)
- **Production:** MicroK8s uses `microk8s-hostpath` storage class
- **Action Required:** Update storage class in PVCs

### 3. **Image Registry**
- **Local:** Images built locally, no push required
- **Production:** Need container registry (Docker Hub, GitHub Container Registry, or private registry)
- **Action Required:** Setup image registry and push images

### 4. **Domain & SSL**
- **Local:** localhost with self-signed certs
- **Production:** Real domain with Let's Encrypt certificates
- **Action Required:** Configure DNS and update ingress

### 5. **Resource Allocation**
- **Local:** Minimal resources (development mode)
- **Production:** Production-grade resources with HPA
- **Action Required:** Already configured in prod overlay

### 6. **Build Process**
- **Local:** Skaffold with local build
- **Production:** CI/CD or manual build + push
- **Action Required:** Setup deployment pipeline

---

## Pre-Migration Checklist

### VPS Requirements
- [ ] Ubuntu 20.04 or later
- [ ] Minimum 8GB RAM (16GB+ recommended)
- [ ] Minimum 4 CPU cores (6+ recommended)
- [ ] 100GB+ disk space
- [ ] Public IP address
- [ ] Domain name configured

### Access Requirements
- [ ] SSH access to VPS
- [ ] Domain DNS access
- [ ] Container registry credentials
- [ ] SSL certificate email address

---

## Step-by-Step Migration Guide

## Phase 1: VPS Setup

### Step 1: Install MicroK8s on Ubuntu VPS

```bash
# SSH into your VPS
ssh user@your-vps-ip

# Update system
sudo apt update && sudo apt upgrade -y

# Install MicroK8s
sudo snap install microk8s --classic --channel=1.28/stable

# Add your user to microk8s group
sudo usermod -a -G microk8s $USER
sudo chown -f -R $USER ~/.kube

# Restart session
newgrp microk8s

# Verify installation
microk8s status --wait-ready

# Enable required addons
microk8s enable dns
microk8s enable hostpath-storage
microk8s enable ingress
microk8s enable cert-manager
microk8s enable metrics-server
microk8s enable rbac

# Optional but recommended
microk8s enable prometheus
microk8s enable registry  # If you want local registry

# Setup kubectl alias
echo "alias kubectl='microk8s kubectl'" >> ~/.bashrc
source ~/.bashrc

# Verify
kubectl get nodes
kubectl get pods -A
```

### Step 2: Configure Firewall

```bash
# Allow necessary ports
sudo ufw allow 22/tcp      # SSH
sudo ufw allow 80/tcp      # HTTP
sudo ufw allow 443/tcp     # HTTPS
sudo ufw allow 16443/tcp   # Kubernetes API (optional, for remote access)

# Enable firewall
sudo ufw enable

# Check status
sudo ufw status
```

---

## Phase 2: Configuration Adaptations

### Step 3: Update Storage Class

Create a production storage patch:

```bash
# On your local machine
cat > infrastructure/kubernetes/overlays/prod/storage-patch.yaml <<EOF
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: model-storage
  namespace: bakery-ia
spec:
  storageClassName: microk8s-hostpath  # Changed from 'standard'
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 50Gi  # Increased for production
EOF
```

Update `infrastructure/kubernetes/overlays/prod/kustomization.yaml`:

```yaml
# Add to patchesStrategicMerge section
patchesStrategicMerge:
  - storage-patch.yaml
```

### Step 4: Configure Domain and Ingress

Update `infrastructure/kubernetes/overlays/prod/prod-ingress.yaml`:

```yaml
# Replace these placeholder domains with your actual domains:
# - bakery.yourdomain.com  → bakery.example.com
# - api.yourdomain.com     → api.example.com
# - monitoring.yourdomain.com → monitoring.example.com

# Update CORS origins with your actual domains
```

**DNS Configuration:**
Point your domains to your VPS public IP:
```
Type    Host                Value           TTL
A       bakery              YOUR_VPS_IP     300
A       api                 YOUR_VPS_IP     300
A       monitoring          YOUR_VPS_IP     300
```

### Step 5: Setup Container Registry

#### Option A: Docker Hub (Recommended for simplicity)

```bash
# On your local machine
docker login

# Update skaffold.yaml for production
```

Create `skaffold-prod.yaml`:

```yaml
apiVersion: skaffold/v2beta28
kind: Config
metadata:
  name: bakery-ia-prod

build:
  local:
    push: true  # Push to registry
  tagPolicy:
    gitCommit:
      variant: AbbrevCommitSha
  artifacts:
    # Update all images with your Docker Hub username
    - image: YOUR_DOCKERHUB_USERNAME/bakery-gateway
      context: .
      docker:
        dockerfile: gateway/Dockerfile

    - image: YOUR_DOCKERHUB_USERNAME/bakery-dashboard
      context: ./frontend
      docker:
        dockerfile: Dockerfile.kubernetes

    # ... (repeat for all services)

deploy:
  kustomize:
    paths:
      - infrastructure/kubernetes/overlays/prod
```

Update `infrastructure/kubernetes/overlays/prod/kustomization.yaml`:

```yaml
images:
  - name: bakery/auth-service
    newName: YOUR_DOCKERHUB_USERNAME/bakery-auth-service
    newTag: latest
  - name: bakery/tenant-service
    newName: YOUR_DOCKERHUB_USERNAME/bakery-tenant-service
    newTag: latest
  # ... (repeat for all services)
```

#### Option B: MicroK8s Built-in Registry

```bash
# On VPS
microk8s enable registry

# Get registry address
kubectl get service -n container-registry

# On local machine, configure insecure registry
# Add to /etc/docker/daemon.json:
{
  "insecure-registries": ["YOUR_VPS_IP:32000"]
}

# Restart Docker
sudo systemctl restart docker

# Tag and push images
docker tag bakery/auth-service YOUR_VPS_IP:32000/bakery/auth-service
docker push YOUR_VPS_IP:32000/bakery/auth-service
```

---

## Phase 3: Secrets and Configuration

### Step 6: Update Production Secrets

```bash
# On your local machine
# Generate strong production secrets
openssl rand -base64 32  # For database passwords
openssl rand -hex 32     # For API keys

# Update infrastructure/kubernetes/base/secrets.yaml with production values
# NEVER commit real production secrets to git!
```

**Best Practice:** Use external secret management:

```bash
# On VPS - Option: Use sealed-secrets
microk8s kubectl apply -f https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.24.0/controller.yaml

# Or use HashiCorp Vault, AWS Secrets Manager, etc.
```

### Step 7: Update ConfigMap for Production

Already configured in `infrastructure/kubernetes/overlays/prod/prod-configmap.yaml`, but verify:

```yaml
data:
  ENVIRONMENT: "production"
  DEBUG: "false"
  LOG_LEVEL: "INFO"
  DOMAIN: "bakery.example.com"  # Update with your domain
  # ... other production settings
```

---

## Phase 4: Deployment

### Step 8: Build and Push Images

#### Using Skaffold (Recommended):

```bash
# On your local machine
# Build and push all images
skaffold build -f skaffold-prod.yaml

# This will:
# 1. Build all Docker images
# 2. Tag them with git commit SHA
# 3. Push to your container registry
```

#### Manual Build (Alternative):

```bash
# Build all images with production tag
docker build -t YOUR_REGISTRY/bakery-gateway:v1.0.0 -f gateway/Dockerfile .
docker build -t YOUR_REGISTRY/bakery-dashboard:v1.0.0 -f frontend/Dockerfile.kubernetes ./frontend
# ... repeat for all services

# Push to registry
docker push YOUR_REGISTRY/bakery-gateway:v1.0.0
# ... repeat for all images
```

### Step 9: Deploy to MicroK8s

#### Option A: Using kubectl

```bash
# Copy manifests to VPS
scp -r infrastructure/kubernetes user@YOUR_VPS_IP:~/

# SSH into VPS
ssh user@YOUR_VPS_IP

# Apply production configuration
kubectl apply -k ~/kubernetes/overlays/prod

# Monitor deployment
kubectl get pods -n bakery-ia -w

# Check ingress
kubectl get ingress -n bakery-ia

# Check certificates
kubectl get certificate -n bakery-ia
```

#### Option B: Using Skaffold from Local

```bash
# Get kubeconfig from VPS
scp user@YOUR_VPS_IP:/var/snap/microk8s/current/credentials/client.config ~/.kube/microk8s-config

# Merge with local kubeconfig
export KUBECONFIG=~/.kube/config:~/.kube/microk8s-config
kubectl config view --flatten > ~/.kube/config-merged
mv ~/.kube/config-merged ~/.kube/config

# Deploy using skaffold
skaffold run -f skaffold-prod.yaml --kube-context=microk8s
```

### Step 10: Verify Deployment

```bash
# Check all pods are running
kubectl get pods -n bakery-ia

# Check services
kubectl get svc -n bakery-ia

# Check ingress
kubectl get ingress -n bakery-ia

# Check persistent volumes
kubectl get pvc -n bakery-ia

# Check logs
kubectl logs -n bakery-ia deployment/gateway -f

# Test database connectivity
kubectl exec -n bakery-ia deployment/auth-db -it -- psql -U postgres -c "\l"
```

---

## Phase 5: SSL Certificate Configuration

### Step 11: Let's Encrypt SSL Certificates

The cert-manager addon is already enabled. Configure production certificates:

```bash
# Verify cert-manager is running
kubectl get pods -n cert-manager

# Check cluster issuer
kubectl get clusterissuer

# If letsencrypt-production issuer doesn't exist, create it:
cat <<EOF | kubectl apply -f -
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-production
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: your-email@example.com  # Update this
    privateKeySecretRef:
      name: letsencrypt-production
    solvers:
    - http01:
        ingress:
          class: public
EOF

# Monitor certificate issuance
kubectl describe certificate bakery-ia-prod-tls-cert -n bakery-ia

# Check certificate status
kubectl get certificate -n bakery-ia
```

**Troubleshooting certificates:**
```bash
# Check cert-manager logs
kubectl logs -n cert-manager deployment/cert-manager

# Check challenge status
kubectl get challenges -n bakery-ia

# Verify DNS resolution
nslookup bakery.example.com
```

---

## Phase 6: Monitoring and Maintenance

### Step 12: Setup Monitoring

```bash
# Prometheus is already enabled as a MicroK8s addon
kubectl get pods -n monitoring

# Access Grafana (if enabled)
kubectl port-forward -n monitoring svc/grafana 3000:3000

# Or expose via ingress (already configured in prod-ingress.yaml)
```

### Step 13: Setup Backups

Create backup script on VPS:

```bash
cat > ~/backup-databases.sh <<'EOF'
#!/bin/bash
BACKUP_DIR="/backups/$(date +%Y-%m-%d)"
mkdir -p $BACKUP_DIR

# Get all database pods
DBS=$(kubectl get pods -n bakery-ia -l app.kubernetes.io/component=database -o name)

for db in $DBS; do
  DB_NAME=$(echo $db | cut -d'/' -f2)
  echo "Backing up $DB_NAME..."

  kubectl exec -n bakery-ia $db -- pg_dump -U postgres > "$BACKUP_DIR/${DB_NAME}.sql"
done

# Compress backups
tar -czf "$BACKUP_DIR.tar.gz" "$BACKUP_DIR"
rm -rf "$BACKUP_DIR"

# Keep only last 7 days
find /backups -name "*.tar.gz" -mtime +7 -delete

echo "Backup completed: $BACKUP_DIR.tar.gz"
EOF

chmod +x ~/backup-databases.sh

# Setup daily cron job
(crontab -l 2>/dev/null; echo "0 2 * * * ~/backup-databases.sh") | crontab -
```

### Step 14: Setup Log Aggregation (Optional)

```bash
# Enable Loki for log aggregation
microk8s enable observability

# Or use external logging service like ELK, Datadog, etc.
```

---

## Phase 7: Post-Deployment Verification

### Step 15: Health Checks

```bash
# Test frontend
curl -k https://bakery.example.com

# Test API
curl -k https://api.example.com/health

# Test database connectivity
kubectl exec -n bakery-ia deployment/auth-service -- curl localhost:8000/health

# Check all services are healthy
kubectl get pods -n bakery-ia -o wide

# Check resource usage
kubectl top pods -n bakery-ia
kubectl top nodes
```

### Step 16: Performance Testing

```bash
# Install hey (HTTP load testing tool)
go install github.com/rakyll/hey@latest

# Test API endpoint
hey -n 1000 -c 10 https://api.example.com/health

# Monitor during load test
kubectl top pods -n bakery-ia
```

---

## Ongoing Operations

### Updating the Application

```bash
# On local machine
# 1. Make code changes
# 2. Build and push new images
skaffold build -f skaffold-prod.yaml

# 3. Update image tags in prod kustomization
# 4. Apply updates
kubectl apply -k infrastructure/kubernetes/overlays/prod

# 5. Rolling update status
kubectl rollout status deployment/auth-service -n bakery-ia
```

### Scaling Services

```bash
# Manual scaling
kubectl scale deployment auth-service -n bakery-ia --replicas=5

# Or update in kustomization.yaml and reapply
```

### Database Migrations

```bash
# Run migration job
kubectl apply -f infrastructure/kubernetes/base/migrations/auth-migration-job.yaml

# Check migration status
kubectl get jobs -n bakery-ia
kubectl logs -n bakery-ia job/auth-migration
```

---

## Troubleshooting Common Issues

### Issue 1: Pods Not Starting

```bash
# Check pod status
kubectl describe pod POD_NAME -n bakery-ia

# Common causes:
# - Image pull errors: Check registry credentials
# - Resource limits: Check node resources
# - Volume mount issues: Check PVC status
```

### Issue 2: Ingress Not Working

```bash
# Check ingress controller
kubectl get pods -n ingress

# Check ingress resource
kubectl describe ingress bakery-ingress-prod -n bakery-ia

# Check if port 80/443 are open
sudo netstat -tlnp | grep -E '(80|443)'

# Check NGINX logs
kubectl logs -n ingress -l app.kubernetes.io/name=ingress-nginx
```

### Issue 3: SSL Certificate Issues

```bash
# Check certificate status
kubectl describe certificate bakery-ia-prod-tls-cert -n bakery-ia

# Check cert-manager logs
kubectl logs -n cert-manager deployment/cert-manager

# Verify DNS
dig bakery.example.com

# Manual certificate request
kubectl delete certificate bakery-ia-prod-tls-cert -n bakery-ia
kubectl apply -f infrastructure/kubernetes/overlays/prod/prod-ingress.yaml
```

### Issue 4: Database Connection Errors

```bash
# Check database pod
kubectl get pods -n bakery-ia -l app.kubernetes.io/component=database

# Check database logs
kubectl logs -n bakery-ia deployment/auth-db

# Test connection from service pod
kubectl exec -n bakery-ia deployment/auth-service -- nc -zv auth-db 5432
```

### Issue 5: Out of Resources

```bash
# Check node resources
kubectl describe node

# Check resource requests/limits
kubectl describe pod POD_NAME -n bakery-ia

# Adjust resource limits in prod kustomization or scale down
```

---

## Security Hardening Checklist

- [ ] Change all default passwords
- [ ] Enable pod security policies
- [ ] Setup network policies
- [ ] Enable audit logging
- [ ] Regular security updates
- [ ] Implement secrets rotation
- [ ] Setup intrusion detection
- [ ] Enable RBAC properly
- [ ] Regular backup testing
- [ ] Implement rate limiting
- [ ] Setup DDoS protection
- [ ] Enable security scanning

---

## Performance Optimization

### For VPS with Limited Resources

If your VPS has limited resources, consider:

```yaml
# Reduce replica counts in prod kustomization.yaml
replicas:
  - name: auth-service
    count: 2  # Instead of 3
  - name: gateway
    count: 2  # Instead of 3

# Adjust resource limits
resources:
  requests:
    memory: "256Mi"  # Reduced from 512Mi
    cpu: "100m"      # Reduced from 200m
```

### Database Optimization

```bash
# Tune PostgreSQL for production
kubectl exec -n bakery-ia deployment/auth-db -it -- psql -U postgres

# Inside PostgreSQL:
ALTER SYSTEM SET shared_buffers = '256MB';
ALTER SYSTEM SET effective_cache_size = '1GB';
ALTER SYSTEM SET maintenance_work_mem = '64MB';
ALTER SYSTEM SET checkpoint_completion_target = '0.9';
ALTER SYSTEM SET wal_buffers = '16MB';
ALTER SYSTEM SET default_statistics_target = '100';

# Restart database pod
kubectl rollout restart deployment/auth-db -n bakery-ia
```

---

## Rollback Procedure

If something goes wrong:

```bash
# Rollback deployment
kubectl rollout undo deployment/DEPLOYMENT_NAME -n bakery-ia

# Rollback to specific revision
kubectl rollout history deployment/DEPLOYMENT_NAME -n bakery-ia
kubectl rollout undo deployment/DEPLOYMENT_NAME --to-revision=2 -n bakery-ia

# Restore from backup
tar -xzf /backups/2024-01-01.tar.gz
kubectl exec -n bakery-ia deployment/auth-db -- psql -U postgres < auth-db.sql
```

---

## Quick Reference

### Useful Commands

```bash
# View all resources
kubectl get all -n bakery-ia

# Get pod logs
kubectl logs -f POD_NAME -n bakery-ia

# Execute command in pod
kubectl exec -it POD_NAME -n bakery-ia -- /bin/bash

# Port forward for debugging
kubectl port-forward svc/SERVICE_NAME 8000:8000 -n bakery-ia

# Check events
kubectl get events -n bakery-ia --sort-by='.lastTimestamp'

# Resource usage
kubectl top nodes
kubectl top pods -n bakery-ia

# Restart deployment
kubectl rollout restart deployment/DEPLOYMENT_NAME -n bakery-ia

# Scale deployment
kubectl scale deployment/DEPLOYMENT_NAME --replicas=3 -n bakery-ia
```

### Important File Locations on VPS

```
/var/snap/microk8s/current/credentials/  # Kubernetes credentials
/var/snap/microk8s/common/default-storage/  # Default storage location
~/kubernetes/                            # Your manifests
/backups/                               # Database backups
```

---

## Next Steps After Migration

1. **Setup CI/CD Pipeline**
   - GitHub Actions or GitLab CI
   - Automated builds and deployments
   - Automated testing

2. **Implement Monitoring Dashboards**
   - Setup Grafana dashboards
   - Configure alerts
   - Setup uptime monitoring

3. **Disaster Recovery Plan**
   - Document recovery procedures
   - Test backup restoration
   - Setup off-site backups

4. **Cost Optimization**
   - Monitor resource usage
   - Right-size deployments
   - Implement auto-scaling

5. **Documentation**
   - Document custom configurations
   - Create runbooks for common tasks
   - Train team members

---

## Support and Resources

- **MicroK8s Documentation:** https://microk8s.io/docs
- **Kubernetes Documentation:** https://kubernetes.io/docs
- **cert-manager Documentation:** https://cert-manager.io/docs
- **NGINX Ingress:** https://kubernetes.github.io/ingress-nginx

## Conclusion

This migration moves your application from a local development environment to a production-ready deployment. Remember to:

- Test thoroughly before going live
- Have a rollback plan ready
- Monitor closely after deployment
- Keep regular backups
- Stay updated with security patches

Good luck with your deployment! 🚀