bakery-ia/docs/K8S-MIGRATION-GUIDE.md

Kubernetes Migration Guide: Local Dev to Production (MicroK8s)

Overview

This guide covers migrating the Bakery IA platform from local development environment to production on a Clouding.io VPS.

Current Setup (Local Development):

• macOS with Colima
• Kind (Kubernetes in Docker)
• NGINX Ingress Controller
• Local storage
• Development domains (localhost, bakery-ia.local)

Target Setup (Production):

• Ubuntu VPS (Clouding.io)
• MicroK8s
• MicroK8s NGINX Ingress
• Persistent storage
• Production domains (your actual domain)

---

Key Differences & Required Adaptations

1. Ingress Controller

• Local: Custom NGINX installed via manifest
• Production: MicroK8s ingress addon
• Action Required: Enable MicroK8s ingress addon

2. Storage

• Local: Kind uses standard storage class (hostPath)
• Production: MicroK8s uses microk8s-hostpath storage class
• Action Required: Update storage class in PVCs

3. Image Registry

• Local: Images built locally, no push required
• Production: Need container registry (Docker Hub, GitHub Container Registry, or private registry)
• Action Required: Setup image registry and push images

4. Domain & SSL

• Local: localhost with self-signed certs
• Production: Real domain with Let's Encrypt certificates
• Action Required: Configure DNS and update ingress

5. Resource Allocation

• Local: Minimal resources (development mode)
• Production: Production-grade resources with HPA
• Action Required: Already configured in prod overlay

6. Build Process

• Local: Skaffold with local build
• Production: CI/CD or manual build + push
• Action Required: Setup deployment pipeline

---

Pre-Migration Checklist

VPS Requirements

• Ubuntu 20.04 or later
• Minimum 8GB RAM (16GB+ recommended)
• Minimum 4 CPU cores (6+ recommended)
• 100GB+ disk space
• Public IP address
• Domain name configured

Access Requirements

• SSH access to VPS
• Domain DNS access
• Container registry credentials
• SSL certificate email address

---

Step-by-Step Migration Guide

Phase 1: VPS Setup

Step 1: Install MicroK8s on Ubuntu VPS

# SSH into your VPS
ssh user@your-vps-ip

# Update system
sudo apt update && sudo apt upgrade -y

# Install MicroK8s
sudo snap install microk8s --classic --channel=1.28/stable

# Add your user to microk8s group
sudo usermod -a -G microk8s $USER
sudo chown -f -R $USER ~/.kube

# Restart session
newgrp microk8s

# Verify installation
microk8s status --wait-ready

# Enable required addons
microk8s enable dns
microk8s enable hostpath-storage
microk8s enable ingress
microk8s enable cert-manager
microk8s enable metrics-server
microk8s enable rbac

# Optional but recommended
microk8s enable prometheus
microk8s enable registry  # If you want local registry

# Setup kubectl alias
echo "alias kubectl='microk8s kubectl'" >> ~/.bashrc
source ~/.bashrc

# Verify
kubectl get nodes
kubectl get pods -A

Step 2: Configure Firewall

# Allow necessary ports
sudo ufw allow 22/tcp      # SSH
sudo ufw allow 80/tcp      # HTTP
sudo ufw allow 443/tcp     # HTTPS
sudo ufw allow 16443/tcp   # Kubernetes API (optional, for remote access)

# Enable firewall
sudo ufw enable

# Check status
sudo ufw status

---

Phase 2: Configuration Adaptations

Step 3: Update Storage Class

Create a production storage patch:

# On your local machine
cat > infrastructure/kubernetes/overlays/prod/storage-patch.yaml <<EOF
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: model-storage
  namespace: bakery-ia
spec:
  storageClassName: microk8s-hostpath  # Changed from 'standard'
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 50Gi  # Increased for production
EOF

Update infrastructure/kubernetes/overlays/prod/kustomization.yaml:

# Add to patchesStrategicMerge section
patchesStrategicMerge:
  - storage-patch.yaml

Step 4: Configure Domain and Ingress

Update infrastructure/kubernetes/overlays/prod/prod-ingress.yaml:

# Replace these placeholder domains with your actual domains:
# - bakery.yourdomain.com  → bakery.example.com
# - api.yourdomain.com     → api.example.com
# - monitoring.yourdomain.com → monitoring.example.com

# Update CORS origins with your actual domains

DNS Configuration: Point your domains to your VPS public IP:

Type    Host                Value           TTL
A       bakery              YOUR_VPS_IP     300
A       api                 YOUR_VPS_IP     300
A       monitoring          YOUR_VPS_IP     300

Step 5: Setup Container Registry

Option A: Docker Hub (Recommended for simplicity)

# On your local machine
docker login

# Update skaffold.yaml for production

Create skaffold-prod.yaml:

apiVersion: skaffold/v2beta28
kind: Config
metadata:
  name: bakery-ia-prod

build:
  local:
    push: true  # Push to registry
  tagPolicy:
    gitCommit:
      variant: AbbrevCommitSha
  artifacts:
    # Update all images with your Docker Hub username
    - image: YOUR_DOCKERHUB_USERNAME/bakery-gateway
      context: .
      docker:
        dockerfile: gateway/Dockerfile

    - image: YOUR_DOCKERHUB_USERNAME/bakery-dashboard
      context: ./frontend
      docker:
        dockerfile: Dockerfile.kubernetes

    # ... (repeat for all services)

deploy:
  kustomize:
    paths:
      - infrastructure/kubernetes/overlays/prod

Update infrastructure/kubernetes/overlays/prod/kustomization.yaml:

images:
  - name: bakery/auth-service
    newName: YOUR_DOCKERHUB_USERNAME/bakery-auth-service
    newTag: latest
  - name: bakery/tenant-service
    newName: YOUR_DOCKERHUB_USERNAME/bakery-tenant-service
    newTag: latest
  # ... (repeat for all services)

Option B: MicroK8s Built-in Registry

# On VPS
microk8s enable registry

# Get registry address
kubectl get service -n container-registry

# On local machine, configure insecure registry
# Add to /etc/docker/daemon.json:
{
  "insecure-registries": ["YOUR_VPS_IP:32000"]
}

# Restart Docker
sudo systemctl restart docker

# Tag and push images
docker tag bakery/auth-service YOUR_VPS_IP:32000/bakery/auth-service
docker push YOUR_VPS_IP:32000/bakery/auth-service

---

Phase 3: Secrets and Configuration

Step 6: Update Production Secrets

# On your local machine
# Generate strong production secrets
openssl rand -base64 32  # For database passwords
openssl rand -hex 32     # For API keys

# Update infrastructure/kubernetes/base/secrets.yaml with production values
# NEVER commit real production secrets to git!

Best Practice: Use external secret management:

# On VPS - Option: Use sealed-secrets
microk8s kubectl apply -f https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.24.0/controller.yaml

# Or use HashiCorp Vault, AWS Secrets Manager, etc.

Step 7: Update ConfigMap for Production

Already configured in infrastructure/kubernetes/overlays/prod/prod-configmap.yaml, but verify:

data:
  ENVIRONMENT: "production"
  DEBUG: "false"
  LOG_LEVEL: "INFO"
  DOMAIN: "bakery.example.com"  # Update with your domain
  # ... other production settings

---

Phase 4: Deployment

Step 8: Build and Push Images

Using Skaffold (Recommended):

# On your local machine
# Build and push all images
skaffold build -f skaffold-prod.yaml

# This will:
# 1. Build all Docker images
# 2. Tag them with git commit SHA
# 3. Push to your container registry

Manual Build (Alternative):

# Build all images with production tag
docker build -t YOUR_REGISTRY/bakery-gateway:v1.0.0 -f gateway/Dockerfile .
docker build -t YOUR_REGISTRY/bakery-dashboard:v1.0.0 -f frontend/Dockerfile.kubernetes ./frontend
# ... repeat for all services

# Push to registry
docker push YOUR_REGISTRY/bakery-gateway:v1.0.0
# ... repeat for all images

Step 9: Deploy to MicroK8s

Option A: Using kubectl

# Copy manifests to VPS
scp -r infrastructure/kubernetes user@YOUR_VPS_IP:~/

# SSH into VPS
ssh user@YOUR_VPS_IP

# Apply production configuration
kubectl apply -k ~/kubernetes/overlays/prod

# Monitor deployment
kubectl get pods -n bakery-ia -w

# Check ingress
kubectl get ingress -n bakery-ia

# Check certificates
kubectl get certificate -n bakery-ia

Option B: Using Skaffold from Local

# Get kubeconfig from VPS
scp user@YOUR_VPS_IP:/var/snap/microk8s/current/credentials/client.config ~/.kube/microk8s-config

# Merge with local kubeconfig
export KUBECONFIG=~/.kube/config:~/.kube/microk8s-config
kubectl config view --flatten > ~/.kube/config-merged
mv ~/.kube/config-merged ~/.kube/config

# Deploy using skaffold
skaffold run -f skaffold-prod.yaml --kube-context=microk8s

Step 10: Verify Deployment

# Check all pods are running
kubectl get pods -n bakery-ia

# Check services
kubectl get svc -n bakery-ia

# Check ingress
kubectl get ingress -n bakery-ia

# Check persistent volumes
kubectl get pvc -n bakery-ia

# Check logs
kubectl logs -n bakery-ia deployment/gateway -f

# Test database connectivity
kubectl exec -n bakery-ia deployment/auth-db -it -- psql -U postgres -c "\l"

---

Phase 5: SSL Certificate Configuration

Step 11: Let's Encrypt SSL Certificates

The cert-manager addon is already enabled. Configure production certificates:

# Verify cert-manager is running
kubectl get pods -n cert-manager

# Check cluster issuer
kubectl get clusterissuer

# If letsencrypt-production issuer doesn't exist, create it:
cat <<EOF | kubectl apply -f -
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-production
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: your-email@example.com  # Update this
    privateKeySecretRef:
      name: letsencrypt-production
    solvers:
    - http01:
        ingress:
          class: public
EOF

# Monitor certificate issuance
kubectl describe certificate bakery-ia-prod-tls-cert -n bakery-ia

# Check certificate status
kubectl get certificate -n bakery-ia

Troubleshooting certificates:

# Check cert-manager logs
kubectl logs -n cert-manager deployment/cert-manager

# Check challenge status
kubectl get challenges -n bakery-ia

# Verify DNS resolution
nslookup bakery.example.com

---

Phase 6: Monitoring and Maintenance

Step 12: Setup Monitoring

# Prometheus is already enabled as a MicroK8s addon
kubectl get pods -n monitoring

# Access Grafana (if enabled)
kubectl port-forward -n monitoring svc/grafana 3000:3000

# Or expose via ingress (already configured in prod-ingress.yaml)

Step 13: Setup Backups

Create backup script on VPS:

cat > ~/backup-databases.sh <<'EOF'
#!/bin/bash
BACKUP_DIR="/backups/$(date +%Y-%m-%d)"
mkdir -p $BACKUP_DIR

# Get all database pods
DBS=$(kubectl get pods -n bakery-ia -l app.kubernetes.io/component=database -o name)

for db in $DBS; do
  DB_NAME=$(echo $db | cut -d'/' -f2)
  echo "Backing up $DB_NAME..."

  kubectl exec -n bakery-ia $db -- pg_dump -U postgres > "$BACKUP_DIR/${DB_NAME}.sql"
done

# Compress backups
tar -czf "$BACKUP_DIR.tar.gz" "$BACKUP_DIR"
rm -rf "$BACKUP_DIR"

# Keep only last 7 days
find /backups -name "*.tar.gz" -mtime +7 -delete

echo "Backup completed: $BACKUP_DIR.tar.gz"
EOF

chmod +x ~/backup-databases.sh

# Setup daily cron job
(crontab -l 2>/dev/null; echo "0 2 * * * ~/backup-databases.sh") | crontab -

Step 14: Setup Log Aggregation (Optional)

# Enable Loki for log aggregation
microk8s enable observability

# Or use external logging service like ELK, Datadog, etc.

---

Phase 7: Post-Deployment Verification

Step 15: Health Checks

# Test frontend
curl -k https://bakery.example.com

# Test API
curl -k https://api.example.com/health

# Test database connectivity
kubectl exec -n bakery-ia deployment/auth-service -- curl localhost:8000/health

# Check all services are healthy
kubectl get pods -n bakery-ia -o wide

# Check resource usage
kubectl top pods -n bakery-ia
kubectl top nodes

Step 16: Performance Testing

# Install hey (HTTP load testing tool)
go install github.com/rakyll/hey@latest

# Test API endpoint
hey -n 1000 -c 10 https://api.example.com/health

# Monitor during load test
kubectl top pods -n bakery-ia

---

Ongoing Operations

Updating the Application

# On local machine
# 1. Make code changes
# 2. Build and push new images
skaffold build -f skaffold-prod.yaml

# 3. Update image tags in prod kustomization
# 4. Apply updates
kubectl apply -k infrastructure/kubernetes/overlays/prod

# 5. Rolling update status
kubectl rollout status deployment/auth-service -n bakery-ia

Scaling Services

# Manual scaling
kubectl scale deployment auth-service -n bakery-ia --replicas=5

# Or update in kustomization.yaml and reapply

Database Migrations

# Run migration job
kubectl apply -f infrastructure/kubernetes/base/migrations/auth-migration-job.yaml

# Check migration status
kubectl get jobs -n bakery-ia
kubectl logs -n bakery-ia job/auth-migration

---

Troubleshooting Common Issues

Issue 1: Pods Not Starting

# Check pod status
kubectl describe pod POD_NAME -n bakery-ia

# Common causes:
# - Image pull errors: Check registry credentials
# - Resource limits: Check node resources
# - Volume mount issues: Check PVC status

Issue 2: Ingress Not Working

# Check ingress controller
kubectl get pods -n ingress

# Check ingress resource
kubectl describe ingress bakery-ingress-prod -n bakery-ia

# Check if port 80/443 are open
sudo netstat -tlnp | grep -E '(80|443)'

# Check NGINX logs
kubectl logs -n ingress -l app.kubernetes.io/name=ingress-nginx

Issue 3: SSL Certificate Issues

# Check certificate status
kubectl describe certificate bakery-ia-prod-tls-cert -n bakery-ia

# Check cert-manager logs
kubectl logs -n cert-manager deployment/cert-manager

# Verify DNS
dig bakery.example.com

# Manual certificate request
kubectl delete certificate bakery-ia-prod-tls-cert -n bakery-ia
kubectl apply -f infrastructure/kubernetes/overlays/prod/prod-ingress.yaml

Issue 4: Database Connection Errors

# Check database pod
kubectl get pods -n bakery-ia -l app.kubernetes.io/component=database

# Check database logs
kubectl logs -n bakery-ia deployment/auth-db

# Test connection from service pod
kubectl exec -n bakery-ia deployment/auth-service -- nc -zv auth-db 5432

Issue 5: Out of Resources

# Check node resources
kubectl describe node

# Check resource requests/limits
kubectl describe pod POD_NAME -n bakery-ia

# Adjust resource limits in prod kustomization or scale down

---

Security Hardening Checklist

• Change all default passwords
• Enable pod security policies
• Setup network policies
• Enable audit logging
• Regular security updates
• Implement secrets rotation
• Setup intrusion detection
• Enable RBAC properly
• Regular backup testing
• Implement rate limiting
• Setup DDoS protection
• Enable security scanning

---

Performance Optimization

For VPS with Limited Resources

If your VPS has limited resources, consider:

# Reduce replica counts in prod kustomization.yaml
replicas:
  - name: auth-service
    count: 2  # Instead of 3
  - name: gateway
    count: 2  # Instead of 3

# Adjust resource limits
resources:
  requests:
    memory: "256Mi"  # Reduced from 512Mi
    cpu: "100m"      # Reduced from 200m

Database Optimization

# Tune PostgreSQL for production
kubectl exec -n bakery-ia deployment/auth-db -it -- psql -U postgres

# Inside PostgreSQL:
ALTER SYSTEM SET shared_buffers = '256MB';
ALTER SYSTEM SET effective_cache_size = '1GB';
ALTER SYSTEM SET maintenance_work_mem = '64MB';
ALTER SYSTEM SET checkpoint_completion_target = '0.9';
ALTER SYSTEM SET wal_buffers = '16MB';
ALTER SYSTEM SET default_statistics_target = '100';

# Restart database pod
kubectl rollout restart deployment/auth-db -n bakery-ia

---

Rollback Procedure

If something goes wrong:

# Rollback deployment
kubectl rollout undo deployment/DEPLOYMENT_NAME -n bakery-ia

# Rollback to specific revision
kubectl rollout history deployment/DEPLOYMENT_NAME -n bakery-ia
kubectl rollout undo deployment/DEPLOYMENT_NAME --to-revision=2 -n bakery-ia

# Restore from backup
tar -xzf /backups/2024-01-01.tar.gz
kubectl exec -n bakery-ia deployment/auth-db -- psql -U postgres < auth-db.sql

---

Quick Reference

Useful Commands

# View all resources
kubectl get all -n bakery-ia

# Get pod logs
kubectl logs -f POD_NAME -n bakery-ia

# Execute command in pod
kubectl exec -it POD_NAME -n bakery-ia -- /bin/bash

# Port forward for debugging
kubectl port-forward svc/SERVICE_NAME 8000:8000 -n bakery-ia

# Check events
kubectl get events -n bakery-ia --sort-by='.lastTimestamp'

# Resource usage
kubectl top nodes
kubectl top pods -n bakery-ia

# Restart deployment
kubectl rollout restart deployment/DEPLOYMENT_NAME -n bakery-ia

# Scale deployment
kubectl scale deployment/DEPLOYMENT_NAME --replicas=3 -n bakery-ia

Important File Locations on VPS

/var/snap/microk8s/current/credentials/  # Kubernetes credentials
/var/snap/microk8s/common/default-storage/  # Default storage location
~/kubernetes/                            # Your manifests
/backups/                               # Database backups

---

Next Steps After Migration

1. Setup CI/CD Pipeline

   • GitHub Actions or GitLab CI
   • Automated builds and deployments
   • Automated testing

2. Implement Monitoring Dashboards

   • Setup Grafana dashboards
   • Configure alerts
   • Setup uptime monitoring

3. Disaster Recovery Plan

   • Document recovery procedures
   • Test backup restoration
   • Setup off-site backups

4. Cost Optimization

   • Monitor resource usage
   • Right-size deployments
   • Implement auto-scaling

5. Documentation

   • Document custom configurations
   • Create runbooks for common tasks
   • Train team members

---

Support and Resources

• MicroK8s Documentation: https://microk8s.io/docs
• Kubernetes Documentation: https://kubernetes.io/docs
• cert-manager Documentation: https://cert-manager.io/docs
• NGINX Ingress: https://kubernetes.github.io/ingress-nginx

Conclusion

This migration moves your application from a local development environment to a production-ready deployment. Remember to:

• Test thoroughly before going live
• Have a rollback plan ready
• Monitor closely after deployment
• Keep regular backups
• Stay updated with security patches

Good luck with your deployment! 🚀