Add base kubernetes support final fix 2
This commit is contained in:
@@ -1,214 +1,86 @@
|
||||
# Bakery IA Kubernetes Configuration
|
||||
|
||||
This directory contains Kubernetes manifests for deploying the Bakery IA forecasting platform in a local development environment with **permanent localhost access** and **FREE HTTPS support** using cert-manager and NGINX ingress.
|
||||
This directory contains Kubernetes manifests for deploying the Bakery IA platform in local development and production environments with HTTPS support using cert-manager and NGINX ingress.
|
||||
|
||||
## ⚡ Quick Start (5 Commands)
|
||||
## Quick Start
|
||||
|
||||
Deploy the entire platform with these 5 commands:
|
||||
|
||||
```bash
|
||||
# 1. Start Colima
|
||||
colima start --cpu 4 --memory 8 --disk 50 --runtime docker --profile k8s-local
|
||||
# 1. Start Colima with adequate resources
|
||||
colima start --cpu 4 --memory 8 --disk 100 --runtime docker --profile k8s-local
|
||||
|
||||
# 2. Create Kind cluster with permanent localhost access
|
||||
kind create cluster --config kind-config.yaml
|
||||
|
||||
# 3. Install NGINX Ingress Controller
|
||||
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/deploy/static/provider/kind/deploy.yaml && kubectl wait --namespace ingress-nginx --for=condition=ready pod --selector=app.kubernetes.io/component=controller --timeout=300s
|
||||
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/deploy/static/provider/kind/deploy.yaml
|
||||
kubectl wait --namespace ingress-nginx --for=condition=ready pod --selector=app.kubernetes.io/component=controller --timeout=300s
|
||||
|
||||
# 4. Configure permanent localhost access
|
||||
kubectl patch svc ingress-nginx-controller -n ingress-nginx -p '{"spec":{"type":"NodePort","ports":[{"name":"http","port":80,"targetPort":"http","nodePort":30080},{"name":"https","port":443,"targetPort":"https","nodePort":30443}]}}'
|
||||
|
||||
# 5. Deploy your application
|
||||
# 5. Deploy with Skaffold
|
||||
skaffold dev --profile=dev
|
||||
|
||||
# 🎉 Done! Access at: http://localhost
|
||||
# 🎉 Access at: https://localhost
|
||||
```
|
||||
|
||||
## Prerequisites (macOS Local Development)
|
||||
## Prerequisites
|
||||
|
||||
1. **Colima**: Docker runtime for macOS
|
||||
2. **Kind**: Kubernetes in Docker for local clusters
|
||||
3. **kubectl**: Kubernetes command-line tool
|
||||
4. **Skaffold**: For building and deploying applications
|
||||
5. **NGINX Ingress Controller**: For routing traffic (installed automatically)
|
||||
6. **cert-manager**: For automatic TLS certificate management (installed automatically)
|
||||
|
||||
### Install Prerequisites (macOS)
|
||||
Install the following tools on macOS:
|
||||
|
||||
```bash
|
||||
# Install Homebrew (if not already installed)
|
||||
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
|
||||
|
||||
# Install required tools
|
||||
# Install via Homebrew
|
||||
brew install colima kind kubectl skaffold
|
||||
|
||||
# Verify installations
|
||||
colima version
|
||||
kind version
|
||||
kubectl version --client
|
||||
skaffold version
|
||||
colima version && kind version && kubectl version --client && skaffold version
|
||||
```
|
||||
|
||||
## 🔒 HTTPS Setup Options
|
||||
|
||||
### Option 1: Automated HTTPS Setup (Recommended)
|
||||
```bash
|
||||
# Run the automated HTTPS setup script
|
||||
./setup-https.sh
|
||||
```
|
||||
|
||||
### Option 2: HTTP Only (Basic Setup)
|
||||
```bash
|
||||
# Deploy without HTTPS
|
||||
kubectl apply -k infrastructure/kubernetes/overlays/dev/
|
||||
```
|
||||
|
||||
## Kind Configuration for Permanent Localhost Access
|
||||
|
||||
The `kind-config.yaml` file in the root directory provides permanent localhost access without port forwarding:
|
||||
|
||||
```yaml
|
||||
kind: Cluster
|
||||
apiVersion: kind.x-k8s.io/v1alpha4
|
||||
name: bakery-ia-local
|
||||
nodes:
|
||||
- role: control-plane
|
||||
kubeadmConfigPatches:
|
||||
- |
|
||||
kind: InitConfiguration
|
||||
nodeRegistration:
|
||||
kubeletExtraArgs:
|
||||
node-labels: "ingress-ready=true"
|
||||
extraPortMappings:
|
||||
# HTTP ingress
|
||||
- containerPort: 30080
|
||||
hostPort: 80
|
||||
protocol: TCP
|
||||
# HTTPS ingress
|
||||
- containerPort: 30443
|
||||
hostPort: 443
|
||||
protocol: TCP
|
||||
# Direct frontend access (backup)
|
||||
- containerPort: 30300
|
||||
hostPort: 3000
|
||||
protocol: TCP
|
||||
# Direct gateway access (backup)
|
||||
- containerPort: 30800
|
||||
hostPort: 8000
|
||||
protocol: TCP
|
||||
```
|
||||
|
||||
This configuration maps:
|
||||
- Port 80 → localhost:80 (HTTP)
|
||||
- Port 443 → localhost:443 (HTTPS)
|
||||
- Port 3000 → localhost:3000 (Direct frontend)
|
||||
- Port 8000 → localhost:8000 (Direct gateway)
|
||||
|
||||
## Directory Structure
|
||||
|
||||
```
|
||||
infrastructure/kubernetes/
|
||||
├── kind-config.yaml # Kind cluster configuration with port mapping
|
||||
├── base/ # Base Kubernetes resources
|
||||
│ ├── namespace.yaml # Namespace definition
|
||||
│ ├── configmap.yaml # Shared configuration
|
||||
│ ├── secrets.yaml # Secrets (base64 encoded)
|
||||
│ ├── ingress.yaml # HTTP ingress rules
|
||||
│ ├── secrets.yaml # Base64 encoded secrets
|
||||
│ ├── ingress-https.yaml # HTTPS ingress rules
|
||||
│ └── kustomization.yaml # Base kustomization
|
||||
├── components/ # Individual component manifests
|
||||
│ ├── cert-manager/ # Certificate management
|
||||
│ │ ├── cluster-issuer-staging.yaml # Let's Encrypt staging
|
||||
│ │ ├── cluster-issuer-production.yaml # Let's Encrypt production
|
||||
│ │ └── local-ca-issuer.yaml # Local CA for development
|
||||
│ ├── auth/ # Auth service
|
||||
│ ├── tenant/ # Tenant service
|
||||
│ ├── training/ # Training service
|
||||
│ ├── forecasting/ # Forecasting service
|
||||
│ ├── sales/ # Sales service
|
||||
│ ├── external/ # External service
|
||||
│ ├── notification/ # Notification service
|
||||
│ ├── inventory/ # Inventory service
|
||||
│ ├── recipes/ # Recipes service
|
||||
│ ├── suppliers/ # Suppliers service
|
||||
│ ├── pos/ # POS service
|
||||
│ ├── orders/ # Orders service
|
||||
│ ├── production/ # Production service
|
||||
│ ├── alert-processor/ # Alert processor
|
||||
│ ├── frontend/ # Frontend application
|
||||
│ ├── databases/ # Database deployments
|
||||
│ └── infrastructure/ # Infrastructure components (gateway, etc.)
|
||||
│ ├── kustomization.yaml # Base kustomization
|
||||
│ └── components/ # Individual component manifests
|
||||
│ ├── cert-manager/ # Certificate management
|
||||
│ ├── auth/ # Authentication service
|
||||
│ ├── tenant/ # Tenant management
|
||||
│ ├── training/ # ML training service
|
||||
│ ├── forecasting/ # Demand forecasting
|
||||
│ ├── sales/ # Sales management
|
||||
│ ├── external/ # External API service
|
||||
│ ├── notification/ # Notification service
|
||||
│ ├── inventory/ # Inventory management
|
||||
│ ├── recipes/ # Recipe management
|
||||
│ ├── suppliers/ # Supplier management
|
||||
│ ├── pos/ # Point of sale
|
||||
│ ├── orders/ # Order management
|
||||
│ ├── production/ # Production planning
|
||||
│ ├── alert-processor/ # Alert processing
|
||||
│ ├── frontend/ # React frontend
|
||||
│ ├── databases/ # Database deployments
|
||||
│ └── infrastructure/ # Gateway & monitoring
|
||||
└── overlays/
|
||||
└── dev/ # Development environment overlay
|
||||
├── kustomization.yaml # Dev-specific kustomization
|
||||
├── https-kustomization.yaml # HTTPS-specific kustomization
|
||||
├── dev-patches.yaml # Development patches
|
||||
└── ingress-https-patch.yaml # HTTPS ingress patch
|
||||
└── dev/ # Development environment
|
||||
├── kustomization.yaml # Dev-specific configuration
|
||||
└── dev-patches.yaml # Development patches
|
||||
```
|
||||
|
||||
## 🚀 Quick Start (macOS with Kind + Colima)
|
||||
## Access URLs
|
||||
|
||||
### 1. Start Colima and Create Kind Cluster with Permanent Localhost Access
|
||||
### Primary Access (Standard Web Ports)
|
||||
- **Frontend**: https://localhost
|
||||
- **API Gateway**: https://localhost/api
|
||||
|
||||
```bash
|
||||
# Start Colima with proper resources for development
|
||||
colima start --cpu 4 --memory 8 --disk 100 --runtime docker --profile k8s-local
|
||||
|
||||
# Create Kind cluster with permanent port mapping for localhost access
|
||||
kind create cluster --config kind-config.yaml
|
||||
|
||||
# Verify cluster is running and port mappings
|
||||
kubectl cluster-info
|
||||
docker port bakery-ia-local-control-plane
|
||||
```
|
||||
|
||||
The `kind-config.yaml` configuration provides permanent localhost access on ports 80 and 443 without requiring port forwarding!
|
||||
|
||||
### 2. Install NGINX Ingress Controller for Kind
|
||||
|
||||
```bash
|
||||
# Install NGINX Ingress Controller (Kind-specific with permanent localhost access)
|
||||
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/deploy/static/provider/kind/deploy.yaml
|
||||
|
||||
# Wait for ingress controller to be ready
|
||||
kubectl wait --namespace ingress-nginx \
|
||||
--for=condition=ready pod \
|
||||
--selector=app.kubernetes.io/component=controller \
|
||||
--timeout=300s
|
||||
|
||||
# Configure ingress controller for permanent localhost access
|
||||
kubectl patch svc ingress-nginx-controller -n ingress-nginx -p '{"spec":{"type":"NodePort","ports":[{"name":"http","port":80,"targetPort":"http","nodePort":30080},{"name":"https","port":443,"targetPort":"https","nodePort":30443}]}}'
|
||||
```
|
||||
|
||||
### 3. Deploy with Skaffold (No Port Forwarding Required!)
|
||||
|
||||
```bash
|
||||
# Option A: Development mode with auto-rebuild (Recommended)
|
||||
skaffold dev --profile=dev
|
||||
|
||||
# Option B: One-time deployment
|
||||
skaffold run --profile=dev
|
||||
|
||||
# Option C: Debug mode (still includes port forwarding for individual services)
|
||||
skaffold debug --profile=debug
|
||||
|
||||
# Check deployment status
|
||||
kubectl get pods -n bakery-ia
|
||||
kubectl get services -n bakery-ia
|
||||
kubectl get ingress -n bakery-ia
|
||||
```
|
||||
|
||||
**Note**: With the new configuration, skaffold no longer needs port forwarding for frontend access since localhost:80 and localhost:443 are permanently mapped!
|
||||
|
||||
### 4. Access the Application - Permanent Localhost Access! 🎉
|
||||
|
||||
**No /etc/hosts modification needed!** The application is now accessible directly via standard localhost URLs:
|
||||
|
||||
**Primary Access (Recommended):**
|
||||
- **Frontend**: http://localhost or https://localhost
|
||||
- **API Gateway**: http://localhost/api or https://localhost/api
|
||||
|
||||
**Named Host Access (Optional):**
|
||||
If you prefer named hosts, add to your `/etc/hosts` file:
|
||||
### Named Host Access (Optional)
|
||||
Add to `/etc/hosts` for named access:
|
||||
```bash
|
||||
echo "127.0.0.1 bakery-ia.local" | sudo tee -a /etc/hosts
|
||||
echo "127.0.0.1 api.bakery-ia.local" | sudo tee -a /etc/hosts
|
||||
@@ -216,508 +88,186 @@ echo "127.0.0.1 monitoring.bakery-ia.local" | sudo tee -a /etc/hosts
|
||||
```
|
||||
|
||||
Then access via:
|
||||
- Frontend: http://bakery-ia.local or https://bakery-ia.local
|
||||
- API Gateway: http://api.bakery-ia.local or https://api.bakery-ia.local
|
||||
- Monitoring: http://monitoring.bakery-ia.local or https://monitoring.bakery-ia.local
|
||||
- **Frontend**: https://bakery-ia.local
|
||||
- **API**: https://api.bakery-ia.local
|
||||
- **Monitoring**: https://monitoring.bakery-ia.local
|
||||
|
||||
## 🔒 HTTPS Configuration (FREE with Let's Encrypt)
|
||||
### Direct Service Access (Development)
|
||||
- **Frontend**: http://localhost:3000
|
||||
- **Gateway**: http://localhost:8000
|
||||
|
||||
### Automated HTTPS Setup
|
||||
|
||||
The quickest way to enable HTTPS is using the automated setup script:
|
||||
## Development Workflow
|
||||
|
||||
### Start Development Environment
|
||||
```bash
|
||||
# Run the automated HTTPS setup script
|
||||
./setup-https.sh
|
||||
# Start development mode with hot-reload
|
||||
skaffold dev --profile=dev
|
||||
|
||||
# Or one-time deployment
|
||||
skaffold run --profile=dev
|
||||
```
|
||||
|
||||
This script will:
|
||||
- ✅ Install cert-manager (FREE Let's Encrypt client)
|
||||
- ✅ Install NGINX Ingress Controller
|
||||
- ✅ Set up cluster issuers (staging, production, and local CA)
|
||||
- ✅ Deploy your application with HTTPS support
|
||||
- ✅ Generate and configure TLS certificates
|
||||
- ✅ Export CA certificate for browser trust
|
||||
|
||||
### Manual HTTPS Setup
|
||||
|
||||
If you prefer manual setup:
|
||||
|
||||
#### 1. Install cert-manager
|
||||
```bash
|
||||
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.13.2/cert-manager.yaml
|
||||
kubectl wait --for=condition=ready pod -l app.kubernetes.io/instance=cert-manager -n cert-manager --timeout=300s
|
||||
```
|
||||
|
||||
#### 2. Install NGINX Ingress Controller for Kind
|
||||
```bash
|
||||
kubectl apply -f https://kind.sigs.k8s.io/examples/ingress/deploy-ingress-nginx.yaml
|
||||
kubectl wait --namespace ingress-nginx --for=condition=ready pod --selector=app.kubernetes.io/component=controller --timeout=300s
|
||||
```
|
||||
|
||||
#### 3. Apply Certificate Issuers
|
||||
```bash
|
||||
kubectl apply -f infrastructure/kubernetes/base/components/cert-manager/cluster-issuer-staging.yaml
|
||||
kubectl apply -f infrastructure/kubernetes/base/components/cert-manager/local-ca-issuer.yaml
|
||||
kubectl apply -f infrastructure/kubernetes/base/components/cert-manager/cluster-issuer-production.yaml
|
||||
```
|
||||
|
||||
#### 4. Deploy with HTTPS
|
||||
```bash
|
||||
kubectl apply -k infrastructure/kubernetes/overlays/dev/
|
||||
kubectl patch ingress bakery-ingress -n bakery-ia --patch-file infrastructure/kubernetes/overlays/dev/ingress-https-patch.yaml
|
||||
```
|
||||
|
||||
#### 5. Export CA Certificate for Browser Trust
|
||||
```bash
|
||||
kubectl get secret local-ca-key-pair -n cert-manager -o jsonpath='{.data.tls\.crt}' | base64 -d > bakery-ia-ca.crt
|
||||
```
|
||||
|
||||
### Access HTTPS Application
|
||||
|
||||
After HTTPS setup:
|
||||
- **🔐 Frontend:** https://bakery-ia.local
|
||||
- **🔐 API Gateway:** https://api.bakery-ia.local
|
||||
- **🔐 Monitoring:** https://monitoring.bakery-ia.local
|
||||
|
||||
### Trust the CA Certificate
|
||||
|
||||
**For macOS:**
|
||||
```bash
|
||||
open bakery-ia-ca.crt
|
||||
# In Keychain Access, find "bakery-ia-local-ca" and set to "Always Trust"
|
||||
```
|
||||
|
||||
**For Linux:**
|
||||
```bash
|
||||
sudo cp bakery-ia-ca.crt /usr/local/share/ca-certificates/
|
||||
sudo update-ca-certificates
|
||||
```
|
||||
|
||||
### Certificate Management Commands
|
||||
|
||||
```bash
|
||||
# Check certificate status
|
||||
kubectl get certificates -n bakery-ia
|
||||
|
||||
# Check certificate details
|
||||
kubectl describe certificate bakery-ia-tls-cert -n bakery-ia
|
||||
|
||||
# Check cluster issuers
|
||||
kubectl get clusterissuers
|
||||
|
||||
# Check TLS secret
|
||||
kubectl get secret bakery-ia-tls-cert -n bakery-ia
|
||||
```
|
||||
|
||||
### Switching to Production Let's Encrypt
|
||||
|
||||
To use real Let's Encrypt certificates (requires public domain):
|
||||
|
||||
1. Update the cluster issuer in `ingress-https-patch.yaml`:
|
||||
```yaml
|
||||
cert-manager.io/cluster-issuer: "letsencrypt-production" # Change from local-ca-issuer
|
||||
```
|
||||
|
||||
2. Update email in cluster issuers to your real email
|
||||
3. Ensure your domain points to your cluster's external IP
|
||||
|
||||
### Cleanup HTTPS Setup
|
||||
|
||||
```bash
|
||||
# Run cleanup script
|
||||
./cleanup-https.sh
|
||||
|
||||
# Or manually clean up
|
||||
kubectl delete -k infrastructure/kubernetes/overlays/dev/
|
||||
kubectl delete -f https://github.com/cert-manager/cert-manager/releases/download/v1.13.2/cert-manager.yaml
|
||||
rm -f bakery-ia-ca.crt
|
||||
```
|
||||
|
||||
## Port Forwarding for Direct Access
|
||||
|
||||
If you prefer to access services directly without ingress:
|
||||
|
||||
```bash
|
||||
# Frontend
|
||||
kubectl port-forward -n bakery-ia svc/frontend-service 3000:3000
|
||||
|
||||
# Gateway
|
||||
kubectl port-forward -n bakery-ia svc/gateway-service 8000:8000
|
||||
|
||||
# Auth Service
|
||||
kubectl port-forward -n bakery-ia svc/auth-service 8001:8000
|
||||
|
||||
# Redis
|
||||
kubectl port-forward -n bakery-ia svc/redis-service 6379:6379
|
||||
|
||||
# Database example (auth-db)
|
||||
kubectl port-forward -n bakery-ia svc/auth-db-service 5432:5432
|
||||
```
|
||||
|
||||
## Managing the Deployment
|
||||
|
||||
### Check Status
|
||||
### Key Features
|
||||
- ✅ **Hot-reload development** - Automatic rebuilds on code changes
|
||||
- ✅ **Permanent localhost access** - No port forwarding needed
|
||||
- ✅ **HTTPS by default** - Local CA certificates for secure development
|
||||
- ✅ **Microservices architecture** - All services deployed together
|
||||
- ✅ **Database management** - PostgreSQL, Redis, and RabbitMQ included
|
||||
|
||||
### Monitor and Debug
|
||||
```bash
|
||||
# Check all resources
|
||||
kubectl get all -n bakery-ia
|
||||
|
||||
# Check specific resource types
|
||||
kubectl get pods -n bakery-ia
|
||||
kubectl get services -n bakery-ia
|
||||
kubectl get deployments -n bakery-ia
|
||||
kubectl get pvc -n bakery-ia
|
||||
# View logs
|
||||
kubectl logs -n bakery-ia deployment/auth-service -f
|
||||
|
||||
# Check logs
|
||||
kubectl logs -n bakery-ia deployment/auth-service
|
||||
kubectl logs -n bakery-ia deployment/frontend -f # Follow logs
|
||||
# Check ingress status
|
||||
kubectl get ingress -n bakery-ia
|
||||
|
||||
# Debug certificate issues
|
||||
kubectl describe certificate bakery-ia-tls-cert -n bakery-ia
|
||||
```
|
||||
|
||||
### Update Deployments
|
||||
## Certificate Management
|
||||
|
||||
The platform uses cert-manager for automatic HTTPS certificate generation:
|
||||
|
||||
- **Local CA**: For development (default)
|
||||
- **Let's Encrypt Staging**: For testing
|
||||
- **Let's Encrypt Production**: For production deployments
|
||||
|
||||
### Trust Local Certificates
|
||||
```bash
|
||||
# After making changes to manifests
|
||||
kubectl apply -k infrastructure/kubernetes/overlays/dev/
|
||||
# Export CA certificate
|
||||
kubectl get secret local-ca-key-pair -n cert-manager -o jsonpath='{.data.tls\.crt}' | base64 -d > bakery-ia-ca.crt
|
||||
|
||||
# Force restart a deployment
|
||||
kubectl rollout restart -n bakery-ia deployment/auth-service
|
||||
|
||||
# Check rollout status
|
||||
kubectl rollout status -n bakery-ia deployment/auth-service
|
||||
# Trust in macOS
|
||||
open bakery-ia-ca.crt
|
||||
# In Keychain Access, set "bakery-ia-local-ca" to "Always Trust"
|
||||
```
|
||||
|
||||
### Scaling Services
|
||||
|
||||
```bash
|
||||
# Scale a service
|
||||
kubectl scale -n bakery-ia deployment/auth-service --replicas=3
|
||||
|
||||
# Or edit the kustomization.yaml replicas section and reapply
|
||||
```
|
||||
|
||||
### Clean Up (macOS + Kind + Colima + Skaffold)
|
||||
|
||||
```bash
|
||||
# Option 1: Quick cleanup (development session)
|
||||
skaffold delete --profile=dev
|
||||
|
||||
# Option 2: Clean up HTTPS setup
|
||||
./cleanup-https.sh
|
||||
|
||||
# Option 3: Complete cleanup (everything)
|
||||
./complete-cleanup.sh
|
||||
|
||||
# Option 4: Manual cleanup steps
|
||||
kubectl delete namespace bakery-ia
|
||||
kind delete cluster --name bakery-ia-local
|
||||
colima stop --profile k8s-local
|
||||
```
|
||||
|
||||
**📖 For detailed cleanup options, see [CLEANUP-GUIDE.md](../../CLEANUP-GUIDE.md)**
|
||||
|
||||
## Configuration
|
||||
## Configuration Management
|
||||
|
||||
### Secrets
|
||||
Base64-encoded secrets are stored in `base/secrets.yaml`. For production:
|
||||
- Use external secret management (HashiCorp Vault, AWS Secrets Manager)
|
||||
- Never commit real secrets to version control
|
||||
|
||||
The `secrets.yaml` file contains base64-encoded secrets. For production, these should be:
|
||||
1. Generated securely
|
||||
2. Managed through external secret management systems
|
||||
3. Not committed to version control
|
||||
|
||||
To encode/decode secrets:
|
||||
```bash
|
||||
# Encode
|
||||
# Encode secrets
|
||||
echo -n "your-secret-value" | base64
|
||||
|
||||
# Decode
|
||||
# Decode secrets
|
||||
echo "eW91ci1zZWNyZXQtdmFsdWU=" | base64 -d
|
||||
```
|
||||
|
||||
### Environment-Specific Configuration
|
||||
### Environment Configuration
|
||||
Development-specific settings are in `overlays/dev/`:
|
||||
- **Resource limits**: Reduced for local development
|
||||
- **Image pull policy**: Never (for local images)
|
||||
- **Debug settings**: Enabled
|
||||
- **CORS**: Configured for localhost
|
||||
|
||||
Modify the `overlays/dev/` files to customize the development environment:
|
||||
- `kustomization.yaml`: Image tags, replicas, resource references
|
||||
- `dev-patches.yaml`: Environment-specific configuration overrides
|
||||
## Scaling and Resource Management
|
||||
|
||||
### Adding New Services
|
||||
### Scale Services
|
||||
```bash
|
||||
# Scale individual service
|
||||
kubectl scale -n bakery-ia deployment/auth-service --replicas=3
|
||||
|
||||
1. Create a new directory under `components/`
|
||||
2. Add the service YAML manifest
|
||||
3. Update `base/kustomization.yaml` to include the new resource
|
||||
4. Update configuration maps and secrets as needed
|
||||
# Or update kustomization.yaml replicas section
|
||||
```
|
||||
|
||||
### Resource Configuration
|
||||
Development environment uses minimal resources:
|
||||
- **Databases**: 64Mi-256Mi memory, 25m-200m CPU
|
||||
- **Services**: 64Mi-256Mi memory, 25m-200m CPU
|
||||
- **Training Service**: 256Mi-1Gi memory (ML workloads)
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Common Issues
|
||||
|
||||
1. **Images not found**: Ensure images are built and available to the cluster
|
||||
2. **Pending pods**: Check resource requests and cluster capacity
|
||||
3. **CrashLoopBackOff**: Check logs and environment variables
|
||||
4. **Service not accessible**: Verify ingress controller is running and localhost ports are mapped
|
||||
5. **Database corruption**: If PostgreSQL databases show "could not locate a valid checkpoint record", delete the PVC and restart the pod to get fresh storage
|
||||
6. **Port conflicts**: If localhost:80 or localhost:443 are already in use, stop other services or change the Kind configuration
|
||||
7. **HTTPS certificate not issued**: Check cert-manager logs and cluster issuer status
|
||||
8. **Browser security warnings**: Import and trust the CA certificate (`bakery-ia-ca.crt`)
|
||||
9. **Certificate pending**: Wait for cert-manager to issue the certificate (usually takes 30-60 seconds)
|
||||
10. **Kustomize deprecation warnings**: Fixed - using modern `patches` syntax instead of deprecated `patchesStrategicMerge` and `patchesJson6902`
|
||||
1. **Images not found**
|
||||
```bash
|
||||
# Build images with Skaffold
|
||||
skaffold build --profile=dev
|
||||
```
|
||||
|
||||
### Database Recovery Commands
|
||||
2. **Database corruption after restart**
|
||||
```bash
|
||||
# Delete corrupted PVC and restart
|
||||
kubectl delete pod -n bakery-ia -l app.kubernetes.io/name=inventory-db
|
||||
kubectl delete pvc -n bakery-ia inventory-db-pvc
|
||||
```
|
||||
|
||||
If you encounter database corruption (common after improper cluster shutdown):
|
||||
3. **HTTPS certificate not issued**
|
||||
```bash
|
||||
# Check cert-manager logs
|
||||
kubectl logs -n cert-manager deployment/cert-manager
|
||||
kubectl describe certificate bakery-ia-tls-cert -n bakery-ia
|
||||
```
|
||||
|
||||
4. **Port conflicts**
|
||||
```bash
|
||||
# Check what's using ports 80/443
|
||||
sudo lsof -i :80 -i :443
|
||||
```
|
||||
|
||||
### Debug Commands
|
||||
```bash
|
||||
# Check which databases are failing
|
||||
kubectl get pods -n bakery-ia | grep -E "(db|CrashLoopBackOff)"
|
||||
|
||||
# For each corrupted database (example with inventory-db):
|
||||
kubectl delete pod -n bakery-ia -l app.kubernetes.io/name=inventory-db
|
||||
kubectl delete pvc -n bakery-ia inventory-db-pvc
|
||||
|
||||
# The deployment will automatically recreate with fresh storage
|
||||
# Repeat for pos-db-pvc and training-db-pvc if needed
|
||||
```
|
||||
|
||||
### Debugging Commands
|
||||
|
||||
```bash
|
||||
# Describe resources for detailed information
|
||||
kubectl describe pod -n bakery-ia <pod-name>
|
||||
kubectl describe deployment -n bakery-ia <deployment-name>
|
||||
|
||||
# Get events
|
||||
# Get cluster events
|
||||
kubectl get events -n bakery-ia --sort-by='.firstTimestamp'
|
||||
|
||||
# Execute commands in pods
|
||||
kubectl exec -n bakery-ia -it <pod-name> -- bash
|
||||
kubectl exec -n bakery-ia -it <pod-name> -- env
|
||||
|
||||
# Check resource usage
|
||||
# Resource usage
|
||||
kubectl top pods -n bakery-ia
|
||||
kubectl top nodes
|
||||
|
||||
# HTTPS/Certificate debugging
|
||||
kubectl logs -n cert-manager deployment/cert-manager
|
||||
kubectl describe clusterissuer letsencrypt-staging
|
||||
kubectl describe certificate bakery-ia-tls-cert -n bakery-ia
|
||||
kubectl get challenges -n bakery-ia
|
||||
kubectl get certificaterequests -n bakery-ia
|
||||
# Execute in pod
|
||||
kubectl exec -n bakery-ia -it <pod-name> -- bash
|
||||
```
|
||||
|
||||
## Production Considerations
|
||||
|
||||
For production deployment, consider:
|
||||
|
||||
1. **Resource Limits**: Set appropriate CPU and memory limits
|
||||
2. **Persistent Volumes**: Use proper storage classes for databases
|
||||
3. **Secrets Management**: Use external secret management (HashiCorp Vault, AWS Secrets Manager, etc.)
|
||||
4. **Monitoring**: Deploy Prometheus and Grafana
|
||||
5. **Backup**: Implement database backup strategies
|
||||
6. **High Availability**: Use multiple replicas and anti-affinity rules
|
||||
7. **Security**: Network policies, RBAC, pod security policies
|
||||
8. **TLS/HTTPS**: Use production Let's Encrypt certificates for public domains
|
||||
9. **CI/CD**: Integrate with your deployment pipeline
|
||||
|
||||
## Next Steps
|
||||
|
||||
1. Add monitoring with Prometheus and Grafana
|
||||
2. Implement proper logging with ELK stack or similar
|
||||
3. Add health checks and metrics endpoints
|
||||
4. Implement automated testing
|
||||
5. Set up CI/CD pipelines for automated deployments
|
||||
|
||||
## 🚀 Complete Setup Guide (macOS + Kind + Colima) - New Permanent Solution!
|
||||
|
||||
### Method 1: Permanent Localhost Access (Recommended - No Port Forwarding!)
|
||||
## Cleanup
|
||||
|
||||
### Quick Cleanup
|
||||
```bash
|
||||
# 1. Start Colima
|
||||
colima start --cpu 4 --memory 8 --disk 50 --runtime docker --profile k8s-local
|
||||
|
||||
# 2. Create Kind cluster with permanent port mapping
|
||||
kind create cluster --config kind-config.yaml
|
||||
|
||||
# 3. Install NGINX Ingress Controller with NodePort configuration
|
||||
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/deploy/static/provider/kind/deploy.yaml
|
||||
kubectl wait --namespace ingress-nginx --for=condition=ready pod --selector=app.kubernetes.io/component=controller --timeout=300s
|
||||
|
||||
# 4. Configure ingress for permanent localhost access
|
||||
kubectl patch svc ingress-nginx-controller -n ingress-nginx -p '{"spec":{"type":"NodePort","ports":[{"name":"http","port":80,"targetPort":"http","nodePort":30080},{"name":"https","port":443,"targetPort":"https","nodePort":30443}]}}'
|
||||
|
||||
# 5. Deploy with Skaffold
|
||||
skaffold dev --profile=dev
|
||||
|
||||
# 6. Access your application - NO /etc/hosts needed!
|
||||
# Frontend: http://localhost
|
||||
# API: http://localhost/api
|
||||
# HTTPS: https://localhost (with browser security warnings)
|
||||
# Stop Skaffold (Ctrl+C or)
|
||||
skaffold delete --profile=dev
|
||||
```
|
||||
|
||||
### Method 2: Legacy Setup with HTTPS and Named Hosts
|
||||
|
||||
### Complete Cleanup
|
||||
```bash
|
||||
# 1. Start Colima
|
||||
colima start --cpu 4 --memory 8 --disk 50 --runtime docker --profile k8s-local
|
||||
|
||||
# 2. Create standard Kind cluster
|
||||
kind create cluster --name bakery-ia-local
|
||||
|
||||
# 3. Run automated HTTPS setup (includes cert-manager and ingress)
|
||||
./setup-https.sh
|
||||
|
||||
# 4. Deploy with Skaffold
|
||||
skaffold dev --profile=dev
|
||||
|
||||
# 5. Add hosts entries for named hosts
|
||||
sudo tee -a /etc/hosts << EOF
|
||||
127.0.0.1 bakery-ia.local
|
||||
127.0.0.1 api.bakery-ia.local
|
||||
127.0.0.1 monitoring.bakery-ia.local
|
||||
EOF
|
||||
|
||||
# 6. Trust CA certificate (for HTTPS)
|
||||
open bakery-ia-ca.crt
|
||||
# In Keychain Access, set "bakery-ia-local-ca" to "Always Trust"
|
||||
```
|
||||
|
||||
## 🚀 Skaffold Development Workflow
|
||||
|
||||
### Development Mode (Recommended)
|
||||
```bash
|
||||
# Start continuous development mode
|
||||
skaffold dev --profile=dev
|
||||
```
|
||||
|
||||
This will:
|
||||
- ✅ **Build all Docker images** automatically
|
||||
- ✅ **Deploy to your Kind cluster**
|
||||
- ✅ **Watch for file changes** in real-time
|
||||
- ✅ **Automatically rebuild and redeploy** when you save files
|
||||
- ✅ **Stream logs** from all services in one terminal
|
||||
|
||||
### Other Skaffold Commands
|
||||
|
||||
```bash
|
||||
# One-time deployment (no file watching)
|
||||
skaffold run --profile=dev
|
||||
|
||||
# Debug mode with port forwarding
|
||||
skaffold debug --profile=debug
|
||||
|
||||
# Force rebuild and deploy
|
||||
skaffold build --file-output=build.json
|
||||
skaffold deploy --build-artifacts=build.json
|
||||
|
||||
# Clean up deployed resources
|
||||
skaffold delete
|
||||
```
|
||||
|
||||
### Stopping Skaffold
|
||||
|
||||
```bash
|
||||
# Stop Skaffold (press Ctrl+C in the terminal running skaffold dev)
|
||||
# Or run:
|
||||
skaffold delete
|
||||
|
||||
# Complete cleanup
|
||||
# Delete everything
|
||||
kubectl delete namespace bakery-ia
|
||||
kind delete cluster --name bakery-ia-local
|
||||
colima stop --profile k8s-local
|
||||
```
|
||||
|
||||
### 🎯 Key Skaffold Benefits
|
||||
### Restart Sequence
|
||||
```bash
|
||||
# Post-restart startup
|
||||
colima start --cpu 4 --memory 8 --disk 100 --runtime docker --profile k8s-local
|
||||
kind create cluster --config kind-config.yaml
|
||||
skaffold dev --profile=dev
|
||||
```
|
||||
|
||||
1. **🔄 Automated builds**: No manual Docker image building
|
||||
2. **👀 File watching**: Instant rebuilds on code changes
|
||||
3. **📊 Log streaming**: All service logs in one place
|
||||
4. **🔗 Port forwarding**: Easy access to services during development
|
||||
5. **⚡ One command deployment**: `skaffold dev` does everything
|
||||
## Production Considerations
|
||||
|
||||
### 💡 Pro Tips
|
||||
For production deployment:
|
||||
|
||||
- Use `skaffold dev --profile=dev` for daily development
|
||||
- Code changes trigger automatic rebuilds and deployments
|
||||
- Logs are automatically streamed to your terminal
|
||||
- Press `Ctrl+C` to stop and clean up everything
|
||||
- **Security**: Implement RBAC, network policies, pod security standards
|
||||
- **Monitoring**: Deploy Prometheus, Grafana, and alerting
|
||||
- **Backup**: Database backup strategies
|
||||
- **High Availability**: Multi-replica deployments with anti-affinity
|
||||
- **External Secrets**: Use managed secret services
|
||||
- **TLS**: Production Let's Encrypt certificates
|
||||
- **CI/CD**: Automated deployment pipelines
|
||||
|
||||
## 🎉 Summary: What You Get
|
||||
## Next Steps
|
||||
|
||||
### 🚀 NEW: Permanent Localhost Access (No Port Forwarding!)
|
||||
- ✅ **Direct localhost access** at http://localhost and https://localhost
|
||||
- ✅ **Standard web ports** 80 and 443 work directly
|
||||
- ✅ **No /etc/hosts modifications** required for basic access
|
||||
- ✅ **No port forwarding commands** needed during development
|
||||
- ✅ **Bookmark-friendly URLs** like any standard web application
|
||||
- ✅ **Kind cluster configuration** with permanent port mapping
|
||||
|
||||
### Development Environment
|
||||
- ✅ **One-command deployment** with `skaffold dev --profile=dev`
|
||||
- ✅ **Hot-reload development** with automatic rebuilds
|
||||
- ✅ **Complete observability** with streaming logs and metrics
|
||||
- ✅ **Easy cleanup** with `skaffold delete` or cleanup scripts
|
||||
- ✅ **Database corruption protection** with proper PVC management
|
||||
|
||||
### FREE HTTPS with Let's Encrypt (Optional)
|
||||
- ✅ **Automated certificate management** with cert-manager
|
||||
- ✅ **Local development certificates** for offline work
|
||||
- ✅ **Production-ready** Let's Encrypt integration
|
||||
- ✅ **Auto-renewal** of certificates before expiration
|
||||
- ✅ **Browser-trusted certificates** with CA import
|
||||
|
||||
### Security Features
|
||||
- ✅ **TLS 1.3 encryption** for all traffic (when HTTPS is configured)
|
||||
- ✅ **HTTPS redirects** from HTTP (configurable)
|
||||
- ✅ **Secure headers** via NGINX Ingress
|
||||
- ✅ **Certificate transparency** compliance
|
||||
|
||||
### Access URLs - Choose Your Style!
|
||||
|
||||
**🌟 Primary Access (New Permanent Solution):**
|
||||
- **Frontend:** http://localhost or https://localhost
|
||||
- **API Gateway:** http://localhost/api or https://localhost/api
|
||||
|
||||
**🏷️ Named Host Access (Optional with /etc/hosts):**
|
||||
- **Frontend:** http://bakery-ia.local or https://bakery-ia.local
|
||||
- **API:** http://api.bakery-ia.local or https://api.bakery-ia.local
|
||||
- **Monitoring:** http://monitoring.bakery-ia.local or https://monitoring.bakery-ia.local
|
||||
|
||||
**🔧 Direct Service Access (Backup):**
|
||||
- **Frontend Direct:** http://localhost:3000
|
||||
- **Gateway Direct:** http://localhost:8000
|
||||
|
||||
This setup provides production-like development experience with the convenience of standard localhost URLs! 🚀
|
||||
|
||||
|
||||
Pre-Restart Shutdown Sequence:
|
||||
|
||||
1. Stop Skaffold:
|
||||
# If running interactively: Ctrl+C
|
||||
# If running in background:
|
||||
pkill -f skaffold
|
||||
|
||||
2. Delete Kind cluster:
|
||||
kind delete cluster --name bakery-ia-local
|
||||
|
||||
3. Stop Colima:
|
||||
colima stop
|
||||
|
||||
Post-Restart Startup Sequence:
|
||||
|
||||
1. Start Colima:
|
||||
colima start
|
||||
|
||||
2. Create Kind cluster:
|
||||
kind create cluster --config kind-config.yaml --name bakery-ia-local
|
||||
|
||||
3. Start Skaffold with dev profile:
|
||||
skaffold dev -p dev
|
||||
|
||||
What Skaffold Will Do:
|
||||
|
||||
- Check existing Docker images (tagged as :dev)
|
||||
- Skip rebuilds if source code unchanged
|
||||
- Load images to new Kind cluster
|
||||
- Deploy using infrastructure/kubernetes/overlays/dev
|
||||
- Watch for changes and hot-reload
|
||||
|
||||
The -p dev profile ensures consistent tagging and deployment configuration
|
||||
as defined in your skaffold.yaml profiles section.
|
||||
1. Add comprehensive monitoring and logging
|
||||
2. Implement automated testing
|
||||
3. Set up CI/CD pipelines
|
||||
4. Add health checks and metrics endpoints
|
||||
5. Implement proper backup strategies
|
||||
Reference in New Issue
Block a user