diff --git a/PRODUCTION_DEPLOYMENT_GUIDE.md b/PRODUCTION_DEPLOYMENT_GUIDE.md new file mode 100644 index 00000000..d7550afe --- /dev/null +++ b/PRODUCTION_DEPLOYMENT_GUIDE.md @@ -0,0 +1,1373 @@ +# Bakery-IA Production Deployment Guide + +**Complete guide for deploying Bakery-IA to production on a MicroK8s cluster** + +| **Version** | 4.0 | +|-------------|-----| +| **Last Updated** | 2026-01-21 | +| **Target Environment** | VPS with MicroK8s (Ubuntu 22.04 LTS) | +| **Estimated Deployment Time** | 3-5 hours (first-time deployment) | +| **Monthly Cost** | ~€41-81 (10-tenant pilot) | + +--- + +## Table of Contents + +1. [Quick Start Overview](#quick-start-overview) +2. [Prerequisites](#prerequisites) +3. [Phase 0: Transfer Infrastructure Code to Server](#phase-0-transfer-infrastructure-code-to-server) +4. [Phase 1: VPS Setup & MicroK8s Installation](#phase-1-vps-setup--microk8s-installation) +5. [Phase 2: Domain & DNS Configuration](#phase-2-domain--dns-configuration) +6. [Phase 3: Deploy Foundation Layer](#phase-3-deploy-foundation-layer) +7. [Phase 4: Deploy CI/CD Infrastructure](#phase-4-deploy-cicd-infrastructure) +8. [Phase 5: Pre-Pull and Push Base Images to Gitea Registry](#phase-5-pre-pull-and-push-base-images-to-gitea-registry) + - [Step 5.1: Pre-Pull Base Images](#step-51-pre-pull-base-images-and-push-to-registry) + - [Step 5.2: Verify Images in Registry](#step-52-verify-images-in-gitea-registry) + - [Step 5.3: Troubleshooting](#step-53-troubleshooting-image-issues) + - [Step 5.4: Verify CI/CD Access](#step-54-verify-cicd-pipeline-can-access-images) + - [Step 5.5: Build Service Images](#step-55-build-and-push-service-images-first-time-deployment-only) + - [Step 5.6: Verify Service Images](#step-56-verify-all-service-images-are-available) +9. [Phase 6: Deploy Application Services](#phase-6-deploy-application-services) +10. [Phase 7: Deploy Optional Services](#phase-7-deploy-optional-services) +11. [Phase 8: Verification & Validation](#phase-8-verification--validation) +12. [Post-Deployment Operations](#post-deployment-operations) +13. [Troubleshooting Guide](#troubleshooting-guide) +14. [Reference & Resources](#reference--resources) + +--- + +## Quick Start Overview + +### What You're Deploying + +A complete multi-tenant SaaS platform consisting of: + +| Component | Details | +|-----------|---------| +| **Microservices** | 18 Python/FastAPI services | +| **Databases** | 18 PostgreSQL instances with TLS | +| **Cache** | Redis with TLS | +| **Message Broker** | RabbitMQ | +| **Object Storage** | MinIO (S3-compatible) | +| **Email** | Mailu (self-hosted) with Mailgun relay | +| **Monitoring** | SigNoz (unified observability) | +| **CI/CD** | Gitea + Tekton + Flux CD | +| **Security** | TLS everywhere, RBAC, Network Policies | + +### Infrastructure Architecture + +``` +┌─────────────────────────────────────────────────────────────────────────────┐ +│ LAYER 6: APPLICATION │ +│ Frontend │ Gateway │ 18 Microservices │ CronJobs & Workers │ +├─────────────────────────────────────────────────────────────────────────────┤ +│ LAYER 5: MONITORING │ +│ SigNoz (Unified Observability) │ AlertManager │ OTel Collector │ +├─────────────────────────────────────────────────────────────────────────────┤ +│ LAYER 4: PLATFORM SERVICES (Optional) │ +│ Mailu (Email) │ Nominatim (Geocoding) │ CI/CD (Tekton, Flux, Gitea) │ +├─────────────────────────────────────────────────────────────────────────────┤ +│ LAYER 3: DATA & STORAGE │ +│ PostgreSQL (18 DBs) │ Redis │ RabbitMQ │ MinIO │ +├─────────────────────────────────────────────────────────────────────────────┤ +│ LAYER 2: NETWORK & SECURITY │ +│ Unbound DNS │ CoreDNS │ Ingress Controller │ Cert-Manager │ TLS │ +├─────────────────────────────────────────────────────────────────────────────┤ +│ LAYER 1: FOUNDATION │ +│ Namespaces │ Storage Classes │ RBAC │ ConfigMaps │ Secrets │ +├─────────────────────────────────────────────────────────────────────────────┤ +│ LAYER 0: KUBERNETES CLUSTER │ +│ MicroK8s (Production) │ Kind (Local Dev) │ EKS (AWS Alternative) │ +└─────────────────────────────────────────────────────────────────────────────┘ +``` + +### Deployment Order (Critical) + +Components **must** be deployed in this order due to dependencies: + +``` +Phase 0: Transfer code to server (bootstrap) + ↓ +Phase 1: MicroK8s + Addons + ↓ +Phase 2: DNS + Domain configuration + ↓ +Phase 3: Foundation (Namespaces, Cert-Manager, TLS) + ↓ +Phase 4: CI/CD (Gitea → Tekton → Flux) + ↓ +Phase 5: Base & Service Images (Pre-pull base images, build service images) + ↓ +Phase 6: Application Services (21 microservices + Gateway + Data Layer) + ↓ +Phase 7: Optional (Mailu, SigNoz, Nominatim) + ↓ +Phase 8: Verification & Validation +``` + +> **Note on First Deployment:** For the first deployment, you must manually build and push service images (Phase 5, Step 5.5) before applying the production kustomization. After the first deployment, the CI/CD pipeline will automatically build and push images on subsequent commits. + +### Cost Breakdown + +| Service | Provider | Monthly Cost | +|---------|----------|-------------| +| VPS (20GB RAM, 8 vCPU, 200GB SSD) | clouding.io | €40-80 | +| Domain | Namecheap/Cloudflare | ~€1.25 (€15/year) | +| Email Relay | Mailgun (free tier) | €0 | +| SSL Certificates | Let's Encrypt | €0 | +| DNS | Cloudflare | €0 | +| **Total** | | **€41-81/month** | + +--- + +## Prerequisites + +### System Requirements + +| Requirement | Specification | +|-------------|---------------| +| **OS** | Ubuntu 22.04 LTS | +| **RAM** | Minimum 16GB (20GB recommended) | +| **CPU** | 8 vCPU cores | +| **Storage** | 200GB NVMe SSD | +| **Network** | Static public IP, 1 Gbps | + +### Required Accounts + +- [ ] **VPS Provider** (clouding.io, Hetzner, DigitalOcean, etc.) +- [ ] **Domain Registrar** (Namecheap, Cloudflare, etc.) +- [ ] **Cloudflare Account** (recommended for DNS) +- [ ] **Mailgun Account** (for email relay, optional) +- [ ] **Stripe Account** (for payments) + +### Local Machine Requirements + +```bash +# Verify these tools are installed: +kubectl version --client # Kubernetes CLI +docker --version # Container runtime +git --version # Version control +ssh -V # SSH client +helm version # Helm package manager +openssl version # TLS utilities + +# Install if missing (macOS): +brew install kubectl docker git helm openssl + +# Install if missing (Ubuntu): +sudo apt install -y docker.io git openssl +sudo snap install kubectl --classic +sudo snap install helm --classic +``` + +### SSH Configuration (Recommended) + +Set up SSH config for easier access: + +```bash +# Create/edit ~/.ssh/config +cat >> ~/.ssh/config << 'EOF' +Host bakery-vps + HostName 200.234.233.87 + User root + IdentityFile ~/.ssh/bakewise.pem + IdentitiesOnly yes +EOF + +# Set proper permissions on key +chmod 600 ~/.ssh/bakewise.pem + +# Test connection +ssh bakery-vps +``` + +--- + +## Phase 0: Transfer Infrastructure Code to Server + +**Problem:** You need the infrastructure code on the server to deploy Gitea, but Gitea is your target repository. + +### Option 1: Direct Transfer with rsync (Recommended) + +This is the **bootstrap approach** - transfer code directly, then push to Gitea once it's running. + +```bash +# From your LOCAL machine - transfer entire repository +rsync -avz --progress \ + --exclude='.git' \ + --exclude='node_modules' \ + --exclude='__pycache__' \ + --exclude='.venv' \ + --exclude='*.pyc' \ + /Users/urtzialfaro/Documents/bakery-ia/ \ + bakery-vps:/root/bakery-ia/ + +# Verify transfer +ssh bakery-vps "ls -la /root/bakery-ia/infrastructure/" +``` + +### Option 2: SCP Tarball Transfer + +```bash +# Create a tarball locally (excludes unnecessary files) +cd /Users/urtzialfaro/Documents/bakery-ia +tar -czvf /tmp/bakery-ia-infra.tar.gz \ + --exclude='.git' \ + --exclude='node_modules' \ + --exclude='__pycache__' \ + --exclude='.venv' \ + infrastructure/ \ + PRODUCTION_DEPLOYMENT_GUIDE.md \ + docs/ + +# Transfer to server +scp /tmp/bakery-ia-infra.tar.gz bakery-vps:/root/ + +# On server - extract +ssh bakery-vps "cd /root && tar -xzvf bakery-ia-infra.tar.gz" +``` + +### Option 3: Temporary GitHub/GitLab (If Needed) + +Use if rsync/scp are not available: + +1. Push to a **temporary private** GitHub/GitLab repo +2. Clone on the server +3. After Gitea is running, migrate the repo to Gitea +4. Delete the temporary remote repo + +### After Transfer - Push to Gitea (Post Phase 5) + +Once Gitea is deployed (Phase 5), push the full repo: + +```bash +# On the SERVER after Gitea is running +cd /root/bakery-ia +git init +git add . +git commit -m "Initial commit - production deployment" +git remote add origin https://gitea.bakewise.ai/bakery-admin/bakery-ia.git +git push -u origin main +``` + +--- + +## Phase 1: VPS Setup & MicroK8s Installation + +### Step 1.1: Initial Server Setup + +```bash +# SSH into your VPS +ssh bakery-vps + +# Update system +apt update && apt upgrade -y + +# Set hostname +hostnamectl set-hostname bakery-ia-prod + +# Install essential tools +apt install -y curl wget git jq openssl +``` + +### Step 1.2: Install MicroK8s + +```bash +# Install MicroK8s (stable channel) +snap install microk8s --classic --channel=1.28/stable + +# Add user to microk8s group +usermod -a -G microk8s $USER +chown -f -R $USER ~/.kube +newgrp microk8s + +# Wait for MicroK8s to be ready +microk8s status --wait-ready +``` + +### Step 1.3: Enable Required Addons + +```bash +# Enable core addons (in order) +microk8s enable dns # DNS resolution +microk8s enable hostpath-storage # Storage provisioner +microk8s enable ingress # NGINX ingress (class: "public") +microk8s enable cert-manager # Let's Encrypt certificates +microk8s enable metrics-server # HPA autoscaling +microk8s enable rbac # Role-based access control + +# Optional but recommended +microk8s enable prometheus # Metrics collection + +# Setup kubectl alias +echo "alias kubectl='microk8s kubectl'" >> ~/.bashrc +source ~/.bashrc + +# Verify installation +kubectl get nodes # Should show: Ready +kubectl get storageclass # Should show: microk8s-hostpath (default) +kubectl get pods -A # All pods should be Running +``` + +### Step 1.4: Configure kubectl Access + +```bash +# Create kubectl config +mkdir -p ~/.kube +microk8s config > ~/.kube/config +chmod 600 ~/.kube/config + +# Test cluster connectivity +kubectl cluster-info +kubectl top nodes +``` + +### Step 1.5: Install Helm + +```bash +# Install Helm 3 +curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash + +# Verify installation +helm version +``` + +### Step 1.6: Configure Firewall (Optional) + +> **Skip this step if:** Your VPS provider already has firewall rules configured in their dashboard with ports 22, 80, 443 open. Most providers (clouding.io, Hetzner, etc.) manage this at the infrastructure level. + +**Required ports for Bakery-IA:** + +| Port | Protocol | Purpose | +|------|----------|---------| +| 22 | TCP | SSH access | +| 80 | TCP | HTTP (Let's Encrypt ACME challenges) | +| 443 | TCP | HTTPS (application access) | +| 25, 465, 587 | TCP | SMTP/SMTPS (if using Mailu) | +| 143, 993 | TCP | IMAP/IMAPS (if using Mailu) | + +**Only if using UFW on the server:** + +```bash +# Allow necessary ports +ufw allow 22/tcp # SSH +ufw allow 80/tcp # HTTP (required for Let's Encrypt) +ufw allow 443/tcp # HTTPS + +# Enable firewall (if not already enabled) +ufw enable + +# Verify +ufw status verbose +``` + +--- + +## Phase 2: Domain & DNS Configuration + +### Step 2.1: DNS Records Configuration + +Add these DNS records pointing to your VPS IP (`200.234.233.87`): + +| Type | Name | Value | TTL | +|------|------|-------|-----| +| A | @ | 200.234.233.87 | Auto | +| A | www | 200.234.233.87 | Auto | +| A | mail | 200.234.233.87 | Auto | +| A | monitoring | 200.234.233.87 | Auto | +| A | gitea | 200.234.233.87 | Auto | +| A | registry | 200.234.233.87 | Auto | +| A | api | 200.234.233.87 | Auto | +| MX | @ | mail.bakewise.ai | 10 | +| TXT | @ | v=spf1 mx a -all | Auto | +| TXT | _dmarc | v=DMARC1; p=reject; rua=mailto:admin@bakewise.ai | Auto | + +### Step 2.2: Verify DNS Propagation + +```bash +# Test DNS resolution (wait 5-10 minutes after changes) +dig bakewise.ai +short +dig www.bakewise.ai +short +dig mail.bakewise.ai +short +dig gitea.bakewise.ai +short + +# Check MX records +dig bakewise.ai MX +short + +# Use online tools for comprehensive check: +# https://dnschecker.org/ +# https://mxtoolbox.com/ +``` + +### Step 2.3: Cloudflare Configuration (If Using) + +If using Cloudflare for DNS: + +1. **SSL/TLS Mode:** Set to "Full (strict)" +2. **Proxy Status:** Set to "DNS only" (orange cloud OFF) for direct IP access +3. **Edge Certificates:** Let cert-manager handle certificates (not Cloudflare) + +--- + +## Phase 3: Deploy Foundation Layer + +### Step 3.1: Create Namespaces + +```bash +# Apply namespace definitions using kustomize (-k flag) +kubectl apply -k infrastructure/namespaces/ + +# Verify +kubectl get namespaces +# Expected: bakery-ia, flux-system, tekton-pipelines + +# Alternative: Apply individual namespace files directly +# kubectl apply -f infrastructure/namespaces/bakery-ia.yaml +# kubectl apply -f infrastructure/namespaces/flux-system.yaml +# kubectl apply -f infrastructure/namespaces/tekton-pipelines.yaml +``` + +### Step 3.2: Deploy Cert-Manager ClusterIssuers + +```bash +# Apply cert-manager configuration +kubectl apply -k infrastructure/platform/cert-manager/ + +# Verify ClusterIssuers are ready +kubectl get clusterissuer +kubectl describe clusterissuer letsencrypt-production + +# Expected output: +# NAME READY AGE +# letsencrypt-production True 1m +# letsencrypt-staging True 1m +``` + +> **Note:** Common configs (secrets, configmaps) and TLS secrets are automatically included when you apply the prod kustomization in Phase 6. No manual application needed. + +--- + +## Phase 4: Deploy CI/CD Infrastructure + +### Step 4.1: Deploy Gitea (Git Server + Container Registry) + +```bash +# Add Gitea Helm repository +helm repo add gitea https://dl.gitea.io/charts +helm repo update + +# Generate and export admin password (REQUIRED for --production flag) +export GITEA_ADMIN_PASSWORD=$(openssl rand -base64 32) +echo "Gitea Admin Password: $GITEA_ADMIN_PASSWORD" +echo "⚠️ SAVE THIS PASSWORD SECURELY!" + +# Run setup script - creates secrets and init job automatically +# The script will: +# 1. Create gitea namespace (if not exists) +# 2. Create gitea-admin-secret in gitea namespace +# 3. Create gitea-registry-secret in bakery-ia namespace +# 4. Apply gitea-init-job.yaml (creates bakery-ia repo) +cd /root/bakery-ia/infrastructure/cicd/gitea +chmod +x setup-admin-secret.sh +./setup-admin-secret.sh --production +cd /root/bakery-ia + +# Install Gitea with production values +helm upgrade --install gitea gitea/gitea -n gitea \ + -f infrastructure/cicd/gitea/values.yaml \ + -f infrastructure/cicd/gitea/values-prod.yaml \ + --timeout 10m \ + --wait + +# Wait for Gitea to be ready +kubectl wait --for=condition=ready pod -n gitea -l app.kubernetes.io/name=gitea --timeout=300s + +# Verify +kubectl get pods -n gitea + +# Check init job status (creates bakery-ia repository) +kubectl logs -n gitea -l app.kubernetes.io/component=init --tail=50 +``` + +### Step 4.2: Push Repository to Gitea + +```bash +cd /root/bakery-ia + +# Fix Git ownership warning (common when using rsync as different user) +git config --global --add safe.directory /root/bakery-ia + +# Configure git user (required for commits) +git config --global user.email "admin@bakewise.ai" +git config --global user.name "Bakery Admin" + +# Initialize repository +git init + +# Rename branch to main (git init may create 'master' by default) +git branch -m main + +# Add all files and commit +git add . +git commit -m "Initial commit - production deployment" + +# Add remote and push (you'll need the admin password from Step 5.1) +git remote add origin https://gitea.bakewise.ai/bakery-admin/bakery-ia.git + +# Force push to overwrite init job's auto-generated content +# This is safe for initial deployment - your local code is the source of truth +git push -u origin main --force +``` + +### Step 4.3: Verify Registry Secret (Already Created) + +> **Note:** The registry secret `gitea-registry-secret` was already created by `setup-admin-secret.sh` in Step 4.1. + +```bash +# Verify the registry secret exists +kubectl get secret gitea-registry-secret -n bakery-ia + +# Expected output: +# NAME TYPE DATA AGE +# gitea-registry-secret kubernetes.io/dockerconfigjson 1 Xm +``` + +### Step 4.4: Deploy Tekton (CI Pipelines) + +```bash +# Step 1: Install Tekton Pipelines (the controller) +kubectl apply --filename https://storage.googleapis.com/tekton-releases/pipeline/latest/release.yaml + +# Wait for Tekton Pipelines to be ready +kubectl wait --for=condition=ready pod -l app.kubernetes.io/part-of=tekton-pipelines -n tekton-pipelines --timeout=300s + +# Step 2: Install Tekton Triggers (for webhooks) +kubectl apply --filename https://storage.googleapis.com/tekton-releases/triggers/latest/release.yaml +kubectl apply --filename https://storage.googleapis.com/tekton-releases/triggers/latest/interceptors.yaml + +# Wait for Tekton Triggers to be ready +kubectl wait --for=condition=ready pod -l app.kubernetes.io/part-of=tekton-triggers -n tekton-pipelines --timeout=300s + +# Verify Tekton is installed +kubectl get pods -n tekton-pipelines + +# Step 3: Get Gitea password and generate webhook token +export GITEA_ADMIN_PASSWORD=$(kubectl get secret gitea-admin-secret -n gitea -o jsonpath='{.data.password}' | base64 -d) +export TEKTON_WEBHOOK_TOKEN=$(openssl rand -hex 32) +echo "Tekton Webhook Token: $TEKTON_WEBHOOK_TOKEN" +echo "⚠️ SAVE THIS TOKEN - needed to configure Gitea webhook!" + +# Step 4: Deploy Bakery-IA CI/CD pipelines and tasks +helm upgrade --install tekton-cicd infrastructure/cicd/tekton-helm \ + -n tekton-pipelines \ + -f infrastructure/cicd/tekton-helm/values.yaml \ + -f infrastructure/cicd/tekton-helm/values-prod.yaml \ + --set secrets.webhook.token=$TEKTON_WEBHOOK_TOKEN \ + --set secrets.registry.password=$GITEA_ADMIN_PASSWORD \ + --set secrets.git.password=$GITEA_ADMIN_PASSWORD \ + --timeout 5m + +# Verify all components +kubectl get pods -n tekton-pipelines +kubectl get tasks -n tekton-pipelines +kubectl get pipelines -n tekton-pipelines +kubectl get eventlisteners -n tekton-pipelines +``` + +### Step 4.5: Deploy Flux CD (GitOps) + +```bash +# Step 1: Install Flux CLI (required for bootstrap) +curl -s https://fluxcd.io/install.sh | sudo bash + +# Verify Flux CLI installation +flux --version + +# Step 2: Install Flux components (controllers and CRDs) +flux install --namespace=flux-system + +# Wait for Flux controllers to be ready +kubectl wait --for=condition=ready pod -l app.kubernetes.io/part-of=flux -n flux-system --timeout=300s + +# Verify Flux controllers are running +kubectl get pods -n flux-system + +# Step 3: Create Git credentials secret for Flux to access Gitea +export GITEA_ADMIN_PASSWORD=$(kubectl get secret gitea-admin-secret -n gitea -o jsonpath='{.data.password}' | base64 -d) + +kubectl create secret generic gitea-credentials \ + --namespace=flux-system \ + --from-literal=username=bakery-admin \ + --from-literal=password=$GITEA_ADMIN_PASSWORD + +# Step 4: Deploy Bakery-IA Flux configuration (GitRepository + Kustomization) +helm upgrade --install flux-cd infrastructure/cicd/flux \ + -n flux-system \ + --timeout 5m + +# Verify Flux resources +kubectl get gitrepository -n flux-system +kubectl get kustomization -n flux-system + +# Check Flux sync status +flux get sources git -n flux-system +flux get kustomizations -n flux-system +``` + + +## Phase 5: Pre-Pull and Push Base Images to Gitea Registry + +> **Critical Step:** This phase must be completed after Gitea is configured (Phase 5) and before deploying application services (Phase 6). It ensures all required base images are available in the Gitea registry. + +### Overview + +This phase involves two main steps: +1. **Step 5.6.1-5.6.4:** Pre-pull base images from Docker Hub and push them to Gitea registry +2. **Step 5.6.5:** Build and push all service images (first-time deployment only) + +### Base Images Required + +The following base images must be available in the Gitea registry: + +| Category | Image | Used By | +|----------|-------|---------| +| **Python Runtime** | `python:3.11-slim` | All microservices, gateway | +| **Frontend Build** | `node:18-alpine` | Frontend build stage | +| **Frontend Runtime** | `nginx:1.25-alpine` | Frontend production server | +| **Database** | `postgres:17-alpine` | All PostgreSQL instances | +| **Cache** | `redis:7.4-alpine` | Redis cache | +| **Message Broker** | `rabbitmq:4.1-management-alpine` | RabbitMQ | +| **Storage** | `minio/minio:RELEASE.2024-11-07T00-52-20Z` | MinIO object storage | +| **CI/CD** | `gcr.io/kaniko-project/executor:v1.23.0` | Tekton image builds | + +--- + +### Step 5.1: Pre-Pull Base Images and Push to Registry + +```bash +# Navigate to the scripts directory +cd /root/bakery-ia/scripts + +# Make the script executable +chmod +x prepull-base-images-for-prod.sh + +# Run the prepull script in production mode WITH push enabled +# IMPORTANT: Use --push-images flag to push to Gitea registry +./prepull-base-images-for-prod.sh -e prod --push-images + +# The script will: +# 1. Authenticate with Docker Hub (uses embedded credentials or env vars) +# 2. Pull all required base images from Docker Hub/GHCR +# 3. Tag them for Gitea registry (bakery-admin namespace) +# 4. Push them to the Gitea container registry +# 5. Report success/failure for each image +``` + +**Alternative: Specify Custom Registry URL** + +```bash +# If auto-detection fails or you need a specific registry URL: +./prepull-base-images-for-prod.sh -e prod --push-images -r registry.bakewise.ai +``` + +**Handle Docker Hub Rate Limits** + +```bash +# If you hit Docker Hub rate limits, use your own credentials: +export DOCKER_HUB_USERNAME=your_username +export DOCKER_HUB_PASSWORD=your_password_or_token +./prepull-base-images-for-prod.sh -e prod --push-images +``` + +--- + +### Step 5.2: Verify Images in Gitea Registry + +```bash +# Get Gitea admin password +export GITEA_ADMIN_PASSWORD=$(kubectl get secret gitea-admin-secret -n gitea -o jsonpath='{.data.password}' | base64 -d) + +# Login to Gitea registry +# Note: Use registry.bakewise.ai for external access +docker login registry.bakewise.ai -u bakery-admin -p $GITEA_ADMIN_PASSWORD + +# List all images in the registry +curl -s -u bakery-admin:$GITEA_ADMIN_PASSWORD https://registry.bakewise.ai/v2/_catalog | jq + +# Verify specific critical images exist +echo "Checking Python base image..." +curl -s -u bakery-admin:$GITEA_ADMIN_PASSWORD https://registry.bakewise.ai/v2/bakery-admin/python/tags/list | jq + +echo "Checking Node.js base image..." +curl -s -u bakery-admin:$GITEA_ADMIN_PASSWORD https://registry.bakewise.ai/v2/bakery-admin/node/tags/list | jq + +echo "Checking Nginx base image..." +curl -s -u bakery-admin:$GITEA_ADMIN_PASSWORD https://registry.bakewise.ai/v2/bakery-admin/nginx/tags/list | jq +``` + +**Alternative: Verify via Gitea Web Interface** + +1. Visit `https://gitea.bakewise.ai` +2. Login with username: `bakery-admin`, password: (from secret) +3. Navigate to **Packages** > **Container** +4. Verify images are listed under the `bakery-admin` namespace +5. Confirm tags match expected versions (`3.11-slim`, `18-alpine`, `1.25-alpine`, etc.) + +--- + +### Step 5.3: Troubleshooting Image Issues + +**Registry Not Accessible** + +```bash +# Check Gitea pods are running +kubectl get pods -n gitea + +# Check Gitea service +kubectl get svc -n gitea + +# Check ingress for registry +kubectl get ingress -n gitea + +# View Gitea logs for registry errors +kubectl logs -n gitea -l app.kubernetes.io/name=gitea --tail=100 +``` + +**Images Failed to Push** + +```bash +# Verify Docker can reach the registry +docker info | grep -i registry + +# Test registry connectivity +curl -v https://registry.bakewise.ai/v2/ + +# Check for TLS certificate issues +openssl s_client -connect registry.bakewise.ai:443 -servername registry.bakewise.ai +``` + +**Re-run Failed Images Only** + +```bash +# Manually pull and push a specific image +docker pull python:3.11-slim +docker tag python:3.11-slim registry.bakewise.ai/bakery-admin/python:3.11-slim +docker push registry.bakewise.ai/bakery-admin/python:3.11-slim +``` + +--- + +### Step 5.4: Verify CI/CD Pipeline Can Access Images + +```bash +# Verify gitea-registry-secret exists in bakery-ia namespace +kubectl get secret gitea-registry-secret -n bakery-ia + +# Check the secret contains correct registry URL +kubectl get secret gitea-registry-secret -n bakery-ia \ + -o jsonpath='{.data.\.dockerconfigjson}' | base64 -d | jq '.auths | keys[]' + +# Test that Kubernetes can pull images using the secret +# Create a test pod that uses the base image +cat < **If this test fails:** The CI/CD pipeline and application deployments will not be able to pull images. Check: +> 1. Registry URL in the secret matches your setup +> 2. Credentials are correct +> 3. Images were successfully pushed in Step 5.1 + +--- + +### Step 5.5: Build and Push Service Images (First-Time Deployment Only) + +> **Critical:** For the first deployment, service images don't exist in the registry yet. You must build and push them before applying the production kustomization in Phase 6. + +#### Option A: Use the Automated Build Script (Recommended) + +```bash +# Navigate to the repository root +cd /root/bakery-ia + +# Make the script executable +chmod +x scripts/build-all-services.sh + +# Run the build script +# This will build and push all 21 services to the Gitea registry +./scripts/build-all-services.sh +``` + +The script builds the following services: + +| Service | Image Name | Dockerfile | +|---------|------------|------------| +| Gateway | `gateway` | `gateway/Dockerfile` | +| Frontend | `dashboard` | `frontend/Dockerfile.kubernetes` | +| Auth | `auth-service` | `services/auth/Dockerfile` | +| Tenant | `tenant-service` | `services/tenant/Dockerfile` | +| Training | `training-service` | `services/training/Dockerfile` | +| Forecasting | `forecasting-service` | `services/forecasting/Dockerfile` | +| Sales | `sales-service` | `services/sales/Dockerfile` | +| Inventory | `inventory-service` | `services/inventory/Dockerfile` | +| Recipes | `recipes-service` | `services/recipes/Dockerfile` | +| Suppliers | `suppliers-service` | `services/suppliers/Dockerfile` | +| POS | `pos-service` | `services/pos/Dockerfile` | +| Orders | `orders-service` | `services/orders/Dockerfile` | +| Production | `production-service` | `services/production/Dockerfile` | +| Procurement | `procurement-service` | `services/procurement/Dockerfile` | +| Distribution | `distribution-service` | `services/distribution/Dockerfile` | +| External | `external-service` | `services/external/Dockerfile` | +| Notification | `notification-service` | `services/notification/Dockerfile` | +| Orchestrator | `orchestrator-service` | `services/orchestrator/Dockerfile` | +| Alert Processor | `alert-processor` | `services/alert_processor/Dockerfile` | +| AI Insights | `ai-insights-service` | `services/ai_insights/Dockerfile` | +| Demo Session | `demo-session-service` | `services/demo_session/Dockerfile` | + +#### Option B: Trigger CI/CD Pipeline + +If Tekton is properly configured, you can trigger the CI/CD pipeline instead: + +```bash +cd /root/bakery-ia + +# Create an empty commit to trigger the pipeline +git commit --allow-empty -m "Trigger initial CI/CD build" +git push origin main + +# Monitor pipeline execution +kubectl get pipelineruns -n tekton-pipelines --watch + +# Wait for all builds to complete (may take 20-30 minutes) +kubectl wait --for=condition=Succeeded pipelinerun --all -n tekton-pipelines --timeout=1800s +``` + +#### Option C: Build Individual Services Manually + +```bash +# Get credentials +export GITEA_ADMIN_PASSWORD=$(kubectl get secret gitea-admin-secret -n gitea -o jsonpath='{.data.password}' | base64 -d) +export REGISTRY="registry.bakewise.ai/bakery-admin" + +# Login to registry +docker login registry.bakewise.ai -u bakery-admin -p $GITEA_ADMIN_PASSWORD + +# Build and push a single service (example: auth-service) +docker build -t $REGISTRY/auth-service:latest \ + --build-arg BASE_REGISTRY=$REGISTRY \ + --build-arg PYTHON_IMAGE=python:3.11-slim \ + -f services/auth/Dockerfile . +docker push $REGISTRY/auth-service:latest + +# Build and push frontend +docker build -t $REGISTRY/dashboard:latest \ + -f frontend/Dockerfile.kubernetes frontend/ +docker push $REGISTRY/dashboard:latest +``` + +--- + +### Step 5.6: Verify All Service Images Are Available + +```bash +# Get Gitea admin password +export GITEA_ADMIN_PASSWORD=$(kubectl get secret gitea-admin-secret -n gitea -o jsonpath='{.data.password}' | base64 -d) + +# List all images in the registry +echo "=== Images in Gitea Registry ===" +curl -s -u bakery-admin:$GITEA_ADMIN_PASSWORD https://registry.bakewise.ai/v2/_catalog | jq -r '.repositories[]' | sort + +# Verify critical service images exist +for service in gateway dashboard auth-service tenant-service forecasting-service; do + echo -n "Checking $service... " + if curl -s -u bakery-admin:$GITEA_ADMIN_PASSWORD \ + "https://registry.bakewise.ai/v2/bakery-admin/$service/tags/list" | jq -e '.tags' > /dev/null 2>&1; then + echo "✅ OK" + else + echo "❌ MISSING" + fi +done +``` + +> **Ready for Phase 6:** Once all service images are verified in the registry, you can proceed to Phase 6: Deploy Application Services. + +## Phase 6: Deploy Application Services + +> **Prerequisite:** This phase assumes that all service images have been built and pushed to the Gitea registry (completed in Phase 5, Step 5.5). The production kustomization references these pre-built images. + +### Step 6.1: Apply Production Certificate + +```bash +# Apply the production TLS certificate +kubectl apply -f infrastructure/environments/prod/k8s-manifests/prod-certificate.yaml + +# Verify certificate is issued +kubectl get certificate -n bakery-ia +kubectl describe certificate bakery-ia-prod-tls-cert -n bakery-ia +``` + +### Step 6.2: Deploy Application with Kustomize + +```bash +# Apply the complete production configuration +kubectl apply -k infrastructure/environments/prod/k8s-manifests + +# Wait for all deployments to be ready (10-15 minutes) +kubectl wait --for=condition=available --timeout=900s deployment --all -n bakery-ia + +# Monitor deployment progress +kubectl get pods -n bakery-ia --watch +``` + +### Step 6.3: Verify Application Health + +```bash +# Check all pods are running +kubectl get pods -n bakery-ia + +# Check services +kubectl get svc -n bakery-ia + +# Check ingress +kubectl get ingress -n bakery-ia + +# Test gateway health +kubectl exec -n bakery-ia deployment/gateway -- curl -s http://localhost:8000/health +``` + +--- + +## Phase 7: Deploy Optional Services + +### Step 7.1: Deploy Unbound DNS (Required for Mailu) + +```bash +# Deploy Unbound DNS resolver +helm upgrade --install unbound infrastructure/platform/networking/dns/unbound-helm \ + -n bakery-ia \ + -f infrastructure/platform/networking/dns/unbound-helm/values.yaml \ + -f infrastructure/platform/networking/dns/unbound-helm/prod/values.yaml \ + --timeout 5m \ + --wait + +# Get Unbound service IP +UNBOUND_IP=$(kubectl get svc unbound-dns -n bakery-ia -o jsonpath='{.spec.clusterIP}') +echo "Unbound DNS IP: $UNBOUND_IP" +``` + +### Step 7.2: Configure CoreDNS for DNSSEC + +```bash +# Patch CoreDNS to forward to Unbound +kubectl patch configmap coredns -n kube-system --type merge -p "{ + \"data\": { + \"Corefile\": \".:53 {\\n errors\\n health {\\n lameduck 5s\\n }\\n ready\\n kubernetes cluster.local in-addr.arpa ip6.arpa {\\n pods insecure\\n fallthrough in-addr.arpa ip6.arpa\\n ttl 30\\n }\\n prometheus :9153\\n forward . $UNBOUND_IP {\\n max_concurrent 1000\\n }\\n cache 30\\n loop\\n reload\\n loadbalance\\n}\\n\" + } +}" + +# Restart CoreDNS +kubectl rollout restart deployment coredns -n kube-system +kubectl rollout status deployment coredns -n kube-system --timeout=60s +``` + +### Step 7.3: Deploy Mailu Email Server + +```bash +# Add Mailu Helm repository +helm repo add mailu https://mailu.github.io/helm-charts +helm repo update + +# Apply Mailu configuration secrets +# These are pre-configured with secure defaults +kubectl apply -f infrastructure/platform/mail/mailu-helm/configs/mailu-admin-credentials-secret.yaml -n bakery-ia +kubectl apply -f infrastructure/platform/mail/mailu-helm/configs/mailu-certificates-secret.yaml -n bakery-ia + +# Install Mailu with production configuration +# The Helm chart uses the pre-configured secrets for admin credentials and TLS certificates +helm upgrade --install mailu mailu/mailu \ + -n bakery-ia \ + -f infrastructure/platform/mail/mailu-helm/values.yaml \ + -f infrastructure/platform/mail/mailu-helm/prod/values.yaml \ + --timeout 10m + +# Wait for Mailu to be ready +kubectl wait --for=condition=available --timeout=600s deployment/mailu-front -n bakery-ia + +# Verify Mailu pods are running +kubectl get pods -n bakery-ia | grep mailu + +# Get the admin password from the pre-configured secret +MAILU_ADMIN_PASSWORD=$(kubectl get secret mailu-admin-credentials -n bakery-ia -o jsonpath='{.data.password}' | base64 -d) +echo "Mailu Admin Password: $MAILU_ADMIN_PASSWORD" +echo "⚠️ SAVE THIS PASSWORD SECURELY!" + +# Check Mailu initialization status +kubectl logs -n bakery-ia deployment/mailu-front --tail=10 +``` + +> **Important Notes about Mailu Deployment:** +> +> 1. **Pre-Configured Secrets:** Mailu uses pre-configured secrets for admin credentials and TLS certificates. These are defined in the configuration files. +> +> 2. **Password Management:** The admin password is stored in `mailu-admin-credentials-secret.yaml`. For production, you should update this with a secure password before deployment. +> +> 3. **TLS Certificates:** The self-signed certificates in `mailu-certificates-secret.yaml` are for initial setup. For production, replace these with proper certificates from cert-manager (see Step 7.3.1). +> +> 4. **Initialization Time:** Mailu may take 5-10 minutes to fully initialize. During this time, some pods may restart as the system configures itself. +> +> 5. **Accessing Mailu:** +> - Webmail: `https://mail.bakewise.ai/webmail` +> - Admin Interface: `https://mail.bakewise.ai/admin` +> - Username: `admin@bakewise.ai` +> - Password: (from `mailu-admin-credentials-secret.yaml`) +> +> 6. **Mailgun Relay:** The production configuration includes Mailgun SMTP relay. Configure your Mailgun credentials in `mailu-mailgun-credentials-secret.yaml` before deployment. + +### Step 7.3.1: Mailu Configuration Notes + +> **Important Information about Mailu Certificates:** +> +> 1. **Dual Certificate Architecture:** +> - **Internal Communication:** Uses self-signed certificates (`mailu-certificates-secret.yaml`) +> - **External Communication:** Uses Let's Encrypt certificates via NGINX Ingress (`bakery-ia-prod-tls-cert`) +> +> 2. **No Certificate Replacement Needed:** The self-signed certificates are only used for internal communication between Mailu services. External clients connect through the NGINX Ingress Controller which uses the publicly trusted Let's Encrypt certificates. +> +> 3. **Certificate Flow:** +> ``` +> External Client → NGINX Ingress (Let's Encrypt) → Internal Network → Mailu Services (Self-signed) +> ``` +> +> 4. **Security:** This architecture is secure because: +> - External connections use publicly trusted certificates +> - Internal connections are still encrypted (even if self-signed) +> - Ingress terminates TLS, reducing load on Mailu services +> +> 5. **Mailgun Relay Configuration:** For outbound email delivery, configure your Mailgun credentials: +> ```bash +> # Edit the Mailgun credentials secret +> nano infrastructure/platform/mail/mailu-helm/configs/mailu-mailgun-credentials-secret.yaml +> +> # Apply the secret +> kubectl apply -f infrastructure/platform/mail/mailu-helm/configs/mailu-mailgun-credentials-secret.yaml -n bakery-ia +> +> # Restart Mailu to pick up the new relay configuration +> kubectl rollout restart deployment -n bakery-ia -l app.kubernetes.io/instance=mailu +> ``` + +### Step 7.4: Deploy SigNoz Monitoring + +```bash +# Add SigNoz Helm repository +helm repo add signoz https://charts.signoz.io +helm repo update + +# Install SigNoz +helm install signoz signoz/signoz \ + -n bakery-ia \ + -f infrastructure/monitoring/signoz/signoz-values-prod.yaml \ + --set global.storageClass="microk8s-hostpath" \ + --set clickhouse.persistence.enabled=true \ + --set clickhouse.persistence.size=50Gi \ + --timeout 15m + +# Wait for SigNoz to be ready +kubectl wait --for=condition=available --timeout=600s deployment/signoz-frontend -n bakery-ia + +# Verify +kubectl get pods -n bakery-ia -l app.kubernetes.io/instance=signoz +``` + +--- + +## Phase 8: Verification & Validation + +### Step 8.1: Complete Verification Checklist + +```bash +# 1. Check all pods are running +kubectl get pods -n bakery-ia | grep -vE "Running|Completed" +# Should return NO results + +# 2. Check services +kubectl get svc -n bakery-ia + +# 3. Check ingress +kubectl get ingress -n bakery-ia + +# 4. Check certificates +kubectl get certificate -n bakery-ia +kubectl describe certificate bakery-ia-prod-tls-cert -n bakery-ia + +# 5. Check PVCs +kubectl get pvc -n bakery-ia +``` + +### Step 8.2: Test Application Endpoints + +```bash +# Test frontend (from external machine) +curl -I https://bakewise.ai +# Expected: HTTP/2 200 OK + +# Test API health +curl https://bakewise.ai/api/v1/health +# Expected: {"status": "healthy"} + +# Test monitoring +curl -I https://monitoring.bakewise.ai/signoz +# Expected: HTTP/2 200 OK +``` + +### Step 8.3: Test Database Connections + +```bash +# Test PostgreSQL SSL +kubectl exec -n bakery-ia deployment/auth-db -- sh -c \ + 'psql -U auth_user -d auth_db -c "SHOW ssl;"' +# Expected: on + +# Test Redis +kubectl exec -n bakery-ia deployment/redis -- redis-cli ping +# Expected: PONG +``` + +### Step 8.4: Production Validation Checklist + +- [ ] Application accessible at `https://bakewise.ai` +- [ ] Monitoring accessible at `https://monitoring.bakewise.ai` +- [ ] SSL certificates valid (check with browser) +- [ ] All services running and healthy +- [ ] Database connections working with TLS +- [ ] CI/CD pipeline operational +- [ ] Email service working (if deployed) +- [ ] Pilot coupon verified (check tenant-service logs) + +--- + +## Post-Deployment Operations + +### Configure Stripe Keys (Required Before Going Live) + +Before accepting payments, configure your Stripe credentials: + +```bash +# Edit ConfigMap for publishable key +nano infrastructure/environments/common/configs/configmap.yaml +# Add: VITE_STRIPE_PUBLISHABLE_KEY: "pk_live_XXXXXXXXXXXX" + +# Encode your secret keys +echo -n "sk_live_XXXXXXXXXX" | base64 # Your secret key +echo -n "whsec_XXXXXXXXXX" | base64 # Your webhook secret + +# Edit Secrets +nano infrastructure/environments/common/configs/secrets.yaml +# Add to payment-secrets section: +# STRIPE_SECRET_KEY: +# STRIPE_WEBHOOK_SECRET: + +# Apply the updated configuration +kubectl apply -k infrastructure/environments/prod/k8s-manifests + +# Restart services that use Stripe +kubectl rollout restart deployment/payment-service -n bakery-ia +``` + +### Backup Strategy + +```bash +# Create backup script +cat > ~/backup-databases.sh << 'EOF' +#!/bin/bash +BACKUP_DIR="/backups/$(date +%Y-%m-%d)" +mkdir -p $BACKUP_DIR + +# Backup all databases +for db in auth tenant training forecasting ai-insights sales inventory production procurement distribution recipes suppliers pos orders external notification alert-processor orchestrator demo-session; do + echo "Backing up ${db}-db..." + kubectl exec -n bakery-ia deployment/${db}-db -- \ + pg_dump -U ${db}_user -d ${db}_db > "$BACKUP_DIR/${db}.sql" +done + +# Compress +tar -czf "$BACKUP_DIR.tar.gz" "$BACKUP_DIR" +rm -rf "$BACKUP_DIR" + +# Keep only last 7 days +find /backups -name "*.tar.gz" -mtime +7 -delete + +echo "Backup completed: $BACKUP_DIR.tar.gz" +EOF + +chmod +x ~/backup-databases.sh + +# Setup daily cron job (2 AM) +(crontab -l 2>/dev/null; echo "0 2 * * * ~/backup-databases.sh") | crontab - +``` + +### Scaling Guidelines + +| Tenants | RAM | CPU | Storage | Monthly Cost | +|---------|-----|-----|---------|--------------| +| 10 | 20 GB | 8 cores | 200 GB | €40-80 | +| 25 | 32 GB | 12 cores | 300 GB | €80-120 | +| 50 | 48 GB | 16 cores | 500 GB | €150-200 | +| 100+ | Consider multi-node cluster | | | €300+ | + +### Regular Maintenance Tasks + +| Frequency | Task | +|-----------|------| +| Daily | Check logs and alerts | +| Weekly | Review resource utilization | +| Monthly | Update dependencies, security patches | +| Quarterly | Review backup procedures, disaster recovery | + +--- + +## Troubleshooting Guide + +### Common Issues + +#### Pods Stuck in Pending State + +```bash +# Check node resources +kubectl describe nodes + +# Check PVC status +kubectl get pvc -n bakery-ia + +# Check events +kubectl get events -n bakery-ia --sort-by='.lastTimestamp' +``` + +#### Certificate Not Issuing + +```bash +# Check cluster issuer +kubectl get clusterissuer + +# Check certificate status +kubectl describe certificate -n bakery-ia + +# Check cert-manager logs +kubectl logs -n cert-manager deployment/cert-manager + +# Verify ports 80/443 are open +curl -I http://bakewise.ai +``` + +#### Services Not Accessible + +```bash +# Check ingress +kubectl describe ingress -n bakery-ia + +# Check ingress controller logs +kubectl logs -n ingress deployment/nginx-ingress-microk8s-controller + +# Check endpoints +kubectl get endpoints -n bakery-ia +``` + +#### Database Connection Errors + +```bash +# Check database pod +kubectl get pods -n bakery-ia -l app.kubernetes.io/component=database + +# Check database logs +kubectl logs -n bakery-ia deployment/auth-db + +# Test connection from service +kubectl exec -n bakery-ia deployment/auth-service -- nc -zv auth-db 5432 +``` + +#### Out of Resources + +```bash +# Check node resources +kubectl top nodes + +# Check pod resource usage +kubectl top pods -n bakery-ia --sort-by=memory + +# Scale down non-critical services temporarily +kubectl scale deployment monitoring -n bakery-ia --replicas=0 +``` + +--- + +## Reference & Resources + +### Key File Locations + +| Configuration | File Path | +|---------------|-----------| +| ConfigMap | `infrastructure/environments/common/configs/configmap.yaml` | +| Secrets | `infrastructure/environments/common/configs/secrets.yaml` | +| Prod Kustomization | `infrastructure/environments/prod/k8s-manifests/kustomization.yaml` | +| Cert-Manager Issuer | `infrastructure/platform/cert-manager/cluster-issuer-production.yaml` | +| Ingress | `infrastructure/platform/networking/ingress/base/ingress.yaml` | +| Gitea Values | `infrastructure/cicd/gitea/values.yaml` | +| Mailu Values | `infrastructure/platform/mail/mailu-helm/values.yaml` | + +### Production URLs + +| Service | URL | +|---------|-----| +| Main Application | https://bakewise.ai | +| API | https://bakewise.ai/api/v1/... | +| Monitoring | https://monitoring.bakewise.ai | +| Gitea | https://gitea.bakewise.ai | +| Registry | https://registry.bakewise.ai | +| Webmail | https://mail.bakewise.ai/webmail | +| Mail Admin | https://mail.bakewise.ai/admin | + +### External Documentation + +- [MicroK8s Documentation](https://microk8s.io/docs) +- [Kubernetes Documentation](https://kubernetes.io/docs) +- [Let's Encrypt Documentation](https://letsencrypt.org/docs) +- [SigNoz Documentation](https://signoz.io/docs/) +- [OpenTelemetry Documentation](https://opentelemetry.io/docs/) + +### Support Resources + +- **Operations Guide:** [PRODUCTION_OPERATIONS_GUIDE.md](./docs/PRODUCTION_OPERATIONS_GUIDE.md) +- **Pilot Launch Guide:** [PILOT_LAUNCH_GUIDE.md](./docs/PILOT_LAUNCH_GUIDE.md) +- **Infrastructure README:** [infrastructure/README.md](./infrastructure/README.md) + +--- + +## Conclusion + +This guide provides a complete, step-by-step process for deploying Bakery-IA to production. Key highlights: + +1. **Bootstrap Approach:** Transfer code to server first, then push to Gitea +2. **Layered Deployment:** Components deployed in dependency order +3. **Production Ready:** TLS everywhere, monitoring, CI/CD, backups +4. **Scalable:** Designed for 10-100+ tenants with clear scaling path + +For questions or issues, refer to the troubleshooting guide or consult the support resources listed above. diff --git a/Tiltfile b/Tiltfile index da8a1ae3..04ee7958 100644 --- a/Tiltfile +++ b/Tiltfile @@ -818,12 +818,54 @@ local_resource( if [ "$CURRENT_FORWARD" != "$UNBOUND_IP" ]; then echo "Updating CoreDNS to forward to Unbound ($UNBOUND_IP)..." - # Patch CoreDNS ConfigMap - kubectl patch configmap coredns -n kube-system --type merge -p "{ - \"data\": { - \"Corefile\": \".:53 {\\n errors\\n health {\\n lameduck 5s\\n }\\n ready\\n kubernetes cluster.local in-addr.arpa ip6.arpa {\\n pods insecure\\n fallthrough in-addr.arpa ip6.arpa\\n ttl 30\\n }\\n prometheus :9153\\n forward . $UNBOUND_IP {\\n max_concurrent 1000\\n }\\n cache 30 {\\n disable success cluster.local\\n disable denial cluster.local\\n }\\n loop\\n reload\\n loadbalance\\n}\\n\" - } - }" + # Change to project root to ensure correct file paths + cd /Users/urtzialfaro/Documents/bakery-ia + + # Create a temporary Corefile with the forwarding configuration + TEMP_COREFILE=$(mktemp) + cat > "$TEMP_COREFILE" << EOF +.:53 { + errors + health { + lameduck 5s + } + ready + kubernetes cluster.local in-addr.arpa ip6.arpa { + pods insecure + fallthrough in-addr.arpa ip6.arpa + ttl 30 + } + prometheus :9153 + forward . $UNBOUND_IP { + max_concurrent 1000 + } + cache 30 { + disable success cluster.local + disable denial cluster.local + } + loop + reload + loadbalance +} +EOF + + # Create a complete new configmap YAML with the updated Corefile content + cat > /tmp/coredns_updated.yaml << EOF +apiVersion: v1 +kind: ConfigMap +metadata: + name: coredns + namespace: kube-system +data: + Corefile: | +$(sed 's/^/ /' "$TEMP_COREFILE") +EOF + + # Apply the updated configmap + kubectl apply -f /tmp/coredns_updated.yaml + + # Clean up the temporary file + rm "$TEMP_COREFILE" # Restart CoreDNS kubectl rollout restart deployment coredns -n kube-system @@ -887,6 +929,9 @@ local_resource( echo "Environment detected: $ENVIRONMENT" # Install Mailu with appropriate values + # Ensure we're in the project root directory for correct file paths + cd /Users/urtzialfaro/Documents/bakery-ia + if [ "$ENVIRONMENT" = "dev" ]; then helm upgrade --install mailu mailu/mailu \ -n bakery-ia \ @@ -912,6 +957,7 @@ local_resource( # ===================================================== echo "" echo "Applying Mailu ingress configuration..." + cd /Users/urtzialfaro/Documents/bakery-ia kubectl apply -f infrastructure/platform/mail/mailu-helm/mailu-ingress.yaml echo "Mailu ingress applied for mail.bakery-ia.dev" @@ -934,7 +980,7 @@ local_resource( echo " IMAP: mail.bakery-ia.dev:993 (SSL/TLS)" echo "" echo "To create admin user:" - echo " kubectl exec -it -n bakery-ia deployment/mailu-admin -- flask mailu admin admin bakery-ia.local 'YourPassword123!'" + echo " Admin user created automatically via initialAccount feature in Helm values" echo "" echo "To check pod status: kubectl get pods -n bakery-ia | grep mailu" ''', diff --git a/docs/PILOT_LAUNCH_GUIDE.md b/docs/PILOT_LAUNCH_GUIDE.md index d1cdb9a3..91fa4081 100644 --- a/docs/PILOT_LAUNCH_GUIDE.md +++ b/docs/PILOT_LAUNCH_GUIDE.md @@ -173,61 +173,127 @@ Components must be deployed in a specific order due to dependencies: ## ⚠️ CRITICAL: Pre-Deployment Configuration -**READ THIS FIRST:** The Kubernetes configuration requires updates for secure production deployment. +**READ THIS FIRST:** Review and complete these configuration steps before deploying to production. + +### Infrastructure Architecture (Updated) + +The infrastructure has been reorganized with the following structure: + +``` +infrastructure/ +├── environments/ # Environment-specific configs +│ ├── common/configs/ # Shared ConfigMaps and Secrets +│ │ ├── configmap.yaml # Application configuration +│ │ ├── secrets.yaml # All secrets (database, JWT, Redis, etc.) +│ │ └── kustomization.yaml +│ ├── dev/k8s-manifests/ # Development Kustomization +│ └── prod/k8s-manifests/ # Production Kustomization & patches +├── platform/ # Platform-level infrastructure +│ ├── cert-manager/ # TLS certificate issuers (Let's Encrypt) +│ ├── networking/ingress/ # NGINX ingress (base + overlays) +│ ├── storage/ # PostgreSQL, Redis, MinIO +│ ├── gateway/ # API Gateway service +│ └── mail/mailu-helm/ # Email server (Helm chart) +├── services/ # Application services +│ ├── databases/ # 19 PostgreSQL database instances +│ └── microservices/ # 19 microservices +├── cicd/ # CI/CD (deployed via Helm, NOT kustomize) +│ ├── gitea/ # Git server + container registry +│ ├── tekton-helm/ # CI pipelines +│ └── flux/ # GitOps deployment +└── monitoring/signoz/ # SigNoz observability (via Helm) +``` ### 🔴 Configuration Status -Your manifests need the following updates before deploying to production: +| Item | Status | File Location | +|------|--------|---------------| +| Production Secrets | ✅ Configured | `infrastructure/environments/common/configs/secrets.yaml` | +| Cert-Manager Email | ✅ Configured | `infrastructure/platform/cert-manager/cluster-issuer-production.yaml` | +| SigNoz Namespace | ✅ Uses bakery-ia | `infrastructure/environments/prod/k8s-manifests/kustomization.yaml` | +| imagePullSecrets | ✅ Auto-patched | Production kustomization adds `gitea-registry-secret` automatically | +| Image Tags | ⚠️ Update for releases | `infrastructure/environments/prod/k8s-manifests/kustomization.yaml` | +| Stripe Keys | ⚠️ Configure before launch | ConfigMap + Secrets | +| Pilot Coupon | ✅ Auto-seeded | `app/jobs/startup_seeder.py` | ### Required Configuration Changes -#### 1. Remove imagePullSecrets (BLOCKING) -**Why:** Images are public/don't require authentication -**Impact if skipped:** All pods fail with ImagePullBackOff +#### 1. imagePullSecrets - ✅ **AUTOMATICALLY HANDLED** +**Status:** ✅ The production kustomization automatically patches all workloads +**File:** `infrastructure/environments/prod/k8s-manifests/kustomization.yaml` -#### 2. Update Image Tags to Semantic Versions (BLOCKING) +The production overlay adds `gitea-registry-secret` to all Deployments, StatefulSets, Jobs, and CronJobs via Kustomize patches: +```yaml +patches: + - target: + kind: Deployment + patch: |- + - op: add + path: /spec/template/spec/imagePullSecrets + value: + - name: gitea-registry-secret +``` + +**Note:** The `gitea-registry-secret` is created by `infrastructure/cicd/gitea/sync-registry-secret.sh` after Gitea deployment. + +#### 2. Update Image Tags to Semantic Versions (FOR RELEASES) **Why:** Using 'latest' causes non-deterministic deployments **Impact if skipped:** Unpredictable behavior, impossible rollbacks +**File:** `infrastructure/environments/prod/k8s-manifests/kustomization.yaml` -#### 3. Fix SigNoz Namespace References (BLOCKING) - ✅ **ALREADY FIXED** -**Why:** SigNoz must be in bakery-ia namespace -**Impact if skipped:** Kustomize apply fails -**Status:** ✅ Fixed in latest commit +For production releases, update the `images:` section from `latest` to semantic versions. -#### 4. Production Secrets (ALREADY CONFIGURED) ✅ -**Status:** Strong production secrets have been generated and configured -**Impact if skipped:** N/A - This step is already completed +#### 3. Production Secrets - ✅ **ALREADY CONFIGURED** +**Status:** ✅ Strong production secrets have been pre-generated +**File:** `infrastructure/environments/common/configs/secrets.yaml` -#### 5. Update Cert-Manager Email (HIGH PRIORITY) - ✅ **ALREADY FIXED** -**Why:** Receive Let's Encrypt renewal notifications -**Impact if skipped:** Won't receive SSL expiry warnings -**Status:** ✅ Fixed - email is now `admin@bakewise.ai` +Pre-configured secrets include: +- **19 database passwords** (24-character URL-safe random strings) +- **JWT secrets** (256-bit cryptographically secure) +- **Redis password** (24-character random string) +- **RabbitMQ credentials** +- **PostgreSQL monitoring user** for SigNoz metrics collection -#### 6. Update Stripe Publishable Key (HIGH PRIORITY) -**Why:** Payment processing requires production Stripe key +#### 4. Cert-Manager Email - ✅ **ALREADY CONFIGURED** +**Status:** ✅ Email set to `admin@bakewise.ai` +**File:** `infrastructure/platform/cert-manager/cluster-issuer-production.yaml` + +#### 5. Update Stripe Keys (HIGH PRIORITY) +**Why:** Payment processing requires production Stripe keys **Impact if skipped:** Payments will use test mode (no real charges) -**File:** `infrastructure/kubernetes/base/configmap.yaml` line 378 -**Current value:** `pk_test_your_stripe_publishable_key_here` -**Required:** Your Stripe production publishable key from https://dashboard.stripe.com/apikeys -#### 7. Pilot Coupon Configuration (OPTIONAL) -**Why:** Control pilot program settings -**Files:** `infrastructure/kubernetes/base/configmap.yaml` lines 375-377 -**Current values (defaults are correct for pilot):** +**ConfigMap** (`infrastructure/environments/common/configs/configmap.yaml`): +```yaml +VITE_STRIPE_PUBLISHABLE_KEY: "pk_live_XXXXXXXXXXXXXXXXXXXX" +``` + +**Secrets** (`infrastructure/environments/common/configs/secrets.yaml`): +```yaml +# Add to payment-secrets section (base64 encoded) +STRIPE_SECRET_KEY: +STRIPE_WEBHOOK_SECRET: +``` + +Get your keys from: https://dashboard.stripe.com/apikeys + +#### 6. Pilot Coupon Configuration - ✅ **AUTO-SEEDED** +**Status:** ✅ Automatically created when tenant-service starts +**How it works:** `app/jobs/startup_seeder.py` creates the PILOT2025 coupon + +Default pilot settings (in configmap, can be customized): - `VITE_PILOT_MODE_ENABLED: "true"` - Enables pilot UI features - `VITE_PILOT_COUPON_CODE: "PILOT2025"` - Coupon code for 3 months free - `VITE_PILOT_TRIAL_MONTHS: "3"` - Trial extension duration -**Note:** The PILOT2025 coupon is automatically created when tenant-service starts. -No manual seeding required - it's handled by `app/jobs/startup_seeder.py`. - ### ✅ Already Correct (No Changes Needed) -- **Storage Class** - `microk8s-hostpath` is correct for MicroK8s -- **Domain Names** - `bakewise.ai` is your production domain +- **Storage Class** - Uses MicroK8s default storage provisioner +- **Domain Names** - `bakewise.ai` configured in production overlay - **Service Types** - ClusterIP + Ingress is correct architecture -- **Network Policies** - Not required for single-namespace deployment -- **SigNoz Namespace** - ✅ Fixed to use bakery-ia namespace +- **Network Policies** - Defined in `infrastructure/platform/security/network-policies/` +- **SigNoz Namespace** - ✅ Uses `bakery-ia` namespace (unified with application) +- **OTEL Configuration** - ✅ Pre-configured for SigNoz in production patches +- **Replica Counts** - ✅ Production replicas defined in kustomization (2-3 per service) ### Step-by-Step Configuration Script @@ -238,68 +304,91 @@ Run these commands on your **local machine** before deployment: cd /path/to/bakery-ia # ======================================== -# STEP 1: Remove imagePullSecrets +# STEP 1: Verify Infrastructure Structure # ======================================== -echo "Step 1: Removing imagePullSecrets..." -chmod +x infrastructure/kubernetes/remove-imagepullsecrets.sh -./infrastructure/kubernetes/remove-imagepullsecrets.sh - -# Verify removal -grep -r "imagePullSecrets" infrastructure/kubernetes/base/ && \ - echo "⚠️ WARNING: Some files still have imagePullSecrets" || \ - echo "✅ imagePullSecrets removed" +echo "Step 1: Verifying new infrastructure structure..." +echo "Checking directories..." +ls -d infrastructure/environments/common/configs/ && echo "✅ Common configs" +ls -d infrastructure/environments/prod/k8s-manifests/ && echo "✅ Prod kustomization" +ls -d infrastructure/platform/cert-manager/ && echo "✅ Cert-manager" +ls -d infrastructure/cicd/gitea/ && echo "✅ Gitea CI/CD" # ======================================== -# STEP 2: Update Image Tags +# STEP 2: Update Image Tags (for releases) # ======================================== -echo -e "\nStep 2: Updating image tags..." +echo -e "\nStep 2: Updating image tags for release..." export VERSION="1.0.0" # Change this to your version -sed -i.bak "s/newTag: latest/newTag: v${VERSION}/g" infrastructure/kubernetes/overlays/prod/kustomization.yaml -# Verify no 'latest' tags remain -grep "newTag:" infrastructure/kubernetes/overlays/prod/kustomization.yaml | grep "latest" && \ - echo "⚠️ WARNING: Some images still use 'latest'" || \ - echo "✅ All images now use version v${VERSION}" +# Update application image tags in production kustomization +sed -i.bak "s/newTag: latest/newTag: v${VERSION}/g" \ + infrastructure/environments/prod/k8s-manifests/kustomization.yaml + +# Verify (show first 10 image entries) +echo "Current image tags:" +grep -A 1 "name: bakery/" infrastructure/environments/prod/k8s-manifests/kustomization.yaml | head -20 # ======================================== -# STEP 3: Production Secrets (ALREADY DONE) ✅ +# STEP 3: Verify Production Secrets # ======================================== echo -e "\nStep 3: Verifying production secrets..." -echo "✅ Production secrets have been pre-configured with strong passwords" -echo " - JWT secrets: 256-bit cryptographically secure" -echo " - Database passwords: 24-character random strings" -echo " - Redis password: 24-character random string" -echo " - RabbitMQ password: 24-character random string" -echo " - Service API key: 64-character hex string" +echo "✅ Production secrets are pre-configured with strong passwords:" +echo " - 19 database passwords (24-char URL-safe random)" +echo " - JWT secrets (256-bit cryptographically secure)" +echo " - Redis password (24-char random)" +echo " - RabbitMQ credentials" +echo " - PostgreSQL monitoring user for SigNoz" echo "" -echo "All secrets are already set in infrastructure/kubernetes/base/secrets.yaml" -echo "No manual action required for this step." +echo "Location: infrastructure/environments/common/configs/secrets.yaml" + +# Quick verification +grep -c "_DB_PASSWORD:" infrastructure/environments/common/configs/secrets.yaml +echo "database password entries found" # ======================================== -# STEP 4: Cert-Manager Email (ALREADY FIXED) +# STEP 4: Verify Cert-Manager Email # ======================================== echo -e "\nStep 4: Verifying cert-manager email..." -grep "admin@bakewise.ai" infrastructure/kubernetes/base/components/cert-manager/cluster-issuer-production.yaml && \ - echo "✅ Cert-manager email already set to admin@bakewise.ai" || \ - echo "⚠️ WARNING: Cert-manager email needs updating" +grep "email:" infrastructure/platform/cert-manager/cluster-issuer-production.yaml +# Should show: email: admin@bakewise.ai # ======================================== -# STEP 5: Update Stripe Publishable Key +# STEP 5: Verify imagePullSecrets Patch # ======================================== -echo -e "\nStep 5: Stripe Publishable Key Configuration..." +echo -e "\nStep 5: Verifying imagePullSecrets configuration..." +grep -A 5 "gitea-registry-secret" infrastructure/environments/prod/k8s-manifests/kustomization.yaml && \ + echo "✅ imagePullSecrets patch configured" || \ + echo "⚠️ WARNING: imagePullSecrets patch missing" + +# ======================================== +# STEP 6: Configure Stripe Keys (MANUAL) +# ======================================== +echo -e "\nStep 6: Stripe Configuration..." echo "================================================================" echo "⚠️ MANUAL STEP REQUIRED" echo "" -echo "Edit: infrastructure/kubernetes/base/configmap.yaml" -echo "Find: VITE_STRIPE_PUBLISHABLE_KEY: \"pk_test_your_stripe_publishable_key_here\"" -echo "Replace with your production Stripe publishable key from:" -echo " https://dashboard.stripe.com/apikeys" +echo "1. Edit ConfigMap:" +echo " File: infrastructure/environments/common/configs/configmap.yaml" +echo " Add: VITE_STRIPE_PUBLISHABLE_KEY: \"pk_live_XXXX\"" echo "" -echo "Example:" -echo " VITE_STRIPE_PUBLISHABLE_KEY: \"pk_live_XXXXXXXXXXXXXXXXXXXX\"" +echo "2. Edit Secrets:" +echo " File: infrastructure/environments/common/configs/secrets.yaml" +echo " Add to payment-secrets (base64 encoded):" +echo " STRIPE_SECRET_KEY: " +echo " STRIPE_WEBHOOK_SECRET: " echo "" -echo "Press Enter when you've updated the Stripe key..." -read +echo "Get keys from: https://dashboard.stripe.com/apikeys" +echo "================================================================" +read -p "Press Enter when Stripe keys are configured..." + +# ======================================== +# STEP 7: Validate Kustomization Build +# ======================================== +echo -e "\nStep 7: Validating Kustomization..." +cd infrastructure/environments/prod/k8s-manifests +kustomize build . > /dev/null 2>&1 && \ + echo "✅ Kustomization builds successfully" || \ + echo "⚠️ WARNING: Kustomization build failed" +cd - > /dev/null # ======================================== # FINAL VALIDATION @@ -309,15 +398,18 @@ echo "Pre-Deployment Configuration Complete!" echo "========================================" echo "" echo "Validation Checklist:" -echo " ✅ imagePullSecrets removed" +echo " ✅ Infrastructure structure verified" echo " ✅ Image tags updated to v${VERSION}" -echo " ✅ SigNoz namespace fixed (bakery-ia)" -echo " ✅ Production secrets configured with strong passwords" -echo " ✅ Cert-manager email set to admin@bakewise.ai" -echo " ⚠️ Stripe publishable key updated (manual verification required)" -echo " ✅ Pilot coupon auto-seeded on tenant-service startup" +echo " ✅ Production secrets pre-configured" +echo " ✅ Cert-manager email: admin@bakewise.ai" +echo " ✅ imagePullSecrets auto-patched via Kustomize" +echo " ⚠️ Stripe keys configured (manual verification)" +echo " ✅ Pilot coupon auto-seeded on startup" echo "" -echo "Next: Copy manifests to VPS and begin deployment" +echo "Next Steps:" +echo " 1. Deploy CI/CD: Gitea, Tekton, Flux (via Helm)" +echo " 2. Push images to Gitea registry" +echo " 3. Apply Kustomization to cluster" ``` ### Manual Verification @@ -326,25 +418,48 @@ After running the script above: 1. **Verify production secrets are configured:** ```bash - # Verify secrets.yaml has strong passwords (not placeholders) - grep "JWT_SECRET_KEY" infrastructure/kubernetes/base/secrets.yaml - # Should show: dXNNSHc5a1FDUW95cmM3d1BtTWkzYkNscjBsVFk5d3Z6Wm1jVGJBRHZMMD0= - # (This is the base64-encoded production JWT secret) + # Check secrets file has strong passwords + head -80 infrastructure/environments/common/configs/secrets.yaml + # Should show base64-encoded passwords for all 19 databases ``` -2. **Check image tags:** +2. **Check image tags in production overlay:** ```bash - grep "newTag:" infrastructure/kubernetes/overlays/prod/kustomization.yaml - # All should show v1.0.0 (or your version), NOT 'latest' + grep "newTag:" infrastructure/environments/prod/k8s-manifests/kustomization.yaml | head -10 + # For releases: should show v1.0.0 (your version) + # For development: 'latest' is acceptable ``` -3. **Verify SigNoz namespace:** +3. **Verify imagePullSecrets patch:** ```bash - grep -A 3 "name: signoz" infrastructure/kubernetes/overlays/prod/kustomization.yaml - # All should show: namespace: bakery-ia + grep -B 2 -A 6 "imagePullSecrets" infrastructure/environments/prod/k8s-manifests/kustomization.yaml + # Should show patches for Deployment, StatefulSet, Job, CronJob ``` -**⏱️ Estimated Time:** 30-45 minutes +4. **Verify OTEL/SigNoz configuration:** + ```bash + grep "OTEL_EXPORTER" infrastructure/environments/prod/k8s-manifests/kustomization.yaml + # Should show: http://signoz-otel-collector.bakery-ia.svc.cluster.local:4317 + ``` + +5. **Test Kustomize build:** + ```bash + cd infrastructure/environments/prod/k8s-manifests + kustomize build . | kubectl apply --dry-run=client -f - + # Should complete without errors + ``` + +### Key File Locations Reference + +| Configuration | File Path | +|---------------|-----------| +| ConfigMap | `infrastructure/environments/common/configs/configmap.yaml` | +| Secrets | `infrastructure/environments/common/configs/secrets.yaml` | +| Prod Kustomization | `infrastructure/environments/prod/k8s-manifests/kustomization.yaml` | +| Cert-Manager Issuer | `infrastructure/platform/cert-manager/cluster-issuer-production.yaml` | +| Ingress | `infrastructure/platform/networking/ingress/base/ingress.yaml` | +| Gitea Values | `infrastructure/cicd/gitea/values.yaml` | +| Mailu Values | `infrastructure/platform/mail/mailu-helm/values.yaml` | --- @@ -360,10 +475,10 @@ After running the script above: - Sign up at [clouding.io](https://www.clouding.io) - Payment method configured -- [ ] **Email Service** (Choose ONE) - - Option A: Zoho Mail FREE (recommended for full send/receive) - - Option B: Gmail SMTP + domain forwarding - - Option C: Google Workspace (14-day free trial, then €5.75/month) +- [ ] **Email Service** - Self-hosted Mailu with Mailgun relay + - Mailu deployed via Helm chart (see [Mailu Email Server Deployment](#mailu-email-server-deployment)) + - Mailgun account for outbound relay (improves deliverability) + - DNS records configured (MX, SPF, DKIM, DMARC) - [ ] **WhatsApp Business API** - Create Meta Business Account (free) @@ -797,106 +912,35 @@ nano infrastructure/platform/cert-manager/cluster-issuer-production.yaml ## Email & Communication Setup -### Option A: Zoho Mail (FREE, Recommended) +### Self-Hosted Mailu with Mailgun Relay -**Features:** -- ✅ Free forever for 1 domain, 5 users -- ✅ 5GB storage per user -- ✅ Full send/receive capability -- ✅ Web interface + SMTP/IMAP -- ✅ Professional email addresses - -**Setup Steps:** - -1. **Sign up for Zoho Mail:** - ``` - 1. Go to https://www.zoho.com/mail/ - 2. Click "Sign Up for Free" - 3. Choose "Forever Free" plan - 4. Enter your domain name - 5. Complete verification - ``` - -2. **Verify domain ownership:** - ``` - Add TXT record to your DNS: - Type: TXT - Name: @ - Value: zoho-verification=XXXXX.zoho.com - ``` - -3. **Configure MX records:** - ``` - Priority Type Name Value - 10 MX @ mx.zoho.com - 20 MX @ mx2.zoho.com - 50 MX @ mx3.zoho.com - ``` - -4. **Get SMTP credentials:** - ``` - SMTP Host: smtp.zoho.com - SMTP Port: 587 - SMTP Username: noreply@yourdomain.com - SMTP Password: (generate app password in Zoho settings) - ``` - -### Option B: Gmail SMTP + Forwarding - -**Features:** -- ✅ Completely free -- ✅ 500 emails/day (sufficient for pilot) -- ✅ Receive via domain forwarding - -**Setup Steps:** - -1. **Enable 2FA on your Gmail:** - ``` - 1. Go to myaccount.google.com - 2. Security → 2-Step Verification - 3. Enable and complete setup - ``` - -2. **Generate app password:** - ``` - 1. Security → 2-Step Verification → App passwords - 2. Select "Mail" and "Other (Custom name)" - 3. Name it "Bakery-IA SMTP" - 4. Copy the 16-character password - ``` - -3. **Configure domain email forwarding:** - ``` - At your domain registrar or Cloudflare: - - Forward noreply@yourdomain.com → your.gmail@gmail.com - - Forward alerts@yourdomain.com → your.gmail@gmail.com - ``` - -4. **SMTP Settings:** - ``` - SMTP Host: smtp.gmail.com - SMTP Port: 587 - SMTP Username: your.gmail@gmail.com - SMTP Password: (16-char app password from step 2) - From Email: noreply@yourdomain.com - ``` - -### Option C: Self-Hosted Mailu (RECOMMENDED for Production) +**Architecture:** +- **Mailu** - Self-hosted email server (Postfix, Dovecot, Rspamd, Roundcube webmail) +- **Mailgun** - External SMTP relay for improved outbound deliverability +- **Helm deployment** - `infrastructure/platform/mail/mailu-helm/` **Features:** - ✅ Full control over email infrastructure -- ✅ No external dependencies or rate limits +- ✅ Mailgun relay improves deliverability (avoids VPS IP reputation issues) - ✅ Built-in antispam (rspamd) with DNSSEC validation -- ✅ Webmail interface (Roundcube) +- ✅ Webmail interface (Roundcube) at `/webmail` +- ✅ Admin panel at `/admin` - ✅ IMAP/SMTP with TLS -- ✅ Admin panel for user management -- ✅ Integrated with Kubernetes +- ✅ Professional addresses: admin@bakewise.ai, noreply@bakewise.ai -**Why Mailu for Production:** -- Complete email stack (Postfix, Dovecot, Rspamd, ClamAV) -- DNSSEC validation for email authentication (DKIM/SPF/DMARC) -- No monthly email limits or third-party dependencies -- Professional email addresses: admin@bakewise.ai, noreply@bakewise.ai +**Configuration Files:** +| File | Purpose | +|------|---------| +| `infrastructure/platform/mail/mailu-helm/values.yaml` | Base Mailu configuration | +| `infrastructure/platform/mail/mailu-helm/prod/values.yaml` | Production overrides | +| `infrastructure/platform/mail/mailu-helm/configs/mailgun-credentials-secret.yaml` | Mailgun SMTP credentials | + +**Internal SMTP for Application Services:** +```yaml +# Services use Mailu's internal postfix for sending +SMTP_HOST: mailu-postfix.bakery-ia.svc.cluster.local +SMTP_PORT: 587 +``` #### Prerequisites @@ -992,7 +1036,21 @@ rm -rf "$TEMP_DIR" kubectl get secret mailu-certificates -n bakery-ia ``` -#### Step 5: Deploy Mailu via Helm +#### Step 5: Create Admin Credentials Secret + +```bash +# Generate a secure password (or use your own) +ADMIN_PASSWORD=$(openssl rand -base64 16 | tr -d '/+=' | head -c 16) +echo "Admin password: $ADMIN_PASSWORD" +echo "SAVE THIS PASSWORD SECURELY!" + +# Create the admin credentials secret +kubectl create secret generic mailu-admin-credentials \ + --from-literal=password="$ADMIN_PASSWORD" \ + -n bakery-ia +``` + +#### Step 6: Deploy Mailu via Helm ```bash # Add Mailu Helm repository @@ -1000,6 +1058,7 @@ helm repo add mailu https://mailu.github.io/helm-charts helm repo update mailu # Deploy Mailu with production values +# Admin user is created automatically via initialAccount feature helm upgrade --install mailu mailu/mailu \ -n bakery-ia \ --create-namespace \ @@ -1009,18 +1068,9 @@ helm upgrade --install mailu mailu/mailu \ # Wait for pods to be ready (may take 5-10 minutes for ClamAV) kubectl get pods -n bakery-ia -l app.kubernetes.io/instance=mailu -w -``` -#### Step 6: Create Admin User - -```bash -# Create initial admin user -kubectl exec -it -n bakery-ia deployment/mailu-admin -- \ - flask mailu admin admin bakewise.ai 'YourSecurePassword123!' - -# Credentials: -# Email: admin@bakewise.ai -# Password: YourSecurePassword123! +# Admin user (admin@bakewise.ai) is created automatically! +# Password is the one you set in Step 5 ``` #### Step 7: Configure DKIM @@ -1485,6 +1535,998 @@ kubectl exec -n bakery-ia deployment/auth-db -it -- psql -U auth_user -d auth_db --- +## CI/CD Infrastructure Deployment + +This section covers deploying the complete CI/CD stack: Gitea (Git server + container registry), Tekton (CI pipelines), and Flux CD (GitOps deployments). + +### Overview + +``` +┌─────────────────────────────────────────────────────────────────────────────┐ +│ CI/CD ARCHITECTURE │ +│ │ +│ Developer Push │ +│ │ │ +│ ▼ │ +│ ┌─────────┐ Webhook ┌─────────────┐ Build/Test ┌─────────┐ │ +│ │ Gitea │ ───────────────► │ Tekton │ ─────────────────►│ Images │ │ +│ │ (Git) │ │ (Pipelines)│ │(Registry)│ │ +│ └─────────┘ └─────────────┘ └─────────┘ │ +│ │ │ │ │ +│ │ │ Update manifests │ │ +│ │ ▼ │ │ +│ │ ┌─────────────┐ │ │ +│ └──────────────────────►│ Flux CD │◄───────────────────────┘ │ +│ Monitor changes │ (GitOps) │ Pull images │ +│ └─────────────┘ │ +│ │ │ +│ ▼ │ +│ ┌─────────────┐ │ +│ │ Kubernetes │ │ +│ │ Cluster │ │ +│ └─────────────┘ │ +└─────────────────────────────────────────────────────────────────────────────┘ +``` + +### Prerequisites + +Before deploying CI/CD infrastructure: +- [ ] Kubernetes cluster is running +- [ ] Ingress controller is configured +- [ ] TLS certificates are available +- [ ] DNS records configured for `gitea.bakewise.ai` + +### Step 1: Deploy Gitea (Git Server + Container Registry) + +Gitea provides a self-hosted Git server with built-in container registry support. The setup is fully automated - admin user and initial repository are created automatically. + +#### 1.1 Create Secrets and Init Job (One Command) + +The setup script creates all necessary secrets and applies the initialization job. + +**For Production Deployment:** + +```bash +# Generate a secure password (minimum 16 characters required for production) +export GITEA_ADMIN_PASSWORD=$(openssl rand -base64 32) +echo "Gitea Admin Password: $GITEA_ADMIN_PASSWORD" +echo "⚠️ Save this password securely - you'll need it for Tekton setup!" + +# Run the setup script with --production flag +# This enforces password requirements and uses production registry URL +./infrastructure/cicd/gitea/setup-admin-secret.sh --production +``` + +**What the `--production` flag does:** +- Requires `GITEA_ADMIN_PASSWORD` environment variable (won't use defaults) +- Validates password is at least 16 characters +- Uses production registry URL (`registry.bakewise.ai`) +- Hides password in output for security +- Shows production-specific next steps + +This creates: +- `gitea-admin-secret` in `gitea` namespace - admin credentials for Gitea +- `gitea-registry-secret` in `bakery-ia` namespace - for imagePullSecrets +- `gitea-init-job` - Kubernetes Job that creates the `bakery-ia` repository automatically + +> **For dev environments only:** Run without flags to use the default static password: +> ```bash +> ./infrastructure/cicd/gitea/setup-admin-secret.sh +> ``` + +#### 1.2 Install Gitea via Helm + +```bash +# Add Gitea Helm repository +helm repo add gitea https://dl.gitea.io/charts +helm repo update gitea + +# Install Gitea with PRODUCTION values (includes TLS, proper domains, resources) +helm upgrade --install gitea gitea/gitea \ + -n gitea \ + -f infrastructure/cicd/gitea/values.yaml \ + -f infrastructure/cicd/gitea/values-prod.yaml \ + --timeout 10m \ + --wait + +# Wait for Gitea to be ready +kubectl wait --for=condition=ready pod -n gitea -l app.kubernetes.io/name=gitea --timeout=300s + +# Verify Gitea is running +kubectl get pods -n gitea +kubectl get svc -n gitea +``` + +**Production values (`values-prod.yaml`) include:** +- Domain: `gitea.bakewise.ai` and `registry.bakewise.ai` +- TLS via cert-manager with Let's Encrypt production issuer +- 50Gi storage (vs 10Gi in dev) +- Increased resource limits + +#### 1.3 Verify Repository Initialization + +The init job automatically creates the `bakery-ia` repository once Gitea is ready: + +```bash +# Check init job completed successfully +kubectl logs -n gitea job/gitea-init-repo + +# Expected output: +# === Gitea Repository Initialization === +# Gitea is ready! +# Repository 'bakery-ia' created successfully! +``` + +If the job needs to be re-run: +```bash +kubectl delete job gitea-init-repo -n gitea +kubectl apply -f infrastructure/cicd/gitea/gitea-init-job.yaml +``` + +#### 1.4 Configure DNS for Gitea + +Add DNS record pointing to your VPS: +``` +Type Name Value TTL +A gitea YOUR_VPS_IP Auto +``` + +#### 1.5 Verify Gitea Access + +```bash +# Check ingress is configured +kubectl get ingress -n gitea + +# Test access (after DNS propagation) +curl -I https://gitea.bakewise.ai + +# Access web interface +# URL: https://gitea.bakewise.ai +# Username: bakery-admin +# Password: (from step 1.1) + +# Verify repository was created via API +curl -u bakery-admin:$GITEA_ADMIN_PASSWORD \ + https://gitea.bakewise.ai/api/v1/repos/bakery-admin/bakery-ia +``` + +#### 1.6 Push Code to Repository + +The `bakery-ia` repository is already created with a README. Push your code: + +```bash +# Add Gitea as remote and push code +cd /path/to/bakery-ia +git remote add gitea https://gitea.bakewise.ai/bakery-admin/bakery-ia.git +git push gitea main +``` + +### Step 2: Deploy Tekton Pipelines + +Tekton provides cloud-native CI/CD pipelines. + +#### 2.1 Install Tekton Core Components + +```bash +# Create Tekton namespace +kubectl create namespace tekton-pipelines + +# Install Tekton Pipelines +kubectl apply -f https://storage.googleapis.com/tekton-releases/pipeline/latest/release.yaml + +# Wait for Tekton Pipelines to be ready +kubectl wait --for=condition=available --timeout=300s \ + deployment/tekton-pipelines-controller -n tekton-pipelines +kubectl wait --for=condition=available --timeout=300s \ + deployment/tekton-pipelines-webhook -n tekton-pipelines + +# Install Tekton Triggers (for webhook-based automation) +kubectl apply -f https://storage.googleapis.com/tekton-releases/triggers/latest/release.yaml +kubectl apply -f https://storage.googleapis.com/tekton-releases/triggers/latest/interceptors.yaml + +# Wait for Tekton Triggers to be ready +kubectl wait --for=condition=available --timeout=300s \ + deployment/tekton-triggers-controller -n tekton-pipelines +kubectl wait --for=condition=available --timeout=300s \ + deployment/tekton-triggers-webhook -n tekton-pipelines + +# Verify installation +kubectl get pods -n tekton-pipelines +``` + +#### 2.2 Deploy Tekton CI/CD Configuration via Helm + +**For Production Deployment:** + +```bash +# Generate secure webhook token (save this for Gitea webhook configuration) +export TEKTON_WEBHOOK_TOKEN=$(openssl rand -hex 32) +echo "Webhook Token: $TEKTON_WEBHOOK_TOKEN" +echo "⚠️ Save this token - you'll need it for Gitea webhook setup!" + +# Ensure GITEA_ADMIN_PASSWORD is still set from Step 1 +echo "Using Gitea password from: GITEA_ADMIN_PASSWORD" + +# Install Tekton CI/CD with PRODUCTION values +helm upgrade --install tekton-cicd infrastructure/cicd/tekton-helm \ + -n tekton-pipelines \ + -f infrastructure/cicd/tekton-helm/values.yaml \ + -f infrastructure/cicd/tekton-helm/values-prod.yaml \ + --set secrets.webhook.token=$TEKTON_WEBHOOK_TOKEN \ + --set secrets.registry.password=$GITEA_ADMIN_PASSWORD \ + --set secrets.git.password=$GITEA_ADMIN_PASSWORD \ + --timeout 10m \ + --wait + +# Verify resources created +kubectl get pipelines -n tekton-pipelines +kubectl get tasks -n tekton-pipelines +kubectl get eventlisteners -n tekton-pipelines +kubectl get triggerbindings -n tekton-pipelines +kubectl get triggertemplates -n tekton-pipelines +``` + +**What the production values (`values-prod.yaml`) provide:** +- Empty default secrets (must be provided via `--set` flags) +- Increased controller/webhook replicas (2 each) +- Higher resource limits for production workloads +- 10Gi workspace storage (vs 5Gi in dev) + +> **⚠️ Security Note:** Never commit actual secrets to values files. Always pass them via `--set` flags or use external secret management. + +#### 2.3 Configure Gitea Webhook + +1. Go to Gitea repository settings → Webhooks +2. Add webhook: + - **Target URL:** `http://el-bakery-ia-listener.tekton-pipelines.svc.cluster.local:8080` + - **HTTP Method:** POST + - **Content Type:** application/json + - **Secret:** (same as `secrets.webhook.token` from Helm) + - **Trigger on:** Push events +3. Save webhook + +#### 2.4 Test Pipeline Manually + +```bash +# Create a manual PipelineRun to test the CI pipeline +cat <> README.md +git add README.md +git commit -m "Test CI/CD pipeline" + +# 2. Push to Gitea +git push gitea main + +# 3. Watch Tekton pipeline triggered by webhook +kubectl get pipelineruns -n tekton-pipelines -w + +# 4. After pipeline completes, watch Flux sync +flux get kustomizations -n flux-system -w + +# 5. Verify deployment updated +kubectl get deployments -n bakery-ia -o wide +``` + +### CI/CD Troubleshooting + +#### Tekton Pipeline Fails + +```bash +# View pipeline run status +kubectl get pipelineruns -n tekton-pipelines + +# Get detailed logs +tkn pipelinerun describe -n tekton-pipelines +tkn pipelinerun logs -n tekton-pipelines + +# Check EventListener logs (for webhook issues) +kubectl logs -n tekton-pipelines -l app.kubernetes.io/component=eventlistener +``` + +#### Flux Not Syncing + +```bash +# Check GitRepository status +kubectl describe gitrepository bakery-ia -n flux-system + +# Check Kustomization status +kubectl describe kustomization bakery-ia-prod -n flux-system + +# View Flux controller logs +kubectl logs -n flux-system deployment/source-controller +kubectl logs -n flux-system deployment/kustomize-controller + +# Force reconciliation +flux reconcile source git bakery-ia -n flux-system --with-source +``` + +#### Gitea Webhook Not Triggering + +```bash +# Check webhook delivery in Gitea UI +# Settings → Webhooks → Recent Deliveries + +# Verify EventListener is running +kubectl get eventlisteners -n tekton-pipelines +kubectl get svc -n tekton-pipelines | grep listener + +# Check EventListener logs +kubectl logs -n tekton-pipelines -l eventlistener=bakery-ia-listener +``` + +### CI/CD URLs Summary + +| Service | URL | Purpose | +|---------|-----|---------| +| Gitea | https://gitea.bakewise.ai | Git repository & container registry | +| Gitea Registry | https://gitea.bakewise.ai/v2/ | Docker registry API | +| Tekton Dashboard | (install separately if needed) | Pipeline visualization | +| Flux | CLI only | GitOps status via `flux` commands | + +### CI/CD Security Considerations + +The CI/CD infrastructure has been configured with production security in mind: + +#### Secrets Management + +| Secret | Purpose | How to Generate | +|--------|---------|-----------------| +| `GITEA_ADMIN_PASSWORD` | Gitea admin & registry auth | `openssl rand -base64 32` | +| `TEKTON_WEBHOOK_TOKEN` | Webhook signature validation | `openssl rand -hex 32` | + +#### Security Features + +1. **Production Mode Enforcement** + - The `--production` flag on `setup-admin-secret.sh` enforces: + - Mandatory `GITEA_ADMIN_PASSWORD` environment variable + - Minimum 16-character password requirement + - Password hidden from terminal output + +2. **Internal Cluster Communication** + - All CI/CD components communicate via internal cluster DNS + - GitOps updates use `gitea-http.gitea.svc.cluster.local:3000` + - No hardcoded external URLs in pipeline tasks + +3. **Credential Isolation** + - Secrets are passed via `--set` flags, never committed to git + - Registry credentials are scoped per-namespace + - Webhook tokens are unique per installation + +#### Post-Deployment Security Checklist + +```bash +# Verify no default passwords in use +kubectl get secret gitea-admin-secret -n gitea -o jsonpath='{.data.password}' | base64 -d | wc -c +# Should be 32+ characters for production + +# Verify webhook secret is set +kubectl get secret gitea-webhook-secret -n tekton-pipelines -o jsonpath='{.data.secretToken}' | base64 -d | wc -c +# Should be 64 characters (hex-encoded 32 bytes) + +# Verify no hardcoded URLs in tasks +kubectl get task update-gitops -n tekton-pipelines -o yaml | grep -c "bakery-ia.local" +# Should be 0 +``` + +--- + +## Mailu Email Server Deployment + +Mailu is a full-featured, self-hosted email server with built-in antispam, webmail, and admin panel. **Outbound emails are relayed through Mailgun** for improved deliverability and to avoid IP reputation issues. + +### Prerequisites + +Before deploying Mailu: +- [ ] Unbound DNS resolver deployed (for DNSSEC validation) +- [ ] DNS records configured for mail domain +- [ ] TLS certificates available +- [ ] Mailgun account created and domain verified (for outbound email relay) + +### Step 1: Deploy Unbound DNS Resolver + +Mailu requires DNSSEC validation for email authentication (DKIM/SPF/DMARC). + +```bash +# Deploy Unbound via Helm +helm upgrade --install unbound infrastructure/platform/networking/dns/unbound-helm \ + -n bakery-ia \ + --create-namespace \ + -f infrastructure/platform/networking/dns/unbound-helm/values.yaml \ + -f infrastructure/platform/networking/dns/unbound-helm/prod/values.yaml \ + --timeout 5m \ + --wait + +# Verify Unbound is running +kubectl get pods -n bakery-ia | grep unbound + +# Get Unbound service IP +UNBOUND_IP=$(kubectl get svc unbound-dns -n bakery-ia -o jsonpath='{.spec.clusterIP}') +echo "Unbound DNS IP: $UNBOUND_IP" +``` + +### Step 2: Configure CoreDNS for DNSSEC + +```bash +# Get Unbound IP +UNBOUND_IP=$(kubectl get svc unbound-dns -n bakery-ia -o jsonpath='{.spec.clusterIP}') + +# Create updated CoreDNS ConfigMap +cat > /tmp/coredns-config.yaml < SMTP credentials** +2. Note your credentials: + - **SMTP hostname:** `smtp.mailgun.org` + - **Port:** `587` (TLS/STARTTLS) + - **Username:** typically `postmaster@bakewise.ai` + - **Password:** your Mailgun SMTP password (NOT the API key) + +#### 3.3: Create Kubernetes Secret for Mailgun + +```bash +# Edit the secret template with your Mailgun credentials +nano infrastructure/platform/mail/mailu-helm/configs/mailgun-credentials-secret.yaml + +# Replace the placeholder values: +# RELAY_USERNAME: "postmaster@bakewise.ai" +# RELAY_PASSWORD: "your-mailgun-smtp-password" + +# Apply the secret +kubectl apply -f infrastructure/platform/mail/mailu-helm/configs/mailgun-credentials-secret.yaml -n bakery-ia + +# Verify secret created +kubectl get secret mailu-mailgun-credentials -n bakery-ia +``` + +### Step 4: Configure DNS Records for Mail + +Add these DNS records for your domain (e.g., bakewise.ai): + +``` +Type Name Value TTL Priority +A mail YOUR_VPS_IP Auto - +MX @ mail.bakewise.ai Auto 10 +TXT @ v=spf1 include:mailgun.org mx a ~all Auto - +TXT _dmarc v=DMARC1; p=quarantine; rua=... Auto - +``` + +**Mailgun-specific DNS records** (Mailgun will provide exact values): +``` +Type Name Value TTL +TXT (provided by Mailgun) (DKIM key from Mailgun) Auto +TXT (provided by Mailgun) (DKIM key from Mailgun) Auto +``` + +**Note:** +- The SPF record includes `mailgun.org` to authorize Mailgun to send on your behalf +- Add the DKIM records exactly as Mailgun provides them +- Mailu's own DKIM record will be added after deployment (Step 9) + +### Step 5: Create TLS Certificate Secret + +```bash +# Generate self-signed certificate for internal Mailu use +# (Ingress handles external TLS termination) +TEMP_DIR=$(mktemp -d) +cd "$TEMP_DIR" + +openssl req -x509 -nodes -days 365 -newkey rsa:2048 \ + -keyout tls.key -out tls.crt \ + -subj "/CN=mail.bakewise.ai/O=bakewise" + +kubectl create secret tls mailu-certificates \ + --cert=tls.crt \ + --key=tls.key \ + -n bakery-ia + +rm -rf "$TEMP_DIR" + +# Verify secret created +kubectl get secret mailu-certificates -n bakery-ia +``` + +### Step 6: Create Admin Credentials Secret + +The admin account is created automatically during Helm deployment using the `initialAccount` feature. Create a secret with the admin password before deploying. + +```bash +# Generate a secure password (or use your own) +ADMIN_PASSWORD=$(openssl rand -base64 16 | tr -d '/+=' | head -c 16) +echo "Admin password: $ADMIN_PASSWORD" +echo "SAVE THIS PASSWORD SECURELY!" + +# Create the admin credentials secret +kubectl create secret generic mailu-admin-credentials \ + --from-literal=password="$ADMIN_PASSWORD" \ + -n bakery-ia + +# Verify secret created +kubectl get secret mailu-admin-credentials -n bakery-ia +``` + +**Alternative:** Use the provided template file: +```bash +# Edit the secret template with your password (base64 encoded) +nano infrastructure/platform/mail/mailu-helm/configs/mailu-admin-credentials-secret.yaml + +# Apply the secret +kubectl apply -f infrastructure/platform/mail/mailu-helm/configs/mailu-admin-credentials-secret.yaml +``` + +### Step 7: Deploy Mailu via Helm + +```bash +# Add Mailu Helm repository +helm repo add mailu https://mailu.github.io/helm-charts +helm repo update mailu + +# Deploy Mailu with production values +# Note: +# - externalRelay uses Mailgun via the secret created in Step 3 +# - initialAccount creates admin user automatically using the secret from Step 6 +helm upgrade --install mailu mailu/mailu \ + -n bakery-ia \ + --create-namespace \ + -f infrastructure/platform/mail/mailu-helm/values.yaml \ + -f infrastructure/platform/mail/mailu-helm/prod/values.yaml \ + --timeout 10m + +# Wait for pods to be ready (ClamAV may take 5-10 minutes) +kubectl get pods -n bakery-ia -l app.kubernetes.io/instance=mailu -w + +# The admin user (admin@bakewise.ai) is created automatically! +``` + +### Step 8: Apply Mailu Ingress + +```bash +# Apply Mailu-specific ingress configuration +kubectl apply -f infrastructure/platform/mail/mailu-helm/mailu-ingress.yaml + +# Verify ingress +kubectl get ingress -n bakery-ia | grep mailu +``` + +**Admin Credentials (created automatically in Step 7):** +- **Email:** `admin@bakewise.ai` +- **Password:** The password you set in Step 6 (stored in `mailu-admin-credentials` secret) + +To retrieve the password later: +```bash +kubectl get secret mailu-admin-credentials -n bakery-ia -o jsonpath='{.data.password}' | base64 -d +``` + +### Step 9: Configure DKIM + +```bash +# Get DKIM public key from Mailu +kubectl exec -n bakery-ia deployment/mailu-admin -- \ + cat /dkim/bakewise.ai.dkim.pub + +# Add DKIM record to DNS: +# Type: TXT +# Name: dkim._domainkey +# Value: (output from above command) +``` + +### Step 10: Verify Email Setup + +```bash +# Check all Mailu pods are running +kubectl get pods -n bakery-ia | grep mailu +# Expected: All pods in Running state + +# Verify Mailgun secret is configured +kubectl get secret mailu-mailgun-credentials -n bakery-ia +kubectl get secret mailu-mailgun-credentials -n bakery-ia -o jsonpath='{.data.RELAY_USERNAME}' | base64 -d +# Should show: postmaster@bakewise.ai + +# Test internal SMTP connectivity +kubectl run -it --rm smtp-test --image=alpine --restart=Never -- \ + sh -c "apk add swaks && swaks --to test@example.com --from admin@bakewise.ai --server mailu-front.bakery-ia.svc.cluster.local:25" + +# Test outbound email via Mailgun relay (send test email) +kubectl exec -it -n bakery-ia deployment/mailu-admin -- \ + flask mailu alias_create test bakewise.ai 'your-personal-email@gmail.com' +# Then send a test email from webmail to your personal email + +# Access webmail (via port-forward for testing) +kubectl port-forward -n bakery-ia svc/mailu-front 8080:80 +# Open: http://localhost:8080/webmail +``` + +### Mailu Endpoints + +| Service | URL/Address | +|---------|-------------| +| Admin Panel | https://mail.bakewise.ai/admin | +| Webmail | https://mail.bakewise.ai/webmail | +| SMTP (STARTTLS) | mail.bakewise.ai:587 | +| SMTP (SSL) | mail.bakewise.ai:465 | +| IMAP (SSL) | mail.bakewise.ai:993 | + +### Mailu Troubleshooting + +#### Admin Pod CrashLoopBackOff with DNSSEC Error + +```bash +# Verify CoreDNS is forwarding to Unbound +kubectl get configmap coredns -n kube-system -o yaml | grep forward +# Should show: forward . + +# If not configured, re-run Step 2 +``` + +#### Front Pod Stuck in ContainerCreating + +```bash +# Check for missing certificate secret +kubectl describe pod -n bakery-ia -l app.kubernetes.io/component=front | grep -A5 Events + +# If missing mailu-certificates, re-run Step 4 +``` + +#### Cannot Connect to Redis + +```bash +# Verify internal Redis is enabled (not external) +helm get values mailu -n bakery-ia | grep -A5 externalRedis +# Should show: enabled: false + +# If enabled: true, upgrade with correct values +helm upgrade mailu mailu/mailu -n bakery-ia \ + -f infrastructure/platform/mail/mailu-helm/values.yaml \ + -f infrastructure/platform/mail/mailu-helm/prod/values.yaml +``` + +#### Outbound Emails Not Delivered (Mailgun Relay Issues) + +```bash +# Check if Mailgun credentials secret exists +kubectl get secret mailu-mailgun-credentials -n bakery-ia +# If missing, create it (see Step 3) + +# Verify credentials are set correctly +kubectl get secret mailu-mailgun-credentials -n bakery-ia -o jsonpath='{.data.RELAY_USERNAME}' | base64 -d +# Should show your Mailgun username (e.g., postmaster@bakewise.ai) + +# Check Postfix logs for relay errors +kubectl logs -n bakery-ia deployment/mailu-postfix | grep -i "relay\|mailgun\|sasl" +# Look for authentication errors or connection failures + +# Verify Mailgun domain is verified +# Go to Mailgun dashboard > Domain Settings > DNS Records +# All records should show "Verified" status + +# Test Mailgun SMTP connectivity directly +kubectl run -it --rm mailgun-test --image=alpine --restart=Never -- \ + sh -c "apk add swaks && swaks --to test@example.com --from postmaster@bakewise.ai \ + --server smtp.mailgun.org:587 --tls \ + --auth-user 'postmaster@bakewise.ai' \ + --auth-password 'YOUR_MAILGUN_PASSWORD'" +``` + +#### Emails Going to Spam + +1. Verify SPF record includes Mailgun: `v=spf1 include:mailgun.org mx a ~all` +2. Check DKIM records are properly configured in both Mailgun and Mailu +3. Verify DMARC record is set +4. Check your domain reputation at [mail-tester.com](https://www.mail-tester.com) + +--- + +## Nominatim Geocoding Service + +Nominatim provides geocoding (address to coordinates) and reverse geocoding for delivery and distribution features. + +### When to Deploy + +Deploy Nominatim if you need: +- Address autocomplete in the frontend +- Delivery route optimization +- Location-based analytics + +### Step 1: Deploy Nominatim via Helm + +```bash +# Deploy Nominatim with production values +helm upgrade --install nominatim infrastructure/platform/nominatim/nominatim-helm \ + -n bakery-ia \ + --create-namespace \ + -f infrastructure/platform/nominatim/nominatim-helm/values.yaml \ + -f infrastructure/platform/nominatim/nominatim-helm/prod/values.yaml \ + --timeout 15m \ + --wait + +# Verify deployment +kubectl get pods -n bakery-ia | grep nominatim +``` + +**Note:** Initial deployment may take 10-15 minutes as Nominatim downloads and processes geographic data. + +### Step 2: Verify Nominatim Service + +```bash +# Check pod status +kubectl get pods -n bakery-ia -l app=nominatim + +# Check service +kubectl get svc -n bakery-ia | grep nominatim + +# Test geocoding endpoint +kubectl run -it --rm curl-test --image=curlimages/curl --restart=Never -- \ + curl "http://nominatim-service.bakery-ia.svc.cluster.local:8080/search?q=Madrid&format=json" +``` + +### Step 3: Configure Application to Use Nominatim + +Update the application ConfigMap to use the internal Nominatim service: + +```bash +# Edit configmap +kubectl edit configmap bakery-ia-config -n bakery-ia + +# Set: +# NOMINATIM_URL: "http://nominatim-service.bakery-ia.svc.cluster.local:8080" +``` + +### Nominatim Service Information + +| Property | Value | +|----------|-------| +| Service Name | nominatim-service.bakery-ia.svc.cluster.local | +| Port | 8080 | +| Health Check | http://nominatim-service:8080/status | +| Search Endpoint | /search?q={query}&format=json | +| Reverse Endpoint | /reverse?lat={lat}&lon={lon}&format=json | + +--- + +## SigNoz Monitoring Deployment + +SigNoz provides unified observability (traces, metrics, logs) for the entire platform. + +### Step 1: Deploy SigNoz via Helm + +```bash +# Add SigNoz Helm repository +helm repo add signoz https://charts.signoz.io +helm repo update signoz + +# Deploy SigNoz into bakery-ia namespace +helm upgrade --install signoz signoz/signoz \ + -n bakery-ia \ + -f infrastructure/monitoring/signoz/signoz-values-prod.yaml \ + --set frontend.service.type=ClusterIP \ + --set clickhouse.persistence.size=20Gi \ + --set clickhouse.persistence.storageClass=microk8s-hostpath \ + --timeout 15m \ + --wait + +# Wait for all components (may take 10-15 minutes) +kubectl wait --for=condition=ready pod \ + -l app.kubernetes.io/instance=signoz \ + -n bakery-ia \ + --timeout=900s + +# Verify deployment +kubectl get pods -n bakery-ia -l app.kubernetes.io/instance=signoz +``` + +### Step 2: Configure Ingress for SigNoz + +```bash +# Apply SigNoz ingress (if not already included in overlays) +cat < /dev/null 2>&1; do + RETRIES=$((RETRIES + 1)) + if [ $RETRIES -ge $MAX_RETRIES ]; then + echo "ERROR: Gitea did not become ready after $MAX_RETRIES attempts" + exit 1 + fi + echo " Attempt $RETRIES/$MAX_RETRIES - Gitea not ready, waiting ${RETRY_INTERVAL}s..." + sleep $RETRY_INTERVAL + done + echo "Gitea is ready!" + + # Check if repository already exists + echo "" + echo "Checking if repository '$REPO_NAME' exists..." + HTTP_CODE=$(curl -s -o /dev/null -w "%{http_code}" \ + -u "$GITEA_ADMIN_USER:$GITEA_ADMIN_PASSWORD" \ + "$GITEA_URL/api/v1/repos/$GITEA_ADMIN_USER/$REPO_NAME") + + if [ "$HTTP_CODE" = "200" ]; then + echo "Repository '$REPO_NAME' already exists. Nothing to do." + exit 0 + fi + + # Create the repository + echo "Creating repository '$REPO_NAME'..." + RESPONSE=$(curl -s -w "\n%{http_code}" \ + -u "$GITEA_ADMIN_USER:$GITEA_ADMIN_PASSWORD" \ + -X POST "$GITEA_URL/api/v1/user/repos" \ + -H "Content-Type: application/json" \ + -d '{ + "name": "'"$REPO_NAME"'", + "description": "Main repository for Bakery IA project - Automatically created", + "private": false, + "auto_init": true, + "default_branch": "main", + "readme": "Default" + }') + + HTTP_CODE=$(echo "$RESPONSE" | tail -1) + BODY=$(echo "$RESPONSE" | sed '$d') + + if [ "$HTTP_CODE" = "201" ]; then + echo "Repository '$REPO_NAME' created successfully!" + echo "" + echo "Repository URL: $GITEA_URL/$GITEA_ADMIN_USER/$REPO_NAME" + echo "Clone URL: $GITEA_URL/$GITEA_ADMIN_USER/$REPO_NAME.git" + else + echo "ERROR: Failed to create repository (HTTP $HTTP_CODE)" + echo "Response: $BODY" + exit 1 + fi + + # Configure webhook for Tekton (optional - if Tekton is installed) + echo "" + echo "Checking if Tekton EventListener is available..." + TEKTON_URL="http://el-bakery-ia-listener.tekton-pipelines.svc.cluster.local:8080" + if curl -sf "$TEKTON_URL" > /dev/null 2>&1; then + echo "Tekton EventListener found. Creating webhook..." + WEBHOOK_RESPONSE=$(curl -s -w "\n%{http_code}" \ + -u "$GITEA_ADMIN_USER:$GITEA_ADMIN_PASSWORD" \ + -X POST "$GITEA_URL/api/v1/repos/$GITEA_ADMIN_USER/$REPO_NAME/hooks" \ + -H "Content-Type: application/json" \ + -d '{ + "type": "gitea", + "config": { + "url": "'"$TEKTON_URL"'", + "content_type": "json" + }, + "events": ["push"], + "active": true + }') + + WEBHOOK_CODE=$(echo "$WEBHOOK_RESPONSE" | tail -1) + if [ "$WEBHOOK_CODE" = "201" ]; then + echo "Webhook created successfully!" + else + echo "Warning: Could not create webhook (HTTP $WEBHOOK_CODE). You may need to configure it manually." + fi + else + echo "Tekton EventListener not available. Skipping webhook creation." + fi + + echo "" + echo "=== Initialization Complete ===" +--- +apiVersion: batch/v1 +kind: Job +metadata: + name: gitea-init-repo + namespace: gitea + labels: + app.kubernetes.io/name: gitea + app.kubernetes.io/component: init + annotations: + # Helm hook annotations (if used with Helm) + helm.sh/hook: post-install,post-upgrade + helm.sh/hook-weight: "10" + helm.sh/hook-delete-policy: before-hook-creation +spec: + ttlSecondsAfterFinished: 300 + backoffLimit: 3 + template: + metadata: + labels: + app.kubernetes.io/name: gitea + app.kubernetes.io/component: init + spec: + restartPolicy: OnFailure + containers: + - name: init-repo + image: curlimages/curl:8.5.0 + command: ["/bin/sh", "/scripts/init-repo.sh"] + env: + - name: GITEA_ADMIN_USER + valueFrom: + secretKeyRef: + name: gitea-admin-secret + key: username + - name: GITEA_ADMIN_PASSWORD + valueFrom: + secretKeyRef: + name: gitea-admin-secret + key: password + volumeMounts: + - name: init-script + mountPath: /scripts + resources: + limits: + cpu: 100m + memory: 64Mi + requests: + cpu: 50m + memory: 32Mi + volumes: + - name: init-script + configMap: + name: gitea-init-script + defaultMode: 0755 diff --git a/infrastructure/cicd/gitea/setup-admin-secret.sh b/infrastructure/cicd/gitea/setup-admin-secret.sh index 842b2c05..ef12f9b6 100755 --- a/infrastructure/cicd/gitea/setup-admin-secret.sh +++ b/infrastructure/cicd/gitea/setup-admin-secret.sh @@ -1,41 +1,86 @@ #!/bin/bash -# Setup Gitea Admin Secret +# Setup Gitea Admin Secret and Initialize Gitea # -# This script creates TWO Kubernetes secrets: -# 1. gitea-admin-secret (gitea namespace) - Used by Gitea Helm chart for admin credentials -# 2. gitea-registry-secret (bakery-ia namespace) - Used by pods for imagePullSecrets -# -# Both secrets use the SAME credentials, ensuring consistency. +# This script: +# 1. Creates gitea-admin-secret (gitea namespace) - Used by Gitea Helm chart for admin credentials +# 2. Creates gitea-registry-secret (bakery-ia namespace) - Used by pods for imagePullSecrets +# 3. Applies the gitea-init-job.yaml to create the initial repository # # Usage: -# ./setup-admin-secret.sh [password] +# Development: +# ./setup-admin-secret.sh # Uses default dev password +# ./setup-admin-secret.sh [password] # Uses provided password +# ./setup-admin-secret.sh --secrets-only # Only create secrets, skip init job # -# If password is not provided, a random one will be generated. +# Production: +# export GITEA_ADMIN_PASSWORD=$(openssl rand -base64 32) +# ./setup-admin-secret.sh --production +# ./setup-admin-secret.sh --production --secrets-only +# +# Environment variables: +# GITEA_ADMIN_PASSWORD - Password to use (required for --production) set -e +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" KUBECTL="kubectl" GITEA_NAMESPACE="gitea" BAKERY_NAMESPACE="bakery-ia" REGISTRY_HOST="registry.bakery-ia.local" ADMIN_USERNAME="bakery-admin" -# Static password for consistent dev environment setup -# This ensures the same credentials work across environment recreations -STATIC_ADMIN_PASSWORD="pvYUkGWJijqc0QfIZEXw" +# Default password for dev environment only +# For PRODUCTION: Always set GITEA_ADMIN_PASSWORD environment variable +# Generate secure password with: openssl rand -base64 32 +DEV_DEFAULT_PASSWORD="pvYUkGWJijqc0QfIZEXw" +SECRETS_ONLY=false +IS_PRODUCTION=false # Check if running in microk8s if command -v microk8s &> /dev/null; then KUBECTL="microk8s kubectl" fi -# Get password from argument, environment variable, or use static default -if [ -n "$1" ]; then - ADMIN_PASSWORD="$1" -elif [ -n "$GITEA_ADMIN_PASSWORD" ]; then - ADMIN_PASSWORD="$GITEA_ADMIN_PASSWORD" -else - ADMIN_PASSWORD="$STATIC_ADMIN_PASSWORD" - echo "Using static admin password for dev environment consistency" +# Parse arguments +for arg in "$@"; do + case $arg in + --secrets-only) + SECRETS_ONLY=true + ;; + --production) + IS_PRODUCTION=true + REGISTRY_HOST="registry.bakewise.ai" + ;; + *) + if [ -z "$ADMIN_PASSWORD" ] && [ "$arg" != "--secrets-only" ] && [ "$arg" != "--production" ]; then + ADMIN_PASSWORD="$arg" + fi + ;; + esac +done + +# Get password from argument, environment variable, or use default (dev only) +if [ -z "$ADMIN_PASSWORD" ]; then + if [ -n "$GITEA_ADMIN_PASSWORD" ]; then + ADMIN_PASSWORD="$GITEA_ADMIN_PASSWORD" + echo "Using password from GITEA_ADMIN_PASSWORD environment variable" + elif [ "$IS_PRODUCTION" = true ]; then + echo "ERROR: Production deployment requires GITEA_ADMIN_PASSWORD environment variable" + echo "Generate a secure password with: openssl rand -base64 32" + echo "" + echo "Usage for production:" + echo " export GITEA_ADMIN_PASSWORD=\$(openssl rand -base64 32)" + echo " ./setup-admin-secret.sh --production" + exit 1 + else + ADMIN_PASSWORD="$DEV_DEFAULT_PASSWORD" + echo "WARNING: Using default dev password. For production, set GITEA_ADMIN_PASSWORD" + fi +fi + +# Validate password strength for production +if [ "$IS_PRODUCTION" = true ] && [ ${#ADMIN_PASSWORD} -lt 16 ]; then + echo "ERROR: Production password must be at least 16 characters" + exit 1 fi # Create namespaces if they don't exist @@ -103,9 +148,15 @@ echo "==========================================" echo "Gitea secrets created successfully!" echo "==========================================" echo "" -echo "Credentials (same for both secrets):" +echo "Environment: $([ "$IS_PRODUCTION" = true ] && echo "PRODUCTION" || echo "Development")" +echo "" +echo "Credentials:" echo " Username: $ADMIN_USERNAME" -echo " Password: $ADMIN_PASSWORD" +if [ "$IS_PRODUCTION" = true ]; then + echo " Password: (stored in secret, not displayed for security)" +else + echo " Password: $ADMIN_PASSWORD" +fi echo "" echo "Secrets created:" echo " 1. gitea-admin-secret (namespace: $GITEA_NAMESPACE) - For Gitea Helm chart" @@ -115,5 +166,44 @@ echo "Registry URLs:" echo " External: https://$REGISTRY_HOST" echo " Internal: $INTERNAL_REGISTRY_HOST" echo "" -echo "Now install Gitea with:" -echo " helm install gitea gitea/gitea -n gitea -f infrastructure/cicd/gitea/values.yaml" + +# Apply the init job ConfigMap and Job (but Job won't run until Gitea is installed) +if [ "$SECRETS_ONLY" = false ]; then + INIT_JOB_FILE="$SCRIPT_DIR/gitea-init-job.yaml" + if [ -f "$INIT_JOB_FILE" ]; then + echo "Applying Gitea initialization resources..." + $KUBECTL apply -f "$INIT_JOB_FILE" + echo "" + echo "Init job will create the 'bakery-ia' repository once Gitea is ready." + else + echo "Warning: gitea-init-job.yaml not found at $INIT_JOB_FILE" + fi + echo "" +fi + +echo "Next steps:" +if [ "$IS_PRODUCTION" = true ]; then + echo " 1. Install Gitea for production:" + echo " helm upgrade --install gitea gitea/gitea -n gitea \\" + echo " -f infrastructure/cicd/gitea/values.yaml \\" + echo " -f infrastructure/cicd/gitea/values-prod.yaml" + echo "" + echo " 2. Install Tekton CI/CD for production:" + echo " export TEKTON_WEBHOOK_TOKEN=\$(openssl rand -hex 32)" + echo " helm upgrade --install tekton-cicd infrastructure/cicd/tekton-helm \\" + echo " -n tekton-pipelines \\" + echo " -f infrastructure/cicd/tekton-helm/values.yaml \\" + echo " -f infrastructure/cicd/tekton-helm/values-prod.yaml \\" + echo " --set secrets.webhook.token=\$TEKTON_WEBHOOK_TOKEN \\" + echo " --set secrets.registry.password=\$GITEA_ADMIN_PASSWORD \\" + echo " --set secrets.git.password=\$GITEA_ADMIN_PASSWORD" +else + echo " 1. Install Gitea (if not already installed):" + echo " helm install gitea gitea/gitea -n gitea -f infrastructure/cicd/gitea/values.yaml" +fi +echo "" +echo " $([ "$IS_PRODUCTION" = true ] && echo "3" || echo "2"). Wait for Gitea to be ready:" +echo " kubectl wait --for=condition=ready pod -n gitea -l app.kubernetes.io/name=gitea --timeout=300s" +echo "" +echo " $([ "$IS_PRODUCTION" = true ] && echo "4" || echo "3"). Check init job status:" +echo " kubectl logs -n gitea -l app.kubernetes.io/component=init --tail=50" diff --git a/infrastructure/cicd/gitea/values.yaml b/infrastructure/cicd/gitea/values.yaml index e4326be2..23feb788 100644 --- a/infrastructure/cicd/gitea/values.yaml +++ b/infrastructure/cicd/gitea/values.yaml @@ -1,9 +1,12 @@ # Gitea Helm values configuration for Bakery-IA CI/CD # This configuration sets up Gitea with registry support and appropriate storage # +# Prerequisites: +# 1. Run setup-admin-secret.sh to create the gitea-admin-secret +# 2. Apply the post-install job: kubectl apply -f gitea-init-job.yaml +# # Installation: # helm repo add gitea https://dl.gitea.io/charts -# kubectl create namespace gitea # helm install gitea gitea/gitea -n gitea -f infrastructure/cicd/gitea/values.yaml # # NOTE: The namespace is determined by the -n flag during helm install, not in this file. @@ -43,22 +46,6 @@ ingress: hosts: - gitea.bakery-ia.local - registry.bakery-ia.local - # Additional ingress for container registry (same backend, different hostname) - apiIngress: - enabled: true - className: nginx - annotations: - nginx.ingress.kubernetes.io/ssl-redirect: "true" - nginx.ingress.kubernetes.io/proxy-body-size: "500m" - hosts: - - host: registry.bakery-ia.local - paths: - - path: / - pathType: Prefix - tls: - - secretName: bakery-dev-tls-cert - hosts: - - registry.bakery-ia.local persistence: enabled: true @@ -68,39 +55,44 @@ persistence: # For Kind: leave empty or use "standard" storageClass: "" +# ============================================================================= +# ADMIN USER CONFIGURATION +# ============================================================================= +# The admin user is automatically created on first install. +# Credentials are read from the 'gitea-admin-secret' Kubernetes secret. +# +# Create the secret BEFORE installing Gitea: +# ./setup-admin-secret.sh +# +# The secret must contain: +# - username: admin username (default: bakery-admin) +# - password: admin password +# ============================================================================= gitea: admin: username: bakery-admin - # IMPORTANT: Override this with --set gitea.admin.password= - # or use existingSecret - password: "" email: admin@bakery-ia.local + # Use existing secret for admin credentials (created by setup-admin-secret.sh) existingSecret: gitea-admin-secret + # keepUpdated ensures password changes in secret are applied + passwordMode: keepUpdated + config: server: DOMAIN: gitea.bakery-ia.local SSH_DOMAIN: gitea.bakery-ia.local + SSH_PORT: 2222 # Use HTTPS for external access; TLS termination happens at ingress ROOT_URL: https://gitea.bakery-ia.local HTTP_PORT: 3000 - # Enable package registry - PACKAGES_ENABLED: true # Disable built-in HTTPS since ingress handles TLS PROTOCOL: http repository: ENABLE_PUSH_CREATE_USER: true ENABLE_PUSH_CREATE_ORG: true + DEFAULT_BRANCH: main packages: ENABLED: true - registry: - ENABLE: true - ROOT: /var/lib/gitea-registry - STORAGE_TYPE: local - # NOTE: PORT config here is internal - registry is accessed via HTTP port on /v2/ path - # Additional registry configuration for proper external access - docker: - ENABLE: true - REGISTRY_SSL_REDIRECT: false # SSL termination happens at ingress webhook: ALLOWED_HOST_LIST: "*" # Allow internal cluster URLs for Tekton EventListener @@ -109,21 +101,6 @@ gitea: DISABLE_REGISTRATION: false REQUIRE_SIGNIN_VIEW: false - # Initial repositories to create automatically after Gitea installation - # These will be created with the admin user as owner - initialRepositories: - - name: bakery-ia - description: "Main repository for Bakery IA project - Automatically created by Helm" - private: false - auto_init: true - default_branch: main - owner: "{{ .Values.gitea.admin.username }}" - # Enable issues, wiki, and other features - enable_issues: true - enable_wiki: true - enable_pull_requests: true - enable_projects: true - # Use embedded SQLite for simpler local development # For production, enable postgresql postgresql: diff --git a/infrastructure/cicd/tekton-helm/README.md b/infrastructure/cicd/tekton-helm/README.md index 742be0d8..9b059cb4 100644 --- a/infrastructure/cicd/tekton-helm/README.md +++ b/infrastructure/cicd/tekton-helm/README.md @@ -18,10 +18,30 @@ kubectl apply -f https://storage.googleapis.com/tekton-releases/pipeline/latest/ Then install the chart: +### Development Installation + ```bash -helm repo add tekton-pipelines https://tekton.dev/charts -helm repo update -helm install tekton-cicd infrastructure/helm/tekton --namespace tekton-pipelines --create-namespace +helm install tekton-cicd infrastructure/cicd/tekton-helm \ + --namespace tekton-pipelines \ + --create-namespace +``` + +### Production Installation + +**Important**: Never use default secrets in production. Always provide secure credentials. + +```bash +# Generate secure webhook token +export TEKTON_WEBHOOK_TOKEN=$(openssl rand -hex 32) + +# Use the same password as Gitea admin (from GITEA_ADMIN_PASSWORD) +helm upgrade --install tekton-cicd infrastructure/cicd/tekton-helm \ + -n tekton-pipelines \ + -f infrastructure/cicd/tekton-helm/values.yaml \ + -f infrastructure/cicd/tekton-helm/values-prod.yaml \ + --set secrets.webhook.token=$TEKTON_WEBHOOK_TOKEN \ + --set secrets.registry.password=$GITEA_ADMIN_PASSWORD \ + --set secrets.git.password=$GITEA_ADMIN_PASSWORD ``` ## Configuration diff --git a/infrastructure/cicd/tekton-helm/templates/task-update-gitops.yaml b/infrastructure/cicd/tekton-helm/templates/task-update-gitops.yaml index f38ffdfc..ffc82adb 100644 --- a/infrastructure/cicd/tekton-helm/templates/task-update-gitops.yaml +++ b/infrastructure/cicd/tekton-helm/templates/task-update-gitops.yaml @@ -65,7 +65,8 @@ spec: git config --global user.name "bakery-ia-ci" # Clone the main repository (not a separate gitops repo) - REPO_URL="https://${GIT_USERNAME}:${GIT_PASSWORD}@gitea.bakery-ia.local/bakery-admin/bakery-ia.git" + # Use internal cluster DNS which works in all environments + REPO_URL="https://${GIT_USERNAME}:${GIT_PASSWORD}@gitea-http.gitea.svc.cluster.local:3000/bakery-admin/bakery-ia.git" git clone "$REPO_URL" /tmp/gitops cd /tmp/gitops diff --git a/infrastructure/cicd/tekton-helm/values-prod.yaml b/infrastructure/cicd/tekton-helm/values-prod.yaml new file mode 100644 index 00000000..eefbf06b --- /dev/null +++ b/infrastructure/cicd/tekton-helm/values-prod.yaml @@ -0,0 +1,81 @@ +# Production values for tekton-cicd Helm chart +# This file overrides values.yaml for production deployment +# +# Installation: +# helm upgrade --install tekton-cicd infrastructure/cicd/tekton-helm \ +# -n tekton-pipelines \ +# -f infrastructure/cicd/tekton-helm/values.yaml \ +# -f infrastructure/cicd/tekton-helm/values-prod.yaml \ +# --set secrets.webhook.token=$TEKTON_WEBHOOK_TOKEN \ +# --set secrets.registry.password=$GITEA_ADMIN_PASSWORD \ +# --set secrets.git.password=$GITEA_ADMIN_PASSWORD +# +# Required environment variables: +# TEKTON_WEBHOOK_TOKEN - Secure webhook token (generate with: openssl rand -hex 32) +# GITEA_ADMIN_PASSWORD - Gitea admin password (must match gitea-admin-secret) + +# Global settings for production +global: + # Git configuration + git: + userEmail: "ci@bakewise.ai" + +# Pipeline configuration for production +pipeline: + # Build configuration + build: + verbosity: "warn" # Less verbose in production + + # Test configuration + test: + skipTests: "false" + skipLint: "false" + + # Workspace configuration - ensure storage class exists in production cluster + workspace: + size: "10Gi" + storageClass: "standard" # Adjust to your production storage class + +# Tekton controller settings - increased resources for production +controller: + replicas: 2 + resources: + limits: + cpu: 2000m + memory: 2Gi + requests: + cpu: 200m + memory: 256Mi + +# Tekton webhook settings - increased resources for production +webhook: + replicas: 2 + resources: + limits: + cpu: 1000m + memory: 1Gi + requests: + cpu: 100m + memory: 128Mi + +# Secrets configuration +# IMPORTANT: These MUST be overridden via --set flags during deployment +# DO NOT commit actual secrets to this file +secrets: + # Webhook secret for validating incoming webhooks + # Override with: --set secrets.webhook.token=$TEKTON_WEBHOOK_TOKEN + webhook: + token: "" # MUST be set via --set flag + + # Registry credentials for pushing images + # Override with: --set secrets.registry.password=$GITEA_ADMIN_PASSWORD + registry: + username: "bakery-admin" + password: "" # MUST be set via --set flag + registryUrl: "gitea-http.gitea.svc.cluster.local:3000" + + # Git credentials for GitOps updates + # Override with: --set secrets.git.password=$GITEA_ADMIN_PASSWORD + git: + username: "bakery-admin" + password: "" # MUST be set via --set flag diff --git a/infrastructure/platform/mail/mailu-helm/configs/mailgun-credentials-secret.yaml b/infrastructure/platform/mail/mailu-helm/configs/mailgun-credentials-secret.yaml index c52c7813..cb59b792 100644 --- a/infrastructure/platform/mail/mailu-helm/configs/mailgun-credentials-secret.yaml +++ b/infrastructure/platform/mail/mailu-helm/configs/mailgun-credentials-secret.yaml @@ -3,33 +3,61 @@ # This secret stores Mailgun credentials for outbound email relay. # Mailu uses Mailgun as an external SMTP relay to send all outbound emails. # +# ============================================================================ # HOW TO CONFIGURE: +# ============================================================================ +# # 1. Go to https://www.mailgun.com and create an account -# 2. Add and verify your domain (e.g., bakery-ia.dev or bakewise.ai) -# 3. Go to Domain Settings > SMTP credentials +# +# 2. Add and verify your domain: +# - For dev: bakery-ia.dev +# - For prod: bakewise.ai +# +# 3. Go to Domain Settings > SMTP credentials in Mailgun dashboard +# # 4. Note your SMTP credentials: # - SMTP hostname: smtp.mailgun.org -# - Port: 587 (TLS) -# - Username: usually postmaster@yourdomain.com -# - Password: your Mailgun SMTP password (NOT API key) -# 5. Base64 encode your password: +# - Port: 587 (TLS/STARTTLS) +# - Username: typically postmaster@yourdomain.com +# - Password: your Mailgun SMTP password (NOT the API key) +# +# 5. Base64 encode your credentials: +# echo -n 'postmaster@bakewise.ai' | base64 # echo -n 'your-mailgun-smtp-password' | base64 -# 6. Replace MAILGUN_SMTP_PASSWORD_BASE64 below with the encoded value +# +# 6. Replace the placeholder values below with your encoded credentials +# # 7. Apply this secret: # kubectl apply -f mailgun-credentials-secret.yaml -n bakery-ia # +# ============================================================================ # IMPORTANT NOTES: -# - Use the SMTP password from Mailgun, not the API key -# - The username is typically postmaster@yourdomain.com -# - For sandbox domains, Mailgun requires authorized recipients -# - Production domains need DNS verification (SPF, DKIM, MX records) +# ============================================================================ # +# - Use the SMTP password from Mailgun, NOT the API key +# - The username format is: postmaster@yourdomain.com +# - For sandbox domains, Mailgun requires adding authorized recipients +# - Production domains need DNS verification (SPF, DKIM records) +# +# ============================================================================ # DNS RECORDS REQUIRED FOR MAILGUN: -# You will need to add these DNS records for your domain: -# - SPF: TXT record for email authentication -# - DKIM: TXT records for email signing (Mailgun provides these) -# - MX: If you want to receive emails via Mailgun (optional for relay-only) +# ============================================================================ # +# Add these DNS records to your domain for proper email delivery: +# +# 1. SPF Record (TXT): +# Name: @ +# Value: v=spf1 include:mailgun.org ~all +# +# 2. DKIM Records (TXT): +# Mailgun will provide two DKIM keys to add as TXT records +# (check your Mailgun domain settings for exact values) +# +# 3. MX Records (optional, only if receiving via Mailgun): +# Priority 10: mxa.mailgun.org +# Priority 10: mxb.mailgun.org +# +# ============================================================================ --- apiVersion: v1 kind: Secret @@ -39,39 +67,28 @@ metadata: labels: app: mailu component: external-relay + annotations: + description: "Mailgun SMTP credentials for Mailu external relay" type: Opaque -data: - # Base64 encoded Mailgun SMTP password - # To encode: echo -n 'your-password' | base64 - # To decode: echo 'encoded-value' | base64 -d - RELAY_PASSWORD: MAILGUN_SMTP_PASSWORD_BASE64 ---- -# Development environment secret (separate for different Mailgun domain) -apiVersion: v1 -kind: Secret -metadata: - name: mailu-mailgun-credentials-dev - namespace: bakery-ia - labels: - app: mailu - component: external-relay - environment: dev -type: Opaque -data: - # Mailgun credentials for bakery-ia.dev domain - RELAY_PASSWORD: MAILGUN_DEV_SMTP_PASSWORD_BASE64 ---- -# Production environment secret -apiVersion: v1 -kind: Secret -metadata: - name: mailu-mailgun-credentials-prod - namespace: bakery-ia - labels: - app: mailu - component: external-relay - environment: prod -type: Opaque -data: - # Mailgun credentials for bakewise.ai domain - RELAY_PASSWORD: MAILGUN_PROD_SMTP_PASSWORD_BASE64 +stringData: + # ============================================================================ + # REPLACE THESE VALUES WITH YOUR MAILGUN CREDENTIALS + # ============================================================================ + # + # Option 1: Use stringData (plain text - Kubernetes will encode automatically) + # This is easier for initial setup but shows credentials in the file + # + RELAY_USERNAME: "postmaster@sandboxc1bff891532b4f0c83056a68ae080b4c.mailgun.org" + RELAY_PASSWORD: "2e47104abadad8eb820d00042ea6d5eb-77c6c375-89c7ea55" + # + # ============================================================================ + # ALTERNATIVE: Use pre-encoded values (more secure for version control) + # ============================================================================ + # Comment out stringData above and uncomment data below: + # + # data: + # # Base64 encoded values + # # echo -n 'postmaster@bakewise.ai' | base64 + # RELAY_USERNAME: cG9zdG1hc3RlckBiYWtld2lzZS5haQ== + # # echo -n 'your-password' | base64 + # RELAY_PASSWORD: WU9VUl9NQUlMR1VOX1NNVFBfUEFTU1dPUkQ= diff --git a/infrastructure/platform/mail/mailu-helm/configs/mailu-admin-credentials-secret.yaml b/infrastructure/platform/mail/mailu-helm/configs/mailu-admin-credentials-secret.yaml new file mode 100644 index 00000000..88fa149f --- /dev/null +++ b/infrastructure/platform/mail/mailu-helm/configs/mailu-admin-credentials-secret.yaml @@ -0,0 +1,34 @@ +# Mailu Admin Credentials Secret +# This secret stores the initial admin account password for Mailu +# +# The password is used by the Helm chart's initialAccount feature to create +# the admin user automatically during deployment. +# +# IMPORTANT: Replace the base64-encoded password before applying! +# +# To generate a secure password and encode it: +# PASSWORD=$(openssl rand -base64 16 | tr -d '/+=' | head -c 16) +# echo -n "$PASSWORD" | base64 +# +# To apply this secret: +# kubectl apply -f mailu-admin-credentials-secret.yaml -n bakery-ia +# +# After deployment, you can log in to the Mailu admin panel at: +# https://mail./admin +# Username: admin@ +# Password: +# +apiVersion: v1 +kind: Secret +metadata: + name: mailu-admin-credentials + namespace: bakery-ia + labels: + app.kubernetes.io/name: mailu + app.kubernetes.io/component: admin +type: Opaque +data: + # Base64-encoded password + # Example: "changeme123" = Y2hhbmdlbWUxMjM= + # IMPORTANT: Replace with your own secure password! + password: "Y2hhbmdlbWUxMjM=" diff --git a/infrastructure/platform/mail/mailu-helm/dev/values.yaml b/infrastructure/platform/mail/mailu-helm/dev/values.yaml index 3cdb207b..955e8349 100644 --- a/infrastructure/platform/mail/mailu-helm/dev/values.yaml +++ b/infrastructure/platform/mail/mailu-helm/dev/values.yaml @@ -36,17 +36,29 @@ domain: "bakery-ia.dev" hostnames: - "mail.bakery-ia.dev" +# Initial admin account for dev environment +# Password is stored in mailu-admin-credentials secret +initialAccount: + enabled: true + username: "admin" + domain: "bakery-ia.dev" + existingSecret: "mailu-admin-credentials" + existingSecretPasswordKey: "password" + mode: "ifmissing" + # External relay configuration for dev (Mailgun) # All outbound emails will be relayed through Mailgun SMTP # To configure: # 1. Register at mailgun.com and verify your domain (bakery-ia.dev) # 2. Get your SMTP credentials from Mailgun dashboard # 3. Update the secret in configs/mailgun-credentials-secret.yaml -# 4. Apply the secret: kubectl apply -f configs/mailgun-credentials-secret.yaml +# 4. Apply the secret: kubectl apply -f configs/mailgun-credentials-secret.yaml -n bakery-ia externalRelay: host: "[smtp.mailgun.org]:587" - username: "postmaster@bakery-ia.dev" # Your Mailgun SMTP username (usually postmaster@yourdomain) - password: "" # Will be loaded from secret - see configs/mailgun-credentials-secret.yaml + # Credentials loaded from Kubernetes secret + secretName: "mailu-mailgun-credentials" + usernameKey: "RELAY_USERNAME" + passwordKey: "RELAY_PASSWORD" # Environment-specific configurations persistence: @@ -92,6 +104,13 @@ resources: limits: cpu: "200m" memory: "128Mi" + webmail: + requests: + cpu: "50m" + memory: "64Mi" + limits: + cpu: "200m" + memory: "128Mi" clamav: requests: cpu: "100m" diff --git a/infrastructure/platform/mail/mailu-helm/prod/values.yaml b/infrastructure/platform/mail/mailu-helm/prod/values.yaml index 18b58ef1..1f852ce1 100644 --- a/infrastructure/platform/mail/mailu-helm/prod/values.yaml +++ b/infrastructure/platform/mail/mailu-helm/prod/values.yaml @@ -21,17 +21,29 @@ domain: "bakewise.ai" hostnames: - "mail.bakewise.ai" +# Initial admin account for production environment +# Password is stored in mailu-admin-credentials secret +initialAccount: + enabled: true + username: "admin" + domain: "bakewise.ai" + existingSecret: "mailu-admin-credentials" + existingSecretPasswordKey: "password" + mode: "ifmissing" + # External relay configuration for production (Mailgun) # All outbound emails will be relayed through Mailgun SMTP # To configure: # 1. Register at mailgun.com and verify your domain (bakewise.ai) # 2. Get your SMTP credentials from Mailgun dashboard # 3. Update the secret in configs/mailgun-credentials-secret.yaml -# 4. Apply the secret: kubectl apply -f configs/mailgun-credentials-secret.yaml +# 4. Apply the secret: kubectl apply -f configs/mailgun-credentials-secret.yaml -n bakery-ia externalRelay: host: "[smtp.mailgun.org]:587" - username: "postmaster@bakewise.ai" # Your Mailgun SMTP username - password: "" # Will be loaded from secret - see configs/mailgun-credentials-secret.yaml + # Credentials loaded from Kubernetes secret + secretName: "mailu-mailgun-credentials" + usernameKey: "RELAY_USERNAME" + passwordKey: "RELAY_PASSWORD" # Environment-specific configurations persistence: diff --git a/infrastructure/platform/mail/mailu-helm/scripts/deploy-mailu-prod.sh b/infrastructure/platform/mail/mailu-helm/scripts/deploy-mailu-prod.sh index b7a78017..876b764b 100755 --- a/infrastructure/platform/mail/mailu-helm/scripts/deploy-mailu-prod.sh +++ b/infrastructure/platform/mail/mailu-helm/scripts/deploy-mailu-prod.sh @@ -7,8 +7,8 @@ # 1. Unbound DNS deployment (for DNSSEC validation) # 2. CoreDNS configuration (forward to Unbound) # 3. TLS certificate secret creation -# 4. Mailu Helm deployment -# 5. Admin user creation +# 4. Admin credentials secret creation +# 5. Mailu Helm deployment (admin user created automatically via initialAccount) # # Usage: # ./deploy-mailu-prod.sh [--domain DOMAIN] [--admin-password PASSWORD] @@ -174,9 +174,35 @@ else fi # ============================================================================= -# Step 4: Deploy Mailu via Helm +# Step 4: Create Admin Credentials Secret # ============================================================================= -print_step "Step 4: Deploying Mailu via Helm..." +print_step "Step 4: Creating admin credentials secret..." + +if kubectl get secret mailu-admin-credentials -n "$NAMESPACE" &>/dev/null; then + print_success "Admin credentials secret already exists" + # Retrieve existing password for summary output + if [ -z "$ADMIN_PASSWORD" ]; then + ADMIN_PASSWORD=$(kubectl get secret mailu-admin-credentials -n "$NAMESPACE" -o jsonpath='{.data.password}' | base64 -d) + fi +else + if [ -z "$ADMIN_PASSWORD" ]; then + # Generate a random password + ADMIN_PASSWORD=$(openssl rand -base64 16 | tr -d '/+=' | head -c 16) + echo -e "${YELLOW}Generated admin password: $ADMIN_PASSWORD${NC}" + echo -e "${YELLOW}Please save this password securely!${NC}" + fi + + kubectl create secret generic mailu-admin-credentials \ + --from-literal=password="$ADMIN_PASSWORD" \ + -n "$NAMESPACE" + + print_success "Admin credentials secret created" +fi + +# ============================================================================= +# Step 5: Deploy Mailu via Helm +# ============================================================================= +print_step "Step 5: Deploying Mailu via Helm..." # Add Mailu Helm repository helm repo add mailu https://mailu.github.io/helm-charts 2>/dev/null || true @@ -189,12 +215,12 @@ helm upgrade --install mailu mailu/mailu \ -f "$MAILU_HELM_DIR/prod/values.yaml" \ --timeout 10m -print_success "Mailu Helm release deployed" +print_success "Mailu Helm release deployed (admin user will be created automatically)" # ============================================================================= -# Step 5: Wait for Pods to be Ready +# Step 6: Wait for Pods to be Ready # ============================================================================= -print_step "Step 5: Waiting for Mailu pods to be ready..." +print_step "Step 6: Waiting for Mailu pods to be ready..." echo "This may take 5-10 minutes (ClamAV takes time to initialize)..." @@ -212,24 +238,7 @@ echo "" echo "Mailu Pod Status:" kubectl get pods -n "$NAMESPACE" | grep mailu -# ============================================================================= -# Step 6: Create Admin User -# ============================================================================= -print_step "Step 6: Creating admin user..." - -if [ -z "$ADMIN_PASSWORD" ]; then - # Generate a random password - ADMIN_PASSWORD=$(openssl rand -base64 16 | tr -d '/+=' | head -c 16) - echo -e "${YELLOW}Generated admin password: $ADMIN_PASSWORD${NC}" - echo -e "${YELLOW}Please save this password securely!${NC}" -fi - -kubectl exec -n "$NAMESPACE" deployment/mailu-admin -- \ - flask mailu admin admin "$DOMAIN" "$ADMIN_PASSWORD" 2>/dev/null || { - print_warning "Admin user may already exist or failed to create" -} - -print_success "Admin user configured" +print_success "Admin user created automatically via Helm initialAccount" # ============================================================================= # Summary diff --git a/infrastructure/platform/mail/mailu-helm/values.yaml b/infrastructure/platform/mail/mailu-helm/values.yaml index 7d27df8c..5129bac1 100644 --- a/infrastructure/platform/mail/mailu-helm/values.yaml +++ b/infrastructure/platform/mail/mailu-helm/values.yaml @@ -25,6 +25,18 @@ timezone: "Etc/UTC" # Postmaster configuration postmaster: "admin" +# Initial admin account configuration +# This creates an admin user as part of the Helm deployment +# Credentials can be provided directly or via Kubernetes secret +initialAccount: + enabled: true + username: "admin" + domain: "" # Set in environment-specific values (dev/prod) + password: "" # Leave empty to use existingSecret + existingSecret: "mailu-admin-credentials" + existingSecretPasswordKey: "password" + mode: "ifmissing" # Only create if account doesn't exist + # TLS configuration tls: flavor: "notls" # Disable TLS for development @@ -40,16 +52,18 @@ limits: # External relay configuration (Mailgun) # Mailu will relay all outbound emails through Mailgun SMTP -# Credentials should be provided via Kubernetes secret or environment-specific values +# Credentials are loaded from Kubernetes secret for security externalRelay: host: "[smtp.mailgun.org]:587" - username: "" # Set in environment-specific values or via secret - password: "" # Set in environment-specific values or via secret + # Use existing secret for credentials (recommended for security) + secretName: "mailu-mailgun-credentials" + usernameKey: "RELAY_USERNAME" + passwordKey: "RELAY_PASSWORD" # Webmail configuration webmail: enabled: true - flavor: "roundcube" + type: "roundcube" # Antivirus and antispam configuration antivirus: diff --git a/scripts/build-all-services.sh b/scripts/build-all-services.sh new file mode 100755 index 00000000..cecd7cea --- /dev/null +++ b/scripts/build-all-services.sh @@ -0,0 +1,126 @@ +#!/bin/bash + +# Build All Services Script for Bakery-IA +# This script builds and pushes all service images to the Gitea registry +# Used for first-time deployment when CI/CD pipeline isn't available yet + +set -e + +echo "==========================================" +echo "Bakery-IA Services Build Script" +echo "==========================================" +echo "" + +# Check if we're in the correct directory +if [ ! -f "PRODUCTION_DEPLOYMENT_GUIDE.md" ]; then + echo "Error: This script must be run from the root of the bakery-ia repository" + exit 1 +fi + +# Get Gitea admin password +echo "Getting Gitea admin credentials..." +if ! GITEA_ADMIN_PASSWORD=$(kubectl get secret gitea-admin-secret -n gitea -o jsonpath='{.data.password}' | base64 -d 2>/dev/null); then + echo "Error: Could not get Gitea admin password" + echo "Make sure you've completed Phase 5 (CI/CD Infrastructure) first" + exit 1 +fi + +# Login to Gitea registry +echo "Logging in to Gitea registry..." +docker login gitea-http.gitea.svc.cluster.local:3000 -u bakery-admin -p "$GITEA_ADMIN_PASSWORD" + +# Define the registry URL +REGISTRY="gitea-http.gitea.svc.cluster.local:3000/bakery-admin" + +# Define all services to build +# Format: "directory_name:image_name" +# Note: directory names use underscores (e.g., alert_processor), image names use hyphens (e.g., alert-processor) +SERVICES=( + "gateway:gateway" + "frontend:dashboard" + "auth:auth-service" + "tenant:tenant-service" + "training:training-service" + "forecasting:forecasting-service" + "sales:sales-service" + "external:external-service" + "notification:notification-service" + "inventory:inventory-service" + "recipes:recipes-service" + "suppliers:suppliers-service" + "pos:pos-service" + "orders:orders-service" + "production:production-service" + "procurement:procurement-service" + "distribution:distribution-service" + "orchestrator:orchestrator-service" + "alert_processor:alert-processor" + "ai_insights:ai-insights-service" + "demo_session:demo-session-service" +) + +# Build each service +echo "" +echo "Starting build process..." +echo "This may take 15-30 minutes depending on your system." +echo "" + +FAILED_SERVICES=() +SUCCESS_COUNT=0 + +for service_def in "${SERVICES[@]}"; do + IFS=':' read -r service_name image_name <<< "$service_def" + + echo "==========================================" + echo "Building: $service_name -> $image_name" + echo "==========================================" + + if [ "$service_name" = "gateway" ]; then + # Gateway service + docker build -t "$REGISTRY/$image_name:latest" \ + --build-arg BASE_REGISTRY="$REGISTRY" \ + --build-arg PYTHON_IMAGE="python:3.11-slim" \ + -f "gateway/Dockerfile" . + elif [ "$service_name" = "frontend" ]; then + # Frontend service (uses node:18-alpine and nginx:1.25-alpine internally) + docker build -t "$REGISTRY/$image_name:latest" \ + -f "frontend/Dockerfile.kubernetes" frontend/ + else + # Microservices (in services/ directory) + docker build -t "$REGISTRY/$image_name:latest" \ + --build-arg BASE_REGISTRY="$REGISTRY" \ + --build-arg PYTHON_IMAGE="python:3.11-slim" \ + -f "services/$service_name/Dockerfile" . + fi + + # Push the image + echo "Pushing $image_name to registry..." + if docker push "$REGISTRY/$image_name:latest"; then + echo "✅ Successfully built and pushed $image_name" + SUCCESS_COUNT=$((SUCCESS_COUNT + 1)) + else + echo "❌ Failed to push $image_name" + FAILED_SERVICES+=("$image_name") + fi + + echo "" +done + +echo "==========================================" +echo "Build Summary" +echo "==========================================" +echo "Total services: ${#SERVICES[@]}" +echo "Successfully built and pushed: $SUCCESS_COUNT" + +if [ ${#FAILED_SERVICES[@]} -gt 0 ]; then + echo "Failed services: ${#FAILED_SERVICES[@]}" + echo "Failed list: ${FAILED_SERVICES[*]}" + echo "" + echo "⚠️ Some services failed to build/push" + exit 1 +else + echo "✅ All services built and pushed successfully!" + echo "" + echo "You can now proceed to Phase 6: Deploy Application Services" + echo "Run: kubectl apply -k infrastructure/environments/prod/k8s-manifests" +fi \ No newline at end of file diff --git a/scripts/prepull-base-images-for-prod.sh b/scripts/prepull-base-images-for-prod.sh new file mode 100755 index 00000000..f702a421 --- /dev/null +++ b/scripts/prepull-base-images-for-prod.sh @@ -0,0 +1,324 @@ +#!/bin/bash + +# Base Image Pre-Pull Script for Bakery-IA Production +# This script pre-pulls all required base images for production deployment +# Supports both local development and production environments with Gitea registry + +set -e + +# Function to display usage +usage() { + echo "Usage: $0 [options]" + echo "" + echo "Options:" + echo " -e, --environment ENV Set environment (dev|prod) - default: dev" + echo " -r, --registry REG Custom registry URL - default: localhost:5000 (dev) or gitea registry (prod)" + echo " --skip-auth Skip Docker Hub authentication" + echo " --push-images Push images to registry (default: true for dev, false for prod)" + echo " --no-push-images Don't push images to registry" + echo " -h, --help Show this help message" + echo "" + echo "Examples:" + echo " $0 # Run in dev mode with local registry" + echo " $0 -e prod # Run in production mode with Gitea registry" + echo " $0 -e prod -r registry.example.com:5000 # Run in production with custom registry" + echo " $0 --skip-auth # Skip Docker Hub auth (for air-gapped envs)" + exit 1 +} + +# Parse command line arguments +ENVIRONMENT="dev" +REGISTRY="" +SKIP_AUTH=false +PUSH_IMAGES="" +while [[ $# -gt 0 ]]; do + case $1 in + -e|--environment) + ENVIRONMENT="$2" + shift 2 + ;; + -r|--registry) + REGISTRY="$2" + shift 2 + ;; + --skip-auth) + SKIP_AUTH=true + shift + ;; + --push-images) + PUSH_IMAGES=true + shift + ;; + --no-push-images) + PUSH_IMAGES=false + shift + ;; + -h|--help) + usage + ;; + *) + echo "Unknown option: $1" + usage + ;; + esac +done + +# Function to check if required tools are available +check_required_tools() { + local missing_tools=() + + # Check for required tools + for tool in docker curl jq kubectl; do + if ! command -v "$tool" &> /dev/null; then + missing_tools+=("$tool") + fi + done + + if [ ${#missing_tools[@]} -gt 0 ]; then + echo "Error: Missing required tools: ${missing_tools[*]}" + echo "Please install them before running this script." + echo "" + echo "On macOS (with Homebrew):" + echo " brew install docker curl jq kubectl" + echo "" + echo "On Ubuntu/Debian:" + echo " sudo apt-get install docker.io curl jq kubectl" + echo "" + echo "On CentOS/RHEL:" + echo " sudo yum install docker curl jq kubectl" + exit 1 + fi +} + +# Check for required tools +check_required_tools + +echo "==========================================" +echo "Bakery-IA Base Image Pre-Pull Script" +echo "Environment: $ENVIRONMENT" +echo "==========================================" +echo "" + +# Set defaults based on environment +if [ "$ENVIRONMENT" = "prod" ]; then + # Production environment - use Gitea registry + if [ -z "$REGISTRY" ]; then + # Try to get Gitea registry from Kubernetes + if kubectl get secret gitea-registry-secret -n bakery-ia &>/dev/null; then + # Extract registry URL from the secret + REGISTRY_JSON=$(kubectl get secret gitea-registry-secret -n bakery-ia -o jsonpath='{.data.\.dockerconfigjson}' | base64 -d) + REGISTRY=$(echo "$REGISTRY_JSON" | jq -r '.auths | keys[]' | head -n 1) + echo "Detected Gitea registry: $REGISTRY" + else + echo "Error: Could not detect Gitea registry automatically" + echo "Please specify the registry with -r/--registry option" + echo "Example: $0 -e prod -r gitea-http.gitea.svc.cluster.local:3000" + exit 1 + fi + fi + + # Default to not pushing images in production - they should be built by CI/CD + if [ -z "$PUSH_IMAGES" ]; then + PUSH_IMAGES=false + fi +elif [ "$ENVIRONMENT" = "dev" ]; then + # Development environment - use local registry + if [ -z "$REGISTRY" ]; then + REGISTRY="localhost:5000" + fi + + # Default to pushing images in dev + if [ -z "$PUSH_IMAGES" ]; then + PUSH_IMAGES=true + fi +else + echo "Error: Invalid environment. Use 'dev' or 'prod'" + exit 1 +fi + +echo "Registry configuration:" +echo " Environment: $ENVIRONMENT" +echo " Registry: $REGISTRY" +echo " Push Images: $PUSH_IMAGES" +echo "" + +# Docker Hub credentials (use environment variables or defaults) +DOCKER_USERNAME="${DOCKER_HUB_USERNAME:-uals}" +DOCKER_PASSWORD="${DOCKER_HUB_PASSWORD:-dckr_pat_zzEY5Q58x1S0puraIoKEtbpue3A}" + +# Authenticate with Docker Hub if not skipping auth +if [ "$SKIP_AUTH" = false ]; then + echo "Authenticating with Docker Hub..." + if ! echo "$DOCKER_PASSWORD" | docker login -u "$DOCKER_USERNAME" --password-stdin; then + echo "⚠ Warning: Docker Hub authentication failed. Continuing anyway..." + else + echo "✓ Authentication successful" + fi +else + echo "Skipping Docker Hub authentication (--skip-auth flag set)" +fi +echo "" + +# Define all base images used in the project +# These are the base images needed for the services +BASE_IMAGES=( + # Service base images (Python microservices) + "python:3.11-slim" + # Frontend base images (Node.js build + Nginx runtime) + "node:18-alpine" + "nginx:1.25-alpine" + # Database images + "postgres:17-alpine" + "redis:7.4-alpine" + "rabbitmq:4.1-management-alpine" + # Utility images + "busybox:1.36" + "curlimages/curl:latest" + "bitnami/kubectl:latest" + # Alpine variants + "alpine:3.18" + "alpine:3.19" + "alpine/git:2.43.0" + # CI/CD images + "gcr.io/kaniko-project/executor:v1.23.0" + "gcr.io/go-containerregistry/crane:latest" + "registry.k8s.io/kustomize/kustomize:v5.3.0" + # Storage images + "minio/minio:RELEASE.2024-11-07T00-52-20Z" + "minio/mc:RELEASE.2024-11-17T19-35-25Z" + # Geocoding + "mediagis/nominatim:4.4" + # Mail server (Mailu - from GHCR) + "ghcr.io/mailu/nginx:2024.06" + "ghcr.io/mailu/admin:2024.06" + "ghcr.io/mailu/postfix:2024.06" + "ghcr.io/mailu/dovecot:2024.06" + "ghcr.io/mailu/rspamd:2024.06" +) + +# If using registry, verify it's running +if [ "$PUSH_IMAGES" = true ]; then + echo "Checking registry at $REGISTRY..." + if curl -s http://$REGISTRY/v2/ >/dev/null 2>&1; then + echo "✓ Registry is accessible" + elif curl -s https://$REGISTRY/v2/ >/dev/null 2>&1; then + echo "✓ Registry is accessible (HTTPS)" + # Update registry to use HTTPS if needed + REGISTRY="https://$REGISTRY" + else + echo "⚠ Registry is not accessible at $REGISTRY" + echo "Will only pull images locally (no registry push)" + PUSH_IMAGES=false + fi +fi + +echo "" +echo "Base images to pre-pull:" +echo "----------------------------------------" +for image in "${BASE_IMAGES[@]}"; do + echo " - $image" +done +echo "" + +echo "Starting pre-pull process..." +echo "----------------------------------------" + +# Track success/failure +FAILED_IMAGES=() +SUCCESS_COUNT=0 + +# Pull each base image +for image in "${BASE_IMAGES[@]}"; do + echo "Pulling: $image" + + # Pull the image + if ! docker pull "$image"; then + echo " ⚠ Failed to pull $image" + FAILED_IMAGES+=("$image") + continue + fi + + # Tag for registry if enabled + if [ "$PUSH_IMAGES" = true ]; then + # Extract registry host and image name + if [[ "$REGISTRY" == https://* ]]; then + REGISTRY_HOST=${REGISTRY#https://} + else + REGISTRY_HOST=$REGISTRY + fi + + # Format for registry: use bakery-admin namespace and preserve original name/tag + # Extract image name and tag + if [[ "$image" == *:* ]]; then + image_name="${image%:*}" + image_tag="${image#*:}" + else + image_name="$image" + image_tag="latest" + fi + + # Replace slashes with underscores for repository name + repo_name="$(echo "$image_name" | sed 's|/|_|g' | tr '[:upper:]' '[:lower:]')" + + # Use bakery-admin namespace and preserve original tag + registry_image="$REGISTRY_HOST/bakery-admin/${repo_name}:${image_tag}" + + docker tag "$image" "$registry_image" + echo " Tagged as: $registry_image" + + # Push to registry + if docker push "$registry_image"; then + echo " ✓ Pushed to registry" + else + echo " ⚠ Failed to push to registry (image still available locally)" + fi + fi + + echo " ✓ Successfully pulled $image" + SUCCESS_COUNT=$((SUCCESS_COUNT + 1)) + echo "" +done + +echo "==========================================" +echo "Base Image Pre-Pull Complete!" +echo "==========================================" +echo "" +echo "Summary:" +echo " - Total images: ${#BASE_IMAGES[@]}" +echo " - Successfully pulled: $SUCCESS_COUNT" +if [ ${#FAILED_IMAGES[@]} -gt 0 ]; then + echo " - Failed: ${#FAILED_IMAGES[@]}" + echo " - Failed images: ${FAILED_IMAGES[*]}" +fi +echo " - Environment: $ENVIRONMENT" +if [ "$PUSH_IMAGES" = true ]; then + echo " - Registry: $REGISTRY" +else + echo " - Registry: None (local Docker only)" +fi +echo "" + +# Exit with error if any images failed +if [ ${#FAILED_IMAGES[@]} -gt 0 ]; then + echo "⚠ Some images failed to pull. This may be due to Docker Hub rate limits." + echo "Please try again later or configure Docker Hub credentials." + exit 1 +fi + +echo "✓ All images pulled successfully!" + +if [ "$ENVIRONMENT" = "prod" ] && [ "$PUSH_IMAGES" = false ]; then + echo "" + echo "💡 Note: In production mode, images are not pushed to registry." + echo " Images should be built and pushed by your CI/CD pipeline." + echo " Make sure your CI/CD pipeline has built and pushed the required images." + echo "" + echo "💡 To build and push service images to Gitea registry:" + echo " 1. Ensure your CI/CD pipeline is running (Tekton)" + echo " 2. Push a commit to trigger the pipeline: git commit --allow-empty -m 'Trigger build'" + echo " 3. Or manually trigger a pipeline run" + echo "" + echo "💡 Check pipeline status:" + echo " kubectl get pipelineruns -n tekton-pipelines" + echo " kubectl get pods -n tekton-pipelines" +fi \ No newline at end of file diff --git a/scripts/prepull-base-images.sh b/scripts/prepull-base-images.sh index 1e26a3a7..d93f659a 100755 --- a/scripts/prepull-base-images.sh +++ b/scripts/prepull-base-images.sh @@ -2,16 +2,74 @@ # Base Image Pre-Pull Script for Bakery-IA # This script pre-pulls all required base images to reduce Docker Hub usage +# Supports both local development and production environments with Gitea registry # Run this script before building services to cache base images locally set -e +# Function to display usage +usage() { + echo "Usage: $0 [options]" + echo "" + echo "Options:" + echo " -e, --environment ENV Set environment (dev|prod) - default: dev" + echo " -r, --registry REG Custom registry URL - default: localhost:5000 (dev) or gitea registry (prod)" + echo " --skip-auth Skip Docker Hub authentication" + echo " --push-images Push images to registry (default: true for dev, false for prod)" + echo " --no-push-images Don't push images to registry" + echo " -h, --help Show this help message" + echo "" + echo "Examples:" + echo " $0 # Run in dev mode with local registry" + echo " $0 -e prod # Run in production mode with Gitea registry" + echo " $0 -e prod -r registry.example.com:5000 # Run in production with custom registry" + echo " $0 --skip-auth # Skip Docker Hub auth (for air-gapped envs)" + exit 1 +} + +# Parse command line arguments +ENVIRONMENT="dev" +REGISTRY="" +SKIP_AUTH=false +PUSH_IMAGES="" +while [[ $# -gt 0 ]]; do + case $1 in + -e|--environment) + ENVIRONMENT="$2" + shift 2 + ;; + -r|--registry) + REGISTRY="$2" + shift 2 + ;; + --skip-auth) + SKIP_AUTH=true + shift + ;; + --push-images) + PUSH_IMAGES=true + shift + ;; + --no-push-images) + PUSH_IMAGES=false + shift + ;; + -h|--help) + usage + ;; + *) + echo "Unknown option: $1" + usage + ;; + esac +done + # Function to check if required tools are available check_required_tools() { local missing_tools=() # Check for required tools - for tool in docker curl jq; do + for tool in docker curl jq kubectl; do if ! command -v "$tool" &> /dev/null; then missing_tools+=("$tool") fi @@ -22,13 +80,13 @@ check_required_tools() { echo "Please install them before running this script." echo "" echo "On macOS (with Homebrew):" - echo " brew install docker curl jq" + echo " brew install docker curl jq kubectl" echo "" echo "On Ubuntu/Debian:" - echo " sudo apt-get install docker.io curl jq" + echo " sudo apt-get install docker.io curl jq kubectl" echo "" echo "On CentOS/RHEL:" - echo " sudo yum install docker curl jq" + echo " sudo yum install docker curl jq kubectl" exit 1 fi } @@ -38,24 +96,78 @@ check_required_tools echo "==========================================" echo "Bakery-IA Base Image Pre-Pull Script" +echo "Environment: $ENVIRONMENT" echo "==========================================" echo "" -# Docker Hub credentials (use the same as in your Kubernetes setup) -DOCKER_USERNAME="uals" -DOCKER_PASSWORD="dckr_pat_zzEY5Q58x1S0puraIoKEtbpue3A" +# Set defaults based on environment +if [ "$ENVIRONMENT" = "prod" ]; then + # Production environment - use Gitea registry + if [ -z "$REGISTRY" ]; then + # Try to get Gitea registry from Kubernetes + if kubectl get secret gitea-registry-secret -n bakery-ia &>/dev/null; then + # Extract registry URL from the secret + REGISTRY_JSON=$(kubectl get secret gitea-registry-secret -n bakery-ia -o jsonpath='{.data.\.dockerconfigjson}' | base64 -d) + REGISTRY=$(echo "$REGISTRY_JSON" | jq -r '.auths | keys[]' | head -n 1) + echo "Detected Gitea registry: $REGISTRY" + else + echo "Error: Could not detect Gitea registry automatically" + echo "Please specify the registry with -r/--registry option" + echo "Example: $0 -e prod -r gitea-http.gitea.svc.cluster.local:3000" + exit 1 + fi + fi -# Authenticate with Docker Hub -echo "Authenticating with Docker Hub..." -echo "$DOCKER_PASSWORD" | docker login -u "$DOCKER_USERNAME" --password-stdin -echo "✓ Authentication successful" + # Default to not pushing images in production - they should be built by CI/CD + if [ -z "$PUSH_IMAGES" ]; then + PUSH_IMAGES=false + fi +elif [ "$ENVIRONMENT" = "dev" ]; then + # Development environment - use local registry + if [ -z "$REGISTRY" ]; then + REGISTRY="localhost:5000" + fi + + # Default to pushing images in dev + if [ -z "$PUSH_IMAGES" ]; then + PUSH_IMAGES=true + fi +else + echo "Error: Invalid environment. Use 'dev' or 'prod'" + exit 1 +fi + +echo "Registry configuration:" +echo " Environment: $ENVIRONMENT" +echo " Registry: $REGISTRY" +echo " Push Images: $PUSH_IMAGES" +echo "" + +# Docker Hub credentials (use environment variables or defaults) +DOCKER_USERNAME="${DOCKER_HUB_USERNAME:-uals}" +DOCKER_PASSWORD="${DOCKER_HUB_PASSWORD:-dckr_pat_zzEY5Q58x1S0puraIoKEtbpue3A}" + +# Authenticate with Docker Hub if not skipping auth +if [ "$SKIP_AUTH" = false ]; then + echo "Authenticating with Docker Hub..." + if ! echo "$DOCKER_PASSWORD" | docker login -u "$DOCKER_USERNAME" --password-stdin; then + echo "⚠ Warning: Docker Hub authentication failed. Continuing anyway..." + else + echo "✓ Authentication successful" + fi +else + echo "Skipping Docker Hub authentication (--skip-auth flag set)" +fi echo "" # Define all base images used in the project -# All images are cached in local registry for dev environment +# These are the base images needed for the services BASE_IMAGES=( - # Service base images + # Service base images (Python microservices) "python:3.11-slim" + # Frontend base images (Node.js build + Nginx runtime) + "node:18-alpine" + "nginx:1.25-alpine" # Database images "postgres:17-alpine" "redis:7.4-alpine" @@ -85,27 +197,19 @@ BASE_IMAGES=( "ghcr.io/mailu/rspamd:2024.06" ) -# Registry configuration -# Read from environment variables (set by Tiltfile or manually) -# USE_LOCAL_REGISTRY=true to push images to local registry after pulling -USE_LOCAL_REGISTRY="${USE_LOCAL_REGISTRY:-true}" - -echo "Registry configuration:" -echo " USE_LOCAL_REGISTRY=$USE_LOCAL_REGISTRY" -echo "" - -# Use local registry (kind registry) -REGISTRY="localhost:5000" - -# If using local registry, verify it's running -if [ "$USE_LOCAL_REGISTRY" = "true" ]; then - echo "Checking local registry at $REGISTRY..." +# If using registry, verify it's running +if [ "$PUSH_IMAGES" = true ]; then + echo "Checking registry at $REGISTRY..." if curl -s http://$REGISTRY/v2/ >/dev/null 2>&1; then - echo "✓ Local registry is accessible" + echo "✓ Registry is accessible" + elif curl -s https://$REGISTRY/v2/ >/dev/null 2>&1; then + echo "✓ Registry is accessible (HTTPS)" + # Update registry to use HTTPS if needed + REGISTRY="https://$REGISTRY" else - echo "⚠ Local registry is not accessible at $REGISTRY" + echo "⚠ Registry is not accessible at $REGISTRY" echo "Will only pull images locally (no registry push)" - USE_LOCAL_REGISTRY="false" + PUSH_IMAGES=false fi fi @@ -136,17 +240,24 @@ for image in "${BASE_IMAGES[@]}"; do fi # Tag for registry if enabled - if [ "$USE_LOCAL_REGISTRY" = "true" ]; then - # Local registry format: replace /, :, -, and . with _ + if [ "$PUSH_IMAGES" = true ]; then + # Extract registry host and image name + if [[ "$REGISTRY" == https://* ]]; then + REGISTRY_HOST=${REGISTRY#https://} + else + REGISTRY_HOST=$REGISTRY + fi + + # Format for registry: replace /, :, -, and . with _ local_repo="$(echo $image | sed 's|/|_|g' | sed 's|:|_|g' | sed 's|-|_|g' | sed 's|\.|_|g' | tr '[:upper:]' '[:lower:]')" - registry_image="$REGISTRY/${local_repo}:latest" + registry_image="$REGISTRY_HOST/${local_repo}:latest" docker tag "$image" "$registry_image" echo " Tagged as: $registry_image" # Push to registry if docker push "$registry_image"; then - echo " ✓ Pushed to local registry" + echo " ✓ Pushed to registry" else echo " ⚠ Failed to push to registry (image still available locally)" fi @@ -168,8 +279,9 @@ if [ ${#FAILED_IMAGES[@]} -gt 0 ]; then echo " - Failed: ${#FAILED_IMAGES[@]}" echo " - Failed images: ${FAILED_IMAGES[*]}" fi -if [ "$USE_LOCAL_REGISTRY" = "true" ]; then - echo " - Registry: Local ($REGISTRY)" +echo " - Environment: $ENVIRONMENT" +if [ "$PUSH_IMAGES" = true ]; then + echo " - Registry: $REGISTRY" else echo " - Registry: None (local Docker only)" fi @@ -183,3 +295,19 @@ if [ ${#FAILED_IMAGES[@]} -gt 0 ]; then fi echo "✓ All images pulled successfully!" + +if [ "$ENVIRONMENT" = "prod" ] && [ "$PUSH_IMAGES" = false ]; then + echo "" + echo "💡 Note: In production mode, images are not pushed to registry." + echo " Images should be built and pushed by your CI/CD pipeline." + echo " Make sure your CI/CD pipeline has built and pushed the required images." + echo "" + echo "💡 To build and push service images to Gitea registry:" + echo " 1. Ensure your CI/CD pipeline is running (Tekton)" + echo " 2. Push a commit to trigger the pipeline: git commit --allow-empty -m 'Trigger build'" + echo " 3. Or manually trigger a pipeline run" + echo "" + echo "💡 Check pipeline status:" + echo " kubectl get pipelineruns -n tekton-pipelines" + echo " kubectl get pods -n tekton-pipelines" +fi