Files

Urtzi Alfaro dfb7e4b237 Add signoz

2026-01-08 12:58:00 +01:00

31 KiB

Raw Blame History

Bakery-IA Pilot Launch Guide

Complete guide for deploying to production for a 10-tenant pilot program

Last Updated: 2026-01-07 Target Environment: clouding.io VPS with MicroK8s Estimated Cost: €41-81/month Time to Deploy: 2-4 hours (first time)

Executive Summary
Pre-Launch Checklist
VPS Provisioning
Infrastructure Setup
Domain & DNS Configuration
TLS/SSL Certificates
Email & Communication Setup
Kubernetes Deployment
Configuration & Secrets
Database Migrations
Verification & Testing
Post-Deployment

Executive Summary

What You're Deploying

A complete multi-tenant SaaS platform with:

18 microservices (auth, tenant, ML forecasting, inventory, sales, orders, etc.)
14 PostgreSQL databases with TLS encryption
Redis cache with TLS
RabbitMQ message broker
Monitoring stack (Prometheus, Grafana, AlertManager)
Full security (TLS, RBAC, audit logging)

Total Cost Breakdown

Service	Provider	Monthly Cost
VPS Server (20GB RAM, 8 vCPU, 200GB SSD)	clouding.io	€40-80
Domain	Namecheap/Cloudflare	€1.25 (€15/year)
Email	Zoho Free / Gmail	€0
WhatsApp API	Meta Business	€0 (1k free conversations)
DNS	Cloudflare	€0
SSL	Let's Encrypt	€0
TOTAL		€41-81/month

Timeline

Phase	Duration	Description
Pre-Launch Setup	1-2 hours	Domain, VPS provisioning, accounts setup
Infrastructure Setup	1 hour	MicroK8s installation, firewall config
Deployment	30-60 min	Deploy all services and databases
Verification	30-60 min	Test everything works
Total	2-4 hours	First-time deployment

Pre-Launch Checklist

Required Accounts & Services

Domain Name
- Register at Namecheap or Cloudflare (€10-15/year)
- Suggested: bakeryforecast.es or bakery-ia.com
VPS Account
- Sign up at clouding.io
- Payment method configured
Email Service (Choose ONE)
- Option A: Zoho Mail FREE (recommended for full send/receive)
- Option B: Gmail SMTP + domain forwarding
- Option C: Google Workspace (14-day free trial, then €5.75/month)
WhatsApp Business API
- Create Meta Business Account (free)
- Verify business identity
- Phone number ready (non-VoIP)
DNS Access
- Cloudflare account (free, recommended)
- Or domain registrar DNS panel access
Container Registry (Choose ONE)
- Option A: Docker Hub account (recommended)
- Option B: GitHub Container Registry
- Option C: MicroK8s built-in registry

Required Tools on Local Machine

# Verify you have these installed:
kubectl version --client
docker --version
git --version
ssh -V
openssl version

# Install if missing (macOS):
brew install kubectl docker git openssh openssl

Repository Setup

# Clone the repository
git clone https://github.com/yourusername/bakery-ia.git
cd bakery-ia

# Verify structure
ls infrastructure/kubernetes/overlays/prod/

VPS Provisioning

Recommended Configuration

For 10-tenant pilot program:

RAM: 20 GB
CPU: 8 vCPU cores
Storage: 200 GB NVMe SSD (triple replica)
Network: 1 Gbps connection
OS: Ubuntu 22.04 LTS
Monthly Cost: €40-80 (check current pricing)

Why These Specs?

Memory Breakdown:

Application services: 14.1 GB
Databases (18 instances): 4.6 GB
Infrastructure (Redis, RabbitMQ): 0.8 GB
Gateway/Frontend: 1.8 GB
Monitoring: 1.5 GB
System overhead: ~3 GB
Total: ~26 GB capacity needed, 20 GB is sufficient with HPA

Storage Breakdown:

Databases: 36 GB (18 × 2GB)
ML Models: 10 GB
Redis: 1 GB
RabbitMQ: 2 GB
Prometheus metrics: 20 GB
Container images: ~30 GB
Growth buffer: 100 GB
Total: 199 GB

Provisioning Steps

Create VPS at clouding.io:

1. Log in to clouding.io dashboard
2. Click "Create New Server"
3. Select:
   - OS: Ubuntu 22.04 LTS
   - RAM: 20 GB
   - CPU: 8 vCPU
   - Storage: 200 GB NVMe SSD
   - Location: Barcelona (best for Spain)
4. Set hostname: bakery-ia-prod-01
5. Add SSH key (or use password)
6. Create server

Note your server details:

# Save these for later:
VPS_IP="YOUR_VPS_IP_ADDRESS"
VPS_ROOT_PASSWORD="YOUR_ROOT_PASSWORD"  # If not using SSH key

Initial SSH connection:

# Test connection
ssh root@$VPS_IP

# Update system
apt update && apt upgrade -y

Infrastructure Setup

Step 1: Install MicroK8s

# SSH into your VPS
ssh root@$VPS_IP

# Install MicroK8s
snap install microk8s --classic --channel=1.28/stable

# Add your user to microk8s group
usermod -a -G microk8s $USER
chown -f -R $USER ~/.kube
newgrp microk8s

# Verify installation
microk8s status --wait-ready

Step 2: Enable Required Add-ons

# Enable core add-ons
microk8s enable dns
microk8s enable hostpath-storage
microk8s enable ingress
microk8s enable cert-manager
microk8s enable metrics-server
microk8s enable rbac

# Optional but recommended
microk8s enable prometheus  # For monitoring
microk8s enable registry    # If using local registry

# Setup kubectl alias
echo "alias kubectl='microk8s kubectl'" >> ~/.bashrc
source ~/.bashrc

# Verify
kubectl get nodes
kubectl get pods -A

Step 3: Configure Firewall

# Allow necessary ports
ufw allow 22/tcp      # SSH
ufw allow 80/tcp      # HTTP
ufw allow 443/tcp     # HTTPS
ufw allow 16443/tcp   # Kubernetes API (optional)

# Enable firewall
ufw enable

# Check status
ufw status verbose

Step 4: Create Namespace

# Create bakery-ia namespace
kubectl create namespace bakery-ia

# Verify
kubectl get namespaces

Domain & DNS Configuration

Step 1: Register Domain

Go to Namecheap or Cloudflare Registrar
Search for your desired domain
Complete purchase (~€10-15/year)
Save domain credentials

Step 2: Configure Cloudflare DNS (Recommended)

Add site to Cloudflare:

1. Log in to Cloudflare
2. Click "Add a Site"
3. Enter your domain name
4. Choose Free plan
5. Cloudflare will scan existing DNS records

Update nameservers at registrar:

Point your domain's nameservers to Cloudflare:
- NS1: assigned.cloudflare.com
- NS2: assigned.cloudflare.com
(Cloudflare will provide the exact values)

Add DNS records:

Type    Name        Content         TTL     Proxy
A       @           YOUR_VPS_IP     Auto    Yes
A       www         YOUR_VPS_IP     Auto    Yes
A       api         YOUR_VPS_IP     Auto    Yes
A       monitoring  YOUR_VPS_IP     Auto    Yes
CNAME   *           yourdomain.com  Auto    No

Configure SSL/TLS mode:

SSL/TLS tab → Overview → Set to "Full (strict)"

Test DNS propagation:

# Wait 5-10 minutes, then test
nslookup yourdomain.com
nslookup api.yourdomain.com

TLS/SSL Certificates

Understanding Certificate Setup

The platform uses two layers of SSL/TLS:

External (Ingress) SSL: Let's Encrypt for public HTTPS
Internal (Database) SSL: Self-signed certificates for database connections

Step 1: Generate Internal Certificates

# On your local machine
cd infrastructure/tls

# Generate certificates
./generate-certificates.sh

# This creates:
# - ca/ (Certificate Authority)
# - postgres/ (PostgreSQL server certs)
# - redis/ (Redis server certs)

Certificate Details:

Root CA: 10-year validity (expires 2035)
Server certs: 3-year validity (expires October 2028)
Algorithm: RSA 4096-bit
Signature: SHA-256

Step 2: Create Kubernetes Secrets

# Create PostgreSQL TLS secret
kubectl create secret generic postgres-tls \
  --from-file=server-cert.pem=infrastructure/tls/postgres/server-cert.pem \
  --from-file=server-key.pem=infrastructure/tls/postgres/server-key.pem \
  --from-file=ca-cert.pem=infrastructure/tls/postgres/ca-cert.pem \
  -n bakery-ia

# Create Redis TLS secret
kubectl create secret generic redis-tls \
  --from-file=redis-cert.pem=infrastructure/tls/redis/redis-cert.pem \
  --from-file=redis-key.pem=infrastructure/tls/redis/redis-key.pem \
  --from-file=ca-cert.pem=infrastructure/tls/redis/ca-cert.pem \
  -n bakery-ia

# Verify secrets created
kubectl get secrets -n bakery-ia | grep tls

Step 3: Configure Let's Encrypt (External SSL)

cert-manager is already enabled. Configure the ClusterIssuer:

# On VPS, create ClusterIssuer
cat <<EOF | kubectl apply -f -
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-production
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: admin@yourdomain.com  # CHANGE THIS
    privateKeySecretRef:
      name: letsencrypt-production
    solvers:
    - http01:
        ingress:
          class: public
EOF

# Verify ClusterIssuer is ready
kubectl get clusterissuer
kubectl describe clusterissuer letsencrypt-production

Email & Communication Setup

Option A: Zoho Mail (FREE, Recommended)

Features:

✅ Free forever for 1 domain, 5 users
✅ 5GB storage per user
✅ Full send/receive capability
✅ Web interface + SMTP/IMAP
✅ Professional email addresses

Setup Steps:

Sign up for Zoho Mail:

1. Go to https://www.zoho.com/mail/
2. Click "Sign Up for Free"
3. Choose "Forever Free" plan
4. Enter your domain name
5. Complete verification

Verify domain ownership:

Add TXT record to your DNS:
Type: TXT
Name: @
Value: zoho-verification=XXXXX.zoho.com

Configure MX records:

Priority  Type  Name  Value
10        MX    @     mx.zoho.com
20        MX    @     mx2.zoho.com
50        MX    @     mx3.zoho.com

Get SMTP credentials:

SMTP Host: smtp.zoho.com
SMTP Port: 587
SMTP Username: noreply@yourdomain.com
SMTP Password: (generate app password in Zoho settings)

Option B: Gmail SMTP + Forwarding

Features:

✅ Completely free
✅ 500 emails/day (sufficient for pilot)
✅ Receive via domain forwarding

Setup Steps:

Enable 2FA on your Gmail:

1. Go to myaccount.google.com
2. Security → 2-Step Verification
3. Enable and complete setup

Generate app password:

1. Security → 2-Step Verification → App passwords
2. Select "Mail" and "Other (Custom name)"
3. Name it "Bakery-IA SMTP"
4. Copy the 16-character password

Configure domain email forwarding:

At your domain registrar or Cloudflare:
- Forward noreply@yourdomain.com → your.gmail@gmail.com
- Forward alerts@yourdomain.com → your.gmail@gmail.com

SMTP Settings:

SMTP Host: smtp.gmail.com
SMTP Port: 587
SMTP Username: your.gmail@gmail.com
SMTP Password: (16-char app password from step 2)
From Email: noreply@yourdomain.com

WhatsApp Business API Setup

Features:

✅ First 1,000 conversations/month FREE
✅ Perfect for 10 tenants (~500 messages/month)

Setup Steps:

Create Meta Business Account:

1. Go to business.facebook.com
2. Create Business Account
3. Complete business verification

Add WhatsApp Product:

1. Go to developers.facebook.com
2. Create New App → Business
3. Add WhatsApp product
4. Complete setup wizard

Configure Phone Number:

1. Test with your personal number initially
2. Later: Get dedicated business number
3. Verify phone number with SMS code

Create Message Templates:

1. Go to WhatsApp Manager
2. Create templates for:
   - Low inventory alert
   - Expired product alert
   - Forecast summary
   - Order notification
3. Submit for approval (15 min - 24 hours)

Get API Credentials:

Save these values:
- Phone Number ID: (from WhatsApp Manager)
- Access Token: (from App Dashboard)
- Business Account ID: (from WhatsApp Manager)
- Webhook Verify Token: (create your own secure string)

Kubernetes Deployment

Step 1: Prepare Container Images

Option A: Using Docker Hub (Recommended)

# On your local machine
docker login

# Build all images
docker-compose build

# Tag images for Docker Hub
# Replace YOUR_USERNAME with your Docker Hub username
export DOCKER_USERNAME="YOUR_USERNAME"

./scripts/tag-images.sh $DOCKER_USERNAME

# Push to Docker Hub
./scripts/push-images.sh $DOCKER_USERNAME

# Update prod kustomization with your username
# Edit: infrastructure/kubernetes/overlays/prod/kustomization.yaml
# Replace all "bakery/" with "$DOCKER_USERNAME/"

Option B: Using MicroK8s Registry

# On VPS
microk8s enable registry

# Get registry address (usually localhost:32000)
kubectl get service -n container-registry

# On local machine, configure insecure registry
# Edit /etc/docker/daemon.json:
{
  "insecure-registries": ["YOUR_VPS_IP:32000"]
}

# Restart Docker
sudo systemctl restart docker

# Tag and push images
docker tag bakery/auth-service YOUR_VPS_IP:32000/bakery/auth-service
docker push YOUR_VPS_IP:32000/bakery/auth-service
# Repeat for all services...

Step 2: Update Production Configuration

The production configuration is already set up for bakewise.ai domain:

Production URLs:

Main Application: https://bakewise.ai
API Endpoints: https://bakewise.ai/api/v1/...
Monitoring Dashboard: https://monitoring.bakewise.ai/grafana
Prometheus: https://monitoring.bakewise.ai/prometheus
SigNoz (Traces/Metrics/Logs): https://monitoring.bakewise.ai/signoz
AlertManager: https://monitoring.bakewise.ai/alertmanager

# Verify the configuration is correct:
cat infrastructure/kubernetes/overlays/prod/prod-ingress.yaml | grep -A 3 "host:"

# Expected output should show:
# - host: bakewise.ai
# - host: monitoring.bakewise.ai

# Verify CORS configuration
cat infrastructure/kubernetes/overlays/prod/prod-configmap.yaml | grep CORS

# Expected: CORS_ORIGINS: "https://bakewise.ai"

If using a different domain, update these files:

# 1. Update domain names
nano infrastructure/kubernetes/overlays/prod/prod-ingress.yaml
# Replace bakewise.ai with your domain

# 2. Update ConfigMap
nano infrastructure/kubernetes/overlays/prod/prod-configmap.yaml
# Update CORS_ORIGINS

# 3. Verify image names (if using custom registry)
nano infrastructure/kubernetes/overlays/prod/kustomization.yaml

Configuration & Secrets

Step 1: Generate Strong Passwords

# Generate passwords for all services
openssl rand -base64 32  # For each database
openssl rand -hex 32     # For JWT secrets and API keys

# Save all passwords securely!
# Recommended: Use a password manager (1Password, LastPass, Bitwarden)

Step 2: Update Application Secrets

# Edit the secrets file
nano infrastructure/kubernetes/base/secrets.yaml

# Update ALL of these values:
# Database passwords (14 databases):
AUTH_DB_PASSWORD: <base64-encoded-password>
TENANT_DB_PASSWORD: <base64-encoded-password>
# ... (all 14 databases)

# Redis password:
REDIS_PASSWORD: <base64-encoded-password>

# JWT secrets:
JWT_SECRET_KEY: <base64-encoded-secret>
JWT_REFRESH_SECRET_KEY: <base64-encoded-secret>

# SMTP settings (from email setup):
SMTP_HOST: <base64-encoded-host>          # smtp.zoho.com or smtp.gmail.com
SMTP_PORT: <base64-encoded-port>          # 587
SMTP_USERNAME: <base64-encoded-username>  # your email
SMTP_PASSWORD: <base64-encoded-password>  # app password
DEFAULT_FROM_EMAIL: <base64-encoded-email> # noreply@yourdomain.com

# WhatsApp credentials (from WhatsApp setup):
WHATSAPP_ACCESS_TOKEN: <base64-encoded-token>
WHATSAPP_PHONE_NUMBER_ID: <base64-encoded-id>
WHATSAPP_BUSINESS_ACCOUNT_ID: <base64-encoded-id>
WHATSAPP_WEBHOOK_VERIFY_TOKEN: <base64-encoded-token>

# Database connection strings (update with actual passwords):
AUTH_DATABASE_URL: postgresql+asyncpg://auth_user:PASSWORD@auth-db:5432/auth_db?ssl=require
# ... (all 14 databases)

To base64 encode:

echo -n "your-password-here" | base64

CRITICAL: Never commit real secrets to git! Use .gitignore for secrets files.

Step 3: Apply Secrets

# Copy manifests to VPS
scp -r infrastructure/kubernetes user@YOUR_VPS_IP:~/

# SSH to VPS
ssh user@YOUR_VPS_IP

# Apply secrets
kubectl apply -f ~/infrastructure/kubernetes/base/secrets.yaml

# Verify secrets created
kubectl get secrets -n bakery-ia

Database Migrations

Step 1: Deploy Databases

# On VPS
kubectl apply -k ~/kubernetes/overlays/prod

# Wait for databases to be ready (5-10 minutes)
kubectl wait --for=condition=ready pod -l app.kubernetes.io/component=database -n bakery-ia --timeout=600s

# Check status
kubectl get pods -n bakery-ia -l app.kubernetes.io/component=database

Step 2: Run Migrations

Migrations are automatically handled by init containers in each service. Verify they completed:

# Check migration job status
kubectl get jobs -n bakery-ia | grep migration

# All should show "COMPLETIONS = 1/1"

# Check logs if any failed
kubectl logs -n bakery-ia job/auth-migration

Step 3: Verify Database Schemas

# Connect to a database to verify
kubectl exec -n bakery-ia deployment/auth-db -it -- psql -U auth_user -d auth_db

# Inside psql:
\dt          # List tables
\d users     # Describe users table
\q           # Quit

Verification & Testing

Step 1: Check All Pods Running

# View all pods
kubectl get pods -n bakery-ia

# Expected: All pods in "Running" state, none in CrashLoopBackOff

# Check for issues
kubectl get pods -n bakery-ia | grep -vE "Running|Completed"

# View logs for any problematic pods
kubectl logs -n bakery-ia POD_NAME

Step 2: Check Services and Ingress

# View services
kubectl get svc -n bakery-ia

# View ingress
kubectl get ingress -n bakery-ia

# View certificates (should auto-issue from Let's Encrypt)
kubectl get certificate -n bakery-ia

# Describe certificate to check status
kubectl describe certificate bakery-ia-prod-tls-cert -n bakery-ia

Step 3: Test Database Connections

# Test PostgreSQL TLS
kubectl exec -n bakery-ia deployment/auth-db -- sh -c \
  'psql -U auth_user -d auth_db -c "SHOW ssl;"'
# Expected output: on

# Test Redis TLS
kubectl exec -n bakery-ia deployment/redis -- redis-cli \
  --tls \
  --cert /tls/redis-cert.pem \
  --key /tls/redis-key.pem \
  --cacert /tls/ca-cert.pem \
  -a $REDIS_PASSWORD \
  ping
# Expected output: PONG

Step 4: Test Frontend Access

# Test frontend (replace with your domain)
curl -I https://bakery.yourdomain.com

# Expected: HTTP/2 200 OK

# Test API health
curl https://api.yourdomain.com/health

# Expected: {"status": "healthy"}

Step 5: Test Authentication

# Create a test user (using your frontend or API)
curl -X POST https://api.yourdomain.com/api/v1/auth/register \
  -H "Content-Type: application/json" \
  -d '{
    "email": "test@yourdomain.com",
    "password": "TestPassword123!",
    "name": "Test User"
  }'

# Login
curl -X POST https://api.yourdomain.com/api/v1/auth/login \
  -H "Content-Type: application/json" \
  -d '{
    "email": "test@yourdomain.com",
    "password": "TestPassword123!"
  }'

# Expected: JWT token in response

Step 6: Test Email Delivery

# Trigger a password reset to test email
curl -X POST https://api.yourdomain.com/api/v1/auth/forgot-password \
  -H "Content-Type: application/json" \
  -d '{"email": "test@yourdomain.com"}'

# Check your email inbox for the reset link
# Check service logs if email not received:
kubectl logs -n bakery-ia deployment/auth-service | grep -i "email\|smtp"

Step 7: Test WhatsApp (Optional)

# Send a test WhatsApp message
# This requires creating a tenant and configuring WhatsApp in the UI
# Or test via API once authenticated

Post-Deployment

Step 1: Access Monitoring Stack

Your production monitoring stack provides complete observability with multiple tools:

Production Monitoring URLs

Access via domain (recommended):

https://monitoring.bakewise.ai/grafana       # Dashboards & visualization
https://monitoring.bakewise.ai/prometheus    # Metrics & queries
https://monitoring.bakewise.ai/signoz        # Unified observability platform (traces, metrics, logs)
https://monitoring.bakewise.ai/alertmanager  # Alert management

Or via port forwarding (if needed):

# Grafana
kubectl port-forward -n monitoring svc/grafana 3000:3000 &

# Prometheus
kubectl port-forward -n monitoring svc/prometheus-external 9090:9090 &

# SigNoz
kubectl port-forward -n monitoring svc/signoz-frontend 3301:3301 &

# AlertManager
kubectl port-forward -n monitoring svc/alertmanager-external 9093:9093 &

Available Dashboards

Main Dashboards:

Gateway Metrics - HTTP request rates, latencies, error rates
Services Overview - Multi-service health and performance
Circuit Breakers - Reliability metrics

Extended Dashboards: 4. Service Performance Monitoring (SPM) - RED metrics from distributed traces 5. PostgreSQL Database - Database health, connections, query performance 6. Node Exporter Infrastructure - CPU, memory, disk, network per node 7. AlertManager Monitoring - Alert tracking and notification status 8. Business Metrics & KPIs - Tenant activity, ML jobs, forecasts

Quick Health Check

# Verify all monitoring pods are running
kubectl get pods -n monitoring

# Check Prometheus targets (all should be UP)
kubectl port-forward -n monitoring svc/prometheus-external 9090:9090
# Open: http://localhost:9090/targets

# View active alerts
kubectl port-forward -n monitoring svc/prometheus-external 9090:9090
# Open: http://localhost:9090/alerts

Step 2: Configure Alerting

Update AlertManager with your notification email addresses:

# Edit alertmanager configuration
kubectl edit configmap -n monitoring alertmanager-config

# Update recipient emails in the routes section:
# - alerts@bakewise.ai (general alerts)
# - critical-alerts@bakewise.ai (critical issues)
# - oncall@bakewise.ai (on-call rotation)

Test alert delivery:

# Fire a test alert
kubectl run memory-test --image=polinux/stress --restart=Never \
  --namespace=bakery-ia -- stress --vm 1 --vm-bytes 600M --timeout 300s

# Check alert appears in AlertManager
# https://monitoring.bakewise.ai/alertmanager

# Verify email notification received

# Clean up test
kubectl delete pod memory-test -n bakery-ia

Step 3: Configure Backups

# Create backup script on VPS
cat > ~/backup-databases.sh <<'EOF'
#!/bin/bash
BACKUP_DIR="/backups/$(date +%Y-%m-%d)"
mkdir -p $BACKUP_DIR

# Get all database pods
DBS=$(kubectl get pods -n bakery-ia -l app.kubernetes.io/component=database -o name)

for db in $DBS; do
  DB_NAME=$(echo $db | cut -d'/' -f2)
  echo "Backing up $DB_NAME..."

  kubectl exec -n bakery-ia $db -- pg_dump -U postgres > "$BACKUP_DIR/${DB_NAME}.sql"
done

# Compress backups
tar -czf "$BACKUP_DIR.tar.gz" "$BACKUP_DIR"
rm -rf "$BACKUP_DIR"

# Keep only last 7 days
find /backups -name "*.tar.gz" -mtime +7 -delete

echo "Backup completed: $BACKUP_DIR.tar.gz"
EOF

chmod +x ~/backup-databases.sh

# Test backup
./backup-databases.sh

# Setup daily cron job (2 AM)
(crontab -l 2>/dev/null; echo "0 2 * * * ~/backup-databases.sh") | crontab -

Step 3: Setup Alerting

# Update AlertManager configuration with your email
kubectl edit configmap -n monitoring alertmanager-config

# Update recipient emails in the routes section

Step 4: Verify Monitoring is Working

Before proceeding, ensure all monitoring components are operational:

# 1. Check Prometheus targets
# Open: https://monitoring.bakewise.ai/prometheus/targets
# All targets should show "UP" status

# 2. Verify Grafana dashboards load data
# Open: https://monitoring.bakewise.ai/grafana
# Navigate to any dashboard and verify metrics are displaying

# 3. Check SigNoz is receiving traces
# Open: https://monitoring.bakewise.ai/signoz
# Search for traces from "gateway" service

# 4. Verify AlertManager cluster
# Open: https://monitoring.bakewise.ai/alertmanager
# Check that all 3 AlertManager instances are connected

Step 5: Document Everything

Create a secure runbook with all credentials and procedures:

Essential Information to Document:

VPS login credentials (stored securely in password manager)
Database passwords (in password manager)
Grafana admin password
Domain registrar access (for bakewise.ai)
Cloudflare access
Email service credentials (SMTP)
WhatsApp API credentials
Docker Hub / Registry credentials
Emergency contact information
Rollback procedures
Monitoring URLs and access procedures

Step 6: Train Your Team

Conduct a training session covering:

Access monitoring dashboards
- Show how to login to https://monitoring.bakewise.ai/grafana
- Walk through key dashboards (Services Overview, Database, Infrastructure)
- Explain how to interpret metrics and identify issues

Check application logs

# View logs for a service
kubectl logs -n bakery-ia deployment/orders-service --tail=100 -f

# Search for errors
kubectl logs -n bakery-ia deployment/gateway | grep ERROR

Restart services when needed

# Restart a service (rolling update, no downtime)
kubectl rollout restart deployment/orders-service -n bakery-ia

Respond to alerts
- Show how to access AlertManager at https://monitoring.bakewise.ai/alertmanager
- Review common alerts and their resolution steps
- Reference the Production Operations Guide
Share documentation
- PILOT_LAUNCH_GUIDE.md - This guide
- PRODUCTION_OPERATIONS_GUIDE.md - Daily operations
- security-checklist.md - Security procedures
Setup on-call rotation (if applicable)
- Configure in AlertManager
- Document escalation procedures

Troubleshooting

Issue: Pods Not Starting

# Check pod status
kubectl describe pod POD_NAME -n bakery-ia

# Common causes:
# 1. Image pull errors
kubectl get events -n bakery-ia | grep -i "pull"

# 2. Resource limits
kubectl describe node

# 3. Volume mount issues
kubectl get pvc -n bakery-ia

Issue: Certificate Not Issuing

# Check certificate status
kubectl describe certificate bakery-ia-prod-tls-cert -n bakery-ia

# Check cert-manager logs
kubectl logs -n cert-manager deployment/cert-manager

# Check challenges
kubectl get challenges -n bakery-ia

# Verify DNS is correct
nslookup bakery.yourdomain.com

Issue: Database Connection Errors

# Check database pod
kubectl get pods -n bakery-ia -l app.kubernetes.io/component=database

# Check database logs
kubectl logs -n bakery-ia deployment/auth-db

# Test connection from service pod
kubectl exec -n bakery-ia deployment/auth-service -- nc -zv auth-db 5432

Issue: Services Can't Connect to Databases

# Check if SSL is enabled
kubectl exec -n bakery-ia deployment/auth-db -- sh -c \
  'psql -U auth_user -d auth_db -c "SHOW ssl;"'

# Check service logs for SSL errors
kubectl logs -n bakery-ia deployment/auth-service | grep -i "ssl\|tls"

# Restart service to pick up new SSL config
kubectl rollout restart deployment/auth-service -n bakery-ia

Issue: Out of Resources

# Check node resources
kubectl top nodes

# Check pod resource usage
kubectl top pods -n bakery-ia

# Identify resource hogs
kubectl top pods -n bakery-ia --sort-by=memory

# Scale down non-critical services temporarily
kubectl scale deployment monitoring -n bakery-ia --replicas=0

Next Steps After Successful Launch

Monitor for 48 Hours
- Check dashboards daily
- Review error logs
- Monitor resource usage
- Test all functionality
Optimize Based on Metrics
- Adjust resource limits if needed
- Fine-tune autoscaling thresholds
- Optimize database queries if slow
Onboard First Tenant
- Create test tenant
- Upload sample data
- Test all features
- Gather feedback
Scale Gradually
- Add 1-2 tenants at a time
- Monitor resource usage
- Upgrade VPS if needed (see scaling guide)
Plan for Growth
- Review PRODUCTION_OPERATIONS_GUIDE.md
- Implement additional monitoring
- Plan capacity upgrades
- Consider managed services for scale

Cost Scaling Path

Tenants	RAM	CPU	Storage	Monthly Cost
10	20 GB	8 cores	200 GB	€40-80
25	32 GB	12 cores	300 GB	€80-120
50	48 GB	16 cores	500 GB	€150-200
100+	Consider multi-node cluster or managed K8s	€300+

Support Resources

Documentation:

Operations Guide: PRODUCTION_OPERATIONS_GUIDE.md - Daily operations, monitoring, incident response
Security Guide: security-checklist.md - Security procedures and compliance
Database Security: database-security.md - Database operations and TLS configuration
TLS Configuration: tls-configuration.md - Certificate management
RBAC Implementation: rbac-implementation.md - Access control

Monitoring Access:

Grafana: https://monitoring.bakewise.ai/grafana (admin / your-password)
Prometheus: https://monitoring.bakewise.ai/prometheus
SigNoz: https://monitoring.bakewise.ai/signoz
AlertManager: https://monitoring.bakewise.ai/alertmanager

External Resources:

MicroK8s Docs: https://microk8s.io/docs
Kubernetes Docs: https://kubernetes.io/docs
Let's Encrypt: https://letsencrypt.org/docs
Cloudflare DNS: https://developers.cloudflare.com/dns
Monitoring Stack README: infrastructure/kubernetes/base/components/monitoring/README.md

Summary Checklist

Before going live, ensure:

VPS provisioned and accessible
MicroK8s installed and configured
Domain registered and DNS configured
Cloudflare protection enabled
TLS certificates generated
Email service configured and tested
WhatsApp API setup (optional for launch)
Container images built and pushed
Production configs updated (domains, CORS, etc.)
Secrets generated (strong passwords!)
All pods running successfully
Databases accepting TLS connections
Let's Encrypt certificates issued
Frontend accessible via HTTPS
API health check passing
Test user can login
Email delivery working
Monitoring dashboards loading
Backups configured and tested
Team trained on operations
Documentation complete
Emergency procedures documented

🎉 Congratulations! Your Bakery-IA platform is now live in production!

Estimated total time: 2-4 hours for first deployment Subsequent updates: 15-30 minutes

Document Version: 1.0 Last Updated: 2026-01-07 Maintained By: DevOps Team

31 KiB Raw Blame History Unescape Escape

Bakery-IA Pilot Launch Guide

Table of Contents

Executive Summary

What You're Deploying

Total Cost Breakdown

Timeline

Pre-Launch Checklist

Required Accounts & Services

Required Tools on Local Machine

Repository Setup

VPS Provisioning

Recommended Configuration

Why These Specs?

Provisioning Steps

Infrastructure Setup

Step 1: Install MicroK8s

Step 2: Enable Required Add-ons

Step 3: Configure Firewall

Step 4: Create Namespace

Domain & DNS Configuration

Step 1: Register Domain

Step 2: Configure Cloudflare DNS (Recommended)

TLS/SSL Certificates

Understanding Certificate Setup

Step 1: Generate Internal Certificates

Step 2: Create Kubernetes Secrets

Step 3: Configure Let's Encrypt (External SSL)

Email & Communication Setup

Option A: Zoho Mail (FREE, Recommended)

Option B: Gmail SMTP + Forwarding

WhatsApp Business API Setup

Kubernetes Deployment

Step 1: Prepare Container Images

Option A: Using Docker Hub (Recommended)

Option B: Using MicroK8s Registry

Step 2: Update Production Configuration

Configuration & Secrets

Step 1: Generate Strong Passwords

Step 2: Update Application Secrets

Step 3: Apply Secrets

Database Migrations

Step 1: Deploy Databases

Step 2: Run Migrations

Step 3: Verify Database Schemas

Verification & Testing

Step 1: Check All Pods Running

Step 2: Check Services and Ingress

Step 3: Test Database Connections

Step 4: Test Frontend Access

Step 5: Test Authentication

Step 6: Test Email Delivery

Step 7: Test WhatsApp (Optional)

Post-Deployment

Step 1: Access Monitoring Stack

Production Monitoring URLs

Available Dashboards

Quick Health Check

Step 2: Configure Alerting

Step 3: Configure Backups

Step 3: Setup Alerting

Step 4: Verify Monitoring is Working

Step 5: Document Everything

Step 6: Train Your Team

Troubleshooting

Issue: Pods Not Starting

Issue: Certificate Not Issuing

Issue: Database Connection Errors

Issue: Services Can't Connect to Databases

Issue: Out of Resources

Next Steps After Successful Launch

Cost Scaling Path

Support Resources

Summary Checklist

31 KiB

Raw Blame History