Add new infra architecture 5
This commit is contained in:
Urtzi Alfaro
@@ -433,6 +433,45 @@ microk8s enable prometheus
|
||||
microk8s enable registry
|
||||
```
|
||||
|
||||
### Step 3: Enhanced Infrastructure Components
|
||||
|
||||
**The platform includes additional infrastructure components that enhance security, monitoring, and operations:**
|
||||
|
||||
```bash
|
||||
# The platform includes Mailu for email services
|
||||
# Deploy Mailu via Helm (optional but recommended for production):
|
||||
kubectl create namespace bakery-ia --dry-run=client -o yaml | kubectl apply -f -
|
||||
helm repo add mailu https://mailu.github.io/helm-charts
|
||||
helm repo update
|
||||
helm install mailu mailu/mailu \
|
||||
-n bakery-ia \
|
||||
-f infrastructure/platform/mail/mailu-helm/values.yaml \
|
||||
--timeout 10m \
|
||||
--wait
|
||||
|
||||
# Verify Mailu deployment
|
||||
kubectl get pods -n bakery-ia | grep mailu
|
||||
```
|
||||
|
||||
**For development environments, ensure the prepull-base-images script is run:**
|
||||
```bash
|
||||
# On your local machine, run the prepull script to cache base images
|
||||
cd bakery-ia
|
||||
chmod +x scripts/prepull-base-images.sh
|
||||
./scripts/prepull-base-images.sh
|
||||
```
|
||||
|
||||
**For production environments, ensure CI/CD infrastructure is properly configured:**
|
||||
```bash
|
||||
# Tekton Pipelines for CI/CD (optional - can be deployed separately)
|
||||
kubectl create namespace tekton-pipelines
|
||||
kubectl apply -f https://storage.googleapis.com/tekton-releases/pipeline/latest/release.yaml
|
||||
kubectl apply -f https://storage.googleapis.com/tekton-releases/triggers/latest/release.yaml
|
||||
|
||||
# Flux CD for GitOps (already enabled in MicroK8s if needed)
|
||||
# flux install --namespace=flux-system --network-policy=false
|
||||
```
|
||||
|
||||
### Step 3: Configure Firewall
|
||||
|
||||
```bash
|
||||
@@ -917,7 +956,34 @@ echo -n "your-value-here" | base64
|
||||
|
||||
**CRITICAL:** Never commit real secrets to git! The secrets.yaml file should be in `.gitignore`.
|
||||
|
||||
### Step 2: Apply Application Secrets
|
||||
### Step 2: CI/CD Secrets Configuration
|
||||
|
||||
**For production CI/CD setup, additional secrets are required:**
|
||||
|
||||
```bash
|
||||
# Create Docker Hub credentials secret (for image pulls)
|
||||
kubectl create secret docker-registry dockerhub-creds \
|
||||
--docker-server=docker.io \
|
||||
--docker-username=YOUR_DOCKERHUB_USERNAME \
|
||||
--docker-password=YOUR_DOCKERHUB_TOKEN \
|
||||
--docker-email=your-email@example.com \
|
||||
-n bakery-ia
|
||||
|
||||
# Create Gitea registry credentials (if using Gitea for CI/CD)
|
||||
kubectl create secret docker-registry gitea-registry-credentials \
|
||||
-n tekton-pipelines \
|
||||
--docker-server=gitea.bakery-ia.local:5000 \
|
||||
--docker-username=your-username \
|
||||
--docker-password=your-password
|
||||
|
||||
# Create Git credentials for Flux (if using GitOps)
|
||||
kubectl create secret generic gitea-credentials \
|
||||
-n flux-system \
|
||||
--from-literal=username=your-username \
|
||||
--from-literal=password=your-password
|
||||
```
|
||||
|
||||
### Step 3: Apply Application Secrets
|
||||
|
||||
```bash
|
||||
# Copy manifests to VPS (from local machine)
|
||||
@@ -938,7 +1004,30 @@ kubectl get secrets -n bakery-ia
|
||||
|
||||
## Database Migrations
|
||||
|
||||
### Step 0: Deploy SigNoz Monitoring (BEFORE Application)
|
||||
### Step 0: Deploy CI/CD Infrastructure (Optional but Recommended)
|
||||
|
||||
**For production environments, deploy CI/CD infrastructure components:**
|
||||
|
||||
```bash
|
||||
# Deploy Tekton Pipelines for CI/CD (optional but recommended for production)
|
||||
kubectl create namespace tekton-pipelines
|
||||
|
||||
# Install Tekton Pipelines
|
||||
kubectl apply -f https://storage.googleapis.com/tekton-releases/pipeline/latest/release.yaml
|
||||
|
||||
# Install Tekton Triggers
|
||||
kubectl apply -f https://storage.googleapis.com/tekton-releases/triggers/latest/release.yaml
|
||||
|
||||
# Apply Tekton configurations
|
||||
kubectl apply -f ~/infrastructure/cicd/tekton/tasks/
|
||||
kubectl apply -f ~/infrastructure/cicd/tekton/pipelines/
|
||||
kubectl apply -f ~/infrastructure/cicd/tekton/triggers/
|
||||
|
||||
# Verify Tekton deployment
|
||||
kubectl get pods -n tekton-pipelines
|
||||
```
|
||||
|
||||
### Step 1: Deploy SigNoz Monitoring (BEFORE Application)
|
||||
|
||||
**⚠️ CRITICAL:** SigNoz must be deployed BEFORE the application into the **bakery-ia namespace** because the production kustomization patches SigNoz resources.
|
||||
|
||||
@@ -975,7 +1064,7 @@ kubectl get statefulset -n bakery-ia | grep signoz
|
||||
|
||||
**⚠️ Important:** Do NOT create a separate `signoz` namespace. SigNoz must be in `bakery-ia` namespace for the overlays to work correctly.
|
||||
|
||||
### Step 1: Deploy Application and Databases
|
||||
### Step 2: Deploy Application and Databases
|
||||
|
||||
```bash
|
||||
# On VPS
|
||||
@@ -1271,6 +1360,88 @@ kubectl logs -n bakery-ia deployment/signoz-otel-collector --tail=50 | grep -i "
|
||||
kubectl logs -n bakery-ia deployment/signoz-otel-collector | grep filelog
|
||||
```
|
||||
|
||||
### Step 2: Configure CI/CD Infrastructure (Optional but Recommended)
|
||||
|
||||
If you deployed the CI/CD infrastructure, configure it for your workflow:
|
||||
|
||||
#### Gitea Setup (Git Server + Registry)
|
||||
```bash
|
||||
# Access Gitea at: http://gitea.bakery-ia.local (for dev) or http://gitea.bakewise.ai (for prod)
|
||||
# Make sure to add the appropriate hostname to /etc/hosts or configure DNS
|
||||
|
||||
# Create your repositories for each service
|
||||
# Configure webhook to trigger Tekton pipelines
|
||||
```
|
||||
|
||||
#### Tekton Pipeline Configuration
|
||||
```bash
|
||||
# Verify Tekton pipelines are running
|
||||
kubectl get pods -n tekton-pipelines
|
||||
|
||||
# Create a PipelineRun manually to test:
|
||||
kubectl create -f - <<EOF
|
||||
apiVersion: tekton.dev/v1beta1
|
||||
kind: PipelineRun
|
||||
metadata:
|
||||
name: manual-ci-run
|
||||
namespace: tekton-pipelines
|
||||
spec:
|
||||
pipelineRef:
|
||||
name: bakery-ia-ci
|
||||
workspaces:
|
||||
- name: shared-workspace
|
||||
volumeClaimTemplate:
|
||||
spec:
|
||||
accessModes: ["ReadWriteOnce"]
|
||||
resources:
|
||||
requests:
|
||||
storage: 5Gi
|
||||
- name: docker-credentials
|
||||
secret:
|
||||
secretName: gitea-registry-credentials
|
||||
params:
|
||||
- name: git-url
|
||||
value: "http://gitea.bakery-ia.local/bakery/bakery-ia.git"
|
||||
- name: git-revision
|
||||
value: "main"
|
||||
EOF
|
||||
```
|
||||
|
||||
#### Flux CD Configuration (GitOps)
|
||||
```bash
|
||||
# Verify Flux is running
|
||||
kubectl get pods -n flux-system
|
||||
|
||||
# Set up GitRepository and Kustomization resources for GitOps deployment
|
||||
# Example:
|
||||
cat <<EOF | kubectl apply -f -
|
||||
apiVersion: source.toolkit.fluxcd.io/v1
|
||||
kind: GitRepository
|
||||
metadata:
|
||||
name: bakery-ia
|
||||
namespace: flux-system
|
||||
spec:
|
||||
interval: 1m
|
||||
url: https://github.com/your-org/bakery-ia.git
|
||||
ref:
|
||||
branch: main
|
||||
---
|
||||
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
||||
kind: Kustomization
|
||||
metadata:
|
||||
name: bakery-ia
|
||||
namespace: flux-system
|
||||
spec:
|
||||
interval: 5m
|
||||
sourceRef:
|
||||
kind: GitRepository
|
||||
name: bakery-ia
|
||||
path: ./infrastructure/environments/prod/k8s-manifests
|
||||
prune: true
|
||||
validation: client
|
||||
EOF
|
||||
```
|
||||
|
||||
### Step 2: Configure Alerting
|
||||
|
||||
SigNoz includes integrated alerting with AlertManager. Configure it for your team:
|
||||
|
||||
@@ -12,14 +12,15 @@
|
||||
|
||||
1. [Overview](#overview)
|
||||
2. [Monitoring & Observability](#monitoring--observability)
|
||||
3. [Security Operations](#security-operations)
|
||||
4. [Database Management](#database-management)
|
||||
5. [Backup & Recovery](#backup--recovery)
|
||||
6. [Performance Optimization](#performance-optimization)
|
||||
7. [Scaling Operations](#scaling-operations)
|
||||
8. [Incident Response](#incident-response)
|
||||
9. [Maintenance Tasks](#maintenance-tasks)
|
||||
10. [Compliance & Audit](#compliance--audit)
|
||||
3. [CI/CD Operations](#ci-cd-operations)
|
||||
4. [Security Operations](#security-operations)
|
||||
5. [Database Management](#database-management)
|
||||
6. [Backup & Recovery](#backup--recovery)
|
||||
7. [Performance Optimization](#performance-optimization)
|
||||
8. [Scaling Operations](#scaling-operations)
|
||||
9. [Incident Response](#incident-response)
|
||||
10. [Maintenance Tasks](#maintenance-tasks)
|
||||
11. [Compliance & Audit](#compliance--audit)
|
||||
|
||||
---
|
||||
|
||||
@@ -33,6 +34,8 @@
|
||||
- **Capacity:** 10-tenant pilot (scalable to 100+)
|
||||
- **Security:** TLS encryption, RBAC, audit logging
|
||||
- **Monitoring:** Prometheus, Grafana, AlertManager, SigNoz
|
||||
- **CI/CD:** Tekton Pipelines, Gitea, Flux CD (GitOps)
|
||||
- **Email:** Mailu (integrated email server)
|
||||
|
||||
**Key Metrics (10-tenant baseline):**
|
||||
- **Uptime Target:** 99.5% (3.65 hours downtime/month)
|
||||
@@ -46,11 +49,12 @@
|
||||
|
||||
| Role | Responsibilities |
|
||||
|------|------------------|
|
||||
| **DevOps Engineer** | Deployment, infrastructure, scaling |
|
||||
| **DevOps Engineer** | Deployment, infrastructure, scaling, CI/CD |
|
||||
| **SRE** | Monitoring, incident response, performance |
|
||||
| **Security Admin** | Access control, security patches, compliance |
|
||||
| **Database Admin** | Backups, optimization, migrations |
|
||||
| **On-Call Engineer** | 24/7 incident response (if applicable) |
|
||||
| **CI/CD Admin** | Pipeline management, GitOps workflows |
|
||||
|
||||
---
|
||||
|
||||
@@ -73,18 +77,6 @@ SigNoz is a comprehensive, open-source observability platform that provides:
|
||||
- **Database Monitoring** - All 18 PostgreSQL databases + Redis + RabbitMQ
|
||||
- **Kubernetes Monitoring** - Cluster, node, pod, and container metrics
|
||||
|
||||
**Port Forwarding (if ingress not available):**
|
||||
```bash
|
||||
# SigNoz Frontend (Main UI)
|
||||
kubectl port-forward -n bakery-ia svc/signoz 8080:8080
|
||||
|
||||
# SigNoz AlertManager
|
||||
kubectl port-forward -n bakery-ia svc/signoz-alertmanager 9093:9093
|
||||
|
||||
# OTel Collector (for debugging)
|
||||
kubectl port-forward -n bakery-ia svc/signoz-otel-collector 4317:4317 # gRPC
|
||||
kubectl port-forward -n bakery-ia svc/signoz-otel-collector 4318:4318 # HTTP
|
||||
```
|
||||
|
||||
### Key SigNoz Dashboards and Features
|
||||
|
||||
@@ -340,6 +332,116 @@ kubectl logs -n bakery-ia deployment/signoz-otel-collector | grep k8sattributes
|
||||
|
||||
---
|
||||
|
||||
## CI/CD Operations
|
||||
|
||||
### CI/CD Infrastructure Overview
|
||||
|
||||
The platform includes a complete CI/CD pipeline using:
|
||||
- **Gitea** - Git server and container registry
|
||||
- **Tekton** - Pipeline automation
|
||||
- **Flux CD** - GitOps deployment
|
||||
|
||||
### Access CI/CD Systems
|
||||
|
||||
**Gitea (Git Server):**
|
||||
- URL: http://gitea.bakery-ia.local (development) or http://gitea.bakewise.ai (production)
|
||||
- Admin panel: http://gitea.bakery-ia.local/admin
|
||||
|
||||
**Tekton Dashboard:**
|
||||
```bash
|
||||
# Port forward to access Tekton dashboard
|
||||
kubectl port-forward -n tekton-pipelines svc/tekton-dashboard 9097:9097
|
||||
# Access at: http://localhost:9097
|
||||
```
|
||||
|
||||
**Flux Status:**
|
||||
```bash
|
||||
# Check Flux status
|
||||
flux check
|
||||
kubectl get gitrepository -n flux-system
|
||||
kubectl get kustomization -n flux-system
|
||||
```
|
||||
|
||||
### CI/CD Monitoring
|
||||
|
||||
**Check pipeline status:**
|
||||
```bash
|
||||
# List all PipelineRuns
|
||||
kubectl get pipelineruns -n tekton-pipelines
|
||||
|
||||
# Check Tekton controller logs
|
||||
kubectl logs -n tekton-pipelines -l app=tekton-pipelines-controller
|
||||
|
||||
# Check Tekton dashboard logs
|
||||
kubectl logs -n tekton-pipelines -l app=tekton-dashboard
|
||||
```
|
||||
|
||||
**Monitor GitOps synchronization:**
|
||||
```bash
|
||||
# Check GitRepository status
|
||||
kubectl get gitrepository -n flux-system -o wide
|
||||
|
||||
# Check Kustomization status
|
||||
kubectl get kustomization -n flux-system -o wide
|
||||
|
||||
# Get reconciliation history
|
||||
kubectl get events -n flux-system --sort-by='.lastTimestamp'
|
||||
```
|
||||
|
||||
### CI/CD Troubleshooting
|
||||
|
||||
**Pipeline not triggering:**
|
||||
```bash
|
||||
# Check Gitea webhook logs
|
||||
kubectl logs -n tekton-pipelines -l app=tekton-triggers-controller
|
||||
|
||||
# Verify EventListener pods are running
|
||||
kubectl get pods -n tekton-pipelines -l app=tekton-triggers-eventlistener
|
||||
|
||||
# Check TriggerBinding configuration
|
||||
kubectl get triggerbinding -n tekton-pipelines
|
||||
```
|
||||
|
||||
**Build failures:**
|
||||
```bash
|
||||
# Check Kaniko logs for build errors
|
||||
kubectl logs -n tekton-pipelines -l tekton.dev/task=kaniko-build
|
||||
|
||||
# Verify Dockerfile paths are correct
|
||||
kubectl describe taskrun -n tekton-pipelines
|
||||
```
|
||||
|
||||
**Flux not applying changes:**
|
||||
```bash
|
||||
# Check GitRepository status
|
||||
kubectl describe gitrepository -n flux-system
|
||||
|
||||
# Check Kustomization reconciliation
|
||||
kubectl describe kustomization -n flux-system
|
||||
|
||||
# Check Flux logs
|
||||
kubectl logs -n flux-system -l app.kubernetes.io/name=helm-controller
|
||||
```
|
||||
|
||||
### CI/CD Maintenance Tasks
|
||||
|
||||
**Daily Tasks:**
|
||||
- [ ] Check for failed pipeline runs
|
||||
- [ ] Verify GitOps synchronization status
|
||||
- [ ] Clean up old PipelineRun resources
|
||||
|
||||
**Weekly Tasks:**
|
||||
- [ ] Review pipeline performance metrics
|
||||
- [ ] Update pipeline definitions if needed
|
||||
- [ ] Rotate CI/CD secrets
|
||||
|
||||
**Monthly Tasks:**
|
||||
- [ ] Update Tekton and Flux versions
|
||||
- [ ] Review and optimize pipeline performance
|
||||
- [ ] Audit CI/CD access permissions
|
||||
|
||||
---
|
||||
|
||||
## Security Operations
|
||||
|
||||
### Security Posture Overview
|
||||
@@ -1210,6 +1312,8 @@ kubectl exec -n bakery-ia deployment/auth-db -- \
|
||||
- [TLS Configuration](./tls-configuration.md) - Certificate management
|
||||
- [RBAC Implementation](./rbac-implementation.md) - Access control configuration
|
||||
- [Monitoring Stack README](../infrastructure/kubernetes/base/components/monitoring/README.md) - Detailed monitoring documentation
|
||||
- [CI/CD Infrastructure README](../infrastructure/cicd/README.md) - Gitea, Tekton, and Flux CD setup and operations
|
||||
- [SigNoz Monitoring README](../infrastructure/monitoring/signoz/README.md) - SigNoz deployment and configuration
|
||||
|
||||
**External Resources:**
|
||||
- Kubernetes: https://kubernetes.io/docs
|
||||
|
||||
Reference in New Issue
Block a user