Add ci/cd and fix multiple pods issues

This commit is contained in:
Urtzi Alfaro
2026-01-18 09:02:27 +01:00
parent 3c4b5c2a06
commit 21d35ea92b
27 changed files with 3779 additions and 73 deletions

View File

@@ -0,0 +1,294 @@
# Bakery-IA CI/CD Implementation
This directory contains the configuration for the production-grade CI/CD system for Bakery-IA using Gitea, Tekton, and Flux CD.
## Architecture Overview
```mermaid
graph TD
A[Developer] -->|Push Code| B[Gitea]
B -->|Webhook| C[Tekton Pipelines]
C -->|Build/Test| D[Gitea Registry]
D -->|New Image| E[Flux CD]
E -->|kubectl apply| F[MicroK8s Cluster]
F -->|Metrics| G[SigNoz]
```
## Directory Structure
```
infrastructure/ci-cd/
├── gitea/ # Gitea configuration (Git server + registry)
│ ├── values.yaml # Helm values for Gitea
│ └── ingress.yaml # Ingress configuration
├── tekton/ # Tekton CI/CD pipeline configuration
│ ├── tasks/ # Individual pipeline tasks
│ │ ├── git-clone.yaml
│ │ ├── detect-changes.yaml
│ │ ├── kaniko-build.yaml
│ │ └── update-gitops.yaml
│ ├── pipelines/ # Pipeline definitions
│ │ └── ci-pipeline.yaml
│ └── triggers/ # Webhook trigger configuration
│ ├── trigger-template.yaml
│ ├── trigger-binding.yaml
│ ├── event-listener.yaml
│ └── gitlab-interceptor.yaml
├── flux/ # Flux CD GitOps configuration
│ ├── git-repository.yaml # Git repository source
│ └── kustomization.yaml # Deployment kustomization
├── monitoring/ # Monitoring configuration
│ └── otel-collector.yaml # OpenTelemetry collector
└── README.md # This file
```
## Deployment Instructions
### Phase 1: Infrastructure Setup
1. **Deploy Gitea**:
```bash
# Add Helm repo
microk8s helm repo add gitea https://dl.gitea.io/charts
# Create namespace
microk8s kubectl create namespace gitea
# Install Gitea
microk8s helm install gitea gitea/gitea \
-n gitea \
-f infrastructure/ci-cd/gitea/values.yaml
# Apply ingress
microk8s kubectl apply -f infrastructure/ci-cd/gitea/ingress.yaml
```
2. **Deploy Tekton**:
```bash
# Create namespace
microk8s kubectl create namespace tekton-pipelines
# Install Tekton Pipelines
microk8s kubectl apply -f https://storage.googleapis.com/tekton-releases/pipeline/latest/release.yaml
# Install Tekton Triggers
microk8s kubectl apply -f https://storage.googleapis.com/tekton-releases/triggers/latest/release.yaml
# Apply Tekton configurations
microk8s kubectl apply -f infrastructure/ci-cd/tekton/tasks/
microk8s kubectl apply -f infrastructure/ci-cd/tekton/pipelines/
microk8s kubectl apply -f infrastructure/ci-cd/tekton/triggers/
```
3. **Deploy Flux CD** (already enabled in MicroK8s):
```bash
# Verify Flux installation
microk8s kubectl get pods -n flux-system
# Apply Flux configurations
microk8s kubectl apply -f infrastructure/ci-cd/flux/
```
### Phase 2: Configuration
1. **Set up Gitea webhook**:
- Go to your Gitea repository settings
- Add webhook with URL: `http://tekton-triggers.tekton-pipelines.svc.cluster.local:8080`
- Use the secret from `gitea-webhook-secret`
2. **Configure registry credentials**:
```bash
# Create registry credentials secret
microk8s kubectl create secret docker-registry gitea-registry-credentials \
-n tekton-pipelines \
--docker-server=gitea.bakery-ia.local:5000 \
--docker-username=your-username \
--docker-password=your-password
```
3. **Configure Git credentials for Flux**:
```bash
# Create Git credentials secret
microk8s kubectl create secret generic gitea-credentials \
-n flux-system \
--from-literal=username=your-username \
--from-literal=password=your-password
```
### Phase 3: Monitoring
```bash
# Apply OpenTelemetry configuration
microk8s kubectl apply -f infrastructure/ci-cd/monitoring/otel-collector.yaml
```
## Usage
### Triggering a Pipeline
1. **Manual trigger**:
```bash
# Create a PipelineRun manually
microk8s kubectl create -f - <<EOF
apiVersion: tekton.dev/v1beta1
kind: PipelineRun
metadata:
name: manual-ci-run
namespace: tekton-pipelines
spec:
pipelineRef:
name: bakery-ia-ci
workspaces:
- name: shared-workspace
volumeClaimTemplate:
spec:
accessModes: ["ReadWriteOnce"]
resources:
requests:
storage: 5Gi
- name: docker-credentials
secret:
secretName: gitea-registry-credentials
params:
- name: git-url
value: "http://gitea.bakery-ia.local/bakery/bakery-ia.git"
- name: git-revision
value: "main"
EOF
```
2. **Automatic trigger**: Push code to the repository and the webhook will trigger the pipeline automatically.
### Monitoring Pipeline Runs
```bash
# List all PipelineRuns
microk8s kubectl get pipelineruns -n tekton-pipelines
# View logs for a specific PipelineRun
microk8s kubectl logs -n tekton-pipelines <pipelinerun-pod> -c <step-name>
# View Tekton dashboard
microk8s kubectl port-forward -n tekton-pipelines svc/tekton-dashboard 9097:9097
```
## Troubleshooting
### Common Issues
1. **Pipeline not triggering**:
- Check Gitea webhook logs
- Verify EventListener pods are running
- Check TriggerBinding configuration
2. **Build failures**:
- Check Kaniko logs for build errors
- Verify Dockerfile paths are correct
- Ensure registry credentials are valid
3. **Flux not applying changes**:
- Check GitRepository status
- Verify Kustomization reconciliation
- Check Flux logs for errors
### Debugging Commands
```bash
# Check Tekton controller logs
microk8s kubectl logs -n tekton-pipelines -l app=tekton-pipelines-controller
# Check Flux reconciliation
microk8s kubectl get kustomizations -n flux-system -o yaml
# Check Gitea webhook delivery
microk8s kubectl logs -n tekton-pipelines -l app=tekton-triggers-controller
```
## Security Considerations
1. **Secrets Management**:
- Use Kubernetes secrets for sensitive data
- Rotate credentials regularly
- Use RBAC for namespace isolation
2. **Network Security**:
- Configure network policies
- Use internal DNS names
- Restrict ingress access
3. **Registry Security**:
- Enable image scanning
- Use image signing
- Implement cleanup policies
## Maintenance
### Upgrading Components
```bash
# Upgrade Tekton
microk8s kubectl apply -f https://storage.googleapis.com/tekton-releases/pipeline/latest/release.yaml
# Upgrade Flux
microk8s helm upgrade fluxcd fluxcd/flux2 -n flux-system
# Upgrade Gitea
microk8s helm upgrade gitea gitea/gitea -n gitea -f infrastructure/ci-cd/gitea/values.yaml
```
### Backup Procedures
```bash
# Backup Gitea
microk8s kubectl exec -n gitea gitea-0 -- gitea dump -c /data/gitea/conf/app.ini
# Backup Flux configurations
microk8s kubectl get all -n flux-system -o yaml > flux-backup.yaml
# Backup Tekton configurations
microk8s kubectl get all -n tekton-pipelines -o yaml > tekton-backup.yaml
```
## Performance Optimization
1. **Resource Management**:
- Set appropriate resource limits
- Limit concurrent builds
- Use node selectors for build pods
2. **Caching**:
- Configure Kaniko cache
- Use persistent volumes for dependencies
- Cache Docker layers
3. **Parallelization**:
- Build independent services in parallel
- Use matrix builds for different architectures
- Optimize task dependencies
## Integration with Existing System
The CI/CD system integrates with:
- **SigNoz**: For monitoring and observability
- **MicroK8s**: For cluster management
- **Existing Kubernetes manifests**: In `infrastructure/kubernetes/`
- **Current services**: All 19 microservices in `services/`
## Migration Plan
1. **Phase 1**: Set up infrastructure (Gitea, Tekton, Flux)
2. **Phase 2**: Configure pipelines and triggers
3. **Phase 3**: Test with non-critical services
4. **Phase 4**: Gradual rollout to all services
5. **Phase 5**: Decommission old deployment methods
## Support
For issues with the CI/CD system:
- Check logs and monitoring first
- Review the troubleshooting section
- Consult the original implementation plan
- Refer to component documentation:
- [Tekton Documentation](https://tekton.dev/docs/)
- [Flux CD Documentation](https://fluxcd.io/docs/)
- [Gitea Documentation](https://docs.gitea.io/)

View File

@@ -0,0 +1,16 @@
# Flux GitRepository for Bakery-IA
# This resource tells Flux where to find the Git repository
apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
name: bakery-ia
namespace: flux-system
spec:
interval: 1m
url: http://gitea.bakery-ia.local/bakery/bakery-ia.git
ref:
branch: main
secretRef:
name: gitea-credentials
timeout: 60s

View File

@@ -0,0 +1,27 @@
# Flux Kustomization for Bakery-IA Production Deployment
# This resource tells Flux how to deploy the application
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: bakery-ia-prod
namespace: flux-system
spec:
interval: 5m
path: ./infrastructure/kubernetes/overlays/prod
prune: true
sourceRef:
kind: GitRepository
name: bakery-ia
targetNamespace: bakery-ia
timeout: 5m
retryInterval: 1m
healthChecks:
- apiVersion: apps/v1
kind: Deployment
name: auth-service
namespace: bakery-ia
- apiVersion: apps/v1
kind: Deployment
name: gateway
namespace: bakery-ia

View File

@@ -0,0 +1,25 @@
# Gitea Ingress configuration for Bakery-IA CI/CD
# This provides external access to Gitea within the cluster
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: gitea-ingress
namespace: gitea
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /
nginx.ingress.kubernetes.io/proxy-body-size: "0"
nginx.ingress.kubernetes.io/proxy-read-timeout: "600"
nginx.ingress.kubernetes.io/proxy-send-timeout: "600"
spec:
rules:
- host: gitea.bakery-ia.local
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: gitea-http
port:
number: 3000

View File

@@ -0,0 +1,38 @@
# Gitea Helm values configuration for Bakery-IA CI/CD
# This configuration sets up Gitea with registry support and appropriate storage
service:
type: ClusterIP
httpPort: 3000
sshPort: 2222
persistence:
enabled: true
size: 50Gi
storageClass: "microk8s-hostpath"
gitea:
config:
server:
DOMAIN: gitea.bakery-ia.local
SSH_DOMAIN: gitea.bakery-ia.local
ROOT_URL: http://gitea.bakery-ia.local
repository:
ENABLE_PUSH_CREATE_USER: true
ENABLE_PUSH_CREATE_ORG: true
registry:
ENABLED: true
postgresql:
enabled: true
persistence:
size: 20Gi
# Resource configuration for production environment
resources:
limits:
cpu: 1000m
memory: 1Gi
requests:
cpu: 500m
memory: 512Mi

View File

@@ -0,0 +1,70 @@
# OpenTelemetry Collector for Bakery-IA CI/CD Monitoring
# This collects metrics and traces from Tekton pipelines
apiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector
metadata:
name: tekton-otel
namespace: tekton-pipelines
spec:
config: |
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
http:
endpoint: 0.0.0.0:4318
prometheus:
config:
scrape_configs:
- job_name: 'tekton-pipelines'
scrape_interval: 30s
static_configs:
- targets: ['tekton-pipelines-controller.tekton-pipelines.svc.cluster.local:9090']
processors:
batch:
timeout: 5s
send_batch_size: 1000
memory_limiter:
check_interval: 2s
limit_percentage: 75
spike_limit_percentage: 20
exporters:
otlp:
endpoint: "signoz-otel-collector.monitoring.svc.cluster.local:4317"
tls:
insecure: true
retry_on_failure:
enabled: true
initial_interval: 5s
max_interval: 30s
max_elapsed_time: 300s
logging:
logLevel: debug
service:
pipelines:
traces:
receivers: [otlp]
processors: [memory_limiter, batch]
exporters: [otlp, logging]
metrics:
receivers: [otlp, prometheus]
processors: [memory_limiter, batch]
exporters: [otlp, logging]
telemetry:
logs:
level: "info"
encoding: "json"
mode: deployment
resources:
limits:
cpu: 500m
memory: 512Mi
requests:
cpu: 200m
memory: 256Mi

View File

@@ -0,0 +1,83 @@
# Main CI Pipeline for Bakery-IA
# This pipeline orchestrates the build, test, and deploy process
apiVersion: tekton.dev/v1beta1
kind: Pipeline
metadata:
name: bakery-ia-ci
namespace: tekton-pipelines
spec:
workspaces:
- name: shared-workspace
- name: docker-credentials
params:
- name: git-url
type: string
description: Repository URL
- name: git-revision
type: string
description: Git revision/commit hash
- name: registry
type: string
description: Container registry URL
default: "gitea.bakery-ia.local:5000"
tasks:
- name: fetch-source
taskRef:
name: git-clone
workspaces:
- name: output
workspace: shared-workspace
params:
- name: url
value: $(params.git-url)
- name: revision
value: $(params.git-revision)
- name: detect-changes
runAfter: [fetch-source]
taskRef:
name: detect-changed-services
workspaces:
- name: source
workspace: shared-workspace
- name: build-and-push
runAfter: [detect-changes]
taskRef:
name: kaniko-build
when:
- input: "$(tasks.detect-changes.results.changed-services)"
operator: notin
values: ["none"]
workspaces:
- name: source
workspace: shared-workspace
- name: docker-credentials
workspace: docker-credentials
params:
- name: services
value: $(tasks.detect-changes.results.changed-services)
- name: registry
value: $(params.registry)
- name: git-revision
value: $(params.git-revision)
- name: update-gitops-manifests
runAfter: [build-and-push]
taskRef:
name: update-gitops
when:
- input: "$(tasks.detect-changes.results.changed-services)"
operator: notin
values: ["none"]
workspaces:
- name: source
workspace: shared-workspace
params:
- name: services
value: $(tasks.detect-changes.results.changed-services)
- name: registry
value: $(params.registry)
- name: git-revision
value: $(params.git-revision)

View File

@@ -0,0 +1,64 @@
# Tekton Detect Changed Services Task for Bakery-IA CI/CD
# This task identifies which services have changed in the repository
apiVersion: tekton.dev/v1beta1
kind: Task
metadata:
name: detect-changed-services
namespace: tekton-pipelines
spec:
workspaces:
- name: source
results:
- name: changed-services
description: Comma-separated list of changed services
steps:
- name: detect
image: alpine/git
script: |
#!/bin/sh
set -e
cd $(workspaces.source.path)
echo "Detecting changed files..."
# Get list of changed files compared to previous commit
CHANGED_FILES=$(git diff --name-only HEAD~1 HEAD 2>/dev/null || git diff --name-only HEAD)
echo "Changed files: $CHANGED_FILES"
# Map files to services
CHANGED_SERVICES=()
for file in $CHANGED_FILES; do
if [[ $file == services/* ]]; then
SERVICE=$(echo $file | cut -d'/' -f2)
# Only add unique service names
if [[ ! " ${CHANGED_SERVICES[@]} " =~ " ${SERVICE} " ]]; then
CHANGED_SERVICES+=("$SERVICE")
fi
elif [[ $file == frontend/* ]]; then
CHANGED_SERVICES+=("frontend")
break
elif [[ $file == gateway/* ]]; then
CHANGED_SERVICES+=("gateway")
break
fi
done
# If no specific services changed, check for infrastructure changes
if [ ${#CHANGED_SERVICES[@]} -eq 0 ]; then
for file in $CHANGED_FILES; do
if [[ $file == infrastructure/* ]]; then
CHANGED_SERVICES+=("infrastructure")
break
fi
done
fi
# Output result
if [ ${#CHANGED_SERVICES[@]} -eq 0 ]; then
echo "No service changes detected"
echo "none" | tee $(results.changed-services.path)
else
echo "Detected changes in services: ${CHANGED_SERVICES[@]}"
echo $(printf "%s," "${CHANGED_SERVICES[@]}" | sed 's/,$//') | tee $(results.changed-services.path)
fi

View File

@@ -0,0 +1,31 @@
# Tekton Git Clone Task for Bakery-IA CI/CD
# This task clones the source code repository
apiVersion: tekton.dev/v1beta1
kind: Task
metadata:
name: git-clone
namespace: tekton-pipelines
spec:
workspaces:
- name: output
params:
- name: url
type: string
description: Repository URL to clone
- name: revision
type: string
description: Git revision to checkout
default: "main"
steps:
- name: clone
image: alpine/git
script: |
#!/bin/sh
set -e
echo "Cloning repository: $(params.url)"
git clone $(params.url) $(workspaces.output.path)
cd $(workspaces.output.path)
echo "Checking out revision: $(params.revision)"
git checkout $(params.revision)
echo "Repository cloned successfully"

View File

@@ -0,0 +1,40 @@
# Tekton Kaniko Build Task for Bakery-IA CI/CD
# This task builds and pushes container images using Kaniko
apiVersion: tekton.dev/v1beta1
kind: Task
metadata:
name: kaniko-build
namespace: tekton-pipelines
spec:
workspaces:
- name: source
- name: docker-credentials
params:
- name: services
type: string
description: Comma-separated list of services to build
- name: registry
type: string
description: Container registry URL
default: "gitea.bakery-ia.local:5000"
- name: git-revision
type: string
description: Git revision for image tag
default: "latest"
steps:
- name: build-and-push
image: gcr.io/kaniko-project/executor:v1.9.0
args:
- --dockerfile=$(workspaces.source.path)/services/$(params.services)/Dockerfile
- --context=$(workspaces.source.path)
- --destination=$(params.registry)/bakery/$(params.services):$(params.git-revision)
- --verbosity=info
volumeMounts:
- name: docker-config
mountPath: /kaniko/.docker
securityContext:
runAsUser: 0
volumes:
- name: docker-config
emptyDir: {}

View File

@@ -0,0 +1,66 @@
# Tekton Update GitOps Manifests Task for Bakery-IA CI/CD
# This task updates Kubernetes manifests with new image tags
apiVersion: tekton.dev/v1beta1
kind: Task
metadata:
name: update-gitops
namespace: tekton-pipelines
spec:
workspaces:
- name: source
params:
- name: services
type: string
description: Comma-separated list of services to update
- name: registry
type: string
description: Container registry URL
- name: git-revision
type: string
description: Git revision for image tag
steps:
- name: update-manifests
image: bitnami/kubectl
script: |
#!/bin/sh
set -e
cd $(workspaces.source.path)
echo "Updating GitOps manifests for services: $(params.services)"
# Split services by comma
IFS=',' read -ra SERVICES <<< "$(params.services)"
for service in "${SERVICES[@]}"; do
echo "Processing service: $service"
# Find and update Kubernetes manifests
if [ "$service" = "frontend" ]; then
# Update frontend deployment
if [ -f "infrastructure/kubernetes/overlays/prod/frontend-deployment.yaml" ]; then
sed -i "s|image:.*|image: $(params.registry)/bakery/frontend:$(params.git-revision)|g" \
"infrastructure/kubernetes/overlays/prod/frontend-deployment.yaml"
fi
elif [ "$service" = "gateway" ]; then
# Update gateway deployment
if [ -f "infrastructure/kubernetes/overlays/prod/gateway-deployment.yaml" ]; then
sed -i "s|image:.*|image: $(params.registry)/bakery/gateway:$(params.git-revision)|g" \
"infrastructure/kubernetes/overlays/prod/gateway-deployment.yaml"
fi
else
# Update service deployment
DEPLOYMENT_FILE="infrastructure/kubernetes/overlays/prod/${service}-deployment.yaml"
if [ -f "$DEPLOYMENT_FILE" ]; then
sed -i "s|image:.*|image: $(params.registry)/bakery/${service}:$(params.git-revision)|g" \
"$DEPLOYMENT_FILE"
fi
fi
done
# Commit changes
git config --global user.name "bakery-ia-ci"
git config --global user.email "ci@bakery-ia.local"
git add .
git commit -m "CI: Update image tags for $(params.services) to $(params.git-revision)"
git push origin HEAD

View File

@@ -0,0 +1,26 @@
# Tekton EventListener for Bakery-IA CI/CD
# This listener receives webhook events and triggers pipelines
apiVersion: triggers.tekton.dev/v1alpha1
kind: EventListener
metadata:
name: bakery-ia-listener
namespace: tekton-pipelines
spec:
serviceAccountName: tekton-triggers-sa
triggers:
- name: bakery-ia-gitea-trigger
bindings:
- ref: bakery-ia-trigger-binding
template:
ref: bakery-ia-trigger-template
interceptors:
- ref:
name: "gitlab"
params:
- name: "secretRef"
value:
secretName: gitea-webhook-secret
secretKey: secretToken
- name: "eventTypes"
value: ["push"]

View File

@@ -0,0 +1,14 @@
# GitLab/Gitea Webhook Interceptor for Tekton Triggers
# This interceptor validates and processes Gitea webhook events
apiVersion: triggers.tekton.dev/v1alpha1
kind: ClusterInterceptor
metadata:
name: gitlab
spec:
clientConfig:
service:
name: tekton-triggers-core-interceptors
namespace: tekton-pipelines
path: "/v1/webhook/gitlab"
port: 8443

View File

@@ -0,0 +1,16 @@
# Tekton TriggerBinding for Bakery-IA CI/CD
# This binding extracts parameters from Gitea webhook events
apiVersion: triggers.tekton.dev/v1alpha1
kind: TriggerBinding
metadata:
name: bakery-ia-trigger-binding
namespace: tekton-pipelines
spec:
params:
- name: git-repo-url
value: $(body.repository.clone_url)
- name: git-revision
value: $(body.head_commit.id)
- name: git-repo-name
value: $(body.repository.name)

View File

@@ -0,0 +1,43 @@
# Tekton TriggerTemplate for Bakery-IA CI/CD
# This template defines how PipelineRuns are created when triggers fire
apiVersion: triggers.tekton.dev/v1alpha1
kind: TriggerTemplate
metadata:
name: bakery-ia-trigger-template
namespace: tekton-pipelines
spec:
params:
- name: git-repo-url
description: The git repository URL
- name: git-revision
description: The git revision/commit hash
- name: git-repo-name
description: The git repository name
default: "bakery-ia"
resourcetemplates:
- apiVersion: tekton.dev/v1beta1
kind: PipelineRun
metadata:
generateName: bakery-ia-ci-run-$(params.git-repo-name)-
spec:
pipelineRef:
name: bakery-ia-ci
workspaces:
- name: shared-workspace
volumeClaimTemplate:
spec:
accessModes: ["ReadWriteOnce"]
resources:
requests:
storage: 5Gi
- name: docker-credentials
secret:
secretName: gitea-registry-credentials
params:
- name: git-url
value: $(params.git-repo-url)
- name: git-revision
value: $(params.git-revision)
- name: registry
value: "gitea.bakery-ia.local:5000"