bakery-ia/infrastructure/cicd

Bakery-IA CI/CD Implementation

This directory contains the configuration for the production-grade CI/CD system for Bakery-IA using Gitea, Tekton, and Flux CD.

Architecture Overview

graph TD
    A[Developer] -->|Push Code| B[Gitea]
    B -->|Webhook| C[Tekton Pipelines]
    C -->|Build/Test| D[Gitea Registry]
    D -->|New Image| E[Flux CD]
    E -->|kubectl apply| F[MicroK8s Cluster]
    F -->|Metrics| G[SigNoz]

Directory Structure

infrastructure/ci-cd/
├── gitea/                  # Gitea configuration (Git server + registry)
│   └── values.yaml         # Helm values for Gitea (ingress now in main config)
├── tekton/                # Tekton CI/CD pipeline configuration
│   ├── tasks/              # Individual pipeline tasks
│   │   ├── git-clone.yaml
│   │   ├── detect-changes.yaml
│   │   ├── kaniko-build.yaml
│   │   └── update-gitops.yaml
│   ├── pipelines/          # Pipeline definitions
│   │   └── ci-pipeline.yaml
│   └── triggers/           # Webhook trigger configuration
│       ├── trigger-template.yaml
│       ├── trigger-binding.yaml
│       ├── event-listener.yaml
│       └── gitlab-interceptor.yaml
├── flux/                   # Flux CD GitOps Helm chart configuration
│   ├── Chart.yaml          # Helm chart definition
│   ├── values.yaml         # Default configuration values
│   ├── templates/          # Kubernetes manifest templates
│   │   ├── gitrepository.yaml
│   │   ├── kustomization.yaml
│   │   └── namespace.yaml
│   └── values/             # Additional value files
├── monitoring/             # Monitoring configuration
│   └── otel-collector.yaml # OpenTelemetry collector
└── README.md               # This file

Deployment Instructions

Phase 1: Infrastructure Setup

1. Deploy Gitea:

   # Add Helm repo
   microk8s helm repo add gitea https://dl.gitea.io/charts
   
   # Create namespace
   microk8s kubectl create namespace gitea
   
   # Install Gitea
   microk8s helm install gitea gitea/gitea \
     -n gitea \
     -f infrastructure/ci-cd/gitea/values.yaml
   
   # Note: Gitea ingress is now included in the main ingress configuration
   # No separate ingress needs to be applied
   

2. Deploy Tekton:

   # Create namespace
   microk8s kubectl create namespace tekton-pipelines
   
   # Install Tekton Pipelines
   microk8s kubectl apply -f https://storage.googleapis.com/tekton-releases/pipeline/latest/release.yaml
   
   # Install Tekton Triggers
   microk8s kubectl apply -f https://storage.googleapis.com/tekton-releases/triggers/latest/release.yaml
   
   # Apply Tekton configurations
   microk8s kubectl apply -f infrastructure/ci-cd/tekton/tasks/
   microk8s kubectl apply -f infrastructure/ci-cd/tekton/pipelines/
   microk8s kubectl apply -f infrastructure/ci-cd/tekton/triggers/
   

3. Deploy Flux CD (already enabled in MicroK8s):

   # Verify Flux installation
   microk8s kubectl get pods -n flux-system
   
   # Apply Flux configurations using kustomize
   microk8s kubectl apply -k infrastructure/ci-cd/flux/
   

Phase 2: Configuration

1. Set up Gitea webhook:

   • Go to your Gitea repository settings
   • Add webhook with URL: http://tekton-triggers.tekton-pipelines.svc.cluster.local:8080
   • Use the secret from gitea-webhook-secret

2. Configure registry credentials:

   # Create registry credentials secret
   microk8s kubectl create secret docker-registry gitea-registry-credentials \
     -n tekton-pipelines \
     --docker-server=gitea.bakery-ia.local:5000 \
     --docker-username=your-username \
     --docker-password=your-password
   

3. Configure Git credentials for Flux:

   # Create Git credentials secret
   microk8s kubectl create secret generic gitea-credentials \
     -n flux-system \
     --from-literal=username=your-username \
     --from-literal=password=your-password
   

Phase 3: Monitoring

# Apply OpenTelemetry configuration
microk8s kubectl apply -f infrastructure/ci-cd/monitoring/otel-collector.yaml

Usage

Triggering a Pipeline

1. Manual trigger:

   # Create a PipelineRun manually
   microk8s kubectl create -f - <<EOF
   apiVersion: tekton.dev/v1beta1
   kind: PipelineRun
   metadata:
     name: manual-ci-run
     namespace: tekton-pipelines
   spec:
     pipelineRef:
       name: bakery-ia-ci
     workspaces:
       - name: shared-workspace
         volumeClaimTemplate:
           spec:
             accessModes: ["ReadWriteOnce"]
             resources:
               requests:
                 storage: 5Gi
       - name: docker-credentials
         secret:
           secretName: gitea-registry-credentials
     params:
       - name: git-url
         value: "http://gitea.bakery-ia.local/bakery/bakery-ia.git"
       - name: git-revision
         value: "main"
   EOF
   

2. Automatic trigger: Push code to the repository and the webhook will trigger the pipeline automatically.

Monitoring Pipeline Runs

# List all PipelineRuns
microk8s kubectl get pipelineruns -n tekton-pipelines

# View logs for a specific PipelineRun
microk8s kubectl logs -n tekton-pipelines <pipelinerun-pod> -c <step-name>

# View Tekton dashboard
microk8s kubectl port-forward -n tekton-pipelines svc/tekton-dashboard 9097:9097

Troubleshooting

Common Issues

1. Pipeline not triggering:

   • Check Gitea webhook logs
   • Verify EventListener pods are running
   • Check TriggerBinding configuration

2. Build failures:

   • Check Kaniko logs for build errors
   • Verify Dockerfile paths are correct
   • Ensure registry credentials are valid

3. Flux not applying changes:

   • Check GitRepository status
   • Verify Kustomization reconciliation
   • Check Flux logs for errors

Debugging Commands

# Check Tekton controller logs
microk8s kubectl logs -n tekton-pipelines -l app=tekton-pipelines-controller

# Check Flux reconciliation
microk8s kubectl get kustomizations -n flux-system -o yaml

# Check Gitea webhook delivery
microk8s kubectl logs -n tekton-pipelines -l app=tekton-triggers-controller

Security Considerations

1. Secrets Management:

   • Use Kubernetes secrets for sensitive data
   • Rotate credentials regularly
   • Use RBAC for namespace isolation

2. Network Security:

   • Configure network policies
   • Use internal DNS names
   • Restrict ingress access

3. Registry Security:

   • Enable image scanning
   • Use image signing
   • Implement cleanup policies

Maintenance

Upgrading Components

# Upgrade Tekton
microk8s kubectl apply -f https://storage.googleapis.com/tekton-releases/pipeline/latest/release.yaml

# Upgrade Flux
microk8s helm upgrade fluxcd fluxcd/flux2 -n flux-system

# Upgrade Gitea
microk8s helm upgrade gitea gitea/gitea -n gitea -f infrastructure/ci-cd/gitea/values.yaml

Backup Procedures

# Backup Gitea
microk8s kubectl exec -n gitea gitea-0 -- gitea dump -c /data/gitea/conf/app.ini

# Backup Flux configurations
microk8s kubectl get all -n flux-system -o yaml > flux-backup.yaml

# Backup Tekton configurations
microk8s kubectl get all -n tekton-pipelines -o yaml > tekton-backup.yaml

Performance Optimization

1. Resource Management:

   • Set appropriate resource limits
   • Limit concurrent builds
   • Use node selectors for build pods

2. Caching:

   • Configure Kaniko cache
   • Use persistent volumes for dependencies
   • Cache Docker layers

3. Parallelization:

   • Build independent services in parallel
   • Use matrix builds for different architectures
   • Optimize task dependencies

Integration with Existing System

The CI/CD system integrates with:

• SigNoz: For monitoring and observability
• MicroK8s: For cluster management
• Existing Kubernetes manifests: In infrastructure/kubernetes/
• Current services: All 19 microservices in services/

Migration Plan

1. Phase 1: Set up infrastructure (Gitea, Tekton, Flux)
2. Phase 2: Configure pipelines and triggers
3. Phase 3: Test with non-critical services
4. Phase 4: Gradual rollout to all services
5. Phase 5: Decommission old deployment methods

Support

For issues with the CI/CD system:

• Check logs and monitoring first
• Review the troubleshooting section
• Consult the original implementation plan
• Refer to component documentation:
  • Tekton Documentation
  • Flux CD Documentation
  • Gitea Documentation