bakery-ia/infrastructure/cicd/README.md

# Bakery-IA CI/CD Implementation

This directory contains the configuration for the production-grade CI/CD system for Bakery-IA using Gitea, Tekton, and Flux CD.

## Architecture Overview

```mermaid
graph TD
    A[Developer] -->|Push Code| B[Gitea]
    B -->|Webhook| C[Tekton Pipelines]
    C -->|Build/Test| D[Gitea Registry]
    D -->|New Image| E[Flux CD]
    E -->|kubectl apply| F[MicroK8s Cluster]
    F -->|Metrics| G[SigNoz]
```

## Directory Structure

```
infrastructure/ci-cd/
├── gitea/                  # Gitea configuration (Git server + registry)
│   └── values.yaml         # Helm values for Gitea (ingress now in main config)
├── tekton/                # Tekton CI/CD pipeline configuration
│   ├── tasks/              # Individual pipeline tasks
│   │   ├── git-clone.yaml
│   │   ├── detect-changes.yaml
│   │   ├── kaniko-build.yaml
│   │   └── update-gitops.yaml
│   ├── pipelines/          # Pipeline definitions
│   │   └── ci-pipeline.yaml
│   └── triggers/           # Webhook trigger configuration
│       ├── trigger-template.yaml
│       ├── trigger-binding.yaml
│       ├── event-listener.yaml
│       └── gitlab-interceptor.yaml
├── flux/                   # Flux CD GitOps configuration
│   ├── git-repository.yaml # Git repository source
│   └── kustomization.yaml  # Deployment kustomization
├── monitoring/             # Monitoring configuration
│   └── otel-collector.yaml # OpenTelemetry collector
└── README.md               # This file
```

## Deployment Instructions

### Phase 1: Infrastructure Setup

1. **Deploy Gitea**:
   ```bash
   # Add Helm repo
   microk8s helm repo add gitea https://dl.gitea.io/charts
   
   # Create namespace
   microk8s kubectl create namespace gitea
   
   # Install Gitea
   microk8s helm install gitea gitea/gitea \
     -n gitea \
     -f infrastructure/ci-cd/gitea/values.yaml
   
   # Note: Gitea ingress is now included in the main ingress configuration
   # No separate ingress needs to be applied
   ```

2. **Deploy Tekton**:
   ```bash
   # Create namespace
   microk8s kubectl create namespace tekton-pipelines
   
   # Install Tekton Pipelines
   microk8s kubectl apply -f https://storage.googleapis.com/tekton-releases/pipeline/latest/release.yaml
   
   # Install Tekton Triggers
   microk8s kubectl apply -f https://storage.googleapis.com/tekton-releases/triggers/latest/release.yaml
   
   # Apply Tekton configurations
   microk8s kubectl apply -f infrastructure/ci-cd/tekton/tasks/
   microk8s kubectl apply -f infrastructure/ci-cd/tekton/pipelines/
   microk8s kubectl apply -f infrastructure/ci-cd/tekton/triggers/
   ```

3. **Deploy Flux CD** (already enabled in MicroK8s):
   ```bash
   # Verify Flux installation
   microk8s kubectl get pods -n flux-system
   
   # Apply Flux configurations using kustomize
   microk8s kubectl apply -k infrastructure/ci-cd/flux/
   ```

### Phase 2: Configuration

1. **Set up Gitea webhook**:
   - Go to your Gitea repository settings
   - Add webhook with URL: `http://tekton-triggers.tekton-pipelines.svc.cluster.local:8080`
   - Use the secret from `gitea-webhook-secret`

2. **Configure registry credentials**:
   ```bash
   # Create registry credentials secret
   microk8s kubectl create secret docker-registry gitea-registry-credentials \
     -n tekton-pipelines \
     --docker-server=gitea.bakery-ia.local:5000 \
     --docker-username=your-username \
     --docker-password=your-password
   ```

3. **Configure Git credentials for Flux**:
   ```bash
   # Create Git credentials secret
   microk8s kubectl create secret generic gitea-credentials \
     -n flux-system \
     --from-literal=username=your-username \
     --from-literal=password=your-password
   ```

### Phase 3: Monitoring

```bash
# Apply OpenTelemetry configuration
microk8s kubectl apply -f infrastructure/ci-cd/monitoring/otel-collector.yaml
```

## Usage

### Triggering a Pipeline

1. **Manual trigger**:
   ```bash
   # Create a PipelineRun manually
   microk8s kubectl create -f - <<EOF
   apiVersion: tekton.dev/v1beta1
   kind: PipelineRun
   metadata:
     name: manual-ci-run
     namespace: tekton-pipelines
   spec:
     pipelineRef:
       name: bakery-ia-ci
     workspaces:
       - name: shared-workspace
         volumeClaimTemplate:
           spec:
             accessModes: ["ReadWriteOnce"]
             resources:
               requests:
                 storage: 5Gi
       - name: docker-credentials
         secret:
           secretName: gitea-registry-credentials
     params:
       - name: git-url
         value: "http://gitea.bakery-ia.local/bakery/bakery-ia.git"
       - name: git-revision
         value: "main"
   EOF
   ```

2. **Automatic trigger**: Push code to the repository and the webhook will trigger the pipeline automatically.

### Monitoring Pipeline Runs

```bash
# List all PipelineRuns
microk8s kubectl get pipelineruns -n tekton-pipelines

# View logs for a specific PipelineRun
microk8s kubectl logs -n tekton-pipelines <pipelinerun-pod> -c <step-name>

# View Tekton dashboard
microk8s kubectl port-forward -n tekton-pipelines svc/tekton-dashboard 9097:9097
```

## Troubleshooting

### Common Issues

1. **Pipeline not triggering**:
   - Check Gitea webhook logs
   - Verify EventListener pods are running
   - Check TriggerBinding configuration

2. **Build failures**:
   - Check Kaniko logs for build errors
   - Verify Dockerfile paths are correct
   - Ensure registry credentials are valid

3. **Flux not applying changes**:
   - Check GitRepository status
   - Verify Kustomization reconciliation
   - Check Flux logs for errors

### Debugging Commands

```bash
# Check Tekton controller logs
microk8s kubectl logs -n tekton-pipelines -l app=tekton-pipelines-controller

# Check Flux reconciliation
microk8s kubectl get kustomizations -n flux-system -o yaml

# Check Gitea webhook delivery
microk8s kubectl logs -n tekton-pipelines -l app=tekton-triggers-controller
```

## Security Considerations

1. **Secrets Management**:
   - Use Kubernetes secrets for sensitive data
   - Rotate credentials regularly
   - Use RBAC for namespace isolation

2. **Network Security**:
   - Configure network policies
   - Use internal DNS names
   - Restrict ingress access

3. **Registry Security**:
   - Enable image scanning
   - Use image signing
   - Implement cleanup policies

## Maintenance

### Upgrading Components

```bash
# Upgrade Tekton
microk8s kubectl apply -f https://storage.googleapis.com/tekton-releases/pipeline/latest/release.yaml

# Upgrade Flux
microk8s helm upgrade fluxcd fluxcd/flux2 -n flux-system

# Upgrade Gitea
microk8s helm upgrade gitea gitea/gitea -n gitea -f infrastructure/ci-cd/gitea/values.yaml
```

### Backup Procedures

```bash
# Backup Gitea
microk8s kubectl exec -n gitea gitea-0 -- gitea dump -c /data/gitea/conf/app.ini

# Backup Flux configurations
microk8s kubectl get all -n flux-system -o yaml > flux-backup.yaml

# Backup Tekton configurations
microk8s kubectl get all -n tekton-pipelines -o yaml > tekton-backup.yaml
```

## Performance Optimization

1. **Resource Management**:
   - Set appropriate resource limits
   - Limit concurrent builds
   - Use node selectors for build pods

2. **Caching**:
   - Configure Kaniko cache
   - Use persistent volumes for dependencies
   - Cache Docker layers

3. **Parallelization**:
   - Build independent services in parallel
   - Use matrix builds for different architectures
   - Optimize task dependencies

## Integration with Existing System

The CI/CD system integrates with:
- **SigNoz**: For monitoring and observability
- **MicroK8s**: For cluster management
- **Existing Kubernetes manifests**: In `infrastructure/kubernetes/`
- **Current services**: All 19 microservices in `services/`

## Migration Plan

1. **Phase 1**: Set up infrastructure (Gitea, Tekton, Flux)
2. **Phase 2**: Configure pipelines and triggers
3. **Phase 3**: Test with non-critical services
4. **Phase 4**: Gradual rollout to all services
5. **Phase 5**: Decommission old deployment methods

## Support

For issues with the CI/CD system:
- Check logs and monitoring first
- Review the troubleshooting section
- Consult the original implementation plan
- Refer to component documentation:
  - [Tekton Documentation](https://tekton.dev/docs/)
  - [Flux CD Documentation](https://fluxcd.io/docs/)
  - [Gitea Documentation](https://docs.gitea.io/)