Fix resources isues 30

PRODUCTION_DEPLOYMENT_GUIDE.md

@@ -20,6 +20,8 @@
 5. [Phase 2: Domain & DNS Configuration](#phase-2-domain--dns-configuration)
 6. [Phase 3: Deploy Foundation Layer](#phase-3-deploy-foundation-layer)
 7. [Phase 4: Deploy CI/CD Infrastructure](#phase-4-deploy-cicd-infrastructure)
+   - [Step 4.6: Configure Gitea Webhook](#step-46-configure-gitea-webhook-for-cicd-pipeline)
+   - [Step 4.7: Configure PipelineRun Cleanup](#step-47-configure-pipelinerun-cleanup-optional-but-recommended)
 8. [Phase 5: Pre-Pull and Push Base Images to Gitea Registry](#phase-5-pre-pull-and-push-base-images-to-gitea-registry)
    - [Step 5.1: Pre-Pull Base Images](#step-51-pre-pull-base-images-and-push-to-registry)
    - [Step 5.2: Verify Images in Registry](#step-52-verify-images-in-gitea-registry)
@@ -654,6 +656,138 @@ flux get sources git -n flux-system
 flux get kustomizations -n flux-system
 ```
 
+### Step 4.6: Configure Gitea Webhook for CI/CD Pipeline
+
+> **Important:** The Tekton EventListener is exposed via an internal Kubernetes service. For Gitea (running in the same cluster) to trigger pipelines, you need to configure a webhook pointing to the EventListener service.
+
+```bash
+# Get the EventListener service details
+kubectl get svc -n tekton-pipelines | grep el-bakery-ia
+
+# Expected output:
+# el-bakery-ia-event-listener   ClusterIP   10.x.x.x   <none>   8080/TCP,9000/TCP
+```
+
+**Configure Webhook in Gitea UI:**
+
+1. Navigate to: `https://gitea.bakewise.ai/bakery-admin/bakery-ia/settings/hooks`
+2. Click **"Add Webhook"** → **"Gitea"**
+3. Configure the webhook:
+
+| Setting | Value |
+|---------|-------|
+| **Target URL** | `http://el-bakery-ia-event-listener.tekton-pipelines.svc.cluster.local:8080` |
+| **HTTP Method** | POST |
+| **Content Type** | application/json |
+| **Secret** | (leave empty or use `$TEKTON_WEBHOOK_TOKEN` from Step 4.4) |
+| **Trigger On** | Push Events |
+| **Branch Filter** | `main` |
+| **Active** | ✅ Checked |
+
+4. Click **"Add Webhook"**
+5. Click **"Test Delivery"** to verify connectivity
+
+**Verify Webhook Works:**
+
+```bash
+# Watch for new PipelineRuns after pushing a commit
+kubectl get pipelineruns -n tekton-pipelines --watch
+
+# Or make a test push
+cd /root/bakery-ia
+git commit --allow-empty -m "Test CI/CD trigger"
+git push origin main
+
+# Check if pipeline was triggered
+kubectl get pipelineruns -n tekton-pipelines
+```
+
+**Alternative: External Webhook URL (if DNS issues)**
+
+If Gitea cannot resolve the internal service DNS, an ingress was created for external access:
+
+```bash
+# Verify EventListener ingress exists
+kubectl get ingress -n tekton-pipelines
+
+# Use external URL instead:
+# Target URL: https://tekton-webhook.bakewise.ai
+```
+
+> **DNS Note:** The internal URL (`http://el-bakery-ia-event-listener.tekton-pipelines.svc.cluster.local:8080`) should work for Gitea pods in the same cluster. If you encounter DNS resolution errors, use the external ingress URL.
+
+### Step 4.7: Configure PipelineRun Cleanup (Optional but Recommended)
+
+> **Purpose:** Completed PipelineRuns and TaskRuns accumulate over time and consume etcd storage. Configure automatic pruning to keep only recent runs.
+
+**Option A: Configure Tekton Pruner (Recommended)**
+
+```bash
+# Create a CronJob to prune old PipelineRuns and TaskRuns
+cat <<EOF | kubectl apply -f -
+apiVersion: batch/v1
+kind: CronJob
+metadata:
+  name: tekton-resource-pruner
+  namespace: tekton-pipelines
+spec:
+  schedule: "0 2 * * *"  # Run daily at 2 AM
+  jobTemplate:
+    spec:
+      template:
+        spec:
+          serviceAccountName: tekton-pipeline-sa
+          containers:
+          - name: pruner
+            image: bitnami/kubectl:latest
+            command:
+            - /bin/sh
+            - -c
+            - |
+              echo "Pruning PipelineRuns older than 7 days..."
+              kubectl delete pipelineruns -n tekton-pipelines \
+                --field-selector=status.completionTime!="" \
+                --selector='tekton.dev/pipeline=bakery-ia-ci' \
+                --sort-by=.metadata.creationTimestamp \
+                | head -n -5 || true
+              echo "Pruning completed TaskRuns..."
+              kubectl get taskruns -n tekton-pipelines \
+                -o jsonpath='{range .items[?(@.status.completionTime)]}{.metadata.name}{"\n"}{end}' \
+                | head -n -10 \
+                | xargs -r kubectl delete taskrun -n tekton-pipelines || true
+              echo "Cleanup complete"
+          restartPolicy: OnFailure
+EOF
+```
+
+**Option B: Manual Cleanup**
+
+```bash
+# Delete all completed PipelineRuns (keep last 5)
+kubectl get pipelineruns -n tekton-pipelines --sort-by=.metadata.creationTimestamp -o name | head -n -5 | xargs -r kubectl delete -n tekton-pipelines
+
+# Delete all completed TaskRuns (keep last 10)
+kubectl get taskruns -n tekton-pipelines --sort-by=.metadata.creationTimestamp -o name | head -n -10 | xargs -r kubectl delete -n tekton-pipelines
+
+# Delete PipelineRuns older than 7 days
+kubectl get pipelineruns -n tekton-pipelines -o json | \
+  jq -r '.items[] | select(.metadata.creationTimestamp | fromdateiso8601 < (now - 604800)) | .metadata.name' | \
+  xargs -r kubectl delete pipelinerun -n tekton-pipelines
+```
+
+**Check Current Resource Usage:**
+
+```bash
+# Count PipelineRuns
+kubectl get pipelineruns -n tekton-pipelines --no-headers | wc -l
+
+# Count TaskRuns
+kubectl get taskruns -n tekton-pipelines --no-headers | wc -l
+
+# Check etcd storage (if metrics-server is enabled)
+kubectl top pods -n kube-system -l component=etcd
+```
+
 
 ## Phase 5: Pre-Pull and Push Base Images to Gitea Registry
 

infrastructure/cicd/gitea/values-prod.yaml

@@ -44,6 +44,33 @@ gitea:
       SSH_DOMAIN: gitea.bakewise.ai
       ROOT_URL: https://gitea.bakewise.ai
 
+    # =============================================================================
+    # PACKAGE/REGISTRY RETENTION POLICY
+    # =============================================================================
+    # Automatic cleanup of old container images and packages
+    # This prevents the registry from growing indefinitely
+    packages:
+      ENABLED: true
+      # Limit container image versions to prevent storage bloat
+      # 0 = unlimited (default), set a reasonable limit for CI/CD
+      LIMIT_TOTAL_OWNER_SIZE: 10737418240  # 10GB per owner/organization
+      LIMIT_SIZE_CONTAINER: 2147483648      # 2GB per container image
+
+    # Cron job for automatic package cleanup
+    cron:
+      ENABLED: true
+    "cron.cleanup_packages":
+      ENABLED: true
+      # Run daily at 3 AM
+      SCHEDULE: "0 3 * * *"
+      # Keep packages newer than this (in hours) - 168h = 7 days
+      OLDER_THAN: 168h
+      # Number of versions to keep per package (0 = disabled)
+      # This keeps the last 5 versions regardless of age
+      NUMBER_TO_KEEP: 5
+      # Also clean up unreferenced blobs
+      REMOVE_UNUSED_ARTIFACTS: true
+
 # Production resources - adjust based on expected load
 resources:
   limits:

infrastructure/cicd/gitea/values.yaml

@@ -93,6 +93,8 @@ gitea:
       DEFAULT_BRANCH: main
     packages:
       ENABLED: true
+      # Retention policy is configured in values-prod.yaml for production
+      # See: LIMIT_TOTAL_OWNER_SIZE, LIMIT_SIZE_CONTAINER, cron.cleanup_packages
     webhook:
       ALLOWED_HOST_LIST: "*"
       # Allow internal cluster URLs for Tekton EventListener

infrastructure/cicd/tekton-helm/templates/task-kaniko-build.yaml

@@ -158,7 +158,8 @@ spec:
               --build-arg="BASE_REGISTRY=$(params.base-registry)" \
               --build-arg="PYTHON_IMAGE=$(params.python-image)" \
               --cache=true \
-              --cache-repo="$(params.registry)/cache"
+              --cache-repo="$(params.registry)/cache" \
+              --cache-ttl=168h
 
             echo "Successfully built: $(params.registry)/$service:$(params.git-revision)"
           fi

infrastructure/cicd/tekton-helm/templates/trigger-template.yaml

@@ -41,6 +41,9 @@ spec:
           # Track the source commit
           bakery-ia.io/git-revision: $(tt.params.git-revision)
           bakery-ia.io/git-branch: $(tt.params.git-branch)
+          # Automatic cleanup: delete PipelineRun after completion
+          # Default: 1 hour (3600 seconds). Configure via values.yaml
+          tekton.dev/pipelinerunDeletionPropagation: "foreground"
       spec:
         pipelineRef:
           name: bakery-ia-ci

infrastructure/cicd/tekton-helm/values.yaml

@@ -93,6 +93,20 @@ serviceAccounts:
   pipeline:
     name: "tekton-pipeline-sa"
 
+# Cleanup and retention configuration
+cleanup:
+  # PipelineRun retention (requires Tekton Pipelines >= v0.52.0)
+  # Set to 0 to disable automatic cleanup
+  pipelineRuns:
+    # Keep last N successful PipelineRuns per Pipeline
+    keepSuccessful: 3
+    # Keep last N failed PipelineRuns per Pipeline
+    keepFailed: 5
+  # TaskRun retention
+  taskRuns:
+    keepSuccessful: 3
+    keepFailed: 5
+
 # Labels to apply to resources
 labels:
   app: