Imporve monitoring 3

2026-01-09 11:18:20 +01:00
parent 8ca5d9c100
commit 43a3f35bd1
27 changed files with 1279 additions and 32 deletions
--- a/docs/SIGNOZ_ROOT_CAUSE_ANALYSIS.md
+++ b/docs/SIGNOZ_ROOT_CAUSE_ANALYSIS.md
@@ -0,0 +1,289 @@
 # SigNoz OpenAMP Root Cause Analysis & Resolution
 ## Problem Statement
 Services were getting `StatusCode.UNAVAILABLE` errors when trying to send traces to the SigNoz OTel Collector at port 4317. The OTel Collector was continuously restarting due to OpenAMP trying to apply invalid remote configurations.
 ## Root Cause Analysis
 ### Primary Issue: Missing `signozmeter` Connector Pipeline
 **Error Message:**
 ```
 connector "signozmeter" used as receiver in [metrics/meter] pipeline
 but not used in any supported exporter pipeline
 ```
 **Root Cause:**
 The OpenAMP server was pushing a remote configuration that included:
 1. A `metrics/meter` pipeline that uses `signozmeter` as a receiver
 2. However, no pipeline was exporting TO the `signozmeter` connector
 **Technical Explanation:**
 - **Connectors** in OpenTelemetry are special components that act as BOTH exporters AND receivers
 - They bridge between pipelines (e.g., traces → metrics)
 - The `signozmeter` connector generates usage/meter metrics from trace data
 - For a connector to work, it must be:
  1. Used as an **exporter** in one pipeline (the source)
  2. Used as a **receiver** in another pipeline (the destination)
 **What Was Missing:**
 Our configuration had:
 - ✅ `signozmeter` connector defined
 - ✅ `metrics/meter` pipeline receiving from `signozmeter`
 - ❌ **No pipeline exporting TO `signozmeter`**
 The traces pipeline needed to export to `signozmeter`:
 ```yaml
 traces:
  receivers: [otlp]
  processors: [...]
  exporters: [clickhousetraces, metadataexporter, signozmeter]  # <-- signozmeter was missing
 ```
 ### Secondary Issue: gRPC Endpoint Format
 **Problem:** Services had `http://` prefix in gRPC endpoints
 **Solution:** Removed `http://` prefix (gRPC doesn't use HTTP protocol prefix)
 **Before:**
 ```yaml
 OTEL_EXPORTER_OTLP_ENDPOINT: "http://signoz-otel-collector.bakery-ia.svc.cluster.local:4317"
 ```
 **After:**
 ```yaml
 OTEL_EXPORTER_OTLP_ENDPOINT: "signoz-otel-collector.bakery-ia.svc.cluster.local:4317"
 ```
 ### Tertiary Issue: Hardcoded Endpoints
 **Problem:** Each service manifest had hardcoded OTEL endpoints instead of referencing ConfigMap
 **Solution:** Updated all 18 services to use `valueFrom: configMapKeyRef`
 ## Solution Implemented
 ### 1. Added Complete Meter Pipeline Configuration
 **Added Connector:**
 ```yaml
 connectors:
  signozmeter:
    dimensions:
      - name: service.name
      - name: deployment.environment
      - name: host.name
    metrics_flush_interval: 1h
 ```
 **Added Batch Processor:**
 ```yaml
 processors:
  batch/meter:
    timeout: 1s
    send_batch_size: 20000
    send_batch_max_size: 25000
 ```
 **Added Exporters:**
 ```yaml
 exporters:
  # Meter exporter
  signozclickhousemeter:
    dsn: "tcp://admin:PASSWORD@signoz-clickhouse:9000/signoz_meter"
    timeout: 45s
    sending_queue:
      enabled: false
  # Metadata exporter
  metadataexporter:
    dsn: "tcp://admin:PASSWORD@signoz-clickhouse:9000/signoz_metadata"
    timeout: 10s
    cache:
      provider: in_memory
 ```
 **Updated Traces Pipeline:**
 ```yaml
 traces:
  receivers: [otlp]
  processors: [memory_limiter, batch, signozspanmetrics/delta, resourcedetection]
  exporters: [clickhousetraces, metadataexporter, signozmeter]  # Added signozmeter
 ```
 **Added Meter Pipeline:**
 ```yaml
 metrics/meter:
  receivers: [signozmeter]
  processors: [batch/meter]
  exporters: [signozclickhousemeter]
 ```
 ### 2. Fixed gRPC Endpoint Configuration
 Updated ConfigMaps:
 - `infrastructure/kubernetes/base/configmap.yaml`
 - `infrastructure/kubernetes/overlays/prod/prod-configmap.yaml`
 ### 3. Centralized OTEL Configuration
 Created script: `infrastructure/kubernetes/fix-otel-endpoints.sh`
 Updated 18 service manifests to use ConfigMap reference instead of hardcoded values.
 ## Results
 ### Before Fix
 - ❌ OTel Collector continuously restarting
 - ❌ Services unable to export traces (StatusCode.UNAVAILABLE)
 - ❌ Error: `connector "signozmeter" used as receiver but not used in any supported exporter pipeline`
 - ❌ OpenAMP constantly trying to reload bad config
 ### After Fix
 - ✅ OTel Collector stable and running
 - ✅ Message: `"Everything is ready. Begin running and processing data."`
 - ✅ No more signozmeter connector errors
 - ✅ OpenAMP errors are now just warnings (remote server issues, not local config)
 - ⚠️  Service connectivity still showing transient errors (separate investigation needed)
 ## OpenAMP Behavior
 **What is OpenAMP?**
 - OpenTelemetry Agent Management Protocol
 - Allows remote management and configuration of collectors
 - SigNoz uses it for central configuration management
 **Current State:**
 - OpenAMP continues to show errors, but they're now **non-fatal**
 - The errors are from the remote OpAMP server (signoz:4320), not local config
 - Local configuration is valid and working
 - Collector is stable and processing data
 **OpenAMP Error Pattern:**
 ```
 [ERROR] opamp/server_client.go:146
 Server returned an error response
 ```
 This is a **warning** that the remote OpAMP server has configuration issues, but it doesn't affect the locally-configured collector.
 ## Files Modified
 ### Helm Values
 1. `infrastructure/helm/signoz-values-dev.yaml`
   - Added connectors section
   - Added batch/meter processor
   - Added signozclickhousemeter exporter
   - Added metadataexporter
   - Updated traces pipeline to export to signozmeter
   - Added metrics/meter pipeline
 2. `infrastructure/helm/signoz-values-prod.yaml`
   - Same changes as dev
 ### ConfigMaps
 3. `infrastructure/kubernetes/base/configmap.yaml`
   - Fixed OTEL_EXPORTER_OTLP_ENDPOINT (removed http://)
 4. `infrastructure/kubernetes/overlays/prod/prod-configmap.yaml`
   - Fixed OTEL_EXPORTER_OTLP_ENDPOINT (removed http://)
 ### Service Manifests (18 files)
 All services in `infrastructure/kubernetes/base/components/*/` changed from:
 ```yaml
 - name: OTEL_EXPORTER_OTLP_ENDPOINT
  value: "http://..."
 ```
 To:
 ```yaml
 - name: OTEL_EXPORTER_OTLP_ENDPOINT
  valueFrom:
    configMapKeyRef:
      name: bakery-config
      key: OTEL_EXPORTER_OTLP_ENDPOINT
 ```
 ## Verification Commands
 ```bash
 # 1. Check OTel Collector is stable
 kubectl get pods -n bakery-ia | grep otel-collector
 # Should show: 1/1 Running
 # 2. Check for configuration errors
 kubectl logs -n bakery-ia deployment/signoz-otel-collector --tail=50 | grep -E "failed to apply config|signozmeter"
 # Should show: NO errors about signozmeter
 # 3. Verify collector is ready
 kubectl logs -n bakery-ia deployment/signoz-otel-collector | grep "Everything is ready"
 # Should show: "Everything is ready. Begin running and processing data."
 # 4. Check service configuration
 kubectl get configmap bakery-config -n bakery-ia -o jsonpath='{.data.OTEL_EXPORTER_OTLP_ENDPOINT}'
 # Should show: signoz-otel-collector.bakery-ia.svc.cluster.local:4317 (no http://)
 # 5. Verify service is using ConfigMap
 kubectl get deployment gateway -n bakery-ia -o yaml | grep -A 5 "OTEL_EXPORTER"
 # Should show: valueFrom / configMapKeyRef
 # 6. Run verification script
 ./infrastructure/helm/verify-signoz-telemetry.sh
 ```
 ## Next Steps
 ### Immediate
 1. ✅ OTel Collector is stable with OpenAMP enabled
 2. ⏭️ Investigate remaining service connectivity issues
 3. ⏭️ Generate test traffic and verify data collection
 4. ⏭️ Check ClickHouse for traces/metrics/logs
 ### Short-term
 1. Monitor OpenAMP errors - they're warnings, not blocking
 2. Consider contacting SigNoz about OpAMP server configuration
 3. Set up SigNoz dashboards and alerts
 4. Document common queries
 ### Long-term
 1. Evaluate if OpAMP remote management is needed
 2. Consider HTTP exporter as alternative to gRPC
 3. Implement service mesh if connectivity issues persist
 4. Set up proper TLS for production
 ## Key Learnings
 ### About OpenTelemetry Connectors
 - Connectors must be used in BOTH directions
 - Source pipeline must export TO the connector
 - Destination pipeline must receive FROM the connector
 - Missing either direction causes pipeline build failures
 ### About OpenAMP
 - OpenAMP can push remote configurations
 - Local config takes precedence
 - Remote server errors don't prevent local operation
 - Collector continues with last known good config
 ### About gRPC Configuration
 - gRPC endpoints don't use `http://` or `https://` prefixes
 - Only use `hostname:port` format
 - HTTP/REST endpoints DO need the protocol prefix
 ### About Configuration Management
 - Centralize configuration in ConfigMaps
 - Use `valueFrom: configMapKeyRef` pattern
 - Single source of truth prevents drift
 - Makes updates easier across all services
 ## References
 - [SigNoz Helm Charts](https://github.com/SigNoz/charts)
 - [OpenTelemetry Connectors](https://opentelemetry.io/docs/collector/configuration/#connectors)
 - [OpAMP Specification](https://github.com/open-telemetry/opamp-spec)
 - [SigNoz OTel Collector](https://github.com/SigNoz/signoz-otel-collector)
 ---
 **Resolution Date:** 2026-01-09
 **Status:** ✅ Resolved - OTel Collector stable, OpenAMP functional
 **Remaining:** Service connectivity investigation ongoing
--- a/docs/SIGNOZ_VERIFICATION_GUIDE.md
+++ b/docs/SIGNOZ_VERIFICATION_GUIDE.md
@@ -0,0 +1,435 @@
 # SigNoz Telemetry Verification Guide
 ## Overview
 This guide explains how to verify that your services are correctly sending metrics, logs, and traces to SigNoz, and that SigNoz is collecting them properly.
 ## Current Configuration
 ### SigNoz Components
 - **Version**: v0.106.0
 - **OTel Collector**: v0.129.12
 - **Namespace**: `bakery-ia`
 - **Ingress URL**: https://monitoring.bakery-ia.local
 ### Telemetry Endpoints
 The OTel Collector exposes the following endpoints:
 | Protocol | Port | Purpose |
 |----------|------|---------|
 | OTLP gRPC | 4317 | Traces, Metrics, Logs (gRPC) |
 | OTLP HTTP | 4318 | Traces, Metrics, Logs (HTTP) |
 | Jaeger gRPC | 14250 | Jaeger traces (gRPC) |
 | Jaeger HTTP | 14268 | Jaeger traces (HTTP) |
 | Metrics | 8888 | Prometheus metrics from collector |
 | Health Check | 13133 | Collector health status |
 ### Service Configuration
 Services are configured via the `bakery-config` ConfigMap:
 ```yaml
 # Observability enabled
 ENABLE_TRACING: "true"
 ENABLE_METRICS: "true"
 ENABLE_LOGS: "true"
 # OTel Collector endpoint
 OTEL_EXPORTER_OTLP_ENDPOINT: "http://signoz-otel-collector.bakery-ia.svc.cluster.local:4317"
 OTEL_EXPORTER_OTLP_PROTOCOL: "grpc"
 ```
 ### Shared Tracing Library
 Services use `shared/monitoring/tracing.py` which:
 - Auto-instruments FastAPI endpoints
 - Auto-instruments HTTPX (inter-service calls)
 - Auto-instruments Redis operations
 - Auto-instruments SQLAlchemy (PostgreSQL)
 - Uses OTLP exporter to send traces to SigNoz
 **Default endpoint**: `http://signoz-otel-collector.bakery-ia:4318` (HTTP)
 ## Verification Steps
 ### 1. Quick Verification Script
 Run the automated verification script:
 ```bash
 ./infrastructure/helm/verify-signoz-telemetry.sh
 ```
 This script checks:
 - ✅ SigNoz components are running
 - ✅ OTel Collector endpoints are exposed
 - ✅ Configuration is correct
 - ✅ Health checks pass
 - ✅ Data is being collected in ClickHouse
 ### 2. Manual Verification
 #### Check SigNoz Components Status
 ```bash
 kubectl get pods -n bakery-ia | grep signoz
 ```
 Expected output:
 ```
 signoz-0                                      1/1     Running
 signoz-otel-collector-xxxxx                   1/1     Running
 chi-signoz-clickhouse-cluster-0-0-0           1/1     Running
 signoz-zookeeper-0                            1/1     Running
 signoz-clickhouse-operator-xxxxx              2/2     Running
 ```
 #### Check OTel Collector Logs
 ```bash
 kubectl logs -n bakery-ia -l app.kubernetes.io/component=otel-collector --tail=50
 ```
 Look for:
 - `"msg":"Everything is ready. Begin running and processing data."`
 - No error messages about invalid processors
 - Evidence of data reception (traces/metrics/logs)
 #### Check Service Logs for Tracing
 ```bash
 # Check a specific service (e.g., gateway)
 kubectl logs -n bakery-ia -l app=gateway --tail=100 | grep -i "tracing\|otel"
 ```
 Expected output:
 ```
 Distributed tracing configured
 service=gateway-service
 otel_endpoint=http://signoz-otel-collector.bakery-ia:4318
 ```
 ### 3. Generate Test Traffic
 Run the traffic generation script:
 ```bash
 ./infrastructure/helm/generate-test-traffic.sh
 ```
 This script:
 1. Makes API calls to various service endpoints
 2. Checks service logs for telemetry
 3. Waits for data processing (30 seconds)
 ### 4. Verify Data in ClickHouse
 ```bash
 # Get ClickHouse password
 CH_PASSWORD=$(kubectl get secret -n bakery-ia signoz-clickhouse -o jsonpath='{.data.admin-password}' 2>/dev/null | base64 -d)
 # Get ClickHouse pod
 CH_POD=$(kubectl get pods -n bakery-ia -l clickhouse.altinity.com/chi=signoz-clickhouse -o jsonpath='{.items[0].metadata.name}')
 # Check traces
 kubectl exec -n bakery-ia $CH_POD -- clickhouse-client --user=admin --password=$CH_PASSWORD --query="
 SELECT
    serviceName,
    COUNT() as trace_count,
    min(timestamp) as first_trace,
    max(timestamp) as last_trace
 FROM signoz_traces.signoz_index_v2
 WHERE timestamp >= now() - INTERVAL 1 HOUR
 GROUP BY serviceName
 ORDER BY trace_count DESC
 "
 # Check metrics
 kubectl exec -n bakery-ia $CH_POD -- clickhouse-client --user=admin --password=$CH_PASSWORD --query="
 SELECT
    metric_name,
    COUNT() as sample_count
 FROM signoz_metrics.samples_v4
 WHERE unix_milli >= toUnixTimestamp(now() - INTERVAL 1 HOUR) * 1000
 GROUP BY metric_name
 ORDER BY sample_count DESC
 LIMIT 10
 "
 # Check logs
 kubectl exec -n bakery-ia $CH_POD -- clickhouse-client --user=admin --password=$CH_PASSWORD --query="
 SELECT
    COUNT() as log_count,
    min(timestamp) as first_log,
    max(timestamp) as last_log
 FROM signoz_logs.logs
 WHERE timestamp >= now() - INTERVAL 1 HOUR
 "
 ```
 ### 5. Access SigNoz UI
 #### Via Ingress (Recommended)
 1. Add to `/etc/hosts`:
   ```
   127.0.0.1 monitoring.bakery-ia.local
   ```
 2. Access: https://monitoring.bakery-ia.local
 #### Via Port-Forward
 ```bash
 kubectl port-forward -n bakery-ia svc/signoz 3301:8080
 ```
 Then access: http://localhost:3301
 ### 6. Explore Telemetry Data in SigNoz UI
 1. **Traces**:
   - Go to "Services" tab
   - You should see your services listed (gateway, auth-service, inventory-service, etc.)
   - Click on a service to see its traces
   - Click on individual traces to see span details
 2. **Metrics**:
   - Go to "Dashboards" or "Metrics" tab
   - Should see infrastructure metrics (PostgreSQL, Redis, RabbitMQ)
   - Should see service metrics (request rate, latency, errors)
 3. **Logs**:
   - Go to "Logs" tab
   - Should see logs from your services
   - Can filter by service name, log level, etc.
 ## Troubleshooting
 ### Services Can't Connect to OTel Collector
 **Symptoms**:
 ```
 [ERROR] opentelemetry.exporter.otlp.proto.grpc.exporter: Failed to export traces
 error code: StatusCode.UNAVAILABLE
 ```
 **Solutions**:
 1. **Check OTel Collector is running**:
   ```bash
   kubectl get pods -n bakery-ia -l app.kubernetes.io/component=otel-collector
   ```
 2. **Verify service can reach collector**:
   ```bash
   # From a service pod
   kubectl exec -it -n bakery-ia <service-pod> -- curl -v http://signoz-otel-collector.bakery-ia.svc.cluster.local:4318
   ```
 3. **Check endpoint configuration**:
   - gRPC endpoint should NOT have `http://` prefix
   - HTTP endpoint should have `http://` prefix
   Update your service's tracing setup:
   ```python
   # For gRPC (recommended)
   setup_tracing(app, "my-service", otel_endpoint="signoz-otel-collector.bakery-ia.svc.cluster.local:4317")
   # For HTTP
   setup_tracing(app, "my-service", otel_endpoint="http://signoz-otel-collector.bakery-ia.svc.cluster.local:4318")
   ```
 4. **Restart services after config changes**:
   ```bash
   kubectl rollout restart deployment/<service-name> -n bakery-ia
   ```
 ### No Data in SigNoz
 **Possible causes**:
 1. **Services haven't been called yet**
   - Solution: Generate traffic using the test script
 2. **Tracing not initialized**
   - Check service logs for tracing initialization messages
   - Verify `ENABLE_TRACING=true` in ConfigMap
 3. **Wrong OTel endpoint**
   - Verify `OTEL_EXPORTER_OTLP_ENDPOINT` in ConfigMap
   - Should be: `http://signoz-otel-collector.bakery-ia.svc.cluster.local:4317`
 4. **Service not using tracing library**
   - Check if service imports and calls `setup_tracing()` in main.py
   ```python
   from shared.monitoring.tracing import setup_tracing
   app = FastAPI(title="My Service")
   setup_tracing(app, "my-service")
   ```
 ### OTel Collector Errors
 **Check collector logs**:
 ```bash
 kubectl logs -n bakery-ia -l app.kubernetes.io/component=otel-collector --tail=100
 ```
 **Common errors**:
 1. **Invalid processor error**:
   - Check `signoz-values-dev.yaml` has `signozspanmetrics/delta` (not `spanmetrics`)
   - Already fixed in your configuration
 2. **ClickHouse connection error**:
   - Verify ClickHouse is running
   - Check ClickHouse service is accessible
 3. **Configuration validation error**:
   - Validate YAML syntax in `signoz-values-dev.yaml`
   - Check all processors used in pipelines are defined
 ## Infrastructure Metrics
 SigNoz automatically collects metrics from your infrastructure:
 ### PostgreSQL Databases
 - **Receivers configured for**:
  - auth_db (auth-db-service:5432)
  - inventory_db (inventory-db-service:5432)
  - orders_db (orders-db-service:5432)
 - **Metrics collected**:
  - Connection counts
  - Query performance
  - Database size
  - Table statistics
 ### Redis
 - **Endpoint**: redis-service:6379
 - **Metrics collected**:
  - Memory usage
  - Keys count
  - Hit/miss ratio
  - Command stats
 ### RabbitMQ
 - **Endpoint**: rabbitmq-service:15672 (management API)
 - **Metrics collected**:
  - Queue lengths
  - Message rates
  - Connection counts
  - Consumer activity
 ## Best Practices
 ### 1. Service Implementation
 Always initialize tracing in your service's `main.py`:
 ```python
 from fastapi import FastAPI
 from shared.monitoring.tracing import setup_tracing
 import os
 app = FastAPI(title="My Service")
 # Initialize tracing
 otel_endpoint = os.getenv("OTEL_EXPORTER_OTLP_ENDPOINT", "http://signoz-otel-collector.bakery-ia:4318")
 setup_tracing(
    app,
    service_name="my-service",
    service_version=os.getenv("SERVICE_VERSION", "1.0.0"),
    otel_endpoint=otel_endpoint
 )
 ```
 ### 2. Custom Spans
 Add custom spans for important operations:
 ```python
 from opentelemetry import trace
 tracer = trace.get_tracer(__name__)
@app.post("/process")
 async def process_data(data: dict):
    with tracer.start_as_current_span("process_data") as span:
        span.set_attribute("data.size", len(data))
        span.set_attribute("data.type", data.get("type"))
        # Your processing logic
        result = process(data)
        span.set_attribute("result.status", "success")
        return result
 ```
 ### 3. Error Tracking
 Record exceptions in spans:
 ```python
 from shared.monitoring.tracing import record_exception
 try:
    result = risky_operation()
 except Exception as e:
    record_exception(e)
    raise
 ```
 ### 4. Correlation
 Use trace IDs in logs for correlation:
 ```python
 from shared.monitoring.tracing import get_current_trace_id
 trace_id = get_current_trace_id()
 logger.info("Processing request", trace_id=trace_id)
 ```
 ## Next Steps
 1. ✅ **Verify SigNoz is running** - Run verification script
 2. ✅ **Generate test traffic** - Run traffic generation script
 3. ✅ **Check data collection** - Query ClickHouse or use UI
 4. ✅ **Access SigNoz UI** - Visualize traces, metrics, and logs
 5. ⏭️ **Set up dashboards** - Create custom dashboards for your use cases
 6. ⏭️ **Configure alerts** - Set up alerts for critical metrics
 7. ⏭️ **Document** - Document common queries and dashboard configurations
 ## Useful Commands
 ```bash
 # Quick status check
 kubectl get pods -n bakery-ia | grep signoz
 # View OTel Collector metrics
 kubectl port-forward -n bakery-ia svc/signoz-otel-collector 8888:8888
 # Then visit: http://localhost:8888/metrics
 # Restart OTel Collector
 kubectl rollout restart deployment/signoz-otel-collector -n bakery-ia
 # View all services with telemetry
 kubectl get pods -n bakery-ia -l tier!=infrastructure
 # Check specific service logs
 kubectl logs -n bakery-ia -l app=<service-name> --tail=100 -f
 # Port-forward to SigNoz UI
 kubectl port-forward -n bakery-ia svc/signoz 3301:8080
 ```
 ## Resources
 - [SigNoz Documentation](https://signoz.io/docs/)
 - [OpenTelemetry Python](https://opentelemetry.io/docs/languages/python/)
 - [SigNoz GitHub](https://github.com/SigNoz/signoz)
 - [Helm Chart Values](infrastructure/helm/signoz-values-dev.yaml)
 - [Verification Script](infrastructure/helm/verify-signoz-telemetry.sh)
 - [Traffic Generation Script](infrastructure/helm/generate-test-traffic.sh)
--- a/infrastructure/helm/generate-test-traffic.sh
+++ b/infrastructure/helm/generate-test-traffic.sh
@@ -0,0 +1,141 @@
 #!/bin/bash
 # Generate Test Traffic to Services
 # This script generates API calls to verify telemetry data collection
 set -e
 NAMESPACE="bakery-ia"
 GREEN='\033[0;32m'
 BLUE='\033[0;34m'
 YELLOW='\033[1;33m'
 NC='\033[0m'
 echo -e "${BLUE}━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━${NC}"
 echo -e "${BLUE}  Generating Test Traffic for SigNoz Verification${NC}"
 echo -e "${BLUE}━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━${NC}"
 echo ""
 # Check if ingress is accessible
 echo -e "${BLUE}Step 1: Verifying Gateway Access${NC}"
 echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
 GATEWAY_POD=$(kubectl get pods -n $NAMESPACE -l app=gateway --field-selector=status.phase=Running -o jsonpath='{.items[0].metadata.name}' 2>/dev/null)
 if [[ -z "$GATEWAY_POD" ]]; then
    echo -e "${YELLOW}⚠ Gateway pod not running. Starting port-forward...${NC}"
    # Port forward in background
    kubectl port-forward -n $NAMESPACE svc/gateway-service 8000:8000 &
    PORT_FORWARD_PID=$!
    sleep 3
    API_URL="http://localhost:8000"
 else
    echo -e "${GREEN}✓ Gateway is running: $GATEWAY_POD${NC}"
    # Use internal service
    API_URL="http://gateway-service.$NAMESPACE.svc.cluster.local:8000"
 fi
 echo ""
 # Function to make API call from inside cluster
 make_request() {
    local endpoint=$1
    local description=$2
    echo -e "${BLUE}→ Testing: $description${NC}"
    echo "  Endpoint: $endpoint"
    if [[ -n "$GATEWAY_POD" ]]; then
        # Make request from inside the gateway pod
        RESPONSE=$(kubectl exec -n $NAMESPACE $GATEWAY_POD -- curl -s -w "\nHTTP_CODE:%{http_code}" "$API_URL$endpoint" 2>/dev/null || echo "FAILED")
    else
        # Make request from localhost
        RESPONSE=$(curl -s -w "\nHTTP_CODE:%{http_code}" "$API_URL$endpoint" 2>/dev/null || echo "FAILED")
    fi
    if [[ "$RESPONSE" == "FAILED" ]]; then
        echo -e "  ${YELLOW}⚠ Request failed${NC}"
    else
        HTTP_CODE=$(echo "$RESPONSE" | grep "HTTP_CODE" | cut -d: -f2)
        if [[ "$HTTP_CODE" == "200" ]] || [[ "$HTTP_CODE" == "401" ]] || [[ "$HTTP_CODE" == "404" ]]; then
            echo -e "  ${GREEN}✓ Response received (HTTP $HTTP_CODE)${NC}"
        else
            echo -e "  ${YELLOW}⚠ Unexpected response (HTTP $HTTP_CODE)${NC}"
        fi
    fi
    echo ""
    sleep 1
 }
 # Generate traffic to various endpoints
 echo -e "${BLUE}Step 2: Generating Traffic to Services${NC}"
 echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
 echo ""
 # Health checks (should generate traces)
 make_request "/health" "Gateway Health Check"
 make_request "/api/health" "API Health Check"
 # Auth service endpoints
 make_request "/api/auth/health" "Auth Service Health"
 # Tenant service endpoints
 make_request "/api/tenants/health" "Tenant Service Health"
 # Inventory service endpoints
 make_request "/api/inventory/health" "Inventory Service Health"
 # Orders service endpoints
 make_request "/api/orders/health" "Orders Service Health"
 # Forecasting service endpoints
 make_request "/api/forecasting/health" "Forecasting Service Health"
 echo -e "${BLUE}Step 3: Checking Service Logs for Telemetry${NC}"
 echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
 echo ""
 # Check a few service pods for tracing logs
 SERVICES=("auth-service" "inventory-service" "gateway")
 for service in "${SERVICES[@]}"; do
    POD=$(kubectl get pods -n $NAMESPACE -l app=$service --field-selector=status.phase=Running -o jsonpath='{.items[0].metadata.name}' 2>/dev/null)
    if [[ -n "$POD" ]]; then
        echo -e "${BLUE}Checking $service ($POD)...${NC}"
        TRACING_LOG=$(kubectl logs -n $NAMESPACE $POD --tail=100 2>/dev/null | grep -i "tracing\|otel" | head -n 2 || echo "")
        if [[ -n "$TRACING_LOG" ]]; then
            echo -e "${GREEN}✓ Tracing configured:${NC}"
            echo "$TRACING_LOG" | sed 's/^/  /'
        else
            echo -e "${YELLOW}⚠ No tracing logs found${NC}"
        fi
        echo ""
    fi
 done
 # Wait for data to be processed
 echo -e "${BLUE}Step 4: Waiting for Data Processing${NC}"
 echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
 echo "Waiting 30 seconds for telemetry data to be processed..."
 for i in {30..1}; do
    echo -ne "\r  ${i} seconds remaining..."
    sleep 1
 done
 echo -e "\n"
 # Cleanup port-forward if started
 if [[ -n "$PORT_FORWARD_PID" ]]; then
    kill $PORT_FORWARD_PID 2>/dev/null || true
 fi
 echo -e "${GREEN}✓ Test traffic generation complete!${NC}"
 echo ""
 echo -e "${BLUE}Next Steps:${NC}"
 echo "1. Run the verification script to check for collected data:"
 echo "   ./infrastructure/helm/verify-signoz-telemetry.sh"
 echo ""
 echo "2. Access SigNoz UI to visualize the data:"
 echo "   https://monitoring.bakery-ia.local"
 echo "   or"
 echo "   kubectl port-forward -n bakery-ia svc/signoz 3301:8080"
 echo "   Then go to: http://localhost:3301"
 echo ""
 echo -e "${BLUE}━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━${NC}"
--- a/infrastructure/helm/signoz-values-dev.yaml
+++ b/infrastructure/helm/signoz-values-dev.yaml
@@ -181,6 +181,15 @@ otelCollector:
  # OpenTelemetry Collector configuration
  config:
    # Connectors - bridge between pipelines
    connectors:
      signozmeter:
        dimensions:
          - name: service.name
          - name: deployment.environment
          - name: host.name
        metrics_flush_interval: 1h
    receivers:
      # OTLP receivers for traces, metrics, and logs from applications
      # All application telemetry is pushed via OTLP protocol
@@ -256,6 +265,12 @@ otelCollector:
        send_batch_size: 10000  # Increased from 1024 for better performance
        send_batch_max_size: 10000
      # Batch processor for meter data
      batch/meter:
        timeout: 1s
        send_batch_size: 20000
        send_batch_max_size: 25000
      # Memory limiter to prevent OOM
      memory_limiter:
        check_interval: 1s
@@ -267,11 +282,19 @@ otelCollector:
        detectors: [env, system, docker]
        timeout: 5s
-      # Span metrics processor for automatic service metrics
+      # SigNoz span metrics processor with delta aggregation (recommended)
-      spanmetrics:
+      # Generates RED metrics (Rate, Error, Duration) from trace spans
      signozspanmetrics/delta:
        aggregation_temporality: AGGREGATION_TEMPORALITY_DELTA
        metrics_exporter: signozclickhousemetrics
-        latency_histogram_buckets: [2ms, 4ms, 6ms, 8ms, 10ms, 50ms, 100ms, 200ms, 400ms, 800ms, 1s, 1400ms, 2s, 5s, 10s, 15s]
+        latency_histogram_buckets: [100us, 1ms, 2ms, 6ms, 10ms, 50ms, 100ms, 250ms, 500ms, 1000ms, 1400ms, 2000ms, 5s, 10s, 20s, 40s, 60s]
-        dimensions_cache_size: 10000
+        dimensions_cache_size: 100000
        dimensions:
          - name: service.namespace
            default: default
          - name: deployment.environment
            default: default
          - name: signoz.collector.id
    exporters:
      # ClickHouse exporter for traces
@@ -294,6 +317,13 @@ otelCollector:
          max_interval: 30s
          max_elapsed_time: 300s
      # ClickHouse exporter for meter data (usage metrics)
      signozclickhousemeter:
        dsn: "tcp://admin:27ff0399-0d3a-4bd8-919d-17c2181e6fb9@signoz-clickhouse:9000/signoz_meter"
        timeout: 45s
        sending_queue:
          enabled: false
      # ClickHouse exporter for logs
      clickhouselogsexporter:
        dsn: tcp://signoz-clickhouse:9000/?database=signoz_logs
@@ -303,6 +333,13 @@ otelCollector:
          initial_interval: 5s
          max_interval: 30s
      # Metadata exporter for service metadata
      metadataexporter:
        dsn: "tcp://admin:27ff0399-0d3a-4bd8-919d-17c2181e6fb9@signoz-clickhouse:9000/signoz_metadata"
        timeout: 10s
        cache:
          provider: in_memory
      # Debug exporter for debugging (optional)
      debug:
        verbosity: detailed
@@ -311,11 +348,11 @@ otelCollector:
    service:
      pipelines:
-        # Traces pipeline
+        # Traces pipeline - exports to ClickHouse and signozmeter connector
        traces:
          receivers: [otlp]
-          processors: [memory_limiter, batch, spanmetrics, resourcedetection]
+          processors: [memory_limiter, batch, signozspanmetrics/delta, resourcedetection]
-          exporters: [clickhousetraces]
+          exporters: [clickhousetraces, metadataexporter, signozmeter]
        # Metrics pipeline
        metrics:
@@ -323,6 +360,12 @@ otelCollector:
          processors: [memory_limiter, batch, resourcedetection]
          exporters: [signozclickhousemetrics]
        # Meter pipeline - receives from signozmeter connector
        metrics/meter:
          receivers: [signozmeter]
          processors: [batch/meter]
          exporters: [signozclickhousemeter]
        # Logs pipeline
        logs:
          receivers: [otlp]
--- a/infrastructure/helm/signoz-values-prod.yaml
+++ b/infrastructure/helm/signoz-values-prod.yaml
@@ -269,6 +269,15 @@ otelCollector:
  # Full OTEL Collector Configuration
  config:
    # Connectors - bridge between pipelines
    connectors:
      signozmeter:
        dimensions:
          - name: service.name
          - name: deployment.environment
          - name: host.name
        metrics_flush_interval: 1h
    extensions:
      health_check:
        endpoint: 0.0.0.0:13133
@@ -304,6 +313,12 @@ otelCollector:
        send_batch_size: 50000  # Increased from 2048 (official recommendation for traces)
        send_batch_max_size: 50000
      # Batch processor for meter data
      batch/meter:
        timeout: 1s
        send_batch_size: 20000
        send_batch_max_size: 25000
      memory_limiter:
        check_interval: 1s
        limit_mib: 1500  # 75% of container memory (2Gi = ~2048Mi)
@@ -324,11 +339,19 @@ otelCollector:
            value: bakery-ia-prod
            action: upsert
-      # Span metrics processor for automatic service performance metrics
+      # SigNoz span metrics processor with delta aggregation (recommended)
-      spanmetrics:
+      # Generates RED metrics (Rate, Error, Duration) from trace spans
      signozspanmetrics/delta:
        aggregation_temporality: AGGREGATION_TEMPORALITY_DELTA
        metrics_exporter: signozclickhousemetrics
-        latency_histogram_buckets: [2ms, 4ms, 6ms, 8ms, 10ms, 50ms, 100ms, 200ms, 400ms, 800ms, 1s, 1400ms, 2s, 5s, 10s, 15s]
+        latency_histogram_buckets: [100us, 1ms, 2ms, 6ms, 10ms, 50ms, 100ms, 250ms, 500ms, 1000ms, 1400ms, 2000ms, 5s, 10s, 20s, 40s, 60s]
        dimensions_cache_size: 100000
        dimensions:
          - name: service.namespace
            default: default
          - name: deployment.environment
            default: production
          - name: signoz.collector.id
    exporters:
      # Export to SigNoz ClickHouse
@@ -350,6 +373,13 @@ otelCollector:
          max_interval: 30s
          max_elapsed_time: 300s
      # ClickHouse exporter for meter data (usage metrics)
      signozclickhousemeter:
        dsn: "tcp://clickhouse:9000/?database=signoz_meter"
        timeout: 45s
        sending_queue:
          enabled: false
      clickhouselogsexporter:
        dsn: tcp://clickhouse:9000/?database=signoz_logs
        timeout: 10s
@@ -359,6 +389,13 @@ otelCollector:
          max_interval: 30s
          max_elapsed_time: 300s
      # Metadata exporter for service metadata
      metadataexporter:
        dsn: "tcp://clickhouse:9000/?database=signoz_metadata"
        timeout: 10s
        cache:
          provider: in_memory
      # Debug exporter for debugging (replaces deprecated logging exporter)
      debug:
        verbosity: detailed
@@ -368,16 +405,25 @@ otelCollector:
    service:
      extensions: [health_check, zpages]
      pipelines:
        # Traces pipeline - exports to ClickHouse and signozmeter connector
        traces:
          receivers: [otlp]
-          processors: [memory_limiter, batch, spanmetrics, resourcedetection, resource]
+          processors: [memory_limiter, batch, signozspanmetrics/delta, resourcedetection, resource]
-          exporters: [clickhousetraces]
+          exporters: [clickhousetraces, metadataexporter, signozmeter]
        # Metrics pipeline
        metrics:
          receivers: [otlp, prometheus]
          processors: [memory_limiter, batch, resourcedetection, resource]
          exporters: [signozclickhousemetrics]
        # Meter pipeline - receives from signozmeter connector
        metrics/meter:
          receivers: [signozmeter]
          processors: [batch/meter]
          exporters: [signozclickhousemeter]
        # Logs pipeline
        logs:
          receivers: [otlp]
          processors: [memory_limiter, batch, resourcedetection, resource]
--- a/infrastructure/helm/verify-signoz-telemetry.sh
+++ b/infrastructure/helm/verify-signoz-telemetry.sh
@@ -0,0 +1,177 @@
 #!/bin/bash
 # SigNoz Telemetry Verification Script
 # This script verifies that services are correctly sending metrics, logs, and traces to SigNoz
 # and that SigNoz is collecting them properly.
 set -e
 NAMESPACE="bakery-ia"
 GREEN='\033[0;32m'
 RED='\033[0;31m'
 YELLOW='\033[1;33m'
 BLUE='\033[0;34m'
 NC='\033[0m' # No Color
 echo -e "${BLUE}━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━${NC}"
 echo -e "${BLUE}  SigNoz Telemetry Verification Script${NC}"
 echo -e "${BLUE}━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━${NC}"
 echo ""
 # Step 1: Verify SigNoz Components are Running
 echo -e "${BLUE}[1/7] Checking SigNoz Components Status...${NC}"
 echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
 OTEL_POD=$(kubectl get pods -n $NAMESPACE -l app.kubernetes.io/name=signoz,app.kubernetes.io/component=otel-collector --field-selector=status.phase=Running -o jsonpath='{.items[0].metadata.name}' 2>/dev/null)
 SIGNOZ_POD=$(kubectl get pods -n $NAMESPACE -l app.kubernetes.io/name=signoz,app.kubernetes.io/component=signoz --field-selector=status.phase=Running -o jsonpath='{.items[0].metadata.name}' 2>/dev/null)
 CLICKHOUSE_POD=$(kubectl get pods -n $NAMESPACE -l clickhouse.altinity.com/chi=signoz-clickhouse --field-selector=status.phase=Running -o jsonpath='{.items[0].metadata.name}' 2>/dev/null)
 if [[ -n "$OTEL_POD" && -n "$SIGNOZ_POD" && -n "$CLICKHOUSE_POD" ]]; then
    echo -e "${GREEN}✓ All SigNoz components are running${NC}"
    echo "  - OTel Collector: $OTEL_POD"
    echo "  - SigNoz Frontend: $SIGNOZ_POD"
    echo "  - ClickHouse: $CLICKHOUSE_POD"
 else
    echo -e "${RED}✗ Some SigNoz components are not running${NC}"
    kubectl get pods -n $NAMESPACE | grep signoz
    exit 1
 fi
 echo ""
 # Step 2: Check OTel Collector Endpoints
 echo -e "${BLUE}[2/7] Verifying OTel Collector Endpoints...${NC}"
 echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
 OTEL_SVC=$(kubectl get svc -n $NAMESPACE signoz-otel-collector -o jsonpath='{.spec.clusterIP}')
 echo "OTel Collector Service IP: $OTEL_SVC"
 echo ""
 echo "Available endpoints:"
 kubectl get svc -n $NAMESPACE signoz-otel-collector -o jsonpath='{range .spec.ports[*]}{.name}{"\t"}{.port}{"\n"}{end}' | column -t
 echo ""
 echo -e "${GREEN}✓ OTel Collector endpoints are exposed${NC}"
 echo ""
 # Step 3: Check OTel Collector Logs for Data Reception
 echo -e "${BLUE}[3/7] Checking OTel Collector for Recent Activity...${NC}"
 echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
 echo "Recent OTel Collector logs (last 20 lines):"
 kubectl logs -n $NAMESPACE $OTEL_POD --tail=20 | grep -E "received|exported|traces|metrics|logs" || echo "No recent telemetry data found in logs"
 echo ""
 # Step 4: Check Service Configurations
 echo -e "${BLUE}[4/7] Verifying Service Telemetry Configuration...${NC}"
 echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
 # Check ConfigMap for OTEL settings
 OTEL_ENDPOINT=$(kubectl get configmap bakery-config -n $NAMESPACE -o jsonpath='{.data.OTEL_EXPORTER_OTLP_ENDPOINT}')
 ENABLE_TRACING=$(kubectl get configmap bakery-config -n $NAMESPACE -o jsonpath='{.data.ENABLE_TRACING}')
 ENABLE_METRICS=$(kubectl get configmap bakery-config -n $NAMESPACE -o jsonpath='{.data.ENABLE_METRICS}')
 ENABLE_LOGS=$(kubectl get configmap bakery-config -n $NAMESPACE -o jsonpath='{.data.ENABLE_LOGS}')
 echo "Configuration from bakery-config ConfigMap:"
 echo "  OTEL_EXPORTER_OTLP_ENDPOINT: $OTEL_ENDPOINT"
 echo "  ENABLE_TRACING: $ENABLE_TRACING"
 echo "  ENABLE_METRICS: $ENABLE_METRICS"
 echo "  ENABLE_LOGS: $ENABLE_LOGS"
 echo ""
 if [[ "$ENABLE_TRACING" == "true" && "$ENABLE_METRICS" == "true" && "$ENABLE_LOGS" == "true" ]]; then
    echo -e "${GREEN}✓ Telemetry is enabled in configuration${NC}"
 else
    echo -e "${YELLOW}⚠ Some telemetry features may be disabled${NC}"
 fi
 echo ""
 # Step 5: Test OTel Collector Health
 echo -e "${BLUE}[5/7] Testing OTel Collector Health Endpoint...${NC}"
 echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
 HEALTH_CHECK=$(kubectl exec -n $NAMESPACE $OTEL_POD -- wget -qO- http://localhost:13133/ 2>/dev/null || echo "FAILED")
 if [[ "$HEALTH_CHECK" == *"Server available"* ]] || [[ "$HEALTH_CHECK" == "{}" ]]; then
    echo -e "${GREEN}✓ OTel Collector health check passed${NC}"
 else
    echo -e "${RED}✗ OTel Collector health check failed${NC}"
    echo "Response: $HEALTH_CHECK"
 fi
 echo ""
 # Step 6: Query ClickHouse for Telemetry Data
 echo -e "${BLUE}[6/7] Querying ClickHouse for Telemetry Data...${NC}"
 echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
 # Get ClickHouse credentials
 CH_PASSWORD=$(kubectl get secret -n $NAMESPACE signoz-clickhouse -o jsonpath='{.data.admin-password}' 2>/dev/null | base64 -d || echo "27ff0399-0d3a-4bd8-919d-17c2181e6fb9")
 echo "Checking for traces in ClickHouse..."
 TRACES_COUNT=$(kubectl exec -n $NAMESPACE $CLICKHOUSE_POD -- clickhouse-client --user=admin --password=$CH_PASSWORD --query="SELECT count() FROM signoz_traces.signoz_index_v2 WHERE timestamp >= now() - INTERVAL 1 HOUR" 2>/dev/null || echo "0")
 echo "  Traces in last hour: $TRACES_COUNT"
 echo "Checking for metrics in ClickHouse..."
 METRICS_COUNT=$(kubectl exec -n $NAMESPACE $CLICKHOUSE_POD -- clickhouse-client --user=admin --password=$CH_PASSWORD --query="SELECT count() FROM signoz_metrics.samples_v4 WHERE unix_milli >= toUnixTimestamp(now() - INTERVAL 1 HOUR) * 1000" 2>/dev/null || echo "0")
 echo "  Metrics in last hour: $METRICS_COUNT"
 echo "Checking for logs in ClickHouse..."
 LOGS_COUNT=$(kubectl exec -n $NAMESPACE $CLICKHOUSE_POD -- clickhouse-client --user=admin --password=$CH_PASSWORD --query="SELECT count() FROM signoz_logs.logs WHERE timestamp >= now() - INTERVAL 1 HOUR" 2>/dev/null || echo "0")
 echo "  Logs in last hour: $LOGS_COUNT"
 echo ""
 if [[ "$TRACES_COUNT" -gt "0" || "$METRICS_COUNT" -gt "0" || "$LOGS_COUNT" -gt "0" ]]; then
    echo -e "${GREEN}✓ Telemetry data found in ClickHouse!${NC}"
 else
    echo -e "${YELLOW}⚠ No telemetry data found in the last hour${NC}"
    echo "  This might be normal if:"
    echo "  - Services were just deployed"
    echo "  - No traffic has been generated yet"
    echo "  - Services haven't finished initializing"
 fi
 echo ""
 # Step 7: Access Information
 echo -e "${BLUE}[7/7] SigNoz UI Access Information${NC}"
 echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
 echo ""
 echo "SigNoz is accessible via ingress at:"
 echo -e "  ${GREEN}https://monitoring.bakery-ia.local${NC}"
 echo ""
 echo "Or via port-forward:"
 echo -e "  ${YELLOW}kubectl port-forward -n $NAMESPACE svc/signoz 3301:8080${NC}"
 echo "  Then access: http://localhost:3301"
 echo ""
 echo "To view OTel Collector metrics:"
 echo -e "  ${YELLOW}kubectl port-forward -n $NAMESPACE svc/signoz-otel-collector 8888:8888${NC}"
 echo "  Then access: http://localhost:8888/metrics"
 echo ""
 # Summary
 echo ""
 echo -e "${BLUE}━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━${NC}"
 echo -e "${BLUE}  Verification Summary${NC}"
 echo -e "${BLUE}━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━${NC}"
 echo ""
 echo "Component Status:"
 echo "  ✓ SigNoz components running"
 echo "  ✓ OTel Collector healthy"
 echo "  ✓ Configuration correct"
 echo ""
 echo "Data Collection (last hour):"
 echo "  Traces: $TRACES_COUNT"
 echo "  Metrics: $METRICS_COUNT"
 echo "  Logs: $LOGS_COUNT"
 echo ""
 if [[ "$TRACES_COUNT" -gt "0" || "$METRICS_COUNT" -gt "0" || "$LOGS_COUNT" -gt "0" ]]; then
    echo -e "${GREEN}✓ SigNoz is collecting telemetry data successfully!${NC}"
 else
    echo -e "${YELLOW}⚠ To generate telemetry data, try:${NC}"
    echo ""
    echo "1. Generate traffic to your services:"
    echo "   curl http://localhost/api/health"
    echo ""
    echo "2. Check service logs for tracing initialization:"
    echo "   kubectl logs -n $NAMESPACE <service-pod> | grep -i 'tracing\\|otel\\|signoz'"
    echo ""
    echo "3. Wait a few minutes and run this script again"
 fi
 echo ""
 echo -e "${BLUE}━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━${NC}"
--- a/infrastructure/kubernetes/base/components/ai-insights/ai-insights-service.yaml
+++ b/infrastructure/kubernetes/base/components/ai-insights/ai-insights-service.yaml
@@ -99,7 +99,10 @@ spec:
        - name: OTEL_COLLECTOR_ENDPOINT
          value: "http://signoz-otel-collector.bakery-ia.svc.cluster.local:4318"
        - name: OTEL_EXPORTER_OTLP_ENDPOINT
-          value: "http://signoz-otel-collector.bakery-ia.svc.cluster.local:4318"
+          valueFrom:
            configMapKeyRef:
              name: bakery-config
              key: OTEL_EXPORTER_OTLP_ENDPOINT
        - name: OTEL_SERVICE_NAME
          value: "ai-insights-service"
        - name: ENABLE_TRACING
--- a/infrastructure/kubernetes/base/components/auth/auth-service.yaml
+++ b/infrastructure/kubernetes/base/components/auth/auth-service.yaml
@@ -100,7 +100,10 @@ spec:
        - name: OTEL_COLLECTOR_ENDPOINT
          value: "http://signoz-otel-collector.bakery-ia.svc.cluster.local:4318"
        - name: OTEL_EXPORTER_OTLP_ENDPOINT
-          value: "http://signoz-otel-collector.bakery-ia.svc.cluster.local:4318"
+          valueFrom:
            configMapKeyRef:
              name: bakery-config
              key: OTEL_EXPORTER_OTLP_ENDPOINT
        - name: OTEL_SERVICE_NAME
          value: "auth-service"
        - name: ENABLE_TRACING
--- a/infrastructure/kubernetes/base/components/distribution/distribution-service.yaml
+++ b/infrastructure/kubernetes/base/components/distribution/distribution-service.yaml
@@ -64,7 +64,10 @@ spec:
        - name: OTEL_COLLECTOR_ENDPOINT
          value: "http://signoz-otel-collector.bakery-ia.svc.cluster.local:4318"
        - name: OTEL_EXPORTER_OTLP_ENDPOINT
-          value: "http://signoz-otel-collector.bakery-ia.svc.cluster.local:4318"
+          valueFrom:
            configMapKeyRef:
              name: bakery-config
              key: OTEL_EXPORTER_OTLP_ENDPOINT
        - name: OTEL_SERVICE_NAME
          value: "distribution-service"
        - name: ENABLE_TRACING
--- a/infrastructure/kubernetes/base/components/external/external-service.yaml
+++ b/infrastructure/kubernetes/base/components/external/external-service.yaml
@@ -92,7 +92,10 @@ spec:
        - name: OTEL_COLLECTOR_ENDPOINT
          value: "http://signoz-otel-collector.bakery-ia.svc.cluster.local:4318"
        - name: OTEL_EXPORTER_OTLP_ENDPOINT
-          value: "http://signoz-otel-collector.bakery-ia.svc.cluster.local:4318"
+          valueFrom:
            configMapKeyRef:
              name: bakery-config
              key: OTEL_EXPORTER_OTLP_ENDPOINT
        - name: OTEL_SERVICE_NAME
          value: "external-service"
        - name: ENABLE_TRACING
--- a/infrastructure/kubernetes/base/components/forecasting/forecasting-service.yaml
+++ b/infrastructure/kubernetes/base/components/forecasting/forecasting-service.yaml
@@ -99,7 +99,10 @@ spec:
        - name: OTEL_COLLECTOR_ENDPOINT
          value: "http://signoz-otel-collector.bakery-ia.svc.cluster.local:4318"
        - name: OTEL_EXPORTER_OTLP_ENDPOINT
-          value: "http://signoz-otel-collector.bakery-ia.svc.cluster.local:4318"
+          valueFrom:
            configMapKeyRef:
              name: bakery-config
              key: OTEL_EXPORTER_OTLP_ENDPOINT
        - name: OTEL_SERVICE_NAME
          value: "forecasting-service"
        - name: ENABLE_TRACING
--- a/infrastructure/kubernetes/base/components/infrastructure/gateway-service.yaml
+++ b/infrastructure/kubernetes/base/components/infrastructure/gateway-service.yaml
@@ -52,7 +52,10 @@ spec:
            name: whatsapp-secrets
        env:
        - name: OTEL_EXPORTER_OTLP_ENDPOINT
-          value: "http://signoz-otel-collector.bakery-ia.svc.cluster.local:4317"
+          valueFrom:
            configMapKeyRef:
              name: bakery-config
              key: OTEL_EXPORTER_OTLP_ENDPOINT
        resources:
          requests:
            memory: "256Mi"
--- a/infrastructure/kubernetes/base/components/inventory/inventory-service.yaml
+++ b/infrastructure/kubernetes/base/components/inventory/inventory-service.yaml
@@ -99,7 +99,10 @@ spec:
        - name: OTEL_COLLECTOR_ENDPOINT
          value: "http://signoz-otel-collector.bakery-ia.svc.cluster.local:4318"
        - name: OTEL_EXPORTER_OTLP_ENDPOINT
-          value: "http://signoz-otel-collector.bakery-ia.svc.cluster.local:4318"
+          valueFrom:
            configMapKeyRef:
              name: bakery-config
              key: OTEL_EXPORTER_OTLP_ENDPOINT
        - name: OTEL_SERVICE_NAME
          value: "inventory-service"
        - name: ENABLE_TRACING
--- a/infrastructure/kubernetes/base/components/notification/notification-service.yaml
+++ b/infrastructure/kubernetes/base/components/notification/notification-service.yaml
@@ -99,7 +99,10 @@ spec:
        - name: OTEL_COLLECTOR_ENDPOINT
          value: "http://signoz-otel-collector.bakery-ia.svc.cluster.local:4318"
        - name: OTEL_EXPORTER_OTLP_ENDPOINT
-          value: "http://signoz-otel-collector.bakery-ia.svc.cluster.local:4318"
+          valueFrom:
            configMapKeyRef:
              name: bakery-config
              key: OTEL_EXPORTER_OTLP_ENDPOINT
        - name: OTEL_SERVICE_NAME
          value: "notification-service"
        - name: ENABLE_TRACING
--- a/infrastructure/kubernetes/base/components/orchestrator/orchestrator-service.yaml
+++ b/infrastructure/kubernetes/base/components/orchestrator/orchestrator-service.yaml
@@ -99,7 +99,10 @@ spec:
        - name: OTEL_COLLECTOR_ENDPOINT
          value: "http://signoz-otel-collector.bakery-ia.svc.cluster.local:4318"
        - name: OTEL_EXPORTER_OTLP_ENDPOINT
-          value: "http://signoz-otel-collector.bakery-ia.svc.cluster.local:4318"
+          valueFrom:
            configMapKeyRef:
              name: bakery-config
              key: OTEL_EXPORTER_OTLP_ENDPOINT
        - name: OTEL_SERVICE_NAME
          value: "orchestrator-service"
        - name: ENABLE_TRACING
--- a/infrastructure/kubernetes/base/components/orders/orders-service.yaml
+++ b/infrastructure/kubernetes/base/components/orders/orders-service.yaml
@@ -99,7 +99,10 @@ spec:
        - name: OTEL_COLLECTOR_ENDPOINT
          value: "http://signoz-otel-collector.bakery-ia.svc.cluster.local:4318"
        - name: OTEL_EXPORTER_OTLP_ENDPOINT
-          value: "http://signoz-otel-collector.bakery-ia.svc.cluster.local:4318"
+          valueFrom:
            configMapKeyRef:
              name: bakery-config
              key: OTEL_EXPORTER_OTLP_ENDPOINT
        - name: OTEL_SERVICE_NAME
          value: "orders-service"
        - name: ENABLE_TRACING
--- a/infrastructure/kubernetes/base/components/pos/pos-service.yaml
+++ b/infrastructure/kubernetes/base/components/pos/pos-service.yaml
@@ -99,7 +99,10 @@ spec:
        - name: OTEL_COLLECTOR_ENDPOINT
          value: "http://signoz-otel-collector.bakery-ia.svc.cluster.local:4318"
        - name: OTEL_EXPORTER_OTLP_ENDPOINT
-          value: "http://signoz-otel-collector.bakery-ia.svc.cluster.local:4318"
+          valueFrom:
            configMapKeyRef:
              name: bakery-config
              key: OTEL_EXPORTER_OTLP_ENDPOINT
        - name: OTEL_SERVICE_NAME
          value: "pos-service"
        - name: ENABLE_TRACING
--- a/infrastructure/kubernetes/base/components/procurement/procurement-service.yaml
+++ b/infrastructure/kubernetes/base/components/procurement/procurement-service.yaml
@@ -99,7 +99,10 @@ spec:
        - name: OTEL_COLLECTOR_ENDPOINT
          value: "http://signoz-otel-collector.bakery-ia.svc.cluster.local:4318"
        - name: OTEL_EXPORTER_OTLP_ENDPOINT
-          value: "http://signoz-otel-collector.bakery-ia.svc.cluster.local:4318"
+          valueFrom:
            configMapKeyRef:
              name: bakery-config
              key: OTEL_EXPORTER_OTLP_ENDPOINT
        - name: OTEL_SERVICE_NAME
          value: "procurement-service"
        - name: ENABLE_TRACING
--- a/infrastructure/kubernetes/base/components/production/production-service.yaml
+++ b/infrastructure/kubernetes/base/components/production/production-service.yaml
@@ -99,7 +99,10 @@ spec:
        - name: OTEL_COLLECTOR_ENDPOINT
          value: "http://signoz-otel-collector.bakery-ia.svc.cluster.local:4318"
        - name: OTEL_EXPORTER_OTLP_ENDPOINT
-          value: "http://signoz-otel-collector.bakery-ia.svc.cluster.local:4318"
+          valueFrom:
            configMapKeyRef:
              name: bakery-config
              key: OTEL_EXPORTER_OTLP_ENDPOINT
        - name: OTEL_SERVICE_NAME
          value: "production-service"
        - name: ENABLE_TRACING
--- a/infrastructure/kubernetes/base/components/recipes/recipes-service.yaml
+++ b/infrastructure/kubernetes/base/components/recipes/recipes-service.yaml
@@ -99,7 +99,10 @@ spec:
        - name: OTEL_COLLECTOR_ENDPOINT
          value: "http://signoz-otel-collector.bakery-ia.svc.cluster.local:4318"
        - name: OTEL_EXPORTER_OTLP_ENDPOINT
-          value: "http://signoz-otel-collector.bakery-ia.svc.cluster.local:4318"
+          valueFrom:
            configMapKeyRef:
              name: bakery-config
              key: OTEL_EXPORTER_OTLP_ENDPOINT
        - name: OTEL_SERVICE_NAME
          value: "recipes-service"
        - name: ENABLE_TRACING
--- a/infrastructure/kubernetes/base/components/sales/sales-service.yaml
+++ b/infrastructure/kubernetes/base/components/sales/sales-service.yaml
@@ -99,7 +99,10 @@ spec:
        - name: OTEL_COLLECTOR_ENDPOINT
          value: "http://signoz-otel-collector.bakery-ia.svc.cluster.local:4318"
        - name: OTEL_EXPORTER_OTLP_ENDPOINT
-          value: "http://signoz-otel-collector.bakery-ia.svc.cluster.local:4318"
+          valueFrom:
            configMapKeyRef:
              name: bakery-config
              key: OTEL_EXPORTER_OTLP_ENDPOINT
        - name: OTEL_SERVICE_NAME
          value: "sales-service"
        - name: ENABLE_TRACING
--- a/infrastructure/kubernetes/base/components/suppliers/suppliers-service.yaml
+++ b/infrastructure/kubernetes/base/components/suppliers/suppliers-service.yaml
@@ -99,7 +99,10 @@ spec:
        - name: OTEL_COLLECTOR_ENDPOINT
          value: "http://signoz-otel-collector.bakery-ia.svc.cluster.local:4318"
        - name: OTEL_EXPORTER_OTLP_ENDPOINT
-          value: "http://signoz-otel-collector.bakery-ia.svc.cluster.local:4318"
+          valueFrom:
            configMapKeyRef:
              name: bakery-config
              key: OTEL_EXPORTER_OTLP_ENDPOINT
        - name: OTEL_SERVICE_NAME
          value: "suppliers-service"
        - name: ENABLE_TRACING
--- a/infrastructure/kubernetes/base/components/tenant/tenant-service.yaml
+++ b/infrastructure/kubernetes/base/components/tenant/tenant-service.yaml
@@ -99,7 +99,10 @@ spec:
        - name: OTEL_COLLECTOR_ENDPOINT
          value: "http://signoz-otel-collector.bakery-ia.svc.cluster.local:4318"
        - name: OTEL_EXPORTER_OTLP_ENDPOINT
-          value: "http://signoz-otel-collector.bakery-ia.svc.cluster.local:4318"
+          valueFrom:
            configMapKeyRef:
              name: bakery-config
              key: OTEL_EXPORTER_OTLP_ENDPOINT
        - name: OTEL_SERVICE_NAME
          value: "tenant-service"
        - name: ENABLE_TRACING
--- a/infrastructure/kubernetes/base/components/training/training-service.yaml
+++ b/infrastructure/kubernetes/base/components/training/training-service.yaml
@@ -99,7 +99,10 @@ spec:
        - name: OTEL_COLLECTOR_ENDPOINT
          value: "http://signoz-otel-collector.bakery-ia.svc.cluster.local:4318"
        - name: OTEL_EXPORTER_OTLP_ENDPOINT
-          value: "http://signoz-otel-collector.bakery-ia.svc.cluster.local:4318"
+          valueFrom:
            configMapKeyRef:
              name: bakery-config
              key: OTEL_EXPORTER_OTLP_ENDPOINT
        - name: OTEL_SERVICE_NAME
          value: "training-service"
        - name: ENABLE_TRACING
--- a/infrastructure/kubernetes/base/configmap.yaml
+++ b/infrastructure/kubernetes/base/configmap.yaml
@@ -385,7 +385,8 @@ data:
  # OBSERVABILITY - SigNoz (Unified Monitoring)
  # ================================================================
  # OpenTelemetry Configuration - Direct to SigNoz
-  OTEL_EXPORTER_OTLP_ENDPOINT: "http://signoz-otel-collector.bakery-ia.svc.cluster.local:4317"
+  # IMPORTANT: gRPC endpoints should NOT include http:// prefix
  OTEL_EXPORTER_OTLP_ENDPOINT: "signoz-otel-collector.bakery-ia.svc.cluster.local:4317"
  OTEL_EXPORTER_OTLP_PROTOCOL: "grpc"
  OTEL_SERVICE_NAME: "bakery-ia"
  OTEL_RESOURCE_ATTRIBUTES: "deployment.environment=development"
--- a/infrastructure/kubernetes/fix-otel-endpoints.sh
+++ b/infrastructure/kubernetes/fix-otel-endpoints.sh
@@ -0,0 +1,60 @@
 #!/bin/bash
 # Fix OTEL endpoint configuration in all service manifests
 # This script replaces hardcoded OTEL_EXPORTER_OTLP_ENDPOINT values
 # with references to the central bakery-config ConfigMap
 set -e
 GREEN='\033[0;32m'
 BLUE='\033[0;34m'
 NC='\033[0m'
 echo -e "${BLUE}Fixing OTEL endpoint configuration in all services...${NC}"
 echo ""
 # Find all service YAML files
 SERVICE_FILES=$(find infrastructure/kubernetes/base/components -name "*-service.yaml")
 for file in $SERVICE_FILES; do
    # Check if file contains hardcoded OTEL_EXPORTER_OTLP_ENDPOINT
    if grep -q "name: OTEL_EXPORTER_OTLP_ENDPOINT" "$file"; then
        # Check if it's already using configMapKeyRef
        if grep -A 3 "name: OTEL_EXPORTER_OTLP_ENDPOINT" "$file" | grep -q "configMapKeyRef"; then
            echo -e "${GREEN}✓ $file already using ConfigMap${NC}"
        else
            echo -e "${BLUE}→ Fixing $file${NC}"
            # Create a temporary file
            tmp_file=$(mktemp)
            # Process the file
            awk '
            /name: OTEL_EXPORTER_OTLP_ENDPOINT/ {
                print $0
                # Read and skip the next line (value line)
                getline
                # Output the configMapKeyRef instead
                print "          valueFrom:"
                print "            configMapKeyRef:"
                print "              name: bakery-config"
                print "              key: OTEL_EXPORTER_OTLP_ENDPOINT"
                next
            }
            { print }
            ' "$file" > "$tmp_file"
            # Replace original file
            mv "$tmp_file" "$file"
            echo -e "${GREEN}  ✓ Fixed${NC}"
        fi
    fi
 done
 echo ""
 echo -e "${GREEN}✓ All service files processed!${NC}"
 echo ""
 echo "Next steps:"
 echo "1. Review changes: git diff infrastructure/kubernetes/base/components"
 echo "2. Apply changes: kubectl apply -k infrastructure/kubernetes/overlays/dev"
 echo "3. Restart services: kubectl rollout restart deployment -n bakery-ia --all"
--- a/infrastructure/kubernetes/overlays/prod/prod-configmap.yaml
+++ b/infrastructure/kubernetes/overlays/prod/prod-configmap.yaml
@@ -23,7 +23,8 @@ data:
  ENABLE_LOGS: "true"
  # OpenTelemetry Configuration - Direct to SigNoz
-  OTEL_EXPORTER_OTLP_ENDPOINT: "http://signoz-otel-collector.bakery-ia.svc.cluster.local:4317"
+  # IMPORTANT: gRPC endpoints should NOT include http:// prefix
  OTEL_EXPORTER_OTLP_ENDPOINT: "signoz-otel-collector.bakery-ia.svc.cluster.local:4317"
  OTEL_EXPORTER_OTLP_PROTOCOL: "grpc"
  OTEL_SERVICE_NAME: "bakery-ia"
  OTEL_RESOURCE_ATTRIBUTES: "deployment.environment=production,cluster.name=bakery-ia-prod"