New enterprise feature

services/demo_session/README.md

@@ -23,13 +23,92 @@ The **Demo Session Service** creates ephemeral, isolated demo environments for s
 - **Suppliers** - 5+ sample supplier profiles
 - **Team Members** - Sample staff with different roles
 
-### Demo Scenarios
-- **Standard Bakery** - Small neighborhood bakery (1 location)
-- **Multi-Location** - Bakery chain (3 locations)
-- **High-Volume** - Large production bakery
-- **Custom Scenario** - Configurable for specific prospects
-- **Spanish Locale** - Madrid-based bakery examples
-- **Feature Showcase** - Highlight specific capabilities
+### Demo Scenarios (Two-Tier Architecture)
+
+**Professional Tier** (Single Bakery)
+- **Individual Bakery** - Standalone neighborhood bakery
+- **Central Production** - Central production facility (Obrador)
+- **Complete Workflow** - From raw materials to finished products
+- **Full Features** - Inventory, recipes, production, procurement, forecasting, sales
+- **Template-Based Cloning** - Instant duplication from pre-seeded parent template
+- **Data Volume**: ~3,000 records (inventory, recipes, production, orders, sales, forecasts)
+
+**Enterprise Tier** (Multi-Location Chain)
+- **Parent Obrador** - Central production facility (supplies children)
+- **3 Retail Outlets** - Madrid Centro, Barcelona Gràcia, Valencia Ruzafa
+- **Distribution Network** - VRP-optimized delivery routes (Mon/Wed/Fri)
+- **Hierarchical Structure** - Parent produces, children sell finished products only
+- **Cross-Location Analytics** - Aggregate forecasting, distribution planning
+- **Advanced Features** - Enterprise dashboard, multi-location inventory, route optimization
+- **Data Volume**: ~10,000 records (parent + 3 children + distribution history)
+
+### Demo Seeding Architecture
+
+**Two-Phase Template System**
+
+Phase 1: **Parent Template Creation** (Kubernetes Init Jobs)
+- 15 parent seed jobs create base template data for both Professional and Enterprise parent tenants
+- Execution order controlled by Helm hook weights (10-15)
+- Jobs run once during cluster initialization/upgrade
+- Professional parent: `a1b2c3d4-e5f6-47a8-b9c0-d1e2f3a4b5c6` (Individual Bakery)
+- Enterprise parent: `c3d4e5f6-a7b8-49c0-d1e2-f3a4b5c6d7e8` (Obrador Madrid)
+
+Parent Seeds (Hook Weight 10-15):
+1. Tenants (weight 10) - Base tenant configuration
+2. Subscription Plans (weight 11) - Professional/Enterprise tier definitions
+3. Tenant Members (weight 12) - Admin users and roles
+4. Suppliers (weight 12) - Raw material providers
+5. Inventory Products (weight 13) - Raw ingredients + finished products
+6. Recipes (weight 13) - Production formulas and BOMs
+7. Equipment (weight 13) - Ovens, mixers, packaging machines
+8. Quality Templates (weight 13) - QA checkpoints
+9. Stock (weight 14) - Initial inventory levels
+10. Production Batches (weight 14) - Historical production runs
+11. POS Configs (weight 14) - Point-of-sale settings
+12. Forecasts (weight 14) - Demand predictions
+13. Procurement Plans (weight 14) - Supplier ordering strategies
+14. Purchase Orders (weight 14) - Historical procurement
+15. Orders, Customers, Sales, Orchestration Runs, AI Models, Alerts (weight 15)
+
+Phase 2: **Child Retail Template Seeding** (Kubernetes Jobs, Hook Weight 50-57)
+- 8 child seed jobs create retail outlet data for 3 enterprise child tenants
+- Executes AFTER all parent seeds complete
+- Creates retail-specific data (finished products only, no raw ingredients)
+- Child tenants:
+  - `d4e5f6a7-b8c9-40d1-e2f3-a4b5c6d7e8f9` (Madrid Centro)
+  - `e5f6a7b8-c9d0-41e2-f3a4-b5c6d7e8f9a0` (Barcelona Gràcia)
+  - `f6a7b8c9-d0e1-42f3-a4b5-c6d7e8f9a0b1` (Valencia Ruzafa)
+
+Child Retail Seeds (Hook Weight 50-57):
+1. Inventory Retail (weight 50) - Finished products catalog
+2. Stock Retail (weight 51) - Retail inventory levels
+3. Orders Retail (weight 52) - Customer orders
+4. Customers Retail (weight 53) - Retail customer database
+5. Sales Retail (weight 54) - Sales transactions
+6. Forecasts Retail (weight 55) - Store-level demand forecasts
+7. Alerts Retail (weight 56) - Stockout/low-stock alerts
+8. Distribution History (weight 57) - 30 days of Obrador→retail deliveries
+
+**ID Transformation Pattern**
+- **XOR Transformation**: `tenant_specific_id = UUID(int=tenant_id_int ^ base_id_int)`
+- Ensures deterministic, unique IDs across parent and child tenants
+- Maintains referential integrity for related records
+- Used for: inventory products, recipes, equipment, batches, etc.
+
+**Temporal Consistency**
+- **BASE_REFERENCE_DATE**: January 8, 2025, 06:00 UTC
+- All demo data anchored to this reference point
+- Ensures consistent time-based queries and dashboards
+- Historical data: 30-90 days before BASE_REFERENCE_DATE
+- Future forecasts: 14-30 days after BASE_REFERENCE_DATE
+
+**Runtime Cloning** (CloneOrchestrator)
+- When a demo session is created, CloneOrchestrator duplicates template data
+- New tenant ID generated for the demo session
+- All related records cloned with updated tenant_id
+- XOR transformation applied to maintain relationships
+- Typical clone time: 2-5 seconds for Professional, 8-15 seconds for Enterprise
+- Isolated demo environment - changes don't affect template
 
 ### Session Management
 - **Auto-Expiration** - Automatic cleanup after expiry

services/demo_session/app/api/__init__.py

@@ -3,5 +3,6 @@
 from .demo_sessions import router as demo_sessions_router
 from .demo_accounts import router as demo_accounts_router
 from .demo_operations import router as demo_operations_router
+from .internal import router as internal_router
 
-__all__ = ["demo_sessions_router", "demo_accounts_router", "demo_operations_router"]
+__all__ = ["demo_sessions_router", "demo_accounts_router", "demo_operations_router", "internal_router"]

services/demo_session/app/api/demo_sessions.py

@@ -5,6 +5,7 @@ Demo Sessions API - Atomic CRUD operations on DemoSession model
 from fastapi import APIRouter, Depends, HTTPException, Path, Query, Request
 from typing import Optional
 from uuid import UUID
+from datetime import datetime, timezone
 import structlog
 import jwt
 
@@ -54,6 +55,41 @@ async def _background_cloning_task(session_id: str, session_obj_id: UUID, base_t
                 error=str(e),
                 exc_info=True
             )
+            # Attempt to update session status to failed if possible
+            try:
+                from app.core.database import db_manager
+                from app.models import DemoSession
+                from sqlalchemy import select, update
+
+                # Try to update the session directly in DB to mark it as failed
+                async with db_manager.session_factory() as update_db:
+                    from app.models import DemoSessionStatus
+                    update_result = await update_db.execute(
+                        update(DemoSession)
+                        .where(DemoSession.id == session_obj_id)
+                        .values(status=DemoSessionStatus.FAILED, cloning_completed_at=datetime.now(timezone.utc))
+                    )
+                    await update_db.commit()
+            except Exception as update_error:
+                logger.error(
+                    "Failed to update session status to FAILED after background task error",
+                    session_id=session_id,
+                    error=str(update_error)
+                )
+
+
+def _handle_task_result(task, session_id: str):
+    """Handle the result of the background cloning task"""
+    try:
+        # This will raise the exception if the task failed
+        task.result()
+    except Exception as e:
+        logger.error(
+            "Background cloning task failed with exception",
+            session_id=session_id,
+            error=str(e),
+            exc_info=True
+        )
 
 
 @router.post(
@@ -77,6 +113,7 @@ async def create_demo_session(
         session_manager = DemoSessionManager(db, redis)
         session = await session_manager.create_session(
             demo_account_type=request.demo_account_type,
+            subscription_tier=request.subscription_tier,
             user_id=request.user_id,
             ip_address=ip_address,
             user_agent=user_agent
@@ -92,10 +129,14 @@ async def create_demo_session(
         base_tenant_id = demo_config.get("base_tenant_id", str(session.base_demo_tenant_id))
 
         # Start cloning in background task with session ID (not session object)
-        asyncio.create_task(
+        # Store task reference in case we need to track it
+        task = asyncio.create_task(
             _background_cloning_task(session.session_id, session.id, base_tenant_id)
         )
 
+        # Add error handling for the task to prevent silent failures
+        task.add_done_callback(lambda t: _handle_task_result(t, session.session_id))
+
         # Generate session token
         session_token = jwt.encode(
             {
@@ -104,8 +145,8 @@ async def create_demo_session(
                 "demo_account_type": request.demo_account_type,
                 "exp": session.expires_at.timestamp()
             },
-            "demo-secret-key",
-            algorithm="HS256"
+            settings.JWT_SECRET_KEY,
+            algorithm=settings.JWT_ALGORITHM
         )
 
         return {

services/demo_session/app/api/internal.py

@@ -0,0 +1,82 @@
+"""
+Internal API for Demo Session Service
+Handles internal service-to-service operations
+"""
+
+from fastapi import APIRouter, Depends, HTTPException, Header
+from sqlalchemy.ext.asyncio import AsyncSession
+import structlog
+
+from app.core import get_db, settings
+from app.core.redis_wrapper import get_redis, DemoRedisWrapper
+from app.services.data_cloner import DemoDataCloner
+
+logger = structlog.get_logger()
+router = APIRouter()
+
+
+async def verify_internal_api_key(x_internal_api_key: str = Header(None)):
+    """Verify internal API key for service-to-service communication"""
+    required_key = settings.INTERNAL_API_KEY
+    if x_internal_api_key != required_key:
+        logger.warning("Unauthorized internal API access attempted")
+        raise HTTPException(status_code=403, detail="Invalid internal API key")
+    return True
+
+
+@router.post("/internal/demo/cleanup")
+async def cleanup_demo_session_internal(
+    cleanup_request: dict,
+    db: AsyncSession = Depends(get_db),
+    redis: DemoRedisWrapper = Depends(get_redis),
+    _: bool = Depends(verify_internal_api_key)
+):
+    """
+    Internal endpoint to cleanup demo session data for a specific tenant
+    Used by rollback mechanisms
+    """
+    try:
+        tenant_id = cleanup_request.get('tenant_id')
+        session_id = cleanup_request.get('session_id')
+
+        if not all([tenant_id, session_id]):
+            raise HTTPException(
+                status_code=400, 
+                detail="Missing required parameters: tenant_id, session_id"
+            )
+
+        logger.info(
+            "Internal cleanup requested", 
+            tenant_id=tenant_id,
+            session_id=session_id
+        )
+
+        data_cloner = DemoDataCloner(db, redis)
+        
+        # Delete session data for this tenant
+        await data_cloner.delete_session_data(
+            str(tenant_id),
+            session_id
+        )
+
+        logger.info(
+            "Internal cleanup completed",
+            tenant_id=tenant_id,
+            session_id=session_id
+        )
+
+        return {
+            "status": "completed",
+            "tenant_id": tenant_id,
+            "session_id": session_id
+        }
+
+    except Exception as e:
+        logger.error(
+            "Internal cleanup failed", 
+            error=str(e),
+            tenant_id=cleanup_request.get('tenant_id'),
+            session_id=cleanup_request.get('session_id'),
+            exc_info=True
+        )
+        raise HTTPException(status_code=500, detail=f"Failed to cleanup demo session: {str(e)}")

services/demo_session/app/main.py

@@ -10,7 +10,7 @@ import structlog
 from contextlib import asynccontextmanager
 
 from app.core import settings, DatabaseManager
-from app.api import demo_sessions, demo_accounts, demo_operations
+from app.api import demo_sessions, demo_accounts, demo_operations, internal
 from shared.redis_utils import initialize_redis, close_redis
 
 logger = structlog.get_logger()
@@ -81,6 +81,7 @@ async def global_exception_handler(request: Request, exc: Exception):
 app.include_router(demo_sessions.router)
 app.include_router(demo_accounts.router)
 app.include_router(demo_operations.router)
+app.include_router(internal.router)
 
 
 @app.get("/")

services/demo_session/app/services/clone_orchestrator.py

@@ -16,6 +16,10 @@ from app.models.demo_session import CloningStatus
 logger = structlog.get_logger()
 
 
+# Import json for Redis serialization
+import json
+
+
 class ServiceDefinition:
     """Definition of a service that can clone demo data"""
 
@@ -29,9 +33,10 @@ class ServiceDefinition:
 class CloneOrchestrator:
     """Orchestrates parallel demo data cloning across services"""
 
-    def __init__(self):
+    def __init__(self, redis_manager=None):
         from app.core.config import settings
         self.internal_api_key = settings.INTERNAL_API_KEY
+        self.redis_manager = redis_manager  # For real-time progress updates
 
         # Define services that participate in cloning
         # URLs should be internal Kubernetes service names
@@ -110,6 +115,66 @@ class CloneOrchestrator:
             ),
         ]
 
+    async def _update_progress_in_redis(
+        self,
+        session_id: str,
+        progress_data: Dict[str, Any]
+    ):
+        """Update cloning progress in Redis for real-time frontend polling"""
+        if not self.redis_manager:
+            return  # Skip if no Redis manager provided
+
+        try:
+            status_key = f"session:{session_id}:status"
+            client = await self.redis_manager.get_client()
+
+            # Get existing status data or create new
+            existing_data_str = await client.get(status_key)
+            if existing_data_str:
+                status_data = json.loads(existing_data_str)
+            else:
+                # Initialize basic status structure
+                status_data = {
+                    "session_id": session_id,
+                    "status": "pending",
+                    "progress": {},
+                    "total_records_cloned": 0
+                }
+
+            # Update progress field with new data
+            status_data["progress"] = progress_data
+
+            # Calculate total records cloned from progress
+            total_records = 0
+            if "parent" in progress_data and "total_records_cloned" in progress_data["parent"]:
+                total_records += progress_data["parent"]["total_records_cloned"]
+            if "children" in progress_data:
+                for child in progress_data["children"]:
+                    if isinstance(child, dict) and "records_cloned" in child:
+                        total_records += child["records_cloned"]
+
+            status_data["total_records_cloned"] = total_records
+
+            # Update Redis with 2-hour TTL
+            await client.setex(
+                status_key,
+                7200,  # 2 hours
+                json.dumps(status_data)
+            )
+
+            logger.debug(
+                "Updated progress in Redis",
+                session_id=session_id,
+                progress_keys=list(progress_data.keys())
+            )
+        except Exception as e:
+            # Don't fail cloning if progress update fails
+            logger.warning(
+                "Failed to update progress in Redis",
+                session_id=session_id,
+                error=str(e)
+            )
+
     async def clone_all_services(
         self,
         base_tenant_id: str,
@@ -535,6 +600,14 @@ class CloneOrchestrator:
         try:
             # Step 1: Clone parent tenant
             logger.info("Cloning parent tenant", session_id=session_id)
+
+            # Update progress: Parent cloning started
+            await self._update_progress_in_redis(session_id, {
+                "parent": {"overall_status": "pending"},
+                "children": [],
+                "distribution": {}
+            })
+
             parent_result = await self.clone_all_services(
                 base_tenant_id=base_tenant_id,
                 virtual_tenant_id=parent_tenant_id,
@@ -543,6 +616,13 @@ class CloneOrchestrator:
             )
             results["parent"] = parent_result
 
+            # Update progress: Parent cloning completed
+            await self._update_progress_in_redis(session_id, {
+                "parent": parent_result,
+                "children": [],
+                "distribution": {}
+            })
+
             # BUG-006 FIX: Track parent for potential rollback
             if parent_result.get("overall_status") not in ["failed"]:
                 rollback_stack.append({
@@ -599,6 +679,13 @@ class CloneOrchestrator:
                     child_count=len(child_configs)
                 )
 
+                # Update progress: Children cloning started
+                await self._update_progress_in_redis(session_id, {
+                    "parent": parent_result,
+                    "children": [{"status": "pending"} for _ in child_configs],
+                    "distribution": {}
+                })
+
                 child_tasks = []
                 for idx, (child_config, child_id) in enumerate(zip(child_configs, child_tenant_ids)):
                     task = self._clone_child_outlet(
@@ -617,6 +704,13 @@ class CloneOrchestrator:
                     for r in children_results
                 ]
 
+                # Update progress: Children cloning completed
+                await self._update_progress_in_redis(session_id, {
+                    "parent": parent_result,
+                    "children": results["children"],
+                    "distribution": {}
+                })
+
                 # BUG-006 FIX: Track children for potential rollback
                 for child_result in results["children"]:
                     if child_result.get("status") not in ["failed"]:
@@ -630,6 +724,13 @@ class CloneOrchestrator:
             distribution_url = os.getenv("DISTRIBUTION_SERVICE_URL", "http://distribution-service:8000")
             logger.info("Setting up distribution data", session_id=session_id, distribution_url=distribution_url)
 
+            # Update progress: Distribution starting
+            await self._update_progress_in_redis(session_id, {
+                "parent": parent_result,
+                "children": results["children"],
+                "distribution": {"status": "pending"}
+            })
+
             try:
                 async with httpx.AsyncClient(timeout=120.0) as client:  # Increased timeout for distribution setup
                     response = await client.post(
@@ -646,6 +747,13 @@ class CloneOrchestrator:
                     if response.status_code == 200:
                         results["distribution"] = response.json()
                         logger.info("Distribution setup completed successfully", session_id=session_id)
+
+                        # Update progress: Distribution completed
+                        await self._update_progress_in_redis(session_id, {
+                            "parent": parent_result,
+                            "children": results["children"],
+                            "distribution": results["distribution"]
+                        })
                     else:
                         error_detail = response.text if response.text else f"HTTP {response.status_code}"
                         results["distribution"] = {

services/demo_session/app/services/session_manager.py

@@ -27,7 +27,7 @@ class DemoSessionManager:
         self.db = db
         self.redis = redis
         self.repository = DemoSessionRepository(db)
-        self.orchestrator = CloneOrchestrator()
+        self.orchestrator = CloneOrchestrator(redis_manager=redis)  # Pass Redis for real-time progress updates
 
     async def create_session(
         self,