bakery-ia/docs/DATABASE_INITIALIZATION.md

# Database Initialization System

This document explains the automatic database initialization system for the Bakery-IA microservices architecture.

## Overview

The system handles two main scenarios:

1. **Production/First-time Deployment**: Automatically creates tables from SQLAlchemy models and sets up Alembic version tracking
2. **Development Workflow**: Provides easy reset capabilities to start with a clean slate

## Key Features

- ✅ **Automatic Table Creation**: Creates tables from SQLAlchemy models when database is empty
- ✅ **Alembic Integration**: Properly manages migration versions and history
- ✅ **Development Reset**: Easy clean-slate restart for development
- ✅ **Production Ready**: Safe for production deployments
- ✅ **All 14 Services**: Works across all microservices

## How It Works

### 1. Automatic Detection

The system automatically detects the database state:

- **Empty Database**: Creates tables from models and initializes Alembic
- **Existing Database with Alembic**: Runs pending migrations
- **Existing Database without Alembic**: Initializes Alembic on existing schema
- **Force Recreate Mode**: Drops everything and recreates (development only)

### 2. Integration Points

#### Service Startup
```python
# In your service main.py
class AuthService(StandardFastAPIService):
    # Migration verification happens automatically during startup
    pass
```

#### Kubernetes Migration Jobs
```yaml
# Enhanced migration jobs handle automatic table creation
containers:
- name: migrate
  image: bakery/auth-service:${IMAGE_TAG}
  command: ["python", "/app/scripts/run_migrations.py", "auth"]
```

#### Environment Variables
```bash
# Control behavior via environment variables
DB_FORCE_RECREATE=true    # Force recreate tables (development)
DEVELOPMENT_MODE=true     # Enable development features
```

## Usage Scenarios

### 1. First-Time Production Deployment

**What happens:**
1. Migration job detects empty database
2. Creates all tables from SQLAlchemy models
3. Stamps Alembic with the latest migration version
4. Service starts and verifies migration state

**No manual intervention required!**

### 2. Development - Clean Slate Reset

**Option A: Using the Development Script**
```bash
# Reset specific service
./scripts/dev-reset-database.sh --service auth

# Reset all services
./scripts/dev-reset-database.sh --all

# Reset with auto-confirmation
./scripts/dev-reset-database.sh --service auth --yes
```

**Option B: Using the Workflow Script**
```bash
# Clean start with dev profile
./scripts/dev-workflow.sh clean --profile dev

# Reset specific service and restart
./scripts/dev-workflow.sh reset --service auth
```

**Option C: Manual Environment Variable**
```bash
# Set force recreate mode
kubectl patch configmap development-config -n bakery-ia \
  --patch='{"data":{"DB_FORCE_RECREATE":"true"}}'

# Run migration job
kubectl apply -f infrastructure/kubernetes/base/migrations/auth-migration-job.yaml
```

### 3. Regular Development Workflow

```bash
# Start development environment
./scripts/dev-workflow.sh start --profile minimal

# Check status
./scripts/dev-workflow.sh status

# View logs for specific service
./scripts/dev-workflow.sh logs --service auth

# Run migrations only
./scripts/dev-workflow.sh migrate --service auth
```

## Configuration

### Skaffold Profiles

The system supports different deployment profiles:

```yaml
# skaffold.yaml profiles
profiles:
  - name: minimal          # Only auth and inventory
  - name: full            # All services + infrastructure
  - name: single          # Template for single service
  - name: dev             # Full development environment
```

### Environment Variables

| Variable | Description | Default |
|----------|-------------|---------|
| `DB_FORCE_RECREATE` | Force recreate tables | `false` |
| `DEVELOPMENT_MODE` | Enable development features | `false` |
| `DEBUG_LOGGING` | Enable debug logging | `false` |
| `SKIP_MIGRATION_VERSION_CHECK` | Skip version verification | `false` |

### Service Configuration

Each service automatically detects its configuration:

- **Models Module**: `services.{service}.app.models`
- **Alembic Config**: `services/{service}/alembic.ini`
- **Migration Scripts**: `services/{service}/migrations/versions/`

## Development Workflows

### Quick Start
```bash
# 1. Start minimal environment
./scripts/dev-workflow.sh start --profile minimal

# 2. Reset specific service when needed
./scripts/dev-workflow.sh reset --service auth

# 3. Clean restart when you want fresh start
./scripts/dev-workflow.sh clean --profile dev
```

### Database Reset Workflows

#### Scenario 1: "I want to reset auth service only"
```bash
./scripts/dev-reset-database.sh --service auth
```

#### Scenario 2: "I want to start completely fresh"
```bash
./scripts/dev-reset-database.sh --all
# or
./scripts/dev-workflow.sh clean --profile dev
```

#### Scenario 3: "I want to reset and restart in one command"
```bash
./scripts/dev-workflow.sh reset --service auth
```

## Technical Details

### Database Initialization Manager

The core logic is in [`shared/database/init_manager.py`](shared/database/init_manager.py):

```python
# Main initialization method
async def initialize_database(self) -> Dict[str, Any]:
    # Check current database state
    db_state = await self._check_database_state()

    # Handle different scenarios
    if self.force_recreate:
        result = await self._handle_force_recreate()
    elif db_state["is_empty"]:
        result = await self._handle_first_time_deployment()
    # ... etc
```

### Migration Job Enhancement

Migration jobs now use the enhanced runner:

```yaml
containers:
- name: migrate
  command: ["python", "/app/scripts/run_migrations.py", "auth"]
  env:
  - name: AUTH_DATABASE_URL
    valueFrom:
      secretKeyRef:
        name: database-secrets
        key: AUTH_DATABASE_URL
  - name: DB_FORCE_RECREATE
    valueFrom:
      configMapKeyRef:
        name: development-config
        key: DB_FORCE_RECREATE
```

### Service Integration

Services automatically handle table initialization during startup:

```python
async def _handle_database_tables(self):
    # Check if we're in force recreate mode
    force_recreate = os.getenv("DB_FORCE_RECREATE", "false").lower() == "true"

    # Initialize database with automatic table creation
    result = await initialize_service_database(
        database_manager=self.database_manager,
        service_name=self.service_name,
        force_recreate=force_recreate
    )
```

## Troubleshooting

### Common Issues

#### 1. Migration Job Fails
```bash
# Check job logs
kubectl logs -l job-name=auth-migration -n bakery-ia

# Check database connectivity
kubectl exec auth-db-pod -n bakery-ia -- pg_isready
```

#### 2. Service Won't Start
```bash
# Check service logs
kubectl logs -l app.kubernetes.io/name=auth -n bakery-ia

# Check database state
./scripts/dev-workflow.sh status
```

#### 3. Tables Not Created
```bash
# Force recreate mode
./scripts/dev-reset-database.sh --service auth --yes

# Check migration job status
kubectl get jobs -n bakery-ia
```

### Debugging Commands

```bash
# Check all components
./scripts/dev-workflow.sh status

# View specific service logs
./scripts/dev-workflow.sh logs --service auth

# Check migration jobs
kubectl get jobs -l app.kubernetes.io/component=migration -n bakery-ia

# Check ConfigMaps
kubectl get configmaps -n bakery-ia

# View database pods
kubectl get pods -l app.kubernetes.io/component=database -n bakery-ia
```

## Benefits

1. **Zero Manual Setup**: Tables are created automatically on first deployment
2. **Development Friendly**: Easy reset capabilities for clean development
3. **Production Safe**: Handles existing databases gracefully
4. **Alembic Compatible**: Maintains proper migration history and versioning
5. **Service Agnostic**: Works identically across all 14 microservices
6. **Kubernetes Native**: Integrates seamlessly with Kubernetes workflows

## Migration from TODO State

If you have existing services with TODO migrations:

1. **Keep existing models**: Your SQLAlchemy models are the source of truth
2. **Deploy normally**: The system will create tables from models automatically
3. **Alembic versions**: Will be stamped with the latest migration version
4. **No data loss**: Existing data is preserved in production deployments

The system eliminates the need to manually fill in TODO migration files while maintaining proper Alembic version tracking.