bakery-ia/services/orchestrator/migrations/MIGRATION_GUIDE.md

# Migration Guide - Consolidated Schema

## Overview

This guide explains how to use the new consolidated initial schema migration for the Orchestration Service.

## Background

The orchestration service schema was previously split across two migration files:
1. `20251029_1700_add_orchestration_runs.py` - Initial table creation
2. `20251105_add_ai_insights_tracking.py` - AI insights fields addition

These have been consolidated into a single, well-structured initial schema file: `001_initial_schema.py`

## For New Deployments

If you're deploying the orchestration service for the first time:

1. **Use the consolidated migration:**
   ```bash
   cd services/orchestrator
   alembic upgrade head
   ```

2. **The migration will create:**
   - `orchestration_runs` table with all columns
   - `orchestrationstatus` enum type
   - All 15 indexes for optimal query performance

3. **Verify the migration:**
   ```bash
   alembic current
   # Should show: 001_initial_schema (head)
   ```

## For Existing Deployments

If your database already has the orchestration schema from the old migrations:

### Option 1: Keep Existing Migration History (Recommended)

**Do nothing.** Your existing migrations are functionally equivalent to the consolidated version. The schema structure is identical.

- Your alembic_version table will show: `20251105_add_ai_insights`
- The new consolidated migration is for future deployments
- No action needed - your database is up to date

### Option 2: Reset Migration History (For Clean State)

Only do this if you want a clean migration history and can afford downtime:

1. **Backup your database:**
   ```bash
   pg_dump -h localhost -U user orchestrator_db > backup.sql
   ```

2. **Drop and recreate schema:**
   ```bash
   # In PostgreSQL
   DROP SCHEMA public CASCADE;
   CREATE SCHEMA public;
   ```

3. **Apply consolidated migration:**
   ```bash
   cd services/orchestrator
   alembic upgrade head
   ```

4. **Restore data (if needed):**
   ```bash
   psql -h localhost -U user orchestrator_db < backup_data_only.sql
   ```

⚠️ **Warning:** This approach requires downtime and careful data backup/restore.

## Checking Your Current Migration Status

### Check Alembic version:
```bash
cd services/orchestrator
alembic current
```

### Check applied migrations:
```bash
alembic history
```

### Verify table structure:
```sql
-- Check if table exists
SELECT EXISTS (
    SELECT FROM information_schema.tables
    WHERE table_schema = 'public'
    AND table_name = 'orchestration_runs'
);

-- Check column count
SELECT COUNT(*) FROM information_schema.columns
WHERE table_schema = 'public'
AND table_name = 'orchestration_runs';
-- Should return 47 columns

-- Check indexes
SELECT indexname FROM pg_indexes
WHERE tablename = 'orchestration_runs';
-- Should return 15 indexes
```

## Migration File Comparison

### Old Migration Chain
```
None → 20251029_1700 → 20251105_add_ai_insights
```

### New Consolidated Migration
```
None → 001_initial_schema
```

Both result in the exact same database schema, but the new version is:
- ✅ Better organized and documented
- ✅ Easier to understand and maintain
- ✅ Fixes revision ID inconsistencies
- ✅ Properly categorizes fields
- ✅ Eliminates duplicate index definitions

## Troubleshooting

### Issue: "relation already exists"
**Cause:** Database already has the schema from old migrations.

**Solution:**
- For existing deployments, no action needed
- For fresh start, see "Option 2: Reset Migration History" above

### Issue: "enum type already exists"
**Cause:** The orchestrationstatus enum was created by old migration.

**Solution:**
- The migration uses `create_type=False` and `checkfirst=True` to handle this
- Should not be an issue in practice

### Issue: "duplicate key value violates unique constraint on alembic_version"
**Cause:** Trying to apply new migration on database with old migrations.

**Solution:**
- Don't apply the new migration on existing databases
- The old migrations already provide the same schema

## Deprecation Notice

### Files Superseded (Do Not Delete Yet)

The following migration files are superseded but kept for reference:
- `20251029_1700_add_orchestration_runs.py`
- `20251105_add_ai_insights_tracking.py`

**Why keep them?**
- Existing deployments reference these migrations
- Provides migration history for troubleshooting
- Can be removed in future major version

### Future Cleanup

In a future major version (e.g., v2.0.0), after all deployments have migrated:
1. Archive old migration files to `migrations/archive/`
2. Update documentation to reference only consolidated schema
3. Clean up alembic version history

## Best Practices

1. **Always backup before migrations:**
   ```bash
   pg_dump -Fc -h localhost -U user orchestrator_db > backup_$(date +%Y%m%d).dump
   ```

2. **Test migrations in staging first:**
   - Never run migrations directly in production
   - Verify schema changes in staging environment
   - Check application compatibility

3. **Monitor migration performance:**
   - Initial migration should complete in < 1 second for empty database
   - Index creation time scales with data volume

4. **Use version control:**
   - All migration files are in git
   - Never modify existing migration files
   - Create new migrations for schema changes

## Getting Help

If you encounter issues with migrations:

1. Check migration status: `alembic current`
2. Review migration history: `alembic history`
3. Check database schema: See SQL queries in "Checking Your Current Migration Status" section
4. Review logs: Check alembic output for error details
5. Consult SCHEMA_DOCUMENTATION.md for expected schema structure

## Next Steps

After successfully applying migrations:

1. **Verify application startup:**
   ```bash
   docker-compose up orchestrator
   ```

2. **Run health checks:**
   ```bash
   curl http://localhost:8000/health
   ```

3. **Test basic operations:**
   - Create a test orchestration run
   - Query run status
   - Verify data persistence

4. **Monitor logs:**
   ```bash
   docker-compose logs -f orchestrator
   ```

## Related Documentation

- `SCHEMA_DOCUMENTATION.md` - Complete schema reference
- `001_initial_schema.py` - Consolidated migration file
- `../../README.md` - Orchestration service overview