bakery-ia/services/orchestrator/migrations/MIGRATION_GUIDE.md

Migration Guide - Consolidated Schema

Overview

This guide explains how to use the new consolidated initial schema migration for the Orchestration Service.

Background

The orchestration service schema was previously split across two migration files:

1. 20251029_1700_add_orchestration_runs.py - Initial table creation
2. 20251105_add_ai_insights_tracking.py - AI insights fields addition

These have been consolidated into a single, well-structured initial schema file: 001_initial_schema.py

For New Deployments

If you're deploying the orchestration service for the first time:

1. Use the consolidated migration:

   cd services/orchestrator
   alembic upgrade head
   

2. The migration will create:

   • orchestration_runs table with all columns
   • orchestrationstatus enum type
   • All 15 indexes for optimal query performance

3. Verify the migration:

   alembic current
   # Should show: 001_initial_schema (head)
   

For Existing Deployments

If your database already has the orchestration schema from the old migrations:

Option 1: Keep Existing Migration History (Recommended)

Do nothing. Your existing migrations are functionally equivalent to the consolidated version. The schema structure is identical.

• Your alembic_version table will show: 20251105_add_ai_insights
• The new consolidated migration is for future deployments
• No action needed - your database is up to date

Option 2: Reset Migration History (For Clean State)

Only do this if you want a clean migration history and can afford downtime:

1. Backup your database:

   pg_dump -h localhost -U user orchestrator_db > backup.sql
   

2. Drop and recreate schema:

   # In PostgreSQL
   DROP SCHEMA public CASCADE;
   CREATE SCHEMA public;
   

3. Apply consolidated migration:

   cd services/orchestrator
   alembic upgrade head
   

4. Restore data (if needed):

   psql -h localhost -U user orchestrator_db < backup_data_only.sql
   

⚠️ Warning: This approach requires downtime and careful data backup/restore.

Checking Your Current Migration Status

Check Alembic version:

cd services/orchestrator
alembic current

Check applied migrations:

alembic history

Verify table structure:

-- Check if table exists
SELECT EXISTS (
    SELECT FROM information_schema.tables
    WHERE table_schema = 'public'
    AND table_name = 'orchestration_runs'
);

-- Check column count
SELECT COUNT(*) FROM information_schema.columns
WHERE table_schema = 'public'
AND table_name = 'orchestration_runs';
-- Should return 47 columns

-- Check indexes
SELECT indexname FROM pg_indexes
WHERE tablename = 'orchestration_runs';
-- Should return 15 indexes

Migration File Comparison

Old Migration Chain

None → 20251029_1700 → 20251105_add_ai_insights

New Consolidated Migration

None → 001_initial_schema

Both result in the exact same database schema, but the new version is:

• ✅ Better organized and documented
• ✅ Easier to understand and maintain
• ✅ Fixes revision ID inconsistencies
• ✅ Properly categorizes fields
• ✅ Eliminates duplicate index definitions

Troubleshooting

Issue: "relation already exists"

Cause: Database already has the schema from old migrations.

Solution:

• For existing deployments, no action needed
• For fresh start, see "Option 2: Reset Migration History" above

Issue: "enum type already exists"

Cause: The orchestrationstatus enum was created by old migration.

Solution:

• The migration uses create_type=False and checkfirst=True to handle this
• Should not be an issue in practice

Issue: "duplicate key value violates unique constraint on alembic_version"

Cause: Trying to apply new migration on database with old migrations.

Solution:

• Don't apply the new migration on existing databases
• The old migrations already provide the same schema

Deprecation Notice

Files Superseded (Do Not Delete Yet)

The following migration files are superseded but kept for reference:

• 20251029_1700_add_orchestration_runs.py
• 20251105_add_ai_insights_tracking.py

Why keep them?

• Existing deployments reference these migrations
• Provides migration history for troubleshooting
• Can be removed in future major version

Future Cleanup

In a future major version (e.g., v2.0.0), after all deployments have migrated:

1. Archive old migration files to migrations/archive/
2. Update documentation to reference only consolidated schema
3. Clean up alembic version history

Best Practices

1. Always backup before migrations:

   pg_dump -Fc -h localhost -U user orchestrator_db > backup_$(date +%Y%m%d).dump
   

2. Test migrations in staging first:

   • Never run migrations directly in production
   • Verify schema changes in staging environment
   • Check application compatibility

3. Monitor migration performance:

   • Initial migration should complete in < 1 second for empty database
   • Index creation time scales with data volume

4. Use version control:

   • All migration files are in git
   • Never modify existing migration files
   • Create new migrations for schema changes

Getting Help

If you encounter issues with migrations:

1. Check migration status: alembic current
2. Review migration history: alembic history
3. Check database schema: See SQL queries in "Checking Your Current Migration Status" section
4. Review logs: Check alembic output for error details
5. Consult SCHEMA_DOCUMENTATION.md for expected schema structure

Next Steps

After successfully applying migrations:

1. Verify application startup:

   docker-compose up orchestrator
   

2. Run health checks:

   curl http://localhost:8000/health
   

3. Test basic operations:

   • Create a test orchestration run
   • Query run status
   • Verify data persistence

4. Monitor logs:

   docker-compose logs -f orchestrator
   

Related Documentation

• SCHEMA_DOCUMENTATION.md - Complete schema reference
• 001_initial_schema.py - Consolidated migration file
• ../../README.md - Orchestration service overview