Initial commit - production deployment
This commit is contained in:
232
services/orchestrator/migrations/MIGRATION_GUIDE.md
Normal file
232
services/orchestrator/migrations/MIGRATION_GUIDE.md
Normal file
@@ -0,0 +1,232 @@
|
||||
# Migration Guide - Consolidated Schema
|
||||
|
||||
## Overview
|
||||
|
||||
This guide explains how to use the new consolidated initial schema migration for the Orchestration Service.
|
||||
|
||||
## Background
|
||||
|
||||
The orchestration service schema was previously split across two migration files:
|
||||
1. `20251029_1700_add_orchestration_runs.py` - Initial table creation
|
||||
2. `20251105_add_ai_insights_tracking.py` - AI insights fields addition
|
||||
|
||||
These have been consolidated into a single, well-structured initial schema file: `001_initial_schema.py`
|
||||
|
||||
## For New Deployments
|
||||
|
||||
If you're deploying the orchestration service for the first time:
|
||||
|
||||
1. **Use the consolidated migration:**
|
||||
```bash
|
||||
cd services/orchestrator
|
||||
alembic upgrade head
|
||||
```
|
||||
|
||||
2. **The migration will create:**
|
||||
- `orchestration_runs` table with all columns
|
||||
- `orchestrationstatus` enum type
|
||||
- All 15 indexes for optimal query performance
|
||||
|
||||
3. **Verify the migration:**
|
||||
```bash
|
||||
alembic current
|
||||
# Should show: 001_initial_schema (head)
|
||||
```
|
||||
|
||||
## For Existing Deployments
|
||||
|
||||
If your database already has the orchestration schema from the old migrations:
|
||||
|
||||
### Option 1: Keep Existing Migration History (Recommended)
|
||||
|
||||
**Do nothing.** Your existing migrations are functionally equivalent to the consolidated version. The schema structure is identical.
|
||||
|
||||
- Your alembic_version table will show: `20251105_add_ai_insights`
|
||||
- The new consolidated migration is for future deployments
|
||||
- No action needed - your database is up to date
|
||||
|
||||
### Option 2: Reset Migration History (For Clean State)
|
||||
|
||||
Only do this if you want a clean migration history and can afford downtime:
|
||||
|
||||
1. **Backup your database:**
|
||||
```bash
|
||||
pg_dump -h localhost -U user orchestrator_db > backup.sql
|
||||
```
|
||||
|
||||
2. **Drop and recreate schema:**
|
||||
```bash
|
||||
# In PostgreSQL
|
||||
DROP SCHEMA public CASCADE;
|
||||
CREATE SCHEMA public;
|
||||
```
|
||||
|
||||
3. **Apply consolidated migration:**
|
||||
```bash
|
||||
cd services/orchestrator
|
||||
alembic upgrade head
|
||||
```
|
||||
|
||||
4. **Restore data (if needed):**
|
||||
```bash
|
||||
psql -h localhost -U user orchestrator_db < backup_data_only.sql
|
||||
```
|
||||
|
||||
⚠️ **Warning:** This approach requires downtime and careful data backup/restore.
|
||||
|
||||
## Checking Your Current Migration Status
|
||||
|
||||
### Check Alembic version:
|
||||
```bash
|
||||
cd services/orchestrator
|
||||
alembic current
|
||||
```
|
||||
|
||||
### Check applied migrations:
|
||||
```bash
|
||||
alembic history
|
||||
```
|
||||
|
||||
### Verify table structure:
|
||||
```sql
|
||||
-- Check if table exists
|
||||
SELECT EXISTS (
|
||||
SELECT FROM information_schema.tables
|
||||
WHERE table_schema = 'public'
|
||||
AND table_name = 'orchestration_runs'
|
||||
);
|
||||
|
||||
-- Check column count
|
||||
SELECT COUNT(*) FROM information_schema.columns
|
||||
WHERE table_schema = 'public'
|
||||
AND table_name = 'orchestration_runs';
|
||||
-- Should return 47 columns
|
||||
|
||||
-- Check indexes
|
||||
SELECT indexname FROM pg_indexes
|
||||
WHERE tablename = 'orchestration_runs';
|
||||
-- Should return 15 indexes
|
||||
```
|
||||
|
||||
## Migration File Comparison
|
||||
|
||||
### Old Migration Chain
|
||||
```
|
||||
None → 20251029_1700 → 20251105_add_ai_insights
|
||||
```
|
||||
|
||||
### New Consolidated Migration
|
||||
```
|
||||
None → 001_initial_schema
|
||||
```
|
||||
|
||||
Both result in the exact same database schema, but the new version is:
|
||||
- ✅ Better organized and documented
|
||||
- ✅ Easier to understand and maintain
|
||||
- ✅ Fixes revision ID inconsistencies
|
||||
- ✅ Properly categorizes fields
|
||||
- ✅ Eliminates duplicate index definitions
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Issue: "relation already exists"
|
||||
**Cause:** Database already has the schema from old migrations.
|
||||
|
||||
**Solution:**
|
||||
- For existing deployments, no action needed
|
||||
- For fresh start, see "Option 2: Reset Migration History" above
|
||||
|
||||
### Issue: "enum type already exists"
|
||||
**Cause:** The orchestrationstatus enum was created by old migration.
|
||||
|
||||
**Solution:**
|
||||
- The migration uses `create_type=False` and `checkfirst=True` to handle this
|
||||
- Should not be an issue in practice
|
||||
|
||||
### Issue: "duplicate key value violates unique constraint on alembic_version"
|
||||
**Cause:** Trying to apply new migration on database with old migrations.
|
||||
|
||||
**Solution:**
|
||||
- Don't apply the new migration on existing databases
|
||||
- The old migrations already provide the same schema
|
||||
|
||||
## Deprecation Notice
|
||||
|
||||
### Files Superseded (Do Not Delete Yet)
|
||||
|
||||
The following migration files are superseded but kept for reference:
|
||||
- `20251029_1700_add_orchestration_runs.py`
|
||||
- `20251105_add_ai_insights_tracking.py`
|
||||
|
||||
**Why keep them?**
|
||||
- Existing deployments reference these migrations
|
||||
- Provides migration history for troubleshooting
|
||||
- Can be removed in future major version
|
||||
|
||||
### Future Cleanup
|
||||
|
||||
In a future major version (e.g., v2.0.0), after all deployments have migrated:
|
||||
1. Archive old migration files to `migrations/archive/`
|
||||
2. Update documentation to reference only consolidated schema
|
||||
3. Clean up alembic version history
|
||||
|
||||
## Best Practices
|
||||
|
||||
1. **Always backup before migrations:**
|
||||
```bash
|
||||
pg_dump -Fc -h localhost -U user orchestrator_db > backup_$(date +%Y%m%d).dump
|
||||
```
|
||||
|
||||
2. **Test migrations in staging first:**
|
||||
- Never run migrations directly in production
|
||||
- Verify schema changes in staging environment
|
||||
- Check application compatibility
|
||||
|
||||
3. **Monitor migration performance:**
|
||||
- Initial migration should complete in < 1 second for empty database
|
||||
- Index creation time scales with data volume
|
||||
|
||||
4. **Use version control:**
|
||||
- All migration files are in git
|
||||
- Never modify existing migration files
|
||||
- Create new migrations for schema changes
|
||||
|
||||
## Getting Help
|
||||
|
||||
If you encounter issues with migrations:
|
||||
|
||||
1. Check migration status: `alembic current`
|
||||
2. Review migration history: `alembic history`
|
||||
3. Check database schema: See SQL queries in "Checking Your Current Migration Status" section
|
||||
4. Review logs: Check alembic output for error details
|
||||
5. Consult SCHEMA_DOCUMENTATION.md for expected schema structure
|
||||
|
||||
## Next Steps
|
||||
|
||||
After successfully applying migrations:
|
||||
|
||||
1. **Verify application startup:**
|
||||
```bash
|
||||
docker-compose up orchestrator
|
||||
```
|
||||
|
||||
2. **Run health checks:**
|
||||
```bash
|
||||
curl http://localhost:8000/health
|
||||
```
|
||||
|
||||
3. **Test basic operations:**
|
||||
- Create a test orchestration run
|
||||
- Query run status
|
||||
- Verify data persistence
|
||||
|
||||
4. **Monitor logs:**
|
||||
```bash
|
||||
docker-compose logs -f orchestrator
|
||||
```
|
||||
|
||||
## Related Documentation
|
||||
|
||||
- `SCHEMA_DOCUMENTATION.md` - Complete schema reference
|
||||
- `001_initial_schema.py` - Consolidated migration file
|
||||
- `../../README.md` - Orchestration service overview
|
||||
Reference in New Issue
Block a user