Add migration services
This commit is contained in:
Urtzi Alfaro
309
docs/DATABASE_INITIALIZATION.md
Normal file
309
docs/DATABASE_INITIALIZATION.md
Normal file
@@ -0,0 +1,309 @@
|
||||
# Database Initialization System
|
||||
|
||||
This document explains the automatic database initialization system for the Bakery-IA microservices architecture.
|
||||
|
||||
## Overview
|
||||
|
||||
The system handles two main scenarios:
|
||||
|
||||
1. **Production/First-time Deployment**: Automatically creates tables from SQLAlchemy models and sets up Alembic version tracking
|
||||
2. **Development Workflow**: Provides easy reset capabilities to start with a clean slate
|
||||
|
||||
## Key Features
|
||||
|
||||
- ✅ **Automatic Table Creation**: Creates tables from SQLAlchemy models when database is empty
|
||||
- ✅ **Alembic Integration**: Properly manages migration versions and history
|
||||
- ✅ **Development Reset**: Easy clean-slate restart for development
|
||||
- ✅ **Production Ready**: Safe for production deployments
|
||||
- ✅ **All 14 Services**: Works across all microservices
|
||||
|
||||
## How It Works
|
||||
|
||||
### 1. Automatic Detection
|
||||
|
||||
The system automatically detects the database state:
|
||||
|
||||
- **Empty Database**: Creates tables from models and initializes Alembic
|
||||
- **Existing Database with Alembic**: Runs pending migrations
|
||||
- **Existing Database without Alembic**: Initializes Alembic on existing schema
|
||||
- **Force Recreate Mode**: Drops everything and recreates (development only)
|
||||
|
||||
### 2. Integration Points
|
||||
|
||||
#### Service Startup
|
||||
```python
|
||||
# In your service main.py
|
||||
class AuthService(StandardFastAPIService):
|
||||
# Migration verification happens automatically during startup
|
||||
pass
|
||||
```
|
||||
|
||||
#### Kubernetes Migration Jobs
|
||||
```yaml
|
||||
# Enhanced migration jobs handle automatic table creation
|
||||
containers:
|
||||
- name: migrate
|
||||
image: bakery/auth-service:${IMAGE_TAG}
|
||||
command: ["python", "/app/scripts/run_migrations.py", "auth"]
|
||||
```
|
||||
|
||||
#### Environment Variables
|
||||
```bash
|
||||
# Control behavior via environment variables
|
||||
DB_FORCE_RECREATE=true # Force recreate tables (development)
|
||||
DEVELOPMENT_MODE=true # Enable development features
|
||||
```
|
||||
|
||||
## Usage Scenarios
|
||||
|
||||
### 1. First-Time Production Deployment
|
||||
|
||||
**What happens:**
|
||||
1. Migration job detects empty database
|
||||
2. Creates all tables from SQLAlchemy models
|
||||
3. Stamps Alembic with the latest migration version
|
||||
4. Service starts and verifies migration state
|
||||
|
||||
**No manual intervention required!**
|
||||
|
||||
### 2. Development - Clean Slate Reset
|
||||
|
||||
**Option A: Using the Development Script**
|
||||
```bash
|
||||
# Reset specific service
|
||||
./scripts/dev-reset-database.sh --service auth
|
||||
|
||||
# Reset all services
|
||||
./scripts/dev-reset-database.sh --all
|
||||
|
||||
# Reset with auto-confirmation
|
||||
./scripts/dev-reset-database.sh --service auth --yes
|
||||
```
|
||||
|
||||
**Option B: Using the Workflow Script**
|
||||
```bash
|
||||
# Clean start with dev profile
|
||||
./scripts/dev-workflow.sh clean --profile dev
|
||||
|
||||
# Reset specific service and restart
|
||||
./scripts/dev-workflow.sh reset --service auth
|
||||
```
|
||||
|
||||
**Option C: Manual Environment Variable**
|
||||
```bash
|
||||
# Set force recreate mode
|
||||
kubectl patch configmap development-config -n bakery-ia \
|
||||
--patch='{"data":{"DB_FORCE_RECREATE":"true"}}'
|
||||
|
||||
# Run migration job
|
||||
kubectl apply -f infrastructure/kubernetes/base/migrations/auth-migration-job.yaml
|
||||
```
|
||||
|
||||
### 3. Regular Development Workflow
|
||||
|
||||
```bash
|
||||
# Start development environment
|
||||
./scripts/dev-workflow.sh start --profile minimal
|
||||
|
||||
# Check status
|
||||
./scripts/dev-workflow.sh status
|
||||
|
||||
# View logs for specific service
|
||||
./scripts/dev-workflow.sh logs --service auth
|
||||
|
||||
# Run migrations only
|
||||
./scripts/dev-workflow.sh migrate --service auth
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
### Skaffold Profiles
|
||||
|
||||
The system supports different deployment profiles:
|
||||
|
||||
```yaml
|
||||
# skaffold.yaml profiles
|
||||
profiles:
|
||||
- name: minimal # Only auth and inventory
|
||||
- name: full # All services + infrastructure
|
||||
- name: single # Template for single service
|
||||
- name: dev # Full development environment
|
||||
```
|
||||
|
||||
### Environment Variables
|
||||
|
||||
| Variable | Description | Default |
|
||||
|----------|-------------|---------|
|
||||
| `DB_FORCE_RECREATE` | Force recreate tables | `false` |
|
||||
| `DEVELOPMENT_MODE` | Enable development features | `false` |
|
||||
| `DEBUG_LOGGING` | Enable debug logging | `false` |
|
||||
| `SKIP_MIGRATION_VERSION_CHECK` | Skip version verification | `false` |
|
||||
|
||||
### Service Configuration
|
||||
|
||||
Each service automatically detects its configuration:
|
||||
|
||||
- **Models Module**: `services.{service}.app.models`
|
||||
- **Alembic Config**: `services/{service}/alembic.ini`
|
||||
- **Migration Scripts**: `services/{service}/migrations/versions/`
|
||||
|
||||
## Development Workflows
|
||||
|
||||
### Quick Start
|
||||
```bash
|
||||
# 1. Start minimal environment
|
||||
./scripts/dev-workflow.sh start --profile minimal
|
||||
|
||||
# 2. Reset specific service when needed
|
||||
./scripts/dev-workflow.sh reset --service auth
|
||||
|
||||
# 3. Clean restart when you want fresh start
|
||||
./scripts/dev-workflow.sh clean --profile dev
|
||||
```
|
||||
|
||||
### Database Reset Workflows
|
||||
|
||||
#### Scenario 1: "I want to reset auth service only"
|
||||
```bash
|
||||
./scripts/dev-reset-database.sh --service auth
|
||||
```
|
||||
|
||||
#### Scenario 2: "I want to start completely fresh"
|
||||
```bash
|
||||
./scripts/dev-reset-database.sh --all
|
||||
# or
|
||||
./scripts/dev-workflow.sh clean --profile dev
|
||||
```
|
||||
|
||||
#### Scenario 3: "I want to reset and restart in one command"
|
||||
```bash
|
||||
./scripts/dev-workflow.sh reset --service auth
|
||||
```
|
||||
|
||||
## Technical Details
|
||||
|
||||
### Database Initialization Manager
|
||||
|
||||
The core logic is in [`shared/database/init_manager.py`](shared/database/init_manager.py):
|
||||
|
||||
```python
|
||||
# Main initialization method
|
||||
async def initialize_database(self) -> Dict[str, Any]:
|
||||
# Check current database state
|
||||
db_state = await self._check_database_state()
|
||||
|
||||
# Handle different scenarios
|
||||
if self.force_recreate:
|
||||
result = await self._handle_force_recreate()
|
||||
elif db_state["is_empty"]:
|
||||
result = await self._handle_first_time_deployment()
|
||||
# ... etc
|
||||
```
|
||||
|
||||
### Migration Job Enhancement
|
||||
|
||||
Migration jobs now use the enhanced runner:
|
||||
|
||||
```yaml
|
||||
containers:
|
||||
- name: migrate
|
||||
command: ["python", "/app/scripts/run_migrations.py", "auth"]
|
||||
env:
|
||||
- name: AUTH_DATABASE_URL
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: database-secrets
|
||||
key: AUTH_DATABASE_URL
|
||||
- name: DB_FORCE_RECREATE
|
||||
valueFrom:
|
||||
configMapKeyRef:
|
||||
name: development-config
|
||||
key: DB_FORCE_RECREATE
|
||||
```
|
||||
|
||||
### Service Integration
|
||||
|
||||
Services automatically handle table initialization during startup:
|
||||
|
||||
```python
|
||||
async def _handle_database_tables(self):
|
||||
# Check if we're in force recreate mode
|
||||
force_recreate = os.getenv("DB_FORCE_RECREATE", "false").lower() == "true"
|
||||
|
||||
# Initialize database with automatic table creation
|
||||
result = await initialize_service_database(
|
||||
database_manager=self.database_manager,
|
||||
service_name=self.service_name,
|
||||
force_recreate=force_recreate
|
||||
)
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Common Issues
|
||||
|
||||
#### 1. Migration Job Fails
|
||||
```bash
|
||||
# Check job logs
|
||||
kubectl logs -l job-name=auth-migration -n bakery-ia
|
||||
|
||||
# Check database connectivity
|
||||
kubectl exec auth-db-pod -n bakery-ia -- pg_isready
|
||||
```
|
||||
|
||||
#### 2. Service Won't Start
|
||||
```bash
|
||||
# Check service logs
|
||||
kubectl logs -l app.kubernetes.io/name=auth -n bakery-ia
|
||||
|
||||
# Check database state
|
||||
./scripts/dev-workflow.sh status
|
||||
```
|
||||
|
||||
#### 3. Tables Not Created
|
||||
```bash
|
||||
# Force recreate mode
|
||||
./scripts/dev-reset-database.sh --service auth --yes
|
||||
|
||||
# Check migration job status
|
||||
kubectl get jobs -n bakery-ia
|
||||
```
|
||||
|
||||
### Debugging Commands
|
||||
|
||||
```bash
|
||||
# Check all components
|
||||
./scripts/dev-workflow.sh status
|
||||
|
||||
# View specific service logs
|
||||
./scripts/dev-workflow.sh logs --service auth
|
||||
|
||||
# Check migration jobs
|
||||
kubectl get jobs -l app.kubernetes.io/component=migration -n bakery-ia
|
||||
|
||||
# Check ConfigMaps
|
||||
kubectl get configmaps -n bakery-ia
|
||||
|
||||
# View database pods
|
||||
kubectl get pods -l app.kubernetes.io/component=database -n bakery-ia
|
||||
```
|
||||
|
||||
## Benefits
|
||||
|
||||
1. **Zero Manual Setup**: Tables are created automatically on first deployment
|
||||
2. **Development Friendly**: Easy reset capabilities for clean development
|
||||
3. **Production Safe**: Handles existing databases gracefully
|
||||
4. **Alembic Compatible**: Maintains proper migration history and versioning
|
||||
5. **Service Agnostic**: Works identically across all 14 microservices
|
||||
6. **Kubernetes Native**: Integrates seamlessly with Kubernetes workflows
|
||||
|
||||
## Migration from TODO State
|
||||
|
||||
If you have existing services with TODO migrations:
|
||||
|
||||
1. **Keep existing models**: Your SQLAlchemy models are the source of truth
|
||||
2. **Deploy normally**: The system will create tables from models automatically
|
||||
3. **Alembic versions**: Will be stamped with the latest migration version
|
||||
4. **No data loss**: Existing data is preserved in production deployments
|
||||
|
||||
The system eliminates the need to manually fill in TODO migration files while maintaining proper Alembic version tracking.
|
||||
Reference in New Issue
Block a user