8.2 KiB
Timezone-Aware Datetime Fix
Date: 2025-10-09 Status: ✅ RESOLVED
Problem
Error in forecasting service logs:
[error] Failed to get cached prediction
error=can't compare offset-naive and offset-aware datetimes
Root Cause
The forecasting service database uses DateTime(timezone=True) for all timestamp columns, which means they store timezone-aware datetime objects. However, the code was using datetime.utcnow() throughout, which returns timezone-naive datetime objects.
When comparing these two types (e.g., checking if cache has expired), Python raises:
TypeError: can't compare offset-naive and offset-aware datetimes
Database Schema
All datetime columns in forecasting service models use DateTime(timezone=True):
# From app/models/predictions.py
class PredictionCache(Base):
forecast_date = Column(DateTime(timezone=True), nullable=False)
created_at = Column(DateTime(timezone=True), default=lambda: datetime.now(timezone.utc))
expires_at = Column(DateTime(timezone=True), nullable=False) # ← Compared with datetime.utcnow()
# ... other columns
class ModelPerformanceMetric(Base):
evaluation_date = Column(DateTime(timezone=True), nullable=False)
created_at = Column(DateTime(timezone=True), default=lambda: datetime.now(timezone.utc))
# ... other columns
# From app/models/forecasts.py
class Forecast(Base):
forecast_date = Column(DateTime(timezone=True), nullable=False, index=True)
created_at = Column(DateTime(timezone=True), default=lambda: datetime.now(timezone.utc))
class PredictionBatch(Base):
requested_at = Column(DateTime(timezone=True), default=lambda: datetime.now(timezone.utc))
completed_at = Column(DateTime(timezone=True))
Solution
Replaced all datetime.utcnow() calls with datetime.now(timezone.utc) throughout the forecasting service.
Before (BROKEN):
# Returns timezone-naive datetime
cache_entry.expires_at < datetime.utcnow() # ❌ TypeError!
After (WORKING):
# Returns timezone-aware datetime
cache_entry.expires_at < datetime.now(timezone.utc) # ✅ Works!
Files Fixed
1. Import statements updated
Added timezone to imports in all affected files:
from datetime import datetime, timedelta, timezone
2. All datetime.utcnow() replaced
Fixed in 9 files across the forecasting service:
-
services/forecasting/app/repositories/prediction_cache_repository.py
- Line 53: Cache expiration time calculation
- Line 105: Cache expiry check (the main error)
- Line 175: Cleanup expired cache entries
- Line 212: Cache statistics query
-
services/forecasting/app/repositories/prediction_batch_repository.py
- Lines 84, 113, 143, 184: Batch completion timestamps
- Line 273: Recent activity queries
- Line 318: Cleanup old batches
- Line 357: Batch progress calculations
-
services/forecasting/app/repositories/forecast_repository.py
- Lines 162, 241: Forecast accuracy and trend analysis date ranges
-
services/forecasting/app/repositories/performance_metric_repository.py
- Line 101: Performance trends date range calculation
-
services/forecasting/app/repositories/base.py
- Lines 116, 118: Recent records queries
- Lines 124, 159, 161: Cleanup and statistics
-
services/forecasting/app/services/forecasting_service.py
- Lines 292, 365, 393, 409, 447, 553: Processing time calculations and timestamps
-
services/forecasting/app/api/forecasting_operations.py
- Line 274: API response timestamps
-
services/forecasting/app/api/scenario_operations.py
- Lines 68, 134, 163: Scenario simulation timestamps
-
services/forecasting/app/services/messaging.py
- Message timestamps
Verification
# Before fix
$ grep -r "datetime\.utcnow()" services/forecasting/app --include="*.py" | wc -l
20
# After fix
$ grep -r "datetime\.utcnow()" services/forecasting/app --include="*.py" | wc -l
0
Why This Matters
Timezone-Naive (datetime.utcnow())
>>> datetime.utcnow()
datetime.datetime(2025, 10, 9, 9, 10, 37, 123456) # No timezone info
Timezone-Aware (datetime.now(timezone.utc))
>>> datetime.now(timezone.utc)
datetime.datetime(2025, 10, 9, 9, 10, 37, 123456, tzinfo=datetime.timezone.utc) # Has timezone
When PostgreSQL stores DateTime(timezone=True) columns, it stores them as timezone-aware. Comparing these with timezone-naive datetimes fails.
Impact
This fix resolves:
- ✅ Cache expiration checks
- ✅ Batch status updates
- ✅ Performance metric queries
- ✅ Forecast analytics date ranges
- ✅ Cleanup operations
- ✅ Recent activity queries
Best Practice
Always use timezone-aware datetimes with PostgreSQL DateTime(timezone=True) columns:
# ✅ GOOD
created_at = Column(DateTime(timezone=True), default=lambda: datetime.now(timezone.utc))
expires_at = datetime.now(timezone.utc) + timedelta(hours=24)
if record.created_at < datetime.now(timezone.utc):
...
# ❌ BAD
created_at = Column(DateTime(timezone=True), default=datetime.utcnow) # No timezone!
expires_at = datetime.utcnow() + timedelta(hours=24) # Naive!
if record.created_at < datetime.utcnow(): # TypeError!
...
Additional Issue Found and Fixed
Local Import Shadowing
After the initial fix, a new error appeared:
[error] Multi-day forecast generation failed
error=cannot access local variable 'timezone' where it is not associated with a value
Cause: In forecasting_service.py line 428, there was a local import inside a conditional block that shadowed the module-level import:
# Module level (line 9)
from datetime import datetime, date, timedelta, timezone
# Inside function (line 428) - PROBLEM
if day_offset > 0:
from datetime import timedelta, timezone # ← Creates LOCAL variable
current_date = current_date + timedelta(days=day_offset)
# Later in same function (line 447)
processing_time = (datetime.now(timezone.utc) - start_time) # ← Error! timezone not accessible
When Python sees the local import on line 428, it creates a local variable timezone that only exists within that if block. When line 447 tries to use timezone.utc, Python looks for the local variable but can't find it (it's out of scope), resulting in: "cannot access local variable 'timezone' where it is not associated with a value".
Fix: Removed the redundant local import since timezone is already imported at module level:
# Before (BROKEN)
if day_offset > 0:
from datetime import timedelta, timezone
current_date = current_date + timedelta(days=day_offset)
# After (WORKING)
if day_offset > 0:
current_date = current_date + timedelta(days=day_offset)
File: services/forecasting/app/services/forecasting_service.py
Deployment
# Restart forecasting service to apply changes
kubectl -n bakery-ia rollout restart deployment forecasting-service
# Monitor for errors
kubectl -n bakery-ia logs -f deployment/forecasting-service | grep -E "(can't compare|cannot access)"
Related Issues
This same issue may exist in other services. Search for:
# Find services using timezone-aware columns
grep -r "DateTime(timezone=True)" services/*/app/models --include="*.py"
# Find services using datetime.utcnow()
grep -r "datetime\.utcnow()" services/*/app --include="*.py"
References
- Python datetime docs: https://docs.python.org/3/library/datetime.html#aware-and-naive-objects
- SQLAlchemy DateTime: https://docs.sqlalchemy.org/en/20/core/type_basics.html#sqlalchemy.types.DateTime
- PostgreSQL TIMESTAMP WITH TIME ZONE: https://www.postgresql.org/docs/current/datatype-datetime.html