.svg){kind=small}
.svg)
.svg)
In-House ML for European Energy Price ForecastingBuilding a custom, transparent machine learning system that consistently outperforms expensive, "black-box" commercial market forecasts.
Beating the Market: ML Models That Outperform Commercial Energy Price Forecasts
In European energy markets, accurate price forecasting is everything. A few percentage points of improved accuracy on day-ahead spot prices can translate into millions in better trading decisions. Yet most energy companies depend on expensive commercial forecast providers with black-box solutions, limited customization and no transparency on methodology.
At Acciona Energia, one of Europe's leading renewable energy producers, I took on the challenge of building something better, from scratch.
I designed and deployed a full in-house ML forecasting system covering the entire lifecycle: automated data ingestion from multiple market and grid sources, domain-specific feature engineering, model training with state-of-the-art time series algorithms (TiDE, Temporal Fusion Transformer, etc.), daily prediction serving, and continuous performance monitoring.
But technical accuracy alone isn't enough. Production reliability is what separates a good model from a real business tool. The pipeline runs on a robust MLOps infrastructure: Docker and Kubernetes for containerization and orchestration, Airflow for scheduling, MLflow for experiment tracking and model registry, Hydra for configuration management, and GitHub Actions for CI/CD. The architecture is deployment-agnostic, running both on-premise and on cloud environments.
The result: our in-house models consistently outperformed leading commercial forecasts on day-ahead electricity spot prices, delivering superior accuracy at a fraction of the cost, while giving the trading team full control and transparency over every prediction.
Building on this success, the approach was extended to three additional forecasting use cases: grid tension days, long-term spot prices, and fast reserve pricing, each with direct business impact on trading and risk management.
Acciona Energia, a major renewable energy producer, needed accurate day-ahead electricity spot price forecasts to optimize its trading decisions on European energy markets. The existing approach relied on expensive commercial forecast providers, which offered limited customization, black-box models, and insufficient accuracy, directly impacting trading margins and risk management.
I designed and deployed an end-to-end ML forecasting pipeline for day-ahead spot prices, built entirely in-house. The system covers the full ML lifecycle: automated data acquisition from multiple market sources, feature engineering tailored to energy market dynamics, model training using advanced time series algorithms (Prophet, TiDE, TFT, Scikit-Learn), prediction serving, and continuous monitoring.
The solution leverages a robust MLOps stack: containerization (Docker, Kubernetes), orchestration (Airflow), experiment tracking and model registry (MLflow), configuration management (Hydra), and CI/CD automation (GitHub Actions). The architecture supports both on-premise and cloud deployment, ensuring flexibility and scalability. Everything is documented and designed for team collaboration via Confluence, Jira, and GitHub.
Forecast accuracy: outperforming commercial providers where it matters most
The in-house deep learning model was rigorously benchmarked against two leading commercial energy price forecast providers over the same evaluation period.
The in-house model achieved the best performance on the metrics most critical for energy trading:
- RMSE: 22.98 vs. 24.19 (Commercial A) and 25.77 (Commercial B) => a 5% to 11% reduction in overall prediction error. This is the key metric for trading, as it heavily penalizes large forecast misses that lead to costly decisions.
- SMAPE: 71.75 vs. 72.49 and 80.58 => best relative accuracy across all providers.
- R²: 0.623 vs. 0.582 and 0.526 => the in-house model explains 7% to 18% more price variance, giving traders a more reliable signal.
The model also outperforms Commercial B across every single metric, and matches or exceeds Commercial A on the metrics with the highest business impact (RMSE, SMAPE, R²).
Beyond accuracy: operational and strategic impact
- Cost reduction: replaced dependency on commercial forecast subscriptions costing several thousands of euros per year with a fully owned, maintainable solution.
- Transparency: traders now have full visibility into model logic, features, and confidence levels, bringing a decisive advantage over black-box commercial outputs.
- Scalability: the proven pipeline architecture was replicated across three additional forecasting use cases (grid tension days, long-term spot prices, fast reserve pricing), multiplying business value with minimal incremental effort.
- Production reliability: fully automated daily runs with monitoring, ensuring consistent delivery without manual intervention.
.svg){kind=icon}