Financial Inputs Prediction with Machine Learning and Covariance Matrix Applications

Abstract

Reliable estimation of the time-varying covariance matrix of asset returns is indispensable for portfolio construction, risk budgeting, and automated advisory services. Conventional estimators-rolling-window sample covariances, EWMA filters, and GARCH families-remain anchored to historical prices and therefore adapt slowly when market conditions pivot. To overcome this latency, we propose an end-to-end, machine-learning-driven framework that forecasts future covariances directly from high-frequency equity data, largely decoupling risk estimation from past observations. Our pipeline ingests heterogeneous stock feeds through a real-time API, applies error-minimising imputation (forward/backward fill, spline, VAR, wavelet, and co-kriging), and standardises returns via empirically selected scaling schemes. The processed features are then fed to a model zoo comprising linear and penalised regressions, tree ensembles (Random Forest, XGBoost, LightGBM, CatBoost), and kernel-based Support Vector Regression. Weekly walk-forward evaluation on a universe of Turkish insurance equities shows that LightGBM and SVR cut out-of-sample covariance prediction error by up to 35 % versus classical benchmarks. We embed the predicted matrices into five allocation engines-Markowitz mean-variance, Black-Litterman, minimum-variance, Risk Parity, and CVaR optimisation-demonstrating that Risk Parity delivers the most consistent variance reduction across 15 stock pairs, while Black-Litterman excels for idiosyncratic combinations such as ANSGR-AKGRT. A granular analysis reveals that day-to-day sign changes in returns create structural breaks that generic regressors miss; augmenting the architecture with a volatility-state classifier and regime-specific learners markedly sharpens turning-point detection. Beyond statistical gains, the framework is production-ready: it is fully implemented in Python, runs on cloud notebooks, and plugs into robo-advisory dashboards. The study thus bridges academic advances in covariance prediction with operational portfolio management, paving the way for broader cross-sector deployment and future research on deep sequential models, transaction-cost awareness, and multi-asset scalability. © 2025 Elsevier B.V., All rights reserved.