Optimizing stock portfolios by minimizing downside volatility using a random forest classifier and reinforcement learning

Abstract

In our study, we evaluate the performance of different computational strategies for mitigating downside risk in equity portfolios - a critical challenge, as standard mean-variance models often struggle with the non-linear, asymmetric nature of market drawdowns. Drawing on data from the Swiss Market Index (SMI), we investigate whether a Random Forest (RF) classifier, a Reinforcement Learning (RL) framework, or a hybrid of the two offers the most robust path toward long-term capital preservation. The research utilizes a multifaceted feature set, incorporating momentum indicators, volatility metrics, and volume-driven sentiment signals. We consider three specific architectures: RF-Only: A classification approach for stock selection paired with a simple equal-weight allocation, RL-Only: Utilizing Proximal Policy Optimization (PPO) to handle both selection and weighting dynamically, and Hybrid RF-RL: An integrated system designed to leverage the predictive strengths of both techniques. The models were trained on a two-decade historical window (2001–2020) and subjected to out-of-sample testing during the period 2021–2025. Our results indicate that the more streamlined RF-only strategy yielded the most favorable outcomes. With total returns of 54% and a notably low downside deviation of 0.0015, it consistently outperformed the more computationally intensive RL and hybrid models. These findings suggest a “complexity paradox” in algorithmic trading: while integrated AI systems are theoretically more powerful, focused optimization methodologies often provide more reliable risk-adjusted results in practice. For researchers and practitioners in portfolio management, this highlights the continued value of robust, interpretable classification models over increasingly opaque end-to-end architectures.