Article Dans Une Revue Reinforcement Learning Journal Année : 2025

Multiple-Frequencies Population-Based Training

Auguste Lehuger
  • Fonction : Auteur
  • PersonId : 1580655
Marin Toromanoff
  • Fonction : Auteur
  • PersonId : 1291396
Valentin Charraut
  • Fonction : Auteur
  • PersonId : 1580656
Thibault Buhet
  • Fonction : Auteur
  • PersonId : 1580657
Fabien Moutarde

Résumé

Reinforcement Learning's high sensitivity to hyperparameters is a source of instability and inefficiency, creating significant challenges for practitioners. Hyperparameter Optimization (HPO) algorithms have been developed to address this issue, among them Population-Based Training (PBT) stands out for its ability to generate hyperparameters schedules instead of fixed configurations. PBT trains a population of agents, each with its own hyperparameters, frequently ranking them and replacing the worst performers with mutations of the best agents. These intermediate selection steps can cause PBT to focus on short-term improvements, leading it to get stuck in local optima and eventually fall behind vanilla Random Search over longer timescales. This paper studies how this greediness issue is connected to the choice of evolution frequency, the rate at which the selection is done. We propose Multiple-Frequencies Population-Based Training (MF-PBT), a novel HPO algorithm that addresses greediness by employing sub-populations, each evolving at distinct frequencies. MF-PBT introduces a migration process to transfer information between sub-populations, with an asymmetric design to balance short and long-term optimization.

Contribution(s)

1. We investigate the impact of evolution frequency on PBT and its connection to greediness.

Context: PBT (Jaderberg et al., 2017) introduces a parameter, denoted t ready , which controls the evolution frequency of its genetic process. Previous extensions of PBT (Parker-

Fichier principal
Vignette du fichier
RLJ_RLC_2025_274.pdf (8.56 Mo) Télécharger le fichier
Origine Fichiers éditeurs autorisés sur une archive ouverte
Licence

Dates et versions

hal-05248147 , version 1 (10-09-2025)

Licence

Identifiants

  • HAL Id : hal-05248147 , version 1

Citer

Wael Doulazmi, Auguste Lehuger, Marin Toromanoff, Valentin Charraut, Thibault Buhet, et al.. Multiple-Frequencies Population-Based Training. Reinforcement Learning Journal, 2025. ⟨hal-05248147⟩
77 Consultations
171 Téléchargements

Partager

  • More