LENS: Localization enhanced by NeRF synthesis - CAO et robotique (CAOR) Accéder directement au contenu
Communication Dans Un Congrès Année : 2021

LENS: Localization enhanced by NeRF synthesis

Arthur Moreau
  • Fonction : Auteur
  • PersonId : 1122704
Bogdan Stanciulescu
Arnaud de La Fortelle

Résumé

Neural Radiance Fields (NeRF) have recently demonstrated photo-realistic results for the task of novel view synthesis. In this paper, we propose to apply novel view synthesis to the robot relocalization problem: we demonstrate improvement of camera pose regression thanks to an additional synthetic dataset rendered by the NeRF class of algorithm. To avoid spawning novel views in irrelevant places we selected virtual camera locations from NeRF internal representation of the 3D geometry of the scene. We further improved localization accuracy of pose regressors using synthesized realistic and geometry consistent images as data augmentation during training. At the time of publication, our approach improved state of the art with a 60% lower error on Cambridge Landmarks and 7-scenes datasets. Hence, the resulting accuracy becomes comparable to structure-based methods, without any architecture modification or domain adaptation constraints. Since our method allows almost infinite generation of training data, we investigated limitations of camera pose regression depending on size and distribution of data used for training on public benchmarks. We concluded that pose regression accuracy is mostly bounded by relatively small and biased datasets rather than capacity of the pose regression model to solve the localization task.

Dates et versions

hal-03520671 , version 1 (11-01-2022)

Identifiants

Citer

Arthur Moreau, Nathan Piasco, Dzmitry Tsishkou, Bogdan Stanciulescu, Arnaud de La Fortelle. LENS: Localization enhanced by NeRF synthesis. 5th Annual Conference on Robot Learning, Nov 2021, London, United Kingdom. ⟨hal-03520671⟩
57 Consultations
0 Téléchargements

Altmetric

Partager

Gmail Facebook X LinkedIn More