Prune Truong

Learning Accurate Dense Correspondences and When to Trust Them

CVPR 2021 - ORAL

Prune Truong             Martin Danelljan             Luc Van Gool             Radu Timofte



intro
Estimating dense correspondences between the query (a) and the reference (b) image. The query is warped according to the resulting flows (c)-(d). The baseline (c) does not estimate an uncertainty map and is therefore unable to filter the inaccurate flows at, for example occluded and homogeneous regions. In contrast, our PDC-Net (d) not only estimates accurate correspondences, but also when to trust them. It predicts a robust uncertainty map that identifies accurate matches and excludes incorrect and unmatched pixels (red).

Abstract

Establishing dense correspondences between a pair of images is an important and general problem. However, dense flow estimation is often inaccurate in the case of large displacements or homogeneous regions. For most applications and down-stream tasks, such as pose estimation, image manipulation, or 3D reconstruction, it is crucial to know when and where to trust the estimated correspondences.

In this work, we aim to estimate a dense flow field relating two images, coupled with a robust pixel-wise confidence map indicating the reliability and accuracy of the prediction. We develop a flexible probabilistic approach that jointly learns the flow prediction and its uncertainty. In particular, we parametrize the predictive distribution as a constrained mixture model, ensuring better modelling of both accurate flow predictions and outliers. Moreover, we develop an architecture and training strategy tailored for robust and generalizable uncertainty prediction in the context of self-supervised training. Our approach obtains state-of-the-art results on multiple challenging geometric matching and optical flow datasets. We further validate the usefulness of our probabilistic confidence estimation for the task of pose estimation. Code and models will be released here.

Visual Results:

  • Aligning images of the Aachen city

Here, we warp the query images toward the reference images. Our approch PDC-Net also predicts a confidence mask along with the dense correspondences. We show the warped query only in the estimated confident regions.

Query
Reference
PDC-Net
  • Texture transfer

Query
Reference
PDC-Net

or with larger appearance transformations...

Query
Reference
PDC-Net
Query
Reference
PDC-Net
  • Aligning images of KITTI-2015

We here show qualitative examples of our approach PDC-Net applied to images of KITTI-2015. We plot directly the estimated flow field for each image pair.

kitti2015
  • Aligning images of MegaDepth

We show qualitative examples of our approach PDC-Net and corresponding non-probabilistic baseline GLU-Net-GOCor*, applied to images of the MegaDepth dataset. In the 3rd and 4th columns, we visualize the query images warped according to the flow fields estimated by the GLU-Net-GOCor* and PDC-Net respectively. PDC-Net also predicts a confidence map, according to which the regions represented in red, are unreliable or inaccurate matching regions. In the last column, we overlay the reference image with the warped query from PDC-Net, in the identified accurate matching regions (lighter color).

mega 1
  • 3D reconstruction of Aachen

Aachen 0 07 3D iso HR copie

How to cite:

@inproceedings{pdcnet,
  author    = {Prune Truong and
               Martin Danelljan and
               Luc Van Gool and
               Radu Timofte},
  title     = {Learning Accurate Dense Correspondences and When to Trust Them},
  booktitle = {{IEEE/CVF} Conference on Computer Vision and Pattern Recognition, {CVPR}},
  year      = {2021},
  url       = {https://arxiv.org/abs/2101.01710}
}
© Prune Truong 2021