Prune Truong

GOCor: Bringing Globally Optimized Correspondence Volumes into Your Neural Network

NeurIPS 2020

Prune Truong *             Martin Danelljan *             Luc Van Gool             Radu Timofte



corr diff iteration
Visualization of the matching confidences computed between the indicated location (green) in the reference image and all locations of either the reference image itself or the query image. The feature correlation generates undistinctive and inaccurate confidences due to similar regions and repetitive patterns. In contrast, our GOCor predicts a distinct high-confidence value at the correct location.

Abstract

The feature correlation layer serves as a key neural network module in numerous computer vision problems that involve dense correspondences between image pairs. It predicts a correspondence volume by evaluating dense scalar products between feature vectors extracted from pairs of locations in two images. However, this point-to-point feature comparison is insufficient when disambiguating multiple similar regions in an image, severely affecting the performance of the end task. We propose GOCor, a fully differentiable dense matching module, acting as a direct replacement to the feature correlation layer. The correspondence volume generated by our module is the result of an internal optimization procedure that explicitly accounts for similar regions in the scene. Moreover, our approach is capable of effectively learning spatial matching priors to resolve further matching ambiguities. We analyze our GOCor module in extensive ablative experiments. When integrated into state-of-the-art networks, our approach significantly outperforms the feature correlation layer for the tasks of geometric matching, optical flow, and dense semantic matching. The code and trained models will be made available here.

method diagram
Schematic overview of the feature correlation layer (a) and our GOCor module (b).

Visual Results:

  • Aligning images of the MegaDepth dataset

Here, we warp the query image toward the reference image. We compare our GLU-Net-GOCor with original network GLU-Net.

megadepth glunet
  • Aligning images of the ETH3D dataset

Here, we warp the query image toward the reference image. We compare our GLU-Net-GOCor with original network GLU-Net.

ETH3D visual glunet
  • Aligning images of the KITTI dataset (optical flow dataset)

kitti pwcnet
  • Aligning images of the Sintel dataset (optical flow dataset)

sintel pwcnet

How to cite:

@inproceedings{GOCor_Truong_2020,
      title = {{GOCor}: Bringing Globally Optimized Correspondence Volumes into Your Neural Network},
      author    = {Prune Truong 
                   and Martin Danelljan 
                   and Luc Van Gool 
                   and Radu Timofte},
      year = {2020},
      booktitle = {Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information
                   Processing Systems 2020, {NeurIPS} 2020}
}
© Prune Truong 2021