Prune Truong

Probabilistic Warp Consistency for Weakly-Supervised Semantic Correspondences

CVPR 2022

Prune Truong             Martin Danelljan            Fisher Yu             Luc Van Gool


We propose Probabilistic Warp Consistency, a weakly-supervised learning objective for semantic matching. Our approach directly supervises the dense matching scores predicted by the network, encoded as a conditional probability distribution. We first construct an image triplet by applying a known warp to one of the images in a pair depicting different instances of the same object class. Our probabilistic learning objectives are then derived using the constraints arising from the resulting image triplet. We further account for occlusion and background clutter present in real image pairs by extending our probabilistic output space with a learnable unmatched state. To supervise it, we design an objective between image pairs depicting different object classes. We validate our method by applying it to four recent semantic matching architectures. Our weakly-supervised approach sets a new state-of-the-art on four challenging semantic matching benchmarks. Lastly, we demonstrate that our objective also brings substantial improvements in the strongly-supervised regime, when combined with keypoint annotations.

Overview of our Probabilistic Warp Consistency approach.
intro reversed

Visual Results

Example predictions of baseline SF-Net (left) and our approach PWarpC-SF-Net (right) on image pairs of the PF-Willow dataset. Correct matches are shown in green.
sfnet pfwillow

How to cite:

  author    = {Prune Truong and
               Martin Danelljan and
               Fisher Yu and 
               Luc Van Gool},
  title     = {Probabilistic Warp Consistency for Weakly-Supervised Semantic Correspondences},
  booktitle = {{IEEE/CVF} Conference on Computer Vision and Pattern Recognition, {CVPR}},
  year = {2022},
  url  = {}
© Prune Truong 2023