TokenCut: Segmenting Objects in Images and Videos with Self-supervised Transformer and Normalized Cut - Multimodal Perception and Sociable Interaction Accéder directement au contenu
Pré-Publication, Document De Travail Année : 2022

TokenCut: Segmenting Objects in Images and Videos with Self-supervised Transformer and Normalized Cut

Résumé

In this paper, we describe a graph-based algorithm that uses the features obtained by a self-supervised transformer to detect and segment salient objects in images and videos. With this approach, the image patches that compose an image or video are organised into a fully connected graph, where the edge between each pair of patches is labeled with a similarity score between patches using features learned by the transformer. Detection and segmentation of salient objects is then formulated as a graph-cut problem and solved using the classical Normalized Cut algorithm. Despite the simplicity of this approach, it achieves state-of-the-art results on several common image and video detection and segmentation tasks. For unsupervised object discovery, this approach outperforms the competing approaches by a margin of 6.1%, 5.7%, and 2.6%, respectively, when tested with the VOC07, VOC12, and COCO20K datasets. For the unsupervised saliency detection task in images, this method improves the score for Intersection over Union (IoU) by 4.4%, 5.6% and 5.2%. When tested with the ECSSD, DUTS, and DUT-OMRON datasets, respectively, compared to current state-of-the-art techniques. This method also achieves competitive results for unsupervised video object segmentation tasks with the DAVIS, SegTV2, and FBMS datasets.
Fichier principal
Vignette du fichier
TokenCut_video.pdf (8.45 Mo) Télécharger le fichier
Origine Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-03765422 , version 1 (31-08-2022)
hal-03765422 , version 2 (09-09-2022)
hal-03765422 , version 3 (30-11-2023)

Identifiants

Citer

Yangtao Wang, Xi Shen, Yuan Yuan, Yuming Du, Maomao Li, et al.. TokenCut: Segmenting Objects in Images and Videos with Self-supervised Transformer and Normalized Cut. 2022. ⟨hal-03765422v1⟩
149 Consultations
69 Téléchargements

Altmetric

Partager

Gmail Mastodon Facebook X LinkedIn More