Open-Vocabulary Semantic Segmentation Using Test-Time Distillation

Nir Zabari*, Yedid Hoshen

*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

Abstract

Semantic segmentation is a key computer vision task that has been actively researched for decades. In recent years, supervised methods have reached unprecedented accuracy; however, obtaining pixel-level annotation is very time-consuming and expensive. In this paper, we propose a novel open-vocabulary approach to creating semantic segmentation masks, without the need for training segmentation networks or seeing any segmentation masks. At test time, our method takes as input the image-level labels of the categories present in the image. We utilize a vision-language embedding model to create a rough segmentation map for each class via model interpretability methods and refine the maps using a test-time augmentation technique. The output of this stage provides pixel-level pseudo-labels, which are utilized by single-image segmentation techniques to obtain high-quality output segmentations. Our method is shown quantitatively and qualitatively to outperform methods that use a similar amount of supervision, and to be competitive with weakly-supervised semantic-segmentation techniques.

Original languageAmerican English
Title of host publicationComputer Vision – ECCV 2022 Workshops, Proceedings
EditorsLeonid Karlinsky, Tomer Michaeli, Ko Nishino
PublisherSpringer Science and Business Media Deutschland GmbH
Pages56-72
Number of pages17
ISBN (Print)9783031250620
DOIs
StatePublished - 2023
Event17th European Conference on Computer Vision, ECCV 2022 - Tel Aviv, Israel
Duration: 23 Oct 202227 Oct 2022

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume13802 LNCS
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349

Conference

Conference17th European Conference on Computer Vision, ECCV 2022
Country/TerritoryIsrael
CityTel Aviv
Period23/10/2227/10/22

Bibliographical note

Publisher Copyright:
© 2023, The Author(s), under exclusive license to Springer Nature Switzerland AG.

Keywords

  • Language-based segmentation
  • Open-vocabulary segmentation
  • Semantic segmentation

Fingerprint

Dive into the research topics of 'Open-Vocabulary Semantic Segmentation Using Test-Time Distillation'. Together they form a unique fingerprint.

Cite this