Multi-scale Context Intertwining for Semantic Segmentation

Di Lin, Yuanfeng Ji, Dani Lischinski, Daniel Cohen-Or, Hui Huang*

*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

24 Scopus citations


Accurate semantic image segmentation requires the joint consideration of local appearance, semantic information, and global scene context. In today’s age of pre-trained deep networks and their powerful convolutional features, state-of-the-art semantic segmentation approaches differ mostly in how they choose to combine together these different kinds of information. In this work, we propose a novel scheme for aggregating features from different scales, which we refer to as Multi-Scale Context Intertwining (MSCI). In contrast to previous approaches, which typically propagate information between scales in a one-directional manner, we merge pairs of feature maps in a bidirectional and recurrent fashion, via connections between two LSTM chains. By training the parameters of the LSTM units on the segmentation task, the above approach learns how to extract powerful and effective features for pixel-level semantic segmentation, which are then combined hierarchically. Furthermore, rather than using fixed information propagation routes, we subdivide images into super-pixels, and use the spatial relationship between them in order to perform image-adapted context aggregation. Our extensive evaluation on public benchmarks indicates that all of the aforementioned components of our approach increase the effectiveness of information propagation throughout the network, and significantly improve its eventual segmentation accuracy.

Original languageAmerican English
Title of host publicationComputer Vision – ECCV 2018 - 15th European Conference, 2018, Proceedings
EditorsVittorio Ferrari, Cristian Sminchisescu, Martial Hebert, Yair Weiss
PublisherSpringer Verlag
Number of pages17
ISBN (Print)9783030012182
StatePublished - 2018
Event15th European Conference on Computer Vision, ECCV 2018 - Munich, Germany
Duration: 8 Sep 201814 Sep 2018

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume11207 LNCS
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349


Conference15th European Conference on Computer Vision, ECCV 2018

Bibliographical note

Publisher Copyright:
© 2018, Springer Nature Switzerland AG.


  • Convolutional neural network
  • Deep learning
  • Long short-term memory
  • Semantic segmentation


Dive into the research topics of 'Multi-scale Context Intertwining for Semantic Segmentation'. Together they form a unique fingerprint.

Cite this