All Together Now! The Benefits of Adaptively Fusing Pre-trained Deep Representations

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

Abstract

Pre-trained deep neural networks, powerful models trained on large datasets, have become a popular tool in computer vision for transfer learning. However, the standard approach of using a single network potentially misses out on valuable information contained in other readily available models. In this work, we study the Mixture of Experts (MoE) approach for adaptively fusing multiple pre-trained models for each individual input image. In particular, we explore how far we can get by combining diverse pre-trained representations in a customized way that maximizes their potential in a lightweight framework. Our approach is motivated by an empirical study of the predictions made by popular pre-trained nets across various datasets, finding that both performance and agreement between models vary across datasets. We further propose a miniature CNN gating mechanism operating on a thumbnail version of the input image, and show this is enough to guide a good fusion. Finally, we explore a multi-modal blend of visual and natural-language representations, using a label-space embedding to inject pre-trained word-vectors. Across multiple datasets, we demonstrate that an adaptive fusion of pre-trained models can obtain favorable results.

Original languageAmerican English
Title of host publicationICPRAM 2019 - Proceedings of the 8th International Conference on Pattern Recognition Applications and Methods, Volume 1
EditorsMaria De Marsico, Gabriella Sanniti di Baja, Ana L.N. Fred
PublisherScience and Technology Publications, Lda
Pages135-144
Number of pages10
ISBN (Print)9789897583513
DOIs
StatePublished - 2019
Externally publishedYes
Event8th International Conference on Pattern Recognition Applications and Methods , ICPRAM 2019 - Prague, Czech Republic
Duration: 19 Feb 201921 Feb 2019

Publication series

NameInternational Conference on Pattern Recognition Applications and Methods
Volume1
ISSN (Electronic)2184-4313

Conference

Conference8th International Conference on Pattern Recognition Applications and Methods , ICPRAM 2019
Country/TerritoryCzech Republic
CityPrague
Period19/02/1921/02/19

Bibliographical note

Publisher Copyright:
© 2019 by SCITEPRESS - Science and Technology Publications, Lda. All rights reserved.

Keywords

  • Deep Learning
  • Fusion

Fingerprint

Dive into the research topics of 'All Together Now! The Benefits of Adaptively Fusing Pre-trained Deep Representations'. Together they form a unique fingerprint.

Cite this