Image Separation With Side Information: A Connected Auto-Encoders Based Approach

Wei Pu*, Barak Sober, Nathan Daly, Chao Zhou, Zahra Sabetsarvestani, Catherine Higgitt, Ingrid Daubechies, Miguel R.D. Rodrigues

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

Abstract

X-radiography (X-ray imaging) is a widely used imaging technique in art investigation. It can provide information about the condition of a painting as well as insights into an artist's techniques and working methods, often revealing hidden information invisible to the naked eye. X-radiograpy of double-sided paintings results in a mixed X-ray image and this paper deals with the problem of separating this mixed image. Using the visible color images (RGB images) from each side of the painting, we propose a new Neural Network architecture, based upon 'connected' auto-encoders, designed to separate the mixed X-ray image into two simulated X-ray images corresponding to each side. This connected auto-encoders architecture is such that the encoders are based on convolutional learned iterative shrinkage thresholding algorithms (CLISTA) designed using algorithm unrolling techniques, whereas the decoders consist of simple linear convolutional layers; the encoders extract sparse codes from the visible image of the front and rear paintings and mixed X-ray image, whereas the decoders reproduce both the original RGB images and the mixed X-ray image. The learning algorithm operates in a totally self-supervised fashion without requiring a sample set that contains both the mixed X-ray images and the separated ones. The methodology was tested on images from the double-sided wing panels of the Ghent Altarpiece, painted in 1432 by the brothers Hubert and Jan van Eyck. These tests show that the proposed approach outperforms other state-of-the-art X-ray image separation methods for art investigation applications.

Original languageAmerican English
Pages (from-to)2931-2946
Number of pages16
JournalIEEE Transactions on Image Processing
Volume32
DOIs
StatePublished - 2023
Externally publishedYes

Bibliographical note

Publisher Copyright:
© 1992-2012 IEEE.

Keywords

  • Image separation
  • auto-encoders
  • convolutional neural networks
  • deep neural networks
  • image unmixing
  • side information

Fingerprint

Dive into the research topics of 'Image Separation With Side Information: A Connected Auto-Encoders Based Approach'. Together they form a unique fingerprint.

Cite this