Learning an optimisable semantic segmentation map with image conditioned variational autoencoder

Pengcheng Zhuang, Yusuke Sekikawa, Kosuke Hara, Hideo Saito

Research output: Chapter in Book/Report/Conference proceedingConference contribution


Recent semantic segmentation systems have achieved significant improvement by performing pixel-wise training with hierarchical features using deep convolutional neural network models. While the learning process usually requires pixel-level annotated images, it is difficult to get desirable amounts of fine-labeled data and thus the training set size is more likely to be limited, often in thousands. This means that top methods for a dataset can be fine-tuned for a specific situation, making the generalization ability unclear. In real-world applications like self-driving systems, ambiguous region or lack of context information can cause errors in the predicted results. Resolving such ambiguities is crucial for subsequent operations to be performed safely. We are inspired by work from CodeSLAM where optimizable pixel-wise depth representation is learned. We modify the regression method to work on the pixel-wise classification problem. By training a variational auto-encoder network conditioned with a color image, the computed latent space works as a low-dimensional representation of semantic segmentation, which can be efficiently optimized. As a consequence, our model can correct the error or ambiguity of the prediction during the inference phase given useful scene information. We show how this approach works by giving partial scene truth and perform optimization on the latent variable.

Original languageEnglish
Title of host publicationImage Analysis and Processing – ICIAP 2019 - 20th International Conference, Proceedings
EditorsElisa Ricci, Nicu Sebe, Samuel Rota Bulò, Cees Snoek, Oswald Lanz, Stefano Messelodi
PublisherSpringer Verlag
Number of pages11
ISBN (Print)9783030306441
Publication statusPublished - 2019
Event20th International Conference on Image Analysis and Processing, ICIAP 2019 - Trento, Italy
Duration: 2019 Sept 92019 Sept 13

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume11752 LNCS
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349


Conference20th International Conference on Image Analysis and Processing, ICIAP 2019


  • Optimization
  • Semantic segmentation
  • Variational autoencoder

ASJC Scopus subject areas

  • Theoretical Computer Science
  • Computer Science(all)


Dive into the research topics of 'Learning an optimisable semantic segmentation map with image conditioned variational autoencoder'. Together they form a unique fingerprint.

Cite this