SyncGAN: Synchronize the Latent Space of Cross-modal Generative Networks

Generative adversarial network (GAN) has achieved impressive success on cross-domain generation, but it faces difficulty in cross-modal generation due to the lack of a common distribution between heterogeneous data. Most existing methods of conditional based cross-modal GANs adopt the strategy of one-directional transfer and have achieved preliminary success on text-to-image transfer. Instead of learning the transfer between different modalities, we aim to learn a synchronous latent space representing the cross-modal common concept. A novel network component named synchronizer is proposed in this work to judge whether the paired data is synchronous/ corresponding or not, which can constrain the latent space of generators in the GANs. Our GAN model, named as SyncGAN, can successfully generate synchronous data (e.g., a pair of image and sound) from identical random noise. For transforming data from one modality to another, we recover the latent code by inverting the mappings of a generator and use it to generate data of different modality. In addition, the proposed model can achieve semi-supervised learning, which makes our model more flexible for practical applications.

BibTeX

@INPROCEEDINGS{8486594,
  author={W. Chen and C. Chen and M. Hu},
  booktitle={2018 IEEE International Conference on Multimedia and Expo (ICME)},
  title={Syncgan: Synchronize the Latent Spaces of Cross-Modal Generative Adversarial Networks},
  year={2018},
  volume={},
  number={},
  pages={1-6},
  abstract={Generative adversarial network (GAN) has achieved impressive success on cross-domain generation, but it
  faces difficulty in cross-modal generation due to the lack of a common distribution between heterogeneous data. Most
  existing methods of conditional based cross-modal GANs adopt the strategy of one-directional transfer and have
  achieved preliminary success on text-to-image transfer. Instead of learning the transfer between different
  modalities, we aim to learn a synchronous latent space representing the cross-modal common concept. A novel network
  component named synchronizer is proposed in this work to judge whether the paired data is synchronous/corresponding
  or not, which can constrain the latent space of generators in the GANs. Our GAN model, named as SyncGAN, can
  successfully generate synchronous data (e.g., a pair of image and sound) from identical random noise. For
  transforming data from one modality to another, we recover the latent code by inverting the mappings of a generator
  and use it to generate data of different modality. In addition, the proposed model can achieve semi-supervised
  learning, which makes our model more flexible for practical applications.},
  keywords={image processing;learning (artificial intelligence);latent spaces;cross-modal generative adversarial
  networks;generative adversarial network;cross-domain generation;cross-modal generation;heterogeneous
  data;conditional based cross-modal GANs;one-directional transfer;text-to-image transfer;synchronous latent
  space;cross-modal common concept;novel network component named synchronizer;paired data;GAN model;synchronous
  data;latent code;Synchronization;Gallium nitride;Generators;Data models;Training;Generative adversarial
  networks;Training data;Generative adversarial network (GAN);cross-domain;cross-modal;SyncGAN},
  doi={10.1109/ICME.2018.8486594},
  ISSN={1945-788X},
  month={July},}
}

SyncGAN: Synchronize the Latent Space of Cross-modal Generative Networks

BibTeX

Paper Download