Sim2Real: Generative AI to enhance photorealism through domain transfer with GAN and seven-chanel-360°-paired-images dataset

Date published

2024-12-23

Free to read from

Supervisor/s

Journal Title

Journal ISSN

Volume Title

Publisher

MDPI

Department

Type

Article

ISSN

Format

Citation

Bresson M, Xing Y, Guo W. (2023) Sim2Real: Generative AI to enhance photorealism through domain transfer with GAN and seven-chanel-360°-paired-images dataset. Sensors, Volume 24, Issue 1, December 2023, Article number 94.

Abstract

This work aims at providing a solution to data scarcity by allowing end users to generate new images while carefully controlling building shapes and environments. While Generative Adversarial Networks (GANs) are the most common network type for image generation tasks, recent studies have only focused on RGB-to-RGB domain transfer tasks. This study utilises a state-of-the-art GAN network for domain transfer that effectively transforms a multi-channel image from a 3D scene into a photorealistic image. It relies on a custom dataset that pairs 360° images from a simulated domain with corresponding 360° street views. The simulated domain includes depth, segmentation map, and surface normal (stored in seven-channel images), while the target domain is composed of photos from Paris. Samples come in pairs thanks to careful virtual camera positioning. To enhance the simulated images into photorealistic views, the generator is designed to preserve semantic information throughout the layers. The study concludes with photorealistic-generated samples from the city of Paris, along with strategies to further refine model performance. The output samples are realistic enough to be used to train and improve future AI models.

Description

Software Description

Software Language

Github

Keywords

generative AI, GAN, simulation, domain transfer, 360° images, perspective augmentation, photorealism, dataset

DOI

Rights

Attribution 4.0 International

Relationships

Relationships

Supplements

Funder/s