ImmerseSketch: Exploring the Use of Generative Models in VR-Based Art Creation Assistance Systems

With Virtual Reality (VR) becoming more commonplace in recent years, many have attempted to design art programs and prototyping applications to leverage VR’s strengths of high immersion and interactivity. VR is capable of immersing users in diverse and fantastical environments, thus stimulating their creativity. However, many existing systems have not yet given users a multitude of unique environments while providing them with tools to create within VR space. In this study, we propose ImmerseSketch, a VR system that generates appro priate panoramic images based on user prompts and transforms them into a 3D environment through depth estimation, allowing users to immerse themselves in the environment for creative work. The system also offers object generation func tionality, enabling users to freely select elements from the environment and use generative AI to generate corresponding 3D object models as reference materials for their creations.