Procedural modelling and generative AI for augmented contextual 3D visualisation of planned urban change

Abstract

Street-level visualisations play an important role in urban street redesign by helping stakeholders and non-specialist audiences understand proposed spatial transformations. However, conventional image-editing workflows are often time-consuming, while purely image-based generative AI approaches provide limited control over object position, scale, and spatial relationships. This study presents a novel, highly automated workflow combining georeferenced street-level smartphone imagery, procedural 3D modelling, semantic image segmentation, and controlled generative AI to produce realistic visualisations of planned street transformations. Two-dimensional CAD drawings of planned changes are procedurally converted into simplified 3D models. Semantic masks are rendered from georeferenced camera viewpoints in the 3D model and combined with masks extracted from the street-level smartphone images. The resulting semantic and inpainting masks condition a Stable Diffusion XL-based inpainting model through segmentation and Canny-edge ControlNets. Text prompts control the overall visual appearance, while an IP-Adapter enables selected elements, such as vegetation and tree species, to be refined using reference images. The workflow was applied to two street redesign scenarios. The results show that planned elements can be generated at positions and with dimensions closely corresponding to the planning data, while different visual and seasonal variants can be explored without changing the spatial configuration. The method therefore provides a scalable approach for generating street-level visualisations while preserving geometric control.