Publication | IEEE International Conference on Computer Vision and Pattern Recognition (CVPR) 2022
CLIP-Forge
Towards Zero-Shot Text-to-Shape Generation
We propose a zero-shot text-to-shape generation method named CLIP-Forge. Without training on any shape-text pairing labels, our method generates meaningful shapes that correctly reflect the common name, (sub-)category, and semantic attribute information.
Our approach is among the pioneering techniques that can convert text to 3D shapes without the need for costly inference time optimization. Furthermore, it enables the production of multiple shapes for a given text.
This paper was presented at the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR)
The dataset for this paper is available at Autodesk AI Lab on Github.
Download publicationAbstract
CLIP-Forge: Towards Zero-Shot Text-to-Shape Generation
Aditya Sanghi, Hang Chu, Joseph G. Lambourne, Ye Wang, Chin-Yi Cheng, Marco Fumero, Kamal Rahimi Malekshan
IEEE International Conference on Computer Vision and Pattern Recognition (CVPR) 2022
Generating shapes using natural language can enable new ways of imagining and creating the things around us. While significant recent progress has been made in text-to-image generation, text-to-shape generation remains a challenging problem due to the unavailability of paired text and shape data at a large scale. We present a simple yet effective method for zero-shot text-to-shape generation that circumvents such data scarcity. Our proposed method, named CLIP-Forge, is based on a two-stage training process, which only depends on an unlabelled shape dataset and a pre-trained image-text network such as CLIP. Our method has the benefits of avoiding expensive inference time optimization, as well as the ability to generate multiple shapes for a given text. We not only demonstrate promising zero-shot generalization of the CLIP-Forge model qualitatively and quantitatively, but also provide extensive comparative evaluations to better understand its behavior.
Related Resources
2023
Leveraging Graph Neural Networks for Graph Regression and Effective Enumeration ReductionGraph-based framework represents aspects of optimal thermal management…
2023
Extracting Design Knowledge from Optimization Data: Enhancing Engineering Design in Fluid Based Thermal Management SystemsExtracting knowledge from optimization data in multi-split thermal…
2023
WorldSmith: Iterative and Expressive Prompting for World Building with a Generative AIUsing multi-modal generative AI to quickly and iteratively visualize…
2022
UNIST: Unpaired Neural Implicit Shape Translation NetworkWe introduce UNIST, the first deep neural implicit modelfor…
Get in touch
Something pique your interest? Get in touch if you’d like to learn more about Autodesk Research, our projects, people, and potential collaboration opportunities.
Contact us