TexGen: Text-Guided 3D Texture Generation with Multi-view Sampling and Resampling

Overview of our proposed method, where AGVS and T\(\mathbf{^2}\)GR denote Attention-Guided View Sampling and Text&Texture-Guided Resampling, respectively. First of all, we sample \(N\) viewpoints across the objects. As shown in (a), our texture sampling strategy is an interleaved process of texture generation and diffusion denoising. Specifically, our texture sampling process is structured into \(T\) desnoising steps of diffusion process, and a complete RGB texture map (\(\hat{U}_{t}^N\)) is generated at the end of each step. As shown in (b), at denoising step \(t\), each AGVS module receives noisy latent features \(x_{t}^i\) as input to sample an image and produce a partial texture map \(\hat{U}_{t}^i\), along with noise estimation \(\epsilon_\theta(x_t^i)\). The generated \(\hat{U}_{t}^i\) serves as guidance for sampling the subsequent view. Subsequently, a complete texture map \(\hat{U}_{t}^N\) is employed to refine the noise estimation of each view within T\(^2\)GR modules, facilitating the prediction of noisy features for the ensuing denoising step (\(x_{t-1}^{1...N}\)).

Details of denoising for view \(i+1\) at step \(t\). The AGVS module is designed to generate denoised observation \(\hat{x}_0^{i+1}(x_t^{i+1})\) which will be assembled onto UV space to form intermediate texture \(\hat{U}_{t}^{i+1}\). The attention guidance is omitted in the figure for simplification. After iterating over all sampled views starting from \(i=1\) to \(N\), we obtain a complete texture map \(\hat{U}_{t}^N\) for each denoising step. Conditioned on the current aggragated texture map, the T\(^2\)GR module will update the noise estimation \(\epsilon_\theta(x_t^i)\) with the multi-conditioned classifier-free guidance (CFG) to calculate the noisy latent feature \(x_{t-1}^{i+1}\) of the next denoising step.