Generating Stunning High-Resolution Images: Introducing FAM Diffusion

In the rapidly evolving field of artificial intelligence, diffusion models have emerged as powerful tools for generating high-quality images. However, a significant hurdle has been their inability to produce images at resolutions higher than those they were trained on without encountering issues like repetitive patterns or distorted structures. Retraining these models for higher resolutions is often computationally prohibitive. This blog post delves into a novel solution, FAM Diffusion, a method designed to enable pre-existing diffusion models to generate superior high-resolution images with flexible test-time resolutions, all without requiring additional training. We will explore the background of this challenge, the innovative methodology behind FAM Diffusion, the results, both qualitatively and quantitatively, discussing its significance in the landscape of image generation.

The Challenge of High-Resolution Image Generation: Background and Previous Efforts

Diffusion models excel at creating images by progressively removing noise from an initial random signal, guided by a learned process. While effective at their native training resolutions (e.g., 512×512 or 1024×1024 pixels), directly attempting to generate images at, for instance, 4K resolution often leads to severe object repetition and unrealistic local patterns.

Previous attempts to overcome this limitation can be broadly categorized:

Figure 1. Comparison at 3x higher resolution than native, i.e. 3072 x 3072px between our approach and prior state-of-the-art. Notice that prior work suffers from repetitive patterns, unnatural texture and structure.

FAM Diffusion aims to take the best of these worlds: a single-pass generation strategy for low latency, while effectively leveraging the native resolution generation to guide the high-resolution process, and critically, addressing both global and local consistency issues often overlooked. See Figure 1 for a visualisation of some of these problems and the result of our approach, which largely alleviates these challenges.

Our approach, FAM Diffusion: A Two-Pronged Approach to High-Resolution Synthesis

FAM Diffusion introduces two simple yet effective modules - Frequency Modulation (FM) and Attention Modulation (AM) - that can be seamlessly integrated into any latent diffusion model without additional training or architectural changes. The process generally follows a test-time diffuse-denoise strategy: an image is first generated at the model's native resolution, then upsampled, noise is added, and finally, a denoising process generates the high-resolution output.

1. Frequency Modulation (FM) for Global Consistency

When upscaling images, maintaining global structural integrity is paramount. Simply steering the high-resolution denoising process with an upsampled low-resolution image (as done by "skip residuals" in some prior work) can indiscriminately force features, potentially harming high-frequency details.

The FM module uses the Fourier domain, where global structure is represented by low frequencies and finer details by high frequencies. During the high-resolution denoising stage, the FM module selectively conditions the low-frequency components of the image being generated using the low-frequency information from the diffused, upsampled native-resolution image. Crucially, it allows the denoising process full control over the high-frequency components, enabling the generation of new, sharp details.

In essence, high frequencies from the ongoing denoising are combined with low frequencies from the guidance image. In the time domain, this operation is equivalent to adding a low-frequency update to the denoised latent, directed towards the diffused latent, effectively giving the UNet a global receptive field without architectural changes.

2. Attention Modulation (AM) for Local Texture Consistency

While FM effectively handles global structure and prevents issues like object duplication, inconsistencies in local textures can still arise. For example, fur texture might incorrectly appear on a shirt collar, or facial features might be distorted compared to the native resolution image. We hypothesize this stems from incorrect attention maps during high-resolution denoising.

To address this, the AM module leverages attention maps from the denoising process at the native resolution to guide the attention maps at the high resolution. Since native resolution attention maps encode semantic relationships between image regions, they can regularize the high-resolution denoising towards generating consistent finer textures.

The self-attention mechanism in diffusion models computes an attention matrix M from query (Q), key (K), and value (V) projections of an input tensor. AM modifies the attention matrix of the high-resolution denoising process at specific UNet layers (primarily up-blocks known to preserve layout information). This helps preserve local structures and semantic consistency.

Figure 2. Overview of the FAM diffusion. (a) We first generate an image at native resolution, followed by a test-time diffuse-denoise process. We incorporate our Frequency Modulation module and Attention Modulation during highres denoising to control global structure and fine local texture, respectively. (b) Details of the Frequency Modulation, where we use the Fourier domain to selectively condition low-frequency components during high-res denoising while leaving high-frequency components fully controllable. (c) Details of Attention Modulation, where attention maps from the native image denoising are used to correct the highres denoising.

Results: State-of-the-Art Performance and Efficiency

Across various scaling factors and base models (including SDXL and HiDiffusion), FAM Diffusion consistently achieved state-of-the-art performance on. Crucially, FAM Diffusion achieves these quality gains with only negligible latency overheads compared to direct inference at the target resolution. For SDXL, this was approximately 0.2, 0.3, and 0.7 minutes extra for 2×, 3×, and 4× scales, respectively. This is a stark contrast to methods like DemoFusion, which added significantly more latency (e.g., 14.2 minutes more than direct SDXL inference at 4×). FAM Diffusion was also shown to enhance the performance of single-pass methods like HiDiffusion while maintaining fast generation.

Figure 3. Qualitive comparison with prior state-of-the-art approaches. Note, that FAMFusion results in consistent and realistic structure, avoiding over sharpening and spurious artifacts.

Discussion: FAM Diffusion – toward highly accurate training free high resolution image generation

While previous methods often focused more on global consistency, sometimes neglecting local texture artifacts or introducing significant computational costs. FAM Diffusion's FM module directly targets global consistency by leveraging the robust low-frequency information from the native resolution image in the Fourier domain. The AM module is a specifically addresses the often-overlooked problem of inconsistent local textures by using attention maps from the native resolution to guide the high-resolution generation.

Additionally, unlike methods requiring retraining or complex architectural modifications, FAM Diffusion is training-free and integrates seamlessly into existing latent diffusion models. Its single-pass nature and the computational efficiency of its modules result in negligible latency overhead, making it a practical solution for generating very high-resolution images. This contrasts sharply with patch-based methods that suffer from high latency due to redundant forward passes.

Conclusion: A New State-of-the-Art in High-Resolution Image Generation

Generating high-resolution images with diffusion models without sacrificing quality or incurring prohibitive costs has been a significant challenge. FAM Diffusion presents a compelling, training-free solution by introducing two key innovations: Frequency Modulation (FM) for robust global structure consistency and Attention Modulation (AM) for coherent local texture patterns.

This method allows existing, powerful diffusion models to operate at flexible, much higher resolutions than they were trained for, producing images with superior qualitative and quantitative results compared to previous state-of-the-art techniques. Its ability to achieve this with negligible latency overhead and without any need for retraining or architectural modifications makes FAM Diffusion a highly practical and impactful advancement in the field of generative AI. It sets a new standard for high-resolution image generation, opening up new possibilities for applications requiring large-scale, detailed visual content.

Link to the paper

https://arxiv.org/pdf/2411.18552

View More

Close