<aside> ℹ️

By L. Zhang et al.

Stanford Univeristy

Marr Prize of ICCV 2023

[Hugging Face] [arXiv] [GitHub]

</aside>

1. Introduction

Motivation

• text-to-image models are limited in the control in spatial composition of the image; and it is difficult for text prompt to express complex layouts, poses, shapes and forms
• want to develop a finer grained spatial control by letting users provide additional images that directly specify their desired image composition.

Challenge

• training data for specific condition may be significantly smaller than general dataset
  • may cause overfitting or forgetting
• designing deeper or more customized neural architectures might be necessary for handling in-the-wild conditioning images with complex shapes and diverse high-level semantics

2. Related Work

2.1. Finetuning Neural Networks

One way to finetune a neural network is to directly continue training it with the additional training data. But this approach can lead to overfitting, mode collapse, and catastrophic for- getting. Extensive research has focused on developing fine- tuning strategies that avoid such issues.

• HyperNetwork
  • originated from NLP, train a small RNN to influence the weights of a larger one.
  • been applied to image generation with GANs.
  • Heathen et al. and Kurumuz implement HyperNetworks for SD to change the artistic style of its output images.
• Adapter
  • widely used in NLP for customizing a pretrained transformer model to other tasks by embedding new module layers into it
  • in vision task, adapters are used for incremental learning and domain adaption
  • often used with CLIP for transferring pretrained models to different tasks
  • vision trainsformers, ViT-Adapter, T2I-Adapter
• Additive Learning
  • circumvents forgetting by freezing the original model weights and adding a small number of new parameters using learned weight masks, pruning, or hard attention
  • side-tuning uses a side branche model to learn extra functionality by linearly blending the outputs of a frozen model and an added network, with a predefined blending weight schedule
• Low-Rank Adaptation (LoRA)
  • prevents catastrophic forgetting by learning the offset of parameters with low rank matrices, based on the observation that many overparameterized models reside in a low intrinsic dimension subspace
• Zero-Initialized Layers
  • used by ControlNet for connecting network blocks
  • works dicsussthe initialization and manipulation of network weights
    • Gaussian initialization of weights can be less risky than initializing with zeros
    • how to scale the initial weight of convolution layers in a diffusion model to improve the training
    • Stability’s model cards also mention the use of zero weights in the neural layers. Manipulating the initial convolution weights is also dicussed in ProGAN, StyleGAN, and Noise2Noise

2.2. Image Diffusion

• Image Diffusion Models
  • first introduced by Sohl Dickstein et al.
  • Latent Diffusion Models (LDM) performs the diffusion steps in the latent image space, reducing computational cost
  • Text-to-image diffusion models achieve SOTA image generation results by encoding text inputs into latent vectors via pretrained language models like CLIP
  • Glide: image generation and editing
  • Disco Diffusion: large-scale implementation of latent diffusion
  • Imagen: directly diffuses pixels
  • Commercial products: DALL-E2 and Midjourney
• Controlling Image Diffusion Models: wide variety of ways to control diffusion models
  • Control directly in image diffusion process
  • Text-guided control: adjust prompts, manipulate CLIP features, and modify corss-attention

2.3. Image-to-Image Translation