ControlNet

last edited

ControlNet is a neural network architecture that conditions text-to-image diffusion on structural inputs such as edge maps, depth maps, segmentation masks and human pose. A February 2023 paper introduced it. The artist supplies the structure, and the prompt describes the content that the model fills in. ControlNet is now a family of conditioned adapters across many base models.

Architecture

A ControlNet locks the pretrained diffusion model and reuses its encoding layers as a deep backbone for learning the condition. Zero-initialized convolutions connect the added layers to the locked model. This keeps the original weights intact, so training on a small dataset does not damage the base model. The paper tested edges, depth, segmentation and pose on an early open diffusion model.

Structural control

Words specify position poorly, and ControlNet addresses that gap. A pose skeleton allows the same character to be restaged. A depth map allows a render or photo to be restyled while its geometry is preserved. See depth-map best practices for details.

Control strength and start and end step timing set how much freedom the model keeps. Several ControlNets can run together, and structural control can be paired with style transfer, as covered in combined IP-Adapter + ControlNet workflows. The same signals also support img2img-style workflows.

Sources

The original paper and the official repository document the architecture, training process and supported conditioning signals for ControlNet.