II · THE IDEA · ARTIFICIAL INTELLIGENCE
Efficient ViT with Shifted Windows
▶ Listen · narrated
A vision transformer that attends over every pixel pair becomes impractical as resolution rises. Windowed attention keeps cost linear; shifting the windows restores the paths that pure locality would sever.
At a glance
- Core idea
- Self-attention inside non-overlapping windows, shifted across layers
- Cost
- Linear in number of patches, not quadratic
- Hierarchy
- Patch merging builds multi-scale feature maps stage by stage
- Cross-window
- A half-window shift links regions that prior layers kept separate
- V2 focus
- Scaling capacity and resolution with norm, attention and bias changes
Think of a large mural divided into tiles, and a group of restorers who may only discuss colour within one tile at a time. That keeps the meeting small no matter how big the wall is. If they always used the same tile lines, the left half of the mural would never influence the right. So every other day they chalk new lines shifted halfway across. Yesterday’s edge becomes today’s centre, and news from one tile can hop into the next over a few days.
The Swin Transformer does the same with image patches. Self-attention—the “discussion”—runs only inside non-overlapping windows, so cost grows with the number of patches, not with the square of that number. The next layer shifts the windows so patches that were separated can meet. Between larger stages, small groups of patches are merged into coarser tokens, building a pyramid from fine detail to broad layout. Nothing in a single layer sees the whole image at once; depth and the shifting cuts create a global view.
Swin replaces global multi-head self-attention (MSA) with window MSA (W-MSA): the H×W patch grid is partitioned into non-overlapping M×M windows; attention is computed independently per window with complexity O(HW × M²) rather than O((HW)²). Two successive blocks alternate W-MSA and shifted-window MSA (SW-MSA). SW-MSA offsets the partition by (⌊M/2⌋, ⌊M/2⌋). In practice the feature map is cyclic-shifted so windows stay rectangular; an attention mask zeros logits between patches that are adjacent only because of the wrap. Relative position bias B ∈ R^{M²×M²} is added to the attention logits inside each window.
Stages are separated by patch-merging layers that concatenate 2×2 neighbours and apply a linear projection, halving spatial resolution and increasing channel dimension. The stack therefore emits a feature pyramid suitable for dense heads.
Swin V2 targets scale and resolution transfer: residual-post-norm style changes improve training stability at larger capacity; attention logit scaling curbs extreme dot products; relative position bias is parameterised in a continuous log-spaced form so biases can be interpolated when window size or input resolution changes at fine-tuning. The shifted-window geometry itself remains the same; V2 is primarily about making that geometry trainable and portable at higher capacity and resolution.
Limitations follow from the design. Cross-window communication requires depth; a shallow stack still has limited receptive field. Border masks and shifts add implementation complexity. Performance remains sensitive to M relative to object scale. The sources do not claim universality over every later efficient-attention variant; they establish a hierarchical, linear-cost baseline that matches convolutional multi-scale practice.
Look closer
Windows that do not overlap
An image is split into patches, then those patches are grouped into fixed, non-overlapping windows. Multi-head self-attention runs only inside each window. Patches in different windows never exchange information in that layer, so the cost of attention scales with window size rather than with the full image. For ordinary resolutions this turns a quadratic bill into a linear one.
The shift between consecutive layers
If every layer used the same window grid, information would stay trapped inside those blocks. The next layer therefore displaces the partitioning by roughly half a window. New windows straddle the old boundaries, so a patch can attend to neighbours it could not reach one layer earlier. Alternating regular and shifted configurations builds a global receptive field without ever computing full global attention.
Hierarchy by patch merging
Between stages, neighbouring patches are merged—typically by concatenating features from a small spatial block and projecting them—so the token grid grows coarser while channels deepen. Later stages therefore operate on lower-resolution maps with a broader view per token, in the same spirit as feature pyramids in convolutional backbones, and the same windowed attention recipe applies at every scale.
The story
A plain vision transformer treats an image as a sequence of patches and lets every patch attend to every other. That design is simple and expressive, but the attention matrix grows with the square of the number of patches. Double the height and width of the input and the compute and memory for self-attention rise by a factor of sixteen. Dense prediction tasks—detection, segmentation—want high-resolution inputs and multi-scale features; global attention fights both needs at once.
The Swin Transformer keeps the transformer block but changes the geometry of attention. Patches are partitioned into non-overlapping windows of fixed size. Self-attention is computed independently inside each window. Because the window size is constant, the cost per window is constant, and the total cost grows only with the number of windows—that is, linearly with image area. The paper calls this window multi-head self-attention.
Pure local windows would isolate regions permanently. The remedy is a shift of the window partition in the following layer. The grid is offset by half a window, so the new blocks cut across the previous edges. A patch that sat at the corner of a window can now sit near the centre of a shifted window and attend to patches from what used to be three neighbouring blocks. Alternating unshifted and shifted layers propagates information across the whole image while every individual attention operation remains local.
Efficient implementation matters as much as the idea. A naive shift would produce irregular window shapes at the borders. The usual approach is to cycle-shift the feature map, run attention on the resulting regular windows with an attention mask that blocks the artificial adjacencies created by wrapping, then cycle-shift back. The masking keeps the mathematics honest without sacrificing the batch-friendly rectangular layout.
The architecture is hierarchical. Early stages work on fine grids. Between stages, patch merging concatenates features from neighbouring tokens and projects them, reducing spatial resolution and increasing channel depth. Later stages therefore see coarser maps where each token already summarises a larger region. Windowed attention is repeated at every stage, so the same linear-cost recipe applies from local detail to broad context. That multi-scale structure aligns the backbone with the needs of detection and segmentation heads that expect feature pyramids.
Relative position bias inside each window supplies a learnable term keyed to the offset between query and key patches, rather than relying only on absolute embeddings. The bias is shared across windows of the same size and gives the model a consistent notion of left, right, above and below within the local block.
Swin Transformer V2 takes the same skeleton and addresses what breaks when capacity and resolution grow. Layer normalisation is placed differently (post-norm to pre-norm style adjustments in the residual path), attention logit scales are stabilised, and relative position bias is reformulated—using a log-spaced continuous approach—so that biases learned at one resolution can transfer more cleanly when fine-tuning at another. The V2 paper is therefore less about inventing a new attention pattern and more about making the shifted-window hierarchy trainable and transferable at larger scale.
Taken together, the design keeps the transformer’s ability to model long-range dependencies through depth, while the per-layer work stays proportional to image size. The shift is the small geometric trick that stops locality from becoming a hard wall.
Why it mattered then
When the original Swin paper appeared, vision transformers had shown strong classification results but were awkward workhorses for dense prediction. Global attention’s quadratic cost made high-resolution inputs expensive, and a single-scale token grid sat poorly beside the multi-resolution heads used in detection and segmentation. Convolutional backbones still dominated those pipelines because their inductive biases and feature pyramids were already engineered for the task. Shifted-window attention offered a practical compromise: retain self-attention’s flexibility inside manageable blocks, restore cross-block communication with a simple partition shift, and grow a hierarchy by patch merging so the backbone emits multi-scale maps. Linear complexity meant resolutions that mattered for dense tasks became feasible without exotic memory schemes. That combination is why the architecture was taken up quickly as a general-purpose visual backbone rather than only as a classifier trunk.
Why it matters now
Windowed and hierarchical attention remain standard tools whenever someone needs transformer features at high resolution under a fixed compute budget. The same tension—global modelling versus quadratic cost—reappears in video, large medical volumes, and remote-sensing tiles. Swin’s pattern of local attention plus a cheap mechanism for cross-window flow is still a reference design when full global attention is unaffordable. V2’s scaling lessons also travel. Stabilising attention logits, adjusting residual normalisation, and making relative position biases resolution-tolerant are recurring concerns whenever a backbone pre-trained at one size is fine-tuned at another. Understanding shifted windows is less about one named model and more about a reusable way to keep receptive fields growing while attention cost stays linear.
The surprising detail
The cross-window links are not produced by a second attention module or by dilated sampling. They come from literally sliding the partition grid by half a window between layers. The model never computes a global attention matrix; depth plus alternating geometry does the work that a single global layer would do in one shot. The elegance is almost administrative: change where the cuts fall, and information that was sealed off becomes local to a new block.
What is disputed
Reported speed and accuracy gains depend on window size, image resolution, hardware kernels and the downstream task. The papers show linear complexity in the attention pattern itself; wall-clock gains assume implementations that exploit the regular windows and masks efficiently. Exact trade-offs against later sparse or linear-attention variants are outside the sources used here.
Remember this
Non-overlapping windows make attention linear; shifting those windows across layers restores cross-region paths and, with patch merging, a global multi-scale receptive field.
Test yourself
If every Swin layer used the same non-overlapping window grid and never shifted, what would fail for a patch near the centre of a window as depth increases—and what does the half-window shift change about that patch’s possible interactions?
Without shifts, that patch could only ever attend inside its fixed block. Deeper layers would refine features within the window but could not bring in evidence from outside it, so the receptive field would stall at window size. The half-window shift re-partitions the grid so the same patch falls into a new block that straddles old boundaries; it can then attend directly to patches that previously belonged to neighbouring windows. Alternating configurations let information hop across the image over successive layers without any single layer paying for global attention.
Go deeper
- [2103.14030] Swin Transformer: Hierarchical Vision Transformer using Shifted Windows · arxiv.org
- [2111.09883] Swin Transformer V2: Scaling Up Capacity and Resolution · arxiv.org
Image: Original diagram, The Daily Triptych. Licence: Original work. Source.