Anti-Aliasing: How MSAA, SSAA, and TAA Kill the Jaggies

The Core Idea: Aliasing Is a Sampling Theorem Violation

A rendered image is a discrete sampling of a continuous 2D signal — the ideal scene. The Nyquist–Shannon theorem says you must sample at ≥ 2× the highest spatial frequency to reconstruct a signal faithfully. A geometric edge is a step function, whose Fourier spectrum has energy at all frequencies — it is infinite-bandwidth. No finite pixel grid can sample it without aliasing; high frequencies fold down into low ones, producing the visible staircase and the shimmering edge crawl as objects move.

The correct answer for each pixel is not a point sample but an integral: the average of the scene over the pixel's footprint, ideally weighted by a reconstruction filter (box, tent, or Gaussian):

color(px) = ∫∫ scene(x,y) · filter(x−cx, y−cy) dx dy

The pixel should be 30% triangle, 70% background when the edge crosses it at that ratio. Every AA method is a numerical approximation of that integral. The invariant: a pixel's output must be a coverage-weighted blend of the visible surfaces within its area, not a single point's color. SSAA estimates the integral with a dense grid of point samples; MSAA estimates it cheaply on edges; TAA estimates it by accumulating samples across time.

SSAA: Brute-Force Supersampling

Supersampled anti-aliasing (SSAA), the original technique from Ed Catmull and Frank Crow's 1970s work, is the honest Monte-Carlo estimator: render the whole scene at k× resolution, then downsample (box-filter average) to display resolution.

• Render at k·n pixels (e.g. 2×2 grid ⇒ k = 4, at double width and height).
• Shade every sub-sample independently — full pixel shader runs k times per output pixel.
• Resolve: average each k-sample block into one output pixel.

Because it shades at the higher rate, SSAA fixes all aliasing — geometry edges, texture minification, shader specular highlights, alpha-tested foliage. That's its unique strength. Its cost is the killer: for n output pixels and k samples, time is O(n·k) for shading and rasterization plus O(n·k) for the resolve, and space is O(n·k) for the enlarged color and depth buffers. At k = 4 that's 4× the fill rate, 4× the bandwidth, and 4× the VRAM footprint of the framebuffer — which is why real-time engines reserve SSAA (or its ordered-grid variant OGSSAA / sparse-grid SGSSAA) for screenshots and 'downsampling' modes like DLDSR/VSR.

MSAA: Shade Once, Test Coverage Many Times

Multisample anti-aliasing (MSAA) is the key insight that made hardware AA practical (GeForce 3 / DirectX 8 era, ~2001): edge aliasing comes from the coverage/visibility test, not from shading. So decouple them. MSAA keeps k depth/coverage sub-samples per pixel but runs the pixel shader only once per pixel, at the pixel center (or at the centroid of covered samples).

• The rasterizer computes, per pixel, a coverage mask: which of the k sub-sample positions fall inside the triangle. This is a k-bit bitmask, cheap integer work.
• The Z-buffer is expanded to k samples per pixel; depth is tested per sub-sample so edges resolve correctly against other geometry.
• The single shaded color is written to every covered sub-sample. Uncovered samples keep the background.
• At frame end, a resolve pass averages the k sub-samples down to the final pixel.

A pixel straddling an edge ends up with, say, 2 of 4 samples = triangle color and 2 = background ⇒ a 50/50 blend. Smooth edge, one shader invocation. Complexity: shading is O(n) (same as no AA), while coverage and depth are O(n·k) but with tiny constants (bit tests, not shader math); space is O(n·k) for the multisampled render target. That asymmetry — O(n) shading, O(n·k) coverage — is exactly why 4×/8× MSAA on forward-rendered geometry costs far less than 4×/8× SSAA. Modern GPUs add CSAA/EQAA, which store even more coverage samples than color/depth samples, decoupling coverage precision from storage for near-free edge quality.

The Complexity Ledger and the Filter Behind the Resolve

Let n = output pixels, k = sample count, s = shading cost per invocation (often the dominant constant — a modern PBR shader is hundreds of ALU ops and several texture fetches).

• No AA: time Θ(n·s), space Θ(n).
• SSAA k×: time Θ(n·k·s), space Θ(n·k). Both shading and coverage scale by k.
• MSAA k×: time Θ(n·s + n·k·c) where c ≪ s is the per-sample coverage/depth cost, space Θ(n·k). The n·s term is independent of k.
• FXAA: time Θ(n·s + n·f), space Θ(n) — one extra full-screen filter pass, f a small constant.
• TAA: time Θ(n·s + n·r), space Θ(n) for current + Θ(n) for history — reprojection cost r is a couple of texture taps per pixel.

The resolve filter matters more than students expect. A plain box filter (unweighted average of the k samples) is what most hardware does by default, but a box filter is a poor low-pass reconstruction — it lets high-frequency energy leak through (residual aliasing) and softens edges. Higher-quality pipelines resolve with a tent or Gaussian weighting over a small neighborhood, and choose sample positions from a rotated or sparse grid (Poisson-disk / n-rooks pattern) rather than a regular grid, because a regular grid still aliases near-vertical and near-horizontal edges — the very orientations the eye notices most. This is the same reason ray tracers use stratified/jittered sampling: decorrelate the samples so the error becomes high-frequency noise (which we tolerate) instead of low-frequency structure (which we see).

Where MSAA Breaks: Deferred Shading, Alpha Test, and Shader Aliasing

MSAA's O(n) shading shortcut assumes the surface is smooth within a pixel except at its silhouette. That assumption fails in several important regimes, and knowing them is a favorite interview probe:

• Deferred rendering: lighting is computed in a screen-space pass after the G-buffer is filled. To MSAA it, you must store k samples of position/normal/albedo in the G-buffer (huge bandwidth and memory) and shade each edge sample — collapsing MSAA back toward SSAA cost. This is the #1 reason modern AAA engines abandoned MSAA for TAA once they went deferred.
• Alpha-tested geometry (chain-link fences, foliage cutouts): the transparency edge is inside the polygon, not on its silhouette, so MSAA's coverage test misses it entirely. The fix is alpha-to-coverage (A2C): convert the alpha value into a sub-sample coverage mask so MSAA can partially resolve the cutout edge.
• Specular / shader aliasing: a sharp highlight or high-frequency normal map produces sub-pixel color variation that a single shade sample can't capture. MSAA does nothing here; you need SSAA, pre-filtered normals (LEAN/Toksvig mapping), or temporal accumulation.
• Transparency ordering: MSAA doesn't solve order-dependent blending; it interacts awkwardly with sorted alpha blending.

The upshot: MSAA is superb for forward-rendered opaque geometry — mobile GPUs (tile-based, where MSAA sub-samples live in fast on-chip tile memory) still love 4× MSAA — but it is not a universal AA solution.

Post-Process and Temporal AA: FXAA, MLAA, and the TAA Workhorse

When MSAA is too expensive or incompatible (deferred pipelines), two families took over.

Post-process AA operates on the finished color buffer with no extra coverage data. MLAA (Reshetov, Intel, 2009) and its GPU descendant FXAA (Timothy Lottes, NVIDIA, 2011) detect edges from luminance discontinuities and blend across them. Cost is one full-screen pass, O(n) time, O(n) space, provider-agnostic (works on any renderer, even video). The weakness is fundamental: with only final pixels to look at, there's no sub-pixel coverage information — FXAA guesses edge geometry, so it can smooth jaggies but also blurs text and fine detail and can't fix temporal crawl.

Temporal AA (TAA) is the modern default. Each frame the projection matrix is jittered by a sub-pixel offset (typically a low-discrepancy Halton(2,3) sequence), so over N frames you accumulate N different sample positions — effectively 8×, 16×, even 64× supersampling spread across time, at roughly single-sample cost per frame.

• Reproject last frame's history buffer into the current frame using per-pixel motion vectors (velocity buffer).
• Blend current sample with the reprojected history via an exponential moving average, e.g. out = lerp(history, current, α) with α ≈ 0.1.
• Reject/clamp stale history (disoccluded or shading-changed pixels) by clamping it to the AABB of the current pixel's neighborhood in color space — the crucial step that prevents smearing.

TAA gives near-SSAA quality, fixes shader and edge aliasing alike, and is nearly free — which is why it underpins temporal upscalers like DLSS, FSR, TSR, and XeSS. Its cost is ghosting behind moving objects, blur in motion, and instability on thin features when history rejection misfires.

Choosing an AA Method in Practice

The decision follows the rendering architecture and platform more than personal taste:

• Forward + opaque, desktop or mobile: MSAA 4× is the sweet spot — crisp edges, no ghosting, and on tile-based mobile GPUs the sub-samples never leave on-chip memory, making it remarkably cheap. VR strongly prefers MSAA because temporal ghosting is nauseating in a headset.
• Deferred / modern AAA: TAA (or a temporal upscaler) is essentially mandatory; MSAA's G-buffer blowup is untenable. Pair with sharpening (CAS) to counter TAA blur.
• Cheap universal fallback / older hardware: FXAA — one pass, works anywhere, acceptable if a little soft. Often offered alongside MSAA.
• Reference / screenshots / offline: SSAA or path-traced supersampling with hundreds of jittered samples per pixel.

Real systems expose these directly: OpenGL via glTexImage2DMultisample and GL_MULTISAMPLE, Direct3D via DXGI_SAMPLE_DESC and ResolveSubresource, Vulkan via rasterizationSamples in the pipeline multisample state. Unreal and Unity ship TAA/TSR as the default and MSAA as an option for forward paths. The one rule that never changes: you are approximating a coverage integral over each pixel, and every technique is a different budget-vs-accuracy point on that same integral.

Anti-aliasing techniques: cost, quality, and where each wins

Technique	Shading cost	Coverage/edge quality	Kills interior aliasing?	Main weakness
SSAA 4×	4× per pixel	Excellent (true 4×)	Yes (textures, shaders)	4× fill + bandwidth
MSAA 4×	1× per pixel	Excellent on geometry edges	No (shading aliased)	Alpha-test/deferred break it
FXAA	1× + 1 filter pass	Good, blurs some	Post-hoc smoothing only	Blurs text, no subpixel info
TAA	≈1× + reproject	Excellent, ~64× effective	Yes (jitter over time)	Ghosting, motion blur
No AA	1× per pixel	Staircased	No	Jaggies + crawling