Visual comparison

Pixel comparison of settled states and per-frame divergence of filmstrips — and why pixels are never critical by default.

On this page
  1. Settled screenshots
  2. Frame divergence
  3. Why pixels are a warning
  4. Per engine

Animation data says how things moved. Pixels say what the user saw at the end, and when during the motion the two versions looked different. MotionPR uses both, and keeps them apart.

Settled screenshots

Every capture checkpoint takes a PNG of the settled state at the end of its window (turn it off per checkpoint with screenshot: false). screenshot checkpoints take one without animation data. The two versions are compared with pixelmatch:

StatusMeaning
identicalNo differing pixel (or the very same asset)
within-thresholdSome pixels differ, at most visual.maxDiffRatio of them (0.2%)
changedMore than visual.maxDiffRatio differ: a visual difference
not-comparableA screenshot is missing on one side, unreadable, or the sizes differ
skippedcomparison.visual.enabled is false

A checkpoint with differing pixels gets a diff image: differing pixels in peach (blue where the candidate is darker), anti-aliased pixels in violet, the rest faded. The report shows it in its Pixel diff mode with the counts and the threshold.

OptionDefaultEffect
comparison.visual.pixelThreshold0.1Per-pixel colour threshold (0 strict … 1 lenient)
comparison.visual.maxDiffRatio0.002Share of differing pixels above which it is changed
comparison.visual.includeAntialiasingfalseCount anti-aliased pixels as differences
comparison.visual.ignoreRegions[]Viewport rectangles { x, y, width, height } excluded
comparison.visual.frameDivergencetrueCompare filmstrip frames pairwise

Frame divergence

With PNG frames on both sides, MotionPR compares time-aligned filmstrip frames and stores the share of differing pixels per frame. The report draws it above the timeline as the frame divergence strip: you can see when the versions start to look different, and when they meet again. JPEG frames (capture.frames.format: "jpeg") skip this, because lossy compression would make every frame differ.

Why pixels are a warning

Fonts and anti-aliasing render differently on macOS, Windows and Linux, and between browser builds. A settled screenshot that differs by a few hundred pixels is often the platform, not your change. So the visual rule is a warning by default and never critical, while timing and easing differences, which do not depend on rendering, can be. When baseline and candidate come from different platforms or browser versions, the comparison adds PLATFORM_DIFFERS or BROWSER_VERSION_DIFFERS.

To keep visual comparison useful:

  1. Capture the baseline where CI runs (Baseline management).
  2. Mask content that changes on its own: privacy.maskSelectors and ignore.selectors for live clocks and carousels, visual.ignoreRegions for fixed areas.
  3. Raise visual.maxDiffRatio a little, or turn the rule off: comparison.severity: { visual: "off" }.

Per engine

Screenshots are compared within one engine: each engine has its own baseline, so a Firefox frame is compared with a Firefox frame. Within one engine, rendering is deterministic: MotionPR's tests capture the same build twice in Chromium, Firefox and WebKit and require byte-identical filmstrip frames and an identical settled screenshot. The default thresholds are the same for every engine for that reason; a threshold loose enough to hide engine differences would also hide real regressions.

Screenshots are taken at the viewport's deviceScaleFactor, so a screenshot's size is the viewport times that factor. When two captures use different factors, their screenshots are not-comparable (the sizes differ) and the comparison adds DEVICE_SCALE_FACTOR_DIFFERS; they are never rescaled to look comparable.

Between two engines (compare --against, --cross-browser), a changed settled screenshot is reported as uncertain with the engines named, never as a warning: engines rasterize text, borders and anti-aliasing differently, and the diff image is for investigating, not for failing a build. See Cross-browser testing.

Inputs, textareas, selects and contenteditable regions are masked in every screenshot and frame by default (privacy.maskInputs). See Privacy.