Skip to content

feat(compositor): the macOS shader effects pass (Metal) - #1206

Merged
EtienneLescot merged 3 commits into
integration/shader-effectsfrom
feat/shader-effects-macos
Oct 14, 2026
Merged

EtienneLescot merged 3 commits into
integration/shader-effectsfrom
feat/shader-effects-macos

Conversation

@EtienneLescot

@EtienneLescot EtienneLescot commented Oct 14, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Third PR of agent-written shader effects, section L.3 of docs/shader-effects.md: the macOS (Metal) pass. An effect now draws on macOS, in preview and export alike; with the Windows and Linux passes, every backend draws effects. Targets integration/shader-effects, which carries #1199, #1202 and the Linux pass.

  • Translation and compile (shader_fx.rs).
    • naga 24 → MSL, with the prelude's bindings mapped explicitly: texture 0, the sampler inlined as a constexpr (linear, clamp to edge, like sampNV), ctx and params at buffers 2 and 3. Without that map, naga emits [[user(fake0)]], which is not MSL.
    • new_library_with_source and a pipeline state with no blending, on a background thread under its own autorelease pool.
    • checkShaderEffect compiles for Metal too, so "accepted" means "will draw".
    • naga's force_loop_bounding is off. The validation has already proved every loop bounded, and naga's guard (a volatile read on every iteration) cost 7 to 10 % of the pass on an M1.
  • The pass (compositor_macos.rs).
    • Same as Windows: the cache, the blocking/non-blocking split and the lane order.
    • Each effect is encoded in the frame's command buffer: a blit of its target into ann_copy, then a render pass that loads the target (an effect may discard).
    • The text target: draw_annotations opened its pass on self.rt; it now takes the texture it draws on. The text layer is cleared in a pass of its own, receives the annotations, goes through its effects, and is drawn over the image by pipeline_fs_tex (premultiplied "over").
  • live.rs: every backend draws effects now, so the live view's non-blocking mode and its recompose-on-landing lose their cfg.
  • Cargo.toml: msl-out is declared for macOS instead of arriving through wgpu's features.
  • Docs: native-compositor.md (the section; the known gap goes, every backend draws), the spec (C.7, D.4, D.5, new H.3, next to the Linux figures), rendering-performance.md, and a row in the e2e log.

Related issue

Part of the shader effects work (#1199, #1202, and the Linux pass on integration/shader-effects).

Type of change

  • Feature
  • Documentation

Release impact

  • No release note needed: nothing is reachable from the app until the document model and the tools land

Desktop impact

  • macOS
  • Windows and Linux: untouched, apart from live.rs losing a cfg that now covers every backend

Testing

  • Rust: cargo test -p openscreen-compositor --lib --tests passes on an M1 (macOS 26.5), 445 lib tests on top of integration/shader-effects. On the Metal device:
    • the six GPU tests of the Windows pass: region, preview landing, broken effect or unknown contract, color parameter, lane order, the three targets;
    • plus a real annotation, which an image effect leaves alone and a text effect repaints, while the image under it stays intact;
    • the MSL bindings, and the spec's C.6 ripple and the two examples of the MCP resource compiling through Metal.
  • Calibration (shader_fx_tier_calibration, release, M1): about 0.12 ms per frame() read at 1080p and 0.5 ms at 4K, thirty times the RTX 4070 Ti. A heavy effect (128 reads) costs 16 ms per 1080p frame. The reads still dominate, so the tiers hold.
  • Compile time: 120 ms in the median for a new source, up to 375 ms for the first in a process; a known source comes back in 0.4 ms. All of it runs off the render thread.
  • End to end, on a local branch with feat(effects): shader effects in the document, and the agent tools #1202 merged in (never pushed), through electron . --user-data-dir=<test profile> mcp, headless:
    • addEffect with a typo is refused in 6 ms at the agent's line; the ripple is accepted in 152 ms, real Metal compile included;
    • exportVideo + getFrames show the ripple after each click;
    • live view against the MP4 (BT.709): 3.1/255 over the frame with the effect, 3.0 without. On the pixels the effect changes, 13 to 16/255 with the effect against 14 to 16 without on the same pixels: codec noise on sharp edges;
    • export of 180 frames at 1752×1080: 2469 ms without, 2428 ms with the ripple. The live view held 29.7 fps at a 30 fps source with and without.
  • Not run: Windows and Linux, which this PR barely touches; the editor UI (no effect track yet). No CI runs on a PR to integration/shader-effects (ci.yml triggers on main, feat/ai-edition and release/**).

@coderabbitai

coderabbitai Bot commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration
  • Configuration used: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: d016aa54-40ab-474a-89d0-fe32d2298bd1

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

EtienneLescot added a commit that referenced this pull request Oct 14, 2026
The parent folder's checks, adapted to the Metal pass (#1206) and to the
agent tools of #1202: the MCP loop, the live view against the export,
the codec floor on the effect's own pixels, and export time. The live
view now gets the inspector's settings through setParam, which the
Windows script skipped.
@EtienneLescot
EtienneLescot force-pushed the feat/shader-effects-macos branch from 6338417 to 80c6dde Compare October 14, 2026 15:11
@EtienneLescot
EtienneLescot changed the base branch from feat/shader-effects-contract to integration/shader-effects October 14, 2026 15:11
Agent-written WGSL effects (docs/shader-effects.md) now draw on macOS, in
preview and export alike. With the Windows and Linux passes, every
backend draws them.

- shader_fx.rs: naga 24 to MSL with the prelude's bindings mapped
  explicitly (texture 0, an inline constexpr sampler, ctx and params at
  buffers 2 and 3), so naga never invents a binding. Compiled with
  new_library_with_source and a Replace pipeline state on a background
  thread, under its own autorelease pool. checkShaderEffect compiles for
  Metal too: accepted means it will draw.
- naga's force_loop_bounding is off: the validation already proved every
  loop bounded, and its volatile guard cost 7 to 10 % of the pass.
- compositor_macos.rs: the same cache, blocking and lane order as the
  other passes. Each pass blits its target into ann_copy in the frame's
  command buffer, then repaints it in a pass that loads the target.
  draw_annotations takes the texture it draws on, which is how the text
  layer gets the annotations for `text` effects.
- live.rs: every backend draws effects now, so the live view's
  non-blocking mode and its recompose-on-landing lose their cfg.
- msl-out is declared for macOS instead of arriving through wgpu.

Tests on the Metal device: the six GPU tests of the Windows pass, plus a
real annotation that an `image` effect leaves alone and a `text` effect
repaints, and the spec's ripple and the two MCP examples compiling.

Measured on an M1: about 0.12 ms per frame() read at 1080p and 0.5 ms at
4K, thirty times the RTX 4070 Ti; a new source compiles in 120 ms (up to
375 ms for the first in a process). An agent's addEffect took 152 ms
through the mcp CLI; preview and export agree within codec noise; export
time and preview frame rate do not change.
@EtienneLescot EtienneLescot mentioned this pull request Oct 14, 2026
3 tasks done
Same two lines as the Windows and Linux passes (#1213): the footage
quad and its aspect, so fx_to_footage works under Metal too.
… the footage plane

Two things from the review of integration/shader-effects (#1213 and the
review fix) that the Metal pass did not have yet.

- The compiled-effect cache never evicts an effect the current scene
  shows (shader_fx::eviction_victim), as on Windows and Linux: set_scene
  installs the scene before it starts the compiles.
- The Windows GPU test of fx_to_footage and fx_from_footage, ported to
  the Metal device: the footage plane maps back to what the compositor
  drew, flat, under a fixed angle (bilinear) and under the real camera
  (homography). It also checks their MSL translation.
@EtienneLescot
EtienneLescot merged commit 8431e83 into integration/shader-effects Oct 14, 2026
16 checks passed
EtienneLescot added a commit that referenced this pull request Oct 14, 2026
…the tiers on an M1

A Metal compile takes 120 ms for a new source on an M1, up to 375 ms for
the first in a process (#1206), against 10 ms for FXC. checkShaderEffect
ran it on the main process's thread.

- The addon's checkShaderEffect is an AsyncTask on the libuv pool, like
  segmentFrame. The service and the checker type return a promise; the
  service awaits it inside its try, so a rejecting addon still reads as
  "not checked", as a throwing one did.
- runDocumentTool asks first, for the code addEffect or
  setEffectDefinition will store (shaderEffectToCheck), then hands the
  answer to the executor, which stays synchronous, as the cursor read and
  the auto-zoom plan already do. Edits off: nothing is asked.
- addEffect's description and the MCP contract resource give each tier
  in ms on the RTX 4070 Ti and on the M1, as the spec's C.7 asks.
@EtienneLescot
EtienneLescot deleted the feat/shader-effects-macos branch October 14, 2026 17:51
EtienneLescot added a commit that referenced this pull request Oct 14, 2026
…the tiers on an M1

A Metal compile takes 120 ms for a new source on an M1, up to 375 ms for
the first in a process (#1206), against 10 ms for FXC. checkShaderEffect
ran it on the main process's thread.

- The addon's checkShaderEffect is an AsyncTask on the libuv pool, like
  segmentFrame. The service and the checker type return a promise; the
  service awaits it inside its try, so a rejecting addon still reads as
  "not checked", as a throwing one did.
- runDocumentTool asks first, for the code addEffect or
  setEffectDefinition will store (shaderEffectToCheck), then hands the
  answer to the executor, which stays synchronous, as the cursor read and
  the auto-zoom plan already do. Edits off: nothing is asked.
- addEffect's description and the MCP contract resource give each tier
  in ms on the RTX 4070 Ti and on the M1, as the spec's C.7 asks.
EtienneLescot added a commit that referenced this pull request Oct 15, 2026
…the tiers on an M1

A Metal compile takes 120 ms for a new source on an M1, up to 375 ms for
the first in a process (#1206), against 10 ms for FXC. checkShaderEffect
ran it on the main process's thread.

- The addon's checkShaderEffect is an AsyncTask on the libuv pool, like
  segmentFrame. The service and the checker type return a promise; the
  service awaits it inside its try, so a rejecting addon still reads as
  "not checked", as a throwing one did.
- runDocumentTool asks first, for the code addEffect or
  setEffectDefinition will store (shaderEffectToCheck), then hands the
  answer to the executor, which stays synchronous, as the cursor read and
  the auto-zoom plan already do. Edits off: nothing is asked.
- addEffect's description and the MCP contract resource give each tier
  in ms on the RTX 4070 Ti and on the M1, as the spec's C.7 asks.
EtienneLescot added a commit that referenced this pull request Oct 15, 2026
…the tiers on an M1

A Metal compile takes 120 ms for a new source on an M1, up to 375 ms for
the first in a process (#1206), against 10 ms for FXC. checkShaderEffect
ran it on the main process's thread.

- The addon's checkShaderEffect is an AsyncTask on the libuv pool, like
  segmentFrame. The service and the checker type return a promise; the
  service awaits it inside its try, so a rejecting addon still reads as
  "not checked", as a throwing one did.
- runDocumentTool asks first, for the code addEffect or
  setEffectDefinition will store (shaderEffectToCheck), then hands the
  answer to the executor, which stays synchronous, as the cursor read and
  the auto-zoom plan already do. Edits off: nothing is asked.
- addEffect's description and the MCP contract resource give each tier
  in ms on the RTX 4070 Ti and on the M1, as the spec's C.7 asks.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant