Repository navigation
feat: integration of finished branches (shader effects, export test fix) - #1209
Closed
EtienneLescot wants to merge 5 commits into
Closed
EtienneLescot wants to merge 5 commits into
EtienneLescot wants to merge 5 commits into
Conversation
The full design after the Windows spike: the document model (definitions and clip-anchored regions), the WGSL contract v1 (named params, region progress, last clicks, footage and camera rects, deterministic hash), indicative cost tiers instead of budgets, the three targets (image, text, both), the DSL tools and what agents read back, rendering per platform, the UI, SSOT and safety, tests and the PR breakdown.
Agent-written WGSL effects (docs/shader-effects.md), the compositor half: the contract, its validation and the D3D11 pass. The document model and the agent tools come next. - shader_fx.rs: contract v1. The agent's code, then a generated FxParams struct (named, typed parameters at b3), then the prelude: frame(), ctx (resolution, aspect, programme time, region progress, pointer, the last four clicks, footage and camera rects in output uv), fx_px, fx_hash. - Validation walks naga's IR: every loop is a `for` with literal bounds, at most 4096 iterations per pixel, refused otherwise with its line. The same walk estimates the cost in frame() reads per pixel, as an indicative tier; nothing is refused on cost. - Windows: effects compile off the render thread (naga 24 -> HLSL SM 5.0 -> FXC), cached by contract, code and parameter layout, 64 kept. Three targets: image (before annotations and captions), text (on a transparent text layer drawn over the image), both (last); lanes order them. The export waits for a compile, the live view skips and then recomposes. - plan_cursor hands effects the pointer and the recent clicks; the napi addon gains checkShaderEffect (error, tier, reads, iterations). Calibrated on an RTX 4070 Ti: about 4 us per frame() read at 1080p and 17 us at 4K; a light effect costs at most 0.06 ms per 1080p frame. Metal and wgpu parse the effects but do not draw them yet.
Agent-written shader effects (docs/shader-effects.md), the document half: where an effect lives, how agents write and re-read it, and what the compositor receives. Stacked on the contract v1 PR. - Schema: `shaderEffects` (definitions: WGSL code, name, description, contract, declared params, cached cost) and `shaderEffectRanges` (clip-anchored regions with their values, target and lane). Optional, like `vocabulary`: a document without effects carries neither key. - `RegionKind` gains "shaderEffect" through the whole modifier pipeline, so an effect moves, trims and goes with its clip. A definition goes with its last region. - Tools: addEffect (writes and poses, or reuses by effectId), setEffect, setEffectDefinition, getEffect; removeModifier removes a region. The compositor checks the code first (checkShaderEffect, wired for the MCP server and the in-app agent); a compile error comes back at the agent's line. Results give the cost tier and a recommendation when a tier runs long, never a refusal. One lane holds no overlap. - getCurrentDocument lists effects and definitions without their code. - The scene gets each fragment with its definition, normalised values and the whole pill's span on the programme clock, so `ctx.region` runs across a cut. - MCP: the resource openscreen://effects/contract holds the full contract and two examples; a test keeps it in step with the prelude and a Rust test compiles its examples. - `openscreen info` counts effects; a workbench scenario (ripple on click) and document invariants for effects; the demo skill and the architecture docs say how it works.
The wgpu compositor now draws agent-written effects, as the Windows one does: same targets (image, text layer, both), same lanes, same blocking rule (the export waits for the pipeline, the live view skips it until it lands). - The WGSL goes to wgpu as it is. A background thread builds the module and the pipeline against an explicit layout of the prelude's four bindings. - wgpu panics on a validation error: `analyze` now also refuses any binding the prelude does not declare, so nothing invalid reaches it. - `check_for_agent` translates the effect to SPIR-V on Linux. - live.rs turns the preview's non-blocking mode on for Linux too. Tested on lavapipe under WSL: the six GPU tests of the Windows pass, ported, and the full compositor suite. Tier calibration on lavapipe goes to rendering-performance.md and the spec (C.7, H.2).
Two exportVideo tests slept 20 ms and assumed the first export was running by then. Under the full suite's load it was not, and the second call was not refused. The fake runner now resolves a promise when a held export starts, which is after the server marked it running and read the project.
Contributor
|
Note Currently processing new changes in this PR. This may take a few minutes, please wait... ⚙️ Run configuration
⛔ Files ignored due to path filters: 1
📒 Files selected for processing: 47
✨ Finishing Touches
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
6 tasks done
Collaborator
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The integration branch for finished work: current
main, with each finished branch replayed on top in order. It is where the pieces are tested together, and it gives CI to the stacked PRs, which have none while they target another branch.What it holds
feat/shader-effects-contractfeat/shader-effects-documentfeat/shader-effects-linuxexportVideotest fixtest/flaky-export-concurrencyHow it is kept
mainby cherry-picking each branch's commits, so the history stays linear (the repo merges by rebase only).Merging. Merging this PR lands every row at once. Each PR in the table then has its commits in
mainalready and should be closed rather than merged. To land them one by one instead, merge the rows in order and rebase this branch after each.Related issue
None.
Type of change
Release impact
Desktop impact
Testing
On this branch, locally:
npx tsc --noEmit,npx tsc -p tsconfig.test.json --noEmit,npm run lint(warnings only).npm run testbefore row 4: everything green but the flaky test that row 4 fixes (it passes alone). The fix itself was run against the full suite on its own branch.cargo test -p openscreen-compositor -p compositor-view-napi --lib, 445 + 1 passed.cargo test -p openscreen-compositor --lib --tests, 481 passed plus the integration tests.🤖 Generated with Claude Code
Summary by CodeRabbit