fix(execution): stop in-flight workflow runs when an account is deleted, removed or banned by waleedlatif1 · Pull Request #9145 · simstudioai/sim

waleedlatif1 · 2026-10-16T22:12:06Z

Summary

No removal path stopped in-flight workflow executions; #9089 stops Copilot runs only. On 2026-10-16 a prod run kept spending a hosted fal.ai key for five minutes after its owner deleted their account, then crashed on foreign-key errors. Banned accounts' runs also kept spending until their timeout.

One mechanism for every removal path

lib/execution/removal-stop-records.ts (light, safe for page graphs):
- collects the executions a removal must stop: running log rows, plus claimed resumes, which run under their own execution id;
- records them as durable execution.removal-stop outbox events, one per workspace;
- signals them through markExecutionCancelled + abortManualExecution, 10 at a time with a 10 s deadline, and never throws.
lib/execution/removal-stops.ts (outbox handler): signals again, then sweeps both job queues for the workspace.
New JobQueueBackend.cancelByWorkspace:
- Trigger: runs.list on the existing workspaceId:<id> tag, filtered to the five workflow task types (workflow, schedule, webhook, resume, workflow-group cell) and active statuses, with each result's tags rechecked exactly. Pages of 100, 10 concurrent cancels, capped.
- Database: UPDATE async_jobs filtered on status, the same five job types, and metadata->>'workspaceId'; aborts this process's inline controllers.
- Queued runs are cancelled before they start a machine.

Wired into

Account deletion: stops are collected and enqueued inside the deletion transaction, after the existing workspace FOR UPDATE, so a rollback enqueues nothing. After commit the signal is fired and forgotten before any other step. The sweep runs in the outbox worker. The deletion response and the storage purge never wait on Trigger.
Ban (disableUserResources): after the archives commit, stops are recorded for the workspaces the ban archived and signalled immediately. A failure to record them is logged, not thrown, so the committed ban never 500s. Bans are admin-only.
Admin /admin/remove-user:
- Better Auth's user.delete.before collects the user's owned workspaces and the workflows they created; both cascade with the user.
- .after enqueues and signals the stops.
- Collection never blocks the removal.

Runs stopped this way still log a noisy ending until #9144 lands; that PR makes it quiet.

Type of Change

Bug fix

Testing

Trigger backend tests use a fake that stores the tags enqueue actually sends. They prove:
- producer tags match the sweep filter;
- only the five task types are cancelled;
- the cap holds across calls;
- partial cancel failures still throw after the rest are cancelled;
- at most 10 cancels run concurrently.
Handler tests run against the real cancellation module:
- no Redis completes;
- a refused Redis write retries;
- a sweep error retries;
- hitting the cap after progress continues;
- hitting the cap with no progress retries.
database.test.ts pins the job-type filter.
Each guard was mutated and went red, then green when restored.
removal-stops.integration.ts (real Postgres + Redis) covers:
- deletion: flags set, queued job cancelled, a non-workflow cleanup-logs job and a bystander's run untouched;
- ban: immediate signal, sweep through the worker;
- admin removal: a workflow the user created in another workspace is stopped, a bystander's is not.
check-tool-registry-boundary --check: all graphs within baseline.
An independent adversarial review checked tag exact-match, job-type filtering on both backends, Better Auth hook semantics (1.6.27), FK cascades behind the admin path, and outbox retry behavior.

Checklist

Code follows project style guidelines
Self-reviewed my changes
Tests added/updated and passing
No new warnings introduced

vercel · 2026-10-16T22:12:09Z

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated
docs	Skipped		Oct 16, 2026 11:38pm UTC

cubic-dev-ai

All reported issues were addressed across 15 files

_{This PR changes concurrency-sensitive code such as locks, queues, or retries. Ultrareviews find 2.4x more serious bugs than standard reviews. Comment @cubic-dev-ai ultrareview to run one.

Turn on auto-fix | Re-trigger cubic}

greptile-apps · 2026-10-16T22:18:05Z

Confidence Score: 5/5

[Critical impact] The PR appears safe to merge; no new blocking issue was found.

Summary

This PR adds workflow cancellation to account deletion, admin removal, and bans. It records stops for later retries, signals running executions, and cancels workspace jobs through both queue backends.

Deleted, banned, and admin-removed accounts stop affected workflow runs.
Removal stops reach active runs and queued jobs through the outbox.
Queues can cancel active workflow jobs by workspace.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Account deletion, admin removal, or ban] --> B[Collect executions and record stop events]
  B --> C[Signal running executions]
  B --> D[Outbox worker]
  D --> E[Signal executions again]
  D --> F[Cancel workspace jobs in both queues]
  E --> G[Retry failures]
  F --> G

_{Reviews (6) · Last reviewed commit: "fix(execution): sweep a removed workspac..." · Reviewed by Greptile}

waleedlatif1 · 2026-10-16T22:48:05Z

@greptile

waleedlatif1 · 2026-10-16T22:48:07Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-10-16T22:48:12Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

All reported issues were addressed across 17 files

_{Reply with feedback, questions, or to request a fix.

Turn on auto-fix | Re-trigger cubic}

waleedlatif1 · 2026-10-16T23:02:05Z

@greptile

waleedlatif1 · 2026-10-16T23:02:07Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-10-16T23:02:12Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

…ed, removed or banned No removal path stopped workflow executions. Account deletion cascades the doomed workspaces' log rows away, after which no cancel path can find the run, so it kept spending hosted provider keys for an account that no longer existed until it crashed on foreign keys at finalization. A ban archived the workspaces and stopped nothing; an admin removal cascaded them the same way. - Account deletion reads the running executions (and claimed resumes, which run under their own id) of the doomed workspaces in its transaction, while the workspace rows are locked, and records one outbox stop per workspace with the erasure. After commit the stops are delivered a few at a time; a stop that cannot be delivered stays in the outbox for its worker, and the committed deletion is never reported as failed. - A stop sets the cancellation flag each engine watches and sweeps the job queue for the workspace: a new JobQueueBackend.cancelByWorkspace cancels queued, delayed and running workflow jobs by the workspaceId tag on Trigger.dev (pages of 100, ten cancels at a time, 1,000 runs per attempt, continued without spending an attempt) and by metadata.workspaceId on the database queue, so queued runs no longer fail preprocessing later. - A missing Redis completes the stop: the flag is process-local by design and no retry could reach further. A refused Redis write or a failed sweep retries. - A ban stops the runs of the workspaces it archived; the admin remove-user hooks read the runs before the bare user delete and record them after it commits. Both sit in page module graphs that cannot load the delivery, so they signal in-flight runs at once and leave the queue sweep to the outbox worker.

…paths - Account deletion no longer awaits the full stop delivery before purging storage. A slow Trigger.dev listing could hold the response for minutes and widen the window in which a restart orphans erased objects. It now signals in-flight runs fire-and-forget right after commit, ahead of every other post-commit step, under a 10-second deadline; the committed outbox stops repeat the signal and sweep the job queue in the outbox worker. - A failure to record a ban's stops is logged instead of thrown. The ban and archives have committed, so a 500 helped no one and a retry finds no unarchived workspace to stop. - Claimed resumes are found from the scope's running and paused log rows and joined through resume_queue's parent index, not by scanning resume_queue for claimed rows inside the deletion transaction. - Tests pin the database queue's job-type filter, so a workspace sweep never cancels a non-workflow job such as log cleanup.

- A ban records and signals each workspace's stops as soon as its archive commits, through the archive's post-commit stop hook, instead of after every archive's socket and webhook cleanup. It also stops runs in the owner's workspaces archived before the ban, which an archive leaves running. - The outbox handler re-reads a workspace's running executions when it runs. An archived workspace keeps its rows, so a run admitted before the ban that logged after its stop was recorded is still signalled. - The admin remove-user hook handlers move to lib/auth/admin-removal-stops.ts, and signal the removed runs even when recording their stops fails. - The database queue's workspace sweep works in batches of 100 up to the caller's cap and checks its abort signal between them, like the Trigger.dev sweep, rather than one unbounded update reported as complete.

- disableUserResources reads owned workspaces once and stops earlier-archived ones up front, dropping the post-archive read that could fail the ban - Trigger.dev cancelByWorkspace reuses scanAndCancelTriggerRuns; workspace tag shares one builder with enqueue - database queue shares abortInlineJob with cancelJob and returns completeness from control flow - removal stop handler parses its payload with zod and signals while it sweeps - drop the unused event-id return and redundant bucketing state

- The removal stop handler loads both job queues independently and sweeps whichever loaded, so a primary queue that fails to load no longer skips the database queue's sweep. The failure is still rethrown for the outbox to retry. - A claimed resume is found under a parent log row in the legacy `paused` status too, one of the statuses a resume claims from. - The database queue re-checks for a matching active job after a short batch, since a selected job that finished before the update reached it shrinks the batch while later jobs still match. - The ban's completion log no longer reports its runs stopped when recording their stops failed.

cubic-dev-ai

Review completed against the latest diff

_{Reply with feedback, questions, or to request a fix.

Turn on auto-fix | Re-trigger cubic}

…an skip them The signal deadline bounds only the Redis flag writes. Aborting a run this process holds needs no Redis, so every execution is aborted first, even after the deadline has passed. Corrects the ban's note on why a failed stop record is logged: banning again records the stops again.

waleedlatif1 · 2026-10-16T23:10:59Z

@greptile

waleedlatif1 · 2026-10-16T23:11:00Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-10-16T23:11:07Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

2 issues found across 18 files

_{Reply with feedback, questions, or to request a fix.

Turn on auto-fix | Re-trigger cubic}

Prompt for AI agents (unresolved issues)


Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="apps/sim/lib/workflows/lifecycle.ts">

<violation number="1" location="apps/sim/lib/workflows/lifecycle.ts:348">
P2: The archived-workspace stop path can cancel a successor owner's runs after a concurrent ownership transfer. Revalidate and lock ownership in the same transaction before recording stops, skipping workspace IDs no longer owned by `userId`.

(Based on your team's feedback about rechecking ownership during ban archival.) [4566fdf2-4a7b-4c10-8836-e8222b12fa54]</violation>
</file>

<file name="apps/sim/lib/core/async-jobs/backends/trigger-dev.ts">

<violation number="1" location="apps/sim/lib/core/async-jobs/backends/trigger-dev.ts:698">
P2: `cancelByWorkspace` charges `maxRuns` when a run is selected, not when it is cancelled. If the first `maxRuns` listed runs keep failing `runs.cancel`, every retry re-selects them and later active runs are never reached. Only count runs whose cancellation succeeded, or skip already-failed IDs on later pages.</violation>
</file>

- A ban records stops only for workspaces the user still owns, re-read in the recording transaction and held FOR SHARE until it commits, so a concurrent ownership transfer waits rather than having its new owner's runs stopped. Only workspace rows are locked, so no lock order is inverted. - Trigger.dev's workspace sweep counts only cancelled runs toward its cap, with a separate bound on failures, so runs that keep refusing cancellation no longer hold every retry at the front of the listing. cancelByExecution is unchanged.

waleedlatif1 · 2026-10-16T23:24:05Z

@greptile

waleedlatif1 · 2026-10-16T23:24:07Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-10-16T23:24:12Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

1 issue found across 18 files

Confidence score: 3/5

removal-stop-records.ts can miss executions created between the pre-delete read and the user delete, leaving records behind after an admin deletion. Synchronize the collection and delete in one transaction, or use an equivalent deletion/run-admission lock.

Prompt for AI agents (unresolved issues)


Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="apps/sim/lib/execution/removal-stop-records.ts">

<violation number="1" location="apps/sim/lib/execution/removal-stop-records.ts:127">
P2: Admin deletion can miss executions created between this pre-delete read and the user delete. Synchronize collection with the delete in one transaction or add an equivalent deletion/run-admission lock so every execution erased by the cascade is recorded before commit.</violation>
</file>

_{You've manually re-run cubic several times on this PR. Each manual re-review checks the full PR again and counts toward your usage quota. To preserve your usage limits, we recommend letting cubic automatically review new commits.

Turn on auto-fix | Re-trigger cubic}

The workspace sweep searched only runs created in the last 14 days, so an older run still queued or waiting survived the removal. It now passes only an upper time bound, which Trigger.dev treats as no lower bound, and falls back to the retention window only if Trigger.dev refuses that search as too expensive before returning a run. cancelByExecution is unchanged.

waleedlatif1 · 2026-10-16T23:38:58Z

@greptile

waleedlatif1 · 2026-10-16T23:38:59Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-10-16T23:39:04Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

No issues found across 18 files

Confidence score: 5/5

Automated review surfaced no issues in the provided summaries.
No files require special attention.

_{You've manually re-run cubic several times on this PR. Each manual re-review checks the full PR again and counts toward your usage quota. To preserve your usage limits, we recommend letting cubic automatically review new commits.

Turn on auto-fix | Re-trigger cubic}

vercel Bot temporarily deployed to Preview October 16, 2026 22:12 Inactive

cubic-dev-ai Bot reviewed Oct 16, 2026

View reviewed changes

greptile-apps Bot reviewed Oct 16, 2026

View reviewed changes

Comment thread apps/sim/lib/workflows/lifecycle.ts Outdated

Comment thread apps/sim/lib/workflows/lifecycle.ts

Comment thread apps/sim/lib/execution/removal-stops.integration.ts Outdated

vercel Bot temporarily deployed to Preview October 16, 2026 22:22 Inactive

vercel Bot temporarily deployed to Preview October 16, 2026 22:35 Inactive

vercel Bot temporarily deployed to Preview October 16, 2026 22:43 Inactive

cubic-dev-ai Bot reviewed Oct 16, 2026

View reviewed changes

Comment thread apps/sim/lib/execution/removal-stops.ts Outdated

Comment thread apps/sim/lib/execution/removal-stop-records.ts Outdated

Comment thread apps/sim/lib/workflows/lifecycle.ts Outdated

greptile-apps Bot reviewed Oct 16, 2026

View reviewed changes

Comment thread apps/sim/lib/core/async-jobs/backends/database.ts Outdated

vercel Bot temporarily deployed to Preview October 16, 2026 23:00 Inactive

waleedlatif1 added 6 commits October 16, 2026 16:02

test(execution): give each removal-stop fixture workflow a unique name

b65ee91

waleedlatif1 force-pushed the fix/stop-runs-on-account-removal branch from 770804e to cd54396 Compare October 16, 2026 23:03

vercel Bot temporarily deployed to Preview October 16, 2026 23:03 Inactive

cubic-dev-ai Bot reviewed Oct 16, 2026

View reviewed changes

Comment thread apps/sim/lib/workflows/lifecycle.ts Outdated

Comment thread apps/sim/lib/execution/removal-stop-records.ts

vercel Bot temporarily deployed to Preview October 16, 2026 23:10 Inactive

waleedlatif1 mentioned this pull request Oct 16, 2026

chore: red proof for #9145 — do not merge #9149

Closed

cubic-dev-ai Bot reviewed Oct 16, 2026

View reviewed changes

Comment thread apps/sim/lib/workflows/lifecycle.ts

Comment thread apps/sim/lib/core/async-jobs/backends/trigger-dev.ts Outdated

vercel Bot temporarily deployed to Preview October 16, 2026 23:23 Inactive

cubic-dev-ai Bot reviewed Oct 16, 2026

View reviewed changes

Comment thread apps/sim/lib/core/async-jobs/backends/trigger-dev.ts Outdated

Comment thread apps/sim/lib/execution/removal-stop-records.ts

Comment thread apps/sim/lib/execution/removal-stop-records.ts

vercel Bot temporarily deployed to Preview October 16, 2026 23:38 Inactive

cubic-dev-ai Bot reviewed Oct 16, 2026

View reviewed changes

waleedlatif1 merged commit 84f41ea into staging Oct 16, 2026
53 checks passed

waleedlatif1 deleted the fix/stop-runs-on-account-removal branch October 16, 2026 23:47

This was referenced Oct 17, 2026

fix(billing): stop account deletion from resetting the free allowance #9141

Merged

v0.9.26: abuse prevention, project files #9155

Open

Conversation

waleedlatif1 commented Oct 16, 2026

Summary

Type of Change

Testing

Checklist

Uh oh!

vercel Bot commented Oct 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

greptile-apps Bot commented Oct 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Confidence Score: 5/5

Summary

Diagram

Uh oh!

Uh oh!

Uh oh!

Uh oh!

waleedlatif1 commented Oct 16, 2026

Uh oh!

waleedlatif1 commented Oct 16, 2026

Uh oh!

cubic-dev-ai Bot commented Oct 16, 2026

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

waleedlatif1 commented Oct 16, 2026

Uh oh!

waleedlatif1 commented Oct 16, 2026

Uh oh!

cubic-dev-ai Bot commented Oct 16, 2026

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

waleedlatif1 commented Oct 16, 2026

Uh oh!

waleedlatif1 commented Oct 16, 2026

Uh oh!

cubic-dev-ai Bot commented Oct 16, 2026

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

waleedlatif1 commented Oct 16, 2026

Uh oh!

waleedlatif1 commented Oct 16, 2026

Uh oh!

cubic-dev-ai Bot commented Oct 16, 2026

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

waleedlatif1 commented Oct 16, 2026

vercel Bot commented Oct 16, 2026 •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading

greptile-apps Bot commented Oct 16, 2026 •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading