fix(webapp): use composite keyset cursor for run pagination#3852
Open
matt-aitken wants to merge 1 commit into
Open
fix(webapp): use composite keyset cursor for run pagination#3852matt-aitken wants to merge 1 commit into
matt-aitken wants to merge 1 commit into
Conversation
listRunIds/listRuns order by the composite key (created_at, run_id) but the cursor predicate cut on run_id alone. That is only sound when run_id lexicographic order matches created_at order. When a burst of runs is created such that the two diverge, keyset pagination both re-includes already-returned runs (duplicates) and drops runs it should return (skips). For bulk replay this produced duplicate runs; for the dashboard and runs.list it could silently skip or repeat runs at page boundaries. - Encode cursors as the composite (created_at, run_id) key (v2_<createdAtMs>_<runId>) and cut on the matching tuple predicate ((created_at, run_id) < / > (...)). The ORDER BY is unchanged, so the table's primary-key alignment (and query performance) is preserved. - Cursors are server-issued opaque tokens (the SDK just echoes pagination.next/previous back), so this needs no client update. Legacy bare-run_id cursors decode to the old run_id-only predicate for backwards compatibility with in-flight cursors. - Add listRunIdsWithCursor for forward-only batch iteration (bulk actions) so the created_at component is sourced from the same query that orders the rows. - ClickHouse getTaskRunsQueryBuilder now also selects toUnixTimestamp64Milli(created_at) AS created_at_ms. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
Contributor
|
Too much diff to scan? Review this PR in Change Stack to start with the highest-impact changes. WalkthroughThis PR fixes a correctness bug in keyset pagination for 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
🔴 getPendingVersionIdsQueryBuilder shares schema that now requires created_at_ms, but its SQL only selects run_id — zod parse will fail at runtime
The PR adds a required created_at_ms: z.number().int() field to TaskRunV2QueryResult (internal-packages/clickhouse/src/taskRuns.ts:372), and updates the getTaskRunsQueryBuilder base query to include toUnixTimestamp64Milli(created_at) AS created_at_ms. However, getPendingVersionIdsQueryBuilder at line 400 reuses the same TaskRunV2QueryResult schema but its base query ("SELECT run_id FROM trigger_dev.task_runs_v2") does NOT select created_at_ms. At runtime, when ClickHouse returns rows with only run_id, the zod parse at client.ts:207 (z.array(req.schema).safeParse(unparsedRows)) will fail because created_at_ms is required but missing. This returns a QueryError, which the caller in clickhousePendingVersionLookup.server.ts:84-92 catches and silently converts to { runIds: [] } — effectively breaking all PENDING_VERSION run processing.
(Refers to lines 398-403)
Prompt for agents
The getPendingVersionIdsQueryBuilder function at internal-packages/clickhouse/src/taskRuns.ts:394-404 shares the TaskRunV2QueryResult schema with getTaskRunsQueryBuilder, but its base SQL query only selects run_id, not created_at_ms. After this PR added created_at_ms as a required field in TaskRunV2QueryResult, zod validation will fail for every query made by getPendingVersionIdsQueryBuilder.
Two possible fixes:
1. Give getPendingVersionIdsQueryBuilder its own schema that only requires run_id (e.g. a new PendingVersionQueryResult = z.object({ run_id: z.string() })).
2. Add created_at_ms to the base query of getPendingVersionIdsQueryBuilder as well (SELECT run_id, toUnixTimestamp64Milli(created_at) AS created_at_ms FROM ...), though the caller doesn't need it.
Option 1 is cleaner since the pending version lookup only needs run_id.
Was this helpful? React with 👍 or 👎 to provide feedback.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI
Review profile: CHILL
Plan: Pro
Run ID: 2cca079c-b500-45ee-ba9d-11e7970832e0
📒 Files selected for processing (7)
.server-changes/bulk-action-cursor-pagination.mdapps/webapp/app/services/runsRepository/clickhouseRunsRepository.server.tsapps/webapp/app/services/runsRepository/runsCursor.server.tsapps/webapp/app/services/runsRepository/runsRepository.server.tsapps/webapp/app/v3/services/bulk/BulkActionV2.server.tsapps/webapp/test/runsRepositoryCursor.test.tsinternal-packages/clickhouse/src/taskRuns.ts
📜 Review details
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (20)
- GitHub Check: webapp / 🧪 Unit Tests: Webapp (7, 8)
- GitHub Check: webapp / 🧪 Unit Tests: Webapp (5, 8)
- GitHub Check: webapp / 🧪 Unit Tests: Webapp (8, 8)
- GitHub Check: webapp / 🧪 Unit Tests: Webapp (6, 8)
- GitHub Check: webapp / 🧪 Unit Tests: Webapp (1, 8)
- GitHub Check: webapp / 🧪 Unit Tests: Webapp (3, 8)
- GitHub Check: webapp / 🧪 Unit Tests: Webapp (4, 8)
- GitHub Check: webapp / 🧪 Unit Tests: Webapp (2, 8)
- GitHub Check: internal / 🧪 Unit Tests: Internal (8, 8)
- GitHub Check: internal / 🧪 Unit Tests: Internal (7, 8)
- GitHub Check: internal / 🧪 Unit Tests: Internal (3, 8)
- GitHub Check: internal / 🧪 Unit Tests: Internal (2, 8)
- GitHub Check: internal / 🧪 Unit Tests: Internal (4, 8)
- GitHub Check: internal / 🧪 Unit Tests: Internal (5, 8)
- GitHub Check: internal / 🧪 Unit Tests: Internal (1, 8)
- GitHub Check: internal / 🧪 Unit Tests: Internal (6, 8)
- GitHub Check: typecheck / typecheck
- GitHub Check: e2e-webapp / 🧪 E2E Tests: Webapp
- GitHub Check: audit
- GitHub Check: Analyze (javascript-typescript)
🧰 Additional context used
📓 Path-based instructions (11)
**/*.{ts,tsx}
📄 CodeRabbit inference engine (.github/copilot-instructions.md)
**/*.{ts,tsx}: Use types over interfaces for TypeScript
Avoid using enums; prefer string unions or const objects insteadImport from
@trigger.dev/sdkwhen writing Trigger.dev tasks. Never use@trigger.dev/sdk/v3or deprecatedclient.defineJob
Files:
apps/webapp/app/services/runsRepository/runsRepository.server.tsapps/webapp/app/services/runsRepository/runsCursor.server.tsinternal-packages/clickhouse/src/taskRuns.tsapps/webapp/app/v3/services/bulk/BulkActionV2.server.tsapps/webapp/test/runsRepositoryCursor.test.tsapps/webapp/app/services/runsRepository/clickhouseRunsRepository.server.ts
{packages/core,apps/webapp}/**/*.{ts,tsx}
📄 CodeRabbit inference engine (.github/copilot-instructions.md)
Use zod for validation in packages/core and apps/webapp
Files:
apps/webapp/app/services/runsRepository/runsRepository.server.tsapps/webapp/app/services/runsRepository/runsCursor.server.tsapps/webapp/app/v3/services/bulk/BulkActionV2.server.tsapps/webapp/test/runsRepositoryCursor.test.tsapps/webapp/app/services/runsRepository/clickhouseRunsRepository.server.ts
**/*.{ts,tsx,js,jsx}
📄 CodeRabbit inference engine (.github/copilot-instructions.md)
Use function declarations instead of default exports
**/*.{ts,tsx,js,jsx}: Prefer static imports over dynamic imports. Only use dynamicimport()when circular dependencies cannot be resolved, code splitting is needed for performance, or the module must be loaded conditionally at runtime
Import subpaths only frompackages/core(@trigger.dev/core), never import from the root
Files:
apps/webapp/app/services/runsRepository/runsRepository.server.tsapps/webapp/app/services/runsRepository/runsCursor.server.tsinternal-packages/clickhouse/src/taskRuns.tsapps/webapp/app/v3/services/bulk/BulkActionV2.server.tsapps/webapp/test/runsRepositoryCursor.test.tsapps/webapp/app/services/runsRepository/clickhouseRunsRepository.server.ts
**/*.ts
📄 CodeRabbit inference engine (.cursor/rules/otel-metrics.mdc)
**/*.ts: When creating or editing OTEL metrics (counters, histograms, gauges), ensure metric attributes have low cardinality by using only enums, booleans, bounded error codes, or bounded shard IDs
Do not use high-cardinality attributes in OTEL metrics such as UUIDs/IDs (envId, userId, runId, projectId, organizationId), unbounded integers (itemCount, batchSize, retryCount), timestamps (createdAt, startTime), or free-form strings (errorMessage, taskName, queueName)
When exporting OTEL metrics via OTLP to Prometheus, be aware that the exporter automatically adds unit suffixes to metric names (e.g., 'my_duration_ms' becomes 'my_duration_ms_milliseconds', 'my_counter' becomes 'my_counter_total'). Account for these transformations when writing Grafana dashboards or Prometheus queries
Files:
apps/webapp/app/services/runsRepository/runsRepository.server.tsapps/webapp/app/services/runsRepository/runsCursor.server.tsinternal-packages/clickhouse/src/taskRuns.tsapps/webapp/app/v3/services/bulk/BulkActionV2.server.tsapps/webapp/test/runsRepositoryCursor.test.tsapps/webapp/app/services/runsRepository/clickhouseRunsRepository.server.ts
apps/webapp/**/*.{ts,tsx}
📄 CodeRabbit inference engine (.cursor/rules/webapp.mdc)
apps/webapp/**/*.{ts,tsx}: Access environment variables through theenvexport ofenv.server.tsinstead of directly accessingprocess.env
Use subpath exports from@trigger.dev/corepackage instead of importing from the root@trigger.dev/corepathUse named constants for sentinel/placeholder values (e.g.
const UNSET_VALUE = '__unset__') instead of raw string literals scattered across comparisons
Files:
apps/webapp/app/services/runsRepository/runsRepository.server.tsapps/webapp/app/services/runsRepository/runsCursor.server.tsapps/webapp/app/v3/services/bulk/BulkActionV2.server.tsapps/webapp/test/runsRepositoryCursor.test.tsapps/webapp/app/services/runsRepository/clickhouseRunsRepository.server.ts
apps/webapp/**/*.server.ts
📄 CodeRabbit inference engine (apps/webapp/CLAUDE.md)
apps/webapp/**/*.server.ts: Never userequest.signalfor detecting client disconnects. UsegetRequestAbortSignal()fromapp/services/httpAsyncStorage.server.tsinstead, which is wired directly to Expressres.on('close')and fires reliably
Access environment variables viaenvexport fromapp/env.server.ts. Never useprocess.envdirectly
Always usefindFirstinstead offindUniquein Prisma queries.findUniquehas an implicit DataLoader that batches concurrent calls and has active bugs even in Prisma 6.x (uppercase UUIDs returning null, composite key SQL correctness issues, 5-10x worse performance).findFirstis never batched and avoids this entire class of issues
Files:
apps/webapp/app/services/runsRepository/runsRepository.server.tsapps/webapp/app/services/runsRepository/runsCursor.server.tsapps/webapp/app/v3/services/bulk/BulkActionV2.server.tsapps/webapp/app/services/runsRepository/clickhouseRunsRepository.server.ts
**/*.{js,ts,tsx,jsx,css,json,md}
📄 CodeRabbit inference engine (AGENTS.md)
Use Prettier for code formatting and run
pnpm run formatbefore committing
Files:
apps/webapp/app/services/runsRepository/runsRepository.server.tsapps/webapp/app/services/runsRepository/runsCursor.server.tsinternal-packages/clickhouse/src/taskRuns.tsapps/webapp/app/v3/services/bulk/BulkActionV2.server.tsapps/webapp/test/runsRepositoryCursor.test.tsapps/webapp/app/services/runsRepository/clickhouseRunsRepository.server.ts
**/*.{test,spec}.{ts,tsx}
📄 CodeRabbit inference engine (.github/copilot-instructions.md)
Use vitest for all tests in the Trigger.dev repository
Files:
apps/webapp/test/runsRepositoryCursor.test.ts
apps/webapp/**/*.test.{ts,tsx}
📄 CodeRabbit inference engine (.cursor/rules/webapp.mdc)
Do not import
env.server.tsdirectly or indirectly into test files; instead pass environment-dependent values through options/parameters to make code testableFor testable code, never import
env.server.tsin test files. Pass configuration as options instead (e.g.,realtimeClient.server.tstakes config as constructor arg,realtimeClientGlobal.server.tscreates singleton with env config)
Files:
apps/webapp/test/runsRepositoryCursor.test.ts
**/*.test.{ts,tsx}
📄 CodeRabbit inference engine (CLAUDE.md)
**/*.test.{ts,tsx}: Never mock anything in tests - use testcontainers instead
Test files should be placed next to source files (e.g.,MyService.ts->MyService.test.ts)
Files:
apps/webapp/test/runsRepositoryCursor.test.ts
**/*.test.{js,ts,tsx}
📄 CodeRabbit inference engine (AGENTS.md)
**/*.test.{js,ts,tsx}: Test files should live beside the files under test and use descriptivedescribeanditblocks
Use vitest for unit testing
Tests should avoid mocks or stubs and use helpers from@internal/testcontainerswhen Redis or Postgres are needed
Files:
apps/webapp/test/runsRepositoryCursor.test.ts
🧠 Learnings (37)
📓 Common learnings
Learnt from: ericallam
Repo: triggerdotdev/trigger.dev PR: 3417
File: apps/webapp/app/services/sessionsRepository/clickhouseSessionsRepository.server.ts:27-40
Timestamp: 2026-04-20T15:08:59.789Z
Learning: In `apps/webapp/app/services/sessionsRepository/clickhouseSessionsRepository.server.ts`, the cursor predicate in `listSessionIds` compares only `session_id` while the `ORDER BY` clause uses `(created_at, session_id)`. This is intentional and consistent with the same pattern in `ClickHouseRunsRepository` and the waitpoints repository. Do not flag this as a skip/duplicate pagination bug in isolation — any fix must land across all three repositories at once as a shared follow-up.
Learnt from: matt-aitken
Repo: triggerdotdev/trigger.dev PR: 2264
File: apps/webapp/app/services/runsRepository.server.ts:172-174
Timestamp: 2025-07-12T18:06:04.133Z
Learning: In apps/webapp/app/services/runsRepository.server.ts, the in-memory status filtering after fetching runs from Prisma is intentionally used as a workaround for ClickHouse data delays. This approach is acceptable because the result set is limited to a maximum of 100 runs due to pagination, making the performance impact negligible.
Learnt from: ericallam
Repo: triggerdotdev/trigger.dev PR: 3417
File: apps/webapp/app/services/sessionsReplicationService.server.ts:204-215
Timestamp: 2026-04-20T15:08:55.358Z
Learning: In `apps/webapp/app/services/sessionsReplicationService.server.ts` and `apps/webapp/app/services/runsReplicationService.server.ts`, the `getKey` function in `ConcurrentFlushScheduler` uses `${item.event}_${item.session.id}` / `${item.event}_${item.run.id}` respectively. This pattern is intentionally kept identical across both replication services for consistency. Any change to the deduplication key shape (e.g., keying solely by session/run id) must be applied to both services together, never to one service in isolation. Tracking as a cross-service follow-up.
Learnt from: ericallam
Repo: triggerdotdev/trigger.dev PR: 3368
File: apps/webapp/app/components/logs/LogsTaskFilter.tsx:135-163
Timestamp: 2026-04-16T14:21:17.695Z
Learning: In `triggerdotdev/trigger.dev` PR `#3368`, the `TaskIdentifier` table has a `@unique([runtimeEnvironmentId, slug])` DB constraint, guaranteeing one row per (environment, slug). In components like `apps/webapp/app/components/logs/LogsTaskFilter.tsx` and `apps/webapp/app/components/runs/v3/RunFilters.tsx`, using `key={item.slug}` for SelectItem list items is correct and unique. Do NOT flag `key={item.slug}` as potentially non-unique — the old duplicate-(slug, triggerSource) issue only existed with the legacy `DISTINCT` query, which this registry replaces.
📚 Learning: 2026-04-20T15:08:59.789Z
Learnt from: ericallam
Repo: triggerdotdev/trigger.dev PR: 3417
File: apps/webapp/app/services/sessionsRepository/clickhouseSessionsRepository.server.ts:27-40
Timestamp: 2026-04-20T15:08:59.789Z
Learning: In `apps/webapp/app/services/sessionsRepository/clickhouseSessionsRepository.server.ts`, the cursor predicate in `listSessionIds` compares only `session_id` while the `ORDER BY` clause uses `(created_at, session_id)`. This is intentional and consistent with the same pattern in `ClickHouseRunsRepository` and the waitpoints repository. Do not flag this as a skip/duplicate pagination bug in isolation — any fix must land across all three repositories at once as a shared follow-up.
Applied to files:
.server-changes/bulk-action-cursor-pagination.mdapps/webapp/app/services/runsRepository/runsRepository.server.tsapps/webapp/app/services/runsRepository/runsCursor.server.tsapps/webapp/test/runsRepositoryCursor.test.tsapps/webapp/app/services/runsRepository/clickhouseRunsRepository.server.ts
📚 Learning: 2025-07-12T18:06:04.133Z
Learnt from: matt-aitken
Repo: triggerdotdev/trigger.dev PR: 2264
File: apps/webapp/app/services/runsRepository.server.ts:172-174
Timestamp: 2025-07-12T18:06:04.133Z
Learning: In apps/webapp/app/services/runsRepository.server.ts, the in-memory status filtering after fetching runs from Prisma is intentionally used as a workaround for ClickHouse data delays. This approach is acceptable because the result set is limited to a maximum of 100 runs due to pagination, making the performance impact negligible.
Applied to files:
.server-changes/bulk-action-cursor-pagination.mdapps/webapp/app/services/runsRepository/runsRepository.server.tsapps/webapp/app/services/runsRepository/runsCursor.server.tsinternal-packages/clickhouse/src/taskRuns.tsapps/webapp/app/v3/services/bulk/BulkActionV2.server.tsapps/webapp/test/runsRepositoryCursor.test.tsapps/webapp/app/services/runsRepository/clickhouseRunsRepository.server.ts
📚 Learning: 2026-04-20T15:08:55.358Z
Learnt from: ericallam
Repo: triggerdotdev/trigger.dev PR: 3417
File: apps/webapp/app/services/sessionsReplicationService.server.ts:204-215
Timestamp: 2026-04-20T15:08:55.358Z
Learning: In `apps/webapp/app/services/sessionsReplicationService.server.ts` and `apps/webapp/app/services/runsReplicationService.server.ts`, the `getKey` function in `ConcurrentFlushScheduler` uses `${item.event}_${item.session.id}` / `${item.event}_${item.run.id}` respectively. This pattern is intentionally kept identical across both replication services for consistency. Any change to the deduplication key shape (e.g., keying solely by session/run id) must be applied to both services together, never to one service in isolation. Tracking as a cross-service follow-up.
Applied to files:
.server-changes/bulk-action-cursor-pagination.mdapps/webapp/app/services/runsRepository/runsCursor.server.tsapps/webapp/app/services/runsRepository/clickhouseRunsRepository.server.ts
📚 Learning: 2026-05-14T14:54:39.095Z
Learnt from: ericallam
Repo: triggerdotdev/trigger.dev PR: 3545
File: .server-changes/agent-view-sessions.md:10-10
Timestamp: 2026-05-14T14:54:39.095Z
Learning: In the `trigger.dev` repository, do not flag inconsistent dot vs slash notation in route/path strings inside `.server-changes/*.md` files. These markdown files are consumed verbatim into the changelog, so the mixed notation (e.g., `resources.orgs.../runs.$runParam/...`) is intentional and should be preserved as-is.
Applied to files:
.server-changes/bulk-action-cursor-pagination.md
📚 Learning: 2026-04-16T14:19:16.330Z
Learnt from: CR
Repo: triggerdotdev/trigger.dev PR: 0
File: apps/webapp/CLAUDE.md:0-0
Timestamp: 2026-04-16T14:19:16.330Z
Learning: Applies to apps/webapp/app/v3/services/{cancelTaskRun,batchTriggerV3}.server.ts : When editing services that branch on `RunEngineVersion` to support both V1 and V2 (e.g., `cancelTaskRun.server.ts`, `batchTriggerV3.server.ts`), only modify V2 code paths
Applied to files:
apps/webapp/app/services/runsRepository/runsRepository.server.tsapps/webapp/app/services/runsRepository/runsCursor.server.tsinternal-packages/clickhouse/src/taskRuns.tsapps/webapp/app/v3/services/bulk/BulkActionV2.server.tsapps/webapp/app/services/runsRepository/clickhouseRunsRepository.server.ts
📚 Learning: 2026-03-22T13:26:12.060Z
Learnt from: ericallam
Repo: triggerdotdev/trigger.dev PR: 3244
File: apps/webapp/app/components/code/TextEditor.tsx:81-86
Timestamp: 2026-03-22T13:26:12.060Z
Learning: In the triggerdotdev/trigger.dev codebase, do not flag `navigator.clipboard.writeText(...)` calls for `missing-await`/`unhandled-promise` issues. These clipboard writes are intentionally invoked without `await` and without `catch` handlers across the project; keep that behavior consistent when reviewing TypeScript/TSX files (e.g., usages like in `apps/webapp/app/components/code/TextEditor.tsx`).
Applied to files:
apps/webapp/app/services/runsRepository/runsRepository.server.tsapps/webapp/app/services/runsRepository/runsCursor.server.tsinternal-packages/clickhouse/src/taskRuns.tsapps/webapp/app/v3/services/bulk/BulkActionV2.server.tsapps/webapp/test/runsRepositoryCursor.test.tsapps/webapp/app/services/runsRepository/clickhouseRunsRepository.server.ts
📚 Learning: 2026-03-22T19:24:14.403Z
Learnt from: matt-aitken
Repo: triggerdotdev/trigger.dev PR: 3187
File: apps/webapp/app/v3/services/alerts/deliverErrorGroupAlert.server.ts:200-204
Timestamp: 2026-03-22T19:24:14.403Z
Learning: In the triggerdotdev/trigger.dev codebase, webhook URLs are not expected to contain embedded credentials/secrets (e.g., fields like `ProjectAlertWebhookProperties` should only hold credential-free webhook endpoints). During code review, if you see logging or inclusion of raw webhook URLs in error messages, do not automatically treat it as a credential-leak/secrets-in-logs issue by default—first verify the URL does not contain embedded credentials (for example, no username/password in the URL, no obvious secret/token query params or fragments). If the URL is credential-free per this project’s conventions, allow the logging.
Applied to files:
apps/webapp/app/services/runsRepository/runsRepository.server.tsapps/webapp/app/services/runsRepository/runsCursor.server.tsinternal-packages/clickhouse/src/taskRuns.tsapps/webapp/app/v3/services/bulk/BulkActionV2.server.tsapps/webapp/test/runsRepositoryCursor.test.tsapps/webapp/app/services/runsRepository/clickhouseRunsRepository.server.ts
📚 Learning: 2026-05-18T08:21:27.694Z
Learnt from: d-cs
Repo: triggerdotdev/trigger.dev PR: 3632
File: apps/webapp/sentry.server.ts:4-21
Timestamp: 2026-05-18T08:21:27.694Z
Learning: When handling Prisma error P1001 ("Can't reach database server") in TypeScript, don’t assume a single error shape. Prisma can surface P1001 via two different error classes/fields: `PrismaClientKnownRequestError` exposes it as `err.code === "P1001"` (common during mid-query connection drops), while `PrismaClientInitializationError` exposes it as `err.errorCode === "P1001"` (common on client startup failure). Therefore, predicates should use `err.code === "P1001" || err.errorCode === "P1001"`. Do not flag `err.code === "P1001"` as “unreachable/never matches,” as it is expected in production.
Applied to files:
apps/webapp/app/services/runsRepository/runsRepository.server.tsapps/webapp/app/services/runsRepository/runsCursor.server.tsinternal-packages/clickhouse/src/taskRuns.tsapps/webapp/app/v3/services/bulk/BulkActionV2.server.tsapps/webapp/test/runsRepositoryCursor.test.tsapps/webapp/app/services/runsRepository/clickhouseRunsRepository.server.ts
📚 Learning: 2026-05-18T08:21:27.694Z
Learnt from: d-cs
Repo: triggerdotdev/trigger.dev PR: 3632
File: apps/webapp/sentry.server.ts:4-21
Timestamp: 2026-05-18T08:21:27.694Z
Learning: When handling Prisma errors for P1001 ("Can't reach database server"), do not assume it only appears under a single property name. Prisma may surface P1001 via either `PrismaClientKnownRequestError` (`err.code === "P1001"`, e.g., mid-query connection drops) or `PrismaClientInitializationError` (`err.errorCode === "P1001"`, e.g., client startup connection failure). To reliably detect the condition, check `err.code === "P1001" || err.errorCode === "P1001"`, and avoid review rules that would incorrectly flag `err.code === "P1001"` as unreachable/never-matching.
Applied to files:
apps/webapp/app/services/runsRepository/runsRepository.server.tsapps/webapp/app/services/runsRepository/runsCursor.server.tsinternal-packages/clickhouse/src/taskRuns.tsapps/webapp/app/v3/services/bulk/BulkActionV2.server.tsapps/webapp/test/runsRepositoryCursor.test.tsapps/webapp/app/services/runsRepository/clickhouseRunsRepository.server.ts
📚 Learning: 2026-03-26T09:02:07.973Z
Learnt from: myftija
Repo: triggerdotdev/trigger.dev PR: 3274
File: apps/webapp/app/services/runsReplicationService.server.ts:922-924
Timestamp: 2026-03-26T09:02:07.973Z
Learning: When parsing Trigger.dev task run annotations in server-side services, keep `TaskRun.annotations` strictly conforming to the `RunAnnotations` schema from `trigger.dev/core/v3`. If the code already uses `RunAnnotations.safeParse` (e.g., in a `#parseAnnotations` helper), treat that as intentional/necessary for atomic, schema-accurate annotation handling. Do not recommend relaxing the annotation payload schema or using a permissive “passthrough” parse path, since the annotations are expected to be written atomically in one operation and should not contain partial/legacy payloads that would require a looser parser.
Applied to files:
apps/webapp/app/services/runsRepository/runsRepository.server.tsapps/webapp/app/services/runsRepository/runsCursor.server.tsapps/webapp/app/services/runsRepository/clickhouseRunsRepository.server.ts
📚 Learning: 2026-05-05T09:38:02.512Z
Learnt from: d-cs
Repo: triggerdotdev/trigger.dev PR: 3523
File: apps/webapp/app/routes/api.v3.batches.ts:178-181
Timestamp: 2026-05-05T09:38:02.512Z
Learning: When reviewing code that catches `ServiceValidationError` in `*.server.ts` files, do not blindly forward `error.status` to HTTP responses, because SVEs may be thrown with non-default statuses (e.g., 400/500) and forwarding them can cause client-visible behavioral regressions (e.g., surfacing 500s to clients). Prefer a safe default response status of `error.status ?? 422`, but only after confirming via the reachable call graph that the caught `ServiceValidationError` instances are expected to carry those non-default statuses; otherwise, normalize to `422` to avoid unexpected client-visible 5xx behavior.
Applied to files:
apps/webapp/app/services/runsRepository/runsRepository.server.tsapps/webapp/app/services/runsRepository/runsCursor.server.tsapps/webapp/app/v3/services/bulk/BulkActionV2.server.tsapps/webapp/app/services/runsRepository/clickhouseRunsRepository.server.ts
📚 Learning: 2026-05-12T21:04:05.815Z
Learnt from: ericallam
Repo: triggerdotdev/trigger.dev PR: 3542
File: apps/webapp/app/components/sessions/v1/SessionStatus.tsx:1-3
Timestamp: 2026-05-12T21:04:05.815Z
Learning: In this Remix + TypeScript codebase, do not flag a server/client boundary violation when a file imports only types from a module matching `*.server`.
Specifically, it’s safe to import types using `import type { Foo } from "*.server"` or `import { type Foo } from "*.server"` because TypeScript erases type-only imports at compile time and they emit no JavaScript, so they won’t cross the Remix server/client bundle boundary.
Only raise the boundary concern for value imports (e.g., `import { Foo }` without `type`, or `import Foo`), since those produce JavaScript output.
Applied to files:
apps/webapp/app/services/runsRepository/runsRepository.server.tsapps/webapp/app/services/runsRepository/runsCursor.server.tsapps/webapp/app/v3/services/bulk/BulkActionV2.server.tsapps/webapp/test/runsRepositoryCursor.test.tsapps/webapp/app/services/runsRepository/clickhouseRunsRepository.server.ts
📚 Learning: 2026-06-04T18:16:35.386Z
Learnt from: nicktrn
Repo: triggerdotdev/trigger.dev PR: 3836
File: apps/supervisor/src/backpressure/backpressureMonitor.ts:3-5
Timestamp: 2026-06-04T18:16:35.386Z
Learning: When reviewing TypeScript in this repo, apply the rule “prefer type aliases over interfaces” only to data/object shapes and union/intersection type modeling. If an interface is being used as a behavioral contract for collaborators to implement (e.g., method-shape interfaces that define required behavior, such as `BackpressureLogger` / `BackpressureSignalSource` in `apps/supervisor/src/backpressure/backpressureMonitor.ts`), keep it as an `interface` and do not flag it as a type-alias-vs-interface violation.
Applied to files:
apps/webapp/app/services/runsRepository/runsRepository.server.tsapps/webapp/app/services/runsRepository/runsCursor.server.tsinternal-packages/clickhouse/src/taskRuns.tsapps/webapp/app/v3/services/bulk/BulkActionV2.server.tsapps/webapp/test/runsRepositoryCursor.test.tsapps/webapp/app/services/runsRepository/clickhouseRunsRepository.server.ts
📚 Learning: 2026-04-20T15:09:12.730Z
Learnt from: ericallam
Repo: triggerdotdev/trigger.dev PR: 3417
File: internal-packages/clickhouse/src/sessions.ts:174-180
Timestamp: 2026-04-20T15:09:12.730Z
Learning: In `internal-packages/clickhouse/src/sessions.ts`, `getSessionTagsQueryBuilder` intentionally queries `trigger_dev.sessions_v1` WITHOUT `FINAL`, mirroring `getTaskRunTagsQueryBuilder` which queries `task_runs_v2` without `FINAL`. The DISTINCT arrayJoin tag-listing read can tolerate an occasional stale tag from a superseded ReplacingMergeTree row; the FINAL cost on a large table is considered not worth it. If FINAL is ever added, both tag query builders (sessions and runs) will be updated together. Do not flag the missing FINAL in either tag query builder as a consistency or stale-data issue.
Applied to files:
internal-packages/clickhouse/src/taskRuns.ts
📚 Learning: 2026-04-16T14:19:16.330Z
Learnt from: CR
Repo: triggerdotdev/trigger.dev PR: 0
File: apps/webapp/CLAUDE.md:0-0
Timestamp: 2026-04-16T14:19:16.330Z
Learning: Applies to apps/webapp/{app/v3/services/triggerTask.server.ts,app/v3/services/batchTriggerV3.server.ts} : In `triggerTask.server.ts` and `batchTriggerV3.server.ts`, do NOT add database queries. Task defaults (TTL, etc.) are resolved via `backgroundWorkerTask.findFirst()` in the queue concern (`queues.server.ts`). Piggyback on the existing query instead of adding new ones
Applied to files:
internal-packages/clickhouse/src/taskRuns.tsapps/webapp/app/v3/services/bulk/BulkActionV2.server.ts
📚 Learning: 2026-05-12T21:04:13.550Z
Learnt from: ericallam
Repo: triggerdotdev/trigger.dev PR: 3542
File: apps/webapp/app/routes/api.v1.deployments.current.ts:19-32
Timestamp: 2026-05-12T21:04:13.550Z
Learning: In triggerdotdev/trigger.dev, `ApiDeploymentListResponseItem` (packages/core/src/v3/schemas/api.ts) does NOT include an `updatedAt` field. Its fields are: id, createdAt, shortCode, version, runtime, runtimeVersion, status, deployedAt, git, error. Do not flag the `api.v1.deployments.current` loader for a missing `updatedAt` field — the response shape matches the schema as-is.
Applied to files:
internal-packages/clickhouse/src/taskRuns.ts
📚 Learning: 2026-03-25T15:29:25.889Z
Learnt from: CR
Repo: triggerdotdev/trigger.dev PR: 0
File: .cursor/rules/writing-tasks.mdc:0-0
Timestamp: 2026-03-25T15:29:25.889Z
Learning: Applies to **/trigger/**/*.{ts,tsx,js,jsx} : Use `task()` from `trigger.dev/sdk` for basic task definitions with `id` and `run` properties
Applied to files:
internal-packages/clickhouse/src/taskRuns.ts
📚 Learning: 2026-04-13T21:44:00.032Z
Learnt from: ericallam
Repo: triggerdotdev/trigger.dev PR: 3368
File: apps/webapp/app/services/taskIdentifierRegistry.server.ts:24-67
Timestamp: 2026-04-13T21:44:00.032Z
Learning: In `apps/webapp/app/services/taskIdentifierRegistry.server.ts`, the sequential upsert/updateMany/findMany writes in `syncTaskIdentifiers` are intentionally NOT wrapped in a Prisma transaction. This function runs only during deployment-change events (low-concurrency path), and any partial `isInLatestDeployment` state is acceptable because it self-corrects on the next deployment. Do not flag this as a missing-transaction/atomicity issue in future reviews.
Applied to files:
apps/webapp/app/v3/services/bulk/BulkActionV2.server.ts
📚 Learning: 2026-03-10T17:56:20.938Z
Learnt from: samejr
Repo: triggerdotdev/trigger.dev PR: 3201
File: apps/webapp/app/v3/services/setSeatsAddOn.server.ts:25-29
Timestamp: 2026-03-10T17:56:20.938Z
Learning: Do not implement local userId-to-organizationId authorization checks inside org-scoped service classes (e.g., SetSeatsAddOnService, SetBranchesAddOnService) in the web app. Rely on route-layer authentication (requireUserId(request)) and org membership enforcement via the _app.orgs.$organizationSlug layout route. Any userId/organizationId that reaches these services from org-scoped routes has already been validated. Apply this pattern across all org-scoped services to avoid redundant auth checks and maintain consistency.
Applied to files:
apps/webapp/app/v3/services/bulk/BulkActionV2.server.ts
📚 Learning: 2026-03-29T19:16:28.864Z
Learnt from: nicktrn
Repo: triggerdotdev/trigger.dev PR: 3291
File: apps/webapp/app/v3/featureFlags.ts:53-65
Timestamp: 2026-03-29T19:16:28.864Z
Learning: When reviewing TypeScript code that uses Zod v3, treat `z.coerce.*()` schemas as their direct Zod type (e.g., `z.coerce.boolean()` returns a `ZodBoolean` with `_def.typeName === "ZodBoolean"`) rather than a `ZodEffects`. Only `.preprocess()`, `.refine()`/`.superRefine()`, and `.transform()` are expected to wrap schemas in `ZodEffects`. Therefore, in reviewers’ logic like `getFlagControlType`, do not flag/unblock failures that require unwrapping `ZodEffects` when the input schema is a `z.coerce.*` schema.
Applied to files:
apps/webapp/app/v3/services/bulk/BulkActionV2.server.ts
📚 Learning: 2026-05-14T08:21:07.614Z
Learnt from: d-cs
Repo: triggerdotdev/trigger.dev PR: 3614
File: apps/webapp/app/v3/mollifier/mollifierGate.server.ts:48-52
Timestamp: 2026-05-14T08:21:07.614Z
Learning: When using Trigger.dev v3 feature flags in the webapp, prefer the existing per-org gating mechanism supported by `flag()` via the `overrides` argument. Pass `Organization.featureFlags` (from `environment.organization.featureFlags`) as the `overrides` value; overrides must take precedence over the global `featureFlag` row. Do not require schema changes or add an `orgId` field to `FlagsOptions` for per-org gating—use the overrides pattern consistently (e.g., in gate flows like `resolveOrgFlag` and any server code that threads `environment.organization.featureFlags` into the gate call).
Applied to files:
apps/webapp/app/v3/services/bulk/BulkActionV2.server.ts
📚 Learning: 2026-03-02T12:43:25.254Z
Learnt from: CR
Repo: triggerdotdev/trigger.dev PR: 0
File: internal-packages/run-engine/CLAUDE.md:0-0
Timestamp: 2026-03-02T12:43:25.254Z
Learning: Applies to internal-packages/run-engine/src/engine/tests/**/*.test.ts : Implement tests for RunEngine in `src/engine/tests/` using testcontainers for Redis and PostgreSQL containerization
Applied to files:
apps/webapp/test/runsRepositoryCursor.test.ts
📚 Learning: 2026-04-16T13:45:22.317Z
Learnt from: ericallam
Repo: triggerdotdev/trigger.dev PR: 3368
File: apps/webapp/test/engine/taskIdentifierRegistry.test.ts:3-19
Timestamp: 2026-04-16T13:45:22.317Z
Learning: In `apps/webapp/test/engine/taskIdentifierRegistry.test.ts`, the `vi.mock` calls for `~/services/taskIdentifierCache.server` (stubbing `getTaskIdentifiersFromCache` and `populateTaskIdentifierCache`), `~/models/task.server` (stubbing `getAllTaskIdentifiers`), and `~/db.server` (stubbing `prisma` and `$replica`) are intentional. The suite uses real Postgres via testcontainers for all `TaskIdentifier` DB operations, but isolates the Redis cache layer and legacy query fallback as separate concerns not exercised in this test file. Do not flag these mocks as violations of the no-mocks policy in future reviews.
Applied to files:
apps/webapp/test/runsRepositoryCursor.test.ts
📚 Learning: 2026-04-07T14:12:18.946Z
Learnt from: matt-aitken
Repo: triggerdotdev/trigger.dev PR: 3331
File: apps/webapp/test/engine/batchPayloads.test.ts:5-24
Timestamp: 2026-04-07T14:12:18.946Z
Learning: In `apps/webapp/test/engine/batchPayloads.test.ts`, using `vi.mock` for `~/v3/objectStore.server` (stubbing `hasObjectStoreClient` and `uploadPacketToObjectStore`), `~/env.server` (overriding offload thresholds), and `~/v3/tracer.server` (stubbing `startActiveSpan`) is intentional and acceptable. Simulating controlled transient upload failures (e.g., fail N times then succeed) to verify `p-retry` behavior cannot be reproduced with real services or testcontainers. This file is an explicit exception to the repo's general no-mocks policy.
Applied to files:
apps/webapp/test/runsRepositoryCursor.test.ts
📚 Learning: 2025-11-27T16:26:37.432Z
Learnt from: CR
Repo: triggerdotdev/trigger.dev PR: 0
File: .github/copilot-instructions.md:0-0
Timestamp: 2025-11-27T16:26:37.432Z
Learning: Applies to **/*.{test,spec}.{ts,tsx} : Use vitest for all tests in the Trigger.dev repository
Applied to files:
apps/webapp/test/runsRepositoryCursor.test.ts
📚 Learning: 2025-11-27T16:26:44.496Z
Learnt from: CR
Repo: triggerdotdev/trigger.dev PR: 0
File: .cursor/rules/executing-commands.mdc:0-0
Timestamp: 2025-11-27T16:26:44.496Z
Learning: For running tests, navigate into the package directory and run `pnpm run test --run` to enable single-file test execution (e.g., `pnpm run test ./src/engine/tests/ttl.test.ts --run`)
Applied to files:
apps/webapp/test/runsRepositoryCursor.test.ts
📚 Learning: 2026-03-03T13:07:33.177Z
Learnt from: ericallam
Repo: triggerdotdev/trigger.dev PR: 3166
File: internal-packages/run-engine/src/batch-queue/tests/index.test.ts:711-713
Timestamp: 2026-03-03T13:07:33.177Z
Learning: In `internal-packages/run-engine/src/batch-queue/tests/index.test.ts`, test assertions for rate limiter stubs can use `toBeGreaterThanOrEqual` rather than exact equality (`toBe`) because the consumer loop may call the rate limiter during empty pops in addition to actual item processing, and this over-calling is acceptable in integration tests.
Applied to files:
apps/webapp/test/runsRepositoryCursor.test.ts
📚 Learning: 2026-06-02T21:20:56.997Z
Learnt from: CR
Repo: triggerdotdev/trigger.dev PR: 0
File: AGENTS.md:0-0
Timestamp: 2026-06-02T21:20:56.997Z
Learning: Applies to **/*.test.{js,ts,tsx} : Use vitest for unit testing
Applied to files:
apps/webapp/test/runsRepositoryCursor.test.ts
📚 Learning: 2026-03-02T12:43:43.173Z
Learnt from: CR
Repo: triggerdotdev/trigger.dev PR: 0
File: packages/redis-worker/CLAUDE.md:0-0
Timestamp: 2026-03-02T12:43:43.173Z
Learning: Applies to packages/redis-worker/**/redis-worker/**/*.{test,spec}.{ts,tsx} : Use testcontainers for Redis in test files for redis-worker
Applied to files:
apps/webapp/test/runsRepositoryCursor.test.ts
📚 Learning: 2026-06-01T15:01:35.175Z
Learnt from: d-cs
Repo: triggerdotdev/trigger.dev PR: 3756
File: apps/webapp/app/v3/services/resetIdempotencyKey.server.ts:65-94
Timestamp: 2026-06-01T15:01:35.175Z
Learning: In `apps/webapp/app/v3/services/resetIdempotencyKey.server.ts` (triggerdotdev/trigger.dev), a transient `buffer.resetIdempotency()` failure when `pgCount > 0` does NOT warrant a 503 and should return success. The mollifier `ack` and `fail` Lua scripts always DEL the idempotency lookup key as part of the run's natural lifecycle (drain→ack or terminal→fail or cancel-bifurcation), so stale buffered idempotency lookups converge automatically without caller retries. Only when `pgCount === 0 && bufferResetFailed` is a 503 appropriate, because then the run's existence is genuinely unobservable (the buffer outage hides a potentially matching buffered run). The test "returns success when PG cleared >=1 run, even if the buffer reset throws" documents this contract explicitly.
Applied to files:
apps/webapp/test/runsRepositoryCursor.test.ts
📚 Learning: 2026-06-02T21:20:56.997Z
Learnt from: CR
Repo: triggerdotdev/trigger.dev PR: 0
File: AGENTS.md:0-0
Timestamp: 2026-06-02T21:20:56.997Z
Learning: Applies to **/*.test.{js,ts,tsx} : Tests should avoid mocks or stubs and use helpers from `internal/testcontainers` when Redis or Postgres are needed
Applied to files:
apps/webapp/test/runsRepositoryCursor.test.ts
📚 Learning: 2026-05-07T12:25:18.271Z
Learnt from: d-cs
Repo: triggerdotdev/trigger.dev PR: 3531
File: apps/webapp/test/sentryTraceContext.server.test.ts:9-47
Timestamp: 2026-05-07T12:25:18.271Z
Learning: In the triggerdotdev/trigger.dev webapp test suite, it is acceptable to leave `createInMemoryTracing()` calls that register a global `NodeTracerProvider` without `afterEach`/`afterAll` teardown. Do not flag this as a test-ordering risk when the code follows the established pattern used across webapp tests (e.g., replication service/benchmark/backfiller tests). This is considered safe because `trace.getActiveSpan()` when called outside a `context.with(...)` block reads `AsyncLocalStorage.getStore()` (undefined when no `run()` scope exists), so it falls back to `ROOT_CONTEXT` with no attached span—regardless of which provider is registered.
Applied to files:
apps/webapp/test/runsRepositoryCursor.test.ts
📚 Learning: 2026-05-18T14:40:02.173Z
Learnt from: ericallam
Repo: triggerdotdev/trigger.dev PR: 3658
File: packages/core/src/v3/realtimeStreams/manager.test.ts:1-147
Timestamp: 2026-05-18T14:40:02.173Z
Learning: In the triggerdotdev/trigger.dev repo, the policy “Never mock anything — use testcontainers instead” should only be enforced for integration tests that interact with real external services (e.g., Redis, Postgres) via actual infrastructure. For unit tests that exercise pure in-memory logic (e.g., cache semantics) it is OK to stub collaborators such as `ApiClient` using Vitest (`vi.fn()`) to assert call counts or control behavior. Do not flag `vi.fn()`-based `ApiClient` stubs in unit tests as violations of the testcontainers policy.
Applied to files:
apps/webapp/test/runsRepositoryCursor.test.ts
📚 Learning: 2026-05-28T20:02:10.647Z
Learnt from: myftija
Repo: triggerdotdev/trigger.dev PR: 3772
File: apps/webapp/test/findOrCreateBackgroundWorker.test.ts:1-1
Timestamp: 2026-05-28T20:02:10.647Z
Learning: In the triggerdotdev/trigger.dev monorepo, for the `apps/webapp` package use the established convention of storing Vitest tests (unit, integration, and e2e) under `apps/webapp/test/` rather than colocating them next to source files. Do not flag files located in `apps/webapp/test/` as violating any rule that says to colocate tests with source.
Applied to files:
apps/webapp/test/runsRepositoryCursor.test.ts
📚 Learning: 2026-04-17T13:20:14.259Z
Learnt from: ericallam
Repo: triggerdotdev/trigger.dev PR: 3333
File: apps/webapp/app/services/runsReplicationService.server.ts:655-685
Timestamp: 2026-04-17T13:20:14.259Z
Learning: In `apps/webapp/app/services/runsReplicationService.server.ts`, the per-ClickHouse-group inserts inside `#flushBatch` are intentionally serialized (sequential) by default. Parallelizing group flushes causes Linux socket write-buffer pressure that required kernel tuning (`net.ipv4.tcp_wmem` set to `4096 20480 8388608`). Parallel flushing may be exposed as an opt-in via a `groupFlushStrategy` env var/option, but sequential must remain the safe default. Do not flag the sequential loop as a performance issue without acknowledging this constraint.
Applied to files:
apps/webapp/app/services/runsRepository/clickhouseRunsRepository.server.ts
📚 Learning: 2025-06-14T08:07:46.625Z
Learnt from: ericallam
Repo: triggerdotdev/trigger.dev PR: 2175
File: apps/webapp/app/services/environmentMetricsRepository.server.ts:202-207
Timestamp: 2025-06-14T08:07:46.625Z
Learning: In apps/webapp/app/services/environmentMetricsRepository.server.ts, the ClickHouse methods (getTaskActivity, getCurrentRunningStats, getAverageDurations) intentionally do not filter by the `tasks` parameter at the ClickHouse level, even though the tasks parameter is accepted by the public methods. This is done on purpose as there is not much benefit from adding that filtering at the ClickHouse layer.
Applied to files:
apps/webapp/app/services/runsRepository/clickhouseRunsRepository.server.ts
🔇 Additional comments (8)
apps/webapp/app/services/runsRepository/runsCursor.server.ts (1)
1-39: LGTM!apps/webapp/app/services/runsRepository/clickhouseRunsRepository.server.ts (3)
24-81: LGTM!
88-105: LGTM!
130-172: LGTM!apps/webapp/app/services/runsRepository/runsRepository.server.ts (1)
133-141: LGTM!Also applies to: 181-196
apps/webapp/app/v3/services/bulk/BulkActionV2.server.ts (1)
162-172: LGTM!Also applies to: 295-295
apps/webapp/test/runsRepositoryCursor.test.ts (1)
1-299: LGTM!.server-changes/bulk-action-cursor-pagination.md (1)
1-16: LGTM!
Comment on lines
367
to
373
| export const TaskRunV2QueryResult = z.object({ | ||
| run_id: z.string(), | ||
| // Milliseconds since epoch. Returned as a JSON number because the client sets | ||
| // output_format_json_quote_64bit_integers: 0. Used to build composite keyset | ||
| // cursors over (created_at, run_id) — see runsRepository.server.ts. | ||
| created_at_ms: z.number().int(), | ||
| }); |
Contributor
There was a problem hiding this comment.
Critical: Schema change breaks getPendingVersionIdsQueryBuilder.
The modified TaskRunV2QueryResult schema now requires both run_id and created_at_ms, but getPendingVersionIdsQueryBuilder (lines 394-404) only selects run_id. When that function executes, Zod validation will fail because created_at_ms is missing from the ClickHouse result, causing a runtime error.
🔧 Recommended fix
Create a separate schema for pending version queries:
+export const PendingVersionIdQueryResult = z.object({
+ run_id: z.string(),
+});
+
+export type PendingVersionIdQueryResult = z.infer<typeof PendingVersionIdQueryResult>;
+
export const TaskRunV2QueryResult = z.object({
run_id: z.string(),
// Milliseconds since epoch. Returned as a JSON number because the client sets
// output_format_json_quote_64bit_integers: 0. Used to build composite keyset
// cursors over (created_at, run_id) — see runsRepository.server.ts.
created_at_ms: z.number().int(),
});Then update getPendingVersionIdsQueryBuilder:
export function getPendingVersionIdsQueryBuilder(
ch: ClickhouseReader,
settings?: ClickHouseSettings
) {
return ch.queryBuilder({
name: "getPendingVersionIds",
baseQuery: "SELECT run_id FROM trigger_dev.task_runs_v2",
- schema: TaskRunV2QueryResult,
+ schema: PendingVersionIdQueryResult,
settings,
});
}
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
ClickHouseRunsRepository.listRunIds/listRunsorder results by the composite key(created_at, run_id), but the cursor predicate cut onrun_idalone:This is only sound when
run_idlexicographic order matchescreated_atorder.run_ids are cuids — only coarsely time-sortable — so when a burst of runs is created within a sub-second window, the two orders can diverge. When they do, the next-page predicate (run_id < cursor, wherecursoris the last page element = the smallestcreated_at, not necessarily the smallestrun_id):For bulk replay this caused runs to be replayed more than once (replay has no idempotency guard). For the dashboard and the
runs.listAPI it could silently repeat or skip runs at page boundaries.Fix
Make the cursor predicate match the composite ordering:
(created_at, run_id)key asv2_<createdAtMs>_<runId>, and the query cuts on the matching tuple —(created_at, run_id) < (…)forward /> (…)backward.ORDER BYis unchanged, so the query stays aligned with the table's primary key — no performance regression (the tuple range predicate is actually more index-friendly thanrun_id <alone).pagination.next/pagination.previousback), so this needs no client/SDK update. Legacy bare-run_idcursors decode to the oldrun_id-only predicate, so in-flight cursors keep working and drain naturally.listRunIdsWithCursorfor forward-only batch iteration (bulk actions), so thecreated_atcomponent is sourced from the same query that orders the rows.getTaskRunsQueryBuildernow also selectstoUnixTimestamp64Milli(created_at) AS created_at_ms.Tests
New
runsRepositoryCursor.test.ts(testcontainer-backed, real Postgres→ClickHouse replication):run_idorder is the reverse ofcreated_atorder (reproduces the duplicate/skip bug — fails onmain),run_idcursor still uses the old predicate (backwards compatibility).All existing
runsRepositorysuites (15 tests) still pass.Notes
listRuns' backward display-slicing (rows.slice(1, size+1)whenhasMore) has an off-by-one that can return a straddled page. Tracked separately.🤖 Generated with Claude Code