Remove module indexing by habdelra · Pull Request #3984 · cardstack/boxel

habdelra · 2026-02-11T23:59:57Z

This PR removes module indexing. This PR is meant to be backwards compatible so that if there are still module index entries we can ignore them safely.

After this PR gets deployed to staging and production, I'll create a followup PR to remove all the module index entries and streamline the queries since we can safely assume those wont exist.

github-actions · 2026-02-12T00:08:05Z

Preview deployments

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c4d4043e92

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

packages/runtime-common/index-writer.ts

Copilot

Pull request overview

This PR removes “module” rows from the search index flow while remaining tolerant of existing type='module' rows in boxel_index. Module dependency/error information is now sourced from the modules cache (via DefinitionLookup) and instance deps are expanded transitively so module rows are no longer needed for invalidation.

Changes:

Stop writing/querying type='module' index entries; filter out existing module rows in boxel_index queries and counts.
Expand instance deps (and instance error deps) with transitive module dependencies using the module cache; switch dependency-error collection to read from cached module errors.
Plumb definitionLookup through worker/task args and add DefinitionLookup.getModuleCacheEntries() to support batched cache reads.

Reviewed changes

Copilot reviewed 18 out of 18 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
packages/runtime-common/worker.ts	Creates and passes a `CachingDefinitionLookup` into task args.
packages/runtime-common/tasks/index.ts	Extends `TaskArgs` to include `definitionLookup`.
packages/runtime-common/tasks/indexer.ts	Passes `definitionLookup` and `realmOwnerUserId` into `IndexRunner`.
packages/runtime-common/index-runner.ts	Removes module indexing; expands instance deps transitively via module cache; reads module errors from cache instead of index.
packages/runtime-common/definition-lookup.ts	Exports cache types and adds `getModuleCacheEntries()` for batched module cache retrieval.
packages/runtime-common/index-writer.ts	Removes module entry support; filters out module rows in reads/copy/counts; rejects `module-error` writes.
packages/runtime-common/index-structure.ts	Removes `'module'` from `BoxelIndexTable.type`.
packages/runtime-common/index-query-engine.ts	Removes `getModule()` and associated module result types.
packages/runtime-common/realm-index-query-engine.ts	Removes `module()` query surface.
packages/runtime-common/realm-index-updater.ts	Drops module stats fields from initializer.
packages/realm-server/worker-manager.ts	Stops emitting `module-error` index entries for failed paths.
packages/realm-server/tests/server-endpoints/maintenance-endpoints-test.ts	Updates expected stats shape (no module counters).
packages/realm-server/tests/indexing-test.ts	Updates tests away from module index queries; validates behavior via instance deps and module cache rows.
packages/realm-server/tests/full-reindex-test.ts	Updates task args construction for `definitionLookup`.
packages/realm-server/tests/billing-test.ts	Updates task args construction for `definitionLookup`.
packages/host/tests/unit/index-writer-test.ts	Updates invalidation expectations to rely on instance deps instead of module rows; adjusts copy assertions.
packages/host/tests/unit/index-query-engine-test.ts	Adds `getModuleCacheEntries` to the mock `DefinitionLookup`.
packages/host/tests/integration/components/ai-module-creation-test.gts	Switches from module index assertions to file index assertions for module files.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

packages/runtime-common/index-runner.ts

github-actions · 2026-02-12T00:18:01Z

Host Test Results

1 files ±0 1 suites ±0 1h 38m 59s ⏱️ -54s
1 842 tests +4 1 829 ✅ +4 13 💤 ±0 0 ❌ ±0
1 857 runs +4 1 844 ✅ +4 13 💤 ±0 0 ❌ ±0

Results for commit 730cad3. ± Comparison against base commit cd80b2b.

This pull request removes 6 and adds 10 tests. Note that renamed tests count towards both.

Chrome ‑ Unit | index-writer: can get compiled module and source from WIP index
Chrome ‑ Unit | index-writer: can get compiled module and source when requested with file extension
Chrome ‑ Unit | index-writer: can get compiled module and source when requested without file extension
Chrome ‑ Unit | index-writer: can get error doc for module
Chrome ‑ Unit | index-writer: can perform invalidations for a module entry
Chrome ‑ Unit | index-writer: returns undefined when getting a deleted module

Chrome ‑ Integration | operator-mode | card catalog: Recents section filters cards by search key
Chrome ‑ Integration | operator-mode | card catalog: Recents section is present when search sheet is open (expanded mode)
Chrome ‑ Integration | operator-mode | card catalog: SearchResultHeader is hidden in compact (prompt) mode
Chrome ‑ Integration | operator-mode | card catalog: Show more button reveals more cards when search has many results
Chrome ‑ Integration | operator-mode | card catalog: Show only focuses a section and collapses others
Chrome ‑ Integration | operator-mode | card catalog: compact mode shows no full header and recents remain clickable
Chrome ‑ Integration | operator-mode | card catalog: empty state shows when URL has no card
Chrome ‑ Integration | operator-mode | card catalog: search results are grouped by realm with section headers
Chrome ‑ Integration | operator-mode | card catalog: view toggle updates layout (grid vs strip)
Chrome ‑ Unit | index-writer: can perform invalidations for a module change via instance deps

♻️ This comment has been updated with latest results.

Copilot

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

Copilot

Pull request overview

Copilot reviewed 25 out of 25 changed files in this pull request and generated 4 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

packages/postgres/migrations/1770889690032_index-row-size-max-error.js

packages/postgres/scripts/convert-to-sqlite.ts

packages/runtime-common/index-writer.ts

packages/realm-server/tests/indexing-test.ts

…ctions preventing tests from running on top of one another

…-indexing

lukemelia · 2026-02-16T17:53:52Z

packages/realm-server/server.ts

+    let moduleExtensions = ['.gts', '.ts', '.js'];
+    for (let extension of moduleExtensions) {
+      if (existsSync(join(realm.dir, `${localPath}${extension}`))) {
+        return true;
+      }
+      if (existsSync(join(realm.dir, localPath, `index${extension}`))) {
+        return true;
+      }
+    }


This is up to 6 IO ops on a hot path (serving a card). Should we consider a cache, or consider doing this in the case of a database miss only?

fixed in 730cad3

lukemelia · 2026-02-16T17:56:00Z

packages/realm-server/tests/indexing-test.ts

+        if (surfacedAuthorizationError) {
+          assert.strictEqual(
+            stats.instanceErrors,
+            1,
+            'instance errors surfaced',
+          );
+          assert.strictEqual(stats.instancesIndexed, 0, 'no instances indexed');
+        } else {
+          assert.true(
+            indexingFailedBeforeStatsWereProduced,
+            `indexing failed before stats were produced: ${JSON.stringify(
+              stats,
+            )}`,
+          );
+        }


Seems weird to have a conditional like this in the test. Shouldn't we know what's going to happen?

fixed in 730cad3

lukemelia · 2026-02-16T17:57:36Z

packages/runtime-common/cache/module-cache-invalidation.ts

+const importSpecifierExpression =
+  /(?:import|export)\s+(?:[^'"`]*?\sfrom\s+)?['"]([^'"]+)['"]|import\(\s*['"]([^'"]+)['"]\s*\)/g;


Have we always used a regexp for this kind of thing?

actually no. we do have a module syntax component in runtime common that uses an AST. let me refactor

fixed in 730cad3

Remove module indexing

c4d4043

habdelra requested a review from Copilot February 12, 2026 00:00

Copilot started reviewing on behalf of habdelra February 12, 2026 00:00 View session

chatgpt-codex-connector bot reviewed Feb 12, 2026

View reviewed changes

packages/runtime-common/index-writer.ts Show resolved Hide resolved

Copilot AI reviewed Feb 12, 2026

View reviewed changes

packages/runtime-common/index-runner.ts Outdated Show resolved Hide resolved

packages/runtime-common/index-runner.ts Outdated Show resolved Hide resolved

habdelra added 3 commits February 12, 2026 09:28

updated based on code review feedback

adff75c

fixing module dependency invalidation test failures

6a8da53

fix broken test

c9ed4be

habdelra requested a review from Copilot February 12, 2026 16:39

Copilot started reviewing on behalf of habdelra February 12, 2026 16:40 View session

Copilot AI reviewed Feb 12, 2026

View reviewed changes

fix tests and fix sqlite conversion

ea64757

habdelra requested a review from Copilot February 12, 2026 19:54

Copilot started reviewing on behalf of habdelra February 12, 2026 19:55 View session

Copilot AI reviewed Feb 12, 2026

View reviewed changes

habdelra added 7 commits February 12, 2026 15:14

better migration script

684d91a

remove unnecessary code

9fce939

fix based on co-pilot review

45d5dc5

fix lint

4f250b8

codex 5.3 likes to do work while running tests so need to give instru…

7ecc7be

…ctions preventing tests from running on top of one another

fix tests

835b960

fix test

92c61f4

habdelra force-pushed the cs-9987-remove-module-indexing branch from d6d1894 to 92c61f4 Compare February 13, 2026 01:04

habdelra added 5 commits February 12, 2026 20:53

debug

4eaf5a4

more visibility into failing start scripts

13e8410

Merge remote-tracking branch 'origin/main' into cs-9987-remove-module…

c4e3aa9

…-indexing

add more logs for CI

6c78fb5

more logging

c0c6b53

habdelra added 2 commits February 12, 2026 22:31

debug with skip submission in host tests

4a40c0e

skip submission realm setup in host and matrix tests

8f39cb6

habdelra force-pushed the cs-9987-remove-module-indexing branch from d72f868 to 8f39cb6 Compare February 13, 2026 04:40

habdelra added 7 commits February 12, 2026 23:53

cleanup

4e046ed

fix matrix test

85d957f

Merge remote-tracking branch 'origin/main' into cs-9987-remove-module…

c64558f

…-indexing

server start script cli annotation

2973d15

fixing tests and removing unnecessary column in modules table

99a64f0

test timeout

8400839

fix test flakiness

0f28771

habdelra requested a review from a team February 13, 2026 19:39

habdelra mentioned this pull request Feb 16, 2026

Remove obsolete module entries from index #4001

Merged

lukemelia approved these changes Feb 16, 2026

View reviewed changes

fixed issues based on review feedback

730cad3

lukemelia approved these changes Feb 16, 2026

View reviewed changes

habdelra merged commit 1fa23fa into main Feb 16, 2026
106 checks passed

		const importSpecifierExpression =
		/(?:import\|export)\s+(?:[^'"`]?\sfrom\s+)?['"]([^'"]+)['"]\|import\(\s['"]([^'"]+)['"]\s*\)/g;

Comments

Conversation

habdelra commented Feb 11, 2026

Uh oh!

github-actions bot commented Feb 12, 2026

Preview deployments

Uh oh!

chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Feb 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Host Test Results

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

lukemelia Feb 16, 2026

Choose a reason for hiding this comment

Uh oh!

habdelra Feb 16, 2026

Choose a reason for hiding this comment

Uh oh!

lukemelia Feb 16, 2026

Choose a reason for hiding this comment

Uh oh!

habdelra Feb 16, 2026

Choose a reason for hiding this comment

Uh oh!

lukemelia Feb 16, 2026

Choose a reason for hiding this comment

Uh oh!

habdelra Feb 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

habdelra Feb 16, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

github-actions bot commented Feb 12, 2026 •

edited

Loading

habdelra Feb 16, 2026 •

edited

Loading