Add normalized hash-based test IDs to draft2020-12/enum.json (POC for #698) #796

AnirudhJindal · 2025-11-25T10:21:18Z

Context

This is a small proof-of-concept for #698, implementing option 3 from our discussion:

an MD5 hash of the normalized schema + test data + valid flag.

The goal is:

Same test across dialects → same ID
IDs change automatically when the semantics of a test change
Fully checkable in CI by recomputing the ID and comparing

normalize.js

Imported and adapted the AST-builder gist from @jdesrosiers.
Added a handler for the dialect-specific keyword draft-2020-12/dynamicRef.
Stripped top-level file: $id during normalization so registerSchema() doesn’t reject test schemas.
Exposed normalize() to return a stable, dialect-aware, $ref-resolved schema representation.

load-remotes.js

Recursively loaded all remotes/ JSON files and registered them under http://localhost:1234/....
Added deduplication so the same remote URL is never registered twice.
Ensured remotes only load for matching dialects.

add-test-ids.js

For each test missing an id, normalized its schema.
Generated an ID using md5(normalizedSchema + data + valid).
Wrote the updated tests back to disk (in this PR: only enum.json).

check-test-ids.js

Re-normalized schema and recomputed IDs for each test.
Reported missing, mismatched, or duplicate IDs.
Exited with error when inconsistencies were found (CI-ready behavior).

enum.json

Added id fields to each test using the MD5-based formula.
No other changes to schema structure or test contents.

How to run

# Add IDs to enum.json
node scripts/add-test-ids.js tests/draft2020-12/enum.json

# Verify IDs within draft2020-12
node scripts/check-test-ids.js tests/draft2020-12

For now, only enum.json has IDs. The checker will still report "Missing ID"
for other files in draft2020-12

Questions for reviewers

If this direction looks good, I can:

Extend the tooling as needed based on feedback

Add a small GitHub Action that runs check-test-ids.js on the touched version

Then start updating more files, one small PR at a time.

…POC for json-schema-org#698)

jdesrosiers · 2025-11-25T18:43:37Z

This should have been opened as draft PR. I've converted it this time. Next time you open a PR that's a work in progress for the purpose of getting feedback, a draft PR is the way to go.

jdesrosiers

I've reviewed the first file. I'll have to get to the rest another time, but that should help clear up some misunderstandings and hopefully, you can apply some of that to the rest as well.

One more thing I noticed is that you're using the wrong indentation for JSON files. In this repo, the convention is 4 spaces. Make sure you're matching that convention or it makes your changes very difficult to review. (2 spaces for JavaScript is fine. Just JSON needs to be 4 spaces.)

jdesrosiers · 2025-11-25T18:47:03Z

scripts/add-test-ids.js

+  if (!Array.isArray(tests)) {
+    console.log("Expected an array at top level, got:", typeof tests);
+    return;
+  }


This shouldn't be necessary. There's already automation to check the test file against the schema. By the time we run this script, we should be confident that the test file is in the right format.

jdesrosiers · 2025-11-25T18:47:45Z

scripts/add-test-ids.js

+  }
+
+  for (const testCase of tests) {
+    if (!Array.isArray(testCase.tests)) continue;


More unnecessary defensive programming.

jdesrosiers · 2025-11-25T18:49:21Z

scripts/add-test-ids.js

+  for (const testCase of tests) {
+    if (!Array.isArray(testCase.tests)) continue;
+
+    const dialectUri = getDialectUri(testCase.schema || {});


Unnecessary defensive programming.

Suggested change

const dialectUri = getDialectUri(testCase.schema || {});

const dialectUri = getDialectUri(testCase.schema);

jdesrosiers · 2025-11-25T18:50:46Z

scripts/add-test-ids.js

+    if (!Array.isArray(testCase.tests)) continue;
+
+    const dialectUri = getDialectUri(testCase.schema || {});
+    const normalizedSchema = await normalize(testCase.schema || true, dialectUri);


Unnecessary defensive programming.

Suggested change

const normalizedSchema = await normalize(testCase.schema || true, dialectUri);

const normalizedSchema = await normalize(testCase.schema, dialectUri);

jdesrosiers · 2025-11-25T19:13:31Z

scripts/add-test-ids.js

+const DIALECT_MAP = {
+  "https://json-schema.org/draft/2020-12/schema": "https://json-schema.org/draft/2020-12/schema",
+  "https://json-schema.org/draft/2019-09/schema": "https://json-schema.org/draft/2019-09/schema",
+  "http://json-schema.org/draft-07/schema#": "http://json-schema.org/draft-07/schema#",
+  "http://json-schema.org/draft-06/schema#": "http://json-schema.org/draft-06/schema#",
+  "http://json-schema.org/draft-04/schema#": "http://json-schema.org/draft-04/schema#"
+};
+
+function getDialectUri(schema) {
+  if (schema.$schema && DIALECT_MAP[schema.$schema]) {
+    return DIALECT_MAP[schema.$schema];
+  }
+  return "https://json-schema.org/draft/2020-12/schema";
+}


There are a couple of issues here.

First, you can't use 2020-12 as the default dialect. The default needs to be the dialect associated with the directory the test file is in. For example, the convention to use $schema in every test case schema was only adopted starting in 2019-09. So, tests for draft-04/6/7 don't have $schema. That means this script will treat all of those tests as 2020-12 schemas, which would be very wrong.

This whole function should be unnecessary anyway. This is already handled internally by @hyperjump/json-schema. The result of this function is only used to pass to the registerSchema function, which should be the dialect from the directory the test file is in, not from the schema.

jdesrosiers · 2025-11-25T19:14:32Z

scripts/add-test-ids.js

+    .digest("hex");
+}
+
+async function addIdsToFile(filePath) {


You should need to pass the dialect based on the directory to this function.

jdesrosiers · 2025-11-25T19:18:32Z

scripts/add-test-ids.js

+    }
+  }
+
+  if (changed) {


Do you really need a changed flag? Can't you just check if added is greater than 0?

jdesrosiers · 2025-11-25T19:26:55Z

scripts/add-test-ids.js

+// Load remotes for all dialects
+const remotesPaths = ["./remotes"];
+for (const dialectUri of Object.values(DIALECT_MAP)) {
+  for (const path of remotesPaths) {
+    if (fs.existsSync(path)) {
+      loadRemotes(dialectUri, path);
+    }
+  }
+}


You're not going to be able to add remotes for all dialects at the same time. You're going to have to run this script separately for each dialect. Most remotes don't have $schema and are expected to run as the dialect of test case that's being tested. So, you can't load them at the same time because you can't have two schemas with the same id and different dialects. Your edits to loadRemotes are a result of this misunderstanding. I don't think you should need to make changes to that function.

I suggest making the first argument of the script the local draft identifier, then convert that to the right dialect URI to pass to addIdsToFile. Then the second argument can be the optional filePath, where all files are processed if the none is given.

jdesrosiers · 2025-11-25T19:28:15Z

scripts/add-test-ids.js

+}
+
+const filePath = process.argv[2] || "tests/draft2020-12/enum.json";
+addIdsToFile(filePath).catch(console.error);


The catch is unnecessary here. Just let it throw if there's an error.

AnirudhJindal added 2 commits November 25, 2025 15:37

Add test IDs to draft2020-12/enum.json using normalized schema hash (…

44fc7b2

…POC for json-schema-org#698)

Allow optional id property on tests in test-schema

d2980e4

AnirudhJindal marked this pull request as ready for review November 25, 2025 10:38

AnirudhJindal requested a review from a team as a code owner November 25, 2025 10:38

jdesrosiers marked this pull request as draft November 25, 2025 18:37

jdesrosiers reviewed Nov 25, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Add normalized hash-based test IDs to draft2020-12/enum.json (POC for #698) #796

Add normalized hash-based test IDs to draft2020-12/enum.json (POC for #698) #796

Uh oh!

AnirudhJindal commented Nov 25, 2025

Uh oh!

jdesrosiers commented Nov 25, 2025

Uh oh!

jdesrosiers left a comment

Uh oh!

jdesrosiers Nov 25, 2025

Uh oh!

jdesrosiers Nov 25, 2025

Uh oh!

jdesrosiers Nov 25, 2025

Uh oh!

jdesrosiers Nov 25, 2025

Uh oh!

jdesrosiers Nov 25, 2025 •

edited

Loading

Uh oh!

jdesrosiers Nov 25, 2025

Uh oh!

jdesrosiers Nov 25, 2025

Uh oh!

jdesrosiers Nov 25, 2025 •

edited

Loading

Uh oh!

jdesrosiers Nov 25, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

	const dialectUri = getDialectUri(testCase.schema \|\| {});
	const dialectUri = getDialectUri(testCase.schema);

	const normalizedSchema = await normalize(testCase.schema \|\| true, dialectUri);
	const normalizedSchema = await normalize(testCase.schema, dialectUri);

Uh oh!

Add normalized hash-based test IDs to draft2020-12/enum.json (POC for #698) #796

Are you sure you want to change the base?

Add normalized hash-based test IDs to draft2020-12/enum.json (POC for #698) #796

Uh oh!

Conversation

AnirudhJindal commented Nov 25, 2025

Context

The goal is:

How to run

Questions for reviewers

Uh oh!

jdesrosiers commented Nov 25, 2025

Uh oh!

jdesrosiers left a comment

Choose a reason for hiding this comment

Uh oh!

jdesrosiers Nov 25, 2025

Choose a reason for hiding this comment

Uh oh!

jdesrosiers Nov 25, 2025

Choose a reason for hiding this comment

Uh oh!

jdesrosiers Nov 25, 2025

Choose a reason for hiding this comment

Uh oh!

jdesrosiers Nov 25, 2025

Choose a reason for hiding this comment

Uh oh!

jdesrosiers Nov 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jdesrosiers Nov 25, 2025

Choose a reason for hiding this comment

Uh oh!

jdesrosiers Nov 25, 2025

Choose a reason for hiding this comment

Uh oh!

jdesrosiers Nov 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jdesrosiers Nov 25, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

jdesrosiers Nov 25, 2025 •

edited

Loading

jdesrosiers Nov 25, 2025 •

edited

Loading