From d048f7064543f86bc7a6fe27b6b83de44baecf89 Mon Sep 17 00:00:00 2001 From: atlowChemi Date: Thu, 30 Apr 2026 19:24:30 +0300 Subject: [PATCH] test_runner: add experimental tag-based test filtering Add a `tags` option on `test()` / `it()` / `suite()` / `describe()` and the `--experimental-test-tag-filter` CLI flag (plus the `testTagFilters` option on `run()`) that accepts a Vitest-style boolean expression with `and`/`&&`, `or`/`||`, `not`/`!`, parentheses, and `*` wildcards. Tags inherit from suite to child tests by union. Filtering composes by AND with name patterns, skip patterns, and `.only`. Untagged tests are filtered out under any include expression; `not X` is true against an untagged test. Tag matching is case-insensitive with lowercase canonical storage. Signed-off-by: atlowChemi --- doc/api/cli.md | 24 + doc/api/test.md | 184 +++++++ doc/node.1 | 6 + lib/internal/test_runner/harness.js | 2 + lib/internal/test_runner/runner.js | 45 ++ lib/internal/test_runner/tag_filter.js | 508 ++++++++++++++++++ lib/internal/test_runner/test.js | 63 ++- lib/internal/test_runner/tests_stream.js | 19 +- lib/internal/test_runner/utils.js | 36 +- src/node_options.cc | 5 + src/node_options.h | 1 + test/fixtures/test-runner/tagged.js | 17 + test/parallel/test-runner-tag-filter-cli.mjs | 149 +++++ .../test-runner-tag-filter-parser.mjs | 257 +++++++++ test/parallel/test-runner-tags-events.mjs | 96 ++++ .../test-runner-tags-experimental-warning.mjs | 97 ++++ .../parallel/test-runner-tags-inheritance.mjs | 104 ++++ test/parallel/test-runner-tags-validation.mjs | 123 +++++ 18 files changed, 1721 insertions(+), 15 deletions(-) create mode 100644 lib/internal/test_runner/tag_filter.js create mode 100644 test/fixtures/test-runner/tagged.js create mode 100644 test/parallel/test-runner-tag-filter-cli.mjs create mode 100644 test/parallel/test-runner-tag-filter-parser.mjs create mode 100644 test/parallel/test-runner-tags-events.mjs create mode 100644 test/parallel/test-runner-tags-experimental-warning.mjs create mode 100644 test/parallel/test-runner-tags-inheritance.mjs create mode 100644 test/parallel/test-runner-tags-validation.mjs diff --git a/doc/api/cli.md b/doc/api/cli.md index e979ec95c4259d..de298bba944cf0 100644 --- a/doc/api/cli.md +++ b/doc/api/cli.md @@ -1373,6 +1373,29 @@ Enable module mocking in the test runner. This feature requires `--allow-worker` if used with the [Permission Model][]. +### `--experimental-test-tag-filter=''` + + + +> Stability: 1.0 - Early development + +Run only tests that match the provided boolean tag-filter expression. Tests +declare tags via the `tags` option on `test()`, `it()`, `suite()`, or +`describe()`. Tags inherit from suites to nested tests by union. + +The expression supports boolean operators (`and`/`&&`, `or`/`||`, +`not`/`!`), parentheses for grouping, and `*` wildcards inside identifiers. +Standard precedence applies: `not` binds tighter than `and`, which binds +tighter than `or`. See [Test tags][] for the full grammar and behavior. + +The flag may be specified more than once; multiple expressions are combined +with AND, so a test must satisfy every expression to run. + +A malformed expression causes the test runner to exit with a non-zero status +before running any tests. + ### `--experimental-vm-modules` + +> Stability: 1.0 - Early development + +Tags annotate tests and suites with arbitrary string labels. The +[`--experimental-test-tag-filter`][] CLI flag (or the `testTagFilters` +option on [`run()`][]) selects tests by a boolean expression over those +labels. + +Tags are an alternative to encoding metadata into test names. They are +useful for cross-cutting axes such as subsystem, speed bucket, flakiness, +or environment, where a name pattern would be brittle. + +### Authoring tagged tests + +Pass a `tags` array on any of `test()`, `it()`, `suite()`, or `describe()`. +Tags inherit from a suite to its child tests by union — a test inside a +suite tagged `['db']` that declares its own `tags: ['integration']` +effectively has both tags. + +```mjs +import { describe, it } from 'node:test'; + +describe('database', { tags: ['db'] }, () => { + it('reads a row'); // tags: ['db'] + it('writes a row', { tags: ['integration'] }); // tags: ['db', 'integration'] + it('reconnects after disconnect', { tags: ['flaky'] }); // tags: ['db', 'flaky'] +}); +``` + +```cjs +const { describe, it } = require('node:test'); + +describe('database', { tags: ['db'] }, () => { + it('reads a row'); // tags: ['db'] + it('writes a row', { tags: ['integration'] }); // tags: ['db', 'integration'] + it('reconnects after disconnect', { tags: ['flaky'] }); // tags: ['db', 'flaky'] +}); +``` + +Tag values must be non-empty strings that contain no whitespace, no +operator characters (`& | ! ( ) *`), and are not the reserved words +`'and'`, `'or'`, or `'not'` in any casing. Tags are matched +case-insensitively; the canonical form is lowercase. Duplicates within a +single `tags` array are collapsed on the lowercased form, preserving the +first-seen declaration order. + +Hooks (`before`, `after`, `beforeEach`, `afterEach`) do not declare their +own tags. They run as part of their owning suite, which carries the +suite's tags. + +### Filtering syntax + +The filter expression supports: + +* Identifiers — any non-whitespace, non-operator characters. A literal + identifier matches a tag of the same value (case-insensitive). +* `*` wildcards inside an identifier match any sequence of characters. + A bare `*` matches any tagged test. +* Boolean operators with two equivalent forms: + * `and` / `&&` + * `or` / `||` + * `not` / `!` +* Parentheses for grouping. + +The word forms (`and`, `or`, `not`) require whitespace separation; the +punctuation forms do not. + +#### Operator precedence + +The expression is evaluated with the standard precedence +`not > and > or`. Binary operators are left-associative. + +| Expression | Equivalent grouping | +| -------------- | ------------------- | +| `a or b and c` | `a or (b and c)` | +| `not a and b` | `(not a) and b` | + +Use parentheses to override: + +| Expression | Selects | +| ------------------------------ | ------------------------------------------ | +| `(unit or smoke) and not slow` | unit-or-smoke tests that are not also slow | +| `db && !flaky` | db tests that are not flaky | +| `*` | every tagged test | + +#### Untagged tests + +Untagged tests behave as if they have an empty tag set. As a result: + +* Any include expression (a tag, wildcard, `and`, or `or`) is **false** + for an untagged test, so untagged tests are excluded under any positive + filter. +* `not X` is **true** for an untagged test, so excluding tags does not + accidentally remove untagged tests. + +For example, `--experimental-test-tag-filter='not flaky'` runs every test +that is not tagged `flaky`, including all untagged tests. + +#### Composing multiple filters + +[`--experimental-test-tag-filter`][] may be specified more than once on the +command line. Multiple expressions compose by AND — a test must satisfy +every expression to run. The same applies to passing an array to +`testTagFilters` on [`run()`][]. The tag filter is also AND'd with +[`--test-name-pattern`][], [`--test-skip-pattern`][], and `.only` +filtering. + +#### Reading tags from inside a test + +The [`TestContext`][] object exposes the test's tags as a frozen array +through [`context.tags`][], so tests can branch on their own metadata. + +#### Errors + +A tag value that violates the validation rules above throws +`ERR_INVALID_ARG_VALUE` at the registration site, before any test runs. +A non-array `tags` value throws `ERR_INVALID_ARG_TYPE`. A malformed +filter expression on the CLI causes the test runner to exit with a +non-zero status before running any test files. + ## Extraneous asynchronous activity Once a test function finishes executing, the results are reported as quickly @@ -750,6 +875,8 @@ test runner functionality: * `--test` - Prevented to avoid recursive test execution * `--experimental-test-coverage` - Managed by the test runner +* `--experimental-test-tag-filter` - Filter expressions are validated by the parent + process and re-emitted to child processes * `--watch` - Watch mode is handled at the parent level * `--experimental-default-config-file` - Config file loading is handled by the parent * `--test-reporter` - Reporting is managed by the parent process @@ -1569,6 +1696,9 @@ added: - v18.9.0 - v16.19.0 changes: + - version: REPLACEME + pr-url: https://github.com/nodejs/node/pull/63054 + description: Added the `testTagFilters` option. - version: - v25.6.0 - v24.14.0 @@ -1657,6 +1787,11 @@ changes: For each test that is executed, any corresponding test hooks, such as `beforeEach()`, are also run. **Default:** `undefined`. + * `testTagFilters` {string|string\[]} A boolean expression, or an array of + boolean expressions, used to filter tests by their declared tags. + Multiple expressions compose by AND. Equivalent to passing + [`--experimental-test-tag-filter`][] on the command line. See + [Test tags][]. **Default:** `undefined`. * `timeout` {number} A number of milliseconds the test execution will fail after. If unspecified, subtests inherit this value from their parent. @@ -1800,6 +1935,9 @@ added: - v18.0.0 - v16.17.0 changes: + - version: REPLACEME + pr-url: https://github.com/nodejs/node/pull/63054 + description: Added the `tags` option. - version: - v20.2.0 - v18.17.0 @@ -1843,6 +1981,10 @@ changes: * `skip` {boolean|string} If truthy, the test is skipped. If a string is provided, that string is displayed in the test results as the reason for skipping the test. **Default:** `false`. + * `tags` {string\[]} An array of string labels associated with the test. + Used together with [`--experimental-test-tag-filter`][] to filter which + tests run. Tags inherit from suites to nested tests by union. See + [Test tags][]. **Default:** `[]`. * `todo` {boolean|string} If truthy, the test marked as `TODO`. If a string is provided, that string is displayed in the test results as the reason why the test is `TODO`. **Default:** `false`. @@ -3431,6 +3573,9 @@ Emitted when code coverage is enabled and all tests have completed. `undefined` if the test was run through the REPL. * `name` {string} The test name. * `nesting` {number} The nesting level of the test. + * `tags` {string\[]} The flattened lowercased tags declared on the test + and its ancestor suites, in declaration order. Empty for untagged tests. + See [Test tags][]. * `testId` {number} A numeric identifier for this test instance, unique within the test file's process. Consistent across all events for the same test instance, enabling reliable correlation in custom reporters. @@ -3454,6 +3599,9 @@ The corresponding declaration ordered events are `'test:pass'` and `'test:fail'` `undefined` if the test was run through the REPL. * `name` {string} The test name. * `nesting` {number} The nesting level of the test. + * `tags` {string\[]} The flattened lowercased tags declared on the test + and its ancestor suites, in declaration order. Empty for untagged tests. + See [Test tags][]. * `testId` {number} A numeric identifier for this test instance, unique within the test file's process. Consistent across all events for the same test instance, enabling reliable correlation in custom reporters. @@ -3495,6 +3643,9 @@ defined. `undefined` if the test was run through the REPL. * `name` {string} The test name. * `nesting` {number} The nesting level of the test. + * `tags` {string\[]} The flattened lowercased tags declared on the test + and its ancestor suites, in declaration order. Empty for untagged tests. + See [Test tags][]. * `testId` {number} A numeric identifier for this test instance, unique within the test file's process. Consistent across all events for the same test instance, enabling reliable correlation in custom reporters. @@ -3521,6 +3672,9 @@ Emitted when a test is enqueued for execution. `undefined` if the test was run through the REPL. * `name` {string} The test name. * `nesting` {number} The nesting level of the test. + * `tags` {string\[]} The flattened lowercased tags declared on the test + and its ancestor suites, in declaration order. Empty for untagged tests. + See [Test tags][]. * `testId` {number} A numeric identifier for this test instance, unique within the test file's process. Consistent across all events for the same test instance, enabling reliable correlation in custom reporters. @@ -3580,6 +3734,9 @@ since the parent runner only knows about file-level tests. When using `undefined` if the test was run through the REPL. * `name` {string} The test name. * `nesting` {number} The nesting level of the test. + * `tags` {string\[]} The flattened lowercased tags declared on the test + and its ancestor suites, in declaration order. Empty for untagged tests. + See [Test tags][]. * `testId` {number} A numeric identifier for this test instance, unique within the test file's process. Consistent across all events for the same test instance, enabling reliable correlation in custom reporters. @@ -3619,6 +3776,9 @@ defined. `undefined` if the test was run through the REPL. * `name` {string} The test name. * `nesting` {number} The nesting level of the test. + * `tags` {string\[]} The flattened lowercased tags declared on the test + and its ancestor suites, in declaration order. Empty for untagged tests. + See [Test tags][]. * `testId` {number} A numeric identifier for this test instance, unique within the test file's process. Consistent across all events for the same test instance, enabling reliable correlation in custom reporters. @@ -4122,6 +4282,20 @@ The attempt number of the test. This value is zero-based, so the first attempt i the second attempt is `1`, and so on. This property is useful in conjunction with the `--test-rerun-failures` option to determine which attempt the test is currently running. +### `context.tags` + + + +> Stability: 1.0 - Early development + +* Type: {string\[]} + +A frozen array of the test's flattened lowercased tags, in declaration +order, including any tags inherited from ancestor suites. Empty when the +test has no tags. See [Test tags][]. + ### `context.workerId`