New fuzzer mode: Fuzz against JavaScript by kripken · Pull Request #8655 · WebAssembly/binaryen

kripken · 2026-04-28T22:26:39Z

The new fuzzer flag --fuzz-against-js tells the fuzzer we will only run the
wasm against JS - not link it to wasm or something else. This lets it make
changes that are valid from JS's point of view, like refining things on the
boundary while not changing the arity.

For example, if we sent JS an anyref, but the actual type we send is
(ref $A) then we can refine to that type (or any type between it and
anyref). We can do this for both export results and import params, as in
both cases we send things to JS and know their type.

Original idea was @tlively 's. This is useful for fuzzers that generate JS
and let Binaryen mutate the wasm: they can emit anyrefs on the
boundary, and Binaryen will be able to add new GC types in the
module and even refine the boundary to those types. Such a fuzzer
does not even need to emit GC types itself (it can emit anyref and send
only nulls). cc @rmahdav

kripken · 2026-04-28T22:42:08Z

@sbc100 I'm confused by the ruff lint errors here

Why is it showing errors on files I didn't touch? e.g. scripts/fuzz_passes_wast.py. Maybe something isn't working right?
It complains about PLR0911 Too many return statements but that seems like the best way to write that code...? Should I disable the check in general, or just in this function..?

sbc100 · 2026-04-28T22:47:50Z

@sbc100 I'm confused by the ruff lint errors here

Why is it showing errors on files I didn't touch? e.g. scripts/fuzz_passes_wast.py. Maybe something isn't working right?

It complains about PLR0911 Too many return statements but that seems like the best way to write that code...? Should I disable the check in general, or just in this function..?

Oops, looks like a bad merge in scripts/fuzz_passes_wast.py.. maybe my fault. If so sorry, will fix now.

Regarding PLR0911, yes you can add it to the exception list in .ruff.toml, or you can add just # noqa: PLR0911 to that one function

sbc100 · 2026-04-28T22:49:02Z

Fixed scripts/fuzz_passes_wast.py on main

kripken · 2026-04-28T23:38:05Z

@sbc100 thanks, I'll disable lint on that one function then.

tlively · 2026-04-29T05:17:19Z

+      if (newHeapType.isBottom()) {
+        options.push_back(oldHeapType);


In cases where the old heap type is also bottom, this will end up with two copies of bottom in the options. Not incorrect, but wastes a bit of randomness.

Hmm, yeah, but keeping the code simple seems good enough here.

tlively · 2026-04-29T05:22:53Z

+        if (type == Type::unreachable) {
+          // Nothing sent here, so use the declared type - what we refine to
+          // must still validate even though this call is unreachable.
+          type = declaredParams[i];
+        }


Using the declared type here will prevent any refinement. For reference types we should use the relevant non-nullable reference to bottom type to preserve maximum optionality.

Good point, done.

tlively · 2026-04-29T05:24:42Z

+    if (map[func->name].reffed) {
+      continue;
+    }


We might consider redirecting the references to new wrapper functions that have the original type and forward to the original function with its refined type.

Added a TODO.

tlively · 2026-04-29T05:30:44Z

+    if (!lub.noted()) {
+      continue;
+    }


Could use non-nullable bottom reference types here, too.

I suppose, though in this case the import/export is never actually reached?

Yes, but who knows what might end up happening in the engine 🤷 Might as well exercise as many situations as we can. Without looking at the V8 code, it seems unlikely but plausible that there would be some different code path taken on traps or exceptions, or when doing a JSPI suspension or something, depending on the return type even when the function never does a normal return.

Hmm, but what would we refine it to? A totally random type..?

A random subtype of the old result type, yes. The maybeRefine call below should not need to change.

tlively · 2026-04-29T05:35:16Z

+  (import "module" "base" (func $import (param i32 anyref) (result eqref)))
+  (import "module" "base" (func $import-reffed (param i32 anyref) (result eqref)))
+
+  ;; Two exports, one which will be reffed.


Might as well expand "reffed" to "referenced" to keep hypothetical future spell checkers running in CI happier.

tlively · 2026-04-29T05:41:44Z

What if we use a script similar to this to find an input module / seed for which the fuzzer generates output that contains all of the modifications we're interested in. Then we could just check in the input and output as a normal lit test and maybe check in the script in scripts/. That way there would be no need to read and understand any code to see that the test is testing the intended behavior.

But the seed would constantly change, leading to a lot of churn and toil?

Yeah, I guess if something unrelated changes the output we would need to entirely regenerate the test.

I still wish there were a more declarative way of doing this kind of property testing on our fuzzers, though. Can we at least refactor this into a generic utility for running the fuzzer iteratively and checking that the output satisfies various user-provided predicates with given probabilities? Laying this into reasonable abstractions would help make it more palatable.

If we want something more generic here, another option is to move this code in the fuzzer itself. Each fuzzer could persist state over time, then run "variety testing" after enough iterations. It would basically be the same code as here, but running when the fuzzer runs, not in the unit tests. What do you think?

Oh, yes, I kind of like that idea. I don't know about persisting the output modules over several iterations, but persisting statistics about the various predicates and raising an error once there is high confidence that there is problem sounds reasonable.

Hmm, thinking more about it, I'm not sure it makes sense in the fuzzer. We need to start with a fixed testcase so we can see what mutations we added on top, like this test does, while the fuzzer starts with something totally random each time.

I pushed a refactor instead along the lines of your suggestion, a generic utility that is now used in a modular way.

kripken · 2026-04-29T15:38:56Z

Last commit fixes the exactness logic, which was wrong - it could pick an intermediate heap type and make it exact.

tlively · 2026-04-29T18:20:34Z

+    if (!lub.noted()) {
+      continue;
+    }


Yes, but who knows what might end up happening in the engine 🤷 Might as well exercise as many situations as we can. Without looking at the V8 code, it seems unlikely but plausible that there would be some different code path taken on traps or exceptions, or when doing a JSPI suspension or something, depending on the return type even when the function never does a normal return.

tlively · 2026-04-29T18:25:12Z

Yeah, I guess if something unrelated changes the output we would need to entirely regenerate the test.

I still wish there were a more declarative way of doing this kind of property testing on our fuzzers, though. Can we at least refactor this into a generic utility for running the fuzzer iteratively and checking that the output satisfies various user-provided predicates with given probabilities? Laying this into reasonable abstractions would help make it more palatable.

tlively · 2026-04-29T18:26:45Z

+    if (newHeapType != new_.getHeapType() || newHeapType.isBasic()) {
      newExactness = Inexact;
    }


Might as well check these conditions before burning a bit to generate a new exactness above.

tlively · 2026-05-04T21:39:44Z

+    if (!lub.noted()) {
+      continue;
+    }


A random subtype of the old result type, yes. The maybeRefine call below should not need to change.

tlively · 2026-05-04T21:40:34Z

+    if (newHeapType != new_.getHeapType() || newHeapType.isBasic()) {
      newExactness = Inexact;
    }


tlively · 2026-05-04T21:46:34Z

+
+    # Given the types we saw for params or results, look in detail for the
+    # things we expect to see.
+    def found_expected(self, data):


IIUC, this looks for a single iteration where all of these things are true. But that's stricter than we want. Can we make each of these conditions a bool on the class, set them to true whenever we see them, and pass the test once all have been seen?

kripken added 30 commits April 23, 2026 15:28

start

91c30a0

work

65c2a31

work

d08c29f

work

f3a07c8

work

823c665

work

767c8b4

work

c7d7f1b

work

fc981c7

work

ebb1106

work

fa2da6e

help

2c3c192

work

53b1438

work

255250d

work

169f4c8

work

2fe14e0

work

300c953

work

39f57ee

work

cffacbc

work

41cd320

work

063b415

work

12d35ef

work

e5d4ea5

work

d968315

work

45ce1dc

work

c1fccda

work

a7b5bfd

work

35c498f

clean

98e7b09

go

26d3f3e

Merge remote-tracking branch 'origin/main' into fuzz.against.js

c0c6dae

kripken added 5 commits April 28, 2026 14:43

cleanup

90c18af

cleanup

f3d11a6

fix

44fe786

fix

79e65d7

go

98bd0ac

kripken requested a review from tlively April 28, 2026 22:26

kripken requested a review from a team as a code owner April 28, 2026 22:26

fix lint

4179d31

kripken added 2 commits April 28, 2026 16:37

lint

2e2d028

Merge remote-tracking branch 'origin/main' into fuzz.against.js

a7bf270

tlively reviewed Apr 29, 2026

View reviewed changes

fix

1f7e875

kripken added 3 commits April 29, 2026 10:03

least restrictions when unreachable

06357e5

wrapper TODO

a079f2e

avoid typos in comments

a7b5125

tlively reviewed Apr 29, 2026

View reviewed changes

kripken added 3 commits April 29, 2026 11:54

typo

149f300

Refactor

130029f

linkt

ca52c2d

tlively reviewed May 4, 2026

View reviewed changes

		if (newHeapType.isBottom()) {
		options.push_back(oldHeapType);

Conversation

kripken commented Apr 28, 2026

Uh oh!

kripken commented Apr 28, 2026

Uh oh!

sbc100 commented Apr 28, 2026

Uh oh!

sbc100 commented Apr 28, 2026

Uh oh!

kripken commented Apr 28, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kripken commented Apr 29, 2026

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants