gh-143732: add specialization for `FOR_ITER` by NekoAsakura · Pull Request #148745 · python/cpython

NekoAsakura · 2026-04-19T06:19:34Z

	main	branch	ratio
for_iter_dict_items	110.23 ms ± 1.74 ms	99.29 ms ± 1.24 ms	1.11× faster
for_iter_dict_keys	85.20 ms ± 1.03 ms	76.20 ms ± 1.02 ms	1.12× faster
for_iter_dict_values	84.91 ms ± 1.15 ms	76.27 ms ± 1.24 ms	1.11× faster
for_iter_set	96.88 ms ± 1.05 ms	87.46 ms ± 1.16 ms	1.11× faster
for_iter_reversed	79.99 ms ± 0.77 ms	72.03 ms ± 0.79 ms	1.11× faster
for_iter_enumerate	123.65 ms ± 1.94 ms	110.14 ms ± 1.82 ms	1.12× faster
for_iter_zip	131.47 ms ± 1.38 ms	122.17 ms ± 2.87 ms	1.08× faster
for_iter_list	59.21 ms ± 1.17 ms	59.48 ms ± 1.06 ms	1.00× slower
for_iter_tuple	53.78 ms ± 0.61 ms	53.77 ms ± 0.74 ms	1.00× faster
for_iter_range	72.62 ms ± 0.80 ms	72.55 ms ± 1.08 ms	1.00× faster

Issue: Broader specialization in the Specializing Adaptive Interpreter for better JIT performance #143732

cocolato

LGTM, thanks for doing this!

cocolato · 2026-04-19T15:03:55Z

        }

-        macro(FOR_ITER) = _SPECIALIZE_FOR_ITER + _FOR_ITER;
+        macro(FOR_ITER) = _SPECIALIZE_FOR_ITER + _RECORD_NOS_GEN_FUNC + _RECORD_NOS_TYPE + _FOR_ITER;


cpython/Python/optimizer_symbols.c

Lines 725 to 726 in ad7d361

case JIT_SYM_RECORDED_GEN_FUNC_TAG:

return &PyGen_Type;

We don't need _RECORD_NOS_GEN_FUNC here, because the GEN_FUNC_TAG recorded here does not contribute to the current optimization.

Every FOR_ITER specialisation's record list must be a prefix of FOR_ITER's.
_RECORD_NOS_GEN_FUNC writes a gen func or NULL to slot 0, matching what FOR_ITER_GEN reads from it.
https://github.com/python/cpython/actions/runs/24623062259/job/71997168333

cocolato · 2026-04-19T15:06:18Z

+                PyType_Watch(TYPE_WATCHER_ID, (PyObject *)probable);
+                _Py_BloomFilter_Add(dependencies, probable);
+                sym_set_type(iter, probable);
+                int32_t orig_target = (this_instr - 1)->target;


We should add an assert to make sure the last uop is _RECORD_NOS_TYPE

markshannon

I've a few suggestions inline.

markshannon · 2026-05-01T10:43:10Z

+        }
+
+        tier2 op(_ITER_NEXT_INLINE, (iternext_fn/4, iter, null_or_index -- iter, null_or_index, next)) {
+            volatile iternextfunc iternext_v = (iternextfunc)iternext_fn;


Why volatile? It shouldn't be necessary.
Also function pointers may not be the same size as normal pointers.
Can you add assert(sizeof(iternextfunc) == sizeof(uintptr_t)); to be on the safe side.

Because mmap pages usually sit too far from tp_iternext for the offset to reach, we have to use volatile to force compiler to emit callq *%rax (target read from a register) instead of callq <rel32> (target baked in as a fixed offset). Otherwise the call jumps to the wrong address and segfaults.

Does this fail just for x86 or for AArch64 as well?

This is a bug in the JIT and we fix it, rather than add workarounds.
I think we should be adding a trampoline here, but it seems that we are not.
If you're looking at the stencils, can you see how it is being patched (what patch function is being called)?

@brandtbucher

Both. (aarch64 was tested under qemu-user)
Trampoline path doesn't appear to be designed for _JIT_* symbols:

cpython/Tools/jit/_stencils.py

Lines 434 to 446 in 7c9ad27

def symbol_to_value(symbol: str) -> tuple[HoleValue, str | None]:

"""

Convert a symbol name to a HoleValue and a symbol name.

Some symbols (starting with "_JIT_") are special and are converted to their

own HoleValues.

"""

if symbol.startswith("_JIT_"):

try:

return HoleValue[symbol.removeprefix("_JIT_")], None

except KeyError:

pass

return HoleValue.ZERO, symbol

cpython/Tools/jit/_stencils.py

Lines 288 to 292 in 7c9ad27

if (

hole.kind

in {"R_AARCH64_CALL26", "R_AARCH64_JUMP26", "ARM64_RELOC_BRANCH26"}

and hole.value is HoleValue.ZERO

and hole.symbol not in self.symbols

cpython/Tools/jit/_stencils.py

Lines 305 to 309 in 7c9ad27

# x86_64 Darwin trampolines for external symbols

elif (

hole.kind == "X86_64_RELOC_BRANCH"

and hole.value is HoleValue.ZERO

and hole.symbol not in self.symbols

Even if we extended it to those, we'd need a stub per instance. I don't think it gains us anything.

We only need a trampoline if the jitted code is too far from the executable. See #148822

bedevere-app · 2026-05-01T10:50:28Z

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phrase I have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.

And if you don't make the requested changes, you will be poked with soft cushions!

read-the-docs-community · 2026-05-02T00:24:01Z

Documentation build overview

📚 cpython-previews | 🛠️ Build #32515427 | 📁 Comparing 49baba9 against main (4b33308)

🔍 Preview build

97 files changed · + 1 added · ± 96 modified

+ Added

c-api/sentinel.html

± Modified

markshannon · 2026-05-03T09:47:29Z

OK, well let's go with your approach for now. We already do the same here

I've created #149316 so we won't need the "volatile"

markshannon · 2026-05-03T09:55:47Z


        macro(FOR_ITER_VIRTUAL) =
            unused/1 +  // Skip over the counter
+            _RECORD_NOS +


Is this necessary?

Yeah, this is by design from #148730. When picking the family-wide recorder, the cases generator only looks at members, not the head:

cpython/Tools/cases_generator/record_function_generator.py

Lines 88 to 89 in 7c9ad27

member_records = [instruction_records[m.name] for m in family_members]

all_member_names = {n for names in member_records for n in names}

So without _RECORD_NOS on a member somewhere, _RECORD_NOS_TYPE on FOR_ITER and _RECORD_NOS_GEN_FUNC on FOR_ITER_GEN will clash. I can add a comment about this.

markshannon · 2026-05-03T09:59:58Z

+    op(_FOR_ITER_TIER_TWO, (iter, null_or_index -- iter, null_or_index, next)) {
+        PyTypeObject *type = sym_get_type(iter);
+        if (type != NULL && type != &PyGen_Type && type->tp_iternext != NULL) {
+            ADD_OP(_ITER_NEXT_INLINE, 0, (uintptr_t)type->tp_iternext);


We need the watcher here as well.

What we do elsewhere is something like this:

bool definite = true; PyTypeObject *type = sym_get_type(iter); if (type == NULL) { type = sym_get_probable_type(iter); definite = false; } if (type != NULL) { // Add if not definite, otherwise NOP // Add watcher(s) }

pythongh-143732: add specialization for FOR_ITER

dc169d8

NekoAsakura requested review from Fidget-Spinner, markshannon, savannahostrowski and tomasr8 as code owners April 19, 2026 06:19

bedevere-app Bot added the awaiting review label Apr 19, 2026

bedevere-app Bot mentioned this pull request Apr 19, 2026

Broader specialization in the Specializing Adaptive Interpreter for better JIT performance #143732

Open

NekoAsakura added 3 commits April 19, 2026 02:41

Merge branch 'main' into pythongh-143732/for-iter-type-recording

66c11de

Make slot compatible with FOR_ITER_GEN

3a934bf

Minimise diff

5700f6a

cocolato reviewed Apr 19, 2026

View reviewed changes

cocolato added the skip news label Apr 19, 2026

Add uop assert

1f48326

markshannon requested changes May 1, 2026

View reviewed changes

bedevere-app Bot added awaiting changes and removed awaiting review labels May 1, 2026

NekoAsakura added 3 commits May 1, 2026 16:15

address feedback

06ce7a4

Merge branch 'main' into pythongh-143732/for-iter-type-recording

4003e8c

uniform nos recording across FOR_ITER

0f69071

NekoAsakura added 2 commits May 2, 2026 09:28

Merge branch 'main' into pythongh-143732/for-iter-type-recording

724e285

Merge branch 'main' into pythongh-148871/load-common-constant

cbd17a2

markshannon mentioned this pull request May 3, 2026

JIT does not emit trampolines for jumps to addresses encoded as operands #149316

Open

markshannon self-requested a review May 3, 2026 09:54

markshannon reviewed May 3, 2026

View reviewed changes

NekoAsakura added 2 commits May 3, 2026 10:07

address feedback

49baba9

Merge branch 'main' into pythongh-143732/for-iter-type-recording

fc9da2c

	def symbol_to_value(symbol: str) -> tuple[HoleValue, str \| None]:
	"""
	Convert a symbol name to a HoleValue and a symbol name.

	Some symbols (starting with "_JIT_") are special and are converted to their
	own HoleValues.
	"""
	if symbol.startswith("_JIT_"):
	try:
	return HoleValue[symbol.removeprefix("_JIT_")], None
	except KeyError:
	pass
	return HoleValue.ZERO, symbol

	if (
	hole.kind
	in {"R_AARCH64_CALL26", "R_AARCH64_JUMP26", "ARM64_RELOC_BRANCH26"}
	and hole.value is HoleValue.ZERO
	and hole.symbol not in self.symbols

	# x86_64 Darwin trampolines for external symbols
	elif (
	hole.kind == "X86_64_RELOC_BRANCH"
	and hole.value is HoleValue.ZERO
	and hole.symbol not in self.symbols

	member_records = [instruction_records[m.name] for m in family_members]
	all_member_names = {n for names in member_records for n in names}

Uh oh!

Conversation

NekoAsakura commented Apr 19, 2026 • edited by bedevere-app Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cocolato left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cocolato Apr 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

markshannon left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

markshannon May 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bedevere-app Bot commented May 1, 2026

Uh oh!

read-the-docs-community Bot commented May 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Documentation build overview

Uh oh!

markshannon commented May 3, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

NekoAsakura commented Apr 19, 2026 •

edited by bedevere-app Bot

Loading

cocolato Apr 19, 2026 •

edited

Loading

markshannon May 1, 2026 •

edited

Loading

read-the-docs-community Bot commented May 2, 2026 •

edited

Loading