Reference implementation for PEP 828

Doc/library/exceptions.rst

@@ -524,8 +524,19 @@ The following exceptions are the exceptions that are usually raised.
    Must be raised by :meth:`~object.__anext__` method of an
    :term:`asynchronous iterator` object to stop the iteration.
 
+   .. attribute:: StopAsyncIteration.value
+
+      This is given as an argument when constructing the exception, and
+      defaults to :const:`None`. This is used for the result of
+      ``async yield from`` expressions (see :ref:`async-yield-from`).
+
+      .. versionadded: next
+
    .. versionadded:: 3.5
 
+   .. versionchanged:: next
+      Added the ``value`` attribute.
+
 .. exception:: SyntaxError(message, details)
 
    Raised when the parser encounters a syntax error.  This may occur in an

Doc/reference/expressions.rst

@@ -629,6 +629,7 @@ Yield expressions
    yield_atom: "(" `yield_expression` ")"
    yield_from: "yield" "from" `expression`
    yield_expression: "yield" `yield_list` | `yield_from`
+   async_yield_from: "async" "yield" "from" `expression`
 
 The yield expression is used when defining a :term:`generator` function
 or an :term:`asynchronous generator` function and
@@ -708,6 +709,10 @@ the yield expression. It can be either set explicitly when raising
 .. versionchanged:: 3.3
    Added ``yield from <expr>`` to delegate control flow to a subiterator.
 
+.. versionchanged:: next
+   ``yield from`` is now allowed to be used in an async generator.
+   Previously, it would raise a :class:`SyntaxError`.
+
 The parentheses may be omitted when the yield expression is the sole expression
 on the right hand side of an assignment statement.
 
@@ -728,6 +733,10 @@ on the right hand side of an assignment statement.
       The proposal that expanded on :pep:`492` by adding generator capabilities to
       coroutine functions.
 
+   :pep:`828` - Supporting ``yield from`` in asynchronous generators
+      The proposal that expanded on :pep:`380` by adding subgenerator delegation
+      to asynchronous generators.
+
 .. index:: pair: object; generator
 .. _generator-methods:
 
@@ -913,7 +922,89 @@ registered *finalizer* to be called upon finalization. For a reference example
 of a *finalizer* method see the implementation of
 ``asyncio.Loop.shutdown_asyncgens`` in :source:`Lib/asyncio/base_events.py`.
 
-The expression ``yield from <expr>`` is a syntax error when used in an
+.. _async-yield-from:
+
+Asynchronous ``yield from``
+^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Conceptually, ``async yield from`` is very similar to ``yield from``, but
+operates solely on asynchronous constructs rather than synchronous ones.
+In particular:
+
+.. list-table::
+   :widths: auto
+   :header-rows: 1
+
+   * * ``yield from`` construct
+     * ``async yield from`` construct
+   * * :meth:`~object.__iter__`
+     * :meth:`~object.__aiter__`
+   * * :meth:`~generator.__next__`
+     * :meth:`~agen.__anext__`
+   * * :meth:`~generator.send`
+     * :meth:`~agen.asend`
+   * * :class:`StopIteration`
+     * :class:`StopAsyncIteration`
+
+To describe the above:
+
+* The object being delegated to must be asynchronously iterable (that is, it
+  must implement ``__aiter__`` instead of ``__iter__``).
+* When ``anext`` is called on the parent generator (the one that contains
+  ``async yield from``), ``__anext__`` will be invoked on the subgenerator.
+  In contrast, a synchronous ``yield from`` would invoke ``__next__`` instead.
+  (Note that calling ``asend`` with a ``None`` value is equivalent to calling
+  ``anext()``, and thus applies here.)
+* All calls to ``asend``, ``athrow``, and ``aclose`` are delegated to the
+  subgenerator (the object returned by ``__aiter__`` in this case). This means
+  that a call to ``parent_generator.asend(x)`` is semantically equivalent to
+  ``sub_generator.asend(x)``, where ``parent_generator`` is currently executing
+  an ``async yield from`` on ``sub_generator``.
+* The result of the expression is retrieved through
+  :attr:`StopAsyncIteration.value` instead of :attr:`StopIteration.value`.
+
+An example of usage for ``async yield from``:
+
+.. code-block:: pycon
+
+   >>> import asyncio
+   >>> async def sleepy_count(number):
+   ...     for num in range(number):
+   ...         await asyncio.sleep(1)
+   ...         result = yield num
+   ...         print(f"Got result: {result}")
+   ...
+   >>> async def counter():
+   ...     final_number = async yield from sleepy_count(5)
+   ...     yield final_number
+   ...
+   >>> await anext(ag)
+   0
+   >>> await anext(ag)
+   Got result: None
+   1
+   >>> await ag.asend(42)
+   Got result: 42
+   2
+   >>> await ag.athrow(ValueError("Nobody expects the Spanish Inquisition"))
+   Traceback (most recent call last):
+   File "/home/python/cpython/Lib/concurrent/futures/_base.py", line 450, in result
+      return self.__get_result()
+            ~~~~~~~~~~~~~~~~~^^
+   File "/home/python/cpython/Lib/concurrent/futures/_base.py", line 395, in __get_result
+      raise self._exception
+   File "<python-input-4>", line 1, in <module>
+      await ag.athrow(ValueError("Nobody expects the Spanish Inquisition"))
+   File "<python-input-0>", line 8, in counter
+      final_number = async yield from sleepy_count(4)
+                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+   File "<python-input-0>", line 3, in sleepy_count
+      result = yield num
+               ^^^^^^^^^
+   ValueError: Nobody expects the Spanish Inquisition
+
+
+``async yield from`` is a :class:`SyntaxError` when used outside of an
 asynchronous generator function.
 
 .. index:: pair: object; asynchronous-generator

Doc/reference/simple_stmts.rst

@@ -534,11 +534,13 @@ statement. For example, the yield statements ::
 
   yield <expr>
   yield from <expr>
+  async yield from <expr>
 
 are equivalent to the yield expression statements ::
 
   (yield <expr>)
   (yield from <expr>)
+  (async yield from <expr>)
 
 Yield expressions and statements are only used when defining a :term:`generator`
 function, and are only used in the body of the generator function.  Using :keyword:`yield`

Doc/whatsnew/3.15.rst

@@ -87,6 +87,7 @@ Summary -- Release highlights
   <whatsnew315-pybyteswriter>`
 * :pep:`803`: :ref:`Stable ABI for Free-Threaded Builds <whatsnew315-abi3t>`
 * :pep:`831`: :ref:`Frame pointers everywhere <whatsnew315-frame-pointers>`
+* :pep:`828`: :ref:`'yield from' in async generators <whatsnew315-async-yield-from>`
 * :ref:`The JIT compiler has been significantly upgraded <whatsnew315-jit>`
 * :ref:`Improved error messages <whatsnew315-improved-error-messages>`
 * :ref:`The official Windows 64-bit binaries now use the tail-calling interpreter
@@ -447,6 +448,54 @@ If not using a build tool -- or when writing such a tool -- you can select
 ``abi3t`` by setting the macro :c:macro:`!Py_TARGET_ABI3T` as discussed
 in :ref:`abi3-compiling`.
 
+.. _whatsnew315-async-yield-from:
+
+:pep:`828`: Supporting ``yield from`` in asynchronous generators
+----------------------------------------------------------------
+
+Use of the :keyword:`yield from <yield>` construct and the :keyword:`return`
+statement with a non-``None`` value is now allowed in an
+:term:`asynchronous generator function <asynchronous generator>`. For example,
+the following code would previously raise a :class:`SyntaxError`:
+
+.. code-block:: python
+
+   async def agenerator():
+      yield 1
+      yield from range(2, 5)  # Now allowed!
+      return 5  # Now allowed!
+
+Additionally, there is a new ``async yield from`` statement to delegate to an
+asynchronous generator:
+
+.. code-block:: python
+
+   async def sleepy_range(start, stop):
+      for number in range(start, stop):
+         await asyncio.sleep(1)
+         yield number
+
+   async def agenerator():
+      yield 1
+      async yield from sleepy_range(2, 5)
+      yield 5
+
+
+Similar to ``yield from``, ``async yield from`` may also be used as an expression,
+in which case the ``return`` of the async generator is used.
+
+For example:
+
+.. code-block:: python
+
+   async def asubgenerator():
+      yield 1
+      yield 2
+      return 3
+
+   async def agenerator():
+      value = async yield from asubgenerator()
+      print(value)  # 3
 
 .. _whatsnew315-improved-error-messages:
 

Grammar/python.gram

@@ -127,7 +127,7 @@ simple_stmt[stmt_ty] (memo):
     | &'raise' raise_stmt
     | &'pass' pass_stmt
     | &'del' del_stmt
-    | &'yield' yield_stmt
+    | &('yield' | 'async') yield_stmt
     | &'assert' assert_stmt
     | &'break' break_stmt
     | &'continue' continue_stmt
@@ -727,6 +727,7 @@ if_expression[expr_ty]:
 yield_expr[expr_ty]:
     | 'yield' 'from' a=expression { _PyAST_YieldFrom(a, EXTRA) }
     | 'yield' a=[star_expressions] { _PyAST_Yield(a, EXTRA) }
+    | 'async' 'yield' 'from' a=expression { _PyAST_AsyncYieldFrom(a, EXTRA) }
 
 star_expressions[expr_ty]:
     | a=star_expression b=(',' c=star_expression { c })+ [','] {

Include/cpython/pyerrors.h

@@ -73,6 +73,11 @@ typedef struct {
     PyObject *value;
 } PyStopIterationObject;
 
+typedef struct {
+    PyException_HEAD
+    PyObject *value;
+} PyStopAsyncIterationObject;
+
 typedef struct {
     PyException_HEAD
     PyObject *name;

Include/internal/pycore_genobject.h

@@ -32,13 +32,15 @@ PyAPI_FUNC(int) _PyGen_FetchStopIterationValue(PyObject **);
 
 PyAPI_FUNC(PyObject *)_PyCoro_GetAwaitableIter(PyObject *o);
 PyAPI_FUNC(PyObject *)_PyAsyncGenValueWrapperNew(PyThreadState *state, PyObject *);
+PyAPI_FUNC(PyObject *)_PyAsyncGenYieldFrom_New(PyThreadState *state, PyObject *);
 
 // Exported for external JIT support
 PyAPI_FUNC(PyObject *) _PyCoro_ComputeOrigin(int origin_depth, _PyInterpreterFrame *current_frame);
 
 extern PyTypeObject _PyCoroWrapper_Type;
 extern PyTypeObject _PyAsyncGenWrappedValue_Type;
 extern PyTypeObject _PyAsyncGenAThrow_Type;
+extern PyTypeObject _PyAsyncGenYieldFrom_Type;
 
 #ifdef __cplusplus
 }

Include/internal/pycore_interp_structs.h

@@ -529,7 +529,7 @@ struct _py_func_state {
    If you add a new static type to the standard library, you may have to
    update one of these numbers.
    */
-#define _Py_NUM_MANAGED_PREINITIALIZED_TYPES 120
+#define _Py_NUM_MANAGED_PREINITIALIZED_TYPES 121
 #define _Py_MAX_MANAGED_STATIC_BUILTIN_TYPES \
     (_Py_NUM_MANAGED_PREINITIALIZED_TYPES + 83)
 #define _Py_MAX_MANAGED_STATIC_EXT_TYPES 10

Include/internal/pycore_intrinsics.h

@@ -18,8 +18,9 @@
 #define INTRINSIC_TYPEVARTUPLE                   9
 #define INTRINSIC_SUBSCRIPT_GENERIC             10
 #define INTRINSIC_TYPEALIAS                     11
+#define INSTRINSIC_ASYNC_GEN_WRAP_YIELD_FROM    12
 
-#define MAX_INTRINSIC_1                         11
+#define MAX_INTRINSIC_1                         12
 
 
 /* Binary Functions: */

Include/internal/pycore_magic_number.h

@@ -296,6 +296,7 @@ Known values:
     Python 3.15a8 3663 (Merge GET_ITER and GET_YIELD_FROM_ITER. Modify SEND to make it a bit more like FOR_ITER)
     Python 3.15a8 3664 (Fix __qualname__ for __annotate__ functions)
     Python 3.15a8 3665 (Add FOR_ITER_VIRTUAL and GET_ITER specializations)
+    Python 3.15a6 3666 (PEP 828: yield from for asyncgens)
 
 
     Python 3.16 will start with 3700
@@ -309,7 +310,7 @@ PC/launcher.c must also be updated.
 
 */
 
-#define PYC_MAGIC_NUMBER 3665
+#define PYC_MAGIC_NUMBER 3666
 /* This is equivalent to converting PYC_MAGIC_NUMBER to 2 bytes
    (little-endian) and then appending b'\r\n'. */
 #define PYC_MAGIC_NUMBER_TOKEN \