python – asyncio.as_completed是否会产生期货或协同程序？

来自asyncio docs：

asyncio.as_completed(aws,*,loop=None,timeout=None)

Run awaitable objects in the aws set concurrently. Return an iterator
of Future objects. Each Future object returned represents the earliest
result from the set of the remaining awaitables.

我假设这些Future对象中的每一个都具有asyncio.Future中描述的方法：.cancelled(),. exception()和.result().但似乎所产生的元素只是协程,而不是Future对象.我错过了什么？

这似乎打败了.as_completed()的描述.如果我需要等待它,协程如何“完成”？

>>> import asyncio
>>> import aiohttp
>>> 
>>> async def get(session,url):
...     async with session.request('GET',url) as resp:
...         t = await resp.text()
...         return t
... 
>>> async def bulk_as_completed(urls):
...     async with aiohttp.ClientSession() as session:
...         aws = [get(session,url) for url in urls]
...         for future in asyncio.as_completed(aws):
...             for i in ('cancelled','exception','result'):
...                 print(hasattr(future,i))
...             print(type(future))
...             try:
...                 result = await future
...             except:
...                 pass
...             else:
...                 print(type(result))
...                 print()
... 
>>> 
>>> urls = (
...     'https://docs.python.org/3/library/asyncio-task.html',...     'https://docs.python.org/3/library/select.html',...     'https://docs.python.org/3/library/this-page-will-404.html',... )
>>> 
>>> asyncio.run(bulk_as_completed(urls))
False
False
False

最终,我关心这个的原因是因为我想让异常像asyncio.gather(…,return_exceptions = True)那样冒泡.考虑在调用session.request()时添加一个伪造的URL：

urls = (
    'https://docs.python.org/3/library/asyncio-task.html','https://docs.python.org/3/library/select.html','https://docs.python.org/3/library/this-page-will-404.html',# This URL will raise on session.request().  How can I propagate
    # that exception to the iterator of results?
    'https://asdfasdfasdf-does-not-exist-asdfasdfasdf.com'
)

我希望能够做的是这样的事情(使用Future对象的方法,但这些都不是Future对象,这就是问题)：

async def bulk_as_completed(urls):
    async with aiohttp.ClientSession() as session:
        aws = [get(session,url) for url in urls]
        for future in asyncio.as_completed(aws):
            if future.cancelled():
                res = futures.CancelledError()
            else:
                exc = future.exception()
                if exc is not None:
                    res = exc
                else:
                    res = future.result()
            # ...
            # [Do something with `res`]

最佳答案

What I would like to be able to do is something like this […]

也许不那么方便,但您应该能够使用以下代码提取异常：

async def bulk_as_completed(urls):
    async with aiohttp.ClientSession() as session:
        aws = [get(session,url) for url in urls]
        for future in asyncio.as_completed(aws):
            try:
                res = await future
            except Exception as e:
                res = e
            # ...
            # [Do something with `res`]

This [yielding coroutines rather than futures] seems to defeat the description of .as_completed(). How is the coroutine “completed” if I need to await it?

不是.首次实现asyncio.as_completed时,异步迭代器不存在.没有异步迭代就没有办法在它们完成时返回期货,所以as_completed类型通过屈服(立即)虚拟等待来伪造它,人们必须等待获得实际结果.

即使as_completed产生了实际的期货,也不会对你的用例有所帮助,因为如果没有等待他们的人,这些期货就无法完成.为了提供as_completed让步完成的期货的预期语义,as_completed需要实现异步迭代,其等价的__next__可以等待.

as_completed的惊人行为之前已经提出,我已经通过提供异步迭代提交了an issue来修复它.一旦实现,您的原始代码将只用于更改为async for.