问题描述
我想要我的脚本
我的问题是 2。我有一个示例脚本,但它同步获取 url。我想让它异步。
from pyppeteer import launch
urls = ['https://www.example.com']
async def main():
browser = await launch(headless=False)
for url in urls:
page1 = await browser.newPage()
page2 = await browser.newPage()
page3 = await browser.newPage()
await page1.goto(url)
await page2.goto(url)
await page3.goto(url)
title1= await page1.title()
title2= await page2.title()
title3= await page3.title()
print(title1)
print(title2)
print(title3)
#await browser.close()
asyncio.get_event_loop().run_until_complete(main())
另外,正如你所看到的,代码并不那么简洁。我如何让它异步?
此外,如果有帮助,我还有其他不适合我需要的 pyppeteer 脚本,以防万一转换这些脚本会更容易
import asyncio
from pyppeteer import launch
url = 'http://www.example.com'
browser = None
async def fetchUrl(url):
# Define browser as a global variable to ensure that the browser window is only created once in the entire process
global browser
if browser is None:
browser = await launch(headless=False)
page = await browser.newPage()
await page.goto(url)
#await asyncio.wait([page.waitForNavigation()])
#str = await page.content()
#print(str)
# Execute this function multiple times for testing
asyncio.get_event_loop().run_until_complete(fetchUrl(url))
asyncio.get_event_loop().run_until_complete(fetchUrl(url))
该脚本是异步的,但它一次执行一个事件循环,因此它与同步一样好。
# cat test.py
import asyncio
import time
from pyppeteer import launch
WEBSITE_LIST = [
'http://envato.com','http://amazon.co.uk','http://example.com',]
start = time.time()
async def fetch(url):
browser = await launch(headless=False,args=['--no-sandBox'])
page = await browser.newPage()
await page.goto(f'{url}',{'waitUntil': 'load'})
print(f'{url}')
await asyncio.sleep(1)
await page.close()
#await browser.close()
async def run():
tasks = []
for url in WEBSITE_LIST:
task = asyncio.ensure_future(fetch(url))
tasks.append(task)
responses = await asyncio.gather(*tasks)
loop = asyncio.get_event_loop()
future = asyncio.ensure_future(run())
loop.run_until_complete(future)
print(f'It took {time.time()-start} seconds.')
该脚本是异步的,但它为每个 url 启动一个单独的浏览器,这最终会占用太多资源。
解决方法
这将在单独的标签中打开每个 URL:
import asyncio
import traceback
from pyppeteer import launch
URLS = [
"http://envato.com","http://amazon.co.uk","http://example.com",]
async def fetch(browser,url):
page = await browser.newPage()
try:
await page.goto(f"{url}",{"waitUntil": "load"})
except Exception:
traceback.print_exc()
else:
html = await page.content()
return (url,html)
finally:
await page.close()
async def main():
tasks = []
browser = await launch(headless=True,args=["--no-sandbox"])
for url in URLS:
tasks.append(asyncio.create_task(fetch(browser,url)))
for coro in asyncio.as_completed(tasks):
url,html = await coro
print(f"{url}: ({len(html)})")
await browser.close()
if __name__ == "__main__":
main = asyncio.run(main())