问题描述
我的问题是我的状态为502或504。但这并非总是如此。有时蜘蛛可以正常工作，因为它应该工作。但是有时某些查询返回502或504。但是，这些查询始终是随机请求。有时还可能是蜘蛛首先返回某个页面的502或504状态，但后来，scrapy通知我该页面已成功收集并保存（此变化显示在日志中）。怎么会这样？以及如何摆脱这些问题？
import os
import re
import scrapy
from scrapy_splash import SplashRequest
from .utils.config import HTML_DIR
class CqSpider(scrapy.Spider):
name = "cq"
def start_requests(self):
urls = [
'https://tools.cisco.com/security/center/content/CiscoSecurityAdvisory/cisco-sa-ucs-director-xss-O7T8ORYR','https://tools.cisco.com/security/center/content/CiscoSecurityAdvisory/cisco-sa-code-exec-wH3BNFb','https://tools.cisco.com/security/center/content/CiscoSecurityAdvisory/cisco-sa-dcnm-stored-xss-yJyqBJGU','https://tools.cisco.com/security/center/content/CiscoSecurityAdvisory/cisco-sa-20190807-wms-oredirect','https://tools.cisco.com/security/center/content/CiscoSecurityAdvisory/cisco-sa-20190501-asaftd-saml-vpn',]
for url in urls:
yield SplashRequest(
url=url,callback=self.save_response,endpoint='render.html',cb_kwargs=dict(path_dir=HTML_DIR),args={
'wait': 1.0,'timeout': 90.0,},)
def save_response(self,response,path_dir,filename=None):
if not filename:
filename = self.norm_url(response.url)
print('===')
print(filename)
print('===')
with open(path_dir / filename,'wb') as f:
f.write(response.body)
settings.py
BOT_NAME = 'splash'
SPIDER_MODULES = ['splash.spiders']
NEWSPIDER_MODULE = 'splash.spiders'
ROBOTSTXT_OBEY = False
DOWNLOAD_DELAY = 2
SPLASH_URL = 'http://172.35.0.1:8050'
DOWNLOADER_MIDDLEWARES = {
'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware': None,'scrapy_user_agents.middlewares.RandomUserAgentMiddleware': 400,'scrapy_splash.SplashCookiesMiddleware': 723,'scrapy_splash.SplashMiddleware': 725,'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware': 810,}
SPIDER_MIDDLEWARES = {
'scrapy_splash.SplashDeduplicateArgsMiddleware': 100,}
DUPEFILTER_CLASS = 'scrapy_splash.SplashAwareDupeFilter'
日志：
docker-compose -f $(pwd)/tor/docker-compose.yml up -d
Creating network "tor_tor_net" with driver "bridge"
Creating tor ... done
docker-compose -f $(pwd)/splash/docker-compose.yml up -d
Creating network "splash_splash_net" with driver "bridge"
Creating splash ... done
docker run --rm --env-file .secrets --env-file .settings \
--name cisco-crawler \
-e data_dir=/data \
-v $(pwd)/data/crawlers/cisco:/data \
-v $(pwd)/crawlers:/code \
docker-secdev.ptsecurity.ru/crawler-base:fixed_cisco_crawler \
cisco
2020-08-18 15:51:01 [scrapy.utils.log] INFO: Scrapy 2.3.0 started (bot: splash)
2020-08-18 15:51:01 [scrapy.utils.log] INFO: Versions: lxml 4.5.2.0,libxml2 2.9.10,cssselect 1.1.0,parsel 1.6.0,w3lib 1.22.0,Twisted 20.3.0,Python 3.7.8 (default,Aug 5 2020,08:30:24) - [GCC 8.3.0],pyOpenSSL 19.1.0 (OpenSSL 1.1.1g 21 Apr 2020),cryptography 3.0,Platform Linux-5.4.0-42-generic-x86_64-with-debian-10.5
2020-08-18 15:51:01 [scrapy.utils.log] DEBUG: Using reactor: twisted.internet.epollreactor.EPollReactor
2020-08-18 15:51:01 [scrapy.crawler] INFO: Overridden settings:
{'BOT_NAME': 'splash','DOWNLOAD_DELAY': 2,'DUPEFILTER_CLASS': 'scrapy_splash.SplashAwareDupeFilter','NEWSPIDER_MODULE': 'splash.spiders','SPIDER_MODULES': ['splash.spiders']}
2020-08-18 15:51:01 [scrapy.extensions.telnet] INFO: Telnet Password: a2db66b61d0f2c0d
2020-08-18 15:51:01 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats','scrapy.extensions.telnet.TelnetConsole','scrapy.extensions.memusage.MemoryUsage','scrapy.extensions.logstats.LogStats']
2020-08-18 15:51:03 [scrapy_user_agents.user_agent_picker] WARNING: [UnsupportedbrowserType] Family: Android
2020-08-18 15:51:03 [scrapy_user_agents.user_agent_picker] WARNING: [UnsupportedDeviceType] Family: Other,Brand: None,Model: None
2020-08-18 15:51:03 [scrapy_user_agents.user_agent_picker] WARNING: [UnsupportedDeviceType] Family: Other,Model: None
2020-08-18 15:51:04 [scrapy_user_agents.user_agent_picker] WARNING: [UnsupportedDeviceType] Family: Other,Model: None
2020-08-18 15:51:04 [scrapy_user_agents.user_agent_picker] WARNING: [UnsupportedDeviceType] Family: Sony BDV13,Brand: Sony,Model: BDV13
2020-08-18 15:51:04 [scrapy_user_agents.user_agent_picker] WARNING: [UnsupportedDeviceType] Family: Nintendo DSi,Brand: Nintendo,Model: DSi
2020-08-18 15:51:04 [scrapy_user_agents.user_agent_picker] WARNING: [UnsupportedDeviceType] Family: Other,Model: None
2020-08-18 15:51:04 [scrapy_user_agents.user_agent_picker] WARNING: [UnsupportedbrowserType] Family: Other
2020-08-18 15:51:04 [scrapy_user_agents.user_agent_picker] WARNING: [UnsupportedDeviceType] Family: LYF F90M,Brand: LYF,Model: F90M
2020-08-18 15:51:05 [scrapy_user_agents.user_agent_picker] WARNING: [UnsupportedDeviceType] Family: Sony BDV14,Model: BDV14
2020-08-18 15:51:05 [scrapy_user_agents.user_agent_picker] WARNING: [UnsupportedDeviceType] Family: LG Web0S SmartTV,Brand: LG,Model: Web0S SmartTV
2020-08-18 15:51:05 [scrapy_user_agents.user_agent_picker] WARNING: [UnsupportedDeviceType] Family: Other,Model: None
2020-08-18 15:51:06 [scrapy_user_agents.user_agent_picker] WARNING: [UnsupportedDeviceType] Family: Other,Model: None
2020-08-18 15:51:06 [scrapy_user_agents.user_agent_picker] WARNING: [UnsupportedDeviceType] Family: Sony BDV11,Model: BDV11
2020-08-18 15:51:06 [scrapy_user_agents.user_agent_picker] WARNING: [UnsupportedDeviceType] Family: Other,Model: None
2020-08-18 15:51:07 [scrapy_user_agents.user_agent_picker] WARNING: [UnsupportedDeviceType] Family: Nintendo DSi,Model: DSi
2020-08-18 15:51:07 [scrapy_user_agents.user_agent_picker] WARNING: [UnsupportedDeviceType] Family: LYF LF-2403N,Model: LF-2403N
2020-08-18 15:51:07 [scrapy_user_agents.user_agent_picker] WARNING: [UnsupportedDeviceType] Family: LYF F90M,Model: F90M
2020-08-18 15:51:07 [scrapy_user_agents.user_agent_picker] WARNING: [UnsupportedDeviceType] Family: Other,Model: None
2020-08-18 15:51:08 [scrapy_user_agents.user_agent_picker] WARNING: [UnsupportedDeviceType] Family: Other,Model: None
2020-08-18 15:51:08 [scrapy_user_agents.user_agent_picker] WARNING: [UnsupportedbrowserType] Family: applebot
2020-08-18 15:51:08 [scrapy_user_agents.user_agent_picker] WARNING: [UnsupportedDeviceType] Family: Other,Model: None
2020-08-18 15:51:08 [scrapy_user_agents.user_agent_picker] WARNING: [UnsupportedbrowserType] Family: applebot
2020-08-18 15:51:09 [scrapy_user_agents.user_agent_picker] WARNING: [UnsupportedbrowserType] Family: Other
2020-08-18 15:51:09 [scrapy_user_agents.user_agent_picker] WARNING: [UnsupportedbrowserType] Family: Other
2020-08-18 15:51:09 [scrapy_user_agents.user_agent_picker] WARNING: [UnsupportedbrowserType] Family: PhantomJS
2020-08-18 15:51:09 [scrapy_user_agents.user_agent_picker] WARNING: [UnsupportedbrowserType] Family: SMTBot
2020-08-18 15:51:10 [scrapy_user_agents.user_agent_picker] WARNING: [UnsupportedbrowserType] Family: PhantomJS
2020-08-18 15:51:10 [scrapy_user_agents.user_agent_picker] WARNING: [UnsupportedbrowserType] Family: PhantomJS
2020-08-18 15:51:10 [scrapy_user_agents.user_agent_picker] WARNING: [UnsupportedbrowserType] Family: WebKit Nightly
2020-08-18 15:51:10 [scrapy_user_agents.user_agent_picker] WARNING: [UnsupportedbrowserType] Family: PhantomJS
2020-08-18 15:51:10 [scrapy_user_agents.user_agent_picker] WARNING: [UnsupportedDeviceType] Family: Other,Model: None
2020-08-18 15:51:11 [scrapy_user_agents.user_agent_picker] WARNING: [UnsupportedbrowserType] Family: PhantomJS
2020-08-18 15:51:11 [scrapy_user_agents.user_agent_picker] WARNING: [UnsupportedbrowserType] Family: PhantomJS
2020-08-18 15:51:11 [scrapy_user_agents.user_agent_picker] WARNING: [UnsupportedbrowserType] Family: Robot
2020-08-18 15:51:11 [scrapy_user_agents.user_agent_picker] WARNING: [UnsupportedDeviceType] Family: Other,Model: None
2020-08-18 15:51:11 [scrapy_user_agents.user_agent_picker] WARNING: [UnsupportedbrowserType] Family: Zune
2020-08-18 15:51:11 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware','scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware','scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware','scrapy_user_agents.middlewares.RandomUserAgentMiddleware','scrapy.downloadermiddlewares.retry.RetryMiddleware','scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware','scrapy.downloadermiddlewares.redirect.RedirectMiddleware','scrapy.downloadermiddlewares.cookies.CookiesMiddleware','scrapy_splash.SplashCookiesMiddleware','scrapy_splash.SplashMiddleware','scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware','scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware','scrapy.downloadermiddlewares.stats.DownloaderStats']
2020-08-18 15:51:11 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware','scrapy_splash.SplashDeduplicateArgsMiddleware','scrapy.spidermiddlewares.offsite.OffsiteMiddleware','scrapy.spidermiddlewares.referer.RefererMiddleware','scrapy.spidermiddlewares.urllength.UrlLengthMiddleware','scrapy.spidermiddlewares.depth.DepthMiddleware']
2020-08-18 15:51:11 [scrapy.middleware] INFO: Enabled item pipelines:
[]
2020-08-18 15:51:11 [scrapy.core.engine] INFO: Spider opened
2020-08-18 15:51:11 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min),scraped 0 items (at 0 items/min)
2020-08-18 15:51:11 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023
2020-08-18 15:51:11 [py.warnings] WARNING: /usr/local/lib/python3.7/site-packages/scrapy_splash/request.py:41: ScrapyDeprecationWarning: Call to deprecated function to_native_str. Use to_unicode instead.
url = to_native_str(url)
2020-08-18 15:51:11 [scrapy_user_agents.middlewares] DEBUG: Assigned User-Agent Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML,like Gecko) Chrome/66.0.3359.181 Safari/537.36
2020-08-18 15:51:11 [scrapy_user_agents.middlewares] DEBUG: Assigned User-Agent Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML,like Gecko) Chrome/60.0.3112.113 Safari/537.36
2020-08-18 15:51:11 [scrapy_user_agents.middlewares] DEBUG: Assigned User-Agent Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML,like Gecko) Chrome/67.0.3396.62 Safari/537.36
2020-08-18 15:51:11 [scrapy_user_agents.middlewares] DEBUG: Assigned User-Agent Mozilla/5.0 (Windows NT 6.0) AppleWebKit/537.36 (KHTML,like Gecko) Chrome/49.0.2623.112 Safari/537.36
2020-08-18 15:51:11 [scrapy_user_agents.middlewares] DEBUG: Assigned User-Agent Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML,like Gecko) Chrome/50.0.2661.75 Safari/537.36
2020-08-18 15:51:11 [scrapy_user_agents.middlewares] DEBUG: Assigned User-Agent Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML,like Gecko) Chrome/63.0.3239.132 Safari/537.36
2020-08-18 15:51:11 [scrapy_user_agents.middlewares] DEBUG: Assigned User-Agent Mozilla/5.0 (Windows NT 5.1; WOW64) AppleWebKit/537.36 (KHTML,like Gecko) Chrome/46.0.2490.80 Safari/537.36
2020-08-18 15:51:11 [scrapy_user_agents.middlewares] DEBUG: Assigned User-Agent Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML,like Gecko) Chrome/63.0.3239.84 Safari/537.36
2020-08-18 15:51:11 [scrapy_user_agents.middlewares] DEBUG: Assigned User-Agent Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML,like Gecko) Chrome/56.0.2924.87 Safari/537.36
2020-08-18 15:51:11 [scrapy_user_agents.middlewares] DEBUG: Assigned User-Agent Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML,like Gecko) Chrome/62.0.3202.89 Safari/537.36
2020-08-18 15:51:31 [scrapy.downloadermiddlewares.retry] DEBUG: retrying <GET https://tools.cisco.com/security/center/content/CiscoSecurityAdvisory/cisco-sa-ucs-director-xss-O7T8ORYR via http://172.35.0.1:8050/render.html> (Failed 1 times): 502 Bad Gateway
2020-08-18 15:51:31 [scrapy_user_agents.middlewares] DEBUG: Assigned User-Agent Mozilla/5.0 (Windows NT 5.1; WOW64) AppleWebKit/537.36 (KHTML,like Gecko) Chrome/52.0.2743.116 Safari/537.36
2020-08-18 15:51:31 [scrapy.downloadermiddlewares.retry] DEBUG: retrying <GET https://tools.cisco.com/security/center/content/CiscoSecurityAdvisory/cisco-sa-dcnm-stored-xss-yJyqBJGU via http://172.35.0.1:8050/render.html> (Failed 1 times): 502 Bad Gateway
2020-08-18 15:51:31 [scrapy_user_agents.middlewares] DEBUG: Assigned User-Agent Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML,like Gecko) Chrome/56.0.2924.87 Safari/537.36
2020-08-18 15:51:38 [scrapy.downloadermiddlewares.retry] DEBUG: retrying <GET https://tools.cisco.com/security/center/content/CiscoSecurityAdvisory/cisco-sa-code-exec-wH3BNFb via http://172.35.0.1:8050/render.html> (Failed 1 times): 502 Bad Gateway
2020-08-18 15:51:38 [scrapy_user_agents.middlewares] DEBUG: Assigned User-Agent Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML,like Gecko) Chrome/39.0.2171.95 Safari/537.36
2020-08-18 15:51:41 [scrapy.downloadermiddlewares.retry] DEBUG: retrying <GET https://tools.cisco.com/security/center/content/CiscoSecurityAdvisory/cisco-sa-dcnm-stored-xss-yJyqBJGU via http://172.35.0.1:8050/render.html> (Failed 2 times): 502 Bad Gateway
2020-08-18 15:51:41 [scrapy_user_agents.middlewares] DEBUG: Assigned User-Agent Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML,like Gecko) Chrome/50.0.2661.94 Safari/537.36
2020-08-18 15:51:46 [scrapy.downloadermiddlewares.retry] DEBUG: retrying <GET https://tools.cisco.com/security/center/content/CiscoSecurityAdvisory/cisco-sa-code-exec-wH3BNFb via http://172.35.0.1:8050/render.html> (Failed 2 times): 502 Bad Gateway
2020-08-18 15:51:46 [scrapy_user_agents.middlewares] DEBUG: Assigned User-Agent Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML,like Gecko) Chrome/58.0.3029.81 Safari/537.36
2020-08-18 15:52:06 [scrapy.downloadermiddlewares.retry] DEBUG: retrying <GET https://tools.cisco.com/security/center/content/CiscoSecurityAdvisory/cisco-sa-ucs-director-xss-O7T8ORYR via http://172.35.0.1:8050/render.html> (Failed 2 times): 502 Bad Gateway
2020-08-18 15:52:06 [scrapy_user_agents.middlewares] DEBUG: Assigned User-Agent Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML,like Gecko) Chrome/66.0.3359.170 Safari/537.36
2020-08-18 15:52:06 [scrapy.downloadermiddlewares.retry] DEBUG: retrying <GET https://tools.cisco.com/security/center/content/CiscoSecurityAdvisory/cisco-sa-20190807-wms-oredirect via http://172.35.0.1:8050/render.html> (Failed 1 times): 502 Bad Gateway
2020-08-18 15:52:06 [scrapy_user_agents.middlewares] DEBUG: Assigned User-Agent Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML,like Gecko) Chrome/50.0.2661.94 Safari/537.36
2020-08-18 15:52:11 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min),scraped 0 items (at 0 items/min)
2020-08-18 15:52:51 [scrapy.downloadermiddlewares.retry] DEBUG: retrying <GET https://tools.cisco.com/security/center/content/CiscoSecurityAdvisory/cisco-sa-20190501-asaftd-saml-vpn via http://172.35.0.1:8050/render.html> (Failed 1 times): 504 Gateway Time-out
2020-08-18 15:52:51 [scrapy_user_agents.middlewares] DEBUG: Assigned User-Agent Mozilla/5.0 (Windows NT 6.2; Win64; x64) AppleWebKit/537.36 (KHTML,like Gecko) Chrome/57.0.2987.133 Safari/537.36
2020-08-18 15:52:59 [scrapy.downloadermiddlewares.retry] ERROR: Gave up retrying <GET https://tools.cisco.com/security/center/content/CiscoSecurityAdvisory/cisco-sa-code-exec-wH3BNFb via http://172.35.0.1:8050/render.html> (Failed 3 times): 502 Bad Gateway
2020-08-18 15:52:59 [scrapy.core.engine] DEBUG: Crawled (502) <GET https://tools.cisco.com/security/center/content/CiscoSecurityAdvisory/cisco-sa-code-exec-wH3BNFb via http://172.35.0.1:8050/render.html> (referer: None)
2020-08-18 15:52:59 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <502 https://tools.cisco.com/security/center/content/CiscoSecurityAdvisory/cisco-sa-code-exec-wH3BNFb>: HTTP status code is not handled or not allowed
2020-08-18 15:53:11 [scrapy.downloadermiddlewares.retry] ERROR: Gave up retrying <GET https://tools.cisco.com/security/center/content/CiscoSecurityAdvisory/cisco-sa-dcnm-stored-xss-yJyqBJGU via http://172.35.0.1:8050/render.html> (Failed 3 times): 504 Gateway Time-out
2020-08-18 15:53:11 [scrapy.core.engine] DEBUG: Crawled (504) <GET https://tools.cisco.com/security/center/content/CiscoSecurityAdvisory/cisco-sa-dcnm-stored-xss-yJyqBJGU via http://172.35.0.1:8050/render.html> (referer: None)
2020-08-18 15:53:11 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <504 https://tools.cisco.com/security/center/content/CiscoSecurityAdvisory/cisco-sa-dcnm-stored-xss-yJyqBJGU>: HTTP status code is not handled or not allowed
2020-08-18 15:53:11 [scrapy.extensions.logstats] INFO: Crawled 2 pages (at 2 pages/min),scraped 0 items (at 0 items/min)
2020-08-18 15:53:25 [scrapy.downloadermiddlewares.retry] ERROR: Gave up retrying <GET https://tools.cisco.com/security/center/content/CiscoSecurityAdvisory/cisco-sa-ucs-director-xss-O7T8ORYR via http://172.35.0.1:8050/render.html> (Failed 3 times): 502 Bad Gateway
2020-08-18 15:53:25 [scrapy.core.engine] DEBUG: Crawled (502) <GET https://tools.cisco.com/security/center/content/CiscoSecurityAdvisory/cisco-sa-ucs-director-xss-O7T8ORYR via http://172.35.0.1:8050/render.html> (referer: None)
2020-08-18 15:53:25 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <502 https://tools.cisco.com/security/center/content/CiscoSecurityAdvisory/cisco-sa-ucs-director-xss-O7T8ORYR>: HTTP status code is not handled or not allowed
2020-08-18 15:53:39 [scrapy.downloadermiddlewares.retry] DEBUG: retrying <GET https://tools.cisco.com/security/center/content/CiscoSecurityAdvisory/cisco-sa-20190807-wms-oredirect via http://172.35.0.1:8050/render.html> (Failed 2 times): 504 Gateway Time-out
2020-08-18 15:53:39 [scrapy_user_agents.middlewares] DEBUG: Assigned User-Agent Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML,like Gecko) Chrome/64.0.3282.186 Safari/537.36
2020-08-18 15:54:11 [scrapy.extensions.logstats] INFO: Crawled 3 pages (at 1 pages/min),scraped 0 items (at 0 items/min)
2020-08-18 15:54:21 [scrapy.downloadermiddlewares.retry] DEBUG: retrying <GET https://tools.cisco.com/security/center/content/CiscoSecurityAdvisory/cisco-sa-20190501-asaftd-saml-vpn via http://172.35.0.1:8050/render.html> (Failed 2 times): 504 Gateway Time-out
2020-08-18 15:54:21 [scrapy_user_agents.middlewares] DEBUG: Assigned User-Agent Mozilla/5.0 (Windows NT 6.0; Win64; x64) AppleWebKit/537.36 (KHTML,like Gecko) Chrome/62.0.3202.89 Safari/537.36
2020-08-18 15:55:03 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://tools.cisco.com/security/center/content/CiscoSecurityAdvisory/cisco-sa-20190501-asaftd-saml-vpn via http://172.35.0.1:8050/render.html> (referer: None)
===
tools.cisco.com-security-center-content-CiscoSecurityAdvisory-cisco-sa-20190501-asaftd-saml-vpn
===
2020-08-18 15:55:05 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://tools.cisco.com/security/center/content/CiscoSecurityAdvisory/cisco-sa-20190807-wms-oredirect via http://172.35.0.1:8050/render.html> (referer: None)
===
tools.cisco.com-security-center-content-CiscoSecurityAdvisory-cisco-sa-20190807-wms-oredirect
===
2020-08-18 15:55:05 [scrapy.core.engine] INFO: Closing spider (finished)
2020-08-18 15:55:05 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 12060,'downloader/request_count': 15,'downloader/request_method_count/POST': 15,'downloader/response_bytes': 454960,'downloader/response_count': 15,'downloader/response_status_count/200': 2,'downloader/response_status_count/502': 9,'downloader/response_status_count/504': 4,'elapsed_time_seconds': 233.602478,'finish_reason': 'finished','finish_time': datetime.datetime(2020,8,18,15,55,5,265746),'httperror/response_ignored_count': 3,'httperror/response_ignored_status_count/502': 2,'httperror/response_ignored_status_count/504': 1,'log_count/DEBUG': 35,'log_count/ERROR': 3,'log_count/INFO': 16,'log_count/WARNING': 46,'memusage/max': 59252736,'memusage/startup': 58208256,'response_received_count': 5,'retry/count': 10,'retry/max_reached': 3,'retry/reason_count/502 Bad Gateway': 7,'retry/reason_count/504 Gateway Time-out': 3,'scheduler/dequeued': 20,'scheduler/dequeued/memory': 20,'scheduler/enqueued': 20,'scheduler/enqueued/memory': 20,'splash/render.html/request_count': 5,'splash/render.html/response_count/200': 2,'splash/render.html/response_count/502': 9,'splash/render.html/response_count/504': 4,'start_time': datetime.datetime(2020,51,11,663268)}
2020-08-18 15:55:05 [scrapy.core.engine] INFO: Spider closed (finished)
docker-compose -f $(pwd)/splash/docker-compose.yml down
Stopping splash ... done
Removing splash ... done
Removing network splash_splash_net
docker-compose -f $(pwd)/tor/docker-compose.yml down
Stopping tor ... done
Removing tor ... done
Removing network tor_tor_net`
解决方法
暂无找到可以解决该程序问题的有效方法，小编努力寻找整理中！
如果你已经找到好的解决方法，欢迎将解决方案带上本链接一起发送给小编。
小编邮箱:dio#foxmail.com (将#修改为@）