问题描述
我的GUI有一个“更新数据库”按钮,每当用户按下该按钮时,我都想启动一个Scrapy蜘蛛,将抓取的数据存储到sqlite3数据库中。我将qt5reactor
的实现方式设置为this answer suggests,但是当我第二次按下更新按钮时,却遇到了ReactorNotRestartable
错误。我该如何解决?我尝试从CrawlerRunner
切换到CrawlerProcess
,但是它仍然会引发相同的错误(但是,也许我做错了)。我也不能使用this answer,因为q.get()
锁定了事件循环,因此在运行Spider时GUI冻结。我是多处理人员的新手,如果我遗漏了一些非常明显的内容,对不起。
在main.py
... # PyQt5 imports
import qt5reactor
from scrapy import crawler
from twisted.internet import reactor
from currency_scraper.currency_scraper.spiders.investor import InvestorSpider
class MyGUI(QMainWindow):
def __init__(self):
self.update_db_button.clicked.connect(self.on_clicked_update)
...
def on_clicked_update(self):
"""Gives command to run scraper and fetch data from the website"""
runner = crawler.CrawlerRunner(
{
"USER_AGENT": "currency scraper","SCRAPY_SETTINGS_MODULE": "currency_scraper.currency_scraper.settings","ITEM_PIPELInes": {
"currency_scraper.currency_scraper.pipelines.sqlite3Pipeline": 300,}
}
)
deferred = runner.crawl(InvestorSpider)
deferred.addBoth(lambda _: reactor.stop())
reactor.run() # has to be run here or the crawling doesn't start
update_notification()
... # other stuff
if __name__ == "__main__":
open_window()
qt5reactor.install()
reactor.run()
错误日志:
Traceback (most recent call last):
File "c:/Users/Familia/Documents/ProgramaþÒo/Python/Projetos/Currency_converter/main.py",line 330,in on_clicked_update
reactor.run()
File "c:\Users\Familia\Documents\ProgramaþÒo\Python\Projetos\Currency_converter\venv\lib\site-packages\twisted\internet\base.py",line 1282,in run
self.startRunning(installSignalHandlers=installSignalHandlers)
File "c:\Users\Familia\Documents\ProgramaþÒo\Python\Projetos\Currency_converter\venv\lib\site-packages\twisted\internet\base.py",line 1262,in startRunning
ReactorBase.startRunning(self)
File "c:\Users\Familia\Documents\ProgramaþÒo\Python\Projetos\Currency_converter\venv\lib\site-packages\twisted\internet\base.py",line 765,in startRunning
raise error.ReactorNotRestartable()
twisted.internet.error.ReactorNotRestartable
解决方法
暂无找到可以解决该程序问题的有效方法,小编努力寻找整理中!
如果你已经找到好的解决方法,欢迎将解决方案带上本链接一起发送给小编。
小编邮箱:dio#foxmail.com (将#修改为@)