使用Pyinstaller将Scrapy打包为更大程序的一部分

编程问答 2022-08-14

问题描述

我正在尝试使用Pyinstaller将Scrapy打包为更大程序的一部分。
在执行代码形式的源代码时，所有内容均按预期运行，但是从可执行文件运行时，一切将返回：

[scrapy.utils.log] INFO: Scrapy 2.3.0 started (bot: NGL)
[scrapy.utils.log] INFO: Versions: lxml 4.5.1.0,libxml2 2.9.10,cssselect 1.1.0,parsel 1.6.0,w3lib 1.22.0,Twisted 20.3.0,Python 3.8.5 (default,Jul 27 2020,08:42:51) - [GCC 10.1.0],pyOpenSSL 19.1.0 (OpenSSL 1.1.1g  21 Apr 2020),cryptography 3.0,Platform Linux-5.8.3-arch1-1-x86_64-with-glibc2.4
Traceback (most recent call last):
  File "scrapy/spiderloader.py",line 76,in load
KeyError: 'ngl'

During handling of the above exception,another exception occurred:

Traceback (most recent call last):
  File "cli.py",line 9,in <module>
  File "ui.py",line 159,in main
  File "ui.py",line 147,in start
  File "ui.py",line 67,in print
  File "NGL/spiders/NGL.py",line 127,in main
  File "scrapy/crawler.py",line 191,in crawl
  File "scrapy/crawler.py",line 224,in create_crawler
  File "scrapy/crawler.py",line 228,in _create_crawler
  File "scrapy/spiderloader.py",line 78,in load
KeyError: 'Spider not found: ngl'

不幸的是，我不知道如何实际调试pyinstaller软件包：/

这是目录树：

.
├── LICENSE
├── main
│   ├── checkers.py
│   ├── cli.py
│   ├── decorators.py
│   ├── fixers.py
│   ├── licences.py
│   ├── NGL
│   │   ├── __init__.py
│   │   ├── items.py
│   │   ├── middlewares.py
│   │   ├── pipelines.py
│   │   └── spiders
│   │       ├── __init__.py
│   │       └── NGL.py
│   ├── scrapy.cfg
│   ├── terminal_tools.py
│   ├── text_tools.py
│   └── ui.py

NGL 中没有 settings.py ，因为我是在 /main/NGL/spiders/NGL.py 中注入直形蜘蛛>与：

def main(url,save_path):
    folder = save_path
    if os.path.exists(folder):
        shutil.rmtree(folder)
    process = CrawlerProcess(
        settings={
            "LOG_ENABLED": True,"LOG_FORMAT": "[%(name)s] %(levelname)s: %(message)s","LOG_LEVEL": "INGO","BOT_NAME": "NGL","SPIDER_MODULES": ["NGL.spiders"],"NEWSPIDER_MODULE": "NGL.spiders","IMAGES_STORE": folder,"ROBOTSTXT_OBEY": True,"ITEM_PIPELINES": {
                "NGL.pipelines.DownloadPipeline": 300,"NGL.pipelines.CleanerPipeline": 600,},"IMAGES_URLS_FIELD": "Image Url","FEEDS": {
                f"{folder}/data.csv": {
                    "format": "csv","encoding": "utf8","fields": [
                        "Inventory number","Full title","Date made","Artist","Artist dates","Medium and support","Dimensions","Overview","In-Depth","Copywright","Image Url","Artwork Url",],}
            },)
    process.crawl("ngl",start_urls=[url])
    process.start()

以及用于打包的命令：

pyinstaller main/cli.py  --clean --onefile --name NGA_linux -p main:main/NGL:main/NGL/spiders

解决方法

暂无找到可以解决该程序问题的有效方法，小编努力寻找整理中！

如果你已经找到好的解决方法，欢迎将解决方案带上本链接一起发送给小编。

小编邮箱:dio#foxmail.com (将#修改为@）

macospyinstallerpythonpython-importscrapy