如何在Linux服务器上使用docker-compose安装phantomjs和selenium？

编程问答 2022-06-03

问题描述

我正在使用Selenium和phantomjs作为我的Web抓取工具。 我的测试Windows应用程序一切正常。 尝试将此代码更新添加到我的主应用程序中，并使用docker-compose进行了部署，我得到了： m = df.Number > 5 df_that_meets_condition,df_that_does_not_meet_condition = df[m],df[~m]

我该如何解决？ 当前我的docker-compose.yml具有以下代码：

selenium.common.exceptions.WebDriverException: Message: 'phantomjs' executable needs to be in PATH.

还有我的python代码：

version: '3.1'

services:

  tgbot:
    container_name: bot
    build:
      context: .
    command: python app.py
    restart: always
    environment:
      WEBAPP_PORT: 3001
    env_file:
      - ".env"
    # bot start after load db
    ports:
      - 8443:3001
    networks:
      - botnet

  phantomjs:
    image: shufo/phantomjs
    command: --webdriver 8901

networks:
      botnet:
        driver: bridge

文档文件：

from selenium import webdriver
driver = webdriver.PhantomJS()

P.S。我使用phantomjs，因为我正在抓取的网页包含JS。不适用于chrome

解决方法

您的配置存在一些问题：

  1. 您的漫游器代码在不同的容器中工作。不能启动phantomjs。这就是为什么它找不到可执行文件。
  2. 您运行的phantomjs容器与代码不在同一网络中
  3. 有些无用的配置似乎是从其他示例中复制过来的。
  4. 您强制容器重新启动。即使退出代码成功，它也会重新启动。

这是如何运行所有内容的完整示例：

  1. 创建一个空文件夹myfolder并放置在app.py中，其中包含以下内容：
from selenium import webdriver
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities
driver = webdriver.Remote(command_executor='http://phantomjs:8901/wd/hub/',desired_capabilities=DesiredCapabilities.PHANTOMJS)
  1. 将requirements.txt文件放入myfolder
  2. 将Dockerfile后面的内容放入myfolder：
FROM python:latest

WORKDIR /configs
COPY requirements.txt requirements.txt
RUN pip install -r requirements.txt
  1. 将docker-compose.yml之后的内容放入myfolder：
version: '3.1'

services:

  tgbot:
    build: .
    container_name: bot
    volumes:
      - .:/apps
    command: python /apps/app.py
    depends_on:
      - phantomjs
    networks:
      - botnet

  phantomjs:
    container_name: phantomjs
    image: shufo/phantomjs
    command: --webdriver 8901
    networks:
      - botnet

networks:
  botnet:
    driver: bridge
  1. cd myfloder，docker-compose up

输出： 

phantomjs    | [INFO  - 2020-11-10T15:18:11.049Z] GhostDriver - Main - running on port 8901
phantomjs    | [INFO  - 2020-11-10T15:18:11.425Z] Session [f2091fe0-2367-11eb-bcd7-956b9cd40e54] - page.settings - {"XSSAuditingEnabled":false,"javascriptCanCloseWindows":true,"javascriptCanOpenWindows":true,"javascriptEnabled":true,"loadImages":true,"localToRemoteUrlAccessEnabled":false,"userAgent":"Mozilla/5.0 (Unknown; Linux x86_64) AppleWebKit/538.1 (KHTML,like Gecko) PhantomJS/2.1.1 Safari/538.1","webSecurityEnabled":true}
phantomjs    | [INFO  - 2020-11-10T15:18:11.425Z] Session [f2091fe0-2367-11eb-bcd7-956b9cd40e54] - page.customHeaders:  - {}
phantomjs    | [INFO  - 2020-11-10T15:18:11.425Z] Session [f2091fe0-2367-11eb-bcd7-956b9cd40e54] - Session.negotiatedCapabilities - {"browserName":"phantomjs","version":"2.1.1","driverName":"ghostdriver","driverVersion":"1.2.0","platform":"linux-unknown-64bit","takesScreenshot":true,"handlesAlerts":false,"databaseEnabled":false,"locationContextEnabled":false,"applicationCacheEnabled":false,"browserConnectionEnabled":false,"cssSelectorsEnabled":true,"webStorageEnabled":false,"rotatable":false,"acceptSslCerts":false,"nativeEvents":true,"proxy":{"proxyType":"direct"}}
phantomjs    | [INFO  - 2020-11-10T15:18:11.425Z] SessionManagerReqHand - _postNewSessionCommand - New Session Created: f2091fe0-2367-11eb-bcd7-956b9cd40e54
bot exited with code 0
dockerdockerdocker-composephantomjspythonseleniumselenium