当 CSS 被禁用时，HTMLUnit 不会执行 JavaScript 并加载所有内容

问题描述

我一直在尝试使用 HTMLUnit 从 ethermine.org 上的矿工地址仪表板中抓取页面数据。一个示例 URL 是：

https://ethermine.org/miners/32885f190a878d6db68cbC2889a3FF3abE3C257C/dashboard

调用 webClient.getoptions().setCssEnabled(false); 会阻止 javascript 加载表数据。此方法的文档是：

启用/禁用 CSS 支持。默认情况下，启用此属性。如果禁用 HtmlUnit 将不会下载链接的 css 文件，也不会触发相关的 onload/onerror 事件。

所以，我假设发生的事情是仅在 CSS 成功加载时才执行 Javascript。启用 CSS 会使我的用例加载页面时间过长。所以，我在问是否有一种方法可以让 Javascript 运行以从此页面加载表数据，同时禁用 CSS。也许这可以通过触发 onload 事件来完成？如何才能做到这一点。谢谢

这是代码和日志输出：

public class Main {


    public static void main(String[] args) {
        WebClient webClient = new WebClient();
        webClient.getoptions().setCssEnabled(false); // comment this line out and the active worker table data loads just fine
        webClient.setAjaxController(new NicelyResynchronizingAjaxController());

        try {

            WebRequest request = new WebRequest(new URL("https://ethermine.org/miners/32885f190a878d6db68cbC2889a3FF3abE3C257C/dashboard"));
            HtmlPage page = webClient.getPage(request);

            webClient.waitForBackgroundJavaScript(5000); // long enough for all javascript to execute

            System.out.println(page.asText());

        } catch (IOException e) {
            e.printstacktrace();
        }
    }
}

这是禁用 CSS 的输出：

Feb 08,2021 5:17:09 PM com.gargoylesoftware.htmlunit.WebConsole error
SEVERE: TypeError: Cannot find function get in object .
Feb 08,2021 5:17:09 PM com.gargoylesoftware.htmlunit.WebConsole error
SEVERE: TypeError: Cannot find function isEqualNode in object [object HTMLLinkElement].
Feb 08,2021 5:17:10 PM com.gargoylesoftware.htmlunit.NicelyResynchronizingAjaxController processSynchron
INFO: Re-synchronized call to https://ethermine.org/css/chunk-vendors.391ed6f9.css
Feb 08,2021 5:17:10 PM com.gargoylesoftware.htmlunit.NicelyResynchronizingAjaxController processSynchron
INFO: Re-synchronized call to https://ethermine.org/css/app.69e6b25a.css
Feb 08,2021 5:17:10 PM com.gargoylesoftware.htmlunit.NicelyResynchronizingAjaxController processSynchron
INFO: Re-synchronized call to https://ethermine.org/css/Dashboard.eb03c99b.css
Feb 08,2021 5:17:11 PM com.gargoylesoftware.htmlunit.WebConsole warn
WARNING: cssVars(): variable "--fa-primary-color" is undefined
Feb 08,2021 5:17:11 PM com.gargoylesoftware.htmlunit.WebConsole warn
WARNING: cssVars(): variable "--fa-primary-opacity" is undefined
Feb 08,2021 5:17:11 PM com.gargoylesoftware.htmlunit.WebConsole warn
WARNING: cssVars(): variable "--fa-secondary-color" is undefined
Feb 08,2021 5:17:11 PM com.gargoylesoftware.htmlunit.WebConsole warn
WARNING: cssVars(): variable "--fa-secondary-opacity" is undefined
Feb 08,2021 5:17:11 PM com.gargoylesoftware.htmlunit.WebConsole warn
WARNING: cssVars(): variable "--fa-primary-opacity" is undefined
Home - Ethermine - Ethereum (ETH) mining pool
ETH
ETH
ZEC
YEC
ETC
BEAM
RVN
unchecked
Home
Statistics
Luck
Support
Start Mining
News
News
Legal Notices
ImprintPrivacy PolicyTerms of Operation
Resources
Help Desk3rd Party AppsAPI
Pools
ethermine.orgetc.ethermine.orgzcash.flypool.orgycash.flypool.orgbeam.flypool.orgravencoin.flypool.org
Other Platforms
etherchain.orgpoolwatch.iozcha.inyec.zcha.inbeaconcha.in

这是启用 CSS 的日志输出：

Feb 08,2021 5:18:02 PM com.gargoylesoftware.htmlunit.WebConsole error
SEVERE: TypeError: Cannot find function get in object .
Feb 08,2021 5:18:02 PM com.gargoylesoftware.htmlunit.WebConsole error
SEVERE: TypeError: Cannot find function isEqualNode in object [object HTMLLinkElement].
Feb 08,2021 5:18:03 PM com.gargoylesoftware.htmlunit.NicelyResynchronizingAjaxController processSynchron
INFO: Re-synchronized call to https://api.ethermine.org/miner/32885f190a878d6db68cbC2889a3FF3abE3C257C/dashboard
Feb 08,2021 5:18:03 PM com.gargoylesoftware.htmlunit.NicelyResynchronizingAjaxController processSynchron
INFO: Re-synchronized call to https://api.ethermine.org/miner/32885f190a878d6db68cbC2889a3FF3abE3C257C/dashboard/payouts
Feb 08,2021 5:18:03 PM com.gargoylesoftware.htmlunit.NicelyResynchronizingAjaxController processSynchron
INFO: Re-synchronized call to https://api.ethermine.org/poolStats
Feb 08,2021 5:18:03 PM com.gargoylesoftware.htmlunit.NicelyResynchronizingAjaxController processSynchron
INFO: Re-synchronized call to https://api.ethermine.org/miner/32885f190a878d6db68cbC2889a3FF3abE3C257C/settings
Feb 08,2021 5:18:03 PM com.gargoylesoftware.htmlunit.NicelyResynchronizingAjaxController processSynchron
INFO: Re-synchronized call to https://api.ethermine.org/miner/32885f190a878d6db68cbC2889a3FF3abE3C257C/dashboard/msg
Feb 08,2021 5:18:07 PM com.gargoylesoftware.htmlunit.NicelyResynchronizingAjaxController processSynchron
INFO: Re-synchronized call to https://ethermine.org/css/chunk-vendors.391ed6f9.css
Feb 08,2021 5:18:07 PM com.gargoylesoftware.htmlunit.NicelyResynchronizingAjaxController processSynchron
INFO: Re-synchronized call to https://ethermine.org/css/app.69e6b25a.css
Feb 08,2021 5:18:07 PM com.gargoylesoftware.htmlunit.NicelyResynchronizingAjaxController processSynchron
INFO: Re-synchronized call to https://ethermine.org/css/Dashboard.eb03c99b.css
Feb 08,2021 5:18:08 PM com.gargoylesoftware.htmlunit.WebConsole warn
WARNING: cssVars(): variable "--fa-primary-color" is undefined
Feb 08,2021 5:18:08 PM com.gargoylesoftware.htmlunit.WebConsole warn
WARNING: cssVars(): variable "--fa-primary-opacity" is undefined
Feb 08,2021 5:18:08 PM com.gargoylesoftware.htmlunit.WebConsole warn
WARNING: cssVars(): variable "--fa-secondary-color" is undefined
Feb 08,2021 5:18:08 PM com.gargoylesoftware.htmlunit.WebConsole warn
WARNING: cssVars(): variable "--fa-secondary-opacity" is undefined
Feb 08,2021 5:18:08 PM com.gargoylesoftware.htmlunit.WebConsole warn
WARNING: cssVars(): variable "--fa-primary-opacity" is undefined
Home - Ethermine - Ethereum (ETH) mining pool
ETH
ZEC
YEC
ETC
BEAM
RVN
unchecked
Home
Statistics
Luck
Support
Start Mining
News
0x32885f190a878d6db68cbC2889a3FF3abE3C257C
Dashboard
Payouts
Settings
Help Desk
Auto-Updateunchecked
Workers Active / Inactive
0/0
Unpaid Balance
0.00012 ETH
Estimated Earnings
0.00016 ETH
Daily
Hashrate
Current (GH/s)
0.0
Average (GH/s)
0.0
Reported (GH/s)
0.0
Shares
Valid 99%
0
Stale 1%
0
Invalid
0
Created with Highcharts 7.1.2HashrateHashrateCurrent HashrateAverage HashrateReported Hashrate18:0020:0022:008. Feb02:0004:0006:0008:0010:0012:0014:0016:0010.0 GH/s10.2 GH/s10.4 GH/s10.6 GH/s10.8 GH/s
Created with Highcharts 7.1.2SharesWorkeRSShares & WorkersValid SharesWorkeRSStale Shares2021
Active Workers (57)
Name    Reported Hashrate   Current Hashrate    Valid Shares    Stale Shares    Invalid Shares  Last seen
1-1 183.7   188.8   168 3   0   8 minutes ago
1-2 183.8   188.8   168 3   0   9 minutes ago
1-3 183.8   161.8   145 1   0   8 minutes ago
1-4 183.8   179.9   160 3   0   8 minutes ago
10-1    192.2   182.2   162 3   0   9 minutes ago
10-2    194.5   192.9   173 1   0   8 minutes ago
10-3    192.3   196.3   176 1   0   8 minutes ago
10-4    189.3   209.2   185 5   0   9 minutes ago
11-1    192.0   179.2   158 5   0   9 minutes ago
11-2    168.9   149.9   133 3   0   8 minutes ago
11-3    194.8   177.7   158 3   0   8 minutes ago
11-4    198.6   163.7   146 2   0   9 minutes ago
12-1    190.9   207.8   187 0   0   9 minutes ago
12-2    163.9   169.9   151 3   0   8 minutes ago
12-3    170.1   188.5   169 1   0   9 minutes ago
12-4    170.2   145.9   130 2   0   8 minutes ago
13-1    167.9   182.2   162 3   0   8 minutes ago
13-2    182.4   157.4   141 1   0   8 minutes ago
13-3    186.4   187.0   167 2   0   8 minutes ago
13-4    166.2   156.3   140 1   0   9 minutes ago
14-1    202.2   185.8   164 5   0   8 minutes ago
14-2    269.4   283.3   253 3   0   8 minutes ago
15-1    265.2   245.7   214 11  0   8 minutes ago
15-2    167.8   186.7   168 0   0   8 minutes ago
15-3    161.3   154.4   139 0   0   8 minutes ago
2-1 181.5   164.4   146 3   0   8 minutes ago
2-2 174.7   181.4   162 2   0   8 minutes ago
2-3 188.3   188.4   167 4   0   8 minutes ago
2-4 188.3   198.1   175 5   0   8 minutes ago
3-1 192.6   170.3   152 2   0   8 minutes ago
3-2 183.7   179.6   161 1   0   9 minutes ago
3-3 188.3   189.6   170 1   0   8 minutes ago
3-4 164.4   162.5   143 5   0   9 minutes ago
4-1 168.4   150.0   135 0   0   8 minutes ago
4-2 192.7   191.1   172 0   0   8 minutes ago
4-3 192.5   211.1   190 0   0   8 minutes ago
4-4 192.4   182.9   164 1   0   8 minutes ago
5-1 168.4   176.6   157 3   0   9 minutes ago
5-2 192.5   210.3   188 2   0   9 minutes ago
5-3 171.3   183.2   161 6   0   8 minutes ago
5-4 193.5   178.5   160 1   0   8 minutes ago
6-1 140.4   145.2   130 1   0   8 minutes ago
6-2 190.9   181.1   163 0   0   8 minutes ago
6-3 191.6   206.7   186 0   0   8 minutes ago
6-4 190.4   185.1   164 4   0   8 minutes ago
7-1 194.5   186.3   167 1   0   8 minutes ago
7-2 193.2   179.2   160 2   0   8 minutes ago
7-3 197.2   208.1   186 2   0   8 minutes ago
7-4 194.7   187.0   167 2   0   8 minutes ago
8-1 193.1   188.5   169 1   0   8 minutes ago
8-2 192.3   187.0   167 2   0   8 minutes ago
8-3 146.0   145.4   127 6   0   8 minutes ago
8-4 196.4   218.1   195 2   0   8 minutes ago
9-1 194.6   207.8   187 0   0   8 minutes ago
9-2 191.7   167.8   151 0   0   10 minutes ago
9-3 152.4   151.4   135 2   0   9 minutes ago
9-4 194.2   207.7   185 3   0   9 minutes ago
Legal Notices
ImprintPrivacy PolicyTerms of Operation
Resources
Help Desk3rd Party AppsAPI
Pools
ethermine.orgetc.ethermine.orgzcash.flypool.orgycash.flypool.orgbeam.flypool.orgravencoin.flypool.org
Other Platforms
etherchain.orgpoolwatch.iozcha.inyec.zcha.inbeaconcha.in

解决方法

所以，我假设发生的事情是仅在 CSS 成功加载时才执行 Javascript。

那是错误的 - 没有这样的依赖。

也许 js 代码停止了，因为它依赖于一些 css 东西的存在。检查您的日志，请尝试添加

webClient.getOptions().setThrowExceptionOnScriptError(false);

在第一个错误时不停止 js 执行。 如果这没有帮助，请在 GitHub 上打开一个问题。