问题描述
我想抓取 flashscore.com 网页，该网页在 Javascript 访问期间完全呈现。我使用 HtmlUnit 进行渲染，现在已经出现了第一个问题，根本无法抓取页面。
@PostMapping("/startScraping")
public ResponseEntity<FlashScraper> startScraping(@NonNull @RequestBody FlashScraper flashScraper) throws FailingHttpStatusCodeException,MalformedURLException,IOException {
logger.info("startScraping request incomming");
logger.info("Call URL: " + flashScraper.getScrapeUrl());
final WebClient webClient = new WebClient(browserVersion.BEST_SUPPORTED);
HtmlPage scrapePage = webClient.getPage(flashScraper.getScrapeUrl());
webClient.getoptions().setJavaScriptEnabled(true);
webClient.getoptions().setThrowExceptionOnScriptError(false);
webClient.waitForBackgroundJavaScript(3000);
System.out.println(scrapePage.getByXPath("//*[@id=\"g_25_rwPxTVj1\"]"));
return new ResponseEntity(flashScraper,HttpStatus.OK);
}
向 startScraping 端点发送 POST 请求后，出现以下异常
2021-07-04 14:43:57.569 WARN 14872 --- [nio-8080-exec-2] c.g.htmlunit.DefaultCssErrorHandler : CSS warning: 'https://www.flashscore.com/res/_fs/build/livetableresponsive.2da0223.css' [1:8910] Ignoring the whole rule.
2021-07-04 14:43:58.035 WARN 14872 --- [nio-8080-exec-2] c.g.htmlunit.IncorrectnessListenerImpl : Obsolete content type encountered: 'text/javascript'.
2021-07-04 14:43:58.175 ERROR 14872 --- [nio-8080-exec-2] c.g.h.j.DefaultJavaScriptErrorListener : Error during JavaScript execution com.gargoylesoftware.htmlunit.ScriptException: TypeError: Cannot find function entries in object function Object() { [native code] }. (script in https://www.flashscore.com/unsupported/ from (31,9) to (53,10)#35)
at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine$HtmlUnitContextAction.run(JavaScriptEngine.java:949) ~[htmlunit-2.50.0.jar:2.50.0]
at net.sourceforge.htmlunit.corejs.javascript.Context.call(Context.java:598) ~[htmlunit-core-js-2.50.0.jar:na]
at net.sourceforge.htmlunit.corejs.javascript.ContextFactory.call(ContextFactory.java:487) ~[htmlunit-core-js-2.50.0.jar:na]
at com.gargoylesoftware.htmlunit.javascript.HtmlUnitContextFactory.callSecured(HtmlUnitContextFactory.java:353) ~[htmlunit-2.50.0.jar:2.50.0]
您是否怀疑问题出在哪里？
谢谢
解决方法
获取页面前请设置选项
final WebClient webClient = new WebClient(BrowserVersion.BEST_SUPPORTED);
webClient.getOptions().setJavaScriptEnabled(true);
webClient.getOptions().setThrowExceptionOnScriptError(false);
HtmlPage scrapePage = webClient.getPage(flashScraper.getScrapeUrl());
webClient.waitForBackgroundJavaScript(3000);
我想错误仍然存在，但是因为您设置了 setThrowExceptionOnScriptError(false)，执行将不再因错误而停止。
,更改参数顺序后，我仍然遇到以下错误 - HtmlUnit 浏览器也被转发到 flashscore.com/unsupported
2021-07-06 08:46:15.869 WARN 14464 --- [nio-8080-exec-2] c.g.htmlunit.DefaultCssErrorHandler : CSS warning: 'https://www.flashscore.com/res/_fs/build/livetableresponsive.2da0223.css' [1:8910] Ignoring the whole rule.
2021-07-06 08:46:16.638 WARN 14464 --- [nio-8080-exec-2] c.g.htmlunit.IncorrectnessListenerImpl : Obsolete content type encountered: 'text/javascript'.
2021-07-06 08:46:16.783 ERROR 14464 --- [nio-8080-exec-2] c.g.h.j.DefaultJavaScriptErrorListener : Error during JavaScript execution
com.gargoylesoftware.htmlunit.ScriptException: TypeError: Cannot find function entries in object function Object() { [native code] }. (script in https://www.flashscore.com/unsupported/ from (31,9) to (53,10)#35)
at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine$HtmlUnitContextAction.run(JavaScriptEngine.java:949) ~[htmlunit-2.50.0.jar:2.50.0]
at net.sourceforge.htmlunit.corejs.javascript.Context.call(Context.java:598) ~[htmlunit-core-js-2.50.0.jar:na]
at net.sourceforge.htmlunit.corejs.javascript.ContextFactory.call(ContextFactory.java:487) ~[htmlunit-core-js-2.50.0.jar:na]
at com.gargoylesoftware.htmlunit.javascript.HtmlUnitContextFactory.callSecured(HtmlUnitContextFactory.java:353) ~[htmlunit-2.50.0.jar:2.50.0]
at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine.execute(JavaScriptEngine.java:829) ~[htmlunit-2.50.0.jar:2.50.0]
at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine.execute(JavaScriptEngine.java:805) ~[htmlunit-2.50.0.jar:2.50.0]
at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine.execute(JavaScriptEngine.java:796) ~[htmlunit-2.50.0.jar:2.50.0]
at com.gargoylesoftware.htmlunit.html.HtmlPage.executeJavaScript(HtmlPage.java:942) ~[htmlunit-2.50.0.jar:2.50.0]
at com.gargoylesoftware.htmlunit.html.ScriptElementSupport.executeInlineScriptIfNeeded(ScriptElementSupport.java:378) ~[htmlunit-2.50.0.jar:2.50.0]
at com.gargoylesoftware.htmlunit.html.ScriptElementSupport.executeScriptIfNeeded(ScriptElementSupport.java:230) ~[htmlunit-2.50.0.jar:2.50.0]
at com.gargoylesoftware.htmlunit.html.ScriptElementSupport$1.execute(ScriptElementSupport.java:120) ~[htmlunit-2.50.0.jar:2.50.0]
at com.gargoylesoftware.htmlunit.html.ScriptElementSupport.onAllChildrenAddedToPage(ScriptElementSupport.java:143) ~[htmlunit-2.50.0.jar:2.50.0]
at com.gargoylesoftware.htmlunit.html.HtmlScript.onAllChildrenAddedToPage(HtmlScript.java:191) ~[htmlunit-2.50.0.jar:2.50.0]
at com.gargoylesoftware.htmlunit.html.parser.neko.HtmlUnitNekoDOMBuilder.endElement(HtmlUnitNekoDOMBuilder.java:551) ~[htmlunit-2.50.0.jar:2.50.0]
at org.apache.xerces.parsers.AbstractSAXParser.endElement(Unknown Source) ~[xercesImpl-2.12.0.jar:na]
at com.gargoylesoftware.htmlunit.html.parser.neko.HtmlUnitNekoDOMBuilder.endElement(HtmlUnitNekoDOMBuilder.java:503) ~[htmlunit-2.50.0.jar:2.50.0]
at net.sourceforge.htmlunit.cyberneko.HTMLTagBalancer.callEndElement(HTMLTagBalancer.java:1216) ~[neko-htmlunit-2.50.0.jar:2.50.0]
at net.sourceforge.htmlunit.cyberneko.HTMLTagBalancer.endElement(HTMLTagBalancer.java:1156) ~[neko-htmlunit-2.50.0.jar:2.50.0]
at net.sourceforge.htmlunit.cyberneko.filters.DefaultFilter.endElement(DefaultFilter.java:219) ~[neko-htmlunit-2.50.0.jar:2.50.0]
at net.sourceforge.htmlunit.cyberneko.filters.NamespaceBinder.endElement(NamespaceBinder.java:312) ~[neko-htmlunit-2.50.0.jar:2.50.0]
at net.sourceforge.htmlunit.cyberneko.HTMLScanner$ContentScanner.scanEndElement(HTMLScanner.java:3189) ~[neko-htmlunit-2.50.0.jar:2.50.0]
at net.sourceforge.htmlunit.cyberneko.HTMLScanner$ContentScanner.scan(HTMLScanner.java:2114) ~[neko-htmlunit-2.50.0.jar:2.50.0]
at net.sourceforge.htmlunit.cyberneko.HTMLScanner.scanDocument(HTMLScanner.java:937) ~[neko-htmlunit-2.50.0.jar:2.50.0]
at net.sourceforge.htmlunit.cyberneko.HTMLConfiguration.parse(HTMLConfiguration.java:443) ~[neko-htmlunit-2.50.0.jar:2.50.0]
at net.sourceforge.htmlunit.cyberneko.HTMLConfiguration.parse(HTMLConfiguration.java:394) ~[neko-htmlunit-2.50.0.jar:2.50.0]
at org.apache.xerces.parsers.XMLParser.parse(Unknown Source) ~[xercesImpl-2.12.0.jar:na]
at com.gargoylesoftware.htmlunit.html.parser.neko.HtmlUnitNekoDOMBuilder.parse(HtmlUnitNekoDOMBuilder.java:751) ~[htmlunit-2.50.0.jar:2.50.0]
at com.gargoylesoftware.htmlunit.html.parser.neko.HtmlUnitNekoHtmlParser.parse(HtmlUnitNekoHtmlParser.java:208) ~[htmlunit-2.50.0.jar:2.50.0]
at com.gargoylesoftware.htmlunit.DefaultPageCreator.createHtmlPage(DefaultPageCreator.java:296) ~[htmlunit-2.50.0.jar:2.50.0]
at com.gargoylesoftware.htmlunit.DefaultPageCreator.createPage(DefaultPageCreator.java:216) ~[htmlunit-2.50.0.jar:2.50.0]
at com.gargoylesoftware.htmlunit.WebClient.loadWebResponseInto(WebClient.java:682) ~[htmlunit-2.50.0.jar:2.50.0]
at com.gargoylesoftware.htmlunit.WebClient.loadDownloadedResponses(WebClient.java:2493) ~[htmlunit-2.50.0.jar:2.50.0]
at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine.doProcessPostponedActions(JavaScriptEngine.java:977) ~[htmlunit-2.50.0.jar:2.50.0]
at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine$HtmlUnitContextAction.run(JavaScriptEngine.java:944) ~[htmlunit-2.50.0.jar:2.50.0]
at net.sourceforge.htmlunit.corejs.javascript.Context.call(Context.java:598) ~[htmlunit-core-js-2.50.0.jar:na]
at net.sourceforge.htmlunit.corejs.javascript.ContextFactory.call(ContextFactory.java:487) ~[htmlunit-core-js-2.50.0.jar:na]
at com.gargoylesoftware.htmlunit.javascript.HtmlUnitContextFactory.callSecured(HtmlUnitContextFactory.java:353) ~[htmlunit-2.50.0.jar:2.50.0]
at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine.execute(JavaScriptEngine.java:829) ~[htmlunit-2.50.0.jar:2.50.0]
at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine.execute(JavaScriptEngine.java:805) ~[htmlunit-2.50.0.jar:2.50.0]
at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine.execute(JavaScriptEngine.java:111) ~[htmlunit-2.50.0.jar:2.50.0]
at com.gargoylesoftware.htmlunit.html.HtmlPage.loadExternalJavaScriptFile(HtmlPage.java:1022) ~[htmlunit-2.50.0.jar:2.50.0]
at com.gargoylesoftware.htmlunit.html.ScriptElementSupport.executeScriptIfNeeded(ScriptElementSupport.java:196) ~[htmlunit-2.50.0.jar:2.50.0]
at com.gargoylesoftware.htmlunit.html.ScriptElementSupport$1.execute(ScriptElementSupport.java:120) ~[htmlunit-2.50.0.jar:2.50.0]
at com.gargoylesoftware.htmlunit.html.ScriptElementSupport.onAllChildrenAddedToPage(ScriptElementSupport.java:143) ~[htmlunit-2.50.0.jar:2.50.0]
at com.gargoylesoftware.htmlunit.html.HtmlScript.onAllChildrenAddedToPage(HtmlScript.java:191) ~[htmlunit-2.50.0.jar:2.50.0]
at com.gargoylesoftware.htmlunit.html.parser.neko.HtmlUnitNekoDOMBuilder.endElement(HtmlUnitNekoDOMBuilder.java:551) ~[htmlunit-2.50.0.jar:2.50.0]
at org.apache.xerces.parsers.AbstractSAXParser.endElement(Unknown Source) ~[xercesImpl-2.12.0.jar:na]
at com.gargoylesoftware.htmlunit.html.parser.neko.HtmlUnitNekoDOMBuilder.endElement(HtmlUnitNekoDOMBuilder.java:503) ~[htmlunit-2.50.0.jar:2.50.0]
at net.sourceforge.htmlunit.cyberneko.HTMLTagBalancer.callEndElement(HTMLTagBalancer.java:1216) ~[neko-htmlunit-2.50.0.jar:2.50.0]
at net.sourceforge.htmlunit.cyberneko.HTMLTagBalancer.endElement(HTMLTagBalancer.java:1156) ~[neko-htmlunit-2.50.0.jar:2.50.0]
at net.sourceforge.htmlunit.cyberneko.filters.DefaultFilter.endElement(DefaultFilter.java:219) ~[neko-htmlunit-2.50.0.jar:2.50.0]
at net.sourceforge.htmlunit.cyberneko.filters.NamespaceBinder.endElement(NamespaceBinder.java:312) ~[neko-htmlunit-2.50.0.jar:2.50.0]
at net.sourceforge.htmlunit.cyberneko.HTMLScanner$ContentScanner.scanEndElement(HTMLScanner.java:3189) ~[neko-htmlunit-2.50.0.jar:2.50.0]
at net.sourceforge.htmlunit.cyberneko.HTMLScanner$ContentScanner.scan(HTMLScanner.java:2114) ~[neko-htmlunit-2.50.0.jar:2.50.0]
at net.sourceforge.htmlunit.cyberneko.HTMLScanner.scanDocument(HTMLScanner.java:937) ~[neko-htmlunit-2.50.0.jar:2.50.0]
at net.sourceforge.htmlunit.cyberneko.HTMLConfiguration.parse(HTMLConfiguration.java:443) ~[neko-htmlunit-2.50.0.jar:2.50.0]
at net.sourceforge.htmlunit.cyberneko.HTMLConfiguration.parse(HTMLConfiguration.java:394) ~[neko-htmlunit-2.50.0.jar:2.50.0]
at org.apache.xerces.parsers.XMLParser.parse(Unknown Source) ~[xercesImpl-2.12.0.jar:na]
at com.gargoylesoftware.htmlunit.html.parser.neko.HtmlUnitNekoDOMBuilder.parse(HtmlUnitNekoDOMBuilder.java:751) ~[htmlunit-2.50.0.jar:2.50.0]
at com.gargoylesoftware.htmlunit.html.parser.neko.HtmlUnitNekoHtmlParser.parse(HtmlUnitNekoHtmlParser.java:208) ~[htmlunit-2.50.0.jar:2.50.0]
at com.gargoylesoftware.htmlunit.DefaultPageCreator.createHtmlPage(DefaultPageCreator.java:296) ~[htmlunit-2.50.0.jar:2.50.0]
at com.gargoylesoftware.htmlunit.DefaultPageCreator.createPage(DefaultPageCreator.java:216) ~[htmlunit-2.50.0.jar:2.50.0]
at com.gargoylesoftware.htmlunit.WebClient.loadWebResponseInto(WebClient.java:682) ~[htmlunit-2.50.0.jar:2.50.0]
at com.gargoylesoftware.htmlunit.WebClient.loadWebResponseInto(WebClient.java:584) ~[htmlunit-2.50.0.jar:2.50.0]
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:499) ~[htmlunit-2.50.0.jar:2.50.0]
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:411) ~[htmlunit-2.50.0.jar:2.50.0]
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:546) ~[htmlunit-2.50.0.jar:2.50.0]
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:527) ~[htmlunit-2.50.0.jar:2.50.0]
at de.toniritter.controller.FlashScraperController.startScraping(FlashScraperController.java:68) ~[classes/:na]
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na]
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:64) ~[na:na]
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na]
at java.base/java.lang.reflect.Method.invoke(Method.java:564) ~[na:na]
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:197) ~[spring-web-5.3.8.jar:5.3.8]
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:141) ~[spring-web-5.3.8.jar:5.3.8]
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:106) ~[spring-webmvc-5.3.8.jar:5.3.8]
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:894) ~[spring-webmvc-5.3.8.jar:5.3.8]
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:808) ~[spring-webmvc-5.3.8.jar:5.3.8]
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87) ~[spring-webmvc-5.3.8.jar:5.3.8]
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1063) ~[spring-webmvc-5.3.8.jar:5.3.8]
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:963) ~[spring-webmvc-5.3.8.jar:5.3.8]
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006) ~[spring-webmvc-5.3.8.jar:5.3.8]
at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:909) ~[spring-webmvc-5.3.8.jar:5.3.8]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:681) ~[tomcat-embed-core-9.0.48.jar:4.0.FR]
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883) ~[spring-webmvc-5.3.8.jar:5.3.8]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:764) ~[tomcat-embed-core-9.0.48.jar:4.0.FR]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:228) ~[tomcat-embed-core-9.0.48.jar:9.0.48]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:163) ~[tomcat-embed-core-9.0.48.jar:9.0.48]
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53) ~[tomcat-embed-websocket-9.0.48.jar:9.0.48]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:190) ~[tomcat-embed-core-9.0.48.jar:9.0.48]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:163) ~[tomcat-embed-core-9.0.48.jar:9.0.48]
at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100) ~[spring-web-5.3.8.jar:5.3.8]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) ~[spring-web-5.3.8.jar:5.3.8]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:190) ~[tomcat-embed-core-9.0.48.jar:9.0.48]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:163) ~[tomcat-embed-core-9.0.48.jar:9.0.48]
at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93) ~[spring-web-5.3.8.jar:5.3.8]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) ~[spring-web-5.3.8.jar:5.3.8]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:190) ~[tomcat-embed-core-9.0.48.jar:9.0.48]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:163) ~[tomcat-embed-core-9.0.48.jar:9.0.48]
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201) ~[spring-web-5.3.8.jar:5.3.8]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) ~[spring-web-5.3.8.jar:5.3.8]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:190) ~[tomcat-embed-core-9.0.48.jar:9.0.48]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:163) ~[tomcat-embed-core-9.0.48.jar:9.0.48]
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202) ~[tomcat-embed-core-9.0.48.jar:9.0.48]
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97) ~[tomcat-embed-core-9.0.48.jar:9.0.48]
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:542) ~[tomcat-embed-core-9.0.48.jar:9.0.48]
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:143) ~[tomcat-embed-core-9.0.48.jar:9.0.48]
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92) ~[tomcat-embed-core-9.0.48.jar:9.0.48]
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78) ~[tomcat-embed-core-9.0.48.jar:9.0.48]
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:357) ~[tomcat-embed-core-9.0.48.jar:9.0.48]
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:382) ~[tomcat-embed-core-9.0.48.jar:9.0.48]
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65) ~[tomcat-embed-core-9.0.48.jar:9.0.48]
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:893) ~[tomcat-embed-core-9.0.48.jar:9.0.48]
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1723) ~[tomcat-embed-core-9.0.48.jar:9.0.48]
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) ~[tomcat-embed-core-9.0.48.jar:9.0.48]
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) ~[na:na]
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) ~[na:na]
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) ~[tomcat-embed-core-9.0.48.jar:9.0.48]
at java.base/java.lang.Thread.run(Thread.java:832) ~[na:na]
Caused by: net.sourceforge.htmlunit.corejs.javascript.EcmaError: TypeError: Cannot find function entries in object function Object() { [native code] }. (script in https://www.flashscore.com/unsupported/ from (31,10)#35)
at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.constructError(ScriptRuntime.java:4527) ~[htmlunit-core-js-2.50.0.jar:na]
at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.constructError(ScriptRuntime.java:4508) ~[htmlunit-core-js-2.50.0.jar:na]
at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.typeError(ScriptRuntime.java:4540) ~[htmlunit-core-js-2.50.0.jar:na]
at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.typeErrorById(ScriptRuntime.java:4545) ~[htmlunit-core-js-2.50.0.jar:na]
at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.notFunctionError(ScriptRuntime.java:4624) ~[htmlunit-core-js-2.50.0.jar:na]
at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.getPropFunctionAndThisHelper(ScriptRuntime.java:2570) ~[htmlunit-core-js-2.50.0.jar:na]
at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.getPropFunctionAndThis(ScriptRuntime.java:2553) ~[htmlunit-core-js-2.50.0.jar:na]
at net.sourceforge.htmlunit.corejs.javascript.Interpreter.interpretLoop(Interpreter.java:1663) ~[htmlunit-core-js-2.50.0.jar:na]
at net.sourceforge.htmlunit.corejs.javascript.Interpreter.interpret(Interpreter.java:1053) ~[htmlunit-core-js-2.50.0.jar:na]
at net.sourceforge.htmlunit.corejs.javascript.InterpretedFunction.call(InterpretedFunction.java:111) ~[htmlunit-core-js-2.50.0.jar:na]
at net.sourceforge.htmlunit.corejs.javascript.ContextFactory.doTopCall(ContextFactory.java:395) ~[htmlunit-core-js-2.50.0.jar:na]
at com.gargoylesoftware.htmlunit.javascript.HtmlUnitContextFactory.doTopCall(HtmlUnitContextFactory.java:339) ~[htmlunit-2.50.0.jar:2.50.0]
at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.doTopCall(ScriptRuntime.java:3852) ~[htmlunit-core-js-2.50.0.jar:na]
at net.sourceforge.htmlunit.corejs.javascript.InterpretedFunction.exec(InterpretedFunction.java:123) ~[htmlunit-core-js-2.50.0.jar:na]
at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine$2.doRun(JavaScriptEngine.java:820) ~[htmlunit-2.50.0.jar:2.50.0]
at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine$HtmlUnitContextAction.run(JavaScriptEngine.java:934) ~[htmlunit-2.50.0.jar:2.50.0]
... 118 common frames omitted
2021-07-06 08:46:17.757 INFO 14464 --- [Client@49a9c8bb] c.gargoylesoftware.htmlunit.WebConsole : *** cookie 'fonce_current_user' set (all pages): _ls_tracking=db75abda-de25-11eb-bb62-5e9401b680a6; _ga_3NV6GD9ZTT=GS1.1.1625553977.1.0.1625553977.60; _ga_HRK2668K68=GS1.1.1625553977.1.0.1625553977.0; _ga=GA1.1.1320121497.1625553977; fonce_current_session=0; fonce_current_day=0,2021-07-06; fonce_current_user=0
[]
这段代码在这里毫无例外地工作 - 我猜你没有使用最新版本。
String url = "https://flashScore.com";
try (final WebClient webClient = new WebClient(BrowserVersion.BEST_SUPPORTED)) {
HtmlPage page = webClient.getPage(url);
webClient.waitForBackgroundJavaScript(3_000);
System.out.println();
System.out.println();
System.out.println("----------------");
System.out.println(page.asNormalizedText());
System.out.println("----------------");
}
但我最终在这个页面上
----------------
FlashScore.com
We can´t display the page,because your browser doesn´t support the required functions.
If you want to visit the lite version of our website,tap on the following link.
Go to www.flashscore.mobi
You can also download one of the modern browsers:
Chrome | Firefox | Edge
----------------
我不知道浏览器会检查页面。也许您可以使用此示例代码并查看网络流量（使用 Charles 或 Fiddler）以了解问题所在。或者也许检查是在客户端上用普通的 javascript 完成的。如果你认为你找到了原因，请随时在 github 上打开一个问题，我会尽量让 HtmlUnit 更兼容。