仅当我添加另一个代理时，才能使用Selenium + Chrome驱动程序+ Browsermob代理登录->为什么？

编程问答 2022-08-14

问题描述

我有下面的代码来初始化Selenium + Chrome驱动程序+ browsermob代理的设置，以登录到某个网站。

如果我仅使用上面的设置，则登录失败并显示HTTP403。但是，如果我有另一个代理服务器（Selenium + Chrome驱动程序+ browsermob代理服务器+例如fiddler Everywhere，请参阅“将请求从代理服务器传递到fiddler的后面”到处”），并强制fiddler到处代理来自browsermob代理的请求，然后登录正常。

任何人都可以向我提示如何调试和/或清除此问题吗？

我研究了两腿之间的标头：

  • browsermob添加了“ Via：1.1 browsermobproxy”，但是从browsermob中删除了“ Via”标头后，我重播了登录请求，但没有任何效果。
  • browserMob正在删除“连接：保持活动”。这可能是原因（或暗示原因）吗？

我还检查了

  • 请求的主体在整个链中都没有改变。
  • HTTP协议版本不会在整个链（HTTP / 1.1）上更改。

以下是没有fiddler的详细请求日志：

POST /api/sts/v2/oauth/auth?client_request_id=34634696-b122-40de-b144-ab812a4b3784 HTTP/1.1
Host: www.example.com
Connection: keep-alive
Content-Length: 147
AccountType: Real
X-STS-ClientTime: 2020-08-27T18:10:51
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML,like Gecko) Chrome/84.0.4147.135 Safari/537.36
X-STS-AppDomain: https://www.example.com
Content-Type: application/json;charset=UTF-8
Accept: application/json,text/plain,*/*
X-STS-GatewayAppId: 90631448-9A01-4860-9FA5-B4EBCDE5EA1D
ApplicationIdentifier: Rexample
ApplicationVersion: 268.0.1
X-DEVICE-ID: 77f281a4-7914-4417-b144-4811998c3e5b
Origin: https://www.example.com
sec-fetch-site: same-origin
sec-fetch-mode: cors
Sec-Fetch-Dest: empty
Referer: https://www.example.com/login
Accept-Encoding: gzip,deflate,br
Accept-Language: en-GB,en-US;q=0.9,en;q=0.8
Cookie: __cfduid=d722b1913f887e5df9b0c740ccef4744a1598548223; exampleLocale=en-gb; TS01047baf=01f1b32d7e1354b9f541c1bab1a81960be72277a638bce5a4865b5b1d5b2e0a9cfddddbcba0d21185b68a7d599391e972d5f065031; __cflb=02DiuEAg8LPSYevheyiyx9rnANkoz3ETfpNySVtv8fQVr; TMIS2=9a74f2a90237486aab4dcb9d44dec14a0520aaf521b9e87e81f271264e25038c2919d29ee219162aa2ad6bdb68fbe78a86ce282ef510aa77456a3876e4e790d3fdbb0ca31af4d206c67f7b9b5fda93e11b80782ed9095c6a7d8766b4642a69785e7ba9d0a86cd98f3b42c2172a8821bbe6906b6fafc8178f0d742271591193; __cf_bm=50d03de4ce1301af67fa03c59387373fba4d8b3e-1598548241-1800-AZ39AzpjNXxDQ+qFhgfI0OYbMauZ3vXOA1MUejwqJsfE3kOEmEVmKXtsstKg5vZZ5ie/d3yHbgzM8RzzLqqSQEVnd5Vy+l06FUm5Z90kFf54ffUqgq+ywM4EocghdP22aetxTezQvFieWgfScPvH0SzqDmfpvrmZJmRuLvnywgcLX14Ys9uA2tMCLWSlDesiQA==; mp_dbbd7bd9566da85f012f7ca5d8c6c944_mixpanel=%7B%22distinct_id%22%3A%20%2217430e66d1679d-015c94d6c30eda-31647304-fa000-17430e66d17591%22%2C%22%24device_id%22%3A%20%2217430e66d1679d-015c94d6c30eda-31647304-fa000-17430e66d17591%22%2C%22%24initial_referrer%22%3A%20%22%24direct%22%2C%22%24initial_referring_domain%22%3A%20%22%24direct%22%7D; _gcl_au=1.1.706805090.1598548242; G_ENABLED_IDPS=google; _uetsid=5c779e7394432399f739848297b9a350; _uetvid=d421e48c4acc1f88b81573014b4fa429; _fbp=fb.1.1598548244752.1489311425; __adal_ses=*; __adal_id=c4b052ee-96b4-4643-b30e-af4af0f7460d.1598548246.1.1598548246.1598548246.2a5a85bd-5415-4275-942c-e02a8048acc2; __adal_ca=so%3Ddirect%26me%3Dnone%26ca%3Ddirect%26co%3D%28not%2520set%29%26ke%3D%28not%2520set%29; __adal_cw=1598548245533; _ga=GA1.2.436171435.1598548249; _gid=GA1.2.1493657885.1598548249; _gat=1; _gat_UA-2056847-65=1
DEBUG [LittleProxy-0-ClientToProxyWorker-2] (ClientToProxyConnection.java:183) - (AWAITING_INITIAL) [id: 0x367d8cd0,L:/[0:0:0:0:0:0:0:1]:64108 - R:/[0:0:0:0:0:0:0:1]:64132]: Received raw request: DefaultHttpRequest(decodeResult: success,version: HTTP/1.1)
POST /api/sts/v2/oauth/auth?client_request_id=34634696-b122-40de-b144-ab812a4b3784 HTTP/1.1
Host: www.example.com
Connection: keep-alive
Content-Length: 147
AccountType: Real
X-STS-ClientTime: 2020-08-27T18:10:51
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML,en;q=0.8
Cookie: __cfduid=d722b1913f887e5df9b0c740ccef4744a1598548223; exampleLocale=en-gb; TS01047baf=01f1b32d7e1354b9f541c1bab1a81960be72277a638bce5a4865b5b1d5b2e0a9cfddddbcba0d21185b68a7d599391e972d5f065031; __cflb=02DiuEAg8LPSYevheyiyx9rnANkoz3ETfpNySVtv8fQVr; TMIS2=9a74f2a90237486aab4dcb9d44dec14a0520aaf521b9e87e81f271264e25038c2919d29ee219162aa2ad6bdb68fbe78a86ce282ef510aa77456a3876e4e790d3fdbb0ca31af4d206c67f7b9b5fda93e11b80782ed9095c6a7d8766b4642a69785e7ba9d0a86cd98f3b42c2172a8821bbe6906b6fafc8178f0d742271591193; __cf_bm=50d03de4ce1301af67fa03c59387373fba4d8b3e-1598548241-1800-AZ39AzpjNXxDQ+qFhgfI0OYbMauZ3vXOA1MUejwqJsfE3kOEmEVmKXtsstKg5vZZ5ie/d3yHbgzM8RzzLqqSQEVnd5Vy+l06FUm5Z90kFf54ffUqgq+ywM4EocghdP22aetxTezQvFieWgfScPvH0SzqDmfpvrmZJmRuLvnywgcLX14Ys9uA2tMCLWSlDesiQA==; mp_dbbd7bd9566da85f012f7ca5d8c6c944_mixpanel=%7B%22distinct_id%22%3A%20%2217430e66d1679d-015c94d6c30eda-31647304-fa000-17430e66d17591%22%2C%22%24device_id%22%3A%20%2217430e66d1679d-015c94d6c30eda-31647304-fa000-17430e66d17591%22%2C%22%24initial_referrer%22%3A%20%22%24direct%22%2C%22%24initial_referring_domain%22%3A%20%22%24direct%22%7D; _gcl_au=1.1.706805090.1598548242; G_ENABLED_IDPS=google; _uetsid=5c779e7394432399f739848297b9a350; _uetvid=d421e48c4acc1f88b81573014b4fa429; _fbp=fb.1.1598548244752.1489311425; __adal_ses=*; __adal_id=c4b052ee-96b4-4643-b30e-af4af0f7460d.1598548246.1.1598548246.1598548246.2a5a85bd-5415-4275-942c-e02a8048acc2; __adal_ca=so%3Ddirect%26me%3Dnone%26ca%3Ddirect%26co%3D%28not%2520set%29%26ke%3D%28not%2520set%29; __adal_cw=1598548245533; _ga=GA1.2.436171435.1598548249; _gid=GA1.2.1493657885.1598548249; _gat=1; _gat_UA-2056847-65=1
DEBUG [LittleProxy-0-ClientToProxyWorker-2] (ClientToProxyConnection.java:269) - (AWAITING_INITIAL) [id: 0x367d8cd0,L:/[0:0:0:0:0:0:0:1]:64108 - R:/[0:0:0:0:0:0:0:1]:64132]: Ensuring that hostAndPort are available in /api/sts/v2/oauth/auth?client_request_id=34634696-b122-40de-b144-ab812a4b3784
DEBUG [LittleProxy-0-ClientToProxyWorker-2] (ClientToProxyConnection.java:281) - (AWAITING_INITIAL) [id: 0x367d8cd0,L:/[0:0:0:0:0:0:0:1]:64108 - R:/[0:0:0:0:0:0:0:1]:64132]: Finding ProxyToServerConnection for: www.example.com
DEBUG [LittleProxy-0-ClientToProxyWorker-2] (ClientToProxyConnection.java:331) - (AWAITING_INITIAL) [id: 0x367d8cd0,L:/[0:0:0:0:0:0:0:1]:64108 - R:/[0:0:0:0:0:0:0:1]:64132]: Reusing existing server connection: org.littleshoot.proxy.impl.ProxyToServerConnection@7c392fd2
DEBUG [LittleProxy-0-ClientToProxyWorker-2] (ClientToProxyConnection.java:1084) - (AWAITING_INITIAL) [id: 0x367d8cd0,L:/[0:0:0:0:0:0:0:1]:64108 - R:/[0:0:0:0:0:0:0:1]:64132]: Modifying request for proxy chaining
DEBUG [LittleProxy-0-ClientToProxyWorker-2] (ClientToProxyConnection.java:1088) - (AWAITING_INITIAL) [id: 0x367d8cd0,L:/[0:0:0:0:0:0:0:1]:64108 - R:/[0:0:0:0:0:0:0:1]:64132]: Stripped host from uri: /api/sts/v2/oauth/auth?client_request_id=34634696-b122-40de-b144-ab812a4b3784    yielding: /api/sts/v2/oauth/auth?client_request_id=34634696-b122-40de-b144-ab812a4b3784
DEBUG [LittleProxy-0-ClientToProxyWorker-2] (ClientToProxyConnection.java:1093) - (AWAITING_INITIAL) [id: 0x367d8cd0,L:/[0:0:0:0:0:0:0:1]:64108 - R:/[0:0:0:0:0:0:0:1]:64132]: Modifying request headers for proxying



DEBUG [LittleProxy-0-ClientToProxyWorker-2] (ClientToProxyConnection.java:350) - (AWAITING_INITIAL) [id: 0x367d8cd0,L:/[0:0:0:0:0:0:0:1]:64108 - R:/[0:0:0:0:0:0:0:1]:64132]: Writing request to ProxyToServerConnection
DEBUG [LittleProxy-0-ClientToProxyWorker-2] (ProxyToServerConnection.java:316) - (AWAITING_INITIAL) [id: 0xba528f83,L:/172.20.10.2:64134 - R:www.example.com/104.18.19.67:443]: Requested write of DefaultHttpRequest(decodeResult: success,version: HTTP/1.1)
POST /api/sts/v2/oauth/auth?client_request_id=34634696-b122-40de-b144-ab812a4b3784 HTTP/1.1
Host: www.example.com
Content-Length: 147
AccountType: Real
X-STS-ClientTime: 2020-08-27T18:10:51
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML,*/*
X-STS-GatewayAppId: 90631448-9A01-4860-9FA5-B4EBCDE5EA1D
ApplicationIdentifier: Rexample
ApplicationVersion: 268.0.1
X-DEVICE-ID: 77f281a4-7914-4417-b144-4811998c3e5b
Origin: https://www.example.com
sec-fetch-site: same-origin
sec-fetch-mode: cors
Sec-Fetch-Dest: empty
Referer: https://www.example.com/login
Accept-Language: en-GB,en;q=0.8
Cookie: __cfduid=d722b1913f887e5df9b0c740ccef4744a1598548223; exampleLocale=en-gb; TS01047baf=01f1b32d7e1354b9f541c1bab1a81960be72277a638bce5a4865b5b1d5b2e0a9cfddddbcba0d21185b68a7d599391e972d5f065031; __cflb=02DiuEAg8LPSYevheyiyx9rnANkoz3ETfpNySVtv8fQVr; TMIS2=9a74f2a90237486aab4dcb9d44dec14a0520aaf521b9e87e81f271264e25038c2919d29ee219162aa2ad6bdb68fbe78a86ce282ef510aa77456a3876e4e790d3fdbb0ca31af4d206c67f7b9b5fda93e11b80782ed9095c6a7d8766b4642a69785e7ba9d0a86cd98f3b42c2172a8821bbe6906b6fafc8178f0d742271591193; __cf_bm=50d03de4ce1301af67fa03c59387373fba4d8b3e-1598548241-1800-AZ39AzpjNXxDQ+qFhgfI0OYbMauZ3vXOA1MUejwqJsfE3kOEmEVmKXtsstKg5vZZ5ie/d3yHbgzM8RzzLqqSQEVnd5Vy+l06FUm5Z90kFf54ffUqgq+ywM4EocghdP22aetxTezQvFieWgfScPvH0SzqDmfpvrmZJmRuLvnywgcLX14Ys9uA2tMCLWSlDesiQA==; mp_dbbd7bd9566da85f012f7ca5d8c6c944_mixpanel=%7B%22distinct_id%22%3A%20%2217430e66d1679d-015c94d6c30eda-31647304-fa000-17430e66d17591%22%2C%22%24device_id%22%3A%20%2217430e66d1679d-015c94d6c30eda-31647304-fa000-17430e66d17591%22%2C%22%24initial_referrer%22%3A%20%22%24direct%22%2C%22%24initial_referring_domain%22%3A%20%22%24direct%22%7D; _gcl_au=1.1.706805090.1598548242; G_ENABLED_IDPS=google; _uetsid=5c779e7394432399f739848297b9a350; _uetvid=d421e48c4acc1f88b81573014b4fa429; _fbp=fb.1.1598548244752.1489311425; __adal_ses=*; __adal_id=c4b052ee-96b4-4643-b30e-af4af0f7460d.1598548246.1.1598548246.1598548246.2a5a85bd-5415-4275-942c-e02a8048acc2; __adal_ca=so%3Ddirect%26me%3Dnone%26ca%3Ddirect%26co%3D%28not%2520set%29%26ke%3D%28not%2520set%29; __adal_cw=1598548245533; _ga=GA1.2.436171435.1598548249; _gid=GA1.2.1493657885.1598548249; _gat=1; _gat_UA-2056847-65=1
Accept-Encoding: gzip,br
Via: 1.1 browsermobproxy
DEBUG [LittleProxy-0-ClientToProxyWorker-2] (ProxyToServerConnection.java:348) - (AWAITING_INITIAL) [id: 0xba528f83,L:/172.20.10.2:64134 - R:www.example.com/104.18.19.67:443]: Using existing connection to: www.example.com/104.18.19.67:443
DEBUG [LittleProxy-0-ClientToProxyWorker-2] (ProxyConnection.java:228) - (AWAITING_INITIAL) [id: 0xba528f83,L:/172.20.10.2:64134 - R:www.example.com/104.18.19.67:443]: Writing: DefaultHttpRequest(decodeResult: success,br
Via: 1.1 browsermobproxy
DEBUG [LittleProxy-0-ClientToProxyWorker-2] (ProxyConnection.java:237) - (AWAITING_INITIAL) [id: 0xba528f83,L:/172.20.10.2:64134 - R:www.example.com/104.18.19.67:443]: Wrote: DefaultHttpRequest(decodeResult: success,br
Via: 1.1 browsermobproxy
DEBUG [LittleProxy-0-ClientToProxyWorker-2] (ProxyConnection.java:111) - (AWAITING_CHUNK) [id: 0x367d8cd0,L:/[0:0:0:0:0:0:0:1]:64108 - R:/[0:0:0:0:0:0:0:1]:64132]: Reading: DefaultLastHttpContent(data: SlicedAbstractByteBuf(ridx: 0,widx: 147,cap: 147/147,unwrapped: UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeNoCleanerDirectByteBuf(ridx: 147,cap: 176)),decoderResult: success)
DEBUG [LittleProxy-0-ClientToProxyWorker-2] (ProxyToServerConnection.java:316) - (AWAITING_INITIAL) [id: 0xba528f83,L:/172.20.10.2:64134 - R:www.example.com/104.18.19.67:443]: Requested write of DefaultLastHttpContent(data: SlicedAbstractByteBuf(ridx: 0,decoderResult: success)
DEBUG [LittleProxy-0-ClientToProxyWorker-2] (ProxyToServerConnection.java:319) - (AWAITING_INITIAL) [id: 0xba528f83,L:/172.20.10.2:64134 - R:www.example.com/104.18.19.67:443]: Retaining reference counted message
DEBUG [LittleProxy-0-ClientToProxyWorker-2] (ProxyToServerConnection.java:348) - (AWAITING_INITIAL) [id: 0xba528f83,L:/172.20.10.2:64134 - R:www.example.com/104.18.19.67:443]: Writing: DefaultLastHttpContent(data: SlicedAbstractByteBuf(ridx: 0,decoderResult: success)
DEBUG [LittleProxy-0-ClientToProxyWorker-2] (ProxyConnection.java:249) - (AWAITING_INITIAL) [id: 0xba528f83,L:/172.20.10.2:64134 - R:www.example.com/104.18.19.67:443]: Writing an empty buffer to signal the end of our chunked transfer
DEBUG [LittleProxy-0-ClientToProxyWorker-2] (ProxyConnection.java:237) - (AWAITING_INITIAL) [id: 0xba528f83,L:/172.20.10.2:64134 - R:www.example.com/104.18.19.67:443]: Wrote: DefaultLastHttpContent(data: SlicedAbstractByteBuf(ridx: 0,decoderResult: success)
DEBUG [LittleProxy-0-ProxyToServerWorker-0] (ProxyConnection.java:111) - (AWAITING_INITIAL) [id: 0xba528f83,L:/172.20.10.2:64134 - R:www.example.com/104.18.19.67:443]: Reading: DefaultHttpResponse(decodeResult: success,version: HTTP/1.1)
HTTP/1.1 403 Forbidden
Date: Thu,27 Aug 2020 17:10:51 GMT
Content-Type: text/html; charset=UTF-8
transfer-encoding: chunked
Connection: keep-alive
x-frame-options: SAMEORIGIN
Cache-Control: private,max-age=0,no-store,no-cache,must-revalidate,post-check=0,pre-check=0
Expires: Thu,01 Jan 1970 00:00:01 GMT
cf-request-id: 04d27fab5700001bfece21f200000001
Expect-CT: max-age=604800,report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
vary: Accept-Encoding
Strict-Transport-Security: max-age=15552000; includeSubDomains; preload
X-Content-Type-Options: nosniff
Server: cloudflare
CF-RAY: 5c97688bbb9f1bfe-OSL
content-encoding: br
alt-svc: h3-27=":443"; ma=86400,h3-28=":443"; ma=86400,h3-29=":443"; ma=86400
DEBUG [LittleProxy-0-ProxyToServerWorker-0] (ProxyToServerConnection.java:220) - (AWAITING_INITIAL) [id: 0xba528f83,L:/172.20.10.2:64134 - R:www.example.com/104.18.19.67:443]: Received raw response: DefaultHttpResponse(decodeResult: success,version: HTTP/1.1)

谢谢！

    System.setProperty("webdriver.chrome.driver","/usr/local/bin/chromedriver");
    System.setProperty("webdriver.chrome.silentOutput","true");
    System.setProperty("webdriver.chrome.verboseLogging","false");
    ChromeOptions options = new ChromeOptions();
    
    // route Chrome requests to proxy server
    proxy = new browserMobProxyServer();
    
    // to pass requests from proxy server to fiddler Everywhere
    // InetSocketAddress inetsockaddr = new InetSocketAddress('localhost',8866);
    // proxy.setTrustAllServers(true);
    // proxy.setChainedProxy(inetsockaddr);
    
    // start proxy server
    proxy.start(0);
    Proxy seleniumProxy = ClientUtil.createSeleniumProxy(proxy);
    seleniumProxy.setHttpProxy("localhost:"+ proxy.getPort());
    seleniumProxy.setSslProxy("localhost:"+ proxy.getPort());
    options.setCapability(CapabilityType.PROXY,seleniumProxy);
    options.setCapability(CapabilityType.ACCEPT_SSL_CERTS,true);
    options.setCapability (CapabilityType.ACCEPT_INSECURE_CERTS,true);
    
    //options.addArguments("headless");
    //options.addArguments("window-size=1200x600");
    options.setExperimentalOption("excludeSwitches",Collections.singletonList("enable-automation"));
    options.setExperimentalOption("useAutomationExtension",false);
    options.addArguments("--disable-blink-features=AutomationControlled");
    
    driver = new ChromeDriver(options);`

解决方法

通过替换JDK（从OpenJDK到Oracle）解决了这个问题

browsermob-proxyfiddlerfiddlerseleniumseleniumselenium-chromedriverssl

相关问答

Selenium Web驱动程序和Java元素在(x，y)点处不可单击其他元素将获得点击?
Selenium Web驱动程序和Java。元素在(x，y)点处不可单击。其...
Python-如何使用点“” 访问字典成员？
Python-如何使用点“。” 访问字典成员？
Java 字符串是不可变的到底是什么意思？
Java 字符串是不可变的。到底是什么意思？
Java中的“ final”关键字如何工作？我仍然可以修改对象
Java中的“ final”关键字如何工作？（我仍然可以修改对象。...
“loop:”在Java代码中这是什么，为什么要编译？
“loop:”在Java代码中。这是什么，为什么要编译？
java.lang.ClassNotFoundException：sun.jdbc.odbc.JdbcOdbcDriver发生异常为什么？
java.lang.ClassNotFoundException：sun.jdbc.odbc.JdbcOdbc...