问题描述
我在玩 Goutte,但无法让它连接到某个网站。所有其他 URL 似乎都运行良好,我正在努力了解阻止它连接的原因。它只是挂起,直到 30 秒后超时。如果我取消超时,150 秒后也会发生同样的情况。
注意事项:
- 此超时/挂起仅发生在我目前发现的 tesco.com 上。 asda.com、google.com 等工作正常并返回结果。
- 该网站会立即在网络浏览器 (Chrome) 中加载(与 IP 或 ISP 无关)。
- 如果我在 Postman 中向同一个 URL 发出 GET 请求,我得到的结果返回正常。
- 似乎与用户代理无关。
<?PHP
namespace App\Http\Controllers;
use Goutte\Client;
use GuzzleHttp\Client as GuzzleClient;
class ScraperController extends Controller
{
public function scrape()
{
$goutteClient = new Client();
$goutteClient->setHeader('user-agent','Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML,like Gecko) Chrome/88.0.4324.96 Safari/537.36');
$guzzleClient = new GuzzleClient(array(
'timeout' => 30,'verify' => true,'debug' => true,));
$goutteClient->setClient($guzzleClient);
$crawler = $goutteClient->request('GET','https://www.tesco.com/');
dump($crawler);
/*$crawler->filter('.result__title .result__a')->each(function ($node) {
dump($node->text());
});*/
}
}
* Trying 104.123.91.150:443... * TCP_NODELAY set * Connected to www.tesco.com (104.123.91.150) port 443 (#0) * ALPN,offering http/1.1 * successfully set certificate verify locations: * CAfile: /etc/ssl/certs/ca-certificates.crt CApath: /etc/ssl/certs * SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 * ALPN,server accepted to use http/1.1 * Server certificate: * subject: C=GB; L=Welwyn Garden City; jurisdictionC=GB; O=Tesco PLC; businessCategory=Private Organization; serialNumber=00445790; CN=www.tesco.com * start date: Feb 4 11:09:23 2020 GMT * expire date: Feb 3 11:39:21 2022 GMT * subjectAltName: host "www.tesco.com" matched cert's "www.tesco.com" * issuer: C=US; O=entrust,Inc.; OU=See www.entrust.net/legal-terms; OU=(c) 2014 entrust,Inc. - for authorized use only; CN=entrust Certification Authority - L1M * SSL certificate verify ok. > GET / HTTP/1.1 Host: www.tesco.com user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML,like Gecko) Chrome/88.0.4324.96 Safari/537.36 * old SSL session ID is stale,removing * Operation timed out after 30001 milliseconds with 0 bytes received * Closing connection 0
GuzzleHttp\Exception\ConnectException
cURL error 28: Operation timed out after 30001 milliseconds with 0 bytes received (see https://curl.haxx.se/libcurl/c/libcurl-errors.html)
http://localhost/scrape
谁能明白我为什么完全没有收到回复?
解决方法
通过添加更多标题设法解决了这个问题:
<?php
namespace App\Http\Controllers;
use Goutte\Client;
use GuzzleHttp\Client as GuzzleClient;
class ScraperController extends Controller
{
public function scrape()
{
$goutteClient = new Client();
$goutteClient->setHeader('accept','text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9');
$goutteClient->setHeader('accept-encoding','gzip,deflate,br');
$goutteClient->setHeader('accept-language','en-GB,en-US;q=0.9,en;q=0.8');
$goutteClient->setHeader('upgrade-insecure-requests','1');
$goutteClient->setHeader('user-agent','Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML,like Gecko) Chrome/88.0.4324.96 Safari/537.36');
$goutteClient->setHeader('connection','keep-alive');
$guzzleClient = new GuzzleClient(array(
'timeout' => 5,'verify' => true,'debug' => true,'cookies' => true,));
$goutteClient->setClient($guzzleClient);
$crawler = $goutteClient->request('GET','https://www.tesco.com/');
dump($crawler);
/*$crawler->filter('.result__title .result__a')->each(function ($node) {
dump($node->text());
});*/
}
}