CasperJs 不抓取一些网站

问题描述

我也尝试更改用户代理，但仍然无法正常工作，还有一些其他网站的 ajax 数据没有被抓取。当我尝试使用 file_get_contents() 打开抓取这些网站时，它返回“无法打开流”。但是使用 CURL 可以正常工作。

GET customers/_search
{
    "query": {
        "bool": {
            "must": [
                {
                    "query_string": {
                        "default_field": "properties.name","query": "Joe*"
                    }
                }
            ],"should": [                    
                {
                    "match": {
                        "properties.role": "admin"
                    }
                },{
                    "match": {
                        "properties.role": "sysop"
                    }
                },{
                    "match": {
                        "properties.role": "client"
                    }
                },{
                    "match": {
                        "properties.status": "public"
                    }
                },{
                    "match": {
                        "properties.status": "public"
                    }
                }
            ],"must_not": [
                {
                    "match": {
                        "properties.status": "hide_from_search_results"
                    }
                },{
                    "match": {
                        "properties.status": "deleted"
                    }
                },{
                    "match": {
                        "properties.status": "banned"
                    }
                },{
                    "match": {
                        "properties.status": "hide_from_search_results"
                    }
                },{
                    "match": {
                        "properties.status": "banned"
                    }
                }
            ]
        }
    },"size": 30,"sort": [
        {
            "_score": {
                "order": "desc"
            }
        },{
            "_script": {
                "type": "string","order": "desc","script": {
                    "lang": "painless","source": "return doc['_index'][0] == 'customers' && doc.containsKey('properties.videoCount')?doc['properties.videoCount'].value:0"
                }
            }
        },"source": "long timestampNow = new Date().getTime(); return doc['_index'][0] == 'customers' && doc.containsKey('properties.subscriptions.features.allow-application')?(timestampNow < doc['properties.subscriptions.features.first-on-search'].value.getMillis()):false"
                }
            }
        },"source": "return doc['_index'][0] == 'customers' && doc.containsKey('properties.videoCount')?doc['properties.videoCount'].value:0"
                }
            }
        }
    ]
}

解决方法

暂无找到可以解决该程序问题的有效方法，小编努力寻找整理中！

如果你已经找到好的解决方法，欢迎将解决方案带上本链接一起发送给小编。

小编邮箱:dio#foxmail.com (将#修改为@）