问题描述
说我在看这个网页
我想提取该医生个人资料的链接,但是当我尝试网络抓取时,即使使用 CSS 选择器也找不到该元素。
from requests_html import HTMLSession
firstname = 'robert'
lastname = 'b'
city = 'Palo_Alto'
url = 'https://openpaymentsdata.cms.gov/search/physicians/by-name-and-location?firstname='\
+ firstname + '&lastname=' + lastname + '&city=' + city
session = HTMLSession()
r = session.get(url)
sel = 'body > div.siteOuterWrapper > div.siteInnerWrapper > div.siteContentWrapper'
print(r.html.find(sel,first=True).text)
这一切都有效,直到我到达内容包装器,在那里我再也看不到任何元素。为什么是这样?有什么原因我看不到这个元素吗?我一开始以为是 Javascript 的原因,但是这个库声称拥有完整的 javascript 支持 https://requests-html.kennethreitz.org/
解决方法
您提到的网站从 API - this 获取数据。
您可以使用 GET
直接向该 API 发出 requests
请求并获取您的数据。
您可以使用 Chrome Devtools 找到 API 端点。
,下面的 HTTP 请求应返回您要查找的数据。 (在浏览器中做 F12 > Network > XHR)
HTTP GET https://openpaymentsdata.cms.gov/resource/khdp-6xuy.json?%24select=%3Aid%2Cphysician_profile_id%2Cphysician_profile_last_name%2Cphysician_profile_middle_name%2Cphysician_profile_first_name%2Cphysician_profile_suffix%2Cphysician_profile_primary_specialty%2Cphysician_profile_address_line_1%2Cphysician_profile_address_line_2%2Cphysician_profile_city%2Cphysician_profile_state%2Cphysician_profile_province_name%2Cphysician_profile_country_name%2Cphysician_profile_zipcode%2Cphysician_profile_alternate_first_name1%2Cphysician_profile_alternate_last_name1%2Cphysician_profile_alternate_first_name2%2Cphysician_profile_alternate_last_name2%2Cphysician_profile_alternate_first_name3%2Cphysician_profile_alternate_last_name3%2Cphysician_profile_alternate_first_name4%2Cphysician_profile_alternate_last_name4%2Cphysician_profile_alternate_first_name5%2Cphysician_profile_alternate_last_name5%2Clocation&%24where=STARTS_WITH(UPPER(physician_profile_first_name)%2C%20%27ROBERT%27)%20AND%20STARTS_WITH(UPPER(physician_profile_last_name)%2C%20%27B%27)%20AND%20STARTS_WITH(UPPER(physician_profile_city)%2C%20%27PALO_ALTO%27)&%24order=physician_profile_last_name%20ASC%2Cphysician_profile_first_name%20ASC&%24limit=300
使用请求
print(requests.get('https://openpaymentsdata.cms.gov/resource/khdp-6xuy.json?%24select=%3Aid%2Cphysician_profile_id%2Cphysician_profile_last_name%2Cphysician_profile_middle_name%2Cphysician_profile_first_name%2Cphysician_profile_suffix%2Cphysician_profile_primary_specialty%2Cphysician_profile_address_line_1%2Cphysician_profile_address_line_2%2Cphysician_profile_city%2Cphysician_profile_state%2Cphysician_profile_province_name%2Cphysician_profile_country_name%2Cphysician_profile_zipcode%2Cphysician_profile_alternate_first_name1%2Cphysician_profile_alternate_last_name1%2Cphysician_profile_alternate_first_name2%2Cphysician_profile_alternate_last_name2%2Cphysician_profile_alternate_first_name3%2Cphysician_profile_alternate_last_name3%2Cphysician_profile_alternate_first_name4%2Cphysician_profile_alternate_last_name4%2Cphysician_profile_alternate_first_name5%2Cphysician_profile_alternate_last_name5%2Clocation&%24where=STARTS_WITH(UPPER(physician_profile_first_name)%2C%20%27ROBERT%27)%20AND%20STARTS_WITH(UPPER(physician_profile_last_name)%2C%20%27B%27)%20AND%20STARTS_WITH(UPPER(physician_profile_city)%2C%20%27PALO_ALTO%27)&%24order=physician_profile_last_name%20ASC%2Cphysician_profile_first_name%20ASC&%24limit=300').json())
输出
[{':id': 'row-9mfk-w6hd-ejup','physician_profile_id': '966387','physician_profile_last_name': 'BOCIAN','physician_profile_middle_name': 'C','physician_profile_first_name': 'ROBERT','physician_profile_primary_specialty': 'Allopathic & Osteopathic Physicians|Allergy & Immunology|Allergy','physician_profile_address_line_1': '795 EL CAMINO REAL','physician_profile_city': 'PALO ALTO','physician_profile_state': 'CA','physician_profile_country_name': 'UNITED STATES','physician_profile_zipcode': '94301-2302','physician_profile_alternate_first_name1': 'ROBERT','physician_profile_alternate_last_name1': 'BOCIAN'}]