使用RSelenium从LinkedIn抓取“体验”部分-网络抓取该部分：id =“ experience-section”

问题描述

这是我的原始here的后续文章，但这是针对使用R进行网络抓取（但是，我遵循BeautifulSoup和selenium Python库）。

我有以下R代码：

（它的作用是打开一个外部Firebox浏览器，导航到LinkedIn登录页面，输入我自己的用户名和密码，然后单击enter。因此我已登录LinkedIn，接下来打开Hadley Wickhams LinkedIn个人资料（通过remDr$navigate），然后下载该页面的页面源，最后它使用rvest网页抓取，我保留了一个Web抓取个人资料名称profileName的示例有点代码的工作原理，但我试图抓取页面的“体验”部分。

library(RSelenium)
library(rvest)
library(stringr)
library(xml2)
library(tidyverse)

userID = "myEmailAddress"
passID = "myPassword"

try(rsDriver(port = 4444L,browser = 'firefox'))
remDr <- remoteDriver()
remDr$open()
remDr$navigate("https://www.linkedin.com/login")    # go to login page

user <- remDr$findElement(using = 'id',"username")
user$sendKeysToElement(list(userID,key="tab"))

pass <- remDr$findElement(using = 'id',"password")
pass$sendKeysToElement(list(passID,key="enter"))    # logs in the user

Sys.sleep(5) # give the page time to fully load
remDr$navigate("https://www.linkedin.com/in/hadleywickham/")    # go to Hadley Wickhams profile page

Sys.sleep(5) # give the page time to fully load
html <- remDr$getPageSource()[[1]]                             # downloads the page source code

signals <- read_html(html)

# Collect the profile header tag - name,location,etc.

profileHeaderTag <- signals %>% 
  html_nodes("div") %>% 
  html_nodes("[class='flex-1 mr5']")                      # locates the relevant tag for profile info


profileName <- profileHeaderTag %>% 
  html_nodes("ul") %>% 
  html_node("li") %>% 
  html_text() %>%                             # extract just the name of the profile person
  str_trim() %>%                              # removes unwanted white space (similar to .strip in Python)
  .[1]

profileName # which prints "Hadley Wickham"

######## Stuck here: ##############

signals %>% 
  html_nodes("ul")

# @ minute 22.06 of this video: https://www.youtube.com/watch?v=QVnBCtgS_-U
# The python code is:
# exp_section = soup.find('section',{'id': 'experience-section'})
# Where "soup" is the authors version of my "signals" R code.
# I want to find the 'experience-section' part but cannot get it working.

# When I click "inspect element" on the LinkedIn profile source code I find a section called:
#<section id="experience-section" class="pv-profile-section experience-section ember-view">

我想从个人资料中抓取“体验”部分。从第22.06分钟的以下Youtube video开始，我已经在Python中看到了这是如何完成的，Python代码是exp_section = soup.find('section',{'id': 'experience-section'})。

我遇到的问题是尝试在R中获得此部分，当我转到LinkedIn个人资料并右键单击并按“检查元素”时，我可以导航到显示<section id="experience-section" class="pv-profile-section experience-section ember-view">的源代码。但是，这部分我似乎无法在R会话中处理。 （似乎返回一个空单元格。）

在Python中，我遵循了原始的Youtubes代码，即使在Python中，我仍然无法获取“体验部分”来打印，并且它返回了“无”值。

Python代码：

import requests,time,random
from bs4 import BeautifulSoup
from selenium import webdriver

browser = webdriver.Chrome('chromedriver.exe')     # need to download chromedriver and put it in the same directory
browser.get('https://www.linkedin.com/uas/login')
userID = "myEmailAddress"
passID = "myPassword"

elementID = browser.find_element_by_id('username')
elementID.send_keys(userID)

elementID = browser.find_element_by_id('password')
elementID.send_keys(passID)

elementID.submit()

link = 'https://www.linkedin.com/in/hadleywickham/'
browser.get(link)

SCROLL_PAUSE_TIME = 5

# Get scroll height
last_height = browser.execute_script("return document.body.scrollHeight")

for i in range(3):
    # Scroll down to bottom
    browser.execute_script("window.scrollTo(0,document.body.scrollHeight);")

    # Wait to load page
    time.sleep(SCROLL_PAUSE_TIME)

    # Calculate new scroll height and compare with last scroll height
    new_height = browser.execute_script("return document.body.scrollHeight")
    if new_height == last_height:
        break
    last_height = new_height

src = browser.page_source
soup = BeautifulSoup(src,'lxml')

name_div = soup.find('div',{'class': 'flex-1 mr5'})
name_div

name_loc = name_div.find_all('ul')
name = name_loc[0].find('li').get_text().strip()
name

exp_section = soup.find('section',{'id': 'experience-section'})
exp_section

print(exp_section) # which returns "None"

解决方法

暂无找到可以解决该程序问题的有效方法，小编努力寻找整理中！

如果你已经找到好的解决方法，欢迎将解决方案带上本链接一起发送给小编。

小编邮箱:dio#foxmail.com (将#修改为@）