Python：使用Selenium进行仪表板Web抓取-无法滚动内部仪表板栏

问题描述

我想从以下在线发布的仪表板的第三页表中提取数据：

link

我设法编写了一个Python脚本，该脚本使用“ driver.find_element_by_xpath”遍历可见的表元素，但是我无法使用selenium方法自动滚动表。我只能提取表格的20个可见元素而无需滚动。

我的代码在下面：

import pandas as pd
from selenium import webdriver


chromeOptions = webdriver.ChromeOptions()
chromeOptions.add_experimental_option('useAutomationExtension',False)

# path to the chromeDriver
chromedriver = 'C:/Users/chromedriver.exe'

# I dont have full admin rights on my machine and that was the only way to run driver
driver = webdriver.Chrome(chromedriver,chrome_options=chromeOptions,desired_capabilities=chromeOptions.to_capabilities())

#link to the dashboard
url2 = 'https://app.powerbi.com/view?r=eyJrIjoiM2UzZjZjNGItYzc0OS00NjVkLTgyYzYtYjk4NmRhYjQ3OTQ0IiwidCI6IjA3ZTZlZTM1LTY4MTQtNDc5MC04NjY5LTgwNzY3Njk0YzI4ZCIsImMiOjl9'

driver.get(url2)

#at this point I was going manually to the third tab of the opened browser

### Main loop to extract elements

# declare empty tables
df_all= pd.DataFrame()
# 12 as we have 12 columns in dashboard
df = pd.DataFrame(columns=['1','2','3','4','5','6','7','8','9','10','11','12'])

#iterate through columns
for j in range(1,13):
    
    # put extracted data here
    ticker_list = []
    
    # set variable for column numeration
    col = j
    
    # as Xpath for columns 11 and 12 has a different argument {k} it has to be adjusted
    if j <= 10:
        k = 1
        # set x to get value of j
        x = j
    else:
        k = 2
        
        # set x to get last number from j ( 1 from 11,2 from 12)
        x = int(str(j)[1:])
        
    # Iterate over rows. I am able to extract 20 rows at once so no point of declaring bigger range    
    for i in range(1,21):
        row = i

        try:
            print(i)
            # get value from dashboard
            tickerpath_2 = f"""//*[@id="pvExplorationHost"]/div/div/exploration/div/explore-canvas-modern/div/div[2]/div/div[2]/div[2]/visual-container-repeat/visual-container-modern[2]/transform/div/div[3]/div/visual-modern/div/div/div[2]/div[1]/div[4]/div/div[{k}]/div[{x}]/div[{row}]"""
            ticker = driver.find_element_by_xpath(tickerpath_2)
            
            print(ticker.text)
            # append element to the list
            ticker_list.append(ticker.text)

        
        except:
            continue
    # append whole list as a pandas column
    df[str(col)] = ticker_list

df_all = df_all.append(df)

现在在变量df_all中，您应该具有表的前20行。没关系但是，我无法自动滚动右侧的仪表板栏。我尝试了类似帖子中的各种选项，例如：

driver.execute_script("window.scrollTo(0,document.body.scrollHeight)") 

或

scr1 = driver.find_element_by_xpath('/')
driver.execute_script("arguments[0].scrollTop = arguments[0].scrollHeight",scr1)

或

read_more_buttons = driver.find_elements_by_class_name("scroll-bar-part-bar")
driver.execute_script("arguments[0].scrollIntoView(true);",read_more_buttons[1])
driver.execute_script("arguments[0].click()",read_more_buttons[1])

似乎这些方法在Dashboard的内部滚动条上不起作用。也许有人遇到过类似的问题？ 否则，我将必须手动滚动每20行之后停止脚本：（

感谢您提前获得的所有帮助

解决方法

暂无找到可以解决该程序问题的有效方法，小编努力寻找整理中！

如果你已经找到好的解决方法，欢迎将解决方案带上本链接一起发送给小编。

小编邮箱:dio#foxmail.com (将#修改为@）