问题描述
我希望每个线程都以不同的IP运行。是否可以为每个线程使用不同的IP? 到目前为止,我无法为每个线程分配不同的IP地址。目标是在不被禁止的情况下进行网页抓取。
f1-> ip X
f2-> ip y
f3-> ip z
这是我的代码:
import requests
from stem.control import Controller
from stem import Signal
import json
import time
import concurrent.futures
from fake_useragent import UserAgent
def get_tor_session():
# initialize a requests Session
session = requests.Session()
# setting the proxy of both http & https to the localhost:9050
# this requires a running Tor service in your machine and listening on port 9050 (by default)
session.proxies = {"http": "socks5h://localhost:9150","https": "socks5h://localhost:9150"}
return session
def renew_connection():
with Controller.from_port(port=9051) as c:
c.authenticate()
# send NEWNYM signal to establish a new clean connection through the Tor network
c.signal(Signal.NEWNYM)
ua=UserAgent()
headers = {'User-Agent':ua.random}
urlA = 'https://api.ipify.org'
def serveur():
renew_connection()
s = get_tor_session()
rA = s.get(urlA,headers=headers)
print(rA.text)
def Th():
with concurrent.futures.ThreadPoolExecutor() as executor:
f1 = executor.submit(serveur)
f2 = executor.submit(serveur)
f3 = executor.submit(serveur)
Th()
这是当前收益:
185.220.101.14
185.220.101.14
185.220.101.14
解决方法
这可能有用:
我有一个任务来实现自动更新多代理。
我用Tor来做。这是一个小包,说明如何运行多个Tor实例:https://github.com/detonavomek/tor-pool
我使用macOS。还有privoxy
代理的助手。这是一个privoxy Python软件包:https://github.com/detonavomek/privoxy-pool
有关如何使用它的信息在这里:https://github.com/detonavomek/privoxy-pool/blob/master/test_controller.py
要安装:
pip install git+https://github.com/detonavomek/privoxy-pool.git@https://github.com/detonavomek/privoxy-pool.git