Scrapy请求-嵌套请求中未调用回调函数

问题描述

我正尝试从亚马逊刮擦一些产品，以获取有关我的竞争对手的一些信息。这是我正在采用的过程：

Make a query in the search bar ->
Visit every product page of the results gotten from the query -> 
Gather information from that product ->
Check if the product matches the quantity that we looked for (I.E. We might want to collect only products sold in a pack of n items ... like a kit of n toner cartridges)
    -> If it does,yield the item.
    -> If not,find a variation in that ad that represents a pack of such n items
         -> If such a variation exists,go visit that variation of the product,modify some information of the item (such as price and asin) and yield that item.

我在这里有一个特殊的情况。我不会发布我拥有的所有功能，而是发布一些代表性的功能（为了使它更简短，更通用，以便将来对其他人有用 ）。

这是我的代码的结构：

def start_requests(self):
        for i,prod in enumerate(products):
            url = 'https://www.amazon.it/s?' + urlencode({'k': prod['query']})
            competitors = scrapy.Request(url=url,callback=self.parse_keyword_response,meta={'prod':prod})
            yield competitors


def parse_keyword_response(self,response):
        # Function that loops on the results of the query made,# and collects all the products that actually match our search
        products = response.xpath('//*[@data-asin]')
        prod = response.meta['prod']

        competitors =[]

        for product in products:
            asin = product.xpath('@data-asin').extract_first()
            product_url = f"https://www.amazon.it/dp/{asin}"
            competitor = scrapy.Request(url=product_url,callback=self.parse_competitor_product_page,meta={'asin': asin,'prod':prod})
            yield competitor
            competitors.append(competitor)


def parse_competitor_product_page(self,response):
        # Function that scrapes information from a product page and yields the competitor
        # only if it actually matches our search.

        ' Do some work and scrape required product attributes'

        competitor = ProductItem()
        competitor['product'] = prod_name
        competitor['asin'] = asin
        competitor['Title'] = title
        competitor['producer'] = producer
        competitor['MainImage'] = image
        competitor['Rating'] = rating
        competitor['NumberOfReviews'] = number_of_reviews
        competitor['price'] = price
        competitor['AvailableSizes'] = sizes
        competitor['AvailableColors'] = colors
        competitor['Varieties'] = varieties
        competitor['BulletPoints'] = bullet_points
        competitor['SellerRank'] = seller_rank

        if self.is_right_product(prod,competitor,response):
            yield competitor

def is_right_product(self,product,response):
       # Function that checks whether a resulting competitor actually matches the product that 
       # we looked for. It returns a boolean if it does. It also alters some attributes of that
       # competitor if a right variation is found on its page.

      ' I will omit some if else branches as those work well and I will only post the faulty 
           branch (which happens to be the one that should modify the competitor object because 
           a right variation is found on its page. '

      if product_is_right_quantity(competitor):
           return True
      else:
           variation = find_variation_of_right_quantity(product['quantity'],competitor)
           if vatiation is not None:
                competitor = self..update_product_to_right_variation(competitor,variation,response)
                print("variation check done")
                return True
           else:
                return False

def update_product_to_right_variation(self,variation_name,response):
        print("IN UPDATE PRODUCT TO RIGHT VARIATION")
        variation_asin = response.xpath(f'//div[@id="variation_color_name"]/ul/li[contains(@title,\'{variation_name}\')]/@data-defaultasin').get()
        product_url = f"https://www.amazon.it/dp/{variation_asin}"
        print(product_url)
        yield scrapy.Request(url=product_url,callback=self.update_competitor_from_product_page,errback=self.errback_http,meta={'prod':product,'asin':variation_asin})

def update_competitor_from_product_page(self,response):
        print("INSIIDE UPDATE COMPETITOR FROM PRODUCT PAGE")
        prod = response.meta['prod']
        asin = response.meta['asin']

        price = response.xpath('//*[@id="priceblock_ourprice"]/text()').extract_first()

        prod['price'] = price
        prod['Title'] = title
        prod['asin'] = asin

        response.meta['prod'] = prod
        print(prod['price'])
        return prod

如您所见，我放置了一些打印语句以用于调试。

update_competitor_from_product_page中的打印语句永远不会输出。

所有其他人都这样做。因此，永远不会调用应用作update_product_to_right_variation中发出的请求的回调函数的函数。结果，竞争者对象保持不变。

我是异步编程的新手，也是Scrapy的新手。

首先，我想知道为什么从未调用过我的回调函数。其次，我该怎么做？

解决方法

我无法测试它，但是问题可能是您尝试在函数yield Request中执行的函数parse_competitor_product_page()中is_right_product()-在parse_competitor_product_page()中执行-函数yield中的return / parse_competitor_product_page()无法直接将其发送到Scrapy Engine，但会将其发送到先前的函数is_right_product()，该函数应yield / {{1 }}到先前的功能return-在parse_competitor_product_page()中，您应该parse_competitor_product_page()，然后将其发送给yield引擎，该引擎将执行它。

在您的代码中，您从Scrapy yield Request到parse_competitor_product_page()，但是在is_right_product()中，您发送了is_right_product() / return True，因此它不会发送return False到Request，并且无法将其发送到Scrapy引擎

---

我认为您需要这样的东西

parse_competitor_product_page()