Hi Guys,
I am scrapping a product website, where products are divided into different
categories and when you click on the each category it load the product,
however product pagination is handled by AJAX.
What I mean to say is, Initially you say only few products say, 50 once
you scroll to bottom of the page new products are loaded. I have figure
out the AJAX request which is being used to to load the new products.
The problem is, I don't know how many pages it has and so I don't know how
to stop sending more request. Following is the code.
*I am not able to figure out how to stop sending more request from test
function. I tried the return statment in callback function test2, however
it ain''t working. I tried setting one variable as, however it ain't
working either. *
def test(self, response):
page=1
while not self.got_all:
data={'ProductShowcaseInput':
'{"PgControlId":46836,"IsConfigured":true,"ConfigurationType":"","CombiIds":"","PageNo":%i,"DivClientId":"ctl00_ContentPlaceHolder1_ctl00_ctl03_Showcase","SortingValues":"CS","ShowViewType":"H","PropertyBag":null,"IsRefineExsists":false,"CID":"CK00002495","CT":0,"TabId":"0","LocationIds":"4729","CurrencyCode":"INR","ContentType":"B"}'
%page
}
yield
FormRequest(url="https://www.bestpricewholesale.co.in/Handler/ProductShowcaseHandler.ashx";,
formdata=data,
callback=self.test2,
method='GET')
page += 1
def test2(self, response):
item = BestpricewholesaleItem()
hxs = HtmlXPathSelector(response)
sku = hxs.select("//div[@class='sku']/text()").extract()
if self.prv_sku == sku:
self.got_all = True
offer_price=hxs.select("//label[@class='mtb-ofr']/span[@class='sp_amt']/text()").extract()
mrp=hxs.select("//label[@class='mtb-mrp']/span[@class='sp_amt']/text()").extract()
self.prv_sku = sku
item['sku'] = sku
item['offer_price'] = offer_price
item['mrp'] = mrp
--
You received this message because you are subscribed to the Google Groups
"scrapy-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to scrapy-users+unsubscr...@googlegroups.com.
To post to this group, send email to scrapy-users@googlegroups.com.
Visit this group at http://groups.google.com/group/scrapy-users.
For more options, visit https://groups.google.com/d/optout.
