Glad I was able to help :) I also just realized my code snippet doesnt have `yield item` at the end of the productpage function, but I assume you know you have to do that.
On Thu, Oct 8, 2015 at 6:17 AM, house_scraper <[email protected]> wrote: > > Key is not yielding the item on that first pass, but rather the request > to jump into the 'more details' function > > AHA! I knew there would be a way to do this working with the twisted > philosophy rather than against it! Thanks! > > > > On Wednesday, 7 October 2015 13:56:26 UTC-7, Jeremy D wrote: >> >> Hey, >> >> I posted this in another thread >> >> http://pastebin.com/H7zLw1FK >> >> You're right, it will process all listings first and then do the more >> details listings. There really isnt any 'games' or matching up, its pretty >> seamless to do this. Key is not yielding the item on that first pass, but >> rather the request to jump into the 'more details' function. >> >> Someone may know a way to have scrapy view the 'more details' listing >> before moving on to the second listing, but I don't. >> >> On Wed, Oct 7, 2015 at 4:50 PM, house_scraper <[email protected]> wrote: >> >>> Hi all, >>> >>> the page I am scraping has the following structure: >>> >>> >>> *Listing 1* >>> >>> *Link to more details for listing 1* >>> >>> *Listing 2* >>> >>> *Link to more details for listing 2* >>> >>> *Listing 2* >>> >>> *Link to more details for listing 2* >>> >>> >>> >>> >>> >>> I want to follow the "*Link to more details for listing 1" * from my >>> spider code that processes listing 1, fill a field for item 1, and then >>> move onto listing 2, follow the listing 2 link, etc and iterate through all >>> the items in order. >>> >>> However, scrapy's twistedreactor/callback philosophy seems to prevent >>> this, and forces me to *first *do all the items without following >>> links, and *then *follow all the links using a linkextractor and play >>> games to match up each followed link to the listing/item it came from. >>> >>> Can I just confirm that this is correct? >>> >>> >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "scrapy-users" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To post to this group, send email to [email protected]. >>> Visit this group at http://groups.google.com/group/scrapy-users. >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> -- > You received this message because you are subscribed to the Google Groups > "scrapy-users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at http://groups.google.com/group/scrapy-users. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "scrapy-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/scrapy-users. For more options, visit https://groups.google.com/d/optout.
