Hey, I posted this in another thread
http://pastebin.com/H7zLw1FK You're right, it will process all listings first and then do the more details listings. There really isnt any 'games' or matching up, its pretty seamless to do this. Key is not yielding the item on that first pass, but rather the request to jump into the 'more details' function. Someone may know a way to have scrapy view the 'more details' listing before moving on to the second listing, but I don't. On Wed, Oct 7, 2015 at 4:50 PM, house_scraper <[email protected]> wrote: > Hi all, > > the page I am scraping has the following structure: > > > *Listing 1* > > *Link to more details for listing 1* > > *Listing 2* > > *Link to more details for listing 2* > > *Listing 2* > > *Link to more details for listing 2* > > > > > > I want to follow the "*Link to more details for listing 1" * from my > spider code that processes listing 1, fill a field for item 1, and then > move onto listing 2, follow the listing 2 link, etc and iterate through all > the items in order. > > However, scrapy's twistedreactor/callback philosophy seems to prevent > this, and forces me to *first *do all the items without following links, > and *then *follow all the links using a linkextractor and play games to > match up each followed link to the listing/item it came from. > > Can I just confirm that this is correct? > > > > -- > You received this message because you are subscribed to the Google Groups > "scrapy-users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at http://groups.google.com/group/scrapy-users. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "scrapy-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/scrapy-users. For more options, visit https://groups.google.com/d/optout.
