Being somewhat impatient myself, I can understand where Ivanov is coming from, but I think this is a common reality in open source, and it is only complicated if there is a language issue. Anyone with a basic understanding of the topic (and a lot of people who don't even seem to have that) and who wants to make a name for themselves can throw up an introductory youtube video, but advanced topics require advanced understanding, and most people don't need or want that. So the 'return' for an advanced topic blog or video is smaller, because it reaches a smaller audience, an audience more likely to know a thing or two themselves and so not let a tutorial creator get away with just phoning it in. In short, you just have to deal with it.
That said, I do recommend, and have viewed many times, both the relatively new Reddit video that Tricia talks about as well as Asheesh's. To that list let me add https://www.youtube.com/playlist?list=PL51BA5190961CFEE3, which is the only long form, multiple episode Scrapy video tutorial I've seen that talks about advanced topics like getting your results into a db, rather than stopping at telling you it can be done. Nevertheless, as a wise person recently told me, you just have to let it fly and see what happens. You learn a lot that way, and it will be directly applicable to your specific projects. And if you then go back to the official docs, I think you will find after that experience that the docs start to make more sense. On Friday, August 21, 2015 at 12:31:09 AM UTC-5, Travis Leleu wrote: > > One might suggest, since this excellent tool is proving value to you, that > you either provide financial support or help with the documentation > yourself. > > Otherwise it would be easy to mistake your comments as coming from someone > without any sense of gratitude, or perhaps an inflated sense of entitlement. > > > > On Aug 20, 2015, at 9:11 PM, ivanov <[email protected] <javascript:>> > wrote: > > Hi Jakob :) > > Thanks for your help & for your attention. > > Unfortunately, I can hardly grasp the official docs. And I think I have > already told you from the beginning, that the official docs is killing me > :D If the docs can be undesrtood easily, I won't be here. haha... > > Oya, I found a good alternative to learn "scrapy pipeline". I think this > blog and forum is good for a newbie like me: > > > http://www.smallsurething.com/web-scraping-article-extraction-and-sentiment-analysis-with-scrapy-goose-and-textblob/ > > > https://stackoverflow.com/questions/29946989/renaming-downloaded-images-in-scrapy-0-24-with-content-from-an-item-field-while > > Thanks Guys :) > > > *NB:* SCRAPY SHOULD HIRE SOMEBODY TO RE-WRITE IT'S DOCS. IT'S VERY > FRUSTATING TO READ IT. > > > > On Monday, August 17, 2015 at 7:14:41 PM UTC+7, Jakob de Maeyer wrote: >> >> Hey Ivanov, >> >> now I'm unsure whether you received my private mail from the 11th, so >> here it is again: >> >> Hey Ivanov, >> >> I can point you in the right direction, but really, it's all there in >> the docs >> >> Pipelines are a really easy concept: Every Item that is scraped (i.e. >> yielded or returned) by the Spider is given to the process_item() method >> of all pipelines. This method can then inspect and modify the item and >> must do one of two things: >> - if it returns the Item, it will be processed by the next pipeline, or >> if there is no further pipeline, go to the feed exports (see >> >> http://doc.scrapy.org/en/latest/intro/tutorial.html#storing-the-scraped-data) >> >> >> - if it raises scrapy.exceptions.DropItem, this particular item will >> stop being processed, end of story. You can use this if you want to >> filter your items for certain characteristics. >> >> There are a couple of extra methods you *can* implement if you want, >> e.g. to open/close files or database connections, but literally all that >> a pipeline *must* do is have a process_item() method. All methods, their >> signatures, and their use cases are explained here: >> >> http://doc.scrapy.org/en/latest/topics/item-pipeline.html#writing-your-own-item-pipeline >> >> >> The most common use case for pipelines is to write scraped data to a >> database. The docs have an example for MongoDB: >> >> http://doc.scrapy.org/en/latest/topics/item-pipeline.html#write-items-to-mongodb >> >> >> You can have multiple pipelines, and the items will be processed in the >> order you set in your ITEM_PIPELINES setting (which you set in your >> settings.py file), as explained here: >> >> http://doc.scrapy.org/en/latest/topics/item-pipeline.html#activating-an-item-pipeline-component >> >> >> Whether you need item pipelines at all really depends on what you want >> to do. >> >> >> Cheers, >> -Jakob >> >> >> On 08/17/2015 01:35 PM, ivanov wrote: >> > Can anyone teach me to use pipeline properly? Or maybe you can tell me >> > a tutorial blog about pipeline. >> > >> > Please don't recommend the official docs. >> > >> > -- >> > You received this message because you are subscribed to a topic in the >> > Google Groups "scrapy-users" group. >> > To unsubscribe from this topic, visit >> > https://groups.google.com/d/topic/scrapy-users/ttaAatl0LCg/unsubscribe. >> >> > To unsubscribe from this group and all its topics, send an email to >> > [email protected] >> > <mailto:[email protected]>. >> > To post to this group, send email to [email protected] >> > <mailto:[email protected]>. >> > Visit this group at http://groups.google.com/group/scrapy-users. >> > For more options, visit https://groups.google.com/d/optout. >> > -- > You received this message because you are subscribed to the Google Groups > "scrapy-users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected] <javascript:>. > To post to this group, send email to [email protected] > <javascript:>. > Visit this group at http://groups.google.com/group/scrapy-users. > For more options, visit https://groups.google.com/d/optout. > > -- You received this message because you are subscribed to the Google Groups "scrapy-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/scrapy-users. For more options, visit https://groups.google.com/d/optout.
