Mongo isn't perfect. And it (was/is) over-touted. But some of the issues in that article are no longer relevant. I'll go through the big ones, just to ensure the record is correct.
Locking - seems like < 2.2 it's an issue. MongoDB now is on 3.2, so this criticism isn't really valid anymore. Space efficiency - Mongo has significantly improved this with WiredTiger. I think it still allocates a decent chunk of disk for a new collection, so this is still an issue for some people. Too many databases - same root issue as above: disk space. The other criticisms are still valid issues with Mongo. If you want really flexible schema in your data layer (you'll have to do all your validation in code), I would suggest looking into the Postgres JSON data type. On Tue, May 3, 2016 at 9:23 AM, Tiago Lira <[email protected]> wrote: > Hi, Uncharted > > I took a read on the post that you mentioned, and I noticed that several > of the issues reported have been solved or minimized. And I think that the > purpose of their use of mongodb was different and heavier, it was to > store/query items that could reach millions. The purpose of mongodb here on > this project is just to queue/dequeue the tasks to be performed by scrapyd > (which was previously being held with SQLite). > > But it was nice to know they are using HBase, I'll take a look and try to > add to the library an interface for those who find it better to use HBase. > Thank you for the advice! > > > > Em terça-feira, 3 de maio de 2016 10:24:01 UTC-3, Uncharted escreveu: >> >> Hi >> >> I'm currently starting to work on the same kind of use case. >> I found this article which does not recommend mongodb : >> https://blog.scrapinghub.com/2013/05/13/mongo-bad-for-scraped-data/ >> >> They say that you'll have the same lock contention with mongodb : the >> article was written in 2013 so maybe it's not the case anymore. >> >> And they migrated to HBase which seems to be the right backend, It is >> used also in the Apache Nutch project. >> >> >> >> -- > You received this message because you are subscribed to the Google Groups > "scrapy-users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at https://groups.google.com/group/scrapy-users. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "scrapy-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/scrapy-users. For more options, visit https://groups.google.com/d/optout.
