I am sure that Renato (if he is watching) can plugin maybe as well. We find in Gora that in every sense of the word, native Hadoop stores such as Avro, HBase and Accumulo when we execute a query with GiraInputFormat via getParitions we retrieve GoraInputSplits natively which means splits are obtained for MapReduce jobs... such as many of the jobs we run in Nutch as well. On the other hand (currently) stores such as Cassandra and Web service stores such as DynamoDB do not support Hadoop out of the box (the former we are working on and hope to have implemented in Gora soon) therefore it is not as simple to get partitions in the same way we would in a Hadoop native store. We therefore obtain one partition to be used as an InputSplit for the MR job. This is certainly an area for concern and right now a bottleneck for some operations. We continue to work on this.
On Wednesday, August 7, 2013, Julien Nioche <[email protected]> wrote: > Hi Otis > > Definitely *not *the fetching speed. Actually everything but *not* the > fetching speed. The fetcher is pretty much the same as 1.x and anyway the > performance with fetching is pretty much always limited by the politeness > settings, not the implementation. > > Re-backend : some backend implementations are more mature than others. The > one for HBase is probably the one most widely used, the Cassandra one has > been greatly improved in particular performance-wise , the SQL one is > broken etc... we need to measure this as this is just a gut feeling at this > stage > > Now for what is slower and why, again this has to be measured but I expect > 2.x to be slower partly because of [1], i.e. the filtering of entries is > not done by the backends (some might provide a way of doing it) but this is > done on the client side, when we create the input for mapred. In other > words we pull things from the backend just to discard it. Since 2.x does > not have segments like 1.x (which the fetch + parse mapreduce jobs take as > single input) we scan the whole table even if we want to fetch or parse a > handful of entries. > > On the other hand, 2.x specifies what columns to retrieve for a given job, > whereas 1.x will for instance deserialize the crawldatum entirely. The > metadata objects are costly to read/write so 2.x might have the upper hand > from that point of view since it pulls and deserializes only what it needs. > > Finally the most costly steps in a large crawl in 1.x are the generation > and update as we have to read/write the crawldb entirely. The way the > updates are done in 2.x is different and should be a lot faster. > > Please could anyone correct me if I am wrong. Some of this is based on my > understanding of 2.x which dates back from quite a while and some of the > stuff might have changed in the meantime. The performance would probably > vary a lot based on the fine tuning of each backend implementation but > having some basic comparison would confirm some of the assertions above. > > Julien > > > [1] https://issues.apache.org/jira/browse/GORA-119 > > > Julien, could you please elaborate a bit about your comment about speed >> depending on the backend used? >> >> Yes, you were the person I was referring to :) >> >> Oh, and *believe* you said it was the fetching speed that was different >> between 1.x and 2.x. Is that right? Or is some other phase slower in 2.x? >> >> Thanks, >> Otis >> ---- >> Performance Monitoring for Solr / ElasticSearch / Hadoop / HBase - >> http://sematext.com/spm >> >> >> >> >> >________________________________ >> > From: Julien Nioche <[email protected]> >> >To: "[email protected]" <[email protected]> >> >Sent: Tuesday, August 6, 2013 10:54 AM >> >Subject: Re: 2.x vs. 1.x speed >> > >> > >> >Hi Otis, >> > >> >That certainly depends on the backend used but on the whole it wouldn't be >> >surprising. Would be good to have some data to substantiate it. I am >> >planning to put my intern on the case and have some basic comparison as >> >soon as she gets a good grip of Hadoop / Nutch etc... but if someone else >> >wants to do it please go ahead. >> > >> >In case I happen to be the person who told you that Otis, well at least I >> >am consistent ;-) >> > >> >Julien >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> >On 6 August 2013 09:08, Otis Gospodnetic <[email protected]> >> wrote: >> > >> >> Hello, >> >> >> >> At some point earlier this year I spoke to a person who told me 2.x is >> >> (a little?) slower than 1.x. Is that still the case? >> >> >> >> Thanks, >> >> Otis >> >> -- >> >> Solr & ElasticSearch Support -- http://sematext.com/ >> >> Performance Monitoring -- http://sematext.com/spm >> >> >> > >> > >> > >> >-- >> >* >> >*Open Source Solutions for Text Engineering >> > >> >http://digitalpebble.blogspot.com/ >> >http://www.digitalpebble.com >> >http://twitter.com/digitalpebble >> > >> > >> > >> > > > > -- > * > *Open Source Solutions for Text Engineering > > http://digitalpebble.blogspot.com/ > http://www.digitalpebble.com > http://twitter.com/digitalpebble > -- *Lewis*
