Thanks for the answer we will look in to it and update The impala is impala parquet table
-----Original Message----- From: Gabriel Reid [mailto:[email protected]] Sent: Tuesday, December 23, 2014 2:27 PM To: [email protected] Subject: Re: CSV bulk loading using map reduce Hi Noam, I think that the things that most typically can affect MR loading performance are: * number of regions (as this affects the number of reducers used to create the HFiles) * amount of memory used for sort buffers * use of compression on map output With your 32-region salted table, it sounds like things should be well-balanced, so that shouldn't be an issue. I would expect the biggest impact you can possibly get is by tuning the sort buffers. You can spot if there are potential issues here by looking at the "Spilled records" counter in the list of job counters in the job tracker. If you're getting multiple times the number of spilled records vs the number of output records, then it'll probably be advantageous to up the sort buffer settings. There's an old but good overview of these concepts starting from slide 12 in this presentation: http://www.slideshare.net/cloudera/mr-perf About the compression on map output, the mapreduce.map.output.compress should almost certainly be set to true, with (probably) snappy used for compression. Having this set to true compresses the output of the map task before it's sent across the network to the reducers. Since HBase KeyValues are very bulky, having this setting turned on can have quite an impact on throughput. As far as the performance comparison with Impala, I assume you're referring to impala that is backed by text files or Parquet, correct? - Gabriel On Tue, Dec 23, 2014 at 9:43 AM, Bulvik, Noam <[email protected]> wrote: > We are testing the CSV bulk loader tool (using MR) and trying to > understand what are the parameter that can be used to enhanced its > performance. > > we have 8 nodes with 4 disks each (1 core per disk) we created 32 > regions and pre-split our table using salt, the table has 55 columns > > > > So far we managed to load ~40,000 rows per second (from csv file of ~ > 8G), when we compare this loading to impala loading on the same > cluster, same table structure we see rates of about 300,000 rows per seconds. > > > > Any idea which parameters of Hbase /Phoenix can we use to enhance the > performance > > > > Regards, > > Noam > > > > Information in this e-mail and its attachments is confidential and > privileged under the TEOCO confidentiality terms that can be reviewed here. Information in this e-mail and its attachments is confidential and privileged under the TEOCO confidentiality terms that can be reviewed here<http://www.teoco.com/email-disclaimer>.
