Could you increase this number (Mebbe three times the current value) and see if it has any impact on throughput:
--hiveconf mapreduce.input.fileinputformat.split.maxsize=33554432 \ Hey Namaskara~Nalama~Guten Tag~Bonjour -- Keigu Deepak 73500 12833 www.simtree.net, dee...@simtree.net deic...@gmail.com LinkedIn: www.linkedin.com/in/deicool Skype: thumsupdeicool Google talk: deicool Blog: http://loveandfearless.wordpress.com Facebook: http://www.facebook.com/deicool "Contribute to the world, environment and more : http://www.gridrepublic.org " On Mon, Jun 20, 2016 at 8:13 PM, David Nies <david.n...@adition.com> wrote: > Dear Hive mailing list, > > in my setup, network throughput from the HiveServer2 to the client seems > to be the bottleneck and I’m seeking a way do increase throughput. Let me > elaborate my use case: > > I’m using Hive version 1.1.0 that is bundeled with Clouders 5.5.1. > > I want to fetch a huge amount of data from our Hive cluster. By huge I > mean something around 100 million rows. The Hive table I’m querying is an > external table whose data is stored in .avro. On HDFS, the data I want to > fetch (i.e. the aforementioned 100 million rows) is about 5GB in size. A > cleverer filtering strategy (to reduce the amount of data) is no option, > sadly, since I need all the data. > > I was able to reduce the time the MapReduce job takes to an agreeable > interval fiddling around with > `mapreduce.input.fileinputformat.split.maxsize`. The part that is taking > ages comes after MapReduce. I’m observing that the Hadoop namenode that is > hosting the HiveServer2 is merely sending data with around 3 MB/sec. Our > network is capable of much more. Playing around with `fetchSize` did not > increase throughput. > > As I identified network throughput to be the bottleneck, I restricted my > efforts to trying to increase it. For this, I simply run the query I’d > normally run through JDBC (from Clojure/Java) via `beeline` and dumping the > output to `/dev/null`. My `beeline` query looks something like that: > > beeline \ > -u jdbc:hive2://srv:10000/db \ > -n user -p password \ > --outputformat=csv2 \ > --incremental=true \ > --hiveconf mapreduce.input.fileinputformat.split.maxsize=33554432 \ > -e 'SELECT <a lot of columns> FROM `db`.`table` WHERE (year=2016 AND > month=6 AND day=1 AND hour=10)' > /dev/null > > I already tried playing around with additional `—hiveconf`s: > > --hiveconf hive.exec.compress.output=true \ > --hiveconf mapred.output.compression.type=BLOCK \ > --hiveconf > mapred.output.compression.codec=org.apache.hadoop.io.compress.SnappyCodec \ > > without success. > > In all cases, Hive is able only to utilize a tiny fraction of the > bandwidth that is available. Is there a possibility to increase network > throughput? > > Thank you in advance! > > Yours > > David Nies > Entwickler Business Intelligence > ADITION technologies AG > > Oststraße 55, D-40211 Düsseldorf > Schwarzwaldstraße 78b, D-79117 Freiburg im Breisgau > > T +49 211 987400 30 > F +49 211 987400 33 > E david.n...@adition.com > > Technischen Support erhalten Sie unter der +49 1805 2348466 > (Festnetzpreis: 14 ct/min; Mobilfunkpreise: maximal 42 ct/min) > > Abonnieren Sie uns auf XING oder besuchen Sie uns unter www.adition.com. > > Vorstände: Andreas Kleiser, Jörg Klekamp, Dr. Lutz Lowis, Marcus Schlüter > Aufsichtsratsvorsitzender: Joachim Schneidmadl > Eingetragen beim Amtsgericht Düsseldorf unter HRB 54076 > UStIDNr.: DE 218 858 434 > > >
