mapred.min.split.size = mapreduce.input.fileinputformat.split.maxsize = 1 mapred.max.split.size = mapreduce.input.fileinputformat.split.maxsize = 134217728 hive.hadoop.supports.splittable.combineinputformat = false
My average file size is pretty small... it's usually between 500K and 20MB. So it looks like the splittable support is turned off? I've been seeing some posts on the mailing list saying there's correctness problems when using this and LZO. Is this still the case? Can I turn this on with LZ4? Thanks! On Wed, Sep 30, 2015 at 1:38 PM, Ryan Harris <ryan.har...@zionsbancorp.com> wrote: > Also... > > mapreduce.input.fileinputformat.split.maxsize > > > > and, what is the size of your input files? > > > > *From:* Ryan Harris > *Sent:* Wednesday, September 30, 2015 2:37 PM > *To:* 'user@hive.apache.org' > *Subject:* RE: CombineHiveInputFormat not working > > > > what are your values for: > > mapred.min.split.size > > mapred.max.split.size > > hive.hadoop.supports.splittable.combineinputformat > > > > > > *From:* Pradeep Gollakota [mailto:pradeep...@gmail.com] > *Sent:* Wednesday, September 30, 2015 2:20 PM > *To:* user@hive.apache.org > *Subject:* CombineHiveInputFormat not working > > > > Hi all, > > > > I have an external table of with the following DDL. > > > > ``` > > DROP TABLE IF EXISTS raw_events; > > CREATE EXTERNAL TABLE IF NOT EXISTS raw_events ( > > raw_event_string string) > > PARTITIONED BY (dc string, community string, dt string) > > STORED AS TEXTFILE > > LOCATION '/lithium/events/{dc}/{community}/events/{year}/{month}/{day}' > > ``` > > > > The files are loaded externally and are LZ4 compressed. When I run a query > on this table for a single day, I'm getting 1 mapper per file even though > the input format is set to CombineHiveInputFormat. > > > > Does anyone know if CombineHiveInputFormat does not work with LZ4 > compressed files or have any idea why split combination is not working? > > > > Thanks! > > Pradeep > ------------------------------ > THIS ELECTRONIC MESSAGE, INCLUDING ANY ACCOMPANYING DOCUMENTS, IS > CONFIDENTIAL and may contain information that is privileged and exempt from > disclosure under applicable law. If you are neither the intended recipient > nor responsible for delivering the message to the intended recipient, > please note that any dissemination, distribution, copying or the taking of > any action in reliance upon the message is strictly prohibited. If you have > received this communication in error, please notify the sender immediately. > Thank you. >
