The planning phase needs to do work for every hive partition and every hadoop files. If you have a lot of 'small' files or many partitions this can take a long time. Also the planning phase that happens on the job tracker is single threaded. Also the new yarn stuff requires back and forth to allocated containers.
Sometimes raising the heap to for the hive-cli/launching process helps because the default heap of 1 GB may not be a lot of space to deal with all of the partition information and memory overhead will make this go faster. Sometimes setting the min split size higher launches less map tasks which speeds up everything. So the answer...Try to tune everything, start hive like this: bin/hive -hiveconf hive.root.logger=DEBUG,console And record where the longest spaces with no output are, that is what you should try to tune first. On Fri, Jul 18, 2014 at 9:36 AM, diogo <di...@uken.com> wrote: > This is probably a simple question, but I'm noticing that for queries that > run on 1+TB of data, it can take Hive up to 30 minutes to actually start > the first map-reduce stage. What is it doing? I imagine it's gathering > information about the data somehow, this 'startup' time is clearly a > function of the amount of data I'm trying to process. > > Cheers, >
