Hi, Yang, you're running your mapreduce jobs in Hadoop's local mode, and in that mode all the Hive MR logging is handled through log4j on your local machine, which is what this log file is about.
The log location and naming is controlled by the property log4j.appender.FA.File in the Hive log4j properties file. Cheers, Andre On 19 July 2014 15:24, Yang <teddyyyy...@gmail.com> wrote: > 2014-07-18 15:03:37,774 INFO mr.ExecDriver > (SessionState.java:printInfo(537)) - Execution log at: > /tmp/myuser/myuser_2014071815030 > 3_56bf6bb0-db30-4dbc-807c-9023ce4103f4.log > 2014-07-18 15:03:37,864 WARN conf.Configuration > (Configuration.java:loadProperty(2358)) - > file:/tmp/myuser/hive_2014-07-18_15-03-30_423_ > 6799963466906099923-1/-local-10011/jobconf.xml:an attempt to override > final parameter: mapreduce.job.end-notification.max.retry.interval; Igno > ring. > 2014-07-18 15:03:37,871 WARN conf.Configuration > (Configuration.java:loadProperty(2358)) - > file:/tmp/myuser/hive_2014-07-18_15-03-30_423_ > 6799963466906099923-1/-local-10011/jobconf.xml:an attempt to override > final parameter: mapreduce.job.end-notification.max.attempts; Ignoring. > 2014-07-18 15:03:37,951 INFO log.PerfLogger > (PerfLogger.java:PerfLogBegin(108)) - <PERFLOG method=deserializePlan > from=org.apache.hadoop.hive. > ql.exec.Utilities> > 2014-07-18 15:03:37,951 INFO exec.Utilities > (Utilities.java:deserializePlan(822)) - Deserializing MapredLocalWork via > kryo > 2014-07-18 15:03:38,237 INFO log.PerfLogger > (PerfLogger.java:PerfLogEnd(135)) - </PERFLOG method=deserializePlan > start=1405721017951 end=14057 > 21018237 duration=286 from=org.apache.hadoop.hive.ql.exec.Utilities> > 2014-07-18 15:03:38,246 INFO mr.MapredLocalTask > (SessionState.java:printInfo(537)) - 2014-07-18 03:03:38 Starting to > launch local task t > o process map join; maximum memory = 4261937152 > 2014-07-18 15:03:38,261 INFO mr.MapredLocalTask > (MapredLocalTask.java:initializeOperators(406)) - fetchoperator for > null-subquery2:a-subquery2 > :dpkg_cntr:dpkg_wtransaction_p2_id_user_30m created > 2014-07-18 15:03:38,263 INFO mr.MapredLocalTask > (MapredLocalTask.java:initializeOperators(406)) - fetchoperator for > null-subquery2:a-subquery2 > :dpkg:dpkg_wtransaction_p2_id_user_30m created > 2014-07-18 15:03:38,264 INFO mr.MapredLocalTask > (MapredLocalTask.java:initializeOperators(406)) - fetchoperator for > null-subquery2:a-subquery2 > :xclick:b:wtrans_data_map_p2_30m created > 2014-07-18 15:03:38,266 INFO mr.MapredLocalTask > (MapredLocalTask.java:initializeOperators(406)) - fetchoperator for > null-subquery1:a-subquery1 > :dpkg_cntr:dpkg_wtransaction_id_user_30m created > 2014-07-18 15:03:38,268 INFO mr.MapredLocalTask > (MapredLocalTask.java:initializeOperators(406)) - fetchoperator for > null-subquery1:a-subquery1 > :dpkg:dpkg_wtransaction_id_user_30m created > 2014-07-18 15:03:38,269 INFO mr.MapredLocalTask > (MapredLocalTask.java:initializeOperators(406)) - fetchoperator for > null-subquery1:a-subquery1 > :xclick:b:wtrans_data_map_30m created > --------------------------------------------- > > whole bunch of stuff omitted here > > -------------------------------------- > > > -------------------------------------- > 2014-07-18 15:04:08,678 INFO exec.HashTableSinkOperator > (HashTableSinkOperator.java:flushToFile(278)) - Temp URI for side table: > file:/tmp/myuser/hive_2014-07-18_15-03-30_423_6799963466906099923-1/-local-10008/HashTable-Stage-2 > 2014-07-18 15:04:08,678 INFO exec.HashTableSinkOperator > (SessionState.java:printInfo(537)) - 2014-07-18 03:04:08 Dump the > side-table into file: > file:/tmp/myuser/hive_2014-07-18_15-03-30_423_6799963466906099923-1/-local-10008/HashTable-Stage-2/MapJoin-mapfile11--.hashtable > 2014-07-18 15:04:09,943 INFO exec.HashTableSinkOperator > (SessionState.java:printInfo(537)) - 2014-07-18 03:04:09 Uploaded 1 > File to: > file:/tmp/myuser/hive_2014-07-18_15-03-30_423_6799963466906099923-1/-local-10008/HashTable-Stage-2/MapJoin-mapfile11--.hashtable > (58010217 bytes) > 2014-07-18 15:04:09,943 INFO exec.HashTableSinkOperator > (Operator.java:close(591)) - 6 Close done > 2014-07-18 15:04:09,943 INFO exec.SelectOperator > (Operator.java:close(591)) - 5 Close done > 2014-07-18 15:04:09,943 INFO exec.TableScanOperator > (Operator.java:close(591)) - 4 Close done > 2014-07-18 15:04:09,951 INFO mapred.FileInputFormat > (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1 > 2014-07-18 15:04:10,008 INFO mapred.FileInputFormat > (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1 > 2014-07-18 15:04:11,526 INFO exec.HashTableSinkOperator > (SessionState.java:printInfo(537)) - 2014-07-18 03:04:11 Processing > rows: > 200000 Hashtable size: 199999 Memory usage: 190041576 > percentage: 0.045 > 2014-07-18 15:04:11,950 INFO exec.HashTableSinkOperator > (SessionState.java:printInfo(537)) - 2014-07-18 03:04:11 Processing > rows: > 300000 Hashtable size: 299999 Memory usage: 250890416 > percentage: 0.059 > 2014-07-18 15:04:12,456 INFO exec.HashTableSinkOperator > (SessionState.java:printInfo(537)) - 2014-07-18 03:04:12 Processing > rows: > 400000 Hashtable size: 399999 Memory usage: 304697120 > percentage: 0.071 > 2014-07-18 15:04:12,744 INFO exec.TableScanOperator > (Operator.java:close(574)) - 11 finished. closing... > 2014-07-18 15:04:12,745 INFO exec.FilterOperator > (Operator.java:close(574)) - 12 finished. closing... > 2014-07-18 15:04:12,745 INFO exec.FilterOperator > (Operator.java:logStats(838)) - PASSED:471403 > 2014-07-18 15:04:12,745 INFO exec.FilterOperator > (Operator.java:logStats(838)) - FILTERED:278619 > 2014-07-18 15:04:12,745 INFO exec.SelectOperator > (Operator.java:close(574)) - 13 finished. closing... > 2014-07-18 15:04:12,745 INFO exec.HashTableSinkOperator > (Operator.java:close(574)) - 14 finished. closing... > 2014-07-18 15:04:12,745 INFO exec.HashTableSinkOperator > (HashTableSinkOperator.java:flushToFile(278)) - Temp URI for side table: > file:/tmp/myuser/hive_2014-07-18_15-03-30_423_6799963466906099923-1/-local-10008/HashTable-Stage-2 > 2014-07-18 15:04:12,745 INFO exec.HashTableSinkOperator > (SessionState.java:printInfo(537)) - 2014-07-18 03:04:12 Dump the > side-table into file: > file:/tmp/myuser/hive_2014-07-18_15-03-30_423_6799963466906099923-1/-local-10008/HashTable-Stage-2/MapJoin-mapfile31--.hashtable > 2014-07-18 15:04:13,836 INFO exec.HashTableSinkOperator > (SessionState.java:printInfo(537)) - 2014-07-18 03:04:13 Uploaded 1 > File to: > file:/tmp/myuser/hive_2014-07-18_15-03-30_423_6799963466906099923-1/-local-10008/HashTable-Stage-2/MapJoin-mapfile31--.hashtable > (47251316 bytes) > 2014-07-18 15:04:13,836 INFO exec.HashTableSinkOperator > (Operator.java:close(591)) - 14 Close done > 2014-07-18 15:04:13,836 INFO exec.SelectOperator > (Operator.java:close(591)) - 13 Close done > 2014-07-18 15:04:13,836 INFO exec.FilterOperator > (Operator.java:close(591)) - 12 Close done > 2014-07-18 15:04:13,836 INFO exec.TableScanOperator > (Operator.java:close(591)) - 11 Close done > 2014-07-18 15:04:13,837 INFO mr.MapredLocalTask > (SessionState.java:printInfo(537)) - 2014-07-18 03:04:13 End of local > task; Time Taken: 35.59 sec. > > -- André Araújo Big Data Consultant/Solutions Architect The Pythian Group - Australia - www.pythian.com Office (calls from within Australia): 1300 366 021 x1270 Office (international): +61 2 8016 7000 x270 *OR* +1 613 565 8696 x1270 Mobile: +61 410 323 559 Fax: +61 2 9805 0544 IM: pythianaraujo @ AIM/MSN/Y! or ara...@pythian.com @ GTalk “Success is not about standing at the top, it's the steps you leave behind.” — Iker Pou (rock climber) -- --
