I'll let Hive folks answer the questions about the Hive counters. In terms of the CPU counter - that was a bug in Tez-0.4.0, which has been fixed in 0.5.0.
COMMITTED_HEAP_BYTES just represents the memory available to the JVM (Runtime.getRuntime().totalMemory()). This will only vary if the VM is started with a different Xms and Xmx option. In terms of Tez, the application logs are currently the best place. Hive may expose these in a more accessible manner though. On Wed, Aug 20, 2014 at 11:16 PM, Suma Shivaprasad < sumasai.shivapra...@gmail.com> wrote: > Hi, > > Needed info on where I can get detailed job counters for Hive on Tez. Am > running this on a HDP cluster with Hive 0.13 and see only the following job > counters through Hive Tez in Yarn application logs which I got through( > yarn logs -applicationId ...) . > > a. Cannot see any ReduceOperator counters and also only DESERIALIZE_ERRORS > is the only counter present in MapOperator > b. The CPU_MILLISECONDS in some cases in -ve. Is CPU_MILLISECONDS accurate > c. What does COMMITTED_HEAP_BYTES indicate? > d. Is there any other place I should be checking the counters? > > [[File System Counters > FILE: BYTES_READ=512, > FILE: BYTES_WRITTEN=3079881, > FILE: READ_OPS=0, FILE: LARGE_READ_OPS=0, FILE: WRITE_OPS=0, HDFS: > BYTES_READ=8215153, HDFS: BYTES_WRITTEN=0, HDFS: READ_OPS=3, HDFS: > LARGE_READ_OPS=0, HDFS: WRITE_OPS=0] > > [org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=222543, > GC_TIME_MILLIS=172, *CPU_MILLISECONDS=-19700*, > PHYSICAL_MEMORY_BYTES=667566080, VIRTUAL_MEMORY_BYTES=1887797248, > COMMITTED_HEAP_BYTES=1011023872, INPUT_RECORDS_PROCESSED=222543, > OUTPUT_RECORDS=222543, > OUTPUT_BYTES=23543896, > OUTPUT_BYTES_WITH_OVERHEAD=23989024, OUTPUT_BYTES_PHYSICAL=3079369, > ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0, > ADDITIONAL_SPILL_COUNT=0] > > > [*org.apache.hadoop.hive.ql.exec.MapOperator*$Counter > DESERIALIZE_ERRORS=0]] > > Thanks > Suma >
