[
https://issues.apache.org/jira/browse/TEZ-4499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
László Bodor updated TEZ-4499:
------------------------------
Description:
Both ShuffleManager and ShuffleScheduler has fields like:
{code}
this.approximateInputRecords =
inputContext.getCounters().findCounter(TaskCounter.APPROXIMATE_INPUT_RECORDS);
this.shuffledInputsCounter =
inputContext.getCounters().findCounter(TaskCounter.NUM_SHUFFLED_INPUTS);
this.failedShufflesCounter =
inputContext.getCounters().findCounter(TaskCounter.NUM_FAILED_SHUFFLE_INPUTS);
this.bytesShuffledCounter =
inputContext.getCounters().findCounter(TaskCounter.SHUFFLE_BYTES);
this.decompressedDataSizeCounter =
inputContext.getCounters().findCounter(TaskCounter.SHUFFLE_BYTES_DECOMPRESSED);
this.bytesShuffledToDiskCounter =
inputContext.getCounters().findCounter(TaskCounter.SHUFFLE_BYTES_TO_DISK);
this.bytesShuffledToMemCounter =
inputContext.getCounters().findCounter(TaskCounter.SHUFFLE_BYTES_TO_MEM);
this.bytesShuffledDirectDiskCounter =
inputContext.getCounters().findCounter(TaskCounter.SHUFFLE_BYTES_DISK_DIRECT);
{code}
we should collect then into a separate class like "ShuffleCounters" initialized
by an inputContext and increment them through an interface
> Refactor shuffle counters to a common class
> -------------------------------------------
>
> Key: TEZ-4499
> URL: https://issues.apache.org/jira/browse/TEZ-4499
> Project: Apache Tez
> Issue Type: Sub-task
> Reporter: László Bodor
> Priority: Major
>
> Both ShuffleManager and ShuffleScheduler has fields like:
> {code}
> this.approximateInputRecords =
> inputContext.getCounters().findCounter(TaskCounter.APPROXIMATE_INPUT_RECORDS);
> this.shuffledInputsCounter =
> inputContext.getCounters().findCounter(TaskCounter.NUM_SHUFFLED_INPUTS);
> this.failedShufflesCounter =
> inputContext.getCounters().findCounter(TaskCounter.NUM_FAILED_SHUFFLE_INPUTS);
> this.bytesShuffledCounter =
> inputContext.getCounters().findCounter(TaskCounter.SHUFFLE_BYTES);
> this.decompressedDataSizeCounter =
> inputContext.getCounters().findCounter(TaskCounter.SHUFFLE_BYTES_DECOMPRESSED);
> this.bytesShuffledToDiskCounter =
> inputContext.getCounters().findCounter(TaskCounter.SHUFFLE_BYTES_TO_DISK);
> this.bytesShuffledToMemCounter =
> inputContext.getCounters().findCounter(TaskCounter.SHUFFLE_BYTES_TO_MEM);
> this.bytesShuffledDirectDiskCounter =
> inputContext.getCounters().findCounter(TaskCounter.SHUFFLE_BYTES_DISK_DIRECT);
> {code}
> we should collect then into a separate class like "ShuffleCounters"
> initialized by an inputContext and increment them through an interface
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
