[jira] [Resolved] (IMPALA-6870) SummaryStatsCounter should be included in averaged profile
[ https://issues.apache.org/jira/browse/IMPALA-6870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tim Armstrong resolved IMPALA-6870. --- Fix Version/s: Impala 4.0 Resolution: Fixed commit 9429bd779de986d3e61858bef7e258bd73a2cacd Author: Tim Armstrong Date: Sun May 17 16:37:46 2020 -0700 IMPALA-9382: part 2/3: aggregate profiles sent to coordinator This reworks the status reporting so that serialized AggregatedRuntimeProfile objects are sent from executors to coordinators. These profiles are substantially denser and faster to process for higher mt_dop values. The aggregation is also done in a single step, merging the aggregated thrift profile from the executor directly into the final aggregated profile, instead of converting it to an unaggregated profile first. The changes required were: * A new Update() method for AggregatedRuntimeProfile that updates the profile from a serialised AggregateRuntimeProfile for a subset of the instances. The code is generalized from the existing InitFromThrift() code path. * Per-fragment reports included in the status report protobuf when --gen_experimental_profile=true. * Logic on the coordinator that either consumes serialized AggregatedRuntimeProfile per fragment, when --gen_experimental_profile=true, or consumes a serialized RuntimeProfile per finstance otherwise. This also adds support for event sequences and time series in the aggregated profile, so the amount of information in the aggregated profile is now on par with the basic profile. We also finish off support for JSON profile. The JSON profile is more stripped down because we do not need to round-trip profiles via JSON and it is a much less dense profile representation. Part 3 will clean up and improve the display of the profile. Testing: * Add sanity tests for aggregated runtime profile. * Add unit tests to exercise aggregation of the various counter types * Ran core tests. Change-Id: Ic680cbfe94c939c2a8fad9d0943034ed058c6bca Reviewed-on: http://gerrit.cloudera.org:8080/16057 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins > SummaryStatsCounter should be included in averaged profile > -- > > Key: IMPALA-6870 > URL: https://issues.apache.org/jira/browse/IMPALA-6870 > Project: IMPALA > Issue Type: Improvement > Components: Backend >Affects Versions: Impala 2.12.0 >Reporter: Tim Armstrong >Assignee: Tim Armstrong >Priority: Major > Labels: observability > Fix For: Impala 4.0 > > > Summary stats like FooterProcessingTime don't show up in the averaged > fragment. We should be able to merge these counters to produce overall > statistics: -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-6870) SummaryStatsCounter should be included in averaged profile
[ https://issues.apache.org/jira/browse/IMPALA-6870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tim Armstrong resolved IMPALA-6870. --- Fix Version/s: Impala 4.0 Resolution: Fixed commit 9429bd779de986d3e61858bef7e258bd73a2cacd Author: Tim Armstrong Date: Sun May 17 16:37:46 2020 -0700 IMPALA-9382: part 2/3: aggregate profiles sent to coordinator This reworks the status reporting so that serialized AggregatedRuntimeProfile objects are sent from executors to coordinators. These profiles are substantially denser and faster to process for higher mt_dop values. The aggregation is also done in a single step, merging the aggregated thrift profile from the executor directly into the final aggregated profile, instead of converting it to an unaggregated profile first. The changes required were: * A new Update() method for AggregatedRuntimeProfile that updates the profile from a serialised AggregateRuntimeProfile for a subset of the instances. The code is generalized from the existing InitFromThrift() code path. * Per-fragment reports included in the status report protobuf when --gen_experimental_profile=true. * Logic on the coordinator that either consumes serialized AggregatedRuntimeProfile per fragment, when --gen_experimental_profile=true, or consumes a serialized RuntimeProfile per finstance otherwise. This also adds support for event sequences and time series in the aggregated profile, so the amount of information in the aggregated profile is now on par with the basic profile. We also finish off support for JSON profile. The JSON profile is more stripped down because we do not need to round-trip profiles via JSON and it is a much less dense profile representation. Part 3 will clean up and improve the display of the profile. Testing: * Add sanity tests for aggregated runtime profile. * Add unit tests to exercise aggregation of the various counter types * Ran core tests. Change-Id: Ic680cbfe94c939c2a8fad9d0943034ed058c6bca Reviewed-on: http://gerrit.cloudera.org:8080/16057 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins > SummaryStatsCounter should be included in averaged profile > -- > > Key: IMPALA-6870 > URL: https://issues.apache.org/jira/browse/IMPALA-6870 > Project: IMPALA > Issue Type: Improvement > Components: Backend >Affects Versions: Impala 2.12.0 >Reporter: Tim Armstrong >Assignee: Tim Armstrong >Priority: Major > Labels: observability > Fix For: Impala 4.0 > > > Summary stats like FooterProcessingTime don't show up in the averaged > fragment. We should be able to merge these counters to produce overall > statistics: -- This message was sent by Atlassian Jira (v8.3.4#803005)
