Qifan Chen has uploaded a new patch set (#8). (
http://gerrit.cloudera.org:8080/16474 )
Change subject: WIP IMPALA-10178 Run-time profile shall report skews
......................................................................
WIP IMPALA-10178 Run-time profile shall report skews
This fix addresses the current limitation in runtime profile that
skews existing in certain operators such as the rows read counter
(RowsRead) in the scan operators are not reported. A skew condition
exists when the number of rows processed at each operator instance
is not about the same and can be detected through standard deviation
(stddev). A high stddev (say > 5) usually implies the existence of
skew.
With the fix and in the average fragment profile, such skew is
reported as follows:
1. In the skew summary section which lists the names of the
operators with skews;
2. In each corresponding operator, the name of the counters
and the corresponding stddev values.
Example of the skew reported for a hdfs scan.
Averaged Fragment F00:(Total: 1s075ms, non-child: 26.919ms, ...
...
num instances: 3
skew found at: HDFS_SCAN_NODE (id=0)
HDFS_SCAN_NODE (id=0):(Total: 1s032ms, non-child: 1s032ms, ...
skew details: RowsRead (stddev=62578.853590)
...
TODO:
1. Check out joins;
2. Add some unit tests;
3. Run core tests.
Change-Id: I91041f2856eef8293ea78f1721f97469062589a1
---
M be/src/runtime/coordinator-backend-state.cc
M be/src/util/CMakeLists.txt
A be/src/util/runtime-profile-counters.cc
M be/src/util/runtime-profile-counters.h
M be/src/util/runtime-profile.cc
M be/src/util/runtime-profile.h
M be/src/util/stat-util.h
7 files changed, 125 insertions(+), 8 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/74/16474/8
--
To view, visit http://gerrit.cloudera.org:8080/16474
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I91041f2856eef8293ea78f1721f97469062589a1
Gerrit-Change-Number: 16474
Gerrit-PatchSet: 8
Gerrit-Owner: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Sahil Takiar <stak...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <tarmstr...@cloudera.com>
