Csaba Ringhofer has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/15835 )
Change subject: IMPALA-9699: Skip '-1' values when aggregating num_null statistics ...................................................................... IMPALA-9699: Skip '-1' values when aggregating num_null statistics This change partially reverts IMPALA-8566 to make Impala backward compatible with the old incremental partition stats. IMPALA-7659 added collecting the number of null value statistics and IMPALA-8566 changed the initial value of the incremental partition statistics from '-1' to '0', because with '-1' the estimates were off by 1. The old statistics in a new release can make the table metadata inaccessible when the column stats are recomputed from the incremental partition stats, which can be triggered by a partition level 'COMPUTE INCREMENTAL STATS'. In this case the old '-1' values can be aggregated to a '<-1' 'num_nulls' value that later can fail a Precondition check during table load. The new behavior ensures that if any incremental partition stat has a value of '-1' for 'num_nulls', the aggregated stats will be '-1', regardless of whether or not other partitions have valid values for 'num_nulls'. This will prevent the planner from utilizing incomplete statistics and the users will be notified about the missing statistics with the general warning in the profile: 'The following tables are missing relevant table and/or column statistics.' Testing: - Added unit test to verify the accepted values and aggregation result Change-Id: I3fdf48a6c88378145078e068e12ade48c460f956 Reviewed-on: http://gerrit.cloudera.org:8080/15835 Reviewed-by: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Tested-by: Impala Public Jenkins <impala-public-jenk...@cloudera.com> --- M be/src/exec/incr-stats-util-test.cc M be/src/exec/incr-stats-util.cc M be/src/exec/incr-stats-util.h 3 files changed, 124 insertions(+), 70 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/15835 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I3fdf48a6c88378145078e068e12ade48c460f956 Gerrit-Change-Number: 15835 Gerrit-PatchSet: 8 Gerrit-Owner: Tamas Mate <tm...@cloudera.com> Gerrit-Reviewer: Csaba Ringhofer <csringho...@cloudera.com> Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Gerrit-Reviewer: Sahil Takiar <stak...@cloudera.com> Gerrit-Reviewer: Tamas Mate <tm...@cloudera.com>
