Sahil Takiar has posted comments on this change. ( http://gerrit.cloudera.org:8080/16098 )
Change subject: IMPALA-9744: Treat corrupt table stats as missing to avoid bad plans ...................................................................... Patch Set 6: (5 comments) adding Tim to the review as well http://gerrit.cloudera.org:8080/#/c/16098/6/fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java File fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java: http://gerrit.cloudera.org:8080/#/c/16098/6/fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java@1218 PS6, Line 1218: if (numRows < -1 || (numRows == 0 && tbl_.getTotalHdfsBytes() > 0)) { : hasCorruptTableStats_ = true; : } : return numRows; what about for unpartitioned tables? it looks like if the table stats are corrupted, and numRows == 0, we still return numRows = 0 from this method. would be good to add a test case for unpartitioned tables as well. http://gerrit.cloudera.org:8080/#/c/16098/6/testdata/data/alltypes_parquet_year2009_month01.parquet File testdata/data/alltypes_parquet_year2009_month01.parquet: http://gerrit.cloudera.org:8080/#/c/16098/6/testdata/data/alltypes_parquet_year2009_month01.parquet@1 PS6, Line 1: PAR1°ºLì Ø ô×a b c d e f g h i j k l m n o p q r s N O P Q R S T U V W X Y Z [ \ ] ^ _ ` t u v w x y z { | } ~ ! " # $ % & do you need to create a new file for this test? shouldn't one of the existing files work? http://gerrit.cloudera.org:8080/#/c/16098/6/tests/metadata/test_compute_stats.py File tests/metadata/test_compute_stats.py: http://gerrit.cloudera.org:8080/#/c/16098/6/tests/metadata/test_compute_stats.py@197 PS6, Line 197: DROP TABLE {0}.{1} PURGE; you shouldn't need this since the table is created in a unique database http://gerrit.cloudera.org:8080/#/c/16098/6/tests/metadata/test_compute_stats.py@198 PS6, Line 198: set hive.stats.autogather=true; do you need this? http://gerrit.cloudera.org:8080/#/c/16098/6/tests/metadata/test_compute_stats.py@213 PS6, Line 213: TBLPROPERTIES ("transactional"="true", "transactional_properties"="insert_only"); do you need this? -- To view, visit http://gerrit.cloudera.org:8080/16098 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9f4c64616ff7c0b6d5a48f2b5331325feeff3576 Gerrit-Change-Number: 16098 Gerrit-PatchSet: 6 Gerrit-Owner: Qifan Chen <qc...@cloudera.com> Gerrit-Reviewer: Aman Sinha <amsi...@cloudera.com> Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com> Gerrit-Reviewer: Sahil Takiar <stak...@cloudera.com> Gerrit-Comment-Date: Mon, 22 Jun 2020 21:26:00 +0000 Gerrit-HasComments: Yes
