[jira] [Commented] (HIVE-3959) Update Partition Statistics in Metastore Layer
[ https://issues.apache.org/jira/browse/HIVE-3959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13811189#comment-13811189 ] Hive QA commented on HIVE-3959: --- {color:red}Overall{color}: -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12611523/HIVE-3959.6.patch {color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 4547 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucketmapjoin7 org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_parallel_orderby org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_smb_mapjoin_8 {noformat} Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/105/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/105/console Messages: {noformat} Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests failed with: TestsFailedException: 3 tests failed {noformat} This message is automatically generated. > Update Partition Statistics in Metastore Layer > -- > > Key: HIVE-3959 > URL: https://issues.apache.org/jira/browse/HIVE-3959 > Project: Hive > Issue Type: Improvement > Components: Metastore, Statistics >Reporter: Bhushan Mandhani >Assignee: Ashutosh Chauhan >Priority: Minor > Attachments: HIVE-3959.1.patch, HIVE-3959.2.patch, HIVE-3959.3.patch, > HIVE-3959.3.patch, HIVE-3959.4.patch, HIVE-3959.4.patch, HIVE-3959.5.patch, > HIVE-3959.6.patch, HIVE-3959.patch.1, HIVE-3959.patch.11.txt, > HIVE-3959.patch.12.txt, HIVE-3959.patch.2 > > > When partitions are created using queries ("insert overwrite" and "insert > into") then the StatsTask updates all stats. However, when partitions are > added directly through metadata-only partitions (either CLI or direct calls > to Thrift Metastore) no stats are populated even if hive.stats.reliable is > set to true. This puts us in a situation where we can't decide if stats are > truly reliable or not. > We propose that the "fast stats" (numFiles and totalSize) which don't require > a scan of the data should always be populated and be completely reliable. For > now we are still excluding rowCount and rawDataSize because that will make > these operations very expensive. Currently they are quick metadata-only ops. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HIVE-3959) Update Partition Statistics in Metastore Layer
[ https://issues.apache.org/jira/browse/HIVE-3959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13811172#comment-13811172 ] Hive QA commented on HIVE-3959: --- {color:red}Overall{color}: -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12611523/HIVE-3959.6.patch {color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 4547 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucketmapjoin7 org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_parallel_orderby {noformat} Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/104/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/104/console Messages: {noformat} Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests failed with: TestsFailedException: 2 tests failed {noformat} This message is automatically generated. > Update Partition Statistics in Metastore Layer > -- > > Key: HIVE-3959 > URL: https://issues.apache.org/jira/browse/HIVE-3959 > Project: Hive > Issue Type: Improvement > Components: Metastore, Statistics >Reporter: Bhushan Mandhani >Assignee: Ashutosh Chauhan >Priority: Minor > Attachments: HIVE-3959.1.patch, HIVE-3959.2.patch, HIVE-3959.3.patch, > HIVE-3959.3.patch, HIVE-3959.4.patch, HIVE-3959.4.patch, HIVE-3959.5.patch, > HIVE-3959.6.patch, HIVE-3959.patch.1, HIVE-3959.patch.11.txt, > HIVE-3959.patch.12.txt, HIVE-3959.patch.2 > > > When partitions are created using queries ("insert overwrite" and "insert > into") then the StatsTask updates all stats. However, when partitions are > added directly through metadata-only partitions (either CLI or direct calls > to Thrift Metastore) no stats are populated even if hive.stats.reliable is > set to true. This puts us in a situation where we can't decide if stats are > truly reliable or not. > We propose that the "fast stats" (numFiles and totalSize) which don't require > a scan of the data should always be populated and be completely reliable. For > now we are still excluding rowCount and rawDataSize because that will make > these operations very expensive. Currently they are quick metadata-only ops. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HIVE-3959) Update Partition Statistics in Metastore Layer
[ https://issues.apache.org/jira/browse/HIVE-3959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13811015#comment-13811015 ] Hive QA commented on HIVE-3959: --- {color:red}Overall{color}: -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12611523/HIVE-3959.6.patch {color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 4547 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucket_num_reducers org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucketmapjoin7 org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_parallel_orderby {noformat} Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/95/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/95/console Messages: {noformat} Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests failed with: TestsFailedException: 3 tests failed {noformat} This message is automatically generated. > Update Partition Statistics in Metastore Layer > -- > > Key: HIVE-3959 > URL: https://issues.apache.org/jira/browse/HIVE-3959 > Project: Hive > Issue Type: Improvement > Components: Metastore, Statistics >Reporter: Bhushan Mandhani >Assignee: Ashutosh Chauhan >Priority: Minor > Attachments: HIVE-3959.1.patch, HIVE-3959.2.patch, HIVE-3959.3.patch, > HIVE-3959.3.patch, HIVE-3959.4.patch, HIVE-3959.4.patch, HIVE-3959.5.patch, > HIVE-3959.6.patch, HIVE-3959.patch.1, HIVE-3959.patch.11.txt, > HIVE-3959.patch.12.txt, HIVE-3959.patch.2 > > > When partitions are created using queries ("insert overwrite" and "insert > into") then the StatsTask updates all stats. However, when partitions are > added directly through metadata-only partitions (either CLI or direct calls > to Thrift Metastore) no stats are populated even if hive.stats.reliable is > set to true. This puts us in a situation where we can't decide if stats are > truly reliable or not. > We propose that the "fast stats" (numFiles and totalSize) which don't require > a scan of the data should always be populated and be completely reliable. For > now we are still excluding rowCount and rawDataSize because that will make > these operations very expensive. Currently they are quick metadata-only ops. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HIVE-3959) Update Partition Statistics in Metastore Layer
[ https://issues.apache.org/jira/browse/HIVE-3959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13810886#comment-13810886 ] Thejas M Nair commented on HIVE-3959: - +1 > Update Partition Statistics in Metastore Layer > -- > > Key: HIVE-3959 > URL: https://issues.apache.org/jira/browse/HIVE-3959 > Project: Hive > Issue Type: Improvement > Components: Metastore, Statistics >Reporter: Bhushan Mandhani >Assignee: Ashutosh Chauhan >Priority: Minor > Attachments: HIVE-3959.1.patch, HIVE-3959.2.patch, HIVE-3959.3.patch, > HIVE-3959.3.patch, HIVE-3959.4.patch, HIVE-3959.4.patch, HIVE-3959.5.patch, > HIVE-3959.6.patch, HIVE-3959.patch.1, HIVE-3959.patch.11.txt, > HIVE-3959.patch.12.txt, HIVE-3959.patch.2 > > > When partitions are created using queries ("insert overwrite" and "insert > into") then the StatsTask updates all stats. However, when partitions are > added directly through metadata-only partitions (either CLI or direct calls > to Thrift Metastore) no stats are populated even if hive.stats.reliable is > set to true. This puts us in a situation where we can't decide if stats are > truly reliable or not. > We propose that the "fast stats" (numFiles and totalSize) which don't require > a scan of the data should always be populated and be completely reliable. For > now we are still excluding rowCount and rawDataSize because that will make > these operations very expensive. Currently they are quick metadata-only ops. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HIVE-3959) Update Partition Statistics in Metastore Layer
[ https://issues.apache.org/jira/browse/HIVE-3959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13810730#comment-13810730 ] Thejas M Nair commented on HIVE-3959: - Can you please rebase the patch (as HIVE-5610 changes are in) ? > Update Partition Statistics in Metastore Layer > -- > > Key: HIVE-3959 > URL: https://issues.apache.org/jira/browse/HIVE-3959 > Project: Hive > Issue Type: Improvement > Components: Metastore, Statistics >Reporter: Bhushan Mandhani >Assignee: Ashutosh Chauhan >Priority: Minor > Attachments: HIVE-3959.1.patch, HIVE-3959.2.patch, HIVE-3959.3.patch, > HIVE-3959.3.patch, HIVE-3959.4.patch, HIVE-3959.4.patch, HIVE-3959.patch.1, > HIVE-3959.patch.11.txt, HIVE-3959.patch.12.txt, HIVE-3959.patch.2 > > > When partitions are created using queries ("insert overwrite" and "insert > into") then the StatsTask updates all stats. However, when partitions are > added directly through metadata-only partitions (either CLI or direct calls > to Thrift Metastore) no stats are populated even if hive.stats.reliable is > set to true. This puts us in a situation where we can't decide if stats are > truly reliable or not. > We propose that the "fast stats" (numFiles and totalSize) which don't require > a scan of the data should always be populated and be completely reliable. For > now we are still excluding rowCount and rawDataSize because that will make > these operations very expensive. Currently they are quick metadata-only ops. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HIVE-3959) Update Partition Statistics in Metastore Layer
[ https://issues.apache.org/jira/browse/HIVE-3959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13809854#comment-13809854 ] Thejas M Nair commented on HIVE-3959: - updated review board with comments > Update Partition Statistics in Metastore Layer > -- > > Key: HIVE-3959 > URL: https://issues.apache.org/jira/browse/HIVE-3959 > Project: Hive > Issue Type: Improvement > Components: Metastore, Statistics >Reporter: Bhushan Mandhani >Assignee: Ashutosh Chauhan >Priority: Minor > Attachments: HIVE-3959.1.patch, HIVE-3959.2.patch, HIVE-3959.3.patch, > HIVE-3959.4.patch, HIVE-3959.patch.1, HIVE-3959.patch.11.txt, > HIVE-3959.patch.12.txt, HIVE-3959.patch.2 > > > When partitions are created using queries ("insert overwrite" and "insert > into") then the StatsTask updates all stats. However, when partitions are > added directly through metadata-only partitions (either CLI or direct calls > to Thrift Metastore) no stats are populated even if hive.stats.reliable is > set to true. This puts us in a situation where we can't decide if stats are > truly reliable or not. > We propose that the "fast stats" (numFiles and totalSize) which don't require > a scan of the data should always be populated and be completely reliable. For > now we are still excluding rowCount and rawDataSize because that will make > these operations very expensive. Currently they are quick metadata-only ops. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HIVE-3959) Update Partition Statistics in Metastore Layer
[ https://issues.apache.org/jira/browse/HIVE-3959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13805775#comment-13805775 ] Hive QA commented on HIVE-3959: --- {color:red}Overall{color}: -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12610142/HIVE-3959.4.patch {color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 4481 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_create_like_view org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_star org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_pcr org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_union_view org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_show_create_table_serde {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/1237/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/1237/console Messages: {noformat} Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests failed with: TestsFailedException: 5 tests failed {noformat} This message is automatically generated. > Update Partition Statistics in Metastore Layer > -- > > Key: HIVE-3959 > URL: https://issues.apache.org/jira/browse/HIVE-3959 > Project: Hive > Issue Type: Improvement > Components: Metastore, Statistics >Reporter: Bhushan Mandhani >Assignee: Ashutosh Chauhan >Priority: Minor > Attachments: HIVE-3959.1.patch, HIVE-3959.2.patch, HIVE-3959.3.patch, > HIVE-3959.4.patch, HIVE-3959.patch.1, HIVE-3959.patch.11.txt, > HIVE-3959.patch.12.txt, HIVE-3959.patch.2 > > > When partitions are created using queries ("insert overwrite" and "insert > into") then the StatsTask updates all stats. However, when partitions are > added directly through metadata-only partitions (either CLI or direct calls > to Thrift Metastore) no stats are populated even if hive.stats.reliable is > set to true. This puts us in a situation where we can't decide if stats are > truly reliable or not. > We propose that the "fast stats" (numFiles and totalSize) which don't require > a scan of the data should always be populated and be completely reliable. For > now we are still excluding rowCount and rawDataSize because that will make > these operations very expensive. Currently they are quick metadata-only ops. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HIVE-3959) Update Partition Statistics in Metastore Layer
[ https://issues.apache.org/jira/browse/HIVE-3959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13803456#comment-13803456 ] Ashutosh Chauhan commented on HIVE-3959: [~gangtimliu] Would you like to review the rebased patch? > Update Partition Statistics in Metastore Layer > -- > > Key: HIVE-3959 > URL: https://issues.apache.org/jira/browse/HIVE-3959 > Project: Hive > Issue Type: Improvement > Components: Metastore, Statistics >Reporter: Bhushan Mandhani >Assignee: Ashutosh Chauhan >Priority: Minor > Attachments: HIVE-3959.1.patch, HIVE-3959.patch.1, > HIVE-3959.patch.11.txt, HIVE-3959.patch.12.txt, HIVE-3959.patch.2 > > > When partitions are created using queries ("insert overwrite" and "insert > into") then the StatsTask updates all stats. However, when partitions are > added directly through metadata-only partitions (either CLI or direct calls > to Thrift Metastore) no stats are populated even if hive.stats.reliable is > set to true. This puts us in a situation where we can't decide if stats are > truly reliable or not. > We propose that the "fast stats" (numFiles and totalSize) which don't require > a scan of the data should always be populated and be completely reliable. For > now we are still excluding rowCount and rawDataSize because that will make > these operations very expensive. Currently they are quick metadata-only ops. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HIVE-3959) Update Partition Statistics in Metastore Layer
[ https://issues.apache.org/jira/browse/HIVE-3959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13781438#comment-13781438 ] Gang Tim Liu commented on HIVE-3959: Yes,assign it to dilip > Update Partition Statistics in Metastore Layer > -- > > Key: HIVE-3959 > URL: https://issues.apache.org/jira/browse/HIVE-3959 > Project: Hive > Issue Type: Improvement > Components: Metastore, Statistics >Reporter: Bhushan Mandhani >Assignee: Dilip Joseph >Priority: Minor > Attachments: HIVE-3959.patch.1, HIVE-3959.patch.11.txt, > HIVE-3959.patch.12.txt, HIVE-3959.patch.2 > > > When partitions are created using queries ("insert overwrite" and "insert > into") then the StatsTask updates all stats. However, when partitions are > added directly through metadata-only partitions (either CLI or direct calls > to Thrift Metastore) no stats are populated even if hive.stats.reliable is > set to true. This puts us in a situation where we can't decide if stats are > truly reliable or not. > We propose that the "fast stats" (numFiles and totalSize) which don't require > a scan of the data should always be populated and be completely reliable. For > now we are still excluding rowCount and rawDataSize because that will make > these operations very expensive. Currently they are quick metadata-only ops. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HIVE-3959) Update Partition Statistics in Metastore Layer
[ https://issues.apache.org/jira/browse/HIVE-3959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13780918#comment-13780918 ] Ashutosh Chauhan commented on HIVE-3959: This is useful work. [~bmadhvani] / [~gangtimliu] Do you guys want to refresh this patch? > Update Partition Statistics in Metastore Layer > -- > > Key: HIVE-3959 > URL: https://issues.apache.org/jira/browse/HIVE-3959 > Project: Hive > Issue Type: Improvement > Components: Metastore, Statistics >Reporter: Bhushan Mandhani >Assignee: Gang Tim Liu >Priority: Minor > Attachments: HIVE-3959.patch.1, HIVE-3959.patch.11.txt, > HIVE-3959.patch.12.txt, HIVE-3959.patch.2 > > > When partitions are created using queries ("insert overwrite" and "insert > into") then the StatsTask updates all stats. However, when partitions are > added directly through metadata-only partitions (either CLI or direct calls > to Thrift Metastore) no stats are populated even if hive.stats.reliable is > set to true. This puts us in a situation where we can't decide if stats are > truly reliable or not. > We propose that the "fast stats" (numFiles and totalSize) which don't require > a scan of the data should always be populated and be completely reliable. For > now we are still excluding rowCount and rawDataSize because that will make > these operations very expensive. Currently they are quick metadata-only ops. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3959) Update Partition Statistics in Metastore Layer
[ https://issues.apache.org/jira/browse/HIVE-3959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13620364#comment-13620364 ] Gang Tim Liu commented on HIVE-3959: rebase https://reviews.facebook.net/D9885 > Update Partition Statistics in Metastore Layer > -- > > Key: HIVE-3959 > URL: https://issues.apache.org/jira/browse/HIVE-3959 > Project: Hive > Issue Type: Improvement > Components: Metastore, Statistics >Reporter: Bhushan Mandhani >Assignee: Gang Tim Liu >Priority: Minor > > When partitions are created using queries ("insert overwrite" and "insert > into") then the StatsTask updates all stats. However, when partitions are > added directly through metadata-only partitions (either CLI or direct calls > to Thrift Metastore) no stats are populated even if hive.stats.reliable is > set to true. This puts us in a situation where we can't decide if stats are > truly reliable or not. > We propose that the "fast stats" (numFiles and totalSize) which don't require > a scan of the data should always be populated and be completely reliable. For > now we are still excluding rowCount and rawDataSize because that will make > these operations very expensive. Currently they are quick metadata-only ops. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3959) Update Partition Statistics in Metastore Layer
[ https://issues.apache.org/jira/browse/HIVE-3959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13570910#comment-13570910 ] Bhushan Mandhani commented on HIVE-3959: Diff out at https://reviews.facebook.net/D8271 Still need some minor updates before I can submit the patch. > Update Partition Statistics in Metastore Layer > -- > > Key: HIVE-3959 > URL: https://issues.apache.org/jira/browse/HIVE-3959 > Project: Hive > Issue Type: Improvement > Components: Metastore, Statistics >Reporter: Bhushan Mandhani >Assignee: Bhushan Mandhani >Priority: Minor > > When partitions are created using queries ("insert overwrite" and "insert > into") then the StatsTask updates all stats. However, when partitions are > added directly through metadata-only partitions (either CLI or direct calls > to Thrift Metastore) no stats are populated even if hive.stats.reliable is > set to true. This puts us in a situation where we can't decide if stats are > truly reliable or not. > We propose that the "fast stats" (numFiles and totalSize) which don't require > a scan of the data should always be populated and be completely reliable. For > now we are still excluding rowCount and rawDataSize because that will make > these operations very expensive. Currently they are quick metadata-only ops. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira