[jira] [Commented] (HIVE-21225) ACID: getAcidState() should cache a recursive dir listing locally
[ https://issues.apache.org/jira/browse/HIVE-21225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17049879#comment-17049879 ] Aditya Shah commented on HIVE-21225: [~gopalv] I further noticed Hive-22001. It seems we are swallowing the fnf exception in the case where we do the listing to populate the cache. So, we could have always done this in case of multiple listings as well since the snapshot will be consistent once the valid Txn Write Ids list is made. And as I already pointed out the performance loss due to this, should we have avoided this? > ACID: getAcidState() should cache a recursive dir listing locally > - > > Key: HIVE-21225 > URL: https://issues.apache.org/jira/browse/HIVE-21225 > Project: Hive > Issue Type: Improvement > Components: Transactions >Reporter: Gopal Vijayaraghavan >Assignee: Vaibhav Gumashta >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-21225.1.patch, HIVE-21225.10.patch, > HIVE-21225.11.patch, HIVE-21225.12.patch, HIVE-21225.13.patch, > HIVE-21225.14.patch, HIVE-21225.15.patch, HIVE-21225.15.patch, > HIVE-21225.16.patch, HIVE-21225.17.patch, HIVE-21225.2.patch, > HIVE-21225.3.patch, HIVE-21225.4.patch, HIVE-21225.4.patch, > HIVE-21225.5.patch, HIVE-21225.6.patch, HIVE-21225.7.patch, > HIVE-21225.7.patch, HIVE-21225.8.patch, HIVE-21225.9.patch, async-pid-44-2.svg > > > Currently getAcidState() makes 3 calls into the FS api which could be > answered by making a single recursive listDir call and reusing the same data > to check for isRawFormat() and isValidBase(). > All delta operations for a single partition can go against a single listed > directory snapshot instead of interacting with the NameNode or ObjectStore > within the inner loop. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-21225) ACID: getAcidState() should cache a recursive dir listing locally
[ https://issues.apache.org/jira/browse/HIVE-21225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17038066#comment-17038066 ] Gopal Vijayaraghavan commented on HIVE-21225: - bq. I have reopened the issue as I couldn't attach new files Since this is a task ticket titled "cache file listing" and the fix is to do exactly that, this shouldn't be reopened. If we have a performance issue from this patch, that should be a different issue - it is too late to layer on an addendum to this. Also bq. I'm not using S3guard which might have resulted in a significant amount of time for S3 calls. ACID doesn't work without S3guard on S3 - compactions will break the world for you. > ACID: getAcidState() should cache a recursive dir listing locally > - > > Key: HIVE-21225 > URL: https://issues.apache.org/jira/browse/HIVE-21225 > Project: Hive > Issue Type: Improvement > Components: Transactions >Reporter: Gopal Vijayaraghavan >Assignee: Vaibhav Gumashta >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-21225.1.patch, HIVE-21225.10.patch, > HIVE-21225.11.patch, HIVE-21225.12.patch, HIVE-21225.13.patch, > HIVE-21225.14.patch, HIVE-21225.15.patch, HIVE-21225.15.patch, > HIVE-21225.16.patch, HIVE-21225.17.patch, HIVE-21225.2.patch, > HIVE-21225.3.patch, HIVE-21225.4.patch, HIVE-21225.4.patch, > HIVE-21225.5.patch, HIVE-21225.6.patch, HIVE-21225.7.patch, > HIVE-21225.7.patch, HIVE-21225.8.patch, HIVE-21225.9.patch, async-pid-44-2.svg > > > Currently getAcidState() makes 3 calls into the FS api which could be > answered by making a single recursive listDir call and reusing the same data > to check for isRawFormat() and isValidBase(). > All delta operations for a single partition can go against a single listed > directory snapshot instead of interacting with the NameNode or ObjectStore > within the inner loop. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-21225) ACID: getAcidState() should cache a recursive dir listing locally
[ https://issues.apache.org/jira/browse/HIVE-21225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17036891#comment-17036891 ] Aditya Shah commented on HIVE-21225: Still couldn't attach files. Image hosted: 1. [https://imgur.com/tpgP37g] 2. [https://imgur.com/Pradd7e] > ACID: getAcidState() should cache a recursive dir listing locally > - > > Key: HIVE-21225 > URL: https://issues.apache.org/jira/browse/HIVE-21225 > Project: Hive > Issue Type: Improvement > Components: Transactions >Reporter: Gopal Vijayaraghavan >Assignee: Vaibhav Gumashta >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-21225.1.patch, HIVE-21225.10.patch, > HIVE-21225.11.patch, HIVE-21225.12.patch, HIVE-21225.13.patch, > HIVE-21225.14.patch, HIVE-21225.15.patch, HIVE-21225.15.patch, > HIVE-21225.16.patch, HIVE-21225.17.patch, HIVE-21225.2.patch, > HIVE-21225.3.patch, HIVE-21225.4.patch, HIVE-21225.4.patch, > HIVE-21225.5.patch, HIVE-21225.6.patch, HIVE-21225.7.patch, > HIVE-21225.7.patch, HIVE-21225.8.patch, HIVE-21225.9.patch, async-pid-44-2.svg > > > Currently getAcidState() makes 3 calls into the FS api which could be > answered by making a single recursive listDir call and reusing the same data > to check for isRawFormat() and isValidBase(). > All delta operations for a single partition can go against a single listed > directory snapshot instead of interacting with the NameNode or ObjectStore > within the inner loop. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-21225) ACID: getAcidState() should cache a recursive dir listing locally
[ https://issues.apache.org/jira/browse/HIVE-21225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17036870#comment-17036870 ] Aditya Shah commented on HIVE-21225: [~gopalv] Thanks for your reply. I'm not using S3guard which might have resulted in a significant amount of time for S3 calls. But, as far as the numbers are concerned I did attach a profiler for the same and have attached the flame graphs for cases (row 2, col 2) and (row 2, col 3) from the above comment: As you can see the listing takes around 96% of the time. Even the amazonHttpClient calls were 29k vs 520k. My concerns/doubts were the following two: 1) As you pointed out the correctness issue. Any plan to backport to Hive3.1.1. 2) Should we have additional optimization for listing in place (something similar to getInputPath's threadpool) > ACID: getAcidState() should cache a recursive dir listing locally > - > > Key: HIVE-21225 > URL: https://issues.apache.org/jira/browse/HIVE-21225 > Project: Hive > Issue Type: Improvement > Components: Transactions >Reporter: Gopal Vijayaraghavan >Assignee: Vaibhav Gumashta >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-21225.1.patch, HIVE-21225.10.patch, > HIVE-21225.11.patch, HIVE-21225.12.patch, HIVE-21225.13.patch, > HIVE-21225.14.patch, HIVE-21225.15.patch, HIVE-21225.15.patch, > HIVE-21225.16.patch, HIVE-21225.17.patch, HIVE-21225.2.patch, > HIVE-21225.3.patch, HIVE-21225.4.patch, HIVE-21225.4.patch, > HIVE-21225.5.patch, HIVE-21225.6.patch, HIVE-21225.7.patch, > HIVE-21225.7.patch, HIVE-21225.8.patch, HIVE-21225.9.patch, async-pid-44-2.svg > > > Currently getAcidState() makes 3 calls into the FS api which could be > answered by making a single recursive listDir call and reusing the same data > to check for isRawFormat() and isValidBase(). > All delta operations for a single partition can go against a single listed > directory snapshot instead of interacting with the NameNode or ObjectStore > within the inner loop. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-21225) ACID: getAcidState() should cache a recursive dir listing locally
[ https://issues.apache.org/jira/browse/HIVE-21225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17036574#comment-17036574 ] Gopal Vijayaraghavan commented on HIVE-21225: - bq. The overhead compared to the benefit where we use the statuses once per delta directory (After HIVE-21177) to determine RawFormat seems very high. This was also a correctness fix even for HDFS, the caching is necessary for the listing to be exactly once as the older compacted dirs can be deleted while a query is halfway through this loop (& there's a file-not-found [~aditya-shah]: Since your numbers seem way off from mine, I suspect you might be hitting another set of issues here & need a svg from an async-profiler attached to the AM. My builds do have HADOOP-16801, which makes a significant difference here. > ACID: getAcidState() should cache a recursive dir listing locally > - > > Key: HIVE-21225 > URL: https://issues.apache.org/jira/browse/HIVE-21225 > Project: Hive > Issue Type: Improvement > Components: Transactions >Reporter: Gopal Vijayaraghavan >Assignee: Vaibhav Gumashta >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-21225.1.patch, HIVE-21225.10.patch, > HIVE-21225.11.patch, HIVE-21225.12.patch, HIVE-21225.13.patch, > HIVE-21225.14.patch, HIVE-21225.15.patch, HIVE-21225.15.patch, > HIVE-21225.16.patch, HIVE-21225.17.patch, HIVE-21225.2.patch, > HIVE-21225.3.patch, HIVE-21225.4.patch, HIVE-21225.4.patch, > HIVE-21225.5.patch, HIVE-21225.6.patch, HIVE-21225.7.patch, > HIVE-21225.7.patch, HIVE-21225.8.patch, HIVE-21225.9.patch, async-pid-44-2.svg > > > Currently getAcidState() makes 3 calls into the FS api which could be > answered by making a single recursive listDir call and reusing the same data > to check for isRawFormat() and isValidBase(). > All delta operations for a single partition can go against a single listed > directory snapshot instead of interacting with the NameNode or ObjectStore > within the inner loop. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-21225) ACID: getAcidState() should cache a recursive dir listing locally
[ https://issues.apache.org/jira/browse/HIVE-21225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17036142#comment-17036142 ] Aditya Shah commented on HIVE-21225: [~vgumashta] I had some doubts similar to what [~vgarg] raised before. Introducing caching which stores the whole status object of each directory is quite expensive for S3. Since we only did listStatus prior to this in getAcidState it was very fast. The overhead compared to the benefit where we use the statuses once per delta directory (After HIVE-21177) to determine RawFormat seems very high. I evaluated 2 examples of tables. One (non-partitioned) with around 900 files in each delta directory and 3 deltas, and other (100 partitions, 40 deltas and 45 files each). The matrix for time for split computation in each was as follows: ||Table||Hive version 3.1.1||With HIVE-21177||With HIVE-21225, HIVE-22537, and HIVE-21177|| |3 deltas, 900 files|798s|1s|367s| |100 partitions,40 deltas, 45 files|12952s|70s|942s| Am I missing something here? > ACID: getAcidState() should cache a recursive dir listing locally > - > > Key: HIVE-21225 > URL: https://issues.apache.org/jira/browse/HIVE-21225 > Project: Hive > Issue Type: Improvement > Components: Transactions >Reporter: Gopal Vijayaraghavan >Assignee: Vaibhav Gumashta >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-21225.1.patch, HIVE-21225.10.patch, > HIVE-21225.11.patch, HIVE-21225.12.patch, HIVE-21225.13.patch, > HIVE-21225.14.patch, HIVE-21225.15.patch, HIVE-21225.15.patch, > HIVE-21225.16.patch, HIVE-21225.17.patch, HIVE-21225.2.patch, > HIVE-21225.3.patch, HIVE-21225.4.patch, HIVE-21225.4.patch, > HIVE-21225.5.patch, HIVE-21225.6.patch, HIVE-21225.7.patch, > HIVE-21225.7.patch, HIVE-21225.8.patch, HIVE-21225.9.patch, async-pid-44-2.svg > > > Currently getAcidState() makes 3 calls into the FS api which could be > answered by making a single recursive listDir call and reusing the same data > to check for isRawFormat() and isValidBase(). > All delta operations for a single partition can go against a single listed > directory snapshot instead of interacting with the NameNode or ObjectStore > within the inner loop. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-21225) ACID: getAcidState() should cache a recursive dir listing locally
[ https://issues.apache.org/jira/browse/HIVE-21225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16891379#comment-16891379 ] Vineet Garg commented on HIVE-21225: +1 pending minor review comments > ACID: getAcidState() should cache a recursive dir listing locally > - > > Key: HIVE-21225 > URL: https://issues.apache.org/jira/browse/HIVE-21225 > Project: Hive > Issue Type: Improvement > Components: Transactions >Reporter: Gopal V >Assignee: Vaibhav Gumashta >Priority: Major > Attachments: HIVE-21225.1.patch, HIVE-21225.10.patch, > HIVE-21225.11.patch, HIVE-21225.12.patch, HIVE-21225.13.patch, > HIVE-21225.14.patch, HIVE-21225.15.patch, HIVE-21225.15.patch, > HIVE-21225.16.patch, HIVE-21225.2.patch, HIVE-21225.3.patch, > HIVE-21225.4.patch, HIVE-21225.4.patch, HIVE-21225.5.patch, > HIVE-21225.6.patch, HIVE-21225.7.patch, HIVE-21225.7.patch, > HIVE-21225.8.patch, HIVE-21225.9.patch, async-pid-44-2.svg > > > Currently getAcidState() makes 3 calls into the FS api which could be > answered by making a single recursive listDir call and reusing the same data > to check for isRawFormat() and isValidBase(). > All delta operations for a single partition can go against a single listed > directory snapshot instead of interacting with the NameNode or ObjectStore > within the inner loop. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (HIVE-21225) ACID: getAcidState() should cache a recursive dir listing locally
[ https://issues.apache.org/jira/browse/HIVE-21225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16889613#comment-16889613 ] Hive QA commented on HIVE-21225: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12975325/HIVE-21225.16.patch {color:green}SUCCESS:{color} +1 due to 6 test(s) being added or modified. {color:green}SUCCESS:{color} +1 due to 16682 tests passed Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/18121/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18121/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18121/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase {noformat} This message is automatically generated. ATTACHMENT ID: 12975325 - PreCommit-HIVE-Build > ACID: getAcidState() should cache a recursive dir listing locally > - > > Key: HIVE-21225 > URL: https://issues.apache.org/jira/browse/HIVE-21225 > Project: Hive > Issue Type: Improvement > Components: Transactions >Reporter: Gopal V >Assignee: Vaibhav Gumashta >Priority: Major > Attachments: HIVE-21225.1.patch, HIVE-21225.10.patch, > HIVE-21225.11.patch, HIVE-21225.12.patch, HIVE-21225.13.patch, > HIVE-21225.14.patch, HIVE-21225.15.patch, HIVE-21225.15.patch, > HIVE-21225.16.patch, HIVE-21225.2.patch, HIVE-21225.3.patch, > HIVE-21225.4.patch, HIVE-21225.4.patch, HIVE-21225.5.patch, > HIVE-21225.6.patch, HIVE-21225.7.patch, HIVE-21225.7.patch, > HIVE-21225.8.patch, HIVE-21225.9.patch, async-pid-44-2.svg > > > Currently getAcidState() makes 3 calls into the FS api which could be > answered by making a single recursive listDir call and reusing the same data > to check for isRawFormat() and isValidBase(). > All delta operations for a single partition can go against a single listed > directory snapshot instead of interacting with the NameNode or ObjectStore > within the inner loop. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (HIVE-21225) ACID: getAcidState() should cache a recursive dir listing locally
[ https://issues.apache.org/jira/browse/HIVE-21225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16889608#comment-16889608 ] Hive QA commented on HIVE-21225: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 45s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m 57s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 29s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 35s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m 8s{color} | {color:blue} ql in master has 2250 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 30s{color} | {color:blue} hcatalog/streaming in master has 11 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 28s{color} | {color:blue} streaming in master has 2 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 38s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 48s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 26s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 53s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 27s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 27s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 50s{color} | {color:red} ql: The patch generated 47 new + 1414 unchanged - 11 fixed = 1461 total (was 1425) {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 13s{color} | {color:red} hcatalog/streaming: The patch generated 1 new + 229 unchanged - 0 fixed = 230 total (was 229) {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 15s{color} | {color:red} streaming: The patch generated 3 new + 454 unchanged - 0 fixed = 457 total (was 454) {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s{color} | {color:red} The patch has 44 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m 14s{color} | {color:red} ql generated 1 new + 2250 unchanged - 0 fixed = 2251 total (was 2250) {color} | | {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 58s{color} | {color:red} ql generated 2 new + 98 unchanged - 2 fixed = 100 total (was 100) {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 13s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 36m 59s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | FindBugs | module:ql | | | Field only ever set to null:null: org.apache.hadoop.hive.ql.io.AcidUtils$HdfsDirSnapshotImpl.fileId In AcidUtils.java | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-18121/dev-support/hive-personality.sh | | git revision | master / ac78f79 | | Default Java | 1.8.0_111 | | findbugs | v3.0.0 | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-18121/yetus/diff-checkstyle-ql.txt | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-18121/yetus/diff-checkstyle-hcatalog_streaming.txt | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-18121/yetus/diff-checksty
[jira] [Commented] (HIVE-21225) ACID: getAcidState() should cache a recursive dir listing locally
[ https://issues.apache.org/jira/browse/HIVE-21225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16889267#comment-16889267 ] Hive QA commented on HIVE-21225: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12975247/HIVE-21225.15.patch {color:green}SUCCESS:{color} +1 due to 6 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 16682 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.llap.daemon.impl.TestTaskExecutorService.testSetCapacity (batchId=358) org.apache.hadoop.hive.ql.TestTxnExIm.testImport (batchId=330) org.apache.hadoop.hive.ql.TestTxnExIm.testImportNoTarget (batchId=330) org.apache.hadoop.hive.ql.TestTxnLoadData.loadData (batchId=306) org.apache.hadoop.hive.ql.TestTxnLoadData.loadDataNonAcid2AcidConversion (batchId=306) org.apache.hadoop.hive.ql.TestTxnLoadData.loadDataUpdate (batchId=306) org.apache.hadoop.hive.ql.TestTxnLoadData.testMultiStatement (batchId=306) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/18109/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18109/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18109/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 7 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12975247 - PreCommit-HIVE-Build > ACID: getAcidState() should cache a recursive dir listing locally > - > > Key: HIVE-21225 > URL: https://issues.apache.org/jira/browse/HIVE-21225 > Project: Hive > Issue Type: Improvement > Components: Transactions >Reporter: Gopal V >Assignee: Vaibhav Gumashta >Priority: Major > Attachments: HIVE-21225.1.patch, HIVE-21225.10.patch, > HIVE-21225.11.patch, HIVE-21225.12.patch, HIVE-21225.13.patch, > HIVE-21225.14.patch, HIVE-21225.15.patch, HIVE-21225.15.patch, > HIVE-21225.2.patch, HIVE-21225.3.patch, HIVE-21225.4.patch, > HIVE-21225.4.patch, HIVE-21225.5.patch, HIVE-21225.6.patch, > HIVE-21225.7.patch, HIVE-21225.7.patch, HIVE-21225.8.patch, > HIVE-21225.9.patch, async-pid-44-2.svg > > > Currently getAcidState() makes 3 calls into the FS api which could be > answered by making a single recursive listDir call and reusing the same data > to check for isRawFormat() and isValidBase(). > All delta operations for a single partition can go against a single listed > directory snapshot instead of interacting with the NameNode or ObjectStore > within the inner loop. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (HIVE-21225) ACID: getAcidState() should cache a recursive dir listing locally
[ https://issues.apache.org/jira/browse/HIVE-21225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16889255#comment-16889255 ] Hive QA commented on HIVE-21225: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 32s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 15s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 30s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 37s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m 9s{color} | {color:blue} ql in master has 2250 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 30s{color} | {color:blue} hcatalog/streaming in master has 11 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 27s{color} | {color:blue} streaming in master has 2 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 40s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 48s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 25s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 55s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 30s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 30s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 52s{color} | {color:red} ql: The patch generated 47 new + 1414 unchanged - 11 fixed = 1461 total (was 1425) {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 13s{color} | {color:red} hcatalog/streaming: The patch generated 1 new + 229 unchanged - 0 fixed = 230 total (was 229) {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 15s{color} | {color:red} streaming: The patch generated 3 new + 454 unchanged - 0 fixed = 457 total (was 454) {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s{color} | {color:red} The patch has 44 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m 17s{color} | {color:red} ql generated 1 new + 2250 unchanged - 0 fixed = 2251 total (was 2250) {color} | | {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 1m 1s{color} | {color:red} ql generated 2 new + 98 unchanged - 2 fixed = 100 total (was 100) {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 13s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 37m 31s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | FindBugs | module:ql | | | Field only ever set to null:null: org.apache.hadoop.hive.ql.io.AcidUtils$HdfsDirSnapshotImpl.fileId In AcidUtils.java | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-18109/dev-support/hive-personality.sh | | git revision | master / 5ea614d | | Default Java | 1.8.0_111 | | findbugs | v3.0.0 | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-18109/yetus/diff-checkstyle-ql.txt | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-18109/yetus/diff-checkstyle-hcatalog_streaming.txt | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-18109/yetus/diff-checksty
[jira] [Commented] (HIVE-21225) ACID: getAcidState() should cache a recursive dir listing locally
[ https://issues.apache.org/jira/browse/HIVE-21225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16886324#comment-16886324 ] Jason Dere commented on HIVE-21225: --- [~gopalv] and [~vgumashta] brought this one to my attention as a fix for HIVE-22001, since using just a single recursive file listing will prevent the problem in HIVE-22001 where the directory was listed once, one or more files are deleted from that directory, and then getAcidState() tries to do a subsequent listing on a deleted file. However this patch will need additional modifications as the error in HIVE-22001 occurred in the file-ids path, which still issues multiple FS calls in the current patch. > ACID: getAcidState() should cache a recursive dir listing locally > - > > Key: HIVE-21225 > URL: https://issues.apache.org/jira/browse/HIVE-21225 > Project: Hive > Issue Type: Improvement > Components: Transactions >Reporter: Gopal V >Assignee: Vaibhav Gumashta >Priority: Major > Attachments: HIVE-21225.1.patch, HIVE-21225.10.patch, > HIVE-21225.11.patch, HIVE-21225.12.patch, HIVE-21225.13.patch, > HIVE-21225.2.patch, HIVE-21225.3.patch, HIVE-21225.4.patch, > HIVE-21225.4.patch, HIVE-21225.5.patch, HIVE-21225.6.patch, > HIVE-21225.7.patch, HIVE-21225.7.patch, HIVE-21225.8.patch, > HIVE-21225.9.patch, async-pid-44-2.svg > > > Currently getAcidState() makes 3 calls into the FS api which could be > answered by making a single recursive listDir call and reusing the same data > to check for isRawFormat() and isValidBase(). > All delta operations for a single partition can go against a single listed > directory snapshot instead of interacting with the NameNode or ObjectStore > within the inner loop. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (HIVE-21225) ACID: getAcidState() should cache a recursive dir listing locally
[ https://issues.apache.org/jira/browse/HIVE-21225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16885179#comment-16885179 ] Hive QA commented on HIVE-21225: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12974709/HIVE-21225.13.patch {color:red}ERROR:{color} -1 due to build exiting with an error Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/18031/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18031/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18031/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Tests exited with: Exception: Patch URL https://issues.apache.org/jira/secure/attachment/12974709/HIVE-21225.13.patch was found in seen patch url's cache and a test was probably run already on it. Aborting... {noformat} This message is automatically generated. ATTACHMENT ID: 12974709 - PreCommit-HIVE-Build > ACID: getAcidState() should cache a recursive dir listing locally > - > > Key: HIVE-21225 > URL: https://issues.apache.org/jira/browse/HIVE-21225 > Project: Hive > Issue Type: Improvement > Components: Transactions >Reporter: Gopal V >Assignee: Vaibhav Gumashta >Priority: Major > Attachments: HIVE-21225.1.patch, HIVE-21225.10.patch, > HIVE-21225.11.patch, HIVE-21225.12.patch, HIVE-21225.13.patch, > HIVE-21225.2.patch, HIVE-21225.3.patch, HIVE-21225.4.patch, > HIVE-21225.4.patch, HIVE-21225.5.patch, HIVE-21225.6.patch, > HIVE-21225.7.patch, HIVE-21225.7.patch, HIVE-21225.8.patch, > HIVE-21225.9.patch, async-pid-44-2.svg > > > Currently getAcidState() makes 3 calls into the FS api which could be > answered by making a single recursive listDir call and reusing the same data > to check for isRawFormat() and isValidBase(). > All delta operations for a single partition can go against a single listed > directory snapshot instead of interacting with the NameNode or ObjectStore > within the inner loop. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (HIVE-21225) ACID: getAcidState() should cache a recursive dir listing locally
[ https://issues.apache.org/jira/browse/HIVE-21225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16885176#comment-16885176 ] Hive QA commented on HIVE-21225: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12974709/HIVE-21225.13.patch {color:green}SUCCESS:{color} +1 due to 6 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 16669 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[acid_bucket_pruning] (batchId=155) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[mm_all] (batchId=157) org.apache.hadoop.hive.ql.exec.tez.TestDynamicPartitionPruner.testSingleSourceMultipleFiltersOrdering1 (batchId=340) org.apache.hadoop.hive.ql.io.orc.TestInputOutputFormat.testFSCallsVectorizedOrcAcidRowBatchReader (batchId=321) org.apache.hadoop.hive.ql.txn.compactor.TestCompactor.testStatsAfterCompactionPartTblNew (batchId=246) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/18030/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18030/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18030/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 5 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12974709 - PreCommit-HIVE-Build > ACID: getAcidState() should cache a recursive dir listing locally > - > > Key: HIVE-21225 > URL: https://issues.apache.org/jira/browse/HIVE-21225 > Project: Hive > Issue Type: Improvement > Components: Transactions >Reporter: Gopal V >Assignee: Vaibhav Gumashta >Priority: Major > Attachments: HIVE-21225.1.patch, HIVE-21225.10.patch, > HIVE-21225.11.patch, HIVE-21225.12.patch, HIVE-21225.13.patch, > HIVE-21225.2.patch, HIVE-21225.3.patch, HIVE-21225.4.patch, > HIVE-21225.4.patch, HIVE-21225.5.patch, HIVE-21225.6.patch, > HIVE-21225.7.patch, HIVE-21225.7.patch, HIVE-21225.8.patch, > HIVE-21225.9.patch, async-pid-44-2.svg > > > Currently getAcidState() makes 3 calls into the FS api which could be > answered by making a single recursive listDir call and reusing the same data > to check for isRawFormat() and isValidBase(). > All delta operations for a single partition can go against a single listed > directory snapshot instead of interacting with the NameNode or ObjectStore > within the inner loop. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (HIVE-21225) ACID: getAcidState() should cache a recursive dir listing locally
[ https://issues.apache.org/jira/browse/HIVE-21225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16885127#comment-16885127 ] Hive QA commented on HIVE-21225: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 51s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 38s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 28s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 35s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m 13s{color} | {color:blue} ql in master has 2253 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 29s{color} | {color:blue} hcatalog/streaming in master has 11 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 27s{color} | {color:blue} streaming in master has 2 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 40s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 51s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 27s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 55s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 28s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 28s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 50s{color} | {color:red} ql: The patch generated 35 new + 1418 unchanged - 7 fixed = 1453 total (was 1425) {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 13s{color} | {color:red} hcatalog/streaming: The patch generated 1 new + 229 unchanged - 0 fixed = 230 total (was 229) {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 15s{color} | {color:red} streaming: The patch generated 3 new + 454 unchanged - 0 fixed = 457 total (was 454) {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s{color} | {color:red} The patch has 44 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m 17s{color} | {color:red} ql generated 1 new + 2253 unchanged - 0 fixed = 2254 total (was 2253) {color} | | {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 59s{color} | {color:red} ql generated 2 new + 98 unchanged - 2 fixed = 100 total (was 100) {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 15s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 38m 13s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | FindBugs | module:ql | | | Field only ever set to null:null: org.apache.hadoop.hive.ql.io.AcidUtils$HdfsDirSnapshotImpl.fileId In AcidUtils.java | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-18030/dev-support/hive-personality.sh | | git revision | master / 2350662 | | Default Java | 1.8.0_111 | | findbugs | v3.0.0 | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-18030/yetus/diff-checkstyle-ql.txt | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-18030/yetus/diff-checkstyle-hcatalog_streaming.txt | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-18030/yetus/diff-checkstyl
[jira] [Commented] (HIVE-21225) ACID: getAcidState() should cache a recursive dir listing locally
[ https://issues.apache.org/jira/browse/HIVE-21225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16884823#comment-16884823 ] Hive QA commented on HIVE-21225: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12974648/HIVE-21225.12.patch {color:green}SUCCESS:{color} +1 due to 7 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 29 failed/errored test(s), 16647 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_nullscan] (batchId=73) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_stats5] (batchId=23) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_4] (batchId=13) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mm_all] (batchId=74) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[row__id] (batchId=86) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[stats_nonpart] (batchId=14) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[stats_part2] (batchId=22) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[stats_part] (batchId=52) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[stats_sizebug] (batchId=89) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[acid_bucket_pruning] (batchId=155) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[mm_all] (batchId=157) org.apache.hadoop.hive.llap.cache.TestBuddyAllocator.testMTT[2] (batchId=358) org.apache.hadoop.hive.ql.TestAcidOnTez.testBucketedAcidInsertWithRemoveUnion (batchId=244) org.apache.hadoop.hive.ql.TestTxnCommands2.testNonAcidToAcidConversion2 (batchId=330) org.apache.hadoop.hive.ql.TestTxnCommands2.testNonAcidToAcidConversion3 (batchId=330) org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.testNonAcidToAcidConversion2 (batchId=344) org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.testNonAcidToAcidConversion3 (batchId=344) org.apache.hadoop.hive.ql.TestTxnNoBuckets.testCompactStatsGather (batchId=330) org.apache.hadoop.hive.ql.TestTxnNoBucketsVectorized.testCompactStatsGather (batchId=332) org.apache.hadoop.hive.ql.io.orc.TestInputOutputFormat.testFSCallsVectorizedOrcAcidRowBatchReader (batchId=321) org.apache.hadoop.hive.ql.parse.TestReplAcidTablesWithJsonMessage.testDumpAcidTableWithPartitionDirMissing (batchId=267) org.apache.hadoop.hive.ql.parse.TestReplAcidTablesWithJsonMessage.testDumpAcidTableWithTableDirMissing (batchId=267) org.apache.hadoop.hive.ql.parse.TestReplAcidTablesWithJsonMessage.testMultiDBTxn (batchId=267) org.apache.hadoop.hive.ql.parse.TestReplicationScenariosAcidTables.testDumpAcidTableWithPartitionDirMissing (batchId=269) org.apache.hadoop.hive.ql.parse.TestReplicationScenariosAcidTables.testDumpAcidTableWithTableDirMissing (batchId=269) org.apache.hadoop.hive.ql.parse.TestReplicationScenariosAcidTables.testMultiDBTxn (batchId=269) org.apache.hadoop.hive.ql.txn.compactor.TestCompactor.testStatsAfterCompactionPartTbl (batchId=246) org.apache.hadoop.hive.ql.txn.compactor.TestCompactor.testStatsAfterCompactionPartTblNew (batchId=246) org.apache.hadoop.hive.ql.util.TestUpgradeTool.testPostUpgrade (batchId=306) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/18026/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18026/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18026/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 29 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12974648 - PreCommit-HIVE-Build > ACID: getAcidState() should cache a recursive dir listing locally > - > > Key: HIVE-21225 > URL: https://issues.apache.org/jira/browse/HIVE-21225 > Project: Hive > Issue Type: Improvement > Components: Transactions >Reporter: Gopal V >Assignee: Vaibhav Gumashta >Priority: Major > Attachments: HIVE-21225.1.patch, HIVE-21225.10.patch, > HIVE-21225.11.patch, HIVE-21225.12.patch, HIVE-21225.2.patch, > HIVE-21225.3.patch, HIVE-21225.4.patch, HIVE-21225.4.patch, > HIVE-21225.5.patch, HIVE-21225.6.patch, HIVE-21225.7.patch, > HIVE-21225.7.patch, HIVE-21225.8.patch, HIVE-21225.9.patch, async-pid-44-2.svg > > > Currently getAcidState() makes 3 calls into the FS api which could be > answered by making a single recursive listDir call and reusing the same data > to check for isRawFormat() and isValidBase(). > All d
[jira] [Commented] (HIVE-21225) ACID: getAcidState() should cache a recursive dir listing locally
[ https://issues.apache.org/jira/browse/HIVE-21225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16884814#comment-16884814 ] Hive QA commented on HIVE-21225: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 49s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 17s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 22s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 38s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m 10s{color} | {color:blue} ql in master has 2253 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 29s{color} | {color:blue} hcatalog/streaming in master has 11 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 27s{color} | {color:blue} streaming in master has 2 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 39s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 50s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 26s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 51s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 24s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 24s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 51s{color} | {color:red} ql: The patch generated 33 new + 1419 unchanged - 7 fixed = 1452 total (was 1426) {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 12s{color} | {color:red} hcatalog/streaming: The patch generated 1 new + 229 unchanged - 0 fixed = 230 total (was 229) {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 15s{color} | {color:red} streaming: The patch generated 3 new + 454 unchanged - 0 fixed = 457 total (was 454) {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s{color} | {color:red} The patch has 45 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m 12s{color} | {color:red} ql generated 1 new + 2253 unchanged - 0 fixed = 2254 total (was 2253) {color} | | {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 1m 0s{color} | {color:red} ql generated 2 new + 98 unchanged - 2 fixed = 100 total (was 100) {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 13s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 37m 16s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | FindBugs | module:ql | | | Field only ever set to null:null: org.apache.hadoop.hive.ql.io.AcidUtils$HdfsDirSnapshotImpl.fileId In AcidUtils.java | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-18026/dev-support/hive-personality.sh | | git revision | master / a669ca4 | | Default Java | 1.8.0_111 | | findbugs | v3.0.0 | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-18026/yetus/diff-checkstyle-ql.txt | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-18026/yetus/diff-checkstyle-hcatalog_streaming.txt | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-18026/yetus/diff-checkstyl
[jira] [Commented] (HIVE-21225) ACID: getAcidState() should cache a recursive dir listing locally
[ https://issues.apache.org/jira/browse/HIVE-21225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16884775#comment-16884775 ] Hive QA commented on HIVE-21225: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12974644/HIVE-21225.11.patch {color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 10 failed/errored test(s), 16625 tests executed *Failed tests:* {noformat} TestDataSourceProviderFactory - did not produce a TEST-*.xml file (likely timed out) (batchId=232) TestObjectStore - did not produce a TEST-*.xml file (likely timed out) (batchId=232) org.apache.hadoop.hive.ql.TestAcidOnTez.testBucketedAcidInsertWithRemoveUnion (batchId=244) org.apache.hadoop.hive.ql.TestAcidOnTez.testCrudMajorCompactionSplitGrouper (batchId=244) org.apache.hadoop.hive.ql.TestTxnNoBuckets.testCompactStatsGather (batchId=330) org.apache.hadoop.hive.ql.TestTxnNoBucketsVectorized.testCompactStatsGather (batchId=332) org.apache.hadoop.hive.ql.io.orc.TestInputOutputFormat.testFSCallsVectorizedOrcAcidRowBatchReader (batchId=321) org.apache.hadoop.hive.ql.txn.compactor.TestCompactor.testStatsAfterCompactionPartTbl (batchId=246) org.apache.hadoop.hive.ql.txn.compactor.TestCompactor.testStatsAfterCompactionPartTblNew (batchId=246) org.apache.hadoop.hive.ql.util.TestUpgradeTool.testPostUpgrade (batchId=306) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/18024/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18024/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18024/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 10 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12974644 - PreCommit-HIVE-Build > ACID: getAcidState() should cache a recursive dir listing locally > - > > Key: HIVE-21225 > URL: https://issues.apache.org/jira/browse/HIVE-21225 > Project: Hive > Issue Type: Improvement > Components: Transactions >Reporter: Gopal V >Assignee: Vaibhav Gumashta >Priority: Major > Attachments: HIVE-21225.1.patch, HIVE-21225.10.patch, > HIVE-21225.11.patch, HIVE-21225.2.patch, HIVE-21225.3.patch, > HIVE-21225.4.patch, HIVE-21225.4.patch, HIVE-21225.5.patch, > HIVE-21225.6.patch, HIVE-21225.7.patch, HIVE-21225.7.patch, > HIVE-21225.8.patch, HIVE-21225.9.patch, async-pid-44-2.svg > > > Currently getAcidState() makes 3 calls into the FS api which could be > answered by making a single recursive listDir call and reusing the same data > to check for isRawFormat() and isValidBase(). > All delta operations for a single partition can go against a single listed > directory snapshot instead of interacting with the NameNode or ObjectStore > within the inner loop. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (HIVE-21225) ACID: getAcidState() should cache a recursive dir listing locally
[ https://issues.apache.org/jira/browse/HIVE-21225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16884765#comment-16884765 ] Hive QA commented on HIVE-21225: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 42s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m 49s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 45s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 6s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m 1s{color} | {color:blue} ql in master has 2253 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 39s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 24s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 26s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 8s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 48s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 48s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 47s{color} | {color:red} ql: The patch generated 34 new + 1189 unchanged - 5 fixed = 1223 total (was 1194) {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s{color} | {color:red} The patch has 45 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m 13s{color} | {color:red} ql generated 1 new + 2253 unchanged - 0 fixed = 2254 total (was 2253) {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 23s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 14s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 30m 15s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | FindBugs | module:ql | | | Field only ever set to null:null: org.apache.hadoop.hive.ql.io.AcidUtils$HdfsDirSnapshotImpl.fileId In AcidUtils.java | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-18024/dev-support/hive-personality.sh | | git revision | master / a669ca4 | | Default Java | 1.8.0_111 | | findbugs | v3.0.0 | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-18024/yetus/diff-checkstyle-ql.txt | | whitespace | http://104.198.109.242/logs//PreCommit-HIVE-Build-18024/yetus/whitespace-eol.txt | | findbugs | http://104.198.109.242/logs//PreCommit-HIVE-Build-18024/yetus/new-findbugs-ql.html | | modules | C: ql itests/hive-unit U: . | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-18024/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > ACID: getAcidState() should cache a recursive dir listing locally > - > > Key: HIVE-21225 > URL: https://issues.apache.org/jira/browse/HIVE-21225 > Project: Hive > Issue Type: Improvement > Components: Transactions >Reporter: Gopal V >Assignee: Vaibhav Gumashta >Priority: Major > Attachments: HIVE-21225.1.patch, HIVE-21225.10.patch, > HIVE-21225.11.patch, HIVE-21225.2.patch, HIVE-212
[jira] [Commented] (HIVE-21225) ACID: getAcidState() should cache a recursive dir listing locally
[ https://issues.apache.org/jira/browse/HIVE-21225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16884632#comment-16884632 ] Hive QA commented on HIVE-21225: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12974627/HIVE-21225.10.patch {color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 16 failed/errored test(s), 16627 tests executed *Failed tests:* {noformat} TestMiniLlapLocalCliDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=177) [union_top_level.q,vector_create_struct_table.q,llap_uncompressed.q,schema_evol_text_vecrow_part_all_primitive.q,materialized_view_partitioned_2.q,murmur_hash_migration.q,update_where_partitioned.q,materialized_view_rewrite_1.q,materialized_view_create_rewrite_time_window_2.q,drop_partition_with_stats.q,vectorization_1.q,skiphf_aggr.q,fold_varchar.q,auto_join_filters.q,join0.q,insert_orig_table.q,mergejoin.q,vector_if_expr_2.q,orc_split_elimination.q,vector_outer_join0.q,vector_udf1.q,schema_evol_text_vec_part_all_primitive.q,vector_complex_all.q,union3.q,windowing_windowspec2.q,auto_smb_mapjoin_14.q,vector_mapjoin_complex_values.q,results_cache_truncate.q,vector_join_filters.q,reduce_deduplicate_extended.q] org.apache.hadoop.hive.ql.TestAcidOnTez.testBucketedAcidInsertWithRemoveUnion (batchId=244) org.apache.hadoop.hive.ql.TestAcidOnTez.testCrudMajorCompactionSplitGrouper (batchId=244) org.apache.hadoop.hive.ql.TestTxnNoBuckets.testCompactStatsGather (batchId=330) org.apache.hadoop.hive.ql.TestTxnNoBucketsVectorized.testCompactStatsGather (batchId=332) org.apache.hadoop.hive.ql.io.orc.TestInputOutputFormat.testACIDReaderFooterSerialize (batchId=321) org.apache.hadoop.hive.ql.io.orc.TestInputOutputFormat.testACIDReaderFooterSerializeWithDeltas (batchId=321) org.apache.hadoop.hive.ql.io.orc.TestInputOutputFormat.testACIDReaderNoFooterSerialize (batchId=321) org.apache.hadoop.hive.ql.io.orc.TestInputOutputFormat.testACIDReaderNoFooterSerializeWithDeltas (batchId=321) org.apache.hadoop.hive.ql.io.orc.TestInputOutputFormat.testFSCallsVectorizedOrcAcidRowBatchReader (batchId=321) org.apache.hadoop.hive.ql.io.orc.TestInputOutputFormat.testSplitGenReadOps (batchId=321) org.apache.hadoop.hive.ql.io.orc.TestInputOutputFormat.testSplitGenReadOpsLocalCache (batchId=321) org.apache.hadoop.hive.ql.io.orc.TestInputOutputFormat.testSplitGenReadOpsLocalCacheChangeFileLen (batchId=321) org.apache.hadoop.hive.ql.io.orc.TestInputOutputFormat.testSplitGenReadOpsLocalCacheChangeModificationTime (batchId=321) org.apache.hadoop.hive.ql.txn.compactor.TestCompactor.testStatsAfterCompactionPartTblNew (batchId=246) org.apache.hadoop.hive.ql.util.TestUpgradeTool.testPostUpgrade (batchId=306) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/18023/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18023/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18023/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 16 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12974627 - PreCommit-HIVE-Build > ACID: getAcidState() should cache a recursive dir listing locally > - > > Key: HIVE-21225 > URL: https://issues.apache.org/jira/browse/HIVE-21225 > Project: Hive > Issue Type: Improvement > Components: Transactions >Reporter: Gopal V >Assignee: Vaibhav Gumashta >Priority: Major > Attachments: HIVE-21225.1.patch, HIVE-21225.10.patch, > HIVE-21225.2.patch, HIVE-21225.3.patch, HIVE-21225.4.patch, > HIVE-21225.4.patch, HIVE-21225.5.patch, HIVE-21225.6.patch, > HIVE-21225.7.patch, HIVE-21225.7.patch, HIVE-21225.8.patch, > HIVE-21225.9.patch, async-pid-44-2.svg > > > Currently getAcidState() makes 3 calls into the FS api which could be > answered by making a single recursive listDir call and reusing the same data > to check for isRawFormat() and isValidBase(). > All delta operations for a single partition can go against a single listed > directory snapshot instead of interacting with the NameNode or ObjectStore > within the inner loop. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (HIVE-21225) ACID: getAcidState() should cache a recursive dir listing locally
[ https://issues.apache.org/jira/browse/HIVE-21225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16884614#comment-16884614 ] Hive QA commented on HIVE-21225: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 34s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m 59s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 51s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 4s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m 2s{color} | {color:blue} ql in master has 2253 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 39s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 20s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 27s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 8s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 45s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 45s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 48s{color} | {color:red} ql: The patch generated 36 new + 1189 unchanged - 5 fixed = 1225 total (was 1194) {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s{color} | {color:red} The patch has 45 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m 14s{color} | {color:red} ql generated 1 new + 2253 unchanged - 0 fixed = 2254 total (was 2253) {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 21s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 14s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 30m 22s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | FindBugs | module:ql | | | Field only ever set to null:null: org.apache.hadoop.hive.ql.io.AcidUtils$HdfsDirSnapshotImpl.fileId In AcidUtils.java | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-18023/dev-support/hive-personality.sh | | git revision | master / a669ca4 | | Default Java | 1.8.0_111 | | findbugs | v3.0.0 | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-18023/yetus/diff-checkstyle-ql.txt | | whitespace | http://104.198.109.242/logs//PreCommit-HIVE-Build-18023/yetus/whitespace-eol.txt | | findbugs | http://104.198.109.242/logs//PreCommit-HIVE-Build-18023/yetus/new-findbugs-ql.html | | modules | C: ql itests/hive-unit U: . | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-18023/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > ACID: getAcidState() should cache a recursive dir listing locally > - > > Key: HIVE-21225 > URL: https://issues.apache.org/jira/browse/HIVE-21225 > Project: Hive > Issue Type: Improvement > Components: Transactions >Reporter: Gopal V >Assignee: Vaibhav Gumashta >Priority: Major > Attachments: HIVE-21225.1.patch, HIVE-21225.10.patch, > HIVE-21225.2.patch, HIVE-21225.3.patch, HIVE-2122
[jira] [Commented] (HIVE-21225) ACID: getAcidState() should cache a recursive dir listing locally
[ https://issues.apache.org/jira/browse/HIVE-21225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16884608#comment-16884608 ] Vaibhav Gumashta commented on HIVE-21225: - [~vgarg] FYI, with respect to the 3 calls: isRawFormated, isValidBase and isCompactedBase, only isRawFormated does an fs listing. > ACID: getAcidState() should cache a recursive dir listing locally > - > > Key: HIVE-21225 > URL: https://issues.apache.org/jira/browse/HIVE-21225 > Project: Hive > Issue Type: Improvement > Components: Transactions >Reporter: Gopal V >Assignee: Vaibhav Gumashta >Priority: Major > Attachments: HIVE-21225.1.patch, HIVE-21225.2.patch, > HIVE-21225.3.patch, HIVE-21225.4.patch, HIVE-21225.4.patch, > HIVE-21225.5.patch, HIVE-21225.6.patch, HIVE-21225.7.patch, > HIVE-21225.7.patch, HIVE-21225.8.patch, HIVE-21225.9.patch, async-pid-44-2.svg > > > Currently getAcidState() makes 3 calls into the FS api which could be > answered by making a single recursive listDir call and reusing the same data > to check for isRawFormat() and isValidBase(). > All delta operations for a single partition can go against a single listed > directory snapshot instead of interacting with the NameNode or ObjectStore > within the inner loop. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (HIVE-21225) ACID: getAcidState() should cache a recursive dir listing locally
[ https://issues.apache.org/jira/browse/HIVE-21225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16884602#comment-16884602 ] Vaibhav Gumashta commented on HIVE-21225: - Looks like the export import integration with mm-tables also creates a weird dir structure like: {{mm_table_import/delta_001_001_/export_old_delta_001_001_/00_0}} ... > ACID: getAcidState() should cache a recursive dir listing locally > - > > Key: HIVE-21225 > URL: https://issues.apache.org/jira/browse/HIVE-21225 > Project: Hive > Issue Type: Improvement > Components: Transactions >Reporter: Gopal V >Assignee: Vaibhav Gumashta >Priority: Major > Attachments: HIVE-21225.1.patch, HIVE-21225.2.patch, > HIVE-21225.3.patch, HIVE-21225.4.patch, HIVE-21225.4.patch, > HIVE-21225.5.patch, HIVE-21225.6.patch, HIVE-21225.7.patch, > HIVE-21225.7.patch, HIVE-21225.8.patch, HIVE-21225.9.patch, async-pid-44-2.svg > > > Currently getAcidState() makes 3 calls into the FS api which could be > answered by making a single recursive listDir call and reusing the same data > to check for isRawFormat() and isValidBase(). > All delta operations for a single partition can go against a single listed > directory snapshot instead of interacting with the NameNode or ObjectStore > within the inner loop. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (HIVE-21225) ACID: getAcidState() should cache a recursive dir listing locally
[ https://issues.apache.org/jira/browse/HIVE-21225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16882283#comment-16882283 ] Vineet Garg commented on HIVE-21225: bq. I think MM tables retain these subdirs since its MoveTask doesn't rename any files Yes I noticed this with MM tables > ACID: getAcidState() should cache a recursive dir listing locally > - > > Key: HIVE-21225 > URL: https://issues.apache.org/jira/browse/HIVE-21225 > Project: Hive > Issue Type: Improvement > Components: Transactions >Reporter: Gopal V >Assignee: Vaibhav Gumashta >Priority: Major > Attachments: HIVE-21225.1.patch, HIVE-21225.2.patch, > HIVE-21225.3.patch, HIVE-21225.4.patch, HIVE-21225.4.patch, > HIVE-21225.5.patch, HIVE-21225.6.patch, HIVE-21225.7.patch, > HIVE-21225.7.patch, HIVE-21225.8.patch, HIVE-21225.9.patch, async-pid-44-2.svg > > > Currently getAcidState() makes 3 calls into the FS api which could be > answered by making a single recursive listDir call and reusing the same data > to check for isRawFormat() and isValidBase(). > All delta operations for a single partition can go against a single listed > directory snapshot instead of interacting with the NameNode or ObjectStore > within the inner loop. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21225) ACID: getAcidState() should cache a recursive dir listing locally
[ https://issues.apache.org/jira/browse/HIVE-21225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16881708#comment-16881708 ] Eugene Koifman commented on HIVE-21225: --- Full Acid uses stmtId for Insert ... as select union all which it does somewhere in MoveTask I think MM tables retain these subdirs since its MoveTask doesn't rename any files > ACID: getAcidState() should cache a recursive dir listing locally > - > > Key: HIVE-21225 > URL: https://issues.apache.org/jira/browse/HIVE-21225 > Project: Hive > Issue Type: Improvement > Components: Transactions >Reporter: Gopal V >Assignee: Vaibhav Gumashta >Priority: Major > Attachments: HIVE-21225.1.patch, HIVE-21225.2.patch, > HIVE-21225.3.patch, HIVE-21225.4.patch, HIVE-21225.4.patch, > HIVE-21225.5.patch, HIVE-21225.6.patch, HIVE-21225.7.patch, > HIVE-21225.7.patch, HIVE-21225.8.patch, HIVE-21225.9.patch, async-pid-44-2.svg > > > Currently getAcidState() makes 3 calls into the FS api which could be > answered by making a single recursive listDir call and reusing the same data > to check for isRawFormat() and isValidBase(). > All delta operations for a single partition can go against a single listed > directory snapshot instead of interacting with the NameNode or ObjectStore > within the inner loop. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21225) ACID: getAcidState() should cache a recursive dir listing locally
[ https://issues.apache.org/jira/browse/HIVE-21225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16881703#comment-16881703 ] Gopal V commented on HIVE-21225: bq. union_mm/delta_001_001_0001/HIVE_UNION_SUBDIR_1 Are you seeing stuff like that? That looks like a bug to me. The output dir should have delta_001_001_0001 & delta_001_001_0002, but no further sub-dirs in it. > ACID: getAcidState() should cache a recursive dir listing locally > - > > Key: HIVE-21225 > URL: https://issues.apache.org/jira/browse/HIVE-21225 > Project: Hive > Issue Type: Improvement > Components: Transactions >Reporter: Gopal V >Assignee: Vaibhav Gumashta >Priority: Major > Attachments: HIVE-21225.1.patch, HIVE-21225.2.patch, > HIVE-21225.3.patch, HIVE-21225.4.patch, HIVE-21225.4.patch, > HIVE-21225.5.patch, HIVE-21225.6.patch, HIVE-21225.7.patch, > HIVE-21225.7.patch, HIVE-21225.8.patch, HIVE-21225.9.patch, async-pid-44-2.svg > > > Currently getAcidState() makes 3 calls into the FS api which could be > answered by making a single recursive listDir call and reusing the same data > to check for isRawFormat() and isValidBase(). > All delta operations for a single partition can go against a single listed > directory snapshot instead of interacting with the NameNode or ObjectStore > within the inner loop. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21225) ACID: getAcidState() should cache a recursive dir listing locally
[ https://issues.apache.org/jira/browse/HIVE-21225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16881618#comment-16881618 ] Vineet Garg commented on HIVE-21225: bq. While building directory snapshots, check for isValidBase, isCompactedBase and isRawFormated and cache those for later within each snapshot. [~vgumashta] I think it makes more sense to cache them whenever they are called the first time. It looks like these functions might not be called for all the base directories and not all the time. So we should pay penalty only when necessary. bq. For the union case, let me look at what the rest of the code is doing, but in any case, we will need to map union_mm/delta_001_001_0002/HIVE_UNION_SUBDIR_2 to delta_001_001_0002 I guess doing this should fix the second issue > ACID: getAcidState() should cache a recursive dir listing locally > - > > Key: HIVE-21225 > URL: https://issues.apache.org/jira/browse/HIVE-21225 > Project: Hive > Issue Type: Improvement > Components: Transactions >Reporter: Gopal V >Assignee: Vaibhav Gumashta >Priority: Major > Attachments: HIVE-21225.1.patch, HIVE-21225.2.patch, > HIVE-21225.3.patch, HIVE-21225.4.patch, HIVE-21225.4.patch, > HIVE-21225.5.patch, HIVE-21225.6.patch, HIVE-21225.7.patch, > HIVE-21225.7.patch, HIVE-21225.8.patch, HIVE-21225.9.patch, async-pid-44-2.svg > > > Currently getAcidState() makes 3 calls into the FS api which could be > answered by making a single recursive listDir call and reusing the same data > to check for isRawFormat() and isValidBase(). > All delta operations for a single partition can go against a single listed > directory snapshot instead of interacting with the NameNode or ObjectStore > within the inner loop. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21225) ACID: getAcidState() should cache a recursive dir listing locally
[ https://issues.apache.org/jira/browse/HIVE-21225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16881614#comment-16881614 ] Vaibhav Gumashta commented on HIVE-21225: - [~vgarg] Thanks for the review: let me update the patch with feedback. On the 2 issues you raised, here is what I am thinking: 1. While building directory snapshots, check for isValidBase, isCompactedBase and isRawFormated and cache those for later within each snapshot. 2. For the union case, let me look at what the rest of the code is doing, but in any case, we will need to map union_mm/delta_001_001_0002/HIVE_UNION_SUBDIR_2 to delta_001_001_0002. I am guessing in other places as well we may be parsing delta_001_001_0002 portion of it, but I can verify and use the same approach. > ACID: getAcidState() should cache a recursive dir listing locally > - > > Key: HIVE-21225 > URL: https://issues.apache.org/jira/browse/HIVE-21225 > Project: Hive > Issue Type: Improvement > Components: Transactions >Reporter: Gopal V >Assignee: Vaibhav Gumashta >Priority: Major > Attachments: HIVE-21225.1.patch, HIVE-21225.2.patch, > HIVE-21225.3.patch, HIVE-21225.4.patch, HIVE-21225.4.patch, > HIVE-21225.5.patch, HIVE-21225.6.patch, HIVE-21225.7.patch, > HIVE-21225.7.patch, HIVE-21225.8.patch, HIVE-21225.9.patch, async-pid-44-2.svg > > > Currently getAcidState() makes 3 calls into the FS api which could be > answered by making a single recursive listDir call and reusing the same data > to check for isRawFormat() and isValidBase(). > All delta operations for a single partition can go against a single listed > directory snapshot instead of interacting with the NameNode or ObjectStore > within the inner loop. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21225) ACID: getAcidState() should cache a recursive dir listing locally
[ https://issues.apache.org/jira/browse/HIVE-21225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16881610#comment-16881610 ] Vineet Garg commented on HIVE-21225: The patch is building directory snapshot (dir path to file lists). For example in this case snapshot keys look like as following: * union_mm/delta_001_001_0002/HIVE_UNION_SUBDIR_2 * union_mm/delta_001_001_0001/HIVE_UNION_SUBDIR_1 This breaks the logic where base and delta directories are identified using directory name prefix. Since none of these begin with base_ or delta_ they are skipped. We could update this logic to look for base_ or delta_ in the whole path but I don't know if this is correct and safe. > ACID: getAcidState() should cache a recursive dir listing locally > - > > Key: HIVE-21225 > URL: https://issues.apache.org/jira/browse/HIVE-21225 > Project: Hive > Issue Type: Improvement > Components: Transactions >Reporter: Gopal V >Assignee: Vaibhav Gumashta >Priority: Major > Attachments: HIVE-21225.1.patch, HIVE-21225.2.patch, > HIVE-21225.3.patch, HIVE-21225.4.patch, HIVE-21225.4.patch, > HIVE-21225.5.patch, HIVE-21225.6.patch, HIVE-21225.7.patch, > HIVE-21225.7.patch, HIVE-21225.8.patch, HIVE-21225.9.patch, async-pid-44-2.svg > > > Currently getAcidState() makes 3 calls into the FS api which could be > answered by making a single recursive listDir call and reusing the same data > to check for isRawFormat() and isValidBase(). > All delta operations for a single partition can go against a single listed > directory snapshot instead of interacting with the NameNode or ObjectStore > within the inner loop. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21225) ACID: getAcidState() should cache a recursive dir listing locally
[ https://issues.apache.org/jira/browse/HIVE-21225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16881575#comment-16881575 ] Vineet Garg commented on HIVE-21225: Went over the patch once again and I am struggling to understand how this patch solves the problem cited in the jira. This patch is still calling FS in {{isValidBase}}, {{isCompactedBase}} and {{isRawFormated}}. May be I am misunderstanding something here? Currently file/dir listing is done to retrieve top level directories/files only (and then a recursive call within original (non-acid) directories if necessary). Multiple calls are done for {{isCompactedBase}} which makes a call to FS to read metadata file and for {{isRawFormatted}} which beside reading metadata file also makes FS call to look for files matching particular pattern. This seems to be done multiple times for the same base. I believe the low hanging fruit here is to cache this information and reuse it. This should reduce unnecessary calls to FS. Doing a recursive call instead of only looking for top level subdirectories/files are causing issue where directory layout is different (.e.g. CTAS with union). I am not sure how should the current logic be tuned/fixed to cater to this case. > ACID: getAcidState() should cache a recursive dir listing locally > - > > Key: HIVE-21225 > URL: https://issues.apache.org/jira/browse/HIVE-21225 > Project: Hive > Issue Type: Improvement > Components: Transactions >Reporter: Gopal V >Assignee: Vaibhav Gumashta >Priority: Major > Attachments: HIVE-21225.1.patch, HIVE-21225.2.patch, > HIVE-21225.3.patch, HIVE-21225.4.patch, HIVE-21225.4.patch, > HIVE-21225.5.patch, HIVE-21225.6.patch, HIVE-21225.7.patch, > HIVE-21225.7.patch, HIVE-21225.8.patch, HIVE-21225.9.patch, async-pid-44-2.svg > > > Currently getAcidState() makes 3 calls into the FS api which could be > answered by making a single recursive listDir call and reusing the same data > to check for isRawFormat() and isValidBase(). > All delta operations for a single partition can go against a single listed > directory snapshot instead of interacting with the NameNode or ObjectStore > within the inner loop. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21225) ACID: getAcidState() should cache a recursive dir listing locally
[ https://issues.apache.org/jira/browse/HIVE-21225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16881291#comment-16881291 ] Hive QA commented on HIVE-21225: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12973993/HIVE-21225.9.patch {color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 45 failed/errored test(s), 16630 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_nullscan] (batchId=73) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_stats5] (batchId=23) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_4] (batchId=13) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mm_all] (batchId=74) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[row__id] (batchId=86) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[stats_nonpart] (batchId=14) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[stats_part2] (batchId=22) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[stats_part] (batchId=52) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[stats_sizebug] (batchId=89) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[acid_bucket_pruning] (batchId=155) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[mm_all] (batchId=157) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[mm_exim] (batchId=186) org.apache.hadoop.hive.ql.TestTxnCommands.testMmExim (batchId=341) org.apache.hadoop.hive.ql.TestTxnCommands2.testNonAcidToAcidConversion1 (batchId=322) org.apache.hadoop.hive.ql.TestTxnCommands2.testOriginalFileReaderWhenNonAcidConvertedToAcid (batchId=322) org.apache.hadoop.hive.ql.TestTxnCommands2.updateDeletePartitioned (batchId=322) org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.testNonAcidToAcidConversion1 (batchId=336) org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.testOriginalFileReaderWhenNonAcidConvertedToAcid (batchId=336) org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.updateDeletePartitioned (batchId=336) org.apache.hadoop.hive.ql.TestTxnCommandsWithSplitUpdateAndVectorization.testMmExim (batchId=324) org.apache.hadoop.hive.ql.TestTxnExIm.testImport (batchId=322) org.apache.hadoop.hive.ql.TestTxnExIm.testImportNoTarget (batchId=322) org.apache.hadoop.hive.ql.TestTxnExIm.testMM (batchId=322) org.apache.hadoop.hive.ql.TestTxnExIm.testMMCreate (batchId=322) org.apache.hadoop.hive.ql.TestTxnLoadData.loadData (batchId=298) org.apache.hadoop.hive.ql.TestTxnLoadData.loadDataNonAcid2AcidConversion (batchId=298) org.apache.hadoop.hive.ql.TestTxnLoadData.loadDataUpdate (batchId=298) org.apache.hadoop.hive.ql.TestTxnLoadData.testMultiStatement (batchId=298) org.apache.hadoop.hive.ql.TestTxnNoBuckets.testCompactStatsGather (batchId=322) org.apache.hadoop.hive.ql.TestTxnNoBuckets.testEmptyCompactionResult (batchId=322) org.apache.hadoop.hive.ql.TestTxnNoBuckets.testToAcidConversionMultiBucket (batchId=322) org.apache.hadoop.hive.ql.TestTxnNoBucketsVectorized.testCompactStatsGather (batchId=324) org.apache.hadoop.hive.ql.TestTxnNoBucketsVectorized.testEmptyCompactionResult (batchId=324) org.apache.hadoop.hive.ql.TestTxnNoBucketsVectorized.testToAcidConversionMultiBucket (batchId=324) org.apache.hadoop.hive.ql.io.TestAcidUtils.testObsoleteOriginals (batchId=310) org.apache.hadoop.hive.ql.parse.TestReplAcidTablesWithJsonMessage.testDumpAcidTableWithPartitionDirMissing (batchId=260) org.apache.hadoop.hive.ql.parse.TestReplAcidTablesWithJsonMessage.testDumpAcidTableWithTableDirMissing (batchId=260) org.apache.hadoop.hive.ql.parse.TestReplAcidTablesWithJsonMessage.testMultiDBTxn (batchId=260) org.apache.hadoop.hive.ql.parse.TestReplicationScenariosAcidTables.testDumpAcidTableWithPartitionDirMissing (batchId=262) org.apache.hadoop.hive.ql.parse.TestReplicationScenariosAcidTables.testDumpAcidTableWithTableDirMissing (batchId=262) org.apache.hadoop.hive.ql.parse.TestReplicationScenariosAcidTables.testMultiDBTxn (batchId=262) org.apache.hadoop.hive.ql.parse.TestReplicationScenariosIncrementalLoadAcidTables.testAcidTableIncrementalReplication (batchId=263) org.apache.hadoop.hive.ql.txn.compactor.TestCompactor.testStatsAfterCompactionPartTbl (batchId=245) org.apache.hadoop.hive.ql.txn.compactor.TestCompactor.testStatsAfterCompactionPartTblNew (batchId=245) org.apache.hadoop.hive.ql.util.TestUpgradeTool.testPostUpgrade (batchId=298) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/17926/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17926/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17926/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.
[jira] [Commented] (HIVE-21225) ACID: getAcidState() should cache a recursive dir listing locally
[ https://issues.apache.org/jira/browse/HIVE-21225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16881253#comment-16881253 ] Hive QA commented on HIVE-21225: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 48s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 59s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 48s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 5s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m 10s{color} | {color:blue} ql in master has 2252 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 38s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 23s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 26s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 4s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 47s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 47s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 45s{color} | {color:red} ql: The patch generated 21 new + 903 unchanged - 1 fixed = 924 total (was 904) {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s{color} | {color:red} The patch has 27 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m 11s{color} | {color:red} ql generated 1 new + 2252 unchanged - 0 fixed = 2253 total (was 2252) {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 23s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 13s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 30m 21s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | FindBugs | module:ql | | | Unread field:AcidUtils.java:[line 1409] | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-17926/dev-support/hive-personality.sh | | git revision | master / f629501 | | Default Java | 1.8.0_111 | | findbugs | v3.0.0 | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-17926/yetus/diff-checkstyle-ql.txt | | whitespace | http://104.198.109.242/logs//PreCommit-HIVE-Build-17926/yetus/whitespace-eol.txt | | findbugs | http://104.198.109.242/logs//PreCommit-HIVE-Build-17926/yetus/new-findbugs-ql.html | | modules | C: ql itests/hive-unit U: . | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-17926/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > ACID: getAcidState() should cache a recursive dir listing locally > - > > Key: HIVE-21225 > URL: https://issues.apache.org/jira/browse/HIVE-21225 > Project: Hive > Issue Type: Improvement > Components: Transactions >Reporter: Gopal V >Assignee: Vaibhav Gumashta >Priority: Major > Attachments: HIVE-21225.1.patch, HIVE-21225.2.patch, > HIVE-21225.3.patch, HIVE-21225.4.patch, HIVE-21225.4.patch, > HIVE-21225.5.patch, HIVE-21225.6.patch, HIVE-21225.7.patch, > HIVE-212
[jira] [Commented] (HIVE-21225) ACID: getAcidState() should cache a recursive dir listing locally
[ https://issues.apache.org/jira/browse/HIVE-21225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16880809#comment-16880809 ] Vineet Garg commented on HIVE-21225: I have removed {{hiddenDirFilter}} in {{HIVE-21225.9.patch}} but there are still few cases producing wrong results (as mentioned earlier) > ACID: getAcidState() should cache a recursive dir listing locally > - > > Key: HIVE-21225 > URL: https://issues.apache.org/jira/browse/HIVE-21225 > Project: Hive > Issue Type: Improvement > Components: Transactions >Reporter: Gopal V >Assignee: Vaibhav Gumashta >Priority: Major > Attachments: HIVE-21225.1.patch, HIVE-21225.2.patch, > HIVE-21225.3.patch, HIVE-21225.4.patch, HIVE-21225.4.patch, > HIVE-21225.5.patch, HIVE-21225.6.patch, HIVE-21225.7.patch, > HIVE-21225.7.patch, HIVE-21225.8.patch, HIVE-21225.9.patch, async-pid-44-2.svg > > > Currently getAcidState() makes 3 calls into the FS api which could be > answered by making a single recursive listDir call and reusing the same data > to check for isRawFormat() and isValidBase(). > All delta operations for a single partition can go against a single listed > directory snapshot instead of interacting with the NameNode or ObjectStore > within the inner loop. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21225) ACID: getAcidState() should cache a recursive dir listing locally
[ https://issues.apache.org/jira/browse/HIVE-21225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16880807#comment-16880807 ] Vineet Garg commented on HIVE-21225: [~vgumashta] {{hiddenDirFilter}} is filtering out directories for temporary table causing wrong results. Is this filter required at all? There is another issue with tables loaded with UNION queries. Directory layout of such table could be {{//HIVE_UNION_SUBDIR_1}} but {{getHdfsDirSnapshots}} ends up returning only one sub-dir. Can you create pull request for this patch? > ACID: getAcidState() should cache a recursive dir listing locally > - > > Key: HIVE-21225 > URL: https://issues.apache.org/jira/browse/HIVE-21225 > Project: Hive > Issue Type: Improvement > Components: Transactions >Reporter: Gopal V >Assignee: Vaibhav Gumashta >Priority: Major > Attachments: HIVE-21225.1.patch, HIVE-21225.2.patch, > HIVE-21225.3.patch, HIVE-21225.4.patch, HIVE-21225.4.patch, > HIVE-21225.5.patch, HIVE-21225.6.patch, HIVE-21225.7.patch, > HIVE-21225.7.patch, HIVE-21225.8.patch, async-pid-44-2.svg > > > Currently getAcidState() makes 3 calls into the FS api which could be > answered by making a single recursive listDir call and reusing the same data > to check for isRawFormat() and isValidBase(). > All delta operations for a single partition can go against a single listed > directory snapshot instead of interacting with the NameNode or ObjectStore > within the inner loop. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21225) ACID: getAcidState() should cache a recursive dir listing locally
[ https://issues.apache.org/jira/browse/HIVE-21225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16880346#comment-16880346 ] Hive QA commented on HIVE-21225: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12973895/HIVE-21225.8.patch {color:red}ERROR:{color} -1 due to build exiting with an error Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/17907/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17907/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17907/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Tests exited with: Exception: Patch URL https://issues.apache.org/jira/secure/attachment/12973895/HIVE-21225.8.patch was found in seen patch url's cache and a test was probably run already on it. Aborting... {noformat} This message is automatically generated. ATTACHMENT ID: 12973895 - PreCommit-HIVE-Build > ACID: getAcidState() should cache a recursive dir listing locally > - > > Key: HIVE-21225 > URL: https://issues.apache.org/jira/browse/HIVE-21225 > Project: Hive > Issue Type: Improvement > Components: Transactions >Reporter: Gopal V >Assignee: Vaibhav Gumashta >Priority: Major > Attachments: HIVE-21225.1.patch, HIVE-21225.2.patch, > HIVE-21225.3.patch, HIVE-21225.4.patch, HIVE-21225.4.patch, > HIVE-21225.5.patch, HIVE-21225.6.patch, HIVE-21225.7.patch, > HIVE-21225.7.patch, HIVE-21225.8.patch, async-pid-44-2.svg > > > Currently getAcidState() makes 3 calls into the FS api which could be > answered by making a single recursive listDir call and reusing the same data > to check for isRawFormat() and isValidBase(). > All delta operations for a single partition can go against a single listed > directory snapshot instead of interacting with the NameNode or ObjectStore > within the inner loop. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21225) ACID: getAcidState() should cache a recursive dir listing locally
[ https://issues.apache.org/jira/browse/HIVE-21225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16880183#comment-16880183 ] Hive QA commented on HIVE-21225: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12973895/HIVE-21225.8.patch {color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 51 failed/errored test(s), 16361 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[delete_tmp_table] (batchId=56) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[insert_values_tmp_table] (batchId=5) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mm_all] (batchId=74) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mm_cttas] (batchId=51) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[temp_table_options1] (batchId=3) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorized_mapjoin2] (batchId=26) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorized_mapjoin3] (batchId=13) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[mm_all] (batchId=157) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[mm_cttas] (batchId=156) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[approx_distinct] (batchId=170) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[delete_tmp_table] (batchId=174) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_tmp_table] (batchId=161) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[lvj_mapjoin] (batchId=165) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[mm_exim] (batchId=186) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[strict_managed_tables1] (batchId=181) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[temp_table] (batchId=185) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_nway_join] (batchId=184) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr_2] (batchId=177) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_like_2] (batchId=183) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_udf2] (batchId=163) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_union_null] (batchId=163) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorized_insert_into_bucketed_table] (batchId=166) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorized_mapjoin3] (batchId=163) org.apache.hadoop.hive.ql.TestTxnCommands.testDeleteIn (batchId=341) org.apache.hadoop.hive.ql.TestTxnCommands.testMmExim (batchId=341) org.apache.hadoop.hive.ql.TestTxnCommands2.updateDeletePartitioned (batchId=322) org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.updateDeletePartitioned (batchId=336) org.apache.hadoop.hive.ql.TestTxnCommandsWithSplitUpdateAndVectorization.testDeleteIn (batchId=322) org.apache.hadoop.hive.ql.TestTxnCommandsWithSplitUpdateAndVectorization.testMmExim (batchId=322) org.apache.hadoop.hive.ql.TestTxnExIm.testImport (batchId=322) org.apache.hadoop.hive.ql.TestTxnExIm.testImportNoTarget (batchId=322) org.apache.hadoop.hive.ql.TestTxnExIm.testMM (batchId=322) org.apache.hadoop.hive.ql.TestTxnExIm.testMMCreate (batchId=322) org.apache.hadoop.hive.ql.TestTxnLoadData.loadData (batchId=298) org.apache.hadoop.hive.ql.TestTxnLoadData.loadDataNonAcid2AcidConversion (batchId=298) org.apache.hadoop.hive.ql.TestTxnLoadData.loadDataUpdate (batchId=298) org.apache.hadoop.hive.ql.TestTxnLoadData.testMultiStatement (batchId=298) org.apache.hadoop.hive.ql.TestTxnNoBuckets.testCompactStatsGather (batchId=322) org.apache.hadoop.hive.ql.TestTxnNoBuckets.testEmptyCompactionResult (batchId=322) org.apache.hadoop.hive.ql.TestTxnNoBucketsVectorized.testCompactStatsGather (batchId=322) org.apache.hadoop.hive.ql.TestTxnNoBucketsVectorized.testEmptyCompactionResult (batchId=322) org.apache.hadoop.hive.ql.io.TestAcidUtils.testObsoleteOriginals (batchId=310) org.apache.hadoop.hive.ql.parse.TestReplAcidTablesWithJsonMessage.testDumpAcidTableWithPartitionDirMissing (batchId=260) org.apache.hadoop.hive.ql.parse.TestReplAcidTablesWithJsonMessage.testDumpAcidTableWithTableDirMissing (batchId=260) org.apache.hadoop.hive.ql.parse.TestReplAcidTablesWithJsonMessage.testMultiDBTxn (batchId=260) org.apache.hadoop.hive.ql.parse.TestReplicationScenariosAcidTables.testDumpAcidTableWithPartitionDirMissing (batchId=262) org.apache.hadoop.hive.ql.parse.TestReplicationScenariosAcidTables.testDumpAcidTableWithTableDirMissing (batchId=262) org.apache.hadoop.hive.ql.parse.TestReplicationScenariosAcidTables.testMultiDBTxn (batchId=262) org.apache.hadoop.hive.ql.parse.TestReplicationS
[jira] [Commented] (HIVE-21225) ACID: getAcidState() should cache a recursive dir listing locally
[ https://issues.apache.org/jira/browse/HIVE-21225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16880154#comment-16880154 ] Hive QA commented on HIVE-21225: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 50s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 39s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 48s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 4s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m 3s{color} | {color:blue} ql in master has 2252 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 39s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 25s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 27s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 8s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 46s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 47s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 47s{color} | {color:red} ql: The patch generated 21 new + 903 unchanged - 1 fixed = 924 total (was 904) {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s{color} | {color:red} The patch has 27 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m 22s{color} | {color:red} ql generated 1 new + 2252 unchanged - 0 fixed = 2253 total (was 2252) {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 25s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 13s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 31m 19s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | FindBugs | module:ql | | | Unread field:AcidUtils.java:[line 1409] | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-17904/dev-support/hive-personality.sh | | git revision | master / 67e515f | | Default Java | 1.8.0_111 | | findbugs | v3.0.0 | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-17904/yetus/diff-checkstyle-ql.txt | | whitespace | http://104.198.109.242/logs//PreCommit-HIVE-Build-17904/yetus/whitespace-eol.txt | | findbugs | http://104.198.109.242/logs//PreCommit-HIVE-Build-17904/yetus/new-findbugs-ql.html | | modules | C: ql itests/hive-unit U: . | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-17904/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > ACID: getAcidState() should cache a recursive dir listing locally > - > > Key: HIVE-21225 > URL: https://issues.apache.org/jira/browse/HIVE-21225 > Project: Hive > Issue Type: Improvement > Components: Transactions >Reporter: Gopal V >Assignee: Vaibhav Gumashta >Priority: Major > Attachments: HIVE-21225.1.patch, HIVE-21225.2.patch, > HIVE-21225.3.patch, HIVE-21225.4.patch, HIVE-21225.4.patch, > HIVE-21225.5.patch, HIVE-21225.6.patch, HIVE-21225.7.patch, > HIVE-212
[jira] [Commented] (HIVE-21225) ACID: getAcidState() should cache a recursive dir listing locally
[ https://issues.apache.org/jira/browse/HIVE-21225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16879827#comment-16879827 ] Hive QA commented on HIVE-21225: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12973847/HIVE-21225.7.patch {color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 59 failed/errored test(s), 16361 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[delete_tmp_table] (batchId=56) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[insert_values_tmp_table] (batchId=5) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mm_all] (batchId=74) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mm_cttas] (batchId=51) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[temp_table_options1] (batchId=3) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorized_mapjoin2] (batchId=26) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorized_mapjoin3] (batchId=13) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[mm_all] (batchId=157) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[mm_cttas] (batchId=156) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[approx_distinct] (batchId=170) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[delete_tmp_table] (batchId=174) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_tmp_table] (batchId=161) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[lvj_mapjoin] (batchId=165) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[mm_exim] (batchId=186) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[strict_managed_tables1] (batchId=181) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[temp_table] (batchId=185) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_nway_join] (batchId=184) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr_2] (batchId=177) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_like_2] (batchId=183) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_udf2] (batchId=163) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_union_null] (batchId=163) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorized_insert_into_bucketed_table] (batchId=166) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorized_mapjoin3] (batchId=163) org.apache.hadoop.hive.ql.TestTxnCommands.testDeleteIn (batchId=341) org.apache.hadoop.hive.ql.TestTxnCommands.testMmExim (batchId=341) org.apache.hadoop.hive.ql.TestTxnCommands2.updateDeletePartitioned (batchId=322) org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.updateDeletePartitioned (batchId=336) org.apache.hadoop.hive.ql.TestTxnCommandsWithSplitUpdateAndVectorization.testDeleteIn (batchId=322) org.apache.hadoop.hive.ql.TestTxnCommandsWithSplitUpdateAndVectorization.testMmExim (batchId=322) org.apache.hadoop.hive.ql.TestTxnExIm.testImport (batchId=322) org.apache.hadoop.hive.ql.TestTxnExIm.testImportNoTarget (batchId=322) org.apache.hadoop.hive.ql.TestTxnExIm.testMM (batchId=322) org.apache.hadoop.hive.ql.TestTxnExIm.testMMCreate (batchId=322) org.apache.hadoop.hive.ql.TestTxnLoadData.loadData (batchId=298) org.apache.hadoop.hive.ql.TestTxnLoadData.loadDataNonAcid2AcidConversion (batchId=298) org.apache.hadoop.hive.ql.TestTxnLoadData.loadDataUpdate (batchId=298) org.apache.hadoop.hive.ql.TestTxnLoadData.testMultiStatement (batchId=298) org.apache.hadoop.hive.ql.TestTxnNoBuckets.testCompactStatsGather (batchId=322) org.apache.hadoop.hive.ql.TestTxnNoBuckets.testEmptyCompactionResult (batchId=322) org.apache.hadoop.hive.ql.TestTxnNoBucketsVectorized.testCompactStatsGather (batchId=322) org.apache.hadoop.hive.ql.TestTxnNoBucketsVectorized.testEmptyCompactionResult (batchId=322) org.apache.hadoop.hive.ql.io.TestAcidUtils.testObsoleteOriginals (batchId=310) org.apache.hadoop.hive.ql.io.orc.TestInputOutputFormat.testACIDReaderFooterSerialize (batchId=313) org.apache.hadoop.hive.ql.io.orc.TestInputOutputFormat.testACIDReaderFooterSerializeWithDeltas (batchId=313) org.apache.hadoop.hive.ql.io.orc.TestInputOutputFormat.testACIDReaderNoFooterSerialize (batchId=313) org.apache.hadoop.hive.ql.io.orc.TestInputOutputFormat.testACIDReaderNoFooterSerializeWithDeltas (batchId=313) org.apache.hadoop.hive.ql.io.orc.TestInputOutputFormat.testSplitGenReadOps (batchId=313) org.apache.hadoop.hive.ql.io.orc.TestInputOutputFormat.testSplitGenReadOpsLocalCache (batchId=313) org.apache.hadoop.hive.ql.io.orc.TestInputOutputFormat.testSplitGenReadOpsLocalCacheChangeFileLen (batchId=3
[jira] [Commented] (HIVE-21225) ACID: getAcidState() should cache a recursive dir listing locally
[ https://issues.apache.org/jira/browse/HIVE-21225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16879818#comment-16879818 ] Hive QA commented on HIVE-21225: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 45s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m 18s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 50s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 3s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m 6s{color} | {color:blue} ql in master has 2252 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 39s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 23s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 27s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 9s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 47s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 47s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 45s{color} | {color:red} ql: The patch generated 19 new + 692 unchanged - 1 fixed = 711 total (was 693) {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s{color} | {color:red} The patch has 27 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m 12s{color} | {color:red} ql generated 1 new + 2252 unchanged - 0 fixed = 2253 total (was 2252) {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 27s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 15s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 30m 47s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | FindBugs | module:ql | | | Unread field:AcidUtils.java:[line 1409] | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-17893/dev-support/hive-personality.sh | | git revision | master / 71adb04 | | Default Java | 1.8.0_111 | | findbugs | v3.0.0 | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-17893/yetus/diff-checkstyle-ql.txt | | whitespace | http://104.198.109.242/logs//PreCommit-HIVE-Build-17893/yetus/whitespace-eol.txt | | findbugs | http://104.198.109.242/logs//PreCommit-HIVE-Build-17893/yetus/new-findbugs-ql.html | | modules | C: ql itests/hive-unit U: . | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-17893/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > ACID: getAcidState() should cache a recursive dir listing locally > - > > Key: HIVE-21225 > URL: https://issues.apache.org/jira/browse/HIVE-21225 > Project: Hive > Issue Type: Improvement > Components: Transactions >Reporter: Gopal V >Assignee: Vaibhav Gumashta >Priority: Major > Attachments: HIVE-21225.1.patch, HIVE-21225.2.patch, > HIVE-21225.3.patch, HIVE-21225.4.patch, HIVE-21225.4.patch, > HIVE-21225.5.patch, HIVE-21225.6.patch, HIVE-21225.7.patch, async-pid-4
[jira] [Commented] (HIVE-21225) ACID: getAcidState() should cache a recursive dir listing locally
[ https://issues.apache.org/jira/browse/HIVE-21225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16878650#comment-16878650 ] Hive QA commented on HIVE-21225: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12973614/HIVE-21225.6.patch {color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 59 failed/errored test(s), 16361 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_nullscan] (batchId=73) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_stats5] (batchId=23) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_4] (batchId=13) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mm_all] (batchId=74) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[row__id] (batchId=86) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[stats_nonpart] (batchId=14) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[stats_part2] (batchId=22) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[stats_part] (batchId=52) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[stats_sizebug] (batchId=89) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[acid_bucket_pruning] (batchId=155) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[mm_all] (batchId=157) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[mm_exim] (batchId=186) org.apache.hadoop.hive.ql.TestTxnCommands.testMmExim (batchId=341) org.apache.hadoop.hive.ql.TestTxnCommands.testNonAcidToAcidConversion01 (batchId=341) org.apache.hadoop.hive.ql.TestTxnCommands2.testNonAcidToAcidConversion02 (batchId=322) org.apache.hadoop.hive.ql.TestTxnCommands2.testNonAcidToAcidConversion1 (batchId=322) org.apache.hadoop.hive.ql.TestTxnCommands2.testNonAcidToAcidConversion2 (batchId=322) org.apache.hadoop.hive.ql.TestTxnCommands2.testNonAcidToAcidConversion3 (batchId=322) org.apache.hadoop.hive.ql.TestTxnCommands2.testOriginalFileReaderWhenNonAcidConvertedToAcid (batchId=322) org.apache.hadoop.hive.ql.TestTxnCommands2.updateDeletePartitioned (batchId=322) org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.testNonAcidToAcidConversion02 (batchId=336) org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.testNonAcidToAcidConversion1 (batchId=336) org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.testNonAcidToAcidConversion2 (batchId=336) org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.testNonAcidToAcidConversion3 (batchId=336) org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.testOriginalFileReaderWhenNonAcidConvertedToAcid (batchId=336) org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.updateDeletePartitioned (batchId=336) org.apache.hadoop.hive.ql.TestTxnCommandsWithSplitUpdateAndVectorization.testMmExim (batchId=322) org.apache.hadoop.hive.ql.TestTxnExIm.testImport (batchId=322) org.apache.hadoop.hive.ql.TestTxnExIm.testImportNoTarget (batchId=322) org.apache.hadoop.hive.ql.TestTxnExIm.testMM (batchId=322) org.apache.hadoop.hive.ql.TestTxnExIm.testMMCreate (batchId=322) org.apache.hadoop.hive.ql.TestTxnLoadData.loadData (batchId=298) org.apache.hadoop.hive.ql.TestTxnLoadData.loadDataNonAcid2AcidConversion (batchId=298) org.apache.hadoop.hive.ql.TestTxnLoadData.loadDataUpdate (batchId=298) org.apache.hadoop.hive.ql.TestTxnLoadData.testMultiStatement (batchId=298) org.apache.hadoop.hive.ql.TestTxnNoBuckets.testCompactStatsGather (batchId=322) org.apache.hadoop.hive.ql.TestTxnNoBuckets.testEmptyCompactionResult (batchId=322) org.apache.hadoop.hive.ql.TestTxnNoBuckets.testToAcidConversionMultiBucket (batchId=322) org.apache.hadoop.hive.ql.TestTxnNoBucketsVectorized.testCompactStatsGather (batchId=322) org.apache.hadoop.hive.ql.TestTxnNoBucketsVectorized.testEmptyCompactionResult (batchId=322) org.apache.hadoop.hive.ql.TestTxnNoBucketsVectorized.testToAcidConversionMultiBucket (batchId=322) org.apache.hadoop.hive.ql.io.orc.TestInputOutputFormat.testACIDReaderFooterSerialize (batchId=313) org.apache.hadoop.hive.ql.io.orc.TestInputOutputFormat.testACIDReaderFooterSerializeWithDeltas (batchId=313) org.apache.hadoop.hive.ql.io.orc.TestInputOutputFormat.testACIDReaderNoFooterSerialize (batchId=313) org.apache.hadoop.hive.ql.io.orc.TestInputOutputFormat.testACIDReaderNoFooterSerializeWithDeltas (batchId=313) org.apache.hadoop.hive.ql.io.orc.TestInputOutputFormat.testSplitGenReadOps (batchId=313) org.apache.hadoop.hive.ql.io.orc.TestInputOutputFormat.testSplitGenReadOpsLocalCache (batchId=313) org.apache.hadoop.hive.ql.io.orc.TestInputOutputFormat.testSplitGenReadOpsLocalCacheChangeFileLen (batchId=313) org.apache.hadoop.hive.ql.io.orc.TestInputOutputFormat.testSplitGenReadOpsLocalCacheCh
[jira] [Commented] (HIVE-21225) ACID: getAcidState() should cache a recursive dir listing locally
[ https://issues.apache.org/jira/browse/HIVE-21225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16878620#comment-16878620 ] Hive QA commented on HIVE-21225: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 44s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 5s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 48s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 57s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m 6s{color} | {color:blue} ql in master has 2253 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 41s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 23s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 27s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 8s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 46s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 46s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 40s{color} | {color:red} ql: The patch generated 16 new + 169 unchanged - 1 fixed = 185 total (was 170) {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s{color} | {color:red} The patch has 26 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m 12s{color} | {color:red} ql generated 1 new + 2253 unchanged - 0 fixed = 2254 total (was 2253) {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 23s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 13s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 30m 17s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | FindBugs | module:ql | | | Unread field:AcidUtils.java:[line 1402] | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-17849/dev-support/hive-personality.sh | | git revision | master / 3223bda | | Default Java | 1.8.0_111 | | findbugs | v3.0.0 | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-17849/yetus/diff-checkstyle-ql.txt | | whitespace | http://104.198.109.242/logs//PreCommit-HIVE-Build-17849/yetus/whitespace-eol.txt | | findbugs | http://104.198.109.242/logs//PreCommit-HIVE-Build-17849/yetus/new-findbugs-ql.html | | modules | C: ql itests/hive-unit U: . | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-17849/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > ACID: getAcidState() should cache a recursive dir listing locally > - > > Key: HIVE-21225 > URL: https://issues.apache.org/jira/browse/HIVE-21225 > Project: Hive > Issue Type: Improvement > Components: Transactions >Reporter: Gopal V >Assignee: Vaibhav Gumashta >Priority: Major > Attachments: HIVE-21225.1.patch, HIVE-21225.2.patch, > HIVE-21225.3.patch, HIVE-21225.4.patch, HIVE-21225.4.patch, > HIVE-21225.5.patch, HIVE-21225.6.patch, async-pid-44-2.svg > > > Curren
[jira] [Commented] (HIVE-21225) ACID: getAcidState() should cache a recursive dir listing locally
[ https://issues.apache.org/jira/browse/HIVE-21225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16874707#comment-16874707 ] Hive QA commented on HIVE-21225: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12973126/HIVE-21225.5.patch {color:red}ERROR:{color} -1 due to build exiting with an error Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/17785/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17785/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17785/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Tests exited with: Exception: Patch URL https://issues.apache.org/jira/secure/attachment/12973126/HIVE-21225.5.patch was found in seen patch url's cache and a test was probably run already on it. Aborting... {noformat} This message is automatically generated. ATTACHMENT ID: 12973126 - PreCommit-HIVE-Build > ACID: getAcidState() should cache a recursive dir listing locally > - > > Key: HIVE-21225 > URL: https://issues.apache.org/jira/browse/HIVE-21225 > Project: Hive > Issue Type: Improvement > Components: Transactions >Reporter: Gopal V >Assignee: Vaibhav Gumashta >Priority: Major > Attachments: HIVE-21225.1.patch, HIVE-21225.2.patch, > HIVE-21225.3.patch, HIVE-21225.4.patch, HIVE-21225.4.patch, > HIVE-21225.5.patch, async-pid-44-2.svg > > > Currently getAcidState() makes 3 calls into the FS api which could be > answered by making a single recursive listDir call and reusing the same data > to check for isRawFormat() and isValidBase(). > All delta operations for a single partition can go against a single listed > directory snapshot instead of interacting with the NameNode or ObjectStore > within the inner loop. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21225) ACID: getAcidState() should cache a recursive dir listing locally
[ https://issues.apache.org/jira/browse/HIVE-21225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16874596#comment-16874596 ] Hive QA commented on HIVE-21225: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12973126/HIVE-21225.5.patch {color:red}ERROR:{color} -1 due to build exiting with an error Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/17781/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17781/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17781/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Tests exited with: NonZeroExitCodeException Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ date '+%Y-%m-%d %T.%3N' 2019-06-28 01:52:29.427 + [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]] + export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 + JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 + export PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games + PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games + export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m ' + ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m ' + export 'MAVEN_OPTS=-Xmx1g ' + MAVEN_OPTS='-Xmx1g ' + cd /data/hiveptest/working/ + tee /data/hiveptest/logs/PreCommit-HIVE-Build-17781/source-prep.txt + [[ false == \t\r\u\e ]] + mkdir -p maven ivy + [[ git = \s\v\n ]] + [[ git = \g\i\t ]] + [[ -z master ]] + [[ -d apache-github-source-source ]] + [[ ! -d apache-github-source-source/.git ]] + [[ ! -d apache-github-source-source ]] + date '+%Y-%m-%d %T.%3N' 2019-06-28 01:52:29.431 + cd apache-github-source-source + git fetch origin + git reset --hard HEAD HEAD is now at 57c4217 HIVE-15177: Authentication with hive fails when kerberos auth type is set to fromSubject and principal contains _HOST (Oliver Draese, reviewed by Gopal V) + git clean -f -d + git checkout master Already on 'master' Your branch is up-to-date with 'origin/master'. + git reset --hard origin/master HEAD is now at 57c4217 HIVE-15177: Authentication with hive fails when kerberos auth type is set to fromSubject and principal contains _HOST (Oliver Draese, reviewed by Gopal V) + git merge --ff-only origin/master Already up-to-date. + date '+%Y-%m-%d %T.%3N' 2019-06-28 01:52:30.881 + rm -rf ../yetus_PreCommit-HIVE-Build-17781 + mkdir ../yetus_PreCommit-HIVE-Build-17781 + git gc + cp -R . ../yetus_PreCommit-HIVE-Build-17781 + mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-17781/yetus + patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh + patchFilePath=/data/hiveptest/working/scratch/build.patch + [[ -f /data/hiveptest/working/scratch/build.patch ]] + chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh + /data/hiveptest/working/scratch/smart-apply-patch.sh /data/hiveptest/working/scratch/build.patch error: a/itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCrudCompactorOnTez.java: does not exist in index error: a/ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java: does not exist in index error: a/ql/src/java/org/apache/hadoop/hive/ql/io/HdfsUtils.java: does not exist in index error: a/ql/src/test/org/apache/hadoop/hive/ql/io/TestAcidUtils.java: does not exist in index Going to apply patch with: git apply -p1 /data/hiveptest/working/scratch/build.patch:10: trailing whitespace. /data/hiveptest/working/scratch/build.patch:46: trailing whitespace. /data/hiveptest/working/scratch/build.patch:94: trailing whitespace. // Okay, we're going to need these originals. /data/hiveptest/working/scratch/build.patch:109: trailing whitespace. /data/hiveptest/working/scratch/build.patch:124: trailing whitespace. } warning: squelched 21 whitespace errors warning: 26 lines add whitespace errors. + [[ maven == \m\a\v\e\n ]] + rm -rf /data/hiveptest/working/maven/org/apache/hive + mvn -B clean install -DskipTests -T 4 -q -Dmaven.repo.local=/data/hiveptest/working/maven protoc-jar: executing: [/tmp/protoc2011211136641073270.exe, --version] protoc-jar: executing: [/tmp/protoc2011211136641073270.exe, -I/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/src/main/protobuf/org/apache/hadoop/hive/metastore, --java_out=/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/target/generated-sources, /data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/src/main/protobuf/org/apache/hadoop/hive/metastore/metastore.proto] libprotoc 2.5.0 ANTLR Parser Generator Version 3.5.2 protoc-jar: executing: [/tmp/protoc3865194964592695274.exe, --version] libprotoc 2.5.0 ANTLR Parser Generator Version
[jira] [Commented] (HIVE-21225) ACID: getAcidState() should cache a recursive dir listing locally
[ https://issues.apache.org/jira/browse/HIVE-21225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16872752#comment-16872752 ] Hive QA commented on HIVE-21225: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12972852/HIVE-21225.4.patch {color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 60 failed/errored test(s), 16339 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_nullscan] (batchId=73) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_stats5] (batchId=23) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_4] (batchId=13) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mm_all] (batchId=74) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[row__id] (batchId=86) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[stats_nonpart] (batchId=14) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[stats_part2] (batchId=22) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[stats_part] (batchId=52) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[stats_sizebug] (batchId=89) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[acid_bucket_pruning] (batchId=155) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[mm_all] (batchId=157) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[mm_exim] (batchId=186) org.apache.hadoop.hive.ql.TestTxnCommands.testMmExim (batchId=341) org.apache.hadoop.hive.ql.TestTxnCommands.testNonAcidToAcidConversion01 (batchId=341) org.apache.hadoop.hive.ql.TestTxnCommands2.testNonAcidToAcidConversion02 (batchId=322) org.apache.hadoop.hive.ql.TestTxnCommands2.testNonAcidToAcidConversion1 (batchId=322) org.apache.hadoop.hive.ql.TestTxnCommands2.testNonAcidToAcidConversion2 (batchId=322) org.apache.hadoop.hive.ql.TestTxnCommands2.testNonAcidToAcidConversion3 (batchId=322) org.apache.hadoop.hive.ql.TestTxnCommands2.testOriginalFileReaderWhenNonAcidConvertedToAcid (batchId=322) org.apache.hadoop.hive.ql.TestTxnCommands2.updateDeletePartitioned (batchId=322) org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.testNonAcidToAcidConversion02 (batchId=336) org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.testNonAcidToAcidConversion1 (batchId=336) org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.testNonAcidToAcidConversion2 (batchId=336) org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.testNonAcidToAcidConversion3 (batchId=336) org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.testOriginalFileReaderWhenNonAcidConvertedToAcid (batchId=336) org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.updateDeletePartitioned (batchId=336) org.apache.hadoop.hive.ql.TestTxnCommandsWithSplitUpdateAndVectorization.testMmExim (batchId=322) org.apache.hadoop.hive.ql.TestTxnExIm.testImport (batchId=322) org.apache.hadoop.hive.ql.TestTxnExIm.testImportNoTarget (batchId=322) org.apache.hadoop.hive.ql.TestTxnExIm.testMM (batchId=322) org.apache.hadoop.hive.ql.TestTxnExIm.testMMCreate (batchId=322) org.apache.hadoop.hive.ql.TestTxnLoadData.loadData (batchId=298) org.apache.hadoop.hive.ql.TestTxnLoadData.loadDataNonAcid2AcidConversion (batchId=298) org.apache.hadoop.hive.ql.TestTxnLoadData.loadDataUpdate (batchId=298) org.apache.hadoop.hive.ql.TestTxnLoadData.testMultiStatement (batchId=298) org.apache.hadoop.hive.ql.TestTxnNoBuckets.testCompactStatsGather (batchId=322) org.apache.hadoop.hive.ql.TestTxnNoBuckets.testEmptyCompactionResult (batchId=322) org.apache.hadoop.hive.ql.TestTxnNoBuckets.testToAcidConversionMultiBucket (batchId=322) org.apache.hadoop.hive.ql.TestTxnNoBucketsVectorized.testCompactStatsGather (batchId=322) org.apache.hadoop.hive.ql.TestTxnNoBucketsVectorized.testEmptyCompactionResult (batchId=322) org.apache.hadoop.hive.ql.TestTxnNoBucketsVectorized.testToAcidConversionMultiBucket (batchId=322) org.apache.hadoop.hive.ql.io.TestAcidUtils.testObsoleteOriginals (batchId=310) org.apache.hadoop.hive.ql.io.orc.TestInputOutputFormat.testACIDReaderFooterSerialize (batchId=313) org.apache.hadoop.hive.ql.io.orc.TestInputOutputFormat.testACIDReaderFooterSerializeWithDeltas (batchId=313) org.apache.hadoop.hive.ql.io.orc.TestInputOutputFormat.testACIDReaderNoFooterSerialize (batchId=313) org.apache.hadoop.hive.ql.io.orc.TestInputOutputFormat.testACIDReaderNoFooterSerializeWithDeltas (batchId=313) org.apache.hadoop.hive.ql.io.orc.TestInputOutputFormat.testSplitGenReadOps (batchId=313) org.apache.hadoop.hive.ql.io.orc.TestInputOutputFormat.testSplitGenReadOpsLocalCache (batchId=313) org.apache.hadoop.hive.ql.io.orc.TestInputOutputFormat.testSplitGenReadOpsLocalCacheChangeFileLen (batchId=313) org.apa
[jira] [Commented] (HIVE-21225) ACID: getAcidState() should cache a recursive dir listing locally
[ https://issues.apache.org/jira/browse/HIVE-21225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16872724#comment-16872724 ] Hive QA commented on HIVE-21225: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 1s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 52s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 23s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 54s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 2s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m 13s{color} | {color:blue} ql in master has 2253 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 41s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 26s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 30s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 17s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 52s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 52s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 43s{color} | {color:red} ql: The patch generated 16 new + 169 unchanged - 1 fixed = 185 total (was 170) {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s{color} | {color:red} The patch has 26 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m 21s{color} | {color:red} ql generated 1 new + 2253 unchanged - 0 fixed = 2254 total (was 2253) {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 27s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 15s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 31m 54s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | FindBugs | module:ql | | | Unread field:AcidUtils.java:[line 1400] | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-17734/dev-support/hive-personality.sh | | git revision | master / aed7500 | | Default Java | 1.8.0_111 | | findbugs | v3.0.0 | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-17734/yetus/diff-checkstyle-ql.txt | | whitespace | http://104.198.109.242/logs//PreCommit-HIVE-Build-17734/yetus/whitespace-eol.txt | | findbugs | http://104.198.109.242/logs//PreCommit-HIVE-Build-17734/yetus/new-findbugs-ql.html | | modules | C: ql itests/hive-unit U: . | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-17734/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > ACID: getAcidState() should cache a recursive dir listing locally > - > > Key: HIVE-21225 > URL: https://issues.apache.org/jira/browse/HIVE-21225 > Project: Hive > Issue Type: Improvement > Components: Transactions >Reporter: Gopal V >Assignee: Vaibhav Gumashta >Priority: Major > Attachments: HIVE-21225.1.patch, HIVE-21225.2.patch, > HIVE-21225.3.patch, HIVE-21225.4.patch, HIVE-21225.4.patch, async-pid-44-2.svg > > > Currently getAcidState() makes 3 calls into the F
[jira] [Commented] (HIVE-21225) ACID: getAcidState() should cache a recursive dir listing locally
[ https://issues.apache.org/jira/browse/HIVE-21225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16872237#comment-16872237 ] Hive QA commented on HIVE-21225: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12972793/HIVE-21225.4.patch {color:red}ERROR:{color} -1 due to build exiting with an error Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/17724/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17724/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17724/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Tests exited with: NonZeroExitCodeException Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ date '+%Y-%m-%d %T.%3N' 2019-06-25 10:21:10.591 + [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]] + export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 + JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 + export PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games + PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games + export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m ' + ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m ' + export 'MAVEN_OPTS=-Xmx1g ' + MAVEN_OPTS='-Xmx1g ' + cd /data/hiveptest/working/ + tee /data/hiveptest/logs/PreCommit-HIVE-Build-17724/source-prep.txt + [[ false == \t\r\u\e ]] + mkdir -p maven ivy + [[ git = \s\v\n ]] + [[ git = \g\i\t ]] + [[ -z master ]] + [[ -d apache-github-source-source ]] + [[ ! -d apache-github-source-source/.git ]] + [[ ! -d apache-github-source-source ]] + date '+%Y-%m-%d %T.%3N' 2019-06-25 10:21:10.595 + cd apache-github-source-source + git fetch origin + git reset --hard HEAD HEAD is now at 84b5ba7 HIVE-21913: GenericUDTFGetSplits should handle usernames in the same way as LLAP (Prasanth Jayachandran reviewed by Jason Dere) + git clean -f -d Removing standalone-metastore/metastore-server/src/gen/ + git checkout master Already on 'master' Your branch is up-to-date with 'origin/master'. + git reset --hard origin/master HEAD is now at 84b5ba7 HIVE-21913: GenericUDTFGetSplits should handle usernames in the same way as LLAP (Prasanth Jayachandran reviewed by Jason Dere) + git merge --ff-only origin/master Already up-to-date. + date '+%Y-%m-%d %T.%3N' 2019-06-25 10:21:11.747 + rm -rf ../yetus_PreCommit-HIVE-Build-17724 + mkdir ../yetus_PreCommit-HIVE-Build-17724 + git gc + cp -R . ../yetus_PreCommit-HIVE-Build-17724 + mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-17724/yetus + patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh + patchFilePath=/data/hiveptest/working/scratch/build.patch + [[ -f /data/hiveptest/working/scratch/build.patch ]] + chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh + /data/hiveptest/working/scratch/smart-apply-patch.sh /data/hiveptest/working/scratch/build.patch error: a/itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCrudCompactorOnTez.java: does not exist in index error: a/ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java: does not exist in index error: a/ql/src/java/org/apache/hadoop/hive/ql/io/HdfsUtils.java: does not exist in index error: a/ql/src/test/org/apache/hadoop/hive/ql/io/TestAcidUtils.java: does not exist in index Going to apply patch with: git apply -p1 /data/hiveptest/working/scratch/build.patch:10: trailing whitespace. /data/hiveptest/working/scratch/build.patch:46: trailing whitespace. /data/hiveptest/working/scratch/build.patch:85: trailing whitespace. // Okay, we're going to need these originals. /data/hiveptest/working/scratch/build.patch:100: trailing whitespace. /data/hiveptest/working/scratch/build.patch:115: trailing whitespace. } warning: squelched 21 whitespace errors warning: 26 lines add whitespace errors. + [[ maven == \m\a\v\e\n ]] + rm -rf /data/hiveptest/working/maven/org/apache/hive + mvn -B clean install -DskipTests -T 4 -q -Dmaven.repo.local=/data/hiveptest/working/maven protoc-jar: executing: [/tmp/protoc2366243295147497200.exe, --version] libprotoc 2.5.0 protoc-jar: executing: [/tmp/protoc2366243295147497200.exe, -I/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/src/main/protobuf/org/apache/hadoop/hive/metastore, --java_out=/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/target/generated-sources, /data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/src/main/protobuf/org/apache/hadoop/hive/metastore/metastore.proto] ANTLR Parser Generator Version 3.5.2 protoc-jar: executing: [/tmp/protoc8182810356046681495.exe, --version] libprotoc 2.5.0 ANTLR Parser Generator Version
[jira] [Commented] (HIVE-21225) ACID: getAcidState() should cache a recursive dir listing locally
[ https://issues.apache.org/jira/browse/HIVE-21225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16871097#comment-16871097 ] Hive QA commented on HIVE-21225: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12972698/HIVE-21225.3.patch {color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 64 failed/errored test(s), 16339 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_nullscan] (batchId=73) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_stats5] (batchId=23) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_4] (batchId=13) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mm_all] (batchId=74) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[row__id] (batchId=86) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[stats_nonpart] (batchId=14) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[stats_part2] (batchId=22) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[stats_part] (batchId=52) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[stats_sizebug] (batchId=89) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[acid_bucket_pruning] (batchId=155) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[mm_all] (batchId=157) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[mm_exim] (batchId=186) org.apache.hadoop.hive.ql.TestTxnCommands.testMmExim (batchId=341) org.apache.hadoop.hive.ql.TestTxnCommands2.testNonAcidToAcidConversion1 (batchId=322) org.apache.hadoop.hive.ql.TestTxnCommands2.testNonAcidToAcidConversion2 (batchId=322) org.apache.hadoop.hive.ql.TestTxnCommands2.testNonAcidToAcidConversion3 (batchId=322) org.apache.hadoop.hive.ql.TestTxnCommands2.testOriginalFileReaderWhenNonAcidConvertedToAcid (batchId=322) org.apache.hadoop.hive.ql.TestTxnCommands2.updateDeletePartitioned (batchId=322) org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.testNonAcidToAcidConversion1 (batchId=336) org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.testNonAcidToAcidConversion2 (batchId=336) org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.testNonAcidToAcidConversion3 (batchId=336) org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.testOriginalFileReaderWhenNonAcidConvertedToAcid (batchId=336) org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.updateDeletePartitioned (batchId=336) org.apache.hadoop.hive.ql.TestTxnCommandsWithSplitUpdateAndVectorization.testMmExim (batchId=322) org.apache.hadoop.hive.ql.TestTxnExIm.testImport (batchId=322) org.apache.hadoop.hive.ql.TestTxnExIm.testImportNoTarget (batchId=322) org.apache.hadoop.hive.ql.TestTxnExIm.testMM (batchId=322) org.apache.hadoop.hive.ql.TestTxnExIm.testMMCreate (batchId=322) org.apache.hadoop.hive.ql.TestTxnLoadData.loadData (batchId=298) org.apache.hadoop.hive.ql.TestTxnLoadData.loadDataNonAcid2AcidConversion (batchId=298) org.apache.hadoop.hive.ql.TestTxnLoadData.loadDataUpdate (batchId=298) org.apache.hadoop.hive.ql.TestTxnLoadData.testMultiStatement (batchId=298) org.apache.hadoop.hive.ql.TestTxnNoBuckets.testCompactStatsGather (batchId=322) org.apache.hadoop.hive.ql.TestTxnNoBuckets.testEmptyCompactionResult (batchId=322) org.apache.hadoop.hive.ql.TestTxnNoBuckets.testToAcidConversionMultiBucket (batchId=322) org.apache.hadoop.hive.ql.TestTxnNoBucketsVectorized.testCompactStatsGather (batchId=322) org.apache.hadoop.hive.ql.TestTxnNoBucketsVectorized.testEmptyCompactionResult (batchId=322) org.apache.hadoop.hive.ql.TestTxnNoBucketsVectorized.testToAcidConversionMultiBucket (batchId=322) org.apache.hadoop.hive.ql.io.TestAcidUtils.testBaseDeltas (batchId=310) org.apache.hadoop.hive.ql.io.TestAcidUtils.testBaseWithDeleteDeltas (batchId=310) org.apache.hadoop.hive.ql.io.TestAcidUtils.testObsoleteOriginals (batchId=310) org.apache.hadoop.hive.ql.io.TestAcidUtils.testOriginal (batchId=310) org.apache.hadoop.hive.ql.io.TestAcidUtils.testOriginalDeltas (batchId=310) org.apache.hadoop.hive.ql.io.orc.TestInputOutputFormat.testACIDReaderFooterSerialize (batchId=313) org.apache.hadoop.hive.ql.io.orc.TestInputOutputFormat.testACIDReaderFooterSerializeWithDeltas (batchId=313) org.apache.hadoop.hive.ql.io.orc.TestInputOutputFormat.testACIDReaderNoFooterSerialize (batchId=313) org.apache.hadoop.hive.ql.io.orc.TestInputOutputFormat.testACIDReaderNoFooterSerializeWithDeltas (batchId=313) org.apache.hadoop.hive.ql.io.orc.TestInputOutputFormat.testEmptyFile (batchId=313) org.apache.hadoop.hive.ql.io.orc.TestInputOutputFormat.testSplitGenReadOps (batchId=313) org.apache.hadoop.hive.ql.io.orc.TestInputOutputFormat.testSplitGenReadOpsLocalCache (batchId=313) org.apache.hadoop.hive.ql.io.
[jira] [Commented] (HIVE-21225) ACID: getAcidState() should cache a recursive dir listing locally
[ https://issues.apache.org/jira/browse/HIVE-21225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16871068#comment-16871068 ] Hive QA commented on HIVE-21225: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 52s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 26s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 55s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 1s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m 14s{color} | {color:blue} ql in master has 2254 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 44s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 30s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 30s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 10s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 53s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 53s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 42s{color} | {color:red} ql: The patch generated 16 new + 169 unchanged - 1 fixed = 185 total (was 170) {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s{color} | {color:red} The patch has 26 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m 27s{color} | {color:red} ql generated 1 new + 2254 unchanged - 0 fixed = 2255 total (was 2254) {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 28s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 15s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 32m 5s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | FindBugs | module:ql | | | Unread field:AcidUtils.java:[line 1398] | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-17705/dev-support/hive-personality.sh | | git revision | master / 11f7856 | | Default Java | 1.8.0_111 | | findbugs | v3.0.0 | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-17705/yetus/diff-checkstyle-ql.txt | | whitespace | http://104.198.109.242/logs//PreCommit-HIVE-Build-17705/yetus/whitespace-eol.txt | | findbugs | http://104.198.109.242/logs//PreCommit-HIVE-Build-17705/yetus/new-findbugs-ql.html | | modules | C: ql itests/hive-unit U: . | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-17705/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > ACID: getAcidState() should cache a recursive dir listing locally > - > > Key: HIVE-21225 > URL: https://issues.apache.org/jira/browse/HIVE-21225 > Project: Hive > Issue Type: Improvement > Components: Transactions >Reporter: Gopal V >Assignee: Vaibhav Gumashta >Priority: Major > Attachments: HIVE-21225.1.patch, HIVE-21225.2.patch, > HIVE-21225.3.patch, async-pid-44-2.svg > > > Currently getAcidState() makes 3 calls into the FS api which could be > answered by maki
[jira] [Commented] (HIVE-21225) ACID: getAcidState() should cache a recursive dir listing locally
[ https://issues.apache.org/jira/browse/HIVE-21225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16861948#comment-16861948 ] Hive QA commented on HIVE-21225: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12971494/HIVE-21225.3.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:red}ERROR:{color} -1 due to 338 failed/errored test(s), 16051 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[orc_buckets] (batchId=289) org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[orc_format_nonpart] (batchId=289) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_insert_overwrite] (batchId=53) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_nullscan] (batchId=73) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_stats5] (batchId=23) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_table_directories_test] (batchId=38) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_table_stats] (batchId=58) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_4] (batchId=13) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[delete_orig_table] (batchId=43) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mm_all] (batchId=74) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[orc_merge11] (batchId=43) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[orc_merge13] (batchId=94) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[orc_merge1] (batchId=22) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[orc_merge_incompat1] (batchId=74) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[orc_merge_incompat2] (batchId=92) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[orc_ppd_same_table_multiple_aliases] (batchId=11) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[orc_ppd_schema_evol_1b] (batchId=37) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[orc_ppd_schema_evol_2b] (batchId=16) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[row__id] (batchId=86) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[stats_nonpart] (batchId=14) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[stats_part2] (batchId=22) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[stats_part] (batchId=52) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[stats_sizebug] (batchId=89) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[update_orig_table] (batchId=67) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_between_columns] (batchId=76) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_coalesce_3] (batchId=63) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_delete_orig_table] (batchId=2) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[acid_bucket_pruning] (batchId=155) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[mm_all] (batchId=157) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[mm_dp] (batchId=158) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[acid_vectorization_original] (batchId=182) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[alter_merge_2_orc] (batchId=181) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[alter_merge_orc] (batchId=178) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[approx_distinct] (batchId=170) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez2] (batchId=163) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[check_constraint] (batchId=164) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[dynpart_sort_opt_vectorization] (batchId=174) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[dynpart_sort_optimization2] (batchId=160) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_into_default_keyword] (batchId=160) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_overwrite] (batchId=165) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid] (batchId=183) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_decimal64_reader] (batchId=173) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_smb] (batchId=186) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_uncompressed] (batchId=176) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[load_data_using_job] (batchId=163) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_create] (batchId=180) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_create_rewrite_2] (batchId=186) org.apache.ha
[jira] [Commented] (HIVE-21225) ACID: getAcidState() should cache a recursive dir listing locally
[ https://issues.apache.org/jira/browse/HIVE-21225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16861907#comment-16861907 ] Hive QA commented on HIVE-21225: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m 45s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 10s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 41s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m 7s{color} | {color:blue} ql in master has 2260 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 1s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 28s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 9s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 9s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 43s{color} | {color:red} ql: The patch generated 10 new + 121 unchanged - 0 fixed = 131 total (was 121) {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s{color} | {color:red} The patch has 18 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m 24s{color} | {color:red} ql generated 1 new + 2260 unchanged - 0 fixed = 2261 total (was 2260) {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 3s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 15s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 25m 25s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | FindBugs | module:ql | | | Unread field:HdfsUtils.java:[line 155] | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-17533/dev-support/hive-personality.sh | | git revision | master / f4be42c | | Default Java | 1.8.0_111 | | findbugs | v3.0.0 | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-17533/yetus/diff-checkstyle-ql.txt | | whitespace | http://104.198.109.242/logs//PreCommit-HIVE-Build-17533/yetus/whitespace-eol.txt | | findbugs | http://104.198.109.242/logs//PreCommit-HIVE-Build-17533/yetus/new-findbugs-ql.html | | modules | C: ql U: ql | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-17533/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > ACID: getAcidState() should cache a recursive dir listing locally > - > > Key: HIVE-21225 > URL: https://issues.apache.org/jira/browse/HIVE-21225 > Project: Hive > Issue Type: Improvement > Components: Transactions >Reporter: Gopal V >Assignee: Vaibhav Gumashta >Priority: Major > Attachments: HIVE-21225.1.patch, HIVE-21225.2.patch, > HIVE-21225.3.patch, async-pid-44-2.svg > > > Currently getAcidState() makes 3 calls into the FS api which could be > answered by making a single recursive listDir call and reusing the same data > to check for isRawFormat() and isValidBase(). > All delta operations for a single partition can go against a single listed > directory snapshot instead of interacting with the NameNode or ObjectStore > within the inner loop. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21225) ACID: getAcidState() should cache a recursive dir listing locally
[ https://issues.apache.org/jira/browse/HIVE-21225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16861234#comment-16861234 ] Vaibhav Gumashta commented on HIVE-21225: - DFSClient, which returns HdfsLocatedFileStatus which is needed for file-id, does not support recursive listing. Modified the code to get recursive listing for non file-id cases (which should cover cloud object stores etc). > ACID: getAcidState() should cache a recursive dir listing locally > - > > Key: HIVE-21225 > URL: https://issues.apache.org/jira/browse/HIVE-21225 > Project: Hive > Issue Type: Improvement > Components: Transactions >Reporter: Gopal V >Assignee: Vaibhav Gumashta >Priority: Major > Attachments: HIVE-21225.1.patch, HIVE-21225.2.patch, > HIVE-21225.3.patch, async-pid-44-2.svg > > > Currently getAcidState() makes 3 calls into the FS api which could be > answered by making a single recursive listDir call and reusing the same data > to check for isRawFormat() and isValidBase(). > All delta operations for a single partition can go against a single listed > directory snapshot instead of interacting with the NameNode or ObjectStore > within the inner loop. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21225) ACID: getAcidState() should cache a recursive dir listing locally
[ https://issues.apache.org/jira/browse/HIVE-21225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16856492#comment-16856492 ] Hive QA commented on HIVE-21225: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12970871/HIVE-21225.2.patch {color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 1132 failed/errored test(s), 16078 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[materialized_view_create_rewrite] (batchId=286) org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[orc_buckets] (batchId=289) org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[orc_format_nonpart] (batchId=289) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_insert_overwrite] (batchId=53) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_join] (batchId=17) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_mapjoin] (batchId=12) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_nullscan] (batchId=73) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_stats2] (batchId=51) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_stats3] (batchId=16) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_stats4] (batchId=67) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_stats5] (batchId=23) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_stats] (batchId=43) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_subquery] (batchId=43) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_table_directories_test] (batchId=38) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_table_stats] (batchId=58) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_vectorization] (batchId=71) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_vectorization_partition] (batchId=79) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_vectorization_project] (batchId=21) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_view_delete] (batchId=36) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[authorization_delete] (batchId=91) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[authorization_delete_own_table] (batchId=73) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[authorization_update] (batchId=9) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[authorization_update_own_table] (batchId=85) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_4] (batchId=13) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucket_if_with_path_filter] (batchId=7) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[char_serde] (batchId=59) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[convert_decimal64_to_decimal] (batchId=54) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[create_transactional_full_acid] (batchId=82) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[create_transactional_insert_only] (batchId=64) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ctas] (batchId=7) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[date_serde] (batchId=49) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[dbtxnmgr_showlocks] (batchId=87) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[decimal_join2] (batchId=43) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[delete_all_non_partitioned] (batchId=31) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[delete_all_partitioned] (batchId=31) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[delete_orig_table] (batchId=43) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[delete_tmp_table] (batchId=56) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[delete_where_no_match] (batchId=31) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[delete_where_non_partitioned] (batchId=42) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[delete_where_partitioned] (batchId=44) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[delete_whole_partition] (batchId=10) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[dynamic_partition_insert] (batchId=61) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[dynpart_sort_opt_bucketing] (batchId=95) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[empty_array] (batchId=73) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[explain_locks] (batchId=48) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[implicit_cast_during_insert] (batchId=56) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[insert_acid_dynamic_partition] (batchId=22) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[insert_acid_not_bucketed] (batchId=76) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[insert_nonacid_from_acid]
[jira] [Commented] (HIVE-21225) ACID: getAcidState() should cache a recursive dir listing locally
[ https://issues.apache.org/jira/browse/HIVE-21225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16856469#comment-16856469 ] Hive QA commented on HIVE-21225: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m 17s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m 47s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 0s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 19s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 22s{color} | {color:blue} shims/common in master has 6 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 27s{color} | {color:blue} shims/0.23 in master has 7 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 5m 9s{color} | {color:blue} ql in master has 2256 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 38s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 45s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 19s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 53s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 53s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 12s{color} | {color:red} shims/common: The patch generated 4 new + 94 unchanged - 0 fixed = 98 total (was 94) {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 12s{color} | {color:red} shims/0.23: The patch generated 1 new + 65 unchanged - 0 fixed = 66 total (was 65) {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 57s{color} | {color:red} ql: The patch generated 6 new + 521 unchanged - 9 fixed = 527 total (was 530) {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s{color} | {color:red} The patch has 19 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 41s{color} | {color:red} shims/0.23 generated 1 new + 7 unchanged - 0 fixed = 8 total (was 7) {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 5m 23s{color} | {color:red} ql generated 1 new + 2256 unchanged - 0 fixed = 2257 total (was 2256) {color} | | {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 1m 11s{color} | {color:red} ql generated 1 new + 99 unchanged - 1 fixed = 100 total (was 100) {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 19s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 37m 54s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | FindBugs | module:shims/0.23 | | | Should org.apache.hadoop.hive.shims.Hadoop23Shims$HdfsDirSnapshotImpl be a _static_ inner class? At Hadoop23Shims.java:inner class? At Hadoop23Shims.java:[lines 771-796] | | FindBugs | module:ql | | | Non-virtual method call in org.apache.hadoop.hive.ql.io.AcidUtils.getAcidState(FileSystem, Path, Configuration, ValidWriteIdList, Ref, boolean, Map) passes null for non-null parameter of getChildState(FileStatus, HadoopShims$HdfsDirSnapshot, ValidWriteIdList, List, List, List, List, AcidUtils$TxnBase, boolean, List, Map, FileSystem, ValidTxnList) At AcidUtils.java:Path, Configuration, ValidWriteIdList, Ref, boolean, Map) passes null for non-null parameter of getChildState(FileStatus, HadoopShims$HdfsDirSnapshot, ValidWriteIdList, List, List, List, List, AcidUtils$TxnBase, boolean, List, Map, FileSystem, ValidTxnList) At AcidUtils.java:[line 1210] | \\ \\ || Subsystem || Report/
[jira] [Commented] (HIVE-21225) ACID: getAcidState() should cache a recursive dir listing locally
[ https://issues.apache.org/jira/browse/HIVE-21225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16856184#comment-16856184 ] Gopal V commented on HIVE-21225: The DirSnapshot can also implement .exists(), so that you can avoid having methods like {code} public static boolean isRawFormat(Path baseOrDeltaDir, FileSystem fs, List files) throws IOException {code} instead be able to do isRawFormat(HdfsDirSnapshot) directly > ACID: getAcidState() should cache a recursive dir listing locally > - > > Key: HIVE-21225 > URL: https://issues.apache.org/jira/browse/HIVE-21225 > Project: Hive > Issue Type: Improvement > Components: Transactions >Reporter: Gopal V >Assignee: Vaibhav Gumashta >Priority: Major > Attachments: HIVE-21225.1.patch, HIVE-21225.2.patch, > async-pid-44-2.svg > > > Currently getAcidState() makes 3 calls into the FS api which could be > answered by making a single recursive listDir call and reusing the same data > to check for isRawFormat() and isValidBase(). > All delta operations for a single partition can go against a single listed > directory snapshot instead of interacting with the NameNode or ObjectStore > within the inner loop. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21225) ACID: getAcidState() should cache a recursive dir listing locally
[ https://issues.apache.org/jira/browse/HIVE-21225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16856170#comment-16856170 ] Gopal V commented on HIVE-21225: This patch looks good, but it still does 2 things - it generates one FS OP per delta dir (i.e the more delta dirs, the more listing calls). The DirSnapshot can be based off the partition recursive listing one-shot. And please verify if the IOW base_n dirs are actually found when doing the listFiles via the {{_orc_acid_version}} marker. > ACID: getAcidState() should cache a recursive dir listing locally > - > > Key: HIVE-21225 > URL: https://issues.apache.org/jira/browse/HIVE-21225 > Project: Hive > Issue Type: Improvement > Components: Transactions >Reporter: Gopal V >Assignee: Vaibhav Gumashta >Priority: Major > Attachments: HIVE-21225.1.patch, HIVE-21225.2.patch, > async-pid-44-2.svg > > > Currently getAcidState() makes 3 calls into the FS api which could be > answered by making a single recursive listDir call and reusing the same data > to check for isRawFormat() and isValidBase(). > All delta operations for a single partition can go against a single listed > directory snapshot instead of interacting with the NameNode or ObjectStore > within the inner loop. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21225) ACID: getAcidState() should cache a recursive dir listing locally
[ https://issues.apache.org/jira/browse/HIVE-21225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16845330#comment-16845330 ] Gopal V commented on HIVE-21225: [~vgumashta]: is that the right patch for this JIRA? > ACID: getAcidState() should cache a recursive dir listing locally > - > > Key: HIVE-21225 > URL: https://issues.apache.org/jira/browse/HIVE-21225 > Project: Hive > Issue Type: Improvement > Components: Transactions >Reporter: Gopal V >Assignee: Vaibhav Gumashta >Priority: Major > Attachments: HIVE-21225.1.patch, async-pid-44-2.svg > > > Currently getAcidState() makes 3 calls into the FS api which could be > answered by making a single recursive listDir call and reusing the same data > to check for isRawFormat() and isValidBase(). > All delta operations for a single partition can go against a single listed > directory snapshot instead of interacting with the NameNode or ObjectStore > within the inner loop. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21225) ACID: getAcidState() should cache a recursive dir listing locally
[ https://issues.apache.org/jira/browse/HIVE-21225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16778983#comment-16778983 ] Gopal V commented on HIVE-21225: The specific improvement I'm looking at is to make the {{List childrenWithId}} in getAcidState to be a recursive listing holder, which is a local change to a single function. Eugene's suggestion is a bit better for skipping one of the looks ups, but the crucial one I'm looking at is to have that dir listing be all-depth instead of single level (FYI, all-depth listings are faster in S3 than one level deep, since the recursive one is just a prefix-lookup, while the one-level is a prefix-lookup + discard anything which has another "/" later). > ACID: getAcidState() should cache a recursive dir listing locally > - > > Key: HIVE-21225 > URL: https://issues.apache.org/jira/browse/HIVE-21225 > Project: Hive > Issue Type: Improvement > Components: Transactions >Reporter: Gopal V >Assignee: Vaibhav Gumashta >Priority: Major > Attachments: async-pid-44-2.svg > > > Currently getAcidState() makes 3 calls into the FS api which could be > answered by making a single recursive listDir call and reusing the same data > to check for isRawFormat() and isValidBase(). > All delta operations for a single partition can go against a single listed > directory snapshot instead of interacting with the NameNode or ObjectStore > within the inner loop. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21225) ACID: getAcidState() should cache a recursive dir listing locally
[ https://issues.apache.org/jira/browse/HIVE-21225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16778979#comment-16778979 ] Ashutosh Chauhan commented on HIVE-21225: - [~vgumashta] Which approach you are thinking here? [~ekoifman]'s of encoding identifier in file names or [~gopalv]'s single recursive call. If we change names of dirs not sure if that will have any impact on data in existing tables. > ACID: getAcidState() should cache a recursive dir listing locally > - > > Key: HIVE-21225 > URL: https://issues.apache.org/jira/browse/HIVE-21225 > Project: Hive > Issue Type: Improvement > Components: Transactions >Reporter: Gopal V >Assignee: Vaibhav Gumashta >Priority: Major > Attachments: async-pid-44-2.svg > > > Currently getAcidState() makes 3 calls into the FS api which could be > answered by making a single recursive listDir call and reusing the same data > to check for isRawFormat() and isValidBase(). > All delta operations for a single partition can go against a single listed > directory snapshot instead of interacting with the NameNode or ObjectStore > within the inner loop. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21225) ACID: getAcidState() should cache a recursive dir listing locally
[ https://issues.apache.org/jira/browse/HIVE-21225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16763911#comment-16763911 ] Eugene Koifman commented on HIVE-21225: --- perhaps an easier/better solution is to add another suffix to the base/delta dir name to indicate the "type" - i.e. acid or raw. Then {{isRawFormat}} would just look at dir name. > ACID: getAcidState() should cache a recursive dir listing locally > - > > Key: HIVE-21225 > URL: https://issues.apache.org/jira/browse/HIVE-21225 > Project: Hive > Issue Type: Improvement > Components: Transactions >Reporter: Gopal V >Priority: Major > > Currently getAcidState() makes 3 calls into the FS api which could be > answered by making a single recursive listDir call and reusing the same data > to check for isRawFormat() and isValidBase(). > All delta operations for a single partition can go against a single listed > directory snapshot instead of interacting with the NameNode or ObjectStore > within the inner loop. -- This message was sent by Atlassian JIRA (v7.6.3#76005)