[jira] [Commented] (HIVE-23196) Reduce number of delete calls to NN during Context::clear
[ https://issues.apache.org/jira/browse/HIVE-23196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17086338#comment-17086338 ] Hive QA commented on HIVE-23196: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/13000284/HIVE-23196.2.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:green}SUCCESS:{color} +1 due to 17131 tests passed Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/21738/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21738/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21738/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase {noformat} This message is automatically generated. ATTACHMENT ID: 13000284 - PreCommit-HIVE-Build > Reduce number of delete calls to NN during Context::clear > - > > Key: HIVE-23196 > URL: https://issues.apache.org/jira/browse/HIVE-23196 > Project: Hive > Issue Type: Improvement >Reporter: Rajesh Balamohan >Assignee: Attila Magyar >Priority: Major > Attachments: HIVE-23196.1.patch, HIVE-23196.2.patch > > > {{Context::clear()}} ends up deleting same directories (or its subdirs) > multiple times. It would be good to reduce the number of delete calls to NN > for latency sensitive queries. This also has an impact on concurrent queries. > {noformat} > 2020-04-14T04:22:28,703 DEBUG [7c6a6b09-ab37-4bc8-93a5-5da6fb154899 > HiveServer2-Handler-Pool: Thread-378] ql.Context: Deleting result dir: > hdfs://nn1:8020/tmp/hive/xyz/7c6a6b09-ab37-4bc8-93a5-5da6fb154899/hive_2020-04-14_04-22-24_335_8573832618972595103-13/-mr-1 > 2020-04-14T04:22:28,721 DEBUG [7c6a6b09-ab37-4bc8-93a5-5da6fb154899 > HiveServer2-Handler-Pool: Thread-378] ql.Context: Deleting scratch dir: > hdfs://nn1:8020/tmp/hive/xyz/7c6a6b09-ab37-4bc8-93a5-5da6fb154899/hive_2020-04-14_04-22-24_335_8573832618972595103-13 > 2020-04-14T04:22:28,737 DEBUG [7c6a6b09-ab37-4bc8-93a5-5da6fb154899 > HiveServer2-Handler-Pool: Thread-378] ql.Context: Deleting scratch dir: > hdfs://nn1:8020/tmp/hive/xyz/7c6a6b09-ab37-4bc8-93a5-5da6fb154899/hive_2020-04-14_04-22-24_335_8573832618972595103-13/-mr-1/.hive-staging_hive_2020-04-14_04-22-24_335_8573832618972595103-13{noformat} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23196) Reduce number of delete calls to NN during Context::clear
[ https://issues.apache.org/jira/browse/HIVE-23196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17086326#comment-17086326 ] Hive QA commented on HIVE-23196: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m 47s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 5s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 44s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m 1s{color} | {color:blue} ql in master has 1528 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 58s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 31s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 6s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 6s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 45s{color} | {color:red} ql: The patch generated 4 new + 55 unchanged - 1 fixed = 59 total (was 56) {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 14s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 59s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 16s{color} | {color:red} The patch generated 2 ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 27m 6s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-21738/dev-support/hive-personality.sh | | git revision | master / 15ebf9e | | Default Java | 1.8.0_111 | | findbugs | v3.0.1 | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-21738/yetus/diff-checkstyle-ql.txt | | asflicense | http://104.198.109.242/logs//PreCommit-HIVE-Build-21738/yetus/patch-asflicense-problems.txt | | modules | C: ql U: ql | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-21738/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > Reduce number of delete calls to NN during Context::clear > - > > Key: HIVE-23196 > URL: https://issues.apache.org/jira/browse/HIVE-23196 > Project: Hive > Issue Type: Improvement >Reporter: Rajesh Balamohan >Assignee: Attila Magyar >Priority: Major > Attachments: HIVE-23196.1.patch, HIVE-23196.2.patch > > > {{Context::clear()}} ends up deleting same directories (or its subdirs) > multiple times. It would be good to reduce the number of delete calls to NN > for latency sensitive queries. This also has an impact on concurrent queries. > {noformat} > 2020-04-14T04:22:28,703 DEBUG [7c6a6b09-ab37-4bc8-93a5-5da6fb154899 > HiveServer2-Handler-Pool: Thread-378] ql.Context: Deleting result dir: > hdfs://nn1:8020/tmp/hive/xyz/7c6a6b09-ab37-4bc8-93a5-5da6fb154899/hive_2020-04-14_04-22-24_335_8573832618972595103-13/-mr-1 > 2020-04-14T04:22:28,721 DEBUG [7c6a6b09-ab37-4bc8-93a5-5da6fb154899 > HiveServer2-Handler-Pool: Thread-378] ql.Context: Deleting scratch dir: > hdfs://nn1:8020/tmp/hive/xyz/7c6a6b09-ab37-4bc8-93a5-5da6fb154899/hive_2020-04-14_04-22-24_335_8573832618972595103-13 > 2020-04-14T04:22:28,737 DEBUG [7c6a6b09-ab37-4bc8-93a5-5da6fb154899 > HiveServer2-Handler-Pool: Thread-378] ql.Context: Deleting scratch dir: >
[jira] [Commented] (HIVE-23196) Reduce number of delete calls to NN during Context::clear
[ https://issues.apache.org/jira/browse/HIVE-23196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17085822#comment-17085822 ] Ashutosh Chauhan commented on HIVE-23196: - +1 > Reduce number of delete calls to NN during Context::clear > - > > Key: HIVE-23196 > URL: https://issues.apache.org/jira/browse/HIVE-23196 > Project: Hive > Issue Type: Improvement >Reporter: Rajesh Balamohan >Assignee: Attila Magyar >Priority: Major > Attachments: HIVE-23196.1.patch, HIVE-23196.2.patch > > > {{Context::clear()}} ends up deleting same directories (or its subdirs) > multiple times. It would be good to reduce the number of delete calls to NN > for latency sensitive queries. This also has an impact on concurrent queries. > {noformat} > 2020-04-14T04:22:28,703 DEBUG [7c6a6b09-ab37-4bc8-93a5-5da6fb154899 > HiveServer2-Handler-Pool: Thread-378] ql.Context: Deleting result dir: > hdfs://nn1:8020/tmp/hive/xyz/7c6a6b09-ab37-4bc8-93a5-5da6fb154899/hive_2020-04-14_04-22-24_335_8573832618972595103-13/-mr-1 > 2020-04-14T04:22:28,721 DEBUG [7c6a6b09-ab37-4bc8-93a5-5da6fb154899 > HiveServer2-Handler-Pool: Thread-378] ql.Context: Deleting scratch dir: > hdfs://nn1:8020/tmp/hive/xyz/7c6a6b09-ab37-4bc8-93a5-5da6fb154899/hive_2020-04-14_04-22-24_335_8573832618972595103-13 > 2020-04-14T04:22:28,737 DEBUG [7c6a6b09-ab37-4bc8-93a5-5da6fb154899 > HiveServer2-Handler-Pool: Thread-378] ql.Context: Deleting scratch dir: > hdfs://nn1:8020/tmp/hive/xyz/7c6a6b09-ab37-4bc8-93a5-5da6fb154899/hive_2020-04-14_04-22-24_335_8573832618972595103-13/-mr-1/.hive-staging_hive_2020-04-14_04-22-24_335_8573832618972595103-13{noformat} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23196) Reduce number of delete calls to NN during Context::clear
[ https://issues.apache.org/jira/browse/HIVE-23196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17085737#comment-17085737 ] Attila Magyar commented on HIVE-23196: -- [~ashutoshc], [~rajesh.balamohan], The API allows us to set paths which are not within the scratch directory. However the general usage is that resFile and resDir is always under the scratch dir, but these are set externally at 30-40 different places. Similarly the stagingDir is within the scratchDir but these are not enforced rules. So I think it's not safe to completely remove these deletions. I simplified the patch by only adding a few guard clauses before removing the directories. Moving the deletion to a different thread seems to be a bit overkill to me. Please see the updated version: https://reviews.apache.org/r/72371/ > Reduce number of delete calls to NN during Context::clear > - > > Key: HIVE-23196 > URL: https://issues.apache.org/jira/browse/HIVE-23196 > Project: Hive > Issue Type: Improvement >Reporter: Rajesh Balamohan >Assignee: Attila Magyar >Priority: Major > Attachments: HIVE-23196.1.patch, HIVE-23196.2.patch > > > {{Context::clear()}} ends up deleting same directories (or its subdirs) > multiple times. It would be good to reduce the number of delete calls to NN > for latency sensitive queries. This also has an impact on concurrent queries. > {noformat} > 2020-04-14T04:22:28,703 DEBUG [7c6a6b09-ab37-4bc8-93a5-5da6fb154899 > HiveServer2-Handler-Pool: Thread-378] ql.Context: Deleting result dir: > hdfs://nn1:8020/tmp/hive/xyz/7c6a6b09-ab37-4bc8-93a5-5da6fb154899/hive_2020-04-14_04-22-24_335_8573832618972595103-13/-mr-1 > 2020-04-14T04:22:28,721 DEBUG [7c6a6b09-ab37-4bc8-93a5-5da6fb154899 > HiveServer2-Handler-Pool: Thread-378] ql.Context: Deleting scratch dir: > hdfs://nn1:8020/tmp/hive/xyz/7c6a6b09-ab37-4bc8-93a5-5da6fb154899/hive_2020-04-14_04-22-24_335_8573832618972595103-13 > 2020-04-14T04:22:28,737 DEBUG [7c6a6b09-ab37-4bc8-93a5-5da6fb154899 > HiveServer2-Handler-Pool: Thread-378] ql.Context: Deleting scratch dir: > hdfs://nn1:8020/tmp/hive/xyz/7c6a6b09-ab37-4bc8-93a5-5da6fb154899/hive_2020-04-14_04-22-24_335_8573832618972595103-13/-mr-1/.hive-staging_hive_2020-04-14_04-22-24_335_8573832618972595103-13{noformat} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23196) Reduce number of delete calls to NN during Context::clear
[ https://issues.apache.org/jira/browse/HIVE-23196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17085307#comment-17085307 ] Hive QA commented on HIVE-23196: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/13000132/HIVE-23196.1.patch {color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 17134 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[results_cache_diff_fs] (batchId=86) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[results_cache_with_auth] (batchId=89) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[results_cache_1] (batchId=111) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[results_cache_invalidation2] (batchId=102) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[results_cache_invalidation] (batchId=102) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[results_cache_quoted_identifiers] (batchId=105) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[results_cache_transactional] (batchId=95) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/21706/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21706/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21706/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 7 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 13000132 - PreCommit-HIVE-Build > Reduce number of delete calls to NN during Context::clear > - > > Key: HIVE-23196 > URL: https://issues.apache.org/jira/browse/HIVE-23196 > Project: Hive > Issue Type: Improvement >Reporter: Rajesh Balamohan >Assignee: Attila Magyar >Priority: Major > Attachments: HIVE-23196.1.patch > > > {{Context::clear()}} ends up deleting same directories (or its subdirs) > multiple times. It would be good to reduce the number of delete calls to NN > for latency sensitive queries. This also has an impact on concurrent queries. > {noformat} > 2020-04-14T04:22:28,703 DEBUG [7c6a6b09-ab37-4bc8-93a5-5da6fb154899 > HiveServer2-Handler-Pool: Thread-378] ql.Context: Deleting result dir: > hdfs://nn1:8020/tmp/hive/xyz/7c6a6b09-ab37-4bc8-93a5-5da6fb154899/hive_2020-04-14_04-22-24_335_8573832618972595103-13/-mr-1 > 2020-04-14T04:22:28,721 DEBUG [7c6a6b09-ab37-4bc8-93a5-5da6fb154899 > HiveServer2-Handler-Pool: Thread-378] ql.Context: Deleting scratch dir: > hdfs://nn1:8020/tmp/hive/xyz/7c6a6b09-ab37-4bc8-93a5-5da6fb154899/hive_2020-04-14_04-22-24_335_8573832618972595103-13 > 2020-04-14T04:22:28,737 DEBUG [7c6a6b09-ab37-4bc8-93a5-5da6fb154899 > HiveServer2-Handler-Pool: Thread-378] ql.Context: Deleting scratch dir: > hdfs://nn1:8020/tmp/hive/xyz/7c6a6b09-ab37-4bc8-93a5-5da6fb154899/hive_2020-04-14_04-22-24_335_8573832618972595103-13/-mr-1/.hive-staging_hive_2020-04-14_04-22-24_335_8573832618972595103-13{noformat} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23196) Reduce number of delete calls to NN during Context::clear
[ https://issues.apache.org/jira/browse/HIVE-23196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17085298#comment-17085298 ] Hive QA commented on HIVE-23196: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m 50s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 1s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 40s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m 32s{color} | {color:blue} ql in master has 1530 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 53s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 26s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 59s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 59s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 38s{color} | {color:red} ql: The patch generated 42 new + 65 unchanged - 18 fixed = 107 total (was 83) {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 53s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 52s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 14s{color} | {color:red} The patch generated 2 ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 24m 30s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-21706/dev-support/hive-personality.sh | | git revision | master / 661e40a | | Default Java | 1.8.0_111 | | findbugs | v3.0.1 | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-21706/yetus/diff-checkstyle-ql.txt | | asflicense | http://104.198.109.242/logs//PreCommit-HIVE-Build-21706/yetus/patch-asflicense-problems.txt | | modules | C: ql U: ql | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-21706/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > Reduce number of delete calls to NN during Context::clear > - > > Key: HIVE-23196 > URL: https://issues.apache.org/jira/browse/HIVE-23196 > Project: Hive > Issue Type: Improvement >Reporter: Rajesh Balamohan >Assignee: Attila Magyar >Priority: Major > Attachments: HIVE-23196.1.patch > > > {{Context::clear()}} ends up deleting same directories (or its subdirs) > multiple times. It would be good to reduce the number of delete calls to NN > for latency sensitive queries. This also has an impact on concurrent queries. > {noformat} > 2020-04-14T04:22:28,703 DEBUG [7c6a6b09-ab37-4bc8-93a5-5da6fb154899 > HiveServer2-Handler-Pool: Thread-378] ql.Context: Deleting result dir: > hdfs://nn1:8020/tmp/hive/xyz/7c6a6b09-ab37-4bc8-93a5-5da6fb154899/hive_2020-04-14_04-22-24_335_8573832618972595103-13/-mr-1 > 2020-04-14T04:22:28,721 DEBUG [7c6a6b09-ab37-4bc8-93a5-5da6fb154899 > HiveServer2-Handler-Pool: Thread-378] ql.Context: Deleting scratch dir: > hdfs://nn1:8020/tmp/hive/xyz/7c6a6b09-ab37-4bc8-93a5-5da6fb154899/hive_2020-04-14_04-22-24_335_8573832618972595103-13 > 2020-04-14T04:22:28,737 DEBUG [7c6a6b09-ab37-4bc8-93a5-5da6fb154899 > HiveServer2-Handler-Pool: Thread-378] ql.Context: Deleting scratch dir: >
[jira] [Commented] (HIVE-23196) Reduce number of delete calls to NN during Context::clear
[ https://issues.apache.org/jira/browse/HIVE-23196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17085288#comment-17085288 ] Rajesh Balamohan commented on HIVE-23196: - Another possibility: Context::clear can just enqueue the paths and gets deleted by another thread. This would free up delete call from query execution path. > Reduce number of delete calls to NN during Context::clear > - > > Key: HIVE-23196 > URL: https://issues.apache.org/jira/browse/HIVE-23196 > Project: Hive > Issue Type: Improvement >Reporter: Rajesh Balamohan >Assignee: Attila Magyar >Priority: Major > Attachments: HIVE-23196.1.patch > > > {{Context::clear()}} ends up deleting same directories (or its subdirs) > multiple times. It would be good to reduce the number of delete calls to NN > for latency sensitive queries. This also has an impact on concurrent queries. > {noformat} > 2020-04-14T04:22:28,703 DEBUG [7c6a6b09-ab37-4bc8-93a5-5da6fb154899 > HiveServer2-Handler-Pool: Thread-378] ql.Context: Deleting result dir: > hdfs://nn1:8020/tmp/hive/xyz/7c6a6b09-ab37-4bc8-93a5-5da6fb154899/hive_2020-04-14_04-22-24_335_8573832618972595103-13/-mr-1 > 2020-04-14T04:22:28,721 DEBUG [7c6a6b09-ab37-4bc8-93a5-5da6fb154899 > HiveServer2-Handler-Pool: Thread-378] ql.Context: Deleting scratch dir: > hdfs://nn1:8020/tmp/hive/xyz/7c6a6b09-ab37-4bc8-93a5-5da6fb154899/hive_2020-04-14_04-22-24_335_8573832618972595103-13 > 2020-04-14T04:22:28,737 DEBUG [7c6a6b09-ab37-4bc8-93a5-5da6fb154899 > HiveServer2-Handler-Pool: Thread-378] ql.Context: Deleting scratch dir: > hdfs://nn1:8020/tmp/hive/xyz/7c6a6b09-ab37-4bc8-93a5-5da6fb154899/hive_2020-04-14_04-22-24_335_8573832618972595103-13/-mr-1/.hive-staging_hive_2020-04-14_04-22-24_335_8573832618972595103-13{noformat} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23196) Reduce number of delete calls to NN during Context::clear
[ https://issues.apache.org/jira/browse/HIVE-23196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17085084#comment-17085084 ] Ashutosh Chauhan commented on HIVE-23196: - instead of intercepting delete calls in Context won't it be better to just remove those delete calls from caller side to eliminate unneeded delete ? > Reduce number of delete calls to NN during Context::clear > - > > Key: HIVE-23196 > URL: https://issues.apache.org/jira/browse/HIVE-23196 > Project: Hive > Issue Type: Improvement >Reporter: Rajesh Balamohan >Assignee: Attila Magyar >Priority: Major > Attachments: HIVE-23196.1.patch > > > {{Context::clear()}} ends up deleting same directories (or its subdirs) > multiple times. It would be good to reduce the number of delete calls to NN > for latency sensitive queries. This also has an impact on concurrent queries. > {noformat} > 2020-04-14T04:22:28,703 DEBUG [7c6a6b09-ab37-4bc8-93a5-5da6fb154899 > HiveServer2-Handler-Pool: Thread-378] ql.Context: Deleting result dir: > hdfs://nn1:8020/tmp/hive/xyz/7c6a6b09-ab37-4bc8-93a5-5da6fb154899/hive_2020-04-14_04-22-24_335_8573832618972595103-13/-mr-1 > 2020-04-14T04:22:28,721 DEBUG [7c6a6b09-ab37-4bc8-93a5-5da6fb154899 > HiveServer2-Handler-Pool: Thread-378] ql.Context: Deleting scratch dir: > hdfs://nn1:8020/tmp/hive/xyz/7c6a6b09-ab37-4bc8-93a5-5da6fb154899/hive_2020-04-14_04-22-24_335_8573832618972595103-13 > 2020-04-14T04:22:28,737 DEBUG [7c6a6b09-ab37-4bc8-93a5-5da6fb154899 > HiveServer2-Handler-Pool: Thread-378] ql.Context: Deleting scratch dir: > hdfs://nn1:8020/tmp/hive/xyz/7c6a6b09-ab37-4bc8-93a5-5da6fb154899/hive_2020-04-14_04-22-24_335_8573832618972595103-13/-mr-1/.hive-staging_hive_2020-04-14_04-22-24_335_8573832618972595103-13{noformat} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23196) Reduce number of delete calls to NN during Context::clear
[ https://issues.apache.org/jira/browse/HIVE-23196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17084313#comment-17084313 ] Ashutosh Chauhan commented on HIVE-23196: - [~amagyar] can you please take a look at this one? > Reduce number of delete calls to NN during Context::clear > - > > Key: HIVE-23196 > URL: https://issues.apache.org/jira/browse/HIVE-23196 > Project: Hive > Issue Type: Improvement >Reporter: Rajesh Balamohan >Priority: Major > > {{Context::clear()}} ends up deleting same directories (or its subdirs) > multiple times. It would be good to reduce the number of delete calls to NN > for latency sensitive queries. This also has an impact on concurrent queries. > {noformat} > 2020-04-14T04:22:28,703 DEBUG [7c6a6b09-ab37-4bc8-93a5-5da6fb154899 > HiveServer2-Handler-Pool: Thread-378] ql.Context: Deleting result dir: > hdfs://nn1:8020/tmp/hive/xyz/7c6a6b09-ab37-4bc8-93a5-5da6fb154899/hive_2020-04-14_04-22-24_335_8573832618972595103-13/-mr-1 > 2020-04-14T04:22:28,721 DEBUG [7c6a6b09-ab37-4bc8-93a5-5da6fb154899 > HiveServer2-Handler-Pool: Thread-378] ql.Context: Deleting scratch dir: > hdfs://nn1:8020/tmp/hive/xyz/7c6a6b09-ab37-4bc8-93a5-5da6fb154899/hive_2020-04-14_04-22-24_335_8573832618972595103-13 > 2020-04-14T04:22:28,737 DEBUG [7c6a6b09-ab37-4bc8-93a5-5da6fb154899 > HiveServer2-Handler-Pool: Thread-378] ql.Context: Deleting scratch dir: > hdfs://nn1:8020/tmp/hive/xyz/7c6a6b09-ab37-4bc8-93a5-5da6fb154899/hive_2020-04-14_04-22-24_335_8573832618972595103-13/-mr-1/.hive-staging_hive_2020-04-14_04-22-24_335_8573832618972595103-13{noformat} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23196) Reduce number of delete calls to NN during Context::clear
[ https://issues.apache.org/jira/browse/HIVE-23196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17082987#comment-17082987 ] Rajesh Balamohan commented on HIVE-23196: - Ref: [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/Context.java#L846] [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/Context.java#L869] > Reduce number of delete calls to NN during Context::clear > - > > Key: HIVE-23196 > URL: https://issues.apache.org/jira/browse/HIVE-23196 > Project: Hive > Issue Type: Improvement >Reporter: Rajesh Balamohan >Priority: Major > > {{Context::clear()}} ends up deleting same directories (or its subdirs) > multiple times. It would be good to reduce the number of delete calls to NN > for latency sensitive queries. This also has an impact on concurrent queries. > {noformat} > 2020-04-14T04:22:28,703 DEBUG [7c6a6b09-ab37-4bc8-93a5-5da6fb154899 > HiveServer2-Handler-Pool: Thread-378] ql.Context: Deleting result dir: > hdfs://nn1:8020/tmp/hive/xyz/7c6a6b09-ab37-4bc8-93a5-5da6fb154899/hive_2020-04-14_04-22-24_335_8573832618972595103-13/-mr-1 > 2020-04-14T04:22:28,721 DEBUG [7c6a6b09-ab37-4bc8-93a5-5da6fb154899 > HiveServer2-Handler-Pool: Thread-378] ql.Context: Deleting scratch dir: > hdfs://nn1:8020/tmp/hive/xyz/7c6a6b09-ab37-4bc8-93a5-5da6fb154899/hive_2020-04-14_04-22-24_335_8573832618972595103-13 > 2020-04-14T04:22:28,737 DEBUG [7c6a6b09-ab37-4bc8-93a5-5da6fb154899 > HiveServer2-Handler-Pool: Thread-378] ql.Context: Deleting scratch dir: > hdfs://nn1:8020/tmp/hive/xyz/7c6a6b09-ab37-4bc8-93a5-5da6fb154899/hive_2020-04-14_04-22-24_335_8573832618972595103-13/-mr-1/.hive-staging_hive_2020-04-14_04-22-24_335_8573832618972595103-13{noformat} -- This message was sent by Atlassian Jira (v8.3.4#803005)