[Impala-ASF-CR] IMPALA-12389: Use -skipTrash for HadoopFsCommandLineClient::delete file dir
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/20386 ) Change subject: IMPALA-12389: Use -skipTrash for HadoopFsCommandLineClient::delete_file_dir .. Patch Set 8: Code-Review+1 This change seems to add ~15 minutes to EE tests, taking them from 3h30m to 3h45m. There are some clumps in test_scanners.py that we might be able to improve, but mostly looks like broad-based slowdown from invoking the CLI. -- To view, visit http://gerrit.cloudera.org:8080/20386 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2d304113596aaf70a122202a33276fc7c3d599e8 Gerrit-Change-Number: 20386 Gerrit-PatchSet: 8 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Thu, 29 Feb 2024 18:25:59 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12389: Use -skipTrash for HadoopFsCommandLineClient::delete file dir
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20386 ) Change subject: IMPALA-12389: Use -skipTrash for HadoopFsCommandLineClient::delete_file_dir .. Patch Set 8: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/20386 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2d304113596aaf70a122202a33276fc7c3d599e8 Gerrit-Change-Number: 20386 Gerrit-PatchSet: 8 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Thu, 29 Feb 2024 03:09:03 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12389: Use -skipTrash for HadoopFsCommandLineClient::delete file dir
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20386 ) Change subject: IMPALA-12389: Use -skipTrash for HadoopFsCommandLineClient::delete_file_dir .. Patch Set 8: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10321/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/20386 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2d304113596aaf70a122202a33276fc7c3d599e8 Gerrit-Change-Number: 20386 Gerrit-PatchSet: 8 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Wed, 28 Feb 2024 22:16:42 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12389: Use -skipTrash for HadoopFsCommandLineClient::delete file dir
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20386 ) Change subject: IMPALA-12389: Use -skipTrash for HadoopFsCommandLineClient::delete_file_dir .. Patch Set 7: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/9766/ -- To view, visit http://gerrit.cloudera.org:8080/20386 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2d304113596aaf70a122202a33276fc7c3d599e8 Gerrit-Change-Number: 20386 Gerrit-PatchSet: 7 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Wed, 27 Sep 2023 20:27:53 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12389: Use -skipTrash for HadoopFsCommandLineClient::delete file dir
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/20386 ) Change subject: IMPALA-12389: Use -skipTrash for HadoopFsCommandLineClient::delete_file_dir .. Patch Set 7: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/20386 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2d304113596aaf70a122202a33276fc7c3d599e8 Gerrit-Change-Number: 20386 Gerrit-PatchSet: 7 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Wed, 27 Sep 2023 16:24:14 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12389: Use -skipTrash for HadoopFsCommandLineClient::delete file dir
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20386 ) Change subject: IMPALA-12389: Use -skipTrash for HadoopFsCommandLineClient::delete_file_dir .. Patch Set 7: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/9766/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/20386 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2d304113596aaf70a122202a33276fc7c3d599e8 Gerrit-Change-Number: 20386 Gerrit-PatchSet: 7 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Wed, 27 Sep 2023 15:51:41 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12389: Use -skipTrash for HadoopFsCommandLineClient::delete file dir
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20386 ) Change subject: IMPALA-12389: Use -skipTrash for HadoopFsCommandLineClient::delete_file_dir .. Patch Set 6: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/9751/ -- To view, visit http://gerrit.cloudera.org:8080/20386 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2d304113596aaf70a122202a33276fc7c3d599e8 Gerrit-Change-Number: 20386 Gerrit-PatchSet: 6 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Fri, 22 Sep 2023 20:22:32 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12389: Use -skipTrash for HadoopFsCommandLineClient::delete file dir
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/20386 ) Change subject: IMPALA-12389: Use -skipTrash for HadoopFsCommandLineClient::delete_file_dir .. Patch Set 6: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/20386 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2d304113596aaf70a122202a33276fc7c3d599e8 Gerrit-Change-Number: 20386 Gerrit-PatchSet: 6 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Fri, 22 Sep 2023 19:31:40 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12389: Use -skipTrash for HadoopFsCommandLineClient::delete file dir
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20386 ) Change subject: IMPALA-12389: Use -skipTrash for HadoopFsCommandLineClient::delete_file_dir .. Patch Set 6: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/14058/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/20386 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2d304113596aaf70a122202a33276fc7c3d599e8 Gerrit-Change-Number: 20386 Gerrit-PatchSet: 6 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Fri, 22 Sep 2023 16:08:31 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12389: Use -skipTrash for HadoopFsCommandLineClient::delete file dir
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20386 ) Change subject: IMPALA-12389: Use -skipTrash for HadoopFsCommandLineClient::delete_file_dir .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/14057/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/20386 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2d304113596aaf70a122202a33276fc7c3d599e8 Gerrit-Change-Number: 20386 Gerrit-PatchSet: 5 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Fri, 22 Sep 2023 16:07:31 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12389: Use -skipTrash for HadoopFsCommandLineClient::delete file dir
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/20386 ) Change subject: IMPALA-12389: Use -skipTrash for HadoopFsCommandLineClient::delete_file_dir .. Patch Set 6: Still tuning the timing for test_recursive_listing.py. Let's see if this helps. -- To view, visit http://gerrit.cloudera.org:8080/20386 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2d304113596aaf70a122202a33276fc7c3d599e8 Gerrit-Change-Number: 20386 Gerrit-PatchSet: 6 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Fri, 22 Sep 2023 15:42:14 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12389: Use -skipTrash for HadoopFsCommandLineClient::delete file dir
Hello Michael Smith, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/20386 to look at the new patch set (#6). Change subject: IMPALA-12389: Use -skipTrash for HadoopFsCommandLineClient::delete_file_dir .. IMPALA-12389: Use -skipTrash for HadoopFsCommandLineClient::delete_file_dir The default behavior for deleting files on Hadoop is to move them to a trash folder. The trash folder can be aged out, but Impala's developer environment sets the trash to live forever. This is a problem, because the trash contents will continue to accumulate. This changes HadoopFsCommandLineClient's delete_file_dir to use -skipTrash to avoid accumulating the trash for this case. It also modifies DelegatingHdfsClient to use HadoopFsCommandLineClient for delete_file_dir. The WebHDFS client doesn't have the option to skip trash. TestHdfsEncryption and TestHdfsPermissions used WebHDFS-style paths without the leading slash. This is incompatible with using the HDFS commandline for delete_file_dir, so it switches those tests to normal paths. This should be safe, because we always use the delegating client which removes slashes when it uses the WebHDFS client. This relaxes the timing for TestRecursiveListing, because deletes via the Hadoop commandline are slower than deletes through the WebHDFS client. Testing: - Ran tests locally and examined the trash directory Change-Id: I2d304113596aaf70a122202a33276fc7c3d599e8 --- M tests/metadata/test_hdfs_encryption.py M tests/metadata/test_hdfs_permissions.py M tests/metadata/test_recursive_listing.py M tests/util/hdfs_util.py 4 files changed, 18 insertions(+), 19 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/86/20386/6 -- To view, visit http://gerrit.cloudera.org:8080/20386 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I2d304113596aaf70a122202a33276fc7c3d599e8 Gerrit-Change-Number: 20386 Gerrit-PatchSet: 6 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith
[Impala-ASF-CR] IMPALA-12389: Use -skipTrash for HadoopFsCommandLineClient::delete file dir
Hello Michael Smith, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/20386 to look at the new patch set (#5). Change subject: IMPALA-12389: Use -skipTrash for HadoopFsCommandLineClient::delete_file_dir .. IMPALA-12389: Use -skipTrash for HadoopFsCommandLineClient::delete_file_dir The default behavior for deleting files on Hadoop is to move them to a trash folder. The trash folder can be aged out, but Impala's developer environment sets the trash to live forever. This is a problem, because the trash contents will continue to accumulate. This changes HadoopFsCommandLineClient's delete_file_dir to use -skipTrash to avoid accumulating the trash for this case. It also modifies DelegatingHdfsClient to use HadoopFsCommandLineClient for delete_file_dir. The WebHDFS client doesn't have the option to skip trash. TestHdfsEncryption and TestHdfsPermissions used WebHDFS-style paths without the leading slash. This is incompatible with using the HDFS commandline for delete_file_dir, so it switches those tests to normal paths. This should be safe, because we always use the delegating client which removes slashes when it uses the WebHDFS client. This relaxes the timing for TestRecursiveListing, because deletes via the Hadoop commandline are slower than deletes through the WebHDFS client. Testing: - Ran tests locally and examined the trash directory Change-Id: I2d304113596aaf70a122202a33276fc7c3d599e8 --- M tests/metadata/test_hdfs_encryption.py M tests/metadata/test_hdfs_permissions.py M tests/metadata/test_recursive_listing.py M tests/util/hdfs_util.py 4 files changed, 18 insertions(+), 19 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/86/20386/5 -- To view, visit http://gerrit.cloudera.org:8080/20386 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I2d304113596aaf70a122202a33276fc7c3d599e8 Gerrit-Change-Number: 20386 Gerrit-PatchSet: 5 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith
[Impala-ASF-CR] IMPALA-12389: Use -skipTrash for HadoopFsCommandLineClient::delete file dir
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20386 ) Change subject: IMPALA-12389: Use -skipTrash for HadoopFsCommandLineClient::delete_file_dir .. Patch Set 6: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/9751/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/20386 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2d304113596aaf70a122202a33276fc7c3d599e8 Gerrit-Change-Number: 20386 Gerrit-PatchSet: 6 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Fri, 22 Sep 2023 15:42:33 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12389: Use -skipTrash for HadoopFsCommandLineClient::delete file dir
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20386 ) Change subject: IMPALA-12389: Use -skipTrash for HadoopFsCommandLineClient::delete_file_dir .. Patch Set 4: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/9748/ -- To view, visit http://gerrit.cloudera.org:8080/20386 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2d304113596aaf70a122202a33276fc7c3d599e8 Gerrit-Change-Number: 20386 Gerrit-PatchSet: 4 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Fri, 22 Sep 2023 04:26:09 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12389: Use -skipTrash for HadoopFsCommandLineClient::delete file dir
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20386 ) Change subject: IMPALA-12389: Use -skipTrash for HadoopFsCommandLineClient::delete_file_dir .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/14055/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/20386 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2d304113596aaf70a122202a33276fc7c3d599e8 Gerrit-Change-Number: 20386 Gerrit-PatchSet: 4 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Fri, 22 Sep 2023 00:13:25 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12389: Use -skipTrash for HadoopFsCommandLineClient::delete file dir
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20386 ) Change subject: IMPALA-12389: Use -skipTrash for HadoopFsCommandLineClient::delete_file_dir .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/14054/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/20386 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2d304113596aaf70a122202a33276fc7c3d599e8 Gerrit-Change-Number: 20386 Gerrit-PatchSet: 3 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Fri, 22 Sep 2023 00:12:07 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12389: Use -skipTrash for HadoopFsCommandLineClient::delete file dir
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/20386 ) Change subject: IMPALA-12389: Use -skipTrash for HadoopFsCommandLineClient::delete_file_dir .. Patch Set 4: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/20386 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2d304113596aaf70a122202a33276fc7c3d599e8 Gerrit-Change-Number: 20386 Gerrit-PatchSet: 4 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Thu, 21 Sep 2023 23:51:23 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12389: Use -skipTrash for HadoopFsCommandLineClient::delete file dir
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/20386 ) Change subject: IMPALA-12389: Use -skipTrash for HadoopFsCommandLineClient::delete_file_dir .. Patch Set 3: (1 comment) http://gerrit.cloudera.org:8080/#/c/20386/3/tests/metadata/test_hdfs_encryption.py File tests/metadata/test_hdfs_encryption.py: http://gerrit.cloudera.org:8080/#/c/20386/3/tests/metadata/test_hdfs_encryption.py@83 PS3, Line 83: , > flake8: E231 missing whitespace after ',' Done -- To view, visit http://gerrit.cloudera.org:8080/20386 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2d304113596aaf70a122202a33276fc7c3d599e8 Gerrit-Change-Number: 20386 Gerrit-PatchSet: 3 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Thu, 21 Sep 2023 23:46:52 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12389: Use -skipTrash for HadoopFsCommandLineClient::delete file dir
Hello Michael Smith, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/20386 to look at the new patch set (#4). Change subject: IMPALA-12389: Use -skipTrash for HadoopFsCommandLineClient::delete_file_dir .. IMPALA-12389: Use -skipTrash for HadoopFsCommandLineClient::delete_file_dir The default behavior for deleting files on Hadoop is to move them to a trash folder. The trash folder can be aged out, but Impala's developer environment sets the trash to live forever. This is a problem, because the trash contents will continue to accumulate. This changes HadoopFsCommandLineClient's delete_file_dir to use -skipTrash to avoid accumulating the trash for this case. It also modifies DelegatingHdfsClient to use HadoopFsCommandLineClient for delete_file_dir. The WebHDFS client doesn't have the option to skip trash. TestHdfsEncryption and TestHdfsPermissions used WebHDFS-style paths without the leading slash. This is incompatible with using the HDFS commandline for delete_file_dir, so it switches those tests to normal paths. This should be safe, because we always use the delegating client which removes slashes when it uses the WebHDFS client. This relaxes the timing for TestRecursiveListing, because deletes via the Hadoop commandline are slower than deletes through the WebHDFS client. Testing: - Ran tests locally and examined the trash directory Change-Id: I2d304113596aaf70a122202a33276fc7c3d599e8 --- M tests/metadata/test_hdfs_encryption.py M tests/metadata/test_hdfs_permissions.py M tests/metadata/test_recursive_listing.py M tests/util/hdfs_util.py 4 files changed, 18 insertions(+), 19 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/86/20386/4 -- To view, visit http://gerrit.cloudera.org:8080/20386 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I2d304113596aaf70a122202a33276fc7c3d599e8 Gerrit-Change-Number: 20386 Gerrit-PatchSet: 4 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith
[Impala-ASF-CR] IMPALA-12389: Use -skipTrash for HadoopFsCommandLineClient::delete file dir
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20386 ) Change subject: IMPALA-12389: Use -skipTrash for HadoopFsCommandLineClient::delete_file_dir .. Patch Set 4: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/9748/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/20386 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2d304113596aaf70a122202a33276fc7c3d599e8 Gerrit-Change-Number: 20386 Gerrit-PatchSet: 4 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Thu, 21 Sep 2023 23:48:40 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12389: Use -skipTrash for HadoopFsCommandLineClient::delete file dir
Hello Michael Smith, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/20386 to look at the new patch set (#3). Change subject: IMPALA-12389: Use -skipTrash for HadoopFsCommandLineClient::delete_file_dir .. IMPALA-12389: Use -skipTrash for HadoopFsCommandLineClient::delete_file_dir The default behavior for deleting files on Hadoop is to move them to a trash folder. The trash folder can be aged out, but Impala's developer environment sets the trash to live forever. This is a problem, because the trash contents will continue to accumulate. This changes HadoopFsCommandLineClient's delete_file_dir to use -skipTrash to avoid accumulating the trash for this case. It also modifies DelegatingHdfsClient to use HadoopFsCommandLineClient for delete_file_dir. The WebHDFS client doesn't have the option to skip trash. TestHdfsEncryption and TestHdfsPermissions used WebHDFS-style paths without the leading slash. This is incompatible with using the HDFS commandline for delete_file_dir, so it switches those tests to normal paths. This should be safe, because we always use the delegating client which removes slashes when it uses the WebHDFS client. This relaxes the timing for TestRecursiveListing, because deletes via the Hadoop commandline are slower than deletes through the WebHDFS client. Testing: - Ran tests locally and examined the trash directory Change-Id: I2d304113596aaf70a122202a33276fc7c3d599e8 --- M tests/metadata/test_hdfs_encryption.py M tests/metadata/test_hdfs_permissions.py M tests/metadata/test_recursive_listing.py M tests/util/hdfs_util.py 4 files changed, 18 insertions(+), 19 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/86/20386/3 -- To view, visit http://gerrit.cloudera.org:8080/20386 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I2d304113596aaf70a122202a33276fc7c3d599e8 Gerrit-Change-Number: 20386 Gerrit-PatchSet: 3 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith
[Impala-ASF-CR] IMPALA-12389: Use -skipTrash for HadoopFsCommandLineClient::delete file dir
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20386 ) Change subject: IMPALA-12389: Use -skipTrash for HadoopFsCommandLineClient::delete_file_dir .. Patch Set 3: (1 comment) http://gerrit.cloudera.org:8080/#/c/20386/3/tests/metadata/test_hdfs_encryption.py File tests/metadata/test_hdfs_encryption.py: http://gerrit.cloudera.org:8080/#/c/20386/3/tests/metadata/test_hdfs_encryption.py@83 PS3, Line 83: , flake8: E231 missing whitespace after ',' -- To view, visit http://gerrit.cloudera.org:8080/20386 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2d304113596aaf70a122202a33276fc7c3d599e8 Gerrit-Change-Number: 20386 Gerrit-PatchSet: 3 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Thu, 21 Sep 2023 23:45:13 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12389: Use -skipTrash for HadoopFsCommandLineClient::delete file dir
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20386 ) Change subject: IMPALA-12389: Use -skipTrash for HadoopFsCommandLineClient::delete_file_dir .. Patch Set 2: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/9745/ -- To view, visit http://gerrit.cloudera.org:8080/20386 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2d304113596aaf70a122202a33276fc7c3d599e8 Gerrit-Change-Number: 20386 Gerrit-PatchSet: 2 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Thu, 21 Sep 2023 22:19:43 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12389: Use -skipTrash for HadoopFsCommandLineClient::delete file dir
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/20386 ) Change subject: IMPALA-12389: Use -skipTrash for HadoopFsCommandLineClient::delete_file_dir .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/20386 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2d304113596aaf70a122202a33276fc7c3d599e8 Gerrit-Change-Number: 20386 Gerrit-PatchSet: 2 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Thu, 21 Sep 2023 17:46:02 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12389: Use -skipTrash for HadoopFsCommandLineClient::delete file dir
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20386 ) Change subject: IMPALA-12389: Use -skipTrash for HadoopFsCommandLineClient::delete_file_dir .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/9745/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/20386 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2d304113596aaf70a122202a33276fc7c3d599e8 Gerrit-Change-Number: 20386 Gerrit-PatchSet: 2 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Thu, 21 Sep 2023 17:46:01 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12389: Use -skipTrash for HadoopFsCommandLineClient::delete file dir
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20386 ) Change subject: IMPALA-12389: Use -skipTrash for HadoopFsCommandLineClient::delete_file_dir .. Patch Set 1: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/9610/ -- To view, visit http://gerrit.cloudera.org:8080/20386 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2d304113596aaf70a122202a33276fc7c3d599e8 Gerrit-Change-Number: 20386 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Tue, 22 Aug 2023 03:49:02 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12389: Use -skipTrash for HadoopFsCommandLineClient::delete file dir
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/20386 ) Change subject: IMPALA-12389: Use -skipTrash for HadoopFsCommandLineClient::delete_file_dir .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/20386 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2d304113596aaf70a122202a33276fc7c3d599e8 Gerrit-Change-Number: 20386 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Smith Gerrit-Comment-Date: Mon, 21 Aug 2023 23:49:26 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12389: Use -skipTrash for HadoopFsCommandLineClient::delete file dir
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20386 ) Change subject: IMPALA-12389: Use -skipTrash for HadoopFsCommandLineClient::delete_file_dir .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/13795/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/20386 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2d304113596aaf70a122202a33276fc7c3d599e8 Gerrit-Change-Number: 20386 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 21 Aug 2023 23:47:53 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12389: Use -skipTrash for HadoopFsCommandLineClient::delete file dir
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20386 ) Change subject: IMPALA-12389: Use -skipTrash for HadoopFsCommandLineClient::delete_file_dir .. Patch Set 1: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/9610/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/20386 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2d304113596aaf70a122202a33276fc7c3d599e8 Gerrit-Change-Number: 20386 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 21 Aug 2023 23:22:58 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12389: Use -skipTrash for HadoopFsCommandLineClient::delete file dir
Joe McDonnell has uploaded this change for review. ( http://gerrit.cloudera.org:8080/20386 Change subject: IMPALA-12389: Use -skipTrash for HadoopFsCommandLineClient::delete_file_dir .. IMPALA-12389: Use -skipTrash for HadoopFsCommandLineClient::delete_file_dir The default behavior for deleting files on Hadoop is to move them to a trash folder. The trash folder can be aged out, but Impala's developer environment sets the trash to live forever. This is a problem, because the trash contents will continue to accumulate. This changes HadoopFsCommandLineClient's delete_file_dir to use -skipTrash to avoid accumulating the trash for this case. It also modifies DelegatingHdfsClient to use HadoopFsCommandLineClient for delete_file_dir. The WebHDFS client doesn't have the option to skip trash. Testing: - Ran tests locally and examined the trash directory Change-Id: I2d304113596aaf70a122202a33276fc7c3d599e8 --- M tests/util/hdfs_util.py 1 file changed, 3 insertions(+), 4 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/86/20386/1 -- To view, visit http://gerrit.cloudera.org:8080/20386 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I2d304113596aaf70a122202a33276fc7c3d599e8 Gerrit-Change-Number: 20386 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell
