[Impala-ASF-CR] IMPALA-11683: Support Aliyun OSS File System
hezhenhua.hzh has posted comments on this change. ( http://gerrit.cloudera.org:8080/19165 ) Change subject: IMPALA-11683: Support Aliyun OSS File System .. Patch Set 16: (1 comment) http://gerrit.cloudera.org:8080/#/c/19165/16//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/19165/16//COMMIT_MSG@20 PS16, Line 20: Run CORE tests > Just curious, are there any failed tests? i just running ./bin/run-all-tests.sh, and seems everything is ok -- To view, visit http://gerrit.cloudera.org:8080/19165 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I267e6531da58e3ac97029fea4c5e075724587910 Gerrit-Change-Number: 19165 Gerrit-PatchSet: 16 Gerrit-Owner: hezhenhua.hzh Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Yida Wu Gerrit-Reviewer: hezhenhua.hzh Gerrit-Comment-Date: Wed, 16 Nov 2022 03:55:33 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11683: Support Aliyun OSS File System
hezhenhua.hzh has posted comments on this change. ( http://gerrit.cloudera.org:8080/19165 ) Change subject: IMPALA-11683: Support Aliyun OSS File System .. Patch Set 17: (2 comments) http://gerrit.cloudera.org:8080/#/c/19165/16/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java File fe/src/main/java/org/apache/impala/catalog/HdfsTable.java: http://gerrit.cloudera.org:8080/#/c/19165/16/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@872 PS16, Line 872: File owner and group are : // persisted, but the permissions model is not enforced. Authorization occurs at : // the level of the entire Aliyun account via Aliyun Resource Access Management. : // The append operation is not supported. : if (FileSystemUtil.isOSSFileSystem(fs)) return true; > This part seems unrelated to read/write permissions. Can we move them to so Done http://gerrit.cloudera.org:8080/#/c/19165/16/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@879 PS16, Line 879: : /** > seems unrelated too the append operation is related about write. so i still keep comment here. -- To view, visit http://gerrit.cloudera.org:8080/19165 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I267e6531da58e3ac97029fea4c5e075724587910 Gerrit-Change-Number: 19165 Gerrit-PatchSet: 17 Gerrit-Owner: hezhenhua.hzh Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Yida Wu Gerrit-Reviewer: hezhenhua.hzh Gerrit-Comment-Date: Wed, 16 Nov 2022 03:45:10 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11683: Support Aliyun OSS File System
Hello Quanlong Huang, Yida Wu, lipeng...@sensorsdata.cn, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/19165 to look at the new patch set (#17). Change subject: IMPALA-11683: Support Aliyun OSS File System .. IMPALA-11683: Support Aliyun OSS File System This patch adds support for OSS (Aliyun Object Storage Service). Using the hadoop-aliyun, the implementation is similar to other remote FileSystems. Tests: - Prepare: Initialize OSS-related environment variables: OSS_ACCESS_KEY_ID, OSS_SECRET_ACCESS_KEY, OSS_ACCESS_ENDPOINT. Compile and create hdfs test data on a ECS instance. Upload test data to an OSS bucket. - Modify all locations in HMS DB to point to the OSS bucket. Remove some hdfs caching params. Run CORE tests. Change-Id: I267e6531da58e3ac97029fea4c5e075724587910 --- M be/src/exec/hdfs-table-sink.cc M bin/impala-config.sh M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/common/FileSystemUtil.java M fe/src/test/java/org/apache/impala/testutil/TestUtils.java M java/executor-deps/pom.xml M java/pom.xml M testdata/bin/create-load-data.sh M testdata/cluster/node_templates/common/etc/hadoop/conf/core-site.xml.py M tests/common/impala_test_suite.py M tests/common/skip.py M tests/custom_cluster/test_hdfs_fd_caching.py M tests/custom_cluster/test_metastore_service.py M tests/util/filesystem_utils.py 14 files changed, 88 insertions(+), 17 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/65/19165/17 -- To view, visit http://gerrit.cloudera.org:8080/19165 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I267e6531da58e3ac97029fea4c5e075724587910 Gerrit-Change-Number: 19165 Gerrit-PatchSet: 17 Gerrit-Owner: hezhenhua.hzh Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Yida Wu Gerrit-Reviewer: hezhenhua.hzh
[Impala-ASF-CR] IMPALA-11683: Support Aliyun OSS File System
hezhenhua.hzh has posted comments on this change. ( http://gerrit.cloudera.org:8080/19165 ) Change subject: IMPALA-11683: Support Aliyun OSS File System .. Patch Set 15: (1 comment) http://gerrit.cloudera.org:8080/#/c/19165/15/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java File fe/src/main/java/org/apache/impala/catalog/HdfsTable.java: http://gerrit.cloudera.org:8080/#/c/19165/15/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@872 PS15, Line 872: rename() > Done who can help with +2 & merge? could you help with that? -- To view, visit http://gerrit.cloudera.org:8080/19165 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I267e6531da58e3ac97029fea4c5e075724587910 Gerrit-Change-Number: 19165 Gerrit-PatchSet: 15 Gerrit-Owner: hezhenhua.hzh Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Yida Wu Gerrit-Reviewer: hezhenhua.hzh Gerrit-Comment-Date: Thu, 10 Nov 2022 03:59:26 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11683: Support Aliyun OSS File System
hezhenhua.hzh has posted comments on this change. ( http://gerrit.cloudera.org:8080/19165 ) Change subject: IMPALA-11683: Support Aliyun OSS File System .. Patch Set 15: (1 comment) http://gerrit.cloudera.org:8080/#/c/19165/15/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java File fe/src/main/java/org/apache/impala/catalog/HdfsTable.java: http://gerrit.cloudera.org:8080/#/c/19165/15/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@872 PS15, Line 872: rename() > AFAIK, e.g. s3(Amazon Simple Storage Service) neither does not support atom Every rename request is converted into a copy + delete operation. Therefore, for the rename operation of an object (file), in the worst case, a large number of copied target objects will be generated, and finally the rename src file will be deleted. -- To view, visit http://gerrit.cloudera.org:8080/19165 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I267e6531da58e3ac97029fea4c5e075724587910 Gerrit-Change-Number: 19165 Gerrit-PatchSet: 15 Gerrit-Owner: hezhenhua.hzh Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Yida Wu Gerrit-Reviewer: hezhenhua.hzh Gerrit-Comment-Date: Wed, 09 Nov 2022 12:08:45 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11683: Support Aliyun OSS File System
Hello Yida Wu, lipeng...@sensorsdata.cn, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/19165 to look at the new patch set (#16). Change subject: IMPALA-11683: Support Aliyun OSS File System .. IMPALA-11683: Support Aliyun OSS File System This patch adds support for OSS (Aliyun Object Storage Service). Using the hadoop-aliyun, the implementation is similar to other remote FileSystems. Tests: - Prepare: Initialize OSS-related environment variables: OSS_ACCESS_KEY_ID, OSS_SECRET_ACCESS_KEY, OSS_ACCESS_ENDPOINT. Compile and create hdfs test data on a ECS instance. Upload test data to an OSS bucket. - Modify all locations in HMS DB to point to the OSS bucket. Remove some hdfs caching params. Run CORE tests. Change-Id: I267e6531da58e3ac97029fea4c5e075724587910 --- M be/src/exec/hdfs-table-sink.cc M bin/impala-config.sh M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/common/FileSystemUtil.java M fe/src/test/java/org/apache/impala/testutil/TestUtils.java M java/executor-deps/pom.xml M java/pom.xml M testdata/bin/create-load-data.sh M testdata/cluster/node_templates/common/etc/hadoop/conf/core-site.xml.py M tests/common/impala_test_suite.py M tests/common/skip.py M tests/custom_cluster/test_hdfs_fd_caching.py M tests/custom_cluster/test_metastore_service.py M tests/util/filesystem_utils.py 14 files changed, 91 insertions(+), 15 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/65/19165/16 -- To view, visit http://gerrit.cloudera.org:8080/19165 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I267e6531da58e3ac97029fea4c5e075724587910 Gerrit-Change-Number: 19165 Gerrit-PatchSet: 16 Gerrit-Owner: hezhenhua.hzh Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Yida Wu Gerrit-Reviewer: hezhenhua.hzh
[Impala-ASF-CR] IMPALA-11683: Support Aliyun OSS File System
hezhenhua.hzh has posted comments on this change. ( http://gerrit.cloudera.org:8080/19165 ) Change subject: IMPALA-11683: Support Aliyun OSS File System .. Patch Set 13: (1 comment) http://gerrit.cloudera.org:8080/#/c/19165/13//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/19165/13//COMMIT_MSG@18 PS13, Line 18: a > nit. an Thanks for your quick review. I updated according to your comments, not sure if this version meets your requirements, not very sure if the commit msg content needs to be less than 72 is it for the description of the second paragraph or the whole commit msg? -- To view, visit http://gerrit.cloudera.org:8080/19165 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I267e6531da58e3ac97029fea4c5e075724587910 Gerrit-Change-Number: 19165 Gerrit-PatchSet: 13 Gerrit-Owner: hezhenhua.hzh Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Yida Wu Gerrit-Reviewer: hezhenhua.hzh Gerrit-Comment-Date: Wed, 09 Nov 2022 03:24:57 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11683: Support Aliyun OSS File System
Hello Yida Wu, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/19165 to look at the new patch set (#14). Change subject: IMPALA-11683: Support Aliyun OSS File System .. IMPALA-11683: Support Aliyun OSS File System Add Aliyun OSS support This patch adds support for OSS (Aliyun Object Storage Service). Using the hadoop-aliyun, the implementation is similar to other remote FileSystems. Tests: - Prepare: Initialize OSS-related environment variables: OSS_ACCESS_KEY_ID, OSS_SECRET_ACCESS_KEY, OSS_ACCESS_ENDPOINT. Compile and create hdfs test data on a ECS instance. Upload test data to an OSS bucket. - Modify all locations in HMS DB to point to the OSS bucket. Remove some hdfs caching params. Run CORE tests. Change-Id: I267e6531da58e3ac97029fea4c5e075724587910 --- M be/src/exec/hdfs-table-sink.cc M bin/impala-config.sh M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/common/FileSystemUtil.java M fe/src/test/java/org/apache/impala/testutil/TestUtils.java M java/executor-deps/pom.xml M java/pom.xml M testdata/bin/create-load-data.sh M testdata/cluster/node_templates/common/etc/hadoop/conf/core-site.xml.py M tests/common/impala_test_suite.py M tests/common/skip.py M tests/custom_cluster/test_hdfs_fd_caching.py M tests/custom_cluster/test_metastore_service.py M tests/util/filesystem_utils.py 14 files changed, 91 insertions(+), 15 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/65/19165/14 -- To view, visit http://gerrit.cloudera.org:8080/19165 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I267e6531da58e3ac97029fea4c5e075724587910 Gerrit-Change-Number: 19165 Gerrit-PatchSet: 14 Gerrit-Owner: hezhenhua.hzh Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Yida Wu Gerrit-Reviewer: hezhenhua.hzh
[Impala-ASF-CR] IMPALA-11683: Support Aliyun OSS File System
hezhenhua.hzh has posted comments on this change. ( http://gerrit.cloudera.org:8080/19165 ) Change subject: IMPALA-11683: Support Aliyun OSS File System .. Patch Set 11: (3 comments) http://gerrit.cloudera.org:8080/#/c/19165/11//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/19165/11//COMMIT_MSG@9 PS11, Line 9: Oobject > nit. Object Done http://gerrit.cloudera.org:8080/#/c/19165/11//COMMIT_MSG@14 PS11, Line 14: Mdddanual ACID test verification. > Could you elaborate a bit on how did you test on this feature? For example, Done http://gerrit.cloudera.org:8080/#/c/19165/11/fe/src/main/java/org/apache/impala/common/FileSystemUtil.java File fe/src/main/java/org/apache/impala/common/FileSystemUtil.java: http://gerrit.cloudera.org:8080/#/c/19165/11/fe/src/main/java/org/apache/impala/common/FileSystemUtil.java@447 PS11, Line 447: a > nit. an Done -- To view, visit http://gerrit.cloudera.org:8080/19165 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I267e6531da58e3ac97029fea4c5e075724587910 Gerrit-Change-Number: 19165 Gerrit-PatchSet: 11 Gerrit-Owner: hezhenhua.hzh Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Yida Wu Gerrit-Reviewer: hezhenhua.hzh Gerrit-Comment-Date: Tue, 08 Nov 2022 09:20:30 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11683: Support Aliyun OSS File System
Hello Yida Wu, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/19165 to look at the new patch set (#13). Change subject: IMPALA-11683: Support Aliyun OSS File System .. IMPALA-11683: Support Aliyun OSS File System Add Aliyun OSS support This patch adds support for OSS (Aliyun Object Storage Service). Using the hadoop-aliyun, the implementation is similar to other remote FileSystems. For more detail about OSS see: https://www.alibabacloud.com/en/product/object-storage-service Tests: - Prepare: Initialize OSS-related environment variables: OSS_ACCESS_KEY_ID, OSS_SECRET_ACCESS_KEY, OSS_ACCESS_ENDPOINT. Compile and create hdfs test data on a ECS instance. Upload test data to a OSS bucket. - Modify all locations in HMS DB to point to the OSS bucket. Remove some hdfs caching params. Run CORE tests. Change-Id: I267e6531da58e3ac97029fea4c5e075724587910 --- M be/src/exec/hdfs-table-sink.cc M bin/impala-config.sh M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/common/FileSystemUtil.java M fe/src/test/java/org/apache/impala/testutil/TestUtils.java M java/executor-deps/pom.xml M java/pom.xml M testdata/bin/create-load-data.sh M testdata/cluster/node_templates/common/etc/hadoop/conf/core-site.xml.py M tests/common/impala_test_suite.py M tests/common/skip.py M tests/custom_cluster/test_hdfs_fd_caching.py M tests/custom_cluster/test_metastore_service.py M tests/util/filesystem_utils.py 14 files changed, 91 insertions(+), 15 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/65/19165/13 -- To view, visit http://gerrit.cloudera.org:8080/19165 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I267e6531da58e3ac97029fea4c5e075724587910 Gerrit-Change-Number: 19165 Gerrit-PatchSet: 13 Gerrit-Owner: hezhenhua.hzh Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Yida Wu Gerrit-Reviewer: hezhenhua.hzh
[Impala-ASF-CR] IMPALA-11683: Support Aliyun OSS File System
Hello Yida Wu, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/19165 to look at the new patch set (#12). Change subject: IMPALA-11683: Support Aliyun OSS File System .. IMPALA-11683: Support Aliyun OSS File System Add Aliyun OSS support This patch adds support for OSS (Aliyun Open Strorage Service). Using the hadoop-aliyun, the implementation is similar to other remote FileSystems. For more detail about OSS see: https://www.alibabacloud.com/en/product/object-storage-service Tests: - Prepare: Initialize OSS-related environment variables: OSS_ACCESS_KEY_ID, OSS_SECRET_ACCESS_KEY, OSS_ACCESS_ENDPOINT. Compile and create hdfs test data on a ECS instance. Upload test data to a OSS bucket. - Modify all locations in HMS DB to point to the OSS bucket. Remove some hdfs caching params. Run CORE tests. Change-Id: I267e6531da58e3ac97029fea4c5e075724587910 --- M be/src/exec/hdfs-table-sink.cc M bin/impala-config.sh M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/common/FileSystemUtil.java M fe/src/test/java/org/apache/impala/testutil/TestUtils.java M java/executor-deps/pom.xml M java/pom.xml M testdata/bin/create-load-data.sh M testdata/cluster/node_templates/common/etc/hadoop/conf/core-site.xml.py M tests/common/impala_test_suite.py M tests/common/skip.py M tests/custom_cluster/test_hdfs_fd_caching.py M tests/custom_cluster/test_metastore_service.py M tests/util/filesystem_utils.py 14 files changed, 91 insertions(+), 15 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/65/19165/12 -- To view, visit http://gerrit.cloudera.org:8080/19165 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I267e6531da58e3ac97029fea4c5e075724587910 Gerrit-Change-Number: 19165 Gerrit-PatchSet: 12 Gerrit-Owner: hezhenhua.hzh Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Yida Wu Gerrit-Reviewer: hezhenhua.hzh
[Impala-ASF-CR] IMPALA-11683: Support Aliyun OSS File System
hezhenhua.hzh has posted comments on this change. ( http://gerrit.cloudera.org:8080/19165 ) Change subject: IMPALA-11683: Support Aliyun OSS File System .. Patch Set 7: (4 comments) http://gerrit.cloudera.org:8080/#/c/19165/5//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/19165/5//COMMIT_MSG@7 PS5, Line 7: Support Aliyun OSS File System > thanks for your comment, i just run the core tests by, and do manual ACID Done http://gerrit.cloudera.org:8080/#/c/19165/7//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/19165/7//COMMIT_MSG@9 PS7, Line 9: Add Aliyun OSS support > This sentence is probably good to delete, otherwise a full stop is needed. Done http://gerrit.cloudera.org:8080/#/c/19165/7//COMMIT_MSG@11 PS7, Line 11: for > nit: need a capital letter. Done http://gerrit.cloudera.org:8080/#/c/19165/7//COMMIT_MSG@14 PS7, Line 14: Manual ACID test verification > nit: would be good to have a full stop. Done -- To view, visit http://gerrit.cloudera.org:8080/19165 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I267e6531da58e3ac97029fea4c5e075724587910 Gerrit-Change-Number: 19165 Gerrit-PatchSet: 7 Gerrit-Owner: hezhenhua.hzh Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Yida Wu Gerrit-Reviewer: hezhenhua.hzh Gerrit-Comment-Date: Tue, 01 Nov 2022 06:28:39 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11683: Support Aliyun OSS File System
Hello Yida Wu, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/19165 to look at the new patch set (#10). Change subject: IMPALA-11683: Support Aliyun OSS File System .. IMPALA-11683: Support Aliyun OSS File System This patch adds support for OSS (Aliyun Oobject Strorage Service). Using the hadoop-aliyun, the implementation is similar to other remote FileSystems. For more detail about OSS see: https://www.alibabacloud.com/en/product/object-storage-service Tests: Run CORE tests. Mdddanual ACID test verification. Change-Id: I267e6531da58e3ac97029fea4c5e075724587910 --- M be/src/exec/hdfs-table-sink.cc M bin/impala-config.sh M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/common/FileSystemUtil.java M fe/src/test/java/org/apache/impala/testutil/TestUtils.java M java/executor-deps/pom.xml M java/pom.xml M testdata/bin/create-load-data.sh M testdata/cluster/node_templates/common/etc/hadoop/conf/core-site.xml.py M tests/common/impala_test_suite.py M tests/common/skip.py M tests/custom_cluster/test_hdfs_fd_caching.py M tests/custom_cluster/test_metastore_service.py M tests/util/filesystem_utils.py 14 files changed, 91 insertions(+), 15 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/65/19165/10 -- To view, visit http://gerrit.cloudera.org:8080/19165 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I267e6531da58e3ac97029fea4c5e075724587910 Gerrit-Change-Number: 19165 Gerrit-PatchSet: 10 Gerrit-Owner: hezhenhua.hzh Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Yida Wu Gerrit-Reviewer: hezhenhua.hzh
[Impala-ASF-CR] IMPALA-11683: Support Aliyun OSS File System
Hello Yida Wu, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/19165 to look at the new patch set (#8). Change subject: IMPALA-11683: Support Aliyun OSS File System .. IMPALA-11683: Support Aliyun OSS File System This patch adds support for OSS (Aliyun Oobject Strorage Service). Using the hadoop-aliyun, the implementation is similar to other remote FileSystems. For more detail about OSS see: https://www.alibabacloud.com/en/product/object-storage-service Tests: Run CORE tests. Mdddanual ACID test verification. Change-Id: I267e6531da58e3ac97029fea4c5e075724587910 --- M be/src/exec/hdfs-table-sink.cc M bin/impala-config.sh M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/common/FileSystemUtil.java M fe/src/test/java/org/apache/impala/planner/ExplainTest.java M fe/src/test/java/org/apache/impala/planner/PlannerTest.java M fe/src/test/java/org/apache/impala/testutil/TestUtils.java M java/executor-deps/pom.xml M java/pom.xml M testdata/bin/create-load-data.sh M testdata/cluster/node_templates/common/etc/hadoop/conf/core-site.xml.py M testdata/workloads/functional-planner/queries/PlannerTest/scan-node-fs-scheme.test M tests/common/impala_test_suite.py M tests/common/skip.py M tests/custom_cluster/test_hdfs_fd_caching.py M tests/custom_cluster/test_metastore_service.py M tests/util/filesystem_utils.py 17 files changed, 113 insertions(+), 15 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/65/19165/8 -- To view, visit http://gerrit.cloudera.org:8080/19165 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I267e6531da58e3ac97029fea4c5e075724587910 Gerrit-Change-Number: 19165 Gerrit-PatchSet: 8 Gerrit-Owner: hezhenhua.hzh Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Yida Wu Gerrit-Reviewer: hezhenhua.hzh
[Impala-ASF-CR] IMPALA-11683: Support Aliyun OSS File System
hezhenhua.hzh has posted comments on this change. ( http://gerrit.cloudera.org:8080/19165 ) Change subject: IMPALA-11683: Support Aliyun OSS File System .. Patch Set 7: (2 comments) http://gerrit.cloudera.org:8080/#/c/19165/5//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/19165/5//COMMIT_MSG@7 PS5, Line 7: Support Aliyun OSS File System > Would be good to have a detailed commit message, including the description thanks for your comment, i just run the core tests by, and do manual ACID test verification, not sure is it enough for you. btw, the cdp_components-31397203 missing hadoop-aliyun-*.jar and aliyun-sdk-oss-*.jar, should i raise a issue for this ? http://gerrit.cloudera.org:8080/#/c/19165/5/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java File fe/src/main/java/org/apache/impala/catalog/HdfsTable.java: http://gerrit.cloudera.org:8080/#/c/19165/5/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@872 PS5, Line 872: // Atomic operations: delete() and rename() are implemented by recursive : // file-by-file operations. > It is good to have such explanations for the atomicity of OSS. But not sure you are right, rename a directory or a file in OSS are both not atomic. rename directory in OSS or in S3-like storage will be recursive file-by-file operations and rename a single file will be split into copy + delete. i will update this explanations. -- To view, visit http://gerrit.cloudera.org:8080/19165 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I267e6531da58e3ac97029fea4c5e075724587910 Gerrit-Change-Number: 19165 Gerrit-PatchSet: 7 Gerrit-Owner: hezhenhua.hzh Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Yida Wu Gerrit-Reviewer: hezhenhua.hzh Gerrit-Comment-Date: Wed, 26 Oct 2022 08:42:44 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11683: Support Aliyun OSS File System
Hello Yida Wu, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/19165 to look at the new patch set (#6). Change subject: IMPALA-11683: Support Aliyun OSS File System .. IMPALA-11683: Support Aliyun OSS File System Add Aliyun OSS support This patch adds support for OSS (Aliyun Oobject Strorage Service). Using the hadoop-aliyun, the implementation is similar to other remote FileSystems. for more detail about OSS see: https://www.alibabacloud.com/en/product/object-storage-service Tests: Run CORE tests. Manual ACID test verification Change-Id: I267e6531da58e3ac97029fea4c5e075724587910 --- M be/src/exec/hdfs-table-sink.cc M bin/impala-config.sh M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/common/FileSystemUtil.java M fe/src/test/java/org/apache/impala/planner/ExplainTest.java M java/executor-deps/pom.xml M java/pom.xml M testdata/bin/create-load-data.sh M testdata/cluster/node_templates/common/etc/hadoop/conf/core-site.xml.py M tests/common/impala_test_suite.py M tests/common/skip.py M tests/custom_cluster/test_hdfs_fd_caching.py M tests/custom_cluster/test_metastore_service.py M tests/util/filesystem_utils.py 14 files changed, 93 insertions(+), 13 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/65/19165/6 -- To view, visit http://gerrit.cloudera.org:8080/19165 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I267e6531da58e3ac97029fea4c5e075724587910 Gerrit-Change-Number: 19165 Gerrit-PatchSet: 6 Gerrit-Owner: hezhenhua.hzh Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Yida Wu
[Impala-ASF-CR] IMPALA-11683: Support Aliyun OSS File System
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/19165 to look at the new patch set (#5). Change subject: IMPALA-11683: Support Aliyun OSS File System .. IMPALA-11683: Support Aliyun OSS File System Change-Id: I267e6531da58e3ac97029fea4c5e075724587910 --- M be/src/exec/hdfs-table-sink.cc M be/src/runtime/io/disk-io-mgr-test.cc M bin/impala-config.sh M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/common/FileSystemUtil.java M fe/src/test/java/org/apache/impala/planner/ExplainTest.java M java/executor-deps/pom.xml M java/pom.xml M testdata/bin/create-load-data.sh M testdata/cluster/node_templates/common/etc/hadoop/conf/core-site.xml.py M tests/common/impala_test_suite.py M tests/common/skip.py M tests/custom_cluster/test_hdfs_fd_caching.py M tests/custom_cluster/test_metastore_service.py M tests/util/filesystem_utils.py 15 files changed, 94 insertions(+), 13 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/65/19165/5 -- To view, visit http://gerrit.cloudera.org:8080/19165 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I267e6531da58e3ac97029fea4c5e075724587910 Gerrit-Change-Number: 19165 Gerrit-PatchSet: 5 Gerrit-Owner: hezhenhua.hzh Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-11683: Support Aliyun OSS File System
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/19165 to look at the new patch set (#4). Change subject: IMPALA-11683: Support Aliyun OSS File System .. IMPALA-11683: Support Aliyun OSS File System Change-Id: I267e6531da58e3ac97029fea4c5e075724587910 --- M be/src/exec/hdfs-table-sink.cc M be/src/runtime/io/disk-io-mgr-test.cc M bin/impala-config.sh M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/common/FileSystemUtil.java M fe/src/test/java/org/apache/impala/planner/ExplainTest.java M java/executor-deps/pom.xml M java/pom.xml M testdata/bin/create-load-data.sh M testdata/cluster/node_templates/common/etc/hadoop/conf/core-site.xml.py M tests/common/impala_test_suite.py M tests/common/skip.py M tests/custom_cluster/test_hdfs_fd_caching.py M tests/custom_cluster/test_metastore_service.py M tests/util/filesystem_utils.py 15 files changed, 92 insertions(+), 11 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/65/19165/4 -- To view, visit http://gerrit.cloudera.org:8080/19165 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I267e6531da58e3ac97029fea4c5e075724587910 Gerrit-Change-Number: 19165 Gerrit-PatchSet: 4 Gerrit-Owner: hezhenhua.hzh Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-11683: Support Aliyun OSS File System
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/19165 to look at the new patch set (#2). Change subject: IMPALA-11683: Support Aliyun OSS File System .. IMPALA-11683: Support Aliyun OSS File System Change-Id: I267e6531da58e3ac97029fea4c5e075724587910 --- M be/src/exec/hdfs-table-sink.cc M be/src/runtime/io/disk-io-mgr-test.cc M bin/impala-config.sh M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/common/FileSystemUtil.java M fe/src/test/java/org/apache/impala/planner/ExplainTest.java M java/executor-deps/pom.xml M java/pom.xml M testdata/bin/create-load-data.sh M testdata/cluster/node_templates/common/etc/hadoop/conf/core-site.xml.py M tests/common/impala_test_suite.py M tests/common/skip.py M tests/custom_cluster/test_hdfs_fd_caching.py M tests/custom_cluster/test_metastore_service.py M tests/util/filesystem_utils.py 15 files changed, 90 insertions(+), 10 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/65/19165/2 -- To view, visit http://gerrit.cloudera.org:8080/19165 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I267e6531da58e3ac97029fea4c5e075724587910 Gerrit-Change-Number: 19165 Gerrit-PatchSet: 2 Gerrit-Owner: hezhenhua.hzh Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-11683: Support Aliyun OSS File System
hezhenhua.hzh has uploaded this change for review. ( http://gerrit.cloudera.org:8080/19165 Change subject: IMPALA-11683: Support Aliyun OSS File System .. IMPALA-11683: Support Aliyun OSS File System Change-Id: I267e6531da58e3ac97029fea4c5e075724587910 --- M be/src/exec/hdfs-table-sink.cc M be/src/runtime/io/disk-io-mgr-test.cc M bin/impala-config.sh M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/common/FileSystemUtil.java M fe/src/test/java/org/apache/impala/planner/ExplainTest.java M java/executor-deps/pom.xml M java/pom.xml M testdata/bin/create-load-data.sh M testdata/cluster/node_templates/common/etc/hadoop/conf/core-site.xml.py M tests/common/impala_test_suite.py M tests/common/skip.py M tests/custom_cluster/test_hdfs_fd_caching.py M tests/custom_cluster/test_metastore_service.py M tests/util/filesystem_utils.py 15 files changed, 90 insertions(+), 11 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/65/19165/1 -- To view, visit http://gerrit.cloudera.org:8080/19165 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I267e6531da58e3ac97029fea4c5e075724587910 Gerrit-Change-Number: 19165 Gerrit-PatchSet: 1 Gerrit-Owner: hezhenhua.hzh
