[Impala-ASF-CR] IMPALA-11070: Create TestSection class.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18119 ) Change subject: IMPALA-11070: Create TestSection class. .. Patch Set 3: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/7716/ -- To view, visit http://gerrit.cloudera.org:8080/18119 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I91b811d131c3c399ac4f01d7da87759c418905fd Gerrit-Change-Number: 18119 Gerrit-PatchSet: 3 Gerrit-Owner: Steve Carlin Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 05 Jan 2022 05:45:09 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11070: Create TestSection class.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18119 ) Change subject: IMPALA-11070: Create TestSection class. .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/9968/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18119 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I91b811d131c3c399ac4f01d7da87759c418905fd Gerrit-Change-Number: 18119 Gerrit-PatchSet: 3 Gerrit-Owner: Steve Carlin Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 05 Jan 2022 04:29:08 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11070: Create TestSection class.
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/18119 to look at the new patch set (#3). Change subject: IMPALA-11070: Create TestSection class. .. IMPALA-11070: Create TestSection class. Create a TestSection class that holds logic regarding sections in a ".test" file. This commit will contain only movement of code without any logic changes. Future commits will move some of the section related code into TestSection where it belongs, including string replacements, and related sections code (e.g. DML_RESULTS and RESULTS). Change-Id: I91b811d131c3c399ac4f01d7da87759c418905fd --- M testdata/bin/generate-schema-statements.py M tests/common/impala_test_suite.py M tests/common/test_result_verifier.py A tests/common/test_section.py M tests/metadata/test_show_create_table.py M tests/metadata/test_views_compatibility.py M tests/util/test_file_parser.py 7 files changed, 281 insertions(+), 192 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/19/18119/3 -- To view, visit http://gerrit.cloudera.org:8080/18119 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I91b811d131c3c399ac4f01d7da87759c418905fd Gerrit-Change-Number: 18119 Gerrit-PatchSet: 3 Gerrit-Owner: Steve Carlin Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-11070: Create TestSection class.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18119 ) Change subject: IMPALA-11070: Create TestSection class. .. Patch Set 3: (1 comment) http://gerrit.cloudera.org:8080/#/c/18119/3/testdata/bin/generate-schema-statements.py File testdata/bin/generate-schema-statements.py: http://gerrit.cloudera.org:8080/#/c/18119/3/testdata/bin/generate-schema-statements.py@679 PS3, Line 679: E flake8: E501 line too long (107 > 90 characters) -- To view, visit http://gerrit.cloudera.org:8080/18119 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I91b811d131c3c399ac4f01d7da87759c418905fd Gerrit-Change-Number: 18119 Gerrit-PatchSet: 3 Gerrit-Owner: Steve Carlin Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 05 Jan 2022 04:08:09 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11070: Create TestSection class.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18119 ) Change subject: IMPALA-11070: Create TestSection class. .. Patch Set 3: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7716/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/18119 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I91b811d131c3c399ac4f01d7da87759c418905fd Gerrit-Change-Number: 18119 Gerrit-PatchSet: 3 Gerrit-Owner: Steve Carlin Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 05 Jan 2022 04:08:03 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10934 (Part 2): Enable table definition over a single file
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18033 ) Change subject: IMPALA-10934 (Part 2): Enable table definition over a single file .. Patch Set 8: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/9967/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18033 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia4f57f07a9f543c660b102ebf307a6cf590a6784 Gerrit-Change-Number: 18033 Gerrit-PatchSet: 8 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Wed, 05 Jan 2022 03:55:34 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10934 (Part 1): Enable table definition over a single file
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17878 ) Change subject: IMPALA-10934 (Part 1): Enable table definition over a single file .. Patch Set 10: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/9966/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/17878 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I32be936243aa4c8320f5d06d2b7fbf98822f82e7 Gerrit-Change-Number: 17878 Gerrit-PatchSet: 10 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Wed, 05 Jan 2022 03:51:59 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10934 (Part 2): Enable table definition over a single file
Aman Sinha has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/18033 ) Change subject: IMPALA-10934 (Part 2): Enable table definition over a single file .. IMPALA-10934 (Part 2): Enable table definition over a single file This patch adds an end-to-end test to validate and characterize HMS' behavior with respect to external table creation after HIVE-25569 via which a user is allowed to create an external table associated with a single file. Change-Id: Ia4f57f07a9f543c660b102ebf307a6cf590a6784 Reviewed-on: http://gerrit.cloudera.org:8080/18033 Tested-by: Impala Public Jenkins Reviewed-by: Aman Sinha --- M bin/rat_exclude_files.txt A testdata/data/sfs_d1.parq A testdata/data/sfs_d2.txt A testdata/data/sfs_d3.parq A testdata/data/sfs_d4.txt A testdata/workloads/functional-query/queries/QueryTest/sfs.test M tests/common/file_utils.py A tests/query_test/test_sfs.py 8 files changed, 360 insertions(+), 6 deletions(-) Approvals: Impala Public Jenkins: Verified Aman Sinha: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/18033 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ia4f57f07a9f543c660b102ebf307a6cf590a6784 Gerrit-Change-Number: 18033 Gerrit-PatchSet: 9 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell
[Impala-ASF-CR] IMPALA-10934 (Part 1): Enable table definition over a single file
Aman Sinha has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/17878 ) Change subject: IMPALA-10934 (Part 1): Enable table definition over a single file .. IMPALA-10934 (Part 1): Enable table definition over a single file Implements an abstraction layer to show files in a single directory. Impala side part - filesystem drivers are in HIVE-25569. Suppose that the filesystem has a directory in which there are multiple files: hdfs://somedir/f1.txt hdfs://somedir/f2.txt In case of a HMS backed table(s) - the contents of a directory could be considered as table. This patch enables a new file system wrapper 'sfs+' (sfs = single file system) which provides a view of a single file in a directory.' The '+' indicates that this wrapper can be added on top of multiple underlying file systems/object storage such as HDFS, S3 etc. The directory which contains the file could be specified: sfs+hdfs://somedir/f1.txt/#SINGLEFILE# This will be a directory containing only the f1.txt and nothing else. This patch was tested locally - with a custom build of Hive version which also had HIVE-25569. Change-Id: I32be936243aa4c8320f5d06d2b7fbf98822f82e7 Reviewed-on: http://gerrit.cloudera.org:8080/17878 Reviewed-by: Aman Sinha Tested-by: Aman Sinha --- M be/src/exec/hdfs-table-sink.cc M be/src/runtime/io/disk-io-mgr-test.cc M be/src/runtime/io/disk-io-mgr.cc M be/src/runtime/io/disk-io-mgr.h M be/src/util/hdfs-util.cc M be/src/util/hdfs-util.h M fe/src/main/java/org/apache/impala/common/FileSystemUtil.java M java/shaded-deps/hive-exec/pom.xml 8 files changed, 32 insertions(+), 2 deletions(-) Approvals: Aman Sinha: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/17878 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I32be936243aa4c8320f5d06d2b7fbf98822f82e7 Gerrit-Change-Number: 17878 Gerrit-PatchSet: 11 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell
[Impala-ASF-CR] IMPALA-10934 (Part 1): Enable table definition over a single file
Aman Sinha has uploaded a new patch set (#10) to the change originally created by k...@rxd.hu. ( http://gerrit.cloudera.org:8080/17878 ) Change subject: IMPALA-10934 (Part 1): Enable table definition over a single file .. IMPALA-10934 (Part 1): Enable table definition over a single file Implements an abstraction layer to show files in a single directory. Impala side part - filesystem drivers are in HIVE-25569. Suppose that the filesystem has a directory in which there are multiple files: hdfs://somedir/f1.txt hdfs://somedir/f2.txt In case of a HMS backed table(s) - the contents of a directory could be considered as table. This patch enables a new file system wrapper 'sfs+' (sfs = single file system) which provides a view of a single file in a directory.' The '+' indicates that this wrapper can be added on top of multiple underlying file systems/object storage such as HDFS, S3 etc. The directory which contains the file could be specified: sfs+hdfs://somedir/f1.txt/#SINGLEFILE# This will be a directory containing only the f1.txt and nothing else. This patch was tested locally - with a custom build of Hive version which also had HIVE-25569. Change-Id: I32be936243aa4c8320f5d06d2b7fbf98822f82e7 --- M be/src/exec/hdfs-table-sink.cc M be/src/runtime/io/disk-io-mgr-test.cc M be/src/runtime/io/disk-io-mgr.cc M be/src/runtime/io/disk-io-mgr.h M be/src/util/hdfs-util.cc M be/src/util/hdfs-util.h M fe/src/main/java/org/apache/impala/common/FileSystemUtil.java M java/shaded-deps/hive-exec/pom.xml 8 files changed, 32 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/78/17878/10 -- To view, visit http://gerrit.cloudera.org:8080/17878 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I32be936243aa4c8320f5d06d2b7fbf98822f82e7 Gerrit-Change-Number: 17878 Gerrit-PatchSet: 10 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell
[Impala-ASF-CR] IMPALA-10934 (Part 2): Enable table definition over a single file
Aman Sinha has posted comments on this change. ( http://gerrit.cloudera.org:8080/18033 ) Change subject: IMPALA-10934 (Part 2): Enable table definition over a single file .. Patch Set 7: Code-Review+2 Carry forward prior +2 -- To view, visit http://gerrit.cloudera.org:8080/18033 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia4f57f07a9f543c660b102ebf307a6cf590a6784 Gerrit-Change-Number: 18033 Gerrit-PatchSet: 7 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Wed, 05 Jan 2022 03:23:49 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11070: Create TestSection class.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18119 ) Change subject: IMPALA-11070: Create TestSection class. .. Patch Set 2: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/7715/ -- To view, visit http://gerrit.cloudera.org:8080/18119 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I91b811d131c3c399ac4f01d7da87759c418905fd Gerrit-Change-Number: 18119 Gerrit-PatchSet: 2 Gerrit-Owner: Steve Carlin Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 05 Jan 2022 03:08:48 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10934 (Part 2): Enable table definition over a single file
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18033 ) Change subject: IMPALA-10934 (Part 2): Enable table definition over a single file .. Patch Set 7: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/18033 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia4f57f07a9f543c660b102ebf307a6cf590a6784 Gerrit-Change-Number: 18033 Gerrit-PatchSet: 7 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Wed, 05 Jan 2022 02:49:11 + Gerrit-HasComments: No
[native-toolchain-CR] IMPALA-10951 (preparation): Update Kudu to a more recent version
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/18120 ) Change subject: IMPALA-10951 (preparation): Update Kudu to a more recent version .. Patch Set 1: Code-Review+1 Thank you to solve the missing symbol issue. Will give +2 after passing Impala asan build. -- To view, visit http://gerrit.cloudera.org:8080/18120 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If551dbd4454221e8a5c485c5e24cfad2eaa5f2a7 Gerrit-Change-Number: 18120 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Wed, 05 Jan 2022 01:36:40 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11070: Create TestSection class.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18119 ) Change subject: IMPALA-11070: Create TestSection class. .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7715/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/18119 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I91b811d131c3c399ac4f01d7da87759c418905fd Gerrit-Change-Number: 18119 Gerrit-PatchSet: 2 Gerrit-Owner: Steve Carlin Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 05 Jan 2022 01:26:58 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11070: Create TestSection class.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18119 ) Change subject: IMPALA-11070: Create TestSection class. .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/9965/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18119 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I91b811d131c3c399ac4f01d7da87759c418905fd Gerrit-Change-Number: 18119 Gerrit-PatchSet: 2 Gerrit-Owner: Steve Carlin Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 05 Jan 2022 01:14:43 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11070: Create TestSection class.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18119 ) Change subject: IMPALA-11070: Create TestSection class. .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/9964/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18119 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I91b811d131c3c399ac4f01d7da87759c418905fd Gerrit-Change-Number: 18119 Gerrit-PatchSet: 1 Gerrit-Owner: Steve Carlin Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 05 Jan 2022 00:58:49 + Gerrit-HasComments: No
[native-toolchain-CR] IMPALA-10951 (preparation): Update Kudu to a more recent version
Joe McDonnell has uploaded this change for review. (
http://gerrit.cloudera.org:8080/18120
Change subject: IMPALA-10951 (preparation): Update Kudu to a more recent version
..
IMPALA-10951 (preparation): Update Kudu to a more recent version
Kudu has fixed IMPALA-3334, where some protobuf symbols were
being exposed in the Kudu client and causing conflicts when
Impala upgrades protobuf. This updates Kudu to pull in that fix.
As a part of updating Kudu, building with Clang failed at link
time with this error:
libprotobuf.a(dynamic_message.o):dynamic_message.cc:function
google::protobuf::DynamicMessage::~DynamicMessage(): warning: relocation refers
to discarded section
This bumps the version of binutils, which improves the error message
significantly:
TargetPassConfig.cpp:function llvm::TargetPassConfig::createRegAllocPass(bool):
error: relocation refers to global symbol "std::call_once(std::once_flag&, void (&)())::{lambda()#2}::_FUN()", which is defined in
a discarded section
section group signature:
"_ZZSt9call_onceIRFvvEJEEvRSt9once_flagOT_DpOT0_ENKUlvE0_clEv"
prevailing definition is from
../../build/debug/security/libsecurity.a(openssl_util.cc.o)
This also adds the calloncehack component, which builds a shared
library with the missing symbol. When this shared library is linked
ahead of kudu_client, it provides the missing symbol and
avoids the issue.
Change-Id: If551dbd4454221e8a5c485c5e24cfad2eaa5f2a7
---
M buildall.sh
M init.sh
A source/calloncehack/build.sh
A source/calloncehack/calloncehack/CMakeLists.txt
A source/calloncehack/calloncehack/calloncehack.cc
A source/calloncehack/calloncehack/calloncehack.h
6 files changed, 164 insertions(+), 2 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/native-toolchain
refs/changes/20/18120/1
--
To view, visit http://gerrit.cloudera.org:8080/18120
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: native-toolchain
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: If551dbd4454221e8a5c485c5e24cfad2eaa5f2a7
Gerrit-Change-Number: 18120
Gerrit-PatchSet: 1
Gerrit-Owner: Joe McDonnell
[Impala-ASF-CR] IMPALA-11070: Create TestSection class.
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/18119 to look at the new patch set (#2). Change subject: IMPALA-11070: Create TestSection class. .. IMPALA-11070: Create TestSection class. Create a TestSection class that holds logic regarding sections in a ".test" file. This commit will contain only movement of code without any logic changes. Future commits will move some of the section related code into TestSection where it belongs, including string replacements, and related sections code (e.g. DML_RESULTS and RESULTS). Change-Id: I91b811d131c3c399ac4f01d7da87759c418905fd --- M tests/common/impala_test_suite.py M tests/common/test_result_verifier.py A tests/common/test_section.py M tests/metadata/test_show_create_table.py M tests/metadata/test_views_compatibility.py M tests/util/test_file_parser.py 6 files changed, 258 insertions(+), 170 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/19/18119/2 -- To view, visit http://gerrit.cloudera.org:8080/18119 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I91b811d131c3c399ac4f01d7da87759c418905fd Gerrit-Change-Number: 18119 Gerrit-PatchSet: 2 Gerrit-Owner: Steve Carlin Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-10868: cmake dependency on the latest datasketches 3.3.0
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18118 ) Change subject: IMPALA-10868: cmake dependency on the latest datasketches 3.3.0 .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/9963/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18118 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I502ae31d8efd775b5bdeaf272530f7adff04e5b8 Gerrit-Change-Number: 18118 Gerrit-PatchSet: 2 Gerrit-Owner: Alexander Saydakov Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 05 Jan 2022 00:39:33 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11070: Create TestSection class.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18119 ) Change subject: IMPALA-11070: Create TestSection class. .. Patch Set 1: (14 comments) http://gerrit.cloudera.org:8080/#/c/18119/1/tests/common/impala_test_suite.py File tests/common/impala_test_suite.py: http://gerrit.cloudera.org:8080/#/c/18119/1/tests/common/impala_test_suite.py@692 PS1, Line 692: \ flake8: E502 the backslash is redundant between brackets http://gerrit.cloudera.org:8080/#/c/18119/1/tests/common/impala_test_suite.py@744 PS1, Line 744: \ flake8: E502 the backslash is redundant between brackets http://gerrit.cloudera.org:8080/#/c/18119/1/tests/common/test_section.py File tests/common/test_section.py: http://gerrit.cloudera.org:8080/#/c/18119/1/tests/common/test_section.py@27 PS1, Line 27: class TestSection(object): flake8: E302 expected 2 blank lines, found 1 http://gerrit.cloudera.org:8080/#/c/18119/1/tests/common/test_section.py@57 PS1, Line 57: , flake8: W602 deprecated form of raising exception http://gerrit.cloudera.org:8080/#/c/18119/1/tests/common/test_section.py@71 PS1, Line 71: , flake8: W602 deprecated form of raising exception http://gerrit.cloudera.org:8080/#/c/18119/1/tests/common/test_section.py@75 PS1, Line 75: = flake8: E711 comparison to None should be 'if cond is None:' http://gerrit.cloudera.org:8080/#/c/18119/1/tests/common/test_section.py@80 PS1, Line 80: , flake8: W602 deprecated form of raising exception http://gerrit.cloudera.org:8080/#/c/18119/1/tests/common/test_section.py@93 PS1, Line 93: , flake8: W602 deprecated form of raising exception http://gerrit.cloudera.org:8080/#/c/18119/1/tests/common/test_section.py@107 PS1, Line 107: , flake8: W602 deprecated form of raising exception http://gerrit.cloudera.org:8080/#/c/18119/1/tests/common/test_section.py@111 PS1, Line 111: , flake8: W602 deprecated form of raising exception http://gerrit.cloudera.org:8080/#/c/18119/1/tests/common/test_section.py@128 PS1, Line 128: flake8: W291 trailing whitespace http://gerrit.cloudera.org:8080/#/c/18119/1/tests/common/test_section.py@128 PS1, Line 128: return default_val line has trailing whitespace http://gerrit.cloudera.org:8080/#/c/18119/1/tests/common/test_section.py@139 PS1, Line 139: def test_section_create(section_text, valid_section_names=None, skip_unknown_sections=False): flake8: E302 expected 2 blank lines, found 1 http://gerrit.cloudera.org:8080/#/c/18119/1/tests/common/test_section.py@139 PS1, Line 139: e flake8: E501 line too long (93 > 90 characters) -- To view, visit http://gerrit.cloudera.org:8080/18119 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I91b811d131c3c399ac4f01d7da87759c418905fd Gerrit-Change-Number: 18119 Gerrit-PatchSet: 1 Gerrit-Owner: Steve Carlin Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 05 Jan 2022 00:35:40 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11070: Create TestSection class.
Steve Carlin has uploaded this change for review. ( http://gerrit.cloudera.org:8080/18119 Change subject: IMPALA-11070: Create TestSection class. .. IMPALA-11070: Create TestSection class. Create a TestSection class that holds logic regarding sections in a ".test" file. This commit will contain only movement of code without any logic changes. Future commits will move some of the section related code into TestSection where it belongs, including string replacements, and related sections code (e.g. DML_RESULTS and RESULTS). Change-Id: I91b811d131c3c399ac4f01d7da87759c418905fd --- M tests/common/impala_test_suite.py M tests/common/test_result_verifier.py A tests/common/test_section.py M tests/metadata/test_show_create_table.py M tests/metadata/test_views_compatibility.py M tests/util/test_file_parser.py 6 files changed, 255 insertions(+), 170 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/19/18119/1 -- To view, visit http://gerrit.cloudera.org:8080/18119 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I91b811d131c3c399ac4f01d7da87759c418905fd Gerrit-Change-Number: 18119 Gerrit-PatchSet: 1 Gerrit-Owner: Steve Carlin
[Impala-ASF-CR] IMPALA-10868: cmake dependency on the latest datasketches 3.3.0
Alexander Saydakov has uploaded this change for review. ( http://gerrit.cloudera.org:8080/18118 Change subject: IMPALA-10868: cmake dependency on the latest datasketches 3.3.0 .. IMPALA-10868: cmake dependency on the latest datasketches 3.3.0 - added dependency on datasketches-cpp-3.3.0 using cmake - removed older checked-in version of datasketches Change-Id: I502ae31d8efd775b5bdeaf272530f7adff04e5b8 --- M CMakeLists.txt M be/CMakeLists.txt M be/src/exprs/CMakeLists.txt M be/src/exprs/aggregate-functions-ir.cc M be/src/exprs/datasketches-common.cc M be/src/exprs/datasketches-common.h M be/src/exprs/datasketches-functions-ir.cc M be/src/exprs/datasketches-test.cc D be/src/thirdparty/datasketches/AuxHashMap-internal.hpp D be/src/thirdparty/datasketches/AuxHashMap.hpp D be/src/thirdparty/datasketches/CompositeInterpolationXTable-internal.hpp D be/src/thirdparty/datasketches/CompositeInterpolationXTable.hpp D be/src/thirdparty/datasketches/CouponHashSet-internal.hpp D be/src/thirdparty/datasketches/CouponHashSet.hpp D be/src/thirdparty/datasketches/CouponList-internal.hpp D be/src/thirdparty/datasketches/CouponList.hpp D be/src/thirdparty/datasketches/CubicInterpolation-internal.hpp D be/src/thirdparty/datasketches/CubicInterpolation.hpp D be/src/thirdparty/datasketches/HarmonicNumbers-internal.hpp D be/src/thirdparty/datasketches/HarmonicNumbers.hpp D be/src/thirdparty/datasketches/Hll4Array-internal.hpp D be/src/thirdparty/datasketches/Hll4Array.hpp D be/src/thirdparty/datasketches/Hll6Array-internal.hpp D be/src/thirdparty/datasketches/Hll6Array.hpp D be/src/thirdparty/datasketches/Hll8Array-internal.hpp D be/src/thirdparty/datasketches/Hll8Array.hpp D be/src/thirdparty/datasketches/HllArray-internal.hpp D be/src/thirdparty/datasketches/HllArray.hpp D be/src/thirdparty/datasketches/HllSketch-internal.hpp D be/src/thirdparty/datasketches/HllSketchImpl-internal.hpp D be/src/thirdparty/datasketches/HllSketchImpl.hpp D be/src/thirdparty/datasketches/HllSketchImplFactory.hpp D be/src/thirdparty/datasketches/HllUnion-internal.hpp D be/src/thirdparty/datasketches/HllUtil.hpp D be/src/thirdparty/datasketches/LICENSE D be/src/thirdparty/datasketches/MurmurHash3.h D be/src/thirdparty/datasketches/README.md D be/src/thirdparty/datasketches/RelativeErrorTables-internal.hpp D be/src/thirdparty/datasketches/RelativeErrorTables.hpp D be/src/thirdparty/datasketches/binomial_bounds.hpp D be/src/thirdparty/datasketches/bounds_binomial_proportions.hpp D be/src/thirdparty/datasketches/bounds_on_ratios_in_sampled_sets.hpp D be/src/thirdparty/datasketches/bounds_on_ratios_in_theta_sketched_sets.hpp D be/src/thirdparty/datasketches/ceiling_power_of_2.hpp D be/src/thirdparty/datasketches/common_defs.hpp D be/src/thirdparty/datasketches/compression_data.hpp D be/src/thirdparty/datasketches/conditional_back_inserter.hpp D be/src/thirdparty/datasketches/conditional_forward.hpp D be/src/thirdparty/datasketches/count_zeros.hpp D be/src/thirdparty/datasketches/coupon_iterator-internal.hpp D be/src/thirdparty/datasketches/coupon_iterator.hpp D be/src/thirdparty/datasketches/cpc_common.hpp D be/src/thirdparty/datasketches/cpc_compressor.hpp D be/src/thirdparty/datasketches/cpc_compressor_impl.hpp D be/src/thirdparty/datasketches/cpc_confidence.hpp D be/src/thirdparty/datasketches/cpc_sketch.hpp D be/src/thirdparty/datasketches/cpc_sketch_impl.hpp D be/src/thirdparty/datasketches/cpc_union.hpp D be/src/thirdparty/datasketches/cpc_union_impl.hpp D be/src/thirdparty/datasketches/cpc_util.hpp D be/src/thirdparty/datasketches/hll.hpp D be/src/thirdparty/datasketches/hll.private.hpp D be/src/thirdparty/datasketches/icon_estimator.hpp D be/src/thirdparty/datasketches/inv_pow2_table.hpp D be/src/thirdparty/datasketches/kll_helper.hpp D be/src/thirdparty/datasketches/kll_helper_impl.hpp D be/src/thirdparty/datasketches/kll_quantile_calculator.hpp D be/src/thirdparty/datasketches/kll_quantile_calculator_impl.hpp D be/src/thirdparty/datasketches/kll_sketch.hpp D be/src/thirdparty/datasketches/kll_sketch_impl.hpp D be/src/thirdparty/datasketches/kxp_byte_lookup.hpp D be/src/thirdparty/datasketches/memory_operations.hpp D be/src/thirdparty/datasketches/serde.hpp D be/src/thirdparty/datasketches/theta_a_not_b.hpp D be/src/thirdparty/datasketches/theta_a_not_b_impl.hpp D be/src/thirdparty/datasketches/theta_comparators.hpp D be/src/thirdparty/datasketches/theta_constants.hpp D be/src/thirdparty/datasketches/theta_helpers.hpp D be/src/thirdparty/datasketches/theta_intersection.hpp D be/src/thirdparty/datasketches/theta_intersection_base.hpp D be/src/thirdparty/datasketches/theta_intersection_base_impl.hpp D be/src/thirdparty/datasketches/theta_intersection_impl.hpp D be/src/thirdparty/datasketches/theta_jaccard_similarity.hpp D be/src/thirdparty/datasketches/theta_jaccard_similarity_base.hpp D be/src/thirdparty/datasketches/theta_set_difference_base.hpp D
[Impala-ASF-CR] IMPALA-11050: Skip filemetadata reloading in processing AlterPartition event from event processor
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18083 ) Change subject: IMPALA-11050: Skip filemetadata reloading in processing AlterPartition event from event processor .. Patch Set 8: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/7713/ -- To view, visit http://gerrit.cloudera.org:8080/18083 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I238b169f7f1122c62cbeb1434dbb675629a1e5f2 Gerrit-Change-Number: 18083 Gerrit-PatchSet: 8 Gerrit-Owner: Sourabh Goyal Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sourabh Goyal Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yu-Wen Lai Gerrit-Comment-Date: Tue, 04 Jan 2022 22:22:58 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10934 (Part 2): Enable table definition over a single file
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18033 ) Change subject: IMPALA-10934 (Part 2): Enable table definition over a single file .. Patch Set 7: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/9962/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18033 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia4f57f07a9f543c660b102ebf307a6cf590a6784 Gerrit-Change-Number: 18033 Gerrit-PatchSet: 7 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Tue, 04 Jan 2022 20:30:24 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10934 (Part 2): Enable table definition over a single file
Fang-Yu Rao has posted comments on this change. ( http://gerrit.cloudera.org:8080/18033 ) Change subject: IMPALA-10934 (Part 2): Enable table definition over a single file .. Patch Set 7: > Patch Set 7: > > > Patch Set 6: > > > > > Patch Set 6: > > > > > > (1 comment) > > > > > > > Patch Set 6: Verified-1 > > > > > > > > Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/7712/ > > > > > > Fang-Yu, could you address the LOCATION related failure as mentioned > > > below? > > > More details of the error: > > > https://jenkins.impala.io/job/ubuntu-16.04-dockerised-tests/5080/ > > > > Taking a closer look at the console output of the Jenkins job, I think I > > should use $DEFAULT_FS to replace each occurrence of > > "hdfs://localhost:20500" since $DEFAULT_FS is not necessarily > > "hdfs://localhost:20500" in every test environment. > > In the updated patch set (patch set 7), I replaced each occurrence of > "localhost" with $INTERNAL_LISTEN_HOST since this is a string that will be > replaced in our test framework according to > https://github.com/apache/impala/blob/master/tests/common/impala_test_suite.py#L534L535. The link above was wrong. I think it should be https://github.com/apache/impala/blob/master/tests/common/impala_test_suite.py#L470 instead. But the patch should still pass according to my current understanding. -- To view, visit http://gerrit.cloudera.org:8080/18033 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia4f57f07a9f543c660b102ebf307a6cf590a6784 Gerrit-Change-Number: 18033 Gerrit-PatchSet: 7 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Tue, 04 Jan 2022 20:29:19 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10934 (Part 2): Enable table definition over a single file
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18033 ) Change subject: IMPALA-10934 (Part 2): Enable table definition over a single file .. Patch Set 7: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7714/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/18033 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia4f57f07a9f543c660b102ebf307a6cf590a6784 Gerrit-Change-Number: 18033 Gerrit-PatchSet: 7 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Tue, 04 Jan 2022 20:27:49 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10934 (Part 2): Enable table definition over a single file
Fang-Yu Rao has posted comments on this change. ( http://gerrit.cloudera.org:8080/18033 ) Change subject: IMPALA-10934 (Part 2): Enable table definition over a single file .. Patch Set 7: Hi all, I have fixed sfs.test to not hardcode the host name to "localhost". Hopefully this time the pre-commit tests could pass. -- To view, visit http://gerrit.cloudera.org:8080/18033 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia4f57f07a9f543c660b102ebf307a6cf590a6784 Gerrit-Change-Number: 18033 Gerrit-PatchSet: 7 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Tue, 04 Jan 2022 20:12:42 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10934 (Part 2): Enable table definition over a single file
Fang-Yu Rao has posted comments on this change. ( http://gerrit.cloudera.org:8080/18033 ) Change subject: IMPALA-10934 (Part 2): Enable table definition over a single file .. Patch Set 7: > Patch Set 6: > > > Patch Set 6: > > > > (1 comment) > > > > > Patch Set 6: Verified-1 > > > > > > Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/7712/ > > > > Fang-Yu, could you address the LOCATION related failure as mentioned below? > > More details of the error: > > https://jenkins.impala.io/job/ubuntu-16.04-dockerised-tests/5080/ > > Taking a closer look at the console output of the Jenkins job, I think I > should use $DEFAULT_FS to replace each occurrence of "hdfs://localhost:20500" > since $DEFAULT_FS is not necessarily "hdfs://localhost:20500" in every test > environment. In the updated patch set (patch set 7), I replaced each occurrence of "localhost" with $INTERNAL_LISTEN_HOST since this is a string that will be replaced in our test framework according to https://github.com/apache/impala/blob/master/tests/common/impala_test_suite.py#L534L535. -- To view, visit http://gerrit.cloudera.org:8080/18033 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia4f57f07a9f543c660b102ebf307a6cf590a6784 Gerrit-Change-Number: 18033 Gerrit-PatchSet: 7 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Tue, 04 Jan 2022 20:11:13 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10934 (Part 2): Enable table definition over a single file
Fang-Yu Rao has uploaded a new patch set (#7). ( http://gerrit.cloudera.org:8080/18033 ) Change subject: IMPALA-10934 (Part 2): Enable table definition over a single file .. IMPALA-10934 (Part 2): Enable table definition over a single file This patch adds an end-to-end test to validate and characterize HMS' behavior with respect to external table creation after HIVE-25569 via which a user is allowed to create an external table associated with a single file. Change-Id: Ia4f57f07a9f543c660b102ebf307a6cf590a6784 --- M bin/rat_exclude_files.txt A testdata/data/sfs_d1.parq A testdata/data/sfs_d2.txt A testdata/data/sfs_d3.parq A testdata/data/sfs_d4.txt A testdata/workloads/functional-query/queries/QueryTest/sfs.test M tests/common/file_utils.py A tests/query_test/test_sfs.py 8 files changed, 360 insertions(+), 6 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/33/18033/7 -- To view, visit http://gerrit.cloudera.org:8080/18033 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ia4f57f07a9f543c660b102ebf307a6cf590a6784 Gerrit-Change-Number: 18033 Gerrit-PatchSet: 7 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell
[Impala-ASF-CR] IMPALA-11046: Change GetTupleIdx assert to a fail.
Steve Carlin has abandoned this change. ( http://gerrit.cloudera.org:8080/18065 ) Change subject: IMPALA-11046: Change GetTupleIdx assert to a fail. .. Abandoned -- To view, visit http://gerrit.cloudera.org:8080/18065 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: abandon Gerrit-Change-Id: I15767b7adc30b4e6fdcf2e76b673a61739e5a289 Gerrit-Change-Number: 18065 Gerrit-PatchSet: 1 Gerrit-Owner: Steve Carlin Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-10934 (Part 2): Enable table definition over a single file
Fang-Yu Rao has posted comments on this change. ( http://gerrit.cloudera.org:8080/18033 ) Change subject: IMPALA-10934 (Part 2): Enable table definition over a single file .. Patch Set 6: > Patch Set 6: > > (1 comment) > > > Patch Set 6: Verified-1 > > > > Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/7712/ > > Fang-Yu, could you address the LOCATION related failure as mentioned below? > More details of the error: > https://jenkins.impala.io/job/ubuntu-16.04-dockerised-tests/5080/ Taking a closer look at the console output of the Jenkins job, I think I should use $DEFAULT_FS to replace each occurrence of "hdfs://localhost:20500" since $DEFAULT_FS is not necessarily "hdfs://localhost:20500" in every test environment. -- To view, visit http://gerrit.cloudera.org:8080/18033 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia4f57f07a9f543c660b102ebf307a6cf590a6784 Gerrit-Change-Number: 18033 Gerrit-PatchSet: 6 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Tue, 04 Jan 2022 17:30:08 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10910, IMPALA-5509: Runtime filter: dictionary filter support
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/18017 ) Change subject: IMPALA-10910, IMPALA-5509: Runtime filter: dictionary filter support .. Patch Set 4: (1 comment) http://gerrit.cloudera.org:8080/#/c/18017/4/be/src/exec/parquet/hdfs-parquet-scanner.cc File be/src/exec/parquet/hdfs-parquet-scanner.cc: http://gerrit.cloudera.org:8080/#/c/18017/4/be/src/exec/parquet/hdfs-parquet-scanner.cc@1918 PS4, Line 1918: break; > My understanding is that runtime filters are in AND relationship, so all of Yeah, sometimes there is no clear answer to who rejected the row group. Probably we should just increment the "Filter processed" counter for the filters that were actually evaluated. And use 'num_dict_filtered_row_groups_counter_' to count the number of rejected row groups, either by conjuncts or runtime filters, or by their combination. Sometimes there's a clear answer to who rejected the row group. E.g. if a filter was evaluated to all entries and it was always false, then we have a clear winner. -- To view, visit http://gerrit.cloudera.org:8080/18017 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ida0ada8799774be34312eaa4be47336149f637c7 Gerrit-Change-Number: 18017 Gerrit-PatchSet: 4 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Amogh Margoor Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 04 Jan 2022 13:50:21 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11050: Skip filemetadata reloading in processing AlterPartition event from event processor
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18083 ) Change subject: IMPALA-11050: Skip filemetadata reloading in processing AlterPartition event from event processor .. Patch Set 8: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7713/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/18083 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I238b169f7f1122c62cbeb1434dbb675629a1e5f2 Gerrit-Change-Number: 18083 Gerrit-PatchSet: 8 Gerrit-Owner: Sourabh Goyal Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sourabh Goyal Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yu-Wen Lai Gerrit-Comment-Date: Tue, 04 Jan 2022 12:22:45 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11050: Skip filemetadata reloading in processing AlterPartition event from event processor
Sourabh Goyal has posted comments on this change. ( http://gerrit.cloudera.org:8080/18083 ) Change subject: IMPALA-11050: Skip filemetadata reloading in processing AlterPartition event from event processor .. Patch Set 8: Please review this patch and share your feedback. -- To view, visit http://gerrit.cloudera.org:8080/18083 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I238b169f7f1122c62cbeb1434dbb675629a1e5f2 Gerrit-Change-Number: 18083 Gerrit-PatchSet: 8 Gerrit-Owner: Sourabh Goyal Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sourabh Goyal Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yu-Wen Lai Gerrit-Comment-Date: Tue, 04 Jan 2022 12:19:39 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11050: Skip filemetadata reloading in processing AlterPartition event from event processor
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/18083 to look at the new patch set (#8). Change subject: IMPALA-11050: Skip filemetadata reloading in processing AlterPartition event from event processor .. IMPALA-11050: Skip filemetadata reloading in processing AlterPartition event from event processor HdfsPartition in catalogD is a collection of files and each file is represented by a FileDescriptor. Currently when processing an AlterPartitionEvent, EventProcessor reloads partition metadata along with its file metadata. File metadata reloading is an expensive operation as it involves listing files in the underlying filesystem. This patch makes an improvement alter partition event processing for external tables. The proposed solution is to compare storage descriptor of HMS partition object received from AlterPartition event with sd of already cached HdfsPartition. Reload file metadata if any of the following is true * HdfsParition's sd does not match sd of HMS partition object * Cache directive is present for HdfsPartition. File metadata reloading is required to fetch latest file blocks info. Testing: * Added new unit tests Change-Id: I238b169f7f1122c62cbeb1434dbb675629a1e5f2 --- A fe/src/main/java/org/apache/impala/catalog/FileMetadataLoadOpts.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java 5 files changed, 315 insertions(+), 29 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/83/18083/8 -- To view, visit http://gerrit.cloudera.org:8080/18083 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I238b169f7f1122c62cbeb1434dbb675629a1e5f2 Gerrit-Change-Number: 18083 Gerrit-PatchSet: 8 Gerrit-Owner: Sourabh Goyal Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-10910, IMPALA-5509: Runtime filter: dictionary filter support
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/18017 ) Change subject: IMPALA-10910, IMPALA-5509: Runtime filter: dictionary filter support .. Patch Set 4: (1 comment) http://gerrit.cloudera.org:8080/#/c/18017/4/be/src/exec/parquet/hdfs-parquet-scanner.cc File be/src/exec/parquet/hdfs-parquet-scanner.cc: http://gerrit.cloudera.org:8080/#/c/18017/4/be/src/exec/parquet/hdfs-parquet-scanner.cc@1918 PS4, Line 1918: break; > This is interesting. Maybe we can also get rid of the 'vector column_ My understanding is that runtime filters are in AND relationship, so all of them has to pass, otherwise the row will be dropped. See https://github.com/apache/impala/blob/9ed4b3689784670532e840c5cb0389bdd9d5c0e8/be/src/exec/hdfs-scanner.cc#L897 The same logic should be applied to dictionary entries to filter whole row groups. About accounting "who rejected the row group" a similar logic is implemented in https://github.com/apache/impala/blob/master/be/src/exec/hdfs-scan-node-base.cc#L928 for files. It only registers the first filter that failed and doesn't check the filters after that. Actually in case of dictionaries, we cannot really say "who" rejected the row group, as one dict entry can be rejected by a predicate, another by a runtime filter, a third one by a different runtime filter, etc. My idea would be the the track the "last" runtime filter (rf_idx) that rejected entries - if the the row group is skipped, then all entries were rejected in one way ore another, and if if max rf_idx was e.g. 3, so we never reached 4, that means that we would have skipped it even if filter 4 was not there at all. -- To view, visit http://gerrit.cloudera.org:8080/18017 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ida0ada8799774be34312eaa4be47336149f637c7 Gerrit-Change-Number: 18017 Gerrit-PatchSet: 4 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Amogh Margoor Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 04 Jan 2022 09:40:43 + Gerrit-HasComments: Yes
