[Impala-ASF-CR] IMPALA-5351: Support storing column comment of kudu table
Hello Thomas Marshall, Fredy Wijaya, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/12977
to look at the new patch set (#9).
Change subject: IMPALA-5351: Support storing column comment of kudu table
..
IMPALA-5351: Support storing column comment of kudu table
This patch intends to support storing column comment of kudu table
on impala side.
Belows tests passed:
1) creata kudu-table with column comment;
2) alter kudu-table with (add/alter[delete] column comment);
3) show create kudu table;
4) describe kudu-table;
5) invalidate metadata;
6) comment on column is { '' | null | 'comment' }
Change-Id: Ifb3b37eed364f12bdb3c1d7ef5be128f1475936c
---
M fe/src/main/java/org/apache/impala/analysis/AlterTableAlterColStmt.java
M fe/src/main/java/org/apache/impala/catalog/KuduColumn.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/service/KuduCatalogOpExecutor.java
M fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java
M tests/metadata/test_ddl.py
M tests/metadata/test_ddl_base.py
M tests/query_test/test_kudu.py
8 files changed, 95 insertions(+), 24 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/77/12977/9
--
To view, visit http://gerrit.cloudera.org:8080/12977
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ifb3b37eed364f12bdb3c1d7ef5be128f1475936c
Gerrit-Change-Number: 12977
Gerrit-PatchSet: 9
Gerrit-Owner: helifu
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Thomas Marshall
Gerrit-Reviewer: helifu
[Impala-ASF-CR] IMPALA-5351: Support storing column comment of kudu table
helifu has posted comments on this change. (
http://gerrit.cloudera.org:8080/12977 )
Change subject: IMPALA-5351: Support storing column comment of kudu table
..
Patch Set 8:
(1 comment)
--helifu Apr 10 8:47 AM↩
Patch Set 1:
The latest kudu client is required, otherwise the compilation fails
http://gerrit.cloudera.org:8080/#/c/12977/8/tests/metadata/test_ddl.py
File tests/metadata/test_ddl.py:
http://gerrit.cloudera.org:8080/#/c/12977/8/tests/metadata/test_ddl.py@737
PS8, Line 737: self.client.execute("drop table {0}".format(table))
> Instead of dropping, we can just create two different tables for the first
Done
--
To view, visit http://gerrit.cloudera.org:8080/12977
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifb3b37eed364f12bdb3c1d7ef5be128f1475936c
Gerrit-Change-Number: 12977
Gerrit-PatchSet: 8
Gerrit-Owner: helifu
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Thomas Marshall
Gerrit-Reviewer: helifu
Gerrit-Comment-Date: Sun, 28 Apr 2019 05:25:05 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-5351: Support storing column comment of kudu table
Fredy Wijaya has posted comments on this change. (
http://gerrit.cloudera.org:8080/12977 )
Change subject: IMPALA-5351: Support storing column comment of kudu table
..
Patch Set 8:
(1 comment)
> Patch Set 8:
>
> Build Failed
>
> https://jenkins.impala.io/job/gerrit-code-review-checks/2951/ : Initial code
> review checks failed. See linked job for details on the failure.
Looks like some compilation error:
02:30:00 [ERROR] COMPILATION ERROR :
02:30:00 [ERROR]
/home/ubuntu/tmp.Myz4JNT6Cp/fe/src/main/java/org/apache/impala/catalog/KuduColumn.java:[88,32]
cannot find symbol
02:30:00 [ERROR]
/home/ubuntu/tmp.Myz4JNT6Cp/fe/src/main/java/org/apache/impala/catalog/KuduColumn.java:[88,67]
cannot find symbol
02:30:00 [ERROR]
/home/ubuntu/tmp.Myz4JNT6Cp/fe/src/main/java/org/apache/impala/service/KuduCatalogOpExecutor.java:[135,10]
cannot find symbol
02:30:00 [ERROR]
/home/ubuntu/tmp.Myz4JNT6Cp/fe/src/main/java/org/apache/impala/service/KuduCatalogOpExecutor.java:[299,23]
cannot find symbol
02:30:00 [ERROR]
/home/ubuntu/tmp.Myz4JNT6Cp/fe/src/main/java/org/apache/impala/service/KuduCatalogOpExecutor.java:[299,58]
cannot find symbol
02:30:00 [ERROR]
/home/ubuntu/tmp.Myz4JNT6Cp/fe/src/main/java/org/apache/impala/service/KuduCatalogOpExecutor.java:[504,24]
cannot find symbol
02:30:00 [INFO] BUILD FAILURE
02:30:00 [ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:3.3:compile (default-compile) on
project impala-frontend: Compilation failure: Compilation failure:
02:30:00 [ERROR]
/home/ubuntu/tmp.Myz4JNT6Cp/fe/src/main/java/org/apache/impala/catalog/KuduColumn.java:[88,32]
cannot find symbol
02:30:00 [ERROR] symbol: method getComment()
02:30:00 [ERROR] location: variable colSchema of type
org.apache.kudu.ColumnSchema
02:30:00 [ERROR]
/home/ubuntu/tmp.Myz4JNT6Cp/fe/src/main/java/org/apache/impala/catalog/KuduColumn.java:[88,67]
cannot find symbol
02:30:00 [ERROR] symbol: method getComment()
02:30:00 [ERROR] location: variable colSchema of type
org.apache.kudu.ColumnSchema
02:30:00 [ERROR]
/home/ubuntu/tmp.Myz4JNT6Cp/fe/src/main/java/org/apache/impala/service/KuduCatalogOpExecutor.java:[135,10]
cannot find symbol
02:30:00 [ERROR] symbol: method comment(java.lang.String)
02:30:00 [ERROR] location: variable csb of type
org.apache.kudu.ColumnSchema.ColumnSchemaBuilder
02:30:00 [ERROR]
/home/ubuntu/tmp.Myz4JNT6Cp/fe/src/main/java/org/apache/impala/service/KuduCatalogOpExecutor.java:[299,23]
cannot find symbol
02:30:00 [ERROR] symbol: method getComment()
02:30:00 [ERROR] location: variable colSchema of type
org.apache.kudu.ColumnSchema
02:30:00 [ERROR]
/home/ubuntu/tmp.Myz4JNT6Cp/fe/src/main/java/org/apache/impala/service/KuduCatalogOpExecutor.java:[299,58]
cannot find symbol
02:30:00 [ERROR] symbol: method getComment()
02:30:00 [ERROR] location: variable colSchema of type
org.apache.kudu.ColumnSchema
02:30:00 [ERROR]
/home/ubuntu/tmp.Myz4JNT6Cp/fe/src/main/java/org/apache/impala/service/KuduCatalogOpExecutor.java:[504,24]
cannot find symbol
02:30:00 [ERROR] symbol: method
changeComment(java.lang.String,java.lang.String)
02:30:00 [ERROR] location: variable alterTableOptions of type
org.apache.kudu.client.AlterTableOptions
02:30:00 [ERROR] -> [Help 1]
02:30:00 [ERROR]
02:30:00 [ERROR] To see the full stack trace of the errors, re-run Maven with
the -e switch.
02:30:00 [ERROR] Re-run Maven using the -X switch to enable full debug logging.
02:30:00 [ERROR]
02:30:00 [ERROR] For more information about the errors and possible solutions,
please read the following articles:
02:30:00 [ERROR] [Help 1]
http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
http://gerrit.cloudera.org:8080/#/c/12977/8/tests/metadata/test_ddl.py
File tests/metadata/test_ddl.py:
http://gerrit.cloudera.org:8080/#/c/12977/8/tests/metadata/test_ddl.py@737
PS8, Line 737: self.client.execute("drop table {0}".format(table))
Instead of dropping, we can just create two different tables for the first one
and second one. They will be automatically deleted after the test completes.
--
To view, visit http://gerrit.cloudera.org:8080/12977
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifb3b37eed364f12bdb3c1d7ef5be128f1475936c
Gerrit-Change-Number: 12977
Gerrit-PatchSet: 8
Gerrit-Owner: helifu
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Thomas Marshall
Gerrit-Reviewer: helifu
Gerrit-Comment-Date: Sun, 28 Apr 2019 04:56:19 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-5351: Support storing column comment of kudu table
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12977 ) Change subject: IMPALA-5351: Support storing column comment of kudu table .. Patch Set 8: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/2951/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/12977 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifb3b37eed364f12bdb3c1d7ef5be128f1475936c Gerrit-Change-Number: 12977 Gerrit-PatchSet: 8 Gerrit-Owner: helifu Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: helifu Gerrit-Comment-Date: Sun, 28 Apr 2019 02:30:06 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7971: Add support for insert events in event processor.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12889 ) Change subject: IMPALA-7971: Add support for insert events in event processor. .. Patch Set 23: Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/4100/ -- To view, visit http://gerrit.cloudera.org:8080/12889 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7c48c5ca4bde18d532c582980aebbc25f1bf1c52 Gerrit-Change-Number: 12889 Gerrit-PatchSet: 23 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Sun, 28 Apr 2019 01:55:08 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5351: Support storing column comment of kudu table
Hello Thomas Marshall, Fredy Wijaya, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/12977
to look at the new patch set (#8).
Change subject: IMPALA-5351: Support storing column comment of kudu table
..
IMPALA-5351: Support storing column comment of kudu table
This patch intends to support storing column comment of kudu table
on impala side.
Belows tests passed:
1) creata kudu-table with column comment;
2) alter kudu-table with (add/alter[delete] column comment);
3) show create kudu table;
4) describe kudu-table;
5) invalidate metadata;
6) comment on column is { '' | null | 'comment' }
Change-Id: Ifb3b37eed364f12bdb3c1d7ef5be128f1475936c
---
M fe/src/main/java/org/apache/impala/analysis/AlterTableAlterColStmt.java
M fe/src/main/java/org/apache/impala/catalog/KuduColumn.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/service/KuduCatalogOpExecutor.java
M fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java
M tests/metadata/test_ddl.py
M tests/metadata/test_ddl_base.py
M tests/query_test/test_kudu.py
8 files changed, 95 insertions(+), 24 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/77/12977/8
--
To view, visit http://gerrit.cloudera.org:8080/12977
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ifb3b37eed364f12bdb3c1d7ef5be128f1475936c
Gerrit-Change-Number: 12977
Gerrit-PatchSet: 8
Gerrit-Owner: helifu
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Thomas Marshall
Gerrit-Reviewer: helifu
[Impala-ASF-CR] IMPALA-5351: Support storing column comment of kudu table
helifu has posted comments on this change. (
http://gerrit.cloudera.org:8080/12977 )
Change subject: IMPALA-5351: Support storing column comment of kudu table
..
Patch Set 7:
(5 comments)
http://gerrit.cloudera.org:8080/#/c/12977/7/fe/src/main/java/org/apache/impala/catalog/KuduColumn.java
File fe/src/main/java/org/apache/impala/catalog/KuduColumn.java:
http://gerrit.cloudera.org:8080/#/c/12977/7/fe/src/main/java/org/apache/impala/catalog/KuduColumn.java@116
PS7, Line 116:
> nit: remove two extra spaces
Done
http://gerrit.cloudera.org:8080/#/c/12977/7/fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java
File fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java:
http://gerrit.cloudera.org:8080/#/c/12977/7/fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java@2607
PS7, Line 2607:
> nit: remove one extra space for the indentation
Done
http://gerrit.cloudera.org:8080/#/c/12977/7/tests/metadata/test_ddl.py
File tests/metadata/test_ddl.py:
http://gerrit.cloudera.org:8080/#/c/12977/7/tests/metadata/test_ddl.py@733
PS7, Line 733: create table {0} (i int PRIMARY KEY) STORED AS KUDU
> nit: use lower case to be consistent with the style in this file
Done
http://gerrit.cloudera.org:8080/#/c/12977/7/tests/metadata/test_ddl.py@735
PS7, Line 735: comment = self._get_column_comment(table, 'i')
> can we have similar test case for "create table {0} (x int comment 'x', pri
Done
http://gerrit.cloudera.org:8080/#/c/12977/7/tests/metadata/test_ddl_base.py
File tests/metadata/test_ddl_base.py:
http://gerrit.cloudera.org:8080/#/c/12977/7/tests/metadata/test_ddl_base.py@122
PS7, Line 122: or len(cols) == 9
> if len(cols) <= 9 is better since it will work on both kudu and on kudu tab
Done
--
To view, visit http://gerrit.cloudera.org:8080/12977
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifb3b37eed364f12bdb3c1d7ef5be128f1475936c
Gerrit-Change-Number: 12977
Gerrit-PatchSet: 7
Gerrit-Owner: helifu
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Thomas Marshall
Gerrit-Reviewer: helifu
Gerrit-Comment-Date: Sun, 28 Apr 2019 01:35:20 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-5031: signed overflow in TimestampValue
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11919 ) Change subject: IMPALA-5031: signed overflow in TimestampValue .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2950/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11919 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iaad158e6634314a5690a43a0cc04426c1aba8f41 Gerrit-Change-Number: 11919 Gerrit-PatchSet: 4 Gerrit-Owner: Jim Apple Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple Gerrit-Reviewer: Jim Apple Gerrit-Comment-Date: Sun, 28 Apr 2019 00:05:16 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7971: Add support for insert events in event processor.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12889 ) Change subject: IMPALA-7971: Add support for insert events in event processor. .. Patch Set 23: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4100/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/12889 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7c48c5ca4bde18d532c582980aebbc25f1bf1c52 Gerrit-Change-Number: 12889 Gerrit-PatchSet: 23 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Sun, 28 Apr 2019 00:06:02 + Gerrit-HasComments: No
[Impala-ASF-CR] Bump CDH BUILD NUMBER to 1046268.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13140 ) Change subject: Bump CDH_BUILD_NUMBER to 1046268. .. Patch Set 1: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/13140 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7b367d803d396196988842f3fb77896725145ff2 Gerrit-Change-Number: 13140 Gerrit-PatchSet: 1 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Sat, 27 Apr 2019 23:47:33 + Gerrit-HasComments: No
[Impala-ASF-CR] Bump CDH BUILD NUMBER to 1046268.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/13140 ) Change subject: Bump CDH_BUILD_NUMBER to 1046268. .. Bump CDH_BUILD_NUMBER to 1046268. This change brings in the CDH version which fixes an issue preventing hive from firing insert events for insert overwrites into dynamically loaded partitions. This was affecting an incoming impala patch to support insert events processing IMPALA-7971. Testing: Ran a full exaustive build using imapala-private-parametrized job. Change-Id: I7b367d803d396196988842f3fb77896725145ff2 Reviewed-on: http://gerrit.cloudera.org:8080/13140 Reviewed-by: Fredy Wijaya Tested-by: Impala Public Jenkins --- M bin/impala-config.sh 1 file changed, 1 insertion(+), 1 deletion(-) Approvals: Fredy Wijaya: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/13140 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I7b367d803d396196988842f3fb77896725145ff2 Gerrit-Change-Number: 13140 Gerrit-PatchSet: 2 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-8309: add user authorization provider flag
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/12901 ) Change subject: IMPALA-8309: add user authorization_provider flag .. IMPALA-8309: add user authorization_provider flag This commit adds a `authorization_provider` user-facing flag in order to provide a more human-readable alternative to the `authorization_factory_class` for internally-provided authorization strategies. The `authorization_factory_class` flag is retained, but no longer takes a default value if not specified. The default for `authorization_provider` is "sentry" in order to retain backwards-compatibility. If specified, `authorization_factory_class` will take precedence. Testing: - Manually started minicluster with each of following flags and verified correct authorization strategy chosen: - provider='' factory='' => sentry - provider=sentry factory='' => sentry - provider=ranger factory='' => ranger - provider='' factory=sentry => sentry - provider='' factory=ranger => ranger - provider=sentry factory=sentry => sentry - provider=ranger factory=sentry => sentry - provider=sentry factory=ranger => ranger - provider=ranger factory=ranger => ranger - Wrote unit tests to capture above assertions - Ran fe unit and e2e tests - Wrote e2e test to verify new flag behavior Change-Id: I001c20505ba4f0562b60fdef73d15308e8500c19 Reviewed-on: http://gerrit.cloudera.org:8080/12901 Reviewed-by: Fredy Wijaya Tested-by: Impala Public Jenkins --- M be/src/service/frontend.cc M be/src/util/backend-gflag-util.cc M common/thrift/BackendGflags.thrift M fe/src/main/java/org/apache/impala/authorization/AuthorizationConfig.java M fe/src/main/java/org/apache/impala/authorization/AuthorizationProvider.java M fe/src/main/java/org/apache/impala/authorization/NoopAuthorizationFactory.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationConfig.java M fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizationConfig.java M fe/src/main/java/org/apache/impala/service/BackendConfig.java M fe/src/main/java/org/apache/impala/service/JniCatalog.java M fe/src/main/java/org/apache/impala/service/JniFrontend.java A fe/src/main/java/org/apache/impala/util/AuthorizationUtil.java M fe/src/test/java/org/apache/impala/common/FrontendTestBase.java A fe/src/test/java/org/apache/impala/util/AuthorizationUtilTest.java A tests/authorization/test_provider.py M tests/authorization/test_ranger.py 16 files changed, 340 insertions(+), 62 deletions(-) Approvals: Fredy Wijaya: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/12901 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I001c20505ba4f0562b60fdef73d15308e8500c19 Gerrit-Change-Number: 12901 Gerrit-PatchSet: 24 Gerrit-Owner: radford nguyen Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: radford nguyen
[Impala-ASF-CR] IMPALA-8309: add user authorization provider flag
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12901 ) Change subject: IMPALA-8309: add user authorization_provider flag .. Patch Set 23: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/12901 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I001c20505ba4f0562b60fdef73d15308e8500c19 Gerrit-Change-Number: 12901 Gerrit-PatchSet: 23 Gerrit-Owner: radford nguyen Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: radford nguyen Gerrit-Comment-Date: Sat, 27 Apr 2019 23:26:28 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5031: signed overflow in TimestampValue
Jim Apple has posted comments on this change. ( http://gerrit.cloudera.org:8080/11919 ) Change subject: IMPALA-5031: signed overflow in TimestampValue .. Patch Set 4: > Uploaded patch set 4. This has no benchmark implications -- To view, visit http://gerrit.cloudera.org:8080/11919 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iaad158e6634314a5690a43a0cc04426c1aba8f41 Gerrit-Change-Number: 11919 Gerrit-PatchSet: 4 Gerrit-Owner: Jim Apple Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple Gerrit-Reviewer: Jim Apple Gerrit-Comment-Date: Sat, 27 Apr 2019 23:21:24 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5031: signed overflow in TimestampValue
Jim Apple has uploaded a new patch set (#4) to the change originally created by Jim Apple. ( http://gerrit.cloudera.org:8080/11919 ) Change subject: IMPALA-5031: signed overflow in TimestampValue .. IMPALA-5031: signed overflow in TimestampValue The standard says that overflow for signed arithmetic operations is undefined behavior; see [expr]: If during the evaluation of an expression, the result is not mathematically defined or not in the range of representable values for its type, the behavior is undefined. This patch fixes a signed overflow with the folowing backtrace (uninteresting parts elided): runtime/timestamp-value.inline.h:67:13: runtime error: signed integer overflow: -9223372036854775808 + -9223372037 cannot be represented in type 'long' #0 TimestampValue::FromUnixTimeNanos(long, long, cctz::time_zone const&) runtime/timestamp-value.inline.h:67:13 #1 TimestampValue::FromSubsecondUnixTime(double, cctz::time_zone const&) runtime/timestamp-value.inline.h:62:10 #2 CastFunctions::CastToTimestampVal(impala_udf::FunctionContext*, impala_udf::FloatVal const&) exprs/cast-functions-ir.cc:248:172 #3 impala_udf::TimestampVal ScalarFnCall::InterpretEval(ScalarExprEvaluator*, TupleRow const*) const exprs/scalar-fn-call.cc:485:208 #4 ScalarFnCall::GetTimestampVal(ScalarExprEvaluator*, TupleRow const*) const exprs/scalar-fn-call.cc:608:44 #5 ScalarExprEvaluator::GetValue(ScalarExpr const&, TupleRow const*) exprs/scalar-expr-evaluator.cc:314:41 #6 ScalarExprEvaluator::GetValue(TupleRow const*) exprs/scalar-expr-evaluator.cc:250:10 #7 void Tuple::MaterializeExprs(TupleRow*, TupleDescriptor const&, ScalarExprEvaluator* const*, MemPool*, StringValue**, int*, int*) runtime/tuple.cc:222:27 #8 void Tuple::MaterializeExprs(TupleRow*, TupleDescriptor const&, vector const&, MemPool*, vector*, int*) runtime/tuple.h:174:5 #9 UnionNode::MaterializeExprs(vector const&, TupleRow*, unsigned char*, RowBatch*) exec/union-node-ir.cc:29:14 #10 UnionNode::GetNextConst(RuntimeState*, RowBatch*) exec/union-node.cc:263:5 #11 UnionNode::GetNext(RuntimeState*, RowBatch*, bool*) exec/union-node.cc:296:45 This was seen in the backend test ExprTest.CastExprs. Change-Id: Iaad158e6634314a5690a43a0cc04426c1aba8f41 --- M be/src/runtime/timestamp-value.inline.h 1 file changed, 3 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/19/11919/4 -- To view, visit http://gerrit.cloudera.org:8080/11919 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iaad158e6634314a5690a43a0cc04426c1aba8f41 Gerrit-Change-Number: 11919 Gerrit-PatchSet: 4 Gerrit-Owner: Jim Apple Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple Gerrit-Reviewer: Jim Apple
[Impala-ASF-CR] IMPALA-8149 : Add support for alter database events
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13049 ) Change subject: IMPALA-8149 : Add support for alter_database events .. Patch Set 6: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/13049 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iaf020e85cae04163bf32e31363eb4119d624640b Gerrit-Change-Number: 13049 Gerrit-PatchSet: 6 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Sat, 27 Apr 2019 22:32:46 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8149 : Add support for alter database events
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/13049 ) Change subject: IMPALA-8149 : Add support for alter_database events .. IMPALA-8149 : Add support for alter_database events This change adds support for alter_database events in two parts: One is adding catalogServiceId and catalogVersion in db parameters when alter database. The other is adding alter database event, check if it's self event during process, if true do nothing, if false replace caralog cached db with event db. Testing: Enabled testAlterDisableFlagFromDb in MetastoreEventsProcessorTest. Change-Id: Iaf020e85cae04163bf32e31363eb4119d624640b Reviewed-on: http://gerrit.cloudera.org:8080/13049 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M fe/src/main/java/org/apache/impala/catalog/Catalog.java M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/Db.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java 6 files changed, 337 insertions(+), 111 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/13049 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Iaf020e85cae04163bf32e31363eb4119d624640b Gerrit-Change-Number: 13049 Gerrit-PatchSet: 7 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-8465: fix TestJsonEndpoints for remote clusters
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13136 ) Change subject: IMPALA-8465: fix TestJsonEndpoints for remote clusters .. Patch Set 3: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/13136 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8cf6ecfb72ec18be0b5274e360825fb3f203dd20 Gerrit-Change-Number: 13136 Gerrit-PatchSet: 3 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Sat, 27 Apr 2019 22:02:43 + Gerrit-HasComments: No
[Impala-ASF-CR] Bump CDH BUILD NUMBER to 1046268.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13140 ) Change subject: Bump CDH_BUILD_NUMBER to 1046268. .. Patch Set 1: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4099/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/13140 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7b367d803d396196988842f3fb77896725145ff2 Gerrit-Change-Number: 13140 Gerrit-PatchSet: 1 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Sat, 27 Apr 2019 18:15:56 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8309: add user authorization provider flag
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/12901 ) Change subject: IMPALA-8309: add user authorization_provider flag .. Patch Set 23: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/12901 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I001c20505ba4f0562b60fdef73d15308e8500c19 Gerrit-Change-Number: 12901 Gerrit-PatchSet: 23 Gerrit-Owner: radford nguyen Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: radford nguyen Gerrit-Comment-Date: Sat, 27 Apr 2019 18:16:34 + Gerrit-HasComments: No
[Impala-ASF-CR] Bump CDH BUILD NUMBER to 1046268.
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/13140 ) Change subject: Bump CDH_BUILD_NUMBER to 1046268. .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13140 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7b367d803d396196988842f3fb77896725145ff2 Gerrit-Change-Number: 13140 Gerrit-PatchSet: 1 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Sat, 27 Apr 2019 18:15:38 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8309: add user authorization provider flag
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12901 ) Change subject: IMPALA-8309: add user authorization_provider flag .. Patch Set 23: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4098/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/12901 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I001c20505ba4f0562b60fdef73d15308e8500c19 Gerrit-Change-Number: 12901 Gerrit-PatchSet: 23 Gerrit-Owner: radford nguyen Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: radford nguyen Gerrit-Comment-Date: Sat, 27 Apr 2019 18:15:21 + Gerrit-HasComments: No
[Impala-ASF-CR] Bump CDH BUILD NUMBER to 1046268.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13140 ) Change subject: Bump CDH_BUILD_NUMBER to 1046268. .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2949/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13140 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7b367d803d396196988842f3fb77896725145ff2 Gerrit-Change-Number: 13140 Gerrit-PatchSet: 1 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Sat, 27 Apr 2019 17:46:57 + Gerrit-HasComments: No
[Impala-ASF-CR] Bump CDH BUILD NUMBER to 1046268.
Anurag Mantripragada has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13140 Change subject: Bump CDH_BUILD_NUMBER to 1046268. .. Bump CDH_BUILD_NUMBER to 1046268. This change brings in the CDH version which fixes an issue preventing hive from firing insert events for insert overwrites into dynamically loaded partitions. This was affecting an incoming impala patch to support insert events processing IMPALA-7971. Testing: Ran a full exaustive build using imapala-private-parametrized job. Change-Id: I7b367d803d396196988842f3fb77896725145ff2 --- M bin/impala-config.sh 1 file changed, 1 insertion(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/40/13140/1 -- To view, visit http://gerrit.cloudera.org:8080/13140 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I7b367d803d396196988842f3fb77896725145ff2 Gerrit-Change-Number: 13140 Gerrit-PatchSet: 1 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya
[Impala-ASF-CR] IMPALA-8149 : Add support for alter database events
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13049 ) Change subject: IMPALA-8149 : Add support for alter_database events .. Patch Set 6: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4096/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/13049 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iaf020e85cae04163bf32e31363eb4119d624640b Gerrit-Change-Number: 13049 Gerrit-PatchSet: 6 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Sat, 27 Apr 2019 16:49:35 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8149 : Add support for alter database events
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/13049 ) Change subject: IMPALA-8149 : Add support for alter_database events .. Patch Set 6: I think this was probably IMPALA-8466 -- To view, visit http://gerrit.cloudera.org:8080/13049 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iaf020e85cae04163bf32e31363eb4119d624640b Gerrit-Change-Number: 13049 Gerrit-PatchSet: 6 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Sat, 27 Apr 2019 16:49:15 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8465: fix TestJsonEndpoints for remote clusters
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13136 ) Change subject: IMPALA-8465: fix TestJsonEndpoints for remote clusters .. Patch Set 3: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4095/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/13136 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8cf6ecfb72ec18be0b5274e360825fb3f203dd20 Gerrit-Change-Number: 13136 Gerrit-PatchSet: 3 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Sat, 27 Apr 2019 16:47:42 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8309: add user authorization provider flag
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/12901 ) Change subject: IMPALA-8309: add user authorization_provider flag .. Patch Set 23: > Patch Set 23: Verified-1 > > Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/4094/ Failure in unrelated test, possibly flaky test: ImpalaBeeswaxException: ImpalaBeeswaxException: INNER EXCEPTION: MESSAGE: TableLoadingException: Loading file and block metadata for 1 paths for table cachedb.cached_tbl_part: failed to load 1 paths. Check the catalog server log for more details. -- To view, visit http://gerrit.cloudera.org:8080/12901 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I001c20505ba4f0562b60fdef73d15308e8500c19 Gerrit-Change-Number: 12901 Gerrit-PatchSet: 23 Gerrit-Owner: radford nguyen Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: radford nguyen Gerrit-Comment-Date: Sat, 27 Apr 2019 13:32:58 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8149 : Add support for alter database events
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/13049 ) Change subject: IMPALA-8149 : Add support for alter_database events .. Patch Set 6: > Patch Set 6: Verified-1 > > Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/4092/ Failure in unrelated test, possibly flaky test: ImpalaBeeswaxException: ImpalaBeeswaxException: INNER EXCEPTION: MESSAGE: TableLoadingException: Loading file and block metadata for 1 paths for table cachedb.cached_tbl_part: failed to load 1 paths. Check the catalog server log for more details. -- To view, visit http://gerrit.cloudera.org:8080/13049 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iaf020e85cae04163bf32e31363eb4119d624640b Gerrit-Change-Number: 13049 Gerrit-PatchSet: 6 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Sat, 27 Apr 2019 13:32:45 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8309: add user authorization provider flag
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12901 ) Change subject: IMPALA-8309: add user authorization_provider flag .. Patch Set 23: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/4094/ -- To view, visit http://gerrit.cloudera.org:8080/12901 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I001c20505ba4f0562b60fdef73d15308e8500c19 Gerrit-Change-Number: 12901 Gerrit-PatchSet: 23 Gerrit-Owner: radford nguyen Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: radford nguyen Gerrit-Comment-Date: Sat, 27 Apr 2019 10:05:29 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8341: Data cache for remote reads
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12987 ) Change subject: IMPALA-8341: Data cache for remote reads .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2948/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12987 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I734803c1c1787c858dc3ffa0a2c0e33e77b12edc Gerrit-Change-Number: 12987 Gerrit-PatchSet: 5 Gerrit-Owner: Michael Ho Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Sat, 27 Apr 2019 08:06:04 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8341: Data cache for remote reads
Michael Ho has posted comments on this change. ( http://gerrit.cloudera.org:8080/12987 ) Change subject: IMPALA-8341: Data cache for remote reads .. Patch Set 4: (1 comment) http://gerrit.cloudera.org:8080/#/c/12987/4//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/12987/4//COMMIT_MSG@61 PS4, Line 61: Testing done: a new BE test was added; core test with cache enabled. > I have a bit of trouble in getting this to work in the mini-cluster. May be Ended up adding a startup flag to force use of the cache even for local reads. A new custom cluster test was also added for sanity check. -- To view, visit http://gerrit.cloudera.org:8080/12987 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I734803c1c1787c858dc3ffa0a2c0e33e77b12edc Gerrit-Change-Number: 12987 Gerrit-PatchSet: 4 Gerrit-Owner: Michael Ho Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Sat, 27 Apr 2019 07:27:42 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8341: Data cache for remote reads
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12987 ) Change subject: IMPALA-8341: Data cache for remote reads .. Patch Set 5: (1 comment) http://gerrit.cloudera.org:8080/#/c/12987/5/tests/custom_cluster/test_data_cache.py File tests/custom_cluster/test_data_cache.py: http://gerrit.cloudera.org:8080/#/c/12987/5/tests/custom_cluster/test_data_cache.py@22 PS5, Line 22: class TestDataCache(CustomClusterTestSuite): flake8: E302 expected 2 blank lines, found 1 -- To view, visit http://gerrit.cloudera.org:8080/12987 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I734803c1c1787c858dc3ffa0a2c0e33e77b12edc Gerrit-Change-Number: 12987 Gerrit-PatchSet: 5 Gerrit-Owner: Michael Ho Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Sat, 27 Apr 2019 07:23:38 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8341: Data cache for remote reads
Michael Ho has uploaded a new patch set (#5). ( http://gerrit.cloudera.org:8080/12987 ) Change subject: IMPALA-8341: Data cache for remote reads .. IMPALA-8341: Data cache for remote reads This is a patch based on PhilZ's prototype: https://gerrit.cloudera.org/#/c/12683/ This change implements an IO data cache which is backed by local storage. It implicitly relies on the OS page cache management to shuffle data between memory and the storage device. This is useful for caching data read from remote filesystems (e.g. remote HDFS data node, S3, ABFS, ADLS). A data cache is divided into one or more partitions based on the configuration string which is a list of directories, separated by comma, followed by the storage capacity per directory. An example configuration string is like the following: --data_cache_config=/data/0,/data/1:150GB In the configuration above, the cache may use up to 300GB of storage space, with 150GB max for /data/0 and /data/1 respectively. Each partition has a meta-data cache which tracks the mappings of cache keys to the locations of the cached data. A cache key is a tuple of (file's name, file's modification time, file offset) and a cache entry is a tuple of (backing file, offset in the backing file, length of the cached data, optional checksum). Note that the cache currently doesn't support overlapping ranges. In other words, if the cache contains an entry of a file for range [m, m+4MB), a lookup for [m+4K, m+8K) will miss in the cache. In practice, we haven't seen this as a problem but this may require further evaluation in the future. Each partition stores its set of cached data in backing files created on local storage. When inserting new data into the cache, the data is appended to the current backing file in use. The storage consumption of each cache entry counts towards the quota of that partition. When a partition reaches its capacity, the least recently used (LRU) data in that partition is evicted. Evicted data is removed from the underlying storage by punching holes in the backing file it's stored in. As a backing file reaches a certain size (by default 4TB), new data will stop being appended to it and a new file will be created instead. Note that due to hole punching, the backing file is actually sparse. When the number of backing files per partition exceeds, --data_cache_max_files_per_partition, files are deleted in the order in which they are created. Stale cache entries referencing deleted files are erased lazily or evicted due to inactivity. Optionally, checksumming can be enabled to verify read from the cache is consistent with what was inserted and to verify that multiple attempted insertions with the same cache key have the same cache content. Checksumming is enabled by default for debug builds. To probe for cached data in the cache, the interface Lookup() is used; To insert data into the cache, the interface Store() is used. Please note that eviction happens inline currently during Store(). This patch also added two startup flags for start-impala-cluster.py: '--data_cache_dir' specifies the base directory in which each Impalad creates the caching directory '--data_cache_size' specifies the capacity string for each cache directory. Testing done: - added a new BE and EE test - exhaustive (debug, release) builds with cache enabled - core ASAN build with cache enabled Perf: - 16-streams TPCDS at 3TB in a 20 node S3 cluster shows about 30% improvement over runs without the cache. Each node has a cache size of 150GB per node. The performance is at parity with a configuration of a HDFS cluster using EBS as the storage. Change-Id: I734803c1c1787c858dc3ffa0a2c0e33e77b12edc --- M be/src/exec/hdfs-scan-node-base.cc M be/src/exec/hdfs-scan-node-base.h M be/src/runtime/io/CMakeLists.txt A be/src/runtime/io/data-cache-test.cc A be/src/runtime/io/data-cache.cc A be/src/runtime/io/data-cache.h M be/src/runtime/io/disk-io-mgr.cc M be/src/runtime/io/disk-io-mgr.h M be/src/runtime/io/hdfs-file-reader.cc M be/src/runtime/io/hdfs-file-reader.h M be/src/runtime/io/request-context.h M be/src/util/filesystem-util-test.cc M be/src/util/filesystem-util.cc M be/src/util/filesystem-util.h M be/src/util/impalad-metrics.cc M be/src/util/impalad-metrics.h M bin/start-impala-cluster.py M common/thrift/metrics.json A testdata/workloads/functional-query/queries/QueryTest/data-cache.test M tests/common/custom_cluster_test_suite.py M tests/common/impala_test_suite.py A tests/custom_cluster/test_data_cache.py M tests/custom_cluster/test_krpc_metrics.py 23 files changed, 2,059 insertions(+), 55 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/87/12987/5 -- To view, visit http://gerrit.cloudera.org:8080/12987 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id:
[Impala-ASF-CR] IMPALA-8341: Data cache for remote reads
Michael Ho has posted comments on this change. (
http://gerrit.cloudera.org:8080/12987 )
Change subject: IMPALA-8341: Data cache for remote reads
..
Patch Set 5:
(32 comments)
http://gerrit.cloudera.org:8080/#/c/12987/4//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/12987/4//COMMIT_MSG@61
PS4, Line 61: '--data_cache_dir' specifies the base directory in which each
Impalad
> Can we add a custom cluster test to sanity check that it works end to end.
I have a bit of trouble in getting this to work in the mini-cluster. May be
it's easier with the dockerised test.
http://gerrit.cloudera.org:8080/#/c/12987/4//COMMIT_MSG@62
PS4, Line 62: creates the caching directory
> Do we want to consider enabling this by default for the dockerised tests as
Definitely. Will also do so for S3 builds.
http://gerrit.cloudera.org:8080/#/c/12987/4/be/src/exec/hdfs-scan-node-base.cc
File be/src/exec/hdfs-scan-node-base.cc:
http://gerrit.cloudera.org:8080/#/c/12987/4/be/src/exec/hdfs-scan-node-base.cc@365
PS4, Line 365: "DataCacheHitCount", TUnit::UNIT);
> Do we have tests for these to make sure they show up in the profiles and wo
Done
http://gerrit.cloudera.org:8080/#/c/12987/4/be/src/runtime/io/data-cache-test.cc
File be/src/runtime/io/data-cache-test.cc:
http://gerrit.cloudera.org:8080/#/c/12987/4/be/src/runtime/io/data-cache-test.cc@138
PS4, Line 138:
> Move to the other test constants (TEMP_BUFFER_SIZE etc)?
Done
http://gerrit.cloudera.org:8080/#/c/12987/4/be/src/runtime/io/data-cache-test.cc@226
PS4, Line 226:
> nit: This could now be 4 * FLAGS_data_cache_file_max_size ?
Done
http://gerrit.cloudera.org:8080/#/c/12987/4/be/src/runtime/io/data-cache-test.cc@349
PS4, Line 349: num_entries = 0
> Can they be to separate tests?
Done
http://gerrit.cloudera.org:8080/#/c/12987/4/be/src/runtime/io/data-cache-test.cc@410
PS4, Line 410: mp_buf
> nit: typo
Done
http://gerrit.cloudera.org:8080/#/c/12987/4/be/src/runtime/io/data-cache.h
File be/src/runtime/io/data-cache.h:
http://gerrit.cloudera.org:8080/#/c/12987/4/be/src/runtime/io/data-cache.h@83
PS4, Line 83: eviction from it happen s
> That's controlled by --data_cache_write_concurrency, right? Mention here?
Done
http://gerrit.cloudera.org:8080/#/c/12987/4/be/src/runtime/io/data-cache.h@87
PS4, Line 87: /// of 4KB so any data inserted will be rounded up to the nearest
multiple of 4KB.
> Do we plan to look into picking partitions on faster disks with higher prob
Ideally, we want to keep the hotter data in the faster media while keeping the
lukewarm data in the slower media. Added a TODO.
http://gerrit.cloudera.org:8080/#/c/12987/4/be/src/runtime/io/data-cache.h@88
PS4, Line 88: ///
> Yeah this scenario is a bit concerning for me still since it's conceivable
I added a "simple" implementation with rw-lock and lazy cache entry eviction.
If it's deemed too complicated, please let me know and I can undo it. Also
added a test case for it.
http://gerrit.cloudera.org:8080/#/c/12987/4/be/src/runtime/io/data-cache.h@145
PS4, Line 145: ,
> nit: formatting
Done
http://gerrit.cloudera.org:8080/#/c/12987/4/be/src/runtime/io/data-cache.h@197
PS4, Line 197: /// - removes any stale backing file in this partition
> Should we also delete the files when we close them? There's a distinction i
This was needed for data-cache-test.cc as we need to close the files before
verifying their sizes. However, it seems that we can hide all those internal
details in VerifyFileSizes(), which is renamed to CloseAndVerifyFileSizes();
http://gerrit.cloudera.org:8080/#/c/12987/4/be/src/runtime/io/data-cache.h@203
PS4, Line 203:
> Should we pass const CacheKey& here and convert it in the implementation?
Done
http://gerrit.cloudera.org:8080/#/c/12987/4/be/src/runtime/io/data-cache.h@224
PS4, Line 224: void EvictedEnt
> nit: VerifyFileSizes
Done
http://gerrit.cloudera.org:8080/#/c/12987/4/be/src/runtime/io/data-cache.cc
File be/src/runtime/io/data-cache.cc:
http://gerrit.cloudera.org:8080/#/c/12987/4/be/src/runtime/io/data-cache.cc@72
PS4, Line 72: "(Advanced) Enable checksumming for the cached buffer.");
> static const char*?
This is actually a static class member of DataCache.
http://gerrit.cloudera.org:8080/#/c/12987/4/be/src/runtime/io/data-cache.cc@75
PS4, Line 75: namespace io {
> Should this be a class, given it has a c'tor and d'tor?
Done
http://gerrit.cloudera.org:8080/#/c/12987/4/be/src/runtime/io/data-cache.cc@173
PS4, Line 173: ock(lock_.get_lock());
: if (UNLIKELY(!file_)) return fals
> I think you can merge these two lines, which also reduces the risk that som
Done
http://gerrit.cloudera.org:8080/#/c/12987/4/be/src/runtime/io/data-cache.cc@187
PS4, Line 187: inline
> nit: missing include, but we might generally omit this one.
Not sure which one you are referring to ? Isn't it in #include "common/names.h"
?
[Impala-ASF-CR] IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13005 ) Change subject: IMPALA-8369 : Impala should be able to interoperate with Hive 3.1.0 .. Patch Set 11: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/4093/ -- To view, visit http://gerrit.cloudera.org:8080/13005 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I45a4dadbdfe30a02f722dbd917a49bc182fc6436 Gerrit-Change-Number: 13005 Gerrit-PatchSet: 11 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Sudhanshu Arora Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Sat, 27 Apr 2019 07:21:45 + Gerrit-HasComments: No
