[Impala-ASF-CR] [tools] Add Dev Container support for Impala development.
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/20380 ) Change subject: [tools] Add Dev Container support for Impala development. .. Patch Set 5: > Patch Set 3: Code-Review+2 > > LGTM. Welcome back to the community, Fredy! We really miss you. > > BTW, it'd be nice if you could also update the wiki about this: > https://cwiki.apache.org/confluence/display/IMPALA/Impala+Development+Environment+inside+Docker Thanks Quanlong for the LGTM :) Done. Updated the wiki. -- To view, visit http://gerrit.cloudera.org:8080/20380 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I50508a09710641ec2a299b001fef3e7fefb0b7d5 Gerrit-Change-Number: 20380 Gerrit-PatchSet: 5 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Wed, 23 Aug 2023 16:00:30 + Gerrit-HasComments: No
[Impala-ASF-CR] [tools] Add Dev Container support for Impala development.
Fredy Wijaya has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/20380 ) Change subject: [tools] Add Dev Container support for Impala development. .. [tools] Add Dev Container support for Impala development. Currently only VS Code is supported since IntelliJ/CLion support for Dev Container is still beta at the time of this writing. To use it, simply open Impala source code. $ git clone https://github.com/apache/impala.git $ cd impala $ code . The bootstrap_development.sh will be automatically executed post Docker container creation and all necessary extensions for IDE-like experience will be automatically installed. For C++, it will use clangd that uses compilation database instead of the Microsoft C++ extension since it works better with Clang related tools. Change-Id: I50508a09710641ec2a299b001fef3e7fefb0b7d5 --- A .devcontainer/Dockerfile A .devcontainer/devcontainer.json M bin/rat_exclude_files.txt 3 files changed, 33 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/80/20380/3 -- To view, visit http://gerrit.cloudera.org:8080/20380 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I50508a09710641ec2a299b001fef3e7fefb0b7d5 Gerrit-Change-Number: 20380 Gerrit-PatchSet: 3 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Michael Smith
[Impala-ASF-CR] [tools] Add Dev Container support for Impala development.
Fredy Wijaya has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/20380 ) Change subject: [tools] Add Dev Container support for Impala development. .. [tools] Add Dev Container support for Impala development. Currently only VS Code is supported since IntelliJ/CLion support for Dev Container is still beta at the time of this writing. To use it, simply open Impala source code. $ git clone https://github.com/apache/impala.git $ cd impala $ code . The bootstrap_development.sh will be automatically executed post Docker container creation and all necessary extensions for IDE-like experience will be automatically installed. For C++, it will use clangd that uses compilation database instead of the Microsoft C++ extension since it works better with Clang related tools. Change-Id: I50508a09710641ec2a299b001fef3e7fefb0b7d5 --- A .devcontainer/Dockerfile A .devcontainer/devcontainer.json 2 files changed, 32 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/80/20380/2 -- To view, visit http://gerrit.cloudera.org:8080/20380 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I50508a09710641ec2a299b001fef3e7fefb0b7d5 Gerrit-Change-Number: 20380 Gerrit-PatchSet: 2 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Michael Smith
[Impala-ASF-CR] [tools] Add Dev Container support for Impala development.
Fredy Wijaya has uploaded this change for review. ( http://gerrit.cloudera.org:8080/20380 Change subject: [tools] Add Dev Container support for Impala development. .. [tools] Add Dev Container support for Impala development. Currently only VS Code is supported since IntelliJ/CLion support for Dev Container is still beta at the time of this writing. To use it, simply open Impala source code. $ git clone https://github.com/apache/impala.git $ cd impala $ code . The bootstrap_development.sh will be automatically executed post Docker container creation and all necesary extensions for IDE-like experience will be automatically installed. For C++, it uses clangd that uses compilation database instead of the Microsoft C++ extension since it works better with Clang related tools. Change-Id: I50508a09710641ec2a299b001fef3e7fefb0b7d5 --- A .devcontainer/Dockerfile A .devcontainer/devcontainer.json 2 files changed, 32 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/80/20380/1 -- To view, visit http://gerrit.cloudera.org:8080/20380 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I50508a09710641ec2a299b001fef3e7fefb0b7d5 Gerrit-Change-Number: 20380 Gerrit-PatchSet: 1 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal
[Impala-ASF-CR] IMPALA-10051: impala-shell exits with ValueError with WITH clauses
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/16389 ) Change subject: IMPALA-10051: impala-shell exits with ValueError with WITH clauses .. Patch Set 5: > Patch Set 5: > > Hi Fredy, > Just checking in if you had some time to review the current version. LGTM. Thanks for fixing this! -- To view, visit http://gerrit.cloudera.org:8080/16389 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I442d3bc65b90a55c73c847948d5179a8586d71ad Gerrit-Change-Number: 16389 Gerrit-PatchSet: 5 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Tamas Mate Gerrit-Comment-Date: Tue, 15 Sep 2020 16:30:48 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10051: impala-shell exits with ValueError with WITH clauses
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/16389 ) Change subject: IMPALA-10051: impala-shell exits with ValueError with WITH clauses .. Patch Set 5: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/16389 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I442d3bc65b90a55c73c847948d5179a8586d71ad Gerrit-Change-Number: 16389 Gerrit-PatchSet: 5 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Tamas Mate Gerrit-Comment-Date: Tue, 15 Sep 2020 16:30:23 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10051: impala-shell exits with ValueError with WITH clauses
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/16389 ) Change subject: IMPALA-10051: impala-shell exits with ValueError with WITH clauses .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/16389 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I442d3bc65b90a55c73c847948d5179a8586d71ad Gerrit-Change-Number: 16389 Gerrit-PatchSet: 3 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Comment-Date: Tue, 01 Sep 2020 14:28:06 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10051: impala-shell exits with ValueError with WITH clauses
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/16389 ) Change subject: IMPALA-10051: impala-shell exits with ValueError with WITH clauses .. Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/16389/2/shell/impala_shell.py File shell/impala_shell.py: http://gerrit.cloudera.org:8080/#/c/16389/2/shell/impala_shell.py@1280 PS2, Line 1280: if self.DML_REGEX.match(query_type.lower()): looks like there were failed tests in the dry-run nit: this code can be simplified like below. is_dml = self.DML_REGEX.match(query_type.lower()) return self._execute_stmt(query, is_dml=is_dml, print_web_link=True) -- To view, visit http://gerrit.cloudera.org:8080/16389 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I442d3bc65b90a55c73c847948d5179a8586d71ad Gerrit-Change-Number: 16389 Gerrit-PatchSet: 2 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 31 Aug 2020 22:06:18 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8228: Ownership support for Ranger authz
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/14106 ) Change subject: IMPALA-8228: Ownership support for Ranger authz .. Patch Set 11: Code-Review+2 (3 comments) LGTM. http://gerrit.cloudera.org:8080/#/c/14106/11/fe/src/main/java/org/apache/impala/authorization/Authorizable.java File fe/src/main/java/org/apache/impala/authorization/Authorizable.java: http://gerrit.cloudera.org:8080/#/c/14106/11/fe/src/main/java/org/apache/impala/authorization/Authorizable.java@65 PS11, Line 65: int ownerHash = getOwnerUser() == null ? 0 : getOwnerUser().hashCode(); : int nameHash = getName().hashCode(); : return nameHash * 31 + ownerHash; nit: You can use Objects.hash(nameHash, ownerHash) http://gerrit.cloudera.org:8080/#/c/14106/11/fe/src/test/java/org/apache/impala/authorization/AuthorizationStmtTest.java File fe/src/test/java/org/apache/impala/authorization/AuthorizationStmtTest.java: http://gerrit.cloudera.org:8080/#/c/14106/11/fe/src/test/java/org/apache/impala/authorization/AuthorizationStmtTest.java@3068 PS11, Line 3068: nit: I think Java 8 can infer this without explicitly specifying the type parameters. http://gerrit.cloudera.org:8080/#/c/14106/11/fe/src/test/java/org/apache/impala/authorization/AuthorizationStmtTest.java@3069 PS11, Line 3069: .put(authorize("select count(*) from functional.alltypes"), : selectError("functional.alltypes")) : .put(authorize("select id from functional.alltypes"), : selectError("functional.alltypes")) : .put(authorize("select id from functional.alltypes_view"), : selectError("functional.alltypes_view")) : .put(authorize("show create table functional.alltypes"), : accessError("functional.alltypes")) : .put(authorize("describe functional.alltypes"), : accessError("functional.alltypes")) : .put(authorize("show create table functional.alltypes_view"), : accessError("functional.alltypes_view")) : .put(authorize("describe functional.alltypes_view"), : accessError("functional.alltypes_view")) : .put(authorize("describe functional.allcomplextypes.int_struct_col"), : accessError("functional.allcomplextypes")) : .put(authorize("refresh functional.alltypes"), : refreshError("functional.alltypes")) : .put(authorize("invalidate metadata functional.alltypes"), : refreshError("functional.alltypes")) : .put(authorize("compute stats functional.alltypes"), : alterError("functional.alltypes")) : .put(authorize("drop stats functional.alltypes"), : alterError("functional.alltypes")) : .put(authorize("create table functional.test_tbl(a int)"), : createError("functional")) : .put(authorize("create table functional.test_tbl like functional.alltypes"), : accessError("functional.alltypes")) : .put(authorize("create table functional.test_tbl as select 1"), : createError("functional")) : .put(authorize("create view functional.test_view as select 1"), : createError("functional")) : .put(authorize("alter table functional.alltypes add column c1 int"), : alterError("functional")) : .put(authorize("drop table functional.alltypes"), : dropError("functional")) : .put(authorize("drop view functional.alltypes_view"), : dropError("functional")) : .put(authorize("alter view functional.alltypes_view as select 1"), : alterError("functional.alltypes_view")) : .put(authorize("alter database functional set owner user foo"), : accessError(true, "functional")) : .build(); nit: should be indented. -- To view, visit http://gerrit.cloudera.org:8080/14106 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I737b7164a3e7afb9996b3402e6872effd663f7b4 Gerrit-Change-Number: 14106 Gerrit-PatchSet: 11 Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fr
[Impala-ASF-CR] IMPALA-8228: Ownership support for Ranger authz
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/14106 ) Change subject: IMPALA-8228: Ownership support for Ranger authz .. Patch Set 6: (9 comments) http://gerrit.cloudera.org:8080/#/c/14106/6/fe/src/main/java/org/apache/impala/authorization/Authorizable.java File fe/src/main/java/org/apache/impala/authorization/Authorizable.java: http://gerrit.cloudera.org:8080/#/c/14106/6/fe/src/main/java/org/apache/impala/authorization/Authorizable.java@62 PS6, Line 62: return getName().hashCode(); If we're comparing owner, we should update the hashCode() as well. http://gerrit.cloudera.org:8080/#/c/14106/6/fe/src/main/java/org/apache/impala/authorization/Authorizable.java@70 PS6, Line 70: temp.getOwnerUser() == this.getOwnerUser() Shouldn't this use equals()? http://gerrit.cloudera.org:8080/#/c/14106/6/fe/src/main/java/org/apache/impala/authorization/AuthorizableColumn.java File fe/src/main/java/org/apache/impala/authorization/AuthorizableColumn.java: http://gerrit.cloudera.org:8080/#/c/14106/6/fe/src/main/java/org/apache/impala/authorization/AuthorizableColumn.java@31 PS6, Line 31: private final String ownerUser_; For consistency, annotate with @Nullable. http://gerrit.cloudera.org:8080/#/c/14106/6/fe/src/main/java/org/apache/impala/authorization/AuthorizableColumn.java@34 PS6, Line 34: String ownerUser I think it's more readable to also annotate the parameter with @Nullable. http://gerrit.cloudera.org:8080/#/c/14106/6/fe/src/main/java/org/apache/impala/authorization/AuthorizableDb.java File fe/src/main/java/org/apache/impala/authorization/AuthorizableDb.java: http://gerrit.cloudera.org:8080/#/c/14106/6/fe/src/main/java/org/apache/impala/authorization/AuthorizableDb.java@33 PS6, Line 33: String ownerUser Same comment about @Nullable parameter. http://gerrit.cloudera.org:8080/#/c/14106/6/fe/src/main/java/org/apache/impala/authorization/AuthorizableFactory.java File fe/src/main/java/org/apache/impala/authorization/AuthorizableFactory.java: http://gerrit.cloudera.org:8080/#/c/14106/6/fe/src/main/java/org/apache/impala/authorization/AuthorizableFactory.java@45 PS6, Line 45: /** :* Creates a new instance of column {@link Authorizable} for a given database name and :* gives access to all tables and columns. :*/ : Authorizable newColumnAllTbls(String dbName, @Nullable String dbOwnerUser); : : /** :* Creates a new instance of column {@link Authorizable} for given database and table :* names and gives access to all columns. :*/ : Authorizable newColumnInTable( : String dbName, String tableName, @Nullable String tblOwnerUser); : : /** :* Creates a new instance of column {@link Authorizable} for given database, table, and :* column names. :*/ : Authorizable newColumnInTable( : String dbName, String tableName, String columnName, @Nullable String tblOwnerUser); nit: update javadoc about the new owner parameter. http://gerrit.cloudera.org:8080/#/c/14106/6/fe/src/main/java/org/apache/impala/authorization/AuthorizableTable.java File fe/src/main/java/org/apache/impala/authorization/AuthorizableTable.java: http://gerrit.cloudera.org:8080/#/c/14106/6/fe/src/main/java/org/apache/impala/authorization/AuthorizableTable.java@34 PS6, Line 34: String ownerUser Same comment about @Nullable parameter. http://gerrit.cloudera.org:8080/#/c/14106/6/fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java File fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java: http://gerrit.cloudera.org:8080/#/c/14106/6/fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java@94 PS6, Line 94: resources.add(new RangerImpalaResourceBuilder() : .database("*").table("*").column("*").build()); : resources.add(new RangerImpalaResourceBuilder() : .database("*").function("*").build()); : resources.add(new RangerImpalaResourceBuilder().uri("*").build()); Should we add .owner() here as well? http://gerrit.cloudera.org:8080/#/c/14106/6/fe/src/main/java/org/apache/impala/catalog/Db.java File fe/src/main/java/org/apache/impala/catalog/Db.java: http://gerrit.cloudera.org:8080/#/c/14106/6/fe/src/main/java/org/apache/impala/catalog/Db.java@544 PS6, Line 544: if (db == null) return null; : return db.getOwnerName(); nit: can be inlined: return db == null ? null : db.getOwnerName() -- To view, visit http://gerrit.cloudera.org:8080/14106 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Messag
[Impala-ASF-CR] IMPALA-8750: Fix profile observability tests
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/13851 ) Change subject: IMPALA-8750: Fix profile observability tests .. Patch Set 5: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13851 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I65a1e81870e808f0e261f8a6097efdcc6903912a Gerrit-Change-Number: 13851 Gerrit-PatchSet: 5 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Comment-Date: Thu, 18 Jul 2019 05:45:50 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8750: Fix profile observability tests
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/13851 ) Change subject: IMPALA-8750: Fix profile observability tests .. Patch Set 1: (4 comments) http://gerrit.cloudera.org:8080/#/c/13851/1/tests/common/impala_service.py File tests/common/impala_service.py: http://gerrit.cloudera.org:8080/#/c/13851/1/tests/common/impala_service.py@90 PS1, Line 90: 'flags' nit: it's better to not mix quotes. For consistency, maybe use "" instead. http://gerrit.cloudera.org:8080/#/c/13851/1/tests/common/impala_service.py@90 PS1, Line 90: for var in varz.get('flags'): : if var["name"] == flag: : return var["current"] what should this method return when varz.get("flags") is empty? http://gerrit.cloudera.org:8080/#/c/13851/1/tests/query_test/test_observability.py File tests/query_test/test_observability.py: http://gerrit.cloudera.org:8080/#/c/13851/1/tests/query_test/test_observability.py@326 PS1, Line 326: 'lineage_event_log_dir' nit: use double quotes for consistency http://gerrit.cloudera.org:8080/#/c/13851/1/tests/query_test/test_observability.py@327 PS1, Line 327: print (lineage_event_log_dir_value) remove debug information -- To view, visit http://gerrit.cloudera.org:8080/13851 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I65a1e81870e808f0e261f8a6097efdcc6903912a Gerrit-Change-Number: 13851 Gerrit-PatchSet: 1 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Comment-Date: Mon, 15 Jul 2019 04:50:12 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8734: Reload table schema on TBLPROPERTIES change
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/13785 ) Change subject: IMPALA-8734: Reload table schema on TBLPROPERTIES change .. Patch Set 4: (3 comments) http://gerrit.cloudera.org:8080/#/c/13785/3/tests/metadata/test_ddl.py File tests/metadata/test_ddl.py: http://gerrit.cloudera.org:8080/#/c/13785/3/tests/metadata/test_ddl.py@697 PS3, Line 697: self.execute_query_expect_success(self.client, "create table {0}.{1} (c1 string)" > Is there a reason for not creating a new test? The new code seems to be com No reason, I thought it would be simpler, but yeah it seems creating a new test is better. Done. http://gerrit.cloudera.org:8080/#/c/13785/3/tests/metadata/test_ddl.py@702 PS3, Line 702: ct_s > Using a less "null like" value could make the test's intention clearer - it I was following the test case, but yeah using a different value is more readable. Done. http://gerrit.cloudera.org:8080/#/c/13785/3/tests/metadata/test_ddl.py@701 PS3, Line 701:file_data="\nfoo\n") : self.execute_query_expect_success(self.client, : "alter table {0}.{1} set tblproperties" : "('serialization.null.format'='foo')" : .format(unique_database, tbl_name)) : result = self.execute_query_expect_success(self.client, :"select * from {0}.{1}" : .format(unique_database, tbl_name)) : assert len(result.data) == 2 : assert result.data[0] == '' > First I thought that it would be nice to create a utility function for this TIL. Done. -- To view, visit http://gerrit.cloudera.org:8080/13785 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2a43a962c2a456f3ddc078b2924f551fccb5c2ad Gerrit-Change-Number: 13785 Gerrit-PatchSet: 4 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 02 Jul 2019 19:52:12 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8734: Reload table schema on TBLPROPERTIES change
Fredy Wijaya has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/13785 ) Change subject: IMPALA-8734: Reload table schema on TBLPROPERTIES change .. IMPALA-8734: Reload table schema on TBLPROPERTIES change Prior to this patch, an INVALIDATE METADATA was required when altering the TBLPROPERTIES for the changes to take effect. With this patch the table schema is automatically reloaded on TBLPROPERTIES change. Testing: - Added a new test in test_ddl.py - Ran test_ddl.py Change-Id: I2a43a962c2a456f3ddc078b2924f551fccb5c2ad --- M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M tests/metadata/test_ddl.py 2 files changed, 20 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/85/13785/4 -- To view, visit http://gerrit.cloudera.org:8080/13785 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I2a43a962c2a456f3ddc078b2924f551fccb5c2ad Gerrit-Change-Number: 13785 Gerrit-PatchSet: 4 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-8734: Reload table schema on TBLPROPERTIES change
Fredy Wijaya has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13785 Change subject: IMPALA-8734: Reload table schema on TBLPROPERTIES change .. IMPALA-8734: Reload table schema on TBLPROPERTIES change Prior to this patch, an INVALIDATE METADATA was required when altering the TBLPROPERTIES for the changes to take effect. With this patch the table schema is automatically reloaded on TBLPROPERTIES change. Testing: - Added a new test in test_ddl.py - Ran test_ddl.py Change-Id: I2a43a962c2a456f3ddc078b2924f551fccb5c2ad --- M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M tests/metadata/test_ddl.py 2 files changed, 25 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/85/13785/3 -- To view, visit http://gerrit.cloudera.org:8080/13785 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I2a43a962c2a456f3ddc078b2924f551fccb5c2ad Gerrit-Change-Number: 13785 Gerrit-PatchSet: 3 Gerrit-Owner: Fredy Wijaya
[Impala-ASF-CR] IMPALA-7734: Catalog and Statestore memz page shows useless memory
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/13670 ) Change subject: IMPALA-7734: Catalog and Statestore memz page shows useless memory .. Patch Set 10: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13670 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1a42434b98aec1d9e0be6ae52325049378fdde23 Gerrit-Change-Number: 13670 Gerrit-PatchSet: 10 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tamas Mate Gerrit-Comment-Date: Tue, 02 Jul 2019 13:22:48 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8716: Log a group of privileges into a single audit event.
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/13744 ) Change subject: IMPALA-8716: Log a group of privileges into a single audit event. .. Patch Set 5: Code-Review+2 Carry Bharath's +2. -- To view, visit http://gerrit.cloudera.org:8080/13744 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib53102bc1ceaf9d62544090dc00f3231fae0efca Gerrit-Change-Number: 13744 Gerrit-PatchSet: 5 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Mon, 01 Jul 2019 21:41:39 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8716: Log a group of privileges into a single audit event.
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/13744 ) Change subject: IMPALA-8716: Log a group of privileges into a single audit event. .. Patch Set 5: (1 comment) http://gerrit.cloudera.org:8080/#/c/13744/2/fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java File fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java: http://gerrit.cloudera.org:8080/#/c/13744/2/fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java@319 PS2, Line 319: // so that we know all the audit events generated by the temporary audit : // handler are contained to the given resource. : RangerBufferAuditHandler tmpAuditHandler = originalAuditHandler == null ? : null : new RangerBufferAuditHandler(originalAuditHandler); : for (Privilege impliedPrivilege: privile > I think you missed the comment. Done -- To view, visit http://gerrit.cloudera.org:8080/13744 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib53102bc1ceaf9d62544090dc00f3231fae0efca Gerrit-Change-Number: 13744 Gerrit-PatchSet: 5 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Mon, 01 Jul 2019 21:41:32 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8716: Log a group of privileges into a single audit event.
Fredy Wijaya has uploaded a new patch set (#5). ( http://gerrit.cloudera.org:8080/13744 ) Change subject: IMPALA-8716: Log a group of privileges into a single audit event. .. IMPALA-8716: Log a group of privileges into a single audit event. This patch updates the audit log handler to group a privilege that consists of multiple privileges into a single audit event. For example if we run "show partitions foo.bar" and we have SELECT privilege on table "foo.bar", before this patch, we would be creating 2 audit events: - Attempt to check if there's INSERT privilege on table "foo.bar" Result: denied, access type: insert, resource: foo.bar - Attempt to check if there's SELECT privilege on table "foo.bar" Result: allowed, access type: select, resource: foo.bar After this patch, we will only create a single audit event, e.g. Result: allowed, access type: view_metadata, resource: foo.bar Testing: - Updated tests in RangerAuditLogTest - Ran FE tests Change-Id: Ib53102bc1ceaf9d62544090dc00f3231fae0efca --- M fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerBufferAuditHandler.java M fe/src/test/java/org/apache/impala/authorization/ranger/RangerAuditLogTest.java 3 files changed, 101 insertions(+), 24 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/44/13744/5 -- To view, visit http://gerrit.cloudera.org:8080/13744 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ib53102bc1ceaf9d62544090dc00f3231fae0efca Gerrit-Change-Number: 13744 Gerrit-PatchSet: 5 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon
[Impala-ASF-CR] IMPALA-8716: Log a group of privileges into a single audit event.
Fredy Wijaya has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/13744 ) Change subject: IMPALA-8716: Log a group of privileges into a single audit event. .. IMPALA-8716: Log a group of privileges into a single audit event. This patch updates the audit log handler to group a privilege that consists of multiple privileges into a single audit event. For example if we run "show partitions foo.bar" and we have SELECT privilege on table "foo.bar", before this patch, we would be creating 2 audit events: - Attempt to check if there's INSERT privilege on table "foo.bar" Result: denied, access type: insert, resource: foo.bar - Attempt to check if there's SELECT privilege on table "foo.bar" Result: allowed, access type: select, resource: foo.bar After this patch, we will only create a single audit event, e.g. Result: allowed, access type: view_metadata, resource: foo.bar Testing: - Updated tests in RangerAuditLogTest - Ran FE tests Change-Id: Ib53102bc1ceaf9d62544090dc00f3231fae0efca --- M fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerBufferAuditHandler.java M fe/src/test/java/org/apache/impala/authorization/ranger/RangerAuditLogTest.java 3 files changed, 95 insertions(+), 24 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/44/13744/3 -- To view, visit http://gerrit.cloudera.org:8080/13744 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ib53102bc1ceaf9d62544090dc00f3231fae0efca Gerrit-Change-Number: 13744 Gerrit-PatchSet: 3 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon
[Impala-ASF-CR] IMPALA-8716: Log a group of privileges into a single audit event.
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/13744 ) Change subject: IMPALA-8716: Log a group of privileges into a single audit event. .. Patch Set 3: (4 comments) http://gerrit.cloudera.org:8080/#/c/13744/2/fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java File fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java: http://gerrit.cloudera.org:8080/#/c/13744/2/fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java@318 PS2, Line 318: fferAu > nit: authorized ? (same below) Done http://gerrit.cloudera.org:8080/#/c/13744/2/fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java@319 PS2, Line 319: null : new RangerBufferAuditHandler(originalAuditHandler); : for (Privilege impliedPrivilege: privilege.getImpliedPrivileges()) { : if (authorizeResource(authzCtx, resource, user, impliedPrivilege, : tmpAuditHandler)) { : authorized = true; > I think this could use a comment. Also, create a copy c'tor? (same below) Good idea about copy constructor. Done. http://gerrit.cloudera.org:8080/#/c/13744/2/fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java@385 PS2, Line 385: RangerBufferAuditHandler auditHandler) throws InternalException { > just checking, is the accessResult automatically inferred from newAuditEven Yeah since we're assigning newAuditEvent to an existing instance and mutating it, so it will have the accessResult. http://gerrit.cloudera.org:8080/#/c/13744/2/fe/src/test/java/org/apache/impala/authorization/ranger/RangerAuditLogTest.java File fe/src/test/java/org/apache/impala/authorization/ranger/RangerAuditLogTest.java: http://gerrit.cloudera.org:8080/#/c/13744/2/fe/src/test/java/org/apache/impala/authorization/ranger/RangerAuditLogTest.java@141 PS2, Line 141: 1 > Mind adding a couple more tests? Those test cases have already been been covered: accessResult = 0, isAny = true in L193-L199 accessResult = 1, isAny = false in L63-L136 --> isAny = false is basically all privileges in https://github.com/apache/impala/blob/c353cf7a648651244ac39677d0cb028e704281d0/fe/src/main/java/org/apache/impala/authorization/Privilege.java#L28-L35 (anyOf_ = false) -- To view, visit http://gerrit.cloudera.org:8080/13744 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib53102bc1ceaf9d62544090dc00f3231fae0efca Gerrit-Change-Number: 13744 Gerrit-PatchSet: 3 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Mon, 01 Jul 2019 19:23:20 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8716: Log a group of privileges into a single audit event.
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/13744 ) Change subject: IMPALA-8716: Log a group of privileges into a single audit event. .. Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/13744/2/fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java File fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java: http://gerrit.cloudera.org:8080/#/c/13744/2/fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java@408 PS2, Line 408: RangerAccessResult result = plugin_.isAccessAllowed(request, auditHandler); > Instead of modifying entries in a tmpAuditHandler, can we make it simple by Tthe plugin._isAccessAllowed() does some logic that passes the correct RangerAccessResult including populating some default values. Calling it manually means we need to always keep the logic similar to the one implemented by isAccessAllowed. -- To view, visit http://gerrit.cloudera.org:8080/13744 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib53102bc1ceaf9d62544090dc00f3231fae0efca Gerrit-Change-Number: 13744 Gerrit-PatchSet: 2 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Mon, 01 Jul 2019 14:16:22 + Gerrit-HasComments: Yes
[Impala-ASF-CR] [IMPALA-8587] Show inherited privileges with Ranger show grant
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/13673 ) Change subject: [IMPALA-8587] Show inherited privileges with Ranger show grant .. Patch Set 3: (3 comments) http://gerrit.cloudera.org:8080/#/c/13673/3/fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpaladAuthorizationManager.java File fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpaladAuthorizationManager.java: http://gerrit.cloudera.org:8080/#/c/13673/3/fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpaladAuthorizationManager.java@367 PS3, Line 367: private static class RangerResourceResult { : List server = new ArrayList<>(); : List uri = new ArrayList<>(); : List database = new ArrayList<>(); : List udf = new ArrayList<>(); : List table = new ArrayList<>(); : List column = new ArrayList<>(); make these private http://gerrit.cloudera.org:8080/#/c/13673/3/fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpaladAuthorizationManager.java@392 PS3, Line 392: public RangerResourceResult addUdfResult(RangerResultRow result) { : udf.add(result); : return this; : } : : public RangerResourceResult addUriResult(RangerResultRow result) { : uri.add(result); : return this; : } These two methods are unused, that means uri and udf will always be empty. http://gerrit.cloudera.org:8080/#/c/13673/3/fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpaladAuthorizationManager.java@407 PS3, Line 407: public List getResultRows() { : List results = new ArrayList<>(); : : results.addAll(filterIfAll(server)); : results.addAll(filterIfAll(database)); : results.addAll(filterIfAll(table)); : results.addAll(filterIfAll(column)); : results.addAll(filterIfAll(udf)); : results.addAll(filterIfAll(uri)); : : return results; : } I don't quite follow the logic why we have to filter ALL. A comment will be helpful. -- To view, visit http://gerrit.cloudera.org:8080/13673 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5c4c9327acb12abc12d130ef5c1ace6a08ed193c Gerrit-Change-Number: 13673 Gerrit-PatchSet: 3 Gerrit-Owner: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 28 Jun 2019 20:38:14 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8612: Fix sporadic NPE when dropping an authorized table
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/13508 ) Change subject: IMPALA-8612: Fix sporadic NPE when dropping an authorized table .. Patch Set 5: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13508 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I70bd7ca4796b24920ee156436bf8bbc682e7d952 Gerrit-Change-Number: 13508 Gerrit-PatchSet: 5 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Thu, 27 Jun 2019 16:40:52 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8612: Fix sporadic NPE when dropping an authorized table
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/13508 ) Change subject: IMPALA-8612: Fix sporadic NPE when dropping an authorized table .. Patch Set 4: Code-Review+2 (1 comment) http://gerrit.cloudera.org:8080/#/c/13508/4/fe/src/main/java/org/apache/impala/analysis/DropTableOrViewStmt.java File fe/src/main/java/org/apache/impala/analysis/DropTableOrViewStmt.java: http://gerrit.cloudera.org:8080/#/c/13508/4/fe/src/main/java/org/apache/impala/analysis/DropTableOrViewStmt.java@131 PS4, Line 131: tableName_.toString() Just an FYI usually it's better to not call toString() explicitly in the logger since toString() will be lazily called by the logger. For example if we had a very expensive toString() call, calling LOG.info() on a different log level such as WARN or ERROR would not incur an expensive toString() call. I don't think it matters for this, though. -- To view, visit http://gerrit.cloudera.org:8080/13508 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I70bd7ca4796b24920ee156436bf8bbc682e7d952 Gerrit-Change-Number: 13508 Gerrit-PatchSet: 4 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Thu, 27 Jun 2019 15:18:36 + Gerrit-HasComments: Yes
[Impala-ASF-CR] Add ext-py/bitarray-0.9.0 in .gitignore
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/13745 ) Change subject: Add ext-py/bitarray-0.9.0 in .gitignore .. Patch Set 2: > Patch Set 2: Verified-1 > > Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/4559/ Looks like flaky Maven issue. Retrying the merge. 02:24:28 [ERROR] Failed to execute goal on project impala-minimal-hive-exec: Could not resolve dependencies for project org.apache.impala:impala-minimal-hive-exec:jar:0.1-SNAPSHOT: Could not transfer artifact io.dropwizard.metrics:metrics-core:jar:3.1.0 from/to impala.cdp.repo (https://native-toolchain.s3.amazonaws.com/build/cdp_components/1153860/maven): Access denied to: https://native-toolchain.s3.amazonaws.com/build/cdp_components/1153860/maven/io/dropwizard/metrics/metrics-core/3.1.0/metrics-core-3.1.0.jar , ReasonPhrase:Forbidden. -> [Help 1] 02:24:28 [ERROR] 02:24:28 [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. 02:24:28 [ERROR] Re-run Maven using the -X switch to enable full debug logging. 02:24:28 [ERROR] 02:24:28 [ERROR] For more information about the errors and possible solutions, please read the following articles: 02:24:28 [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException 02:24:28 mvn -U -B install -DskipTests exited with code 0 02:24:28 shaded-deps/CMakeFiles/shaded-deps.dir/build.make:57: recipe for target 'shaded-deps/CMakeFiles/shaded-deps' failed 02:24:28 make[2]: *** [shaded-deps/CMakeFiles/shaded-deps] Error 1 02:24:28 CMakeFiles/Makefile2:14411: recipe for target 'shaded-deps/CMakeFiles/shaded-deps.dir/all' failed 02:24:28 make[1]: *** [shaded-deps/CMakeFiles/shaded-deps.dir/all] Error 2 02:24:28 Makefile:94: recipe for target 'all' failed 02:24:28 make: *** [all] Error -- To view, visit http://gerrit.cloudera.org:8080/13745 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I22abd82eb2c5a4a52fd56fd2266b636a0dbde071 Gerrit-Change-Number: 13745 Gerrit-PatchSet: 2 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 27 Jun 2019 15:14:04 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8716: Log a group of privileges into a single audit event.
Fredy Wijaya has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/13744 ) Change subject: IMPALA-8716: Log a group of privileges into a single audit event. .. IMPALA-8716: Log a group of privileges into a single audit event. This patch updates the audit log handler to group a privilege that consists of multiple privileges into a single audit event. For example if we run "show partitions foo.bar" and we have SELECT privilege on table "foo.bar", before this patch, we would be creating 2 audit events: - Attempt to check if there's INSERT privilege on table "foo.bar" Result: denied, access type: insert, resource: foo.bar - Attempt to check if there's SELECT privilege on table "foo.bar" Result: allowed, access type: select, resource: foo.bar After this patch, we will only create a single audit event, e.g. Result: allowed, access type: view_metadata, resource: foo.bar Testing: - Updated tests in RangerAuditLogTest - Ran FE tests Change-Id: Ib53102bc1ceaf9d62544090dc00f3231fae0efca --- M fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java M fe/src/test/java/org/apache/impala/authorization/ranger/RangerAuditLogTest.java 2 files changed, 87 insertions(+), 20 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/44/13744/2 -- To view, visit http://gerrit.cloudera.org:8080/13744 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ib53102bc1ceaf9d62544090dc00f3231fae0efca Gerrit-Change-Number: 13744 Gerrit-PatchSet: 2 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon
[Impala-ASF-CR] Add ext-py/bitarray-0.9.0 in .gitignore
Fredy Wijaya has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13745 Change subject: Add ext-py/bitarray-0.9.0 in .gitignore .. Add ext-py/bitarray-0.9.0 in .gitignore Change-Id: I22abd82eb2c5a4a52fd56fd2266b636a0dbde071 --- M shell/.gitignore 1 file changed, 2 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/45/13745/1 -- To view, visit http://gerrit.cloudera.org:8080/13745 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I22abd82eb2c5a4a52fd56fd2266b636a0dbde071 Gerrit-Change-Number: 13745 Gerrit-PatchSet: 1 Gerrit-Owner: Fredy Wijaya
[Impala-ASF-CR] IMPALA-8716: Log a group of privileges into a single audit event.
Fredy Wijaya has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13744 Change subject: IMPALA-8716: Log a group of privileges into a single audit event. .. IMPALA-8716: Log a group of privileges into a single audit event. This patch updates the audit log handler to group a privilege that consists of multiple privileges into a single audit event. For example if we run "show partitions foo.bar" and we have SELECT privilege on table "foo.bar", before this patch, we would be creating 2 audit events: - Attempt to check if there's INSERT privilege on table "foo.bar" Result: denied, access type: insert, resource: foo.bar - Attempt to check if there's SELECT privilege on table "foo.bar" Result: allowed, access type: select, resource: foo.bar After this patch, we will only create a single audit event, e.g. Result: allowed, access type: view_metadata, resource: foo.bar Testing: - Updated tests in RangerAuditLogTest - Ran FE tests Change-Id: Ib53102bc1ceaf9d62544090dc00f3231fae0efca --- M fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java M fe/src/test/java/org/apache/impala/authorization/ranger/RangerAuditLogTest.java 2 files changed, 81 insertions(+), 20 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/44/13744/1 -- To view, visit http://gerrit.cloudera.org:8080/13744 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ib53102bc1ceaf9d62544090dc00f3231fae0efca Gerrit-Change-Number: 13744 Gerrit-PatchSet: 1 Gerrit-Owner: Fredy Wijaya
[Impala-ASF-CR] IMPALA-8476: Replace Sentry admin check workaround with proper Sentry API
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/13346 ) Change subject: IMPALA-8476: Replace Sentry admin check workaround with proper Sentry API .. Patch Set 15: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13346 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5a27140d401494bc372ad0da96ada57bda94cd82 Gerrit-Change-Number: 13346 Gerrit-PatchSet: 15 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 26 Jun 2019 18:41:37 + Gerrit-HasComments: No
[Impala-ASF-CR] Fix incorrect timestamp units in the logs
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/13731 ) Change subject: Fix incorrect timestamp units in the logs .. Patch Set 2: Code-Review+2 > Patch Set 2: Verified-1 > > Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/4547/ Unrelated flaky tests. Re-running the merge. -- To view, visit http://gerrit.cloudera.org:8080/13731 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I132a007404dcf5291e70f2642ed3e9f995d694b1 Gerrit-Change-Number: 13731 Gerrit-PatchSet: 2 Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 26 Jun 2019 17:25:59 + Gerrit-HasComments: No
[Impala-ASF-CR] Fix incorrect timestamp units in the logs
Fredy Wijaya has removed a vote on this change. Change subject: Fix incorrect timestamp units in the logs .. Removed Verified-1 by Impala Public Jenkins -- To view, visit http://gerrit.cloudera.org:8080/13731 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: deleteVote Gerrit-Change-Id: I132a007404dcf5291e70f2642ed3e9f995d694b1 Gerrit-Change-Number: 13731 Gerrit-PatchSet: 2 Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-8612: Fix sporadic NPE when dropping an authorized table
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/13508 ) Change subject: IMPALA-8612: Fix sporadic NPE when dropping an authorized table .. Patch Set 3: (3 comments) http://gerrit.cloudera.org:8080/#/c/13508/3/fe/src/main/java/org/apache/impala/analysis/DropTableOrViewStmt.java File fe/src/main/java/org/apache/impala/analysis/DropTableOrViewStmt.java: http://gerrit.cloudera.org:8080/#/c/13508/3/fe/src/main/java/org/apache/impala/analysis/DropTableOrViewStmt.java@40 PS3, Line 40: Expr Shouldn't this be DropTableOrViewStmt? http://gerrit.cloudera.org:8080/#/c/13508/3/fe/src/main/java/org/apache/impala/analysis/DropTableOrViewStmt.java@131 PS3, Line 131: "Ignoring TableLoadingException for " + dbName_ + "." + tableName_ nit: it's usually preferable to use the SLF4J log format, e.g. LOG.info("Ignoring TableLoadingException for {}.{}", dbName_, getTbl()); http://gerrit.cloudera.org:8080/#/c/13508/3/fe/src/main/java/org/apache/impala/analysis/DropTableOrViewStmt.java@134 PS3, Line 134: tableName_ tableName_.toString() may print "db.tbl": https://github.com/apache/impala/blob/572c1278566ca329f3e47e5c2d7e13dd0c7967f7/fe/src/main/java/org/apache/impala/analysis/TableName.java#L85-L92, should we use getTbl() instead similar to L115 and L120? -- To view, visit http://gerrit.cloudera.org:8080/13508 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I70bd7ca4796b24920ee156436bf8bbc682e7d952 Gerrit-Change-Number: 13508 Gerrit-PatchSet: 3 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 26 Jun 2019 17:24:30 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8476: Replace Sentry admin check workaround with proper Sentry API
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/13346 ) Change subject: IMPALA-8476: Replace Sentry admin check workaround with proper Sentry API .. Patch Set 11: (5 comments) Thanks for the fixes! This is getting close. I'm ready to give a +2 once all the comments are resolved. http://gerrit.cloudera.org:8080/#/c/13346/11/fe/src/main/java/org/apache/impala/authorization/sentry/SentryPolicyService.java File fe/src/main/java/org/apache/impala/authorization/sentry/SentryPolicyService.java: http://gerrit.cloudera.org:8080/#/c/13346/11/fe/src/main/java/org/apache/impala/authorization/sentry/SentryPolicyService.java@535 PS11, Line 535: public boolean isSentryAdmin(User requestingUser) throws InternalException, can we add javadoc? http://gerrit.cloudera.org:8080/#/c/13346/11/fe/src/main/java/org/apache/impala/authorization/sentry/SentryPolicyService.java@535 PS11, Line 535: throws InternalException, nit: I think it's more readable to put throws InternalException, SentryUserException in the next line. http://gerrit.cloudera.org:8080/#/c/13346/11/fe/src/main/java/org/apache/impala/authorization/sentry/SentryPolicyService.java@537 PS11, Line 537: SentryServiceClient client = new SentryServiceClient(); : try { : return client.get().isAdmin(requestingUser.getName()); : } finally { : client.close(); : } nit: using try-resource is shorter, i.e. try (SentryServiceClient client = new SentryServiceClient()) { return client.get().isAdmin(requestingUser.getName()); } http://gerrit.cloudera.org:8080/#/c/13346/11/fe/src/main/java/org/apache/impala/service/JniCatalog.java File fe/src/main/java/org/apache/impala/service/JniCatalog.java: http://gerrit.cloudera.org:8080/#/c/13346/11/fe/src/main/java/org/apache/impala/service/JniCatalog.java@304 PS11, Line 304: try { : boolean isSentryAdmin = ((SentryCatalogdAuthorizationManager) : catalogOpExecutor_.getAuthzManager()).isSentryAdmin(user); : response.setIs_admin(isSentryAdmin); : } catch (SentryUserException e) { : // When a user is not defined in Sentry, isAdmin() will throw : // SentryUserException, we will consider this requesting user : // as a non-Sentry administrator. : response.setIs_admin(false); : } Sorry for the back and forth, but putting this logic in JniCatalog doesn't feel quite right since JniCatalog is supposed to be a thin JNI wrapper. It makes more sense to move the try/catch logic in the SentryProxy instead. http://gerrit.cloudera.org:8080/#/c/13346/11/tests/authorization/test_sentry.py File tests/authorization/test_sentry.py: http://gerrit.cloudera.org:8080/#/c/13346/11/tests/authorization/test_sentry.py@51 PS11, Line 51: root nit: Calling it non_admin makes it easier to understand. -- To view, visit http://gerrit.cloudera.org:8080/13346 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5a27140d401494bc372ad0da96ada57bda94cd82 Gerrit-Change-Number: 13346 Gerrit-PatchSet: 11 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 26 Jun 2019 03:57:18 + Gerrit-HasComments: Yes
[Impala-ASF-CR] Fix incorrect timestamp units in the logs
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/13731 ) Change subject: Fix incorrect timestamp units in the logs .. Patch Set 1: Code-Review+2 Thanks for fixing this! -- To view, visit http://gerrit.cloudera.org:8080/13731 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I132a007404dcf5291e70f2642ed3e9f995d694b1 Gerrit-Change-Number: 13731 Gerrit-PatchSet: 1 Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 26 Jun 2019 03:40:32 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8589: Re-enable flaky test query event hooks.py
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/13713 ) Change subject: IMPALA-8589: Re-enable flaky test_query_event_hooks.py .. Patch Set 8: Code-Review+2 (2 comments) Carry Bharath's +2. http://gerrit.cloudera.org:8080/#/c/13713/7/fe/src/test/java/org/apache/impala/testutil/DummyQueryEventHook.java File fe/src/test/java/org/apache/impala/testutil/DummyQueryEventHook.java: http://gerrit.cloudera.org:8080/#/c/13713/7/fe/src/test/java/org/apache/impala/testutil/DummyQueryEventHook.java@24 PS7, Line 24: /** > Add a class comment on how this is supposed to work? Done http://gerrit.cloudera.org:8080/#/c/13713/7/tests/custom_cluster/test_query_event_hooks.py File tests/custom_cluster/test_query_event_hooks.py: http://gerrit.cloudera.org:8080/#/c/13713/7/tests/custom_cluster/test_query_event_hooks.py@112 PS7, Line 112: > remove? Done -- To view, visit http://gerrit.cloudera.org:8080/13713 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia64550e986b5eba59a1d77657943932bb977d470 Gerrit-Change-Number: 13713 Gerrit-PatchSet: 8 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: radford nguyen Gerrit-Comment-Date: Tue, 25 Jun 2019 18:24:21 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8589: Re-enable flaky test query event hooks.py
Fredy Wijaya has uploaded a new patch set (#8). ( http://gerrit.cloudera.org:8080/13713 ) Change subject: IMPALA-8589: Re-enable flaky test_query_event_hooks.py .. IMPALA-8589: Re-enable flaky test_query_event_hooks.py This patch fixes the flaky test_query_event_hooks.py. The patch also cuts down the waiting time for impalad timeout to 5 seconds from the default 60 seconds especially for those tests that will fail Impala startup. Testing: - Ran test_query_event_hooks.py in a loop. Change-Id: Ia64550e986b5eba59a1d77657943932bb977d470 --- M fe/src/main/java/org/apache/impala/hooks/QueryEventHookManager.java M fe/src/test/java/org/apache/impala/testutil/AlwaysErrorQueryEventHook.java M fe/src/test/java/org/apache/impala/testutil/DummyQueryEventHook.java M fe/src/test/java/org/apache/impala/testutil/PostQueryErrorEventHook.java M tests/common/custom_cluster_test_suite.py M tests/custom_cluster/test_query_event_hooks.py 6 files changed, 126 insertions(+), 196 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/13/13713/8 -- To view, visit http://gerrit.cloudera.org:8080/13713 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ia64550e986b5eba59a1d77657943932bb977d470 Gerrit-Change-Number: 13713 Gerrit-PatchSet: 8 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: radford nguyen
[Impala-ASF-CR] IMPALA-8589: Re-enable flaky test query event hooks.py
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/13713 ) Change subject: IMPALA-8589: Re-enable flaky test_query_event_hooks.py .. Patch Set 7: Rebased and re-running the dry-run. -- To view, visit http://gerrit.cloudera.org:8080/13713 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia64550e986b5eba59a1d77657943932bb977d470 Gerrit-Change-Number: 13713 Gerrit-PatchSet: 7 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: radford nguyen Gerrit-Comment-Date: Tue, 25 Jun 2019 15:51:09 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8589: Re-enable flaky test query event hooks.py
Fredy Wijaya has uploaded a new patch set (#7). ( http://gerrit.cloudera.org:8080/13713 ) Change subject: IMPALA-8589: Re-enable flaky test_query_event_hooks.py .. IMPALA-8589: Re-enable flaky test_query_event_hooks.py This patch fixes the flaky test_query_event_hooks.py. The patch also cuts down the waiting time for impalad timeout to 5 seconds from the default 60 seconds especially for those tests that will fail Impala startup. Testing: - Ran test_query_event_hooks.py in a loop. Change-Id: Ia64550e986b5eba59a1d77657943932bb977d470 --- M fe/src/test/java/org/apache/impala/testutil/DummyQueryEventHook.java M tests/common/custom_cluster_test_suite.py M tests/custom_cluster/test_query_event_hooks.py 3 files changed, 67 insertions(+), 142 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/13/13713/7 -- To view, visit http://gerrit.cloudera.org:8080/13713 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ia64550e986b5eba59a1d77657943932bb977d470 Gerrit-Change-Number: 13713 Gerrit-PatchSet: 7 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: radford nguyen
[Impala-ASF-CR] IMPALA-8589: Re-enable flaky test query event hooks.py
Fredy Wijaya has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13713 Change subject: IMPALA-8589: Re-enable flaky test_query_event_hooks.py .. IMPALA-8589: Re-enable flaky test_query_event_hooks.py This patch fixes the flaky test_query_event_hooks.py. The patch also cuts down the waiting time for impalad timeout to 5 seconds from the default 60 seconds especially for those tests that will fail Impala startup. Testing: - Ran test_query_event_hooks.py in a loop. Change-Id: Ia64550e986b5eba59a1d77657943932bb977d470 --- M fe/src/test/java/org/apache/impala/testutil/DummyQueryEventHook.java M tests/common/custom_cluster_test_suite.py M tests/custom_cluster/test_query_event_hooks.py 3 files changed, 67 insertions(+), 142 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/13/13713/6 -- To view, visit http://gerrit.cloudera.org:8080/13713 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ia64550e986b5eba59a1d77657943932bb977d470 Gerrit-Change-Number: 13713 Gerrit-PatchSet: 6 Gerrit-Owner: Fredy Wijaya
[Impala-ASF-CR] IMPALA-8652 Illegal delimiter error in shell has unknown error
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/13690 ) Change subject: IMPALA-8652 Illegal delimiter error in shell has unknown error .. Patch Set 2: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/13690 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7ee2fccd305b104b3aff44c57659b6f14f2f4a05 Gerrit-Change-Number: 13690 Gerrit-PatchSet: 2 Gerrit-Owner: Jiawei Wang Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jiawei Wang Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 20 Jun 2019 22:00:46 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8652 Illegal delimiter error in shell has unknown error
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/13690 ) Change subject: IMPALA-8652 Illegal delimiter error in shell has unknown error .. Patch Set 1: (2 comments) http://gerrit.cloudera.org:8080/#/c/13690/1/shell/impala_shell.py File shell/impala_shell.py: http://gerrit.cloudera.org:8080/#/c/13690/1/shell/impala_shell.py@1733 PS1, Line 1733: if options.output_delimiter != "\\t": Can we not check if the delimiter is not empty then perform the validation? http://gerrit.cloudera.org:8080/#/c/13690/1/shell/impala_shell.py@1738 PS1, Line 1738: sys.exit(1) Raise FatalShellException to be consistent with the rest of the code here. Can we also add/update the test to catch this issue? -- To view, visit http://gerrit.cloudera.org:8080/13690 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7ee2fccd305b104b3aff44c57659b6f14f2f4a05 Gerrit-Change-Number: 13690 Gerrit-PatchSet: 1 Gerrit-Owner: Jiawei Wang Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 20 Jun 2019 20:24:48 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8682: Add authorized proxy user/group test coverage with Ranger
Fredy Wijaya has uploaded a new patch set (#5). ( http://gerrit.cloudera.org:8080/13679 ) Change subject: IMPALA-8682: Add authorized proxy user/group test coverage with Ranger .. IMPALA-8682: Add authorized proxy user/group test coverage with Ranger This patch adds a test coverage for authorized proxy user/group with Ranger. This patch also moves the authorized proxy tests into a separate file, test_authorized_proxy and refactors the tests to be more readable and reusable. Testing: - Added a new test_authorized_proxy.py - Ran all E2E authorization tests Change-Id: If6f797600720e6432b85cac8f13afe8fa5624596 --- M tests/authorization/test_authorization.py A tests/authorization/test_authorized_proxy.py 2 files changed, 275 insertions(+), 145 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/79/13679/5 -- To view, visit http://gerrit.cloudera.org:8080/13679 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: If6f797600720e6432b85cac8f13afe8fa5624596 Gerrit-Change-Number: 13679 Gerrit-PatchSet: 5 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon
[Impala-ASF-CR] IMPALA-8682: Add authorized proxy user/group test coverage with Ranger
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/13679 ) Change subject: IMPALA-8682: Add authorized proxy user/group test coverage with Ranger .. Patch Set 5: (2 comments) http://gerrit.cloudera.org:8080/#/c/13679/4/tests/authorization/test_authorized_proxy.py File tests/authorization/test_authorized_proxy.py: http://gerrit.cloudera.org:8080/#/c/13679/4/tests/authorization/test_authorized_proxy.py@195 PS4, Line 195: s > what are these commented-out lines? I meant to delete that. Done. http://gerrit.cloudera.org:8080/#/c/13679/4/tests/authorization/test_authorized_proxy.py@231 PS4, Line 231: > typo? Done -- To view, visit http://gerrit.cloudera.org:8080/13679 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If6f797600720e6432b85cac8f13afe8fa5624596 Gerrit-Change-Number: 13679 Gerrit-PatchSet: 5 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Thu, 20 Jun 2019 17:14:38 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8443: Record time spent in authorization in the runtime profile
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/13353 ) Change subject: IMPALA-8443: Record time spent in authorization in the runtime profile .. Patch Set 10: > Patch Set 10: Verified-1 > > Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/4500/ This looks like some flaky Maven issue. I'm re-running the merge. 23:59:12 ] [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.3:compile (default-compile) on project query-event-hook-api: Execution default-compile of goal org.apache.maven.plugins:maven-compiler-plugin:3.3:compile failed: Plugin org.apache.maven.plugins:maven-compiler-plugin:3.3 or one of its dependencies could not be resolved: Failed to collect dependencies at org.apache.maven.plugins:maven-compiler-plugin:jar:3.3 -> org.codehaus.plexus:plexus-container-default:jar:1.5.5 -> org.apache.xbean:xbean-reflect:jar:3.4: Failed to read artifact descriptor for org.apache.xbean:xbean-reflect:jar:3.4: Could not transfer artifact org.apache.xbean:xbean-reflect:pom:3.4 from/to central (https://repo.maven.apache.org/maven2): /home/ubuntu/.m2/repository/org/apache/xbean/xbean-reflect/3.4/xbean-reflect-3.4.pom.part (No such file or directory) -> [Help 1] 23:59:12 ] [ERROR] 23:59:12 ] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. 23:59:12 ] [ERROR] Re-run Maven using the -X switch to enable full debug logging. 23:59:12 ] [ERROR] 23:59:12 ] [ERROR] For more information about the errors and possible solutions, please read the following articles: 23:59:12 ] [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginResolutionException 23:59:12 ] mvn -B install -DskipTests exited with code 0 23:59:12 ] query-event-hook-api/CMakeFiles/query-event-hook-api.dir/build.make:57: recipe for target 'query-event-hook-api/CMakeFiles/query-event-hook-api' failed 23:59:12 ] make[3]: *** [query-event-hook-api/CMakeFiles/query-event-hook-api] Error 1 23:59:12 ] CMakeFiles/Makefile2:14640: recipe for target 'query-event-hook-api/CMakeFiles/query-event-hook-api.dir/all' failed 23:59:12 ] make[2]: *** [query-event-hook-api/CMakeFiles/query-event-hook-api.dir/all] Error 2 23:59:12 ] make[2]: *** Waiting for unfinished jobs -- To view, visit http://gerrit.cloudera.org:8080/13353 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5bb85e57fcc75d41f3eb2911e6d375e0da6f82ae Gerrit-Change-Number: 13353 Gerrit-PatchSet: 10 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Comment-Date: Thu, 20 Jun 2019 17:13:41 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8682: Add authorized proxy user/group test coverage with Ranger
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/13679 ) Change subject: IMPALA-8682: Add authorized proxy user/group test coverage with Ranger .. Patch Set 4: (3 comments) http://gerrit.cloudera.org:8080/#/c/13679/3/tests/authorization/test_authorized_proxy.py File tests/authorization/test_authorized_proxy.py: http://gerrit.cloudera.org:8080/#/c/13679/3/tests/authorization/test_authorized_proxy.py@26 PS3, Line 26: from getpass import getuser > flake8: F811 redefinition of unused 'time' from line 21 Done http://gerrit.cloudera.org:8080/#/c/13679/3/tests/authorization/test_authorized_proxy.py@120 PS3, Line 120: t > flake8: E131 continuation line unaligned for hanging indent Done http://gerrit.cloudera.org:8080/#/c/13679/3/tests/authorization/test_authorized_proxy.py@140 PS3, Line 140: t > flake8: E131 continuation line unaligned for hanging indent Done -- To view, visit http://gerrit.cloudera.org:8080/13679 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If6f797600720e6432b85cac8f13afe8fa5624596 Gerrit-Change-Number: 13679 Gerrit-PatchSet: 4 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Wed, 19 Jun 2019 18:51:46 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8682: Add authorized proxy user/group test coverage with Ranger
Fredy Wijaya has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/13679 ) Change subject: IMPALA-8682: Add authorized proxy user/group test coverage with Ranger .. IMPALA-8682: Add authorized proxy user/group test coverage with Ranger This patch adds a test coverage for authorized proxy user/group with Ranger. This patch also moves the authorized proxy tests into a separate file, test_authorized_proxy and refactors the tests to be more readable and reusable. Testing: - Added a new test_authorized_proxy.py - Ran all E2E authorization tests Change-Id: If6f797600720e6432b85cac8f13afe8fa5624596 --- M tests/authorization/test_authorization.py A tests/authorization/test_authorized_proxy.py 2 files changed, 278 insertions(+), 145 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/79/13679/4 -- To view, visit http://gerrit.cloudera.org:8080/13679 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: If6f797600720e6432b85cac8f13afe8fa5624596 Gerrit-Change-Number: 13679 Gerrit-PatchSet: 4 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon
[Impala-ASF-CR] IMPALA-8682: Add authorized proxy user/group test coverage with Ranger
Fredy Wijaya has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13679 Change subject: IMPALA-8682: Add authorized proxy user/group test coverage with Ranger .. IMPALA-8682: Add authorized proxy user/group test coverage with Ranger This patch adds a test coverage for authorized proxy user/group with Ranger. This patch also moves the authorized proxy tests into a separate file, test_authorized_proxy and refactors the tests to be more readable and reusable. Testing: - Added a new test_authorized_proxy.py - Ran all E2E authorization tests Change-Id: If6f797600720e6432b85cac8f13afe8fa5624596 --- M tests/authorization/test_authorization.py A tests/authorization/test_authorized_proxy.py 2 files changed, 279 insertions(+), 144 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/79/13679/3 -- To view, visit http://gerrit.cloudera.org:8080/13679 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: If6f797600720e6432b85cac8f13afe8fa5624596 Gerrit-Change-Number: 13679 Gerrit-PatchSet: 3 Gerrit-Owner: Fredy Wijaya
[Impala-ASF-CR] IMPALA-8443: Record time spent in authorization in the runtime profile
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/13353 ) Change subject: IMPALA-8443: Record time spent in authorization in the runtime profile .. Patch Set 10: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13353 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5bb85e57fcc75d41f3eb2911e6d375e0da6f82ae Gerrit-Change-Number: 13353 Gerrit-PatchSet: 10 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Comment-Date: Wed, 19 Jun 2019 18:28:23 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8658: Populate missing Ranger audit fields
Fredy Wijaya has uploaded a new patch set (#11). ( http://gerrit.cloudera.org:8080/13601 ) Change subject: IMPALA-8658: Populate missing Ranger audit fields .. IMPALA-8658: Populate missing Ranger audit fields This patch adds missing Ranger audit fields, such as: - Client IP - Cluster name This patch also updates the access type to be in upper case to be consistent with Hive Ranger audit. The SQL statement logged will now return a redacted statement when the redacted flag is set. Testing: - Updated the tests in RangerAuditLogTest - Ran all FE tests - Tested the changes against Solr cluster Change-Id: I167bc35411ad9b4164c292077ff082671967cff8 --- M be/src/service/client-request-state.cc M common/thrift/CatalogService.thrift M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java M fe/src/main/java/org/apache/impala/analysis/GrantRevokePrivStmt.java M fe/src/main/java/org/apache/impala/authorization/AuthorizationChecker.java M fe/src/main/java/org/apache/impala/authorization/AuthorizationManager.java M fe/src/main/java/org/apache/impala/authorization/BaseAuthorizationChecker.java M fe/src/main/java/org/apache/impala/authorization/NoopAuthorizationFactory.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationContext.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerBufferAuditHandler.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerCatalogdAuthorizationManager.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpaladAuthorizationManager.java M fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizationChecker.java M fe/src/main/java/org/apache/impala/authorization/sentry/SentryCatalogdAuthorizationManager.java M fe/src/main/java/org/apache/impala/authorization/sentry/SentryImpaladAuthorizationManager.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/test/java/org/apache/impala/authorization/AuthorizationTestBase.java M fe/src/test/java/org/apache/impala/authorization/ranger/RangerAuditLogTest.java M fe/src/test/java/org/apache/impala/common/FrontendTestBase.java M fe/src/test/resources/ranger-hive-audit.xml 22 files changed, 223 insertions(+), 144 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/01/13601/11 -- To view, visit http://gerrit.cloudera.org:8080/13601 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I167bc35411ad9b4164c292077ff082671967cff8 Gerrit-Change-Number: 13601 Gerrit-PatchSet: 11 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon
[Impala-ASF-CR] [IMPALA-8587] Show inherited privileges with Ranger show grant
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/13673 ) Change subject: [IMPALA-8587] Show inherited privileges with Ranger show grant .. Patch Set 3: (5 comments) http://gerrit.cloudera.org:8080/#/c/13673/3//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/13673/3//COMMIT_MSG@7 PS3, Line 7: [IMPALA-8587] nit: use IMPALA-8587 format http://gerrit.cloudera.org:8080/#/c/13673/3//COMMIT_MSG@20 PS3, Line 20: They would see no results. After this change, the user will see database : level privileges when executing the previous statement. If a user has : SELECT privilege on DATABASE and on TABLE and issues a show grant on : TABLE, they will only see the SELECT privilege for TABLE. Users will not : see multiple instances of SELECT or any other privilege type in a SHOW : GRANT statemenet. Show what the new output looks like. It'll be much easier to understand. We also need more explanation about this especially since this is relates to how the Ranger policy engine works. http://gerrit.cloudera.org:8080/#/c/13673/3/fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpaladAuthorizationManager.java File fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpaladAuthorizationManager.java: http://gerrit.cloudera.org:8080/#/c/13673/3/fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpaladAuthorizationManager.java@330 PS3, Line 330: != Use equals(). != or == is for reference equality. Sometimes you get lucky because string intern, but we shouldn't rely on that. http://gerrit.cloudera.org:8080/#/c/13673/3/fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpaladAuthorizationManager.java@329 PS3, Line 329: for (RangerResultRow row : resultRows) { : if (row.column_ != "*" && !row.column_.isEmpty()) { : resourceResult.addColumnResult(row); : } else if (row.table_ != "*" && !row.table_.isEmpty()) { : resourceResult.addTableResult(row); : } else if (row.database_ != "*" && !row.database_.isEmpty()) { : resourceResult.addDatabaseResult(row); : } else { : resourceResult.addServerResult(row); : } : } Can you add a comment for this logic? It's not quite clear to me what it's trying to do. http://gerrit.cloudera.org:8080/#/c/13673/3/fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpaladAuthorizationManager.java@422 PS3, Line 422: nit: overly indented, use 4 spaces -- To view, visit http://gerrit.cloudera.org:8080/13673 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5c4c9327acb12abc12d130ef5c1ace6a08ed193c Gerrit-Change-Number: 13673 Gerrit-PatchSet: 3 Gerrit-Owner: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 18 Jun 2019 21:08:45 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7734: Catalog and Statestore memz page shows useless memory...
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/13670 ) Change subject: IMPALA-7734: Catalog and Statestore memz page shows useless memory... .. Patch Set 3: (2 comments) http://gerrit.cloudera.org:8080/#/c/13670/3//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/13670/3//COMMIT_MSG@7 PS3, Line 7: ... remove ... http://gerrit.cloudera.org:8080/#/c/13670/3/be/src/util/default-path-handlers.cc File be/src/util/default-path-handlers.cc: http://gerrit.cloudera.org:8080/#/c/13670/3/be/src/util/default-path-handlers.cc@135 PS3, Line 135: if (mem_tracker != NULL) { It's less nested to do if (mem_tracker == nullptr) return; -- To view, visit http://gerrit.cloudera.org:8080/13670 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1a42434b98aec1d9e0be6ae52325049378fdde23 Gerrit-Change-Number: 13670 Gerrit-PatchSet: 3 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 18 Jun 2019 18:56:00 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8443: Record time spent in authorization in the runtime profile
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/13353 ) Change subject: IMPALA-8443: Record time spent in authorization in the runtime profile .. Patch Set 8: Code-Review+2 Thanks for the patch! -- To view, visit http://gerrit.cloudera.org:8080/13353 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5bb85e57fcc75d41f3eb2911e6d375e0da6f82ae Gerrit-Change-Number: 13353 Gerrit-PatchSet: 8 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Comment-Date: Tue, 18 Jun 2019 16:30:05 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8658: Populate missing Ranger audit fields
Fredy Wijaya has uploaded a new patch set (#8). ( http://gerrit.cloudera.org:8080/13601 ) Change subject: IMPALA-8658: Populate missing Ranger audit fields .. IMPALA-8658: Populate missing Ranger audit fields This patch adds missing Ranger audit fields, such as: - Client IP - Cluster name This patch also updates the access type to be in upper case to be consistent with Hive Ranger audit. The SQL statement logged will now return a redacted statement when the redacted flag is set. Testing: - Updated the tests in RangerAuditLogTest - Ran all FE tests - Tested the changes against Solr cluster Change-Id: I167bc35411ad9b4164c292077ff082671967cff8 --- M be/src/service/client-request-state.cc M common/thrift/CatalogService.thrift M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java M fe/src/main/java/org/apache/impala/analysis/GrantRevokePrivStmt.java M fe/src/main/java/org/apache/impala/authorization/AuthorizationChecker.java M fe/src/main/java/org/apache/impala/authorization/AuthorizationManager.java M fe/src/main/java/org/apache/impala/authorization/BaseAuthorizationChecker.java M fe/src/main/java/org/apache/impala/authorization/NoopAuthorizationFactory.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationContext.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerBufferAuditHandler.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerCatalogdAuthorizationManager.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpaladAuthorizationManager.java M fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizationChecker.java M fe/src/main/java/org/apache/impala/authorization/sentry/SentryCatalogdAuthorizationManager.java M fe/src/main/java/org/apache/impala/authorization/sentry/SentryImpaladAuthorizationManager.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/util/TSessionStateUtil.java M fe/src/test/java/org/apache/impala/authorization/AuthorizationTestBase.java M fe/src/test/java/org/apache/impala/authorization/ranger/RangerAuditLogTest.java M fe/src/test/java/org/apache/impala/common/FrontendTestBase.java M fe/src/test/resources/ranger-hive-audit.xml 23 files changed, 225 insertions(+), 135 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/01/13601/8 -- To view, visit http://gerrit.cloudera.org:8080/13601 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I167bc35411ad9b4164c292077ff082671967cff8 Gerrit-Change-Number: 13601 Gerrit-PatchSet: 8 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon
[Impala-ASF-CR] IMPALA-8599: Create a Maven module for query event hook API
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/13653 ) Change subject: IMPALA-8599: Create a Maven module for query event hook API .. Patch Set 3: > Patch Set 3: Verified-1 > > Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/4482/ Looks like network issue: 23:29:56 ] [ERROR] Failed to execute goal org.jacoco:jacoco-maven-plugin:0.7.6.201602180812:prepare-agent (prepare-jacoco-agent) on project impala-frontend: Execution prepare-jacoco-agent of goal org.jacoco:jacoco-maven-plugin:0.7.6.201602180812:prepare-agent failed: Plugin org.jacoco:jacoco-maven-plugin:0.7.6.201602180812 or one of its dependencies could not be resolved: Could not transfer artifact org.apache.maven.shared:maven-shared-io:jar:1.1 from/to central (https://repo.maven.apache.org/maven2): Failed to transfer file: https://repo.maven.apache.org/maven2/org/apache/maven/shared/maven-shared-io/1.1/maven-shared-io-1.1.jar. Return code is: 503 , ReasonPhrase:Maximum threads for service reached. -> [Help 1] 23:29:56 ] [ERROR] 23:29:56 ] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. 23:29:56 ] [ERROR] Re-run Maven using the -X switch to enable full debug logging. 23:29:56 ] [ERROR] 23:29:56 ] [ERROR] For more information about the errors and possible solutions, please read the following articles: 23:29:56 ] [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginResolutionException I'm going to retry. -- To view, visit http://gerrit.cloudera.org:8080/13653 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I84c422d83c19b75c3d1d7a772b971f4f7704d44c Gerrit-Change-Number: 13653 Gerrit-PatchSet: 3 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Tue, 18 Jun 2019 01:14:54 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8671: Do not re-create RangerAuthorizationChecker instance on catalog update
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/13663 ) Change subject: IMPALA-8671: Do not re-create RangerAuthorizationChecker instance on catalog update .. Patch Set 4: (2 comments) http://gerrit.cloudera.org:8080/#/c/13663/3/fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationFactory.java File fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationFactory.java: http://gerrit.cloudera.org:8080/#/c/13663/3/fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationFactory.java@79 PS3, Line 79: // since RangerAuthorizationChecker is stateless and creating a new instance of > do we need to reinit the ranger plugin for explicit REFRESH AUTHORIZATION c No, we can use the existing plugin instance and call a refreshTagsAndPolicy API instead of re-initing. http://gerrit.cloudera.org:8080/#/c/13663/3/fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationFactory.java@81 PS3, Line 81: // Ranger plugin. > this isn't safe under concurrency unless authzChecker is volatile. Should w Good idea. Done. -- To view, visit http://gerrit.cloudera.org:8080/13663 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iabd5c1b5c7cafc7a03681def0c0f6f775d690c41 Gerrit-Change-Number: 13663 Gerrit-PatchSet: 4 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Mon, 17 Jun 2019 20:22:56 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8671: Do not re-create RangerAuthorizationChecker instance on catalog update
Fredy Wijaya has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/13663 ) Change subject: IMPALA-8671: Do not re-create RangerAuthorizationChecker instance on catalog update .. IMPALA-8671: Do not re-create RangerAuthorizationChecker instance on catalog update This patch fixes the issue where RangerAuthorizationChecker instance gets created on every catalog update, which can impact the performance due to the need to re-initialize the Ranger plugin. Unlike SentryAuthorizationChecker which is stateful (it uses AuthorizationPolicy), RangerAuthorizationChecker is stateless and we can reuse the same instance of it. Testing: - Ran FE tests - Ran E2E authorization tests Change-Id: Iabd5c1b5c7cafc7a03681def0c0f6f775d690c41 --- M fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationFactory.java 1 file changed, 8 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/63/13663/4 -- To view, visit http://gerrit.cloudera.org:8080/13663 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iabd5c1b5c7cafc7a03681def0c0f6f775d690c41 Gerrit-Change-Number: 13663 Gerrit-PatchSet: 4 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon
[Impala-ASF-CR] IMPALA-8671: Do not re-create RangerAuthorizationChecker instance on catalog update
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/13663 ) Change subject: IMPALA-8671: Do not re-create RangerAuthorizationChecker instance on catalog update .. Patch Set 3: (1 comment) http://gerrit.cloudera.org:8080/#/c/13663/2/fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationFactory.java File fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationFactory.java: http://gerrit.cloudera.org:8080/#/c/13663/2/fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationFactory.java@39 PS2, Line 39: authzChecker_ > where does this ever get set? Ah my bad, I pushed the too quick :( I should've checked first before publishing. Sorry about that. -- To view, visit http://gerrit.cloudera.org:8080/13663 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iabd5c1b5c7cafc7a03681def0c0f6f775d690c41 Gerrit-Change-Number: 13663 Gerrit-PatchSet: 3 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Mon, 17 Jun 2019 20:12:44 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8671: Do not re-create RangerAuthorizationChecker instance on catalog update
Fredy Wijaya has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/13663 ) Change subject: IMPALA-8671: Do not re-create RangerAuthorizationChecker instance on catalog update .. IMPALA-8671: Do not re-create RangerAuthorizationChecker instance on catalog update This patch fixes the issue where RangerAuthorizationChecker instance gets created on every catalog update, which can impact the performance due to the need to re-initialize the Ranger plugin. Unlike SentryAuthorizationChecker which is stateful (it uses AuthorizationPolicy), RangerAuthorizationChecker is stateless and we can reuse the same instance of it. Testing: - Ran FE tests - Ran E2E authorization tests Change-Id: Iabd5c1b5c7cafc7a03681def0c0f6f775d690c41 --- M fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationFactory.java 1 file changed, 8 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/63/13663/3 -- To view, visit http://gerrit.cloudera.org:8080/13663 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iabd5c1b5c7cafc7a03681def0c0f6f775d690c41 Gerrit-Change-Number: 13663 Gerrit-PatchSet: 3 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon
[Impala-ASF-CR] IMPALA-8671: Do not re-create RangerAuthorizationChecker instance on catalog update
Fredy Wijaya has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13663 Change subject: IMPALA-8671: Do not re-create RangerAuthorizationChecker instance on catalog update .. IMPALA-8671: Do not re-create RangerAuthorizationChecker instance on catalog update This patch fixes the issue where RangerAuthorizationChecker instance gets created on every catalog update, which can impact the performance due to the need to re-initialize the Ranger plugin. Unlike SentryAuthorizationChecker which is stateful (it uses AuthorizationPolicy), RangerAuthorizationChecker is stateless and we can reuse the same instance of it. Testing: - Ran FE tests - Ran E2E authorization tests Change-Id: Iabd5c1b5c7cafc7a03681def0c0f6f775d690c41 --- M fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationFactory.java 1 file changed, 6 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/63/13663/2 -- To view, visit http://gerrit.cloudera.org:8080/13663 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Iabd5c1b5c7cafc7a03681def0c0f6f775d690c41 Gerrit-Change-Number: 13663 Gerrit-PatchSet: 2 Gerrit-Owner: Fredy Wijaya
[Impala-ASF-CR] IMPALA-8443: Record time spent in authorization in the runtime profile
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/13353 ) Change subject: IMPALA-8443: Record time spent in authorization in the runtime profile .. Patch Set 7: (2 comments) You need to rebase again to resolve the merge conflict. http://gerrit.cloudera.org:8080/#/c/13353/7/fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java File fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java: http://gerrit.cloudera.org:8080/#/c/13353/7/fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java@232 PS7, Line 232: nit: overly indented, use 4 spaces for continued indentation. http://gerrit.cloudera.org:8080/#/c/13353/7/fe/src/main/java/org/apache/impala/util/EventSequence.java File fe/src/main/java/org/apache/impala/util/EventSequence.java: http://gerrit.cloudera.org:8080/#/c/13353/7/fe/src/main/java/org/apache/impala/util/EventSequence.java@44 PS7, Line 44: duration nit: duration in nano seconds. -- To view, visit http://gerrit.cloudera.org:8080/13353 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5bb85e57fcc75d41f3eb2911e6d375e0da6f82ae Gerrit-Change-Number: 13353 Gerrit-PatchSet: 7 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Comment-Date: Mon, 17 Jun 2019 19:59:31 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8658: Populate missing Ranger audit fields
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/13601 ) Change subject: IMPALA-8658: Populate missing Ranger audit fields .. Patch Set 6: (5 comments) http://gerrit.cloudera.org:8080/#/c/13601/5/fe/src/main/java/org/apache/impala/authorization/AuthorizationChecker.java File fe/src/main/java/org/apache/impala/authorization/AuthorizationChecker.java: http://gerrit.cloudera.org:8080/#/c/13601/5/fe/src/main/java/org/apache/impala/authorization/AuthorizationChecker.java@43 PS5, Line 43:* @param sqlStmt the SQL statement to be logged for auditing > nit: add a @param for sessionState? Done http://gerrit.cloudera.org:8080/#/c/13601/5/fe/src/main/java/org/apache/impala/authorization/ranger/RangerBufferAuditHandler.java File fe/src/main/java/org/apache/impala/authorization/ranger/RangerBufferAuditHandler.java: http://gerrit.cloudera.org:8080/#/c/13601/5/fe/src/main/java/org/apache/impala/authorization/ranger/RangerBufferAuditHandler.java@59 PS5, Line 59: sqlStmt_ = Preconditions.checkNotNull(sqlStmt); > should you Preconditions.checkNotNull() for these? or Objects.firstNonNull( Done http://gerrit.cloudera.org:8080/#/c/13601/5/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java: http://gerrit.cloudera.org:8080/#/c/13601/5/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@367 PS5, Line 367: ddlRequest.getHeader(), ddlRequest.g > it seems like 'requestingUser' always is being constructed out of ddlReques Done http://gerrit.cloudera.org:8080/#/c/13601/5/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@367 PS5, Line 367: ddlRequest.getHeader(), ddlRequest.g > it seems like 'requestingUser' always is being constructed out of ddlReques Done http://gerrit.cloudera.org:8080/#/c/13601/5/fe/src/main/java/org/apache/impala/util/TSessionStateUtil.java File fe/src/main/java/org/apache/impala/util/TSessionStateUtil.java: http://gerrit.cloudera.org:8080/#/c/13601/5/fe/src/main/java/org/apache/impala/util/TSessionStateUtil.java@44 PS5, Line 44: InetAddresses.forString(session.getNetwork_address().getHostname()) : .getHostAddress(); > hrm, now I'm wondering why we can't just use getNetowkr_address().getHostna Ranger wants it into the IP address form. It maybe safe to provide Ranger what it needs in case it needs to do a reverse lookup or something. -- To view, visit http://gerrit.cloudera.org:8080/13601 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I167bc35411ad9b4164c292077ff082671967cff8 Gerrit-Change-Number: 13601 Gerrit-PatchSet: 6 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Mon, 17 Jun 2019 19:01:42 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8658: Populate missing Ranger audit fields
Fredy Wijaya has uploaded a new patch set (#6). ( http://gerrit.cloudera.org:8080/13601 ) Change subject: IMPALA-8658: Populate missing Ranger audit fields .. IMPALA-8658: Populate missing Ranger audit fields This patch adds missing Ranger audit fields, such as: - Client IP - Cluster name This patch also updates the access type to be in upper case to be consistent with Hive Ranger audit. The SQL statement logged will now return a redacted statement when the redacted flag is set. Testing: - Updated the tests in RangerAuditLogTest - Ran all FE tests - Tested the changes against Solr cluster Change-Id: I167bc35411ad9b4164c292077ff082671967cff8 --- M be/src/service/client-request-state.cc M common/thrift/CatalogService.thrift M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java M fe/src/main/java/org/apache/impala/analysis/GrantRevokePrivStmt.java M fe/src/main/java/org/apache/impala/authorization/AuthorizationChecker.java M fe/src/main/java/org/apache/impala/authorization/AuthorizationManager.java M fe/src/main/java/org/apache/impala/authorization/BaseAuthorizationChecker.java M fe/src/main/java/org/apache/impala/authorization/NoopAuthorizationFactory.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationContext.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerBufferAuditHandler.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerCatalogdAuthorizationManager.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpaladAuthorizationManager.java M fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizationChecker.java M fe/src/main/java/org/apache/impala/authorization/sentry/SentryCatalogdAuthorizationManager.java M fe/src/main/java/org/apache/impala/authorization/sentry/SentryImpaladAuthorizationManager.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/util/TSessionStateUtil.java M fe/src/test/java/org/apache/impala/authorization/AuthorizationTestBase.java M fe/src/test/java/org/apache/impala/authorization/ranger/RangerAuditLogTest.java M fe/src/test/java/org/apache/impala/common/FrontendTestBase.java M fe/src/test/resources/ranger-hive-audit.xml 23 files changed, 218 insertions(+), 135 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/01/13601/6 -- To view, visit http://gerrit.cloudera.org:8080/13601 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I167bc35411ad9b4164c292077ff082671967cff8 Gerrit-Change-Number: 13601 Gerrit-PatchSet: 6 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon
[Impala-ASF-CR] IMPALA-8599: Create a Maven module for query event hook API
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/13653 ) Change subject: IMPALA-8599: Create a Maven module for query event hook API .. Patch Set 2: > Patch Set 2: Code-Review+2 > > (1 comment) > > Looks fine. We should probably reorganize all this stuff so that the > different modules are submodules within a top-level directory with a pom to > tie them together (multi-module maven project) instead of depending on each > other as separate projects tied together by cmake, but I guess that's out of > scope for this patch. Any interest in taking that on? I agree, the existing project structure isn't the usual Maven project structure. Sure, I can take on fixing it. -- To view, visit http://gerrit.cloudera.org:8080/13653 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I84c422d83c19b75c3d1d7a772b971f4f7704d44c Gerrit-Change-Number: 13653 Gerrit-PatchSet: 2 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Mon, 17 Jun 2019 18:03:39 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8599: Create a Maven module for query event hook API
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/13653 ) Change subject: IMPALA-8599: Create a Maven module for query event hook API .. Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/13653/2/query-event-hook-api/CMakeLists.txt File query-event-hook-api/CMakeLists.txt: http://gerrit.cloudera.org:8080/#/c/13653/2/query-event-hook-api/CMakeLists.txt@18 PS2, Line 18: DEPENDS gen-deps impala-parent > qq: why is this needed? Looking at the hook, seems like a standalone thing. It depends on impala-parent because this module uses impala-parent. As for gen-deps, it's currently not needed, but if we need to use anything from Thrift in the future, then we'll need gen-deps. -- To view, visit http://gerrit.cloudera.org:8080/13653 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I84c422d83c19b75c3d1d7a772b971f4f7704d44c Gerrit-Change-Number: 13653 Gerrit-PatchSet: 2 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Mon, 17 Jun 2019 05:04:33 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8599: Create a Maven module for query event hook API
Fredy Wijaya has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13653 Change subject: IMPALA-8599: Create a Maven module for query event hook API .. IMPALA-8599: Create a Maven module for query event hook API This patch moves the query event event hook API into a separate Maven module so that it can be easily consumed by other projects, such as Atlas. Testing: - Ran make fe Change-Id: I84c422d83c19b75c3d1d7a772b971f4f7704d44c --- M CMakeLists.txt M fe/CMakeLists.txt M fe/pom.xml M impala-parent/pom.xml A query-event-hook-api/CMakeLists.txt A query-event-hook-api/pom.xml R query-event-hook-api/src/main/java/org/apache/impala/hooks/QueryCompleteContext.java R query-event-hook-api/src/main/java/org/apache/impala/hooks/QueryEventHook.java 8 files changed, 92 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/53/13653/2 -- To view, visit http://gerrit.cloudera.org:8080/13653 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I84c422d83c19b75c3d1d7a772b971f4f7704d44c Gerrit-Change-Number: 13653 Gerrit-PatchSet: 2 Gerrit-Owner: Fredy Wijaya
[Impala-ASF-CR] Remove "Could not transfer" exclusion in mvn-quiet.sh
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/13647 ) Change subject: Remove "Could not transfer" exclusion in mvn-quiet.sh .. Patch Set 1: > Patch Set 1: > > We could also consider removing the whole error suppression thing. I added it > a long time ago when I was trying to make Jenkins logs less noisy but I'm now > not sure that it was a good idea. The current mvn-quiet.sh doesn't suppress error though. So, I think we're good here. -- To view, visit http://gerrit.cloudera.org:8080/13647 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ide3367fd98abbbe11eec1fa86fbad8b32eeecb8d Gerrit-Change-Number: 13647 Gerrit-PatchSet: 1 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 14 Jun 2019 16:25:21 + Gerrit-HasComments: No
[Impala-ASF-CR] Remove "Could not transfer" exclusion in mvn-quiet.sh
Fredy Wijaya has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13647 Change subject: Remove "Could not transfer" exclusion in mvn-quiet.sh .. Remove "Could not transfer" exclusion in mvn-quiet.sh "Could not transfer" warning messages are noisy. However, excluding "Could not transfer" words can lead to actual error messages that contain "Could not transfer" to not be shown in the stdout, which can make debugging difficult. This patch updates mvn-quiet.sh to show "Could not transfer" messages. Testing: - Ran FE build Change-Id: Ide3367fd98abbbe11eec1fa86fbad8b32eeecb8d --- M bin/mvn-quiet.sh 1 file changed, 1 insertion(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/47/13647/1 -- To view, visit http://gerrit.cloudera.org:8080/13647 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ide3367fd98abbbe11eec1fa86fbad8b32eeecb8d Gerrit-Change-Number: 13647 Gerrit-PatchSet: 1 Gerrit-Owner: Fredy Wijaya
[Impala-ASF-CR] IMPALA-8658: Populate missing Ranger audit fields
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/13601 ) Change subject: IMPALA-8658: Populate missing Ranger audit fields .. Patch Set 5: (1 comment) http://gerrit.cloudera.org:8080/#/c/13601/3/fe/src/main/java/org/apache/impala/util/TSessionStateUtil.java File fe/src/main/java/org/apache/impala/util/TSessionStateUtil.java: http://gerrit.cloudera.org:8080/#/c/13601/3/fe/src/main/java/org/apache/impala/util/TSessionStateUtil.java@47 PS3, Line 47: } > I guess I'm wondering: would this ever _not_ be a dotted-decimal address? I Good idea. Done. -- To view, visit http://gerrit.cloudera.org:8080/13601 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I167bc35411ad9b4164c292077ff082671967cff8 Gerrit-Change-Number: 13601 Gerrit-PatchSet: 5 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Fri, 14 Jun 2019 01:48:54 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8658: Populate missing Ranger audit fields
Fredy Wijaya has uploaded a new patch set (#5). ( http://gerrit.cloudera.org:8080/13601 ) Change subject: IMPALA-8658: Populate missing Ranger audit fields .. IMPALA-8658: Populate missing Ranger audit fields This patch adds missing Ranger audit fields, such as: - Client IP - Cluster name This patch also updates the access type to be in upper case to be consistent with Hive Ranger audit. The SQL statement logged will now return a redacted statement when the redacted flag is set. Testing: - Updated the tests in RangerAuditLogTest - Ran all FE tests - Tested the changes against Solr cluster Change-Id: I167bc35411ad9b4164c292077ff082671967cff8 --- M be/src/service/client-request-state.cc M common/thrift/CatalogService.thrift M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java M fe/src/main/java/org/apache/impala/analysis/GrantRevokePrivStmt.java M fe/src/main/java/org/apache/impala/authorization/AuthorizationChecker.java M fe/src/main/java/org/apache/impala/authorization/AuthorizationManager.java M fe/src/main/java/org/apache/impala/authorization/BaseAuthorizationChecker.java M fe/src/main/java/org/apache/impala/authorization/NoopAuthorizationFactory.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationContext.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerBufferAuditHandler.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerCatalogdAuthorizationManager.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpaladAuthorizationManager.java M fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizationChecker.java M fe/src/main/java/org/apache/impala/authorization/sentry/SentryCatalogdAuthorizationManager.java M fe/src/main/java/org/apache/impala/authorization/sentry/SentryImpaladAuthorizationManager.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/util/TSessionStateUtil.java M fe/src/test/java/org/apache/impala/authorization/AuthorizationTestBase.java M fe/src/test/java/org/apache/impala/authorization/ranger/RangerAuditLogTest.java M fe/src/test/java/org/apache/impala/common/FrontendTestBase.java M fe/src/test/resources/ranger-hive-audit.xml 23 files changed, 211 insertions(+), 95 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/01/13601/5 -- To view, visit http://gerrit.cloudera.org:8080/13601 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I167bc35411ad9b4164c292077ff082671967cff8 Gerrit-Change-Number: 13601 Gerrit-PatchSet: 5 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon
[Impala-ASF-CR] IMPALA-8658: Populate missing Ranger audit fields
Fredy Wijaya has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/13601 ) Change subject: IMPALA-8658: Populate missing Ranger audit fields .. IMPALA-8658: Populate missing Ranger audit fields This patch adds missing Ranger audit fields, such as: - Client IP - Cluster name This patch also updates the access type to be in upper case to be consistent with Hive Ranger audit. The SQL statement logged will now return a redacted statement when the redacted flag is set. Testing: - Updated the tests in RangerAuditLogTest - Ran all FE tests - Tested the changes against Solr cluster Change-Id: I167bc35411ad9b4164c292077ff082671967cff8 --- M be/src/service/client-request-state.cc M common/thrift/CatalogService.thrift M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java M fe/src/main/java/org/apache/impala/analysis/GrantRevokePrivStmt.java M fe/src/main/java/org/apache/impala/authorization/AuthorizationChecker.java M fe/src/main/java/org/apache/impala/authorization/AuthorizationManager.java M fe/src/main/java/org/apache/impala/authorization/BaseAuthorizationChecker.java M fe/src/main/java/org/apache/impala/authorization/NoopAuthorizationFactory.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationContext.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerBufferAuditHandler.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerCatalogdAuthorizationManager.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpaladAuthorizationManager.java M fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizationChecker.java M fe/src/main/java/org/apache/impala/authorization/sentry/SentryCatalogdAuthorizationManager.java M fe/src/main/java/org/apache/impala/authorization/sentry/SentryImpaladAuthorizationManager.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/util/TSessionStateUtil.java M fe/src/test/java/org/apache/impala/authorization/AuthorizationTestBase.java M fe/src/test/java/org/apache/impala/authorization/ranger/RangerAuditLogTest.java M fe/src/test/java/org/apache/impala/common/FrontendTestBase.java M fe/src/test/resources/ranger-hive-audit.xml 23 files changed, 217 insertions(+), 95 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/01/13601/4 -- To view, visit http://gerrit.cloudera.org:8080/13601 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I167bc35411ad9b4164c292077ff082671967cff8 Gerrit-Change-Number: 13601 Gerrit-PatchSet: 4 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon
[Impala-ASF-CR] IMPALA-8658: Populate missing Ranger audit fields
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/13601 ) Change subject: IMPALA-8658: Populate missing Ranger audit fields .. Patch Set 4: (5 comments) http://gerrit.cloudera.org:8080/#/c/13601/3/common/thrift/JniCatalog.thrift File common/thrift/JniCatalog.thrift: http://gerrit.cloudera.org:8080/#/c/13601/3/common/thrift/JniCatalog.thrift@642 PS3, Line 642: // Parameters of DROP DATABASE commands > Can you name this 'redacted_sql_stmt' or something so that it's clear this Done http://gerrit.cloudera.org:8080/#/c/13601/3/common/thrift/JniCatalog.thrift@640 PS3, Line 640: } : : // Parameters of DROP DATABASE commands : struct TDropDbParams { : // Name of the database to drop : 1: required string db > What do you think about moving these to TCatalogServiceRequestHeader? They Done http://gerrit.cloudera.org:8080/#/c/13601/3/fe/src/main/java/org/apache/impala/analysis/GrantRevokePrivStmt.java File fe/src/main/java/org/apache/impala/analysis/GrantRevokePrivStmt.java: http://gerrit.cloudera.org:8080/#/c/13601/3/fe/src/main/java/org/apache/impala/analysis/GrantRevokePrivStmt.java@111 PS3, Line 111: : > seems weird that this can throw. Is it trying to reverse-lookup or somethin Yeah it's just Java checked exception. I also don't know a case where it'll throw an exception. I'll change it to throw an unchecked exception in getClientIpAddress instead. http://gerrit.cloudera.org:8080/#/c/13601/3/fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java File fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java: http://gerrit.cloudera.org:8080/#/c/13601/3/fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java@222 PS3, Line 222: } : } : retur > similar to elsewhere, it seems like this should never happen if we just hav Done http://gerrit.cloudera.org:8080/#/c/13601/3/fe/src/main/java/org/apache/impala/util/TSessionStateUtil.java File fe/src/main/java/org/apache/impala/util/TSessionStateUtil.java: http://gerrit.cloudera.org:8080/#/c/13601/3/fe/src/main/java/org/apache/impala/util/TSessionStateUtil.java@47 PS3, Line 47: return InetAddress.getByName(session.getNetwork_address().getHostname()) > We could avoid the 'throws UnknownHostException' by just using String.forma Ranger wants it to be an IP address, although technically it's just a string and I'm sure it'll accept anything. -- To view, visit http://gerrit.cloudera.org:8080/13601 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I167bc35411ad9b4164c292077ff082671967cff8 Gerrit-Change-Number: 13601 Gerrit-PatchSet: 4 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Thu, 13 Jun 2019 23:44:19 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8658: Populate missing Ranger audit fields
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/13601 ) Change subject: IMPALA-8658: Populate missing Ranger audit fields .. Patch Set 3: (2 comments) http://gerrit.cloudera.org:8080/#/c/13601/2/fe/src/main/java/org/apache/impala/authorization/ranger/RangerBufferAuditHandler.java File fe/src/main/java/org/apache/impala/authorization/ranger/RangerBufferAuditHandler.java: http://gerrit.cloudera.org:8080/#/c/13601/2/fe/src/main/java/org/apache/impala/authorization/ranger/RangerBufferAuditHandler.java@117 PS2, Line 117: String resourceType = resource != null ? resource.getLeafName() : null; > this seems like the impala IP, not the client's IP. I'm not sure exactly wh Ah Impala client IP. I thought it was Ranger client IP, which is an Impala IP :) Done. http://gerrit.cloudera.org:8080/#/c/13601/2/fe/src/test/resources/ranger-hive-audit.xml File fe/src/test/resources/ranger-hive-audit.xml: http://gerrit.cloudera.org:8080/#/c/13601/2/fe/src/test/resources/ranger-hive-audit.xml@27 PS2, Line 27: ranger.plugin.hive.ambari.cluster.name > if this parameter isn't set by the end user, do we risk an uninterpretable I don't think this field is strictly required by the Ranger audit. It seems to work fine without it. It is okay to pass null when cluster name isn't available. -- To view, visit http://gerrit.cloudera.org:8080/13601 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I167bc35411ad9b4164c292077ff082671967cff8 Gerrit-Change-Number: 13601 Gerrit-PatchSet: 3 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Thu, 13 Jun 2019 19:56:45 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8658: Populate missing Ranger audit fields
Fredy Wijaya has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/13601 ) Change subject: IMPALA-8658: Populate missing Ranger audit fields .. IMPALA-8658: Populate missing Ranger audit fields This patch adds missing Ranger audit fields, such as: - Client IP - Cluster name This patch also updates the access type to be in upper case to be consistent with Hive Ranger audit. The SQL statement logged will now return a redacted statement when the redacted flag is set. Testing: - Updated the tests in RangerAuditLogTest - Ran all FE tests - Tested the changes against Solr cluster Change-Id: I167bc35411ad9b4164c292077ff082671967cff8 --- M common/thrift/JniCatalog.thrift M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java M fe/src/main/java/org/apache/impala/analysis/GrantRevokePrivStmt.java M fe/src/main/java/org/apache/impala/authorization/AuthorizationChecker.java M fe/src/main/java/org/apache/impala/authorization/BaseAuthorizationChecker.java M fe/src/main/java/org/apache/impala/authorization/NoopAuthorizationFactory.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationContext.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerBufferAuditHandler.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerCatalogdAuthorizationManager.java M fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizationChecker.java M fe/src/main/java/org/apache/impala/util/TSessionStateUtil.java M fe/src/test/java/org/apache/impala/authorization/AuthorizationTestBase.java M fe/src/test/java/org/apache/impala/authorization/ranger/RangerAuditLogTest.java M fe/src/test/java/org/apache/impala/common/FrontendTestBase.java M fe/src/test/resources/ranger-hive-audit.xml 16 files changed, 125 insertions(+), 32 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/01/13601/3 -- To view, visit http://gerrit.cloudera.org:8080/13601 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I167bc35411ad9b4164c292077ff082671967cff8 Gerrit-Change-Number: 13601 Gerrit-PatchSet: 3 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon
[Impala-ASF-CR] IMPALA-8443: Record time spent in authorization in the runtime profile
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/13353 ) Change subject: IMPALA-8443: Record time spent in authorization in the runtime profile .. Patch Set 3: > Patch Set 3: > > Hi Fredy, thanks for keeping an eye on this. > > Yes, I am working on it, I just got interrupted in the past couple of weeks. > > I was moving the timeline into the AuthorizationContext here: > https://github.com/apache/impala/blob/ab908d54c22861967f693428ec7d9f6d7008607f/fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java#L424 > > However, in some cases a new AuthorizationContext is created during > AuthorizationChecker 'hasAccess' call. > https://github.com/apache/impala/blob/ab908d54c22861967f693428ec7d9f6d7008607f/fe/src/main/java/org/apache/impala/authorization/BaseAuthorizationChecker.java#L66 > > I am working on to untangle this part at the moment, will get back to you > with a new patch by EOB Monday. No worries :) -- To view, visit http://gerrit.cloudera.org:8080/13353 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5bb85e57fcc75d41f3eb2911e6d375e0da6f82ae Gerrit-Change-Number: 13353 Gerrit-PatchSet: 3 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Comment-Date: Thu, 13 Jun 2019 16:30:40 + Gerrit-HasComments: No
[Impala-ASF-CR](2.x) Ignore IMPALA-7316, IMPALA-7212 and IMPALA-5607
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/13609 ) Change subject: Ignore IMPALA-7316, IMPALA-7212 and IMPALA-5607 .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13609 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: 2.x Gerrit-MessageType: comment Gerrit-Change-Id: I5465fed79b8cb5969a50eceedfb9d147a2a12811 Gerrit-Change-Number: 13609 Gerrit-PatchSet: 1 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 13 Jun 2019 16:30:04 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8658: Populate missing Ranger audit fields
Fredy Wijaya has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13601 Change subject: IMPALA-8658: Populate missing Ranger audit fields .. IMPALA-8658: Populate missing Ranger audit fields This patch adds missing Ranger audit fields, such as: - Client IP - Cluster name This patch also updates the access type to be in upper case to be consistent with Hive Ranger audit. Testing: - Updated the tests in RangerAuditLogTest - Ran all FE tests - Tested the changes against Solr cluster Change-Id: I167bc35411ad9b4164c292077ff082671967cff8 --- M fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerBufferAuditHandler.java M fe/src/test/java/org/apache/impala/authorization/ranger/RangerAuditLogTest.java M fe/src/test/resources/ranger-hive-audit.xml 4 files changed, 26 insertions(+), 4 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/01/13601/2 -- To view, visit http://gerrit.cloudera.org:8080/13601 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I167bc35411ad9b4164c292077ff082671967cff8 Gerrit-Change-Number: 13601 Gerrit-PatchSet: 2 Gerrit-Owner: Fredy Wijaya
[Impala-ASF-CR] IMPALA-8654: Log the SQL statement in the Ranger audit log
Fredy Wijaya has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/13590 ) Change subject: IMPALA-8654: Log the SQL statement in the Ranger audit log .. IMPALA-8654: Log the SQL statement in the Ranger audit log This patch logs the SQL statement to be authorized in the Ranger audit log since it was a required field prior to the fix in RANGER-2463 to avoid a NullPointerException in the Ranger admin that could cause the Ranger audit logs to not show up in the Ranger web UI. Testing: - Updated the RangerAuditLogTest - Tested the changes against Solr cluster - Ran all FE tests Change-Id: Id9f584ac4209604675eb13b6d6f349c6cbb1a387 --- M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java M fe/src/main/java/org/apache/impala/analysis/CreateDbStmt.java M fe/src/main/java/org/apache/impala/authorization/AuthorizationChecker.java M fe/src/main/java/org/apache/impala/authorization/BaseAuthorizationChecker.java M fe/src/main/java/org/apache/impala/authorization/NoopAuthorizationFactory.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerBufferAuditHandler.java M fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizationChecker.java M fe/src/test/java/org/apache/impala/authorization/ranger/RangerAuditLogTest.java M fe/src/test/java/org/apache/impala/common/FrontendTestBase.java 10 files changed, 71 insertions(+), 12 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/90/13590/4 -- To view, visit http://gerrit.cloudera.org:8080/13590 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Id9f584ac4209604675eb13b6d6f349c6cbb1a387 Gerrit-Change-Number: 13590 Gerrit-PatchSet: 4 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon
[Impala-ASF-CR] IMPALA-8443: Record time spent in authorization in the runtime profile
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/13353 ) Change subject: IMPALA-8443: Record time spent in authorization in the runtime profile .. Patch Set 3: > Patch Set 3: Patch Set 2 was rebased Hi Tamas, I saw that you just rebased the patch. Are you going to make a change to make sure the authorization time logged here: https://github.com/apache/impala/blob/ab908d54c22861967f693428ec7d9f6d7008607f/fe/src/main/java/org/apache/impala/authorization/BaseAuthorizationChecker.java#L89 is the same as the one in timeline.markEvent("Authorization finished")? -- To view, visit http://gerrit.cloudera.org:8080/13353 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5bb85e57fcc75d41f3eb2911e6d375e0da6f82ae Gerrit-Change-Number: 13353 Gerrit-PatchSet: 3 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Comment-Date: Wed, 12 Jun 2019 01:23:36 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8654: Log the SQL statement in the Ranger audit log
Fredy Wijaya has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13590 Change subject: IMPALA-8654: Log the SQL statement in the Ranger audit log .. IMPALA-8654: Log the SQL statement in the Ranger audit log This patch logs the SQL statement to be authorized in the Ranger audit log since it was a required field prior to the fix in RANGER-2463 to avoid a NullPointerException in the Ranger admin that could cause the Ranger audit logs to not show up in the Ranger web UI. Logging the SQL statement requires proper implementation of StatementBase::toSql() which unfortunately is not available in all SQL statements. See IMPALA-8655. Testing: - Updated the RangerAuditLogTest - Tested the changes against Solr cluster Change-Id: Id9f584ac4209604675eb13b6d6f349c6cbb1a387 --- M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java M fe/src/main/java/org/apache/impala/analysis/CreateDbStmt.java M fe/src/main/java/org/apache/impala/authorization/AuthorizationChecker.java M fe/src/main/java/org/apache/impala/authorization/BaseAuthorizationChecker.java M fe/src/main/java/org/apache/impala/authorization/NoopAuthorizationFactory.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerBufferAuditHandler.java M fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizationChecker.java M fe/src/test/java/org/apache/impala/authorization/ranger/RangerAuditLogTest.java M fe/src/test/java/org/apache/impala/common/FrontendTestBase.java 10 files changed, 84 insertions(+), 12 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/90/13590/1 -- To view, visit http://gerrit.cloudera.org:8080/13590 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Id9f584ac4209604675eb13b6d6f349c6cbb1a387 Gerrit-Change-Number: 13590 Gerrit-PatchSet: 1 Gerrit-Owner: Fredy Wijaya
[Impala-ASF-CR] IMPALA-7154 : Error making 'dropDatabase' RPC to Hive Metastore. NoSuchObjectException thrown
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/13586 ) Change subject: IMPALA-7154 : Error making 'dropDatabase' RPC to Hive Metastore. NoSuchObjectException thrown .. Patch Set 1: Code-Review+1 (1 comment) http://gerrit.cloudera.org:8080/#/c/13586/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/13586/1//COMMIT_MSG@14 PS1, Line 14: timedout typo: timed out -- To view, visit http://gerrit.cloudera.org:8080/13586 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I253fc00618b28d8d6e626e1b6bf9c5c8290006d7 Gerrit-Change-Number: 13586 Gerrit-PatchSet: 1 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Comment-Date: Tue, 11 Jun 2019 22:26:23 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8649: Fix confusing SHOW GRANT error messages
Fredy Wijaya has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13587 Change subject: IMPALA-8649: Fix confusing SHOW GRANT error messages .. IMPALA-8649: Fix confusing SHOW GRANT error messages Before this patch, PrivilegeSpec had error messages tailored to GRANT and REVOKE statements. However, PrivilegeSpec is also used in SHOW GRANT statement. This patch updates the error messages to be less confusing in SHOW GRANT statement. Testing: - Added new tests in AnalyzeAuthStmtsTest - Ran AnalyzeAuthStmtsTest Change-Id: Ibb88bdc19cd1223902b44e3634f756d086332266 --- M fe/src/main/java/org/apache/impala/analysis/PrivilegeSpec.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeAuthStmtsTest.java 2 files changed, 33 insertions(+), 18 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/87/13587/2 -- To view, visit http://gerrit.cloudera.org:8080/13587 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ibb88bdc19cd1223902b44e3634f756d086332266 Gerrit-Change-Number: 13587 Gerrit-PatchSet: 2 Gerrit-Owner: Fredy Wijaya
[Impala-ASF-CR] IMPALA-8588: Fix revoke grant option with Ranger
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/13450 ) Change subject: IMPALA-8588: Fix revoke grant option with Ranger .. Patch Set 2: Code-Review+2 (1 comment) http://gerrit.cloudera.org:8080/#/c/13450/2//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/13450/2//COMMIT_MSG@20 PS2, Line 20: REVOKE GRANT OPTION FOR SELECT ON DATABASE FROM USER : : This will revoke the grant option for all privileges on this database : resource. It will not revoke the SELECT privilege on the resource. > I think Option 1 makes more sense as it offers better functionality to the Thanks. That works for me. Let's fix the bug first and do another enhancement in a separate CR. -- To view, visit http://gerrit.cloudera.org:8080/13450 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iddfccb442c3be3c266dbc2d8ae85c5674c534d7c Gerrit-Change-Number: 13450 Gerrit-PatchSet: 2 Gerrit-Owner: Austin Nobis Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 11 Jun 2019 19:06:40 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8612: Fix sporadic NPE when dropping an authorized table
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/13508 ) Change subject: IMPALA-8612: Fix sporadic NPE when dropping an authorized table .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/13508/1/fe/src/main/java/org/apache/impala/analysis/DropTableOrViewStmt.java File fe/src/main/java/org/apache/impala/analysis/DropTableOrViewStmt.java: http://gerrit.cloudera.org:8080/#/c/13508/1/fe/src/main/java/org/apache/impala/analysis/DropTableOrViewStmt.java@124 PS1, Line 124: analyzer.addAccessEvent(new TAccessEvent( > Should we log the exception? We should have an E2E test here by having a table that is dropped externally. -- To view, visit http://gerrit.cloudera.org:8080/13508 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I70bd7ca4796b24920ee156436bf8bbc682e7d952 Gerrit-Change-Number: 13508 Gerrit-PatchSet: 1 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 11 Jun 2019 18:30:32 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8551: Make the grant/revoke error messages to be more user friendly
Fredy Wijaya has uploaded a new patch set (#8). ( http://gerrit.cloudera.org:8080/13525 ) Change subject: IMPALA-8551: Make the grant/revoke error messages to be more user friendly .. IMPALA-8551: Make the grant/revoke error messages to be more user friendly This patch updates the grant/revoke error messages to be more user friendly, especially when granting/revoking with an invalid principal. Testing: - Added E2E test to test grant/revoke with invalid principal Change-Id: I8995f5dc88b211cd3af415713802cfeac44fe576 --- M fe/src/main/java/org/apache/impala/authorization/ranger/RangerCatalogdAuthorizationManager.java M tests/authorization/test_ranger.py 2 files changed, 64 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/25/13525/8 -- To view, visit http://gerrit.cloudera.org:8080/13525 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I8995f5dc88b211cd3af415713802cfeac44fe576 Gerrit-Change-Number: 13525 Gerrit-PatchSet: 8 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon
[Impala-ASF-CR] IMPALA-8551: Make the grant/revoke error messages to be more user friendly
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/13525 ) Change subject: IMPALA-8551: Make the grant/revoke error messages to be more user friendly .. Patch Set 8: Code-Review+2 Rebased. Carry Todd's +2. -- To view, visit http://gerrit.cloudera.org:8080/13525 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8995f5dc88b211cd3af415713802cfeac44fe576 Gerrit-Change-Number: 13525 Gerrit-PatchSet: 8 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Tue, 11 Jun 2019 01:02:26 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8551: Make the grant/revoke error messages to be more user friendly
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/13525 ) Change subject: IMPALA-8551: Make the grant/revoke error messages to be more user friendly .. Patch Set 7: (1 comment) http://gerrit.cloudera.org:8080/#/c/13525/6/tests/authorization/test_ranger.py File tests/authorization/test_ranger.py: http://gerrit.cloudera.org:8080/#/c/13525/6/tests/authorization/test_ranger.py@572 PS6, Line 572: assert "Error granting a privilege in Ranger. Ranger error message: " \ > don't these asserts need to be updated? Sorry, I forgot to commit this change. Done. -- To view, visit http://gerrit.cloudera.org:8080/13525 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8995f5dc88b211cd3af415713802cfeac44fe576 Gerrit-Change-Number: 13525 Gerrit-PatchSet: 7 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Mon, 10 Jun 2019 19:25:22 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8551: Make the grant/revoke error messages to be more user friendly
Fredy Wijaya has uploaded a new patch set (#7). ( http://gerrit.cloudera.org:8080/13525 ) Change subject: IMPALA-8551: Make the grant/revoke error messages to be more user friendly .. IMPALA-8551: Make the grant/revoke error messages to be more user friendly This patch updates the grant/revoke error messages to be more user friendly, especially when granting/revoking with an invalid principal. Testing: - Added E2E test to test grant/revoke with invalid principal Change-Id: I8995f5dc88b211cd3af415713802cfeac44fe576 --- M fe/src/main/java/org/apache/impala/authorization/ranger/RangerCatalogdAuthorizationManager.java M tests/authorization/test_ranger.py 2 files changed, 64 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/25/13525/7 -- To view, visit http://gerrit.cloudera.org:8080/13525 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I8995f5dc88b211cd3af415713802cfeac44fe576 Gerrit-Change-Number: 13525 Gerrit-PatchSet: 7 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon
[Impala-ASF-CR] IMPALA-8551: Make the grant/revoke error messages to be more user friendly
Fredy Wijaya has uploaded a new patch set (#6). ( http://gerrit.cloudera.org:8080/13525 ) Change subject: IMPALA-8551: Make the grant/revoke error messages to be more user friendly .. IMPALA-8551: Make the grant/revoke error messages to be more user friendly This patch updates the grant/revoke error messages to be more user friendly, especially when granting/revoking with an invalid principal. Testing: - Added E2E test to test grant/revoke with invalid principal Change-Id: I8995f5dc88b211cd3af415713802cfeac44fe576 --- M fe/src/main/java/org/apache/impala/authorization/ranger/RangerCatalogdAuthorizationManager.java M tests/authorization/test_ranger.py 2 files changed, 64 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/25/13525/6 -- To view, visit http://gerrit.cloudera.org:8080/13525 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I8995f5dc88b211cd3af415713802cfeac44fe576 Gerrit-Change-Number: 13525 Gerrit-PatchSet: 6 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon
[Impala-ASF-CR] IMPALA-8551: Make the grant/revoke error messages to be more user friendly
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/13525 ) Change subject: IMPALA-8551: Make the grant/revoke error messages to be more user friendly .. Patch Set 6: (2 comments) > Patch Set 5: > > (2 comments) > > Looks pretty good modulo some nits. One thing, though: would it be possible > to split this into one commit to bump the CDP_BUILD_NUMBER and one to change > the error messages? I'm afraid when we look through lists of commits to > backport across branches, we'll see this and think "oh it's just a CDP > version bump, we can skip this one and pick up a later one" whereas in fact > it also contains a functional change. Split into two now. https://gerrit.cloudera.org/c/13574/ https://gerrit.cloudera.org/c/13525/ http://gerrit.cloudera.org:8080/#/c/13525/5/fe/src/main/java/org/apache/impala/authorization/ranger/RangerCatalogdAuthorizationManager.java File fe/src/main/java/org/apache/impala/authorization/ranger/RangerCatalogdAuthorizationManager.java: http://gerrit.cloudera.org:8080/#/c/13525/5/fe/src/main/java/org/apache/impala/authorization/ranger/RangerCatalogdAuthorizationManager.java@173 PS5, Line 173: try (AutoFlush auditHandler = RangerBufferAuditHandler.autoFlush()) { > pre-existing issue, but: is it possible to add a DEBUG level log message he Wouldn't be better to use ERROR log level instead? http://gerrit.cloudera.org:8080/#/c/13525/5/fe/src/main/java/org/apache/impala/authorization/ranger/RangerCatalogdAuthorizationManager.java@174 PS5, Line 174: plugin_.get().grantAccess(request, auditHandler); > nit: can we reword this to "in Ranger" instead of "to Ranger" -- the privil Done -- To view, visit http://gerrit.cloudera.org:8080/13525 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8995f5dc88b211cd3af415713802cfeac44fe576 Gerrit-Change-Number: 13525 Gerrit-PatchSet: 6 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Mon, 10 Jun 2019 16:17:45 + Gerrit-HasComments: Yes
[Impala-ASF-CR] Bump CDP BUILD NUMBER to 1153860
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/13574 ) Change subject: Bump CDP_BUILD_NUMBER to 1153860 .. Patch Set 1: Splitting the https://gerrit.cloudera.org/c/13525/ into two commits. -- To view, visit http://gerrit.cloudera.org:8080/13574 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ief3458c7dbf2ee973b586e2d0d54cc0739b6ae3b Gerrit-Change-Number: 13574 Gerrit-PatchSet: 1 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Mon, 10 Jun 2019 16:01:49 + Gerrit-HasComments: No
[Impala-ASF-CR] Bump CDP BUILD NUMBER to 1153860
Fredy Wijaya has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13574 Change subject: Bump CDP_BUILD_NUMBER to 1153860 .. Bump CDP_BUILD_NUMBER to 1153860 This patch bumps the CDP_BUILD_NUMBER to 1153860 to pull RANGER-2437 fix that provides better error messages when granting/revoking with invalid user/group. Testing: - Ran core tests - Ran the build with USE_CDP_HIVE=true Change-Id: Ief3458c7dbf2ee973b586e2d0d54cc0739b6ae3b --- M bin/impala-config.sh 1 file changed, 5 insertions(+), 5 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/74/13574/1 -- To view, visit http://gerrit.cloudera.org:8080/13574 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ief3458c7dbf2ee973b586e2d0d54cc0739b6ae3b Gerrit-Change-Number: 13574 Gerrit-PatchSet: 1 Gerrit-Owner: Fredy Wijaya
[Impala-ASF-CR] IMPALA-8551: Bump CDP BUILD NUMBER to 1153860
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/13525 ) Change subject: IMPALA-8551: Bump CDP_BUILD_NUMBER to 1153860 .. Patch Set 5: (1 comment) http://gerrit.cloudera.org:8080/#/c/13525/4/tests/authorization/test_ranger.py File tests/authorization/test_ranger.py: http://gerrit.cloudera.org:8080/#/c/13525/4/tests/authorization/test_ranger.py@587 PS4, Line 587: H > flake8: E126 continuation line over-indented for hanging indent Done -- To view, visit http://gerrit.cloudera.org:8080/13525 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8995f5dc88b211cd3af415713802cfeac44fe576 Gerrit-Change-Number: 13525 Gerrit-PatchSet: 5 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Fri, 07 Jun 2019 21:18:37 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8551: Bump CDP BUILD NUMBER to 1153860
Fredy Wijaya has uploaded a new patch set (#5). ( http://gerrit.cloudera.org:8080/13525 ) Change subject: IMPALA-8551: Bump CDP_BUILD_NUMBER to 1153860 .. IMPALA-8551: Bump CDP_BUILD_NUMBER to 1153860 This patch bumps CDP_BUILD_NUMBER to 1153860 to pull RANGER-2437 fix that provides better error messages when granting/revoking with invalid user/group. Testing: - Added E2E test to test grant/revoke with invalid principal - Ran core tests - Ran the build with USE_CDP_HIVE=true Change-Id: I8995f5dc88b211cd3af415713802cfeac44fe576 --- M bin/impala-config.sh M fe/src/main/java/org/apache/impala/authorization/ranger/RangerCatalogdAuthorizationManager.java M tests/authorization/test_ranger.py 3 files changed, 63 insertions(+), 7 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/25/13525/5 -- To view, visit http://gerrit.cloudera.org:8080/13525 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I8995f5dc88b211cd3af415713802cfeac44fe576 Gerrit-Change-Number: 13525 Gerrit-PatchSet: 5 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon
[Impala-ASF-CR] IMPALA-8551: Bump CDP BUILD NUMBER to 1153860
Fredy Wijaya has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/13525 ) Change subject: IMPALA-8551: Bump CDP_BUILD_NUMBER to 1153860 .. IMPALA-8551: Bump CDP_BUILD_NUMBER to 1153860 This patch bumps CDP_BUILD_NUMBER to 1153860 to pull RANGER-2437 fix that provides better error messages when granting/revoking with invalid user/group. Testing: - Added E2E test to test grant/revoke with invalid principal - Ran core tests - Ran the build with USE_CDP_HIVE=true Change-Id: I8995f5dc88b211cd3af415713802cfeac44fe576 --- M bin/impala-config.sh M fe/src/main/java/org/apache/impala/authorization/ranger/RangerCatalogdAuthorizationManager.java M tests/authorization/test_ranger.py 3 files changed, 63 insertions(+), 7 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/25/13525/4 -- To view, visit http://gerrit.cloudera.org:8080/13525 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I8995f5dc88b211cd3af415713802cfeac44fe576 Gerrit-Change-Number: 13525 Gerrit-PatchSet: 4 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon
[Impala-ASF-CR] IMPALA-8551: Bump CDP BUILD NUMBER to 1153860
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/13525 ) Change subject: IMPALA-8551: Bump CDP_BUILD_NUMBER to 1153860 .. Patch Set 4: (1 comment) http://gerrit.cloudera.org:8080/#/c/13525/2/tests/authorization/test_ranger.py File tests/authorization/test_ranger.py: http://gerrit.cloudera.org:8080/#/c/13525/2/tests/authorization/test_ranger.py@578 PS2, Line 578: ent in ["grant > hrm. I agree that parsing the error message isn't super robust, but I think I can easily prefix this with "Error granting a privilege to Ranger. Ranger error message: HTTP 403: ...". This way we don't have to worry if Ranger changes their error message and it's pretty clear that the error message came from Ranger. -- To view, visit http://gerrit.cloudera.org:8080/13525 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8995f5dc88b211cd3af415713802cfeac44fe576 Gerrit-Change-Number: 13525 Gerrit-PatchSet: 4 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Fri, 07 Jun 2019 21:16:16 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8638: Fix flaky TestLineage::test create table timestamp
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/13560 ) Change subject: IMPALA-8638: Fix flaky TestLineage::test_create_table_timestamp .. Patch Set 3: Code-Review+2 (1 comment) Carry Lars' +2. http://gerrit.cloudera.org:8080/#/c/13560/2//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/13560/2//COMMIT_MSG@12 PS2, Line 12: to u > missing word Done -- To view, visit http://gerrit.cloudera.org:8080/13560 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5813e4a570c181ba196b9ddf0210c8a0d92e21e8 Gerrit-Change-Number: 13560 Gerrit-PatchSet: 3 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Comment-Date: Fri, 07 Jun 2019 17:48:34 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8638: Fix flaky TestLineage::test create table timestamp
Fredy Wijaya has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/13560 ) Change subject: IMPALA-8638: Fix flaky TestLineage::test_create_table_timestamp .. IMPALA-8638: Fix flaky TestLineage::test_create_table_timestamp This patch fixes the bug TestLineage::test_create_table_timestamp where it uses the same lineage log directory created by TestLineage::test_start_end_timestamp, which could fail the query ID assertion. The fix is to use a different lineage log directory than the one used by TestLineage::test_start_end_timestamp. Testing: - Ran test_lineage.py multiple times and it still passed. Change-Id: I5813e4a570c181ba196b9ddf0210c8a0d92e21e8 --- M tests/custom_cluster/test_lineage.py 1 file changed, 11 insertions(+), 15 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/60/13560/3 -- To view, visit http://gerrit.cloudera.org:8080/13560 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I5813e4a570c181ba196b9ddf0210c8a0d92e21e8 Gerrit-Change-Number: 13560 Gerrit-PatchSet: 3 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker
[Impala-ASF-CR] IMPALA-8638: Fix flaky TestLineage::test create table timestamp
Fredy Wijaya has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13560 Change subject: IMPALA-8638: Fix flaky TestLineage::test_create_table_timestamp .. IMPALA-8638: Fix flaky TestLineage::test_create_table_timestamp This patch fixes the bug TestLineage::test_create_table_timestamp where it uses the same lineage log directory created by TestLineage::test_start_end_timestamp, which could fail the query ID assertion. The fix is to a different lineage log directory than the one used by TestLineage::test_start_end_timestamp. Testing: - Ran test_lineage.py multiple times and it still passed. Change-Id: I5813e4a570c181ba196b9ddf0210c8a0d92e21e8 --- M tests/custom_cluster/test_lineage.py 1 file changed, 11 insertions(+), 15 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/60/13560/2 -- To view, visit http://gerrit.cloudera.org:8080/13560 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I5813e4a570c181ba196b9ddf0210c8a0d92e21e8 Gerrit-Change-Number: 13560 Gerrit-PatchSet: 2 Gerrit-Owner: Fredy Wijaya
[Impala-ASF-CR] IMPALA-8551: Bump CDP BUILD NUMBER to 1153860
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/13525 ) Change subject: IMPALA-8551: Bump CDP_BUILD_NUMBER to 1153860 .. Patch Set 3: (2 comments) http://gerrit.cloudera.org:8080/#/c/13525/2/tests/authorization/test_ranger.py File tests/authorization/test_ranger.py: http://gerrit.cloudera.org:8080/#/c/13525/2/tests/authorization/test_ranger.py@566 PS2, Line 566: ctional.alltypes from user {0}" : .format(getuser())]: : result = self.execute_query_expect_failure(invalid_impala_cli > it's weird that you need two different clients, but you're passing the user This is a known issue. I filed a bug here: https://issues.apache.org/jira/browse/IMPALA-8640 http://gerrit.cloudera.org:8080/#/c/13525/2/tests/authorization/test_ranger.py@578 PS2, Line 578: > is there any way we can get rid of the HTTP error part of the error message We can but we have to parse the error message and that can be somewhat error prone. Ranger throws a generic exception, so we pretty much just rely on the error message returned by them. -- To view, visit http://gerrit.cloudera.org:8080/13525 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8995f5dc88b211cd3af415713802cfeac44fe576 Gerrit-Change-Number: 13525 Gerrit-PatchSet: 3 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Fri, 07 Jun 2019 16:30:15 + Gerrit-HasComments: Yes