[jira] [Commented] (SENTRY-2127) Fix unstable unit test TestColumnEndToEnd.testCrossDbTableOperations
[
https://issues.apache.org/jira/browse/SENTRY-2127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16339478#comment-16339478
]
Na Li commented on SENTRY-2127:
---
The actual problem is MapRedTask sometimes fail with more complicated join
statement. It usually passes when I ran it locally, but may fail when I ran it
in build machine. So it seems to be performance issue for map reduce. I have
checked log and all sentry authorization finished successfully. So this issue
is not related to sentry, and should not cause sentry unit test to fail.
> Fix unstable unit test TestColumnEndToEnd.testCrossDbTableOperations
>
>
> Key: SENTRY-2127
> URL: https://issues.apache.org/jira/browse/SENTRY-2127
> Project: Sentry
> Issue Type: Bug
> Components: Sentry
>Affects Versions: 2.1.0
>Reporter: Na Li
>Assignee: Na Li
>Priority: Major
> Attachments: SENTRY-2127.001.patch
>
>
> Occasionally, the test TestColumnEndToEnd.testCrossDbTableOperations fails at
> statement
> statement.execute("CREATE table db_1.t1 as select tb1.id, tb3.val, tb2.num
> from db_1.tb1,db_2.tb3,db_2.tb2");
> We should make it stable to avoid blocking other updates.
> More info:
> 1) Failed run with the following statement. The reason it fails is not
> related to sentry processing. It is due to the map reduce task fails with the
> more complicated join in the unit test
> {code:java}
> 2018-01-23 19:46:28,766 (5c4ce662-7587-40a5-8b24-b4c31f006b53
> HiveServer2-Handler-Pool: Thread-365) [INFO -
> org.apache.hadoop.hive.ql.Driver.compile(Driver.java:460)] Compiling
> command(queryId=jenkins_20180123194628_73ef2fc1-7819-4329-9d1c-96775dcc839c):
> CREATE table db_1.t1 as select tb1.id, tb3.val, tb2.num from
> db_1.tb1,db_2.tb3,db_2.tb2
> ...
> 2018-01-23 19:46:29,266 (HiveServer2-Background-Pool: Thread-373) [INFO -
> org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1713)] Executing
> command(queryId=jenkins_20180123194628_73ef2fc1-7819-4329-9d1c-96775dcc839c):
> CREATE table db_1.t1 as select tb1.id, tb3.val, tb2.num from
> db_1.tb1,db_2.tb3,db_2.tb2
> ...
> 2018-01-23 19:49:29,229 (HiveServer2-Background-Pool: Thread-373) [INFO -
> org.apache.hadoop.hive.ql.Driver.execute(Driver.java:2021)] Completed
> executing
> command(queryId=jenkins_20180123194628_73ef2fc1-7819-4329-9d1c-96775dcc839c);
> Time taken: 179.962 seconds
> 2018-01-23 19:49:29,235 (HiveServer2-Background-Pool: Thread-373) [ERROR -
> org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:345)]
> Error running hive query:
> org.apache.hive.service.cli.HiveSQLException: Error while processing
> statement: FAILED: Execution Error, return code 2 from
> org.apache.hadoop.hive.ql.exec.mr.MapRedTask
> at
> org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:330)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:254)
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$700(SQLOperation.java:91)
> at
> org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:342)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962)
> at
> org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:354)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> 2) Success run with following statement in unit test
> {code:java}
> statement.execute("CREATE table db_1.t2 as select tb1.id from db_1.tb1");
> 2018-01-24 10:29:01,071 (c042d8d8-423b-4ddf-a50b-63f1dd3e3f50
> HiveServer2-Handler-Pool: Thread-365) [INFO -
> org.apache.hadoop.hive.ql.Driver.compile(Driver.java:460)] Compiling
> command(queryId=jenkins_20180124102901_6376f6e1-0e74-4b05-b0a1-ab3ca5512e94):
> CREATE table db_1.t2 as select tb1.id from db_1.tb1
> 2018-01-24 10:29:01,513 (HiveServer2-Background-Pool: Thread-372) [INFO -
> org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1713)] Executing
> command(queryId=jenkins_20180124102901_6376f6e1-0e74-4b05-b0a1-ab3ca5512e94):
> CREATE table db_1.t2 as select tb1.id from db_1.tb1
> 2018-01-24 10:29:03,636 (HiveServer2-Background-Pool: Thread-372) [INFO -
> org.apache.hadoop.hive.ql.Driver.execute(Driver.java:2021)]
[jira] [Updated] (SENTRY-2127) Fix unstable unit test TestColumnEndToEnd.testCrossDbTableOperations
[
https://issues.apache.org/jira/browse/SENTRY-2127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Na Li updated SENTRY-2127:
--
Attachment: SENTRY-2127.001.patch
> Fix unstable unit test TestColumnEndToEnd.testCrossDbTableOperations
>
>
> Key: SENTRY-2127
> URL: https://issues.apache.org/jira/browse/SENTRY-2127
> Project: Sentry
> Issue Type: Bug
> Components: Sentry
>Affects Versions: 2.1.0
>Reporter: Na Li
>Assignee: Na Li
>Priority: Major
> Attachments: SENTRY-2127.001.patch
>
>
> Occasionally, the test TestColumnEndToEnd.testCrossDbTableOperations fails at
> statement
> statement.execute("CREATE table db_1.t1 as select tb1.id, tb3.val, tb2.num
> from db_1.tb1,db_2.tb3,db_2.tb2");
> We should make it stable to avoid blocking other updates.
> More info:
> 1) Failed run with the following statement. The reason it fails is not
> related to sentry processing. It is due to the map reduce task fails with the
> more complicated join in the unit test
> {code:java}
> 2018-01-23 19:46:28,766 (5c4ce662-7587-40a5-8b24-b4c31f006b53
> HiveServer2-Handler-Pool: Thread-365) [INFO -
> org.apache.hadoop.hive.ql.Driver.compile(Driver.java:460)] Compiling
> command(queryId=jenkins_20180123194628_73ef2fc1-7819-4329-9d1c-96775dcc839c):
> CREATE table db_1.t1 as select tb1.id, tb3.val, tb2.num from
> db_1.tb1,db_2.tb3,db_2.tb2
> ...
> 2018-01-23 19:46:29,266 (HiveServer2-Background-Pool: Thread-373) [INFO -
> org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1713)] Executing
> command(queryId=jenkins_20180123194628_73ef2fc1-7819-4329-9d1c-96775dcc839c):
> CREATE table db_1.t1 as select tb1.id, tb3.val, tb2.num from
> db_1.tb1,db_2.tb3,db_2.tb2
> ...
> 2018-01-23 19:49:29,229 (HiveServer2-Background-Pool: Thread-373) [INFO -
> org.apache.hadoop.hive.ql.Driver.execute(Driver.java:2021)] Completed
> executing
> command(queryId=jenkins_20180123194628_73ef2fc1-7819-4329-9d1c-96775dcc839c);
> Time taken: 179.962 seconds
> 2018-01-23 19:49:29,235 (HiveServer2-Background-Pool: Thread-373) [ERROR -
> org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:345)]
> Error running hive query:
> org.apache.hive.service.cli.HiveSQLException: Error while processing
> statement: FAILED: Execution Error, return code 2 from
> org.apache.hadoop.hive.ql.exec.mr.MapRedTask
> at
> org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:330)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:254)
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$700(SQLOperation.java:91)
> at
> org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:342)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962)
> at
> org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:354)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> 2) Success run with following statement in unit test
> {code:java}
> statement.execute("CREATE table db_1.t2 as select tb1.id from db_1.tb1");
> 2018-01-24 10:29:01,071 (c042d8d8-423b-4ddf-a50b-63f1dd3e3f50
> HiveServer2-Handler-Pool: Thread-365) [INFO -
> org.apache.hadoop.hive.ql.Driver.compile(Driver.java:460)] Compiling
> command(queryId=jenkins_20180124102901_6376f6e1-0e74-4b05-b0a1-ab3ca5512e94):
> CREATE table db_1.t2 as select tb1.id from db_1.tb1
> 2018-01-24 10:29:01,513 (HiveServer2-Background-Pool: Thread-372) [INFO -
> org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1713)] Executing
> command(queryId=jenkins_20180124102901_6376f6e1-0e74-4b05-b0a1-ab3ca5512e94):
> CREATE table db_1.t2 as select tb1.id from db_1.tb1
> 2018-01-24 10:29:03,636 (HiveServer2-Background-Pool: Thread-372) [INFO -
> org.apache.hadoop.hive.ql.Driver.execute(Driver.java:2021)] Completed
> executing
> command(queryId=jenkins_20180124102901_6376f6e1-0e74-4b05-b0a1-ab3ca5512e94);
> Time taken: 2.122 seconds
> OK{code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (SENTRY-2127) Fix unstable unit test TestColumnEndToEnd.testCrossDbTableOperations
[
https://issues.apache.org/jira/browse/SENTRY-2127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Na Li updated SENTRY-2127:
--
Status: Patch Available (was: Open)
> Fix unstable unit test TestColumnEndToEnd.testCrossDbTableOperations
>
>
> Key: SENTRY-2127
> URL: https://issues.apache.org/jira/browse/SENTRY-2127
> Project: Sentry
> Issue Type: Bug
> Components: Sentry
>Affects Versions: 2.1.0
>Reporter: Na Li
>Assignee: Na Li
>Priority: Major
> Attachments: SENTRY-2127.001.patch
>
>
> Occasionally, the test TestColumnEndToEnd.testCrossDbTableOperations fails at
> statement
> statement.execute("CREATE table db_1.t1 as select tb1.id, tb3.val, tb2.num
> from db_1.tb1,db_2.tb3,db_2.tb2");
> We should make it stable to avoid blocking other updates.
> More info:
> 1) Failed run with the following statement. The reason it fails is not
> related to sentry processing. It is due to the map reduce task fails with the
> more complicated join in the unit test
> {code:java}
> 2018-01-23 19:46:28,766 (5c4ce662-7587-40a5-8b24-b4c31f006b53
> HiveServer2-Handler-Pool: Thread-365) [INFO -
> org.apache.hadoop.hive.ql.Driver.compile(Driver.java:460)] Compiling
> command(queryId=jenkins_20180123194628_73ef2fc1-7819-4329-9d1c-96775dcc839c):
> CREATE table db_1.t1 as select tb1.id, tb3.val, tb2.num from
> db_1.tb1,db_2.tb3,db_2.tb2
> ...
> 2018-01-23 19:46:29,266 (HiveServer2-Background-Pool: Thread-373) [INFO -
> org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1713)] Executing
> command(queryId=jenkins_20180123194628_73ef2fc1-7819-4329-9d1c-96775dcc839c):
> CREATE table db_1.t1 as select tb1.id, tb3.val, tb2.num from
> db_1.tb1,db_2.tb3,db_2.tb2
> ...
> 2018-01-23 19:49:29,229 (HiveServer2-Background-Pool: Thread-373) [INFO -
> org.apache.hadoop.hive.ql.Driver.execute(Driver.java:2021)] Completed
> executing
> command(queryId=jenkins_20180123194628_73ef2fc1-7819-4329-9d1c-96775dcc839c);
> Time taken: 179.962 seconds
> 2018-01-23 19:49:29,235 (HiveServer2-Background-Pool: Thread-373) [ERROR -
> org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:345)]
> Error running hive query:
> org.apache.hive.service.cli.HiveSQLException: Error while processing
> statement: FAILED: Execution Error, return code 2 from
> org.apache.hadoop.hive.ql.exec.mr.MapRedTask
> at
> org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:330)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:254)
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$700(SQLOperation.java:91)
> at
> org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:342)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962)
> at
> org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:354)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> 2) Success run with following statement in unit test
> {code:java}
> statement.execute("CREATE table db_1.t2 as select tb1.id from db_1.tb1");
> 2018-01-24 10:29:01,071 (c042d8d8-423b-4ddf-a50b-63f1dd3e3f50
> HiveServer2-Handler-Pool: Thread-365) [INFO -
> org.apache.hadoop.hive.ql.Driver.compile(Driver.java:460)] Compiling
> command(queryId=jenkins_20180124102901_6376f6e1-0e74-4b05-b0a1-ab3ca5512e94):
> CREATE table db_1.t2 as select tb1.id from db_1.tb1
> 2018-01-24 10:29:01,513 (HiveServer2-Background-Pool: Thread-372) [INFO -
> org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1713)] Executing
> command(queryId=jenkins_20180124102901_6376f6e1-0e74-4b05-b0a1-ab3ca5512e94):
> CREATE table db_1.t2 as select tb1.id from db_1.tb1
> 2018-01-24 10:29:03,636 (HiveServer2-Background-Pool: Thread-372) [INFO -
> org.apache.hadoop.hive.ql.Driver.execute(Driver.java:2021)] Completed
> executing
> command(queryId=jenkins_20180124102901_6376f6e1-0e74-4b05-b0a1-ab3ca5512e94);
> Time taken: 2.122 seconds
> OK{code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (SENTRY-2127) Fix unstable unit test TestColumnEndToEnd.testCrossDbTableOperations
[
https://issues.apache.org/jira/browse/SENTRY-2127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Na Li updated SENTRY-2127:
--
Description:
Occasionally, the test TestColumnEndToEnd.testCrossDbTableOperations fails at
statement
statement.execute("CREATE table db_1.t1 as select tb1.id, tb3.val, tb2.num
from db_1.tb1,db_2.tb3,db_2.tb2");
We should make it stable to avoid blocking other updates.
More info:
1) Failed run with the following statement. The reason it fails is not related
to sentry processing. It is due to the map reduce task fails with the more
complicated join in the unit test
{code:java}
2018-01-23 19:46:28,766 (5c4ce662-7587-40a5-8b24-b4c31f006b53
HiveServer2-Handler-Pool: Thread-365) [INFO -
org.apache.hadoop.hive.ql.Driver.compile(Driver.java:460)] Compiling
command(queryId=jenkins_20180123194628_73ef2fc1-7819-4329-9d1c-96775dcc839c):
CREATE table db_1.t1 as select tb1.id, tb3.val, tb2.num from
db_1.tb1,db_2.tb3,db_2.tb2
...
2018-01-23 19:46:29,266 (HiveServer2-Background-Pool: Thread-373) [INFO -
org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1713)] Executing
command(queryId=jenkins_20180123194628_73ef2fc1-7819-4329-9d1c-96775dcc839c):
CREATE table db_1.t1 as select tb1.id, tb3.val, tb2.num from
db_1.tb1,db_2.tb3,db_2.tb2
...
2018-01-23 19:49:29,229 (HiveServer2-Background-Pool: Thread-373) [INFO -
org.apache.hadoop.hive.ql.Driver.execute(Driver.java:2021)] Completed executing
command(queryId=jenkins_20180123194628_73ef2fc1-7819-4329-9d1c-96775dcc839c);
Time taken: 179.962 seconds
2018-01-23 19:49:29,235 (HiveServer2-Background-Pool: Thread-373) [ERROR -
org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:345)]
Error running hive query:
org.apache.hive.service.cli.HiveSQLException: Error while processing statement:
FAILED: Execution Error, return code 2 from
org.apache.hadoop.hive.ql.exec.mr.MapRedTask
at
org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:330)
at
org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:254)
at
org.apache.hive.service.cli.operation.SQLOperation.access$700(SQLOperation.java:91)
at
org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:342)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962)
at
org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:354)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
{code}
2) Success run with following statement in unit test
{code:java}
statement.execute("CREATE table db_1.t2 as select tb1.id from db_1.tb1");
2018-01-24 10:29:01,071 (c042d8d8-423b-4ddf-a50b-63f1dd3e3f50
HiveServer2-Handler-Pool: Thread-365) [INFO -
org.apache.hadoop.hive.ql.Driver.compile(Driver.java:460)] Compiling
command(queryId=jenkins_20180124102901_6376f6e1-0e74-4b05-b0a1-ab3ca5512e94):
CREATE table db_1.t2 as select tb1.id from db_1.tb1
2018-01-24 10:29:01,513 (HiveServer2-Background-Pool: Thread-372) [INFO -
org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1713)] Executing
command(queryId=jenkins_20180124102901_6376f6e1-0e74-4b05-b0a1-ab3ca5512e94):
CREATE table db_1.t2 as select tb1.id from db_1.tb1
2018-01-24 10:29:03,636 (HiveServer2-Background-Pool: Thread-372) [INFO -
org.apache.hadoop.hive.ql.Driver.execute(Driver.java:2021)] Completed executing
command(queryId=jenkins_20180124102901_6376f6e1-0e74-4b05-b0a1-ab3ca5512e94);
Time taken: 2.122 seconds
OK{code}
was:
Occasionally, the test TestColumnEndToEnd.testCrossDbTableOperations fails at
statement
statement.execute("CREATE table db_1.t1 as select tb1.id, tb3.val, tb2.num from
db_1.tb1,db_2.tb3,db_2.tb2");
We should make it stable to avoid blocking other updates.
More info:
1) Failed run with the following statement. The reason it fails is not related
to sentry processing. It is due to the map reduce task fails with the more
complicated join in the unit test
{code:java}
2018-01-23 19:46:28,766 (5c4ce662-7587-40a5-8b24-b4c31f006b53
HiveServer2-Handler-Pool: Thread-365) [INFO -
org.apache.hadoop.hive.ql.Driver.compile(Driver.java:460)] Compiling
command(queryId=jenkins_20180123194628_73ef2fc1-7819-4329-9d1c-96775dcc839c):
CREATE table db_1.t1 as select tb1.id, tb3.val, tb2.num from
db_1.tb1,db_2.tb3,db_2.tb2
...
2018-01-23 19:46:29,266 (HiveServer2-Background-Pool: Thread-373)
[jira] [Created] (SENTRY-2127) Fix unstable unit test TestColumnEndToEnd.testCrossDbTableOperations
Na Li created SENTRY-2127:
-
Summary: Fix unstable unit test
TestColumnEndToEnd.testCrossDbTableOperations
Key: SENTRY-2127
URL: https://issues.apache.org/jira/browse/SENTRY-2127
Project: Sentry
Issue Type: Bug
Components: Sentry
Affects Versions: 2.1.0
Reporter: Na Li
Assignee: Na Li
Occasionally, the test TestColumnEndToEnd.testCrossDbTableOperations fails at
statement
statement.execute("CREATE table db_1.t1 as select tb1.id, tb3.val, tb2.num from
db_1.tb1,db_2.tb3,db_2.tb2");
We should make it stable to avoid blocking other updates.
More info:
1) Failed run with the following statement. The reason it fails is not related
to sentry processing. It is due to the map reduce task fails with the more
complicated join in the unit test
{code:java}
2018-01-23 19:46:28,766 (5c4ce662-7587-40a5-8b24-b4c31f006b53
HiveServer2-Handler-Pool: Thread-365) [INFO -
org.apache.hadoop.hive.ql.Driver.compile(Driver.java:460)] Compiling
command(queryId=jenkins_20180123194628_73ef2fc1-7819-4329-9d1c-96775dcc839c):
CREATE table db_1.t1 as select tb1.id, tb3.val, tb2.num from
db_1.tb1,db_2.tb3,db_2.tb2
...
2018-01-23 19:46:29,266 (HiveServer2-Background-Pool: Thread-373) [INFO -
org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1713)] Executing
command(queryId=jenkins_20180123194628_73ef2fc1-7819-4329-9d1c-96775dcc839c):
CREATE table db_1.t1 as select tb1.id, tb3.val, tb2.num from
db_1.tb1,db_2.tb3,db_2.tb2
...
2018-01-23 19:49:29,229 (HiveServer2-Background-Pool: Thread-373) [INFO -
org.apache.hadoop.hive.ql.Driver.execute(Driver.java:2021)] Completed executing
command(queryId=jenkins_20180123194628_73ef2fc1-7819-4329-9d1c-96775dcc839c);
Time taken: 179.962 seconds
2018-01-23 19:49:29,235 (HiveServer2-Background-Pool: Thread-373) [ERROR -
org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:345)]
Error running hive query:
org.apache.hive.service.cli.HiveSQLException: Error while processing statement:
FAILED: Execution Error, return code 2 from
org.apache.hadoop.hive.ql.exec.mr.MapRedTask
at
org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:330)
at
org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:254)
at
org.apache.hive.service.cli.operation.SQLOperation.access$700(SQLOperation.java:91)
at
org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:342)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962)
at
org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:354)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
{code}
2) Success run with following statement in unit test
statement.execute("CREATE table db_1.t2 as select tb1.id from db_1.tb1");
2018-01-24 10:29:01,071 (c042d8d8-423b-4ddf-a50b-63f1dd3e3f50
HiveServer2-Handler-Pool: Thread-365) [INFO -
org.apache.hadoop.hive.ql.Driver.compile(Driver.java:460)] Compiling
command(queryId=jenkins_20180124102901_6376f6e1-0e74-4b05-b0a1-ab3ca5512e94):
CREATE table db_1.t2 as select tb1.id from db_1.tb1
2018-01-24 10:29:01,513 (HiveServer2-Background-Pool: Thread-372) [INFO -
org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1713)] Executing
command(queryId=jenkins_20180124102901_6376f6e1-0e74-4b05-b0a1-ab3ca5512e94):
CREATE table db_1.t2 as select tb1.id from db_1.tb1
2018-01-24 10:29:03,636 (HiveServer2-Background-Pool: Thread-372) [INFO -
org.apache.hadoop.hive.ql.Driver.execute(Driver.java:2021)] Completed executing
command(queryId=jenkins_20180124102901_6376f6e1-0e74-4b05-b0a1-ab3ca5512e94);
Time taken: 2.122 seconds
OK
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (SENTRY-2091) User-based Privilege is broken by SENTRY-769
[
https://issues.apache.org/jira/browse/SENTRY-2091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Na Li updated SENTRY-2091:
--
Attachment: SENTRY-2091.006.patch
> User-based Privilege is broken by SENTRY-769
>
>
> Key: SENTRY-2091
> URL: https://issues.apache.org/jira/browse/SENTRY-2091
> Project: Sentry
> Issue Type: Bug
> Components: Sentry
>Affects Versions: 2.1.0
>Reporter: Na Li
>Assignee: Na Li
>Priority: Major
> Attachments: SENTRY-2091.001.patch, SENTRY-2091.002.patch,
> SENTRY-2091.003.patch, SENTRY-2091.004.patch, SENTRY-2091.004.patch,
> SENTRY-2091.006.patch
>
>
> SENTRY-769 throws exception when a user has no group. This breaks user-based
> privilege as the exception prevents getting privilege using user-based
> privilege.
> For example, in the following code
> {code}
> Set userPrivileges =
> authProvider.getPolicyEngine().getPrivileges(
> authProvider.getGroupMapping().getGroups(userName),
> Sets.newHashSet(userName),
> hiveAuthzBinding.getActiveRoleSet(),
> hiveAuthzBinding.getAuthServer());
> {code}
> when user has no group, the exception causes the processing stops even when
> user has privilege.
> The solution is to catch the exception, and continue the processing.
> {code}
> try {
> Set groups = null;
> try {
> groups = authProvider.getGroupMapping().getGroups(userName)
> } catch (SentryGroupNotFoundException ex) {
> log.debug(...);
> groups = new HashSet();
> }
> Set userPrivileges =
> authProvider.getPolicyEngine().getPrivileges(
> groups, Sets.newHashSet(userName),
> hiveAuthzBinding.getActiveRoleSet(),
> hiveAuthzBinding.getAuthServer());
> ...
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (SENTRY-1648) Sentry MySQL "Unique" Index
[
https://issues.apache.org/jira/browse/SENTRY-1648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Na Li reassigned SENTRY-1648:
-
Assignee: Na Li
> Sentry MySQL "Unique" Index
> ---
>
> Key: SENTRY-1648
> URL: https://issues.apache.org/jira/browse/SENTRY-1648
> Project: Sentry
> Issue Type: Improvement
>Affects Versions: 1.8.0
>Reporter: BELUGA BEHR
>Assignee: Na Li
>Priority: Minor
>
> {code:sql|title=sentry-mysql-1.8.0.sql}
> CREATE TABLE `SENTRY_DB_PRIVILEGE` (
> `DB_PRIVILEGE_ID` BIGINT NOT NULL,
> `PRIVILEGE_SCOPE` VARCHAR(32) CHARACTER SET utf8 COLLATE utf8_bin NOT NULL,
> `SERVER_NAME` VARCHAR(128) CHARACTER SET utf8 COLLATE utf8_bin NOT NULL,
> `DB_NAME` VARCHAR(128) CHARACTER SET utf8 COLLATE utf8_bin DEFAULT
> '__NULL__',
> `TABLE_NAME` VARCHAR(128) CHARACTER SET utf8 COLLATE utf8_bin DEFAULT
> '__NULL__',
> `COLUMN_NAME` VARCHAR(128) CHARACTER SET utf8 COLLATE utf8_bin DEFAULT
> '__NULL__',
> `URI` VARCHAR(4000) CHARACTER SET utf8 COLLATE utf8_bin DEFAULT '__NULL__',
> `ACTION` VARCHAR(128) CHARACTER SET utf8 COLLATE utf8_bin NOT NULL,
> `CREATE_TIME` BIGINT NOT NULL,
> `WITH_GRANT_OPTION` CHAR(1) NOT NULL
> ) ENGINE=InnoDB DEFAULT CHARSET=utf8;
> ALTER TABLE `SENTRY_DB_PRIVILEGE`
> ADD UNIQUE `SENTRY_DB_PRIV_PRIV_NAME_UNIQ`
> (`SERVER_NAME`,`DB_NAME`,`TABLE_NAME`,`COLUMN_NAME`,`URI`(250),`ACTION`,`WITH_GRANT_OPTION`);
> {code}
> As you can see, only the first 250 characters of URI is considered when
> determining "uniqueness". Typically, a second column would be added
> containing the hash value (MD5/SHA) of the URI and that column would instead
> be used as part of the unique index instead of the field itself.
> Oracle would also benefit from this.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (SENTRY-2123) Specify code path of auth-generated thrift files for Javadoc and exclude them from Javadoc generation
[
https://issues.apache.org/jira/browse/SENTRY-2123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Na Li updated SENTRY-2123:
--
Description:
Thrift source code is generated and put in its separate directory. Without
providing its path to Javadoc, Javadoc compilation of sentry-provider-db and
sentry-hdfs-common cannot find those auto-generated files.
{code:java}
package org.apache.sentry.hdfs.service.thrift does not exist import
org.apache.sentry.hdfs.service.thrift.TPathsDump;{code}
Also, Javadoc should not be geneated for those auto-generated code.
was:Thrift source code is generated and put in its separate directory. In
order for javadoc plugin to find it, we need to add configuration to specify
the path
> Specify code path of auth-generated thrift files for Javadoc and exclude them
> from Javadoc generation
> -
>
> Key: SENTRY-2123
> URL: https://issues.apache.org/jira/browse/SENTRY-2123
> Project: Sentry
> Issue Type: Bug
> Components: Sentry
>Affects Versions: 2.1.0
>Reporter: Na Li
>Assignee: Na Li
>Priority: Major
> Attachments: SENTRY-2123.001.patch, SENTRY-2123.002.patch
>
>
> Thrift source code is generated and put in its separate directory. Without
> providing its path to Javadoc, Javadoc compilation of sentry-provider-db and
> sentry-hdfs-common cannot find those auto-generated files.
> {code:java}
> package org.apache.sentry.hdfs.service.thrift does not exist import
> org.apache.sentry.hdfs.service.thrift.TPathsDump;{code}
> Also, Javadoc should not be geneated for those auto-generated code.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (SENTRY-2123) Specify code path of auth-generated thrift files for Javadoc and exclude them from Javadoc generation
[ https://issues.apache.org/jira/browse/SENTRY-2123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Na Li updated SENTRY-2123: -- Summary: Specify code path of auth-generated thrift files for Javadoc and exclude them from Javadoc generation (was: Specify Thrift source code path for Javadoc) > Specify code path of auth-generated thrift files for Javadoc and exclude them > from Javadoc generation > - > > Key: SENTRY-2123 > URL: https://issues.apache.org/jira/browse/SENTRY-2123 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.1.0 >Reporter: Na Li >Assignee: Na Li >Priority: Major > Attachments: SENTRY-2123.001.patch, SENTRY-2123.002.patch > > > Thrift source code is generated and put in its separate directory. In order > for javadoc plugin to find it, we need to add configuration to specify the > path -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (SENTRY-2123) Specify Thrift source code path for Javadoc
[ https://issues.apache.org/jira/browse/SENTRY-2123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Na Li updated SENTRY-2123: -- Attachment: SENTRY-2123.002.patch > Specify Thrift source code path for Javadoc > --- > > Key: SENTRY-2123 > URL: https://issues.apache.org/jira/browse/SENTRY-2123 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.1.0 >Reporter: Na Li >Assignee: Na Li >Priority: Major > Attachments: SENTRY-2123.001.patch, SENTRY-2123.002.patch > > > Thrift source code is generated and put in its separate directory. In order > for javadoc plugin to find it, we need to add configuration to specify the > path -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (SENTRY-2118) Document Configuration required to make Column authentication work
[ https://issues.apache.org/jira/browse/SENTRY-2118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Na Li closed SENTRY-2118. - > Document Configuration required to make Column authentication work > -- > > Key: SENTRY-2118 > URL: https://issues.apache.org/jira/browse/SENTRY-2118 > Project: Sentry > Issue Type: Task > Components: Sentry >Affects Versions: 2.1.0 >Reporter: Na Li >Assignee: Na Li >Priority: Major > Fix For: 2.1.0 > > > Sentry 2.0 and above is working with newer Hive version, and the > configuration > HiveConf.HIVE_STATS_COLLECT_SCANCOLS must be set true in order for Hive to > set column info into input for sentry. > Without this setting, authorization for column is broken. > We need to document this info. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (SENTRY-2118) Document Configuration required to make Column authentication work
[ https://issues.apache.org/jira/browse/SENTRY-2118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Na Li updated SENTRY-2118: -- Fix Version/s: 2.1.0 > Document Configuration required to make Column authentication work > -- > > Key: SENTRY-2118 > URL: https://issues.apache.org/jira/browse/SENTRY-2118 > Project: Sentry > Issue Type: Task > Components: Sentry >Affects Versions: 2.1.0 >Reporter: Na Li >Assignee: Na Li >Priority: Major > Fix For: 2.1.0 > > > Sentry 2.0 and above is working with newer Hive version, and the > configuration > HiveConf.HIVE_STATS_COLLECT_SCANCOLS must be set true in order for Hive to > set column info into input for sentry. > Without this setting, authorization for column is broken. > We need to document this info. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (SENTRY-2123) Specify Thrift source code path for Javadoc
[ https://issues.apache.org/jira/browse/SENTRY-2123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16330886#comment-16330886 ] Na Li commented on SENTRY-2123: --- Since HDFS-common and provider-db project import generated sentry thrift packages, javadoc plugin needs to know where to find those files. otherwise, the build for those projects may fail in javadoc. > Specify Thrift source code path for Javadoc > --- > > Key: SENTRY-2123 > URL: https://issues.apache.org/jira/browse/SENTRY-2123 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.1.0 >Reporter: Na Li >Assignee: Na Li >Priority: Major > Attachments: SENTRY-2123.001.patch > > > Thrift source code is generated and put in its separate directory. In order > for javadoc plugin to find it, we need to add configuration to specify the > path -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (SENTRY-2123) Specify Thrift source code path for Javadoc
[ https://issues.apache.org/jira/browse/SENTRY-2123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Na Li updated SENTRY-2123: -- Attachment: SENTRY-2123.001.patch > Specify Thrift source code path for Javadoc > --- > > Key: SENTRY-2123 > URL: https://issues.apache.org/jira/browse/SENTRY-2123 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.1.0 >Reporter: Na Li >Assignee: Na Li >Priority: Major > Attachments: SENTRY-2123.001.patch > > > Thrift source code is generated and put in its separate directory. In order > for javadoc plugin to find it, we need to add configuration to specify the > path -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (SENTRY-2123) Specify Thrift source code path for Javadoc
Na Li created SENTRY-2123: - Summary: Specify Thrift source code path for Javadoc Key: SENTRY-2123 URL: https://issues.apache.org/jira/browse/SENTRY-2123 Project: Sentry Issue Type: Bug Components: Sentry Affects Versions: 2.1.0 Reporter: Na Li Assignee: Na Li Thrift source code is generated and put in its separate directory. In order for javadoc plugin to find it, we need to add configuration to specify the path -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (SENTRY-2120) Escape input string for error response message in LogLevelServlet
[
https://issues.apache.org/jira/browse/SENTRY-2120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Na Li updated SENTRY-2120:
--
Summary: Escape input string for error response message in LogLevelServlet
(was: Potential cross-site scripting in LogLevelServlet)
> Escape input string for error response message in LogLevelServlet
> -
>
> Key: SENTRY-2120
> URL: https://issues.apache.org/jira/browse/SENTRY-2120
> Project: Sentry
> Issue Type: Bug
> Components: Sentry
>Affects Versions: 2.0.0, 2.1.0
>Reporter: Na Li
>Assignee: Na Li
> Attachments: SENTRY-2120.001.patch, SENTRY-2120.001.patch
>
>
> LogLevelServlet.java has the following code
> {code}
> public void doGet(HttpServletRequest request, HttpServletResponse response)
> throws ServletException, IOException {
> String logName = getParameter(request, "log");
> String level = getParameter(request, "level");
> response.setContentType("text/html;charset=utf-8");
> response.setStatus(HttpServletResponse.SC_OK);
> PrintWriter out = response.getWriter();
> if (logName != null) {
> Logger logInstance = LogManager.getLogger(logName);
> if (level == null) {
> out.write(String.format(FORMS_GET,
> escapeHtml(logName),
> logInstance.getEffectiveLevel().toString()));
> } else if (isLogLevelValid(level)) {
> logInstance.setLevel(Level.toLevel(level));
> out.write(String.format(FORMS_SET,
> escapeHtml(logName),
> level,
> level,
> logInstance.getEffectiveLevel().toString()));
> } else {
> response.sendError(HttpServletResponse.SC_BAD_REQUEST, "Invalid log
> level: " + level);
> return;
> }
> }
> out.write(FORMS_END);
> out.close();
> response.flushBuffer();
> }
> {code}
> As a result HTTP parameter is directly written to Servlet error page. Echoing
> this untrusted input to error message directly is a bad practice for security
> purpose. For best practice, we should escape the input string.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (SENTRY-2120) Potential cross-site scripting in LogLevelServlet
[
https://issues.apache.org/jira/browse/SENTRY-2120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Na Li updated SENTRY-2120:
--
Description:
LogLevelServlet.java has the following code
{code}
public void doGet(HttpServletRequest request, HttpServletResponse response)
throws ServletException, IOException {
String logName = getParameter(request, "log");
String level = getParameter(request, "level");
response.setContentType("text/html;charset=utf-8");
response.setStatus(HttpServletResponse.SC_OK);
PrintWriter out = response.getWriter();
if (logName != null) {
Logger logInstance = LogManager.getLogger(logName);
if (level == null) {
out.write(String.format(FORMS_GET,
escapeHtml(logName),
logInstance.getEffectiveLevel().toString()));
} else if (isLogLevelValid(level)) {
logInstance.setLevel(Level.toLevel(level));
out.write(String.format(FORMS_SET,
escapeHtml(logName),
level,
level,
logInstance.getEffectiveLevel().toString()));
} else {
response.sendError(HttpServletResponse.SC_BAD_REQUEST, "Invalid log
level: " + level);
return;
}
}
out.write(FORMS_END);
out.close();
response.flushBuffer();
}
{code}
As a result HTTP parameter is directly written to Servlet error page. Echoing
this untrusted input to error message directly is a bad practice for security
purpose. For best practice, we should escape the input string.
was:
LogLevelServlet.java has the following code
{code}
public void doGet(HttpServletRequest request, HttpServletResponse response)
throws ServletException, IOException {
String logName = getParameter(request, "log");
String level = getParameter(request, "level");
response.setContentType("text/html;charset=utf-8");
response.setStatus(HttpServletResponse.SC_OK);
PrintWriter out = response.getWriter();
if (logName != null) {
Logger logInstance = LogManager.getLogger(logName);
if (level == null) {
out.write(String.format(FORMS_GET,
escapeHtml(logName),
logInstance.getEffectiveLevel().toString()));
} else if (isLogLevelValid(level)) {
logInstance.setLevel(Level.toLevel(level));
out.write(String.format(FORMS_SET,
escapeHtml(logName),
level,
level,
logInstance.getEffectiveLevel().toString()));
} else {
response.sendError(HttpServletResponse.SC_BAD_REQUEST, "Invalid log
level: " + level);
return;
}
}
out.write(FORMS_END);
out.close();
response.flushBuffer();
}
{code}
As a result HTTP parameter is directly written to Servlet error page. Echoing
this untrusted input allows for a reflected cross site scripting. See
http://en.wikipedia.org/wiki/Cross-site_scripting for more information.
> Potential cross-site scripting in LogLevelServlet
> -
>
> Key: SENTRY-2120
> URL: https://issues.apache.org/jira/browse/SENTRY-2120
> Project: Sentry
> Issue Type: Bug
> Components: Sentry
>Affects Versions: 2.0.0, 2.1.0
>Reporter: Na Li
>Assignee: Na Li
> Attachments: SENTRY-2120.001.patch, SENTRY-2120.001.patch
>
>
> LogLevelServlet.java has the following code
> {code}
> public void doGet(HttpServletRequest request, HttpServletResponse response)
> throws ServletException, IOException {
> String logName = getParameter(request, "log");
> String level = getParameter(request, "level");
> response.setContentType("text/html;charset=utf-8");
> response.setStatus(HttpServletResponse.SC_OK);
> PrintWriter out = response.getWriter();
> if (logName != null) {
> Logger logInstance = LogManager.getLogger(logName);
> if (level == null) {
> out.write(String.format(FORMS_GET,
> escapeHtml(logName),
> logInstance.getEffectiveLevel().toString()));
> } else if (isLogLevelValid(level)) {
> logInstance.setLevel(Level.toLevel(level));
> out.write(String.format(FORMS_SET,
> escapeHtml(logName),
> level,
> level,
> logInstance.getEffectiveLevel().toString()));
> } else {
> response.sendError(HttpServletResponse.SC_BAD_REQUEST, "Invalid log
> level: " + level);
> return;
> }
> }
> out.write(FORMS_END);
> out.close();
> response.flushBuffer();
> }
> {code}
> As a result HTTP parameter is directly written to Servlet error page. Echoing
> this untrusted input to error message directly is a bad practice for security
> purpose. For best practice, we should escape the input string.
[jira] [Commented] (SENTRY-2119) HMSFollower may not fetch HMS notifications which are out of order
[ https://issues.apache.org/jira/browse/SENTRY-2119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16322583#comment-16322583 ] Na Li commented on SENTRY-2119: --- [~kkalyan] Is it possible for event_id to have permanent gap? If so, then there is no way to know the event_id that is missing is temporary or permanent. If it is temporary, we should get it with multiple retries. If it is permanent, we should move on. However, there is no way to tell if the gap is permanent or temporary. > HMSFollower may not fetch HMS notifications which are out of order > -- > > Key: SENTRY-2119 > URL: https://issues.apache.org/jira/browse/SENTRY-2119 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.1.0 >Reporter: kalyan kumar kalvagadda >Assignee: kalyan kumar kalvagadda > > With the current implementation of HMS and HMS-HA notifications inserted in > NOTIFICATION_LOG table can be out of order. That means, notifications with > smaller event-id's can be inserted into the table later and there is not > clear understanding on on the time difference between them. > When this happens HMSFollower will not be able to fetch these notifications. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (SENTRY-2121) Notifications processed during times when HDFS sync is disabled will not be applied as ACLs when later HDFS sync was to be enabled
[ https://issues.apache.org/jira/browse/SENTRY-2121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16322560#comment-16322560 ] Na Li commented on SENTRY-2121: --- "Another option is to always trigger a full snapshot every time HDFS sync is enabled" Sentry has no way to know HDFS is just enabled, or it is enabled since last restart of services. > Notifications processed during times when HDFS sync is disabled will not be > applied as ACLs when later HDFS sync was to be enabled > -- > > Key: SENTRY-2121 > URL: https://issues.apache.org/jira/browse/SENTRY-2121 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.1.0 >Reporter: Arjun Mishra >Assignee: Arjun Mishra > Labels: triaged > > When HDFS sync is disabled, we don't update the AUTHZ_PATHS_MAPPING table. > However, when HDFS sync is enabled, ACLs that are generated are based on > entries in the AUTHZ_PATHS_MAPPING table. So this means that if we were to > disable HDFS sync and later enable HDFS sync, ACLs for all notifications that > were processed during the times when HDFS sycn was disabled won't be created -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (SENTRY-2121) Notifications processed during times when HDFS sync is disabled will not be applied as ACLs when later HDFS sync was to be enabled
[
https://issues.apache.org/jira/browse/SENTRY-2121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16321458#comment-16321458
]
Na Li commented on SENTRY-2121:
---
To solve this issue, we have two options:
1) When HDFS is disabled, we clear the MAuthzPathsSnapshotId table. So when
HDFS is enabled again, the following condition will be met, and trigger getting
full snapshot. This approach has less overhead when HDFS is disabled, but
higher overhead at the moment when HDFS is enabled again (getting a full
snapshot)
{code}
// Once HDFS sync is enabled, and if MAuthzPathsSnapshotId
// table is still empty, we need to request a full snapshot
if(hdfsSyncEnabled && sentryStore.isAuthzPathsSnapshotEmpty()) {
LOGGER.debug("HDFSSync is enabled and MAuthzPathsSnapshotId table is
empty. Need to request a full snapshot");
return true;
}
{code}
2) When HDFS is disabled, still update MAuthzPathsMapping table. So when HDFS
is enabled again, there is no missing path info. This approach has higher
overhead when HDFS is disabled, but no extra overhead at the moment when HDFS
is enabled again.
> Notifications processed during times when HDFS sync is disabled will not be
> applied as ACLs when later HDFS sync was to be enabled
> --
>
> Key: SENTRY-2121
> URL: https://issues.apache.org/jira/browse/SENTRY-2121
> Project: Sentry
> Issue Type: Bug
> Components: Sentry
>Affects Versions: 2.1.0
>Reporter: Arjun Mishra
>Assignee: Arjun Mishra
> Labels: triaged
>
> When HDFS sync is disabled, we don't update the AUTHZ_PATHS_MAPPING table.
> However, when HDFS sync is enabled, ACLs that are generated are based on
> entries in the AUTHZ_PATHS_MAPPING table. So this means that if we were to
> disable HDFS sync and later enable HDFS sync, ACLs for all notifications that
> were processed during the times when HDFS sycn was disabled won't be created
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (SENTRY-2120) Potential cross-site scripting in LogLevelServlet
[
https://issues.apache.org/jira/browse/SENTRY-2120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Na Li updated SENTRY-2120:
--
Summary: Potential cross-site scripting in LogLevelServlet (was:
cross-site scripting vulnerability in LogLevelServlet)
> Potential cross-site scripting in LogLevelServlet
> -
>
> Key: SENTRY-2120
> URL: https://issues.apache.org/jira/browse/SENTRY-2120
> Project: Sentry
> Issue Type: Bug
> Components: Sentry
>Affects Versions: 2.0.0, 2.1.0
>Reporter: Na Li
>Assignee: Na Li
> Attachments: SENTRY-2120.001.patch, SENTRY-2120.001.patch
>
>
> LogLevelServlet.java has the following code
> {code}
> public void doGet(HttpServletRequest request, HttpServletResponse response)
> throws ServletException, IOException {
> String logName = getParameter(request, "log");
> String level = getParameter(request, "level");
> response.setContentType("text/html;charset=utf-8");
> response.setStatus(HttpServletResponse.SC_OK);
> PrintWriter out = response.getWriter();
> if (logName != null) {
> Logger logInstance = LogManager.getLogger(logName);
> if (level == null) {
> out.write(String.format(FORMS_GET,
> escapeHtml(logName),
> logInstance.getEffectiveLevel().toString()));
> } else if (isLogLevelValid(level)) {
> logInstance.setLevel(Level.toLevel(level));
> out.write(String.format(FORMS_SET,
> escapeHtml(logName),
> level,
> level,
> logInstance.getEffectiveLevel().toString()));
> } else {
> response.sendError(HttpServletResponse.SC_BAD_REQUEST, "Invalid log
> level: " + level);
> return;
> }
> }
> out.write(FORMS_END);
> out.close();
> response.flushBuffer();
> }
> {code}
> As a result HTTP parameter is directly written to Servlet error page. Echoing
> this untrusted input allows for a reflected cross site scripting
> vulnerability. See http://en.wikipedia.org/wiki/Cross-site_scripting for more
> information.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (SENTRY-2120) Potential cross-site scripting in LogLevelServlet
[
https://issues.apache.org/jira/browse/SENTRY-2120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Na Li updated SENTRY-2120:
--
Description:
LogLevelServlet.java has the following code
{code}
public void doGet(HttpServletRequest request, HttpServletResponse response)
throws ServletException, IOException {
String logName = getParameter(request, "log");
String level = getParameter(request, "level");
response.setContentType("text/html;charset=utf-8");
response.setStatus(HttpServletResponse.SC_OK);
PrintWriter out = response.getWriter();
if (logName != null) {
Logger logInstance = LogManager.getLogger(logName);
if (level == null) {
out.write(String.format(FORMS_GET,
escapeHtml(logName),
logInstance.getEffectiveLevel().toString()));
} else if (isLogLevelValid(level)) {
logInstance.setLevel(Level.toLevel(level));
out.write(String.format(FORMS_SET,
escapeHtml(logName),
level,
level,
logInstance.getEffectiveLevel().toString()));
} else {
response.sendError(HttpServletResponse.SC_BAD_REQUEST, "Invalid log
level: " + level);
return;
}
}
out.write(FORMS_END);
out.close();
response.flushBuffer();
}
{code}
As a result HTTP parameter is directly written to Servlet error page. Echoing
this untrusted input allows for a reflected cross site scripting. See
http://en.wikipedia.org/wiki/Cross-site_scripting for more information.
was:
LogLevelServlet.java has the following code
{code}
public void doGet(HttpServletRequest request, HttpServletResponse response)
throws ServletException, IOException {
String logName = getParameter(request, "log");
String level = getParameter(request, "level");
response.setContentType("text/html;charset=utf-8");
response.setStatus(HttpServletResponse.SC_OK);
PrintWriter out = response.getWriter();
if (logName != null) {
Logger logInstance = LogManager.getLogger(logName);
if (level == null) {
out.write(String.format(FORMS_GET,
escapeHtml(logName),
logInstance.getEffectiveLevel().toString()));
} else if (isLogLevelValid(level)) {
logInstance.setLevel(Level.toLevel(level));
out.write(String.format(FORMS_SET,
escapeHtml(logName),
level,
level,
logInstance.getEffectiveLevel().toString()));
} else {
response.sendError(HttpServletResponse.SC_BAD_REQUEST, "Invalid log
level: " + level);
return;
}
}
out.write(FORMS_END);
out.close();
response.flushBuffer();
}
{code}
As a result HTTP parameter is directly written to Servlet error page. Echoing
this untrusted input allows for a reflected cross site scripting vulnerability.
See http://en.wikipedia.org/wiki/Cross-site_scripting for more information.
> Potential cross-site scripting in LogLevelServlet
> -
>
> Key: SENTRY-2120
> URL: https://issues.apache.org/jira/browse/SENTRY-2120
> Project: Sentry
> Issue Type: Bug
> Components: Sentry
>Affects Versions: 2.0.0, 2.1.0
>Reporter: Na Li
>Assignee: Na Li
> Attachments: SENTRY-2120.001.patch, SENTRY-2120.001.patch
>
>
> LogLevelServlet.java has the following code
> {code}
> public void doGet(HttpServletRequest request, HttpServletResponse response)
> throws ServletException, IOException {
> String logName = getParameter(request, "log");
> String level = getParameter(request, "level");
> response.setContentType("text/html;charset=utf-8");
> response.setStatus(HttpServletResponse.SC_OK);
> PrintWriter out = response.getWriter();
> if (logName != null) {
> Logger logInstance = LogManager.getLogger(logName);
> if (level == null) {
> out.write(String.format(FORMS_GET,
> escapeHtml(logName),
> logInstance.getEffectiveLevel().toString()));
> } else if (isLogLevelValid(level)) {
> logInstance.setLevel(Level.toLevel(level));
> out.write(String.format(FORMS_SET,
> escapeHtml(logName),
> level,
> level,
> logInstance.getEffectiveLevel().toString()));
> } else {
> response.sendError(HttpServletResponse.SC_BAD_REQUEST, "Invalid log
> level: " + level);
> return;
> }
> }
> out.write(FORMS_END);
> out.close();
> response.flushBuffer();
> }
> {code}
> As a result HTTP parameter is directly written to Servlet error page. Echoing
> this untrusted input allows for a reflected cross site scripting. See
> http://en.wikipedia.org/wiki/Cross-site_scripting for more
[jira] [Updated] (SENTRY-2120) cross-site scripting vulnerability in LogLevelServlet
[
https://issues.apache.org/jira/browse/SENTRY-2120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Na Li updated SENTRY-2120:
--
Attachment: SENTRY-2120.001.patch
v1. attach again to trigger the test. the failed test is timeout, and it past
when I ran them locally.
> cross-site scripting vulnerability in LogLevelServlet
> -
>
> Key: SENTRY-2120
> URL: https://issues.apache.org/jira/browse/SENTRY-2120
> Project: Sentry
> Issue Type: Bug
> Components: Sentry
>Affects Versions: 2.0.0, 2.1.0
>Reporter: Na Li
>Assignee: Na Li
> Attachments: SENTRY-2120.001.patch, SENTRY-2120.001.patch
>
>
> LogLevelServlet.java has the following code
> {code}
> public void doGet(HttpServletRequest request, HttpServletResponse response)
> throws ServletException, IOException {
> String logName = getParameter(request, "log");
> String level = getParameter(request, "level");
> response.setContentType("text/html;charset=utf-8");
> response.setStatus(HttpServletResponse.SC_OK);
> PrintWriter out = response.getWriter();
> if (logName != null) {
> Logger logInstance = LogManager.getLogger(logName);
> if (level == null) {
> out.write(String.format(FORMS_GET,
> escapeHtml(logName),
> logInstance.getEffectiveLevel().toString()));
> } else if (isLogLevelValid(level)) {
> logInstance.setLevel(Level.toLevel(level));
> out.write(String.format(FORMS_SET,
> escapeHtml(logName),
> level,
> level,
> logInstance.getEffectiveLevel().toString()));
> } else {
> response.sendError(HttpServletResponse.SC_BAD_REQUEST, "Invalid log
> level: " + level);
> return;
> }
> }
> out.write(FORMS_END);
> out.close();
> response.flushBuffer();
> }
> {code}
> As a result HTTP parameter is directly written to Servlet error page. Echoing
> this untrusted input allows for a reflected cross site scripting
> vulnerability. See http://en.wikipedia.org/wiki/Cross-site_scripting for more
> information.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (SENTRY-2120) cross-site scripting vulnerability in LogLevelServlet
[
https://issues.apache.org/jira/browse/SENTRY-2120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Na Li updated SENTRY-2120:
--
Labels: Fortify-Critical (was: )
> cross-site scripting vulnerability in LogLevelServlet
> -
>
> Key: SENTRY-2120
> URL: https://issues.apache.org/jira/browse/SENTRY-2120
> Project: Sentry
> Issue Type: Bug
> Components: Sentry
>Affects Versions: 2.0.0, 2.1.0
>Reporter: Na Li
>Assignee: Na Li
> Labels: Fortify-Critical
> Attachments: SENTRY-2120.001.patch
>
>
> LogLevelServlet.java has the following code
> {code}
> public void doGet(HttpServletRequest request, HttpServletResponse response)
> throws ServletException, IOException {
> String logName = getParameter(request, "log");
> String level = getParameter(request, "level");
> response.setContentType("text/html;charset=utf-8");
> response.setStatus(HttpServletResponse.SC_OK);
> PrintWriter out = response.getWriter();
> if (logName != null) {
> Logger logInstance = LogManager.getLogger(logName);
> if (level == null) {
> out.write(String.format(FORMS_GET,
> escapeHtml(logName),
> logInstance.getEffectiveLevel().toString()));
> } else if (isLogLevelValid(level)) {
> logInstance.setLevel(Level.toLevel(level));
> out.write(String.format(FORMS_SET,
> escapeHtml(logName),
> level,
> level,
> logInstance.getEffectiveLevel().toString()));
> } else {
> response.sendError(HttpServletResponse.SC_BAD_REQUEST, "Invalid log
> level: " + level);
> return;
> }
> }
> out.write(FORMS_END);
> out.close();
> response.flushBuffer();
> }
> {code}
> As a result HTTP parameter is directly written to Servlet error page. Echoing
> this untrusted input allows for a reflected cross site scripting
> vulnerability. See http://en.wikipedia.org/wiki/Cross-site_scripting for more
> information.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (SENTRY-2120) cross-site scripting vulnerability in LogLevelServlet
[
https://issues.apache.org/jira/browse/SENTRY-2120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Na Li updated SENTRY-2120:
--
Attachment: SENTRY-2120.001.patch
> cross-site scripting vulnerability in LogLevelServlet
> -
>
> Key: SENTRY-2120
> URL: https://issues.apache.org/jira/browse/SENTRY-2120
> Project: Sentry
> Issue Type: Bug
> Components: Sentry
>Affects Versions: 2.0.0, 2.1.0
>Reporter: Na Li
>Assignee: Na Li
> Attachments: SENTRY-2120.001.patch
>
>
> LogLevelServlet.java has the following code
> {code}
> public void doGet(HttpServletRequest request, HttpServletResponse response)
> throws ServletException, IOException {
> String logName = getParameter(request, "log");
> String level = getParameter(request, "level");
> response.setContentType("text/html;charset=utf-8");
> response.setStatus(HttpServletResponse.SC_OK);
> PrintWriter out = response.getWriter();
> if (logName != null) {
> Logger logInstance = LogManager.getLogger(logName);
> if (level == null) {
> out.write(String.format(FORMS_GET,
> escapeHtml(logName),
> logInstance.getEffectiveLevel().toString()));
> } else if (isLogLevelValid(level)) {
> logInstance.setLevel(Level.toLevel(level));
> out.write(String.format(FORMS_SET,
> escapeHtml(logName),
> level,
> level,
> logInstance.getEffectiveLevel().toString()));
> } else {
> response.sendError(HttpServletResponse.SC_BAD_REQUEST, "Invalid log
> level: " + level);
> return;
> }
> }
> out.write(FORMS_END);
> out.close();
> response.flushBuffer();
> }
> {code}
> As a result HTTP parameter is directly written to Servlet error page. Echoing
> this untrusted input allows for a reflected cross site scripting
> vulnerability. See http://en.wikipedia.org/wiki/Cross-site_scripting for more
> information.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (SENTRY-2120) cross-site scripting vulnerability in LogLevelServlet
[
https://issues.apache.org/jira/browse/SENTRY-2120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Na Li updated SENTRY-2120:
--
Status: Patch Available (was: Open)
> cross-site scripting vulnerability in LogLevelServlet
> -
>
> Key: SENTRY-2120
> URL: https://issues.apache.org/jira/browse/SENTRY-2120
> Project: Sentry
> Issue Type: Bug
> Components: Sentry
>Affects Versions: 2.0.0, 2.1.0
>Reporter: Na Li
>Assignee: Na Li
> Attachments: SENTRY-2120.001.patch
>
>
> LogLevelServlet.java has the following code
> {code}
> public void doGet(HttpServletRequest request, HttpServletResponse response)
> throws ServletException, IOException {
> String logName = getParameter(request, "log");
> String level = getParameter(request, "level");
> response.setContentType("text/html;charset=utf-8");
> response.setStatus(HttpServletResponse.SC_OK);
> PrintWriter out = response.getWriter();
> if (logName != null) {
> Logger logInstance = LogManager.getLogger(logName);
> if (level == null) {
> out.write(String.format(FORMS_GET,
> escapeHtml(logName),
> logInstance.getEffectiveLevel().toString()));
> } else if (isLogLevelValid(level)) {
> logInstance.setLevel(Level.toLevel(level));
> out.write(String.format(FORMS_SET,
> escapeHtml(logName),
> level,
> level,
> logInstance.getEffectiveLevel().toString()));
> } else {
> response.sendError(HttpServletResponse.SC_BAD_REQUEST, "Invalid log
> level: " + level);
> return;
> }
> }
> out.write(FORMS_END);
> out.close();
> response.flushBuffer();
> }
> {code}
> As a result HTTP parameter is directly written to Servlet error page. Echoing
> this untrusted input allows for a reflected cross site scripting
> vulnerability. See http://en.wikipedia.org/wiki/Cross-site_scripting for more
> information.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (SENTRY-2120) cross-site scripting vulnerability in LogLevelServlet
Na Li created SENTRY-2120:
-
Summary: cross-site scripting vulnerability in LogLevelServlet
Key: SENTRY-2120
URL: https://issues.apache.org/jira/browse/SENTRY-2120
Project: Sentry
Issue Type: Bug
Components: Sentry
Affects Versions: 2.0.0, 2.1.0
Reporter: Na Li
Assignee: Na Li
LogLevelServlet.java has the following code
{code}
public void doGet(HttpServletRequest request, HttpServletResponse response)
throws ServletException, IOException {
String logName = getParameter(request, "log");
String level = getParameter(request, "level");
response.setContentType("text/html;charset=utf-8");
response.setStatus(HttpServletResponse.SC_OK);
PrintWriter out = response.getWriter();
if (logName != null) {
Logger logInstance = LogManager.getLogger(logName);
if (level == null) {
out.write(String.format(FORMS_GET,
escapeHtml(logName),
logInstance.getEffectiveLevel().toString()));
} else if (isLogLevelValid(level)) {
logInstance.setLevel(Level.toLevel(level));
out.write(String.format(FORMS_SET,
escapeHtml(logName),
level,
level,
logInstance.getEffectiveLevel().toString()));
} else {
response.sendError(HttpServletResponse.SC_BAD_REQUEST, "Invalid log
level: " + level);
return;
}
}
out.write(FORMS_END);
out.close();
response.flushBuffer();
}
{code}
As a result HTTP parameter is directly written to Servlet error page. Echoing
this untrusted input allows for a reflected cross site scripting vulnerability.
See http://en.wikipedia.org/wiki/Cross-site_scripting for more information.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Resolved] (SENTRY-2118) Document Configuration required to make Column authentication work
[ https://issues.apache.org/jira/browse/SENTRY-2118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Na Li resolved SENTRY-2118. --- Resolution: Fixed The description is at https://cwiki.apache.org/confluence/display/SENTRY/Sentry+Column+Access+Control+Configuration > Document Configuration required to make Column authentication work > -- > > Key: SENTRY-2118 > URL: https://issues.apache.org/jira/browse/SENTRY-2118 > Project: Sentry > Issue Type: Task > Components: Sentry >Affects Versions: 2.1.0 >Reporter: Na Li >Assignee: Na Li > > Sentry 2.0 and above is working with newer Hive version, and the > configuration > HiveConf.HIVE_STATS_COLLECT_SCANCOLS must be set true in order for Hive to > set column info into input for sentry. > Without this setting, authorization for column is broken. > We need to document this info. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (SENTRY-2118) Document Configuration required to make Column authentication work
[ https://issues.apache.org/jira/browse/SENTRY-2118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16313493#comment-16313493 ] Na Li commented on SENTRY-2118: --- [~coheig] has found this issue, and it is fixed by adding this configuration. > Document Configuration required to make Column authentication work > -- > > Key: SENTRY-2118 > URL: https://issues.apache.org/jira/browse/SENTRY-2118 > Project: Sentry > Issue Type: Task > Components: Sentry >Affects Versions: 2.1.0 >Reporter: Na Li >Assignee: Na Li > > Sentry 2.0 and above is working with newer Hive version, and the > configuration > HiveConf.HIVE_STATS_COLLECT_SCANCOLS must be set true in order for Hive to > set column info into input for sentry. > Without this setting, authorization for column is broken. > We need to document this info. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (SENTRY-2118) Document Configuration required to make Column authentication work
Na Li created SENTRY-2118: - Summary: Document Configuration required to make Column authentication work Key: SENTRY-2118 URL: https://issues.apache.org/jira/browse/SENTRY-2118 Project: Sentry Issue Type: Task Components: Sentry Affects Versions: 2.1.0 Reporter: Na Li Assignee: Na Li Sentry 2.0 and above is working with newer Hive version, and the configuration HiveConf.HIVE_STATS_COLLECT_SCANCOLS must be set true in order for Hive to set column info into input for sentry. Without this setting, authorization for column is broken. We need to document this info. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (SENTRY-2109) Fix the logic of identifying HMS out of Sync
[ https://issues.apache.org/jira/browse/SENTRY-2109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16311884#comment-16311884 ] Na Li commented on SENTRY-2109: --- [~kkalyan] the purging uses ID - 100. It does not guarantee that we have 100 entries in the Notification ID table. If there is a gap as big as 100, we may only have 1 entry in the table. > Fix the logic of identifying HMS out of Sync > > > Key: SENTRY-2109 > URL: https://issues.apache.org/jira/browse/SENTRY-2109 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.1.0 >Reporter: kalyan kumar kalvagadda >Assignee: kalyan kumar kalvagadda > Fix For: 2.1.0 > > Attachments: SENTRY-2109.001.patch, SENTRY-2109.002.patch, > SENTRY-2109.003.patch, Screenshot_HMS_NOTIFICATION_LOG.png > > > Currently HMSFollower proactively checks if sentry is out of sync with HMS > and initiates full snapshot, if needed. > There will be false positives with the current logic if there are gaps in the > event-id in the notification log sequence. > This jira is aimed at making that logic robust. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (SENTRY-2114) Fix behavior of HMSFollower when HDFS sync is not enabled
[ https://issues.apache.org/jira/browse/SENTRY-2114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16311633#comment-16311633 ] Na Li commented on SENTRY-2114: --- We should process notifications just to update the ID and wake up the waiter. So Hive is not blocked on waiting sentry sync with HMS. So no need to get get full snapshot. > Fix behavior of HMSFollower when HDFS sync is not enabled > - > > Key: SENTRY-2114 > URL: https://issues.apache.org/jira/browse/SENTRY-2114 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.1.0 >Reporter: kalyan kumar kalvagadda >Assignee: kalyan kumar kalvagadda > > Currently behavior of HMSFollower when HDFS sync disabled is as below. > # Take full snapshot of HMS > # Take the Notification Id from the snapshot created. > # Persist the notification Id to SENTRY_HMS_NOTIFICATION_ID table > # Fetches notifications and process new notifications after that point. > *This doesn't make sense. Ideally, I think it should work as below.* > # Initially, fetch all the notifications with even-id > 0 and process them. > # Subsequently fetch and process new notifications. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (SENTRY-2114) Fix behavior of HMSFollower when HDFS sync is not enabled
[ https://issues.apache.org/jira/browse/SENTRY-2114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16310252#comment-16310252 ] Na Li commented on SENTRY-2114: --- How to handle the following case correctly? 1. HDFS is disabled 2. HDFS is enabled after some time. The notification indicates many notifications are processed, but they are not in the change log. A full snapshot should be obtained. I suggest to get all scenarios analyzed, decide the behavior that work correctly in all scenarios, and then implement the approach in each scenario. Without this coordination, we can fix one issue and then creating another issue. > Fix behavior of HMSFollower when HDFS sync is not enabled > - > > Key: SENTRY-2114 > URL: https://issues.apache.org/jira/browse/SENTRY-2114 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.1.0 >Reporter: kalyan kumar kalvagadda >Assignee: kalyan kumar kalvagadda > > Currently behavior of HMSFollower when HDFS sync disabled is as below. > # Take full snapshot of HMS > # Take the Notification Id from the snapshot created. > # Persist the notification Id to SENTRY_HMS_NOTIFICATION_ID table > # Fetches notifications and process new notifications after that point. > *This doesn't make sense. Ideally, I think it should work as below.* > # Initially, fetch all the notifications with even-id > 0 and process them. > # Subsequently fetch and process new notifications. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (SENTRY-2114) Fix behavior of HMSFollower when HDFS sync is not enabled
[ https://issues.apache.org/jira/browse/SENTRY-2114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16310249#comment-16310249 ] Na Li commented on SENTRY-2114: --- The default value of sentry last notification ID should be 0. So the step 1 and step 2 can be combined into a single step "fetch and process new notifications > sentry last processed notification ID" > Fix behavior of HMSFollower when HDFS sync is not enabled > - > > Key: SENTRY-2114 > URL: https://issues.apache.org/jira/browse/SENTRY-2114 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.1.0 >Reporter: kalyan kumar kalvagadda >Assignee: kalyan kumar kalvagadda > > Currently behavior of HMSFollower when HDFS sync disabled is as below. > # Take full snapshot of HMS > # Take the Notification Id from the snapshot created. > # Persist the notification Id to SENTRY_HMS_NOTIFICATION_ID table > # Fetches notifications and process new notifications after that point. > *This doesn't make sense. Ideally, I think it should work as below.* > # Initially, fetch all the notifications with even-id > 0 and process them. > # Subsequently fetch and process new notifications. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (SENTRY-2109) Fix the logic of identifying HMS out of Sync
[ https://issues.apache.org/jira/browse/SENTRY-2109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16308834#comment-16308834 ] Na Li commented on SENTRY-2109: --- [~kkalyan] Sentry service periodically calls purgeNotificationIdTable. Have you take this into account in your update? Since there is a gap and the Notification ID could decrease, the logic for getting notification events should be updated to too to make sure sentry gets all notifications. Otherwise, sentry may miss some notifications and will be inconsistent with HMS updates. > Fix the logic of identifying HMS out of Sync > > > Key: SENTRY-2109 > URL: https://issues.apache.org/jira/browse/SENTRY-2109 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.1.0 >Reporter: kalyan kumar kalvagadda >Assignee: kalyan kumar kalvagadda > Fix For: 2.1.0 > > Attachments: SENTRY-2109.001.patch, SENTRY-2109.002.patch, > SENTRY-2109.003.patch, Screenshot_HMS_NOTIFICATION_LOG.png > > > Currently HMSFollower proactively checks if sentry is out of sync with HMS > and initiates full snapshot, if needed. > There will be false positives with the current logic if there are gaps in the > event-id in the notification log sequence. > This jira is aimed at making that logic robust. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (SENTRY-2091) User-based Privilege is broken by SENTRY-769
[
https://issues.apache.org/jira/browse/SENTRY-2091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Na Li updated SENTRY-2091:
--
Attachment: SENTRY-2091.004.patch
v4 again to trigger the test
> User-based Privilege is broken by SENTRY-769
>
>
> Key: SENTRY-2091
> URL: https://issues.apache.org/jira/browse/SENTRY-2091
> Project: Sentry
> Issue Type: Bug
> Components: Sentry
>Affects Versions: 2.1.0
>Reporter: Na Li
>Assignee: Na Li
> Attachments: SENTRY-2091.001.patch, SENTRY-2091.002.patch,
> SENTRY-2091.003.patch, SENTRY-2091.004.patch, SENTRY-2091.004.patch
>
>
> SENTRY-769 throws exception when a user has no group. This breaks user-based
> privilege as the exception prevents getting privilege using user-based
> privilege.
> For example, in the following code
> {code}
> Set userPrivileges =
> authProvider.getPolicyEngine().getPrivileges(
> authProvider.getGroupMapping().getGroups(userName),
> Sets.newHashSet(userName),
> hiveAuthzBinding.getActiveRoleSet(),
> hiveAuthzBinding.getAuthServer());
> {code}
> when user has no group, the exception causes the processing stops even when
> user has privilege.
> The solution is to catch the exception, and continue the processing.
> {code}
> try {
> Set groups = null;
> try {
> groups = authProvider.getGroupMapping().getGroups(userName)
> } catch (SentryGroupNotFoundException ex) {
> log.debug(...);
> groups = new HashSet();
> }
> Set userPrivileges =
> authProvider.getPolicyEngine().getPrivileges(
> groups, Sets.newHashSet(userName),
> hiveAuthzBinding.getActiveRoleSet(),
> hiveAuthzBinding.getAuthServer());
> ...
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (SENTRY-2091) User-based Privilege is broken by SENTRY-769
[
https://issues.apache.org/jira/browse/SENTRY-2091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Na Li updated SENTRY-2091:
--
Attachment: SENTRY-2091.004.patch
v4. fix more hive test failure
> User-based Privilege is broken by SENTRY-769
>
>
> Key: SENTRY-2091
> URL: https://issues.apache.org/jira/browse/SENTRY-2091
> Project: Sentry
> Issue Type: Bug
> Components: Sentry
>Affects Versions: 2.1.0
>Reporter: Na Li
>Assignee: Na Li
> Attachments: SENTRY-2091.001.patch, SENTRY-2091.002.patch,
> SENTRY-2091.003.patch, SENTRY-2091.004.patch
>
>
> SENTRY-769 throws exception when a user has no group. This breaks user-based
> privilege as the exception prevents getting privilege using user-based
> privilege.
> For example, in the following code
> {code}
> Set userPrivileges =
> authProvider.getPolicyEngine().getPrivileges(
> authProvider.getGroupMapping().getGroups(userName),
> Sets.newHashSet(userName),
> hiveAuthzBinding.getActiveRoleSet(),
> hiveAuthzBinding.getAuthServer());
> {code}
> when user has no group, the exception causes the processing stops even when
> user has privilege.
> The solution is to catch the exception, and continue the processing.
> {code}
> try {
> Set groups = null;
> try {
> groups = authProvider.getGroupMapping().getGroups(userName)
> } catch (SentryGroupNotFoundException ex) {
> log.debug(...);
> groups = new HashSet();
> }
> Set userPrivileges =
> authProvider.getPolicyEngine().getPrivileges(
> groups, Sets.newHashSet(userName),
> hiveAuthzBinding.getActiveRoleSet(),
> hiveAuthzBinding.getAuthServer());
> ...
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (SENTRY-2096) Fail unit tests at end
[
https://issues.apache.org/jira/browse/SENTRY-2096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16291392#comment-16291392
]
Na Li commented on SENTRY-2096:
---
[~kkalyan] [~spena] Based on my testing results, we should use *--fail-at-end*.
1) with command "mvn verify --fail-at-end", all tests were run and the build is
marked as failure due to some failed test cases. This is desired behavior
{code}
[INFO]
[INFO] Reactor Summary:
[INFO]
[INFO] Sentry . SUCCESS [ 1.962 s]
[INFO] Sentry Core SUCCESS [ 0.123 s]
[INFO] Sentry Core Common . SUCCESS [ 6.822 s]
[INFO] Sentry Core Model DB ... SUCCESS [ 1.551 s]
[INFO] Sentry Core Model Indexer .. SUCCESS [ 1.086 s]
[INFO] Sentry Core Model Solr . SUCCESS [ 1.579 s]
[INFO] Sentry Core Model Sqoop SUCCESS [ 1.673 s]
[INFO] Sentry Core Model Kafka SUCCESS [ 1.000 s]
[INFO] Sentry Policies SUCCESS [ 0.066 s]
[INFO] Sentry Policy Common ... SUCCESS [ 1.640 s]
[INFO] Sentry Providers ... SUCCESS [ 0.136 s]
[INFO] Sentry Provider Common . SUCCESS [ 1.146 s]
[INFO] Sentry Provider File ... SUCCESS [ 1.719 s]
[INFO] Sentry Policy Engine ... SUCCESS [ 0.303 s]
[INFO] Sentry Bindings SUCCESS [ 0.094 s]
[INFO] Sentry Hive Binding Configuration .. SUCCESS [ 3.029 s]
[INFO] Hive follower for Sentry ... SUCCESS [ 3.789 s]
[INFO] Sentry HDFS SUCCESS [ 0.122 s]
[INFO] Sentry HDFS Common . SUCCESS [ 14.834 s]
[INFO] Sentry Provider DB . SUCCESS [09:36 min]
[INFO] Sentry Binding for Kafka ... SUCCESS [ 8.526 s]
[INFO] Sentry Provider Cache .. SUCCESS [ 1.337 s]
[INFO] Sentry Hive Binding Common . SUCCESS [ 1.586 s]
[INFO] Sentry Binding for Solr SUCCESS [ 14.299 s]
[INFO] Sentry Binding for Sqoop ... SUCCESS [ 6.420 s]
[INFO] Sentry Binding for Hive SUCCESS [ 15.991 s]
[INFO] Sentry Policy for Indexer .. SUCCESS [ 5.829 s]
[INFO] Sentry Solr SUCCESS [ 0.069 s]
[INFO] Solr Sentry handler SUCCESS [ 3.737 s]
[INFO] Sentry Tests ... SUCCESS [ 0.148 s]
[INFO] Sentry Solr Tests .. SUCCESS [01:52 min]
[INFO] Sentry Sqoop Tests . SUCCESS [01:43 min]
[INFO] Sentry Kafka Tests . SUCCESS [03:27 min]
[INFO] Sentry HDFS Service SUCCESS [06:55 min]
[INFO] Sentry HDFS Namenode Plugin SUCCESS [ 5.963 s]
[INFO] Sentry Hive Tests .. FAILURE [ 1.876 s]
[INFO] Sentry HDFS Dist ... SUCCESS [ 2.006 s]
[INFO] sentry-tools ... SUCCESS [ 1.592 s]
[INFO] Sentry Distribution SUCCESS [ 25.775 s]
[INFO]
[INFO] BUILD FAILURE
[INFO]
{code}
2) With command "mvn verify -Dmaven.test.failure.ignore=true", all tests were
run and the build was marked as success even with some failed tests. It could
mislead people to believe the code change does not break unit tests, but in
fact it does.
{code}
[INFO]
[INFO] Reactor Summary:
[INFO]
[INFO] Sentry . SUCCESS [ 12.537 s]
[INFO] Sentry Core SUCCESS [ 0.163 s]
[INFO] Sentry Core Common . SUCCESS [ 13.896 s]
[INFO] Sentry Core Model DB ... SUCCESS [ 3.708 s]
[INFO] Sentry Core Model Indexer .. SUCCESS [ 1.449 s]
[INFO] Sentry Core Model Solr . SUCCESS [ 1.106 s]
[INFO] Sentry Core Model Sqoop SUCCESS [ 1.428 s]
[INFO] Sentry Core Model Kafka SUCCESS [ 1.405 s]
[INFO] Sentry Policies SUCCESS [ 0.089 s]
[INFO] Sentry Policy
[jira] [Commented] (SENTRY-1753) Make HMSFollower initial delay configurable
[ https://issues.apache.org/jira/browse/SENTRY-1753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16289715#comment-16289715 ] Na Li commented on SENTRY-1753: --- and the solution is very simple. > Make HMSFollower initial delay configurable > --- > > Key: SENTRY-1753 > URL: https://issues.apache.org/jira/browse/SENTRY-1753 > Project: Sentry > Issue Type: Task > Components: Sentry >Affects Versions: 2.0.0 >Reporter: Na Li >Assignee: Liam Sargent >Priority: Minor > > HMSFollower initial delay affects how fast it can get notification from Hive > meta store. We are changing it from 60 seconds to 0 second. It may cause some > issue. It is better to make this value configurable, so we can adjust it > based on testing result. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (SENTRY-2100) Create user related ACL at Name Node
Na Li created SENTRY-2100: - Summary: Create user related ACL at Name Node Key: SENTRY-2100 URL: https://issues.apache.org/jira/browse/SENTRY-2100 Project: Sentry Issue Type: Sub-task Reporter: Na Li at Name Node, Sentry only create ACL for group with ACL type as AclEntryType.GROUP. Need to add code to create ACL with type as AclEntryType.USER based on user to role updates -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (SENTRY-2099) Add user to role mapping in thrift response for permission updates
Na Li created SENTRY-2099: - Summary: Add user to role mapping in thrift response for permission updates Key: SENTRY-2099 URL: https://issues.apache.org/jira/browse/SENTRY-2099 Project: Sentry Issue Type: Sub-task Reporter: Na Li The user changes to a role is not included when sending delta update from Sentry to NN. Need to add AddUsers and DropUsers in TRoleChanges -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (SENTRY-1753) Make HMSFollower initial delay configurable
[
https://issues.apache.org/jira/browse/SENTRY-1753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16289606#comment-16289606
]
Na Li edited comment on SENTRY-1753 at 12/13/17 5:42 PM:
-
[~liamsargent] the default value right now is 0
"SENTRY_HMSFOLLOWER_INIT_DELAY_MILLS_DEFAULT = 0;". It would be nice to measure
how long HMS service can start, and then set the initial delay around that
value. So we can avoid a lot of exception in sentry log
{code}
2017-12-11 10:50:00,779 ERROR org.apache.sentry.service.thrift.HMSFollower:
HMSFollower cannot connect to HMS!!
java.lang.reflect.UndeclaredThrowableException
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1980)
at
org.apache.sentry.service.thrift.HiveSimpleConnectionFactory.connect(HiveSimpleConnectionFactory.java:114)
at
org.apache.sentry.service.thrift.SentryHMSClient.connect(SentryHMSClient.java:108)
at
org.apache.sentry.service.thrift.HMSFollower.syncupWithHms(HMSFollower.java:158)
{code}
We want to minimize those exception before HMS service is up and Sentry can
connect to it by {color:red}setting SENTRY_HMSFOLLOWER_INIT_DELAY_MILLS_DEFAULT
a little lower than the time HMS service takes to start{color}. Setting this
value too large will unnecessarily delay sentry service.
Basically, such exception should be ignored before the following message shows
up in sentry log.
{code}
2017-12-01 05:50:03,672 INFO org.apache.sentry.service.thrift.HMSFollower:
Sentry HMS support is ready
{code}
For example, it takes about 5 minutes for HMS service to start, and sentry
service is able to connect to HMS service in one cluster
{code}
2017-12-11 10:49:53
2017-12-11 10:55:18,745 INFO hive.metastore: Connected to metastore.
2017-12-11 10:55:18,832 INFO org.apache.sentry.service.thrift.SentryHMSClient:
Request full HMS snapshot
2017-12-11 10:55:19,147 INFO org.apache.sentry.service.thrift.HMSFollower:
Sentry HMS support is ready
{code}
was (Author: linaataustin):
[~liamsargent] the default value right now is 0
"SENTRY_HMSFOLLOWER_INIT_DELAY_MILLS_DEFAULT = 0;". It would be nice to measure
how long HMS service can start, and then set the initial delay around that
value. So we can avoid a lot of exception in sentry log
{code}
2017-12-11 10:50:00,779 ERROR org.apache.sentry.service.thrift.HMSFollower:
HMSFollower cannot connect to HMS!!
java.lang.reflect.UndeclaredThrowableException
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1980)
at
org.apache.sentry.service.thrift.HiveSimpleConnectionFactory.connect(HiveSimpleConnectionFactory.java:114)
at
org.apache.sentry.service.thrift.SentryHMSClient.connect(SentryHMSClient.java:108)
at
org.apache.sentry.service.thrift.HMSFollower.syncupWithHms(HMSFollower.java:158)
{code}
We want to minimize those exception before HMS service is up and Sentry can
connect to it by {color:red}setting SENTRY_HMSFOLLOWER_INIT_DELAY_MILLS_DEFAULT
a little lower than the time HMS service takes to start{color}. Setting this
value too large will unnecessarily delay sentry service.
Basically, such exception should be ignored before the following message shows
up in sentry log.
{code}
2017-12-01 05:50:03,672 INFO org.apache.sentry.service.thrift.HMSFollower:
Sentry HMS support is ready
{code}
For example, it takes about 5 minutes for HMS service to start, and sentry
service is able to connect to HMS service in one cluster
{code}
2017-12-11 10:49:53
2017-12-11 10:55:18,745 INFO hive.metastore: Connected to metastore.
2017-12-11 10:55:18,832 INFO org.apache.sentry.service.thrift.SentryHMSClient:
Request full HMS snapshot
2017-12-11 10:55:19,147 INFO org.apache.sentry.service.thrift.HMSFollower:
Sentry HMS support is ready
{code}
> Make HMSFollower initial delay configurable
> ---
>
> Key: SENTRY-1753
> URL: https://issues.apache.org/jira/browse/SENTRY-1753
> Project: Sentry
> Issue Type: Task
> Components: Sentry
>Affects Versions: 2.0.0
>Reporter: Na Li
>Priority: Minor
>
> HMSFollower initial delay affects how fast it can get notification from Hive
> meta store. We are changing it from 60 seconds to 0 second. It may cause some
> issue. It is better to make this value configurable, so we can adjust it
> based on testing
[jira] [Commented] (SENTRY-1753) Make HMSFollower initial delay configurable
[
https://issues.apache.org/jira/browse/SENTRY-1753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16289606#comment-16289606
]
Na Li commented on SENTRY-1753:
---
[~liamsargent] the default value right now is 0
"SENTRY_HMSFOLLOWER_INIT_DELAY_MILLS_DEFAULT = 0;". It would be nice to measure
how long HMS service can start, and then set the initial delay around that
value. So we can avoid a lot of exception in sentry log
{code}
2017-12-11 10:50:00,779 ERROR org.apache.sentry.service.thrift.HMSFollower:
HMSFollower cannot connect to HMS!!
java.lang.reflect.UndeclaredThrowableException
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1980)
at
org.apache.sentry.service.thrift.HiveSimpleConnectionFactory.connect(HiveSimpleConnectionFactory.java:114)
at
org.apache.sentry.service.thrift.SentryHMSClient.connect(SentryHMSClient.java:108)
at
org.apache.sentry.service.thrift.HMSFollower.syncupWithHms(HMSFollower.java:158)
{code}
We want to minimize those exception before HMS service is up and Sentry can
connect to it by {color:red}setting SENTRY_HMSFOLLOWER_INIT_DELAY_MILLS_DEFAULT
a little lower than the time HMS service takes to start{color}. Setting this
value too large will unnecessarily delay sentry service.
Basically, such exception should be ignored before the following message shows
up in sentry log.
{code}
2017-12-01 05:50:03,672 INFO org.apache.sentry.service.thrift.HMSFollower:
Sentry HMS support is ready
{code}
For example, it takes about 5 minutes for HMS service to start, and sentry
service is able to connect to HMS service in one cluster
{code}
2017-12-11 10:49:53
2017-12-11 10:55:18,745 INFO hive.metastore: Connected to metastore.
2017-12-11 10:55:18,832 INFO org.apache.sentry.service.thrift.SentryHMSClient:
Request full HMS snapshot
2017-12-11 10:55:19,147 INFO org.apache.sentry.service.thrift.HMSFollower:
Sentry HMS support is ready
{code}
> Make HMSFollower initial delay configurable
> ---
>
> Key: SENTRY-1753
> URL: https://issues.apache.org/jira/browse/SENTRY-1753
> Project: Sentry
> Issue Type: Task
> Components: Sentry
>Affects Versions: 2.0.0
>Reporter: Na Li
>Priority: Minor
>
> HMSFollower initial delay affects how fast it can get notification from Hive
> meta store. We are changing it from 60 seconds to 0 second. It may cause some
> issue. It is better to make this value configurable, so we can adjust it
> based on testing result.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (SENTRY-2097) Sentry privileges model: Can Sentry take a database privileges away from a server privileges?
[
https://issues.apache.org/jira/browse/SENTRY-2097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16288467#comment-16288467
]
Na Li commented on SENTRY-2097:
---
We should document this, so user won't revoke privileges in this way
To fix this issue, require the following changes
1) Revoke behavior
Need to store revoked privileges
2) Privilege checking behavior
Follow the authorization object hierarchy. The finest granularity matched
privilege determines the access.
> Sentry privileges model: Can Sentry take a database privileges away from a
> server privileges?
> -
>
> Key: SENTRY-2097
> URL: https://issues.apache.org/jira/browse/SENTRY-2097
> Project: Sentry
> Issue Type: Bug
> Components: Sentry
>Reporter: Sergio Peña
>Assignee: Na Li
>Priority: Critical
>
> Assume I have a user |jack| and a group |datateam|. The
> user |jack| belongs to group |datateam|.
> Use Sentry for authorization.
> |create role admin; grant role admin to group datateam; grant all on
> server server1 to role admin; |
> Now the role |admin| has the following priveleges.
> {noformat}
> |+---+++-+-+-++---+---+--+--+
> | database | table | partition | column | principal_name |
> principal_type | privilege | grant_option | grant_time | grantor |
> +---+++-+-+-++---+---+--+--+
> | * | | | | admin | ROLE | * | false | 1480985013185000 | -- |
> +---+++-+-+-++---+---+--+--+|
> {noformat}
> Assume I have this database.
> |create database testdb; |
> It is successful. User |jack| created a database |testdb|.
> Use Sentry to revoke the privileges on |testdb|;
> |revoke all on database `testdb` from role admin; |
> The priveleges is still the same.
> {noformat}
> |+---+++-+-+-++---+---+--+--+
> | database | table | partition | column | principal_name |
> principal_type | privilege | grant_option | grant_time | grantor |
> +---+++-+-+-++---+---+--+--+
> | * | | | | admin | ROLE | * | false | 1480985013185000 | -- |
> +---+++-+-+-++---+---+--+--+|
> {noformat}
> Shouldn't Sentry take the privileges on database |testdb| away from the
> server |server1|?
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Assigned] (SENTRY-2097) Sentry privileges model: Can Sentry take a database privileges away from a server privileges?
[
https://issues.apache.org/jira/browse/SENTRY-2097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Na Li reassigned SENTRY-2097:
-
Assignee: Na Li
> Sentry privileges model: Can Sentry take a database privileges away from a
> server privileges?
> -
>
> Key: SENTRY-2097
> URL: https://issues.apache.org/jira/browse/SENTRY-2097
> Project: Sentry
> Issue Type: Bug
> Components: Sentry
>Reporter: Sergio Peña
>Assignee: Na Li
>Priority: Critical
>
> Assume I have a user |jack| and a group |datateam|. The
> user |jack| belongs to group |datateam|.
> Use Sentry for authorization.
> |create role admin; grant role admin to group datateam; grant all on
> server server1 to role admin; |
> Now the role |admin| has the following priveleges.
> {noformat}
> |+---+++-+-+-++---+---+--+--+
> | database | table | partition | column | principal_name |
> principal_type | privilege | grant_option | grant_time | grantor |
> +---+++-+-+-++---+---+--+--+
> | * | | | | admin | ROLE | * | false | 1480985013185000 | -- |
> +---+++-+-+-++---+---+--+--+|
> {noformat}
> Assume I have this database.
> |create database testdb; |
> It is successful. User |jack| created a database |testdb|.
> Use Sentry to revoke the privileges on |testdb|;
> |revoke all on database `testdb` from role admin; |
> The priveleges is still the same.
> {noformat}
> |+---+++-+-+-++---+---+--+--+
> | database | table | partition | column | principal_name |
> principal_type | privilege | grant_option | grant_time | grantor |
> +---+++-+-+-++---+---+--+--+
> | * | | | | admin | ROLE | * | false | 1480985013185000 | -- |
> +---+++-+-+-++---+---+--+--+|
> {noformat}
> Shouldn't Sentry take the privileges on database |testdb| away from the
> server |server1|?
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (SENTRY-2096) Fail unit tests at end
[ https://issues.apache.org/jira/browse/SENTRY-2096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Na Li updated SENTRY-2096: -- Status: Patch Available (was: Open) > Fail unit tests at end > -- > > Key: SENTRY-2096 > URL: https://issues.apache.org/jira/browse/SENTRY-2096 > Project: Sentry > Issue Type: Task > Components: Sentry >Affects Versions: 2.1.0 >Reporter: Na Li >Assignee: Na Li > Attachments: SENTRY-2096.001.patch > > > Currently, when the unit tests fail in one module, the test stops and report > failure. It is better to run tests in all modules and then report all failed > tests. So we can fix test failure at once. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (SENTRY-2096) Fail unit tests at end
[ https://issues.apache.org/jira/browse/SENTRY-2096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Na Li updated SENTRY-2096: -- Attachment: SENTRY-2096.001.patch > Fail unit tests at end > -- > > Key: SENTRY-2096 > URL: https://issues.apache.org/jira/browse/SENTRY-2096 > Project: Sentry > Issue Type: Task > Components: Sentry >Affects Versions: 2.1.0 >Reporter: Na Li >Assignee: Na Li > Attachments: SENTRY-2096.001.patch > > > Currently, when the unit tests fail in one module, the test stops and report > failure. It is better to run tests in all modules and then report all failed > tests. So we can fix test failure at once. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (SENTRY-2096) Fail unit tests at end
[ https://issues.apache.org/jira/browse/SENTRY-2096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16284388#comment-16284388 ] Na Li commented on SENTRY-2096: --- mvn test --fail-at-end should be used in test-patch.py > Fail unit tests at end > -- > > Key: SENTRY-2096 > URL: https://issues.apache.org/jira/browse/SENTRY-2096 > Project: Sentry > Issue Type: Task > Components: Sentry >Affects Versions: 2.1.0 >Reporter: Na Li >Assignee: Na Li > > Currently, when the unit tests fail in one module, the test stops and report > failure. It is better to run tests in all modules and then report all failed > tests. So we can fix test failure at once. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (SENTRY-2096) Fail unit tests at end
Na Li created SENTRY-2096: - Summary: Fail unit tests at end Key: SENTRY-2096 URL: https://issues.apache.org/jira/browse/SENTRY-2096 Project: Sentry Issue Type: Task Components: Sentry Affects Versions: 2.1.0 Reporter: Na Li Assignee: Na Li Currently, when the unit tests fail in one module, the test stops and report failure. It is better to run tests in all modules and then report all failed tests. So we can fix test failure at once. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (SENTRY-2095) Support User-based privilege at Name Node through HDFS sync
Na Li created SENTRY-2095: - Summary: Support User-based privilege at Name Node through HDFS sync Key: SENTRY-2095 URL: https://issues.apache.org/jira/browse/SENTRY-2095 Project: Sentry Issue Type: New Feature Components: Sentry Affects Versions: 2.1.0 Reporter: Na Li Assignee: Na Li When a user is granted with role to access authorizables, that user should be able to access corresponding files at HDFS with the ACL derived from permission info and path info from sentry -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (SENTRY-2083) Add more test cases for policy processor
[ https://issues.apache.org/jira/browse/SENTRY-2083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Na Li resolved SENTRY-2083. --- Resolution: Won't Fix no need to do so. > Add more test cases for policy processor > > > Key: SENTRY-2083 > URL: https://issues.apache.org/jira/browse/SENTRY-2083 > Project: Sentry > Issue Type: Task > Components: Sentry >Affects Versions: 2.0.0 >Reporter: Na Li >Assignee: Na Li > > We need to add more testing cases for policy store processor -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (SENTRY-2091) User-based Privilege is broken by SENTRY-769
[
https://issues.apache.org/jira/browse/SENTRY-2091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Na Li updated SENTRY-2091:
--
Attachment: SENTRY-2091.003.patch
v3. fix hive test failure.
> User-based Privilege is broken by SENTRY-769
>
>
> Key: SENTRY-2091
> URL: https://issues.apache.org/jira/browse/SENTRY-2091
> Project: Sentry
> Issue Type: Bug
> Components: Sentry
>Affects Versions: 2.1.0
>Reporter: Na Li
>Assignee: Na Li
> Attachments: SENTRY-2091.001.patch, SENTRY-2091.002.patch,
> SENTRY-2091.003.patch
>
>
> SENTRY-769 throws exception when a user has no group. This breaks user-based
> privilege as the exception prevents getting privilege using user-based
> privilege.
> For example, in the following code
> {code}
> Set userPrivileges =
> authProvider.getPolicyEngine().getPrivileges(
> authProvider.getGroupMapping().getGroups(userName),
> Sets.newHashSet(userName),
> hiveAuthzBinding.getActiveRoleSet(),
> hiveAuthzBinding.getAuthServer());
> {code}
> when user has no group, the exception causes the processing stops even when
> user has privilege.
> The solution is to catch the exception, and continue the processing.
> {code}
> try {
> Set groups = null;
> try {
> groups = authProvider.getGroupMapping().getGroups(userName)
> } catch (SentryGroupNotFoundException ex) {
> log.debug(...);
> groups = new HashSet();
> }
> Set userPrivileges =
> authProvider.getPolicyEngine().getPrivileges(
> groups, Sets.newHashSet(userName),
> hiveAuthzBinding.getActiveRoleSet(),
> hiveAuthzBinding.getAuthServer());
> ...
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (SENTRY-2091) User-based Privilege is broken by SENTRY-769
[
https://issues.apache.org/jira/browse/SENTRY-2091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Na Li updated SENTRY-2091:
--
Attachment: SENTRY-2091.002.patch
v2. fix solr test failure
> User-based Privilege is broken by SENTRY-769
>
>
> Key: SENTRY-2091
> URL: https://issues.apache.org/jira/browse/SENTRY-2091
> Project: Sentry
> Issue Type: Bug
> Components: Sentry
>Affects Versions: 2.1.0
>Reporter: Na Li
>Assignee: Na Li
> Attachments: SENTRY-2091.001.patch, SENTRY-2091.002.patch
>
>
> SENTRY-769 throws exception when a user has no group. This breaks user-based
> privilege as the exception prevents getting privilege using user-based
> privilege.
> For example, in the following code
> {code}
> Set userPrivileges =
> authProvider.getPolicyEngine().getPrivileges(
> authProvider.getGroupMapping().getGroups(userName),
> Sets.newHashSet(userName),
> hiveAuthzBinding.getActiveRoleSet(),
> hiveAuthzBinding.getAuthServer());
> {code}
> when user has no group, the exception causes the processing stops even when
> user has privilege.
> The solution is to catch the exception, and continue the processing.
> {code}
> try {
> Set groups = null;
> try {
> groups = authProvider.getGroupMapping().getGroups(userName)
> } catch (SentryGroupNotFoundException ex) {
> log.debug(...);
> groups = new HashSet();
> }
> Set userPrivileges =
> authProvider.getPolicyEngine().getPrivileges(
> groups, Sets.newHashSet(userName),
> hiveAuthzBinding.getActiveRoleSet(),
> hiveAuthzBinding.getAuthServer());
> ...
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (SENTRY-2091) User-based Privilege is broken by SENTRY-769
[
https://issues.apache.org/jira/browse/SENTRY-2091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Na Li updated SENTRY-2091:
--
Status: Patch Available (was: Open)
> User-based Privilege is broken by SENTRY-769
>
>
> Key: SENTRY-2091
> URL: https://issues.apache.org/jira/browse/SENTRY-2091
> Project: Sentry
> Issue Type: Bug
> Components: Sentry
>Affects Versions: 2.1.0
>Reporter: Na Li
>Assignee: Na Li
> Attachments: SENTRY-2091.001.patch
>
>
> SENTRY-769 throws exception when a user has no group. This breaks user-based
> privilege as the exception prevents getting privilege using user-based
> privilege.
> For example, in the following code
> {code}
> Set userPrivileges =
> authProvider.getPolicyEngine().getPrivileges(
> authProvider.getGroupMapping().getGroups(userName),
> Sets.newHashSet(userName),
> hiveAuthzBinding.getActiveRoleSet(),
> hiveAuthzBinding.getAuthServer());
> {code}
> when user has no group, the exception causes the processing stops even when
> user has privilege.
> The solution is to catch the exception, and continue the processing.
> {code}
> try {
> Set groups = null;
> try {
> groups = authProvider.getGroupMapping().getGroups(userName)
> } catch (SentryGroupNotFoundException ex) {
> log.debug(...);
> groups = new HashSet();
> }
> Set userPrivileges =
> authProvider.getPolicyEngine().getPrivileges(
> groups, Sets.newHashSet(userName),
> hiveAuthzBinding.getActiveRoleSet(),
> hiveAuthzBinding.getAuthServer());
> ...
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (SENTRY-2091) User-based Privilege is broken by SENTRY-769
[
https://issues.apache.org/jira/browse/SENTRY-2091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Na Li updated SENTRY-2091:
--
Attachment: SENTRY-2091.001.patch
> User-based Privilege is broken by SENTRY-769
>
>
> Key: SENTRY-2091
> URL: https://issues.apache.org/jira/browse/SENTRY-2091
> Project: Sentry
> Issue Type: Bug
> Components: Sentry
>Affects Versions: 2.1.0
>Reporter: Na Li
>Assignee: Na Li
> Attachments: SENTRY-2091.001.patch
>
>
> SENTRY-769 throws exception when a user has no group. This breaks user-based
> privilege as the exception prevents getting privilege using user-based
> privilege.
> For example, in the following code
> {code}
> Set userPrivileges =
> authProvider.getPolicyEngine().getPrivileges(
> authProvider.getGroupMapping().getGroups(userName),
> Sets.newHashSet(userName),
> hiveAuthzBinding.getActiveRoleSet(),
> hiveAuthzBinding.getAuthServer());
> {code}
> when user has no group, the exception causes the processing stops even when
> user has privilege.
> The solution is to catch the exception, and continue the processing.
> {code}
> try {
> Set groups = null;
> try {
> groups = authProvider.getGroupMapping().getGroups(userName)
> } catch (SentryGroupNotFoundException ex) {
> log.debug(...);
> groups = new HashSet();
> }
> Set userPrivileges =
> authProvider.getPolicyEngine().getPrivileges(
> groups, Sets.newHashSet(userName),
> hiveAuthzBinding.getActiveRoleSet(),
> hiveAuthzBinding.getAuthServer());
> ...
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (SENTRY-2092) Drop Role log message shows "Creating role"
[
https://issues.apache.org/jira/browse/SENTRY-2092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Na Li updated SENTRY-2092:
--
Attachment: SENTRY-2092.002.patch
v2. update error message
> Drop Role log message shows "Creating role"
> ---
>
> Key: SENTRY-2092
> URL: https://issues.apache.org/jira/browse/SENTRY-2092
> Project: Sentry
> Issue Type: Bug
> Components: Sentry
>Affects Versions: 2.1.0
>Reporter: Na Li
>Assignee: Na Li
> Labels: bite-sized, newbie
> Attachments: SENTRY-2092.001.patch, SENTRY-2092.002.patch
>
>
> DefaultSentryAccessController.dropRole logs error message as "Error occurred
> when Sentry client creating role: "
> It is confusing, and could waster engineering time to debug such
> inconsistency. One example is in hive log, we see
> org.apache.hadoop.hive.ql.exec.SentryFilterDDLTask. Error occurred when
> Sentry client {color:red}creating role{color}: Role
> :{color:red}TDropSentryRoleRequest{color}(protocol_version:2,
> requestorUserName:hive, roleName:hbase_admin) doesn't exist. Server
> Stacktrace:
> org.apache.sentry.core.common.exception.SentryNoSuchObjectException: Role
> hbase_admin doesn't exist
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore.noSuchRole(SentryStore.java:3695)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore.dropSentryRoleCore(SentryStore.java:1249)
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (SENTRY-2092) Drop Role log message shows "Creating role"
[
https://issues.apache.org/jira/browse/SENTRY-2092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Na Li updated SENTRY-2092:
--
Attachment: SENTRY-2092.001.patch
> Drop Role log message shows "Creating role"
> ---
>
> Key: SENTRY-2092
> URL: https://issues.apache.org/jira/browse/SENTRY-2092
> Project: Sentry
> Issue Type: Bug
> Components: Sentry
>Affects Versions: 2.1.0
>Reporter: Na Li
>Assignee: Na Li
> Attachments: SENTRY-2092.001.patch
>
>
> DefaultSentryAccessController.dropRole logs error message as "Error occurred
> when Sentry client creating role: "
> It is confusing, and could waster engineering time to debug such
> inconsistency. One example is in hive log, we see
> org.apache.hadoop.hive.ql.exec.SentryFilterDDLTask. Error occurred when
> Sentry client {color:red}creating role{color}: Role
> :{color:red}TDropSentryRoleRequest{color}(protocol_version:2,
> requestorUserName:hive, roleName:hbase_admin) doesn't exist. Server
> Stacktrace:
> org.apache.sentry.core.common.exception.SentryNoSuchObjectException: Role
> hbase_admin doesn't exist
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore.noSuchRole(SentryStore.java:3695)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore.dropSentryRoleCore(SentryStore.java:1249)
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (SENTRY-2092) Drop Role log message shows "Creating role"
[
https://issues.apache.org/jira/browse/SENTRY-2092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Na Li updated SENTRY-2092:
--
Status: Patch Available (was: Open)
> Drop Role log message shows "Creating role"
> ---
>
> Key: SENTRY-2092
> URL: https://issues.apache.org/jira/browse/SENTRY-2092
> Project: Sentry
> Issue Type: Bug
> Components: Sentry
>Affects Versions: 2.1.0
>Reporter: Na Li
>Assignee: Na Li
> Attachments: SENTRY-2092.001.patch
>
>
> DefaultSentryAccessController.dropRole logs error message as "Error occurred
> when Sentry client creating role: "
> It is confusing, and could waster engineering time to debug such
> inconsistency. One example is in hive log, we see
> org.apache.hadoop.hive.ql.exec.SentryFilterDDLTask. Error occurred when
> Sentry client {color:red}creating role{color}: Role
> :{color:red}TDropSentryRoleRequest{color}(protocol_version:2,
> requestorUserName:hive, roleName:hbase_admin) doesn't exist. Server
> Stacktrace:
> org.apache.sentry.core.common.exception.SentryNoSuchObjectException: Role
> hbase_admin doesn't exist
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore.noSuchRole(SentryStore.java:3695)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore.dropSentryRoleCore(SentryStore.java:1249)
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (SENTRY-2092) Drop Role log message shows "Creating role"
Na Li created SENTRY-2092:
-
Summary: Drop Role log message shows "Creating role"
Key: SENTRY-2092
URL: https://issues.apache.org/jira/browse/SENTRY-2092
Project: Sentry
Issue Type: Bug
Components: Sentry
Affects Versions: 2.1.0
Reporter: Na Li
Assignee: Na Li
DefaultSentryAccessController.dropRole logs error message as "Error occurred
when Sentry client creating role: "
It is confusing, and could waster engineering time to debug such inconsistency.
One example is in hive log, we see
org.apache.hadoop.hive.ql.exec.SentryFilterDDLTask. Error occurred when Sentry
client {color:red}creating role{color}: Role
:{color:red}TDropSentryRoleRequest{color}(protocol_version:2,
requestorUserName:hive, roleName:hbase_admin) doesn't exist. Server Stacktrace:
org.apache.sentry.core.common.exception.SentryNoSuchObjectException: Role
hbase_admin doesn't exist
at
org.apache.sentry.provider.db.service.persistent.SentryStore.noSuchRole(SentryStore.java:3695)
at
org.apache.sentry.provider.db.service.persistent.SentryStore.dropSentryRoleCore(SentryStore.java:1249)
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (SENTRY-2091) User-based Privilege is broken by SENTRY-769
Na Li created SENTRY-2091:
-
Summary: User-based Privilege is broken by SENTRY-769
Key: SENTRY-2091
URL: https://issues.apache.org/jira/browse/SENTRY-2091
Project: Sentry
Issue Type: Bug
Components: Sentry
Affects Versions: 2.1.0
Reporter: Na Li
Assignee: Na Li
SENTRY-769 throws exception when a user has no group. This breaks user-based
privilege as the exception prevents getting privilege using user-based
privilege.
For example, in the following code
{code}
Set userPrivileges =
authProvider.getPolicyEngine().getPrivileges(
authProvider.getGroupMapping().getGroups(userName),
Sets.newHashSet(userName),
hiveAuthzBinding.getActiveRoleSet(),
hiveAuthzBinding.getAuthServer());
{code}
when user has no group, the exception causes the processing stops even when
user has privilege.
The solution is to catch the exception, and continue the processing.
{code}
try {
Set groups = null;
try {
groups = authProvider.getGroupMapping().getGroups(userName)
} catch (SentryGroupNotFoundException ex) {
log.debug(...);
groups = new HashSet();
}
Set userPrivileges =
authProvider.getPolicyEngine().getPrivileges(
groups, Sets.newHashSet(userName),
hiveAuthzBinding.getActiveRoleSet(),
hiveAuthzBinding.getAuthServer());
...
}
{code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (SENTRY-2083) Add more test cases for policy processor
Na Li created SENTRY-2083: - Summary: Add more test cases for policy processor Key: SENTRY-2083 URL: https://issues.apache.org/jira/browse/SENTRY-2083 Project: Sentry Issue Type: Task Components: Sentry Affects Versions: 2.0.0 Reporter: Na Li Assignee: Na Li We need to add more testing cases for policy store processor -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (SENTRY-1813) LeaderStatusMonitor could get into limbo state upon ZK connection loss
[
https://issues.apache.org/jira/browse/SENTRY-1813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Na Li resolved SENTRY-1813.
---
Resolution: Duplicate
Fix Version/s: (was: 2.1.0)
2.0.0
> LeaderStatusMonitor could get into limbo state upon ZK connection loss
> --
>
> Key: SENTRY-1813
> URL: https://issues.apache.org/jira/browse/SENTRY-1813
> Project: Sentry
> Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Vamsee Yarlagadda
>Assignee: Vamsee Yarlagadda
> Fix For: 2.0.0
>
> Attachments: Screenshot.png
>
>
> I noticed that during failover testing, if there was a connection loss with
> ZK to the sentry servers, the one who is currently the leader gets into a
> limbo state as it interrupts the Curator-LeaderSelector thread which no
> longer gets revived in the running Sentry process (unless the process is
> restarted).
> Relevant code under LeaderStatusMonitor
> http://github.mtv.cloudera.com/CDH/sentry/blob/cdh5-1.5.1/sentry-provider/sentry-provider-db/src/main/java/org/apache/sentry/service/thrift/LeaderStatusMonitor.java#L243-L246
> {code}
>try {
> isLeader = true;
> // Wait until we are interrupted or receive a signal
> cond.await();
> } catch (InterruptedException ignored) {
> Thread.currentThread().interrupt();
> LOG.info("LeaderStatusMonitor: interrupted");
> } finally {
> isLeader = false;
> lock.unlock();
> LOG.info("LeaderStatusMonitor: becoming standby");
> }
> {code}
> I realized even upon the loss of ZK connection, curator framework raises an
> Interrupted Exception in LeaderStausMonitor which attempts to call interrupt
> on Thread.currentThread which is essentially *Curator-LeaderSelector* thread.
>
> So if the LeaderSelector thread is interrupted, this particular Sentry server
> loses the capability of participating in a leader election in the future. And
> if this happens to all the sentry servers in the cluster, any further loss
> could get into a limbo state.
> And during this state, Sentry no longer reads events from HMS and thereby
> users can no longer be able to issue DDL statements like CREATE etc. However
> GRANT, REVOKE still work as they don't go through HMSFollower.
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (SENTRY-2070) TestSentryShellSolr tests are case sensitive to privileges
[ https://issues.apache.org/jira/browse/SENTRY-2070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Na Li updated SENTRY-2070: -- Resolution: Won't Fix Status: Resolved (was: Patch Available) The corresponding test does not fail. So no need to change the test. If the test fails later, we can reopen it. > TestSentryShellSolr tests are case sensitive to privileges > -- > > Key: SENTRY-2070 > URL: https://issues.apache.org/jira/browse/SENTRY-2070 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.0.0 >Reporter: Na Li >Assignee: Na Li > Attachments: SENTRY-2070.001.patch > > > When comparing expected privileges with actual privileges, the string > comparison is case sensitive and therefore may fail. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (SENTRY-2079) Sentry HA leader monitor does not work due to a mix of curator versions in the classpath
[ https://issues.apache.org/jira/browse/SENTRY-2079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16271654#comment-16271654 ] Na Li commented on SENTRY-2079: --- [~spena] Is this related to SENTRY-1813? > Sentry HA leader monitor does not work due to a mix of curator versions in > the classpath > > > Key: SENTRY-2079 > URL: https://issues.apache.org/jira/browse/SENTRY-2079 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.0.0 >Reporter: Sergio Peña >Assignee: Sergio Peña >Priority: Blocker > Fix For: 2.0.0 > > > While testing Sentry 2.0, I couldn't enable HA mode because connections to > Zookeeper were stuck due to a mix of Apache curator dependencies > (curator-client-2.8.0 and curator-client-2.11.1). > We need to exclude the old curator dependencies and also shade the curator > jars so that it does not affect other components. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (SENTRY-379) Db entities are not captured when firing failurehook for SentryAccessDeniedException
[
https://issues.apache.org/jira/browse/SENTRY-379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Na Li resolved SENTRY-379.
--
Resolution: Fixed
test shows the DB name is captured when firing failurehook.
> Db entities are not captured when firing failurehook for
> SentryAccessDeniedException
>
>
> Key: SENTRY-379
> URL: https://issues.apache.org/jira/browse/SENTRY-379
> Project: Sentry
> Issue Type: Bug
>Affects Versions: 1.4.0
>Reporter: Sravya Tirukkovalur
>Priority: Minor
> Fix For: 2.1.0
>
>
> {noformat}
> SentryOnFailureHookContext hookContext = new SentryOnFailureHookContextImpl(
> queryPlan.getQueryString(), new HashSet(),
> new HashSet(), stmtOperation,
> null, null, null, null, subject.getName(), ipAddress,
> new AuthorizationException(e), conf);
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (SENTRY-379) Db entities are not captured when firing failurehook for SentryAccessDeniedException
[
https://issues.apache.org/jira/browse/SENTRY-379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16267026#comment-16267026
]
Na Li commented on SENTRY-379:
--
[~spena] [~kkalyan] I believe this is fixed. And test
TestDbSentryOnFailureHookLoading checks if the DB name and table name are set
in failurehook for SentryAccessDeniedException. If you agree, I can resolve
this issue.
> Db entities are not captured when firing failurehook for
> SentryAccessDeniedException
>
>
> Key: SENTRY-379
> URL: https://issues.apache.org/jira/browse/SENTRY-379
> Project: Sentry
> Issue Type: Bug
>Affects Versions: 1.4.0
>Reporter: Sravya Tirukkovalur
>Priority: Minor
> Fix For: 2.1.0
>
>
> {noformat}
> SentryOnFailureHookContext hookContext = new SentryOnFailureHookContextImpl(
> queryPlan.getQueryString(), new HashSet(),
> new HashSet(), stmtOperation,
> null, null, null, null, subject.getName(), ipAddress,
> new AuthorizationException(e), conf);
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (SENTRY-2070) TestSentryShellSolr tests are case sensitive to privileges
[ https://issues.apache.org/jira/browse/SENTRY-2070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Na Li updated SENTRY-2070: -- Attachment: SENTRY-2070.001.patch > TestSentryShellSolr tests are case sensitive to privileges > -- > > Key: SENTRY-2070 > URL: https://issues.apache.org/jira/browse/SENTRY-2070 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.0.0 >Reporter: Na Li >Assignee: Na Li > Attachments: SENTRY-2070.001.patch > > > When comparing expected privileges with actual privileges, the string > comparison is case sensitive and therefore may fail. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (SENTRY-2070) TestSentryShellSolr tests are case sensitive to privileges
[ https://issues.apache.org/jira/browse/SENTRY-2070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Na Li updated SENTRY-2070: -- Status: Patch Available (was: Open) > TestSentryShellSolr tests are case sensitive to privileges > -- > > Key: SENTRY-2070 > URL: https://issues.apache.org/jira/browse/SENTRY-2070 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.0.0 >Reporter: Na Li >Assignee: Na Li > Attachments: SENTRY-2070.001.patch > > > When comparing expected privileges with actual privileges, the string > comparison is case sensitive and therefore may fail. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (SENTRY-2070) TestSentryShellSolr tests are case sensitive to privileges
Na Li created SENTRY-2070: - Summary: TestSentryShellSolr tests are case sensitive to privileges Key: SENTRY-2070 URL: https://issues.apache.org/jira/browse/SENTRY-2070 Project: Sentry Issue Type: Bug Components: Sentry Affects Versions: 2.0.0 Reporter: Na Li Assignee: Na Li When comparing expected privileges with actual privileges, the string comparison is case sensitive and therefore may fail. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (SENTRY-2024) Drop Index that includes AUTHZ_OBJ_NAME
[
https://issues.apache.org/jira/browse/SENTRY-2024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Na Li updated SENTRY-2024:
--
Attachment: SENTRY-2024.005.patch
v5. Make index name easier to read
> Drop Index that includes AUTHZ_OBJ_NAME
> ---
>
> Key: SENTRY-2024
> URL: https://issues.apache.org/jira/browse/SENTRY-2024
> Project: Sentry
> Issue Type: Bug
> Components: Sentry
>Affects Versions: 2.0.0
>Reporter: Na Li
>Assignee: Na Li
>Priority: Critical
> Attachments: SENTRY-2024.001.patch, SENTRY-2024.001.patch,
> SENTRY-2024.001.patch, SENTRY-2024.002.patch, SENTRY-2024.002.patch,
> SENTRY-2024.003.patch, SENTRY-2024.004.patch, SENTRY-2024.005.patch
>
>
> AUTHZ_OBJ_NAME has (384) chars. It is constructed by sentry from notification
> event as {color:red}DB_NAME + "." + TBL_NAME{color}. To be consistent with
> hive, sentry should use the same char set as what's used by DB_NAME and
> TBL_NAME
> Besides, if its table char set is utf8, the constrain AUTHZOBJNAME will have
> error "Specified key was too long; max key length is 767 bytes" for mysql.
> The solution is to drop the index includes this field, so it works for mysql
> regardless the char set of its table or DB.
> Reference:
> 1) CREATE TABLE AUTHZ_PATHS_MAPPING
> (
> AUTHZ_OBJ_ID BIGINT NOT NULL generated always as identity (start with 1),
> AUTHZ_OBJ_NAME VARCHAR({color:red}384{color}),
> CREATE_TIME_MS BIGINT NOT NULL
> );
> CREATE UNIQUE INDEX AUTHZOBJNAME ON AUTHZ_PATHS_MAPPING
> ({color:red}AUTHZ_OBJ_NAME{color});
> 2) AUTHZ_OBJ_NAME is constructed by sentry from notification event as
> DB_NAME + "." + TBL_NAME. To be consistent with hive, sentry should use the
> same char set as what's used by DB_NAME and TBL_NAME in
> NotificationProcessor.processAlterTable().
> 3) hive uses latin1 as default charset for notification event.
> CREATE TABLE IF NOT EXISTS `NOTIFICATION_LOG`
> (
> `NL_ID` BIGINT(20) NOT NULL,
> `EVENT_ID` BIGINT(20) NOT NULL,
> `EVENT_TIME` INT(11) NOT NULL,
> `EVENT_TYPE` varchar(32) NOT NULL,
> `DB_NAME` varchar(128),
> `TBL_NAME` varchar(128),
> `MESSAGE` mediumtext,
> PRIMARY KEY (`NL_ID`)
> ) ENGINE=InnoDB DEFAULT CHARSET=latin1;
> 4) actual installation shows both fields are in latin1 char set.
> SHOW FULL COLUMNS FROM NOTIFICATION_LOG;
> ++--+---+--+-+-+---+-+-+
> | Field | Type | Collation | Null | Key | Default |
> Extra | Privileges | Comment |
> ++--+---+--+-+-+---+-+-+
> | NL_ID | bigint(20) | NULL | NO | PRI | NULL|
> | select,insert,update,references | |
> | EVENT_ID | bigint(20) | NULL | NO | | NULL|
> | select,insert,update,references | |
> | EVENT_TIME | int(11) | NULL | NO | | NULL|
> | select,insert,update,references | |
> | EVENT_TYPE | varchar(32) | latin1_swedish_ci | NO | | NULL|
> | select,insert,update,references | |
> | DB_NAME| varchar(128) | latin1_swedish_ci | YES | | NULL|
> | select,insert,update,references | |
> | TBL_NAME | varchar(128) | latin1_swedish_ci | YES | | NULL|
> | select,insert,update,references | |
> | MESSAGE| mediumtext | latin1_swedish_ci | YES | | NULL|
> | select,insert,update,references | |
> ++--+---+--+-+-+---+-+-+
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (SENTRY-2024) Drop Index that includes AUTHZ_OBJ_NAME
[
https://issues.apache.org/jira/browse/SENTRY-2024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Na Li updated SENTRY-2024:
--
Attachment: SENTRY-2024.004.patch
v4. Create index for snapshot ID
> Drop Index that includes AUTHZ_OBJ_NAME
> ---
>
> Key: SENTRY-2024
> URL: https://issues.apache.org/jira/browse/SENTRY-2024
> Project: Sentry
> Issue Type: Bug
> Components: Sentry
>Affects Versions: 2.0.0
>Reporter: Na Li
>Assignee: Na Li
>Priority: Critical
> Attachments: SENTRY-2024.001.patch, SENTRY-2024.001.patch,
> SENTRY-2024.001.patch, SENTRY-2024.002.patch, SENTRY-2024.002.patch,
> SENTRY-2024.003.patch, SENTRY-2024.004.patch
>
>
> AUTHZ_OBJ_NAME has (384) chars. It is constructed by sentry from notification
> event as {color:red}DB_NAME + "." + TBL_NAME{color}. To be consistent with
> hive, sentry should use the same char set as what's used by DB_NAME and
> TBL_NAME
> Besides, if its table char set is utf8, the constrain AUTHZOBJNAME will have
> error "Specified key was too long; max key length is 767 bytes" for mysql.
> The solution is to drop the index includes this field, so it works for mysql
> regardless the char set of its table or DB.
> Reference:
> 1) CREATE TABLE AUTHZ_PATHS_MAPPING
> (
> AUTHZ_OBJ_ID BIGINT NOT NULL generated always as identity (start with 1),
> AUTHZ_OBJ_NAME VARCHAR({color:red}384{color}),
> CREATE_TIME_MS BIGINT NOT NULL
> );
> CREATE UNIQUE INDEX AUTHZOBJNAME ON AUTHZ_PATHS_MAPPING
> ({color:red}AUTHZ_OBJ_NAME{color});
> 2) AUTHZ_OBJ_NAME is constructed by sentry from notification event as
> DB_NAME + "." + TBL_NAME. To be consistent with hive, sentry should use the
> same char set as what's used by DB_NAME and TBL_NAME in
> NotificationProcessor.processAlterTable().
> 3) hive uses latin1 as default charset for notification event.
> CREATE TABLE IF NOT EXISTS `NOTIFICATION_LOG`
> (
> `NL_ID` BIGINT(20) NOT NULL,
> `EVENT_ID` BIGINT(20) NOT NULL,
> `EVENT_TIME` INT(11) NOT NULL,
> `EVENT_TYPE` varchar(32) NOT NULL,
> `DB_NAME` varchar(128),
> `TBL_NAME` varchar(128),
> `MESSAGE` mediumtext,
> PRIMARY KEY (`NL_ID`)
> ) ENGINE=InnoDB DEFAULT CHARSET=latin1;
> 4) actual installation shows both fields are in latin1 char set.
> SHOW FULL COLUMNS FROM NOTIFICATION_LOG;
> ++--+---+--+-+-+---+-+-+
> | Field | Type | Collation | Null | Key | Default |
> Extra | Privileges | Comment |
> ++--+---+--+-+-+---+-+-+
> | NL_ID | bigint(20) | NULL | NO | PRI | NULL|
> | select,insert,update,references | |
> | EVENT_ID | bigint(20) | NULL | NO | | NULL|
> | select,insert,update,references | |
> | EVENT_TIME | int(11) | NULL | NO | | NULL|
> | select,insert,update,references | |
> | EVENT_TYPE | varchar(32) | latin1_swedish_ci | NO | | NULL|
> | select,insert,update,references | |
> | DB_NAME| varchar(128) | latin1_swedish_ci | YES | | NULL|
> | select,insert,update,references | |
> | TBL_NAME | varchar(128) | latin1_swedish_ci | YES | | NULL|
> | select,insert,update,references | |
> | MESSAGE| mediumtext | latin1_swedish_ci | YES | | NULL|
> | select,insert,update,references | |
> ++--+---+--+-+-+---+-+-+
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (SENTRY-2024) Drop Index that includes AUTHZ_OBJ_NAME
[
https://issues.apache.org/jira/browse/SENTRY-2024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Na Li updated SENTRY-2024:
--
Attachment: SENTRY-2024.003.patch
v3. drop index that contains the AUTHZ_OBJ_NAME
> Drop Index that includes AUTHZ_OBJ_NAME
> ---
>
> Key: SENTRY-2024
> URL: https://issues.apache.org/jira/browse/SENTRY-2024
> Project: Sentry
> Issue Type: Bug
> Components: Sentry
>Affects Versions: 2.0.0
>Reporter: Na Li
>Assignee: Na Li
>Priority: Critical
> Attachments: SENTRY-2024.001.patch, SENTRY-2024.001.patch,
> SENTRY-2024.001.patch, SENTRY-2024.002.patch, SENTRY-2024.002.patch,
> SENTRY-2024.003.patch
>
>
> AUTHZ_OBJ_NAME has (384) chars. It is constructed by sentry from notification
> event as {color:red}DB_NAME + "." + TBL_NAME{color}. To be consistent with
> hive, sentry should use the same char set as what's used by DB_NAME and
> TBL_NAME
> Besides, if its table char set is utf8, the constrain AUTHZOBJNAME will have
> error "Specified key was too long; max key length is 767 bytes" for mysql.
> The solution is to drop the index includes this field, so it works for mysql
> regardless the char set of its table or DB.
> Reference:
> 1) CREATE TABLE AUTHZ_PATHS_MAPPING
> (
> AUTHZ_OBJ_ID BIGINT NOT NULL generated always as identity (start with 1),
> AUTHZ_OBJ_NAME VARCHAR({color:red}384{color}),
> CREATE_TIME_MS BIGINT NOT NULL
> );
> CREATE UNIQUE INDEX AUTHZOBJNAME ON AUTHZ_PATHS_MAPPING
> ({color:red}AUTHZ_OBJ_NAME{color});
> 2) AUTHZ_OBJ_NAME is constructed by sentry from notification event as
> DB_NAME + "." + TBL_NAME. To be consistent with hive, sentry should use the
> same char set as what's used by DB_NAME and TBL_NAME in
> NotificationProcessor.processAlterTable().
> 3) hive uses latin1 as default charset for notification event.
> CREATE TABLE IF NOT EXISTS `NOTIFICATION_LOG`
> (
> `NL_ID` BIGINT(20) NOT NULL,
> `EVENT_ID` BIGINT(20) NOT NULL,
> `EVENT_TIME` INT(11) NOT NULL,
> `EVENT_TYPE` varchar(32) NOT NULL,
> `DB_NAME` varchar(128),
> `TBL_NAME` varchar(128),
> `MESSAGE` mediumtext,
> PRIMARY KEY (`NL_ID`)
> ) ENGINE=InnoDB DEFAULT CHARSET=latin1;
> 4) actual installation shows both fields are in latin1 char set.
> SHOW FULL COLUMNS FROM NOTIFICATION_LOG;
> ++--+---+--+-+-+---+-+-+
> | Field | Type | Collation | Null | Key | Default |
> Extra | Privileges | Comment |
> ++--+---+--+-+-+---+-+-+
> | NL_ID | bigint(20) | NULL | NO | PRI | NULL|
> | select,insert,update,references | |
> | EVENT_ID | bigint(20) | NULL | NO | | NULL|
> | select,insert,update,references | |
> | EVENT_TIME | int(11) | NULL | NO | | NULL|
> | select,insert,update,references | |
> | EVENT_TYPE | varchar(32) | latin1_swedish_ci | NO | | NULL|
> | select,insert,update,references | |
> | DB_NAME| varchar(128) | latin1_swedish_ci | YES | | NULL|
> | select,insert,update,references | |
> | TBL_NAME | varchar(128) | latin1_swedish_ci | YES | | NULL|
> | select,insert,update,references | |
> | MESSAGE| mediumtext | latin1_swedish_ci | YES | | NULL|
> | select,insert,update,references | |
> ++--+---+--+-+-+---+-+-+
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (SENTRY-2024) Drop Index that includes AUTHZ_OBJ_NAME
[
https://issues.apache.org/jira/browse/SENTRY-2024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Na Li updated SENTRY-2024:
--
Description:
AUTHZ_OBJ_NAME has (384) chars. It is constructed by sentry from notification
event as {color:red}DB_NAME + "." + TBL_NAME{color}. To be consistent with
hive, sentry should use the same char set as what's used by DB_NAME and
TBL_NAME
Besides, if its table char set is utf8, the constrain AUTHZOBJNAME will have
error "Specified key was too long; max key length is 767 bytes" for mysql.
The solution is to drop the index includes this field, so it works for mysql
regardless the char set of its table or DB.
Reference:
1) CREATE TABLE AUTHZ_PATHS_MAPPING
(
AUTHZ_OBJ_ID BIGINT NOT NULL generated always as identity (start with 1),
AUTHZ_OBJ_NAME VARCHAR({color:red}384{color}),
CREATE_TIME_MS BIGINT NOT NULL
);
CREATE UNIQUE INDEX AUTHZOBJNAME ON AUTHZ_PATHS_MAPPING
({color:red}AUTHZ_OBJ_NAME{color});
2) AUTHZ_OBJ_NAME is constructed by sentry from notification event as DB_NAME
+ "." + TBL_NAME. To be consistent with hive, sentry should use the same char
set as what's used by DB_NAME and TBL_NAME in
NotificationProcessor.processAlterTable().
3) hive uses latin1 as default charset for notification event.
CREATE TABLE IF NOT EXISTS `NOTIFICATION_LOG`
(
`NL_ID` BIGINT(20) NOT NULL,
`EVENT_ID` BIGINT(20) NOT NULL,
`EVENT_TIME` INT(11) NOT NULL,
`EVENT_TYPE` varchar(32) NOT NULL,
`DB_NAME` varchar(128),
`TBL_NAME` varchar(128),
`MESSAGE` mediumtext,
PRIMARY KEY (`NL_ID`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1;
4) actual installation shows both fields are in latin1 char set.
SHOW FULL COLUMNS FROM NOTIFICATION_LOG;
++--+---+--+-+-+---+-+-+
| Field | Type | Collation | Null | Key | Default | Extra
| Privileges | Comment |
++--+---+--+-+-+---+-+-+
| NL_ID | bigint(20) | NULL | NO | PRI | NULL|
| select,insert,update,references | |
| EVENT_ID | bigint(20) | NULL | NO | | NULL|
| select,insert,update,references | |
| EVENT_TIME | int(11) | NULL | NO | | NULL|
| select,insert,update,references | |
| EVENT_TYPE | varchar(32) | latin1_swedish_ci | NO | | NULL|
| select,insert,update,references | |
| DB_NAME| varchar(128) | latin1_swedish_ci | YES | | NULL|
| select,insert,update,references | |
| TBL_NAME | varchar(128) | latin1_swedish_ci | YES | | NULL|
| select,insert,update,references | |
| MESSAGE| mediumtext | latin1_swedish_ci | YES | | NULL|
| select,insert,update,references | |
++--+---+--+-+-+---+-+-+
was:
AUTHZ_OBJ_NAME has (384) chars. It is constructed by sentry from notification
event as {color:red}DB_NAME + "." + TBL_NAME{color}. To be consistent with
hive, sentry should use the same char set as what's used by DB_NAME and
TBL_NAME
Besides, if its table char set is utf8, the constrain AUTHZOBJNAME will have
error "Specified key was too long; max key length is 767 bytes" for mysql.
The solution is to specify the char set for this field, so it works for mysql
regardless the char set of its table or DB.
Reference:
1) CREATE TABLE AUTHZ_PATHS_MAPPING
(
AUTHZ_OBJ_ID BIGINT NOT NULL generated always as identity (start with 1),
AUTHZ_OBJ_NAME VARCHAR({color:red}384{color}),
CREATE_TIME_MS BIGINT NOT NULL
);
CREATE UNIQUE INDEX AUTHZOBJNAME ON AUTHZ_PATHS_MAPPING
({color:red}AUTHZ_OBJ_NAME{color});
2) AUTHZ_OBJ_NAME is constructed by sentry from notification event as DB_NAME
+ "." + TBL_NAME. To be consistent with hive, sentry should use the same char
set as what's used by DB_NAME and TBL_NAME in
NotificationProcessor.processAlterTable().
3) hive uses latin1 as default charset for notification event.
CREATE TABLE IF NOT EXISTS `NOTIFICATION_LOG`
(
`NL_ID` BIGINT(20) NOT NULL,
`EVENT_ID` BIGINT(20) NOT NULL,
`EVENT_TIME` INT(11) NOT NULL,
`EVENT_TYPE` varchar(32) NOT NULL,
`DB_NAME` varchar(128),
`TBL_NAME` varchar(128),
`MESSAGE` mediumtext,
PRIMARY KEY (`NL_ID`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1;
4) actual installation shows both fields are in latin1 char set.
SHOW FULL COLUMNS FROM NOTIFICATION_LOG;
++--+---+--+-+-+---+-+-+
| Field | Type | Collation | Null | Key | Default | Extra
| Privileges
[jira] [Updated] (SENTRY-2024) Drop Index that includes AUTHZ_OBJ_NAME
[
https://issues.apache.org/jira/browse/SENTRY-2024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Na Li updated SENTRY-2024:
--
Summary: Drop Index that includes AUTHZ_OBJ_NAME (was: Specify Char Set
for AUTHZ_OBJ_NAME)
> Drop Index that includes AUTHZ_OBJ_NAME
> ---
>
> Key: SENTRY-2024
> URL: https://issues.apache.org/jira/browse/SENTRY-2024
> Project: Sentry
> Issue Type: Bug
> Components: Sentry
>Affects Versions: 2.0.0
>Reporter: Na Li
>Assignee: Na Li
>Priority: Critical
> Attachments: SENTRY-2024.001.patch, SENTRY-2024.001.patch,
> SENTRY-2024.001.patch, SENTRY-2024.002.patch, SENTRY-2024.002.patch
>
>
> AUTHZ_OBJ_NAME has (384) chars. It is constructed by sentry from notification
> event as {color:red}DB_NAME + "." + TBL_NAME{color}. To be consistent with
> hive, sentry should use the same char set as what's used by DB_NAME and
> TBL_NAME
> Besides, if its table char set is utf8, the constrain AUTHZOBJNAME will have
> error "Specified key was too long; max key length is 767 bytes" for mysql.
> The solution is to specify the char set for this field, so it works for mysql
> regardless the char set of its table or DB.
> Reference:
> 1) CREATE TABLE AUTHZ_PATHS_MAPPING
> (
> AUTHZ_OBJ_ID BIGINT NOT NULL generated always as identity (start with 1),
> AUTHZ_OBJ_NAME VARCHAR({color:red}384{color}),
> CREATE_TIME_MS BIGINT NOT NULL
> );
> CREATE UNIQUE INDEX AUTHZOBJNAME ON AUTHZ_PATHS_MAPPING
> ({color:red}AUTHZ_OBJ_NAME{color});
> 2) AUTHZ_OBJ_NAME is constructed by sentry from notification event as
> DB_NAME + "." + TBL_NAME. To be consistent with hive, sentry should use the
> same char set as what's used by DB_NAME and TBL_NAME in
> NotificationProcessor.processAlterTable().
> 3) hive uses latin1 as default charset for notification event.
> CREATE TABLE IF NOT EXISTS `NOTIFICATION_LOG`
> (
> `NL_ID` BIGINT(20) NOT NULL,
> `EVENT_ID` BIGINT(20) NOT NULL,
> `EVENT_TIME` INT(11) NOT NULL,
> `EVENT_TYPE` varchar(32) NOT NULL,
> `DB_NAME` varchar(128),
> `TBL_NAME` varchar(128),
> `MESSAGE` mediumtext,
> PRIMARY KEY (`NL_ID`)
> ) ENGINE=InnoDB DEFAULT CHARSET=latin1;
> 4) actual installation shows both fields are in latin1 char set.
> SHOW FULL COLUMNS FROM NOTIFICATION_LOG;
> ++--+---+--+-+-+---+-+-+
> | Field | Type | Collation | Null | Key | Default |
> Extra | Privileges | Comment |
> ++--+---+--+-+-+---+-+-+
> | NL_ID | bigint(20) | NULL | NO | PRI | NULL|
> | select,insert,update,references | |
> | EVENT_ID | bigint(20) | NULL | NO | | NULL|
> | select,insert,update,references | |
> | EVENT_TIME | int(11) | NULL | NO | | NULL|
> | select,insert,update,references | |
> | EVENT_TYPE | varchar(32) | latin1_swedish_ci | NO | | NULL|
> | select,insert,update,references | |
> | DB_NAME| varchar(128) | latin1_swedish_ci | YES | | NULL|
> | select,insert,update,references | |
> | TBL_NAME | varchar(128) | latin1_swedish_ci | YES | | NULL|
> | select,insert,update,references | |
> | MESSAGE| mediumtext | latin1_swedish_ci | YES | | NULL|
> | select,insert,update,references | |
> ++--+---+--+-+-+---+-+-+
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (SENTRY-2024) Specify Char Set for AUTHZ_OBJ_NAME
[
https://issues.apache.org/jira/browse/SENTRY-2024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16261670#comment-16261670
]
Na Li commented on SENTRY-2024:
---
I am OK to drop this index, so we don't have issue for MySql server. I will
attach a patch for review
> Specify Char Set for AUTHZ_OBJ_NAME
> ---
>
> Key: SENTRY-2024
> URL: https://issues.apache.org/jira/browse/SENTRY-2024
> Project: Sentry
> Issue Type: Bug
> Components: Sentry
>Affects Versions: 2.0.0
>Reporter: Na Li
>Assignee: Na Li
>Priority: Critical
> Attachments: SENTRY-2024.001.patch, SENTRY-2024.001.patch,
> SENTRY-2024.001.patch, SENTRY-2024.002.patch, SENTRY-2024.002.patch
>
>
> AUTHZ_OBJ_NAME has (384) chars. It is constructed by sentry from notification
> event as {color:red}DB_NAME + "." + TBL_NAME{color}. To be consistent with
> hive, sentry should use the same char set as what's used by DB_NAME and
> TBL_NAME
> Besides, if its table char set is utf8, the constrain AUTHZOBJNAME will have
> error "Specified key was too long; max key length is 767 bytes" for mysql.
> The solution is to specify the char set for this field, so it works for mysql
> regardless the char set of its table or DB.
> Reference:
> 1) CREATE TABLE AUTHZ_PATHS_MAPPING
> (
> AUTHZ_OBJ_ID BIGINT NOT NULL generated always as identity (start with 1),
> AUTHZ_OBJ_NAME VARCHAR({color:red}384{color}),
> CREATE_TIME_MS BIGINT NOT NULL
> );
> CREATE UNIQUE INDEX AUTHZOBJNAME ON AUTHZ_PATHS_MAPPING
> ({color:red}AUTHZ_OBJ_NAME{color});
> 2) AUTHZ_OBJ_NAME is constructed by sentry from notification event as
> DB_NAME + "." + TBL_NAME. To be consistent with hive, sentry should use the
> same char set as what's used by DB_NAME and TBL_NAME in
> NotificationProcessor.processAlterTable().
> 3) hive uses latin1 as default charset for notification event.
> CREATE TABLE IF NOT EXISTS `NOTIFICATION_LOG`
> (
> `NL_ID` BIGINT(20) NOT NULL,
> `EVENT_ID` BIGINT(20) NOT NULL,
> `EVENT_TIME` INT(11) NOT NULL,
> `EVENT_TYPE` varchar(32) NOT NULL,
> `DB_NAME` varchar(128),
> `TBL_NAME` varchar(128),
> `MESSAGE` mediumtext,
> PRIMARY KEY (`NL_ID`)
> ) ENGINE=InnoDB DEFAULT CHARSET=latin1;
> 4) actual installation shows both fields are in latin1 char set.
> SHOW FULL COLUMNS FROM NOTIFICATION_LOG;
> ++--+---+--+-+-+---+-+-+
> | Field | Type | Collation | Null | Key | Default |
> Extra | Privileges | Comment |
> ++--+---+--+-+-+---+-+-+
> | NL_ID | bigint(20) | NULL | NO | PRI | NULL|
> | select,insert,update,references | |
> | EVENT_ID | bigint(20) | NULL | NO | | NULL|
> | select,insert,update,references | |
> | EVENT_TIME | int(11) | NULL | NO | | NULL|
> | select,insert,update,references | |
> | EVENT_TYPE | varchar(32) | latin1_swedish_ci | NO | | NULL|
> | select,insert,update,references | |
> | DB_NAME| varchar(128) | latin1_swedish_ci | YES | | NULL|
> | select,insert,update,references | |
> | TBL_NAME | varchar(128) | latin1_swedish_ci | YES | | NULL|
> | select,insert,update,references | |
> | MESSAGE| mediumtext | latin1_swedish_ci | YES | | NULL|
> | select,insert,update,references | |
> ++--+---+--+-+-+---+-+-+
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (SENTRY-2066) DB name is not set for AlterTable
[
https://issues.apache.org/jira/browse/SENTRY-2066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Na Li updated SENTRY-2066:
--
Attachment: SENTRY-2066.003.patch
v3. Simplify logic and remove local variable isDBSet
> DB name is not set for AlterTable
> --
>
> Key: SENTRY-2066
> URL: https://issues.apache.org/jira/browse/SENTRY-2066
> Project: Sentry
> Issue Type: Bug
> Components: Sentry
>Affects Versions: 2.0.0
>Reporter: Na Li
>Assignee: Na Li
>Priority: Critical
> Labels: 2.0
> Attachments: SENTRY-2066.001.patch, SENTRY-2066.002.patch,
> SENTRY-2066.003.patch
>
>
> SENTRY-1087 introduced a regression: the database name is not set in alter
> table column command.
> SENTRY-1087 added a new case below, and it causes the currDB not set because
> its children are "TOK_TABNAME" and "TOK_ALTERTABLE_RENAMECOL" NONE IS
> "TOK_ALTERTABLE_SERIALIZER"
> {code}
> case HiveParser.TOK_ALTERTABLE:
>
> for (Node childNode : ast.getChildren()) {
> ASTNode childASTNode = (ASTNode) childNode;
> if
> ("TOK_ALTERTABLE_SERIALIZER".equals(childASTNode.getText())) {
> ASTNode serdeNode = (ASTNode) childASTNode.getChild(0);
> String serdeClassName =
> BaseSemanticAnalyzer.unescapeSQLString(serdeNode.getText());
> setSerdeURI(serdeClassName);
> currDB = getCanonicalDb();
> }
> }
> break;
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (SENTRY-2067) Refactor HMS Follower syncupWithHms method
[ https://issues.apache.org/jira/browse/SENTRY-2067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16261473#comment-16261473 ] Na Li commented on SENTRY-2067: --- If the notification table is empty, we should get a full snapshot even when HDFS sync is not enabled. Once the full snapshot is done, the notification table is not empty, and no more full snapshot. In this way, When hive creates a table, and waits for sentry to sync the update. What's wrong about this? > Refactor HMS Follower syncupWithHms method > -- > > Key: SENTRY-2067 > URL: https://issues.apache.org/jira/browse/SENTRY-2067 > Project: Sentry > Issue Type: Bug > Components: Sentry >Reporter: Arjun Mishra >Assignee: Arjun Mishra > Labels: bite-sized > Fix For: 2.0.0 > > > Currently we seem to be getting a full snapshot even when HDFS Sync is > disabled. A full snapshot can be very time consuming and puts a lot of strain > on resources and it doesn't make sense to have to get it without persisting > it. Code needs to be refactored to handle all cases with HDFS Sync enabled > and disabled appropriately -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (SENTRY-2024) Specify Char Set for AUTHZ_OBJ_NAME
[
https://issues.apache.org/jira/browse/SENTRY-2024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16261144#comment-16261144
]
Na Li commented on SENTRY-2024:
---
[~spena] getMAuthzPathsMappingCore() is to get he paths associated with a
specific authzObjName. auth_snapshot_id is increased whenever we have a new
full snapshot, and all authzObjName share that ID. The 2nd index for the
authz_obj_name is definitely necessary.
> Specify Char Set for AUTHZ_OBJ_NAME
> ---
>
> Key: SENTRY-2024
> URL: https://issues.apache.org/jira/browse/SENTRY-2024
> Project: Sentry
> Issue Type: Bug
> Components: Sentry
>Affects Versions: 2.0.0
>Reporter: Na Li
>Assignee: Na Li
>Priority: Critical
> Attachments: SENTRY-2024.001.patch, SENTRY-2024.001.patch,
> SENTRY-2024.001.patch, SENTRY-2024.002.patch, SENTRY-2024.002.patch
>
>
> AUTHZ_OBJ_NAME has (384) chars. It is constructed by sentry from notification
> event as {color:red}DB_NAME + "." + TBL_NAME{color}. To be consistent with
> hive, sentry should use the same char set as what's used by DB_NAME and
> TBL_NAME
> Besides, if its table char set is utf8, the constrain AUTHZOBJNAME will have
> error "Specified key was too long; max key length is 767 bytes" for mysql.
> The solution is to specify the char set for this field, so it works for mysql
> regardless the char set of its table or DB.
> Reference:
> 1) CREATE TABLE AUTHZ_PATHS_MAPPING
> (
> AUTHZ_OBJ_ID BIGINT NOT NULL generated always as identity (start with 1),
> AUTHZ_OBJ_NAME VARCHAR({color:red}384{color}),
> CREATE_TIME_MS BIGINT NOT NULL
> );
> CREATE UNIQUE INDEX AUTHZOBJNAME ON AUTHZ_PATHS_MAPPING
> ({color:red}AUTHZ_OBJ_NAME{color});
> 2) AUTHZ_OBJ_NAME is constructed by sentry from notification event as
> DB_NAME + "." + TBL_NAME. To be consistent with hive, sentry should use the
> same char set as what's used by DB_NAME and TBL_NAME in
> NotificationProcessor.processAlterTable().
> 3) hive uses latin1 as default charset for notification event.
> CREATE TABLE IF NOT EXISTS `NOTIFICATION_LOG`
> (
> `NL_ID` BIGINT(20) NOT NULL,
> `EVENT_ID` BIGINT(20) NOT NULL,
> `EVENT_TIME` INT(11) NOT NULL,
> `EVENT_TYPE` varchar(32) NOT NULL,
> `DB_NAME` varchar(128),
> `TBL_NAME` varchar(128),
> `MESSAGE` mediumtext,
> PRIMARY KEY (`NL_ID`)
> ) ENGINE=InnoDB DEFAULT CHARSET=latin1;
> 4) actual installation shows both fields are in latin1 char set.
> SHOW FULL COLUMNS FROM NOTIFICATION_LOG;
> ++--+---+--+-+-+---+-+-+
> | Field | Type | Collation | Null | Key | Default |
> Extra | Privileges | Comment |
> ++--+---+--+-+-+---+-+-+
> | NL_ID | bigint(20) | NULL | NO | PRI | NULL|
> | select,insert,update,references | |
> | EVENT_ID | bigint(20) | NULL | NO | | NULL|
> | select,insert,update,references | |
> | EVENT_TIME | int(11) | NULL | NO | | NULL|
> | select,insert,update,references | |
> | EVENT_TYPE | varchar(32) | latin1_swedish_ci | NO | | NULL|
> | select,insert,update,references | |
> | DB_NAME| varchar(128) | latin1_swedish_ci | YES | | NULL|
> | select,insert,update,references | |
> | TBL_NAME | varchar(128) | latin1_swedish_ci | YES | | NULL|
> | select,insert,update,references | |
> | MESSAGE| mediumtext | latin1_swedish_ci | YES | | NULL|
> | select,insert,update,references | |
> ++--+---+--+-+-+---+-+-+
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (SENTRY-2065) Timestamp in MSentryPathChange and MSentryPermChange should be set by DB server
[ https://issues.apache.org/jira/browse/SENTRY-2065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16261085#comment-16261085 ] Na Li commented on SENTRY-2065: --- We can make DB to assign the timestamps in all entries saved in sentry DB. In this way, it is more consistent. > Timestamp in MSentryPathChange and MSentryPermChange should be set by DB > server > --- > > Key: SENTRY-2065 > URL: https://issues.apache.org/jira/browse/SENTRY-2065 > Project: Sentry > Issue Type: Sub-task > Components: Sentry >Affects Versions: 2.0.0 >Reporter: Na Li >Assignee: Na Li > Fix For: 2.0.0 > > > The timestamp of the path change and perm change entries should be set by DB > server, so when there are multiple sentry servers, the timestamp of the > entries are from the single source, i.e., no need to require the time sync > among sentry servers. When sentry retrieve those entries for HDFS, sentry can > retrieve the entries after the given timestamp from request. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (SENTRY-2066) DB name is not set for AlterTable
[
https://issues.apache.org/jira/browse/SENTRY-2066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Na Li updated SENTRY-2066:
--
Attachment: SENTRY-2066.002.patch
v2. make error message in test more meaningful
> DB name is not set for AlterTable
> --
>
> Key: SENTRY-2066
> URL: https://issues.apache.org/jira/browse/SENTRY-2066
> Project: Sentry
> Issue Type: Bug
> Components: Sentry
>Affects Versions: 2.0.0
>Reporter: Na Li
>Assignee: Na Li
>Priority: Critical
> Labels: 2.0
> Attachments: SENTRY-2066.001.patch, SENTRY-2066.002.patch
>
>
> SENTRY-1087 introduced a regression: the database name is not set in alter
> table column command.
> SENTRY-1087 added a new case below, and it causes the currDB not set because
> its children are "TOK_TABNAME" and "TOK_ALTERTABLE_RENAMECOL" NONE IS
> "TOK_ALTERTABLE_SERIALIZER"
> {code}
> case HiveParser.TOK_ALTERTABLE:
>
> for (Node childNode : ast.getChildren()) {
> ASTNode childASTNode = (ASTNode) childNode;
> if
> ("TOK_ALTERTABLE_SERIALIZER".equals(childASTNode.getText())) {
> ASTNode serdeNode = (ASTNode) childASTNode.getChild(0);
> String serdeClassName =
> BaseSemanticAnalyzer.unescapeSQLString(serdeNode.getText());
> setSerdeURI(serdeClassName);
> currDB = getCanonicalDb();
> }
> }
> break;
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (SENTRY-2064) Sentry client at HDFS should send back the timestamp in last response
[
https://issues.apache.org/jira/browse/SENTRY-2064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16261047#comment-16261047
]
Na Li commented on SENTRY-2064:
---
The proposed Thrift changes
const string UNUSED_TIME_DATASTORE = "0001-01-01 01:01:01";
struct TPathsUpdate {
1: required bool hasFullImage;
2: optional TPathsDump pathsDump;
3: required i64 seqNum;
4: required list pathChanges;
5: optional i64 imgNum = UNUSED_PATH_UPDATE_IMG_NUM;
6: optional string currTimeAtDataStore = UNUSED_TIME_DATASTORE; <- new field
}
struct TPermissionsUpdate {
1: required bool hasfullImage;
2: required i64 seqNum;
3: required map privilegeChanges;
4: required map roleChanges;
5: required string currTimeAtDataStore = UNUSED_TIME_DATASTORE; <- new field
}
struct TAuthzUpdateRequest {
1: required i64 permSeqNum;
2: required i64 pathSeqNum;
3: required i64 pathImgNum;
4: required string permLastTimeAtDataStore = UNUSED_TIME_DATASTORE; <- new
field
5: required string pathLastTimeAtDataStore = UNUSED_TIME_DATASTORE;<- new
field
}
> Sentry client at HDFS should send back the timestamp in last response
> -
>
> Key: SENTRY-2064
> URL: https://issues.apache.org/jira/browse/SENTRY-2064
> Project: Sentry
> Issue Type: Sub-task
> Components: Sentry
>Reporter: Na Li
>Assignee: Na Li
> Fix For: 2.0.0
>
>
> When sentry server sends response to HDFS, it includes the timestamp of the
> current time at the DB server.
> When sentry client at HDFS gets response from sentry, it sets its local
> variable, which is the latest timestamp received from sentry, to be the
> timestamp in the response
> When sentry client sends request, it should include the latest timestamp
> received from sentry in the request.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Comment Edited] (SENTRY-2064) Sentry client at HDFS should send back the timestamp in last response
[
https://issues.apache.org/jira/browse/SENTRY-2064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16261047#comment-16261047
]
Na Li edited comment on SENTRY-2064 at 11/21/17 4:59 PM:
-
The proposed Thrift changes
{code}
const string UNUSED_TIME_DATASTORE = "0001-01-01 01:01:01";
struct TPathsUpdate {
1: required bool hasFullImage;
2: optional TPathsDump pathsDump;
3: required i64 seqNum;
4: required list pathChanges;
5: optional i64 imgNum = UNUSED_PATH_UPDATE_IMG_NUM;
6: optional string currTimeAtDataStore = UNUSED_TIME_DATASTORE; <- new field
}
struct TPermissionsUpdate {
1: required bool hasfullImage;
2: required i64 seqNum;
3: required map privilegeChanges;
4: required map roleChanges;
5: required string currTimeAtDataStore = UNUSED_TIME_DATASTORE; <- new field
}
struct TAuthzUpdateRequest {
1: required i64 permSeqNum;
2: required i64 pathSeqNum;
3: required i64 pathImgNum;
4: required string permLastTimeAtDataStore = UNUSED_TIME_DATASTORE; <- new
field
5: required string pathLastTimeAtDataStore = UNUSED_TIME_DATASTORE;<- new
field
}
{code}
was (Author: linaataustin):
The proposed Thrift changes
const string UNUSED_TIME_DATASTORE = "0001-01-01 01:01:01";
struct TPathsUpdate {
1: required bool hasFullImage;
2: optional TPathsDump pathsDump;
3: required i64 seqNum;
4: required list pathChanges;
5: optional i64 imgNum = UNUSED_PATH_UPDATE_IMG_NUM;
6: optional string currTimeAtDataStore = UNUSED_TIME_DATASTORE; <- new field
}
struct TPermissionsUpdate {
1: required bool hasfullImage;
2: required i64 seqNum;
3: required map privilegeChanges;
4: required map roleChanges;
5: required string currTimeAtDataStore = UNUSED_TIME_DATASTORE; <- new field
}
struct TAuthzUpdateRequest {
1: required i64 permSeqNum;
2: required i64 pathSeqNum;
3: required i64 pathImgNum;
4: required string permLastTimeAtDataStore = UNUSED_TIME_DATASTORE; <- new
field
5: required string pathLastTimeAtDataStore = UNUSED_TIME_DATASTORE;<- new
field
}
> Sentry client at HDFS should send back the timestamp in last response
> -
>
> Key: SENTRY-2064
> URL: https://issues.apache.org/jira/browse/SENTRY-2064
> Project: Sentry
> Issue Type: Sub-task
> Components: Sentry
>Reporter: Na Li
>Assignee: Na Li
> Fix For: 2.0.0
>
>
> When sentry server sends response to HDFS, it includes the timestamp of the
> current time at the DB server.
> When sentry client at HDFS gets response from sentry, it sets its local
> variable, which is the latest timestamp received from sentry, to be the
> timestamp in the response
> When sentry client sends request, it should include the latest timestamp
> received from sentry in the request.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (SENTRY-2064) Sentry client at HDFS should send back the timestamp in last response
[ https://issues.apache.org/jira/browse/SENTRY-2064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Na Li updated SENTRY-2064: -- Description: When sentry server sends response to HDFS, it includes the timestamp of the current time at the DB server. When sentry client at HDFS gets response from sentry, it sets its local variable, which is the latest timestamp received from sentry, to be the timestamp in the response When sentry client sends request, it should include the latest timestamp received from sentry in the request. was:When sentry client at HDFS gets response from sentry, it should update the timestamp. When sending request, it should include the timestamp. > Sentry client at HDFS should send back the timestamp in last response > - > > Key: SENTRY-2064 > URL: https://issues.apache.org/jira/browse/SENTRY-2064 > Project: Sentry > Issue Type: Sub-task > Components: Sentry >Reporter: Na Li >Assignee: Na Li > Fix For: 2.0.0 > > > When sentry server sends response to HDFS, it includes the timestamp of the > current time at the DB server. > When sentry client at HDFS gets response from sentry, it sets its local > variable, which is the latest timestamp received from sentry, to be the > timestamp in the response > When sentry client sends request, it should include the latest timestamp > received from sentry in the request. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (SENTRY-2066) DB name is not set for AlterTable
[
https://issues.apache.org/jira/browse/SENTRY-2066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Na Li updated SENTRY-2066:
--
Status: Patch Available (was: Open)
> DB name is not set for AlterTable
> --
>
> Key: SENTRY-2066
> URL: https://issues.apache.org/jira/browse/SENTRY-2066
> Project: Sentry
> Issue Type: Bug
> Components: Sentry
>Affects Versions: 2.0.0
>Reporter: Na Li
>Assignee: Na Li
>Priority: Critical
> Labels: 2.0
> Attachments: SENTRY-2066.001.patch
>
>
> SENTRY-1087 introduced a regression: the database name is not set in alter
> table column command.
> SENTRY-1087 added a new case below, and it causes the currDB not set because
> its children are "TOK_TABNAME" and "TOK_ALTERTABLE_RENAMECOL" NONE IS
> "TOK_ALTERTABLE_SERIALIZER"
> {code}
> case HiveParser.TOK_ALTERTABLE:
>
> for (Node childNode : ast.getChildren()) {
> ASTNode childASTNode = (ASTNode) childNode;
> if
> ("TOK_ALTERTABLE_SERIALIZER".equals(childASTNode.getText())) {
> ASTNode serdeNode = (ASTNode) childASTNode.getChild(0);
> String serdeClassName =
> BaseSemanticAnalyzer.unescapeSQLString(serdeNode.getText());
> setSerdeURI(serdeClassName);
> currDB = getCanonicalDb();
> }
> }
> break;
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (SENTRY-2066) DB name is not set for AlterTable
[
https://issues.apache.org/jira/browse/SENTRY-2066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Na Li updated SENTRY-2066:
--
Attachment: SENTRY-2066.001.patch
> DB name is not set for AlterTable
> --
>
> Key: SENTRY-2066
> URL: https://issues.apache.org/jira/browse/SENTRY-2066
> Project: Sentry
> Issue Type: Bug
> Components: Sentry
>Affects Versions: 2.0.0
>Reporter: Na Li
>Assignee: Na Li
>Priority: Critical
> Labels: 2.0
> Attachments: SENTRY-2066.001.patch
>
>
> SENTRY-1087 introduced a regression: the database name is not set in alter
> table column command.
> SENTRY-1087 added a new case below, and it causes the currDB not set because
> its children are "TOK_TABNAME" and "TOK_ALTERTABLE_RENAMECOL" NONE IS
> "TOK_ALTERTABLE_SERIALIZER"
> {code}
> case HiveParser.TOK_ALTERTABLE:
>
> for (Node childNode : ast.getChildren()) {
> ASTNode childASTNode = (ASTNode) childNode;
> if
> ("TOK_ALTERTABLE_SERIALIZER".equals(childASTNode.getText())) {
> ASTNode serdeNode = (ASTNode) childASTNode.getChild(0);
> String serdeClassName =
> BaseSemanticAnalyzer.unescapeSQLString(serdeNode.getText());
> setSerdeURI(serdeClassName);
> currDB = getCanonicalDb();
> }
> }
> break;
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (SENTRY-2066) DB name is not set for AlterTable
Na Li created SENTRY-2066:
-
Summary: DB name is not set for AlterTable
Key: SENTRY-2066
URL: https://issues.apache.org/jira/browse/SENTRY-2066
Project: Sentry
Issue Type: Bug
Components: Sentry
Affects Versions: 2.0.0
Reporter: Na Li
Assignee: Na Li
Priority: Critical
SENTRY-1087 introduced a regression: the database name is not set in alter
table column command.
SENTRY-1087 added a new case below, and it causes the currDB not set because
its children are "TOK_TABNAME" and "TOK_ALTERTABLE_RENAMECOL" NONE IS
"TOK_ALTERTABLE_SERIALIZER"
{code}
case HiveParser.TOK_ALTERTABLE:
for (Node childNode : ast.getChildren()) {
ASTNode childASTNode = (ASTNode) childNode;
if ("TOK_ALTERTABLE_SERIALIZER".equals(childASTNode.getText()))
{
ASTNode serdeNode = (ASTNode) childASTNode.getChild(0);
String serdeClassName =
BaseSemanticAnalyzer.unescapeSQLString(serdeNode.getText());
setSerdeURI(serdeClassName);
currDB = getCanonicalDb();
}
}
break;
{code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (SENTRY-2065) Timestamp in MSentryPathChange and MSentryPermChange should be set by DB server
Na Li created SENTRY-2065: - Summary: Timestamp in MSentryPathChange and MSentryPermChange should be set by DB server Key: SENTRY-2065 URL: https://issues.apache.org/jira/browse/SENTRY-2065 Project: Sentry Issue Type: Sub-task Components: Sentry Affects Versions: 2.0.0 Reporter: Na Li Assignee: Na Li The timestamp of the path change and perm change entries should be set by DB server, so when there are multiple sentry servers, the timestamp of the entries are from the single source, i.e., no need to require the time sync among sentry servers. When sentry retrieve those entries for HDFS, sentry can retrieve the entries after the given timestamp from request. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (SENTRY-2064) Sentry client at HDFS should send back the timestamp in last response
Na Li created SENTRY-2064: - Summary: Sentry client at HDFS should send back the timestamp in last response Key: SENTRY-2064 URL: https://issues.apache.org/jira/browse/SENTRY-2064 Project: Sentry Issue Type: Sub-task Reporter: Na Li When sentry client at HDFS gets response from sentry, it should update the timestamp. When sending request, it should include the timestamp. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (SENTRY-2063) Add timestamp in Thrift API for HDFS integration
Na Li created SENTRY-2063: - Summary: Add timestamp in Thrift API for HDFS integration Key: SENTRY-2063 URL: https://issues.apache.org/jira/browse/SENTRY-2063 Project: Sentry Issue Type: Sub-task Components: Sentry Affects Versions: 2.0.0 Reporter: Na Li Assignee: Na Li Add timestamp field in HDFS request to sentry and response from sentry. In this way, sentry server can find when the HDFS client got update, and gets all updates after that time. In this way, we can avoid missing any update -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (SENTRY-2024) Specify Char Set for AUTHZ_OBJ_NAME
[
https://issues.apache.org/jira/browse/SENTRY-2024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16259569#comment-16259569
]
Na Li commented on SENTRY-2024:
---
[~spena] MySql 5.7.20 does not have this issue. So MySql server version decides
if a key length can exceeds 767 bytes. If we support 5.6, then we still need
this fix.
mysql> CREATE TABLE `AUTHZ_PATHS_MAPPING` ( `AUTHZ_OBJ_ID` BIGINT NOT NULL,
`AUTHZ_OBJ_NAME` VARCHAR(384) BINARY CHARACTER SET utf8 COLLATE utf8_bin NOT
NULL, `CREATE_TIME_MS` BIGINT NOT NULL, `AUTHZ_SNAPSHOT_ID` BIGINT NOT NULL)
ENGINE=INNODB;
Query OK, 0 rows affected (0.03 sec)
mysql> CREATE UNIQUE INDEX `AUTHZOBJNAMEID` ON `AUTHZ_PATHS_MAPPING`
(`AUTHZ_OBJ_NAME`, `AUTHZ_SNAPSHOT_ID`);
Query OK, 0 rows affected (0.04 sec)
Records: 0 Duplicates: 0 Warnings: 0
mysql> SELECT @@version;
+---+
| @@version |
+---+
| 5.7.20|
+---+
1 row in set (0.00 sec)
> Specify Char Set for AUTHZ_OBJ_NAME
> ---
>
> Key: SENTRY-2024
> URL: https://issues.apache.org/jira/browse/SENTRY-2024
> Project: Sentry
> Issue Type: Bug
> Components: Sentry
>Affects Versions: 2.0.0
>Reporter: Na Li
>Assignee: Na Li
>Priority: Critical
> Attachments: SENTRY-2024.001.patch, SENTRY-2024.001.patch,
> SENTRY-2024.001.patch, SENTRY-2024.002.patch, SENTRY-2024.002.patch
>
>
> AUTHZ_OBJ_NAME has (384) chars. It is constructed by sentry from notification
> event as {color:red}DB_NAME + "." + TBL_NAME{color}. To be consistent with
> hive, sentry should use the same char set as what's used by DB_NAME and
> TBL_NAME
> Besides, if its table char set is utf8, the constrain AUTHZOBJNAME will have
> error "Specified key was too long; max key length is 767 bytes" for mysql.
> The solution is to specify the char set for this field, so it works for mysql
> regardless the char set of its table or DB.
> Reference:
> 1) CREATE TABLE AUTHZ_PATHS_MAPPING
> (
> AUTHZ_OBJ_ID BIGINT NOT NULL generated always as identity (start with 1),
> AUTHZ_OBJ_NAME VARCHAR({color:red}384{color}),
> CREATE_TIME_MS BIGINT NOT NULL
> );
> CREATE UNIQUE INDEX AUTHZOBJNAME ON AUTHZ_PATHS_MAPPING
> ({color:red}AUTHZ_OBJ_NAME{color});
> 2) AUTHZ_OBJ_NAME is constructed by sentry from notification event as
> DB_NAME + "." + TBL_NAME. To be consistent with hive, sentry should use the
> same char set as what's used by DB_NAME and TBL_NAME in
> NotificationProcessor.processAlterTable().
> 3) hive uses latin1 as default charset for notification event.
> CREATE TABLE IF NOT EXISTS `NOTIFICATION_LOG`
> (
> `NL_ID` BIGINT(20) NOT NULL,
> `EVENT_ID` BIGINT(20) NOT NULL,
> `EVENT_TIME` INT(11) NOT NULL,
> `EVENT_TYPE` varchar(32) NOT NULL,
> `DB_NAME` varchar(128),
> `TBL_NAME` varchar(128),
> `MESSAGE` mediumtext,
> PRIMARY KEY (`NL_ID`)
> ) ENGINE=InnoDB DEFAULT CHARSET=latin1;
> 4) actual installation shows both fields are in latin1 char set.
> SHOW FULL COLUMNS FROM NOTIFICATION_LOG;
> ++--+---+--+-+-+---+-+-+
> | Field | Type | Collation | Null | Key | Default |
> Extra | Privileges | Comment |
> ++--+---+--+-+-+---+-+-+
> | NL_ID | bigint(20) | NULL | NO | PRI | NULL|
> | select,insert,update,references | |
> | EVENT_ID | bigint(20) | NULL | NO | | NULL|
> | select,insert,update,references | |
> | EVENT_TIME | int(11) | NULL | NO | | NULL|
> | select,insert,update,references | |
> | EVENT_TYPE | varchar(32) | latin1_swedish_ci | NO | | NULL|
> | select,insert,update,references | |
> | DB_NAME| varchar(128) | latin1_swedish_ci | YES | | NULL|
> | select,insert,update,references | |
> | TBL_NAME | varchar(128) | latin1_swedish_ci | YES | | NULL|
> | select,insert,update,references | |
> | MESSAGE| mediumtext | latin1_swedish_ci | YES | | NULL|
> | select,insert,update,references | |
> ++--+---+--+-+-+---+-+-+
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (SENTRY-2024) Specify Char Set for AUTHZ_OBJ_NAME
[
https://issues.apache.org/jira/browse/SENTRY-2024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16257795#comment-16257795
]
Na Li commented on SENTRY-2024:
---
[~spena] This is test on 5.5.56-MariaDB, and it also failed to create the
index. Without this fix, some of MySql servers will have this issue.
MariaDB [sentryTest]> SELECT @@character_set_database, @@collation_database;
+--+--+
| @@character_set_database | @@collation_database |
+--+--+
| utf8 | utf8_general_ci |
+--+--+
1 row in set (0.00 sec)
MariaDB [sentryTest]> SHOW GLOBAL VARIABLES LIKE 'innodb_large%';
+-+---+
| Variable_name | Value |
+-+---+
| innodb_large_prefix | OFF |
+-+---+
1 row in set (0.02 sec)
MariaDB [sentryTest]> SELECT @@version;
++
| @@version |
++
| 5.5.56-MariaDB |
++
1 row in set (0.00 sec)
MariaDB [sentryTest]> CREATE TABLE `AUTHZ_PATHS_MAPPING_NOBINARY`
-> (
-> `AUTHZ_OBJ_ID` BIGINT NOT NULL AUTO_INCREMENT,
-> `AUTHZ_OBJ_NAME` VARCHAR(384) CHARACTER SET utf8 COLLATE utf8_bin
NOT NULL,
-> `CREATE_TIME_MS` BIGINT NOT NULL,
-> `AUTHZ_SNAPSHOT_ID` BIGINT NOT NULL,
-> CONSTRAINT `AUTHZ_PATHS_MAPPING_PK` PRIMARY KEY (`AUTHZ_OBJ_ID`)
-> ) ENGINE=INNODB;
Query OK, 0 rows affected (0.09 sec)
MariaDB [sentryTest]> CREATE UNIQUE INDEX `AUTHZOBJNAMEID` ON
`AUTHZ_PATHS_MAPPING_NOBINARY` (`AUTHZ_OBJ_NAME`, `AUTHZ_SNAPSHOT_ID`);
ERROR 1071 (42000): {color:red}Specified key was too long; max key length is
767 bytes{color}
MariaDB [sentryTest]>
> Specify Char Set for AUTHZ_OBJ_NAME
> ---
>
> Key: SENTRY-2024
> URL: https://issues.apache.org/jira/browse/SENTRY-2024
> Project: Sentry
> Issue Type: Bug
> Components: Sentry
>Affects Versions: 2.0.0
>Reporter: Na Li
>Assignee: Na Li
>Priority: Critical
> Attachments: SENTRY-2024.001.patch, SENTRY-2024.001.patch,
> SENTRY-2024.001.patch, SENTRY-2024.002.patch, SENTRY-2024.002.patch
>
>
> AUTHZ_OBJ_NAME has (384) chars. It is constructed by sentry from notification
> event as {color:red}DB_NAME + "." + TBL_NAME{color}. To be consistent with
> hive, sentry should use the same char set as what's used by DB_NAME and
> TBL_NAME
> Besides, if its table char set is utf8, the constrain AUTHZOBJNAME will have
> error "Specified key was too long; max key length is 767 bytes" for mysql.
> The solution is to specify the char set for this field, so it works for mysql
> regardless the char set of its table or DB.
> Reference:
> 1) CREATE TABLE AUTHZ_PATHS_MAPPING
> (
> AUTHZ_OBJ_ID BIGINT NOT NULL generated always as identity (start with 1),
> AUTHZ_OBJ_NAME VARCHAR({color:red}384{color}),
> CREATE_TIME_MS BIGINT NOT NULL
> );
> CREATE UNIQUE INDEX AUTHZOBJNAME ON AUTHZ_PATHS_MAPPING
> ({color:red}AUTHZ_OBJ_NAME{color});
> 2) AUTHZ_OBJ_NAME is constructed by sentry from notification event as
> DB_NAME + "." + TBL_NAME. To be consistent with hive, sentry should use the
> same char set as what's used by DB_NAME and TBL_NAME in
> NotificationProcessor.processAlterTable().
> 3) hive uses latin1 as default charset for notification event.
> CREATE TABLE IF NOT EXISTS `NOTIFICATION_LOG`
> (
> `NL_ID` BIGINT(20) NOT NULL,
> `EVENT_ID` BIGINT(20) NOT NULL,
> `EVENT_TIME` INT(11) NOT NULL,
> `EVENT_TYPE` varchar(32) NOT NULL,
> `DB_NAME` varchar(128),
> `TBL_NAME` varchar(128),
> `MESSAGE` mediumtext,
> PRIMARY KEY (`NL_ID`)
> ) ENGINE=InnoDB DEFAULT CHARSET=latin1;
> 4) actual installation shows both fields are in latin1 char set.
> SHOW FULL COLUMNS FROM NOTIFICATION_LOG;
> ++--+---+--+-+-+---+-+-+
> | Field | Type | Collation | Null | Key | Default |
> Extra | Privileges | Comment |
> ++--+---+--+-+-+---+-+-+
> | NL_ID | bigint(20) | NULL | NO | PRI | NULL|
> | select,insert,update,references | |
> | EVENT_ID | bigint(20) | NULL | NO | | NULL|
> | select,insert,update,references | |
> | EVENT_TIME | int(11) | NULL | NO | | NULL|
> | select,insert,update,references | |
> | EVENT_TYPE | varchar(32) | latin1_swedish_ci | NO | | NULL|
> | select,insert,update,references | |
> | DB_NAME| varchar(128) | latin1_swedish_ci
[jira] [Commented] (SENTRY-1572) SentryMain() shouldn't dynamically load tool class
[
https://issues.apache.org/jira/browse/SENTRY-1572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16257608#comment-16257608
]
Na Li commented on SENTRY-1572:
---
[~xinrantinney] I think the reason the code load class is to avoid dependency
loop. If we want to load the class directly, you may consider to create a new
project for SentryMain, and let it depend on all other sentry projects. In this
way, you can avoid dependency loop.
> SentryMain() shouldn't dynamically load tool class
> --
>
> Key: SENTRY-1572
> URL: https://issues.apache.org/jira/browse/SENTRY-1572
> Project: Sentry
> Issue Type: Improvement
> Components: Sentry
>Reporter: Alexander Kolbasov
>Assignee: Xinran Tinney
> Labels: bite-sized, newbie
>
> TheSentryMain class currently works by mapping the command name to a Java
> class that is then dynamically loaded:
> {code}
> String commandName = commandLine.getOptionValue(COMMAND);
> String commandClazz = COMMANDS.get(commandName);
> Object command;
> try {
> command = Class.forName(commandClazz.trim()).newInstance();
> } catch (Exception e) {
> String msg = "Could not create instance of " + commandClazz + " for
> command " + commandName;
> throw new IllegalStateException(msg, e);
> }
> if (!(command instanceof Command)) {
> String msg = "Command " + command.getClass().getName() + " is not an
> instance of "
> + Command.class.getName();
> throw new IllegalStateException(msg);
> }
> ((Command)command).run(commandLine.getArgs());
> }
> {code}
> This ia too complicated and causes subtle problems at runtime. Instead it
> should just create a new instance of appropriate class and call it directly.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (SENTRY-1964) HDFS sync does not need partition locations (usually)
[ https://issues.apache.org/jira/browse/SENTRY-1964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Na Li updated SENTRY-1964: -- Resolution: Won't Fix Status: Resolved (was: Patch Available) This approach does not work in some scenario. Need to find other way to reduce the memory usage. > HDFS sync does not need partition locations (usually) > - > > Key: SENTRY-1964 > URL: https://issues.apache.org/jira/browse/SENTRY-1964 > Project: Sentry > Issue Type: Improvement > Components: Sentry >Affects Versions: 2.0.0 >Reporter: Na Li >Assignee: Na Li >Priority: Critical > Attachments: SENTRY-1964.001.patch, SENTRY-1964.001.patch, > SENTRY-1964.002.patch > > > Right now, sentry saves partition info from HMS and send it to HDFS. HDFS > only needs database and table info, and does not need partition info for ACL > unless the partion location is not sharing the same prefix of its table. > The partition data amount is huge, and causes performance issue. We can > optimize it by not saving and not sending partition info if it shares the > same path of its table. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (SENTRY-1964) HDFS sync does not need partition locations (usually)
[ https://issues.apache.org/jira/browse/SENTRY-1964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16256448#comment-16256448 ] Na Li commented on SENTRY-1964: --- [~akolb] the only problem is the scenario that the table location is initially prefix of the partition's local, and then changes to another location, but partition's location does not change. How often does this happen? Can we add configuration to control if we send partition to HDFS, and the default behavior is to send HDFS? In this way, for customer who really wants the performance improvement and does not run into the above scenario will be able to enjoy the benefit of not sending partition to HDFS. Later on, the component that user uses to make table location change can be smarter to avoid such situation. For example, when changing table location, will ask user to choose 1) change partition location as well to be sub-directory of table location, or 2) enable sending partition to HDFS. > HDFS sync does not need partition locations (usually) > - > > Key: SENTRY-1964 > URL: https://issues.apache.org/jira/browse/SENTRY-1964 > Project: Sentry > Issue Type: Improvement > Components: Sentry >Affects Versions: 2.0.0 >Reporter: Na Li >Assignee: Na Li >Priority: Critical > Attachments: SENTRY-1964.001.patch, SENTRY-1964.001.patch, > SENTRY-1964.002.patch > > > Right now, sentry saves partition info from HMS and send it to HDFS. HDFS > only needs database and table info, and does not need partition info for ACL > unless the partion location is not sharing the same prefix of its table. > The partition data amount is huge, and causes performance issue. We can > optimize it by not saving and not sending partition info if it shares the > same path of its table. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (SENTRY-1964) HDFS sync does not need partition locations (usually)
[
https://issues.apache.org/jira/browse/SENTRY-1964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16256443#comment-16256443
]
Na Li commented on SENTRY-1964:
---
The problem is to find the authObj from path.
For example, with the code change to not sending partition to HDFS, initially,
the INode hierarchy of the table path "tmp/external/tables/ext2_before" is
below, and the authObj of the partition "tmp/external/tables/ext2_before/i=1"
or "tmp/external/tables/ext2_before/i=1/stuff.txt" can use its parent's
authObj, which is default.ext2. So sentry ACL can be found using the authObj.
1) path name=tmp, type=DIR, authObj=null
2) path name=external, type=PREFIX, authObj=null
3) path name=tables, type=DIR, authObj=null
4) path name=ext2_before, type=AUTHZ_OBJECT, authObj=default.ext2
after "alter table ext2 set location
\'hdfs:///tmp/external/tables/ext2_after\'", the INode hierarchy becomes
1) path name=tmp, type=DIR, authObj=null
2) path name=external, type=PREFIX, authObj=null
3) path name=tables, type=DIR, authObj=null
4) path name=ext2_after, type=AUTHZ_OBJECT, authObj=default.ext2
When finding authObj for the partition "tmp/external/tables/ext2_before/i=1" or
"tmp/external/tables/ext2_before/i=1/stuff.txt", its parent is "tables", and
its authObj is null. So the SentryINodeAttributes won't be used, and no Sentry
ACL
In public List getAclEntries(String[] pathElements) at
SentryAuthorizationInfo, the authObjs are found in path first at
"{color:red}Set authzObjs =
authzPaths.findAuthzObject(pathElements);{color}", then use authObjs to find
the acl. If authObjs is null, then no sentry acl.
{code}
public List getAclEntries(String[] pathElements) {
lock.readLock().lock();
try {
Set authzObjs = authzPaths.findAuthzObject(pathElements);
// Apparently setFAcl throws error if 'group::---' is not present
AclEntry noGroup = AclEntry.parseAclEntry("group::---", true);
Set retSet = new HashSet<>();
retSet.add(noGroup);
if (authzObjs == null) {
retSet.addAll(Collections.emptyList());
return new ArrayList<>(retSet);
}
// No duplicate acls should be added.
for (String authzObj: authzObjs) {
retSet.addAll(authzPermissions.getAcls(authzObj));
}
return new ArrayList<>(retSet);
} finally {
lock.readLock().unlock();
}
}
{code}
> HDFS sync does not need partition locations (usually)
> -
>
> Key: SENTRY-1964
> URL: https://issues.apache.org/jira/browse/SENTRY-1964
> Project: Sentry
> Issue Type: Improvement
> Components: Sentry
>Affects Versions: 2.0.0
>Reporter: Na Li
>Assignee: Na Li
>Priority: Critical
> Attachments: SENTRY-1964.001.patch, SENTRY-1964.001.patch,
> SENTRY-1964.002.patch
>
>
> Right now, sentry saves partition info from HMS and send it to HDFS. HDFS
> only needs database and table info, and does not need partition info for ACL
> unless the partion location is not sharing the same prefix of its table.
> The partition data amount is huge, and causes performance issue. We can
> optimize it by not saving and not sending partition info if it shares the
> same path of its table.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Assigned] (SENTRY-2051) Catch SentryGroupNotFoundException in thrift server handler
[ https://issues.apache.org/jira/browse/SENTRY-2051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Na Li reassigned SENTRY-2051: - Assignee: (was: Na Li) > Catch SentryGroupNotFoundException in thrift server handler > --- > > Key: SENTRY-2051 > URL: https://issues.apache.org/jira/browse/SENTRY-2051 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.0.0 >Reporter: Na Li > > SENTRY-769 changes the behavior when there is no group associated with a > user. Before, the GroupMappingService implementation returns empty set. Now, > SentryGroupNotFoundException is thrown to stop further processing. > This causes problem for client that does not expect exception in this > scenario. > The solution is to catch such exception at sentry server before returning > response to client. So it does not change API behavior while still keeping > the benefit of SENTRY-769 with better error handling. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (SENTRY-2046) Create a full snapshot if AUTHZ_PATHS_SNAPSHOT_ID is empty, even if HMS and Sentry Notifications are in sync
[ https://issues.apache.org/jira/browse/SENTRY-2046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16254546#comment-16254546 ] Na Li commented on SENTRY-2046: --- We should get full snapshot when sentryStore.isAuthzPathsMappingEmpty() only if HDFS is enabled. Otherwise (i.e. HDFS sync is disabled), we will have issue stated in SENTRY-1890 "However, when data store is empty, we decide to get full snapshot. Therefore, every run, we get full snapshot from HMS. This causes big overhead." The reason is that when HDFS sync is disabled, we don't save full snapshot. and sentryStore.isAuthzPathsMappingEmpty() is always true > Create a full snapshot if AUTHZ_PATHS_SNAPSHOT_ID is empty, even if HMS and > Sentry Notifications are in sync > > > Key: SENTRY-2046 > URL: https://issues.apache.org/jira/browse/SENTRY-2046 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.0.0 >Reporter: Arjun Mishra >Assignee: Arjun Mishra > Labels: bite-sized, mewbi > > AUTHZ_PATHS_SNAPSHOT_ID is only populated when a full HMS snapshot is > persisted. If for some reason this does not occur and > SENTRY_HMS_NOTIFICATION_ID table gets populated with newer values, a full > snapshot will not occur unless until HMS and Sentry are out of sync. This > will result in ACL's not being applied unless until HMS and Sentry are out of > sync, since we only send NN snapshots if AUTHZ_PATHS_SNAPSHOT_ID has values > greater than 0 > We should create a full snapshot if hdfsSync is enabled, and > AUTHZ_PATHS_SNAPSHOT_ID is empty -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (SENTRY-2041) Change Index name in Package.jdo to match index name in SQL
[ https://issues.apache.org/jira/browse/SENTRY-2041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Na Li updated SENTRY-2041: -- Resolution: Won't Fix Fix Version/s: 2.0.0 Status: Resolved (was: Patch Available) > Change Index name in Package.jdo to match index name in SQL > --- > > Key: SENTRY-2041 > URL: https://issues.apache.org/jira/browse/SENTRY-2041 > Project: Sentry > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Na Li >Assignee: Na Li > Fix For: 2.0.0 > > Attachments: SENTRY-2041.002.patch > > > Many of the index names in package.jdo for dataNucleus are different from the > corresponding index names in sql script. > If dataNucleus is configured to add schema info into DB, it will result in > creating additional index for the same purpose in some DB (like MySql), or > cause exception for other DB (like Oracle) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (SENTRY-1894) Update field size in package.jdo for dataNucleus to match size in sql
[ https://issues.apache.org/jira/browse/SENTRY-1894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Na Li updated SENTRY-1894: -- Resolution: Won't Fix Status: Resolved (was: Patch Available) SENTRY-2028 does not add dataNucleus schema into DB by default. So with default DN behavior, we don't need to fix this issue. > Update field size in package.jdo for dataNucleus to match size in sql > - > > Key: SENTRY-1894 > URL: https://issues.apache.org/jira/browse/SENTRY-1894 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.0.0 >Reporter: Na Li >Assignee: Na Li > Attachments: SENTRY-1894.001.patch, SENTRY-1894.002.patch > > > The unique index > (`SERVER_NAME`,`DB_NAME`,`TABLE_NAME`,`COLUMN_NAME`,`URI`,`ACTION`,`WITH_GRANT_OPTION`)" > in `SENTRY_DB_PRIVILEGE` at package.jdo file is too large. When running > datanucleus schema tool, it complains > "com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Specified key was > too long; max key length is 767 bytes" > The size of the fields `SERVER_NAME`,`DB_NAME`,`TABLE_NAME`,`COLUMN_NAME` are > 4000, but the corresponding size in sql is only 128. > We should match the size defined in package.jdo to what's in sql script, and > remove `URI` from being part of the unique id. After those change, the > datanucleus schema tool finishes successfully with command "mvn > datanucleus:schema-create -X -e" -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (SENTRY-1922) Remove file-based sentry configuration
[ https://issues.apache.org/jira/browse/SENTRY-1922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16254248#comment-16254248 ] Na Li commented on SENTRY-1922: --- [~kkalyan] We can move it to a release after 2.0.0. We still want to have a jira to track this requirement and remove the support of file-based configuration. > Remove file-based sentry configuration > -- > > Key: SENTRY-1922 > URL: https://issues.apache.org/jira/browse/SENTRY-1922 > Project: Sentry > Issue Type: Improvement > Components: Sentry >Affects Versions: 2.0.0 >Reporter: Na Li > > Once all components use sentry service, the code for file-based configuration > should be removed. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
