[jira] [Commented] (SENTRY-2486) Wrong user name when sentry HMSFollower gets full snapshot from HMS at insecure mode
[
https://issues.apache.org/jira/browse/SENTRY-2486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16752798#comment-16752798
]
Hadoop QA commented on SENTRY-2486:
---
Here are the results of testing the latest attachment
https://issues.apache.org/jira/secure/attachment/12956395/SENTRY-2486.001.patch
against master.
{color:green}Overall:{color} +1 all checks pass
{color:green}SUCCESS:{color} all tests passed
Console output:
https://builds.apache.org/job/PreCommit-SENTRY-Build/4336/console
This message is automatically generated.
> Wrong user name when sentry HMSFollower gets full snapshot from HMS at
> insecure mode
>
>
> Key: SENTRY-2486
> URL: https://issues.apache.org/jira/browse/SENTRY-2486
> Project: Sentry
> Issue Type: Bug
> Components: Sentry
>Affects Versions: 2.2.0
>Reporter: Na Li
>Assignee: Na Li
>Priority: Major
> Attachments: SENTRY-2486.001.patch, SENTRY-2486.001.patch
>
>
> In insecure mode, the current login user name is passed from Sentry to HMS
> server when sentry HMSFollower gets full snapshot from HMS.
> The user name should be "sentry" instead of current login user.
> The followiong code shows how current login user name is used when subject is
> null.
> In UserGroupInformation, if the context does not have subject, the
> getLoginUser() is used as user name
> @Public
> @Evolving
> public static UserGroupInformation getCurrentUser() throws IOException {
> AccessControlContext context = AccessController.getContext();
> Subject subject = Subject.getSubject(context);
> return subject != null && !subject.getPrincipals(User.class).isEmpty() ?
> new UserGroupInformation(subject) : getLoginUser();
> }
> This issue should not happen in production because secure mode is always
> used. Insecure mode is only used in test.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (SENTRY-2491) Sentry High availability unit tests run into deadlock sometimes
[
https://issues.apache.org/jira/browse/SENTRY-2491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16752791#comment-16752791
]
Hadoop QA commented on SENTRY-2491:
---
Here are the results of testing the latest attachment
https://issues.apache.org/jira/secure/attachment/12956388/SENTRY-2491.001.patch
against master.
{color:red}Overall:{color} -1 due to 3 errors
{color:red}ERROR:{color} mvn test exited 1
{color:red}ERROR:{color} Failed:
org.apache.sentry.tests.e2e.dbprovider.TestHmsNotificationProcessingWithOutHdfsSync
{color:red}ERROR:{color} Failed:
org.apache.sentry.tests.e2e.dbprovider.TestOwnerPrivileges
Console output:
https://builds.apache.org/job/PreCommit-SENTRY-Build/4334/console
This message is automatically generated.
> Sentry High availability unit tests run into deadlock sometimes
> ---
>
> Key: SENTRY-2491
> URL: https://issues.apache.org/jira/browse/SENTRY-2491
> Project: Sentry
> Issue Type: Bug
> Components: Sentry
>Affects Versions: 2.2.0
>Reporter: Na Li
>Assignee: Na Li
>Priority: Major
> Attachments: SENTRY-2491.001.patch, SENTRY-2491.002.patch
>
>
> In sentry unit tests, we don't create schema before running a test. Instead,
> we use dataNucleus to create sentry tables when they are accessed. This
> creates potential deadlock when running test for Sentry HA setup.
> For example, the following shows the event sequence that causes deadlock
> 1) thread_1 gets shared lock of SYSTABLES in order to read table
> SENTRY_HMS_NOTIFICATION_ID
> 2) thread_2 gets shared lock of SYSTABLES in order to read table
> SENTRY_HMS_NOTIFICATION_ID
> 3) thread_1 tries to get execution lock to create table
> SENTRY_HMS_NOTIFICATION_ID,
>and wait for execution lock because thread_2 got shared lock already.
> 4) thread_2 tries to get execution lock to create table
> SENTRY_HMS_NOTIFICATION_ID,
>and wait for execution lock because thread_1 got shared lock already.
> The solution is to let the instances of sentry service start with delay.
> Specifically,
> let HMS follower threads separate as far as possible, i.e., half of the
> interval.
>
> This deadlock only exists in unit tests, and does not exist in production
> because schema is created before starting Sentry services. Therefore, there
> is no table creation after service starts.
> Example of the log message when such deadlock happens
> {code}
> 2019-01-04 18:32:46,332 (pool-13-thread-1) [INFO -
> org.apache.sentry.hdfs.DBUpdateForwarder.getAllUpdatesFrom(DBUpdateForwarder.java:140)]
> (org.apache.sentry.hdfs.PermImageRetriever) Requested sequence number 0 is
> less than 0 or requested deltas for that sequence number are not available.
> Fetch a full update
> 2019-01-04 18:32:49,346 (hms-follower) [DEBUG -
> org.apache.sentry.service.thrift.SentryStateBank.enableState(SentryStateBank.java:102)]
> HMSFollower entered state STARTED
> 2019-01-04 18:32:50,091 (hms-follower) [DEBUG -
> org.apache.sentry.service.thrift.SentryStateBank.enableState(SentryStateBank.java:102)]
> HMSFollower entered state STARTED
> 2019-01-04 18:33:00,286 (store-cleaner) [ERROR -
> org.datanucleus.util.Log4JLogger.error(Log4JLogger.java:115)] Error thrown
> executing CREATE TABLE SENTRY_HMS_NOTIFICATION_ID
> (
> NOTIFICATION_ID BIGINT NOT NULL
> ) : A lock could not be obtained due to a deadlock, cycle of locks and
> waiters is:
> Lock : ROW, SYSTABLES, (1,3)
> Waiting XID : {436, X} , SENTRY, CREATE TABLE SENTRY_HMS_NOTIFICATION_ID
> (
> NOTIFICATION_ID BIGINT NOT NULL
> )
> Granted XID : {419, S}
> Lock : ROW, SYSTABLES, (1,3)
> Waiting XID : {419, X} , SENTRY, CREATE TABLE SENTRY_HMS_NOTIFICATION_ID
> (
> NOTIFICATION_ID BIGINT NOT NULL
> )
> Granted XID : {419, S} , {436, S}
> . The selected victim is XID : 436.
> java.sql.SQLTransactionRollbackException: A lock could not be obtained due to
> a deadlock, cycle of locks and waiters is:
> Lock : ROW, SYSTABLES, (1,3)
> Waiting XID : {436, X} , SENTRY, CREATE TABLE SENTRY_HMS_NOTIFICATION_ID
> (
> NOTIFICATION_ID BIGINT NOT NULL
> )
> Granted XID : {419, S}
> Lock : ROW, SYSTABLES, (1,3)
> Waiting XID : {419, X} , SENTRY, CREATE TABLE SENTRY_HMS_NOTIFICATION_ID
> (
> NOTIFICATION_ID BIGINT NOT NULL
> )
> Granted XID : {419, S} , {436, S}
> . The selected victim is XID : 436.
> at
> org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
> at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown
> Source)
> at
> org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown
> Source)
> at
> org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown
> Source)
> at org.apache.derby.impl.jdbc.EmbedConnectio
[jira] [Commented] (SENTRY-2491) Sentry High availability unit tests run into deadlock sometimes
[
https://issues.apache.org/jira/browse/SENTRY-2491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16752760#comment-16752760
]
Hadoop QA commented on SENTRY-2491:
---
Here are the results of testing the latest attachment
https://issues.apache.org/jira/secure/attachment/12956400/SENTRY-2491.002.patch
against master.
{color:red}Overall:{color} -1 due to 2 errors
{color:red}ERROR:{color} mvn test exited 1
{color:red}ERROR:{color} Failed:
org.apache.sentry.hdfs.TestSentryHDFSServiceProcessor
Console output:
https://builds.apache.org/job/PreCommit-SENTRY-Build/4337/console
This message is automatically generated.
> Sentry High availability unit tests run into deadlock sometimes
> ---
>
> Key: SENTRY-2491
> URL: https://issues.apache.org/jira/browse/SENTRY-2491
> Project: Sentry
> Issue Type: Bug
> Components: Sentry
>Affects Versions: 2.2.0
>Reporter: Na Li
>Assignee: Na Li
>Priority: Major
> Attachments: SENTRY-2491.001.patch, SENTRY-2491.002.patch
>
>
> In sentry unit tests, we don't create schema before running a test. Instead,
> we use dataNucleus to create sentry tables when they are accessed. This
> creates potential deadlock when running test for Sentry HA setup.
> For example, the following shows the event sequence that causes deadlock
> 1) thread_1 gets shared lock of SYSTABLES in order to read table
> SENTRY_HMS_NOTIFICATION_ID
> 2) thread_2 gets shared lock of SYSTABLES in order to read table
> SENTRY_HMS_NOTIFICATION_ID
> 3) thread_1 tries to get execution lock to create table
> SENTRY_HMS_NOTIFICATION_ID,
>and wait for execution lock because thread_2 got shared lock already.
> 4) thread_2 tries to get execution lock to create table
> SENTRY_HMS_NOTIFICATION_ID,
>and wait for execution lock because thread_1 got shared lock already.
> The solution is to let the instances of sentry service start with delay.
> Specifically,
> let HMS follower threads separate as far as possible, i.e., half of the
> interval.
>
> This deadlock only exists in unit tests, and does not exist in production
> because schema is created before starting Sentry services. Therefore, there
> is no table creation after service starts.
> Example of the log message when such deadlock happens
> {code}
> 2019-01-04 18:32:46,332 (pool-13-thread-1) [INFO -
> org.apache.sentry.hdfs.DBUpdateForwarder.getAllUpdatesFrom(DBUpdateForwarder.java:140)]
> (org.apache.sentry.hdfs.PermImageRetriever) Requested sequence number 0 is
> less than 0 or requested deltas for that sequence number are not available.
> Fetch a full update
> 2019-01-04 18:32:49,346 (hms-follower) [DEBUG -
> org.apache.sentry.service.thrift.SentryStateBank.enableState(SentryStateBank.java:102)]
> HMSFollower entered state STARTED
> 2019-01-04 18:32:50,091 (hms-follower) [DEBUG -
> org.apache.sentry.service.thrift.SentryStateBank.enableState(SentryStateBank.java:102)]
> HMSFollower entered state STARTED
> 2019-01-04 18:33:00,286 (store-cleaner) [ERROR -
> org.datanucleus.util.Log4JLogger.error(Log4JLogger.java:115)] Error thrown
> executing CREATE TABLE SENTRY_HMS_NOTIFICATION_ID
> (
> NOTIFICATION_ID BIGINT NOT NULL
> ) : A lock could not be obtained due to a deadlock, cycle of locks and
> waiters is:
> Lock : ROW, SYSTABLES, (1,3)
> Waiting XID : {436, X} , SENTRY, CREATE TABLE SENTRY_HMS_NOTIFICATION_ID
> (
> NOTIFICATION_ID BIGINT NOT NULL
> )
> Granted XID : {419, S}
> Lock : ROW, SYSTABLES, (1,3)
> Waiting XID : {419, X} , SENTRY, CREATE TABLE SENTRY_HMS_NOTIFICATION_ID
> (
> NOTIFICATION_ID BIGINT NOT NULL
> )
> Granted XID : {419, S} , {436, S}
> . The selected victim is XID : 436.
> java.sql.SQLTransactionRollbackException: A lock could not be obtained due to
> a deadlock, cycle of locks and waiters is:
> Lock : ROW, SYSTABLES, (1,3)
> Waiting XID : {436, X} , SENTRY, CREATE TABLE SENTRY_HMS_NOTIFICATION_ID
> (
> NOTIFICATION_ID BIGINT NOT NULL
> )
> Granted XID : {419, S}
> Lock : ROW, SYSTABLES, (1,3)
> Waiting XID : {419, X} , SENTRY, CREATE TABLE SENTRY_HMS_NOTIFICATION_ID
> (
> NOTIFICATION_ID BIGINT NOT NULL
> )
> Granted XID : {419, S} , {436, S}
> . The selected victim is XID : 436.
> at
> org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
> at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown
> Source)
> at
> org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown
> Source)
> at
> org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown
> Source)
> at org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unknown
> Source)
> at org.apache.derby.impl.jdbc.ConnectionChild.handleException(Unknown
> Sourc
[jira] [Updated] (SENTRY-2491) Sentry High availability unit tests run into deadlock sometimes
[
https://issues.apache.org/jira/browse/SENTRY-2491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Na Li updated SENTRY-2491:
--
Attachment: SENTRY-2491.002.patch
> Sentry High availability unit tests run into deadlock sometimes
> ---
>
> Key: SENTRY-2491
> URL: https://issues.apache.org/jira/browse/SENTRY-2491
> Project: Sentry
> Issue Type: Bug
> Components: Sentry
>Affects Versions: 2.2.0
>Reporter: Na Li
>Assignee: Na Li
>Priority: Major
> Attachments: SENTRY-2491.001.patch, SENTRY-2491.002.patch
>
>
> In sentry unit tests, we don't create schema before running a test. Instead,
> we use dataNucleus to create sentry tables when they are accessed. This
> creates potential deadlock when running test for Sentry HA setup.
> For example, the following shows the event sequence that causes deadlock
> 1) thread_1 gets shared lock of SYSTABLES in order to read table
> SENTRY_HMS_NOTIFICATION_ID
> 2) thread_2 gets shared lock of SYSTABLES in order to read table
> SENTRY_HMS_NOTIFICATION_ID
> 3) thread_1 tries to get execution lock to create table
> SENTRY_HMS_NOTIFICATION_ID,
>and wait for execution lock because thread_2 got shared lock already.
> 4) thread_2 tries to get execution lock to create table
> SENTRY_HMS_NOTIFICATION_ID,
>and wait for execution lock because thread_1 got shared lock already.
> The solution is to let the instances of sentry service start with delay.
> Specifically,
> let HMS follower threads separate as far as possible, i.e., half of the
> interval.
>
> This deadlock only exists in unit tests, and does not exist in production
> because schema is created before starting Sentry services. Therefore, there
> is no table creation after service starts.
> Example of the log message when such deadlock happens
> {code}
> 2019-01-04 18:32:46,332 (pool-13-thread-1) [INFO -
> org.apache.sentry.hdfs.DBUpdateForwarder.getAllUpdatesFrom(DBUpdateForwarder.java:140)]
> (org.apache.sentry.hdfs.PermImageRetriever) Requested sequence number 0 is
> less than 0 or requested deltas for that sequence number are not available.
> Fetch a full update
> 2019-01-04 18:32:49,346 (hms-follower) [DEBUG -
> org.apache.sentry.service.thrift.SentryStateBank.enableState(SentryStateBank.java:102)]
> HMSFollower entered state STARTED
> 2019-01-04 18:32:50,091 (hms-follower) [DEBUG -
> org.apache.sentry.service.thrift.SentryStateBank.enableState(SentryStateBank.java:102)]
> HMSFollower entered state STARTED
> 2019-01-04 18:33:00,286 (store-cleaner) [ERROR -
> org.datanucleus.util.Log4JLogger.error(Log4JLogger.java:115)] Error thrown
> executing CREATE TABLE SENTRY_HMS_NOTIFICATION_ID
> (
> NOTIFICATION_ID BIGINT NOT NULL
> ) : A lock could not be obtained due to a deadlock, cycle of locks and
> waiters is:
> Lock : ROW, SYSTABLES, (1,3)
> Waiting XID : {436, X} , SENTRY, CREATE TABLE SENTRY_HMS_NOTIFICATION_ID
> (
> NOTIFICATION_ID BIGINT NOT NULL
> )
> Granted XID : {419, S}
> Lock : ROW, SYSTABLES, (1,3)
> Waiting XID : {419, X} , SENTRY, CREATE TABLE SENTRY_HMS_NOTIFICATION_ID
> (
> NOTIFICATION_ID BIGINT NOT NULL
> )
> Granted XID : {419, S} , {436, S}
> . The selected victim is XID : 436.
> java.sql.SQLTransactionRollbackException: A lock could not be obtained due to
> a deadlock, cycle of locks and waiters is:
> Lock : ROW, SYSTABLES, (1,3)
> Waiting XID : {436, X} , SENTRY, CREATE TABLE SENTRY_HMS_NOTIFICATION_ID
> (
> NOTIFICATION_ID BIGINT NOT NULL
> )
> Granted XID : {419, S}
> Lock : ROW, SYSTABLES, (1,3)
> Waiting XID : {419, X} , SENTRY, CREATE TABLE SENTRY_HMS_NOTIFICATION_ID
> (
> NOTIFICATION_ID BIGINT NOT NULL
> )
> Granted XID : {419, S} , {436, S}
> . The selected victim is XID : 436.
> at
> org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
> at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown
> Source)
> at
> org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown
> Source)
> at
> org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown
> Source)
> at org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unknown
> Source)
> at org.apache.derby.impl.jdbc.ConnectionChild.handleException(Unknown
> Source)
> at org.apache.derby.impl.jdbc.EmbedStatement.executeStatement(Unknown
> Source)
> at org.apache.derby.impl.jdbc.EmbedStatement.execute(Unknown Source)
> at org.apache.derby.impl.jdbc.EmbedStatement.execute(Unknown Source)
> at com.jolbox.bonecp.StatementHandle.execute(StatementHandle.java:254)
> at
> org.datanucleus.store.rdbms.table.AbstractTable.executeDdlStatement(AbstractTable.java:879)
> at
> org.datanucleus.store.rdbms.table.Abstr
[jira] [Updated] (SENTRY-2486) Wrong user name when sentry HMSFollower gets full snapshot from HMS at insecure mode
[
https://issues.apache.org/jira/browse/SENTRY-2486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Na Li updated SENTRY-2486:
--
Attachment: SENTRY-2486.001.patch
> Wrong user name when sentry HMSFollower gets full snapshot from HMS at
> insecure mode
>
>
> Key: SENTRY-2486
> URL: https://issues.apache.org/jira/browse/SENTRY-2486
> Project: Sentry
> Issue Type: Bug
> Components: Sentry
>Affects Versions: 2.2.0
>Reporter: Na Li
>Assignee: Na Li
>Priority: Major
> Attachments: SENTRY-2486.001.patch, SENTRY-2486.001.patch
>
>
> In insecure mode, the current login user name is passed from Sentry to HMS
> server when sentry HMSFollower gets full snapshot from HMS.
> The user name should be "sentry" instead of current login user.
> The followiong code shows how current login user name is used when subject is
> null.
> In UserGroupInformation, if the context does not have subject, the
> getLoginUser() is used as user name
> @Public
> @Evolving
> public static UserGroupInformation getCurrentUser() throws IOException {
> AccessControlContext context = AccessController.getContext();
> Subject subject = Subject.getSubject(context);
> return subject != null && !subject.getPrincipals(User.class).isEmpty() ?
> new UserGroupInformation(subject) : getLoginUser();
> }
> This issue should not happen in production because secure mode is always
> used. Insecure mode is only used in test.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (SENTRY-2486) Wrong user name when sentry HMSFollower gets full snapshot from HMS at insecure mode
[
https://issues.apache.org/jira/browse/SENTRY-2486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Na Li updated SENTRY-2486:
--
Description:
In insecure mode, the current login user name is passed from Sentry to HMS
server when sentry HMSFollower gets full snapshot from HMS.
The user name should be "sentry" instead of current login user.
The followiong code shows how current login user name is used when subject is
null.
In UserGroupInformation, if the context does not have subject, the
getLoginUser() is used as user name
@Public
@Evolving
public static UserGroupInformation getCurrentUser() throws IOException {
AccessControlContext context = AccessController.getContext();
Subject subject = Subject.getSubject(context);
return subject != null && !subject.getPrincipals(User.class).isEmpty() ?
new UserGroupInformation(subject) : getLoginUser();
}
This issue should not happen in production because secure mode is always used.
Insecure mode is only used in test.
was:
In insecure mode, the current login user name is passed from Sentry to HMS
server when sentry HMSFollower gets full snapshot from HMS.
The user name should be "sentry" instead of current login user.
The followiong code shows how current login user name is used when subject is
null.
In UserGroupInformation, if the context does not have subject, the
getLoginUser()
@Public
@Evolving
public static UserGroupInformation getCurrentUser() throws IOException {
AccessControlContext context = AccessController.getContext();
Subject subject = Subject.getSubject(context);
return subject != null && !subject.getPrincipals(User.class).isEmpty() ?
new UserGroupInformation(subject) : getLoginUser();
}
This issue should not happen in production because secure mode is always used.
Insecure mode is only used in test.
> Wrong user name when sentry HMSFollower gets full snapshot from HMS at
> insecure mode
>
>
> Key: SENTRY-2486
> URL: https://issues.apache.org/jira/browse/SENTRY-2486
> Project: Sentry
> Issue Type: Bug
> Components: Sentry
>Affects Versions: 2.2.0
>Reporter: Na Li
>Assignee: Na Li
>Priority: Major
> Attachments: SENTRY-2486.001.patch
>
>
> In insecure mode, the current login user name is passed from Sentry to HMS
> server when sentry HMSFollower gets full snapshot from HMS.
> The user name should be "sentry" instead of current login user.
> The followiong code shows how current login user name is used when subject is
> null.
> In UserGroupInformation, if the context does not have subject, the
> getLoginUser() is used as user name
> @Public
> @Evolving
> public static UserGroupInformation getCurrentUser() throws IOException {
> AccessControlContext context = AccessController.getContext();
> Subject subject = Subject.getSubject(context);
> return subject != null && !subject.getPrincipals(User.class).isEmpty() ?
> new UserGroupInformation(subject) : getLoginUser();
> }
> This issue should not happen in production because secure mode is always
> used. Insecure mode is only used in test.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (SENTRY-2486) Wrong user name when sentry HMSFollower gets full snapshot from HMS at insecure mode
[
https://issues.apache.org/jira/browse/SENTRY-2486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16752668#comment-16752668
]
Na Li commented on SENTRY-2486:
---
local test for all tests in
org.apache.sentry.provider.db.generic.service.persistent.TestPrivilegeOperatePersistence
have passed. reattach the same patch.
> Wrong user name when sentry HMSFollower gets full snapshot from HMS at
> insecure mode
>
>
> Key: SENTRY-2486
> URL: https://issues.apache.org/jira/browse/SENTRY-2486
> Project: Sentry
> Issue Type: Bug
> Components: Sentry
>Affects Versions: 2.2.0
>Reporter: Na Li
>Assignee: Na Li
>Priority: Major
> Attachments: SENTRY-2486.001.patch, SENTRY-2486.001.patch
>
>
> In insecure mode, the current login user name is passed from Sentry to HMS
> server when sentry HMSFollower gets full snapshot from HMS.
> The user name should be "sentry" instead of current login user.
> The followiong code shows how current login user name is used when subject is
> null.
> In UserGroupInformation, if the context does not have subject, the
> getLoginUser() is used as user name
> @Public
> @Evolving
> public static UserGroupInformation getCurrentUser() throws IOException {
> AccessControlContext context = AccessController.getContext();
> Subject subject = Subject.getSubject(context);
> return subject != null && !subject.getPrincipals(User.class).isEmpty() ?
> new UserGroupInformation(subject) : getLoginUser();
> }
> This issue should not happen in production because secure mode is always
> used. Insecure mode is only used in test.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (SENTRY-2490) NN is unfamiliar with REFRESH or CREATE privileges and sets FSAction to NONE on the group when it sees it in a full update
[ https://issues.apache.org/jira/browse/SENTRY-2490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arjun Mishra updated SENTRY-2490: - Summary: NN is unfamiliar with REFRESH or CREATE privileges and sets FSAction to NONE on the group when it sees it in a full update (was: NN is unfamiliar with REFRESH privileges and sets FSAction to NONE on the group when it sees it in a full update) > NN is unfamiliar with REFRESH or CREATE privileges and sets FSAction to NONE > on the group when it sees it in a full update > -- > > Key: SENTRY-2490 > URL: https://issues.apache.org/jira/browse/SENTRY-2490 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.1.0 >Reporter: Arjun Mishra >Assignee: Arjun Mishra >Priority: Major > > When NN tries to get the FS action on Sentry managed paths, it checks against > the ACTION_MAPPING map values. Currently this dataset doesn't have REFRESH > listed. When NN sees a REFRESH privilege it will flag it as unsupported and > set the FS action as NONE on that path for that group. This seems to happen > only with full update. It will also override any other privilege that group > had on that path -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (SENTRY-2490) NN is unfamiliar with REFRESH or CREATE privileges and sets FSAction to NONE on the group when it sees it in a full update
[ https://issues.apache.org/jira/browse/SENTRY-2490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arjun Mishra updated SENTRY-2490: - Description: When NN tries to get the FS action on Sentry managed paths, it checks against the ACTION_MAPPING map values. Currently this dataset doesn't have REFRESH or CREATE listed. When NN sees a REFRESH or CREATE privilege it will flag it as unsupported and set the FS action as NONE on that path for that group. This seems to happen only with full update. It will also override any other privilege that group had on that path (was: When NN tries to get the FS action on Sentry managed paths, it checks against the ACTION_MAPPING map values. Currently this dataset doesn't have REFRESH listed. When NN sees a REFRESH privilege it will flag it as unsupported and set the FS action as NONE on that path for that group. This seems to happen only with full update. It will also override any other privilege that group had on that path) > NN is unfamiliar with REFRESH or CREATE privileges and sets FSAction to NONE > on the group when it sees it in a full update > -- > > Key: SENTRY-2490 > URL: https://issues.apache.org/jira/browse/SENTRY-2490 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.1.0 >Reporter: Arjun Mishra >Assignee: Arjun Mishra >Priority: Major > > When NN tries to get the FS action on Sentry managed paths, it checks against > the ACTION_MAPPING map values. Currently this dataset doesn't have REFRESH or > CREATE listed. When NN sees a REFRESH or CREATE privilege it will flag it as > unsupported and set the FS action as NONE on that path for that group. This > seems to happen only with full update. It will also override any other > privilege that group had on that path -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (SENTRY-2486) Wrong user name when sentry HMSFollower gets full snapshot from HMS at insecure mode
[
https://issues.apache.org/jira/browse/SENTRY-2486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16752654#comment-16752654
]
Hadoop QA commented on SENTRY-2486:
---
Here are the results of testing the latest attachment
https://issues.apache.org/jira/secure/attachment/12956392/SENTRY-2486.001.patch
against master.
{color:red}Overall:{color} -1 due to 2 errors
{color:red}ERROR:{color} mvn test exited 1
{color:red}ERROR:{color} Failed:
org.apache.sentry.provider.db.generic.service.persistent.TestPrivilegeOperatePersistence
Console output:
https://builds.apache.org/job/PreCommit-SENTRY-Build/4335/console
This message is automatically generated.
> Wrong user name when sentry HMSFollower gets full snapshot from HMS at
> insecure mode
>
>
> Key: SENTRY-2486
> URL: https://issues.apache.org/jira/browse/SENTRY-2486
> Project: Sentry
> Issue Type: Bug
> Components: Sentry
>Affects Versions: 2.2.0
>Reporter: Na Li
>Assignee: Na Li
>Priority: Major
> Attachments: SENTRY-2486.001.patch
>
>
> In insecure mode, the current login user name is passed from Sentry to HMS
> server when sentry HMSFollower gets full snapshot from HMS.
> The user name should be "sentry" instead of current login user.
> The followiong code shows how current login user name is used when subject is
> null.
> In UserGroupInformation, if the context does not have subject, the
> getLoginUser()
> @Public
> @Evolving
> public static UserGroupInformation getCurrentUser() throws IOException {
> AccessControlContext context = AccessController.getContext();
> Subject subject = Subject.getSubject(context);
> return subject != null && !subject.getPrincipals(User.class).isEmpty() ?
> new UserGroupInformation(subject) : getLoginUser();
> }
> This issue should not happen in production because secure mode is always
> used. Insecure mode is only used in test.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (SENTRY-2489) Fix flaky test testRequestSyncUpdatesWhenPubSubNotifyReturnsFullPathsUpdate
[
https://issues.apache.org/jira/browse/SENTRY-2489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16752646#comment-16752646
]
Hadoop QA commented on SENTRY-2489:
---
Here are the results of testing the latest attachment
https://issues.apache.org/jira/secure/attachment/12956373/SENTRY-2489.001.patch
against master.
{color:green}Overall:{color} +1 all checks pass
{color:green}SUCCESS:{color} all tests passed
Console output:
https://builds.apache.org/job/PreCommit-SENTRY-Build/4333/console
This message is automatically generated.
> Fix flaky test testRequestSyncUpdatesWhenPubSubNotifyReturnsFullPathsUpdate
> ---
>
> Key: SENTRY-2489
> URL: https://issues.apache.org/jira/browse/SENTRY-2489
> Project: Sentry
> Issue Type: Bug
> Components: Sentry
>Affects Versions: 2.1
>Reporter: kalyan kumar kalvagadda
>Assignee: kalyan kumar kalvagadda
>Priority: Major
> Attachments: SENTRY-2489.001.patch
>
>
> testRequestSyncUpdatesWhenPubSubNotifyReturnsFullPathsUpdate is falky and it
> fails pretty commonly.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (SENTRY-2486) Wrong user name when sentry HMSFollower gets full snapshot from HMS at insecure mode
[
https://issues.apache.org/jira/browse/SENTRY-2486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Na Li updated SENTRY-2486:
--
Attachment: SENTRY-2486.001.patch
> Wrong user name when sentry HMSFollower gets full snapshot from HMS at
> insecure mode
>
>
> Key: SENTRY-2486
> URL: https://issues.apache.org/jira/browse/SENTRY-2486
> Project: Sentry
> Issue Type: Bug
> Components: Sentry
>Affects Versions: 2.2.0
>Reporter: Na Li
>Assignee: Na Li
>Priority: Major
> Attachments: SENTRY-2486.001.patch
>
>
> In insecure mode, the current login user name is passed from Sentry to HMS
> server when sentry HMSFollower gets full snapshot from HMS.
> The user name should be "sentry" instead of current login user.
> The followiong code shows how current login user name is used when subject is
> null.
> In UserGroupInformation, if the context does not have subject, the
> getLoginUser()
> @Public
> @Evolving
> public static UserGroupInformation getCurrentUser() throws IOException {
> AccessControlContext context = AccessController.getContext();
> Subject subject = Subject.getSubject(context);
> return subject != null && !subject.getPrincipals(User.class).isEmpty() ?
> new UserGroupInformation(subject) : getLoginUser();
> }
> This issue should not happen in production because secure mode is always
> used. Insecure mode is only used in test.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (SENTRY-2486) Wrong user name when sentry HMSFollower gets full snapshot from HMS at insecure mode
[
https://issues.apache.org/jira/browse/SENTRY-2486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Na Li updated SENTRY-2486:
--
Status: Patch Available (was: Open)
> Wrong user name when sentry HMSFollower gets full snapshot from HMS at
> insecure mode
>
>
> Key: SENTRY-2486
> URL: https://issues.apache.org/jira/browse/SENTRY-2486
> Project: Sentry
> Issue Type: Bug
> Components: Sentry
>Affects Versions: 2.2.0
>Reporter: Na Li
>Assignee: Na Li
>Priority: Major
> Attachments: SENTRY-2486.001.patch
>
>
> In insecure mode, the current login user name is passed from Sentry to HMS
> server when sentry HMSFollower gets full snapshot from HMS.
> The user name should be "sentry" instead of current login user.
> The followiong code shows how current login user name is used when subject is
> null.
> In UserGroupInformation, if the context does not have subject, the
> getLoginUser()
> @Public
> @Evolving
> public static UserGroupInformation getCurrentUser() throws IOException {
> AccessControlContext context = AccessController.getContext();
> Subject subject = Subject.getSubject(context);
> return subject != null && !subject.getPrincipals(User.class).isEmpty() ?
> new UserGroupInformation(subject) : getLoginUser();
> }
> This issue should not happen in production because secure mode is always
> used. Insecure mode is only used in test.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (SENTRY-2491) Sentry High availability unit tests run into deadlock sometimes
[
https://issues.apache.org/jira/browse/SENTRY-2491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Na Li updated SENTRY-2491:
--
Description:
In sentry unit tests, we don't create schema before running a test. Instead, we
use dataNucleus to create sentry tables when they are accessed. This creates
potential deadlock when running test for Sentry HA setup.
For example, the following shows the event sequence that causes deadlock
1) thread_1 gets shared lock of SYSTABLES in order to read table
SENTRY_HMS_NOTIFICATION_ID
2) thread_2 gets shared lock of SYSTABLES in order to read table
SENTRY_HMS_NOTIFICATION_ID
3) thread_1 tries to get execution lock to create table
SENTRY_HMS_NOTIFICATION_ID,
and wait for execution lock because thread_2 got shared lock already.
4) thread_2 tries to get execution lock to create table
SENTRY_HMS_NOTIFICATION_ID,
and wait for execution lock because thread_1 got shared lock already.
The solution is to let the instances of sentry service start with delay.
Specifically,
let HMS follower threads separate as far as possible, i.e., half of the
interval.
This deadlock only exists in unit tests, and does not exist in production
because schema is created before starting Sentry services. Therefore, there is
no table creation after service starts.
Example of the log message when such deadlock happens
{code}
2019-01-04 18:32:46,332 (pool-13-thread-1) [INFO -
org.apache.sentry.hdfs.DBUpdateForwarder.getAllUpdatesFrom(DBUpdateForwarder.java:140)]
(org.apache.sentry.hdfs.PermImageRetriever) Requested sequence number 0 is
less than 0 or requested deltas for that sequence number are not available.
Fetch a full update
2019-01-04 18:32:49,346 (hms-follower) [DEBUG -
org.apache.sentry.service.thrift.SentryStateBank.enableState(SentryStateBank.java:102)]
HMSFollower entered state STARTED
2019-01-04 18:32:50,091 (hms-follower) [DEBUG -
org.apache.sentry.service.thrift.SentryStateBank.enableState(SentryStateBank.java:102)]
HMSFollower entered state STARTED
2019-01-04 18:33:00,286 (store-cleaner) [ERROR -
org.datanucleus.util.Log4JLogger.error(Log4JLogger.java:115)] Error thrown
executing CREATE TABLE SENTRY_HMS_NOTIFICATION_ID
(
NOTIFICATION_ID BIGINT NOT NULL
) : A lock could not be obtained due to a deadlock, cycle of locks and waiters
is:
Lock : ROW, SYSTABLES, (1,3)
Waiting XID : {436, X} , SENTRY, CREATE TABLE SENTRY_HMS_NOTIFICATION_ID
(
NOTIFICATION_ID BIGINT NOT NULL
)
Granted XID : {419, S}
Lock : ROW, SYSTABLES, (1,3)
Waiting XID : {419, X} , SENTRY, CREATE TABLE SENTRY_HMS_NOTIFICATION_ID
(
NOTIFICATION_ID BIGINT NOT NULL
)
Granted XID : {419, S} , {436, S}
. The selected victim is XID : 436.
java.sql.SQLTransactionRollbackException: A lock could not be obtained due to a
deadlock, cycle of locks and waiters is:
Lock : ROW, SYSTABLES, (1,3)
Waiting XID : {436, X} , SENTRY, CREATE TABLE SENTRY_HMS_NOTIFICATION_ID
(
NOTIFICATION_ID BIGINT NOT NULL
)
Granted XID : {419, S}
Lock : ROW, SYSTABLES, (1,3)
Waiting XID : {419, X} , SENTRY, CREATE TABLE SENTRY_HMS_NOTIFICATION_ID
(
NOTIFICATION_ID BIGINT NOT NULL
)
Granted XID : {419, S} , {436, S}
. The selected victim is XID : 436.
at
org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown
Source)
at
org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown
Source)
at
org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown
Source)
at org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unknown
Source)
at org.apache.derby.impl.jdbc.ConnectionChild.handleException(Unknown
Source)
at org.apache.derby.impl.jdbc.EmbedStatement.executeStatement(Unknown
Source)
at org.apache.derby.impl.jdbc.EmbedStatement.execute(Unknown Source)
at org.apache.derby.impl.jdbc.EmbedStatement.execute(Unknown Source)
at com.jolbox.bonecp.StatementHandle.execute(StatementHandle.java:254)
at
org.datanucleus.store.rdbms.table.AbstractTable.executeDdlStatement(AbstractTable.java:879)
at
org.datanucleus.store.rdbms.table.AbstractTable.executeDdlStatementList(AbstractTable.java:830)
at
org.datanucleus.store.rdbms.table.AbstractTable.create(AbstractTable.java:546)
at
org.datanucleus.store.rdbms.table.AbstractTable.exists(AbstractTable.java:609)
at
org.datanucleus.store.rdbms.RDBMSStoreManager$ClassAdder.performTablesValidation(RDBMSStoreManager.java:3385)
at
org.datanucleus.store.rdbms.RDBMSStoreManager$ClassAdder.run(RDBMSStoreManager.java:2896)
at
org.datanucleus.store.rdbms.AbstractSchemaTransaction.execute(AbstractSchemaTransaction.java:119)
at
...[truncated 36339941 chars]...
eImpl(RetryClientInvocationHandler.java:94)] Calling getAllUpdatesFrom
{code}
[jira] [Updated] (SENTRY-2491) Sentry High availability unit tests run into deadlock sometimes
[ https://issues.apache.org/jira/browse/SENTRY-2491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Na Li updated SENTRY-2491: -- Attachment: SENTRY-2491.001.patch > Sentry High availability unit tests run into deadlock sometimes > --- > > Key: SENTRY-2491 > URL: https://issues.apache.org/jira/browse/SENTRY-2491 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.2.0 >Reporter: Na Li >Assignee: Na Li >Priority: Major > Attachments: SENTRY-2491.001.patch > > > In sentry unit tests, we don't create schema before running a test. Instead, > we use dataNucleus to create sentry tables when they are accessed. This > creates potential deadlock when running test for Sentry HA setup. > For example, the following shows the event sequence that causes deadlock > 1) thread_1 gets shared lock of SYSTABLES in order to read table > SENTRY_HMS_NOTIFICATION_ID > 2) thread_2 gets shared lock of SYSTABLES in order to read table > SENTRY_HMS_NOTIFICATION_ID > 3) thread_1 tries to get execution lock to create table > SENTRY_HMS_NOTIFICATION_ID, >and wait for execution lock because thread_2 got shared lock already. > 4) thread_2 tries to get execution lock to create table > SENTRY_HMS_NOTIFICATION_ID, >and wait for execution lock because thread_1 got shared lock already. > The solution is to let the instances of sentry service start with delay. > Specifically, > let HMS follower threads separate as far as possible, i.e., half of the > interval. > > This deadlock only exists in unit tests, and does not exist in production > because schema is created before starting Sentry services. Therefore, there > is no table creation after service starts. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (SENTRY-2491) Sentry High availability unit tests run into deadlock sometimes
[ https://issues.apache.org/jira/browse/SENTRY-2491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Na Li updated SENTRY-2491: -- Status: Patch Available (was: Open) > Sentry High availability unit tests run into deadlock sometimes > --- > > Key: SENTRY-2491 > URL: https://issues.apache.org/jira/browse/SENTRY-2491 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.2.0 >Reporter: Na Li >Assignee: Na Li >Priority: Major > Attachments: SENTRY-2491.001.patch > > > In sentry unit tests, we don't create schema before running a test. Instead, > we use dataNucleus to create sentry tables when they are accessed. This > creates potential deadlock when running test for Sentry HA setup. > For example, the following shows the event sequence that causes deadlock > 1) thread_1 gets shared lock of SYSTABLES in order to read table > SENTRY_HMS_NOTIFICATION_ID > 2) thread_2 gets shared lock of SYSTABLES in order to read table > SENTRY_HMS_NOTIFICATION_ID > 3) thread_1 tries to get execution lock to create table > SENTRY_HMS_NOTIFICATION_ID, >and wait for execution lock because thread_2 got shared lock already. > 4) thread_2 tries to get execution lock to create table > SENTRY_HMS_NOTIFICATION_ID, >and wait for execution lock because thread_1 got shared lock already. > The solution is to let the instances of sentry service start with delay. > Specifically, > let HMS follower threads separate as far as possible, i.e., half of the > interval. > > This deadlock only exists in unit tests, and does not exist in production > because schema is created before starting Sentry services. Therefore, there > is no table creation after service starts. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (SENTRY-2491) Sentry High availability unit tests run into deadlock sometimes
Na Li created SENTRY-2491: - Summary: Sentry High availability unit tests run into deadlock sometimes Key: SENTRY-2491 URL: https://issues.apache.org/jira/browse/SENTRY-2491 Project: Sentry Issue Type: Bug Components: Sentry Affects Versions: 2.2.0 Reporter: Na Li Assignee: Na Li In sentry unit tests, we don't create schema before running a test. Instead, we use dataNucleus to create sentry tables when they are accessed. This creates potential deadlock when running test for Sentry HA setup. For example, the following shows the event sequence that causes deadlock 1) thread_1 gets shared lock of SYSTABLES in order to read table SENTRY_HMS_NOTIFICATION_ID 2) thread_2 gets shared lock of SYSTABLES in order to read table SENTRY_HMS_NOTIFICATION_ID 3) thread_1 tries to get execution lock to create table SENTRY_HMS_NOTIFICATION_ID, and wait for execution lock because thread_2 got shared lock already. 4) thread_2 tries to get execution lock to create table SENTRY_HMS_NOTIFICATION_ID, and wait for execution lock because thread_1 got shared lock already. The solution is to let the instances of sentry service start with delay. Specifically, let HMS follower threads separate as far as possible, i.e., half of the interval. This deadlock only exists in unit tests, and does not exist in production because schema is created before starting Sentry services. Therefore, there is no table creation after service starts. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (SENTRY-2490) NN is unfamiliar with REFRESH privileges and sets FSAction to NONE on the group when it sees it in a full update
[
https://issues.apache.org/jira/browse/SENTRY-2490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16752545#comment-16752545
]
kalyan kumar kalvagadda edited comment on SENTRY-2490 at 1/25/19 6:25 PM:
--
[~arjunmishra13] If you look at the code below
{noformat}
private static FsAction getFAction(String sentryPriv) {
String[] strPrivs = sentryPriv.trim().split(",");
FsAction retVal = FsAction.NONE;
for (String strPriv : strPrivs) {
FsAction action = ACTION_MAPPING.get(strPriv.toUpperCase());
if (action == null) {
// Encountered a privilege that is not supported. Since we do not know
what
// to do with it we just drop all access.
LOG.warn("Unsupported privilege {}, disabling all access", strPriv);
action = FsAction.NONE;
}
retVal = retVal.or(action);
}
return retVal;
}
{noformat}
At this line *retVal = retVal.or(action)*, if the action is NONE, it may not
update the *retval*.
was (Author: kkalyan):
[~arjunmishra13] If you look at the code below
{noformat}
private static FsAction getFAction(String sentryPriv) {
String[] strPrivs = sentryPriv.trim().split(",");
FsAction retVal = FsAction.NONE;
for (String strPriv : strPrivs) {
FsAction action = ACTION_MAPPING.get(strPriv.toUpperCase());
if (action == null) {
// Encountered a privilege that is not supported. Since we do not know
what
// to do with it we just drop all access.
LOG.warn("Unsupported privilege {}, disabling all access", strPriv);
action = FsAction.NONE;
}
retVal = retVal.or(action);
}
return retVal;
}
{noformat}
At this libe *retVal = retVal.or(action)*, if the action is NONE, it may not
update the *retval*.
> NN is unfamiliar with REFRESH privileges and sets FSAction to NONE on the
> group when it sees it in a full update
>
>
> Key: SENTRY-2490
> URL: https://issues.apache.org/jira/browse/SENTRY-2490
> Project: Sentry
> Issue Type: Bug
> Components: Sentry
>Affects Versions: 2.1.0
>Reporter: Arjun Mishra
>Assignee: Arjun Mishra
>Priority: Major
>
> When NN tries to get the FS action on Sentry managed paths, it checks against
> the ACTION_MAPPING map values. Currently this dataset doesn't have REFRESH
> listed. When NN sees a REFRESH privilege it will flag it as unsupported and
> set the FS action as NONE on that path for that group. This seems to happen
> only with full update. It will also override any other privilege that group
> had on that path
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (SENTRY-2490) NN is unfamiliar with REFRESH privileges and sets FSAction to NONE on the group when it sees it in a full update
[
https://issues.apache.org/jira/browse/SENTRY-2490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16752545#comment-16752545
]
kalyan kumar kalvagadda commented on SENTRY-2490:
-
[~arjunmishra13] If you look at the code below
{noformat}
private static FsAction getFAction(String sentryPriv) {
String[] strPrivs = sentryPriv.trim().split(",");
FsAction retVal = FsAction.NONE;
for (String strPriv : strPrivs) {
FsAction action = ACTION_MAPPING.get(strPriv.toUpperCase());
if (action == null) {
// Encountered a privilege that is not supported. Since we do not know
what
// to do with it we just drop all access.
LOG.warn("Unsupported privilege {}, disabling all access", strPriv);
action = FsAction.NONE;
}
retVal = retVal.or(action);
}
return retVal;
}
{noformat}
At this libe *retVal = retVal.or(action)*, if the action is NONE, it may not
update the *retval*.
> NN is unfamiliar with REFRESH privileges and sets FSAction to NONE on the
> group when it sees it in a full update
>
>
> Key: SENTRY-2490
> URL: https://issues.apache.org/jira/browse/SENTRY-2490
> Project: Sentry
> Issue Type: Bug
> Components: Sentry
>Affects Versions: 2.1.0
>Reporter: Arjun Mishra
>Assignee: Arjun Mishra
>Priority: Major
>
> When NN tries to get the FS action on Sentry managed paths, it checks against
> the ACTION_MAPPING map values. Currently this dataset doesn't have REFRESH
> listed. When NN sees a REFRESH privilege it will flag it as unsupported and
> set the FS action as NONE on that path for that group. This seems to happen
> only with full update. It will also override any other privilege that group
> had on that path
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (SENTRY-2490) NN is unfamiliar with REFRESH privileges and sets FSAction to NONE on the group when it sees it in a full update
[ https://issues.apache.org/jira/browse/SENTRY-2490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arjun Mishra updated SENTRY-2490: - Description: When NN tries to get the FS action on Sentry managed paths, it checks against the ACTION_MAPPING map values. Currently this dataset doesn't have REFRESH listed. When NN sees a REFRESH privilege it will flag it as unsupported and set the FS action as NONE on that path for that group. This seems to happen only with full update. It will also override any other privilege that group had on that path (was: When NN tries to get the FS action on Sentry managed paths, it checks against the ACTION_MAPPING map values. Currently this dataset doesn't have REFRESH listed. When NN sees a REFRESH privilege it will flag it as unsupported and set the FS action as NONE on that path for that group. It will also override any other privilege that group had on that path) > NN is unfamiliar with REFRESH privileges and sets FSAction to NONE on the > group when it sees it in a full update > > > Key: SENTRY-2490 > URL: https://issues.apache.org/jira/browse/SENTRY-2490 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.1.0 >Reporter: Arjun Mishra >Assignee: Arjun Mishra >Priority: Major > > When NN tries to get the FS action on Sentry managed paths, it checks against > the ACTION_MAPPING map values. Currently this dataset doesn't have REFRESH > listed. When NN sees a REFRESH privilege it will flag it as unsupported and > set the FS action as NONE on that path for that group. This seems to happen > only with full update. It will also override any other privilege that group > had on that path -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (SENTRY-2490) NN is unfamiliar with REFRESH privileges and sets FSAction to NONE on the group when it sees it in a full update
[ https://issues.apache.org/jira/browse/SENTRY-2490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arjun Mishra updated SENTRY-2490: - Summary: NN is unfamiliar with REFRESH privileges and sets FSAction to NONE on the group when it sees it in a full update (was: NN is unfamiliar with REFRESH privileges and sets FSAction to NONE on the group when it sees it) > NN is unfamiliar with REFRESH privileges and sets FSAction to NONE on the > group when it sees it in a full update > > > Key: SENTRY-2490 > URL: https://issues.apache.org/jira/browse/SENTRY-2490 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.1.0 >Reporter: Arjun Mishra >Assignee: Arjun Mishra >Priority: Major > > When NN tries to get the FS action on Sentry managed paths, it checks against > the ACTION_MAPPING map values. Currently this dataset doesn't have REFRESH > listed. When NN sees a REFRESH privilege it will flag it as unsupported and > set the FS action as NONE on that path for that group. It will also override > any other privilege that group had on that path -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (SENTRY-2490) NN is unfamiliar with REFRESH privileges and sets FSAction to NONE on the group when it sees it
Arjun Mishra created SENTRY-2490: Summary: NN is unfamiliar with REFRESH privileges and sets FSAction to NONE on the group when it sees it Key: SENTRY-2490 URL: https://issues.apache.org/jira/browse/SENTRY-2490 Project: Sentry Issue Type: Bug Components: Sentry Affects Versions: 2.1.0 Reporter: Arjun Mishra Assignee: Arjun Mishra When NN tries to get the FS action on Sentry managed paths, it checks against the ACTION_MAPPING map values. Currently this dataset doesn't have REFRESH listed. When NN sees a REFRESH privilege it will flag it as unsupported and set the FS action as NONE on that path for that group. It will also override any other privilege that group had on that path -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (SENTRY-2489) Fix flaky test testRequestSyncUpdatesWhenPubSubNotifyReturnsFullPathsUpdate
[ https://issues.apache.org/jira/browse/SENTRY-2489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kalyan kumar kalvagadda updated SENTRY-2489: Attachment: SENTRY-2489.001.patch > Fix flaky test testRequestSyncUpdatesWhenPubSubNotifyReturnsFullPathsUpdate > --- > > Key: SENTRY-2489 > URL: https://issues.apache.org/jira/browse/SENTRY-2489 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.1 >Reporter: kalyan kumar kalvagadda >Assignee: kalyan kumar kalvagadda >Priority: Major > Attachments: SENTRY-2489.001.patch > > > testRequestSyncUpdatesWhenPubSubNotifyReturnsFullPathsUpdate is falky and it > fails pretty commonly. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Issue Comment Deleted] (SENTRY-2489) Fix flaky test testRequestSyncUpdatesWhenPubSubNotifyReturnsFullPathsUpdate
[
https://issues.apache.org/jira/browse/SENTRY-2489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
kalyan kumar kalvagadda updated SENTRY-2489:
Comment: was deleted
(was: Here are the results of testing the latest attachment
https://issues.apache.org/jira/secure/attachment/12956355/SENTRY-2489.001.patch
against master.
{color:red}Overall:{color} -1 due to 2 errors
{color:red}ERROR:{color} mvn test exited 1
{color:red}ERROR:{color} Failed:
org.apache.sentry.provider.db.generic.service.persistent.TestPrivilegeOperatePersistence
Console output:
https://builds.apache.org/job/PreCommit-SENTRY-Build/4330/console
This message is automatically generated.)
> Fix flaky test testRequestSyncUpdatesWhenPubSubNotifyReturnsFullPathsUpdate
> ---
>
> Key: SENTRY-2489
> URL: https://issues.apache.org/jira/browse/SENTRY-2489
> Project: Sentry
> Issue Type: Bug
> Components: Sentry
>Affects Versions: 2.1
>Reporter: kalyan kumar kalvagadda
>Assignee: kalyan kumar kalvagadda
>Priority: Major
>
> testRequestSyncUpdatesWhenPubSubNotifyReturnsFullPathsUpdate is falky and it
> fails pretty commonly.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (SENTRY-2489) Fix flaky test testRequestSyncUpdatesWhenPubSubNotifyReturnsFullPathsUpdate
[ https://issues.apache.org/jira/browse/SENTRY-2489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kalyan kumar kalvagadda updated SENTRY-2489: Attachment: (was: SENTRY-2489.001.patch) > Fix flaky test testRequestSyncUpdatesWhenPubSubNotifyReturnsFullPathsUpdate > --- > > Key: SENTRY-2489 > URL: https://issues.apache.org/jira/browse/SENTRY-2489 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.1 >Reporter: kalyan kumar kalvagadda >Assignee: kalyan kumar kalvagadda >Priority: Major > > testRequestSyncUpdatesWhenPubSubNotifyReturnsFullPathsUpdate is falky and it > fails pretty commonly. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (SENTRY-1904) TransactionManager should limit the max time spent by transaction retry
[
https://issues.apache.org/jira/browse/SENTRY-1904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16752463#comment-16752463
]
Hadoop QA commented on SENTRY-1904:
---
Here are the results of testing the latest attachment
https://issues.apache.org/jira/secure/attachment/12956357/SENTRY-1904.009.patch
against master.
{color:green}Overall:{color} +1 all checks pass
{color:green}SUCCESS:{color} all tests passed
Console output:
https://builds.apache.org/job/PreCommit-SENTRY-Build/4331/console
This message is automatically generated.
> TransactionManager should limit the max time spent by transaction retry
> ---
>
> Key: SENTRY-1904
> URL: https://issues.apache.org/jira/browse/SENTRY-1904
> Project: Sentry
> Issue Type: Bug
> Components: Sentry
>Affects Versions: 2.0.0
>Reporter: Alexander Kolbasov
>Assignee: kalyan kumar kalvagadda
>Priority: Major
> Attachments: SENTRY-1904.001.patch, SENTRY-1904.002.patch,
> SENTRY-1904.003.patch, SENTRY-1904.004.patch, SENTRY-1904.005.patch,
> SENTRY-1904.006.patch, SENTRY-1904.007.patch, SENTRY-1904.008.patch,
> SENTRY-1904.009.patch
>
>
> The TransactionManager uses exponential backoff strategy for transaction
> retries. This may cause some transactions to be delayed by a very long time.
> We should also have a constraint on the max time for a transaction so that we
> do not retry for too long.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (SENTRY-2287) Find a way to report that HDFS ACL synchronization is complete.
[
https://issues.apache.org/jira/browse/SENTRY-2287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16752455#comment-16752455
]
Hadoop QA commented on SENTRY-2287:
---
Here are the results of testing the latest attachment
https://issues.apache.org/jira/secure/attachment/12956359/SENTRY-2287.004.patch
against master.
{color:green}Overall:{color} +1 all checks pass
{color:green}SUCCESS:{color} all tests passed
Console output:
https://builds.apache.org/job/PreCommit-SENTRY-Build/4332/console
This message is automatically generated.
> Find a way to report that HDFS ACL synchronization is complete.
> ---
>
> Key: SENTRY-2287
> URL: https://issues.apache.org/jira/browse/SENTRY-2287
> Project: Sentry
> Issue Type: Sub-task
> Components: Sentry
>Affects Versions: 2.1.0
>Reporter: kalyan kumar kalvagadda
>Assignee: kalyan kumar kalvagadda
>Priority: Major
> Attachments: SENTRY-2287.001.patch, SENTRY-2287.002.patch,
> SENTRY-2287.003.patch, SENTRY-2287.004.patch
>
>
> Currently there is no way to confirm that HDFS ACL synchronization is
> complete when snapshot is initiated. We need to identify that and log in
> console and log file as well.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (SENTRY-2489) Fix flaky test testRequestSyncUpdatesWhenPubSubNotifyReturnsFullPathsUpdate
[
https://issues.apache.org/jira/browse/SENTRY-2489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16752312#comment-16752312
]
Hadoop QA commented on SENTRY-2489:
---
Here are the results of testing the latest attachment
https://issues.apache.org/jira/secure/attachment/12956355/SENTRY-2489.001.patch
against master.
{color:red}Overall:{color} -1 due to 2 errors
{color:red}ERROR:{color} mvn test exited 1
{color:red}ERROR:{color} Failed:
org.apache.sentry.provider.db.generic.service.persistent.TestPrivilegeOperatePersistence
Console output:
https://builds.apache.org/job/PreCommit-SENTRY-Build/4330/console
This message is automatically generated.
> Fix flaky test testRequestSyncUpdatesWhenPubSubNotifyReturnsFullPathsUpdate
> ---
>
> Key: SENTRY-2489
> URL: https://issues.apache.org/jira/browse/SENTRY-2489
> Project: Sentry
> Issue Type: Bug
> Components: Sentry
>Affects Versions: 2.1
>Reporter: kalyan kumar kalvagadda
>Assignee: kalyan kumar kalvagadda
>Priority: Major
> Attachments: SENTRY-2489.001.patch
>
>
> testRequestSyncUpdatesWhenPubSubNotifyReturnsFullPathsUpdate is falky and it
> fails pretty commonly.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (SENTRY-2287) Find a way to report that HDFS ACL synchronization is complete.
[ https://issues.apache.org/jira/browse/SENTRY-2287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kalyan kumar kalvagadda updated SENTRY-2287: Attachment: (was: SENTRY-2287.004.patch) > Find a way to report that HDFS ACL synchronization is complete. > --- > > Key: SENTRY-2287 > URL: https://issues.apache.org/jira/browse/SENTRY-2287 > Project: Sentry > Issue Type: Sub-task > Components: Sentry >Affects Versions: 2.1.0 >Reporter: kalyan kumar kalvagadda >Assignee: kalyan kumar kalvagadda >Priority: Major > Attachments: SENTRY-2287.001.patch, SENTRY-2287.002.patch, > SENTRY-2287.003.patch, SENTRY-2287.004.patch > > > Currently there is no way to confirm that HDFS ACL synchronization is > complete when snapshot is initiated. We need to identify that and log in > console and log file as well. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (SENTRY-2287) Find a way to report that HDFS ACL synchronization is complete.
[ https://issues.apache.org/jira/browse/SENTRY-2287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kalyan kumar kalvagadda updated SENTRY-2287: Attachment: SENTRY-2287.004.patch > Find a way to report that HDFS ACL synchronization is complete. > --- > > Key: SENTRY-2287 > URL: https://issues.apache.org/jira/browse/SENTRY-2287 > Project: Sentry > Issue Type: Sub-task > Components: Sentry >Affects Versions: 2.1.0 >Reporter: kalyan kumar kalvagadda >Assignee: kalyan kumar kalvagadda >Priority: Major > Attachments: SENTRY-2287.001.patch, SENTRY-2287.002.patch, > SENTRY-2287.003.patch, SENTRY-2287.004.patch > > > Currently there is no way to confirm that HDFS ACL synchronization is > complete when snapshot is initiated. We need to identify that and log in > console and log file as well. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (SENTRY-2489) Fix flaky test testRequestSyncUpdatesWhenPubSubNotifyReturnsFullPathsUpdate
[ https://issues.apache.org/jira/browse/SENTRY-2489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kalyan kumar kalvagadda updated SENTRY-2489: Attachment: (was: SENTRY-2489.001.patch) > Fix flaky test testRequestSyncUpdatesWhenPubSubNotifyReturnsFullPathsUpdate > --- > > Key: SENTRY-2489 > URL: https://issues.apache.org/jira/browse/SENTRY-2489 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.1 >Reporter: kalyan kumar kalvagadda >Assignee: kalyan kumar kalvagadda >Priority: Major > Attachments: SENTRY-2489.001.patch > > > testRequestSyncUpdatesWhenPubSubNotifyReturnsFullPathsUpdate is falky and it > fails pretty commonly. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (SENTRY-1904) TransactionManager should limit the max time spent by transaction retry
[ https://issues.apache.org/jira/browse/SENTRY-1904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kalyan kumar kalvagadda updated SENTRY-1904: Attachment: (was: SENTRY-1904.009.patch) > TransactionManager should limit the max time spent by transaction retry > --- > > Key: SENTRY-1904 > URL: https://issues.apache.org/jira/browse/SENTRY-1904 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.0.0 >Reporter: Alexander Kolbasov >Assignee: kalyan kumar kalvagadda >Priority: Major > Attachments: SENTRY-1904.001.patch, SENTRY-1904.002.patch, > SENTRY-1904.003.patch, SENTRY-1904.004.patch, SENTRY-1904.005.patch, > SENTRY-1904.006.patch, SENTRY-1904.007.patch, SENTRY-1904.008.patch, > SENTRY-1904.009.patch > > > The TransactionManager uses exponential backoff strategy for transaction > retries. This may cause some transactions to be delayed by a very long time. > We should also have a constraint on the max time for a transaction so that we > do not retry for too long. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (SENTRY-1904) TransactionManager should limit the max time spent by transaction retry
[ https://issues.apache.org/jira/browse/SENTRY-1904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kalyan kumar kalvagadda updated SENTRY-1904: Attachment: SENTRY-1904.009.patch > TransactionManager should limit the max time spent by transaction retry > --- > > Key: SENTRY-1904 > URL: https://issues.apache.org/jira/browse/SENTRY-1904 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.0.0 >Reporter: Alexander Kolbasov >Assignee: kalyan kumar kalvagadda >Priority: Major > Attachments: SENTRY-1904.001.patch, SENTRY-1904.002.patch, > SENTRY-1904.003.patch, SENTRY-1904.004.patch, SENTRY-1904.005.patch, > SENTRY-1904.006.patch, SENTRY-1904.007.patch, SENTRY-1904.008.patch, > SENTRY-1904.009.patch > > > The TransactionManager uses exponential backoff strategy for transaction > retries. This may cause some transactions to be delayed by a very long time. > We should also have a constraint on the max time for a transaction so that we > do not retry for too long. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (SENTRY-2489) Fix flaky test testRequestSyncUpdatesWhenPubSubNotifyReturnsFullPathsUpdate
[ https://issues.apache.org/jira/browse/SENTRY-2489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kalyan kumar kalvagadda updated SENTRY-2489: Attachment: SENTRY-2489.001.patch > Fix flaky test testRequestSyncUpdatesWhenPubSubNotifyReturnsFullPathsUpdate > --- > > Key: SENTRY-2489 > URL: https://issues.apache.org/jira/browse/SENTRY-2489 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.1 >Reporter: kalyan kumar kalvagadda >Assignee: kalyan kumar kalvagadda >Priority: Major > Attachments: SENTRY-2489.001.patch > > > testRequestSyncUpdatesWhenPubSubNotifyReturnsFullPathsUpdate is falky and it > fails pretty commonly. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
