[jira] [Commented] (SENTRY-2300) Move Permission Update due to DDL to HMS Post Event Listener
[
https://issues.apache.org/jira/browse/SENTRY-2300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16642471#comment-16642471
]
Alexander Kolbasov commented on SENTRY-2300:
What is the plan for dealing with failures? The approach is Ok as long as
things work as expected. What if the sync perm update fails? This will
potentially cause incorrect permissions on objects. How would these be
reconciled/fixed?
> Move Permission Update due to DDL to HMS Post Event Listener
>
>
> Key: SENTRY-2300
> URL: https://issues.apache.org/jira/browse/SENTRY-2300
> Project: Sentry
> Issue Type: Bug
> Components: Sentry
>Affects Versions: 2.1.0, 2.2.0
>Reporter: Na Li
>Assignee: Na Li
>Priority: Major
> Attachments: SENTRY-2300.001.patch, SENTRY-2300.002.patch
>
>
> There was a code in MetastorePlugin that modified Sentry privileges on table
> Create/Drop and database Create/Drop. As part of Sentry HA work we moved all
> this logic from Sentry plugin to be driven by notifications which required
> the extra synchronization between HMS and Sentry.
> It should be possible to do permission changes in the post event listener
> itself to avoid blocking for Sentry. This requires some experiments though
> because it may cause strange artifacts since at the time these DDL operations
> are done Sentry may not be aware of the current state - for example you may
> try to change permissions of a table that Sentry doesn’t know about, which
> seems to be OK.
> This update will have the following benefits:
> {code}
> * HMS waits on Sentry polling HMS update takes 0.5 to 1 second. This update
> will remove this delay
> * Sentry knows every DDL update, and therefore can update permission
> correctly. In current approach using notification processing, Sentry could
> miss updates if full snapshot is fetched from HMS, and permission is not
> updated correctly. In the case of table rename, when mission DDL update event
> because of full snapshot, sentry will not move the permissions associated
> with old table to the new table. And the authorization on queries on the
> renamed table will fail.
> {code}
> The proposed approach is:
> {code}
> 1) Add the permission update in
> SentryPolicyStoreProcessor.sentry_notify_hms_event() through
> SentrySyncHMSNotificationsPostEventListener for create/drop/alter table,
> create/drop database commands. This is synchronous processing.
> 2) Remove the permission update from
> NotificationProcessor.processNotificationEvent() for create/drop/alter table,
> create/drop database commands.
> 3) Remove the blocking of HMS in SentrySyncHMSNotificationsPostEventListener
> for Sentry fetching notification events asynchronously from HMS for
> create/drop/alter table, create/drop database commands. The reason is that
> now the permission is updated through
> SentrySyncHMSNotificationsPostEventListener. Therefore, there is no need for
> HMS to wait for Sentry fetching those notification events.
> {code}
> Without this approach, the DDL operation take at least 500 to 1000
> milliseconds. It is easy for HMS to encounter SyncTimeoutException when it
> takes long time for Sentry to get full snapshot or fetch notification events.
> Many users choose to remove the post event listener in configuration (HMS not
> waiting for Sentry fetching notification events asynchronously) to get rid of
> this exception. Users cannot take this workaround if they want owner
> privilege enabled.
> This approach makes DDL operation in HMS independent from Sentry fetching
> full snapshot or notification events, and reduces the DDL operation delay
> significantly. With this approach, the DDL operation is in the range of few
> milliseconds. This approach fixes the timeout issue once for all. And users
> can get rid of timeout issue and enable owner privilege at the same time
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (SENTRY-2300) Move Permission Update due to DDL to HMS Post Event Listener
[
https://issues.apache.org/jira/browse/SENTRY-2300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637185#comment-16637185
]
Alexander Kolbasov commented on SENTRY-2300:
[~LinaAtAustin] Can you, please, provide a description of the proposed solution
here?
> Move Permission Update due to DDL to HMS Post Event Listener
>
>
> Key: SENTRY-2300
> URL: https://issues.apache.org/jira/browse/SENTRY-2300
> Project: Sentry
> Issue Type: Bug
> Components: Sentry
>Affects Versions: 2.1.0, 2.2.0
>Reporter: Na Li
>Assignee: Na Li
>Priority: Major
> Attachments: SENTRY-2300.001.patch, SENTRY-2300.002.patch
>
>
> There was a code in MetastorePlugin that modified Sentry privileges on table
> Create/Drop and database Create/Drop. As part of Sentry HA work we moved all
> this logic from Sentry plugin to be driven by notifications which required
> the extra synchronization between HMS and Sentry.
> It should be possible to do permission changes in the post event listener
> itself to avoid blocking for Sentry. This requires some experiments though
> because it may cause strange artifacts since at the time these DDL operations
> are done Sentry may not be aware of the current state - for example you may
> try to change permissions of a table that Sentry doesn’t know about, which
> seems to be OK.
> This update will have the following benefits:
> {code}
> * HMS waits on Sentry polling HMS update takes 0.5 to 1 second. This update
> will remove this delay
> * Sentry knows every DDL update, and therefore can update permission
> correctly. In current approach using notification processing, Sentry could
> miss updates if full snapshot is fetched from HMS, and permission is not
> updated correctly. In the case of table rename, when mission DDL update event
> because of full snapshot, sentry will not move the permissions associated
> with old table to the new table. And the authorization on queries on the
> renamed table will fail.
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (SENTRY-2416) FullUpdateInitializer metrics are not reset for each new HMS snapshot
[ https://issues.apache.org/jira/browse/SENTRY-2416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16625343#comment-16625343 ] Alexander Kolbasov commented on SENTRY-2416: IMO there is value in both per-snapshot counters as well as globally aggregated counters. What do you think about adding per-snapshot counters and keeping existing ones? > FullUpdateInitializer metrics are not reset for each new HMS snapshot > - > > Key: SENTRY-2416 > URL: https://issues.apache.org/jira/browse/SENTRY-2416 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.0.0 >Reporter: Sergio Peña >Assignee: Sergio Peña >Priority: Major > Attachments: SENTRY-2416.1.patch > > > The counter metrics displayed on the logs for the FullUpdateInitializer keep > increasing whenever new snapshots happen. This is not allowing us to clearly > track the progress of a HMS snapshot in Sentry. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (SENTRY-2414) Add reconnect logic to database
[ https://issues.apache.org/jira/browse/SENTRY-2414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16623944#comment-16623944 ] Alexander Kolbasov commented on SENTRY-2414: Does connection pool (BoneCP or HikariCP) deal with it? > Add reconnect logic to database > --- > > Key: SENTRY-2414 > URL: https://issues.apache.org/jira/browse/SENTRY-2414 > Project: Sentry > Issue Type: Bug > Components: Sentry >Reporter: Arjun Mishra >Assignee: Arjun Mishra >Priority: Major > > As of now if connection between sentry server and database is lost, Sentry > service doesn't recover even if network reestablishes itself -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (SENTRY-2396) Minimize the time sentry blocks HMS threads
[ https://issues.apache.org/jira/browse/SENTRY-2396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16607832#comment-16607832 ] Alexander Kolbasov commented on SENTRY-2396: Looks like SENTRY-2300 is more experimental and it changes semantics in some non-trivial ways. What is proposed here looks like a simpler change - there is no reason not to kick HMSFollower to wake up early. The only issue here is that we may be waiting on the follower side, so these guys would have to wait longer. It would be useful to also decrease the default time between polls. > Minimize the time sentry blocks HMS threads > --- > > Key: SENTRY-2396 > URL: https://issues.apache.org/jira/browse/SENTRY-2396 > Project: Sentry > Issue Type: Sub-task > Components: Sentry >Affects Versions: 2.1.0 >Reporter: kalyan kumar kalvagadda >Assignee: kalyan kumar kalvagadda >Priority: Major > Attachments: SENTRY-2396.001.patch, SENTRY-2396.002.patch > > > Currently HMS is blocked for max of one sec by sentry for below operations > # create database > # drop database > # create table > # drop table > # alter table > Reason for this is to make sure that sentry fetched the notification event > from HMS and processed. > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (SENTRY-2371) Add a new thrift API for getting all privileges a user has
[ https://issues.apache.org/jira/browse/SENTRY-2371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16596810#comment-16596810 ] Alexander Kolbasov commented on SENTRY-2371: I think Sentry can do these kind of queries more efficiently since it can get the result from a single transaction. Otherwise there is always a chance of inconsistency if something changed in between. > Add a new thrift API for getting all privileges a user has > --- > > Key: SENTRY-2371 > URL: https://issues.apache.org/jira/browse/SENTRY-2371 > Project: Sentry > Issue Type: Improvement >Reporter: Hao Hao >Priority: Major > > SENTRY-2256 introduces an API to list the privileges directly assigned to a > user. But there isn't an API to list all privileges a user has including the > ones granted to the group the user is in. Such an API is important when > clients need to find out all privilege a user has. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (SENTRY-2370) Create a cache of PathUpdates to send to NN
[ https://issues.apache.org/jira/browse/SENTRY-2370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16596808#comment-16596808 ] Alexander Kolbasov commented on SENTRY-2370: Can you elaborate on the caching strategy? Is it only for the case when NN failed to get a full snapshot? How do you invalidate the cache if something changed since then? > Create a cache of PathUpdates to send to NN > --- > > Key: SENTRY-2370 > URL: https://issues.apache.org/jira/browse/SENTRY-2370 > Project: Sentry > Issue Type: Sub-task > Components: Sentry >Affects Versions: 2.1.0 >Reporter: Arjun Mishra >Assignee: Arjun Mishra >Priority: Major > Attachments: SENTRY-2370.01.patch, SENTRY-2370.02.patch > > > When NN requests path updates from sentry and if it exceeds the time > threshold, on consecutive attempts sentry will attempt to fetch the full > update from scratch. Instead it should cache it and update the cache before > sending it to NN. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (SENTRY-2367) Implement subsystem to allow for pluggable attribute providers and transports
[ https://issues.apache.org/jira/browse/SENTRY-2367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16592072#comment-16592072 ] Alexander Kolbasov commented on SENTRY-2367: Is it possible to use pom-based dependency on keycloak instead? I don't like the idea of just copying files. Sentry is a security product and this code deals with dynamic loading of jar files. Suppose that there is a security exploit in keycloak that is discovered and fixed upstream - we would hard time figuring out that there is a dependency in Sentry. Also it makes version management difficult. > Implement subsystem to allow for pluggable attribute providers and transports > - > > Key: SENTRY-2367 > URL: https://issues.apache.org/jira/browse/SENTRY-2367 > Project: Sentry > Issue Type: Sub-task > Components: Core >Affects Versions: 2.0.1 >Reporter: Brian Towles >Assignee: Brian Towles >Priority: Major > Fix For: 2.1.0 > > Attachments: SENTRY-2367.001.patch, SENTRY-2367.002.patch > > > Implement a subsystem for Sentry to for the pluggable loading of attribute > providers and transports. This will be done with the Java SPI interface and > mechanisms. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (SENTRY-2367) Implement subsystem to allow for pluggable attribute providers and transports
[ https://issues.apache.org/jira/browse/SENTRY-2367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16591043#comment-16591043 ] Alexander Kolbasov commented on SENTRY-2367: I see that some classes have @author annotations. Where do these come from? > Implement subsystem to allow for pluggable attribute providers and transports > - > > Key: SENTRY-2367 > URL: https://issues.apache.org/jira/browse/SENTRY-2367 > Project: Sentry > Issue Type: Sub-task > Components: Core >Affects Versions: 2.0.1 >Reporter: Brian Towles >Assignee: Brian Towles >Priority: Major > Fix For: 2.1.0 > > Attachments: SENTRY-2367.001.patch, SENTRY-2367.002.patch > > > Implement a subsystem for Sentry to for the pluggable loading of attribute > providers and transports. This will be done with the Java SPI interface and > mechanisms. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (SENTRY-2367) Implement subsystem to allow for pluggable attribute providers and transports
[ https://issues.apache.org/jira/browse/SENTRY-2367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16591038#comment-16591038 ] Alexander Kolbasov commented on SENTRY-2367: So the use case is for users to set up and deploy Sentry and then drop some jars in certain directories and provide a way for this jars to be loaded into Sentry JVM? > Implement subsystem to allow for pluggable attribute providers and transports > - > > Key: SENTRY-2367 > URL: https://issues.apache.org/jira/browse/SENTRY-2367 > Project: Sentry > Issue Type: Sub-task > Components: Core >Affects Versions: 2.0.1 >Reporter: Brian Towles >Assignee: Brian Towles >Priority: Major > Fix For: 2.1.0 > > Attachments: SENTRY-2367.001.patch, SENTRY-2367.002.patch > > > Implement a subsystem for Sentry to for the pluggable loading of attribute > providers and transports. This will be done with the Java SPI interface and > mechanisms. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (SENTRY-2344) create a branch for 2.0.1 based on 2.0.0 branch
[ https://issues.apache.org/jira/browse/SENTRY-2344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16578537#comment-16578537 ] Alexander Kolbasov commented on SENTRY-2344: How can we do this with tags then? Don't we need a branch to set all the versions in the poms correctly?How do other projects do that? > create a branch for 2.0.1 based on 2.0.0 branch > --- > > Key: SENTRY-2344 > URL: https://issues.apache.org/jira/browse/SENTRY-2344 > Project: Sentry > Issue Type: Sub-task > Components: Sentry >Affects Versions: 2.0.1 >Reporter: kalyan kumar kalvagadda >Assignee: kalyan kumar kalvagadda >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (SENTRY-2344) create a branch for 2.0.1 based on 2.0.0 branch
[ https://issues.apache.org/jira/browse/SENTRY-2344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16578487#comment-16578487 ] Alexander Kolbasov commented on SENTRY-2344: Are there any other things that should be done for 2.0.1 release like versions in the pom files? > create a branch for 2.0.1 based on 2.0.0 branch > --- > > Key: SENTRY-2344 > URL: https://issues.apache.org/jira/browse/SENTRY-2344 > Project: Sentry > Issue Type: Sub-task > Components: Sentry >Affects Versions: 2.0.1 >Reporter: kalyan kumar kalvagadda >Assignee: kalyan kumar kalvagadda >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (SENTRY-2320) CLI: Refactor existing Sentry CLIs to use jcommander instead of commons-cli
[ https://issues.apache.org/jira/browse/SENTRY-2320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16551090#comment-16551090 ] Alexander Kolbasov commented on SENTRY-2320: I would also suggest evaluating picocli for this purpose. > CLI: Refactor existing Sentry CLIs to use jcommander instead of commons-cli > --- > > Key: SENTRY-2320 > URL: https://issues.apache.org/jira/browse/SENTRY-2320 > Project: Sentry > Issue Type: Sub-task >Reporter: Anthony Young-Garner >Assignee: Anthony Young-Garner >Priority: Major > > In preparation for ABCM CLI changes discussed in SENTRY-2227 and as discussed > on the mailing list, the existing CLIs will be refactored to use the > jcommander library instead of commons-cli -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (SENTRY-2304) Optimize the time taken for HMS snapshot creation.
[ https://issues.apache.org/jira/browse/SENTRY-2304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16538965#comment-16538965 ] Alexander Kolbasov commented on SENTRY-2304: A simple WHERE clause for DB name doesn't make things slower and all tables for all DBs may be pretty large list for some installations. > Optimize the time taken for HMS snapshot creation. > -- > > Key: SENTRY-2304 > URL: https://issues.apache.org/jira/browse/SENTRY-2304 > Project: Sentry > Issue Type: Sub-task > Components: Sentry >Affects Versions: 2.1.0 >Reporter: kalyan kumar kalvagadda >Priority: Major > > First get all the database names and their locations. > # Create Table Task for each database by providing the db name > # DB Task will get the names and locations for all the tables in that > database. > # Create Partition task for each table in that database by providing the > database and table names. > ## Table task will get the locations of all the partitions in that table. > This approach needs new API’s implemented in HMS to get the names and > locations of the databases/tables/partitions. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (SENTRY-2269) Make SentryStore pluggable
[
https://issues.apache.org/jira/browse/SENTRY-2269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16517651#comment-16517651
]
Alexander Kolbasov commented on SENTRY-2269:
Comment for the actual code change. In this case SentryStore *is* the interface
and existing SentryStore is an implementation so I would argue that instead of
creating {{SentryStoreInterface}} and having {{SentryStore}} as its
implementation it should be another way around.
Please also add a disclaimer that we are free to change/modify the interface at
any time and will break any client using it outside of Sentry code base.
> Make SentryStore pluggable
> --
>
> Key: SENTRY-2269
> URL: https://issues.apache.org/jira/browse/SENTRY-2269
> Project: Sentry
> Issue Type: Improvement
> Components: sentrystore
>Affects Versions: 2.1.0
>Reporter: Fahd Siddiqui
>Assignee: Fahd Siddiqui
>Priority: Major
> Attachments: SENTRY-2269.1.patch
>
>
> Make SentryStore pluggable so a different implementation can be plugged in at
> run-time using a config property ("sentry.service.sentrystore"), similar to
> what we have for processor factories.
> This would entail extracting all public methods of SentryStore to a
> SentryStoreInterface and converting all call sites to program to the
> interface.
> It will default to the existing SentryStore.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (SENTRY-2269) Make SentryStore pluggable
[
https://issues.apache.org/jira/browse/SENTRY-2269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16511669#comment-16511669
]
Alexander Kolbasov commented on SENTRY-2269:
Can you explain why this is a useful feature? What is the use case? Do you
envision other components writing different implementations? Does this affect
stability levels of SentryStore interfaces? Do you intend to include
alternative implementation in a later commit?
> Make SentryStore pluggable
> --
>
> Key: SENTRY-2269
> URL: https://issues.apache.org/jira/browse/SENTRY-2269
> Project: Sentry
> Issue Type: Improvement
> Components: sentrystore
>Affects Versions: 2.1.0
>Reporter: Fahd Siddiqui
>Assignee: Fahd Siddiqui
>Priority: Major
> Attachments: SENTRY-2269.1.patch
>
>
> Make SentryStore pluggable so a different implementation can be plugged in at
> run-time using a config property ("sentry.service.sentrystore"), similar to
> what we have for processor factories.
> This would entail extracting all public methods of SentryStore to a
> SentryStoreInterface and converting all call sites to program to the
> interface.
> It will default to the existing SentryStore.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (SENTRY-2250) Optimize the logic of fetching the notification_log entries
[ https://issues.apache.org/jira/browse/SENTRY-2250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16502611#comment-16502611 ] Alexander Kolbasov commented on SENTRY-2250: We don't really need a thread pool since there is a single stream of notifications, so a single thread is good enough. We just need to replace the existing periodic executor with a thread that can be waken up by either # Timer expiration # New request coming > Optimize the logic of fetching the notification_log entries > --- > > Key: SENTRY-2250 > URL: https://issues.apache.org/jira/browse/SENTRY-2250 > Project: Sentry > Issue Type: Improvement > Components: Sentry >Reporter: kalyan kumar kalvagadda >Assignee: kalyan kumar kalvagadda >Priority: Major > > Currently sentry fetches entries from notification_log in HMS periodically. > This could result in delay 0.5 sec to 1 sec in fetching the events and > processing them. > As HMS threads are blocked till sentry processes the notification events, > this delay in processing notifications would result in delay in query > executions. > > Proposal is to proactively fetch the notification log entires from HMS. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (SENTRY-2225) Generic Attribute Ingestion and Default Implementation
[ https://issues.apache.org/jira/browse/SENTRY-2225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16471707#comment-16471707 ] Alexander Kolbasov commented on SENTRY-2225: Hmm, this is a bit confusing. If HMS already has all the information about these attribute, why would we want to copy then to Sentry so that HMS can (again) get them if it has these attributes in the first place? Are you suggesting implementing column-level masking without storing these attributes in Sentry and by storing them in HMS directly or you propose something else? Can you clarify your proposal? > Generic Attribute Ingestion and Default Implementation > -- > > Key: SENTRY-2225 > URL: https://issues.apache.org/jira/browse/SENTRY-2225 > Project: Sentry > Issue Type: Sub-task > Components: Core >Affects Versions: 2.1.0 >Reporter: Anthony Young-Garner >Priority: Major > Labels: ABAC > > As discussed in the design document linked on SENTRY-2140, attributes and > their mapping to columns are created and stored in an external system. In > order for Sentry to make masking decisions based upon these attributes and > mappings, this information must be ingested from the external system. The > scope of this Jira is to : > # implement the generic extensible framework by which different external > systems may contribute attributes (the specific details of the design are > still under discussion on the parent Jira; whether there is a full plugin > model implemented in Sentry or whether the ingestion process will run > entirely external to Sentry and send the information to Sentry via Thrift API > is not yet decided). > # Implement at least one default implementation (whether this will be an > example implementation only for reference like a static text file or a > full-featured implementation more suitable for production use is under > discussion) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (SENTRY-2140) Attribute based access control
[ https://issues.apache.org/jira/browse/SENTRY-2140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16469254#comment-16469254 ] Alexander Kolbasov commented on SENTRY-2140: This sounds reasonable. Another way to do it is to have top-level ABAC JIRA and have various sub-jiras below it (e.g. column masking, etc). The drawback of that approach is that you can only have subtasks, not sub-jiras so now you can't have a subtask of a subtask which is inconvenient. What may be really confusing if we close ABAC jira and it turns out that there is a lot to be done still. > Attribute based access control > -- > > Key: SENTRY-2140 > URL: https://issues.apache.org/jira/browse/SENTRY-2140 > Project: Sentry > Issue Type: New Feature > Components: Core >Reporter: Steve Moist >Priority: Major > Labels: ABAC > Attachments: Sentry ABAC Proposal v1.1.pdf, Sentry ABAC Proposal.pdf > > > As a user, I want to have finer grain control over which users/roles can view > data in Hive. Some information such as Social Security Number is considered > very confidential information. I want to be able to tag columns in Hive with > "attributes" that prevent users/roles from not accessing or seeing the data. > For users/roles that have that attribute, they should be able to see that > information. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (SENTRY-2140) Attribute based access control
[
https://issues.apache.org/jira/browse/SENTRY-2140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16464528#comment-16464528
]
Alexander Kolbasov commented on SENTRY-2140:
[~anthony.young-gar...@cloudera.com] Thank you for providing detailed
clarifications. This is very useful.
Some comments below.
{quote}
2) Why wouldn't attribute based privileges be granted in beeline rather than
through the Sentry CLI?
In the intended users groups, there are different levels of admins - data
admins and org admins. They want to administer things differently and have
different skill sets. Data admins who grant role based access to data are not
the same admins who assign policies to attributes and apply attributes to data.
However, it may be both possible and useful in the scope of the project to
allow data admins to at least view attribute based policies through beeline.
{quote}
While I do not necessarily buy the argument of different levels of admins -
some orgs may have different levels, some may have the same two admins who do
everything, I understand your thought that all these attributes can come from
external source, so setting them via Beeline may not be practical. I still
think that viewing these through Beeline or Impala shell is important since
this is the primary interface used. That said, I understand that this requires
changes outside of Sentry (and outside your control) so it should be Ok to
delay these.
As for item 3, it seems like the existing proposal only concerns column level
masking, so it isn't really ABAC, right?
{quote}
5) The file-based attribute provider doesn't seem useful in production. What is
its purpose?
The file-based attribute provider is a reference implementation, similar to the
Hadoop Java KeyStore Key Provider in Hadoop. It demonstrates how to implement a
provider using any external attribute source.
{quote}
I think the fundamental issue here isn't the file-based provider but push vs
pull model. So far Sentry uses push model - someone needs to initiate a Thrift
call to the Sentry server to modify its set of privileges, roles, etc. You are
proposing pull model instead which complicates things seriously - now you need
to define some kind of pull plugin architecture. I am not convinced that such
pull model is needed. For example, even if some external providers can not push
to Sentry for whatever reasons you could have a separate daemon/service that
pulls from such providers and pushes updates to Sentry periodically (e.g, using
cron jobs). Moving this functionality into the actual Sentry server isn't
required for implementing column-level masking. And without the pull model
there is no need for file-based provider.
{quote}
6) Why is a cache needed?
I wanted to split the question about a) the architectural need for a cache from
b) the question about how that cache is implemented.
{quote}
{quote}
6a. So, first, the architectural need for a cache is driven by the fact that
attributes reside in an external system (as discussed in item 1). Since the
system of record for attributes is external to Sentry, attribute data must be
either retrieved (pulled) or sent (pushed) from the external system to Sentry.
Rather than retrieving attribute data each time it is needed from the external
system, Sentry will retain some copy of the attribute data locally (to reduce
load on the external system and also to speed up access to attribute data). A
local copy of remote data kept for purposes of faster data access is a cache by
definition, however it is implemented.
{quote}
I think we should distinguish between the mechanism to get attributes to the
Sentry service and the mechanism to store them. Whatever the mechanism is,
currently Sentry uses RDBMS to store attributes. It doesn't really matter where
they come from - Sentry just responds to requests and stores data in RDBMS or
retrieves from RDBMS. You may call RDBMS storage a "cache" if you want. I do
not see a need to introduce a separate storage mechanism for column level
masking metadata - we should keep things simple and use the same mechanisms
unless there are some fundamental reasons not to.
{quote}
6b. Given that we will definitely have a de facto cache of some type (however
implemented) in the design (as discussed in 6a), we investigated using the
existing Sentry datastore as a cache (see SENTRY-2196). We've implemented this
in prototype sufficiently to understand that this will work as long as the
schema in which attributes are stored has no joins with the rest of the Sentry
datastore schema.
{quote}
This should be fine, Why did you decide to abandon this approach?
{quote}
However, in discussion with Impala committers, we understood that the Impala
team was hoping to avoid increasing the number of direct calls that require
copying data from the Sentry database into Impala (this significantly impacts
[jira] [Commented] (SENTRY-2201) Initial Attriubte based access control
[ https://issues.apache.org/jira/browse/SENTRY-2201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16459284#comment-16459284 ] Alexander Kolbasov commented on SENTRY-2201: My understanding is that it is a prototype. Since we are still having a discussion on the design let's hold off putting prototypes in until we agree on the general design. > Initial Attriubte based access control > --- > > Key: SENTRY-2201 > URL: https://issues.apache.org/jira/browse/SENTRY-2201 > Project: Sentry > Issue Type: New Feature > Components: Sentry >Affects Versions: 2.1.0 >Reporter: Steve Moist >Assignee: Steve Moist >Priority: Major > Fix For: 2.1.0 > > Attachments: SENTRY-2201-0004.patch, SENTRY-2201-001.patch, > SENTRY-2201-002.patch, SENTRY-2201-003.patch, SENTRY-2201-005.patch > > > This is the initial version of attribute based access control. It > incorporates a proof of concept that was developed and Liam's > ([https://reviews.apache.org/r/66480/)] attribute ingestor as a starting > point. Not everything is fully complete based on the document posted but we > plan to get there and use this as a starting point for further development > and refinement. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (SENTRY-2170) Update the Sentry-HDFS thrift for user level privileges.
[ https://issues.apache.org/jira/browse/SENTRY-2170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16455574#comment-16455574 ] Alexander Kolbasov commented on SENTRY-2170: Do we need to preserve on the wire compatibility between old protocol and the new one? > Update the Sentry-HDFS thrift for user level privileges. > > > Key: SENTRY-2170 > URL: https://issues.apache.org/jira/browse/SENTRY-2170 > Project: Sentry > Issue Type: Sub-task > Components: Sentry >Affects Versions: 2.1.0 >Reporter: kalyan kumar kalvagadda >Assignee: kalyan kumar kalvagadda >Priority: Major > Attachments: SENTRY-2170.001.patch, SENTRY-2170.002.patch > > > Thrift interface between sentry server and hdfs sentry client should be > updated to send privileges granted to users. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (SENTRY-2140) Attribute based access control
[ https://issues.apache.org/jira/browse/SENTRY-2140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16453211#comment-16453211 ] Alexander Kolbasov commented on SENTRY-2140: [~moist] It would be good if you send email to dev@ alias and explain all this. > Attribute based access control > -- > > Key: SENTRY-2140 > URL: https://issues.apache.org/jira/browse/SENTRY-2140 > Project: Sentry > Issue Type: New Feature > Components: Core >Reporter: Steve Moist >Priority: Major > Attachments: Sentry ABAC Proposal v1.1.pdf, Sentry ABAC Proposal.pdf > > > As a user, I want to have finer grain control over which users/roles can view > data in Hive. Some information such as Social Security Number is considered > very confidential information. I want to be able to tag columns in Hive with > "attributes" that prevent users/roles from not accessing or seeing the data. > For users/roles that have that attribute, they should be able to see that > information. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (SENTRY-2140) Attribute based access control
[ https://issues.apache.org/jira/browse/SENTRY-2140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16453154#comment-16453154 ] Alexander Kolbasov commented on SENTRY-2140: [~moist] What is the proposed timeline for this work? Do you think that it should be in the scope of upcoming Sentry release (which currently is supposed to have FGP changes only) or go to the next one? What are your thoughts on the way to work on this - do you plan on working in master or in a feature branch? Or you plan to do a huge drop in master at some point? > Attribute based access control > -- > > Key: SENTRY-2140 > URL: https://issues.apache.org/jira/browse/SENTRY-2140 > Project: Sentry > Issue Type: New Feature > Components: Core >Reporter: Steve Moist >Priority: Major > Attachments: Sentry ABAC Proposal v1.1.pdf, Sentry ABAC Proposal.pdf > > > As a user, I want to have finer grain control over which users/roles can view > data in Hive. Some information such as Social Security Number is considered > very confidential information. I want to be able to tag columns in Hive with > "attributes" that prevent users/roles from not accessing or seeing the data. > For users/roles that have that attribute, they should be able to see that > information. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (SENTRY-2140) Attribute based access control
[ https://issues.apache.org/jira/browse/SENTRY-2140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16453112#comment-16453112 ] Alexander Kolbasov commented on SENTRY-2140: [~moist] Sentry used to work in the way you described - there was a policy file that was the description of the policy. It moved away from this for obvious reasons - let's avoid getting back to file-based solutions. Sentry is now an API-based service. > Attribute based access control > -- > > Key: SENTRY-2140 > URL: https://issues.apache.org/jira/browse/SENTRY-2140 > Project: Sentry > Issue Type: New Feature > Components: Core >Reporter: Steve Moist >Priority: Major > Attachments: Sentry ABAC Proposal v1.1.pdf, Sentry ABAC Proposal.pdf > > > As a user, I want to have finer grain control over which users/roles can view > data in Hive. Some information such as Social Security Number is considered > very confidential information. I want to be able to tag columns in Hive with > "attributes" that prevent users/roles from not accessing or seeing the data. > For users/roles that have that attribute, they should be able to see that > information. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (SENTRY-2202) Revoking SELECT or INSERT from parent privilege does not get applied in Impala
[ https://issues.apache.org/jira/browse/SENTRY-2202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16449154#comment-16449154 ] Alexander Kolbasov commented on SENTRY-2202: [~LinaAtAustin] I don't understand your last comment. Why do you think it should be the case? What does 'all' privilege allow the user to do exactly? > Revoking SELECT or INSERT from parent privilege does not get applied in Impala > -- > > Key: SENTRY-2202 > URL: https://issues.apache.org/jira/browse/SENTRY-2202 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.1.0 >Reporter: Arjun Mishra >Assignee: Arjun Mishra >Priority: Major > Fix For: 2.1.0 > > > When revoking select or insert from privilege, child privilege should be > appropriately updated. For eg if there is ALL on table and SELECT on database > and SELECT is revoked from database, then table privileges should be changed > from ALL to INSERT. This is not happening in Impala because when looking for > child privilege we only filter by "\*" as opposed to both "\*" or "all" > depending on the original privilege -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (SENTRY-2203) Leader Lock is not released when Sentry service shuts down
[ https://issues.apache.org/jira/browse/SENTRY-2203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16446379#comment-16446379 ] Alexander Kolbasov commented on SENTRY-2203: [~LinaAtAustin] While it isn't a problem it is still good to shutdown cleanly and to clean up ZK state whenever possible, so I suggest we still go ahead with your fix. What do you think? > Leader Lock is not released when Sentry service shuts down > -- > > Key: SENTRY-2203 > URL: https://issues.apache.org/jira/browse/SENTRY-2203 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.1.0 >Reporter: Na Li >Assignee: Na Li >Priority: Critical > Attachments: SENTRY-2203.001.patch > > > In our testing for sentry HA, we found after restarting sentry service > without restarting zookeeper service, it is possible that none of sentry > servers is elected as leader to sync with HMS. > What happened was > 1) When a leader is elected, the sentry server host holds the leader lock. > The lock is identified by the mutexPath. All sentry servers in a cluster use > the same mutexPath. > 2) When sentry service is shutdown, the HAContext is shutdown, so its > contained CuratorFrameworkImpl was shutdown, but the leader lock was still > hold by the sentry server host > 3) When the Interruption signal from shutdown caused the leader election > thread to be interrupted, releasing the leader lock failed because > CuratorFrameworkImpl was not in started state. > 4) When sentry server restarts, acquiring the leader lock failed because it > was not released. So no active sentry servers is leader. > 5) If releasing leader lock happened before CuratorFrameworkImpl was > shutdown, this issue won't happen. If restarting zookeeper after sentry > service restart, this issue won't happen. > To fix this issue, > Sentry LeaderStatusMonitor can deactivate the leader to release the leader > lock when it is closed, so the leader lock can be guaranteed to release > before CuratorFrameworkImpl is shutdown. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (SENTRY-1953) CounterWait does not remove timed out entries from the queue
[ https://issues.apache.org/jira/browse/SENTRY-1953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Kolbasov resolved SENTRY-1953. Resolution: Won't Fix > CounterWait does not remove timed out entries from the queue > > > Key: SENTRY-1953 > URL: https://issues.apache.org/jira/browse/SENTRY-1953 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.0.0 >Reporter: Alexander Kolbasov >Assignee: Alexander Kolbasov >Priority: Major > > When CounterWait waitFor() times out, the waiter is woken up, but the entry > remains in the queue. It doesn't break anything but just consume resources > and represents a potential memory leak. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (SENTRY-2207) Refactor out Sentry CLI from sentry-provider-db into own module
[ https://issues.apache.org/jira/browse/SENTRY-2207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16443162#comment-16443162 ] Alexander Kolbasov commented on SENTRY-2207: Note that we have a Sentry-tools module which contains an interactive CLI. It seems to be a good place to put the original CLI as well. > Refactor out Sentry CLI from sentry-provider-db into own module > --- > > Key: SENTRY-2207 > URL: https://issues.apache.org/jira/browse/SENTRY-2207 > Project: Sentry > Issue Type: Improvement > Components: Sentry >Affects Versions: 2.1.0 >Reporter: Steve Moist >Assignee: Steve Moist >Priority: Major > Fix For: 2.1.0 > > > Refactor out the sentry cli parts into their own module in > sentry-main/sentry-main-cli. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (SENTRY-2208) Refactor out Sentry service into own module from sentry-provider-db
[ https://issues.apache.org/jira/browse/SENTRY-2208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16443158#comment-16443158 ] Alexander Kolbasov commented on SENTRY-2208: This work was done earlier as SENTRY-1205 but was later reverted from master due to difficulties in merging with Sentry HA branch. > Refactor out Sentry service into own module from sentry-provider-db > --- > > Key: SENTRY-2208 > URL: https://issues.apache.org/jira/browse/SENTRY-2208 > Project: Sentry > Issue Type: Improvement > Components: Sentry >Affects Versions: 2.1.0 >Reporter: Steve Moist >Assignee: Steve Moist >Priority: Major > Fix For: 2.1.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (SENTRY-2203) Leader Lock is not released when Sentry service shuts down
[ https://issues.apache.org/jira/browse/SENTRY-2203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16441703#comment-16441703 ] Alexander Kolbasov commented on SENTRY-2203: LeaderStatusMonitor does extend LeaderSelectorListenerAdapter. > Leader Lock is not released when Sentry service shuts down > -- > > Key: SENTRY-2203 > URL: https://issues.apache.org/jira/browse/SENTRY-2203 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.1.0 >Reporter: Na Li >Assignee: Na Li >Priority: Critical > Attachments: SENTRY-2203.001.patch > > > In our testing for sentry HA, we found after restarting sentry service > without restarting zookeeper service, it is possible that none of sentry > servers is elected as leader to sync with HMS. > What happened was > 1) When a leader is elected, the sentry server host holds the leader lock. > The lock is identified by the mutexPath. All sentry servers in a cluster use > the same mutexPath. > 2) When sentry service is shutdown, the HAContext is shutdown, so its > contained CuratorFrameworkImpl was shutdown, but the leader lock was still > hold by the sentry server host > 3) When the Interruption signal from shutdown caused the leader election > thread to be interrupted, releasing the leader lock failed because > CuratorFrameworkImpl was not in started state. > 4) When sentry server restarts, acquiring the leader lock failed because it > was not released. So no active sentry servers is leader. > 5) If releasing leader lock happened before CuratorFrameworkImpl was > shutdown, this issue won't happen. If restarting zookeeper after sentry > service restart, this issue won't happen. > To fix this issue, > Sentry LeaderStatusMonitor can deactivate the leader to release the leader > lock when it is closed, so the leader lock can be guaranteed to release > before CuratorFrameworkImpl is shutdown. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (SENTRY-2203) Leader Lock is not released when Sentry service shuts down
[ https://issues.apache.org/jira/browse/SENTRY-2203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16441668#comment-16441668 ] Alexander Kolbasov commented on SENTRY-2203: How is this different from a leader just crashing while still holding the lock? I would think that ZK/Curator should detect that the server is down and clean up the lock. Is it a known bug for Curator? If it is, is it fixed in some newer Curator release? Should we upgrade to a newer Curator library? > Leader Lock is not released when Sentry service shuts down > -- > > Key: SENTRY-2203 > URL: https://issues.apache.org/jira/browse/SENTRY-2203 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.1.0 >Reporter: Na Li >Assignee: Na Li >Priority: Critical > Attachments: SENTRY-2203.001.patch > > > In our testing for sentry HA, we found after restarting sentry service > without restarting zookeeper service, it is possible that none of sentry > servers is elected as leader to sync with HMS. > What happened was > 1) When a leader is elected, the sentry server host holds the leader lock. > The lock is identified by the mutexPath. All sentry servers in a cluster use > the same mutexPath. > 2) When sentry service is shutdown, the HAContext is shutdown, so its > contained CuratorFrameworkImpl was shutdown, but the leader lock was still > hold by the sentry server host > 3) When the Interruption signal from shutdown caused the leader election > thread to be interrupted, releasing the leader lock failed because > CuratorFrameworkImpl was not in started state. > 4) When sentry server restarts, acquiring the leader lock failed because it > was not released. So no active sentry servers is leader. > 5) If releasing leader lock happened before CuratorFrameworkImpl was > shutdown, this issue won't happen. If restarting zookeeper after sentry > service restart, this issue won't happen. > To fix this issue, > Sentry LeaderStatusMonitor can deactivate the leader to release the leader > lock when it is closed, so the leader lock can be guaranteed to release > before CuratorFrameworkImpl is shutdown. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (SENTRY-2154) Update schema to grant privileges to user
[ https://issues.apache.org/jira/browse/SENTRY-2154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16408653#comment-16408653 ] Alexander Kolbasov commented on SENTRY-2154: [~LinaAtAustin] [~kkalyan] Can you post proposed changes to the model Java classes and package.jdo file? > Update schema to grant privileges to user > - > > Key: SENTRY-2154 > URL: https://issues.apache.org/jira/browse/SENTRY-2154 > Project: Sentry > Issue Type: Sub-task > Components: Sentry >Affects Versions: 2.1.0 >Reporter: Na Li >Assignee: Na Li >Priority: Major > Fix For: 2.1.0 > > > Need to add new DB table to support grant user to privileges > Also, a flag should be added in privilege table to indicate the privilege is > created by user, or created by sentry implicitly. User can view the implicit > privileges, but cannot change it directly -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (SENTRY-2154) Update schema to grant privileges to user
[ https://issues.apache.org/jira/browse/SENTRY-2154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16408641#comment-16408641 ] Alexander Kolbasov commented on SENTRY-2154: We had some discussion a while ago about the relationship between roles and privileges. Currently it is M:N meaning that there is a pool of roles and a pool of privileges and they refer to each other in some way. I was suggesting to change this to a different model where each role may have a bunch of privileges so it is more like 1:N relationship - in the current model privileges do not make much sense outside of a role. Currently Sentry uses role-based model - only roles have privileges. This proposal changes this - now users can have privileges and users are not roles. So now we need to define relationship between users, groups, roles and privileges. Can someone summarize the proposed relationships between all these? > Update schema to grant privileges to user > - > > Key: SENTRY-2154 > URL: https://issues.apache.org/jira/browse/SENTRY-2154 > Project: Sentry > Issue Type: Sub-task > Components: Sentry >Affects Versions: 2.1.0 >Reporter: Na Li >Assignee: Na Li >Priority: Major > Fix For: 2.1.0 > > > Need to add new DB table to support grant user to privileges > Also, a flag should be added in privilege table to indicate the privilege is > created by user, or created by sentry implicitly. User can view the implicit > privileges, but cannot change it directly -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (SENTRY-2185) Performance Issue: Saving MAuthzPathsMapping should be done in batch for path full snapshot
[
https://issues.apache.org/jira/browse/SENTRY-2185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexander Kolbasov updated SENTRY-2185:
---
Description:
>From the code below, for each MAuthzPathsMapping, there is a query to save
>each MAuthzPathsMapping instance. Need to save the changes in batch.
{code:java}
public void persistFullPathsImage(final Map
authzPaths,
final long notificationID) throws Exception {
tm.executeTransactionWithRetry(
pm -> {
pm.setDetachAllOnCommit(false); // No need to detach objects
deleteNotificationsSince(pm, notificationID + 1);
// persist the notidicationID
pm.makePersistent(new MSentryHmsNotification(notificationID));
// persist the full snapshot
long snapshotID = getCurrentAuthzPathsSnapshotID(pm);
long nextSnapshotID = snapshotID + 1;
pm.makePersistent(new MAuthzPathsSnapshotId(nextSnapshotID));
LOGGER.info("Attempting to commit new HMS snapshot with ID = {}",
nextSnapshotID);
for (Map.Entry authzPath :
authzPaths.entrySet()) {
pm.makePersistent(new MAuthzPathsMapping(nextSnapshotID,
authzPath.getKey(), authzPath.getValue()));
}
return null;
});
}{code}
was:
>From the code below, for each MAuthzPathsMapping, there is a query to save
>each MAuthzPathsMapping instance. Need to save the changes in batch.
{code:java}
/**
* Persist an up-to-date HMS snapshot into Sentry DB in a single transaction
with its latest
* notification ID
*
* @param authzPaths paths to be be persisted
* @param notificationID the latest notificationID associated with the snapshot
* @throws Exception
*/public void persistFullPathsImage(final Map
authzPaths,final long notificationID) throws Exception {
tm.executeTransactionWithRetry(
pm -> {
pm.setDetachAllOnCommit(false); // No need to detach
objectsdeleteNotificationsSince(pm, notificationID + 1);// persist the
notidicationIDpm.makePersistent(new MSentryHmsNotification(notificationID));//
persist the full snapshotlong snapshotID =
getCurrentAuthzPathsSnapshotID(pm);long nextSnapshotID = snapshotID + 1;
pm.makePersistent(new MAuthzPathsSnapshotId(nextSnapshotID));
LOGGER.info("Attempting to commit new HMS snapshot with ID = {}",
nextSnapshotID);for (Map.Entry authzPath :
authzPaths.entrySet()) {
pm.makePersistent(new MAuthzPathsMapping(nextSnapshotID, authzPath.getKey(),
authzPath.getValue()));
}return null;
});
}
{code}
> Performance Issue: Saving MAuthzPathsMapping should be done in batch for path
> full snapshot
> ---
>
> Key: SENTRY-2185
> URL: https://issues.apache.org/jira/browse/SENTRY-2185
> Project: Sentry
> Issue Type: Bug
> Components: Sentry
>Affects Versions: 2.1.0
>Reporter: Na Li
>Priority: Major
>
> From the code below, for each MAuthzPathsMapping, there is a query to save
> each MAuthzPathsMapping instance. Need to save the changes in batch.
> {code:java}
> public void persistFullPathsImage(final Map
> authzPaths,
> final long notificationID) throws Exception {
> tm.executeTransactionWithRetry(
> pm -> {
> pm.setDetachAllOnCommit(false); // No need to detach objects
> deleteNotificationsSince(pm, notificationID + 1);
> // persist the notidicationID
> pm.makePersistent(new MSentryHmsNotification(notificationID));
> // persist the full snapshot
> long snapshotID = getCurrentAuthzPathsSnapshotID(pm);
> long nextSnapshotID = snapshotID + 1;
> pm.makePersistent(new MAuthzPathsSnapshotId(nextSnapshotID));
> LOGGER.info("Attempting to commit new HMS snapshot with ID =
> {}", nextSnapshotID);
> for (Map.Entry authzPath :
> authzPaths.entrySet()) {
> pm.makePersistent(new MAuthzPathsMapping(nextSnapshotID,
> authzPath.getKey(), authzPath.getValue()));
> }
> return null;
> });
> }{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (SENTRY-2185) Performance Issue: Saving MAuthzPathsMapping should be done in batch for path full snapshot
[
https://issues.apache.org/jira/browse/SENTRY-2185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexander Kolbasov updated SENTRY-2185:
---
Description:
>From the code below, for each MAuthzPathsMapping, there is a query to save
>each MAuthzPathsMapping instance. Need to save the changes in batch.
{code:java}
/**
* Persist an up-to-date HMS snapshot into Sentry DB in a single transaction
with its latest
* notification ID
*
* @param authzPaths paths to be be persisted
* @param notificationID the latest notificationID associated with the snapshot
* @throws Exception
*/public void persistFullPathsImage(final Map
authzPaths,final long notificationID) throws Exception {
tm.executeTransactionWithRetry(
pm -> {
pm.setDetachAllOnCommit(false); // No need to detach
objectsdeleteNotificationsSince(pm, notificationID + 1);// persist the
notidicationIDpm.makePersistent(new MSentryHmsNotification(notificationID));//
persist the full snapshotlong snapshotID =
getCurrentAuthzPathsSnapshotID(pm);long nextSnapshotID = snapshotID + 1;
pm.makePersistent(new MAuthzPathsSnapshotId(nextSnapshotID));
LOGGER.info("Attempting to commit new HMS snapshot with ID = {}",
nextSnapshotID);for (Map.Entry authzPath :
authzPaths.entrySet()) {
pm.makePersistent(new MAuthzPathsMapping(nextSnapshotID, authzPath.getKey(),
authzPath.getValue()));
}return null;
});
}
{code}
was:
>From the code below, for each MAuthzPathsMapping, there is a query to save
>each MAuthzPathsMapping instance. Need to save the changes in batch.
/**
* Persist an up-to-date HMS snapshot into Sentry DB in a single transaction
with its latest
* notification ID
*
* @param authzPaths paths to be be persisted
* @param notificationID the latest notificationID associated with the snapshot
* @throws Exception
*/public void persistFullPathsImage(final Map
authzPaths,final long notificationID) throws Exception {
tm.executeTransactionWithRetry(
pm -> {
pm.setDetachAllOnCommit(false); // No need to detach
objectsdeleteNotificationsSince(pm, notificationID + 1);// persist the
notidicationIDpm.makePersistent(new MSentryHmsNotification(notificationID));//
persist the full snapshotlong snapshotID =
getCurrentAuthzPathsSnapshotID(pm);long nextSnapshotID = snapshotID + 1;
pm.makePersistent(new MAuthzPathsSnapshotId(nextSnapshotID));
LOGGER.info("Attempting to commit new HMS snapshot with ID = {}",
nextSnapshotID);for (Map.Entry authzPath :
authzPaths.entrySet()) {
pm.makePersistent(new MAuthzPathsMapping(nextSnapshotID, authzPath.getKey(),
authzPath.getValue()));
}return null;
});
}
> Performance Issue: Saving MAuthzPathsMapping should be done in batch for path
> full snapshot
> ---
>
> Key: SENTRY-2185
> URL: https://issues.apache.org/jira/browse/SENTRY-2185
> Project: Sentry
> Issue Type: Bug
> Components: Sentry
>Affects Versions: 2.1.0
>Reporter: Na Li
>Priority: Major
>
> From the code below, for each MAuthzPathsMapping, there is a query to save
> each MAuthzPathsMapping instance. Need to save the changes in batch.
> {code:java}
> /**
> * Persist an up-to-date HMS snapshot into Sentry DB in a single transaction
> with its latest
> * notification ID
> *
> * @param authzPaths paths to be be persisted
> * @param notificationID the latest notificationID associated with the snapshot
> * @throws Exception
> */public void persistFullPathsImage(final Map
> authzPaths,final long notificationID) throws Exception {
> tm.executeTransactionWithRetry(
> pm -> {
> pm.setDetachAllOnCommit(false); // No need to detach
> objectsdeleteNotificationsSince(pm, notificationID + 1);// persist the
> notidicationIDpm.makePersistent(new
> MSentryHmsNotification(notificationID));// persist the full snapshotlong
> snapshotID = getCurrentAuthzPathsSnapshotID(pm);long nextSnapshotID =
> snapshotID + 1;
> pm.makePersistent(new MAuthzPathsSnapshotId(nextSnapshotID));
> LOGGER.info("Attempting to commit new HMS snapshot with ID = {}",
> nextSnapshotID);for (Map.Entry authzPath :
> authzPaths.entrySet()) {
> pm.makePersistent(new MAuthzPathsMapping(nextSnapshotID, authzPath.getKey(),
> authzPath.getValue()));
> }return null;
> });
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (SENTRY-2184) Performance Issue: MPath is queried for each MAuthzPathsMapping in full snapshot
[
https://issues.apache.org/jira/browse/SENTRY-2184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16406677#comment-16406677
]
Alexander Kolbasov commented on SENTRY-2184:
[~LinaAtAustin] Yes, that's exactly what I am talking about on the write side.
In your fix - do you really need to set maxFetchDepth or not?
Did you run any perf tests to measure the difference in performance?
SENTRY-1937 has some examples.
> Performance Issue: MPath is queried for each MAuthzPathsMapping in full
> snapshot
>
>
> Key: SENTRY-2184
> URL: https://issues.apache.org/jira/browse/SENTRY-2184
> Project: Sentry
> Issue Type: Bug
> Components: Sentry
>Affects Versions: 2.1.0
>Reporter: Na Li
>Assignee: Na Li
>Priority: Critical
> Attachments: SENTRY-2184.001.patch
>
>
> MAuthzPathsMapping contains list of MPath instances. From log message, when
> getting path full snapshot at SentryStore.retrieveFullPathsImageCore(),
> DataNucleus issues a query for all MPath instances associated with each
> MAuthzPathsMapping. Therefore, getting full path image may take a very long
> time.
> The solution is to get MPath in a batch when getting full path image.
> Log Message when DataNucleus issues a query for all MPath instances
> associated with each MAuthzPathsMapping
> {code:java}
> 1) Initially, all MAuthzPathsMapping entries for current snapshot is queried.
> 2018-03-14 11:51:23,999 (main) [DEBUG -
> org.datanucleus.util.Log4JLogger.debug(Log4JLogger.java:58)] SELECT
> 'org.apache.sentry.provider.db.service.model.MAuthzPathsMapping' AS
> NUCLEUS_TYPE,A0.AUTHZ_OBJ_NAME,A0.AUTHZ_SNAPSHOT_ID,A0.CREATE_TIME_MS,A0.AUTHZ_OBJ_ID
> FROM AUTHZ_PATHS_MAPPING A0 WHERE A0.AUTHZ_SNAPSHOT_ID = <1>
> 2) call authzToPaths.getPathStrings() causes MPath to be queried for each
> AUTHZ_OBJ_ID
> 2018-03-14 11:52:27,700 (main) [DEBUG -
> org.datanucleus.util.Log4JLogger.debug(Log4JLogger.java:58)] SELECT
> 'org.apache.sentry.provider.db.service.model.MPath' AS
> NUCLEUS_TYPE,A0.PATH_NAME,A0.PATH_ID FROM AUTHZ_PATH A0 WHERE A0.AUTHZ_OBJ_ID
> = <1>{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (SENTRY-2184) Performance Issue: MPath is queried for each MAuthzPathsMapping in full snapshot
[
https://issues.apache.org/jira/browse/SENTRY-2184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16405730#comment-16405730
]
Alexander Kolbasov commented on SENTRY-2184:
I think we may have a somewhat symmetrical issue when writing snapshots - what
do you think about it?
Also, are there any similar cases where using fetch groups would help? They
don't have to be addressed with your fix but since you have good understanding
of the issue it may be good to file JIRA(s) for other places where this can be
relevant.
> Performance Issue: MPath is queried for each MAuthzPathsMapping in full
> snapshot
>
>
> Key: SENTRY-2184
> URL: https://issues.apache.org/jira/browse/SENTRY-2184
> Project: Sentry
> Issue Type: Bug
> Components: Sentry
>Affects Versions: 2.1.0
>Reporter: Na Li
>Assignee: Na Li
>Priority: Critical
> Attachments: SENTRY-2184.001.patch
>
>
> MAuthzPathsMapping contains list of MPath instances. From log message, when
> getting path full snapshot at SentryStore.retrieveFullPathsImageCore(),
> DataNucleus issues a query for all MPath instances associated with each
> MAuthzPathsMapping. Therefore, getting full path image may take a very long
> time.
> The solution is to get MPath in a batch when getting full path image.
> Log Message when DataNucleus issues a query for all MPath instances
> associated with each MAuthzPathsMapping
> {code:java}
> 1) Initially, all MAuthzPathsMapping entries for current snapshot is queried.
> 2018-03-14 11:51:23,999 (main) [DEBUG -
> org.datanucleus.util.Log4JLogger.debug(Log4JLogger.java:58)] SELECT
> 'org.apache.sentry.provider.db.service.model.MAuthzPathsMapping' AS
> NUCLEUS_TYPE,A0.AUTHZ_OBJ_NAME,A0.AUTHZ_SNAPSHOT_ID,A0.CREATE_TIME_MS,A0.AUTHZ_OBJ_ID
> FROM AUTHZ_PATHS_MAPPING A0 WHERE A0.AUTHZ_SNAPSHOT_ID = <1>
> 2) call authzToPaths.getPathStrings() causes MPath to be queried for each
> AUTHZ_OBJ_ID
> 2018-03-14 11:52:27,700 (main) [DEBUG -
> org.datanucleus.util.Log4JLogger.debug(Log4JLogger.java:58)] SELECT
> 'org.apache.sentry.provider.db.service.model.MPath' AS
> NUCLEUS_TYPE,A0.PATH_NAME,A0.PATH_ID FROM AUTHZ_PATH A0 WHERE A0.AUTHZ_OBJ_ID
> = <1>{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (SENTRY-2184) Performance Issue: MPath is queried for each MAuthzPathsMapping in full snapshot
[
https://issues.apache.org/jira/browse/SENTRY-2184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16405601#comment-16405601
]
Alexander Kolbasov commented on SENTRY-2184:
[~LinaAtAustin] It is great to have native DataNucleus solution for this. Could
you, please
# Explain how your fix works
# Describe how you measured performance difference and what was the performance
improvement from your fix
> Performance Issue: MPath is queried for each MAuthzPathsMapping in full
> snapshot
>
>
> Key: SENTRY-2184
> URL: https://issues.apache.org/jira/browse/SENTRY-2184
> Project: Sentry
> Issue Type: Bug
> Components: Sentry
>Affects Versions: 2.1.0
>Reporter: Na Li
>Assignee: Na Li
>Priority: Critical
> Attachments: SENTRY-2184.001.patch
>
>
> MAuthzPathsMapping contains list of MPath instances. From log message, when
> getting path full snapshot at SentryStore.retrieveFullPathsImageCore(),
> DataNucleus issues a query for all MPath instances associated with each
> MAuthzPathsMapping. Therefore, getting full path image may take a very long
> time.
> The solution is to get MPath in a batch when getting full path image.
> Log Message when DataNucleus issues a query for all MPath instances
> associated with each MAuthzPathsMapping
> {code:java}
> 1) Initially, all MAuthzPathsMapping entries for current snapshot is queried.
> 2018-03-14 11:51:23,999 (main) [DEBUG -
> org.datanucleus.util.Log4JLogger.debug(Log4JLogger.java:58)] SELECT
> 'org.apache.sentry.provider.db.service.model.MAuthzPathsMapping' AS
> NUCLEUS_TYPE,A0.AUTHZ_OBJ_NAME,A0.AUTHZ_SNAPSHOT_ID,A0.CREATE_TIME_MS,A0.AUTHZ_OBJ_ID
> FROM AUTHZ_PATHS_MAPPING A0 WHERE A0.AUTHZ_SNAPSHOT_ID = <1>
> 2) call authzToPaths.getPathStrings() causes MPath to be queried for each
> AUTHZ_OBJ_ID
> 2018-03-14 11:52:27,700 (main) [DEBUG -
> org.datanucleus.util.Log4JLogger.debug(Log4JLogger.java:58)] SELECT
> 'org.apache.sentry.provider.db.service.model.MPath' AS
> NUCLEUS_TYPE,A0.PATH_NAME,A0.PATH_ID FROM AUTHZ_PATH A0 WHERE A0.AUTHZ_OBJ_ID
> = <1>{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (SENTRY-1855) Improve scalability of permission delta updates
[
https://issues.apache.org/jira/browse/SENTRY-1855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexander Kolbasov updated SENTRY-1855:
---
Affects Version/s: 2.1.0
> Improve scalability of permission delta updates
> ---
>
> Key: SENTRY-1855
> URL: https://issues.apache.org/jira/browse/SENTRY-1855
> Project: Sentry
> Issue Type: Bug
> Components: Sentry
>Affects Versions: 2.0.0, 2.1.0
>Reporter: Alexander Kolbasov
>Assignee: Na Li
>Priority: Major
> Fix For: 2.1.0
>
> Attachments: SENTRY-1855.001.patch,
> SENTRY-1855.002-sentry-ha-redesign.patch, SENTRY-1855.002.patch,
> SENTRY-1855.003-master.patch, SENTRY-1855.01-sentry-ha-redesign.patch
>
>
> Looking at the latest stress runs, we noticed that some of the transactions
> could fail to commit to the database (with Duplicate key exception) after
> exhausting all the retries.
> This problem has become more evident if we have more number of clients
> connecting to Sentry to issue the permission updates. Was able to reproduce
> consistently with 15 clients doing 100 operations each.
> In the past we introduced exponential backoff (SENTRY-1821) so as part of
> test run increased the defaults to 750ms sleep and 20 retries. But even after
> this, the cluster still shows up the transaction failures. This change also
> increases the latency of every transaction in sentry.
> We need to revisit this and come up with a better way to solve this problem.
> {code}
> 2017-07-13 13:18:14,449 ERROR
> org.apache.sentry.provider.db.service.persistent.TransactionManager: The
> transaction has reached max retry number, Exception thrown when executing
> query
> javax.jdo.JDOException: Exception thrown when executing query
> at
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:596)
> at org.datanucleus.api.jdo.JDOQuery.execute(JDOQuery.java:252)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore.getRole(SentryStore.java:294)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore.alterSentryRoleGrantPrivilegeCore(SentryStore.java:645)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore.access$500(SentryStore.java:101)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore$11.execute(SentryStore.java:601)
> at
> org.apache.sentry.provider.db.service.persistent.TransactionManager.executeTransaction(TransactionManager.java:159)
> at
> org.apache.sentry.provider.db.service.persistent.TransactionManager.access$100(TransactionManager.java:63)
> at
> org.apache.sentry.provider.db.service.persistent.TransactionManager$2.call(TransactionManager.java:213)
> --
> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:971)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3887)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3823)
> at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2435)
> at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2582)
> at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2530)
> at
> com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1907)
> at
> com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2141)
> at
> com.mysql.jdbc.PreparedStatement.executeBatchSerially(PreparedStatement.java:1773)
> ... 33 more
> 2017-07-13 13:18:14,450 ERROR
> org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor:
> Unknown error for request:
> TAlterSentryRoleGrantPrivilegeRequest(protocol_version:2,
> requestorUserName:hive, roleName:2017_07_12_15_06_38_1_2_805,
> privileges:[TSentryPrivilege(privilegeScope:DATABASE, serverName:server1,
> dbName:2017_07_12_15_06_38_1_2, tableName:, URI:, action:*,
> createTime:1499904401222, grantOption:FALSE, columnName:)]), message: The
> transaction has reached max retry number, Exception thrown when executing
> query
> java.lang.Exception: The transaction has reached max retry number, Exception
> thrown when executing query
> at
> org.apache.sentry.provider.db.service.persistent.TransactionManager$ExponentialBackoff.execute(TransactionManager.java:255)
> at
> org.apache.sentry.provider.db.service.persistent.TransactionManager.executeTransactionBlocksWithRetry(TransactionManager.java:209)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore.execute(SentryStore.java:3330)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore.alterSentryRoleGrantPrivilege(SentryStore.java:593)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore.alterSentryRoleGrantPrivileges(SentryStore.java:633)
> at
>
[jira] [Commented] (SENTRY-2183) Increase default sentry-hdfs rpc timeout to 20 mins
[ https://issues.apache.org/jira/browse/SENTRY-2183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16401143#comment-16401143 ] Alexander Kolbasov commented on SENTRY-2183: Why do you think that 20 mins is a good number? What if it takes 25 mins? > Increase default sentry-hdfs rpc timeout to 20 mins > --- > > Key: SENTRY-2183 > URL: https://issues.apache.org/jira/browse/SENTRY-2183 > Project: Sentry > Issue Type: Improvement > Components: Sentry >Affects Versions: 2.1.0 >Reporter: Arjun Mishra >Assignee: Arjun Mishra >Priority: Major > Attachments: SENTRY-2183.001.patch, SENTRY-2183.002.patch, > SENTRY-2183.01.patch > > > Full HMS Snapshots for large amounts of data (millions of databases, or > tables or partitions etc) take a lot more time than the current default of > 3.3mins. Also ideally since a full HMS snapshot should only occur once it is > best to increase the default to 20mins. This highly reduce the chance of > failure while sending a full snapshot which could be critical > Set sentry.hdfs.service.client.server.rpc-connection-timeout = 120 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (SENTRY-2179) Sentry should provide its own HMS listener
[ https://issues.apache.org/jira/browse/SENTRY-2179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Kolbasov reassigned SENTRY-2179: -- Assignee: (was: Alexander Kolbasov) > Sentry should provide its own HMS listener > -- > > Key: SENTRY-2179 > URL: https://issues.apache.org/jira/browse/SENTRY-2179 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.1.0 >Reporter: Alexander Kolbasov >Priority: Major > > Right now Sentry is using DbNotificationListener from Hive which listens to > lots of things that are not relevant for Sentry. Instead Sentry should > provide its own listener. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (SENTRY-2179) Sentry should provide its own HMS listener
[ https://issues.apache.org/jira/browse/SENTRY-2179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Kolbasov reassigned SENTRY-2179: -- Assignee: Alexander Kolbasov > Sentry should provide its own HMS listener > -- > > Key: SENTRY-2179 > URL: https://issues.apache.org/jira/browse/SENTRY-2179 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.1.0 >Reporter: Alexander Kolbasov >Assignee: Alexander Kolbasov >Priority: Major > > Right now Sentry is using DbNotificationListener from Hive which listens to > lots of things that are not relevant for Sentry. Instead Sentry should > provide its own listener. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (SENTRY-2179) Sentry should provide its own HMS listener
Alexander Kolbasov created SENTRY-2179: -- Summary: Sentry should provide its own HMS listener Key: SENTRY-2179 URL: https://issues.apache.org/jira/browse/SENTRY-2179 Project: Sentry Issue Type: Bug Components: Sentry Affects Versions: 2.1.0 Reporter: Alexander Kolbasov Right now Sentry is using DbNotificationListener from Hive which listens to lots of things that are not relevant for Sentry. Instead Sentry should provide its own listener. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (SENTRY-2149) Implement functionality to show groups
[
https://issues.apache.org/jira/browse/SENTRY-2149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16390575#comment-16390575
]
Alexander Kolbasov commented on SENTRY-2149:
We have two sentry CLI implementations - SentryShell and interactive Sentry
CLI. The latter can show all groups.
> Implement functionality to show groups
> --
>
> Key: SENTRY-2149
> URL: https://issues.apache.org/jira/browse/SENTRY-2149
> Project: Sentry
> Issue Type: New Feature
>Reporter: Sachin
>Priority: Major
>
> Sentry allows to list the roles
> SHOW ROLES;
> There should be also a way to show the groups . Currently it seems that this
> is only possible by directly querying the Sentry database. This functionality
> should be provided out-of-the-box similar to the statement above.
> The functionality could look similar to the following statement
> {code:sql}
> SHOW GROUPS;{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (SENTRY-2164) Convert uses of TransactionBlock to lambdas
[ https://issues.apache.org/jira/browse/SENTRY-2164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Kolbasov updated SENTRY-2164: --- Resolution: Fixed Fix Version/s: 2.1.0 Status: Resolved (was: Patch Available) > Convert uses of TransactionBlock to lambdas > --- > > Key: SENTRY-2164 > URL: https://issues.apache.org/jira/browse/SENTRY-2164 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.1.0 >Reporter: Alexander Kolbasov >Assignee: Alexander Kolbasov >Priority: Major > Fix For: 2.1.0 > > Attachments: SENTRY-2164.02.patch > > > Now that we agreed to use Java 8 features for Sentry it makes sense to clean > up SentryStore code and convert anonymous TransactionBlock instances to > lambdas. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (SENTRY-2164) Convert uses of TransactionBlock to lambdas
[ https://issues.apache.org/jira/browse/SENTRY-2164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Kolbasov updated SENTRY-2164: --- Attachment: (was: SENTRY-2164.01.patch) > Convert uses of TransactionBlock to lambdas > --- > > Key: SENTRY-2164 > URL: https://issues.apache.org/jira/browse/SENTRY-2164 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.1.0 >Reporter: Alexander Kolbasov >Assignee: Alexander Kolbasov >Priority: Major > Attachments: SENTRY-2164.02.patch > > > Now that we agreed to use Java 8 features for Sentry it makes sense to clean > up SentryStore code and convert anonymous TransactionBlock instances to > lambdas. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (SENTRY-2165) NotificationProcesser processAlterTable method has logs wrongly flagged as ERROR when they should be INFO
[ https://issues.apache.org/jira/browse/SENTRY-2165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16390529#comment-16390529 ] Alexander Kolbasov commented on SENTRY-2165: I would go even further and log these at DEBUG level. > NotificationProcesser processAlterTable method has logs wrongly flagged as > ERROR when they should be INFO > - > > Key: SENTRY-2165 > URL: https://issues.apache.org/jira/browse/SENTRY-2165 > Project: Sentry > Issue Type: Improvement > Components: Sentry >Reporter: Arjun Mishra >Assignee: Arjun Mishra >Priority: Major > Attachments: SENTRY-2165.01.patch > > > "Alter table event has incomplete information" and "Alter table notification > ignored as neither name nor location has changed" are logged as ERROR. This > can flood the logs and be a distraction to analyzing actual issues. These > should be set at INFO level -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (SENTRY-2164) Convert uses of TransactionBlock to lambdas
[ https://issues.apache.org/jira/browse/SENTRY-2164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Kolbasov updated SENTRY-2164: --- Attachment: SENTRY-2164.02.patch > Convert uses of TransactionBlock to lambdas > --- > > Key: SENTRY-2164 > URL: https://issues.apache.org/jira/browse/SENTRY-2164 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.1.0 >Reporter: Alexander Kolbasov >Assignee: Alexander Kolbasov >Priority: Major > Attachments: SENTRY-2164.01.patch, SENTRY-2164.02.patch > > > Now that we agreed to use Java 8 features for Sentry it makes sense to clean > up SentryStore code and convert anonymous TransactionBlock instances to > lambdas. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (SENTRY-2164) Convert uses of TransactionBlock to lambdas
[ https://issues.apache.org/jira/browse/SENTRY-2164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Kolbasov updated SENTRY-2164: --- Status: Patch Available (was: Open) > Convert uses of TransactionBlock to lambdas > --- > > Key: SENTRY-2164 > URL: https://issues.apache.org/jira/browse/SENTRY-2164 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.1.0 >Reporter: Alexander Kolbasov >Assignee: Alexander Kolbasov >Priority: Major > Attachments: SENTRY-2164.01.patch > > > Now that we agreed to use Java 8 features for Sentry it makes sense to clean > up SentryStore code and convert anonymous TransactionBlock instances to > lambdas. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (SENTRY-2164) Convert uses of TransactionBlock to lambdas
[ https://issues.apache.org/jira/browse/SENTRY-2164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Kolbasov updated SENTRY-2164: --- Attachment: SENTRY-2164.01.patch > Convert uses of TransactionBlock to lambdas > --- > > Key: SENTRY-2164 > URL: https://issues.apache.org/jira/browse/SENTRY-2164 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.1.0 >Reporter: Alexander Kolbasov >Assignee: Alexander Kolbasov >Priority: Major > Attachments: SENTRY-2164.01.patch > > > Now that we agreed to use Java 8 features for Sentry it makes sense to clean > up SentryStore code and convert anonymous TransactionBlock instances to > lambdas. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (SENTRY-2164) Convert uses of TransactionBlock to lambdas
Alexander Kolbasov created SENTRY-2164: -- Summary: Convert uses of TransactionBlock to lambdas Key: SENTRY-2164 URL: https://issues.apache.org/jira/browse/SENTRY-2164 Project: Sentry Issue Type: Bug Components: Sentry Affects Versions: 2.1.0 Reporter: Alexander Kolbasov Assignee: Alexander Kolbasov Now that we agreed to use Java 8 features for Sentry it makes sense to clean up SentryStore code and convert anonymous TransactionBlock instances to lambdas. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (SENTRY-2140) Attribute based access control
[ https://issues.apache.org/jira/browse/SENTRY-2140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16387327#comment-16387327 ] Alexander Kolbasov commented on SENTRY-2140: [~moist], thank you for the proposal. The example part is great, it helps to understand where you are coming from. It would be good to add a bit more technical substance there. In particular: 1) More formal definition of tags and their interaction with Hive privilege model 2) Some discussion of how it all applies (or doesn't apply) to generic privilege model 3) Proposed changes to Sentry thrift API (after all, CLI examples that you mention just speak Sentrish). 4) Proposed changes to the Hive privilege model > Attribute based access control > -- > > Key: SENTRY-2140 > URL: https://issues.apache.org/jira/browse/SENTRY-2140 > Project: Sentry > Issue Type: New Feature > Components: Core >Reporter: Steve Moist >Priority: Major > Attachments: Sentry ABAC Proposal.pdf > > > As a user, I want to have finer grain control over which users/roles can view > data in Hive. Some information such as Social Security Number is considered > very confidential information. I want to be able to tag columns in Hive with > "attributes" that prevent users/roles from not accessing or seeing the data. > For users/roles that have that attribute, they should be able to see that > information. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (SENTRY-1242) Enable getting all privileges on a hive object
[ https://issues.apache.org/jira/browse/SENTRY-1242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16386832#comment-16386832 ] Alexander Kolbasov commented on SENTRY-1242: [~moist] Can you post some example showing the command and the output? > Enable getting all privileges on a hive object > -- > > Key: SENTRY-1242 > URL: https://issues.apache.org/jira/browse/SENTRY-1242 > Project: Sentry > Issue Type: New Feature >Affects Versions: 2.0.0 >Reporter: Sravya Tirukkovalur >Assignee: Steve Moist >Priority: Major > Fix For: 2.1.0 > > Attachments: SENTRY-1242-001.patch, SENTRY-1242-002.diff > > > Enable show grant on table/db . This syntax is already supported by > hive. > This would be really useful for the admin to find out all policies on a hive > object. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (SENTRY-2149) Implement functionality to show groups
[
https://issues.apache.org/jira/browse/SENTRY-2149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16386415#comment-16386415
]
Alexander Kolbasov commented on SENTRY-2149:
The submitter can clarify this, but looks like the request is to be able to see
all groups known to Sentry. Note that you can do this with Sentry CLI.
> Implement functionality to show groups
> --
>
> Key: SENTRY-2149
> URL: https://issues.apache.org/jira/browse/SENTRY-2149
> Project: Sentry
> Issue Type: New Feature
>Reporter: Sachin
>Priority: Major
>
> Sentry allows to list the roles
> SHOW ROLES;
> There should be also a way to show the groups . Currently it seems that this
> is only possible by directly querying the Sentry database. This functionality
> should be provided out-of-the-box similar to the statement above.
> The functionality could look similar to the following statement
> {code:sql}
> SHOW GROUPS;{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (SENTRY-2144) Table Rename should allow database name change
[
https://issues.apache.org/jira/browse/SENTRY-2144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16374860#comment-16374860
]
Alexander Kolbasov commented on SENTRY-2144:
You also need to check the code in {{FullUpdateModifier}}.
There is a unit test for it - {{TestFullUpdateModifier}} - it should be easy to
add a test and see whether it works correctly or not.
> Table Rename should allow database name change
> --
>
> Key: SENTRY-2144
> URL: https://issues.apache.org/jira/browse/SENTRY-2144
> Project: Sentry
> Issue Type: Bug
>Reporter: Na Li
>Assignee: Na Li
>Priority: Major
> Attachments: SENTRY-2144.001.patch
>
>
> Right now, when renaming a table, if the database changes, the privileges
> will be moved to the wrong table.
> For example, original table is db1.tbl1, and user changes to db2.tbl2.
> *Expected behavior*:
> The privileges associated with db1.tbl1 should be removed from db1.tbl1, and
> added to *{color:#205081}db2{color}*.tbl2.
> *Actual behavior*:
> The privileges associated with db1.tbl1 is removed from db1.tbl1, and added
> to {color:red}*db1*{color}.tbl2.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (SENTRY-2143) Table renames should synchronize with Sentry
[
https://issues.apache.org/jira/browse/SENTRY-2143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16374144#comment-16374144
]
Alexander Kolbasov commented on SENTRY-2143:
[~LinaAtAustin] Most likely the test doesn't set up DbNotificationListener and
this causes the effect that you see in debugger.
> Table renames should synchronize with Sentry
>
>
> Key: SENTRY-2143
> URL: https://issues.apache.org/jira/browse/SENTRY-2143
> Project: Sentry
> Issue Type: Bug
> Components: Sentry
>Affects Versions: 2.1.0
>Reporter: Alexander Kolbasov
>Assignee: Na Li
>Priority: Major
> Attachments: SENTRY-2143.001.patch, SENTRY-2143.002.patch,
> SENTRY-2143.003.patch, SENTRY-2143.004.patch
>
>
> Currently table renames are not synchronized from Hive (while table
> creates/drops are). This creates a problem since the renamed table doesn't
> have correct privileges for a bit until it is processed by Sentry. So
> perfectly valid scripts that rename tables and expect the rename table to
> retain the privileges are going to fail.
> The fix is to update {{SentrySyncHMSNotificationsPostEventListener}} to
> synchronize table renames as well.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (SENTRY-2144) Table Rename should allow database name change
[
https://issues.apache.org/jira/browse/SENTRY-2144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16374030#comment-16374030
]
Alexander Kolbasov commented on SENTRY-2144:
Note that if this is, indeed, an issue you also need to verify that it is
handled during taking full snapshot from HMS when there is reconciliation code
for new events that appeared during snapshot creation.
> Table Rename should allow database name change
> --
>
> Key: SENTRY-2144
> URL: https://issues.apache.org/jira/browse/SENTRY-2144
> Project: Sentry
> Issue Type: Bug
>Reporter: Na Li
>Assignee: Na Li
>Priority: Major
> Attachments: SENTRY-2144.001.patch
>
>
> Right now, when renaming a table, if the database changes, the privileges
> will be moved to the wrong table.
> For example, original table is db1.tbl1, and user changes to db2.tbl2.
> *Expected behavior*:
> The privileges associated with db1.tbl1 should be removed from db1.tbl1, and
> added to *{color:#205081}db2{color}*.tbl2.
> *Actual behavior*:
> The privileges associated with db1.tbl1 is removed from db1.tbl1, and added
> to {color:red}*db1*{color}.tbl2.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (SENTRY-2144) Table Rename should allow database name change
[
https://issues.apache.org/jira/browse/SENTRY-2144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16374027#comment-16374027
]
Alexander Kolbasov commented on SENTRY-2144:
* How would it be possible to have ALTER TABLE event that changes both database
name and table name?
* Can you show some example?
* Did you see this actually happening?
* Do you have a test case that demonstrates the problem?
> Table Rename should allow database name change
> --
>
> Key: SENTRY-2144
> URL: https://issues.apache.org/jira/browse/SENTRY-2144
> Project: Sentry
> Issue Type: Bug
>Reporter: Na Li
>Assignee: Na Li
>Priority: Major
> Attachments: SENTRY-2144.001.patch
>
>
> Right now, when renaming a table, if the database changes, the privileges
> will be moved to the wrong table.
> For example, original table is db1.tbl1, and user changes to db2.tbl2.
> *Expected behavior*:
> The privileges associated with db1.tbl1 should be removed from db1.tbl1, and
> added to *{color:#205081}db2{color}*.tbl2.
> *Actual behavior*:
> The privileges associated with db1.tbl1 is removed from db1.tbl1, and added
> to {color:red}*db1*{color}.tbl2.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (SENTRY-2143) Table renames should synchronize with Sentry
[
https://issues.apache.org/jira/browse/SENTRY-2143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16374024#comment-16374024
]
Alexander Kolbasov commented on SENTRY-2143:
[~LinaAtAustin] Note that event is stored at the same time its event ID is
updated, so if there is no event ID it means that event wasn't stored in the
HMS DB in the first place.
> Table renames should synchronize with Sentry
>
>
> Key: SENTRY-2143
> URL: https://issues.apache.org/jira/browse/SENTRY-2143
> Project: Sentry
> Issue Type: Bug
> Components: Sentry
>Affects Versions: 2.1.0
>Reporter: Alexander Kolbasov
>Assignee: Na Li
>Priority: Major
> Attachments: SENTRY-2143.001.patch, SENTRY-2143.002.patch,
> SENTRY-2143.003.patch, SENTRY-2143.004.patch
>
>
> Currently table renames are not synchronized from Hive (while table
> creates/drops are). This creates a problem since the renamed table doesn't
> have correct privileges for a bit until it is processed by Sentry. So
> perfectly valid scripts that rename tables and expect the rename table to
> retain the privileges are going to fail.
> The fix is to update {{SentrySyncHMSNotificationsPostEventListener}} to
> synchronize table renames as well.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (SENTRY-2143) Table renames should synchronize with Sentry
[
https://issues.apache.org/jira/browse/SENTRY-2143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16374022#comment-16374022
]
Alexander Kolbasov commented on SENTRY-2143:
[~LinaAtAustin] Note that if event doesn't have event ID it will never reach
Sentry in the first place, so this means that if you are right, none of the
ALTER TABLE events ever reach Sentry. This definitely isn't the case since I
saw the ALTER TABLE event in the log. So it is worth investigating why
notification ID is not set in your case - it is definitely not the normal
situation.
> Table renames should synchronize with Sentry
>
>
> Key: SENTRY-2143
> URL: https://issues.apache.org/jira/browse/SENTRY-2143
> Project: Sentry
> Issue Type: Bug
> Components: Sentry
>Affects Versions: 2.1.0
>Reporter: Alexander Kolbasov
>Assignee: Na Li
>Priority: Major
> Attachments: SENTRY-2143.001.patch, SENTRY-2143.002.patch,
> SENTRY-2143.003.patch, SENTRY-2143.004.patch
>
>
> Currently table renames are not synchronized from Hive (while table
> creates/drops are). This creates a problem since the renamed table doesn't
> have correct privileges for a bit until it is processed by Sentry. So
> perfectly valid scripts that rename tables and expect the rename table to
> retain the privileges are going to fail.
> The fix is to update {{SentrySyncHMSNotificationsPostEventListener}} to
> synchronize table renames as well.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (SENTRY-2143) Table renames should synchronize with Sentry
[
https://issues.apache.org/jira/browse/SENTRY-2143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16373977#comment-16373977
]
Alexander Kolbasov edited comment on SENTRY-2143 at 2/23/18 6:24 AM:
-
[~LinaAtAustin] I'm a bit confused here. Please clarify a few things.
{quote}
1) SentrySyncHMSNotificationsPostEventListener.syncNotificationEvents does not
do anything for alter table event because HiveMetaStore does not set the
notification ID in the event's parameter
{quote}
The code that you quote was done by [~spena] - Sergio - what do you think about
this?
{quote}
2) Notification ID is set for create database, create table etc, but not set
for alter table event.
{quote}
Why do you think that it doesn't?
In the snippet above isn't {{MetaStoreListenerNotifier.notifyEvent()}} doing
exactly that - creating an event and storing event ID there?
was (Author: akolb):
[~LinaAtAustin] I'm a bit confused here. Please clarify a few things.
{quote}
1) SentrySyncHMSNotificationsPostEventListener.syncNotificationEvents does not
do anything for alter table event because HiveMetaStore does not set the
notification ID in the event's parameter
{quote}
The code that you quote was done by [~spena] - Sergio - what do you think about
this?
{quote}
2) Notification ID is set for create database, create table etc, but not set
for alter table event.
{quote}
Why do you think that it doesn't?
In the snippet above isn't {{MetaStoreListenerNotifier.notifyEvent()}}
> Table renames should synchronize with Sentry
>
>
> Key: SENTRY-2143
> URL: https://issues.apache.org/jira/browse/SENTRY-2143
> Project: Sentry
> Issue Type: Bug
> Components: Sentry
>Affects Versions: 2.1.0
>Reporter: Alexander Kolbasov
>Assignee: Na Li
>Priority: Major
> Attachments: SENTRY-2143.001.patch, SENTRY-2143.002.patch,
> SENTRY-2143.003.patch, SENTRY-2143.004.patch
>
>
> Currently table renames are not synchronized from Hive (while table
> creates/drops are). This creates a problem since the renamed table doesn't
> have correct privileges for a bit until it is processed by Sentry. So
> perfectly valid scripts that rename tables and expect the rename table to
> retain the privileges are going to fail.
> The fix is to update {{SentrySyncHMSNotificationsPostEventListener}} to
> synchronize table renames as well.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (SENTRY-2143) Table renames should synchronize with Sentry
[
https://issues.apache.org/jira/browse/SENTRY-2143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16373977#comment-16373977
]
Alexander Kolbasov commented on SENTRY-2143:
[~LinaAtAustin] I'm a bit confused here. Please clarify a few things.
{quote}
1) SentrySyncHMSNotificationsPostEventListener.syncNotificationEvents does not
do anything for alter table event because HiveMetaStore does not set the
notification ID in the event's parameter
{quote}
The code that you quote was done by [~spena] - Sergio - what do you think about
this?
{quote}
2) Notification ID is set for create database, create table etc, but not set
for alter table event.
{quote}
Why do you think that it doesn't?
In the snippet above isn't {{MetaStoreListenerNotifier.notifyEvent()}}
> Table renames should synchronize with Sentry
>
>
> Key: SENTRY-2143
> URL: https://issues.apache.org/jira/browse/SENTRY-2143
> Project: Sentry
> Issue Type: Bug
> Components: Sentry
>Affects Versions: 2.1.0
>Reporter: Alexander Kolbasov
>Assignee: Na Li
>Priority: Major
> Attachments: SENTRY-2143.001.patch, SENTRY-2143.002.patch,
> SENTRY-2143.003.patch, SENTRY-2143.004.patch
>
>
> Currently table renames are not synchronized from Hive (while table
> creates/drops are). This creates a problem since the renamed table doesn't
> have correct privileges for a bit until it is processed by Sentry. So
> perfectly valid scripts that rename tables and expect the rename table to
> retain the privileges are going to fail.
> The fix is to update {{SentrySyncHMSNotificationsPostEventListener}} to
> synchronize table renames as well.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (SENTRY-2143) Table renames should synchronize with Sentry
Alexander Kolbasov created SENTRY-2143:
--
Summary: Table renames should synchronize with Sentry
Key: SENTRY-2143
URL: https://issues.apache.org/jira/browse/SENTRY-2143
Project: Sentry
Issue Type: Bug
Components: Sentry
Affects Versions: 2.1.0
Reporter: Alexander Kolbasov
Currently table renames are not synchronized from Hive (while table
creates/drops are). This creates a problem since the renamed table doesn't have
correct privileges for a bit until it is processed by Sentry. So perfectly
valid scripts that rename tables and expect the rename table to retain the
privileges are going to fail.
The fix is to update {{SentrySyncHMSNotificationsPostEventListener}} to
synchronize table renames as well.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (SENTRY-2142) NotificationProcessor should ignore index events
[
https://issues.apache.org/jira/browse/SENTRY-2142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexander Kolbasov updated SENTRY-2142:
---
Labels: bite-sized newbie (was: )
> NotificationProcessor should ignore index events
>
>
> Key: SENTRY-2142
> URL: https://issues.apache.org/jira/browse/SENTRY-2142
> Project: Sentry
> Issue Type: Bug
> Components: Sentry
>Affects Versions: 2.0.0
>Reporter: Alexander Kolbasov
>Priority: Major
> Labels: bite-sized, newbie
>
> Currently NotificationProcessor that handles events from HMS complains about
> index-related events ({{CREATE_INDEX}}, {{ALTER_INDEX}}, {{DROP_INDEX}}):
> {code:java}
> 2018-01-28 03:25:30,047 ERROR
> org.apache.sentry.service.thrift.NotificationProcessor: Notification with
> ID:1398 has invalid event type: CREATE_INDEX{code}
> Instead it should just ignore these events.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (SENTRY-2142) NotificationProcessor should ignore index events
Alexander Kolbasov created SENTRY-2142:
--
Summary: NotificationProcessor should ignore index events
Key: SENTRY-2142
URL: https://issues.apache.org/jira/browse/SENTRY-2142
Project: Sentry
Issue Type: Bug
Components: Sentry
Affects Versions: 2.0.0
Reporter: Alexander Kolbasov
Currently NotificationProcessor that handles events from HMS complains about
index-related events ({{CREATE_INDEX}}, {{ALTER_INDEX}}, {{DROP_INDEX}}):
{code:java}
2018-01-28 03:25:30,047 ERROR
org.apache.sentry.service.thrift.NotificationProcessor: Notification with
ID:1398 has invalid event type: CREATE_INDEX{code}
Instead it should just ignore these events.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (SENTRY-2134) Apply Hive URI grants recursively to subdirectories
[ https://issues.apache.org/jira/browse/SENTRY-2134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363434#comment-16363434 ] Alexander Kolbasov commented on SENTRY-2134: I think that supporting URI grants through Sentry may be a reasonable thing to do as long as it is clear hw they interplay with regular grants - what happens if there is table grant and URI grant? Or there is URI grant on a directory and column-level privilege? Would someone care to draft a proposal of the suggested behavior of URI grants? > Apply Hive URI grants recursively to subdirectories > --- > > Key: SENTRY-2134 > URL: https://issues.apache.org/jira/browse/SENTRY-2134 > Project: Sentry > Issue Type: Improvement > Components: Hive Binding >Affects Versions: 1.8.0, 2.0.0, 1.7.1 >Reporter: Ruslan Dautkhanov >Priority: Major > Labels: hive, uri > > Currently we need to add direct grants for all Hive tables' LOCATIONs. > Like, 'hdfs_staging/table1', 'hdfs_staging/table2', etc.. > It's not manageable this way. - we can't add grants for each and every table. > It would be great if we could just do one grant - > 'hdfs_staging/' so it would automatically be applied to > 'hdfs_staging/table1', 'hdfs_staging/table2', and other subdirectories. > There is probably a reason this wasn't implemented earlier? Thanks for > considering this improvement. > Also found another user's request on this - > https://community.cloudera.com/t5/Interactive-Short-cycle-SQL/Impala-Sentry-GRANT-ALL-ON-URI-not-cascaded-down-through/td-p/39928 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (SENTRY-2137) Improve and rework the CLI
[ https://issues.apache.org/jira/browse/SENTRY-2137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363431#comment-16363431 ] Alexander Kolbasov commented on SENTRY-2137: Note that there is also an interactive session-based Sentry CLI. > Improve and rework the CLI > -- > > Key: SENTRY-2137 > URL: https://issues.apache.org/jira/browse/SENTRY-2137 > Project: Sentry > Issue Type: New Feature >Affects Versions: 2.0.0 >Reporter: Steve Moist >Priority: Minor > > Sentry can be improved by moving all of the privilige actions for hive (such > as grant/revoke) from beeline and into a centralized CLI. With this we can > do operations such as show all privileges for a role across HDFS, Hive, > Impala, etc in a single location and administer this in a single location. > In a cluster, it would be good to have the sentry cli as a standalone > executable, so building in a REST API for Sentry use would be needed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (SENTRY-2140) Tag based access control
[ https://issues.apache.org/jira/browse/SENTRY-2140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363430#comment-16363430 ] Alexander Kolbasov commented on SENTRY-2140: It would be nice to see a more detailed description of the proposal once you have it worked out. > Tag based access control > > > Key: SENTRY-2140 > URL: https://issues.apache.org/jira/browse/SENTRY-2140 > Project: Sentry > Issue Type: New Feature > Components: Core >Reporter: Steve Moist >Priority: Major > > As a user, I want to have finer grain control over which users/roles can view > data in Hive. Some information such as Social Security Number is considered > very confidential information. I want to be able to tag columns in Hive with > "tags" that prevent users/roles from not accessing or seeing the data. For > users/roles that have that tag, they should be able to see that information. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (SENTRY-2109) Fix the logic of identifying HMS out of Sync and handle gaps and out-of-sequence notifications.
[ https://issues.apache.org/jira/browse/SENTRY-2109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16350906#comment-16350906 ] Alexander Kolbasov commented on SENTRY-2109: [~kkalyan] I would appreciate if you revert the patch and we complete the discussion review and then you commit (or not commit) whatever comes out of it. This patch had many iterations. There were several reviewers who had comments about earlier versions of the patch and some reviewers posted unanswered comments in this Jira so it is pretty clear that the review wasn't complete even though you got one +1 from a committer. I think that the best course of action is to revert the patch and complete the review. > Fix the logic of identifying HMS out of Sync and handle gaps and > out-of-sequence notifications. > --- > > Key: SENTRY-2109 > URL: https://issues.apache.org/jira/browse/SENTRY-2109 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.1.0 >Reporter: kalyan kumar kalvagadda >Assignee: kalyan kumar kalvagadda >Priority: Major > Fix For: 2.1.0 > > Attachments: SENTRY-2109.001.patch, SENTRY-2109.002.patch, > SENTRY-2109.003.patch, SENTRY-2109.004.patch, SENTRY-2109.005.patch, > SENTRY-2109.006.patch, SENTRY-2109.007.patch, SENTRY-2109.008.patch, > SENTRY-2109.009.patch, SENTRY-2109.010.patch, SENTRY-2109.010.patch, > SENTRY-2109.011.patch, SENTRY-2109.012.patch, SENTRY-2109.012.patch, > SENTRY-2109.012.patch, Screenshot_HMS_NOTIFICATION_LOG.png > > > Currently HMSFollower proactively checks if sentry is out of sync with HMS > and initiates full snapshot, if needed. > There will be false positives with the current logic if there are gaps in the > event-id in the notification log sequence. > This jira is aimed at making that logic robust. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (SENTRY-2140) Tag based access control
[ https://issues.apache.org/jira/browse/SENTRY-2140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16350882#comment-16350882 ] Alexander Kolbasov commented on SENTRY-2140: Is it a Sentry request or Hive request or both? Do you propose to tag columns using some kind of new Hive syntax or something else? What are these tags and how do they relate to privileges? Does this require some kind of user-level privileges? What objects do you want to tag - just columns or something else? > Tag based access control > > > Key: SENTRY-2140 > URL: https://issues.apache.org/jira/browse/SENTRY-2140 > Project: Sentry > Issue Type: New Feature > Components: Core >Reporter: Steve Moist >Priority: Major > > As a user, I want to have finer grain control over which users/roles can view > data in Hive. Some information such as Social Security Number is considered > very confidential information. I want to be able to tag columns in Hive with > "tags" that prevent users/roles from not accessing or seeing the data. For > users/roles that have that tag, they should be able to see that information. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (SENTRY-2138) Sentry REST API
[ https://issues.apache.org/jira/browse/SENTRY-2138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16350878#comment-16350878 ] Alexander Kolbasov commented on SENTRY-2138: CLI and restful API are a bit orthogonal (well, you need *some* API to build CLI). But having restful API isn't an obvious thing - what would it do better then existing API? Do we have a real need to improve/change API? If you think that we do, we should start discussing this. > Sentry REST API > --- > > Key: SENTRY-2138 > URL: https://issues.apache.org/jira/browse/SENTRY-2138 > Project: Sentry > Issue Type: New Feature >Affects Versions: 2.0.0 >Reporter: Steve Moist >Priority: Major > > In an effort to decentralize the Sentry CLI, we need to build out a restful > API to administer Sentry privileges for all the components. > This would also have the benefit of allowing customers to manage and > integrate Sentry permissions in their own applications. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (SENTRY-2137) Improve and rework the CLI
[ https://issues.apache.org/jira/browse/SENTRY-2137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16350869#comment-16350869 ] Alexander Kolbasov commented on SENTRY-2137: I agree that having a nice usable CLI is nice (we have two now to play with) but it doesn't mean that we should move things *out* of beeline. Having REST API is quite possible as well but this is a different ask from CLI, so let's not mix it in one bucket. > Improve and rework the CLI > -- > > Key: SENTRY-2137 > URL: https://issues.apache.org/jira/browse/SENTRY-2137 > Project: Sentry > Issue Type: New Feature >Affects Versions: 2.0.0 >Reporter: Steve Moist >Priority: Minor > > Sentry can be improved by moving all of the privilige actions for hive (such > as grant/revoke) from beeline and into a centralized CLI. With this we can > do operations such as show all privileges for a role across HDFS, Hive, > Impala, etc in a single location and administer this in a single location. > In a cluster, it would be good to have the sentry cli as a standalone > executable, so building in a REST API for Sentry use would be needed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (SENTRY-2109) Fix the logic of identifying HMS out of Sync and handle gaps and out-of-sequence notifications.
[ https://issues.apache.org/jira/browse/SENTRY-2109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16349562#comment-16349562 ] Alexander Kolbasov commented on SENTRY-2109: I asked the question multiple times in this JIRA and didn't get an answer - what problem does this fix address once the issue is fixed on HMS side? That's the basic question to answer before we dive into the implementation details. On Thu, Feb 1, 2018 at 4:09 PM, kalyan kumar kalvagadda (JIRA) < > Fix the logic of identifying HMS out of Sync and handle gaps and > out-of-sequence notifications. > --- > > Key: SENTRY-2109 > URL: https://issues.apache.org/jira/browse/SENTRY-2109 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.1.0 >Reporter: kalyan kumar kalvagadda >Assignee: kalyan kumar kalvagadda >Priority: Major > Fix For: 2.1.0 > > Attachments: SENTRY-2109.001.patch, SENTRY-2109.002.patch, > SENTRY-2109.003.patch, SENTRY-2109.004.patch, SENTRY-2109.005.patch, > SENTRY-2109.006.patch, SENTRY-2109.007.patch, SENTRY-2109.008.patch, > SENTRY-2109.009.patch, SENTRY-2109.010.patch, SENTRY-2109.010.patch, > SENTRY-2109.011.patch, SENTRY-2109.012.patch, SENTRY-2109.012.patch, > SENTRY-2109.012.patch, Screenshot_HMS_NOTIFICATION_LOG.png > > > Currently HMSFollower proactively checks if sentry is out of sync with HMS > and initiates full snapshot, if needed. > There will be false positives with the current logic if there are gaps in the > event-id in the notification log sequence. > This jira is aimed at making that logic robust. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (SENTRY-2109) Fix the logic of identifying HMS out of Sync and handle gaps and out-of-sequence notifications.
[ https://issues.apache.org/jira/browse/SENTRY-2109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16349520#comment-16349520 ] Alexander Kolbasov commented on SENTRY-2109: [~kkalyan] This is complicated patch that took 12 iterations to do - most of the patches didn't compile so it wasn't clear that there is a final patch available for review. Many previous patches didn't compile so I wasn't aware that you are ready for code review - and now it is committed. > Fix the logic of identifying HMS out of Sync and handle gaps and > out-of-sequence notifications. > --- > > Key: SENTRY-2109 > URL: https://issues.apache.org/jira/browse/SENTRY-2109 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.1.0 >Reporter: kalyan kumar kalvagadda >Assignee: kalyan kumar kalvagadda >Priority: Major > Fix For: 2.1.0 > > Attachments: SENTRY-2109.001.patch, SENTRY-2109.002.patch, > SENTRY-2109.003.patch, SENTRY-2109.004.patch, SENTRY-2109.005.patch, > SENTRY-2109.006.patch, SENTRY-2109.007.patch, SENTRY-2109.008.patch, > SENTRY-2109.009.patch, SENTRY-2109.010.patch, SENTRY-2109.010.patch, > SENTRY-2109.011.patch, SENTRY-2109.012.patch, SENTRY-2109.012.patch, > SENTRY-2109.012.patch, Screenshot_HMS_NOTIFICATION_LOG.png > > > Currently HMSFollower proactively checks if sentry is out of sync with HMS > and initiates full snapshot, if needed. > There will be false positives with the current logic if there are gaps in the > event-id in the notification log sequence. > This jira is aimed at making that logic robust. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (SENTRY-2109) Fix the logic of identifying HMS out of Sync and handle gaps and out-of-sequence notifications.
[ https://issues.apache.org/jira/browse/SENTRY-2109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16349502#comment-16349502 ] Alexander Kolbasov commented on SENTRY-2109: Why is it resolved so quickly? I was reviewing this as well and had some earlier comments, you didn't give me a chance to complete review. > Fix the logic of identifying HMS out of Sync and handle gaps and > out-of-sequence notifications. > --- > > Key: SENTRY-2109 > URL: https://issues.apache.org/jira/browse/SENTRY-2109 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.1.0 >Reporter: kalyan kumar kalvagadda >Assignee: kalyan kumar kalvagadda >Priority: Major > Fix For: 2.1.0 > > Attachments: SENTRY-2109.001.patch, SENTRY-2109.002.patch, > SENTRY-2109.003.patch, SENTRY-2109.004.patch, SENTRY-2109.005.patch, > SENTRY-2109.006.patch, SENTRY-2109.007.patch, SENTRY-2109.008.patch, > SENTRY-2109.009.patch, SENTRY-2109.010.patch, SENTRY-2109.010.patch, > SENTRY-2109.011.patch, SENTRY-2109.012.patch, SENTRY-2109.012.patch, > SENTRY-2109.012.patch, Screenshot_HMS_NOTIFICATION_LOG.png > > > Currently HMSFollower proactively checks if sentry is out of sync with HMS > and initiates full snapshot, if needed. > There will be false positives with the current logic if there are gaps in the > event-id in the notification log sequence. > This jira is aimed at making that logic robust. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (SENTRY-2098) Get user to role mapping when retrieving full snapshot from DB
[ https://issues.apache.org/jira/browse/SENTRY-2098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16344086#comment-16344086 ] Alexander Kolbasov commented on SENTRY-2098: [~LinaAtAustin] Can you clarify here what do you mean by "full snapshot update"? What is exactly the ask for this JIRA? > Get user to role mapping when retrieving full snapshot from DB > -- > > Key: SENTRY-2098 > URL: https://issues.apache.org/jira/browse/SENTRY-2098 > Project: Sentry > Issue Type: Sub-task > Components: Sentry >Reporter: Na Li >Priority: Major > > The full snapshot update only contains authorization to roles mapping and > role to group mapping. Need to add role to user mapping in > SentryStore.retrieveFullRoleImageCore -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (SENTRY-1068) Allow user who created a table to have "with grant" over that table by default
[ https://issues.apache.org/jira/browse/SENTRY-1068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16339665#comment-16339665 ] Alexander Kolbasov commented on SENTRY-1068: [~dapengsun] Are you still working on this? > Allow user who created a table to have "with grant" over that table by default > -- > > Key: SENTRY-1068 > URL: https://issues.apache.org/jira/browse/SENTRY-1068 > Project: Sentry > Issue Type: Improvement >Reporter: Dapeng Sun >Assignee: Dapeng Sun >Priority: Major > Labels: hive, roadmap > Fix For: 2.1.0 > > Attachments: SENTRY-1068.001.patch, SENTRY-1068.002.patch > > > The database(table) owner should have "with grant" over that table, so that > the owner can grant the privilege to others -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (SENTRY-1067) Exclude capability for privilege("DENY" privilege support)
[
https://issues.apache.org/jira/browse/SENTRY-1067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16339662#comment-16339662
]
Alexander Kolbasov commented on SENTRY-1067:
[~dapengsun] Are you still working on this? It is marked as "in-progress"
> Exclude capability for privilege("DENY" privilege support)
> --
>
> Key: SENTRY-1067
> URL: https://issues.apache.org/jira/browse/SENTRY-1067
> Project: Sentry
> Issue Type: New Feature
>Reporter: Dapeng Sun
>Assignee: Dapeng Sun
>Priority: Major
> Labels: roadmap
> Attachments: Design Document of Sentry Exclude capability for
> privilege-20160302.pdf
>
>
> Currently Sentry can only grant privileges to object, in some cases, only
> some sensitive data need to be protected. Adding exclude capability can
> simplify the management of access control.
> For example, the table "employee" have many columns, the column likes
> "username", "contact" and other information can be queried by others,but the
> column "salary" can only be queried by specific user.
> With exclude capability, we can grant privilege of table "employee" to user
> and block the column "salary".
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (SENTRY-1034) Fix beeline connecting to db
[ https://issues.apache.org/jira/browse/SENTRY-1034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Kolbasov updated SENTRY-1034: --- Description: A possible info leak in the way how beeline connects to databases and uses the ACLs to prevent seeing unauthorised databases and tables. It turns out that one can connect to a database that one should not see, but listing it afterwards gives no tables. This is still somewhat a security breach as an attacker can gain insight what databases exist. The way the problem got identified: root@prod-vm-cdh-mgr-01 ~]# kinit -kt ~test_mval.keytab test_mval [root@prod-vm-cdh-mgr-01 ~]# beeline Beeline version 1.1.0-cdh5.4.8 by Apache Hive beeline> !connect jdbc:hive2://host:1/testdb;principal=hive/_HOST@ABCD scan complete in 6ms Connecting to jdbc:hive2://host:1/testdb;principal=hive/_HOST@ABCD Enter username for jdbc:hive2://host:1/testdb;principal=hive/_HOST@ABCD: Enter password for jdbc:hive2://host:1/testdb;principal=hive/_HOST@ABCD: Connected to: Apache Hive (version 1.1.0-cdh5.4.8) Driver: Hive JDBC (version 1.1.0-cdh5.4.8) Transaction isolation: TRANSACTION_REPEATABLE_READ 0: jdbc:hive2://host:1/testdb> show databases; -+ database_name -+ test_mvaldb default -+ 2 rows selected (0.726 seconds) 0: jdbc:hive2://host:1/testdb> show tables; ---+ tab_name ---+ ---+ No rows selected (1.033 seconds) 0: jdbc:hive2://host:1/testdb> !quit Closing: 0: jdbc:hive2://host:1/testdb;principal=hive/_HOST@ABCD [root@foo ~]# beeline Beeline version 1.1.0-cdh5.4.8 by Apache Hive beeline> !connect jdbc:hive2://host:1/asdasdasdasd;principal=hive/_HOST@ABCD scan complete in 2ms Connecting to jdbc:hive2://host:1/asdasdasdasd;principal=hive/_HOST@ABCD Enter username for jdbc:hive2://host:1/asdasdasdasd;principal=hive/_HOST@ABCD: Enter password for jdbc:hive2://host:1/asdasdasdasd;principal=hive/_HOST@ABCD: Connected to: Apache Hive (version 1.1.0-cdh5.4.8) Driver: Hive JDBC (version 1.1.0-cdh5.4.8) Transaction isolation: TRANSACTION_REPEATABLE_READ 0: jdbc:hive2://host:1/asdas> show tables; Error: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. Database does not exist: asdasdasdasd (state=08S01,code=1) 0: jdbc:hive2://host:1/asdas> !connect jdbc:hive2://host:1/testdb;principal=hive/_HOST@ABCD Connecting to jdbc:hive2://vm-cdh-01:1/testdb;principal=hive/_HOST@ABCD Enter username for jdbc:hive2://host:1/testdb;principal=hive/_HOST@ABCD: Enter password for jdbc:hive2://host:1/testdb;principal=hive/_HOST@ABCD: Connected to: Apache Hive (version 1.1.0-cdh5.4.8) Driver: Hive JDBC (version 1.1.0-cdh5.4.8) Transaction isolation: TRANSACTION_REPEATABLE_READ 1: jdbc:hive2://host:1/testdb> show tables; ---+ tab_name ---+ ---+ No rows selected (1.09 seconds) 1: jdbc:hive2://host:1/testdb> 1: jdbc:hive2://host:1/testdb> 1: jdbc:hive2://host:1/testdb> 1: jdbc:hive2://host:1/testdb> 1: jdbc:hive2://host:1/testdb> 1: jdbc:hive2://host:1/testdb> !quit; Unknown command: quit; 1: jdbc:hive2://host:1/testdb> !quit; Unknown command: quit; 1: jdbc:hive2://host:1/testdb> !quit was: A possible info leak in the way how beeline connects to databases and uses the ACLs to prevent seeing unauthorised databases and tables. It turns out that one can connect to a database that one should not see, but listing it afterwards gives no tables. This is still somewhat a security breach as an attacker can gain insight what databases exist. The way the problem got identified: root@prod-vm-cdh-mgr-01 ~]# kinit -kt ~test_mval.keytab test_mval [root@prod-vm-cdh-mgr-01 ~]# beeline Beeline version 1.1.0-cdh5.4.8 by Apache Hive beeline> !connect jdbc:hive2://vm-cdh-01:1/testdb;principal=hive/_HOST@MITKDC scan complete in 6ms Connecting to jdbc:hive2://vm-cdh-01:1/testdb;principal=hive/_HOST@MITKDC Enter username for jdbc:hive2://vm-cdh-01:1/testdb;principal=hive/_HOST@MITKDC: Enter password for jdbc:hive2://vm-cdh-01:1/testdb;principal=hive/_HOST@MITKDC: Connected to: Apache Hive (version 1.1.0-cdh5.4.8) Driver: Hive JDBC (version 1.1.0-cdh5.4.8) Transaction isolation: TRANSACTION_REPEATABLE_READ 0: jdbc:hive2://vm-cdh-01:1/testdb> show databases; -+ database_name -+ test_mvaldb default -+ 2 rows selected (0.726 seconds) 0: jdbc:hive2://vm-cdh-01:1/testdb> show tables; ---+ tab_name ---+ ---+ No rows selected (1.033 seconds) 0: jdbc:hive2://vm-cdh-01:1/testdb> !quit Closing: 0: jdbc:hive2://vm-cdh-01:1/testdb;principal=hive/_HOST@MITKDC
[jira] [Commented] (SENTRY-2109) Fix the logic of identifying HMS out of Sync and handle gaps and out-of-sequence notifications.
[ https://issues.apache.org/jira/browse/SENTRY-2109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16339528#comment-16339528 ] Alexander Kolbasov commented on SENTRY-2109: [~kkalyan] ignore the title, it takes care of gaps as well. The fix strictly serializes all updates to notifications using row lock on the notification ID row, so it essentially works as a distributed lock. As a result, all artifacts related to concurrent modifications disappear. > Fix the logic of identifying HMS out of Sync and handle gaps and > out-of-sequence notifications. > --- > > Key: SENTRY-2109 > URL: https://issues.apache.org/jira/browse/SENTRY-2109 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.1.0 >Reporter: kalyan kumar kalvagadda >Assignee: kalyan kumar kalvagadda >Priority: Major > Fix For: 2.1.0 > > Attachments: SENTRY-2109.001.patch, SENTRY-2109.002.patch, > SENTRY-2109.003.patch, SENTRY-2109.004.patch, SENTRY-2109.005.patch, > SENTRY-2109.006.patch, SENTRY-2109.007.patch, SENTRY-2109.008.patch, > SENTRY-2109.009.patch, SENTRY-2109.010.patch, SENTRY-2109.010.patch, > Screenshot_HMS_NOTIFICATION_LOG.png > > > Currently HMSFollower proactively checks if sentry is out of sync with HMS > and initiates full snapshot, if needed. > There will be false positives with the current logic if there are gaps in the > event-id in the notification log sequence. > This jira is aimed at making that logic robust. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (SENTRY-2106) Remove sentry dependency on HMS table NOTIFICATION_SEQUENCE when checking if Sentry is out-of-sync with HMS
[ https://issues.apache.org/jira/browse/SENTRY-2106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16339517#comment-16339517 ] Alexander Kolbasov commented on SENTRY-2106: As mentioned in SENTRY-2109, it is better to fix the root cause on the HMS side. > Remove sentry dependency on HMS table NOTIFICATION_SEQUENCE when checking if > Sentry is out-of-sync with HMS > --- > > Key: SENTRY-2106 > URL: https://issues.apache.org/jira/browse/SENTRY-2106 > Project: Sentry > Issue Type: New Feature > Components: Sentry >Affects Versions: 2.1.0 >Reporter: Arjun Mishra >Assignee: Arjun Mishra >Priority: Major > Labels: features > Fix For: 2.1.0 > > Attachments: SENTRY-2106.01.patch, SENTRY-2106.02.patch, > SENTRY-2106.03.patch, SENTRY-2106.04.patch, SENTRY-2106.05.patch, > SENTRY-2106.06.patch > > > After steady state, a full snapshot is triggered by mainly 2 cases > # Sentry is "ahead" > # Sentry is behind > Case 1 has a dependency on NOTIFICATION_SEQUENCE table. This is not reliable > as it was observed that sometimes NOTIFICATION_SEQUENCE and NOTIFICATION_LOG > are not in sync. As a result of this unnecessary full snapshots can be > triggered. > The fix is to remove this dependency > > PLEASE NOTE THAT THIS SHOULD BE COMMITTED WITH SENTRY-2109 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (SENTRY-2127) Fix unstable unit test TestColumnEndToEnd.testCrossDbTableOperations
[
https://issues.apache.org/jira/browse/SENTRY-2127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16339490#comment-16339490
]
Alexander Kolbasov commented on SENTRY-2127:
[~LinaAtAustin] Thanks for clarification! I think for some simple queries there
is no MR job created, but more complicated ones do need MR job.
> Fix unstable unit test TestColumnEndToEnd.testCrossDbTableOperations
>
>
> Key: SENTRY-2127
> URL: https://issues.apache.org/jira/browse/SENTRY-2127
> Project: Sentry
> Issue Type: Bug
> Components: Sentry
>Affects Versions: 2.1.0
>Reporter: Na Li
>Assignee: Na Li
>Priority: Major
> Attachments: SENTRY-2127.001.patch
>
>
> Occasionally, the test TestColumnEndToEnd.testCrossDbTableOperations fails at
> statement
> statement.execute("CREATE table db_1.t1 as select tb1.id, tb3.val, tb2.num
> from db_1.tb1,db_2.tb3,db_2.tb2");
> We should make it stable to avoid blocking other updates.
> More info:
> 1) Failed run with the following statement. The reason it fails is not
> related to sentry processing. It is due to the map reduce task fails with the
> more complicated join in the unit test
> {code:java}
> 2018-01-23 19:46:28,766 (5c4ce662-7587-40a5-8b24-b4c31f006b53
> HiveServer2-Handler-Pool: Thread-365) [INFO -
> org.apache.hadoop.hive.ql.Driver.compile(Driver.java:460)] Compiling
> command(queryId=jenkins_20180123194628_73ef2fc1-7819-4329-9d1c-96775dcc839c):
> CREATE table db_1.t1 as select tb1.id, tb3.val, tb2.num from
> db_1.tb1,db_2.tb3,db_2.tb2
> ...
> 2018-01-23 19:46:29,266 (HiveServer2-Background-Pool: Thread-373) [INFO -
> org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1713)] Executing
> command(queryId=jenkins_20180123194628_73ef2fc1-7819-4329-9d1c-96775dcc839c):
> CREATE table db_1.t1 as select tb1.id, tb3.val, tb2.num from
> db_1.tb1,db_2.tb3,db_2.tb2
> ...
> 2018-01-23 19:49:29,229 (HiveServer2-Background-Pool: Thread-373) [INFO -
> org.apache.hadoop.hive.ql.Driver.execute(Driver.java:2021)] Completed
> executing
> command(queryId=jenkins_20180123194628_73ef2fc1-7819-4329-9d1c-96775dcc839c);
> Time taken: 179.962 seconds
> 2018-01-23 19:49:29,235 (HiveServer2-Background-Pool: Thread-373) [ERROR -
> org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:345)]
> Error running hive query:
> org.apache.hive.service.cli.HiveSQLException: Error while processing
> statement: FAILED: Execution Error, return code 2 from
> org.apache.hadoop.hive.ql.exec.mr.MapRedTask
> at
> org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:330)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:254)
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$700(SQLOperation.java:91)
> at
> org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:342)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962)
> at
> org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:354)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> 2) Success run with following statement in unit test
> {code:java}
> statement.execute("CREATE table db_1.t2 as select tb1.id from db_1.tb1");
> 2018-01-24 10:29:01,071 (c042d8d8-423b-4ddf-a50b-63f1dd3e3f50
> HiveServer2-Handler-Pool: Thread-365) [INFO -
> org.apache.hadoop.hive.ql.Driver.compile(Driver.java:460)] Compiling
> command(queryId=jenkins_20180124102901_6376f6e1-0e74-4b05-b0a1-ab3ca5512e94):
> CREATE table db_1.t2 as select tb1.id from db_1.tb1
> 2018-01-24 10:29:01,513 (HiveServer2-Background-Pool: Thread-372) [INFO -
> org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1713)] Executing
> command(queryId=jenkins_20180124102901_6376f6e1-0e74-4b05-b0a1-ab3ca5512e94):
> CREATE table db_1.t2 as select tb1.id from db_1.tb1
> 2018-01-24 10:29:03,636 (HiveServer2-Background-Pool: Thread-372) [INFO -
> org.apache.hadoop.hive.ql.Driver.execute(Driver.java:2021)] Completed
> executing
> command(queryId=jenkins_20180124102901_6376f6e1-0e74-4b05-b0a1-ab3ca5512e94);
> Time taken: 2.122 seconds
> OK{code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (SENTRY-2109) Fix the logic of identifying HMS out of Sync and handle gaps and out-of-sequence notifications.
[ https://issues.apache.org/jira/browse/SENTRY-2109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16339448#comment-16339448 ] Alexander Kolbasov commented on SENTRY-2109: HIVE-18526 provides a very simple fix on the Hive side. I think it is better to fix the root cause of the problem rather then create a complicated workaround. > Fix the logic of identifying HMS out of Sync and handle gaps and > out-of-sequence notifications. > --- > > Key: SENTRY-2109 > URL: https://issues.apache.org/jira/browse/SENTRY-2109 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.1.0 >Reporter: kalyan kumar kalvagadda >Assignee: kalyan kumar kalvagadda >Priority: Major > Fix For: 2.1.0 > > Attachments: SENTRY-2109.001.patch, SENTRY-2109.002.patch, > SENTRY-2109.003.patch, SENTRY-2109.004.patch, SENTRY-2109.005.patch, > SENTRY-2109.006.patch, SENTRY-2109.007.patch, SENTRY-2109.008.patch, > SENTRY-2109.009.patch, SENTRY-2109.010.patch, > Screenshot_HMS_NOTIFICATION_LOG.png > > > Currently HMSFollower proactively checks if sentry is out of sync with HMS > and initiates full snapshot, if needed. > There will be false positives with the current logic if there are gaps in the > event-id in the notification log sequence. > This jira is aimed at making that logic robust. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (SENTRY-2127) Fix unstable unit test TestColumnEndToEnd.testCrossDbTableOperations
[
https://issues.apache.org/jira/browse/SENTRY-2127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16338567#comment-16338567
]
Alexander Kolbasov commented on SENTRY-2127:
What is the actual problem?
> Fix unstable unit test TestColumnEndToEnd.testCrossDbTableOperations
>
>
> Key: SENTRY-2127
> URL: https://issues.apache.org/jira/browse/SENTRY-2127
> Project: Sentry
> Issue Type: Bug
> Components: Sentry
>Affects Versions: 2.1.0
>Reporter: Na Li
>Assignee: Na Li
>Priority: Major
> Attachments: SENTRY-2127.001.patch
>
>
> Occasionally, the test TestColumnEndToEnd.testCrossDbTableOperations fails at
> statement
> statement.execute("CREATE table db_1.t1 as select tb1.id, tb3.val, tb2.num
> from db_1.tb1,db_2.tb3,db_2.tb2");
> We should make it stable to avoid blocking other updates.
> More info:
> 1) Failed run with the following statement. The reason it fails is not
> related to sentry processing. It is due to the map reduce task fails with the
> more complicated join in the unit test
> {code:java}
> 2018-01-23 19:46:28,766 (5c4ce662-7587-40a5-8b24-b4c31f006b53
> HiveServer2-Handler-Pool: Thread-365) [INFO -
> org.apache.hadoop.hive.ql.Driver.compile(Driver.java:460)] Compiling
> command(queryId=jenkins_20180123194628_73ef2fc1-7819-4329-9d1c-96775dcc839c):
> CREATE table db_1.t1 as select tb1.id, tb3.val, tb2.num from
> db_1.tb1,db_2.tb3,db_2.tb2
> ...
> 2018-01-23 19:46:29,266 (HiveServer2-Background-Pool: Thread-373) [INFO -
> org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1713)] Executing
> command(queryId=jenkins_20180123194628_73ef2fc1-7819-4329-9d1c-96775dcc839c):
> CREATE table db_1.t1 as select tb1.id, tb3.val, tb2.num from
> db_1.tb1,db_2.tb3,db_2.tb2
> ...
> 2018-01-23 19:49:29,229 (HiveServer2-Background-Pool: Thread-373) [INFO -
> org.apache.hadoop.hive.ql.Driver.execute(Driver.java:2021)] Completed
> executing
> command(queryId=jenkins_20180123194628_73ef2fc1-7819-4329-9d1c-96775dcc839c);
> Time taken: 179.962 seconds
> 2018-01-23 19:49:29,235 (HiveServer2-Background-Pool: Thread-373) [ERROR -
> org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:345)]
> Error running hive query:
> org.apache.hive.service.cli.HiveSQLException: Error while processing
> statement: FAILED: Execution Error, return code 2 from
> org.apache.hadoop.hive.ql.exec.mr.MapRedTask
> at
> org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:330)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:254)
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$700(SQLOperation.java:91)
> at
> org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:342)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962)
> at
> org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:354)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> 2) Success run with following statement in unit test
> {code:java}
> statement.execute("CREATE table db_1.t2 as select tb1.id from db_1.tb1");
> 2018-01-24 10:29:01,071 (c042d8d8-423b-4ddf-a50b-63f1dd3e3f50
> HiveServer2-Handler-Pool: Thread-365) [INFO -
> org.apache.hadoop.hive.ql.Driver.compile(Driver.java:460)] Compiling
> command(queryId=jenkins_20180124102901_6376f6e1-0e74-4b05-b0a1-ab3ca5512e94):
> CREATE table db_1.t2 as select tb1.id from db_1.tb1
> 2018-01-24 10:29:01,513 (HiveServer2-Background-Pool: Thread-372) [INFO -
> org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1713)] Executing
> command(queryId=jenkins_20180124102901_6376f6e1-0e74-4b05-b0a1-ab3ca5512e94):
> CREATE table db_1.t2 as select tb1.id from db_1.tb1
> 2018-01-24 10:29:03,636 (HiveServer2-Background-Pool: Thread-372) [INFO -
> org.apache.hadoop.hive.ql.Driver.execute(Driver.java:2021)] Completed
> executing
> command(queryId=jenkins_20180124102901_6376f6e1-0e74-4b05-b0a1-ab3ca5512e94);
> Time taken: 2.122 seconds
> OK{code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (SENTRY-2109) Fix the logic of identifying HMS out of Sync and handle gaps and out-of-sequence notifications.
[ https://issues.apache.org/jira/browse/SENTRY-2109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16338147#comment-16338147 ] Alexander Kolbasov commented on SENTRY-2109: Added a link. I am not quite right though - the fix is in Hive 3.0 but not in Hive 2 branch. It should be backported to Hive 2. > Fix the logic of identifying HMS out of Sync and handle gaps and > out-of-sequence notifications. > --- > > Key: SENTRY-2109 > URL: https://issues.apache.org/jira/browse/SENTRY-2109 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.1.0 >Reporter: kalyan kumar kalvagadda >Assignee: kalyan kumar kalvagadda >Priority: Major > Fix For: 2.1.0 > > Attachments: SENTRY-2109.001.patch, SENTRY-2109.002.patch, > SENTRY-2109.003.patch, SENTRY-2109.004.patch, SENTRY-2109.005.patch, > SENTRY-2109.006.patch, SENTRY-2109.007.patch, SENTRY-2109.008.patch, > Screenshot_HMS_NOTIFICATION_LOG.png > > > Currently HMSFollower proactively checks if sentry is out of sync with HMS > and initiates full snapshot, if needed. > There will be false positives with the current logic if there are gaps in the > event-id in the notification log sequence. > This jira is aimed at making that logic robust. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (SENTRY-2109) Fix the logic of identifying HMS out of Sync and handle gaps and out-of-sequence notifications.
[ https://issues.apache.org/jira/browse/SENTRY-2109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16338109#comment-16338109 ] Alexander Kolbasov commented on SENTRY-2109: Note that the original problem (HMS notifications duplicates and other weirdness) is fixed in Hive upstream. So this is a work-around for a problem that is already fixed on the HMS side. > Fix the logic of identifying HMS out of Sync and handle gaps and > out-of-sequence notifications. > --- > > Key: SENTRY-2109 > URL: https://issues.apache.org/jira/browse/SENTRY-2109 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.1.0 >Reporter: kalyan kumar kalvagadda >Assignee: kalyan kumar kalvagadda >Priority: Major > Fix For: 2.1.0 > > Attachments: SENTRY-2109.001.patch, SENTRY-2109.002.patch, > SENTRY-2109.003.patch, SENTRY-2109.004.patch, SENTRY-2109.005.patch, > SENTRY-2109.006.patch, SENTRY-2109.007.patch, SENTRY-2109.008.patch, > Screenshot_HMS_NOTIFICATION_LOG.png > > > Currently HMSFollower proactively checks if sentry is out of sync with HMS > and initiates full snapshot, if needed. > There will be false positives with the current logic if there are gaps in the > event-id in the notification log sequence. > This jira is aimed at making that logic robust. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (SENTRY-2109) Fix the logic of identifying HMS out of Sync and handle gaps and out-of-sequence notifications.
[ https://issues.apache.org/jira/browse/SENTRY-2109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16338103#comment-16338103 ] Alexander Kolbasov commented on SENTRY-2109: Can't build the latest patch. > Fix the logic of identifying HMS out of Sync and handle gaps and > out-of-sequence notifications. > --- > > Key: SENTRY-2109 > URL: https://issues.apache.org/jira/browse/SENTRY-2109 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.1.0 >Reporter: kalyan kumar kalvagadda >Assignee: kalyan kumar kalvagadda >Priority: Major > Fix For: 2.1.0 > > Attachments: SENTRY-2109.001.patch, SENTRY-2109.002.patch, > SENTRY-2109.003.patch, SENTRY-2109.004.patch, SENTRY-2109.005.patch, > SENTRY-2109.006.patch, SENTRY-2109.007.patch, SENTRY-2109.008.patch, > Screenshot_HMS_NOTIFICATION_LOG.png > > > Currently HMSFollower proactively checks if sentry is out of sync with HMS > and initiates full snapshot, if needed. > There will be false positives with the current logic if there are gaps in the > event-id in the notification log sequence. > This jira is aimed at making that logic robust. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (SENTRY-1904) TransactionManager should limit the max time spent by transaction retry
[ https://issues.apache.org/jira/browse/SENTRY-1904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16338096#comment-16338096 ] Alexander Kolbasov commented on SENTRY-1904: The latest patch seems to contain unrelated changes. As for the approach - I think that if we use max time we should stop using number of retries as a criteria and just use total time. Using both doesn't make much sense. > TransactionManager should limit the max time spent by transaction retry > --- > > Key: SENTRY-1904 > URL: https://issues.apache.org/jira/browse/SENTRY-1904 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.0.0 >Reporter: Alexander Kolbasov >Assignee: kalyan kumar kalvagadda >Priority: Major > Attachments: SENTRY-1904.001.patch, SENTRY-1904.002.patch, > SENTRY-1904.003.patch > > > The TransactionManager uses exponential backoff strategy for transaction > retries. This may cause some transactions to be delayed by a very long time. > We should also have a constraint on the max time for a transaction so that we > do not retry for too long. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (SENTRY-2124) LeaderStatusMonitor.toString() throws IllegalFormatConversionException with AtomicLong
[
https://issues.apache.org/jira/browse/SENTRY-2124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexander Kolbasov updated SENTRY-2124:
---
Labels: bite-sized newbie (was: )
> LeaderStatusMonitor.toString() throws IllegalFormatConversionException with
> AtomicLong
> --
>
> Key: SENTRY-2124
> URL: https://issues.apache.org/jira/browse/SENTRY-2124
> Project: Sentry
> Issue Type: Bug
> Components: Sentry
>Affects Versions: 2.1.0
>Reporter: Sergio Peña
>Assignee: Xinran Tinney
>Priority: Minor
> Labels: bite-sized, newbie
>
> When I start Sentry, I noticed an exception on the console due to an illegal
> format string in LeaderStatusMonitor.toString()
> {noformat}
> java.util.IllegalFormatConversionException: d !=
> java.util.concurrent.atomic.AtomicLong
> at java.util.Formatter$FormatSpecifier.failConversion(Formatter.java:4302)
> at java.util.Formatter$FormatSpecifier.printInteger(Formatter.java:2793)
> at java.util.Formatter$FormatSpecifier.print(Formatter.java:2747)
> at java.util.Formatter.format(Formatter.java:2520)
> at java.util.Formatter.format(Formatter.java:2455)
> at java.lang.String.format(String.java:2940)
> at
> org.apache.sentry.provider.db.service.persistent.LeaderStatusMonitor.toString(LeaderStatusMonitor.java:282)
>
> at
> org.slf4j.helpers.MessageFormatter.safeObjectAppend(MessageFormatter.java:297)
>
> at
> org.slf4j.helpers.MessageFormatter.deeplyAppendParameter(MessageFormatter.java:269)
>
> at org.slf4j.helpers.MessageFormatter.arrayFormat(MessageFormatter.java:227)
> at org.slf4j.helpers.MessageFormatter.format(MessageFormatter.java:124)
> at org.slf4j.impl.Log4jLoggerAdapter.info(Log4jLoggerAdapter.java:322)
> at
> org.apache.sentry.provider.db.service.persistent.LeaderStatusMonitor.takeLeadership(LeaderStatusMonitor.java:251)
>
> at
> sentry.org.apache.curator.framework.recipes.leader.LeaderSelector$WrappedListener.takeLeadership(LeaderSelector.java:537)
>
> at
> sentry.org.apache.curator.framework.recipes.leader.LeaderSelector.doWork(LeaderSelector.java:399)
>
> at
> sentry.org.apache.curator.framework.recipes.leader.LeaderSelector.doWorkLoop(LeaderSelector.java:444)
>
> at
> sentry.org.apache.curator.framework.recipes.leader.LeaderSelector.access$100(LeaderSelector.java:64)
>
> at
> sentry.org.apache.curator.framework.recipes.leader.LeaderSelector$2.call(LeaderSelector.java:245)
>
> at
> sentry.org.apache.curator.framework.recipes.leader.LeaderSelector$2.call(LeaderSelector.java:239)
>
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748){noformat}
> Seems that the {{leaderCount=%d}} used in the String.format() is causing the
> issue.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (SENTRY-2097) Sentry privileges model: Can Sentry take a database privileges away from a server privileges?
[
https://issues.apache.org/jira/browse/SENTRY-2097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16296045#comment-16296045
]
Alexander Kolbasov commented on SENTRY-2097:
[~kkalyan] Thanks for updating the wiki with clear documentation!
> Sentry privileges model: Can Sentry take a database privileges away from a
> server privileges?
> -
>
> Key: SENTRY-2097
> URL: https://issues.apache.org/jira/browse/SENTRY-2097
> Project: Sentry
> Issue Type: Bug
> Components: Sentry
>Reporter: Sergio Peña
>Assignee: Na Li
>Priority: Minor
>
> Assume I have a user |jack| and a group |datateam|. The
> user |jack| belongs to group |datateam|.
> Use Sentry for authorization.
> |create role admin; grant role admin to group datateam; grant all on
> server server1 to role admin; |
> Now the role |admin| has the following priveleges.
> {noformat}
> |+---+++-+-+-++---+---+--+--+
> | database | table | partition | column | principal_name |
> principal_type | privilege | grant_option | grant_time | grantor |
> +---+++-+-+-++---+---+--+--+
> | * | | | | admin | ROLE | * | false | 1480985013185000 | -- |
> +---+++-+-+-++---+---+--+--+|
> {noformat}
> Assume I have this database.
> |create database testdb; |
> It is successful. User |jack| created a database |testdb|.
> Use Sentry to revoke the privileges on |testdb|;
> |revoke all on database `testdb` from role admin; |
> The priveleges is still the same.
> {noformat}
> |+---+++-+-+-++---+---+--+--+
> | database | table | partition | column | principal_name |
> principal_type | privilege | grant_option | grant_time | grantor |
> +---+++-+-+-++---+---+--+--+
> | * | | | | admin | ROLE | * | false | 1480985013185000 | -- |
> +---+++-+-+-++---+---+--+--+|
> {noformat}
> Shouldn't Sentry take the privileges on database |testdb| away from the
> server |server1|?
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (SENTRY-2102) Switching to HttpServer2 for WebUI access
[ https://issues.apache.org/jira/browse/SENTRY-2102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16293405#comment-16293405 ] Alexander Kolbasov commented on SENTRY-2102: That's great! Just having SSL support is worth it! > Switching to HttpServer2 for WebUI access > - > > Key: SENTRY-2102 > URL: https://issues.apache.org/jira/browse/SENTRY-2102 > Project: Sentry > Issue Type: Improvement >Reporter: Vadim Spector > > There is the implementation of HTTP server which is supposed to be the > standard for all Hadoop components implementing Web UIs: > https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/http/HttpServer2.java > It comes with built-in support of SSL, SPNEGO (Kerberos from browsers), plus > authorization - all configurable. > It comes with pre-configured servlets, such as Stack, LogLevel servlet, > Metrics, JMX, Configuration, all for free. > Sentry code is still using its own SentryWebServer.java implementation. ? > Some of Sentry's servlets are already supported in HttpServer2 (maybe even > improved versions of those). > In addition, HttpServer2 security features satisfy security demands of > commercial deployment, unlike our own Web UI, which security-conscious > customers may be reluctant to activate for good reasons. > Switching to HttpServer2 would be a major improvement of the Sentry product > (and less code to support). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (SENTRY-2078) Have sentry server print an obvious INFO level log message when it becomes the writer
[ https://issues.apache.org/jira/browse/SENTRY-2078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Kolbasov updated SENTRY-2078: --- Resolution: Fixed Status: Resolved (was: Patch Available) [~arjunmishra13] Thank you for your contribution! > Have sentry server print an obvious INFO level log message when it becomes > the writer > - > > Key: SENTRY-2078 > URL: https://issues.apache.org/jira/browse/SENTRY-2078 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 2.1.0 >Reporter: Arjun Mishra >Assignee: Arjun Mishra >Priority: Minor > Labels: newbie > Fix For: 2.1.0 > > Attachments: SENTRY-2078.01.patch, SENTRY-2078.02.patch, > SENTRY-2078.03.patch, SENTRY-2078.04.patch, SENTRY-2078.05.patch, > SENTRY-2078.06.patch > > > Currently sentry's log messaging when it becomes a 'writer' (the sentry > server that reads from HMS) is not obvious, and is really mostly discernable > at the debug level. Add a log message at the INFO level, such as 'This sentry > server has just become the HMS reader/writer' that is printed once when the > sentry server first becomes the HMS reader/writer. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (SENTRY-1960) Use DB auto-increment for incrementing table IDs
[
https://issues.apache.org/jira/browse/SENTRY-1960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexander Kolbasov reassigned SENTRY-1960:
--
Assignee: (was: Alexander Kolbasov)
> Use DB auto-increment for incrementing table IDs
>
>
> Key: SENTRY-1960
> URL: https://issues.apache.org/jira/browse/SENTRY-1960
> Project: Sentry
> Issue Type: Improvement
> Components: Sentry
>Affects Versions: 2.0.0, 2.1.0
>Reporter: Alexander Kolbasov
> Labels: sentry-ha
>
> We noticed that the way package.jdo is configured, it uses DataNucleus
> sequence table to maintain keys for table IDs. For new tables we should use
> datastore-based identity which is much more efficient.
> For example, setting
> {code}
>
>
>
> {code}
> {code}
>
>
>
> {code}
> noticeably improve performance for saving full HMS snapshot in DB.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (SENTRY-1960) Use DB auto-increment for incrementing table IDs
[
https://issues.apache.org/jira/browse/SENTRY-1960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexander Kolbasov updated SENTRY-1960:
---
Status: Open (was: Patch Available)
> Use DB auto-increment for incrementing table IDs
>
>
> Key: SENTRY-1960
> URL: https://issues.apache.org/jira/browse/SENTRY-1960
> Project: Sentry
> Issue Type: Improvement
> Components: Sentry
>Affects Versions: 2.0.0, 2.1.0
>Reporter: Alexander Kolbasov
>Assignee: Alexander Kolbasov
> Labels: sentry-ha
>
> We noticed that the way package.jdo is configured, it uses DataNucleus
> sequence table to maintain keys for table IDs. For new tables we should use
> datastore-based identity which is much more efficient.
> For example, setting
> {code}
>
>
>
> {code}
> {code}
>
>
>
> {code}
> noticeably improve performance for saving full HMS snapshot in DB.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (SENTRY-1960) Use DB auto-increment for incrementing table IDs
[
https://issues.apache.org/jira/browse/SENTRY-1960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexander Kolbasov updated SENTRY-1960:
---
Attachment: HIVE-18247.01.patch
> Use DB auto-increment for incrementing table IDs
>
>
> Key: SENTRY-1960
> URL: https://issues.apache.org/jira/browse/SENTRY-1960
> Project: Sentry
> Issue Type: Improvement
> Components: Sentry
>Affects Versions: 2.0.0, 2.1.0
>Reporter: Alexander Kolbasov
>Assignee: Alexander Kolbasov
> Labels: sentry-ha
> Attachments: HIVE-18247.01.patch
>
>
> We noticed that the way package.jdo is configured, it uses DataNucleus
> sequence table to maintain keys for table IDs. For new tables we should use
> datastore-based identity which is much more efficient.
> For example, setting
> {code}
>
>
>
> {code}
> {code}
>
>
>
> {code}
> noticeably improve performance for saving full HMS snapshot in DB.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (SENTRY-1960) Use DB auto-increment for incrementing table IDs
[
https://issues.apache.org/jira/browse/SENTRY-1960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexander Kolbasov updated SENTRY-1960:
---
Attachment: (was: HIVE-18247.01.patch)
> Use DB auto-increment for incrementing table IDs
>
>
> Key: SENTRY-1960
> URL: https://issues.apache.org/jira/browse/SENTRY-1960
> Project: Sentry
> Issue Type: Improvement
> Components: Sentry
>Affects Versions: 2.0.0, 2.1.0
>Reporter: Alexander Kolbasov
> Labels: sentry-ha
>
> We noticed that the way package.jdo is configured, it uses DataNucleus
> sequence table to maintain keys for table IDs. For new tables we should use
> datastore-based identity which is much more efficient.
> For example, setting
> {code}
>
>
>
> {code}
> {code}
>
>
>
> {code}
> noticeably improve performance for saving full HMS snapshot in DB.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (SENTRY-1960) Use DB auto-increment for incrementing table IDs
[
https://issues.apache.org/jira/browse/SENTRY-1960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexander Kolbasov updated SENTRY-1960:
---
Status: Patch Available (was: In Progress)
> Use DB auto-increment for incrementing table IDs
>
>
> Key: SENTRY-1960
> URL: https://issues.apache.org/jira/browse/SENTRY-1960
> Project: Sentry
> Issue Type: Improvement
> Components: Sentry
>Affects Versions: 2.0.0, 2.1.0
>Reporter: Alexander Kolbasov
>Assignee: Alexander Kolbasov
> Labels: sentry-ha
>
> We noticed that the way package.jdo is configured, it uses DataNucleus
> sequence table to maintain keys for table IDs. For new tables we should use
> datastore-based identity which is much more efficient.
> For example, setting
> {code}
>
>
>
> {code}
> {code}
>
>
>
> {code}
> noticeably improve performance for saving full HMS snapshot in DB.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Assigned] (SENTRY-1960) Use DB auto-increment for incrementing table IDs
[
https://issues.apache.org/jira/browse/SENTRY-1960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexander Kolbasov reassigned SENTRY-1960:
--
Assignee: Alexander Kolbasov
> Use DB auto-increment for incrementing table IDs
>
>
> Key: SENTRY-1960
> URL: https://issues.apache.org/jira/browse/SENTRY-1960
> Project: Sentry
> Issue Type: Improvement
> Components: Sentry
>Affects Versions: 2.0.0, 2.1.0
>Reporter: Alexander Kolbasov
>Assignee: Alexander Kolbasov
> Labels: sentry-ha
>
> We noticed that the way package.jdo is configured, it uses DataNucleus
> sequence table to maintain keys for table IDs. For new tables we should use
> datastore-based identity which is much more efficient.
> For example, setting
> {code}
>
>
>
> {code}
> {code}
>
>
>
> {code}
> noticeably improve performance for saving full HMS snapshot in DB.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (SENTRY-1473) [test]: add more tests to test sentry supports only absolute path for now: s3://bucketname/ is correct; while s3://bucket is not
[ https://issues.apache.org/jira/browse/SENTRY-1473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16289961#comment-16289961 ] Alexander Kolbasov commented on SENTRY-1473: [~anneyu] is no longer active in Sentry, so you'll need to reconstruct the problem by yourself. > [test]: add more tests to test sentry supports only absolute path for now: > s3://bucketname/ is correct; while s3://bucket is not > > > Key: SENTRY-1473 > URL: https://issues.apache.org/jira/browse/SENTRY-1473 > Project: Sentry > Issue Type: Test > Components: Sentry, Test >Affects Versions: 1.8.0 >Reporter: Anne Yu >Assignee: Steve Moist > Labels: aws-s3, test > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
