[
https://issues.apache.org/jira/browse/SENTRY-1784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16033187#comment-16033187
]
Na Li commented on SENTRY-1784:
-------------------------------
Info about the fix:
HDFS starts requesting changeID = 0 instead of 1. DBUpdateForwarder returns
full update if the request changeID <= 0. After first full update, the request
changeID = 1, so only delta update is sent unless clean up removes changes that
are not sent to HDFS. This fixes both issues in this Jira.
1) What happens when NN just starts
NN sends requestedId = 0 to sentry server. Sentry server sends full update
back. The latest changeID for perm and path are returned to NN. NN saves the
latest changeID for perm and path.
NN initial changeID for permission and path is -1. The requestedId is NN
changeID + 1. So initial requestedId from NN to sentry server is 0.
2) What happens once NN gets a full snapshot
Once NN gets a full snapshot, it creats new UpdateableAuthzPermissions for full
perm update; it creates new UpdateableAuthzPaths for full path update;
It saves the latest changeID for perm and path (referred as latestChangeID_path
and latestChangeID_perm). Next interval, NN sends requestedId
(latestChangeID_path + 1, latestChangeID_perm + 1) to Sentry server. Since the
lowest latestChangeID is 0, this requestedId >= 1.
If Sentry server has changes equal or newer than requestedId, it sends back
delta changes to NN. Otherwise, it sends back empty list to NN. The exceptions
are: 1) the requestedId does not exist in SentryStore (maybe it is cleaned up)
or 2) the returned delta change list from SentryStore is empty (maybe the
change table is corrupted). If exception happens, full snapshot is sent back to
NN.
3) What happens when a delta is received from HMS.
NN updates the privilege and path based on the delta changes. It update save
the latest changeID for perm and path.
> DBUpdateForwarder returns empty update list to HDFS instead of full update
> --------------------------------------------------------------------------
>
> Key: SENTRY-1784
> URL: https://issues.apache.org/jira/browse/SENTRY-1784
> Project: Sentry
> Issue Type: Sub-task
> Components: Sentry
> Affects Versions: sentry-ha-redesign
> Reporter: Na Li
> Assignee: Na Li
> Priority: Critical
> Fix For: sentry-ha-redesign
>
> Attachments: SENTRY-1784.001-sentry-ha-redesign.patch,
> SENTRY-1784.001-sentry-ha-redesign.patch,
> SENTRY-1784.002-sentry-ha-redesign.patch,
> SENTRY-1784.003-sentry-ha-redesign.patch
>
>
> When HDFS first asks Sentry for update, the request changeID is 1. If there
> is no entries in permission change table (current changeID is 0), empty
> update list is returned to HDFS because the current code behavior is that no
> update is returned if request changeID is larger than the current chaneID.
> Once this problem is fixed (returning full update when request changeID = 1,
> and current changeID = 0), another issue of the current code is that after a
> full update is sent to client, and there is no entries in permission change
> table (current changeID is 0), the next request changeID is still 1,
> so full update is returned for the second time. Full update will be sent for
> each request until permission change table has at least one entry.
> The desired behavior is for HDFS to get full update when it sends the first
> request to Sentry. Once it gets full update, the following requests should
> get delta updates unless some updates are purged before they are sent to HDFS.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
