[
https://issues.apache.org/jira/browse/SENTRY-1792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16041758#comment-16041758
]
Vamsee Yarlagadda commented on SENTRY-1792:
-------------------------------------------
SENTRY-1795 has solved the issue of holes in the relevant tables. Instead of
using auto-incrementing property, we now add the number along with the
transaction itself. That being said, the current jira is mostly solved and the
only thing that needs to be taken care of is to make sure the ID's returned
from the DB are in order to avoid NN jumping the transaction ID's.
> HDFS sync should be aware of holes in delta table
> -------------------------------------------------
>
> Key: SENTRY-1792
> URL: https://issues.apache.org/jira/browse/SENTRY-1792
> Project: Sentry
> Issue Type: Sub-task
> Components: Sentry
> Affects Versions: sentry-ha-redesign
> Reporter: Na Li
> Assignee: Na Li
> Labels: sentry-ha
> Fix For: sentry-ha-redesign
>
> Attachments: SENTRY-1792.001-sentry-ha-redesign.patch,
> SENTRY-1792.002-sentry-ha-redesign.patch,
> SENTRY-1792.003-sentry-ha-redesign.patch
>
>
> Sentry server returns full permission snapshot almost every 10 seconds. The
> desired behavior is to reduce the chance of getting full snapshot
> significantly.
> From debug message, we can see the returned change list is empty.
> That causes Sentry to send back full snapshot instead of delta changes. It
> has performance penalty because it takes much longer to get a full snapshot
> than getting the delta changes.
> {code}
> 2017-06-05 14:00:21,030 INFO org.apache.sentry.hdfs.DBUpdateForwarder: ####
> GetAllUpdatesFrom return full image with isDeltaAvailable:true,
> isDeltaEmpty:true, reqSeqNum:1076942, curSeqNum:1076948 ; type:
> PermImageRetriever
> 2017-06-05 14:00:31,905 INFO org.apache.sentry.hdfs.DBUpdateForwarder: ####
> GetAllUpdatesFrom return full image with isDeltaAvailable:true,
> isDeltaEmpty:true, reqSeqNum:1076949, curSeqNum:1077064 ; type:
> PermImageRetriever
> 2017-06-05 14:01:00,041 INFO org.apache.sentry.hdfs.DBUpdateForwarder: ####
> GetAllUpdatesFrom return full image with isDeltaAvailable:true,
> isDeltaEmpty:true, reqSeqNum:1077354, curSeqNum:1077361 ; type:
> PermImageRetriever
> 2017-06-05 14:01:25,735 INFO org.apache.sentry.hdfs.DBUpdateForwarder: ####
> GetAllUpdatesFrom return full image with isDeltaAvailable:true,
> isDeltaEmpty:true, reqSeqNum:1077630, curSeqNum:1077636 ; type:
> PermImageRetriever
> 2017-06-05 14:01:42,197 INFO org.apache.sentry.hdfs.DBUpdateForwarder: ####
> GetAllUpdatesFrom return full image with isDeltaAvailable:true,
> isDeltaEmpty:true, reqSeqNum:1077820, curSeqNum:1077826 ; type:
> PermImageRetriever
> 2017-06-05 14:01:53,491 INFO org.apache.sentry.hdfs.DBUpdateForwarder: ####
> GetAllUpdatesFrom return full image with isDeltaAvailable:true,
> isDeltaEmpty:true, reqSeqNum:1077827, curSeqNum:1077962 ; type:
> PermImageRetriever
> 2017-06-05 14:02:09,472 INFO org.apache.sentry.hdfs.DBUpdateForwarder: ####
> GetAllUpdatesFrom return full image with isDeltaAvailable:true,
> isDeltaEmpty:true, reqSeqNum:1078147, curSeqNum:1078155 ; type:
> PermImageRetriever
> 2017-06-05 14:02:21,527 INFO org.apache.sentry.hdfs.DBUpdateForwarder: ####
> GetAllUpdatesFrom return full image with isDeltaAvailable:true,
> isDeltaEmpty:true, reqSeqNum:1078300, curSeqNum:1078305 ; type:
> PermImageRetriever
> 2017-06-05 14:02:35,761 INFO org.apache.sentry.hdfs.DBUpdateForwarder: ####
> GetAllUpdatesFrom return full image with isDeltaAvailable:true,
> isDeltaEmpty:true, reqSeqNum:1078483, curSeqNum:1078489 ; type:
> PermImageRetriever
> 2017-06-05 14:02:50,079 INFO org.apache.sentry.hdfs.DBUpdateForwarder: ####
> GetAllUpdatesFrom return full image with isDeltaAvailable:true,
> isDeltaEmpty:true, reqSeqNum:1078683, curSeqNum:1078686 ; type:
> PermImageRetriever
> 2017-06-05 14:03:09,889 INFO org.apache.sentry.hdfs.DBUpdateForwarder: ####
> GetAllUpdatesFrom return full image with isDeltaAvailable:true,
> isDeltaEmpty:true, reqSeqNum:1078958, curSeqNum:1078961 ; type:
> PermImageRetriever
> 2017-06-05 14:03:21,641 INFO org.apache.sentry.hdfs.DBUpdateForwarder: ####
> GetAllUpdatesFrom return full image with isDeltaAvailable:true,
> isDeltaEmpty:true, reqSeqNum:1079144, curSeqNum:1079150 ; type:
> PermImageRetriever
> 2017-06-05 14:03:32,600 INFO org.apache.sentry.hdfs.DBUpdateForwarder: ####
> GetAllUpdatesFrom return full image with isDeltaAvailable:true,
> isDeltaEmpty:true, reqSeqNum:1079324, curSeqNum:1079331 ; type:
> PermImageRetriever
> 2017-06-05 14:03:46,223 INFO org.apache.sentry.hdfs.DBUpdateForwarder: ####
> GetAllUpdatesFrom return full image with isDeltaAvailable:true,
> isDeltaEmpty:true, reqSeqNum:1079537, curSeqNum:1079546 ; type:
> PermImageRetriever
> 2017-06-05 14:03:59,870 INFO org.apache.sentry.hdfs.DBUpdateForwarder: ####
> GetAllUpdatesFrom return full image with isDeltaAvailable:true,
> isDeltaEmpty:true, reqSeqNum:1079762, curSeqNum:1079769 ; type:
> PermImageRetriever
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
