Vadim Spector created SENTRY-2110:
-------------------------------------
Summary: send HDFS full updates in incremental chunks, to overcome
Thrift 2Gb message size limit
Key: SENTRY-2110
URL: https://issues.apache.org/jira/browse/SENTRY-2110
Project: Sentry
Issue Type: Improvement
Reporter: Vadim Spector
Thrift messages are limited to 2 Gb. Sending full update for millions of
partitions from Sentry to HDFS plugin can at some point exceed the message size
limit. Sample figures: for 15 million partitions, 2 Gb translates to 143 bytes
per partition record which is not too much.
Full update can be split into several pieces. It will require
1. adding some additional fields to Thrift message schema, like the sequence
number of the incremental full update message and the total number of
incremental full update messages (e.g. messge #2 of 10 total)
2. Logic on Sentry side (SentryPlugin) to split full update into chunks and
manage sending full updates in those chunks, watching for acknowledgements from
HDFS plugin and resending chunks if needed.
3. Logic on HDFS plugin to assemble incremental chunks into full updates,
watching for chunk numbers, and asking SentryPlugin to resend a chunk if needed.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
