Na Li created SENTRY-1882:
-----------------------------
Summary: Stream HMS full snapshot
Key: SENTRY-1882
URL: https://issues.apache.org/jira/browse/SENTRY-1882
Project: Sentry
Issue Type: Improvement
Components: Sentry
Affects Versions: 2.0.0
Reporter: Na Li
Priority: Minor
Right now, we hold the meta store full snapshot in memory until it finishes.
Where there are a lot of tables, the size could be really big and may cause out
of memory issue.
It is better to stream the snapshot processing, saving into sentry data store
while getting next table. In this way, the memory requirement for getting full
snapshot can drop dramatically.
When we finish, we save the notificationID right before taking the snapshot.
Then get notifications with ID larger than notificationIDBefore, and process
them as normal notification events.
We don't need atomic write of the snapshot in a single transaction. We may be
able to do it in simpler ways
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
