----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73329/ -----------------------------------------------------------
Review request for atlas, Radhika Kundam and Sarath Subramanian. Bugs: ATLAS-4152 https://issues.apache.org/jira/browse/ATLAS-4152 Repository: atlas Description ------- **Background** As part of ATLAS-4204, HS2 notifications send entity-lineage only (provided the poperty is enabled). When Spooling enabled the order of messages can potentially change. The notification messages coming from HS2 and HMS may not be in the same order as when they arrived with direct notification. Problem: Consider the sequence of arriving messages: This is the sequence of messages for Entity 1 (C = create, U = update, D = delete, L?x = Lineage of type 'x') No problem: C1, U1, L1x, L1y, D1 Problem: C1, U1, D1, L1x, L1y This implementation attempts to handle ths problem mentioned above. **Approach** Used bounded stream approach where an incoming stream of messages is bounded with an indicator that it originates from spool. This helps makes localized decisions on the incoming stream of messages. Diffs ----- intg/src/main/java/org/apache/atlas/model/notification/HookNotification.java 5b5fa04e2 notification/src/main/java/org/apache/atlas/notification/spool/AtlasFileSpool.java 2d7d19595 notification/src/main/java/org/apache/atlas/notification/spool/SpoolConfiguration.java a9a3a78cc webapp/src/main/java/org/apache/atlas/notification/EntityCorrelationManager.java PRE-CREATION webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java 84cc8d813 webapp/src/main/java/org/apache/atlas/notification/preprocessor/EntityPreprocessor.java 89568e236 webapp/src/main/java/org/apache/atlas/notification/preprocessor/HiveDdlLineagePreprocessor.java PRE-CREATION webapp/src/main/java/org/apache/atlas/notification/preprocessor/HivePreprocessor.java e69d63e3a webapp/src/main/java/org/apache/atlas/notification/preprocessor/PreprocessorContext.java 608b4a304 webapp/src/test/resources/json/hive-ddl.json PRE-CREATION Diff: https://reviews.apache.org/r/73329/diff/1/ Testing ------- **Functional tests** Manual verification of scenarios. Thanks, Ashutosh Mestry