-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68844/
-----------------------------------------------------------
(Updated Sept. 26, 2018, 7:38 p.m.)
Review request for atlas, Apoorv Naik, Ashutosh Mestry, keval bhatt, Nixon
Rodrigues, and Sarath Subramanian.
Changes
-------
updated configuration name to include HIVE-20633
Bugs: ATLAS-2891
https://issues.apache.org/jira/browse/ATLAS-2891
Repository: atlas
Description
-------
Processing of hook notifications at Atlas server was updated to skip
hive_column_lineage that is likely to contain incorrect lineage information due
to HIVE-20633. hive_column_lineage is skipped only when both the conditions
listed below are met:
- hive_column_lineage should have more than 5 inputs (configurable via
atlas.notification.consumer.skip.hive_column_lineage.inputs.threshold)
- all hive_column_lineage entities in the message should have same number of
inputs
This is can be disabled by setting configuration
atlas.notification.consumer.skip.hive_column_lineage.inputs.threshold to a
negative value.
Diffs (updated)
-----
addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/AtlasHiveHookContext.java
b9e4256be
addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java
19075f61c
addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/CreateHiveProcess.java
c99a699b2
notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaConsumer.java
e3bb71ca8
webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java
1a567af5c
Diff: https://reviews.apache.org/r/68844/diff/3/
Changes: https://reviews.apache.org/r/68844/diff/2-3/
Testing
-------
Verified that the notification processing skips potentially incorrect
hive_column_lineage; and following WARN logs is printed for each message where
hive_column_lineage was skiped.
2018-09-25 12:41:27,536 WARN - [NotificationHookConsumer thread-0:] ~ removed
120 hive_column_lineage entities, each having 241 inputs. offset=27648,
partition=0 (NotificationHookConsumer:687)
2018-09-25 12:41:46,799 WARN - [NotificationHookConsumer thread-0:] ~ removed
21 hive_column_lineage entities, each having 43 inputs. offset=27786,
partition=0 (NotificationHookConsumer:687)
2018-09-25 12:42:07,646 WARN - [NotificationHookConsumer thread-0:] ~ removed
26 hive_column_lineage entities, each having 53 inputs. offset=27947,
partition=0 (NotificationHookConsumer:687)
2018-09-25 12:42:19,210 WARN - [NotificationHookConsumer thread-0:] ~ removed
28 hive_column_lineage entities, each having 57 inputs. offset=28031,
partition=0 (NotificationHookConsumer:687)
2018-09-25 12:45:26,062 WARN - [NotificationHookConsumer thread-0:] ~ removed
584 hive_column_lineage entities, each having 1172 inputs. offset=28605,
partition=0 (NotificationHookConsumer:687)
2018-09-25 12:45:39,153 WARN - [NotificationHookConsumer thread-0:] ~ removed
584 hive_column_lineage entities, each having 1172 inputs. offset=28628,
partition=0 (NotificationHookConsumer:687)
2018-09-25 12:46:05,157 WARN - [NotificationHookConsumer thread-0:] ~ removed
584 hive_column_lineage entities, each having 1172 inputs. offset=28653,
partition=0 (NotificationHookConsumer:687)
2018-09-25 12:46:14,670 WARN - [NotificationHookConsumer thread-0:] ~ removed
33 hive_column_lineage entities, each having 68 inputs. offset=28665,
partition=0 (NotificationHookConsumer:687)
Thanks,
Madhan Neethiraj
