-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72636/
-----------------------------------------------------------
(Updated July 20, 2020, 10:44 p.m.)
Review request for atlas, Nikhil Bonte, Nixon Rodrigues, and Sarath Subramanian.
Changes
-------
Updates include:
- Addressed the case where synchronization was not handled for addition to set.
Bugs: ATLAS-3874
https://issues.apache.org/jira/browse/ATLAS-3874
Repository: atlas
Description
-------
**Background**
Please see bug description.
**Approach**
At a high-level: Introduce a notion where the individual consumers are aware of
the entities being processed by each other. If there are no entities being
processed concurrently, everything proceeds as usual (the way it is before his
change). If same entity is being procesed by multiple consumers, then one
consumer waits for the other to finish before proceeding.
Classes:
New *UniqueKeysExtractor*: Extracts values of unique keys from
*AtlasEntitiesWithExtInfo*. It navigates *relationshipAttributes* and
*attributes* that has *objectRef* set.
New *UniquenessChecker*: Maintains a set of unique keys provided by
*UniqueKeysExtractor*. It detects the presence of duplicates and waits until
duplicates are resolved.
Modified *NoitficationHookConsumer.createOrUpdate*
- Updates *UniquenessChecker* with output from *UniqueKeysExtractor*. Clears
the keys at the end of entity creation.
- Handles JanusGraph's *PermanentLockingException*. See
[ATLAS-3801](https://issues.apache.org/jira/browse/ATLAS-3801).
Modified *NotificationHookConsumer*: Accepts an shared instance of
*UniqunessChecker*.
**Additional Logging**
Log entry is added when attempt is made to creates entities concurrently.
Diffs (updated)
-----
webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java
3f1ea05e1
webapp/src/main/java/org/apache/atlas/notification/UniqueKeysExtractor.java
PRE-CREATION
webapp/src/main/java/org/apache/atlas/notification/UniquenessChecker.java
PRE-CREATION
webapp/src/test/java/org/apache/atlas/notification/UniquenessCheckerTest.java
PRE-CREATION
Diff: https://reviews.apache.org/r/72636/diff/2/
Changes: https://reviews.apache.org/r/72636/diff/1-2/
Testing
-------
**Unit tests**
Tests added to verify new clases.
*UniquessCheckerTest*
Performs worst case checking by adding 100s of keys that are duplicates and
verifies output of those.
**Functional tests**
Used Spark hook to verify. These sequence of commands will create shell entity
and first class entity. If resolution for first class entity is available, then
no shell entity is created.
Start Spark shell using:
```
sudo -u hdfs spark-shell
```
Spark sql commands:
```
spark.sql("create table default.t1_1381104676(col1 int)")
spark.sql("create table default.t2_1381104676(col2 int)")
spark.sql("select * from t1_1381104676, t2_1381104676 where
col1=col2").write.saveAsTable("t3_1381104676")
```
**Volume test**
Medium-size Kafka dump added.
*Setup* Add same kafka dump to multiple topics. This will result in contention
for almost every message.
**Pre-commit Build**
https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/2003/
Thanks,
Ashutosh Mestry
