-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72666/
-----------------------------------------------------------
(Updated July 29, 2020, 6:23 p.m.)
Review request for atlas, Madhan Neethiraj, Nikhil Bonte, Nixon Rodrigues, and
Sarath Subramanian.
Changes
-------
Updates include: Addressed review comments.
Bugs: ATLAS-3878
https://issues.apache.org/jira/browse/ATLAS-3878
Repository: atlas
Description
-------
**Background**
See JIRA for details.
*Analysis* Using memory profiling tools, it was observed that large number of
notification objects were created. These stayed in memory and later were
promoted to higher generation, thereby taking even longer to be collected.
**Approach**
Using the fixed-buffer approach to address the problem of creating large number
of small objects.
New *FixedBufferList* This is an encapsulation over *ArrayList*. During initial
allocation, list is populated with default values. Features:
- Setting of values to these pre-allocated objects is achieved by first doing a
*get* on the element and then assigning values to it.
- *toList* fetches the sub-list from the encapsulating list. This uses the
state within the class to fetch the right length for the returning array.
New *NamedFixedBufferList* Maintains a per-thread *FixedBufferList*. This is
necessary since the list is now part class's state.
Modified *EntityAuditListenerV2* Uses the new classes.
Modifed *EntityNotificationListener* Uses the new classes.
**Verification**
- Using the test setup, the memory usage was observed over a period of 24 hrs.
- Memory usage and object allocation was obvserved using memory profiler.
Diffs (updated)
-----
intg/src/main/java/org/apache/atlas/AtlasConfiguration.java 2c007ca01
intg/src/main/java/org/apache/atlas/model/Clearable.java PRE-CREATION
intg/src/main/java/org/apache/atlas/model/audit/EntityAuditEventV2.java
63116d467
intg/src/main/java/org/apache/atlas/utils/FixedBufferList.java PRE-CREATION
intg/src/test/java/org/apache/atlas/utils/FixedBufferListTest.java
PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditListenerV2.java
79527acfa
webapp/src/main/java/org/apache/atlas/notification/EntityNotificationSender.java
c2ae5122b
Diff: https://reviews.apache.org/r/72666/diff/9/
Changes: https://reviews.apache.org/r/72666/diff/8-9/
Testing
-------
**Unit testing**
Unit tests added for the new classes.
**Volume testing**
Setup:
- Node: Threads 40, Core: 40, Allocated Memory: 12 GB
- Multiple Kafka queues ingesting data.
- Bulk entity creation using custom script ingesting 100M entities.
Memory usage stayed between 0 and 5% during the 24 hr period.
With:
- Workers: 64
- Batch size: 50 (fewer elements in batch improve commit time and audit write
time).
- Throughput: ~1.2 M entities per hour. Without out of memory error.
**Pre-commit**
https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/2035/
https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/2067/
Thanks,
Ashutosh Mestry
