----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/72666/ -----------------------------------------------------------
(Updated July 30, 2020, 5:55 p.m.) Review request for atlas, Madhan Neethiraj, Nikhil Bonte, Nixon Rodrigues, and Sarath Subramanian. Changes ------- Updates include: Addressed review comments. Bugs: ATLAS-3878 https://issues.apache.org/jira/browse/ATLAS-3878 Repository: atlas Description ------- **Background** See JIRA for details. *Analysis* Using memory profiling tools, it was observed that large number of notification objects were created. These stayed in memory and later were promoted to higher generation, thereby taking even longer to be collected. **Approach** Using the fixed-buffer approach to address the problem of creating large number of small objects. New *FixedBufferList* This is an encapsulation over *ArrayList*. During initial allocation, list is populated with default values. Features: - Setting of values to these pre-allocated objects is achieved by first doing a *get* on the element and then assigning values to it. - *toList* fetches the sub-list from the encapsulating list. This uses the state within the class to fetch the right length for the returning array. New *NamedFixedBufferList* Maintains a per-thread *FixedBufferList*. This is necessary since the list is now part class's state. Modified *EntityAuditListenerV2* Uses the new classes. Modifed *EntityNotificationListener* Uses the new classes. **Verification** - Using the test setup, the memory usage was observed over a period of 24 hrs. - Memory usage and object allocation was obvserved using memory profiler. Diffs (updated) ----- intg/src/main/java/org/apache/atlas/AtlasConfiguration.java 2c007ca01 intg/src/main/java/org/apache/atlas/model/Clearable.java PRE-CREATION intg/src/main/java/org/apache/atlas/model/audit/EntityAuditEventV2.java 63116d467 intg/src/main/java/org/apache/atlas/utils/FixedBufferList.java PRE-CREATION intg/src/test/java/org/apache/atlas/utils/FixedBufferListTest.java PRE-CREATION repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditListenerV2.java 79527acfa webapp/src/main/java/org/apache/atlas/notification/EntityNotificationSender.java c2ae5122b Diff: https://reviews.apache.org/r/72666/diff/10/ Changes: https://reviews.apache.org/r/72666/diff/9-10/ Testing ------- **Unit testing** Unit tests added for the new classes. **Volume testing** Setup: - Node: Threads 40, Core: 40, Allocated Memory: 12 GB - Multiple Kafka queues ingesting data. - Bulk entity creation using custom script ingesting 100M entities. Memory usage stayed between 0 and 5% during the 24 hr period. With: - Workers: 64 - Batch size: 50 (fewer elements in batch improve commit time and audit write time). - Throughput: ~1.2 M entities per hour. Without out of memory error. **Pre-commit** https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/2035/ https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/2067/ Thanks, Ashutosh Mestry