----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/72287/#review220193 -----------------------------------------------------------
Ship it! Ship It! - Madhan Neethiraj On April 2, 2020, 3:27 p.m., Ashutosh Mestry wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/72287/ > ----------------------------------------------------------- > > (Updated April 2, 2020, 3:27 p.m.) > > > Review request for atlas, Madhan Neethiraj, Nikhil Bonte, Nixon Rodrigues, > and Sarath Subramanian. > > > Bugs: ATLAS-3706 > https://issues.apache.org/jira/browse/ATLAS-3706 > > > Repository: atlas > > > Description > ------- > > **Approach** > > 1. Added Metrics to most of the methods in entity creation. (The patch does > not include the additional metrics added to additional places.) > 2. Started importing large number of entities using the > _ZipFileMigrationImporter_. > 3. Observed behavior of import over 24 hours. Observations included CPU > usage, memory usage and the import throughput using the _metric.log_. > 4. Changes were added to the one at a time. Impact of the change was observed > for performance (via metric.log) and accuracy before next change was added. > > **Observations** > * Relationship creation took inordinately large amount of time under load. > The time was spent in _GraphHelper.getAdjacentEdgesByLabel_. This > implementation also caused memory build up of _AtlasEdge_ objects which > stayed in memory for long time. This had the secondary effect of slowing down > entity creation operations after about 6 hours (this duration differed with > node configuration). > * _GraphHelper.getOrCreateEdge_ did a vertex to vertex comparison which is > time consuming. > * _GraphBackedSearchIndexer_ edge label index. Majority of edge creation > operation included lookup by edge label. > > **Configuration** > Cluster: 3 node: 40 cores, 128 GB RAM, 1.5 TB of disk space. > Atlas configuration: 32 GB RAM. > > > Diffs > ----- > > repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java > 5ab9f4d13 > > > Diff: https://reviews.apache.org/r/72287/diff/2/ > > > Testing > ------- > > **Manual tests** > (See above). > Accuracy verification. > > **Unit tests** > Executed existing unit tests. > > **Pre-commit build** > https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/1776/ > > > Thanks, > > Ashutosh Mestry > >
