----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/72384/ -----------------------------------------------------------
(Updated April 24, 2020, 8:10 p.m.) Review request for atlas, Madhan Neethiraj, Nikhil Bonte, Nixon Rodrigues, and Sarath Subramanian. Changes ------- Updates include: - Simplified edge creation approach. Bugs: ATLAS-3737 https://issues.apache.org/jira/browse/ATLAS-3737 Repository: atlas Description ------- **Approach** Please refer to JIRA for details. **Benefits with this Approach** - Memory usage remains constant. - Cores, based on numWorkers, are utilized. **Changes to Migration Import** - Instantiates 2 _EntityStoreV2_, one with non-bulk _AtlasGraph_ and other with bulk loading _AtlasGraph_. - For entities that causes schema change (handled by _IllegalStateException_ or _IllegalArgumentException_), the same entity is attempted with non-bulk instantiated _EntityStoreV2_. Diffs (updated) ----- graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java 0176ba705 graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/query/NativeJanusGraphQuery.java 24859389b repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java 25042c19c repository/src/main/java/org/apache/atlas/discovery/EntityLineageService.java 5548740d0 repository/src/main/java/org/apache/atlas/repository/converters/AtlasInstanceConverter.java 172e187e4 repository/src/main/java/org/apache/atlas/repository/graph/FullTextMapperV2.java 417c96cd8 repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java bff54ede3 repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java dfaa739e8 repository/src/main/java/org/apache/atlas/repository/impexp/ExportService.java 9c57d54de repository/src/main/java/org/apache/atlas/repository/store/graph/v1/DeleteHandlerDelegate.java 512eb4636 repository/src/main/java/org/apache/atlas/repository/store/graph/v1/DeleteHandlerV1.java 540defca9 repository/src/main/java/org/apache/atlas/repository/store/graph/v1/HardDeleteHandlerV1.java a32b67124 repository/src/main/java/org/apache/atlas/repository/store/graph/v1/SoftDeleteHandlerV1.java e614a4e41 repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityGraphDiscoveryV2.java d3d9367fb repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2.java 379150b7b repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasGraphUtilsV2.java f393e515b repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasRelationshipStoreV2.java 7425ac601 repository/src/main/java/org/apache/atlas/repository/store/graph/v2/BulkImporterImpl.java a4d732af0 repository/src/main/java/org/apache/atlas/repository/store/graph/v2/ClassificationAssociator.java 05878f405 repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java 04da41e1a repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java 36bee301d repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityStateChecker.java 2718a0627 repository/src/main/java/org/apache/atlas/repository/store/graph/v2/IDBasedEntityResolver.java 3b9694851 repository/src/main/java/org/apache/atlas/repository/store/graph/v2/UniqAttrBasedEntityResolver.java 14ddc5d7a repository/src/main/java/org/apache/atlas/repository/store/graph/v2/bulkimport/MigrationImport.java ff55e40eb repository/src/main/java/org/apache/atlas/repository/store/graph/v2/bulkimport/RegularImport.java ecce1b0e7 repository/src/main/java/org/apache/atlas/repository/store/graph/v2/bulkimport/pc/EntityConsumer.java e0a457334 repository/src/main/java/org/apache/atlas/repository/store/graph/v2/bulkimport/pc/EntityConsumerBuilder.java 69d33b21e repository/src/test/java/org/apache/atlas/repository/impexp/ExportSkipLineageTest.java f8b7eda71 repository/src/test/java/org/apache/atlas/repository/migration/HiveParititionTest.java 7567daa8f repository/src/test/java/org/apache/atlas/repository/migration/HiveStocksTest.java 90e714cb5 repository/src/test/java/org/apache/atlas/repository/migration/MigrationBaseAsserts.java ce9056b54 repository/src/test/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2Test.java 38228a8ec repository/src/test/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityTestBase.java 368951dfc repository/src/test/java/org/apache/atlas/repository/store/graph/v2/AtlasRelationshipStoreV2Test.java a40cf85a1 repository/src/test/java/org/apache/atlas/repository/store/graph/v2/ClassificationAssociatorTest.java 37acbe5de Diff: https://reviews.apache.org/r/72384/diff/8/ Changes: https://reviews.apache.org/r/72384/diff/7-8/ Testing (updated) ------- **Pre-commit** https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/1830/ https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/1847/ https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/1850/ **Volume testing** Verified with large volume of data. Import parameters: numWorkers: 36 batchSize: 750 Heap: 16 Solr Heap: 60 Hbase memory tuning. +--------------------------+---------------------+-----------------+ | Number of entities | | | | (in millions) | Before | After | +--------------------------+---------------------+-----------------+ | 6.21 | 5 hrs 26 m | 4 hrs 12 mins | +--------------------------+---------------------+-----------------+ | 8.5 | 6 hrs 15 m | 4 hr 40 mins | +--------------------------+---------------------+-----------------+ Thanks, Ashutosh Mestry