> On April 7, 2020, 5:42 p.m., Madhan Neethiraj wrote: > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java > > Lines 1068 (patched) > > <https://reviews.apache.org/r/72330/diff/1/?file=2217104#file2217104line1068> > > > > Not populating uniqPropName will cause UniqAttrBasedEntityResolver to > > not find the entity - as it uses > > AtlasGraphUtilsV2.findByUniqueAttributes(), which in turn looks for entity > > using uniqPropName. This could result in multiple entity instances be > > created with the same uniqAttribute value during migration-import. > > > > To improve throughput during migration, consider enabling bulk-load > > option; refer to https://docs.janusgraph.org/advanced-topics/bulk-loading/.
Bulkloading is already being used in migration. - Ashutosh ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/72330/#review220242 ----------------------------------------------------------- On April 7, 2020, 4:30 p.m., Ashutosh Mestry wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/72330/ > ----------------------------------------------------------- > > (Updated April 7, 2020, 4:30 p.m.) > > > Review request for atlas, Madhan Neethiraj, Nikhil Bonte, and Nixon Rodrigues. > > > Bugs: ATLAS-3715 > https://issues.apache.org/jira/browse/ATLAS-3715 > > > Repository: atlas > > > Description > ------- > > **Approach** > - Create a new flag within _RequestContext_ that indicates if migration > import is in progress. > - Refactored _mapPrimitiveAttribute_ to not create > _AtlasAttributeMutationContext_ for primitive attributes. The new method is > used in entity and relationship mapping. > - Within _mapPrimitiveAttribute_ skip setting the uniqueProperty name. > > **Reason** > - The change does not set the _uniquePropertyKey_. The act of doing this > causes throughput to be improved by ~75%. > - This attribute would be handled post-upgrade of this attribute via the > _UniqueAttributePatch_. > > > Diffs > ----- > > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java > 0d5b6314d > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/bulkimport/pc/EntityConsumer.java > e0a457334 > server-api/src/main/java/org/apache/atlas/RequestContext.java 282a66f1d > > > Diff: https://reviews.apache.org/r/72330/diff/1/ > > > Testing > ------- > > **Volume test** > Enable metric log. > Baseline: 25 secs. > New: 2 secs. > ``` > atlas.migration.mode.batch.size=700 > atlas.migration.mode.workers=24 > ```` > > > Thanks, > > Ashutosh Mestry > >
