----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71025/#review216479 -----------------------------------------------------------
Ship it! Ship It! - Nikhil Bonte On July 9, 2019, 4:42 p.m., Ashutosh Mestry wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/71025/ > ----------------------------------------------------------- > > (Updated July 9, 2019, 4:42 p.m.) > > > Review request for atlas, Madhan Neethiraj, Nikhil Bonte, Nixon Rodrigues, > and Sarath Subramanian. > > > Bugs: ATLAS-3320 > https://issues.apache.org/jira/browse/ATLAS-3320 > > > Repository: atlas > > > Description > ------- > > **Approach** > - Use existing producer-consumer (PC) framework. > - Modify _BulkImporterImpl_ to implement _WorkItemConsumer_. > - Add support for configuring number of workers and batch size within > _AtlasImportRequest_. > > _AtlasImportRequest_ > ``` > { > "options": { > "numWorkers": 8, > "batchSize": 25 > } > } > ``` > > **CURL** > ``` > curl -v -X POST -u admin:admin -H "Content-Type: multipart/form-data" -H > "Cache-Control: no-cache" -F request=@./import-options.json -F > data=@./Default-3-pre.zip http://localhost:21000/api/atlas/admin/import > ``` > > > Diffs > ----- > > > graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java > 499e8d1af > intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java > 0b3ede93f > intg/src/main/resources/atlas-log4j.xml 4f74c2abb > > repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java > 3ded79842 > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/BulkImporterImpl.java > 2f330c093 > > > Diff: https://reviews.apache.org/r/71025/diff/1/ > > > Testing > ------- > > **Unit tests** > Existing tests. > > **Functional tests** > - Verified import for pre-1.0 and post-1.0 exported ZIP files. > > **Volume tests** > - Measure performance with large data. > > +----------+----------+----------+------------------------+ > | File | Before | After | Configuration | > +----------+----------+----------+------------------------+ > | smalldb | 6 min | 3 min | Shards: 4, Threads: 8 | > | (2.2 MB) | | | | > +----------+----------+----------+------------------------+ > | largedb | 3 hrs | 20 mins | Shards: 4, Threads: 16 | > | (40 MB) | | | | > +----------+----------+----------+------------------------+ > > > Thanks, > > Ashutosh Mestry > >
