----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71025/ -----------------------------------------------------------
(Updated July 9, 2019, 4:42 p.m.) Review request for atlas, Madhan Neethiraj, Nikhil Bonte, Nixon Rodrigues, and Sarath Subramanian. Changes ------- Updates include: - Added CURL call and request json. Bugs: ATLAS-3320 https://issues.apache.org/jira/browse/ATLAS-3320 Repository: atlas Description (updated) ------- **Approach** - Use existing producer-consumer (PC) framework. - Modify _BulkImporterImpl_ to implement _WorkItemConsumer_. - Add support for configuring number of workers and batch size within _AtlasImportRequest_. _AtlasImportRequest_ ``` { "options": { "numWorkers": 8, "batchSize": 25 } } ``` **CURL** ``` curl -v -X POST -u admin:admin -H "Content-Type: multipart/form-data" -H "Cache-Control: no-cache" -F request=@./import-options.json -F data=@./Default-3-pre.zip http://localhost:21000/api/atlas/admin/import ``` Diffs ----- graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java 499e8d1af intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java 0b3ede93f intg/src/main/resources/atlas-log4j.xml 4f74c2abb repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java 3ded79842 repository/src/main/java/org/apache/atlas/repository/store/graph/v2/BulkImporterImpl.java 2f330c093 Diff: https://reviews.apache.org/r/71025/diff/1/ Testing ------- **Unit tests** Existing tests. **Functional tests** - Verified import for pre-1.0 and post-1.0 exported ZIP files. **Volume tests** - Measure performance with large data. +----------+----------+----------+------------------------+ | File | Before | After | Configuration | +----------+----------+----------+------------------------+ | smalldb | 6 min | 3 min | Shards: 4, Threads: 8 | | (2.2 MB) | | | | +----------+----------+----------+------------------------+ | largedb | 3 hrs | 20 mins | Shards: 4, Threads: 16 | | (40 MB) | | | | +----------+----------+----------+------------------------+ Thanks, Ashutosh Mestry