----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71426/ -----------------------------------------------------------
(Updated Sept. 9, 2019, 5:35 a.m.) Review request for atlas, Madhan Neethiraj, Nikhil Bonte, Nixon Rodrigues, and Sarath Subramanian. Changes ------- Updates include: - Addressed review comments. - Fixed performance problem that was causing slow ingest for entities with large lineage. - Refactoring for reducing memory usage. Bugs: ATLAS-3396 https://issues.apache.org/jira/browse/ATLAS-3396 Repository: atlas Description ------- **Background** The approach adds another option to _ImportService_ to be able to optionally use a directory on the server for storing data to be imported. **Approach** This is a backward compatible. - - New: _ZipSourceWithBackingDirectory_: Uses a temporary directory for storing contents of the zip file. - Modified: _ImportService_ Now takes _InputStream_ as parameter instead of _ZipSource_. - Modified: _ImportRequest_ Additionl option takes _backingDirectory_ as additional option. Specifying this will use the new _ZipSourceWithBackingDirectory_ during import. **CURL** Use pre-configured temporary directory, set _atlas.import.temporaryDirectory_ in the applicatino properties. _CURL_ ``` curl -v -X POST -u admin:admin -H "Content-Type: multipart/form-data" -H "Cache-Control: no-cache" -F request=@../docs/import-options.json -F data=@../docs/smalldb.zip http://localhost:21000/api/atlas/admin/import ``` **Documentation** Updated. Diffs (updated) ----- distro/src/conf/atlas-application.properties 654a9304e docs/src/site/twiki/Import-API-Options.twiki 4004e7013 repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java b5d8b7c39 repository/src/main/java/org/apache/atlas/repository/impexp/ZipSource.java 016acd743 repository/src/main/java/org/apache/atlas/repository/impexp/ZipSourceWithBackingDirectory.java PRE-CREATION repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStreamForImport.java 90ae15d1e repository/src/main/java/org/apache/atlas/repository/store/graph/v1/BulkImporterImpl.java 26066920a repository/src/main/java/org/apache/atlas/repository/store/graph/v1/EntityImportStream.java d4b6c5505 repository/src/test/java/org/apache/atlas/repository/impexp/ExportIncrementalTest.java a355297a8 repository/src/test/java/org/apache/atlas/repository/impexp/ExportSkipLineageTest.java eaf4602d5 repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java 693a163f8 repository/src/test/java/org/apache/atlas/repository/impexp/ImportTransformsShaperTest.java 06bdaa6d6 repository/src/test/java/org/apache/atlas/repository/impexp/ReplicationEntityAttributeTest.java 868b732d3 repository/src/test/java/org/apache/atlas/repository/impexp/ZipFileResourceTestUtils.java a2a5f58dc webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 8417e7eb0 Diff: https://reviews.apache.org/r/71426/diff/6/ Changes: https://reviews.apache.org/r/71426/diff/5-6/ Testing ------- **Unit tests** Updated to exercise the new implementation. **Volume tests** Imported 4GB ZIP file that took 26 hours. Memory and CPU stayed in normal range during this operation. **Functional tests** Accuracy tests preformed on small size data. **Pre-commit Build** https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/1380/ Thanks, Ashutosh Mestry