----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71426/#review217664 -----------------------------------------------------------
distro/src/conf/atlas-application.properties Lines 69 (patched) <https://reviews.apache.org/r/71426/#comment304958> atlas.import.temporaryDirectory => atlas.import.temp.directory - Madhan Neethiraj On Sept. 9, 2019, 7:16 p.m., Ashutosh Mestry wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/71426/ > ----------------------------------------------------------- > > (Updated Sept. 9, 2019, 7:16 p.m.) > > > Review request for atlas, Madhan Neethiraj, Nikhil Bonte, Nixon Rodrigues, > and Sarath Subramanian. > > > Bugs: ATLAS-3396 > https://issues.apache.org/jira/browse/ATLAS-3396 > > > Repository: atlas > > > Description > ------- > > **Background** > The approach adds another option to _ImportService_ to be able to optionally > use a directory on the server for storing data to be imported. > > **Approach** > This is a backward compatible. > > - - New: _ZipSourceWithBackingDirectory_: Uses a temporary directory for > storing contents of the zip file. > - Modified: _ImportService_ Now takes _InputStream_ as parameter instead of > _ZipSource_. > - Modified: _ImportRequest_ Additionl option takes _backingDirectory_ as > additional option. Specifying this will use the new > _ZipSourceWithBackingDirectory_ during import. > > > **CURL** > > Use pre-configured temporary directory, set _atlas.import.temporaryDirectory_ > in the applicatino properties. > > _CURL_ > ``` > curl -v -X POST -u admin:admin -H "Content-Type: multipart/form-data" -H > "Cache-Control: no-cache" -F request=@../docs/import-options.json -F > data=@../docs/smalldb.zip http://localhost:21000/api/atlas/admin/import > ``` > > **Documentation** > Updated. > > > Diffs > ----- > > common/src/main/java/org/apache/atlas/AtlasConfiguration.java c5357f5a4 > distro/src/conf/atlas-application.properties 654a9304e > docs/src/site/twiki/Import-API-Options.twiki 4004e7013 > > repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java > b5d8b7c39 > repository/src/main/java/org/apache/atlas/repository/impexp/ZipSource.java > 016acd743 > > repository/src/main/java/org/apache/atlas/repository/impexp/ZipSourceWithBackingDirectory.java > PRE-CREATION > > repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStreamForImport.java > 90ae15d1e > > repository/src/main/java/org/apache/atlas/repository/store/graph/v1/BulkImporterImpl.java > 26066920a > > repository/src/main/java/org/apache/atlas/repository/store/graph/v1/EntityImportStream.java > d4b6c5505 > > repository/src/test/java/org/apache/atlas/repository/impexp/ExportImportTestBase.java > a11bcdcb4 > > repository/src/test/java/org/apache/atlas/repository/impexp/ExportIncrementalTest.java > 5ec62e5f9 > > repository/src/test/java/org/apache/atlas/repository/impexp/ExportSkipLineageTest.java > eaf4602d5 > > repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java > 693a163f8 > > repository/src/test/java/org/apache/atlas/repository/impexp/ImportTransformsShaperTest.java > 06bdaa6d6 > > repository/src/test/java/org/apache/atlas/repository/impexp/ReplicationEntityAttributeTest.java > 5e909e198 > > repository/src/test/java/org/apache/atlas/repository/impexp/ZipFileResourceTestUtils.java > a2a5f58dc > webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java > 8417e7eb0 > > > Diff: https://reviews.apache.org/r/71426/diff/8/ > > > Testing > ------- > > **Unit tests** > Updated to exercise the new implementation. > > **Volume tests** > Imported 4GB ZIP file that took 26 hours. Memory and CPU stayed in normal > range during this operation. > > **Functional tests** > Accuracy tests preformed on small size data. > > **Pre-commit Build** > https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/1380/ > > > Thanks, > > Ashutosh Mestry > >
