----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71426/#review217553 -----------------------------------------------------------
docs/src/site/twiki/Import-API-Options.twiki Lines 149 (patched) <https://reviews.apache.org/r/71426/#comment304815> - please consider avoiding the client to specify a directory to write at the server side. The location must be configured at the server-side. - if use of 'backingDirectory' benefits all imports, can this approach be used for all imports, instead of having the client to request for it? In such case, consider creating a directory, perhaps named as a generated guid, under the configured location - to avoid potential reuse contents across multiple requests. Also, care must be taken to delete the contents after the import is completed - successfully or unsuccessfully. - Madhan Neethiraj On Sept. 3, 2019, 5:24 p.m., Ashutosh Mestry wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/71426/ > ----------------------------------------------------------- > > (Updated Sept. 3, 2019, 5:24 p.m.) > > > Review request for atlas, Madhan Neethiraj, Nikhil Bonte, Nixon Rodrigues, > and Sarath Subramanian. > > > Bugs: ATLAS-3396 > https://issues.apache.org/jira/browse/ATLAS-3396 > > > Repository: atlas > > > Description > ------- > > **Background** > The approach adds another option to _ImportService_ to be able to optionally > use a directory on the server for storing data to be imported. > > **Approach** > This is a backward compatible. > > - - New: _ZipSourceWithBackingDirectory_: Uses a temporary directory for > storing contents of the zip file. > - Modified: _ImportService_ Now takes _InputStream_ as parameter instead of > _ZipSource_. > - Modified: _ImportRequest_ Additionl option takes _backingDirectory_ as > additional option. Specifying this will use the new > _ZipSourceWithBackingDirectory_ during import. > > > **CURL** > > Use pre-configured temporary directory: > ``` > { > "options": { > "backingDirectory": "/grid/0/temp" > } > } > ``` > > To use default temp directory: > > ``` > { > "options": { > "backingDirectory": "" > } > } > > `` > > _CURL_ > ``` > curl -v -X POST -u admin:admin -H "Content-Type: multipart/form-data" -H > "Cache-Control: no-cache" -F request=@../docs/import-options.json -F > data=@../docs/smalldb.zip http://localhost:21000/api/atlas/admin/import > ``` > > **Documentation** > Updated. > > > Diffs > ----- > > docs/src/site/twiki/Import-API-Options.twiki 4004e7013 > intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java > 6a9985642 > > repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java > b5d8b7c39 > > repository/src/main/java/org/apache/atlas/repository/impexp/ZipSourceWithBackingDirectory.java > PRE-CREATION > > repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStreamForImport.java > 90ae15d1e > > repository/src/main/java/org/apache/atlas/repository/store/graph/v1/EntityImportStream.java > d4b6c5505 > > repository/src/test/java/org/apache/atlas/repository/impexp/ExportIncrementalTest.java > a355297a8 > > repository/src/test/java/org/apache/atlas/repository/impexp/ExportSkipLineageTest.java > eaf4602d5 > > repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java > 693a163f8 > > repository/src/test/java/org/apache/atlas/repository/impexp/ReplicationEntityAttributeTest.java > 868b732d3 > > repository/src/test/java/org/apache/atlas/repository/impexp/ZipFileResourceTestUtils.java > a2a5f58dc > webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java > 8417e7eb0 > > > Diff: https://reviews.apache.org/r/71426/diff/2/ > > > Testing > ------- > > **Unit tests** > Updated to exercise the new implementation. > > **Volume tests** > Imported 4GB ZIP file that took 26 hours. Memory and CPU stayed in normal > range during this operation. > > **Functional tests** > Accuracy tests preformed on small size data. > > **Pre-commit Build** > https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/1379/console > > > Thanks, > > Ashutosh Mestry > >
