----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/59722/ -----------------------------------------------------------
Review request for atlas and Madhan Neethiraj. Bugs: ATLAS-1851 https://issues.apache.org/jira/browse/ATLAS-1851 Repository: atlas Description ------- **Implementation** - Added additional options to _AtlasImportRequest_. - Additional options: - _startGuid_ - _startPosition_ - Added method for percentage calculation to _AtlasEntityStoreV1_. - Updated logging message to include entity guid, type and position. **CURL** Create file with these contents call it _importTransform.json_: ```javascript { "options": { "startGuid": "bd97c78e-3fa5-4f9c-9f48-3683ca3d1fb1" } } ```javascript ``` curl -g -X POST -u admin:admin -H "Content-Type: multipart/form-data" -H "Cache-Control: no-cache" -F request=@../docs/importTransform.json -F data=@../docs/Stocks-2.zip "http://localhost:21000/api/atlas/admin/import" ``` Steps to use the behavior: - Start an import (using the CURL above) that is fairly long, say about 1000+ entities. - While the import is in progress, stop atlas server (using _atlas_stop.py_). - From the log file located at _/var/log/atlas/application.log_ get the last successfully imported entity GUID or position. - Update the _importTransform.json_ with the guid. - Restart import. You should see that import resumes from where it left off. **Highlights** Specify the _startGuid_ option and notice that the operation resumes from the correct percentage and not 0%. Diffs ----- intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java 4f2c1fbc repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java 8a7e3585 repository/src/main/java/org/apache/atlas/repository/impexp/ZipSource.java 76451c98 repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStoreV1.java 27c0b5d4 repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStreamForImport.java 69140e69 repository/src/main/java/org/apache/atlas/repository/store/graph/v1/EntityImportStream.java 0f711db4 repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStoreV1BulkImportPercentTest.java PRE-CREATION Diff: https://reviews.apache.org/r/59722/diff/1/ Testing ------- **Unit tests** Added tests to cover the new functionality. Note the usage of mock for _Logger_. **Volume tests** - Performed large imports with resume. - Noted the numbers against baseline. Did not observe significant deviation. **Functional tests** - Used common scenarios from test suite. **Accuracy testing** - Not done. Thanks, Ashutosh Mestry