----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/59722/#review176849 -----------------------------------------------------------
intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java Lines 76 (patched) <https://reviews.apache.org/r/59722/#comment250321> Consider adding @JsonIgnore annotation here, so that 'startGuid' and 'startPosition' will not be serialized. repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java Lines 93 (patched) <https://reviews.apache.org/r/59722/#comment250324> It might be clearer if this method only sets the start-position; and have the caller call processEntities(). Consider the following: Replace: processEntitiesUsingStartOption(request, source, result); With: setStartPosition(source, request); processEntities(source, result); repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStoreV1.java Lines 208 (patched) <https://reviews.apache.org/r/59722/#comment250326> Why 'currentIndex + 1'? ZipSource.currentIndex seems to have the number of entities processed so far. Please review. Also 'ZipSource.currentPosition' might be a better name, instead of currentIndex. Please review. repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStreamForImport.java Lines 48 (patched) <https://reviews.apache.org/r/59722/#comment250328> Wouldn't size be always "1"? Given this object is initialized with an instance of AtlasEntityWithExtInfo? repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStreamForImport.java Lines 53 (patched) <https://reviews.apache.org/r/59722/#comment250330> Consider adding a comment like: // not applicable for a single entity stream repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStreamForImport.java Lines 58 (patched) <https://reviews.apache.org/r/59722/#comment250329> It will be useful for getNextEntityWithExtInfo() to maintain a counter, which can be returned from getPosition() - similar to the implementation in ZipSource. - Madhan Neethiraj On June 2, 2017, 8:52 p.m., Ashutosh Mestry wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/59722/ > ----------------------------------------------------------- > > (Updated June 2, 2017, 8:52 p.m.) > > > Review request for atlas and Madhan Neethiraj. > > > Bugs: ATLAS-1851 > https://issues.apache.org/jira/browse/ATLAS-1851 > > > Repository: atlas > > > Description > ------- > > **Implementation** > - Added additional options to _AtlasImportRequest_. > - Additional options: > - _startGuid_ > - _startPosition_ > - Added method for percentage calculation to _AtlasEntityStoreV1_. > - Updated logging message to include entity guid, type and position. > > **CURL** > > Create file with these contents call it _importTransform.json_: > ```javascript > { "options": { > "startGuid": "bd97c78e-3fa5-4f9c-9f48-3683ca3d1fb1" > } > } > ```javascript > > > ``` > curl -g -X POST -u admin:admin -H "Content-Type: multipart/form-data" -H > "Cache-Control: no-cache" -F request=@../docs/importTransform.json -F > data=@../docs/Stocks-2.zip "http://localhost:21000/api/atlas/admin/import"; > ``` > > Steps to use the behavior: > - Start an import (using the CURL above) that is fairly long, say about 1000+ > entities. > - While the import is in progress, stop atlas server (using _atlas_stop.py_). > - From the log file located at _/var/log/atlas/application.log_ get the last > successfully imported entity GUID or position. > - Update the _importTransform.json_ with the guid. > - Restart import. > > You should see that import resumes from where it left off. > > > **Highlights** > Specify the _startGuid_ option and notice that the operation resumes from the > correct percentage and not 0%. > > > Diffs > ----- > > intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java > 4f2c1fbc > > repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java > 8a7e3585 > repository/src/main/java/org/apache/atlas/repository/impexp/ZipSource.java > 76451c98 > > repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStoreV1.java > 27c0b5d4 > > repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStreamForImport.java > 69140e69 > > repository/src/main/java/org/apache/atlas/repository/store/graph/v1/EntityImportStream.java > 0f711db4 > > repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStoreV1BulkImportPercentTest.java > PRE-CREATION > > > Diff: https://reviews.apache.org/r/59722/diff/1/ > > > Testing > ------- > > **Unit tests** > Added tests to cover the new functionality. Note the usage of mock for > _Logger_. > > **Volume tests** > - Performed large imports with resume. > - Noted the numbers against baseline. Did not observe significant deviation. > > **Functional tests** > - Used common scenarios from test suite. > > **Accuracy testing** > - Not done. > > > Thanks, > > Ashutosh Mestry > >
