-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72039/#review219586
-----------------------------------------------------------


Ship it!




Ship It!

- Ashutosh Mestry


On Feb. 13, 2020, 1:02 p.m., Nikhil Bonte wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72039/
> -----------------------------------------------------------
> 
> (Updated Feb. 13, 2020, 1:02 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, Nixon Rodrigues, 
> and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3595
>     https://issues.apache.org/jira/browse/ATLAS-3595
> 
> 
> Repository: atlas
> 
> 
> Description
> -------
> 
> ## ATLAS-3595
> Data migration service added under ATLAS-2461 works with json files as input.
> 
> This improvement will enable Data Migration service to import data using a 
> zip file.
> 
> Approach:
> 
> * Property value for migration should contain full path of file, below is the 
> example. 
> * Data migration service will first check migration property if value refers 
> to a zip file.
> * If true, it will call ZipMigrationImporter to import data in zip file.
> * ZipMigrationImporter internally uses existing ImportService to ingest zip.
> * Audit service & Notification service will also be started as a part of zip 
> import.
> 
> 
> E.g. properties to enable zip data migration:
> 
> * atlas.migration.data.filename=/path/fileName.zip
> * atlas.migration.mode.workers=5
> * atlas.migration.mode.batch.size=20
> 
> 
> ## ATLAS-3603
> Migration import should start when the data (zip) to be ingested is available 
> at the pre-configured location.
> 
> Approach:
> 
> ZipMigrationImporter will first check if zip is available, accessible & 
> completely written.
> 
> If yes, migration will start ingesting zip.
> if not, will poll the location periodically. when existence is detected, 
> import will proceed as usual.
> 
> It uses the file size comparison approach. Once modify event is received, 
> file size is compared in an interval. If sizes match, it means file is 
> available and not being written.
> 
> 
> Diffs
> -----
> 
>   common/src/main/java/org/apache/atlas/service/Services.java 7b36db5ff 
>   
> repository/src/main/java/org/apache/atlas/repository/migration/DataMigrationService.java
>  170136195 
>   
> repository/src/main/java/org/apache/atlas/repository/migration/FileWatcher.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/migration/ZipMigrationImporter.java
>  PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/72039/diff/8/
> 
> 
> Testing
> -------
> 
> PC build - https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/1632/
> 
> 
> Thanks,
> 
> Nikhil Bonte
> 
>

Reply via email to