----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/72039/#review219586 -----------------------------------------------------------
Ship it! Ship It! - Ashutosh Mestry On Feb. 13, 2020, 1:02 p.m., Nikhil Bonte wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/72039/ > ----------------------------------------------------------- > > (Updated Feb. 13, 2020, 1:02 p.m.) > > > Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, Nixon Rodrigues, > and Sarath Subramanian. > > > Bugs: ATLAS-3595 > https://issues.apache.org/jira/browse/ATLAS-3595 > > > Repository: atlas > > > Description > ------- > > ## ATLAS-3595 > Data migration service added under ATLAS-2461 works with json files as input. > > This improvement will enable Data Migration service to import data using a > zip file. > > Approach: > > * Property value for migration should contain full path of file, below is the > example. > * Data migration service will first check migration property if value refers > to a zip file. > * If true, it will call ZipMigrationImporter to import data in zip file. > * ZipMigrationImporter internally uses existing ImportService to ingest zip. > * Audit service & Notification service will also be started as a part of zip > import. > > > E.g. properties to enable zip data migration: > > * atlas.migration.data.filename=/path/fileName.zip > * atlas.migration.mode.workers=5 > * atlas.migration.mode.batch.size=20 > > > ## ATLAS-3603 > Migration import should start when the data (zip) to be ingested is available > at the pre-configured location. > > Approach: > > ZipMigrationImporter will first check if zip is available, accessible & > completely written. > > If yes, migration will start ingesting zip. > if not, will poll the location periodically. when existence is detected, > import will proceed as usual. > > It uses the file size comparison approach. Once modify event is received, > file size is compared in an interval. If sizes match, it means file is > available and not being written. > > > Diffs > ----- > > common/src/main/java/org/apache/atlas/service/Services.java 7b36db5ff > > repository/src/main/java/org/apache/atlas/repository/migration/DataMigrationService.java > 170136195 > > repository/src/main/java/org/apache/atlas/repository/migration/FileWatcher.java > PRE-CREATION > > repository/src/main/java/org/apache/atlas/repository/migration/ZipMigrationImporter.java > PRE-CREATION > > > Diff: https://reviews.apache.org/r/72039/diff/8/ > > > Testing > ------- > > PC build - https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/1632/ > > > Thanks, > > Nikhil Bonte > >
