-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72039/
-----------------------------------------------------------
(Updated Feb. 13, 2020, 1:02 p.m.)
Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, Nixon Rodrigues,
and Sarath Subramanian.
Changes
-------
* Addressed review comments.
* Included ATLAS-3603.
Summary (updated)
-----------------
ATLAS-3595, ATLAS-3603 Migration import: support zip file for data migration
Bugs: ATLAS-3595
https://issues.apache.org/jira/browse/ATLAS-3595
Repository: atlas
Description (updated)
-------
## ATLAS-3595
Data migration service added under ATLAS-2461 works with json files as input.
This improvement will enable Data Migration service to import data using a zip
file.
Approach:
* Property value for migration should contain full path of file, below is the
example.
* Data migration service will first check migration property if value refers to
a zip file.
* If true, it will call ZipMigrationImporter to import data in zip file.
* ZipMigrationImporter internally uses existing ImportService to ingest zip.
* Audit service & Notification service will also be started as a part of zip
import.
E.g. properties to enable zip data migration:
* atlas.migration.data.filename=/path/fileName.zip
* atlas.migration.mode.workers=5
* atlas.migration.mode.batch.size=20
## ATLAS-3603
Migration import should start when the data (zip) to be ingested is available
at the pre-configured location.
Approach:
ZipMigrationImporter will first check if zip is available, accessible &
completely written.
If yes, migration will start ingesting zip.
if not, will poll the location periodically. when existence is detected, import
will proceed as usual.
It uses the file size comparison approach. Once modify event is received, file
size is compared in an interval. If sizes match, it means file is available and
not being written.
Diffs (updated)
-----
common/src/main/java/org/apache/atlas/service/Services.java 7b36db5ff
repository/src/main/java/org/apache/atlas/repository/migration/DataMigrationService.java
170136195
repository/src/main/java/org/apache/atlas/repository/migration/FileWatcher.java
PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/migration/ZipMigrationImporter.java
PRE-CREATION
repository/src/test/java/org/apache/atlas/repository/impexp/MigrationImportTest.java
PRE-CREATION
Diff: https://reviews.apache.org/r/72039/diff/7/
Changes: https://reviews.apache.org/r/72039/diff/6-7/
Testing
-------
PC build - https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/1614/
Thanks,
Nikhil Bonte