----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/66253/ -----------------------------------------------------------
Review request for atlas, Apoorv Naik, Madhan Neethiraj, Ruchi Solani, Sarath Subramanian, and Vishal Suvagia. Bugs: ATLAS-2460 https://issues.apache.org/jira/browse/ATLAS-2460 Repository: atlas Description ------- **Background** This implementation deals with the 'import into' part of the data migration process. It assumes: - Export from older cluster is done. - Generated file has been moved to newer cluster. **Implementation** During _Atlas_ server startup, the configuration parameter is checked, if that parameter exists, all services except _DataMigrationService_ is started. Migration is started. Atlas server is available in _MIGRATION_ mode. It processes REST calls made only to the _AdminResource_. Here's are the udpates: - New configuration parameter has been added: _atlas.migration.mode.filename=<name of the file to be imported>_ This configuration parameter is set by Ambari as part of its migration orchestration. - _DataMigrationService_: New service that performs async migration as soon as Atlas server starts up. - _MigrationProgressService_: Added. Get progress of import. - _AdminResource.getStatus()_ Now supplies additional status about migration. - _ServiceState_ Modified to carry additional status _MIGRATION_. This status is set by looking at the configuration parameter above. - _Services_ modified for special handling of _DataMigrationService_. **CURL** Check status using: ``` curl -X GET -u admin:admin -H "Content-Type: application/json" -H "Cache-Control: no-cache" http://localhost:21000/api/atlas/admin/status ``` **Migration Status** The above URL in migration mode yields JSON like: ``` {"Status":"MIGRATION","MigrationStatus":{"operationStatus":"SUCCESS","startTime":1521738357947,"endTime":1521738359272,"currentIndex":48544}} ``` Diffs ----- addons/models/0000-Area0/0010-base_model.json aebe955d common/src/main/java/org/apache/atlas/AtlasConstants.java f5de1df3 common/src/main/java/org/apache/atlas/service/Services.java 1267dc92 graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java 31d20855 graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java 6820a93c graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphDatabase.java a0060200 graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/AtlasGraphSONReader.java PRE-CREATION graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONTokensTP2.java PRE-CREATION graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtility.java PRE-CREATION graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/MappedElementCache.java PRE-CREATION graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/ReaderStatusManager.java PRE-CREATION graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/RelationshipTypeCache.java PRE-CREATION graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/Titan0Graph.java 44090097 intg/src/main/java/org/apache/atlas/model/impexp/AtlasExportResult.java 1ea961d8 intg/src/main/java/org/apache/atlas/model/impexp/MigrationStatus.java PRE-CREATION intg/src/main/java/org/apache/atlas/store/AtlasTypeDefStore.java c63dc24a pom.xml ee2746a9 repository/pom.xml b1d6b1f9 repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java 5672d9dc repository/src/main/java/org/apache/atlas/repository/impexp/MigrationProgressService.java PRE-CREATION repository/src/main/java/org/apache/atlas/repository/impexp/ZipExportFileNames.java 351b4753 repository/src/main/java/org/apache/atlas/repository/migration/DataMigrationService.java PRE-CREATION repository/src/main/java/org/apache/atlas/repository/migration/RelationshipCacheGenerator.java PRE-CREATION repository/src/main/java/org/apache/atlas/repository/ogm/DTORegistry.java 818960d9 repository/src/main/java/org/apache/atlas/repository/store/bootstrap/AtlasTypeDefStoreInitializer.java 66762001 repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStoreV1.java b5461d4b repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasTypeDefGraphStoreV1.java 1a04418a repository/src/test/java/org/apache/atlas/repository/migration/MigrationServiceTest.java PRE-CREATION repository/src/test/java/org/apache/atlas/repository/migration/RelationshipMappingTest.java PRE-CREATION repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStoreV1Test.java 8257faa1 repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasRelationshipStoreV1Test.java ac35860d repository/src/test/resources/stocks-2-0.8-extended-tag.json PRE-CREATION repository/src/test/resources/stocks-2.zip PRE-CREATION repository/src/test/resources/stocks-2/atlas-export-info.json PRE-CREATION repository/src/test/resources/stocks-2/atlas-export-order.json PRE-CREATION repository/src/test/resources/stocks-2/atlas-typesdef.json PRE-CREATION webapp/src/main/java/org/apache/atlas/web/filters/ActiveServerFilter.java 6681a372 webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 1b3f2c86 webapp/src/main/java/org/apache/atlas/web/security/AtlasSecurityConfig.java f1760e7f webapp/src/main/java/org/apache/atlas/web/service/ServiceState.java 3fe8d18c webapp/src/test/java/org/apache/atlas/web/resources/AdminResourceTest.java aab2bb8f Diff: https://reviews.apache.org/r/66253/diff/1/ Testing ------- **Unit tests** Unit tests for _AtlasGraphSONReader_ added. **Functional tests** Steps to test file-based import: - Place the exported file say _/root/atlas-data_ - Add to _Atlas_ Ambari's customer property: _atlas.migration.mode.filename=/root/atlas-data_ - Ambari will prompt for a restart. Restart Atlas. - On the server view the progress in the logs using: _tail -f /var/log/atlas/application.log_ - Use the CURL call mentioned above and view the status and the progress of the import. Steps to test directory-based import: - Place the exported files say _/root/atlas-data_ - Add to _Atlas_ Ambari's customer property: _atlas.migration.mode.filename=/root/atlas-data_ - Ambari will prompt for a restart. Restart Atlas. - On the server view the progress in the logs using: _tail -f /var/log/atlas/application.log_ - Use the CURL call mentioned above and view the status and the progress of the import. Thanks, Ashutosh Mestry