Cade,
Thanks for the details. I could reproduce the issue – of missing relationship-def in atlas-typedefs.json. I field ATLAS-2984 to track fix for this issue. Fix should be out for review shortly. Thanks, Madhan From: Cade Parker <[email protected]> Reply-To: "[email protected]" <[email protected]> Date: Thursday, November 29, 2018 at 12:39 PM To: "[email protected]" <[email protected]> Subject: Re: Exporting/Importing Relationships Thanks for the reply Madhan. I think my initial assumptions about the import functionality were wrong. I assumed that the relationships weren't being rebuilt on the import and it was only creating the entities themselves. In my first attempt, I exported a type (TypeA) and then removed all instances of that entity. Once all the entities were removed, all of the relationships were deleted automatically by Atlas. Then I removed the relationshipDefs that were referencing TypeA and the TypeA entitydef itself. On my local machine, I made the backwards incompatible changes of TypeA in the same atlas-typedefs.json file that was produced by the export operation, and I edited the entity attributes in the guid.json files of the TypeA entities. I zipped the changes and ran the import operation with the new entities/typedefs, and all of the entities were recreated as expected, but the relationships for TypeA were not recreated. After reading your email, I tried again, but I recreated the relationship definitions that used TypeA with the typedefs API before importing (I did not recreate the TypeA entityDef via the typedef API, and instead let the import operation handle it), and the relationships were rebuilt as expected. Success! Finally, I tried the whole process again, but included the TypeA relationshipDefs in the atlas-typedefs.json file used in the import, and it also gave me the desired result. The process is a little tedious, but it's useful to have in case we want to migrate all entities to a new typedef without having to handle two entity types in the dependent services or keep track of which type is the latest. To make this process a little easier, I would suggest including the relationshipDefs that are used by the exported entities in the atlas-typedefs.json output file, so they are recreated automatically and it minimizes the changes needed in the typedefs file. Thanks again! Cade Parker, Civitas Learning Inc. On Thu, Nov 29, 2018 at 1:10 PM Madhan Neethiraj <[email protected]> wrote: Cade, In addition to including entity data, Atlas export operation includes details of entity-relationships, entity-classifications and all typedefs referenced by the entities. Here is the sample export command line: curl -X POST -u user:password -H "Accept: application/zip" -H "Content-Type: application/json" http://atlas-host:port/api/atlas/admin/export -d '{ "itemsToExport": [ { "typeName": "hive_db", "uniqueAttributes": { "qualifiedName": "hortoniabank@cl1" } } ], "options": { "matchType":"equals", "fetchType":"connected" }}' > export-hive_db-hortoniabank.zip Here is the exported data for ‘hortoniabank’ database entity, which includes details of its relationships with 3 tables – please see inside relationshipAttributes: { "entity": { "guid": "7bede7f3-3ad6-4051-a79a-6d97c8e42de2", "typeName": "hive_db", "attributes": { "qualifiedName": "hortoniabank@cl1", ... }, "classifications": [ { "typeName": "FINANCE", ... } ], "relationshipAttributes": { "tables": [ { "displayText": "ww_customers", "entityStatus": "ACTIVE", "guid": "247887d3-bf6b-4ae7-8ba8-cb3ed9fdbc9d", "relationshipAttributes": { "typeName": "hive_table_db" }, "relationshipGuid": "aacba4bb-9cbd-4b50-b3e0-4d27270c90a3", "relationshipStatus": "ACTIVE", "typeName": "hive_table" }, { "displayText": "us_customers", "entityStatus": "ACTIVE", "guid": "707b380b-3524-4a61-bfde-696f8002ff22", "relationshipAttributes": { "typeName": "hive_table_db" }, "relationshipGuid": "8a5c8185-1b99-4f95-9d02-7d06c303dce0", "relationshipStatus": "ACTIVE", "typeName": "hive_table" }, { "displayText": "eu_countries", "entityStatus": "ACTIVE", "guid": "1bb2aa5e-d589-4350-afa6-b5fd75a41ccc", "relationshipAttributes": { "typeName": "hive_table_db" }, "relationshipGuid": "0b7fddbd-3d0a-417d-aeb0-dd678fb62e01", "relationshipStatus": "ACTIVE", "typeName": "hive_table" } ] }, ... } } Hope this helps. Getting back to your requirement on type-system updates: Atlas doesn’t allow certain changes due to complexities, like: - addition of mandatory attribute would require populating existing entities with default value - removal of attributes might require cleaning up the index stored in Solr (or whatever the index backend is); there may not be a clean way to remove index. Such changes can take a very long time for Atlas instances having large amount of data. However, I understand this is a very critical usecase for you, hence you are taking a tedious approach of export/cleanup/import. If you can enumerate specific changes you are looking for, we can look at the feasibility of adding support for these cases. Thanks, Madhan From: Cade Parker <[email protected]> Reply-To: "[email protected]" <[email protected]> Date: Wednesday, November 28, 2018 at 3:04 PM To: "[email protected]" <[email protected]> Subject: Exporting/Importing Relationships Is it possible to export relationships along with the entities when using the export API? My use case is trying to make a backwards incompatible change to the type system by exporting all entities of the type I want to change, delete all of the entities of that type (including the relationships that use that type), modify the entities and the typedef(s) from the export locally, and import the modified entities with the import API. However, I can't seem to find an option or see any behavior that exporting/importing relationships is possible. Is the expectation that when you export from Atlas and re-import, that the relationships need to be recreated? How would a data migration from one Atlas instance to another work using the Export/Import APIs, and how does it affect the relationships of those entities? Thanks, Cade Parker, Civitas Learning Inc.
