Review Request 67640: Migration Exporter: Removed Usage of Spring to Help Support Earlier Versions
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/67640/ --- Review request for atlas, Apoorv Naik, Madhan Neethiraj, Ruchi Solani, and Sarath Subramanian. Bugs: ATLAS-2761 https://issues.apache.org/jira/browse/ATLAS-2761 Repository: atlas Description --- **Background** Spring got integrated around mid-May of 2017 within Atlas code base. Existing implementation of _Migration Exporter_ uses Spring for instantiating objects. While this is a clean approach, it prevents versions prior to May 2017 from using _Migration Exporter_. **Approach** - Removed use of Spring. - Objects are directly instantiated. - Constructor variations have been identified and care has been taken to support that using Reflection. Diffs - tools/atlas-migration-exporter/pom.xml 8ecd306ff tools/atlas-migration-exporter/src/main/java/org/apache/atlas/migration/Exporter.java 1bda19fb4 tools/atlas-migration-exporter/src/main/resources/migrationContext.xml c0f96598c Diff: https://reviews.apache.org/r/67640/diff/1/ Testing --- **Unit tests** None. **Versions verified** Versions prior to branch-0.8 creation have been verified. Thanks, Ashutosh Mestry
[jira] [Updated] (ATLAS-2761) Migration Exporter: Existing Implementation Does not Support Export from Early Atlas 0.8 Versions
[ https://issues.apache.org/jira/browse/ATLAS-2761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Mestry updated ATLAS-2761: --- Attachment: ATLAS-2761-Removed-dependency-on-Spring-to-support-m.patch > Migration Exporter: Existing Implementation Does not Support Export from > Early Atlas 0.8 Versions > - > > Key: ATLAS-2761 > URL: https://issues.apache.org/jira/browse/ATLAS-2761 > Project: Atlas > Issue Type: Bug >Reporter: Ashutosh Mestry >Assignee: Ashutosh Mestry >Priority: Major > Fix For: 1.0.0, trunk > > Attachments: > ATLAS-2761-Removed-dependency-on-Spring-to-support-m.patch > > > *Background* > Current implementation of _Migration Exporter_ uses Spring to instantiate > objects. This works for most of Atlas v0.8 versions, except the early > versions that did not have Spring integrated. > *Analysis* > The use of Spring in object creation imposes this restriction. Simpler > approach by directly instantiating objects can help in supporting more > versions. > There may be some inconsistencies in parameters passed during object > creation. This needs to be considered. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (ATLAS-2761) Migration Exporter: Existing Implementation Does not Support Export from Early Atlas 0.8 Versions
Ashutosh Mestry created ATLAS-2761: -- Summary: Migration Exporter: Existing Implementation Does not Support Export from Early Atlas 0.8 Versions Key: ATLAS-2761 URL: https://issues.apache.org/jira/browse/ATLAS-2761 Project: Atlas Issue Type: Bug Reporter: Ashutosh Mestry Assignee: Ashutosh Mestry Fix For: trunk, 1.0.0 *Background* Current implementation of _Migration Exporter_ uses Spring to instantiate objects. This works for most of Atlas v0.8 versions, except the early versions that did not have Spring integrated. *Analysis* The use of Spring in object creation imposes this restriction. Simpler approach by directly instantiating objects can help in supporting more versions. There may be some inconsistencies in parameters passed during object creation. This needs to be considered. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (ATLAS-2760) Update Hive hook to create AWS S3 entities for S3 path references
Madhan Neethiraj created ATLAS-2760: --- Summary: Update Hive hook to create AWS S3 entities for S3 path references Key: ATLAS-2760 URL: https://issues.apache.org/jira/browse/ATLAS-2760 Project: Atlas Issue Type: Bug Components: atlas-intg Reporter: Madhan Neethiraj Assignee: Madhan Neethiraj Entity types for AWS S3 have been added recently via ATLAS-2708. Hive hook should be updated to create AWS S3 entities for references to S3 paths - for example in 'insert overwrite' operation. This will capture the lineage between Hive tables and S3 paths. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Deleting entities and relationships
Hi Madhan Could you please look at the following and tell me whether the following look like sensible requirements for Atlas? To support the OMRS interface, I would like to be able to: soft delete an entity/relationship - this can fail if Atlas is configured only for hard-deletes. hard delete (purge) an entity/relationship - this is needed even when Atlas is configured for soft deletes. restore (un-delete) an entity/relationship that was soft-deleted be able to purge (hard delete) an entity/relationship that was already soft-deleted. inquire whether a delete will be soft or hard (before issuing the delete) - I guess being able to safely inspect the configuration would b an option here, but an API might be better. find out whether an entity/relationship was soft- or hard-deleted after the delete operation. For this I could just try to retrieve the entity/relationship but it would be neater to known from the return value. I don't think the EntityMutationResponse includes that detail. I know the requestContext is updated to include the deleted entities but that seems to be before the delegation to either the soft or hard delete handler, so I suspect it cannot indicate whether the delete was ultimately soft or hard). I'm not sure how I would get that information that for relationships. Best regards, Graham Graham Wallis IBM Analytics Internet: graham_wal...@uk.ibm.com IBM Laboratories, Hursley Park, Hursley, Hampshire SO21 2JN Tel: +44 7741 379735 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
Re: [jira] [Commented] (ATLAS-1773) Create the OMRS Connector for Atlas
Hi David, I'm not sure why the 2665 patch failed to apply - it sounds like it contains dependencies that conflict with the current master. May be worth asking Bogdan to rebase it, Your correction to the omag-server/pom.xml looks good to me - for me that is sufficient to remove the cyclic log dependency and allows Atlas to start correctly. Did you then perform a full, clean build? Best regards, Graham Graham Wallis IBM Analytics Internet: graham_wal...@uk.ibm.com IBM Laboratories, Hursley Park, Hursley, Hampshire SO21 2JN Tel: +44 7741 379735 From: "David Radley (JIRA)" To: dev@atlas.apache.org Date: 10/06/2018 10:47 Subject:[jira] [Commented] (ATLAS-1773) Create the OMRS Connector for Atlas [ https://issues.apache.org/jira/browse/ATLAS-1773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16507324#comment-16507324 ] David Radley commented on ATLAS-1773: - [~grahamwallis] I have committed 2745. I then tried to follow the notes. I am on the latest master. The patch for ATLAS-1773 applied. I then ran git am < ATLAS-1773-2018-06-07.patch and got: Applying: ATLAS-1773: Atlas OMRS Connector ✔ ~/atlasreview [master ↑·1|…2] 09:20 $ git am < ATLAS-2665-Add-OMAG-API-in-Atlas-v7.patch Applying: ATLAS-2665 Add OMAG-API in Atlas error: patch failed: omag-api/pom.xml:41 error: omag-api/pom.xml: patch does not apply error: patch failed: omag-server/pom.xml:53 error: omag-server/pom.xml: patch does not apply error: patch failed: omas-assetconsumer/pom.xml:39 error: omas-assetconsumer/pom.xml: patch does not apply error: patch failed: omas-connectedasset/pom.xml:40 error: omas-connectedasset/pom.xml: patch does not apply error: patch failed: omrs/pom.xml:39 error: omrs/pom.xml: patch does not apply Patch failed at 0001 ATLAS-2665 Add OMAG-API in Atlas The copy of the patch that failed is found in: .git/rebase-apply/patch I did a 3 way merge choosing the patch content over head to apply the content. This was all around pom dependancies it did not like. This built successfully. I got the log error even though I have made your suggested change (though possibly made a mistake). I have changed the omag-server pom to: omag-server Open Metadata and Governance (OMAG) Server Open Metadata and Governance (OMAG) server for running open metadata function outside of the Apache Atlas server. jar org.springframework.boot spring-boot-starter 1.5.7.RELEASE ch.qos.logback * org.springframework.boot spring-boot-starter-web 1.5.7.RELEASE > Create the OMRS Connector for Atlas > --- > > Key: ATLAS-1773 > URL: https://issues.apache.org/jira/browse/ATLAS-1773 > Project: Atlas > Issue Type: New Feature > Components: atlas-core >Affects Versions: 1.0.0 >Reporter: Mandy Chessell >Assignee: Graham Wallis >Priority: Major > Attachments: ATLAS-1773-2018-06-07.patch, Notes on running the OMRS stack including the Atlas OMRS connector.txt, RELATIONSHIP_DEFS_AS_ASSOCIATIONS.patch > > > This JIRA provides the definition of the OMRS Connector API and an implementation of this API for a local Apache Atlas metadata repository and for the OMRS REST API. > The OMRS Connector has 3 API groups > * The types API - this is the metadata API for a metadata repository > * The entity and relationships APIs that provide the type-agnostic interfaces that can access any type - even those added dynamically > * The fine-grained type-safe APIs that are generated from the addons models in the build. -- This message was sent by Atlassian JIRA (v7.6.3#76005) Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
