Re: Consensus for migration of git repositories to gitbox
Hi Madhan, That is a good idea - I think a couple of Atlas changes are going in or are in progress make the Atlas connector work, around deletes and saving reference copies, then we could branch, many thanks , David. From: Madhan Neethiraj To: David Radley Cc: "dev@atlas.apache.org" Date: 11/01/2019 17:49 Subject:Re: Consensus for migration of git repositories to gitbox David, ATLAS-2987 (Update component versions of Atlas to use Hadoop3, HBase2 and Solr7) is the only big change I am aware of in the pipeline i.e. under review. To minimize potential impacts due to commits that go into master, I would suggest you to consider using branch-1.0 for your demo. Only bugfixes should go into maint branches (like branch-1.0), hence it should be safer. Thanks, Madhan From: David Radley Date: Thursday, January 10, 2019 at 2:44 AM To: Madhan Neethiraj Cc: "dev@atlas.apache.org" Subject: Re: Consensus for migration of git repositories to gitbox Hi Madhan, +1 I like the idea of being pro active with this. We are looking to demo master Atlas code at a conference in the following week, so we need to have a working Atlas master build, so it would be good to make this change asap. On a related subject, would it be possible for the community to not make any big changes (types, software level changes or API changes) coming up to the week of the 16th of February - so the Atlas / Egeria demo continues to build. If there are big changes you need in, could I request that the community be informed and given a change to assess the impact during this period and potentially delay pervasive changes, many thanks , David. From:Madhan Neethiraj To:"dev@atlas.apache.org" Date:10/01/2019 07:37 Subject:Re: Consensus for migration of git repositories to gitbox +1 Thanks Apoorv for volunteering. Please keep the community updated of the progress and actions to be done by each to access the migrated repo. Thanks, Madhan On 1/9/19, 11:01 PM, "Ashutosh Mestry" wrote: +1 for migrating Atlas repositories. Thanks Apoorv! ~ ashutosh On 1/9/19, 10:58 PM, "Nixon Rodrigues" wrote: Thanks Apoorv for volunteering this effort +1 for migrating Atlas repositories to gitbox server. Regards, Nixon On Thu, Jan 10, 2019 at 11:50 AM Apoorv Naik wrote: > Hi Atlas devs, > > There's an ASF initiative to migrate the git-wip-us-apache.org > repositories > to the gitbox server before Feb 7, after which it'll be mass migrated > without a warning. I volunteer to take this migration effort to avoid any > disruption to commit activity. > > PMCs and committers, please share your +1 or -1 vote for this move (this is > the first step in starting the migration) > > *Apoorv Naik* > Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
[jira] [Commented] (ATLAS-1773) Create the OMRS Connector for Atlas
[ https://issues.apache.org/jira/browse/ATLAS-1773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16739509#comment-16739509 ] David Radley commented on ATLAS-1773: - [~grahamwallis] I wonder if you could add the link to this Jira to the associated RBT(s) for the latest patches. I assume you are looking for the RBTs to be reviewed > Create the OMRS Connector for Atlas > --- > > Key: ATLAS-1773 > URL: https://issues.apache.org/jira/browse/ATLAS-1773 > Project: Atlas > Issue Type: New Feature > Components: atlas-core >Affects Versions: 1.0.0 >Reporter: Mandy Chessell >Assignee: Graham Wallis >Priority: Major > Attachments: ATLAS-1773-2018-12-21.patch, > ATLAS-2810-2019-01-10.patch, ATLAS-2939-2018-12-21.patch, > ATLAS-2985-2019-01-10.patch > > > This JIRA provides the definition of the OMRS Connector API and an > implementation of this API for a local Apache Atlas metadata repository and > for the OMRS REST API. > The OMRS Connector has 3 API groups > * The types API - this is the metadata API for a metadata repository > * The entity and relationships APIs that provide the type-agnostic interfaces > that can access any type - even those added dynamically > * The fine-grained type-safe APIs that are generated from the addons models > in the build. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: Consensus for migration of git repositories to gitbox
Hi Madhan, +1 I like the idea of being pro active with this. We are looking to demo master Atlas code at a conference in the following week, so we need to have a working Atlas master build, so it would be good to make this change asap. On a related subject, would it be possible for the community to not make any big changes (types, software level changes or API changes) coming up to the week of the 16th of February - so the Atlas / Egeria demo continues to build. If there are big changes you need in, could I request that the community be informed and given a change to assess the impact during this period and potentially delay pervasive changes, many thanks , David. From: Madhan Neethiraj To: "dev@atlas.apache.org" Date: 10/01/2019 07:37 Subject:Re: Consensus for migration of git repositories to gitbox +1 Thanks Apoorv for volunteering. Please keep the community updated of the progress and actions to be done by each to access the migrated repo. Thanks, Madhan On 1/9/19, 11:01 PM, "Ashutosh Mestry" wrote: +1 for migrating Atlas repositories. Thanks Apoorv! ~ ashutosh On 1/9/19, 10:58 PM, "Nixon Rodrigues" wrote: Thanks Apoorv for volunteering this effort +1 for migrating Atlas repositories to gitbox server. Regards, Nixon On Thu, Jan 10, 2019 at 11:50 AM Apoorv Naik wrote: > Hi Atlas devs, > > There's an ASF initiative to migrate the git-wip-us-apache.org > repositories > to the gitbox server before Feb 7, after which it'll be mass migrated > without a warning. I volunteer to take this migration effort to avoid any > disruption to commit activity. > > PMCs and committers, please share your +1 or -1 vote for this move (this is > the first step in starting the migration) > > *Apoorv Naik* > Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
Re: [DRAFT] Board report for Apache Atlas: December 2018
Hi Madhan, That is a shame. I hope we can work together so by the next report, the Egeria connector for Atlas will be in the Atlas code, so we will have an even bigger good news story, all the best, David. From: Madhan Neethiraj To: "dev@atlas.apache.org" Date: 13/12/2018 22:42 Subject:Re: [DRAFT] Board report for Apache Atlas: December 2018 David, Thanks for reviewing the report and suggestion to reference Egeria. This would indeed be a good item to include. However, the report was sent to the board yesterday - as it was the last day. I will include Egeria reference in the next report. Hope it is alright with you. Thanks, Madhan On 12/13/18, 1:23 AM, "David Radley" wrote: Hi Madhan, Enhancements have gone into Atlas to enable Atlas to be a reference implementation for Egeria. The Egeria Atlas connector is out for review. This is a very positive story for Atlas, that we could include in the draft board report,all the best, David. From: Madhan Neethiraj To: "dev@atlas.apache.org" Date: 11/12/2018 05:41 Subject:[DRAFT] Board report for Apache Atlas: December 2018 Atlas team, Please review the draft board report below and send your feedback/comments. Thanks, Madhan ## Description: Apache Atlas is a scalable and extensible set of core foundational governance services that enables enterprises to effectively and efficiently meet their compliance requirements within Hadoop and allows integration with the complete enterprise data ecosystem ## Issues: There are no issues requiring board attention at this time. ## Activity: - released 0.8.3 on 10/31/2018 - released 1.1.0 on 09/17/2018 - working on 2.0.0 release, to support Hadoop 3, HBase 2, Solr 7, Kafka 2, Hive 3 - updated to support Hadoop trusted-proxy authentication - updated lineage UI to support entity-type specific icons, customizable depth, option to hide process entities - performance related fixes in Hive hook and notification processing - model enhancements to support soft-ref - export/import enhancements to create audit entries containing summary of the operation ## Health report: - 1 new contributor added in last 3 months: Nikhil Bonte ## PMC changes: - Currently 33 PMC members - No new PMC members added in last 3 months - Last PMC member addition was on 6/21/2017 ## Committer base changes: - Currently 38 committers - 1 new committer was added in last 3 months: Ramesh Mani - Last addition to committer role was on 10/15/2018 ## Releases: 2.0.0plan to release by 12/31/2018 0.8.3was released on 10/31/2018 1.1.0was released on 09/17/2018 1.0.0was released on 06/02/2018 0.8.2was released on 02/05/2018 1.0.0-alpha was released on 01/25/2018 0.8.1was released on 08/29/2017 0.8-incubating was released on 03/16/2017 0.7.1-incubating was released on 01/26/2017 0.7-incubating was released on 07/09/2016 0.6-incubating was released on 12/31/2015 0.5-incubating was released on 07/11/2015 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
Re: [DRAFT] Board report for Apache Atlas: December 2018
Hi Madhan, Enhancements have gone into Atlas to enable Atlas to be a reference implementation for Egeria. The Egeria Atlas connector is out for review. This is a very positive story for Atlas, that we could include in the draft board report,all the best, David. From: Madhan Neethiraj To: "dev@atlas.apache.org" Date: 11/12/2018 05:41 Subject:[DRAFT] Board report for Apache Atlas: December 2018 Atlas team, Please review the draft board report below and send your feedback/comments. Thanks, Madhan ## Description: Apache Atlas is a scalable and extensible set of core foundational governance services that enables enterprises to effectively and efficiently meet their compliance requirements within Hadoop and allows integration with the complete enterprise data ecosystem ## Issues: There are no issues requiring board attention at this time. ## Activity: - released 0.8.3 on 10/31/2018 - released 1.1.0 on 09/17/2018 - working on 2.0.0 release, to support Hadoop 3, HBase 2, Solr 7, Kafka 2, Hive 3 - updated to support Hadoop trusted-proxy authentication - updated lineage UI to support entity-type specific icons, customizable depth, option to hide process entities - performance related fixes in Hive hook and notification processing - model enhancements to support soft-ref - export/import enhancements to create audit entries containing summary of the operation ## Health report: - 1 new contributor added in last 3 months: Nikhil Bonte ## PMC changes: - Currently 33 PMC members - No new PMC members added in last 3 months - Last PMC member addition was on 6/21/2017 ## Committer base changes: - Currently 38 committers - 1 new committer was added in last 3 months: Ramesh Mani - Last addition to committer role was on 10/15/2018 ## Releases: 2.0.0plan to release by 12/31/2018 0.8.3was released on 10/31/2018 1.1.0was released on 09/17/2018 1.0.0was released on 06/02/2018 0.8.2was released on 02/05/2018 1.0.0-alpha was released on 01/25/2018 0.8.1was released on 08/29/2017 0.8-incubating was released on 03/16/2017 0.7.1-incubating was released on 01/26/2017 0.7-incubating was released on 07/09/2016 0.6-incubating was released on 12/31/2015 0.5-incubating was released on 07/11/2015 Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
Re: Review Request 69511: Atlas OMRS Connector and Event Mapper
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/69511/#review211091 --- open-metadata/README.md Lines 19 (patched) <https://reviews.apache.org/r/69511/#comment295989> I was expecting the instructions on how to configure and run the Atlas connector to be in this readme. open-metadata/src/main/java/org/apache/atlas/openmetadata/adapters/eventmapper/AtlasOMRSRepositoryEventMapper.java Lines 64 (patched) <https://reviews.apache.org/r/69511/#comment295974> I am wondering what happens if you run 2 Atlas instances on the same machine - I assume we would need to have the ability to specify different topic names. open-metadata/src/main/java/org/apache/atlas/openmetadata/adapters/repositoryconnector/AtlasEntityMapper.java Lines 194 (patched) <https://reviews.apache.org/r/69511/#comment295987> remove? open-metadata/src/main/java/org/apache/atlas/openmetadata/adapters/repositoryconnector/AtlasEntityMapper.java Lines 344 (patched) <https://reviews.apache.org/r/69511/#comment295988> remove? open-metadata/src/main/java/org/apache/atlas/openmetadata/adapters/repositoryconnector/LocalAtlasOMRSMetadataCollection.java Lines 18 (patched) <https://reviews.apache.org/r/69511/#comment295975> License here twice open-metadata/src/main/java/org/apache/atlas/openmetadata/adapters/repositoryconnector/LocalAtlasOMRSMetadataCollection.java Lines 474 (patched) <https://reviews.apache.org/r/69511/#comment295980> Shouldn't we catch and throw the specific exception types? This pattern is used in all the methods. I am wondering why we need this try catch at all, I wonder if we should remove all these try catches; it seems to be there to just add debug tracing, if we are going to trace shouldn't we be putting out error tracing for the error - or is this done elsewhere? Similar to method processAtlasClassificationDef. open-metadata/src/main/java/org/apache/atlas/openmetadata/adapters/repositoryconnector/LocalAtlasOMRSMetadataCollection.java Lines 11945 (patched) <https://reviews.apache.org/r/69511/#comment295981> I am not sure what the CTOR is open-metadata/src/main/java/org/apache/atlas/openmetadata/adapters/repositoryconnector/LocalAtlasOMRSMetadataCollection.java Lines 15188 (patched) <https://reviews.apache.org/r/69511/#comment295977> is this right it looks like it is processing as a Boolean? Most of the other debug steing do not look right in this method. open-metadata/src/main/java/org/apache/atlas/openmetadata/admin/server/spring/OpenMetadataAdminResource.java Lines 429 (patched) <https://reviews.apache.org/r/69511/#comment295982> Should we remove this ? open-metadata/src/main/java/org/apache/atlas/openmetadata/admin/server/spring/OpenMetadataAdminResource.java Lines 454 (patched) <https://reviews.apache.org/r/69511/#comment295983> should we remove this? open-metadata/src/main/java/org/apache/atlas/openmetadata/admin/server/spring/OpenMetadataAdminResource.java Lines 477 (patched) <https://reviews.apache.org/r/69511/#comment295984> remove? open-metadata/src/main/java/org/apache/atlas/openmetadata/admin/server/spring/OpenMetadataAdminResource.java Lines 496 (patched) <https://reviews.apache.org/r/69511/#comment295985> remove? open-metadata/src/main/java/org/apache/atlas/openmetadata/admin/server/spring/OpenMetadataAdminResource.java Lines 516 (patched) <https://reviews.apache.org/r/69511/#comment295986> remove? - David Radley On Dec. 5, 2018, 3:52 p.m., Graham Wallis wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/69511/ > --- > > (Updated Dec. 5, 2018, 3:52 p.m.) > > > Review request for atlas. > > > Repository: atlas > > > Description > --- > > ATLAS-1773: Atlas OMRS conenctor and event mapper > > > Diffs > - > > open-metadata/README.md PRE-CREATION > open-metadata/pom.xml PRE-CREATION > > open-metadata/src/main/java/org/apache/atlas/openmetadata/adapters/eventmapper/AtlasOMRSRepositoryEventMapper.java > PRE-CREATION > > open-metadata/src/main/java/org/apache/atlas/openmetadata/adapters/eventmapper/AtlasOMRSRepositoryEventMapperProvider.java > PRE-CREATION > > open-metadata/src/main/java/org/apache/atlas/openmetadata/adapters/repositoryconnector/AtlasAttributeDefMapper.java > PRE-CREATION > > open-metadata/src/main/java/org/apache/atlas/openmetadata/adapters/repositoryconnector/AtlasAttributeMapper.java > PRE-CREATION > > open-metadata/src
[jira] [Comment Edited] (ATLAS-2817) Update to JanusGraph 0.3.x
[ https://issues.apache.org/jira/browse/ATLAS-2817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16589863#comment-16589863 ] David Radley edited comment on ATLAS-2817 at 8/23/18 8:01 AM: -- [~apoorvnaik] I notice that Janus 0.3.0 https://github.com/JanusGraph/janusgraph/releases says it supports Apache HBase 1.2.6, 1.3.1, 1.4.4. In Atlas we are using HBase 1.1. It looks like there is a mismatch. Also Atlas is using 5.5.1 and JanusGraph prereqs 5.5.4. I have not checked that Cassandra ES levels that Atlas uses are compatible with this new level of JanusGraph. was (Author: davidrad): [~apoorvnaik] I notice that Janus 0.3.0 https://github.com/JanusGraph/janusgraph/releases says it supports Apache HBase 1.2.6, 1.3.1, 1.4.4. In Atlas we are using HBase 1.1. It looks like there is a mismatch. I have not checked that solr, Cassandra ES levels that Atlas uses are compatible with this new level of JanusGraph. > Update to JanusGraph 0.3.x > -- > > Key: ATLAS-2817 > URL: https://issues.apache.org/jira/browse/ATLAS-2817 > Project: Atlas > Issue Type: Improvement >Reporter: Apoorv Naik >Assignee: Apoorv Naik >Priority: Major > Attachments: 0001-Janusgraph-update-to-0.3.0.patch > > > There are couple of index related improvements in this version which helps > resolve the slowness in few of our DSL queries. This upgrade would speed up > DSL execution. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ATLAS-2817) Update to JanusGraph 0.3.x
[ https://issues.apache.org/jira/browse/ATLAS-2817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16589863#comment-16589863 ] David Radley commented on ATLAS-2817: - [~apoorvnaik] I notice that Janus 0.3.0 https://github.com/JanusGraph/janusgraph/releases says it supports Apache HBase 1.2.6, 1.3.1, 1.4.4. In Atlas we are using HBase 1.1. It looks like there is a mismatch. I have not checked that solr, Cassandra ES levels that Atlas uses are compatible with this new level of JanusGraph. > Update to JanusGraph 0.3.x > -- > > Key: ATLAS-2817 > URL: https://issues.apache.org/jira/browse/ATLAS-2817 > Project: Atlas > Issue Type: Improvement >Reporter: Apoorv Naik >Assignee: Apoorv Naik >Priority: Major > Attachments: 0001-Janusgraph-update-to-0.3.0.patch > > > There are couple of index related improvements in this version which helps > resolve the slowness in few of our DSL queries. This upgrade would speed up > DSL execution. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: Review Request 68412: ATLAS-1773: Atlas OMRS Repository Connector
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/68412/#review207494 --- open-metadata/src/main/java/org/apache/atlas/openmetadata/adapters/eventmapper/AtlasOMRSRepositoryEventMapper.java Lines 346 (patched) <https://reviews.apache.org/r/68412/#comment290894> What is the implication of leaving htis TODO in the code? I suggest at least there should be a Jira numebr here to track ths issue. open-metadata/src/main/java/org/apache/atlas/openmetadata/adapters/eventmapper/AtlasOMRSRepositoryEventMapper.java Lines 376 (patched) <https://reviews.apache.org/r/68412/#comment290896> siple => simple open-metadata/src/main/java/org/apache/atlas/openmetadata/adapters/eventmapper/AtlasOMRSRepositoryEventMapper.java Lines 425 (patched) <https://reviews.apache.org/r/68412/#comment290897> I think this s a duplicate line of the one above open-metadata/src/main/java/org/apache/atlas/openmetadata/adapters/eventmapper/AtlasOMRSRepositoryEventMapper.java Lines 434 (patched) <https://reviews.apache.org/r/68412/#comment290898> typo doesn#t open-metadata/src/main/java/org/apache/atlas/openmetadata/adapters/repositoryconnector/AtlasClassificationDefMapper.java Lines 244 (patched) <https://reviews.apache.org/r/68412/#comment290899> typo fial - David Radley On Aug. 17, 2018, 2:38 p.m., Graham Wallis wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/68412/ > --- > > (Updated Aug. 17, 2018, 2:38 p.m.) > > > Review request for atlas. > > > Repository: atlas > > > Description > --- > > ATLAS-1773: Atlas OMRS Repository Connector > > > Diffs > - > > open-metadata/pom.xml PRE-CREATION > > open-metadata/src/main/java/org/apache/atlas/openmetadata/adapters/eventmapper/AtlasOMRSRepositoryEventMapper.java > PRE-CREATION > > open-metadata/src/main/java/org/apache/atlas/openmetadata/adapters/eventmapper/AtlasOMRSRepositoryEventMapperProvider.java > PRE-CREATION > > open-metadata/src/main/java/org/apache/atlas/openmetadata/adapters/repositoryconnector/AtlasAttributeDefMapper.java > PRE-CREATION > > open-metadata/src/main/java/org/apache/atlas/openmetadata/adapters/repositoryconnector/AtlasAttributeMapper.java > PRE-CREATION > > open-metadata/src/main/java/org/apache/atlas/openmetadata/adapters/repositoryconnector/AtlasBaseTypeDefMapper.java > PRE-CREATION > > open-metadata/src/main/java/org/apache/atlas/openmetadata/adapters/repositoryconnector/AtlasClassificationDefMapper.java > PRE-CREATION > > open-metadata/src/main/java/org/apache/atlas/openmetadata/adapters/repositoryconnector/AtlasEntityDefMapper.java > PRE-CREATION > > open-metadata/src/main/java/org/apache/atlas/openmetadata/adapters/repositoryconnector/AtlasEntityMapper.java > PRE-CREATION > > open-metadata/src/main/java/org/apache/atlas/openmetadata/adapters/repositoryconnector/AtlasRelationshipDefMapper.java > PRE-CREATION > > open-metadata/src/main/java/org/apache/atlas/openmetadata/adapters/repositoryconnector/AtlasRelationshipMapper.java > PRE-CREATION > > open-metadata/src/main/java/org/apache/atlas/openmetadata/adapters/repositoryconnector/AtlasStoresProxy.java > PRE-CREATION > > open-metadata/src/main/java/org/apache/atlas/openmetadata/adapters/repositoryconnector/AtlasStoresProxyImpl.java > PRE-CREATION > > open-metadata/src/main/java/org/apache/atlas/openmetadata/adapters/repositoryconnector/Comparator.java > PRE-CREATION > > open-metadata/src/main/java/org/apache/atlas/openmetadata/adapters/repositoryconnector/DSLQueryHelper.java > PRE-CREATION > > open-metadata/src/main/java/org/apache/atlas/openmetadata/adapters/repositoryconnector/EntityDefMapper.java > PRE-CREATION > > open-metadata/src/main/java/org/apache/atlas/openmetadata/adapters/repositoryconnector/FamousFive.java > PRE-CREATION > > open-metadata/src/main/java/org/apache/atlas/openmetadata/adapters/repositoryconnector/ISpringBridge.java > PRE-CREATION > > open-metadata/src/main/java/org/apache/atlas/openmetadata/adapters/repositoryconnector/LocalAtlasOMRSErrorCode.java > PRE-CREATION > > open-metadata/src/main/java/org/apache/atlas/openmetadata/adapters/repositoryconnector/LocalAtlasOMRSMetadataCollection.java > PRE-CREATION > > open-metadata/src/main/java/org/apache/atlas/openmetadata/adapters/repositoryconnector/LocalAtlasOMRSReposito
Re: Review Request 68188: ATLAS-2808: Enable Atlas to support entity proxies
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/68188/#review207487 --- Ship it! Ship It! - David Radley On Aug. 17, 2018, 12:03 p.m., Graham Wallis wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/68188/ > --- > > (Updated Aug. 17, 2018, 12:03 p.m.) > > > Review request for atlas. > > > Repository: atlas > > > Description > --- > > ATLAS_2808: Support for entity proxies > > > Diffs > - > > common/src/main/java/org/apache/atlas/repository/Constants.java a892551f7 > intg/src/main/java/org/apache/atlas/model/instance/AtlasEntity.java > dc2be412b > repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java > 9c4b9f232 > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java > 2153a5a9e > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java > 52c374564 > > > Diff: https://reviews.apache.org/r/68188/diff/3/ > > > Testing > --- > > Tested with saveRelationshipReferenceCopy between entities that do not exist > in the Atlas repository. > > > Thanks, > > Graham Wallis > >
Re: Review Request 68287: ATLAS-2807: Re-evaluate classification propagation during entity delete
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/68287/#review207066 --- repository/src/main/java/org/apache/atlas/repository/store/graph/v1/HardDeleteHandlerV1.java Line 56 (original) <https://reviews.apache.org/r/68287/#comment290253> I assume this is implementing the reporting text "Hard Delete: In cases of hard delete, the source entity (f1) is deleted from atlas, but its classification vertex (PII) continue to exist and will continue propagating to process1 and t1." I am not sure about this - as if a source entity is PII and it is deleted - I would think any downstream entities should not be PII. - David Radley On Aug. 10, 2018, 2:36 a.m., Sarath Subramanian wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/68287/ > --- > > (Updated Aug. 10, 2018, 2:36 a.m.) > > > Review request for atlas, Apoorv Naik, Ashutosh Mestry, Madhan Neethiraj, and > Nixon Rodrigues. > > > Bugs: ATLAS-2807 > https://issues.apache.org/jira/browse/ATLAS-2807 > > > Repository: atlas > > > Description > --- > > The current behavior - when we delete an entity f1, all tags associate to f1 > which got propagated > > > to downstream entities are removed – PII tag propagated to process1 and t1 is > removed > > > Proposed Change > > > Soft Delete: > > > When entity f1 is deleted, retain the propagated classification edges, so the > downstream entities > > > – process1 and t1 continue to have PII classification associated to them > > > Hard Delete: > > > In cases of hard delete, the source entity (f1) is deleted from atlas, but > its classification vertex > > > (PII) continue to exist and will continue propagating to process1 and t1. > > > Diffs > - > > common/src/main/java/org/apache/atlas/repository/Constants.java 7988d1d81 > intg/src/main/java/org/apache/atlas/AtlasErrorCode.java 328b76734 > intg/src/main/java/org/apache/atlas/model/instance/AtlasClassification.java > 3132379b9 > repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java > d328873f1 > > repository/src/main/java/org/apache/atlas/repository/store/graph/AtlasEntityStore.java > c6be4061b > > repository/src/main/java/org/apache/atlas/repository/store/graph/v1/DeleteHandlerV1.java > 4a0924bbe > > repository/src/main/java/org/apache/atlas/repository/store/graph/v1/HardDeleteHandlerV1.java > a95e6898e > > repository/src/main/java/org/apache/atlas/repository/store/graph/v1/SoftDeleteHandlerV1.java > 41e65d4a3 > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2.java > 28a26ab4f > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java > f57ce99f7 > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java > 8f4faaff8 > > repository/src/test/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2Test.java > 58005c458 > > repository/src/test/java/org/apache/atlas/repository/tagpropagation/ClassificationPropagationTest.java > 8ceeedb01 > webapp/src/main/java/org/apache/atlas/web/resources/EntityResource.java > e01cba92d > webapp/src/main/java/org/apache/atlas/web/rest/EntityREST.java fd331faa0 > webapp/src/test/java/org/apache/atlas/web/adapters/TestEntityREST.java > 78bd53ce7 > > > Diff: https://reviews.apache.org/r/68287/diff/1/ > > > Testing > --- > > Validated basic cases using REST clirnt. > > > PreCommit: In progress > > > Thanks, > > Sarath Subramanian > >
Re: Review Request 68188: ATLAS-2808: Enable Atlas to support entity proxies
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/68188/#review207015 --- common/src/main/java/org/apache/atlas/repository/Constants.java Lines 96 (patched) <https://reviews.apache.org/r/68188/#comment290179> I suggest a comment to document what the flag is. In the style of the homeId comment. repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java Lines 174 (patched) <https://reviews.apache.org/r/68188/#comment290180> I am wondering what your thoughts are on validation in Atlas. Specifically for cases like: - the entity is a proxy but has no homeid. - the entity is a proxy but has none manditory attributes set which should not be returned - the entity is a proxy so should not be in search results As coded the caller is responsible for this sort of validation. - David Radley On Aug. 3, 2018, 1:33 p.m., Graham Wallis wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/68188/ > --- > > (Updated Aug. 3, 2018, 1:33 p.m.) > > > Review request for atlas. > > > Repository: atlas > > > Description > --- > > ATLAS_2808: Support for entity proxies > > > Diffs > - > > common/src/main/java/org/apache/atlas/repository/Constants.java > 7988d1d81595eb3d65ecca2480c0e67d17f53573 > intg/src/main/java/org/apache/atlas/model/instance/AtlasEntity.java > dc2be412b9ec1b07d7c7befc722810273464cd3f > repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java > d328873f1f2695954d79e476a40166f48ffda482 > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java > f57ce99f764a5c779f1cfa4325849c15ac804b2d > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java > 448d03bc069869db85cd86e14021a3fa59b57e33 > > > Diff: https://reviews.apache.org/r/68188/diff/1/ > > > Testing > --- > > Tested with saveRelationshipReferenceCopy between entities that do not exist > in the Atlas repository. > > > Thanks, > > Graham Wallis > >
[jira] [Commented] (ATLAS-2774) Options for hard and soft delete of instances
[ https://issues.apache.org/jira/browse/ATLAS-2774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16532700#comment-16532700 ] David Radley commented on ATLAS-2774: - Sounds like a good change. A couple of thoughts: - we may need to have new return codes if we attempt a purge and it does not work. - I assume a soft delete of an entity soft deletes any associated relationships around it. I guess we may need special logic to handle deletion when there is a composition relationship. It would make sense that deleting the container - deletes the children entities in this case (recursively) or is not allowed until the children are themselves deleted. > Options for hard and soft delete of instances > - > > Key: ATLAS-2774 > URL: https://issues.apache.org/jira/browse/ATLAS-2774 > Project: Atlas > Issue Type: New Feature >Reporter: Graham Wallis >Priority: Major > > *Overview* > For Entities and Relationships, the following delete behaviors are desired. > Atlas can be configured to offer hard delete (only). Alternatively it can be > configured to offer soft-delete. Configuration is achieved by setting the > delete handler configuration property (see atlas-application.properties) to > either the SoftDeleteHandlerV1 or the HardDeleteHandlerV1. The default (for > when the property is not set) is to use the SoftDeleteHandlerV1. > The AtlasEntityStoreV2 deleteById() and deleteByIds() and methods and the > AtlasRelationshipStoreV2 deleteById() method are sensitive to the above > configuration. When the configuration is soft these methods will perform a > soft delete; when it is hard they will perform a hard delete. > In addition to the above methods, the AtlasEntityStoreV2 and > AtlasRelationshipStoreV2 offer a (new) purgeById() method that ALWAYS > performs a hard delete. This is true regardless of which delete handler has > been configured. When the configuration is hard, the purgeById() methods and > deleteById() methods are essentially equivalent. When the configuration is > soft, the purgeById() methods continue to provide a hard delete capability. > The AtlasEntityStoreV2 deleteById(), deleteByIds() and purgeById() methods > will delete the specified entities and any relationships to which they are > connected. The AtlasRelationshipStoreV2 deleteById() and purgeById() methods > will delete the specified relationship. Deletion of a relationship may cause > an upate to an entity to which the relationhsip is connected, if it changes > the propagation of classifications, for example. > In a hard delete or purge operation, an affected entity and relationship will > be removed from the graph and will not be returned in response to any future > queries. This is true regardless of whether the affected entity or > relationship has already been soft deleted or not. i.e. it's status could > initially be ACTIVE or DELETED. Following the operation it will not exist. > In a soft delete, the affected entities and relationships (that initially > have status ACTIVE) are updated to set the status to DELETED. These instances > can still be returned from queries and searches if the appropriate control is > selected (to include deleted instances). Without that control selected they > will not be returned. > *Use Case for soft delete* > Soft delete provides a 'safe' means of deleting instances from the repository > whilst preserving an audit trail and (if supported by the repository) > enabling a deleted instance to be restored (to ACTIVE status). > *Use Cases for hard delete* > One use case for hard delete is to permanently remove soft-deleted items some > period of time after they were soft deleted. > Another use case for hard delete is to is to correct (clean up after) a > mistake in which a potentially large batch of incomplete/corrupted/wrong > metadata is loaded and needs to be fully removed. > *Use Scenario for an OMAS user* > An OMAS provides a delete function that does NOT offer the user a choice of > hard/soft. The OMAS's delete function will invoke the relevant soft delete > method provided by the OMRS - deleteEntity or deleteRelationship. These are > both soft delete methods. > The Atlas OMRS Connector's deleteEntity/deleteRelationship methods will check > whether Atlas is configured for hard or soft delete. If Atlas is configured > for hard delete then soft-delete is not possible, and the method will throw a > FunctionNotSupported exception which is caught by the OMAS. > On catching this exception the OMAS author should automatically escala
[jira] [Commented] (ATLAS-2708) AWS S3 data lake typedefs for Atlas
[ https://issues.apache.org/jira/browse/ATLAS-2708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16507807#comment-16507807 ] David Radley commented on ATLAS-2708: - [~barbara] I have had a quick look at the json file, it looks like you are using the old style of definition. I would recommend that that you use the new RelationshipDef style rather than constraints. This is more descriptive of relationships - it allows you to specify attributes on the relationship as well. I think the constraints style is only there for so that the Hadoop types could continue to work. > AWS S3 data lake typedefs for Atlas > --- > > Key: ATLAS-2708 > URL: https://issues.apache.org/jira/browse/ATLAS-2708 > Project: Atlas > Issue Type: New Feature > Components: atlas-core >Reporter: Barbara Eckman >Assignee: Barbara Eckman >Priority: Critical > Attachments: all_datalake_typedefs.json > > > Currently the base types in Atlas do not include AWS data lake objects. It > would be nice to add typedefs for AWS data lake objects (buckets and > pseudo-directories) and lineage processes that move the data from another > source (e.g., kafka topic) to the data lake. For example: > * AWSS3PseudoDir type represents the pseudo-directory “prefix” of objects in > an S3 bucket. For example, in the case of an object with key > “myWork/Development/Projects1.xls”, “myWork/Development” is the > pseudo-directory. It supports: > ** Array of avro schemas that are associated with the data in the > pseudo-directory (based on Avro schema extensions outlined in ATLAS-2694) > ** what type of data it contains, e.g., avro, json, unstructured > ** time of creation > * AWSS3BucketLifeCycleRule type represents a rule specifying a transition of > the data in a bucket to a storageClass after a specific time interval, or > expiration. For example, transition to GLACIER after 60 days, or expire > (i.e. be deleted) after 90 days: > ** ruleType (e.g., transition or expiration) > ** time interval in days before rule is executed > ** storageClass to which the data is transitioned (null if ruleType is > expiration) > * AWSTag type represents a tag-value pair created by the user and associated > with an AWS object. > ** tag > ** value > * AWSCloudWatchMetric type represents a storage or request metric that is > monitored by AWS CloudWatch and can be configured for a bucket > ** metricName, for example, “AllRequests”, “GetRequests”, > TotalRequestLatency, BucketSizeBytes > ** scope: null if entire bucket; otherwise, the prefixes/tags that filter or > limit the monitoring of the metric. > * AWSS3Bucket type represents a bucket in an S3 instance. It supports: > ** Array of AWSS3PseudoDirectories that are associated with objects stored > in the bucket > ** AWS region > ** IsEncrypted (boolean) > ** encryptionType, e.g., AES-256 > ** S3AccessPolicy, a JSON object expressing access policies, eg GetObject, > PutObject > ** time of creation > ** Array of AWSS3BucketLifeCycleRules that are associated with the bucket > ** Array of AWSS3CloudWatchMetrics that are associated with the bucket or > its tags or prefixes > ** Array of AWSTags that are associated with the bucket > * Generic dataset2Dataset process to represent movement of data from one > dataset to another. It supports: > ** array of transforms performed by the process > ** map of tag/value pairs representing configurationParameters of the process > ** inputs and outputs are arrays of dataset objects, e.g., kafka topic and > S3 pseudo-directory. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ATLAS-1773) Create the OMRS Connector for Atlas
[ https://issues.apache.org/jira/browse/ATLAS-1773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16507324#comment-16507324 ] David Radley commented on ATLAS-1773: - [~grahamwallis] I have committed 2745. I then tried to follow the notes. I am on the latest master. The patch for ATLAS-1773 applied. I then ran git am < ATLAS-1773-2018-06-07.patch and got: Applying: ATLAS-1773: Atlas OMRS Connector ✔ ~/atlasreview [master ↑·1|…2] 09:20 $ git am < ATLAS-2665-Add-OMAG-API-in-Atlas-v7.patch Applying: ATLAS-2665 Add OMAG-API in Atlas error: patch failed: omag-api/pom.xml:41 error: omag-api/pom.xml: patch does not apply error: patch failed: omag-server/pom.xml:53 error: omag-server/pom.xml: patch does not apply error: patch failed: omas-assetconsumer/pom.xml:39 error: omas-assetconsumer/pom.xml: patch does not apply error: patch failed: omas-connectedasset/pom.xml:40 error: omas-connectedasset/pom.xml: patch does not apply error: patch failed: omrs/pom.xml:39 error: omrs/pom.xml: patch does not apply Patch failed at 0001 ATLAS-2665 Add OMAG-API in Atlas The copy of the patch that failed is found in: .git/rebase-apply/patch I did a 3 way merge choosing the patch content over head to apply the content. This was all around pom dependancies it did not like. This built successfully. I got the log error even though I have made your suggested change (though possibly made a mistake). I have changed the omag-server pom to: omag-server Open Metadata and Governance (OMAG) Server Open Metadata and Governance (OMAG) server for running open metadata function outside of the Apache Atlas server. jar org.springframework.boot spring-boot-starter 1.5.7.RELEASE ch.qos.logback * org.springframework.boot spring-boot-starter-web 1.5.7.RELEASE > Create the OMRS Connector for Atlas > --- > > Key: ATLAS-1773 > URL: https://issues.apache.org/jira/browse/ATLAS-1773 > Project: Atlas > Issue Type: New Feature > Components: atlas-core >Affects Versions: 1.0.0 >Reporter: Mandy Chessell >Assignee: Graham Wallis >Priority: Major > Attachments: ATLAS-1773-2018-06-07.patch, Notes on running the OMRS > stack including the Atlas OMRS connector.txt, > RELATIONSHIP_DEFS_AS_ASSOCIATIONS.patch > > > This JIRA provides the definition of the OMRS Connector API and an > implementation of this API for a local Apache Atlas metadata repository and > for the OMRS REST API. > The OMRS Connector has 3 API groups > * The types API - this is the metadata API for a metadata repository > * The entity and relationships APIs that provide the type-agnostic interfaces > that can access any type - even those added dynamically > * The fine-grained type-safe APIs that are generated from the addons models > in the build. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (ATLAS-2745) Atlas EnumDefStore does not remember default value
[ https://issues.apache.org/jira/browse/ATLAS-2745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Radley resolved ATLAS-2745. - Resolution: Fixed commit ce5ffeb710721b78135e8b3c3ebe593bd70d2cdf (HEAD -> master, origin/master, origin/HEAD) Author: Graham Wallis Date: Thu Jun 7 14:25:53 2018 +0100 ATLAS-2745 - AtlasEnumDefStore should remember default value Signed-off-by: David Radley > Atlas EnumDefStore does not remember default value > -- > > Key: ATLAS-2745 > URL: https://issues.apache.org/jira/browse/ATLAS-2745 > Project: Atlas > Issue Type: Bug >Reporter: Graham Wallis >Assignee: Graham Wallis >Priority: Minor > Attachments: ATLAS-2745-2018-06-07.patch > > > If an enum def has a default value it is not remembered when the enum is > stored. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: Review Request 67486: ATLAS-2745 EnumDefStore manages defaultValue
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/67486/#review204487 --- Ship it! Ship It! - David Radley On June 7, 2018, 1:57 p.m., Graham Wallis wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/67486/ > --- > > (Updated June 7, 2018, 1:57 p.m.) > > > Review request for atlas and Madhan Neethiraj. > > > Repository: atlas > > > Description > --- > > ATLAS-2745 - AtlasEnumDefStore should remember default value > > > Diffs > - > > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEnumDefStoreV2.java > 7d7233fe8c6e9b8449df7798afdc7216e30d538b > > > Diff: https://reviews.apache.org/r/67486/diff/1/ > > > Testing > --- > > Bring-up testing loading Open Metadata types including EnumDefs with default > values. > > > Thanks, > > Graham Wallis > >
Re: [VOTE] Release Apache Atlas version 1.0.0 - rc2
Thanks Madhan for putting out the release candidate 2 for Apache Atlas 1.0 - Built with embedded-hbase-solr profile successfully with no test failures - built with clean install with no test failures. - Ran quick start and the UI. +1 for release candidate #2. all the best, David. From: Sarath Subramanian To: dev@atlas.apache.org Date: 31/05/2018 17:17 Subject:Re: [VOTE] Release Apache Atlas version 1.0.0 - rc2 Thanks Madhan for putting out the release candidate 2 for Apache Atlas 1.0 - Verified checksum and signature - Built with embedded-hbase-solr profile successfully with no test failures. - Validated new functionalities - glossary, relationships, lineage, tag propagation and UI enhancements +1 for release candidate #2. Thanks, Sarath Subramanian On Wed, May 30, 2018 at 12:31 AM, Madhan Neethiraj wrote: > Atlas team, > > Apache Atlas 1.0.0 release candidate #2 is now available for a vote within > dev community. > > Following commit went into branch-1.0 since the last release candidate, > #1, was sent yesterday. > ATLAS-2726: updated migration-import to add __name property to > classification edges > > Links to the release artifacts are given below. Please review and vote. > > The vote will be open for at least 72 hours or until necessary votes are > reached. > [ ] +1 approve > [ ] +0 no opinion > [ ] -1 disapprove (and reason why) > > Thanks, > Madhan > > > List of issues addressed in this release: https://issues.apache.org/ > jira/issues/?jql=project%20%3D%20Atlas%20AND%20resolution% > 20!%3D%20Unresolved%20%20AND%20fixVersion%20%3D%201.0.0% > 20ORDER%20BY%20key%20DESC > > Git tag for the release: https://github.com/apache/ > atlas/tree/release-1.0.0-rc2 > > Sources for the release: https://dist.apache.org/repos/ > dist/dev/atlas/1.0.0-rc2/apache-atlas-1.0.0-sources.tar.gz > > Documentation for the release: http://atlas.apache.org/1.0.0- > rc0/index.html > > Source release verification: > PGP Signature: https://dist.apache.org/repos/dist/dev/atlas/1.0.0-rc2/ > apache-atlas-1.0.0-sources.tar.gz.asc > SHA512 Hash: https://dist.apache.org/repos/dist/dev/atlas/1.0.0-rc2/ > apache-atlas-1.0.0-sources.tar.gz.sha512 > > Keys to verify the signature of the release artifacts are available at: > https://dist.apache.org/repos/dist/dev/atlas/KEYS > > > > Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
Re: Problem deleting edges with 1.0.0 - pre rc
Hi Pierre, Just so I am clear, does"Entity A -> Composition Relationship -> Entity B" mean that B or A is the container? If B is the container then - you should be able to delete A. You are saying that B can be deleted and leaves its part A. If this is the case then I would suggest that the code could be enhanced to either stop this sort of deletion or delete it and all of its composed parts. I suggest this is not a stopper for v1.0, as you have a circumvention where you manually delete each entity. If this is an issue - I suggest looking at AtlasEntityStoreV1 deleteById to add some checking around composition relationships prior to deletion, all the best, David. From: Pierre Padovani To: dev@atlas.apache.org Date: 31/05/2018 17:47 Subject:Re: Problem deleting edges with 1.0.0 - pre rc Sorry for the spam... One last data point. It looks like the deletion of B is the issue. I can delete A, but this doesn't delete B. On Thu, May 31, 2018 at 11:21 AM, Pierre Padovani < pierre.padov...@civitaslearning.com> wrote: > The level we are using is a few weeks old, but I didn't see an issue that > fixed this in the interim. > > Problem: > > Entity A -> Composition Relationship -> Entity B > > Delete Entity A and Entity B > > I get the following exception trace: > > java.lang.IllegalArgumentException: Invalid edge label > r:DiscoveryPackTables: expected 2 or 3 label components but found 1 at > org.apache.atlas.repository.graph.AtlasEdgeLabel.(AtlasEdgeLabel.java:37) > at org.apache.atlas.repository.store.graph.v1.DeleteHandlerV1. > getAttributeForEdge(DeleteHandlerV1.java:722) at > org.apache.atlas.repository.store.graph.v1.DeleteHandlerV1.deleteVertex(DeleteHandlerV1.java:865) > at org.apache.atlas.repository.store.graph.v1.DeleteHandlerV1. > deleteTypeVertex(DeleteHandlerV1.java:718) at org.apache.atlas.repository. > store.graph.v1.DeleteHandlerV1.deleteEntities(DeleteHandlerV1.java:140) > at org.apache.atlas.repository.store.graph.v1.AtlasEntityStoreV1. > deleteVertices(AtlasEntityStoreV1.java:704) at > org.apache.atlas.repository.store.graph.v1.AtlasEntityStoreV1.deleteById(AtlasEntityStoreV1.java:297) > at org.apache.atlas.repository.store.graph.v1.AtlasEntityStoreV1$$ > FastClassBySpringCGLIB$$80c00649.invoke() at > org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) > at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation. > invokeJoinpoint(CglibAopProxy.java:738) at org.springframework.aop. > framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) > at org.apache.atlas.GraphTransactionInterceptor.invoke( > GraphTransactionInterceptor.java:75) at org.springframework.aop.framework. > ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) > at org.springframework.aop.framework.CglibAopProxy$ > DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:673) at > org.apache.atlas.repository.store.graph.v1.AtlasEntityStoreV1$$ > EnhancerBySpringCGLIB$$2072786c.deleteById() at > org.apache.atlas.web.rest.EntityREST.deleteByGuid(EntityREST.java:327) at > sun.reflect.GeneratedMethodAccessor231.invoke(Unknown Source) at > sun.reflect.DelegatingMethodAccessorImpl.invoke( > DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) > at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke( > JavaMethodInvokerFactory.java:60) at com.sun.jersey.server.impl. > model.method.dispatch.AbstractResourceMethodDispatch > Provider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185) > at com.sun.jersey.server.impl.model.method.dispatch. > ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75) > at com.sun.jersey.server.impl.uri.rules.HttpMethodRule. > accept(HttpMethodRule.java:302) at com.sun.jersey.server.impl. > uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) at > com.sun.jersey.server.impl.uri.rules.ResourceClassRule. > accept(ResourceClassRule.java:108) at com.sun.jersey.server.impl. > uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) at > com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept( > RootResourceClassesRule.java:84) at com.sun.jersey.server.impl. > application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542) > at com.sun.jersey.server.impl.application.WebApplicationImpl._ > handleRequest(WebApplicationImpl.java:1473) at com.sun.jersey.server.impl. > application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419) > at com.sun.jersey.server.impl.application.WebApplicationImpl. > handleRequest(WebApplicationImpl.java:1409) at > com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409) > at com.sun.jersey.spi.container.servlet.ServletContainer. > service(ServletContainer.java:558) at com.sun.jersey.spi.container. > servlet.ServletContainer.service(ServletContainer.java:733) at > javax.servlet.http.HttpServlet.servi
[jira] [Resolved] (ATLAS-2523) Import relationship, preserving existing GUID
[ https://issues.apache.org/jira/browse/ATLAS-2523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Radley resolved ATLAS-2523. - Resolution: Fixed commit 72cc566a7f23bef86c00afb6c7ef136c1d01b424 (HEAD -> master, origin/master, origin/HEAD) Author: Graham Wallis Date: Fri May 25 13:35:16 2018 +0100 ATLAS-2523 Add HomeId and allow GUIDs to be specified on creates > Import relationship, preserving existing GUID > - > > Key: ATLAS-2523 > URL: https://issues.apache.org/jira/browse/ATLAS-2523 > Project: Atlas > Issue Type: Bug >Affects Versions: 1.0.0 >Reporter: Graham Wallis >Priority: Major > Fix For: 1.0.0 > > Attachments: ATLAS-2523-2018-05-25.patch > > > To add a relationship defined externally (in another metadata repository), I > want to be able to add the relationship and maintain the existing GUID. > This is possible for an entity by performing an import. It does not seem to > be possible for relationships., because the relationship store always > generates a new GUID, which overwrites the existing GUID. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: Review Request 67332: ATLAS-2709: added model for RDBMS types
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/67332/#review203934 --- addons/models/2000-RDBMS/2010-rdbms_model.json Lines 7 (patched) <https://reviews.apache.org/r/67332/#comment286309> I wonder if you have seen the open metadata types. There is quite a sophisticated representation of schemas and databases and the like. See the wiki for the details of the types; the ones involving schemas are documented here: https://cwiki.apache.org/confluence/display/ATLAS/Area+5+-+Standards The idea is that these types and the others would be the standard representations of RDMS concepts. I am interested to see whether these types are rich enough to meet your requirements. - David Radley On May 25, 2018, 11:48 p.m., Madhan Neethiraj wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/67332/ > --- > > (Updated May 25, 2018, 11:48 p.m.) > > > Review request for atlas, Apoorv Naik, Ashutosh Mestry, and Sarath > Subramanian. > > > Bugs: ATLAS-2709 > https://issues.apache.org/jira/browse/ATLAS-2709 > > > Repository: atlas > > > Description > --- > > added model containing RDBMS typedefs - author: Barbara Eckman > > > Diffs > - > > addons/models/2000-RDBMS/2010-rdbms_model.json PRE-CREATION > > > Diff: https://reviews.apache.org/r/67332/diff/1/ > > > Testing > --- > > - verified that Atlas successfully loads RDBMS model file contents > > > Thanks, > > Madhan Neethiraj > >
Re: Review Request 67083: ATLAS-2665 - Add OMAG-API in Atlas
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67083/#review203328
---
omrs/src/main/java/org/apache/atlas/omrs/admin/OMRSConfigurationFactory.java
Line 55 (original), 55 (patched)
<https://reviews.apache.org/r/67083/#comment285453>
This should be plural to be consistent with access-services. there are a
few otehr instances of this in the patch
omrs/src/main/java/org/apache/atlas/omrs/admin/OMRSConfigurationFactory.java
Line 291 (original), 291 (patched)
<https://reviews.apache.org/r/67083/#comment285454>
this is not what the code does. For this example wouldn't the code would
produce urls starting with.
ocalhost:8080/open-metadata/localServerName/open-metadata/repository-service/
omrs/src/main/java/org/apache/atlas/omrs/admin/OMRSConfigurationFactory.java
Line 306 (original), 306 (patched)
<https://reviews.apache.org/r/67083/#comment285455>
should be repository-services to be consistant
omrs/src/main/java/org/apache/atlas/omrs/rest/server/OMRSRepositoryRESTServices.java
Line 202 (original), 201 (patched)
<https://reviews.apache.org/r/67083/#comment285456>
Are we missing the /users/ before the {userId} in the path? The same for
the rest of this file.
omrs/src/main/java/org/apache/atlas/omrs/rest/server/spring/OMRSRepositoryResource.java
Lines 84 (patched)
<https://reviews.apache.org/r/67083/#comment285457>
plural for consistancy ?
webapp/src/main/webapp/WEB-INF/openMetadataContext.xml
Lines 3 (patched)
<https://reviews.apache.org/r/67083/#comment285452>
Bad end of line character on this and the next 6 lines
- David Radley
On May 17, 2018, 12:10 p.m., Bogdan Sava wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67083/
> ---
>
> (Updated May 17, 2018, 12:10 p.m.)
>
>
> Review request for atlas, Daniela Otelea, David Radley, Graham Wallis, Mandy
> Chessell, and Ruxandra Rosu.
>
>
> Repository: atlas
>
>
> Description
> ---
>
> Create configuration for OMAG API in Atlas
>
>
> Diffs
> -
>
> om-fwk-ocf/pom.xml 8b1cefd
> omag-api/pom.xml 2338286
>
> omag-api/src/main/java/org/apache/atlas/omag/admin/OMAGServerAdministration.java
> dd19018
> omag-api/src/main/java/org/apache/atlas/omag/admin/OMAGServerCategory.java
> PRE-CREATION
>
> omag-api/src/main/java/org/apache/atlas/omag/admin/server/OMAGServerAdminRESTServices.java
> PRE-CREATION
>
> omag-api/src/main/java/org/apache/atlas/omag/admin/server/OMAGServerAdminResource.java
> ebc9f2a
>
> omag-api/src/main/java/org/apache/atlas/omag/admin/server/spring/OMAGServerAdminResource.java
> PRE-CREATION
>
> omag-api/src/main/java/org/apache/atlas/omag/configuration/registration/AccessServiceDescription.java
> 1a15215
>
> omag-api/src/main/java/org/apache/atlas/omag/configuration/registration/AccessServiceRegistration.java
> c2950c4
> omag-server/pom.xml 527c5c3
>
> omag-server/src/main/java/org/apache/atlas/omag/application/OMAGApplication.java
> 88397de
> omag-server/src/main/resources/application.properties PRE-CREATION
> omas-assetconsumer/pom.xml 29935e7
>
> omas-assetconsumer/src/main/java/org/apache/atlas/omas/assetconsumer/client/AssetConsumer.java
> 80a98f5
>
> omas-assetconsumer/src/main/java/org/apache/atlas/omas/assetconsumer/publisher/AssetConsumerPublisher.java
>
>
> omas-assetconsumer/src/main/java/org/apache/atlas/omas/assetconsumer/server/AssetConsumerRESTServices.java
> 967be5a
>
> omas-assetconsumer/src/main/java/org/apache/atlas/omas/assetconsumer/server/spring/AssetConsumerOMASResource.java
> PRE-CREATION
> omas-connectedasset/README.md 2f431df
> omas-connectedasset/pom.xml 219c972
>
> omas-connectedasset/src/main/java/org/apache/atlas/omas/connectedasset/server/ConnectedAssetRESTServices.java
> 8ae6c37
>
> omas-connectedasset/src/main/java/org/apache/atlas/omas/connectedasset/server/spring/ConnectedAssetOMASResource.java
> PRE-CREATION
> omrs/pom.xml 50b9898
>
> omrs/src/main/java/org/apache/atlas/omrs/admin/OMRSConfigurationFactory.java
> 7ae87c6
>
> omrs/src/main/java/org/apache/atlas/omrs/rest/server/OMRSRepositoryRESTServices.java
> 5948482
>
> omrs/src/main/java/org/apache/atlas/omrs/rest/server/spring/OMRSRepositoryResource.java
> PRE-CREATION
>
> omrs/src/test/java/org/apache/atlas/omrs/archivemanager/opentypes/TestOpenMetadata
Re: Review Request 66064: ATLAS-2478 Elasticsearch support broken with JanusGraph 0.2.0
> On May 7, 2018, 11:41 p.m., Ashutosh Mestry wrote:
> > On executing ./bin/atlas_start.py displays these messages:
> > ./target/apache-atlas-1.0.0-SNAPSHOT-bin/apache-atlas-1.0.0-SNAPSHOT/conf/atlas-env.sh:
> > line 65: MANAGE_EMBEDDED_CASSANDRA=${cassandra.embedded}: bad substitution
> > ./target/apache-atlas-1.0.0-SNAPSHOT-bin/apache-atlas-1.0.0-SNAPSHOT/conf/atlas-env.sh:
> > line 68: MANAGE_LOCAL_ELASTICSEARCH=${elasticsearch.managed}: bad
> > substitution
> > Exception: ('Could not find hbase-site.xml in %s. Please set env var
> > HBASE_CONF_DIR to the hbase client conf dir',
> > './target/apache-atlas-1.0.0-SNAPSHOT-bin/apache-atlas-1.0.0-SNAPSHOT/hbase/conf')
> > Traceback (most recent call last):
> > File "./bin/atlas_start.py", line 163, in
> > File "./bin/atlas_start.py", line 92, in main
> > Exception: ('Could not find hbase-site.xml in %s. Please set env var
> > HBASE_CONF_DIR to the hbase client conf dir',
> > './target/apache-atlas-1.0.0-SNAPSHOT-bin/apache-atlas-1.0.0-SNAPSHOT/hbase/conf')
>
> Pierre Padovani wrote:
> Fixed in the updated patch.
>
> Pierre Padovani wrote:
> Any updates on this review?
Hi Pierre - there are outstanding issues against this entry - which normally
indicates there is work for the controbuter to do. These need to be fixed or
dropped before a subsequent review.
- David
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66064/#review202603
---
On May 8, 2018, 3 p.m., Pierre Padovani wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66064/
> ---
>
> (Updated May 8, 2018, 3 p.m.)
>
>
> Review request for atlas, David Radley, Madhan Neethiraj, and Sarath
> Subramanian.
>
>
> Bugs: ATLAS-2478
> https://issues.apache.org/jira/browse/ATLAS-2478
>
>
> Repository: atlas
>
>
> Description
> ---
>
> This patch fixes the Elasticsearch support for JanusGraph 0.2.0 and updates
> documentation.
>
> Included with this patch is an update to the berkley-elasticsearch profile to
> automatically download and include elasticsearch as a side application much
> like solr is. Updates to the start/stop/conf scripts are included as well.
>
> NOTE: This patch includes a **BACKWARDS INCOMPATIBLE** change to
> /atlas/common/src/main/java/org/apache/atlas/repository/Constants.java. There
> are six constants that are incorrectly named with a '.' (dot). This is not
> supported in Elasticsearch 5 and beyond when defining a mapping **UNLESS**
> the field names can be collectively thought of as an object. In the case of
> the fields defined in the Constants.java file, 'type' is defined as a string
> field, and 'type.name' is also defined as a string field. Elasticsearch sees
> this as an error, since it cannot convert type to an object. The fix included
> simply changes the field names from using a '.' (dot) to an '_' (underscore).
> This should NOT affect compatibility with hbase/solr for new installs. For
> existing installations, a reindex will be required as the field names will
> have changed.
>
> **Query**: There is a way we can simplify integration/unit tests for the
> in-memory graph store by using a maven plugin that will download and run an
> elasticsearch node. This is nothing more than a maven change, and change to
> the atlas-application.properties to switch to elasticsearch from solr. I did
> not implement this, but am curious if this change would be desired. If so,
> this can be done with a separate ticket.
>
>
> Diffs
> -
>
> common/src/main/java/org/apache/atlas/repository/Constants.java 16a3aaa15
> distro/pom.xml 6431fd86d
> distro/src/bin/atlas_config.py 9062da649
> distro/src/bin/atlas_start.py 61d69eb21
> distro/src/bin/atlas_stop.py 94c3d6d46
> distro/src/conf/atlas-env.sh 053cbd500
> distro/src/main/assemblies/standalone-package.xml dc88add56
> docs/src/site/twiki/Configuration.twiki 63c3fce96
> docs/src/site/twiki/HighAvailability.twiki 4270d0974
> docs/src/site/twiki/InstallationSteps.twiki dca0618e3
> graphdb/janus/pom.xml 016a09c33
> pom.xml 3469a393a
> webapp/pom.xml 03b84087f
>
>
> Diff: https://reviews.apache.org/r/66064/diff/4/
>
>
> Testing
> ---
>
> We currently use this fix with our Atlas setup in a fork of the Atlas code,
> and have found no issues with it. Additionally, with the update to the
> berkley-elasticsearch profile, I have extensively tested that profile to make
> sure that management of Elasticsearch functioned correctly.
>
>
> Thanks,
>
> Pierre Padovani
>
>
Re: Review Request 67148: ATLAS-2685: Impose displayName restrictions on Glossary, Term & Category
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/67148/#review203206 --- repository/src/main/java/org/apache/atlas/glossary/GlossaryService.java Lines 69 (patched) <https://reviews.apache.org/r/67148/#comment285300> I am wondering why the display name is restricted with these characters - does it cause a probem? Where would you use the display name in a dotted qualfied name? I assume the display name could be Janaese characters - I am not sure it would be appropriate to concatinate dots and a Japanese string. If you want to restrict it, presumably you need to remove all punctuation content in all languages. I suggest you be extra safe for now and only allow upper&lower case case letters and blanks - unless you already have nls requirements. Then extend as requirements come in. Out of interest I assume that the Janus keywords checking could effect what term names can be used. Fyi translation is covered in the open metadata model with relationships. - David Radley On May 16, 2018, 6:11 a.m., Apoorv Naik wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/67148/ > --- > > (Updated May 16, 2018, 6:11 a.m.) > > > Review request for atlas, Madhan Neethiraj and Sarath Subramanian. > > > Bugs: ATLAS-2685 > https://issues.apache.org/jira/browse/ATLAS-2685 > > > Repository: atlas > > > Description > --- > > Glossary & term name can only have letters, spaces, underscores and dashes > Category can only have letters and spaces > > Added appropriate error codes around this check > > > Diffs > - > > intg/src/main/java/org/apache/atlas/AtlasErrorCode.java 3a609786a > repository/src/main/java/org/apache/atlas/glossary/GlossaryService.java > bc9fe2a2e > repository/src/test/java/org/apache/atlas/glossary/GlossaryServiceTest.java > f66165050 > > > Diff: https://reviews.apache.org/r/67148/diff/2/ > > > Testing > --- > > PreCommit: > https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/408/console > > > Thanks, > > Apoorv Naik > >
Re: Review Request 67148: ATLAS-2685: Impose displayName restrictions on Glossary, Term & Category
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/67148/#review203205 --- repository/src/main/java/org/apache/atlas/glossary/GlossaryService.java Lines 139 (patched) <https://reviews.apache.org/r/67148/#comment285299> I see qualified name as a unique identifier, being derived from referenencable. In one of our meeting we talked of using the guid and name here. I thought we talked of deriving a path attribute post v1 - using a new generic Atlas mechanism. Display name might not be unique, as we want to support categories with the same name. I see display name is not the hierarchical path. It should be the name. On the policing I wonder what happens with janus keywords and whether mixed case are viewed as different. - David Radley On May 16, 2018, 6:11 a.m., Apoorv Naik wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/67148/ > --- > > (Updated May 16, 2018, 6:11 a.m.) > > > Review request for atlas, Madhan Neethiraj and Sarath Subramanian. > > > Bugs: ATLAS-2685 > https://issues.apache.org/jira/browse/ATLAS-2685 > > > Repository: atlas > > > Description > --- > > Glossary & term name can only have letters, spaces, underscores and dashes > Category can only have letters and spaces > > Added appropriate error codes around this check > > > Diffs > - > > intg/src/main/java/org/apache/atlas/AtlasErrorCode.java 3a609786a > repository/src/main/java/org/apache/atlas/glossary/GlossaryService.java > bc9fe2a2e > repository/src/test/java/org/apache/atlas/glossary/GlossaryServiceTest.java > f66165050 > > > Diff: https://reviews.apache.org/r/67148/diff/2/ > > > Testing > --- > > PreCommit: > https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/408/console > > > Thanks, > > Apoorv Naik > >
Re: Review Request 67083: ATLAS-2665 - Add OMAG-API in Atlas
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/67083/#review203132 --- omag-api/pom.xml Line 42 (original), 42 (patched) <https://reviews.apache.org/r/67083/#comment285217> Atlas (built with embedded hbase) fails to start with this in the application.log: 2018-05-15 16:56:33,093 INFO - [main:] ~ Loading atlas-application.properties from file:/Users/david/atlasreview/distro/target/apache-atlas-1.0.0-SNAPSHOT-bin/apache-atlas-1.0.0-SNAPSHOT/conf/atlas-application.properties (ApplicationProperties:98) 2018-05-15 16:56:33,100 INFO - [main:] ~ Property (set to default) atlas.graph.cache.db-cache = true (ApplicationProperties:242) 2018-05-15 16:56:33,100 INFO - [main:] ~ Property (set to default) atlas.graph.cache.db-cache-clean-wait = 20 (ApplicationProperties:242) 2018-05-15 16:56:33,102 INFO - [main:] ~ Property (set to default) atlas.graph.cache.db-cache-size = 0.5 (ApplicationProperties:242) 2018-05-15 16:56:33,102 INFO - [main:] ~ Property (set to default) atlas.graph.cache.tx-cache.size = 15000 (ApplicationProperties:242) 2018-05-15 16:56:33,102 INFO - [main:] ~ Property (set to default) atlas.graph.cache.tx-dirty-size = 120 (ApplicationProperties:242) 2018-05-15 16:56:33,116 INFO - [main:] ~ Atlas Server (STARTUP) project.name: apache-atlas project.description:Metadata Management and Data Governance Platform over Hadoop build.user: david build.epoch:1526398417725 project.version:1.0.0-SNAPSHOT build.version: 1.0.0-SNAPSHOT-r7e70b1e76324305fb1cd50302206c9d135b2be53 vc.revision:7e70b1e76324305fb1cd50302206c9d135b2be53 vc.source.url: scm:git:git://git.apache.org/atlas.git/atlas-webapp (Atlas:215) 2018-05-15 16:56:33,117 INFO - [main:] ~ >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (Atlas:216) 2018-05-15 16:56:33,117 INFO - [main:] ~ Server starting with TLS ? false on port 21000 (Atlas:217) 2018-05-15 16:56:33,117 INFO - [main:] ~ <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< (Atlas:218) 2018-05-15 16:56:33,694 INFO - [main:] ~ No authentication method configured. Defaulting to simple authentication (LoginProcessor:102) 2018-05-15 16:56:33,806 INFO - [main:] ~ Logged in user david (auth:SIMPLE) (LoginProcessor:77) 2018-05-15 16:56:34,210 INFO - [main:] ~ Not running setup per configuration atlas.server.run.setup.on.start. (SetupSteps$SetupRequired:189) - David Radley On May 15, 2018, 1:35 p.m., Bogdan Sava wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/67083/ > --- > > (Updated May 15, 2018, 1:35 p.m.) > > > Review request for atlas, Daniela Otelea, David Radley, Graham Wallis, Mandy > Chessell, and Ruxandra Rosu. > > > Repository: atlas > > > Description > --- > > Create configuration for OMAG API in Atlas > > > Diffs > - > > om-fwk-ocf/pom.xml 8b1cefd > omag-api/pom.xml 2338286 > > omag-api/src/main/java/org/apache/atlas/omag/admin/OMAGServerAdministration.java > dd19018 > omag-api/src/main/java/org/apache/atlas/omag/admin/OMAGServerCategory.java > PRE-CREATION > > omag-api/src/main/java/org/apache/atlas/omag/admin/server/OMAGServerAdminRESTServices.java > PRE-CREATION > > omag-api/src/main/java/org/apache/atlas/omag/admin/server/OMAGServerAdminResource.java > ebc9f2a > > omag-api/src/main/java/org/apache/atlas/omag/admin/server/spring/OMAGServerAdminResource.java > PRE-CREATION > > omag-api/src/main/java/org/apache/atlas/omag/configuration/registration/AccessServiceDescription.java > 1a15215 > > omag-api/src/main/java/org/apache/atlas/omag/configuration/registration/AccessServiceRegistration.java > c2950c4 > omag-server/pom.xml 527c5c3 > > omag-server/src/main/java/org/apache/atlas/omag/application/OMAGApplication.java > 88397de > omag-server/src/main/resources/application.properties PRE-CREATION > omas-assetconsumer/pom.xml 29935e7 > > omas-assetconsumer/src/main/java/org/apache/atlas/omas/assetconsumer/client/AssetConsumer.java > 80a98f5
Re: Review Request 67083: ATLAS-2665 - Add OMAG-API in Atlas
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/67083/#review203130 --- omag-api/src/main/java/org/apache/atlas/omag/admin/OMAGServerAdministration.java Line 180 (original), 181 (patched) <https://reviews.apache.org/r/67083/#comment285212> you have remove the throws clause in the latest patch but the javadoc is left in. I suggest leaving in the throws clause. I think they are there to indicate these are the exceptions that any validation code we add in the future might throw omag-api/src/main/java/org/apache/atlas/omag/admin/server/OMAGServerAdminRESTServices.java Lines 1143 (patched) <https://reviews.apache.org/r/67083/#comment285213> javadoc of exceptions mismatches the code. It is important this matches as it will be the basis of the the swagger. This is the same for many methods omag-server/src/main/java/org/apache/atlas/omag/application/OMAGApplication.java Line 28 (original), 27 (patched) <https://reviews.apache.org/r/67083/#comment285210> what about omrs? omas-assetconsumer/src/main/java/org/apache/atlas/omas/assetconsumer/server/AssetConsumerRESTServices.java Line 230 (original), 211 (patched) <https://reviews.apache.org/r/67083/#comment285211> this change have removed the PathVariable and RequestParam annotations, it looks like we have lost inforration about which parameters are PathVariables and which are RequestParams omrs/src/main/java/org/apache/atlas/omrs/admin/OMRSConfigurationFactory.java Line 306 (original), 306 (patched) <https://reviews.apache.org/r/67083/#comment285214> shouldn't this url include the open-metadata segment? webapp/src/main/webapp/WEB-INF/openMetadataContext.xml Lines 2 (patched) <https://reviews.apache.org/r/67083/#comment285216> this licenses has bad characters that are casing git rebase errors when I apply the patch. - David Radley On May 15, 2018, 1:35 p.m., Bogdan Sava wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/67083/ > --- > > (Updated May 15, 2018, 1:35 p.m.) > > > Review request for atlas, Daniela Otelea, David Radley, Graham Wallis, Mandy > Chessell, and Ruxandra Rosu. > > > Repository: atlas > > > Description > --- > > Create configuration for OMAG API in Atlas > > > Diffs > - > > om-fwk-ocf/pom.xml 8b1cefd > omag-api/pom.xml 2338286 > > omag-api/src/main/java/org/apache/atlas/omag/admin/OMAGServerAdministration.java > dd19018 > omag-api/src/main/java/org/apache/atlas/omag/admin/OMAGServerCategory.java > PRE-CREATION > > omag-api/src/main/java/org/apache/atlas/omag/admin/server/OMAGServerAdminRESTServices.java > PRE-CREATION > > omag-api/src/main/java/org/apache/atlas/omag/admin/server/OMAGServerAdminResource.java > ebc9f2a > > omag-api/src/main/java/org/apache/atlas/omag/admin/server/spring/OMAGServerAdminResource.java > PRE-CREATION > > omag-api/src/main/java/org/apache/atlas/omag/configuration/registration/AccessServiceDescription.java > 1a15215 > > omag-api/src/main/java/org/apache/atlas/omag/configuration/registration/AccessServiceRegistration.java > c2950c4 > omag-server/pom.xml 527c5c3 > > omag-server/src/main/java/org/apache/atlas/omag/application/OMAGApplication.java > 88397de > omag-server/src/main/resources/application.properties PRE-CREATION > omas-assetconsumer/pom.xml 29935e7 > > omas-assetconsumer/src/main/java/org/apache/atlas/omas/assetconsumer/client/AssetConsumer.java > 80a98f5 > > omas-assetconsumer/src/main/java/org/apache/atlas/omas/assetconsumer/publisher/AssetConsumerPublisher.java > > > omas-assetconsumer/src/main/java/org/apache/atlas/omas/assetconsumer/server/AssetConsumerRESTServices.java > 967be5a > > omas-assetconsumer/src/main/java/org/apache/atlas/omas/assetconsumer/server/spring/AssetConsumerOMASResource.java > PRE-CREATION > omas-connectedasset/README.md 2f431df > omas-connectedasset/pom.xml 219c972 > > omas-connectedasset/src/main/java/org/apache/atlas/omas/connectedasset/server/ConnectedAssetRESTServices.java > 8ae6c37 > > omas-connectedasset/src/main/java/org/apache/atlas/omas/connectedasset/server/spring/ConnectedAssetOMASResource.java > PRE-CREATION > omrs/pom.xml 50b9898 > > omrs/src/main/java/org/apache/atlas/omrs/admin/OMRSConfigurationFactory.java > 7ae87c6 > > omrs/src/main/java/org/apache/atlas/omrs/rest/server/OMRSRepositoryRESTServ
Re: Review Request 67114: ATLAS-2463 Documentation for Atlas Authorization model and simple authorizer.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/67114/#review203019 --- docs/src/site/twiki/Authentication-Authorization.twiki Line 114 (original) <https://reviews.apache.org/r/67114/#comment285070> I suggest adding some words around the nature of this authorization. The fact that Ranger provides instance level authorization. Atlas itself provides authorization mechanisms that allows the user to protect the Atlas metadata. this can be done in a simple manner or using Ranger. docs/src/site/twiki/Authentication-Authorization.twiki Line 121 (original) <https://reviews.apache.org/r/67114/#comment285071> missing docs/src/site/twiki/Authentication-Authorization.twiki Line 143 (original) <https://reviews.apache.org/r/67114/#comment285072> of => for docs/src/site/twiki/Authentication-Authorization.twiki Line 164 (original) <https://reviews.apache.org/r/67114/#comment285073> Can this run without Ambari? docs/src/site/twiki/Authentication-Authorization.twiki Line 166 (original) <https://reviews.apache.org/r/67114/#comment285076> I cannot see the link on the Ranger page to its documentation docs/src/site/twiki/Configure-simple-authorizor.twiki Lines 35 (patched) <https://reviews.apache.org/r/67114/#comment285074> white space issue docs/src/site/twiki/Configure-simple-authorizor.twiki Lines 68 (patched) <https://reviews.apache.org/r/67114/#comment285075> white space - David Radley On May 14, 2018, 1:44 p.m., Nixon Rodrigues wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/67114/ > --- > > (Updated May 14, 2018, 1:44 p.m.) > > > Review request for atlas, Apoorv Naik, Ashutosh Mestry, Madhan Neethiraj, and > Sarath Subramanian. > > > Bugs: ATLAS-2463 > https://issues.apache.org/jira/browse/ATLAS-2463 > > > Repository: atlas > > > Description > --- > > This patch adds documentation to Atlas authorization at entity level for > simple authorizer. > > > Diffs > - > > docs/src/site/twiki/Atlas-Authorization-Model.twiki PRE-CREATION > docs/src/site/twiki/Authentication-Authorization.twiki 9832a92e8 > docs/src/site/twiki/Configure-simple-authorizor.twiki PRE-CREATION > docs/src/site/twiki/index.twiki 258dfbb7f > > > Diff: https://reviews.apache.org/r/67114/diff/1/ > > > Testing > --- > > Tested the documentation after running mvn site > > > Thanks, > > Nixon Rodrigues > >
[jira] [Assigned] (ATLAS-2665) Add OMAG-API in Atlas
[ https://issues.apache.org/jira/browse/ATLAS-2665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Radley reassigned ATLAS-2665: --- Assignee: Bogdan Sava > Add OMAG-API in Atlas > - > > Key: ATLAS-2665 > URL: https://issues.apache.org/jira/browse/ATLAS-2665 > Project: Atlas > Issue Type: Improvement >Affects Versions: 1.0.0 >Reporter: Bogdan Sava >Assignee: Bogdan Sava >Priority: Major > Attachments: 0001-ATLAS-2665-Add-OMAG-API-in-Atlas.patch > > > Create configuration for OMAG API in Atlas using Spring MVC Dispatcher > servlet. > Change base URL for the API to "/open-metadata/access-services" > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Review Request 67089: ATLAS-2674 Unable to set an array value for an attribute in the open metadata
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/67089/ --- Review request for atlas, Graham Wallis and MandyDW MandyDW. Bugs: ATLAS2674 https://issues.apache.org/jira/browse/ATLAS2674 Repository: atlas Description --- Unable to set an array value for an attribute in the open metadata Diffs - omrs/src/main/java/org/apache/atlas/omrs/metadatacollection/properties/instances/ArrayPropertyValue.java 154eb16c9 omrs/src/test/java/org/apache/atlas/omrs/metadatacollection/properties/instances/TestArrayPropertyValue.java PRE-CREATION Diff: https://reviews.apache.org/r/67089/diff/1/ Testing --- Ran junits- that failed without the fix. Thanks, David Radley
[jira] [Created] (ATLAS-2674) Unable to set an array value for an attribute in the open metadata
David Radley created ATLAS-2674: --- Summary: Unable to set an array value for an attribute in the open metadata Key: ATLAS-2674 URL: https://issues.apache.org/jira/browse/ATLAS-2674 Project: Atlas Issue Type: Improvement Components: atlas-core Reporter: David Radley Assignee: David Radley Unable to set an array value for an attribute in the open metadata -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ATLAS-2665) Add OMAG-API in Atlas
[ https://issues.apache.org/jira/browse/ATLAS-2665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16471885#comment-16471885 ] David Radley commented on ATLAS-2665: - Hi Bogdan - please can you combine the patches so they can be git pushed as one commit. > Add OMAG-API in Atlas > - > > Key: ATLAS-2665 > URL: https://issues.apache.org/jira/browse/ATLAS-2665 > Project: Atlas > Issue Type: Improvement >Affects Versions: 1.0.0 >Reporter: Bogdan Sava >Priority: Major > Attachments: 0001-ATLAS-2665-Add-OMAG-API-in-Atlas.patch, > 0002-ATLAS-2665-Added-apache-licence-to-properties-file.patch, > 0003-ATLAS-2665-comments-fixed.patch > > > Create configuration for OMAG API in Atlas using Spring MVC Dispatcher > servlet. > Change base URL for the API to "/open-metadata/access-services" > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: Review Request 67060: ATLAS-2668: Add OMAG Server to distribution
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/67060/#review202921 --- ATLAS-2668a.patch Lines 1 (patched) <https://reviews.apache.org/r/67060/#comment284979> looks like a patch has got into the diff - David Radley On May 11, 2018, 11:18 a.m., Nigel Jones wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/67060/ > --- > > (Updated May 11, 2018, 11:18 a.m.) > > > Review request for atlas, David Radley, Graham Wallis, Madhan Neethiraj, and > Mandy Chessell. > > > Repository: atlas > > > Description > --- > > Added OMAG Server to distribution with an easy to launch jar > (See JIRA for more information) > > > Diffs > - > > ATLAS-2668a.patch PRE-CREATION > distro/pom.xml 6431fd86d > distro/src/main/assemblies/omag-server.xml PRE-CREATION > omag-server/README.md PRE-CREATION > omag-server/pom.xml 4c1c98aa3 > > > Diff: https://reviews.apache.org/r/67060/diff/3/ > > > Testing > --- > > Built atlas > Checked correct files in distribution archive > Checked OMAG server launches with default config > > > Thanks, > > Nigel Jones > >
Re: Review Request 67060: ATLAS-2668: Add OMAG Server to distribution
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/67060/#review202920 --- Ship it! Please make the changes then ship it - David Radley On May 11, 2018, 11:18 a.m., Nigel Jones wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/67060/ > --- > > (Updated May 11, 2018, 11:18 a.m.) > > > Review request for atlas, David Radley, Graham Wallis, Madhan Neethiraj, and > Mandy Chessell. > > > Repository: atlas > > > Description > --- > > Added OMAG Server to distribution with an easy to launch jar > (See JIRA for more information) > > > Diffs > - > > ATLAS-2668a.patch PRE-CREATION > distro/pom.xml 6431fd86d > distro/src/main/assemblies/omag-server.xml PRE-CREATION > omag-server/README.md PRE-CREATION > omag-server/pom.xml 4c1c98aa3 > > > Diff: https://reviews.apache.org/r/67060/diff/3/ > > > Testing > --- > > Built atlas > Checked correct files in distribution archive > Checked OMAG server launches with default config > > > Thanks, > > Nigel Jones > >
Re: Review Request 67061: ATLAS-2670: Check validity of entity guid when dissociating term
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/67061/#review202918 --- repository/src/main/java/org/apache/atlas/glossary/GlossaryTermUtils.java Lines 103 (patched) <https://reviews.apache.org/r/67061/#comment284974> should we check that the associated entity and relationship are not deleted? - David Radley On May 10, 2018, 5:07 p.m., Apoorv Naik wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/67061/ > --- > > (Updated May 10, 2018, 5:07 p.m.) > > > Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, and Sarath > Subramanian. > > > Bugs: ATLAS-2670 > https://issues.apache.org/jira/browse/ATLAS-2670 > > > Repository: atlas > > > Description > --- > > See JIRA > > > Diffs > - > > intg/src/main/java/org/apache/atlas/AtlasErrorCode.java 5e3d81846 > repository/src/main/java/org/apache/atlas/glossary/GlossaryTermUtils.java > 5400b77c3 > > > Diff: https://reviews.apache.org/r/67061/diff/1/ > > > Testing > --- > > PreCommit: > https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/377/ > > > Thanks, > > Apoorv Naik > >
Re: Review Request 67060: ATLAS-2668: Add OMAG Server to distribution
Hi Nigel, Thanks for the explanations. I have indicated ship it! Once you make your amendments - I intend to commit this tomorrow if there are no further issues raised today, fyi - I notice you copied the dev list with incubator in the name - I have copied the non-incubator dev list. all the best, David. From: Nigel Jones To: Mandy Chessell , Graham Wallis , Madhan Neethiraj , David Radley Cc: atlas Date: 11/05/2018 10:51 Subject:Re: Review Request 67060: ATLAS-2668: Add OMAG Server to distribution Sent by:Nigel Jones This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/67060/ On May 11th, 2018, 10:39 a.m. BST, David Radley wrote: distro/src/main/assemblies/omag-server.xml (Diff revision 1) 29 I am wondering if you should add and like KafkaHook does. We should ensure that the jar and sh files are executable and the readme is not. I see -rw-r--r-- 1 david staff 1080 11 May 09:48 README.md -rw-r--r-- 1 david staff 8022 11 May 09:55 omag-server-1.0.0-SNAPSHOT-sources.jar -rw-r--r-- 1 david staff 6561 11 May 09:55 omag-server-1.0.0-SNAPSHOT-test-sources.jar -rw-r--r-- 1 david staff 17222951 11 May 09:55 omag-server-1.0.0-SNAPSHOT.jar It doesn to look like the jar is executable. Also why have we got the sources and test sources jar files? On May 11th, 2018, 10:41 a.m. BST, David Radley wrote: As it works - I guess we do not need to make it executable. Please could you review deleting the otehr jars Intriguingly when I build it I do not see those jars in the distribution which only contains the files as per the jira ie ? ~/src/atlas/distro/target/apache-atlas-1.0.0-SNAPSHOT-omag-server/omag-server-1.0.0-SNAPSHOT [master ?·3?·1|…1? 3] 10:49 $ ls README.md omag-server-1.0.0-SNAPSHOT.jar I did check this beforehand.. I'll do a clean build again to double check Note I am referring here to what goes in the distribution (under distro/target)... I think I'd leave what's in the component build (omag-server/target) On May 11th, 2018, 10:39 a.m. BST, David Radley wrote: distro/src/main/assemblies/omag-server.xml (Diff revision 1) 38 ../omag-server I see none of the other assemblies use ../ I am not sure why we need to? This is the link to the location of the artifacts we want to pull into the distibution . Others do use it such as atlas-storm-hook-bridge.xml, so does hbase. Ok? On May 11th, 2018, 10:39 a.m. BST, David Radley wrote: omag-server/README.md (Diff revision 1) 22 **Launching the standalone server** I suggest changing this to be launching the standalone Open Metadata And Governance (OMAG) Server. Fair comment, though as there's no regression here could do in another jira for expediency? Can remake patch once we are happy with other issues? On May 11th, 2018, 10:39 a.m. BST, David Radley wrote: omag-server/README.md (Diff revision 1) 25 If you lose this whitespace - then we will not get thew white space warning on patch apply Odd. hadn't noticed that. I still need to understand how it gets there. Will do. On May 11th, 2018, 10:39 a.m. BST, David Radley wrote: omag-server/pom.xml (Diff revision 1) 124 org.apache.maven.plugins 124 org.springframework.boot I do not understand this change. How does this effect the Atlas build? It's to do with making an executable jar. Previously the maven-jar-plugin was being used, but would not build a jar you could run - nor did it include dependencies. To do that you can normally use the maven-shade-plugin instead (as we see elsewhere in atlas, and as I used for the gaian ranger plugin). Here you can built a composite jar with dependencies, and can also set the main class in the manifest. However in a spring environment this fails as additional configuration data is needed. This is where the spring maven plugin comes to the rescue and builds a correct jar so you can just now run the omag server as per the readme. The JIRA has a little more explanation - Nigel On May 10th, 2018, 5:23 p.m. BST, Nigel Jones wrote: Review request for atlas, David Radley, Graham Wallis, Madhan Neethiraj, and Mandy Chessell. By Nigel Jones. Updated May 10, 2018, 5:23 p.m. Repository: atlas Description Added OMAG Server to distribution with an easy to launch jar (See JIRA for more information) Testing Built atlas Checked correct files in distribution archive Checked OMAG server launches with default config Diffs distro/pom.xml (6431fd86d) distro/src/main/assemblies/omag-server.xml (PRE-CREATION) omag-server/README.md (PRE-CREATION) omag-server/pom.xml (4c1c98aa3) View Diff Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
Re: Review Request 67060: ATLAS-2668: Add OMAG Server to distribution
> On May 11, 2018, 9:39 a.m., David Radley wrote: > > distro/src/main/assemblies/omag-server.xml > > Lines 29 (patched) > > <https://reviews.apache.org/r/67060/diff/1/?file=2019396#file2019396line29> > > > > I am wondering if you should add and > > like KafkaHook does. We should ensure that the jar and > > sh files are executable and the readme is not. > > > > I see > > -rw-r--r-- 1 david staff 1080 11 May 09:48 README.md > > -rw-r--r-- 1 david staff 8022 11 May 09:55 > > omag-server-1.0.0-SNAPSHOT-sources.jar > > -rw-r--r-- 1 david staff 6561 11 May 09:55 > > omag-server-1.0.0-SNAPSHOT-test-sources.jar > > -rw-r--r-- 1 david staff 17222951 11 May 09:55 > > omag-server-1.0.0-SNAPSHOT.jar > > > > > > It doesn to look like the jar is executable. > > Also why have we got the sources and test sources jar files? As it works - I guess we do not need to make it executable. Please could you review deleting the otehr jars - David --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/67060/#review202903 --- On May 10, 2018, 4:23 p.m., Nigel Jones wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/67060/ > --- > > (Updated May 10, 2018, 4:23 p.m.) > > > Review request for atlas, David Radley, Graham Wallis, Madhan Neethiraj, and > Mandy Chessell. > > > Repository: atlas > > > Description > --- > > Added OMAG Server to distribution with an easy to launch jar > (See JIRA for more information) > > > Diffs > - > > distro/pom.xml 6431fd86d > distro/src/main/assemblies/omag-server.xml PRE-CREATION > omag-server/README.md PRE-CREATION > omag-server/pom.xml 4c1c98aa3 > > > Diff: https://reviews.apache.org/r/67060/diff/1/ > > > Testing > --- > > Built atlas > Checked correct files in distribution archive > Checked OMAG server launches with default config > > > Thanks, > > Nigel Jones > >
Re: Review Request 67060: ATLAS-2668: Add OMAG Server to distribution
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/67060/#review202903 --- distro/src/main/assemblies/omag-server.xml Lines 29 (patched) <https://reviews.apache.org/r/67060/#comment284944> I am wondering if you should add and like KafkaHook does. We should ensure that the jar and sh files are executable and the readme is not. I see -rw-r--r-- 1 david staff 1080 11 May 09:48 README.md -rw-r--r-- 1 david staff 8022 11 May 09:55 omag-server-1.0.0-SNAPSHOT-sources.jar -rw-r--r-- 1 david staff 6561 11 May 09:55 omag-server-1.0.0-SNAPSHOT-test-sources.jar -rw-r--r-- 1 david staff 17222951 11 May 09:55 omag-server-1.0.0-SNAPSHOT.jar It doesn to look like the jar is executable. Also why have we got the sources and test sources jar files? distro/src/main/assemblies/omag-server.xml Lines 38 (patched) <https://reviews.apache.org/r/67060/#comment284948> I see none of the other assemblies use ../ I am not sure why we need to? omag-server/README.md Lines 22 (patched) <https://reviews.apache.org/r/67060/#comment284946> I suggest changing this to be launching the standalone Open Metadata And Governance (OMAG) Server. omag-server/README.md Lines 25 (patched) <https://reviews.apache.org/r/67060/#comment284945> If you lose this whitespace - then we will not get thew white space warning on patch apply omag-server/pom.xml Line 124 (original), 124 (patched) <https://reviews.apache.org/r/67060/#comment284947> I do not understand this change. How does this effect the Atlas build? - David Radley On May 10, 2018, 4:23 p.m., Nigel Jones wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/67060/ > --- > > (Updated May 10, 2018, 4:23 p.m.) > > > Review request for atlas, David Radley, Graham Wallis, Madhan Neethiraj, and > Mandy Chessell. > > > Repository: atlas > > > Description > --- > > Added OMAG Server to distribution with an easy to launch jar > (See JIRA for more information) > > > Diffs > - > > distro/pom.xml 6431fd86d > distro/src/main/assemblies/omag-server.xml PRE-CREATION > omag-server/README.md PRE-CREATION > omag-server/pom.xml 4c1c98aa3 > > > Diff: https://reviews.apache.org/r/67060/diff/1/ > > > Testing > --- > > Built atlas > Checked correct files in distribution archive > Checked OMAG server launches with default config > > > Thanks, > > Nigel Jones > >
[jira] [Commented] (ATLAS-2655) [Glossary] Restrict category child to be within same glossary
[ https://issues.apache.org/jira/browse/ATLAS-2655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16467152#comment-16467152 ] David Radley commented on ATLAS-2655: - It can be very useful to be able to have category children in other glossaries. For example re- using a 'standard' glossary categorizations across an organization. If you need to restrict - could you put it under control of a feature toggle. > [Glossary] Restrict category child to be within same glossary > - > > Key: ATLAS-2655 > URL: https://issues.apache.org/jira/browse/ATLAS-2655 > Project: Atlas > Issue Type: Bug >Reporter: Apoorv Naik >Assignee: Apoorv Naik >Priority: Major > Fix For: 1.0.0 > > > Currently no such check exists. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: Review Request 66949: ATLAS-2523: changes to accept external GUIDs and manage homeIds
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66949/#review202433
---
common/src/main/java/org/apache/atlas/repository/Constants.java
Lines 80 (patched)
<https://reviews.apache.org/r/66949/#comment284251>
I suggest a comment here to describe how an Atlas developer should
understand this field.
A concern I have, is that if the users go through the om aPI then the
entities with a non-null homId will never be updated - apart from as part of
the asynchronous cnotification process from anotehr server. The Atlas developer
using the local Atlas API, needs to be aware not to change the entities and
relationships with a none null homeId. Ideally we should police this.
intg/src/main/java/org/apache/atlas/type/AtlasTypeUtil.java
Lines 61 (patched)
<https://reviews.apache.org/r/66949/#comment284248>
I am wondering why you are adding a Logger and not using it om this change.
intg/src/main/java/org/apache/atlas/type/AtlasTypeUtil.java
Line 384 (original), 389 (patched)
<https://reviews.apache.org/r/66949/#comment284249>
I suggest adding a comment here, around why you are checking for '-' as the
first character.
What happens if another repository creates an identifier starting with a
'-' character?
to cope with theis case - I am suggest that the logic should check for a
null homeId and a guid starting character of '-' (or null) we should say the
guid is not assigned.
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasRelationshipStoreV1.java
Line 709 (original), 709 (patched)
<https://reviews.apache.org/r/66949/#comment284250>
why are we leaving in commented out code
- David Radley
On May 4, 2018, 10:55 a.m., Graham Wallis wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66949/
> ---
>
> (Updated May 4, 2018, 10:55 a.m.)
>
>
> Review request for atlas, Ashutosh Mestry and Madhan Neethiraj.
>
>
> Repository: atlas
>
>
> Description
> ---
>
> ATLAS-2523: Changes to accept external GUIDs and manage homeIds
>
>
> Diffs
> -
>
> common/src/main/java/org/apache/atlas/repository/Constants.java
> c570be2ccbc41a426b0f093b4a33163092223f2f
> intg/src/main/java/org/apache/atlas/model/instance/AtlasEntity.java
> e6a7f19633a1642f6b415db99e20c0641df9463b
> intg/src/main/java/org/apache/atlas/model/instance/AtlasRelationship.java
> d04daa5d283fdba90b917f38eba6c937021352df
> intg/src/main/java/org/apache/atlas/type/AtlasTypeUtil.java
> 1b6af94f24eb7d99953f72815167c868a9bd9efd
> repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java
> edf10da9533803234342dc3313ea1024c5e7030f
>
> repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasRelationshipStoreV1.java
> 9fcba6ddaa1b74b2c81b980bcb455b614f4a4ed7
>
> repository/src/main/java/org/apache/atlas/repository/store/graph/v1/EntityGraphMapper.java
> d51adad80e133d636ef84883d395126ae0150af5
>
> repository/src/main/java/org/apache/atlas/repository/store/graph/v1/EntityGraphRetriever.java
> 3b9a287f6986ff6bca26fc20b2f94ec1700dbe1e
>
>
> Diff: https://reviews.apache.org/r/66949/diff/1/
>
>
> Testing
> ---
>
> Functional testing of these changes to save reference copies of entities and
> relationships
>
>
> Thanks,
>
> Graham Wallis
>
>
[jira] [Commented] (ATLAS-2607) Classification lifecycle through metadata properties and relationships
[ https://issues.apache.org/jira/browse/ATLAS-2607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16460928#comment-16460928 ] David Radley commented on ATLAS-2607: - [~madhan.neethiraj] . when you say "Asset can only be attached to classifications with the status of ACTIVE" -is this the general principle? I had assumed that the Asset could be classified with a draft classification, this would exist in the graph but would not be seen by normal queries. We would need new queries that an workflow approver could use to see the draft classifications. You say "Existing entity-classification-update permission will be used to enforce change to status". I suggest that the workflow permissions should be separate so we can have an approver that can only update the status and everyone else can update all but the status. In the open Metadata archive model I notice: - there is a Draft sate in the open metadata TermRelationshipStatus. I wonder if you should consider using these enum values for classifications. - Open metadata also has the GovernanceClassificationStatus which includes states for Governance action classifications. It is worth considering enum states that will ease mapping from open metadata.. > Classification lifecycle through metadata properties and relationships > -- > > Key: ATLAS-2607 > URL: https://issues.apache.org/jira/browse/ATLAS-2607 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Reporter: Srikanth Venkat >Assignee: Madhan Neethiraj >Priority: Critical > > Currently tags or classifications in Atlas are considered active once they > are defined. For governance and stewardship purposes, it would be important > to attach the notion of what state in its lifecycle a particular > classification is. This would help with workflows to manage the lifecycle > aspects and provide any filtering needed to take appropriate actions. For > example only active classifications should be considered for classification > based policy enforcement. Additionally lifecycle status would help with > filtering and search as well as reporting and compliance/audit scenarios. > Implementation Proposal: > * All tags or classifications have a "Lifecycle Status" property > * They can go through the following list of states during their lifecycle: > DRAFT -> ACTIVE -> RETIRED > * Lifecycle Status can be set as an enum property that is mandatory or > required for all classifications. > * All existing classifications already present in Atlas before this change > will default to an ACTIVE status so that all pre-existing classifications > will continue to work as before. > * All new classifications after this change will start out with DRAFT status > and a steward or an admin with appropriate permissions can move them into an > ACTIVE state (controlled via Metadata security policies) > * Policy enforcement for authorization on classifications can ignore any > that are not in ACTIVE state. > * Asset can only be attached to classifications with the status of ACTIVE > * For a classification in RETIRED state, we might have an optional > relationship with another classification called "Replaced By" which is the > new classification that the current one was remapped or replaced with. The > inverse relationship could be labeled "Replaces" which is on the new > classification and points to the removed classification that it replaces. > * The state RETIRED implies this classification is no longer used and policy > enforcement will ignore any classifications in such states. > * Additionally UI can filter and show by default only classifications that > are not RETIRED and a checkbox to "Show Retired" > * Deletion of classifications should work as it currently does with no > behavioral changes. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ATLAS-2139) Map IGC meta model and Atlas OMRS open metadata types
[ https://issues.apache.org/jira/browse/ATLAS-2139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16458681#comment-16458681 ] David Radley commented on ATLAS-2139: - [~sethv] Mandy is not available this week. My thoughts on your questions are: 1. getSkeletonEntity is javadoc says "Return an entity with the header and type information filled out. The caller only needs to add properties and classifications to complete the set up of the entity." The way I am reading this is that I would not expect it to fill in properties, as the caller is expected to do this. 2. The way the OMRS is, means that entities and relationships are created independently, but you need information from more than 1 entity or relationship, in order to issue the IGC call. If I have interpreted this correctly, my suggestion is : - the IGC local connector keeps the entities and relationships until it has enough information (schema rids and the like) to issue the IGC request. Is there a way you can persist this information in IGC. or -The local connector could query IGC for information that is not supplied in an add entity, this may allow you to access the rid information you need. > Map IGC meta model and Atlas OMRS open metadata types > - > > Key: ATLAS-2139 > URL: https://issues.apache.org/jira/browse/ATLAS-2139 > Project: Atlas > Issue Type: Sub-task >Reporter: Jerry He >Assignee: Jerry He >Priority: Major > Attachments: Relational Tables and Columns.png > > > Create the two-way mapping between IGC meta model and Atlas OMRS open > metadata types. > It is used by the IGC Event Mapper and IGC metadata repository connector. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: Cassandra support for Atlas v1.0
Hi Pierre, >From what I understand, the way Apache projects work is that a group in the community may want a certain capability (for example running with hbase and solr). There may be another group in the community who want to run with other capabilities (Cassandra) that the first group may not be interested in. Each group in the community would test, raise and fix issues and make sure that their capability still worked. The committers and PMC decide what is supported,based on things like their confidence on what has been tested. As far as I am aware, none of the committers or PMCs have tested Cassandra or Scylladb in production, and the code change you did and I committed was for an embedded Cassandra build. Embedded Cassandra seems to be a development style of build - embedded hbase is a different build profile from external hbase - which is the hbase profile used in production. So there are no issues I am aware of, and there is one voice in the community who wants Cassandra support. I would also like Cassandra support in Atlas, but I have not put any resources in to doing rigorous testing of it. I suggest adding the embedded Cassandra for development now , this may encourage other members of the community to use Cassandra and raise, test and fix issues around Cassandra; so that production use could then be easily supported. I suggest this is a point in time decision for v1.0, and does not stop people from using Cassandra in production with v1.0 or from Atlas supporting it in the future. I think the community and Apache Atlas is stronger from being inclusive of diverse usages of Atlas, I am interested if you or others have views on my interpretation, all the best, David. From: Pierre Padovani To: dev@atlas.apache.org Date: 27/04/2018 20:18 Subject:Re: Cassandra support for Atlas v1.0 I don't have an issue with #3, but we will be using it in production, specifically with the Scylladb implementation. Out of curiosity, what exactly are the issues? On Fri, Apr 27, 2018 at 10:45 AM, Apoorv Naik wrote: > #3 sounds like a better option for now. +1 > > On 4/27/18, 12:42 AM, "Nigel Jones" wrote: > > > > On 2018/04/26 16:14:06, David Radley wrote: > > > 3) Document it in the same way as the embedded solr and hbase build. > That > > +1 for this option #3 > > If in future someone has a need for production support I'm sure they > will get engaged and help in testing etc :-) > > > > Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
[jira] [Updated] (ATLAS-2622) Add extra checking for open metatdata entity type attributes
[ https://issues.apache.org/jira/browse/ATLAS-2622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Radley updated ATLAS-2622: Description: Jira 2604 added extra checking and units tests to ensure that relationshipdef enddef names do not conflict with each other or with any locally defined attributes. This Jira is raised to ensure that he same checking occurs for when add /update entitytype, add/update relationshipdefs are created via the API. Also this checking should occur when types are replicated between metadata repositories. Ideally this and related name checking should be done in one place in the code for all these scenarios; 2604 updates code involved with the archive types only - I suggest this checking be pushed down into code that is used for the other scenarios if possible. > Add extra checking for open metatdata entity type attributes > > > Key: ATLAS-2622 > URL: https://issues.apache.org/jira/browse/ATLAS-2622 > Project: Atlas > Issue Type: Improvement > Reporter: David Radley >Priority: Major > > Jira 2604 added extra checking and units tests to ensure that relationshipdef > enddef names do not conflict with each other or with any locally defined > attributes. > This Jira is raised to ensure that he same checking occurs for when add > /update entitytype, add/update relationshipdefs are created via the API. > Also this checking should occur when types are replicated between metadata > repositories. > Ideally this and related name checking should be done in one place in the > code for all these scenarios; 2604 updates code involved with the archive > types only - I suggest this checking be pushed down into code that is used > for the other scenarios if possible. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (ATLAS-2622) Add extra checking for open metatdata entity type attributes
David Radley created ATLAS-2622: --- Summary: Add extra checking for open metatdata entity type attributes Key: ATLAS-2622 URL: https://issues.apache.org/jira/browse/ATLAS-2622 Project: Atlas Issue Type: Improvement Reporter: David Radley -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: ATLAS-2569 ATLAS-2571 - commit?
Hi Nigel, I assumed Graham was going to commit ATLAS-2569 as he had put ship it; Graham let me know if this is not the case. I do not see the link to the review board in the other Jira. I can have a look - if you put the link in, all the best, David. From: Nigel Jones To: Date: 27/04/2018 08:45 Subject:ATLAS-2569 ATLAS-2571 - commit? FAO committers: Both of these now have a patch available. One +1 on the submission so far, but both are minor cleanups it would be good to get committed especially as our poms are regularly changing :-) Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
Cassandra support for Atlas v1.0
Hi there, We had a meeting to discuss how we would like to document the embedded Cassandra support in Atlas v1.0. We talked of 3 options : 1) Include it in the code base but do not document. 2) Document it as a tech preview 3) Document it in the same way as the embedded solr and hbase build. That means it will be document as being supported for development, but not in production. My preference is option 3). Are there any objections to this approach? many thanks, David. Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
[jira] [Updated] (ATLAS-2604) Minor fixups for relationshipdefs in the open metadata archive
[ https://issues.apache.org/jira/browse/ATLAS-2604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Radley updated ATLAS-2604: Attachment: ATLAS2604-3.patch > Minor fixups for relationshipdefs in the open metadata archive > --- > > Key: ATLAS-2604 > URL: https://issues.apache.org/jira/browse/ATLAS-2604 > Project: Atlas > Issue Type: Bug > Reporter: David Radley > Assignee: David Radley >Priority: Major > Attachments: ATLAS2604-2.patch, ATLAS2604-3.patch, ATLAS2604.patch > > > Some relationships were introducing the same attribute name into an entity > type. This Jira is to change the relationshipdefs so all the entitydef > attributes are unique. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: Review Request 66784: ATLAS2604 Minor fixups to open metadata archive types
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/66784/ --- (Updated April 26, 2018, 2:37 p.m.) Review request for atlas, Madhan Neethiraj and Mandy Chessell. Bugs: ATLAS2604 https://issues.apache.org/jira/browse/ATLAS2604 Repository: atlas Description --- ATLAS2604 Minor fixups to open metadata archive types to avoid duplicate attributes due to relationship end names. Diffs (updated) - omrs/src/main/java/org/apache/atlas/omrs/archivemanager/OMRSArchiveBuilder.java 5902c4202 omrs/src/main/java/org/apache/atlas/omrs/archivemanager/OMRSArchiveHelper.java 42d2d8746 omrs/src/main/java/org/apache/atlas/omrs/archivemanager/opentypes/OpenMetadataTypesArchive.java faab96cac omrs/src/main/java/org/apache/atlas/omrs/ffdc/OMRSErrorCode.java 351c910e3 omrs/src/test/java/org/apache/atlas/omrs/archivemanager/opentypes/TestOpenMetadataTypesArchive.java PRE-CREATION omrs/src/test/java/org/apache/atlas/omrs/archivemanager/store/TestOMRSArchiveBuilder.java PRE-CREATION Diff: https://reviews.apache.org/r/66784/diff/3/ Changes: https://reviews.apache.org/r/66784/diff/2-3/ Testing --- Ran junit that failed without the patch applied. I ran this in Intellij using the Maven view to run the test phase on the OMRS module. This successfully ran the test. Thanks, David Radley
[jira] [Updated] (ATLAS-2604) Minor fixups for relationshipdefs in the open metadata archive
[ https://issues.apache.org/jira/browse/ATLAS-2604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Radley updated ATLAS-2604: Attachment: ATLAS2604-2.patch > Minor fixups for relationshipdefs in the open metadata archive > --- > > Key: ATLAS-2604 > URL: https://issues.apache.org/jira/browse/ATLAS-2604 > Project: Atlas > Issue Type: Bug > Reporter: David Radley > Assignee: David Radley >Priority: Major > Attachments: ATLAS2604-2.patch, ATLAS2604.patch > > > Some relationships were introducing the same attribute name into an entity > type. This Jira is to change the relationshipdefs so all the entitydef > attributes are unique. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: Review Request 66784: ATLAS2604 Minor fixups to open metadata archive types
> On April 25, 2018, 8:40 a.m., Mandy Chessell wrote: > > omrs/src/test/java/org/apache/atlas/omrs/archivemanager/opentypes/TestOpenMetadataTypesArchive.java > > Lines 35 (patched) > > <https://reviews.apache.org/r/66784/diff/1/?file=2011193#file2011193line35> > > > > This code is not in the most effective place. It only ensures the open > > metadata types do not have duplicate attributes for an entity and only > > after the archive is built. It should be in the archive builder ensuring > > that no archives can be built with duplicate names. It should also be in > > the repository content manager validating that types from different > > archives do not cause an entity to have duplicate properties. This test > > code should be removed and reformatted to run in the main OMRS code. I have redone the tests. - David --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/66784/#review201902 ------- On April 26, 2018, 12:57 p.m., David Radley wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/66784/ > --- > > (Updated April 26, 2018, 12:57 p.m.) > > > Review request for atlas, Madhan Neethiraj and Mandy Chessell. > > > Bugs: ATLAS2604 > https://issues.apache.org/jira/browse/ATLAS2604 > > > Repository: atlas > > > Description > --- > > ATLAS2604 Minor fixups to open metadata archive types to avoid duplicate > attributes due to relationship end names. > > > Diffs > - > > > omrs/src/main/java/org/apache/atlas/omrs/archivemanager/OMRSArchiveBuilder.java > 5902c4202 > > omrs/src/main/java/org/apache/atlas/omrs/archivemanager/OMRSArchiveHelper.java > 42d2d8746 > > omrs/src/main/java/org/apache/atlas/omrs/archivemanager/opentypes/OpenMetadataTypesArchive.java > faab96cac > omrs/src/main/java/org/apache/atlas/omrs/ffdc/OMRSErrorCode.java 351c910e3 > > omrs/src/test/java/org/apache/atlas/omrs/archivemanager/opentypes/TestOpenMetadataTypesArchive.java > PRE-CREATION > > omrs/src/test/java/org/apache/atlas/omrs/archivemanager/store/TestOMRSArchiveBuilder.java > PRE-CREATION > > > Diff: https://reviews.apache.org/r/66784/diff/2/ > > > Testing > --- > > Ran junit that failed without the patch applied. I ran this in Intellij using > the Maven view to run the test phase on the OMRS module. This successfully > ran the test. > > > Thanks, > > David Radley > >
Re: Review Request 66784: ATLAS2604 Minor fixups to open metadata archive types
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/66784/ --- (Updated April 26, 2018, 12:57 p.m.) Review request for atlas, Madhan Neethiraj and Mandy Chessell. Bugs: ATLAS2604 https://issues.apache.org/jira/browse/ATLAS2604 Repository: atlas Description --- ATLAS2604 Minor fixups to open metadata archive types to avoid duplicate attributes due to relationship end names. Diffs (updated) - omrs/src/main/java/org/apache/atlas/omrs/archivemanager/OMRSArchiveBuilder.java 5902c4202 omrs/src/main/java/org/apache/atlas/omrs/archivemanager/OMRSArchiveHelper.java 42d2d8746 omrs/src/main/java/org/apache/atlas/omrs/archivemanager/opentypes/OpenMetadataTypesArchive.java faab96cac omrs/src/main/java/org/apache/atlas/omrs/ffdc/OMRSErrorCode.java 351c910e3 omrs/src/test/java/org/apache/atlas/omrs/archivemanager/opentypes/TestOpenMetadataTypesArchive.java PRE-CREATION omrs/src/test/java/org/apache/atlas/omrs/archivemanager/store/TestOMRSArchiveBuilder.java PRE-CREATION Diff: https://reviews.apache.org/r/66784/diff/2/ Changes: https://reviews.apache.org/r/66784/diff/1-2/ Testing --- Ran junit that failed without the patch applied. I ran this in Intellij using the Maven view to run the test phase on the OMRS module. This successfully ran the test. Thanks, David Radley
[jira] [Updated] (ATLAS-2604) Minor fixups for relationshipdefs in the open metadata archive
[ https://issues.apache.org/jira/browse/ATLAS-2604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Radley updated ATLAS-2604: Description: Some relationships were introducing the same attribute name into an entity type. This Jira is to change the relationshipdefs so all the entitydef attributes are unique. (was: Some relationships were introducing the same attribute name into an entity type. This Jira is to change the relationships so all the entity attributes are unique.) > Minor fixups for relationshipdefs in the open metadata archive > --- > > Key: ATLAS-2604 > URL: https://issues.apache.org/jira/browse/ATLAS-2604 > Project: Atlas > Issue Type: Bug > Reporter: David Radley > Assignee: David Radley >Priority: Major > Attachments: ATLAS2604.patch > > > Some relationships were introducing the same attribute name into an entity > type. This Jira is to change the relationshipdefs so all the entitydef > attributes are unique. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (ATLAS-2604) Minor fixups for relationshipdefs in the open metadata archive
[ https://issues.apache.org/jira/browse/ATLAS-2604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Radley updated ATLAS-2604: Description: Some relationships were introducing the same attribute name into an entity type. This Jira is to change the relationships so all the entity attributes are unique. > Minor fixups for relationshipdefs in the open metadata archive > --- > > Key: ATLAS-2604 > URL: https://issues.apache.org/jira/browse/ATLAS-2604 > Project: Atlas > Issue Type: Bug > Reporter: David Radley > Assignee: David Radley >Priority: Major > > Some relationships were introducing the same attribute name into an entity > type. This Jira is to change the relationships so all the entity attributes > are unique. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (ATLAS-2604) Minor fixups for relationshipdefs in the open metadata archive
[ https://issues.apache.org/jira/browse/ATLAS-2604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Radley updated ATLAS-2604: Attachment: ATLAS2604.patch > Minor fixups for relationshipdefs in the open metadata archive > --- > > Key: ATLAS-2604 > URL: https://issues.apache.org/jira/browse/ATLAS-2604 > Project: Atlas > Issue Type: Bug > Reporter: David Radley > Assignee: David Radley >Priority: Major > Attachments: ATLAS2604.patch > > > Some relationships were introducing the same attribute name into an entity > type. This Jira is to change the relationships so all the entity attributes > are unique. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Review Request 66784: ATLAS2604 Minor fixups to open metadata archive types
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/66784/ --- Review request for atlas, Madhan Neethiraj and Mandy Chessell. Bugs: ATLAS2604 https://issues.apache.org/jira/browse/ATLAS2604 Repository: atlas Description --- ATLAS2604 Minor fixups to open metadata archive types to avoid duplicate attributes due to relationship end names. Diffs - omrs/src/main/java/org/apache/atlas/omrs/archivemanager/opentypes/OpenMetadataTypesArchive.java faab96cac omrs/src/test/java/org/apache/atlas/omrs/archivemanager/opentypes/TestOpenMetadataTypesArchive.java PRE-CREATION Diff: https://reviews.apache.org/r/66784/diff/1/ Testing --- Ran junit that failed without the patch applied. I ran this in Intellij using the Maven view to run the test phase on the OMRS module. This successfully ran the test. Thanks, David Radley
[jira] [Created] (ATLAS-2604) Minor fixups for relationshipdefs in the open metadata archive
David Radley created ATLAS-2604: --- Summary: Minor fixups for relationshipdefs in the open metadata archive Key: ATLAS-2604 URL: https://issues.apache.org/jira/browse/ATLAS-2604 Project: Atlas Issue Type: Bug Reporter: David Radley Assignee: David Radley -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ATLAS-2601) OMAG server Swagger
[ https://issues.apache.org/jira/browse/ATLAS-2601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16449807#comment-16449807 ] David Radley commented on ATLAS-2601: - [~daniela.otelea] . Ideally we should minimize the dependancies and standardise on one library to implement a capability. I am wondering why we need a new swagger dependancy - when we already have a working one knitted in the Atlas buld. . omag-api is in the atlas web server - so this is not completely independent of Atlas. > OMAG server Swagger > > > Key: ATLAS-2601 > URL: https://issues.apache.org/jira/browse/ATLAS-2601 > Project: Atlas > Issue Type: Improvement >Reporter: Maryna Strelchuk >Assignee: Maryna Strelchuk >Priority: Major > Attachments: 0001-Atlas-2601-OMAG-server-Swagger.patch > > > This Jira addresses the addition of the Swagger for the OMAG server. > Currently Swagger was added as a dependency for > * omag-server > * omag-api > * omas-connectedasset -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ATLAS-2601) OMAG server Swagger
[ https://issues.apache.org/jira/browse/ATLAS-2601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16449723#comment-16449723 ] David Radley commented on ATLAS-2601: - The existing Atlas build uses enunciate to generate swagger. Is there a reason to add a second mechanism to generate Swagger? > OMAG server Swagger > > > Key: ATLAS-2601 > URL: https://issues.apache.org/jira/browse/ATLAS-2601 > Project: Atlas > Issue Type: Improvement >Reporter: Maryna Strelchuk >Assignee: Maryna Strelchuk >Priority: Major > Attachments: 0001-Atlas-2601-OMAG-server-Swagger.patch > > > This Jira addresses the addition of the Swagger for the OMAG server. > Currently Swagger was added as a dependency for > * omag-server > * omag-api > * omas-connectedasset -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: Review Request 65435: ATLAS-2298 - Review of OCF Database Connector_New
> On March 26, 2018, 2 p.m., David Radley wrote: > > ocf-database-connector/src/main/java/org/apache/atlas/ocfdbconnector/connectors/gaian/GaianOCFConnector.java > > Lines 212 (patched) > > <https://reviews.apache.org/r/65435/diff/2/?file=1985285#file1985285line212> > > > > should we return this for an error? > > Yao Li wrote: > i am not quite sure what do you mean the finally is clause in java is driven in all cases - i.e even if you throw an error. I suggest this be moved to the end of the try block > On March 26, 2018, 2 p.m., David Radley wrote: > > ocf-database-connector/src/main/resources/gaian.properties > > Lines 6 (patched) > > <https://reviews.apache.org/r/65435/diff/2/?file=1985292#file1985292line6> > > > > It does not look right to include default passwords here, without > > talking about poc / dev scenarios and not production scenarios > > Yao Li wrote: > for now I am still not sure where do we store the information for the > front-end Gaian. For now I just set it up as a properties file. I dont know > these password will be stored in a central properties file or also OCF > Connection for every asset. I will keep this issue open. > > Yao Li wrote: > I have deleted the properties file for the new patch. for now if we want > to use Gaian, The front-end application have to know where the Gaian is. > GaianOCFConnector offer a constructor which contains all the set up for > connecting to Gaian. Thanks Yau - I will wait for the new patch to be uploaded to Jira and into the review board. Then please close out the issues - when you feel they are addressed. - David --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/65435/#review199965 --- On March 30, 2018, 9:33 a.m., Yao Li wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/65435/ > --- > > (Updated March 30, 2018, 9:33 a.m.) > > > Review request for atlas and Mandy Chessell. > > > Repository: atlas > > > Description > --- > > This is the new review request for ATLAS-2298 OCF Database Connector. The old > review on > [https://reviews.apache.org/r/65123/](https://reviews.apache.org/r/65123/) > will not be updated anymore. > > The OCF Database Connector is the subclass of OCF Connector and it is > designed especially for connection to database to retrieve data. > Here we implement a connector for Gaian (GaianOCFConnector) as an example for > using OCF Database Connector. It is related to Open Connector Framework. The > JIRA can be found https://issues.apache.org/jira/browse/ATLAS-2298 > > > Diffs > - > > ocf-database-connector/README.md PRE-CREATION > ocf-database-connector/pom.xml PRE-CREATION > > ocf-database-connector/src/main/java/org/apache/atlas/ocfdbconnector/connectors/OCFDatabaseConnection.java > PRE-CREATION > > ocf-database-connector/src/main/java/org/apache/atlas/ocfdbconnector/connectors/OCFDatabaseConnector.java > PRE-CREATION > > ocf-database-connector/src/main/java/org/apache/atlas/ocfdbconnector/connectors/OCFDatabaseConnectorProviderBase.java > PRE-CREATION > > ocf-database-connector/src/main/java/org/apache/atlas/ocfdbconnector/connectors/gaian/GaianOCFConnector.java > PRE-CREATION > > ocf-database-connector/src/main/java/org/apache/atlas/ocfdbconnector/connectors/gaian/GaianOCFConnectorProvider.java > PRE-CREATION > > ocf-database-connector/src/main/java/org/apache/atlas/ocfdbconnector/ffdc/DatabaseConnectCheckedException.java > PRE-CREATION > > ocf-database-connector/src/main/java/org/apache/atlas/ocfdbconnector/ffdc/OCFDatabaseCheckedExceptionBase.java > PRE-CREATION > > ocf-database-connector/src/main/java/org/apache/atlas/ocfdbconnector/ffdc/OCFDatabaseConnectorErrorCode.java > PRE-CREATION > > ocf-database-connector/src/main/java/org/apache/atlas/ocfdbconnector/ffdc/OCFDatabaseConnectorRuntimeException.java > PRE-CREATION > > ocf-database-connector/src/test/java/gaianocfconnector/GaianOCFConnectorTest.java > PRE-CREATION > > ocf-database-connector/src/test/java/gaianocfconnector/UseGaianOCFConnector.java > PRE-CREATION > pom.xml ee2746a905f84359d426acfb1fa6d64c6310aaaf > > > Diff: https://reviews.apache.org/r/65435/diff/5/ > > > Testing > --- > > see test folder. > Gaian has to set up in advance > > > File Attachments > > > 0005-ATLAS-2298-05-Feb-code-review.patch > > https://reviews.apache.org/media/uploaded/files/2018/02/05/292c1917-9deb-4ced-8bd6-1689fc45fbd4__0005-ATLAS-2298-05-Feb-code-review.patch > > > Thanks, > > Yao Li > >
Re: Review Request 66522: ATLAS-2523: RelationshipStore accepts existing GUID
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/66522/#review200802 --- repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasRelationshipStoreV1.java Lines 570 (patched) <https://reviews.apache.org/r/66522/#comment281692> For the open metadata usage - would it not be better to accept any value for the guid - rather than restricting it to the the Atlas format. I assume the archive type guids are of this form, but what about a relationship type that is mastered in another repository, this may not conform to the Atlas format for guids. In the Entity case the IGC RID may not conform to an assigned or unassigned Atlas guid. I think we need to consistent in how we treat the relationship and entity type guids - and not enforce the Atlas format on supplied guids - as they are likely to not match the Atlas format. - David Radley On April 10, 2018, 9:01 a.m., Graham Wallis wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/66522/ > --- > > (Updated April 10, 2018, 9:01 a.m.) > > > Review request for atlas. > > > Repository: atlas > > > Description > --- > > ATLAS-2523: RelationshipStore accepts existing GUID > > > Diffs > - > > > repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasRelationshipStoreV1.java > 28636d864f28aea97ae50683a84b79a2b29cd166 > > > Diff: https://reviews.apache.org/r/66522/diff/1/ > > > Testing > --- > > During connector development I have been saving relationships with externally > specified GUIDs from FV tests. > > > Thanks, > > Graham Wallis > >
Re: Review Request 66478: ATLAS-2534: Atlas glossary support
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/66478/#review200630 --- addons/models/-Area0/0010-base_model.json Lines 4 (patched) <https://reviews.apache.org/r/66478/#comment281398> I suggest it would be cleaner to have the atlas glossary entiries and relationships in a new model file called atlasglossary.json. addons/models/-Area0/0010-base_model.json Line 105 (original), 180 (patched) <https://reviews.apache.org/r/66478/#comment281399> You are subtyping __internal for the 3 new entities. Open metadata GlossaryTerm and the other glossary entities are actually referencables. I suggest subtypeing Referenceable to __internalReferencable and then using __internalReferencable as the supertype for the 3 atlas entities you are adding. In this way you stay mappable to the open metadata types. I think the qualified name is important for these glossary entities. intg/src/main/java/org/apache/atlas/AtlasErrorCode.java Lines 132 (patched) <https://reviews.apache.org/r/66478/#comment281400> I cannot see this being used - or what the text has to do with glossary intg/src/main/java/org/apache/atlas/model/glossary/relations/AtlasRelatedTermId.java Lines 25 (patched) <https://reviews.apache.org/r/66478/#comment281401> I am wondering why you need to introduce a new custom Atlas annotation repository/src/main/java/org/apache/atlas/glossary/GlossaryService.java Lines 576 (patched) <https://reviews.apache.org/r/66478/#comment281403> why can't they be empty? repository/src/main/java/org/apache/atlas/glossary/GlossaryService.java Lines 590 (patched) <https://reviews.apache.org/r/66478/#comment281402> Is this tracked by a Jira? repository/src/main/java/org/apache/atlas/glossary/GlossaryService.java Lines 602 (patched) <https://reviews.apache.org/r/66478/#comment281405> Unless it was created by the entity API and the relationship has not been created yet. I suggest the error mentions this - as the user would need to know this to be able to fix the issue. repository/src/main/java/org/apache/atlas/glossary/GlossaryService.java Lines 618 (patched) <https://reviews.apache.org/r/66478/#comment281404> Unless it was created by the entity API and the relationship has not been created yet. I suggest the error mentions this - as the user would need to know this to be able to fix the issue. repository/src/main/java/org/apache/atlas/glossary/GlossaryService.java Lines 643 (patched) <https://reviews.apache.org/r/66478/#comment281406> Is the a Jira for this repository/src/main/java/org/apache/atlas/glossary/GlossaryService.java Lines 663 (patched) <https://reviews.apache.org/r/66478/#comment281407> Is there a Jira for this? repository/src/main/java/org/apache/atlas/glossary/GlossaryService.java Lines 678 (patched) <https://reviews.apache.org/r/66478/#comment281408> I suspect you wanted to put some text here. repository/src/main/java/org/apache/atlas/repository/ogm/glossary/AtlasGlossaryCategoryDTO.java Lines 18 (patched) <https://reviews.apache.org/r/66478/#comment281409> what does the ogm package mean? - David Radley On April 5, 2018, 11:57 p.m., Apoorv Naik wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/66478/ > --- > > (Updated April 5, 2018, 11:57 p.m.) > > > Review request for atlas, keval bhatt, Madhan Neethiraj, and Sarath > Subramanian. > > > Bugs: ATLAS-2534 > https://issues.apache.org/jira/browse/ATLAS-2534 > > > Repository: atlas > > > Description > --- > > Glossary implementation, no UI > > > Diffs > - > > addons/models/-Area0/0010-base_model.json aebe955db > > graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphQuery.java > 7bdbeabf0 > > graphdb/common/src/main/java/org/apache/atlas/repository/graphdb/tinkerpop/query/NativeTinkerpopGraphQuery.java > 75665592e > > graphdb/common/src/main/java/org/apache/atlas/repository/graphdb/tinkerpop/query/TinkerpopGraphQuery.java > 96b9705fa > > graphdb/common/src/main/java/org/apache/atlas/repository/graphdb/tinkerpop/query/expr/OrderByPredicate.java > PRE-CREATION > > graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java > 6820a93c2 > > graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/query/NativeJanusGraphQuery.java > d3c976df5 > > graphdb/titan0/src/main/java/org/apache/atlas/re
[jira] [Created] (ATLAS-2537) when running the omag server, by default it create files that interfere with a build
David Radley created ATLAS-2537: --- Summary: when running the omag server, by default it create files that interfere with a build Key: ATLAS-2537 URL: https://issues.apache.org/jira/browse/ATLAS-2537 Project: Atlas Issue Type: Improvement Reporter: David Radley Assignee: David Radley when running the omagapplication in a development environment, by default a config file is created in the root of the workspace. This file causes build failures due to the licence checker and could also be inadvertently checked into git. This Jira is raised to exclude this file in the license checker and to put it into .gitignore. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: Review Request 65885: ATLAS-2470 - Add JanusGraph Cassandra support to Atlas
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/65885/#review200445 --- Ship it! Ship It! - David Radley On March 28, 2018, 3:28 p.m., Pierre Padovani wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/65885/ > --- > > (Updated March 28, 2018, 3:28 p.m.) > > > Review request for atlas and David Radley. > > > Bugs: ATLAS-2470 > https://issues.apache.org/jira/browse/ATLAS-2470 > > > Repository: atlas > > > Description > --- > > This updates pom's to add the needed cassandra jars, and adds a dist profile > embedded-cassandra-solr. > > > Diffs > - > > addons/storm-bridge-shim/pom.xml c25fbfe5 > addons/storm-bridge/pom.xml 2627d92e > distro/pom.xml 0103bef6 > distro/src/bin/atlas_config.py e6415cf4 > distro/src/bin/atlas_start.py 39be6b7c > distro/src/bin/atlas_stop.py 66edd904 > distro/src/conf/atlas-application.properties 10fb7b86 > distro/src/conf/atlas-env.sh 68b24e93 > distro/src/conf/cassandra.yml.template PRE-CREATION > distro/src/conf/zookeeper/log4j.properties PRE-CREATION > distro/src/conf/zookeeper/zoo.cfg.template PRE-CREATION > distro/src/main/assemblies/standalone-package.xml 1881082e > docs/src/site/twiki/InstallationSteps.twiki 6b9f0313 > graphdb/janus/pom.xml 143b775f > pom.xml 9ddc8e18 > repository/pom.xml b1d6b1f9 > > repository/src/main/java/org/apache/atlas/repository/audit/AbstractStorageBasedAuditRepository.java > PRE-CREATION > > repository/src/main/java/org/apache/atlas/repository/audit/CassandraBasedAuditRepository.java > PRE-CREATION > > repository/src/main/java/org/apache/atlas/repository/audit/HBaseBasedAuditRepository.java > cee7374e > > repository/src/test/java/org/apache/atlas/repository/audit/AuditRepositoryTestBase.java > 4d70b7f7 > > repository/src/test/java/org/apache/atlas/repository/audit/CassandraAuditRepositoryTest.java > PRE-CREATION > repository/src/test/resources/cassandra_test.yml PRE-CREATION > > server-api/src/main/java/org/apache/atlas/listener/ActiveStateChangeHandler.java > 7d282b30 > test-tools/src/main/resources/solr/solr.xml 041f0058 > webapp/pom.xml 284f538f > > > Diff: https://reviews.apache.org/r/65885/diff/6/ > > > Testing > --- > > Full build with the new embedded-cassandra-solr, and testing to make sure > Atlas comes up and is functional. > > Be aware that we have been running Cassandra backed JanusGraph for months > with no issues. > > This patch now includes support for Cassandra as an audit store. I've also > refactored the audit unit tests, and added cassandra specific tests. > > > Thanks, > > Pierre Padovani > >
Re: Review Request 66374: ATLAS-2526: misc model updates
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/66374/#review200231 --- addons/models/-Area0/0010-base_model.json Line 205 (original), 205 (patched) <https://reviews.apache.org/r/66374/#comment280891> I suggest there may be migration implications to renaming these base types. I suggest versioning this change. addons/models/0300-Area3-SubjectArea/0310-Glossary.json Lines 89 (patched) <https://reviews.apache.org/r/66374/#comment280889> I thought that we were thinking of these constraints as legacy and had assumed that we would not be adding any new ones. Doesn't the owned constraint imply that this is a one directional relationship, but the relationship definition is bidirectional. What does it mean that there is an attribute that uses the contraint but is not marked as legacy? I am wondering what we gain by adding the attribute in this way? addons/models/0300-Area3-SubjectArea/0320-CategoryHierarchy.json Lines 32 (patched) <https://reviews.apache.org/r/66374/#comment280890> I wonder why we are adding legacy attributes. I can see it gives us a manditory parent; I thought the relationship design was such that we were dropping this capability. - David Radley On March 30, 2018, 8:16 a.m., Madhan Neethiraj wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/66374/ > --- > > (Updated March 30, 2018, 8:16 a.m.) > > > Review request for atlas. > > > Bugs: ATLAS-2526 > https://issues.apache.org/jira/browse/ATLAS-2526 > > > Repository: atlas > > > Description > --- > > - fixed incorrect syntax/braces in 0120-Collections.json that resulted in > relationshipDefs not to be loaded > - renamed sourceToProcesses as inputToProcess and sinkFromProcesses as > outputFromProcess > - added terms and categories as direct attributes of Glossary > - added anchor, parentCategory and childrenCategories as direct attributes of > GlossaryCategory > - added anchor as direct attribute of GlossaryTerm > > > Diffs > - > > addons/models/-Area0/0010-base_model.json aebe955d > addons/models/0100-Area1-Collaboration/0120-Collections.json 4dbd2adb > addons/models/0300-Area3-SubjectArea/0310-Glossary.json cac2ce2a > addons/models/0300-Area3-SubjectArea/0320-CategoryHierarchy.json 10637970 > addons/models/0300-Area3-SubjectArea/0330-Terms.json f492ddfd > > > Diff: https://reviews.apache.org/r/66374/diff/1/ > > > Testing > --- > > Verified Atlas server starts up successfully with updated models > > > Thanks, > > Madhan Neethiraj > >
Re: Review Request 66203: [ATLAS-2510]: Add support to disable/enable propagated classification in entity
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/66203/#review200036 --- repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java Lines 419 (patched) <https://reviews.apache.org/r/66203/#comment280628> If I have understood correctly; I think the proposal here is that a propogated classification can be disabled - this is implmented using a soft delete. Please could you confirm that any downstream propagated classifications are also disabled? - David Radley On March 27, 2018, 5:26 a.m., Sarath Subramanian wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/66203/ > --- > > (Updated March 27, 2018, 5:26 a.m.) > > > Review request for atlas, Apoorv Naik, Ashutosh Mestry, and Madhan Neethiraj. > > > Bugs: ATLAS-2510 > https://issues.apache.org/jira/browse/ATLAS-2510 > > > Repository: atlas > > > Description > --- > > Currently there is no way to remove propagated classification from an entity. > This Jira provides capability to disable/enable a propagated classification > from an entity using REST. > > The propagated classification name and its associated entity guid is supplied > to the REST to disable/enable right propagated classification. > > > Disable a propagated tag: > curl -X PUT > 'http://localhost:21000/api/atlas/v2/entity/guid/0ea86d75-b92c-4907-b149-4fe3fe341cc3/propagatedClassifications/PII?sourceEntityGuid=fb38e88c-c9a8-4fa2-abaa-6be895b067b3&disablePropagation=true' > -H 'Authorization: Basic YWRtaW46YWRtaW4=' > > Enable a propagated tag: > curl -X PUT > 'http://localhost:21000/api/atlas/v2/entity/guid/0ea86d75-b92c-4907-b149-4fe3fe341cc3/propagatedClassifications/PII?sourceEntityGuid=fb38e88c-c9a8-4fa2-abaa-6be895b067b3&disablePropagation=false' > -H 'Authorization: Basic YWRtaW46YWRtaW4=' > > > Diffs > - > > common/src/main/java/org/apache/atlas/repository/Constants.java 605742dd > intg/src/main/java/org/apache/atlas/AtlasErrorCode.java 4a86670c > intg/src/main/java/org/apache/atlas/model/instance/AtlasClassification.java > 008314b3 > intg/src/main/java/org/apache/atlas/model/instance/AtlasEntity.java > fce46da9 > repository/src/main/java/org/apache/atlas/query/GremlinClause.java 454b3434 > repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java > bfb5a711 > > repository/src/main/java/org/apache/atlas/repository/store/graph/AtlasEntityStore.java > 79e8e3e8 > > repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStoreV1.java > b5461d4b > > repository/src/main/java/org/apache/atlas/repository/store/graph/v1/DeleteHandlerV1.java > 19ac6209 > > repository/src/main/java/org/apache/atlas/repository/store/graph/v1/EntityGraphMapper.java > c76f6405 > > repository/src/main/java/org/apache/atlas/repository/store/graph/v1/EntityGraphRetriever.java > 8fe635ae > > repository/src/test/java/org/apache/atlas/query/GremlinQueryComposerTest.java > 9b4c91d1 > webapp/src/main/java/org/apache/atlas/web/rest/EntityREST.java fdafa2c1 > > > Diff: https://reviews.apache.org/r/66203/diff/6/ > > > Testing > --- > > Validated disable and enable of propagated classification through REST client > - POSTMAN. > > https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/213/console > > > Thanks, > > Sarath Subramanian > >
Re: Review Request 65435: ATLAS-2298 - Review of OCF Database Connector_New
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/65435/#review199965 --- ocf-database-connector/README.md Lines 40 (patched) <https://reviews.apache.org/r/65435/#comment280469> bad eol character ocf-database-connector/README.md Lines 42 (patched) <https://reviews.apache.org/r/65435/#comment280472> typo sequene the sentence mentions we and you - I suggest using one or the other. ocf-database-connector/README.md Lines 58 (patched) <https://reviews.apache.org/r/65435/#comment280470> bad eol character ocf-database-connector/README.md Lines 61 (patched) <https://reviews.apache.org/r/65435/#comment280474> I thought impersonation would require a patch on top of Gaian. If this is the case we need to detail where to get this patch for impersonation to be able to be used here. ocf-database-connector/README.md Lines 69 (patched) <https://reviews.apache.org/r/65435/#comment280471> same ocf-database-connector/README.md Lines 71 (patched) <https://reviews.apache.org/r/65435/#comment280473> 'here' is an extra word Typo Gian. We have not mention LDAP, I think we need to be careful mentioning products that might not be installed. We should not mention default passwords here - as in production this would not make sense. ocf-database-connector/README.md Lines 79 (patched) <https://reviews.apache.org/r/65435/#comment280475> This text is assuming that people have access to a Gaian Ranger plugin. This is not currently available. You say "it is better to use **table function**. My understanding is that using the Ranger plugin requires the table function to be used. You say "You need to change cst to your own table name". What is cst? ocf-database-connector/src/main/java/org/apache/atlas/ocfdbconnector/connectors/gaian/GaianOCFConnector.java Lines 35 (patched) <https://reviews.apache.org/r/65435/#comment280477> incomplete sentence ocf-database-connector/src/main/java/org/apache/atlas/ocfdbconnector/connectors/gaian/GaianOCFConnector.java Lines 159 (patched) <https://reviews.apache.org/r/65435/#comment280478> finally is driven for errors as well - so this debug message is not correct. I suggst moving this debug to the end of the try section. ocf-database-connector/src/main/java/org/apache/atlas/ocfdbconnector/connectors/gaian/GaianOCFConnector.java Lines 212 (patched) <https://reviews.apache.org/r/65435/#comment280479> should we return this for an error? ocf-database-connector/src/main/java/org/apache/atlas/ocfdbconnector/connectors/gaian/GaianOCFConnector.java Lines 232 (patched) <https://reviews.apache.org/r/65435/#comment280480> this if and body is a repeat of an previous if. ocf-database-connector/src/main/java/org/apache/atlas/ocfdbconnector/ffdc/OCFDatabaseConnectorErrorCode.java Lines 76 (patched) <https://reviews.apache.org/r/65435/#comment280481> what is the user's valid information and valid use of OCF Connectors? ocf-database-connector/src/main/resources/gaian.properties Lines 6 (patched) <https://reviews.apache.org/r/65435/#comment280482> It does not look right to include default passwords here, without talking about poc / dev scenarios and not production scenarios ocf-database-connector/src/test/java/gaianocfconnector/GaianOCFConnectorTest.java Lines 56 (patched) <https://reviews.apache.org/r/65435/#comment280483> It looks like this class might a be unit test, but it seems that it requires Gaian to be running. If this cannot be run as a unit test - I suggest including instructions on how to run the tests in this file. ocf-database-connector/src/test/java/gaianocfconnector/UseGaianOCFConnector.java Lines 33 (patched) <https://reviews.apache.org/r/65435/#comment280484> I suggest the test throws the excpetions rather than gobbles them. In this way the test fails if there is an error - David Radley On March 22, 2018, 11:23 a.m., Yao Li wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/65435/ > --- > > (Updated March 22, 2018, 11:23 a.m.) > > > Review request for atlas and Mandy Chessell. > > > Repository: atlas > > > Description > --- > > This is the new review request for ATLAS-2298 OCF Database Connector. The old > review on > [https://reviews.apache.org/r/65123/](https://reviews.apache.org/r/65123/) > will not be updated anymore. > > The OCF Database Connector is the subclass of OCF Connector and it is > desig
Re: Review Request 65885: ATLAS-2470 - Add JanusGraph Cassandra support to Atlas
cutor.runWorker(ThreadPoolExecutor.java:1149) [ERROR] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [ERROR] at java.lang.Thread.run(Thread.java:748) [ERROR] Caused by: java.lang.ClassNotFoundException: com.codahale.metrics.MetricRegistry$MetricSupplier [ERROR] at java.net.URLClassLoader.findClass(URLClassLoader.java:381) [ERROR] at java.lang.ClassLoader.loadClass(ClassLoader.java:424) [ERROR] at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335) [ERROR] at java.lang.ClassLoader.loadClass(ClassLoader.java:357) [ERROR] ... 23 more [ERROR] -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn -rf :atlas-repository repository/src/main/java/org/apache/atlas/repository/audit/CassandraBasedAuditRepository.java Lines 200 (patched) <https://reviews.apache.org/r/65885/#comment280468> Do we need this TODO? - David Radley On March 20, 2018, 12:04 a.m., Pierre Padovani wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/65885/ > --- > > (Updated March 20, 2018, 12:04 a.m.) > > > Review request for atlas and David Radley. > > > Bugs: ATLAS-2470 > https://issues.apache.org/jira/browse/ATLAS-2470 > > > Repository: atlas > > > Description > --- > > This updates pom's to add the needed cassandra jars, and adds a dist profile > embedded-cassandra-solr. > > > Diffs > - > > distro/pom.xml 0103bef6 > distro/src/bin/atlas_config.py e6415cf4 > distro/src/bin/atlas_start.py 39be6b7c > distro/src/bin/atlas_stop.py 66edd904 > distro/src/conf/atlas-application.properties 10fb7b86 > distro/src/conf/atlas-env.sh 68b24e93 > distro/src/conf/cassandra.yml.template PRE-CREATION > distro/src/conf/zookeeper/log4j.properties PRE-CREATION > distro/src/conf/zookeeper/zoo.cfg.template PRE-CREATION > distro/src/main/assemblies/standalone-package.xml 1881082e > docs/src/site/twiki/InstallationSteps.twiki 6b9f0313 > graphdb/janus/pom.xml 143b775f > pom.xml 9ddc8e18 > repository/pom.xml b1d6b1f9 > > repository/src/main/java/org/apache/atlas/repository/audit/AbstractStorageBasedAuditRepository.java > PRE-CREATION > > repository/src/main/java/org/apache/atlas/repository/audit/CassandraBasedAuditRepository.java > PRE-CREATION > > repository/src/main/java/org/apache/atlas/repository/audit/HBaseBasedAuditRepository.java > cee7374e > > repository/src/test/java/org/apache/atlas/repository/audit/AuditRepositoryTestBase.java > 4d70b7f7 > > repository/src/test/java/org/apache/atlas/repository/audit/CassandraAuditRepositoryTest.java > PRE-CREATION > repository/src/test/resources/cassandra_test.yml PRE-CREATION > > server-api/src/main/java/org/apache/atlas/listener/ActiveStateChangeHandler.java > 7d282b30 > > > Diff: https://reviews.apache.org/r/65885/diff/4/ > > > Testing > --- > > Full build with the new embedded-cassandra-solr, and testing to make sure > Atlas comes up and is functional. > > Be aware that we have been running Cassandra backed JanusGraph for months > with no issues. > > This patch now includes support for Cassandra as an audit store. I've also > refactored the audit unit tests, and added cassandra specific tests. > > > Thanks, > > Pierre Padovani > >
Re: Review Request 66174: ATLAS-1831 Virtualiser to dynamically configure an information virtualization layer
ps as more than one consumer could be
listening on the topic.
virtualiser/src/main/java/org/apache/atlas/virtualiser/kafka/KafkaVirtualiserConsumer.java
Lines 120 (patched)
<https://reviews.apache.org/r/66174/#comment279886>
delete
virtualiser/src/main/java/org/apache/atlas/virtualiser/views/ViewsConstructor.java
Lines 41 (patched)
<https://reviews.apache.org/r/66174/#comment279873>
I do not think you need to have this variable. As an empty view indicated
we have nothing mapped.
virtualiser/src/main/java/org/apache/atlas/virtualiser/views/ViewsConstructor.java
Lines 47 (patched)
<https://reviews.apache.org/r/66174/#comment279874>
hich typo
virtualiser/src/main/java/org/apache/atlas/virtualiser/views/ViewsConstructor.java
Lines 54 (patched)
<https://reviews.apache.org/r/66174/#comment279876>
No views could mean this method returns null. If there is an exception then
views will be ("",""}. Unless there is a reason I suggest views are null or
have values.
virtualiser/src/main/java/org/apache/atlas/virtualiser/views/ViewsConstructor.java
Lines 58 (patched)
<https://reviews.apache.org/r/66174/#comment279877>
why dont we test if views == null - do we need noneMapped variable
virtualiser/src/main/java/org/apache/atlas/virtualiser/views/ViewsConstructor.java
Lines 63 (patched)
<https://reviews.apache.org/r/66174/#comment279875>
log the exception
virtualiser/src/main/java/org/apache/atlas/virtualiser/views/ViewsConstructor.java
Lines 101 (patched)
<https://reviews.apache.org/r/66174/#comment279879>
delete
virtualiser/src/main/java/org/apache/atlas/virtualiser/views/ViewsConstructor.java
Lines 109 (patched)
<https://reviews.apache.org/r/66174/#comment279880>
delete
virtualiser/src/main/java/org/apache/atlas/virtualiser/views/ViewsConstructor.java
Lines 181 (patched)
<https://reviews.apache.org/r/66174/#comment279872>
buinsess typo
virtualiser/src/main/resources/input.json
Lines 11 (patched)
<https://reviews.apache.org/r/66174/#comment279871>
I am unsure why we have specifc guid values in the main code. I would think
I would see guid values in the test suite but not in the main java.
virtualiser/src/main/resources/virtualiser.properties
Lines 9 (patched)
<https://reviews.apache.org/r/66174/#comment279881>
This file should be a sample as it has specific values in, rather than as
part of the main code.
virtualiser/src/main/resources/virtualiser.properties
Lines 56 (patched)
<https://reviews.apache.org/r/66174/#comment279882>
We hsould not be using the atlas API and there should not be ing urls in
Atlas.
virtualiser/src/test/java/org/apache/atlas/virtualiser/JsonReadHelper.java
Lines 21 (patched)
<https://reviews.apache.org/r/66174/#comment279883>
delete
virtualiser/src/test/java/org/apache/atlas/virtualiser/JsonReadHelper.java
Lines 33 (patched)
<https://reviews.apache.org/r/66174/#comment279884>
It is not usual to put returns in finally sections. I suggest putting it at
the end of the try or after the finally section.
- David Radley
On March 20, 2018, 2:25 p.m., Yao Li wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66174/
> ---
>
> (Updated March 20, 2018, 2:25 p.m.)
>
>
> Review request for atlas, Kees van de Fliert, Mandy Chessell, and Maryna
> Strelchuk.
>
>
> Repository: atlas
>
>
> Description
> ---
>
> Virtualiser has three main funcitons:
> 1. listen to Information View OMAS (information-view-out-topic) and retrieve
> the json file;
> 2. update business logic table and technical logic table, send the updates to
> Gaian;
> 3. create business view json file and technical view json file, notify
> Information View OMAS the update(information-view-in-topic).
>
> The JIRA can be found https://issues.apache.org/jira/browse/ATLAS-1831
>
>
> Diffs
> -
>
> virtualiser/README.md PRE-CREATION
> virtualiser/pom.xml PRE-CREATION
>
> virtualiser/src/main/java/org/apache/atlas/omas/informationview/events/BusinessTerm.java
> PRE-CREATION
>
> virtualiser/src/main/java/org/apache/atlas/omas/informationview/events/ColumnContextEvent.java
> PRE-CREATION
>
> virtualiser/src/main/java/org/apache/atlas/omas/informationview/events/ColumnDetails.java
> PRE-CREATION
>
> virtualiser/src/main/java/org/apache/atlas/omas/informationview/events/Connection.java
> PRE-CREATION
>
> virtualiser/src/main/java/org/apache/atlas/omas/informationview/events/DerivedColumnDetail.java
> PRE-CREATION
>
> virtualiser/src/main/java/org/apache/
Re: Review Request 65885: ATLAS-2470 - Add JanusGraph Cassandra support to Atlas
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/65885/#review199456 --- docs/src/site/twiki/InstallationSteps.twiki Lines 45 (patched) <https://reviews.apache.org/r/65885/#comment279774> I built Atlas as per these instructions. I then started atlas. Atlas started and worked - but when I looked at the application.log there was only 3 log entries. 2018-03-19 12:22:22,447 WARN - [main:] ~ No meta.properties file under dir /Users/david/atlasreview2/distro/target/apache-atlas-1.0.0-SNAPSHOT-bin/apache-atlas-1.0.0-SNAPSHOT/data/kafka/kafka/meta.properties (Logging$class:87) 2018-03-19 12:22:22,922 WARN - [main:] ~ No meta.properties file under dir /Users/david/atlasreview2/distro/target/apache-atlas-1.0.0-SNAPSHOT-bin/apache-atlas-1.0.0-SNAPSHOT/data/kafka/kafka/meta.properties (Logging$class:87) 2018-03-19 12:22:22,986 INFO - [NotificationHookConsumer thread-0:] ~ [atlas-hook-consumer-thread]: Starting (Logging$class:72) I would have expected much more logging. - David Radley On March 19, 2018, 11:51 a.m., Pierre Padovani wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/65885/ > --- > > (Updated March 19, 2018, 11:51 a.m.) > > > Review request for atlas and David Radley. > > > Bugs: ATLAS-2470 > https://issues.apache.org/jira/browse/ATLAS-2470 > > > Repository: atlas > > > Description > --- > > This updates pom's to add the needed cassandra jars, and adds a dist profile > embedded-cassandra-solr. > > > Diffs > - > > distro/pom.xml 0103bef6 > distro/src/bin/atlas_config.py e6415cf4 > distro/src/bin/atlas_start.py 39be6b7c > distro/src/bin/atlas_stop.py 66edd904 > distro/src/conf/atlas-application.properties 10fb7b86 > distro/src/conf/atlas-env.sh 68b24e93 > distro/src/conf/cassandra.yml.template PRE-CREATION > distro/src/conf/zookeeper/log4j.properties PRE-CREATION > distro/src/conf/zookeeper/zoo.cfg.template PRE-CREATION > distro/src/main/assemblies/standalone-package.xml 1881082e > docs/src/site/twiki/InstallationSteps.twiki 6b9f0313 > graphdb/janus/pom.xml 143b775f > repository/pom.xml b1d6b1f9 > > repository/src/main/java/org/apache/atlas/repository/audit/AbstractStorageBasedAuditRepository.java > PRE-CREATION > > repository/src/main/java/org/apache/atlas/repository/audit/CassandraBasedAuditRepository.java > PRE-CREATION > > repository/src/main/java/org/apache/atlas/repository/audit/HBaseBasedAuditRepository.java > cee7374e > > repository/src/test/java/org/apache/atlas/repository/audit/AuditRepositoryTestBase.java > 4d70b7f7 > > repository/src/test/java/org/apache/atlas/repository/audit/CassandraAuditRepositoryTest.java > PRE-CREATION > repository/src/test/resources/cassandra_test.yml PRE-CREATION > > server-api/src/main/java/org/apache/atlas/listener/ActiveStateChangeHandler.java > 7d282b30 > > > Diff: https://reviews.apache.org/r/65885/diff/3/ > > > Testing > --- > > Full build with the new embedded-cassandra-solr, and testing to make sure > Atlas comes up and is functional. > > Be aware that we have been running Cassandra backed JanusGraph for months > with no issues. > > This patch now includes support for Cassandra as an audit store. I've also > refactored the audit unit tests, and added cassandra specific tests. > > > Thanks, > > Pierre Padovani > >
Re: Review Request 65885: ATLAS-2470 - Add JanusGraph Cassandra support to Atlas
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/65885/#review199398 --- distro/src/conf/atlas-application.properties Line 23 (original), 23 (patched) <https://reviews.apache.org/r/65885/#comment279722> Sorry for the delay in reviewing. Unfortunatley the patch now does not apply to master I get: git am < ../ATLAS-2470-2.patch Applying: ATLAS-2470 - JanusGraph Cassandra support .git/rebase-apply/patch:1146: trailing whitespace. # The number of ticks that the initial error: patch failed: distro/src/conf/atlas-application.properties:24 error: distro/src/conf/atlas-application.properties: patch does not apply .git/rebase-apply/patch:1039: new blank line at EOF. + .git/rebase-apply/patch:2761: new blank line at EOF. + Patch failed at 0001 ATLAS-2470 - JanusGraph Cassandra support - David Radley On March 8, 2018, 6:53 p.m., Pierre Padovani wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/65885/ > --- > > (Updated March 8, 2018, 6:53 p.m.) > > > Review request for atlas and David Radley. > > > Bugs: ATLAS-2470 > https://issues.apache.org/jira/browse/ATLAS-2470 > > > Repository: atlas > > > Description > --- > > This updates pom's to add the needed cassandra jars, and adds a dist profile > embedded-cassandra-solr. > > > Diffs > - > > distro/pom.xml 0103bef6 > distro/src/bin/atlas_config.py e6415cf4 > distro/src/bin/atlas_start.py 39be6b7c > distro/src/bin/atlas_stop.py 66edd904 > distro/src/conf/atlas-application.properties 355abf44 > distro/src/conf/atlas-env.sh 68b24e93 > distro/src/conf/cassandra.yml.template PRE-CREATION > distro/src/conf/zookeeper/log4j.properties PRE-CREATION > distro/src/conf/zookeeper/zoo.cfg.template PRE-CREATION > distro/src/main/assemblies/standalone-package.xml 1881082e > docs/src/site/twiki/InstallationSteps.twiki 6b9f0313 > graphdb/janus/pom.xml 143b775f > repository/pom.xml b1d6b1f9 > > repository/src/main/java/org/apache/atlas/repository/audit/AbstractStorageBasedAuditRepository.java > PRE-CREATION > > repository/src/main/java/org/apache/atlas/repository/audit/CassandraBasedAuditRepository.java > PRE-CREATION > > repository/src/main/java/org/apache/atlas/repository/audit/HBaseBasedAuditRepository.java > cee7374e > > repository/src/test/java/org/apache/atlas/repository/audit/AuditRepositoryTestBase.java > 4d70b7f7 > > repository/src/test/java/org/apache/atlas/repository/audit/CassandraAuditRepositoryTest.java > PRE-CREATION > repository/src/test/resources/cassandra_test.yml PRE-CREATION > > server-api/src/main/java/org/apache/atlas/listener/ActiveStateChangeHandler.java > 7d282b30 > > > Diff: https://reviews.apache.org/r/65885/diff/2/ > > > Testing > --- > > Full build with the new embedded-cassandra-solr, and testing to make sure > Atlas comes up and is functional. > > Be aware that we have been running Cassandra backed JanusGraph for months > with no issues. > > This patch now includes support for Cassandra as an audit store. I've also > refactored the audit unit tests, and added cassandra specific tests. > > > Thanks, > > Pierre Padovani > >
Re: Review Request 66111: AtlasClientV2 should support create of relationship
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/66111/#review199317 --- Ship it! Ship It! - David Radley On March 16, 2018, 8:58 a.m., Nigel Jones wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/66111/ > --- > > (Updated March 16, 2018, 8:58 a.m.) > > > Review request for atlas, David Radley, Graham Wallis, and Madhan Neethiraj. > > > Repository: atlas > > > Description > --- > > Add support to Atlas Client v2 api to allow create of relationships > > > Diffs > - > > client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java 92ad923a > > > Diff: https://reviews.apache.org/r/66111/diff/1/ > > > Testing > --- > > See JIRA. > Still testing, but won't be able to complete for a few weeks so posting for > feedback/comments & for others to try :-) > > > Thanks, > > Nigel Jones > >
Re: AtlasClient API v2
Hi Nigel,
I think you are saying this is a regression.
I notice "Add jackson.core dependency in intg module" has just been
committed. Could this make a difference?
Does it work if you recompile?
all the best, David.
From: Nigel Jones
To:
Date: 13/03/2018 13:02
Subject:AtlasClient API v2
I'm trying to use the AtlasClientV2 api but hitting an exception when
trying to create entities.
This is using code from master (client and server) built in the last few
days.
I'm able to send REST APIs directly, also my type creation worked fine.
What do I need to look at to resolve this error? Is it an error I've made
in my types? Could it be a Jackson version issue (I did try updating some
dependencies for my client code to 2.x)
Creating sample entities:Â
2018-03-13 12:43:12 INFO Â AtlasBaseClient:330 -
--
2018-03-13 12:43:12 INFO  AtlasBaseClient:331 - Call     : POST
api/atlas/v2/entity/
2018-03-13 12:43:12 INFO Â AtlasBaseClient:332 - Content-type :
application/json; charset=UTF-8Â
2018-03-13 12:43:12 INFO  AtlasBaseClient:333 - Accept    :
application/jsonÂ
2018-03-13 12:43:12 INFO  AtlasBaseClient:335 - Request    :
AtlasEntityWithExtInfo{entity=AtlasEntity{AtlasStruct{typeName='gaianSchema',
attributes=[qualifiedName:GAIAN, name:GAIAN, comment:Gaian Database
Schema]}guid='-24212680184574', status=null, createdBy='null',
updatedBy='null', createTime=null, updateTime=null, version=0,
relationshipAttributes=[], classifications=[],
},AtlasEntityExtInfo{referredEntities={}}}
Exception in thread "main" java.lang.AbstractMethodError
   at
org.codehaus.jackson.map.AnnotationIntrospector$Pair.findSerializer(AnnotationIntrospector.java:1153)
   at
org.codehaus.jackson.map.ser.BasicSerializerFactory.findSerializerFromAnnotation(BasicSerializerFactory.java:366)
   at
org.codehaus.jackson.map.ser.BeanSerializerFactory.createSerializer(BeanSerializerFactory.java:252)
   at
org.codehaus.jackson.map.ser.StdSerializerProvider._createUntypedSerializer(StdSerializerProvider.java:782)
   at
org.codehaus.jackson.map.ser.StdSerializerProvider._createAndCacheUntypedSerializer(StdSerializerProvider.java:735)
   at
org.codehaus.jackson.map.ser.StdSerializerProvider.findValueSerializer(StdSerializerProvider.java:344)
   at
org.codehaus.jackson.map.ser.StdSerializerProvider.findTypedValueSerializer(StdSerializerProvider.java:420)
   at
org.codehaus.jackson.map.ser.StdSerializerProvider._serializeValue(StdSerializerProvider.java:601)
   at
org.codehaus.jackson.map.ser.StdSerializerProvider.serializeValue(StdSerializerProvider.java:256)
   at
org.codehaus.jackson.map.ObjectMapper.writeValue(ObjectMapper.java:1613)
   at
org.codehaus.jackson.jaxrs.JacksonJsonProvider.writeTo(JacksonJsonProvider.java:559)
   at
com.sun.jersey.json.impl.provider.entity.JacksonProviderProxy.writeTo(JacksonProviderProxy.java:160)
   at
com.sun.jersey.api.client.RequestWriter.writeRequestEntity(RequestWriter.java:300)
   at
com.sun.jersey.client.urlconnection.URLConnectionClientHandler._invoke(URLConnectionClientHandler.java:217)
   at
com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:153)
   at
com.sun.jersey.api.client.filter.HTTPBasicAuthFilter.handle(HTTPBasicAuthFilter.java:105)
   at com.sun.jersey.api.client.Client.handle(Client.java:652)
   at
com.sun.jersey.api.client.WebResource.handle(WebResource.java:682)
   at
com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74)
   at
com.sun.jersey.api.client.WebResource$Builder.method(WebResource.java:634)
   at
org.apache.atlas.AtlasBaseClient.callAPIWithResource(AtlasBaseClient.java:350)
   at
org.apache.atlas.AtlasBaseClient.callAPIWithResource(AtlasBaseClient.java:323)
   at
org.apache.atlas.AtlasBaseClient.callAPI(AtlasBaseClient.java:211)
   at
org.apache.atlas.AtlasClientV2.createEntity(AtlasClientV2.java:285)
   at
org.apache.atlas.gaian.GaianSampleData.createInstance(GaianSampleData.java:176)
   at
org.apache.atlas.gaian.GaianSampleData.createSchema(GaianSampleData.java:198)
   at
org.apache.atlas.gaian.GaianSampleData.createEntities(GaianSampleData.java:155)
   at
org.apache.atlas.gaian.GaianSampleData.main(GaianSampleData.java:72)
Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number
741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
Re: Review Request 65885: ATLAS-2470 - Add JanusGraph Cassandra support to Atlas
> On March 4, 2018, 11:04 a.m., David Radley wrote: > > distro/pom.xml > > Lines 302 (patched) > > <https://reviews.apache.org/r/65885/diff/1/?file=1969111#file1969111line302> > > > > for consistency should we have as well? > > Pierre Padovani wrote: > You cannot just have cassandra without a solr/elasticsearch install > unless you are using DSE (the Datastax Enterprise with integrates with solr). > Based on cursory reading over on JanusGraph, they do not recommend production > deployments of JanusGraph with Cassandra embedded as the performance profiles > are not entirely predictable. Again I would address this in the main > documentation task. Yes I was thinking that cassandra-embedded to differentiate between the production version - maybe you can add this when the extenral cassandra support is put in. - David --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/65885/#review198584 --- On March 2, 2018, 4:51 p.m., Pierre Padovani wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/65885/ > --- > > (Updated March 2, 2018, 4:51 p.m.) > > > Review request for atlas and David Radley. > > > Bugs: ATLAS-2470 > https://issues.apache.org/jira/browse/ATLAS-2470 > > > Repository: atlas > > > Description > --- > > This updates pom's to add the needed cassandra jars, and adds a dist profile > embedded-cassandra-solr. > > > Diffs > - > > distro/pom.xml 0103bef6 > distro/src/bin/atlas_config.py e6415cf4 > distro/src/bin/atlas_start.py 39be6b7c > distro/src/bin/atlas_stop.py 66edd904 > distro/src/conf/cassandra.yml PRE-CREATION > distro/src/conf/zookeeper/log4j.properties PRE-CREATION > distro/src/conf/zookeeper/zoo.cfg PRE-CREATION > distro/src/main/assemblies/standalone-package.xml 1881082e > docs/src/site/twiki/InstallationSteps.twiki 6b9f0313 > graphdb/janus/pom.xml 143b775f > > > Diff: https://reviews.apache.org/r/65885/diff/1/ > > > Testing > --- > > Full build with the new embedded-cassandra-solr, and testing to make sure > Atlas comes up and is functional. > > Be aware that we have been running Cassandra backed JanusGraph for months > with no issues. > > > Thanks, > > Pierre Padovani > >
Re: Review Request 65885: ATLAS-2470 - Add JanusGraph Cassandra support to Atlas
> On March 4, 2018, 11:07 a.m., David Radley wrote: > > My review comments are from my initial look at the code; I hope to try > > running this patch to verify it works for me > > Pierre Padovani wrote: > I'll update the patch with the above changes today or tomorrow as I have > time. > > Pierre Padovani wrote: > I've updated the ATLAS-2470 with a new patch that should fix all of these > issues. > > Pierre Padovani wrote: > Hi David, I think I'm missing something. The instructions on the 'Using > GIT with Atlas' page are unclear on patch revisions. Do I create another > review, or do I somehow update this review? It is best to use the same review for subsequent patches- so then it is possible to compare between versions. I have updated the wiki with this information - thanks. - David --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/65885/#review198593 --- On March 2, 2018, 4:51 p.m., Pierre Padovani wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/65885/ > ------- > > (Updated March 2, 2018, 4:51 p.m.) > > > Review request for atlas and David Radley. > > > Bugs: ATLAS-2470 > https://issues.apache.org/jira/browse/ATLAS-2470 > > > Repository: atlas > > > Description > --- > > This updates pom's to add the needed cassandra jars, and adds a dist profile > embedded-cassandra-solr. > > > Diffs > - > > distro/pom.xml 0103bef6 > distro/src/bin/atlas_config.py e6415cf4 > distro/src/bin/atlas_start.py 39be6b7c > distro/src/bin/atlas_stop.py 66edd904 > distro/src/conf/cassandra.yml PRE-CREATION > distro/src/conf/zookeeper/log4j.properties PRE-CREATION > distro/src/conf/zookeeper/zoo.cfg PRE-CREATION > distro/src/main/assemblies/standalone-package.xml 1881082e > docs/src/site/twiki/InstallationSteps.twiki 6b9f0313 > graphdb/janus/pom.xml 143b775f > > > Diff: https://reviews.apache.org/r/65885/diff/1/ > > > Testing > --- > > Full build with the new embedded-cassandra-solr, and testing to make sure > Atlas comes up and is functional. > > Be aware that we have been running Cassandra backed JanusGraph for months > with no issues. > > > Thanks, > > Pierre Padovani > >
[jira] [Commented] (ATLAS-2478) Elasticsearch support is broken for JanusGraph
[
https://issues.apache.org/jira/browse/ATLAS-2478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16386672#comment-16386672
]
David Radley commented on ATLAS-2478:
-
[~ppadovani] I am not an expert on this part of Atlas. My thought is that most
atlas customer will not have moved to Janus yet, so will need to perform some
sort of migration of existing data; I would think that this sort of change
could be made during the migration.
The only released version of Atlas is an Alpha release, which is not intended
for production. I suggest getting this change into master, so migration scripts
could take account of this as well.
> Elasticsearch support is broken for JanusGraph
> --
>
> Key: ATLAS-2478
> URL: https://issues.apache.org/jira/browse/ATLAS-2478
> Project: Atlas
> Issue Type: Bug
> Components: atlas-core
>Affects Versions: 1.0.0-alpha
>Reporter: Pierre Padovani
>Assignee: Pierre Padovani
>Priority: Major
> Fix For: 1.0.0
>
>
> With JanusGraph the Elasticsearch support moved to 5.x+. This introduced a
> change where fields that contained '.' (dots) were not allowed unless either
> a specific cluster wide setting was enabled AND the mapping was formatted
> such that each of the fields that contained a '.' could be considered part of
> an object.
> Example:
> {code:java}
> foo.x
> foo.y
> foo.z{code}
> Elasticsearch looks at these fields as if they are truly:
> {code:java}
> foo : {
> x,
> y,
> z
> }{code}
> In the file:
> /atlas/common/src/main/java/org/apache/atlas/repository/Constants.java
> {code:java}
> /**
> * Properties for type store graph.
> */
> public static final String TYPE_CATEGORY_PROPERTY_KEY =
> INTERNAL_PROPERTY_KEY_PREFIX + "type.category";
> public static final String VERTEX_TYPE_PROPERTY_KEY =
> INTERNAL_PROPERTY_KEY_PREFIX + "type";
> public static final String TYPENAME_PROPERTY_KEY =
> INTERNAL_PROPERTY_KEY_PREFIX + "type.name";
> public static final String TYPEDESCRIPTION_PROPERTY_KEY =
> INTERNAL_PROPERTY_KEY_PREFIX + "type.description";
> public static final String TYPEVERSION_PROPERTY_KEY =
> INTERNAL_PROPERTY_KEY_PREFIX + "type.version";
> public static final String TYPEOPTIONS_PROPERTY_KEY =
> INTERNAL_PROPERTY_KEY_PREFIX + "type.options";
> {code}
> These are the only fields that cause Elasticsearch issue. As you can see a
> field called 'type' is created, then additional fields type.name,
> type.description etc. This will cause a mapping conflict exception in
> Elasticsearch and it will refuse to create the mapping.
>
> The easy fix is to simply replace the '.' with an '_' (underscore) but this
> will be a backwards incompatible change for existing customers.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
Re: Review Request 65885: ATLAS-2470 - Add JanusGraph Cassandra support to Atlas
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/65885/#review198593 --- My review comments are from my initial look at the code; I hope to try running this patch to verify it works for me - David Radley On March 2, 2018, 4:51 p.m., Pierre Padovani wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/65885/ > --- > > (Updated March 2, 2018, 4:51 p.m.) > > > Review request for atlas and David Radley. > > > Bugs: ATLAS-2470 > https://issues.apache.org/jira/browse/ATLAS-2470 > > > Repository: atlas > > > Description > --- > > This updates pom's to add the needed cassandra jars, and adds a dist profile > embedded-cassandra-solr. > > > Diffs > - > > distro/pom.xml 0103bef6 > distro/src/bin/atlas_config.py e6415cf4 > distro/src/bin/atlas_start.py 39be6b7c > distro/src/bin/atlas_stop.py 66edd904 > distro/src/conf/cassandra.yml PRE-CREATION > distro/src/conf/zookeeper/log4j.properties PRE-CREATION > distro/src/conf/zookeeper/zoo.cfg PRE-CREATION > distro/src/main/assemblies/standalone-package.xml 1881082e > docs/src/site/twiki/InstallationSteps.twiki 6b9f0313 > graphdb/janus/pom.xml 143b775f > > > Diff: https://reviews.apache.org/r/65885/diff/1/ > > > Testing > --- > > Full build with the new embedded-cassandra-solr, and testing to make sure > Atlas comes up and is functional. > > Be aware that we have been running Cassandra backed JanusGraph for months > with no issues. > > > Thanks, > > Pierre Padovani > >
Re: Review Request 65885: ATLAS-2470 - Add JanusGraph Cassandra support to Atlas
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/65885/#review198584 --- distro/pom.xml Lines 258 (patched) <https://reviews.apache.org/r/65885/#comment278782> I see that you are downloading Zookeeper. Many components require zookeeper - so it may already exist on the system. I think for production you would want to run with an external Zookeeper. I suggest we allow this as a build option. I realise this jira is for the embedded Cssandra build only - but I could envisage you needing non-embedded cassandra builds as well; I thought I would suggest this here - if this is a requirement for you, you may decide you want to address this in a separate Jira. distro/pom.xml Lines 302 (patched) <https://reviews.apache.org/r/65885/#comment278783> for consistency should we have as well? distro/src/bin/atlas_start.py Lines 121 (patched) <https://reviews.apache.org/r/65885/#comment278784> For an embedded hbase build, it will use the hbase zk and the embeded solr. I assume this line ( and the matching stop) should not be called for the embedded hbase build (which is not using the downloaded zk). distro/src/conf/cassandra.yml Lines 22 (patched) <https://reviews.apache.org/r/65885/#comment278780> typo distro/src/conf/cassandra.yml Lines 24 (patched) <https://reviews.apache.org/r/65885/#comment278785> I suggest the install could text replace these to the correct values. Is it possible for it to refer to an environment variable with the absolute path, then the script pick up that environment variable? distro/src/conf/zookeeper/zoo.cfg Lines 20 (patched) <https://reviews.apache.org/r/65885/#comment278781> bad end of line character here and a few places below docs/src/site/twiki/InstallationSteps.twiki Lines 38 (patched) <https://reviews.apache.org/r/65885/#comment278786> We did not have to do with before this change for embedded hbase solr. I think these notes should not apply to the embedded hbase solr profile. - David Radley On March 2, 2018, 4:51 p.m., Pierre Padovani wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/65885/ > --- > > (Updated March 2, 2018, 4:51 p.m.) > > > Review request for atlas and David Radley. > > > Bugs: ATLAS-2470 > https://issues.apache.org/jira/browse/ATLAS-2470 > > > Repository: atlas > > > Description > --- > > This updates pom's to add the needed cassandra jars, and adds a dist profile > embedded-cassandra-solr. > > > Diffs > - > > distro/pom.xml 0103bef6 > distro/src/bin/atlas_config.py e6415cf4 > distro/src/bin/atlas_start.py 39be6b7c > distro/src/bin/atlas_stop.py 66edd904 > distro/src/conf/cassandra.yml PRE-CREATION > distro/src/conf/zookeeper/log4j.properties PRE-CREATION > distro/src/conf/zookeeper/zoo.cfg PRE-CREATION > distro/src/main/assemblies/standalone-package.xml 1881082e > docs/src/site/twiki/InstallationSteps.twiki 6b9f0313 > graphdb/janus/pom.xml 143b775f > > > Diff: https://reviews.apache.org/r/65885/diff/1/ > > > Testing > --- > > Full build with the new embedded-cassandra-solr, and testing to make sure > Atlas comes up and is functional. > > Be aware that we have been running Cassandra backed JanusGraph for months > with no issues. > > > Thanks, > > Pierre Padovani > >
[jira] [Commented] (ATLAS-2465) OMRS - Kafka topic connector
[ https://issues.apache.org/jira/browse/ATLAS-2465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16380673#comment-16380673 ] David Radley commented on ATLAS-2465: - Hi [~kanes] , when you want the patch to be reviewed for inclusion into Atlas, please create one patch file and put the same changes into the Review Board tool. The code reviews occur in the Review Board tool, kind regards, David. > OMRS - Kafka topic connector > - > > Key: ATLAS-2465 > URL: https://issues.apache.org/jira/browse/ATLAS-2465 > Project: Atlas > Issue Type: Improvement > Components: atlas-intg >Reporter: Shrinivas Kane >Assignee: Shrinivas Kane >Priority: Minor > Attachments: kafka-topic_connector-part1.patch, > kafka-topic_connector-part2.patch > > > Add topic connector which will > a) listen to incoming kafka message and distribute across listeners > b) send message to kafka topic based on generated events -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ATLAS-2259) Add Janus Graph Cassandra support
[
https://issues.apache.org/jira/browse/ATLAS-2259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16377245#comment-16377245
]
David Radley commented on ATLAS-2259:
-
Hi [~ppadovani], If you submit a patch I will review it and commit it for you
if there are no concerns with it, this means submitting attaching a patch file
to this Jira and raising a rbt review for the code, all the best, David.
> Add Janus Graph Cassandra support
> -
>
> Key: ATLAS-2259
> URL: https://issues.apache.org/jira/browse/ATLAS-2259
> Project: Atlas
> Issue Type: Improvement
> Components: atlas-core
>Affects Versions: 1.0.0
>Reporter: Pierre Padovani
>Assignee: Pierre Padovani
>Priority: Major
> Fix For: 1.0.0
>
>
> Atlas should have support for Cassandra as a backend for Janus available by
> default. If someone wants this type of configuration, they have to modify the
> pom.xml and rebuild Atlas. Here is the pom.xml modification required to
> enable this support:
> {code:java}
>
> org.janusgraph
> janusgraph-cassandra
> ${janus.version}
>
>
> org.codehaus.jettison
> jettison
>
>
> commons-lang
> commons-lang
>
>
>
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (ATLAS-2270) Supported combinations of persistent store and index backend
[ https://issues.apache.org/jira/browse/ATLAS-2270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16374297#comment-16374297 ] David Radley commented on ATLAS-2270: - Hi Pierre, Great; you ask to be a contributer on the dev list - this allows you to assign a Jira to yourself. Then you create a patch in git and attach it to the Jira and put it into the review board for review. There may be review comments or a "ship it". After that a committer (probably me in this case) should take your patch and push it to master. I was new to git and have created a wiki page with the git commands I use https://cwiki.apache.org/confluence/display/ATLAS/Using+Git+with+Atlas. You may find this useful if you have not used git too much. There is a also a wiki page for committers https://cwiki.apache.org/confluence/display/ATLAS/Developer+Resources you may be interested in . I suggest that the Jira should be scoped just to the fix you want to put in. You could change the words of 2270 or create a new one, all the best, David. From: "Pierre Padovani (JIRA)" To: david...@apache.org Date: 22/02/2018 16:44 Subject:[jira] [Commented] (ATLAS-2270) Supported combinations of persistent store and index backend [ https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_ATLAS-2D2270-3Fpage-3Dcom.atlassian.jira.plugin.system.issuetabpanels-3Acomment-2Dtabpanel-26focusedCommentId-3D16373048-23comment-2D16373048&d=DwICaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=QhpUQPr5YlG95aAgCvZGStEXHg4hBbSYQ9JkRqR_svY&m=XQ5cRTBSZfDIanJGha8zM6Dkgnw0IjOYn8x5TlzRFBc&s=CvAzKNiTZy_yOqiqGZadB1tR74aieY28pWb2P8q3ucI&e= ] Pierre Padovani commented on ATLAS-2270: [~davidrad] I'd be happy to be a contributor and get this in. How does one go about becoming one? Are there a set of guidelines somewhere specific to this project? We've been running the Cassandra + ES 5.x flavor of Atlas as both a self contained docker container for dev purposes, as well as a full blown HA cluster for at least the last two months. So I think it looks like a pretty stable setup. https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_ATLAS-2D2270&d=DwICaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=QhpUQPr5YlG95aAgCvZGStEXHg4hBbSYQ9JkRqR_svY&m=XQ5cRTBSZfDIanJGha8zM6Dkgnw0IjOYn8x5TlzRFBc&s=r6UWXjJszzenyZwhDWtpds09zR_aEaPTnA245uXb-vU&e= indexing backend Atlas 1.0.0 (master) should support. This includes building/running Atlas as a standalone package and running UTs/ITs as part of the Atlas build. databases that will be supported in master/1.0.0. This JIRA deliberately ignores titan1 and janusgraph 0.1.1 as the former should be deprecated/removed and the other is a transient state as we get to janusgraph 0.2.0. combinations of persistent store and indexer. It is suggested that this set is kept unchanged: additional combinations. Cassandra is included in this discussion pending response to ATLAS-2259. continued and the remaining 4 combinations, marked with '?', should be considered. There seems to be evidence of people using all 4 of these combinations, although not necessarily with Atlas. possible to build Atlas as a standalone package with any of the combinations - i.e. that they are mutually exclusive and do not interfere with one another. They currently interfere which makes it impossible to build Atlas with -Pdist,berkeley-elasticsearch because the 'dist' profile will exclude jars that are needed by the berkeley-elasticsearch profile - which leads to class not found exceptions when the Atlas server is started. The solution to this could be very simple, or slightly more sophisticated, depending on how many of the combinations we choose to support. -- This message was sent by Atlassian JIRA (v7.6.3#76005) Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU > Supported combinations of persistent store and index backend > > > Key: ATLAS-2270 > URL: https://issues.apache.org/jira/browse/ATLAS-2270 > Project: Atlas > Issue Type: Bug >Reporter: Graham Wallis >Priority: Major > > We need to discuss and decide which combinations of persistent store and > indexing backend Atlas 1.0.0 (master) should support. This includes > building/running Atlas as a standalone package and running UTs/ITs as part of > the Atlas build. > This JIRA focusses on titan0 and janusgraph 0.2.0, as they are the graph > databases that will be supported in master/1.
Re: [jira] [Commented] (ATLAS-2270) Supported combinations of persistent store and index backend
Hi Pierre,
Great; you ask to be a contributer on the dev list - this allows you to
assign a Jira to yourself.
Then you create a patch in git and attach it to the Jira and put it into
the review board for review. There may be review comments or a "ship it".
After that a committer (probably me in this case) should take your patch
and push it to master.
I was new to git and have created a wiki page with the git commands I use
https://cwiki.apache.org/confluence/display/ATLAS/Using+Git+with+Atlas.
You may find this useful if you have not used git too much.
There is a also a wiki page for committers
https://cwiki.apache.org/confluence/display/ATLAS/Developer+Resources you
may be interested in .
I suggest that the Jira should be scoped just to the fix you want to put
in. You could change the words of 2270 or create a new one,
all the best, David.
From: "Pierre Padovani (JIRA)"
To: david...@apache.org
Date: 22/02/2018 16:44
Subject:[jira] [Commented] (ATLAS-2270) Supported combinations of
persistent store and index backend
[
https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_ATLAS-2D2270-3Fpage-3Dcom.atlassian.jira.plugin.system.issuetabpanels-3Acomment-2Dtabpanel-26focusedCommentId-3D16373048-23comment-2D16373048&d=DwICaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=QhpUQPr5YlG95aAgCvZGStEXHg4hBbSYQ9JkRqR_svY&m=XQ5cRTBSZfDIanJGha8zM6Dkgnw0IjOYn8x5TlzRFBc&s=CvAzKNiTZy_yOqiqGZadB1tR74aieY28pWb2P8q3ucI&e=
]
Pierre Padovani commented on ATLAS-2270:
[~davidrad] I'd be happy to be a contributor and get this in. How does one
go about becoming one? Are there a set of guidelines somewhere specific to
this project?
We've been running the Cassandra + ES 5.x flavor of Atlas as both a self
contained docker container for dev purposes, as well as a full blown HA
cluster for at least the last two months. So I think it looks like a
pretty stable setup.
> Supported combinations of persistent store and index backend
>
>
> Key: ATLAS-2270
> URL:
https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_ATLAS-2D2270&d=DwICaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=QhpUQPr5YlG95aAgCvZGStEXHg4hBbSYQ9JkRqR_svY&m=XQ5cRTBSZfDIanJGha8zM6Dkgnw0IjOYn8x5TlzRFBc&s=r6UWXjJszzenyZwhDWtpds09zR_aEaPTnA245uXb-vU&e=
> Project: Atlas
> Issue Type: Bug
>Reporter: Graham Wallis
>Priority: Major
>
> We need to discuss and decide which combinations of persistent store and
indexing backend Atlas 1.0.0 (master) should support. This includes
building/running Atlas as a standalone package and running UTs/ITs as part
of the Atlas build.
> This JIRA focusses on titan0 and janusgraph 0.2.0, as they are the graph
databases that will be supported in master/1.0.0. This JIRA deliberately
ignores titan1 and janusgraph 0.1.1 as the former should be
deprecated/removed and the other is a transient state as we get to
janusgraph 0.2.0.
> With titan0 as the graph provider, Atlas has supported the following
combinations of persistent store and indexer. It is suggested that this
set is kept unchanged:
> {{
> titan0 solr es
>
> berkeley 0 1
> hbase 1 0
> cassandra0 0
> }}
> With janusgraph (0.2.0) as the graph provider, Atlas *could* support
additional combinations. Cassandra is included in this discussion pending
response to ATLAS-2259.
> {{
> janus 0.2.0 solr es
>
> berkeley ? 1
> hbase 1 ?
> cassandra? ?
> }}
> It is suggested that the combinations marked with '1' should be
continued and the remaining 4 combinations, marked with '?', should be
considered. There seems to be evidence of people using all 4 of these
combinations, although not necessarily with Atlas.
> Depending on the decision made above, we need to ensure that it is
possible to build Atlas as a standalone package with any of the
combinations - i.e. that they are mutually exclusive and do not interfere
with one another. They currently interfere which makes it impossible to
build Atlas with -Pdist,berkeley-elasticsearch because the 'dist' profile
will exclude jars that are needed by the berkeley-elasticsearch profile -
which leads to class not found exceptions when the Atlas server is
started. The solution to this could be very simple, or slightly more
sophisticated, depending on how many of the combinations we choose to
support.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number
741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire
[jira] [Commented] (ATLAS-2270) Supported combinations of persistent store and index backend
[
https://issues.apache.org/jira/browse/ATLAS-2270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16373016#comment-16373016
]
David Radley commented on ATLAS-2270:
-
[~ppadovani] - as far as I am aware there is no one working on this at the
moment. The only permutation that works is the Hbase solr one at the moment in
master. You could become a contribute and contribute a fix, which we could
commit. I think in these situations the person who would like it in, needs to
contribute the fix and would then fix any issues in that stack. At this time ,
this stack is not a priority at the moment for the few committers I have talked
to. Is this patch something you want to contribute i.e. provide a patch to
master and submit on the review board and be up for fixing any issues in that
stack. I so I could commit it for you if it does not break the hbase solr
config.
> Supported combinations of persistent store and index backend
>
>
> Key: ATLAS-2270
> URL: https://issues.apache.org/jira/browse/ATLAS-2270
> Project: Atlas
> Issue Type: Bug
>Reporter: Graham Wallis
>Priority: Major
>
> We need to discuss and decide which combinations of persistent store and
> indexing backend Atlas 1.0.0 (master) should support. This includes
> building/running Atlas as a standalone package and running UTs/ITs as part of
> the Atlas build.
> This JIRA focusses on titan0 and janusgraph 0.2.0, as they are the graph
> databases that will be supported in master/1.0.0. This JIRA deliberately
> ignores titan1 and janusgraph 0.1.1 as the former should be
> deprecated/removed and the other is a transient state as we get to janusgraph
> 0.2.0.
> With titan0 as the graph provider, Atlas has supported the following
> combinations of persistent store and indexer. It is suggested that this set
> is kept unchanged:
> {{
> titan0 solr es
>
> berkeley 0 1
> hbase 1 0
> cassandra0 0
> }}
> With janusgraph (0.2.0) as the graph provider, Atlas *could* support
> additional combinations. Cassandra is included in this discussion pending
> response to ATLAS-2259.
> {{
> janus 0.2.0 solr es
>
> berkeley ? 1
> hbase 1 ?
> cassandra? ?
> }}
> It is suggested that the combinations marked with '1' should be continued and
> the remaining 4 combinations, marked with '?', should be considered. There
> seems to be evidence of people using all 4 of these combinations, although
> not necessarily with Atlas.
> Depending on the decision made above, we need to ensure that it is possible
> to build Atlas as a standalone package with any of the combinations - i.e.
> that they are mutually exclusive and do not interfere with one another. They
> currently interfere which makes it impossible to build Atlas with
> -Pdist,berkeley-elasticsearch because the 'dist' profile will exclude jars
> that are needed by the berkeley-elasticsearch profile - which leads to class
> not found exceptions when the Atlas server is started. The solution to this
> could be very simple, or slightly more sophisticated, depending on how many
> of the combinations we choose to support.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
Re: Review Request 65742: ATLAS-2457: updates to support time-boundary in AtlasClassification associations
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/65742/#review198099 --- repository/src/main/java/org/apache/atlas/repository/store/graph/v1/EntityGraphRetriever.java Lines 221 (patched) <https://reviews.apache.org/r/65742/#comment278209> Do we handle 5 classification of the same type against an entity, but with non-overlapping validity periods. repository/src/main/java/org/apache/atlas/repository/store/graph/v1/EntityGraphRetriever.java Lines 222 (patched) <https://reviews.apache.org/r/65742/#comment278208> My feeling is that there should be a new state for the classification that says it is yet to be activated. repository/src/main/java/org/apache/atlas/repository/store/graph/v1/EntityGraphRetriever.java Lines 226 (patched) <https://reviews.apache.org/r/65742/#comment278207> We should have checking to make sure the lower bounds are sooner than the upper bounds. Do we allow the same value for bottom and top. Do we allow adding classification validity periods in the past? Or crossing the present? Maybe we round the time period to a minute to prevent timeing issues for very small time periods. I think we should police all of these bioundary conditions and have unique messages for each case. Holding and specifying the time in UTC would prevent any quirky issues like daylight savings, 29th of Feb is valid only for leap years. - David Radley On Feb. 22, 2018, 12:19 a.m., Madhan Neethiraj wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/65742/ > --- > > (Updated Feb. 22, 2018, 12:19 a.m.) > > > Review request for atlas. > > > Bugs: ATLAS-2457 > https://issues.apache.org/jira/browse/ATLAS-2457 > > > Repository: atlas > > > Description > --- > > Updated AtlasClassification with addition of 'timeBoundary' attribute. > > > Diffs > - > > common/src/main/java/org/apache/atlas/repository/Constants.java ae528807 > intg/src/main/java/org/apache/atlas/model/TimeBoundary.java PRE-CREATION > intg/src/main/java/org/apache/atlas/model/instance/AtlasClassification.java > a499f793 > > repository/src/main/java/org/apache/atlas/repository/store/graph/v1/EntityGraphMapper.java > 0224bf01 > > repository/src/main/java/org/apache/atlas/repository/store/graph/v1/EntityGraphRetriever.java > d01fb9f0 > > webapp/src/test/java/org/apache/atlas/web/integration/EntityV2JerseyResourceIT.java > dabb2efa > > > Diff: https://reviews.apache.org/r/65742/diff/2/ > > > Testing > --- > > - added IT to validate that the new attribute is correctly stored/retrieved > > > Thanks, > > Madhan Neethiraj > >
Re: Relationships to struct fields
Hi Pierre, That is not the way we have implemented the relationships. It is only possible to have relationships between 2 entities. I think this makes sense to restrict to entities. I am interested why you need this and whether there are alternative ways to model what you need, all the best, David. From: Pierre Padovani To: dev@atlas.apache.org Date: 20/02/2018 14:26 Subject:Relationships to struct fields Quick question, as I could not find any documentation that asserts this is a valid use case or not. I have a type with a struct. I'd like to create a relationship from a field in the struct to another type. Is this even possible? Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
[jira] [Commented] (ATLAS-2456) Implement tag propagation using relationships
[
https://issues.apache.org/jira/browse/ATLAS-2456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16369846#comment-16369846
]
David Radley commented on ATLAS-2456:
-
[~sarath.ku...@gmail.com] Thanks for your reply.
>From your responses, is it true that:
* many propagated classifications of the same type can exist for an entity
* a propagated classification can exist with the same type as an explicitly
defined classification
* on point 7 - does this mean that only one classification type or any of its
sub types can be applied to an entity
I suggest:
* we do not propagate a classification of a given type to an entity if there
is already a classification of that type defined (we should account for sub
types here)
* we should not allow more than one classification of a given type against an
entity across explicitly defined classification and propagated classifications.
This means that a policy only has to consider one classification and its
ramifications. I am not sure how we resolve the potential conflicts; ideally
this would be resolved using some rules and a data steward. If we could search
for these conflicts (maybe with a new conflicted status on the entity) , then a
classification could be then explicitly defined on the entity and would force
which classification values were effective.
> Implement tag propagation using relationships
> -
>
> Key: ATLAS-2456
> URL: https://issues.apache.org/jira/browse/ATLAS-2456
> Project: Atlas
> Issue Type: New Feature
> Components: atlas-core
>Affects Versions: 1.0.0
>Reporter: Sarath Subramanian
>Assignee: Sarath Subramanian
>Priority: Major
> Fix For: 1.0.0
>
> Attachments: TAG PROPAGATION IN ATLAS v.1.pdf,
> tag_propagation_rest_api
>
>
> {color:#172b4d}Scalable way to quickly and efficiently propagate tags for
> efficient searches and tag based security. Likewise tags for derivative
> dataset should be inherited from the parent. For example, if an entity is
> tagged "PII" then resulting entity created from a CTAS operation should also
> be tagged "secret" to maintain the classification of the parent. In the case
> where 2 or more datasets are aggregated the derivative dataset should be a
> union of all parent tags.{color}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (ATLAS-2456) Implement tag propagation using relationships
[
https://issues.apache.org/jira/browse/ATLAS-2456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16368955#comment-16368955
]
David Radley commented on ATLAS-2456:
-
[~sarath.ku...@gmail.com] I have not had time to go through the code in a lot
of detail. I think it looks pretty good. I wanted to check how you handle the
following situations:
1) If there is a classification that is applied to an entity with certain
attributes defined, and you decide to propagate the same classification to that
entity, in this situation there must be 2 classification instances - each with
different property values.
So how does the code handle the case when there are 2 instances of the same
classification with different property values that propagate to the same target
entity.
2) I think the classifications defined in the open type
GovernanceActionclassifications
[https://cwiki.apache.org/confluence/display/ATLAS/Area+4+-+Governance] are
going to be the key classifications around governance. There classifications
are defined by their enums- which have an implied low-high order. So for
example confifidentialityLevel of sensitive is higher than internal. For point
1 scenario- we would want the higher value to win.
4) I think that any classification that is explicitly defined on an entity wins
over propagated values.
5) Can you confirm that only one classification of a type is allowed for a
given entity.
6) I assume that the entitytypes defined on a classificationdef will always
constrain - so classification and propagated classification (or their subtypes)
can only ever classify entities of those types (or their subtypes).
7) I assume that a classification and subtypes of a classification could both
be applied to an entity. This could compromise the idea of one classification
per entity. How do you see this working or is this policed as invalid? For
example a Confidentiality classification and its subclass could be applied to
an entity. - each specifying different levels of confidentiality - which one is
effective?
> Implement tag propagation using relationships
> -
>
> Key: ATLAS-2456
> URL: https://issues.apache.org/jira/browse/ATLAS-2456
> Project: Atlas
> Issue Type: New Feature
> Components: atlas-core
>Affects Versions: 1.0.0
>Reporter: Sarath Subramanian
>Assignee: Sarath Subramanian
>Priority: Major
> Fix For: 1.0.0
>
> Attachments: TAG PROPAGATION IN ATLAS v.1.pdf,
> tag_propagation_rest_api
>
>
> {color:#172b4d}Scalable way to quickly and efficiently propagate tags for
> efficient searches and tag based security. Likewise tags for derivative
> dataset should be inherited from the parent. For example, if an entity is
> tagged "PII" then resulting entity created from a CTAS operation should also
> be tagged "secret" to maintain the classification of the parent. In the case
> where 2 or more datasets are aggregated the derivative dataset should be a
> union of all parent tags.{color}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
Re: Review Request 65700: [ATLAS-2456]: Implement tag propagation using relationships
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/65700/#review197732 --- addons/models/-Area0/0010-base_model.json Line 216 (original), 216 (patched) <https://reviews.apache.org/r/65700/#comment277981> you are adding tag propagation from a DataSet to a Process and from a Process to a dataset. I realise these are aggregation relationships, I wonder what prevents circularities. common/src/main/java/org/apache/atlas/repository/Constants.java Lines 72 (patched) <https://reviews.apache.org/r/65700/#comment277982> Can we use classification instead of traits or tags? dashboardv2/public/js/utils/Enums.js Lines 31 (patched) <https://reviews.apache.org/r/65700/#comment277983> agree with the previous reviewer that these tags seem the same as the TAG_ enums repository/src/main/java/org/apache/atlas/repository/store/graph/v1/EntityGraphRetriever.java Lines 1141 (patched) <https://reviews.apache.org/r/65700/#comment277984> I wonder if you could explain what ths creation of a propagation edge means. This implies that the propagated classifications are stored in the edges. I am wondering how this fits with associating the propagated classifications with the downstream entities, which I assume will be stored as propagation classifications in the downstream entities - have /i understood this correctly. A comment to explain how this works would be helpful. - David Radley On Feb. 18, 2018, 8:27 a.m., Sarath Subramanian wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/65700/ > --- > > (Updated Feb. 18, 2018, 8:27 a.m.) > > > Review request for atlas, Apoorv Naik, Ashutosh Mestry, and Madhan Neethiraj. > > > Bugs: ATLAS-2456 > https://issues.apache.org/jira/browse/ATLAS-2456 > > > Repository: atlas > > > Description > --- > > Scalable way to quickly and efficiently propagate tags for efficient searches > and tag based security. Likewise tags for derivative dataset should be > inherited from the parent. For example, if an entity is tagged "PII" then > resulting entity created from a CTAS operation should also be tagged "secret" > to maintain the classification of the parent. In the case where 2 or more > datasets are aggregated the derivative dataset should be a union of all > parent tags. > > This changeset includes introduction of v2 data structures for notifications > and audit events. > > > Diffs > - > > addons/models/-Area0/0010-base_model.json 0296e8f9 > common/src/main/java/org/apache/atlas/repository/Constants.java 265be788 > dashboardv2/public/js/utils/Enums.js a7b9a8b6 > > graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasElement.java > 42837f49 > > graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasVertex.java > a68d8ebe > > graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusVertex.java > aef20f03 > > graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/Titan0Vertex.java > ca48e3d9 > intg/src/main/java/org/apache/atlas/AtlasErrorCode.java ff09e6c9 > intg/src/main/java/org/apache/atlas/listener/EntityChangeListenerV2.java > PRE-CREATION > intg/src/main/java/org/apache/atlas/model/audit/EntityAuditEventV2.java > PRE-CREATION > intg/src/main/java/org/apache/atlas/model/instance/AtlasClassification.java > f594a814 > intg/src/main/java/org/apache/atlas/model/instance/AtlasEntity.java > 08d1ce11 > > intg/src/main/java/org/apache/atlas/model/notification/EntityNotification.java > b272b733 > > intg/src/main/java/org/apache/atlas/v1/model/notification/EntityNotificationV2.java > PRE-CREATION > > repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditListener.java > 74d3b913 > > repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditListenerV2.java > PRE-CREATION > > repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditRepository.java > 9dc78350 > > repository/src/main/java/org/apache/atlas/repository/audit/HBaseBasedAuditRepository.java > 774934c7 > > repository/src/main/java/org/apache/atlas/repository/audit/InMemoryEntityAuditRepository.java > 22d2a810 > > repository/src/main/java/org/apache/atlas/repository/audit/NoopEntityAuditRepository.java > c3826019 > > repository/src/main/java/org/apache/atlas/
Unable to resolve Jiras
Hi Madhan, I have pushed the changed for Jiras 1095 and 2246. I do not seem to be able to resolve these Jiras. Any thoughts? all the best, David. Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
[jira] [Commented] (ATLAS-2246) Define OMRS Connector interface
[ https://issues.apache.org/jira/browse/ATLAS-2246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16366808#comment-16366808 ] David Radley commented on ATLAS-2246: - I have committed the code commit 8a57e6571c0079c25c147732ad3a6994be8e14ea (HEAD -> master, origin/master, origin/HEAD) Author: Mandy Chessell Date: Thu Feb 15 12:46:01 2018 + ATLAS-2246: OMRS Connector API plus REST and IGC Connector skeleton - 15th February 2018 Signed-off-by: David Radley > Define OMRS Connector interface > --- > > Key: ATLAS-2246 > URL: https://issues.apache.org/jira/browse/ATLAS-2246 > Project: Atlas > Issue Type: Sub-task > Components: atlas-intg >Reporter: Mandy Chessell >Assignee: Mandy Chessell >Priority: Critical > Attachments: > 0001-ATLAS-2246-OMRS-Connector-API-plus-REST-and-IGC-Conn.patch, OMRS > Javadoc.zip > > > This task covers the creation of the OMRS Connector APIs. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: Review Request 63502: ATLAS-2246 - Review request for the OMRS Connector API.
> On Feb. 15, 2018, 6:13 p.m., Jerry He wrote: > > Can we put the omrs module in the top level pom so that it builds the omrs > > module? I agree, this is where it needs to go. We were thinking that we would leave it out of the build for now until omrs is more fleshed out. It is really useful for the community developers to have this code checked in, so that can code to it. The Atlas connector is once piece of code that is being developed - this will allow omrs calls to isue requests to Atlas; I think when we have that, we can link it into the build. Personally I have added the line to the pom.xml to bring omrs into my local builds. - David --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/63502/#review197628 --- On Feb. 15, 2018, 7:03 p.m., Mandy Chessell wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/63502/ > --- > > (Updated Feb. 15, 2018, 7:03 p.m.) > > > Review request for atlas, David Radley and Madhan Neethiraj. > > > Repository: atlas > > > Description > --- > > This is a request for code review of the OMRS. There are still parts of the > code not written - the aim of the review is to make the connector interfaces > available so they can be evaluated by teams buinding components against the > APIs. > > There is a patch connected to > https://issues.apache.org/jira/projects/ATLAS/issues/ATLAS-2246 that has the > code plus other code that is not ready for review. The OMRS Connector API > depends on the OCF - see https://issues.apache.org/jira/browse/ATLAS-1095. > > > Diffs > - > > om-fwk-ocf/src/main/java/org/apache/atlas/ocf/ConnectorBase.java cc2e430 > om-fwk-ocf/src/main/java/org/apache/atlas/ocf/ConnectorProviderBase.java > fafd20c > om-fwk-ocf/src/main/java/org/apache/atlas/ocf/properties/Connection.java > d34e6f4 > om-fwk-ocf/src/main/java/org/apache/atlas/ocf/properties/KeyPattern.java > ce6dae1 > omrs/README.md PRE-CREATION > omrs/pom.xml PRE-CREATION > > omrs/src/main/java/org/apache/atlas/omrs/adapters/atlas/eventmapper/AtlasOMRSRepositoryEventMapper.java > PRE-CREATION > > omrs/src/main/java/org/apache/atlas/omrs/adapters/atlas/eventmapper/AtlasOMRSRepositoryEventMapperProvider.java > PRE-CREATION > > omrs/src/main/java/org/apache/atlas/omrs/adapters/atlas/repositoryconnector/LocalAtlasOMRSMetadataCollection.java > PRE-CREATION > > omrs/src/main/java/org/apache/atlas/omrs/adapters/atlas/repositoryconnector/LocalAtlasOMRSRepositoryConnector.java > PRE-CREATION > > omrs/src/main/java/org/apache/atlas/omrs/adapters/atlas/repositoryconnector/LocalAtlasOMRSRepositoryConnectorProvider.java > PRE-CREATION > > omrs/src/main/java/org/apache/atlas/omrs/adapters/igc/v1/eventmapper/IGCOMRSRepositoryEventMapper.java > PRE-CREATION > > omrs/src/main/java/org/apache/atlas/omrs/adapters/igc/v1/eventmapper/IGCOMRSRepositoryEventMapperProvider.java > PRE-CREATION > > omrs/src/main/java/org/apache/atlas/omrs/adapters/igc/v1/repositoryconnector/IGCOMRSMetadataCollection.java > PRE-CREATION > > omrs/src/main/java/org/apache/atlas/omrs/adapters/igc/v1/repositoryconnector/IGCOMRSRepositoryConnector.java > PRE-CREATION > > omrs/src/main/java/org/apache/atlas/omrs/adapters/igc/v1/repositoryconnector/IGCOMRSRepositoryConnectorProvider.java > PRE-CREATION > > omrs/src/main/java/org/apache/atlas/omrs/adapters/igc/v2/eventmapper/IGCV2OMRSRepositoryEventMapper.java > PRE-CREATION > > omrs/src/main/java/org/apache/atlas/omrs/adapters/igc/v2/eventmapper/IGCV2OMRSRepositoryEventMapperProvider.java > PRE-CREATION > > omrs/src/main/java/org/apache/atlas/omrs/adapters/igc/v2/repositoryconnector/IGCV2OMRSMetadataCollection.java > PRE-CREATION > > omrs/src/main/java/org/apache/atlas/omrs/adapters/igc/v2/repositoryconnector/IGCV2OMRSRepositoryConnector.java > PRE-CREATION > > omrs/src/main/java/org/apache/atlas/omrs/adapters/igc/v2/repositoryconnector/IGCV2OMRSRepositoryConnectorProvider.java > PRE-CREATION > > omrs/src/main/java/org/apache/atlas/omrs/admin/OMRSConfigurationFactory.java > PRE-CREATION > omrs/src/main/java/org/apache/atlas/omrs/admin/OMRSOperationalServices.java > PRE-CREATION > omrs/src/main/java/org/apache/atlas/omrs/admin/properties/CohortConfig.java > PRE-CREATION > > omrs/src/main/java/org/apache/atlas/omrs/admin/properties/Enterprise
Re: Metadata security policies examples
Hi Madhan, My thoughts on the 3 points are : 1) It would seem to me that we should have one way of identifying users at the metadata layer and any normalisation or mapping should be done prior to this so that users are uniquely identified in an organization in the policy / metadata layer.. 2) You say "Wouldn't it be more flexible to allow different set of users to edit and classify entities? Similar to the following scenario." I think it is not an 'or ' situation but an 'and'. I think it is beneficial to restrict based on a list of users as you suggest. It is also useful to be able to have some principles as Mandy suggests, so if you have certain access rights then you get other access rights by virtual of a policy. Having these principles expressed in policies means that the policies can be simpler and more naturally expressed. 3) Agreed. all the best, David. From: Madhan Neethiraj To: "dev@atlas.apache.org" Date: 16/02/2018 02:23 Subject:Re: Metadata security policies examples Mandy, > edit access is limited to the user identified in the createdBy property. It will be possible to support authorization as above. However, there are few issues to be aware of in such approach. For example: - 'createdBy' property may not be present in all entity-types - the username in 'createdBy' property might have to be normalized or mapped, to be able to compare with the username logged into Atlas. - for example john.sc...@example.com vs jscott I would recommend to handle such username normalization/mapping in a later phase. > where a classification can only be added to an entity by a user that has edit access to the entity. > where a classification can only be added to any entity by a user with create rights on the classification. Wouldn't it be more flexible to allow different set of users to edit and classify entities? Similar to the following scenario. > where edit access to an entity is required before a relationship can connect it to something else > and it would be good from a graph decoupling point of view if adding relationships could be done independently of the access rights to either entity. I agree on decoupling authorization to create relationship from access rights to the entities at both ends. Thanks, Madhan On 2/14/18, 10:06 AM, "Mandy Chessell" wrote: Hello Madhan, I was thinking through our common use cases for metadata security. For most metadata entities and relationships, we would want to enforce that metadata is readable by logged on users but edit access is limited to the user identified in the createdBy property. Then we have special cases for entities such as connections and some governance actions. For example there may be a connection to an audit log and that can only be seen by members of the security team since having access to the connection means you can connect to the data store. Some governance actions may be updateable by anyone in the governance team - not just the creator. When it comes to classifications, we have 2 scenarios - where a classification can only be added to an entity by a user that has edit access to the entity. - where a classification can only be added to any entity by a user with create rights on the classification. I was trying to think through similar examples for relationships - for example, where edit access to an entity is required before a relationship can connect it to something else - but I can't think of one - and it would be good from a graph decoupling point of view if adding relationships could be done independently of the access rights to either entity. All the best Mandy ___ Mandy Chessell CBE FREng CEng FBCS IBM Distinguished Engineer Master Inventor Member of the IBM Academy of Technology Visiting Professor, Department of Computer Science, University of Sheffield Email: mandy_chess...@uk.ibm.com LinkedIn: https://urldefense.proofpoint.com/v2/url?u=http-3A__www.linkedin.com_pub_mandy-2Dchessell_22_897_a49&d=DwIFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=QhpUQPr5YlG95aAgCvZGStEXHg4hBbSYQ9JkRqR_svY&m=Qa-FKldwlfxLK0OygoMR9IopavMa6ccJ81Xtg4l11cs&s=20HZiMDEZtwudUrz9j12W32Kb8AklTGhbNDPAg5AN6s&e= Assistant: Janet Brooks - jsbrook...@uk.ibm.com Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
Attempting to run Atlas with external HBase
Hi Madhan, Based on our meeting earlier this week, you suggested that we run with an external HBase rather than the embedded HBase, which you have found has delays around some transaction commits. I am not very familiar with HBase and wondered if you could point me in the right direction. This is what I did: 1) I ran an embedded hbase build and found the solr and hbase tar.gz files 2) I then expanded these archives in a new runtime folder 3) I built Atlas without the embedded hbase option and then copied over the atlas tree into the runtime folder. 4) I found https://atlas.apache.org/InstallationSteps.html . - It indicates I should specify export ATLAS_SERVER_OPTS="-server -XX:SoftRefLRUPolicyMSPerMB=0 -XX:+CMSClassUnloadingEnabled -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:+PrintTenuringDistribution -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=dumps/atlas_server.hprof -Xloggc:logs/gc-worker.log -verbose:gc -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=1m -XX:+PrintGCDetails -XX:+PrintHeapAtGC -XX:+PrintGCTimeStamps" which I put in the atlas_env.sh - As I am on a mac it suggests I specify : export ATLAS_SERVER_OPTS="-Djava.awt.headless=true -Djava.security.krb5.realm= -Djava.security.krb5.kdc=" which I did. - It says I should change the config to: "atlas.graph.storage.hbase.table=atlas atlas.audit.hbase.tablename=apache_atlas_entity_audit " Is this correct - as the application.properties that has been generated is "atlas.graph.storage.hbase.table=apache_atlas_titan". - It says I should start solr with "$SOLR_HOME/bin/solr start -c - z -p 8983". I do not know what to put in for < zookeeper_host:port>. Do I need to specify this if I am using the solr embedded zookeeper? - It then says I should run " $SOLR_BIN/solr create -c vertex_index -d SOLR_CONF -shards #numShards -replicationFactor #replicationFactor $SOLR_BIN/solr create -c edge_index -d SOLR_CONF -shards #numShards -replicationFactor #replicationFactor $SOLR_BIN/solr create -c fulltext_index -d SOLR_CONF -shards #numShards -replicationFactor #replicationFactor" It seems I need to specify numbers for numshards and replicationFactor, can I let these default, if not what do I specify here? - It then says I need to specify "atlas.graph.index.search.backend=solr5 atlas.graph.index.search.solr.mode=cloud atlas.graph.index.search.solr.zookeeper-url= eg: 10.1.6.4:2181,10.1.6.5:2181 atlas.graph.index.search.solr.zookeeper-connect-timeout=. Default value is 6 ms atlas.graph.index.search.solr.zookeeper-session-timeout=. Default value is 6 ms" I am not sure what to put for atlas.graph.index.search.solr.zookeeper-url. should we be using the solr embedded ZK - if do I need this line? I updated HBASE_CONF_DIR to point to the conf folder of the hbase I had expanded. I start solr using $SOLR_HOME/bin/solr start -c -p 8983 I start hbase I start atlas. Atlas says it successfully has started but the last line in the application.log says : 2018-02-15 17:25:23,755 INFO - [main:] ~ Not running setup per configuration atlas.server.run.setup.on.start. (SetupSteps$SetupRequired:189) The installation twiki talks of "If the setup failed due to HBase JanusGraph schema setup errors, it may be necessary to repair the HBase schema. If no data has been stored, one can also disable and drop the HBase tables used by Atlas and run setup again." . It does not indicate what commands I need to run and how. many thanks , David. Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
Re: Review Request 63502: ATLAS-2246 - Review request for the OMRS Connector API.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/63502/#review197619 --- Ship it! Ship It! - David Radley On Feb. 15, 2018, 12:49 p.m., Mandy Chessell wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/63502/ > --- > > (Updated Feb. 15, 2018, 12:49 p.m.) > > > Review request for atlas, David Radley and Madhan Neethiraj. > > > Repository: atlas > > > Description > --- > > This is a request for code review of the OMRS. There are still parts of the > code not written - the aim of the review is to make the connector interfaces > available so they can be evaluated by teams buinding components against the > APIs. > > There is a patch connected to > https://issues.apache.org/jira/projects/ATLAS/issues/ATLAS-2246 that has the > code plus other code that is not ready for review. The OMRS Connector API > depends on the OCF - see https://issues.apache.org/jira/browse/ATLAS-1095. > > > Diffs > - > > om-fwk-ocf/src/main/java/org/apache/atlas/ocf/ConnectorBase.java cc2e430 > om-fwk-ocf/src/main/java/org/apache/atlas/ocf/ConnectorProviderBase.java > fafd20c > om-fwk-ocf/src/main/java/org/apache/atlas/ocf/properties/Connection.java > d34e6f4 > om-fwk-ocf/src/main/java/org/apache/atlas/ocf/properties/KeyPattern.java > ce6dae1 > omrs/README.md PRE-CREATION > omrs/pom.xml PRE-CREATION > > omrs/src/main/java/org/apache/atlas/omrs/adapters/atlas/eventmapper/AtlasOMRSRepositoryEventMapper.java > PRE-CREATION > > omrs/src/main/java/org/apache/atlas/omrs/adapters/atlas/eventmapper/AtlasOMRSRepositoryEventMapperProvider.java > PRE-CREATION > > omrs/src/main/java/org/apache/atlas/omrs/adapters/atlas/repositoryconnector/LocalAtlasOMRSMetadataCollection.java > PRE-CREATION > > omrs/src/main/java/org/apache/atlas/omrs/adapters/atlas/repositoryconnector/LocalAtlasOMRSRepositoryConnector.java > PRE-CREATION > > omrs/src/main/java/org/apache/atlas/omrs/adapters/atlas/repositoryconnector/LocalAtlasOMRSRepositoryConnectorProvider.java > PRE-CREATION > > omrs/src/main/java/org/apache/atlas/omrs/adapters/igc/v1/eventmapper/IGCOMRSRepositoryEventMapper.java > PRE-CREATION > > omrs/src/main/java/org/apache/atlas/omrs/adapters/igc/v1/eventmapper/IGCOMRSRepositoryEventMapperProvider.java > PRE-CREATION > > omrs/src/main/java/org/apache/atlas/omrs/adapters/igc/v1/repositoryconnector/IGCOMRSMetadataCollection.java > PRE-CREATION > > omrs/src/main/java/org/apache/atlas/omrs/adapters/igc/v1/repositoryconnector/IGCOMRSRepositoryConnector.java > PRE-CREATION > > omrs/src/main/java/org/apache/atlas/omrs/adapters/igc/v1/repositoryconnector/IGCOMRSRepositoryConnectorProvider.java > PRE-CREATION > > omrs/src/main/java/org/apache/atlas/omrs/adapters/igc/v2/eventmapper/IGCV2OMRSRepositoryEventMapper.java > PRE-CREATION > > omrs/src/main/java/org/apache/atlas/omrs/adapters/igc/v2/eventmapper/IGCV2OMRSRepositoryEventMapperProvider.java > PRE-CREATION > > omrs/src/main/java/org/apache/atlas/omrs/adapters/igc/v2/repositoryconnector/IGCV2OMRSMetadataCollection.java > PRE-CREATION > > omrs/src/main/java/org/apache/atlas/omrs/adapters/igc/v2/repositoryconnector/IGCV2OMRSRepositoryConnector.java > PRE-CREATION > > omrs/src/main/java/org/apache/atlas/omrs/adapters/igc/v2/repositoryconnector/IGCV2OMRSRepositoryConnectorProvider.java > PRE-CREATION > > omrs/src/main/java/org/apache/atlas/omrs/admin/OMRSConfigurationFactory.java > PRE-CREATION > omrs/src/main/java/org/apache/atlas/omrs/admin/OMRSOperationalServices.java > PRE-CREATION > omrs/src/main/java/org/apache/atlas/omrs/admin/properties/CohortConfig.java > PRE-CREATION > > omrs/src/main/java/org/apache/atlas/omrs/admin/properties/EnterpriseAccessConfig.java > PRE-CREATION > > omrs/src/main/java/org/apache/atlas/omrs/admin/properties/LocalRepositoryConfig.java > PRE-CREATION > > omrs/src/main/java/org/apache/atlas/omrs/admin/properties/OpenMetadataEventProtocolVersion.java > PRE-CREATION > > omrs/src/main/java/org/apache/atlas/omrs/admin/properties/OpenMetadataExchangeRule.java > PRE-CREATION > > omrs/src/main/java/org/apache/atlas/omrs/admin/properties/RepositoryServicesConfig.java > PRE-CREATION > > omrs/src/main/java/org/apache/atlas/omrs/archivemanager/OMRSArchiveBuilder.java > PRE-CREATI
[jira] [Created] (ATLAS-2449) Latest master atlas does not build cleanly with tests enabled locally on Mac
David Radley created ATLAS-2449: --- Summary: Latest master atlas does not build cleanly with tests enabled locally on Mac Key: ATLAS-2449 URL: https://issues.apache.org/jira/browse/ATLAS-2449 Project: Atlas Issue Type: Bug Reporter: David Radley I run mvn clean install -Pdist,embedded-hbase-solr And it fails with: 127.0.0.1 - - [15/Feb/2018:14:06:01 +] "GET //localhost:31000/api/atlas/v2/types/typedef/name/hive_table_v1 HTTP/1.1" 503 372 "-" "Java/1.8.0_151" 127.0.0.1 - - [15/Feb/2018:14:06:06 +] "GET //localhost:31000/api/atlas/v2/types/typedef/name/hive_table_v1 HTTP/1.1" 503 372 "-" "Java/1.8.0_151" 127.0.0.1 - - [15/Feb/2018:14:06:11 +] "GET //localhost:31000/api/atlas/v2/types/typedef/name/hive_process_v1 HTTP/1.1" 503 374 "-" "Java/1.8.0_151" 127.0.0.1 - - [15/Feb/2018:14:06:16 +] "GET //localhost:31000/api/atlas/v2/types/typedef/name/hive_process_v1 HTTP/1.1" 503 374 "-" "Java/1.8.0_151" 127.0.0.1 - - [15/Feb/2018:14:06:21 +] "GET //localhost:31000/api/atlas/v2/types/typedef/name/hive_process_v1 HTTP/1.1" 503 374 "-" "Java/1.8.0_151" 127.0.0.1 - - [15/Feb/2018:14:06:26 +] "GET //localhost:31000/api/atlas/v2/types/typedef/name/hive_process_v1 HTTP/1.1" 503 374 "-" "Java/1.8.0_151" 127.0.0.1 - - [15/Feb/2018:14:06:31 +] "POST //localhost:31000/api/atlas/types HTTP/1.1" 503 342 "-" "Java/1.8.0_151" 127.0.0.1 - - [15/Feb/2018:14:06:36 +] "POST //localhost:31000/api/atlas/types HTTP/1.1" 503 342 "-" "Java/1.8.0_151" 127.0.0.1 - - [15/Feb/2018:14:06:41 +] "POST //localhost:31000/api/atlas/types HTTP/1.1" 503 342 "-" "Java/1.8.0_151" 127.0.0.1 - - [15/Feb/2018:14:06:46 +] "POST //localhost:31000/api/atlas/types HTTP/1.1" 503 342 "-" "Java/1.8.0_151" Tests run: 143, Failures: 11, Errors: 0, Skipped: 125, Time elapsed: 2,422.962 sec <<< FAILURE! - in TestSuite setUp(org.apache.atlas.web.integration.DataSetLineageJerseyResourceIT) Time elapsed: 510.086 sec <<< FAILURE! org.apache.atlas.AtlasServiceException: Metadata service API org.apache.atlas.AtlasClient$API_V1@5161ae92 failed with status 503 (Service Unavailable) Response Body ( Error 503 HTTP ERROR: 503 Problem accessing /api/atlas/types. Reason: Service Unavailable http://eclipse.org/jetty";>Powered by Jetty:// 9.3.14.v20161028 ) at org.apache.atlas.AtlasBaseClient.callAPIWithResource(AtlasBaseClient.java:388) at org.apache.atlas.AtlasBaseClient.callAPIWithResource(AtlasBaseClient.java:323) at org.apache.atlas.AtlasBaseClient.callAPI(AtlasBaseClient.java:211) at org.apache.atlas.AtlasClient.callAPIWithBody(AtlasClient.java:906) at org.apache.atlas.AtlasClient.createType(AtlasClient.java:257) at org.apache.atlas.AtlasClient.createType(AtlasClient.java:275) at org.apache.atlas.web.integration.BaseResourceIT.createType(BaseResourceIT.java:213) at org.apache.atlas.web.integration.BaseResourceIT.createTypeDefinitionsV1(BaseResourceIT.java:360) at org.apache.atlas.web.integration.DataSetLineageJerseyResourceIT.setUp(DataSetLineageJerseyResourceIT.java:60) setUp(org.apache.atlas.notification.NotificationHookConsumerIT) Time elapsed: 362.578 sec <<< FAILURE! org.apache.atlas.AtlasServiceException: Metadata service API org.apache.atlas.AtlasClient$API_V1@5161ae92 failed with status 503 (Service Unavailable) Response Body ( Error 503 HTTP ERROR: 503 Problem accessing /api/atlas/types. Reason: Service Unavailable http://eclipse.org/jetty";>Powered by Jetty:// 9.3.14.v20161028 ) at org.apache.atlas.AtlasBaseClient.callAPIWithResource(AtlasBaseClient.java:388) at org.apache.atlas.AtlasBaseClient.callAPIWithResource(AtlasBaseClient.java:323) at org.apache.atlas.AtlasBaseClient.callAPI(AtlasBaseClient.java:211) at org.apache.atlas.AtlasClient.callAPIWithBody(AtlasClient.java:906) at org.apache.atlas.AtlasClient.createType(AtlasClient.java:257) at org.apache.atlas.AtlasClient.createType(AtlasClient.java:275) at org.apache.atlas.web.integration.BaseResourceIT.createType(BaseResourceIT.java:213) at org.apache.atlas.web.integration.BaseResourceIT.createTypeDefinitionsV1(BaseResourceIT.java:360) at org.apache.atlas.notification.NotificationHookConsumerIT.setUp(NotificationHookConsumerIT.java:54) testAccessforUnauthenticatedResource(org.apache.atlas.web.filters.AtlasAuthenticationSimpleFilterIT) Time elapsed: 0.003 sec <<< FAILURE! java.lang.AssertionError: expected [200] but found [503] at org.testng.Asse
Re: Server appears to start, but connections fail with 503
Hi Ernie, There are 2 ways to build and run Atlas currently: - external Hbase and Solr and ZK - embedded HBase Solr and ZK I am not sure which one you are trying. Sharing your build string would be helpful. If you are using the embedded HBase option then you need to specify environment variables, for the runtime Atlas to work, As per yesterdays call - Madhan suggests running with an external HBase, as he seems a performance delay using the embedded HBase. all the best, David. From: "Ernie Ostic" To: "Atlas" Date: 14/02/2018 11:52 Subject:Server appears to start, but connections fail with 503 Hi all... Using "master", the build appears to be successful, and the server is properly listening on port 21000, but quick start or even direct access fails with 503. Any suggestions on where to look? Thanks! Ernie $ bin/quick_start.py log4j:WARN No such property [maxFileSize] in org.apache.log4j.PatternLayout. log4j:WARN No such property [maxBackupIndex] in org.apache.log4j.PatternLayout. log4j:WARN No such property [maxFileSize] in org.apache.log4j.PatternLayout. log4j:WARN No such property [maxBackupIndex] in org.apache.log4j.PatternLayout. log4j:WARN No such property [maxFileSize] in org.apache.log4j.PatternLayout. Enter username for atlas :- admin Enter password for atlas :- Creating sample types: Exception in thread "main" org.apache.atlas.AtlasServiceException: Metadata service API org.apache.atlas.AtlasClientV2$API_V2@1033576a failed with status 503 (Service Unavailable) Response Body ( Error 503 HTTP ERROR: 503 Problem accessing /api/atlas/v2/types/typedefs/. Reason: Service Unavailable https://urldefense.proofpoint.com/v2/url?u=http-3A__eclipse.org_jetty&d=DwMFAg&c=jf_iaSHvJObTbx-siA1ZOg&r=QhpUQPr5YlG95aAgCvZGStEXHg4hBbSYQ9JkRqR_svY&m=0y7mtLMoBMEc2RNTv8nhllpgWZHIGzqdnqcU0FUYtvg&s=yRGEThijdpp0lFksddATlowggX7r9D_zX0z2DXZR8EA&e= ">Powered by Jetty:// 9.3.14.v20161028 ) at org.apache.atlas.AtlasBaseClient.callAPIWithResource (AtlasBaseClient.java:388) at org.apache.atlas.AtlasBaseClient.callAPIWithResource (AtlasBaseClient.java:323) at org.apache.atlas.AtlasBaseClient.callAPI(AtlasBaseClient.java:211) at org.apache.atlas.AtlasClientV2.createAtlasTypeDefs (AtlasClientV2.java:227) at org.apache.atlas.examples.QuickStartV2.createTypes (QuickStartV2.java:185) at org.apache.atlas.examples.QuickStartV2.runQuickstart (QuickStartV2.java:141) at org.apache.atlas.examples.QuickStartV2.main(QuickStartV2.java:126) No sample data added to Apache Atlas Server. Ernie Ostic Worldwide Technical Sales InfoSphere Information Server IBM Analytics Cell: (617) 331 8238 --- Apache Atlas Update! https://dsrealtime.wordpress.com/2017/11/16/apache-atlas-update-have-you-been-watching/ Open IGC is here! https://dsrealtime.wordpress.com/2015/07/29/open-igc-is-here/ Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
Re: Review Request 63502: ATLAS-2246 - Review request for the OMRS Connector API.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/63502/#review197399 --- omrs/src/main/java/org/apache/atlas/omrs/metadatacollection/properties/instances/EntitySummary.java Lines 71 (patched) <https://reviews.apache.org/r/63502/#comment277546> It would be normal to only expose List here are the return value and not expose the type of List, so you do not expose the List implementation. The same for the setter. - David Radley On Jan. 29, 2018, 9:51 a.m., Mandy Chessell wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/63502/ > --- > > (Updated Jan. 29, 2018, 9:51 a.m.) > > > Review request for atlas, David Radley and Madhan Neethiraj. > > > Repository: atlas > > > Description > --- > > This is a request for code review of the OMRS. There are still parts of the > code not written - the aim of the review is to make the connector interfaces > available so they can be evaluated by teams buinding components against the > APIs. > > There is a patch connected to > https://issues.apache.org/jira/projects/ATLAS/issues/ATLAS-2246 that has the > code plus other code that is not ready for review. The OMRS Connector API > depends on the OCF - see https://issues.apache.org/jira/browse/ATLAS-1095. > > > Diffs > - > > omrs/README.md PRE-CREATION > omrs/pom.xml PRE-CREATION > > omrs/src/main/java/org/apache/atlas/omrs/adapters/atlas/eventmapper/AtlasOMRSRepositoryEventMapper.java > PRE-CREATION > > omrs/src/main/java/org/apache/atlas/omrs/adapters/atlas/eventmapper/AtlasOMRSRepositoryEventMapperProvider.java > PRE-CREATION > > omrs/src/main/java/org/apache/atlas/omrs/adapters/atlas/repositoryconnector/LocalAtlasOMRSMetadataCollection.java > PRE-CREATION > > omrs/src/main/java/org/apache/atlas/omrs/adapters/atlas/repositoryconnector/LocalAtlasOMRSRepositoryConnector.java > PRE-CREATION > > omrs/src/main/java/org/apache/atlas/omrs/adapters/atlas/repositoryconnector/LocalAtlasOMRSRepositoryConnectorProvider.java > PRE-CREATION > > omrs/src/main/java/org/apache/atlas/omrs/adapters/igc/eventmapper/IGCOMRSRepositoryEventMapper.java > PRE-CREATION > > omrs/src/main/java/org/apache/atlas/omrs/adapters/igc/eventmapper/IGCOMRSRepositoryEventMapperProvider.java > PRE-CREATION > > omrs/src/main/java/org/apache/atlas/omrs/adapters/igc/repositoryconnector/IGCOMRSMetadataCollection.java > PRE-CREATION > > omrs/src/main/java/org/apache/atlas/omrs/adapters/igc/repositoryconnector/IGCOMRSRepositoryConnector.java > PRE-CREATION > > omrs/src/main/java/org/apache/atlas/omrs/adapters/igc/repositoryconnector/IGCOMRSRepositoryConnectorProvider.java > PRE-CREATION > > omrs/src/main/java/org/apache/atlas/omrs/admin/OMRSConfigurationFactory.java > PRE-CREATION > omrs/src/main/java/org/apache/atlas/omrs/admin/OMRSOperationalServices.java > PRE-CREATION > omrs/src/main/java/org/apache/atlas/omrs/admin/properties/CohortConfig.java > PRE-CREATION > > omrs/src/main/java/org/apache/atlas/omrs/admin/properties/EnterpriseAccessConfig.java > PRE-CREATION > > omrs/src/main/java/org/apache/atlas/omrs/admin/properties/LocalRepositoryConfig.java > PRE-CREATION > > omrs/src/main/java/org/apache/atlas/omrs/admin/properties/OpenMetadataEventProtocolVersion.java > PRE-CREATION > > omrs/src/main/java/org/apache/atlas/omrs/admin/properties/OpenMetadataExchangeRule.java > PRE-CREATION > > omrs/src/main/java/org/apache/atlas/omrs/admin/properties/RepositoryServicesConfig.java > PRE-CREATION > > omrs/src/main/java/org/apache/atlas/omrs/archivemanager/OMRSArchiveManager.java > PRE-CREATION > > omrs/src/main/java/org/apache/atlas/omrs/archivemanager/store/OpenMetadataArchiveStore.java > PRE-CREATION > > omrs/src/main/java/org/apache/atlas/omrs/archivemanager/store/OpenMetadataArchiveStoreConnectorBase.java > PRE-CREATION > > omrs/src/main/java/org/apache/atlas/omrs/archivemanager/store/OpenMetadataArchiveStoreProviderBase.java > PRE-CREATION > > omrs/src/main/java/org/apache/atlas/omrs/archivemanager/store/properties/OpenMetadataArchive.java > PRE-CREATION > > omrs/src/main/java/org/apache/atlas/omrs/archivemanager/store/properties/OpenMetadataArchiveInstanceStore.java > PRE-CREATION > > omrs/src/main/java/org/apache/atlas/omrs/archivemanager/store/properties/Op
[jira] [Commented] (ATLAS-1095) Open connector framework
[ https://issues.apache.org/jira/browse/ATLAS-1095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16342239#comment-16342239 ] David Radley commented on ATLAS-1095: - I have committed the patch, the details are : Author Mandy Chessell Fri, 19 Jan 2018 09:27:26 + (09:27 +) committer David Radley Sat, 27 Jan 2018 00:40:56 + (00:40 +) commit cbfdd7fcc2ccb011c6f3000e3cb7a9de79268445 tree bc710344fde1adac92c65ec74e1042cc34960db4 tree | snapshot parent 804c4635e66e05fae627ad1e9344c1b11251ad35 commit | diff > Open connector framework > > > Key: ATLAS-1095 > URL: https://issues.apache.org/jira/browse/ATLAS-1095 > Project: Atlas > Issue Type: New Feature >Affects Versions: 1.0.0 >Reporter: Stephanie Hazlewood >Assignee: David Radley >Priority: Major > Labels: VirtualDataConnector > Attachments: 0001-ATLAS-1095-final-code-drop-for-OCF.patch, OCF > JavaDoc.zip, Open Connector Framework - 20th June 2017.doc, Open Connector > Framework - 9th May 2017.doc > > > Atlas provides a common approach to metadata management and governance across > all systems and data within an organization. Today Atlas provides access to > metadata. A connector provides access to a data source. As connectors are > the proxy of all data, they can also be explicit providers of metadata. > This JIRA proposes an open connector framework to manage connectors that > provide access to both data and the metadata Atlas provides together through > a single connector interface. > This will help data tools to to better the exchange of information between > platforms. It also offers new opportunities for the consistent enforcement of > the governance policies and rules (e.g., rules of visibility). Source > connector/connection metadata provides the nucleus around which all other > metadata describing the data builds. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (ATLAS-1095) Open connector framework
[ https://issues.apache.org/jira/browse/ATLAS-1095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Radley reassigned ATLAS-1095: --- Assignee: David Radley (was: Mandy Chessell) > Open connector framework > > > Key: ATLAS-1095 > URL: https://issues.apache.org/jira/browse/ATLAS-1095 > Project: Atlas > Issue Type: New Feature >Affects Versions: 1.0.0 >Reporter: Stephanie Hazlewood >Assignee: David Radley >Priority: Major > Labels: VirtualDataConnector > Attachments: 0001-ATLAS-1095-final-code-drop-for-OCF.patch, OCF > JavaDoc.zip, Open Connector Framework - 20th June 2017.doc, Open Connector > Framework - 9th May 2017.doc > > > Atlas provides a common approach to metadata management and governance across > all systems and data within an organization. Today Atlas provides access to > metadata. A connector provides access to a data source. As connectors are > the proxy of all data, they can also be explicit providers of metadata. > This JIRA proposes an open connector framework to manage connectors that > provide access to both data and the metadata Atlas provides together through > a single connector interface. > This will help data tools to to better the exchange of information between > platforms. It also offers new opportunities for the consistent enforcement of > the governance policies and rules (e.g., rules of visibility). Source > connector/connection metadata provides the nucleus around which all other > metadata describing the data builds. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: Review Request 63503: ATLAS-1095 - Review of OCF - the Open Connector Framework
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/63503/#review196317 --- Ship it! Ship It! - David Radley On Jan. 19, 2018, 9:30 a.m., Mandy Chessell wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/63503/ > --- > > (Updated Jan. 19, 2018, 9:30 a.m.) > > > Review request for atlas and Madhan Neethiraj. > > > Repository: atlas > > > Description > --- > > This patch contains the open connector framework code. This code is located > in the om-fwk-ocf component and is described in JIRA > https://issues.apache.org/jira/browse/ATLAS-1095 > > I have added a new patch to the Jira with fixes from Yao's comments. > Upgraded Maven to 352 and rebuilt/rerun tests. > > > Diffs > - > > om-fwk-ocf/README.md PRE-CREATION > om-fwk-ocf/pom.xml PRE-CREATION > om-fwk-ocf/src/main/java/org/apache/atlas/ocf/Connector.java PRE-CREATION > om-fwk-ocf/src/main/java/org/apache/atlas/ocf/ConnectorBase.java > PRE-CREATION > om-fwk-ocf/src/main/java/org/apache/atlas/ocf/ConnectorBroker.java > PRE-CREATION > om-fwk-ocf/src/main/java/org/apache/atlas/ocf/ConnectorProvider.java > PRE-CREATION > om-fwk-ocf/src/main/java/org/apache/atlas/ocf/ConnectorProviderBase.java > PRE-CREATION > > om-fwk-ocf/src/main/java/org/apache/atlas/ocf/ffdc/ConnectionCheckedException.java > PRE-CREATION > > om-fwk-ocf/src/main/java/org/apache/atlas/ocf/ffdc/ConnectorCheckedException.java > PRE-CREATION > > om-fwk-ocf/src/main/java/org/apache/atlas/ocf/ffdc/OCFCheckedExceptionBase.java > PRE-CREATION > om-fwk-ocf/src/main/java/org/apache/atlas/ocf/ffdc/OCFErrorCode.java > PRE-CREATION > om-fwk-ocf/src/main/java/org/apache/atlas/ocf/ffdc/OCFRuntimeException.java > PRE-CREATION > > om-fwk-ocf/src/main/java/org/apache/atlas/ocf/ffdc/PropertyServerException.java > PRE-CREATION > om-fwk-ocf/src/main/java/org/apache/atlas/ocf/ffdc/README.md PRE-CREATION > > om-fwk-ocf/src/main/java/org/apache/atlas/ocf/properties/AdditionalProperties.java > PRE-CREATION > om-fwk-ocf/src/main/java/org/apache/atlas/ocf/properties/Annotation.java > PRE-CREATION > > om-fwk-ocf/src/main/java/org/apache/atlas/ocf/properties/AnnotationStatus.java > PRE-CREATION > om-fwk-ocf/src/main/java/org/apache/atlas/ocf/properties/Annotations.java > PRE-CREATION > > om-fwk-ocf/src/main/java/org/apache/atlas/ocf/properties/AssetDescriptor.java > PRE-CREATION > om-fwk-ocf/src/main/java/org/apache/atlas/ocf/properties/AssetDetail.java > PRE-CREATION > > om-fwk-ocf/src/main/java/org/apache/atlas/ocf/properties/AssetPropertyBase.java > PRE-CREATION > > om-fwk-ocf/src/main/java/org/apache/atlas/ocf/properties/AssetPropertyIteratorBase.java > PRE-CREATION > om-fwk-ocf/src/main/java/org/apache/atlas/ocf/properties/AssetSummary.java > PRE-CREATION > om-fwk-ocf/src/main/java/org/apache/atlas/ocf/properties/AssetUniverse.java > PRE-CREATION > om-fwk-ocf/src/main/java/org/apache/atlas/ocf/properties/Certification.java > PRE-CREATION > > om-fwk-ocf/src/main/java/org/apache/atlas/ocf/properties/Certifications.java > PRE-CREATION > > om-fwk-ocf/src/main/java/org/apache/atlas/ocf/properties/Classification.java > PRE-CREATION > > om-fwk-ocf/src/main/java/org/apache/atlas/ocf/properties/Classifications.java > PRE-CREATION > om-fwk-ocf/src/main/java/org/apache/atlas/ocf/properties/Comment.java > PRE-CREATION > om-fwk-ocf/src/main/java/org/apache/atlas/ocf/properties/CommentType.java > PRE-CREATION > om-fwk-ocf/src/main/java/org/apache/atlas/ocf/properties/Comments.java > PRE-CREATION > > om-fwk-ocf/src/main/java/org/apache/atlas/ocf/properties/ConnectedAssetProperties.java > PRE-CREATION > om-fwk-ocf/src/main/java/org/apache/atlas/ocf/properties/Connection.java > PRE-CREATION > om-fwk-ocf/src/main/java/org/apache/atlas/ocf/properties/Connections.java > PRE-CREATION > om-fwk-ocf/src/main/java/org/apache/atlas/ocf/properties/ConnectorType.java > PRE-CREATION > > om-fwk-ocf/src/main/java/org/apache/atlas/ocf/properties/DerivedSchemaElement.java > PRE-CREATION > om-fwk-ocf/src/main/java/org/apache/atlas/ocf/properties/ElementHeader.java > PRE-CREATION > om-fwk-ocf/src/main/java/org/apache/atlas/ocf/properties/ElementOrigin.java > PRE-CREATION > om-fwk-oc
