[jira] [Comment Edited] (ATLAS-1690) Introduce top level relationships

[Suma Shivaprasad (JIRA)]

[ 
https://issues.apache.org/jira/browse/ATLAS-1690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15999062#comment-15999062
 ] 

Suma Shivaprasad edited comment on ATLAS-1690 at 5/5/17 10:30 PM:
--

In response to [~davidrad] 's comment , 
3. Interesting idea. You are right - tag propagation is important. As these are 
not bidirectional relationships, there is not an implied direction-it is not 
obvious which way the propagation should flow. I could see us wanting to do 
this sort of thing for composition (which are not modelled by a top level 
relationship). I suspect it is less likely that for 2 independent entities we 
would want to propagate tags using flags in the types. 
Ideally propagation of tags should be defined in Ranger rules - as we are 
likely to want different tag propagation in different contexts. Can I suggest 
we deal with classification in a subsequent design? We can then cover Semantic 
classification which will actually be a relationship. 

--> Can you pls elaborate or point me to jira/design docs where tag propagation 
has been detailed out?

4. Yes this is the same as Suma's suggestion of a category; I was thinking of 
"bidirectional”, “aggregation” - as association could be directional and 
contains could be composition . I will put this in, as we can then put this 
value into the edge - so we can identify which edges we need to expand into 
attributes. Are you OK with these amendments?

--> I think we should have four categories initially - composition, 
aggregation, bidirectional, directional. Its important to distinguish between  
aggregation and composition I think, since tag propagation and entity mutation 
cascade policies like delete etc can be controlled based on this. Also 
capturing bidirectional vs directional would help us in migrating the current 
technical model where a unidirectional relation could just have ownedRef on one 
side and there s no reverseAttribute on the referred end and the default could 
be bidirectional associations.


was (Author: suma.shivaprasad):
In response to [~davidrad] 's comment , 

4. Yes this is the same as Suma's suggestion of a category; I was thinking of 
"bidirectional”, “aggregation” - as association could be directional and 
contains could be composition . I will put this in, as we can then put this 
value into the edge - so we can identify which edges we need to expand into 
attributes. Are you OK with these amendments?

--> I think we should have four categories initially - composition, 
aggregation, bidirectional, directional. Its important to distinguish between  
aggregation and composition I think, since tag propagation and entity mutation 
cascade policies like delete etc can be controlled based on this. Also 
capturing bidirectional vs directional would help us in migrating the current 
technical model where a unidirectional relation could just have ownedRef on one 
side and there s no reverseAttribute on the referred end and the default could 
be bidirectional associations.

> Introduce top level relationships
> -
>
> Key: ATLAS-1690
> URL: https://issues.apache.org/jira/browse/ATLAS-1690
> Project: Atlas
>  Issue Type: Improvement
>Reporter: David Radley
>Assignee: David Radley
>  Labels: VirtualDataConnector
> Attachments: Atlas_RelationDef_Json_Structure_v1.pdf, Atlas 
> Relationships proposal v1.0.pdf, Atlas Relationships proposal v1.1.pdf, Atlas 
> Relationships proposal v1.2.pdf, Atlas Relationships proposal v1.3.pdf, Atlas 
> Relationships proposal v1.4.pdf, Atlas Relationships proposal v1.5.pdf, Atlas 
> Relationships proposal v1.6.pdf, Atlas Relationships proposal v1.7.pdf
>
>
> Introduce top level relationships including support for 
> -many to many relationships
> - relationship names including the name for both ends and the relationship.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (ATLAS-1690) Introduce top level relationships

[Suma Shivaprasad (JIRA)]

[ 
https://issues.apache.org/jira/browse/ATLAS-1690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15999062#comment-15999062
 ] 

Suma Shivaprasad commented on ATLAS-1690:
-

In response to [~davidrad] 's comment , 

4. Yes this is the same as Suma's suggestion of a category; I was thinking of 
"bidirectional”, “aggregation” - as association could be directional and 
contains could be composition . I will put this in, as we can then put this 
value into the edge - so we can identify which edges we need to expand into 
attributes. Are you OK with these amendments?

--> I think we should have four categories initially - composition, 
aggregation, bidirectional, directional. Its important to distinguish between  
aggregation and composition I think, since tag propagation and entity mutation 
cascade policies like delete etc can be controlled based on this. Also 
capturing bidirectional vs directional would help us in migrating the current 
technical model where a unidirectional relation could just have ownedRef on one 
side and there s no reverseAttribute on the referred end and the default could 
be bidirectional associations.

> Introduce top level relationships
> -
>
> Key: ATLAS-1690
> URL: https://issues.apache.org/jira/browse/ATLAS-1690
> Project: Atlas
>  Issue Type: Improvement
>Reporter: David Radley
>Assignee: David Radley
>  Labels: VirtualDataConnector
> Attachments: Atlas_RelationDef_Json_Structure_v1.pdf, Atlas 
> Relationships proposal v1.0.pdf, Atlas Relationships proposal v1.1.pdf, Atlas 
> Relationships proposal v1.2.pdf, Atlas Relationships proposal v1.3.pdf, Atlas 
> Relationships proposal v1.4.pdf, Atlas Relationships proposal v1.5.pdf, Atlas 
> Relationships proposal v1.6.pdf, Atlas Relationships proposal v1.7.pdf
>
>
> Introduce top level relationships including support for 
> -many to many relationships
> - relationship names including the name for both ends and the relationship.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: Review Request 58985: Type deletion should invalidate property key in titan

[David Radley]

> On May 5, 2017, 8:14 a.m., David Radley wrote:
> > graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphManagement.java
> > Lines 69 (patched)
> > 
> >
> > It does not seem right to rename property keys in old types. This seems 
> > to me to be changing history - which is not great for audit. 
> > 
> > I do not understand why the name type is restricted to the old name - 
> > can we not change this logic? 
> > I would suggest we tolerate more than one type of the same name - only 
> > one of which can be active at any one time - each type chould have their 
> > own shape and guid. All references to a type should be by guid - so we 
> > should not get confused by reusing a name.
> 
> Apoorv Naik wrote:
> The reason we're renaming the property key name is that once the type is 
> deleted, the data type information (schema) still persists in Titan. Now when 
> we try to recreate that type with same attribute but different data type, the 
> schema is violated and it fails. Titan docs recommend changing the name (as 
> it can't be deleted by titan) in order to redefine the key. The test will 
> make it more clear, without this change the new test would fail.

Great thanks Apoorv. I thought there would be a good reason


- David


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58985/#review174009
---


On May 4, 2017, 10:07 p.m., Apoorv Naik wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/58985/
> ---
> 
> (Updated May 4, 2017, 10:07 p.m.)
> 
> 
> Review request for atlas.
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> This change supports the use-case where the user creates a type and deletes 
> it sometime later, now the redefinition of the type is restricted to use the 
> same data type for the attributes used during the initial creation. 
> 
> Solution: Rename the propertyKey corresponding to that attribute using the 
> titan management API by suffing _deleted_xxx where xxx is an increasing 
> integer sequence starting from 0. The reason for increment of xxx is that the 
> type create, delete and re-create can be done multiple times without any 
> conflicting key in Titan.
> 
> 
> Diffs
> -
> 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphManagement.java
>  5efd7c0f 
>   
> graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/Titan0GraphManagement.java
>  9a17d9ee 
>   
> graphdb/titan1/src/main/java/org/apache/atlas/repository/graphdb/titan1/Titan1GraphManagement.java
>  12faeda5 
>   intg/src/main/java/org/apache/atlas/listener/TypeDefChangeListener.java 
> 124f106f 
>   intg/src/main/java/org/apache/atlas/type/AtlasTypeRegistry.java ea2a7038 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java
>  47dccf19 
>   
> repository/src/test/java/org/apache/atlas/repository/impexp/ExportServiceTest.java
>  a7fc24cf 
>   
> repository/src/test/java/org/apache/atlas/repository/store/graph/AtlasTypeDefGraphStoreTest.java
>  84ad72c8 
> 
> 
> Diff: https://reviews.apache.org/r/58985/diff/2/
> 
> 
> Testing
> ---
> 
> mvn clean package and mvn clean install run successfully.
> 
> 
> Thanks,
> 
> Apoorv Naik
> 
>

Re: Review Request 58803: ATLAS-1390: IBM graph implementation

[Graham Wallis]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58803/#review174038
---



Hi Apoorv,

Thanks for rebasing - it's definitely better than before, but I still get a 
compile error when I try to build using the ibm-graph profile. I'm not sure 
that it's strictly part of the IBM-Graph changes to Atlas - it seems to be 
caused by the org.apache.atlas.repository.impexp package not having a maven 
dependency on org.apache.solr.common - so ZipFileResourceTestUtils cannot 
locate StringUtils.

- Graham Wallis


On May 4, 2017, 4:35 p.m., Apoorv Naik wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/58803/
> ---
> 
> (Updated May 4, 2017, 4:35 p.m.)
> 
> 
> Review request for atlas, David Radley, David Kantor, Graham Wallis, Neeru 
> Gupta, and Jeff Hagelberg.
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> See https://reviews.apache.org/r/56724/
> 
> Since the review is outdated and lot of conflicting changes have happened 
> over the course of time. This review addresses those conflicts.
> 
> 
> Diffs
> -
> 
>   common/src/main/java/org/apache/atlas/GraphInitializationException.java 
> PRE-CREATION 
>   common/src/main/java/org/apache/atlas/groovy/LiteralExpression.java 
> 14074994 
>   
> common/src/main/java/org/apache/atlas/groovy/VariableAssignmentExpression.java
>  1aa74435 
>   distro/pom.xml f0962b6c 
>   distro/src/conf/atlas-application.properties b2b8e745 
>   distro/src/conf/credentials.json PRE-CREATION 
>   distro/src/main/assemblies/standalone-package.xml 215cb236 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java 
> a3a27bfd 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/GraphDatabase.java
>  3dfc6e8d 
>   graphdb/graphdb-impls/pom.xml feafe742 
>   graphdb/ibm-graph/pom.xml PRE-CREATION 
>   graphdb/ibm-graph/readme-multitenancy-support.txt PRE-CREATION 
>   graphdb/ibm-graph/readme.txt PRE-CREATION 
>   
> graphdb/ibm-graph/src/main/java/org/apache/atlas/ibmgraph/AtlasPropertyKeyToPropertyKey.java
>  PRE-CREATION 
>   
> graphdb/ibm-graph/src/main/java/org/apache/atlas/ibmgraph/ElementDeletedCheckingInvocationHandler.java
>  PRE-CREATION 
>   
> graphdb/ibm-graph/src/main/java/org/apache/atlas/ibmgraph/GraphPerTenantStrategy.java
>  PRE-CREATION 
>   
> graphdb/ibm-graph/src/main/java/org/apache/atlas/ibmgraph/GraphReadLockInvocationHandler.java
>  PRE-CREATION 
>   
> graphdb/ibm-graph/src/main/java/org/apache/atlas/ibmgraph/IBMGraphDatabase.java
>  PRE-CREATION 
>   graphdb/ibm-graph/src/main/java/org/apache/atlas/ibmgraph/IBMGraphEdge.java 
> PRE-CREATION 
>   
> graphdb/ibm-graph/src/main/java/org/apache/atlas/ibmgraph/IBMGraphElement.java
>  PRE-CREATION 
>   
> graphdb/ibm-graph/src/main/java/org/apache/atlas/ibmgraph/IBMGraphGraph.java 
> PRE-CREATION 
>   
> graphdb/ibm-graph/src/main/java/org/apache/atlas/ibmgraph/IBMGraphGraphQuery.java
>  PRE-CREATION 
>   
> graphdb/ibm-graph/src/main/java/org/apache/atlas/ibmgraph/IBMGraphIndex.java 
> PRE-CREATION 
>   
> graphdb/ibm-graph/src/main/java/org/apache/atlas/ibmgraph/IBMGraphIndexQuery.java
>  PRE-CREATION 
>   
> graphdb/ibm-graph/src/main/java/org/apache/atlas/ibmgraph/IBMGraphIndexQueryResult.java
>  PRE-CREATION 
>   
> graphdb/ibm-graph/src/main/java/org/apache/atlas/ibmgraph/IBMGraphManagement.java
>  PRE-CREATION 
>   
> graphdb/ibm-graph/src/main/java/org/apache/atlas/ibmgraph/IBMGraphMetadata.java
>  PRE-CREATION 
>   
> graphdb/ibm-graph/src/main/java/org/apache/atlas/ibmgraph/IBMGraphPropertyKey.java
>  PRE-CREATION 
>   
> graphdb/ibm-graph/src/main/java/org/apache/atlas/ibmgraph/IBMGraphVertex.java 
> PRE-CREATION 
>   
> graphdb/ibm-graph/src/main/java/org/apache/atlas/ibmgraph/IBMGraphVertexQuery.java
>  PRE-CREATION 
>   
> graphdb/ibm-graph/src/main/java/org/apache/atlas/ibmgraph/MultiTenancyDisabledStrategy.java
>  PRE-CREATION 
>   
> graphdb/ibm-graph/src/main/java/org/apache/atlas/ibmgraph/PartitionPerTenantStrategy.java
>  PRE-CREATION 
>   
> graphdb/ibm-graph/src/main/java/org/apache/atlas/ibmgraph/TenantGraphStrategy.java
>  PRE-CREATION 
>   
> graphdb/ibm-graph/src/main/java/org/apache/atlas/ibmgraph/api/Cardinality.java
>  PRE-CREATION 
>   
> graphdb/ibm-graph/src/main/java/org/apache/atlas/ibmgraph/api/GraphDatabaseClient.java
>  PRE-CREATION 
>   
> graphdb/ibm-graph/src/main/java/org/apache/atlas/ibmgraph/api/GraphDatabaseConfiguration.java
>  PRE-CREATION 
>   
> graphdb/ibm-graph/src/main/java/org/apache/atlas/ibmgraph/api/IGraphDatabaseClient.java
>  PRE-CREATION 
>   
> graphdb/ibm-graph/src/main/java/org/apache/atlas/ibmgraph/api/IndexStatus.java
>  PRE-CREATION 
>   
> graphdb/ibm-graph/src/main/java/org/

Re: Review Request 58985: Type deletion should invalidate property key in titan

[Apoorv Naik]

> On May 5, 2017, 8:14 a.m., David Radley wrote:
> > graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphManagement.java
> > Lines 69 (patched)
> > 
> >
> > It does not seem right to rename property keys in old types. This seems 
> > to me to be changing history - which is not great for audit. 
> > 
> > I do not understand why the name type is restricted to the old name - 
> > can we not change this logic? 
> > I would suggest we tolerate more than one type of the same name - only 
> > one of which can be active at any one time - each type chould have their 
> > own shape and guid. All references to a type should be by guid - so we 
> > should not get confused by reusing a name.

The reason we're renaming the property key name is that once the type is 
deleted, the data type information (schema) still persists in Titan. Now when 
we try to recreate that type with same attribute but different data type, the 
schema is violated and it fails. Titan docs recommend changing the name (as it 
can't be deleted by titan) in order to redefine the key. The test will make it 
more clear, without this change the new test would fail.


- Apoorv


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58985/#review174009
---


On May 4, 2017, 10:07 p.m., Apoorv Naik wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/58985/
> ---
> 
> (Updated May 4, 2017, 10:07 p.m.)
> 
> 
> Review request for atlas.
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> This change supports the use-case where the user creates a type and deletes 
> it sometime later, now the redefinition of the type is restricted to use the 
> same data type for the attributes used during the initial creation. 
> 
> Solution: Rename the propertyKey corresponding to that attribute using the 
> titan management API by suffing _deleted_xxx where xxx is an increasing 
> integer sequence starting from 0. The reason for increment of xxx is that the 
> type create, delete and re-create can be done multiple times without any 
> conflicting key in Titan.
> 
> 
> Diffs
> -
> 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphManagement.java
>  5efd7c0f 
>   
> graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/Titan0GraphManagement.java
>  9a17d9ee 
>   
> graphdb/titan1/src/main/java/org/apache/atlas/repository/graphdb/titan1/Titan1GraphManagement.java
>  12faeda5 
>   intg/src/main/java/org/apache/atlas/listener/TypeDefChangeListener.java 
> 124f106f 
>   intg/src/main/java/org/apache/atlas/type/AtlasTypeRegistry.java ea2a7038 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java
>  47dccf19 
>   
> repository/src/test/java/org/apache/atlas/repository/impexp/ExportServiceTest.java
>  a7fc24cf 
>   
> repository/src/test/java/org/apache/atlas/repository/store/graph/AtlasTypeDefGraphStoreTest.java
>  84ad72c8 
> 
> 
> Diff: https://reviews.apache.org/r/58985/diff/2/
> 
> 
> Testing
> ---
> 
> mvn clean package and mvn clean install run successfully.
> 
> 
> Thanks,
> 
> Apoorv Naik
> 
>

[jira] [Created] (ATLAS-1768) Create common types for Apache Atlas

[Mandy Chessell (JIRA)]

Mandy Chessell created ATLAS-1768:
-

 Summary: Create common types for Apache Atlas
 Key: ATLAS-1768
 URL: https://issues.apache.org/jira/browse/ATLAS-1768
 Project: Atlas
  Issue Type: New Feature
  Components:  atlas-core
Affects Versions: 0.9-incubating
Reporter: Mandy Chessell
Assignee: Mandy Chessell


This JIRA describes a proposal for standard types for common metadata entities 
and relationships.  For example, glossaries, database definitions, rules, 
policies, ...

The value of having standard definitions for metadata is to enable type safe 
APIs and business level UIs plus be able to exchange metadata between different 
instances of metadata repositories.

The implementation of these common definitions can be divided into 2 groups:
 - standard system attributes for all entities and relationships
 - standard types for commonly used metadata

Adaptation and flexibility are key in metadata environments so these common 
definitions must be extensible - and we still need to support the ad hoc 
definition of new types in Atlas.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (ATLAS-1767) Support KNOX SSO Token based authentication on Atlas REST API calls

[Nixon Rodrigues (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-1767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nixon Rodrigues updated ATLAS-1767:
---
Attachment: ATLAS-1767.patch

> Support KNOX SSO Token based authentication on Atlas REST API calls
> ---
>
> Key: ATLAS-1767
> URL: https://issues.apache.org/jira/browse/ATLAS-1767
> Project: Atlas
>  Issue Type: Bug
>  Components: atlas-webui
>Affects Versions: 0.8-incubating
>Reporter: Nixon Rodrigues
>Assignee: Nixon Rodrigues
> Fix For: 0.9-incubating
>
> Attachments: ATLAS-1767.patch
>
>
> Currently Authentication for Atlas with Knox SSO is supported for broswer 
> based userAgent only.
> Removing this restriction to support KNOX SSO Token based authentication on 
> Atlas REST API calls from all user-agents with hadoop-jwt cookie in header.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Review Request 59023: ATLAS-1767 :- Support KNOX SSO Token based authentication on Atlas REST API calls

[Nixon Rodrigues]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59023/
---

Review request for atlas, Apoorv Naik, Ayub Pathan, Madhan Neethiraj, and 
Sarath Subramanian.


Repository: atlas


Description
---

This patch includes change to support knox sso authentication from all 
user-agent when knox cookie header are being in request header.


Diffs
-

  
webapp/src/main/java/org/apache/atlas/web/filters/AtlasKnoxSSOAuthenticationFilter.java
 c3219b9 


Diff: https://reviews.apache.org/r/59023/diff/1/


Testing
---

Verified authentication from various scenarios.

* From Browser with Knox Authentication

* From Login Page browser

* From from Curl client with various header option

* With knox hadoop-jwt-cookie header
curl -kivL --negotiate -u :  http://:21000/api/atlas/session --cookie 
"hadoop-jwt=eyJhbGciOiJSUzI1NiJ9.eyJzdWIiOiJhZG1pbiIsImlzcyI6IktOT1hTU08iLCJleHAiOjE0OTQyODc3MDF9.aj5E0bqTlmnn42gxBm0geogNpX6XwXaEFhDrt-0ANbQpsF2gL2Fn8Lb9hBu9x2R9WFiCv9Wc2H58mW93H_k17yQFqHnJc5OfFOnQ-oI4BYABe7Wi4HkmD3ocRHXc6coY9RbR7Tu6FYT-8ZNH99FgmU20XqXFnmTUgxQYSosjXzU"

* With Kerberos negotiate header
curl -kivL --negotiate -u :  http://:21000/api/atlas/admin/session

* With Basic Auth Header
curl -kivL -u admin:admin  http://:21000/api/atlas/session

* From Jersey Java Client ie verified Quick Start script


Thanks,

Nixon Rodrigues

[jira] [Commented] (ATLAS-1690) Introduce top level relationships

[Christopher Grote (JIRA)]

[ 
https://issues.apache.org/jira/browse/ATLAS-1690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15998393#comment-15998393
 ] 

Christopher Grote commented on ATLAS-1690:
--

Fair points on appropriate layer for tag propagation -- I think you're right, 
it probably doesn't make sense to try to embed that "logic" into OMRS itself, 
but rather leave it to the consumption-side to appropriately branch-out 
(propagate) the tags as needed for their target consumption-audience.

> Introduce top level relationships
> -
>
> Key: ATLAS-1690
> URL: https://issues.apache.org/jira/browse/ATLAS-1690
> Project: Atlas
>  Issue Type: Improvement
>Reporter: David Radley
>Assignee: David Radley
>  Labels: VirtualDataConnector
> Attachments: Atlas_RelationDef_Json_Structure_v1.pdf, Atlas 
> Relationships proposal v1.0.pdf, Atlas Relationships proposal v1.1.pdf, Atlas 
> Relationships proposal v1.2.pdf, Atlas Relationships proposal v1.3.pdf, Atlas 
> Relationships proposal v1.4.pdf, Atlas Relationships proposal v1.5.pdf, Atlas 
> Relationships proposal v1.6.pdf, Atlas Relationships proposal v1.7.pdf
>
>
> Introduce top level relationships including support for 
> -many to many relationships
> - relationship names including the name for both ends and the relationship.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (ATLAS-1754) Self-Service User Interfaces for Atlas

[Mandy Chessell (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-1754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mandy Chessell updated ATLAS-1754:
--
Description: 
Apache Atlas collects information about an organization's data assets.  This 
JIRA provides a role-based user interface to enable people in the organization 
to locate, use, manage and govern these assets.  The personas that it will 
support are:
 - Data scientists and citizen analysts (business people working with data) 
wishing to locate and use data.
 - Application developers wanting to use standard structures in their APIs and 
to understand the rules that apply to the data they are processing with the API.
 - Data engineers and information architects supporting the data platforms and 
integrations that support the business.  These people use metadata to create 
strong information supply chains that deliver data to the organization. They 
also deploy analytics created by data scientists and citizen analytics into 
production and build analytical processes themselves.
 - Data owners (any role) wanting to improve the description of their data 
asset, to classify their asset and to understand how their data assets are 
being used.
 - Governance team members from CDO, to data owners to data stewards and 
custodians that define and monitor the metadata used to manage and govern the 
data.


  was:
Apache Atlas collects information about an organization's data assets.  This 
JIRA provides a role-based user interface to enable people in the organization 
to locate, use, manage and govern these assets.  The personas that it will 
support are:
 - Data scientists and citizen analysts (business people working with data) 
wishing to locate and use data.
 - Data engineers and information architects supporting the data platforms and 
integrations that support the business.  These people use metadata to create 
strong information supply chains that deliver data to the organization. They 
also deploy analytics created by data scientists and citizen analytics into 
production and build analytical processes themselves.
 - Governance team members from CDO, to data owners to data stewards and 
custodians that define and monitor the metadata used to manage and govern the 
data.



> Self-Service User Interfaces for Atlas
> --
>
> Key: ATLAS-1754
> URL: https://issues.apache.org/jira/browse/ATLAS-1754
> Project: Atlas
>  Issue Type: New Feature
>  Components: atlas-webui
>Affects Versions: 0.9-incubating
> Environment: Providing access to Atlas metadata for information 
> consumers and data governance team.
>Reporter: Mandy Chessell
>Assignee: Mandy Chessell
> Attachments: Pharma - Personas - 17th June 2016.ppt, Pharma - 
> Personas and Roles in Action - 4th July 2016.pptx
>
>
> Apache Atlas collects information about an organization's data assets.  This 
> JIRA provides a role-based user interface to enable people in the 
> organization to locate, use, manage and govern these assets.  The personas 
> that it will support are:
>  - Data scientists and citizen analysts (business people working with data) 
> wishing to locate and use data.
>  - Application developers wanting to use standard structures in their APIs 
> and to understand the rules that apply to the data they are processing with 
> the API.
>  - Data engineers and information architects supporting the data platforms 
> and integrations that support the business.  These people use metadata to 
> create strong information supply chains that deliver data to the 
> organization. They also deploy analytics created by data scientists and 
> citizen analytics into production and build analytical processes themselves.
>  - Data owners (any role) wanting to improve the description of their data 
> asset, to classify their asset and to understand how their data assets are 
> being used.
>  - Governance team members from CDO, to data owners to data stewards and 
> custodians that define and monitor the metadata used to manage and govern the 
> data.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (ATLAS-1095) Open connector framework

[Mandy Chessell (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-1095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mandy Chessell updated ATLAS-1095:
--
Description: 
Atlas provides a common approach to metadata management and governance across 
all systems and data within an organization.  Today Atlas provides access to 
metadata.   A connector provides access to a data source.  As connectors are 
the proxy of all data, they can also be explicit providers of metadata.   

This JIRA proposes an open connector framework to manage connectors that 
provide access to both data and the metadata Atlas provides together through a 
single connector interface.  

This will help data tools to to better the exchange of information between 
platforms. It also offers new opportunities for the consistent enforcement of 
the governance policies and rules (e.g., rules of visibility).  Source 
connector/connection metadata provides the nucleus around which all other 
metadata describing the data builds.  

Introducing this framework:

The Connector Broker is used to access to (finding and instantiating) a 
connector instance. 

Provides extension API supporting both partial and incremental adoption of the 
framework by existing connector provider runtimes allowing connector 
providers/contributors to adapt what they have vs re-write.

Introducing the APIs for metadata-enabled open connectors:

Provides an API for the normalized access to the metadata describing a 
connector (asset type, connection type).  This is called the Asset OMAS

The key personas using the framework are the tool developers and developers.

As a connector developer, I want to create a new connector (or register an 
existing connector) to plug into the framework so that I can retrieve and 
manage metadata about what the source provides and ensure data used from that 
source has governance policies (e.g., rules of visibility) consistently 
configured and enforced. 

The tool developer uses a connector available through the framework to 
implement an application. Leveraging the APIs of the connector framework 
applications will be able to find relevant sources systems and connect to them. 



  was:
Atlas is intended to provide a common approach to data governance and lineage 
across all systems and data within an organization.  Today Atlas provides 
access to metadata.   A connector provides access to a data source.  As 
connectors are the proxy of all data, they can also be explicit providers of 
metadata.   An open connector framework is being proposed to provide access to 
both data and the metadata Atlas provides together – the framework being the 
point at which data and metadata can be associated. This will help to better 
the exchange of information between platforms. It also offers new opportunities 
for the consistent enforcement of the governance policies and rules (e.g., 
rules of visibility).  Source connector/connection metadata provides the 
nucleus around which all other metadata describing the data builds.  

Introducing this framework:

Provides the necessary broker capabilities for normalized access to (finding 
and instantiating) a connector instance. 

Provides extension API supporting both partial and incremental adoption of the 
framework by existing connector provider runtimes allowing connector 
providers/contributors to adapt what they have vs re-write.

Introducing the notion of metadata-enabled open connectors:

Provides an API for the normalized access to the metadata describing a 
connector (asset type, connection type)

Provides base APIs for creating new connectors of different types and 
interaction styles.

The key personas using the framework are the tool developers and developers.  

As a connector developer, I want to create a new connector (or register an 
existing connector) to plug into the framework so that I can retrieve and 
manage metadata about what the source provides and ensure data used from that 
source has governance policies (e.g., rules of visibility) consistently 
configured and enforced. 

The tool developer uses a connector available through the framework to 
implement an application. Leveraging the APIs of the connector framework 
applications will be able to find relevant sources systems and connect to them. 




> Open connector framework
> 
>
> Key: ATLAS-1095
> URL: https://issues.apache.org/jira/browse/ATLAS-1095
> Project: Atlas
>  Issue Type: New Feature
>Affects Versions: 0.8-incubating
>Reporter: Stephanie Hazlewood
>Assignee: Mandy Chessell
>  Labels: VirtualDataConnector
>
> Atlas provides a common approach to metadata management and governance across 
> all systems and data within an organization.  Today Atlas provides access to 
> metadata.   A connector provides access to a data source.  As connectors are 
> the proxy of al

[jira] [Updated] (ATLAS-1095) Open connector framework

[Mandy Chessell (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-1095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mandy Chessell updated ATLAS-1095:
--
Summary: Open connector framework  (was: Define Requirements: Data and 
metadata association via open connector framework)

> Open connector framework
> 
>
> Key: ATLAS-1095
> URL: https://issues.apache.org/jira/browse/ATLAS-1095
> Project: Atlas
>  Issue Type: New Feature
>Affects Versions: 0.8-incubating
>Reporter: Stephanie Hazlewood
>Assignee: Mandy Chessell
>  Labels: VirtualDataConnector
>
> Atlas is intended to provide a common approach to data governance and lineage 
> across all systems and data within an organization.  Today Atlas provides 
> access to metadata.   A connector provides access to a data source.  As 
> connectors are the proxy of all data, they can also be explicit providers of 
> metadata.   An open connector framework is being proposed to provide access 
> to both data and the metadata Atlas provides together – the framework being 
> the point at which data and metadata can be associated. This will help to 
> better the exchange of information between platforms. It also offers new 
> opportunities for the consistent enforcement of the governance policies and 
> rules (e.g., rules of visibility).  Source connector/connection metadata 
> provides the nucleus around which all other metadata describing the data 
> builds.  
> Introducing this framework:
> Provides the necessary broker capabilities for normalized access to (finding 
> and instantiating) a connector instance. 
> Provides extension API supporting both partial and incremental adoption of 
> the framework by existing connector provider runtimes allowing connector 
> providers/contributors to adapt what they have vs re-write.
> Introducing the notion of metadata-enabled open connectors:
> Provides an API for the normalized access to the metadata describing a 
> connector (asset type, connection type)
> Provides base APIs for creating new connectors of different types and 
> interaction styles.
> The key personas using the framework are the tool developers and developers.  
> As a connector developer, I want to create a new connector (or register an 
> existing connector) to plug into the framework so that I can retrieve and 
> manage metadata about what the source provides and ensure data used from that 
> source has governance policies (e.g., rules of visibility) consistently 
> configured and enforced. 
> The tool developer uses a connector available through the framework to 
> implement an application. Leveraging the APIs of the connector framework 
> applications will be able to find relevant sources systems and connect to 
> them. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (ATLAS-1763) UI : Atlas lands in the page of Deleted tag (the only tag) on clicking TAGS tab

[Keval Bhatt (JIRA)]

[ 
https://issues.apache.org/jira/browse/ATLAS-1763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15998052#comment-15998052
 ] 

Keval Bhatt commented on ATLAS-1763:


[~madhan.neethiraj] You mean to say application cache ?

> UI : Atlas lands in the page of Deleted tag (the only tag) on clicking TAGS 
> tab 
> 
>
> Key: ATLAS-1763
> URL: https://issues.apache.org/jira/browse/ATLAS-1763
> Project: Atlas
>  Issue Type: Bug
>  Components: atlas-webui
>Affects Versions: 0.9-incubating
>Reporter: Sharmadha Sainath
>Assignee: Keval Bhatt
>Priority: Trivial
>
> 1. Created tag tag1
> 2. Clicked on TAGS tab which landed in 
> http://localhost:21000/index.html#!/tag/tagAttribute/tag1
> 3. Deleted the tag through REST.
> 4. Clicked on the TAGS tab. The TAGS tab listed tag1 . But on disabling cache 
> , tag1 is removed.
> 5.But Atlas lands in 
> http://localhost:21000/index.html#!/tag/tagAttribute/tag1 which throws 404 
> exception while clicking Tags tab .
> On opening Atlas in another incognito window , this issue is not seen . Atlas 
> possibly caches tag1 which lands in /tag/tagAttribute/tag1 and causes issue.
> When there are 1 or more ACTIVE tags , this issue is not seen as it lands in 
> /tag/tagAttribute/.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (ATLAS-1690) Introduce top level relationships

[David Radley (JIRA)]

[ 
https://issues.apache.org/jira/browse/ATLAS-1690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15997971#comment-15997971
 ] 

David Radley commented on ATLAS-1690:
-

Thank you very much for the feedback [~sarath.ku...@gmail.com] and [~cmgrote] . 
I am very much looking forward to your responses.

To respond to the Sarath's points : 

1. I do not think we should have "to" and "from". This implies a direction. For 
bidirection associations we do not want to imply there is any direction. 
Aggregations have a containership relationship - I think the important piece 
here is the container side of things not the direction.  
2. I am wondering about the use case for a relationship to have more than one 
type. We can allow more than one type here if there is a compelling use case.
3. Interesting idea. You are right - tag propagation is important. As these are 
not bidirectional relationships, there is not an implied direction-it is not 
obvious which way the propagation should flow. I could see us wanting to do 
this sort of thing for composition (which are not modelled by a top level 
relationship). I suspect it is less likely that for 2 independent entities we 
would want to propagate tags using flags in the types. 
Ideally propagation of tags should be defined in Ranger rules - as we are 
likely to want different tag propagation in different contexts. Can I suggest 
we deal with classification in a subsequent design? We can then cover Semantic 
classification which will actually be a relationship. 
4.  Yes this is the same as Suma's suggestion of a category; I was thinking of 
"bidirectional”, “aggregation” - as association could be directional and 
contains could be composition . I will put this in, as we can then put this 
value into the edge - so we can identify which edges we need to expand into 
attributes. Are you OK with these amendments?  
 
I am thinking we still need an isContains flag to indicate the container end of 
a relationship.
   
On your examples
1. I think a collection to member relationship would be an aggregation. 
2. and 3. are compositions. There is a case to model composition in the top 
level relationship as well. I suggest we consider that later to reduce the 
migration impact of this change. 
4 I have not looked at lineage. 

Responding to Chis's comments 

I can see that tag propagation direction is important; I am not sure the type 
itself should have these indicators. In the wider VDC architecture proposed in 
the Atlas wiki,  the OMRS would not be responsible for tag propagation. A 
second layer -the OMAS layer exposes data in a particular shape for consumption 
in a context, eg around assets or the glossary or catalog search. Each of these 
OMAS layers are likely to have different tag propagation requirements. I am 
therefore reticent to have tag propagation at this lower layer. By not 
propagating at the OMRS layer classifications are defined once and are not 
duplicated. As above, I suggest we enhance the classifications design after the 
glossary. 











> Introduce top level relationships
> -
>
> Key: ATLAS-1690
> URL: https://issues.apache.org/jira/browse/ATLAS-1690
> Project: Atlas
>  Issue Type: Improvement
>Reporter: David Radley
>Assignee: David Radley
>  Labels: VirtualDataConnector
> Attachments: Atlas_RelationDef_Json_Structure_v1.pdf, Atlas 
> Relationships proposal v1.0.pdf, Atlas Relationships proposal v1.1.pdf, Atlas 
> Relationships proposal v1.2.pdf, Atlas Relationships proposal v1.3.pdf, Atlas 
> Relationships proposal v1.4.pdf, Atlas Relationships proposal v1.5.pdf, Atlas 
> Relationships proposal v1.6.pdf, Atlas Relationships proposal v1.7.pdf
>
>
> Introduce top level relationships including support for 
> -many to many relationships
> - relationship names including the name for both ends and the relationship.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: Review Request 58985: Type deletion should invalidate property key in titan

[David Radley]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58985/#review174009
---




graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphManagement.java
Lines 69 (patched)


It does not seem right to rename property keys in old types. This seems to 
me to be changing history - which is not great for audit. 

I do not understand why the name type is restricted to the old name - can 
we not change this logic? 
I would suggest we tolerate more than one type of the same name - only one 
of which can be active at any one time - each type chould have their own shape 
and guid. All references to a type should be by guid - so we should not get 
confused by reusing a name.


- David Radley


On May 4, 2017, 10:07 p.m., Apoorv Naik wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/58985/
> ---
> 
> (Updated May 4, 2017, 10:07 p.m.)
> 
> 
> Review request for atlas.
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> This change supports the use-case where the user creates a type and deletes 
> it sometime later, now the redefinition of the type is restricted to use the 
> same data type for the attributes used during the initial creation. 
> 
> Solution: Rename the propertyKey corresponding to that attribute using the 
> titan management API by suffing _deleted_xxx where xxx is an increasing 
> integer sequence starting from 0. The reason for increment of xxx is that the 
> type create, delete and re-create can be done multiple times without any 
> conflicting key in Titan.
> 
> 
> Diffs
> -
> 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphManagement.java
>  5efd7c0f 
>   
> graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/Titan0GraphManagement.java
>  9a17d9ee 
>   
> graphdb/titan1/src/main/java/org/apache/atlas/repository/graphdb/titan1/Titan1GraphManagement.java
>  12faeda5 
>   intg/src/main/java/org/apache/atlas/listener/TypeDefChangeListener.java 
> 124f106f 
>   intg/src/main/java/org/apache/atlas/type/AtlasTypeRegistry.java ea2a7038 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java
>  47dccf19 
>   
> repository/src/test/java/org/apache/atlas/repository/impexp/ExportServiceTest.java
>  a7fc24cf 
>   
> repository/src/test/java/org/apache/atlas/repository/store/graph/AtlasTypeDefGraphStoreTest.java
>  84ad72c8 
> 
> 
> Diff: https://reviews.apache.org/r/58985/diff/2/
> 
> 
> Testing
> ---
> 
> mvn clean package and mvn clean install run successfully.
> 
> 
> Thanks,
> 
> Apoorv Naik
> 
>

[jira] [Commented] (ATLAS-1690) Introduce top level relationships

[Christopher Grote (JIRA)]

[ 
https://issues.apache.org/jira/browse/ATLAS-1690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15997909#comment-15997909
 ] 

Christopher Grote commented on ATLAS-1690:
--

While in general I agree with [~davidrad]'s prior suggestions that in general 
relationships should be bi-directional in nature, so am at first hesitant to 
think in terms of "from" and "to", nonetheless I think 
[~sarath.ku...@gmail.com] is raising a good point about the explicit 
directionality being potentially important for tag propagation.

To further describe the inherent bi-directionality, would it be worthwhile to 
capture a description within the "from" and "to" as well, e.g.

{code:javascript}
{
  "relationDef": {
"name" : "CollectionToEntityRelation",
"description" : "Entity Collection relationship",
"from": {
  "types" : [ "Collection" ],
  "cardinality": "SINGLE",
  "name" : "group",
  "description": "Has entities"
},
"to": {
  "types" : ["entity"],
  "cardinality": "SET",
  "name": "member",
  "description": "Within collection"
},
"propagateTags": true,
"relationType" : "association"
  }
}
{code}

(Though in this particular example, I would further assume that a particular 
entity could be in more than one collection, so not sure the cardinality is 
correct on the "from" side; though perhaps it's the relationType of 
"association" rather than "contains" that addresses this?)

What about "propagatesTags" being an enum rather than a simple boolean, e.g. 
"FROM-TO", "TO-FROM", "BI-DIRECTIONAL"; in case there are scenarios where it 
would make sense to apply the super-set of tags (from both sides of the 
relationship)?

> Introduce top level relationships
> -
>
> Key: ATLAS-1690
> URL: https://issues.apache.org/jira/browse/ATLAS-1690
> Project: Atlas
>  Issue Type: Improvement
>Reporter: David Radley
>Assignee: David Radley
>  Labels: VirtualDataConnector
> Attachments: Atlas_RelationDef_Json_Structure_v1.pdf, Atlas 
> Relationships proposal v1.0.pdf, Atlas Relationships proposal v1.1.pdf, Atlas 
> Relationships proposal v1.2.pdf, Atlas Relationships proposal v1.3.pdf, Atlas 
> Relationships proposal v1.4.pdf, Atlas Relationships proposal v1.5.pdf, Atlas 
> Relationships proposal v1.6.pdf, Atlas Relationships proposal v1.7.pdf
>
>
> Introduce top level relationships including support for 
> -many to many relationships
> - relationship names including the name for both ends and the relationship.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)