[jira] [Created] (ATLAS-1960) Import command fired on passive server throws Exception

[Sharmadha Sainath (JIRA)]

Sharmadha Sainath created ATLAS-1960:


 Summary: Import command fired on passive server throws Exception
 Key: ATLAS-1960
 URL: https://issues.apache.org/jira/browse/ATLAS-1960
 Project: Atlas
  Issue Type: Bug
  Components:  atlas-core
Affects Versions: 0.9-incubating
Reporter: Sharmadha Sainath
Priority: Critical
 Attachments: ImportOnPassiveHost.txt

1.Fired import command on an ACTIVE host which resulted in successful import.
2. On firing import command  with a zip file containing exported items of a 
hive_table ,on PASSIVE host, following exception is thrown :

{code}
{"errorCode":"ATLAS-500-00-001","errorMessage":"org.apache.atlas.exception.AtlasBaseException:
 org.apache.atlas.typesystem.exception.TypeNotFoundException: Unknown datatype: 
hive_table"}
{code}

Expected 30X with redirection URL.

Attached the complete exception stack trace found in PASSIVE host's application 
logs.




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (ATLAS-1961) Basic search improvement in use of Solr index for attribute filtering

[Madhan Neethiraj (JIRA)]

Madhan Neethiraj created ATLAS-1961:
---

 Summary: Basic search improvement in use of Solr index for 
attribute filtering
 Key: ATLAS-1961
 URL: https://issues.apache.org/jira/browse/ATLAS-1961
 Project: Atlas
  Issue Type: Improvement
  Components:  atlas-core
Affects Versions: 0.9-incubating, 0.8.1-incubating
Reporter: Madhan Neethiraj
Assignee: Madhan Neethiraj


Current basic search implementation uses Solr only when both type and attribute 
filters can be done via Solr. When the type specified has too many subtypes, 
Solr is not used - even though Solr could be used for attribute filtering. Use 
of Solr for attribute filtering can be help quicker response.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Review Request 60940: ATLAS-1961: Basic search improvement in use of Solr index for attribute filtering

[Madhan Neethiraj]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60940/
---

Review request for atlas.


Bugs: ATLAS-1961
https://issues.apache.org/jira/browse/ATLAS-1961


Repository: atlas


Description
---

Basic search improvement in use of Solr index for attribute filtering


Diffs
-

  
repository/src/main/java/org/apache/atlas/discovery/EntitySearchProcessor.java 
605cb15d 
  repository/src/main/java/org/apache/atlas/discovery/SearchContext.java 
55a07f3d 
  repository/src/main/java/org/apache/atlas/discovery/SearchProcessor.java 
14818094 


Diff: https://reviews.apache.org/r/60940/diff/1/


Testing
---

Validated that the search completes much quicker with this update.


Thanks,

Madhan Neethiraj

[jira] [Updated] (ATLAS-1961) Basic search improvement in use of Solr index for attribute filtering

[Madhan Neethiraj (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-1961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Madhan Neethiraj updated ATLAS-1961:

Attachment: ATLAS-1961.patch

Review board URL: https://reviews.apache.org/r/60940/

> Basic search improvement in use of Solr index for attribute filtering
> -
>
> Key: ATLAS-1961
> URL: https://issues.apache.org/jira/browse/ATLAS-1961
> Project: Atlas
>  Issue Type: Improvement
>  Components:  atlas-core
>Affects Versions: 0.9-incubating, 0.8.1-incubating
>Reporter: Madhan Neethiraj
>Assignee: Madhan Neethiraj
> Fix For: 0.9-incubating, 0.8.1-incubating
>
> Attachments: ATLAS-1961.patch
>
>
> Current basic search implementation uses Solr only when both type and 
> attribute filters can be done via Solr. When the type specified has too many 
> subtypes, Solr is not used - even though Solr could be used for attribute 
> filtering. Use of Solr for attribute filtering can be help quicker response.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (ATLAS-1839) Area 2 of the open metadata model

[David Radley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-1839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Radley updated ATLAS-1839:

Attachment: (was: 0250RelatedTerms.json)

> Area 2 of the open metadata model
> -
>
> Key: ATLAS-1839
> URL: https://issues.apache.org/jira/browse/ATLAS-1839
> Project: Atlas
>  Issue Type: Task
>  Components:  atlas-core
>Affects Versions: 0.9-incubating
>Reporter: Mandy Chessell
>Assignee: David Radley
>  Labels: OpenMetadata, VirtualDataConnector
>
> This task delivers the JSON files for the new models that describe types for 
> Area 2 in the open metadata model. This area covers the glossary.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (ATLAS-1839) Area 2 of the open metadata model

[David Radley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-1839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Radley updated ATLAS-1839:

Attachment: (was: 0270SemanticAssignment.json)

> Area 2 of the open metadata model
> -
>
> Key: ATLAS-1839
> URL: https://issues.apache.org/jira/browse/ATLAS-1839
> Project: Atlas
>  Issue Type: Task
>  Components:  atlas-core
>Affects Versions: 0.9-incubating
>Reporter: Mandy Chessell
>Assignee: David Radley
>  Labels: OpenMetadata, VirtualDataConnector
>
> This task delivers the JSON files for the new models that describe types for 
> Area 2 in the open metadata model. This area covers the glossary.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (ATLAS-1839) Area 2 of the open metadata model

[David Radley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-1839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Radley updated ATLAS-1839:

Attachment: (was: 0230Terms.json)

> Area 2 of the open metadata model
> -
>
> Key: ATLAS-1839
> URL: https://issues.apache.org/jira/browse/ATLAS-1839
> Project: Atlas
>  Issue Type: Task
>  Components:  atlas-core
>Affects Versions: 0.9-incubating
>Reporter: Mandy Chessell
>Assignee: David Radley
>  Labels: OpenMetadata, VirtualDataConnector
>
> This task delivers the JSON files for the new models that describe types for 
> Area 2 in the open metadata model. This area covers the glossary.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (ATLAS-1839) Area 2 of the open metadata model

[David Radley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-1839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Radley updated ATLAS-1839:

Attachment: (was: 0220CategoryHierarchy.json)

> Area 2 of the open metadata model
> -
>
> Key: ATLAS-1839
> URL: https://issues.apache.org/jira/browse/ATLAS-1839
> Project: Atlas
>  Issue Type: Task
>  Components:  atlas-core
>Affects Versions: 0.9-incubating
>Reporter: Mandy Chessell
>Assignee: David Radley
>  Labels: OpenMetadata, VirtualDataConnector
>
> This task delivers the JSON files for the new models that describe types for 
> Area 2 in the open metadata model. This area covers the glossary.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (ATLAS-1839) Area 2 of the open metadata model

[David Radley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-1839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Radley updated ATLAS-1839:

Attachment: (was: 0280SpineObjects.json)

> Area 2 of the open metadata model
> -
>
> Key: ATLAS-1839
> URL: https://issues.apache.org/jira/browse/ATLAS-1839
> Project: Atlas
>  Issue Type: Task
>  Components:  atlas-core
>Affects Versions: 0.9-incubating
>Reporter: Mandy Chessell
>Assignee: David Radley
>  Labels: OpenMetadata, VirtualDataConnector
>
> This task delivers the JSON files for the new models that describe types for 
> Area 2 in the open metadata model. This area covers the glossary.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (ATLAS-1839) Area 2 of the open metadata model

[David Radley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-1839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Radley updated ATLAS-1839:

Attachment: (was: 0260Contexts.json)

> Area 2 of the open metadata model
> -
>
> Key: ATLAS-1839
> URL: https://issues.apache.org/jira/browse/ATLAS-1839
> Project: Atlas
>  Issue Type: Task
>  Components:  atlas-core
>Affects Versions: 0.9-incubating
>Reporter: Mandy Chessell
>Assignee: David Radley
>  Labels: OpenMetadata, VirtualDataConnector
>
> This task delivers the JSON files for the new models that describe types for 
> Area 2 in the open metadata model. This area covers the glossary.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (ATLAS-1839) Area 2 of the open metadata model

[David Radley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-1839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Radley updated ATLAS-1839:

Attachment: (was: 0210Glossary.json)

> Area 2 of the open metadata model
> -
>
> Key: ATLAS-1839
> URL: https://issues.apache.org/jira/browse/ATLAS-1839
> Project: Atlas
>  Issue Type: Task
>  Components:  atlas-core
>Affects Versions: 0.9-incubating
>Reporter: Mandy Chessell
>Assignee: David Radley
>  Labels: OpenMetadata, VirtualDataConnector
>
> This task delivers the JSON files for the new models that describe types for 
> Area 2 in the open metadata model. This area covers the glossary.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (ATLAS-1839) Area 2 of the open metadata model

[David Radley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-1839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Radley updated ATLAS-1839:

Attachment: 0210Glossary.json
0220CategoryHierarchy.json
0230Terms.json
0250RelatedTerms.json
0260Contexts.json
0270SemanticAssignment.json
0280SpineObjects.json
0240Dictionary.json
0005LinkedMediaTypes.json

Making all fields optional

> Area 2 of the open metadata model
> -
>
> Key: ATLAS-1839
> URL: https://issues.apache.org/jira/browse/ATLAS-1839
> Project: Atlas
>  Issue Type: Task
>  Components:  atlas-core
>Affects Versions: 0.9-incubating
>Reporter: Mandy Chessell
>Assignee: David Radley
>  Labels: OpenMetadata, VirtualDataConnector
> Attachments: 0005LinkedMediaTypes.json, 0210Glossary.json, 
> 0220CategoryHierarchy.json, 0230Terms.json, 0240Dictionary.json, 
> 0250RelatedTerms.json, 0260Contexts.json, 0270SemanticAssignment.json, 
> 0280SpineObjects.json
>
>
> This task delivers the JSON files for the new models that describe types for 
> Area 2 in the open metadata model. This area covers the glossary.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Comment Edited] (ATLAS-1839) Area 2 of the open metadata model

[David Radley (JIRA)]

[ 
https://issues.apache.org/jira/browse/ATLAS-1839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16091257#comment-16091257
 ] 

David Radley edited comment on ATLAS-1839 at 7/18/17 8:06 AM:
--

Making all fields optional in the latest attachments.


was (Author: davidrad):
Making all fields optional

> Area 2 of the open metadata model
> -
>
> Key: ATLAS-1839
> URL: https://issues.apache.org/jira/browse/ATLAS-1839
> Project: Atlas
>  Issue Type: Task
>  Components:  atlas-core
>Affects Versions: 0.9-incubating
>Reporter: Mandy Chessell
>Assignee: David Radley
>  Labels: OpenMetadata, VirtualDataConnector
> Attachments: 0005LinkedMediaTypes.json, 0210Glossary.json, 
> 0220CategoryHierarchy.json, 0230Terms.json, 0240Dictionary.json, 
> 0250RelatedTerms.json, 0260Contexts.json, 0270SemanticAssignment.json, 
> 0280SpineObjects.json
>
>
> This task delivers the JSON files for the new models that describe types for 
> Area 2 in the open metadata model. This area covers the glossary.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (ATLAS-1768) Create common types for Open Metadata

[Graham Wallis (JIRA)]

[ 
https://issues.apache.org/jira/browse/ATLAS-1768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16091391#comment-16091391
 ] 

Graham Wallis commented on ATLAS-1768:
--

In Area 2 (Glossary) within Spine Objects, do we also need an IsIdentifiedBy 
relationship between a (GlossaryTerm that is classified as a) SpineObject and a 
(GlossaryTerm that is classified as an) ObjectIdentifier?


> Create common types for Open Metadata
> -
>
> Key: ATLAS-1768
> URL: https://issues.apache.org/jira/browse/ATLAS-1768
> Project: Atlas
>  Issue Type: New Feature
>  Components:  atlas-core
>Affects Versions: 0.9-incubating
>Reporter: Mandy Chessell
>Assignee: Mandy Chessell
>  Labels: OpenMetadata
>
> This JIRA describes a proposal for standard types for open metadata entities 
> and relationships.  For example, glossaries, database definitions, rules, 
> policies, ...
> The value of having standard definitions for metadata is to enable type safe 
> APIs and business level UIs plus be able to exchange metadata between 
> different instances of metadata repositories.
> The implementation of these common types is divided into 8 areas:
> * Area 0 - for extensions to Apache Atlas's base model
> * Area 1 - for definitions of the data-related assets we are governing and 
> using
> * Area 2 - for a glossary of meanings and semantic relationships
> * Area 3 - for information about asset use, crowd-sourced definitions and 
> collaboration around the data-related assets
> * Area 4 - for governance such as policies, rules and classifications
> * Area 5 - for reference models and reference data
> * Area 6 - for metadata discovery processes (see 
> https://issues.apache.org/jira/browse/ATLAS-1748)
> * Area 7 - for lineage
> Adaptation and flexibility are key in metadata environments so these common 
> definitions must be extensible - and we still need to support the ad hoc 
> definition of new types in Atlas.
> Apache Atlas supports meta-types that are used in the definition of new 
> types.  These are currently enumeration, struct, classification and entity.  
> JIRA https://issues.apache.org/jira/browse/ATLAS-1690 adds relationships to 
> this list.  The open metadata models make use of all of these meta-types.  
> These are represented by sterotypes on the classes of the open metadata 
> definitions.
> The Atlas wiki has the models as a set of linked pages which are probably the 
> easiest way to view the models.
> Start here: 
> https://cwiki.apache.org/confluence/display/ATLAS/Building+out+the+Apache+Atlas+Typesystem



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (ATLAS-1962) User/group mapping rules similar to Hadoop's auth_to_local

[Hari Sekhon (JIRA)]

Hari Sekhon created ATLAS-1962:
--

 Summary: User/group mapping rules similar to Hadoop's auth_to_local
 Key: ATLAS-1962
 URL: https://issues.apache.org/jira/browse/ATLAS-1962
 Project: Atlas
  Issue Type: New Feature
  Components:  atlas-core, atlas-intg
Affects Versions: 0.8-incubating
 Environment: HDP 2.6 + Kerberos + AD LDAP multi-domain forest
Reporter: Hari Sekhon
Priority: Blocker


Feature Request to add user/group mapping rules similar to Hadoop's 
auth_to_local.

This will allow munging users/groups and rule based remappings to differentiate 
duplicate users in multi-domain Active Directory forests where the LDAP results 
returned from the global catalog include duplicate usernames which need to be 
translated with a prefix/suffix in order to differentiate between domains to 
prevent users from different domains sharing logins, permissions etc.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (ATLAS-1768) Create common types for Open Metadata

[Graham Wallis (JIRA)]

[ 
https://issues.apache.org/jira/browse/ATLAS-1768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16091399#comment-16091399
 ] 

Graham Wallis commented on ATLAS-1768:
--

Within Semantic Assignment does this only apply to Asset. If so should we use 
Asset rather than Referenceable in the model?

> Create common types for Open Metadata
> -
>
> Key: ATLAS-1768
> URL: https://issues.apache.org/jira/browse/ATLAS-1768
> Project: Atlas
>  Issue Type: New Feature
>  Components:  atlas-core
>Affects Versions: 0.9-incubating
>Reporter: Mandy Chessell
>Assignee: Mandy Chessell
>  Labels: OpenMetadata
>
> This JIRA describes a proposal for standard types for open metadata entities 
> and relationships.  For example, glossaries, database definitions, rules, 
> policies, ...
> The value of having standard definitions for metadata is to enable type safe 
> APIs and business level UIs plus be able to exchange metadata between 
> different instances of metadata repositories.
> The implementation of these common types is divided into 8 areas:
> * Area 0 - for extensions to Apache Atlas's base model
> * Area 1 - for definitions of the data-related assets we are governing and 
> using
> * Area 2 - for a glossary of meanings and semantic relationships
> * Area 3 - for information about asset use, crowd-sourced definitions and 
> collaboration around the data-related assets
> * Area 4 - for governance such as policies, rules and classifications
> * Area 5 - for reference models and reference data
> * Area 6 - for metadata discovery processes (see 
> https://issues.apache.org/jira/browse/ATLAS-1748)
> * Area 7 - for lineage
> Adaptation and flexibility are key in metadata environments so these common 
> definitions must be extensible - and we still need to support the ad hoc 
> definition of new types in Atlas.
> Apache Atlas supports meta-types that are used in the definition of new 
> types.  These are currently enumeration, struct, classification and entity.  
> JIRA https://issues.apache.org/jira/browse/ATLAS-1690 adds relationships to 
> this list.  The open metadata models make use of all of these meta-types.  
> These are represented by sterotypes on the classes of the open metadata 
> definitions.
> The Atlas wiki has the models as a set of linked pages which are probably the 
> easiest way to view the models.
> Start here: 
> https://cwiki.apache.org/confluence/display/ATLAS/Building+out+the+Apache+Atlas+Typesystem



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Review Request 60705: ATLAS-1927 : UI - changing the look and feel of attribute in details page.

[Kalyani Kashikar]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60705/
---

(Updated July 18, 2017, 11:04 a.m.)


Review request for atlas, keval bhatt, Madhan Neethiraj, and Nixon Rodrigues.


Changes
---

Attribute table size issue fixed and change values alignment based on Data-type.


Bugs: ATLAS-1927
https://issues.apache.org/jira/browse/ATLAS-1927


Repository: atlas


Description
---

Show the attributes of entities in tag format instead of showing comma(,) 
separator.


Diffs (updated)
-

  dashboardv2/public/css/scss/tag.scss fc0652b 
  dashboardv2/public/js/views/detail_page/DetailPageLayoutView.js c45755f 
  dashboardv2/public/js/views/tag/TagDetailTableLayoutView.js 45cdd6b 
  dashboardv2/public/js/views/tag/addTagModalView.js 97d80bd 


Diff: https://reviews.apache.org/r/60705/diff/5/

Changes: https://reviews.apache.org/r/60705/diff/4-5/


Testing
---

verified the attributes of entities is showing in tag format instead of showing 
comma(,) separator.


Thanks,

Kalyani Kashikar

[jira] [Updated] (ATLAS-1927) UI - changing the look and feel of attribute in details page.

[Kalyani Kashikar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-1927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kalyani Kashikar updated ATLAS-1927:

Attachment: ATLAS-1927.4.patch

> UI - changing the look and feel of attribute in details page.
> -
>
> Key: ATLAS-1927
> URL: https://issues.apache.org/jira/browse/ATLAS-1927
> Project: Atlas
>  Issue Type: Bug
>Reporter: Kalyani Kashikar
>Assignee: Kalyani Kashikar
> Attachments: ATLAS-1927.1.patch, ATLAS-1927.2.patch, 
> ATLAS-1927.3.patch, ATLAS-1927.4.patch, ATLAS-1927.patch
>
>
> Show the attributes of entities in tag format instead of showing comma(,) 
> separator.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (ATLAS-1698) Create Glossary OMAS API

[Graham Wallis (JIRA)]

[ 
https://issues.apache.org/jira/browse/ATLAS-1698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16091419#comment-16091419
 ] 

Graham Wallis commented on ATLAS-1698:
--

Regarding CRUD:
To expand on the listed CRUD operations, is it fair to say that a user would 
need to:
* Create - either a Glossary, a Term, Category or Classification
* Create - a relationship between a pair of Categories (for hierarchy), a 
Category and a Term, or between a pair of Terms (for any of context, related 
terms, spine objects).
* Create - a classification relationship between a Classification and either a 
Glossary, Category or Term
* Create - a relationship between a Referenceable and a Term (for Semantic 
Assignment)
* Read - any of a Glossary, Term, Category, Classification or any of the 
relationships listed above
* Do they also need to read the Entity in a Semantic Assignment, and if so does 
that include the entity's attributes and/or relationships?
* Update of any of the above.
* Delete of any of the above.

Regarding graph formats:
There are a number of graph serialisation formats - including both JSON and 
XML. The most popular are GraphML (XML) and GraphSON (JSON). They are both well 
supported by Tinkerpop. Personally I think GraphSON is quite verbose, but 
that's just my opinion. Regarding jsongraph (as suggested in the PDF and which 
is different to GraphSON) I suspect that it may not currently be sufficiently 
expressive for Atlas. The lack of edge ids is an example. We could of course 
contribute to jsongraph, but we should choose carefully what format(s) to 
support.

Regarding graph-style visualization:
D3 is a good visualization tool, but be aware that it's scalability is limited 
in relation to drawing graphs - a few thousand elements is OK but SVG will max 
out above that. Canvas scales better. WebGL scales better still (but is harder 
to use).

Regarding the REST API:
Do you intentionally have both v2/omas/glossary/{guid} and 
v2/omas/glossary/guid/{guid} ?
Wouldn't all v2/omas/glossary/... URIs need the glossary {guid)? 
Wouldn't v2/omas/glossary/category/classifications also need the category 
{guid}? 
Wouldn't v2/omas/glossary/term/classifications also need the term {guid}? 
Should we also have v2/omas/glossary/{guid}/term/{guid}/assets, to allow for 
retrieval of all assets with a specified term?
And v2/omas/glossary/{guid}/assets/{guid}/terms to retrieve all the terms 
associated with an Asset?
Do we also need to allow navigation via Term to (set of) Category - i.e. 
v2/omas/glossary/{guid}/term/{guid}/categories?
Does the REST API also need to support exploration by relationship type? e.g. 
retrieve Terms that are Synonyms of a specified Term? I guess that might look 
something like  v2/omas/glossary/{guid}/term/{guid}/synonyms. Similarly for 
other types of Term<->Term relationship.

> Create Glossary OMAS API
> 
>
> Key: ATLAS-1698
> URL: https://issues.apache.org/jira/browse/ATLAS-1698
> Project: Atlas
>  Issue Type: Sub-task
>Reporter: David Radley
>Assignee: David Radley
>  Labels: VirtualDataConnector
> Attachments: apidocs-4.zip, Atlas Glossary OMAS API proposal v1.0 
> .pdf, Atlas Glossary OMAS API proposal v1.1.pdf
>
>
> The Glossary OMAS provides a specialized API for glossary applications to 
> retrieve and store their glossary metadata and link assets of different types 
> to these glossary entries.
> The Glossary OMAS makes heavy use of the Area 2 open metadata model.  See 
> https://cwiki.apache.org/confluence/display/ATLAS/Area+2+-+Glossary



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (ATLAS-1963) Relationship instance creation does not result in the expected entity instance content

[David Radley (JIRA)]

David Radley created ATLAS-1963:
---

 Summary: Relationship instance creation does not result in the 
expected entity instance content 
 Key: ATLAS-1963
 URL: https://issues.apache.org/jira/browse/ATLAS-1963
 Project: Atlas
  Issue Type: Bug
Reporter: David Radley






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (ATLAS-1963) Relationship instance creation does not result in the expected entity instance content

[David Radley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-1963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Radley updated ATLAS-1963:

Attachment: 0210Glossary.json
0005LinkedMediaTypes.json
0220CategoryHierarchy.json

> Relationship instance creation does not result in the expected entity 
> instance content 
> ---
>
> Key: ATLAS-1963
> URL: https://issues.apache.org/jira/browse/ATLAS-1963
> Project: Atlas
>  Issue Type: Bug
>Reporter: David Radley
> Attachments: 0005LinkedMediaTypes.json, 0210Glossary.json, 
> 0220CategoryHierarchy.json
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (ATLAS-1963) Relationship instance creation does not result in the expected entity instance content

[David Radley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-1963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Radley updated ATLAS-1963:

Description: 
I create types using the attached  json files. I then create 2 GlossaryCategory 
instances. 
I then create a relationship from one category to another with 
CategoryHierarchyLink.

{
"end1": {
"guid"   : "12bcfbd3-1574-452f-8139-4d14147fa56b",
"typeName"   : "GlossaryCategory"
},
"end2": {
"guid"   : "1b7b5834-abc6-4ea4-bb6e-b10ae066724e",
"typeName"   : "GlossaryCategory"
},
"typeName"  : "CategoryHierarchyLink"
} 

I get one of the categories and do not see the populated parent or children 
category relationship in the relationship attributes.

> Relationship instance creation does not result in the expected entity 
> instance content 
> ---
>
> Key: ATLAS-1963
> URL: https://issues.apache.org/jira/browse/ATLAS-1963
> Project: Atlas
>  Issue Type: Bug
>Reporter: David Radley
> Attachments: 0005LinkedMediaTypes.json, 0210Glossary.json, 
> 0220CategoryHierarchy.json
>
>
> I create types using the attached  json files. I then create 2 
> GlossaryCategory instances. 
> I then create a relationship from one category to another with 
> CategoryHierarchyLink.
> {
> "end1": {
> "guid" : "12bcfbd3-1574-452f-8139-4d14147fa56b",
> "typeName" : "GlossaryCategory"
> },
> "end2": {
> "guid" : "1b7b5834-abc6-4ea4-bb6e-b10ae066724e",
> "typeName" : "GlossaryCategory"
> },
> "typeName"  : "CategoryHierarchyLink"
> } 
> I get one of the categories and do not see the populated parent or children 
> category relationship in the relationship attributes.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (ATLAS-1839) Area 2 of the open metadata model

[David Radley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-1839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Radley updated ATLAS-1839:

Attachment: (was: 0220CategoryHierarchy.json)

> Area 2 of the open metadata model
> -
>
> Key: ATLAS-1839
> URL: https://issues.apache.org/jira/browse/ATLAS-1839
> Project: Atlas
>  Issue Type: Task
>  Components:  atlas-core
>Affects Versions: 0.9-incubating
>Reporter: Mandy Chessell
>Assignee: David Radley
>  Labels: OpenMetadata, VirtualDataConnector
> Attachments: 0005LinkedMediaTypes.json, 0210Glossary.json, 
> 0240Dictionary.json, 0250RelatedTerms.json, 0260Contexts.json, 
> 0270SemanticAssignment.json, 0280SpineObjects.json
>
>
> This task delivers the JSON files for the new models that describe types for 
> Area 2 in the open metadata model. This area covers the glossary.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (ATLAS-1839) Area 2 of the open metadata model

[David Radley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-1839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Radley updated ATLAS-1839:

Attachment: (was: 0230Terms.json)

> Area 2 of the open metadata model
> -
>
> Key: ATLAS-1839
> URL: https://issues.apache.org/jira/browse/ATLAS-1839
> Project: Atlas
>  Issue Type: Task
>  Components:  atlas-core
>Affects Versions: 0.9-incubating
>Reporter: Mandy Chessell
>Assignee: David Radley
>  Labels: OpenMetadata, VirtualDataConnector
> Attachments: 0005LinkedMediaTypes.json, 0210Glossary.json, 
> 0240Dictionary.json, 0250RelatedTerms.json, 0260Contexts.json, 
> 0270SemanticAssignment.json, 0280SpineObjects.json
>
>
> This task delivers the JSON files for the new models that describe types for 
> Area 2 in the open metadata model. This area covers the glossary.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (ATLAS-1839) Area 2 of the open metadata model

[David Radley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-1839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Radley updated ATLAS-1839:

Attachment: 0230Terms.json
0220CategoryHierarchy.json

> Area 2 of the open metadata model
> -
>
> Key: ATLAS-1839
> URL: https://issues.apache.org/jira/browse/ATLAS-1839
> Project: Atlas
>  Issue Type: Task
>  Components:  atlas-core
>Affects Versions: 0.9-incubating
>Reporter: Mandy Chessell
>Assignee: David Radley
>  Labels: OpenMetadata, VirtualDataConnector
> Attachments: 0005LinkedMediaTypes.json, 0210Glossary.json, 
> 0220CategoryHierarchy.json, 0230Terms.json, 0240Dictionary.json, 
> 0250RelatedTerms.json, 0260Contexts.json, 0270SemanticAssignment.json, 
> 0280SpineObjects.json
>
>
> This task delivers the JSON files for the new models that describe types for 
> Area 2 in the open metadata model. This area covers the glossary.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (ATLAS-1698) Create Glossary OMAS API

[David Radley (JIRA)]

[ 
https://issues.apache.org/jira/browse/ATLAS-1698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16091541#comment-16091541
 ] 

David Radley commented on ATLAS-1698:
-

Thanks [~grahamwallis]
great feedback. somments in <> sections

Regarding CRUD:
To expand on the listed CRUD operations, is it fair to say that a user would 
need to:
Create - either a Glossary, a Term, Category or Classification <>
Create - a relationship between a pair of Categories (for hierarchy), a 
Category and a Term, or between a pair of Terms (for any of context, related 
terms, spine objects).  <>
Create - a classification relationship between a Classification and either a 
Glossary, Category or Term <>- 
Create - a relationship between a Referenceable and a Term (for Semantic 
Assignment) <>
Read - any of a Glossary, Term, Category, Classification or any of the 
relationships listed above <>
Do they also need to read the Entity in a Semantic Assignment, and if so does 
that include the entity's attributes and/or relationships? <>
Update of any of the above.
Delete of any of the above.
Regarding graph formats:
There are a number of graph serialisation formats - including both JSON and 
XML. The most popular are GraphML (XML) and GraphSON (JSON). They are both well 
supported by Tinkerpop. Personally I think GraphSON is quite verbose, but 
that's just my opinion. Regarding jsongraph (as suggested in the PDF and which 
is different to GraphSON) I suspect that it may not currently be sufficiently 
expressive for Atlas. The lack of edge ids is an example. We could of course 
contribute to jsongraph, but we should choose carefully what format(s) to 
support.<>
Regarding graph-style visualization:
D3 is a good visualization tool, but be aware that it's scalability is limited 
in relation to drawing graphs - a few thousand elements is OK but SVG will max 
out above that. Canvas scales better. WebGL scales better still (but is harder 
to use).
Regarding the REST API:
Do you intentionally have both v2/omas/glossary/
{guid} and v2/omas/glossary/guid/{guid} 
?  <>
Wouldn't all v2/omas/glossary/... URIs need the glossary {guid)?  <>
Wouldn't v2/omas/glossary/category/classifications also need the category 
<>
{guid}? 
Wouldn't v2/omas/glossary/term/classifications also need the term {guid} 
<>
? 
Should we also have v2/omas/glossary/
{guid}/term/{guid} 
/assets, to allow for retrieval of all assets with a specified term? <> 
And v2/omas/glossary/
{guid}/assets/{guid} <>  
/terms to retrieve all the terms associated with an Asset? <> 
Do we also need to allow navigation via Term to (set of) Category - i.e. 
v2/omas/glossary
{guid}/term/{guid} / <>
/categories?
Does the REST API also need to support exploration by relationship type? e.g. 
retrieve Terms that are Synonyms of a specified Term? I guess that might look 
something like v2/omas/glossary/
{guid}/term/{guid} 
/synonyms. Similarly for other types of Term<->Term relationship. <>


> Create Glossary OMAS API
> 
>
> Key: ATLAS-1698
> URL: https://issues.apache.org/jira/browse/ATLAS-1698
> Project: Atlas
>  Issue Type: Sub-task
>Reporter: David Radley
>Assignee: David Radley
>  Labels: VirtualDataConnector
> Attachments: apidocs-4.zip, Atlas Glossary OMAS API proposal v1.0 
> .pdf, Atlas Glossary OMAS API proposal v1.1.pdf
>
>
> The Glossary OMAS provides a specialized API for glossary applications to 
> retrieve and store their glossary metadata and link assets of different types 
> to these glossary entries.
> The Glossary OMAS makes heavy use of the Area 2 open metadata model.  See 
> https://cwiki.apache.org/confluence/display/ATLAS/Area+2+-+Glossary



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (ATLAS-1964) UI : Support to resize and order columns in Search table

[Keval Bhatt (JIRA)]

Keval Bhatt created ATLAS-1964:
--

 Summary: UI : Support to resize and order columns in Search table
 Key: ATLAS-1964
 URL: https://issues.apache.org/jira/browse/ATLAS-1964
 Project: Atlas
  Issue Type: Bug
Affects Versions: 0.9-incubating
Reporter: Keval Bhatt
Assignee: Keval Bhatt






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (ATLAS-1698) Create Glossary OMAS API

[Graham Wallis (JIRA)]

[ 
https://issues.apache.org/jira/browse/ATLAS-1698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16091636#comment-16091636
 ] 

Graham Wallis commented on ATLAS-1698:
--

Thanks - and apologies that my example URIs did not format in a very readable 
manner! 

> Create Glossary OMAS API
> 
>
> Key: ATLAS-1698
> URL: https://issues.apache.org/jira/browse/ATLAS-1698
> Project: Atlas
>  Issue Type: Sub-task
>Reporter: David Radley
>Assignee: David Radley
>  Labels: VirtualDataConnector
> Attachments: apidocs-4.zip, Atlas Glossary OMAS API proposal v1.0 
> .pdf, Atlas Glossary OMAS API proposal v1.1.pdf
>
>
> The Glossary OMAS provides a specialized API for glossary applications to 
> retrieve and store their glossary metadata and link assets of different types 
> to these glossary entries.
> The Glossary OMAS makes heavy use of the Area 2 open metadata model.  See 
> https://cwiki.apache.org/confluence/display/ATLAS/Area+2+-+Glossary



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Review Request 60705: ATLAS-1927 : UI - changing the look and feel of attribute in details page.

[Madhan Neethiraj]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60705/#review180812
---


Ship it!




Ship It!

- Madhan Neethiraj


On July 18, 2017, 11:04 a.m., Kalyani Kashikar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60705/
> ---
> 
> (Updated July 18, 2017, 11:04 a.m.)
> 
> 
> Review request for atlas, keval bhatt, Madhan Neethiraj, and Nixon Rodrigues.
> 
> 
> Bugs: ATLAS-1927
> https://issues.apache.org/jira/browse/ATLAS-1927
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Show the attributes of entities in tag format instead of showing comma(,) 
> separator.
> 
> 
> Diffs
> -
> 
>   dashboardv2/public/css/scss/tag.scss fc0652b 
>   dashboardv2/public/js/views/detail_page/DetailPageLayoutView.js c45755f 
>   dashboardv2/public/js/views/tag/TagDetailTableLayoutView.js 45cdd6b 
>   dashboardv2/public/js/views/tag/addTagModalView.js 97d80bd 
> 
> 
> Diff: https://reviews.apache.org/r/60705/diff/5/
> 
> 
> Testing
> ---
> 
> verified the attributes of entities is showing in tag format instead of 
> showing comma(,) separator.
> 
> 
> Thanks,
> 
> Kalyani Kashikar
> 
>

Re: Review Request 60940: ATLAS-1961: Basic search improvement in use of Solr index for attribute filtering

[Apoorv Naik]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60940/#review180814
---


Ship it!




Ship It!

- Apoorv Naik


On July 18, 2017, 7:32 a.m., Madhan Neethiraj wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60940/
> ---
> 
> (Updated July 18, 2017, 7:32 a.m.)
> 
> 
> Review request for atlas.
> 
> 
> Bugs: ATLAS-1961
> https://issues.apache.org/jira/browse/ATLAS-1961
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Basic search improvement in use of Solr index for attribute filtering
> 
> 
> Diffs
> -
> 
>   
> repository/src/main/java/org/apache/atlas/discovery/EntitySearchProcessor.java
>  605cb15d 
>   repository/src/main/java/org/apache/atlas/discovery/SearchContext.java 
> 55a07f3d 
>   repository/src/main/java/org/apache/atlas/discovery/SearchProcessor.java 
> 14818094 
> 
> 
> Diff: https://reviews.apache.org/r/60940/diff/1/
> 
> 
> Testing
> ---
> 
> Validated that the search completes much quicker with this update.
> 
> 
> Thanks,
> 
> Madhan Neethiraj
> 
>

Re: Review Request 60927: ATLAS-1957: Multi-vertex atlas graph query

[Madhan Neethiraj]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60927/#review180774
---




graphdb/common/src/main/java/org/apache/atlas/repository/graphdb/titan/query/NativeTitanMultiVertexQuery.java
Lines 29 (patched)


This interface doesn't seem to be used. Please review. If not required, 
please remove.


- Madhan Neethiraj


On July 17, 2017, 10:31 p.m., Apoorv Naik wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60927/
> ---
> 
> (Updated July 17, 2017, 10:31 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, Sarath 
> Subramanian, and Suma Shivaprasad.
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Initial draft for multi-vertex graph query in atlas
> 
> 
> Diffs
> -
> 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java 
> dded76f8 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasMultiVertexQuery.java
>  PRE-CREATION 
>   
> graphdb/common/src/main/java/org/apache/atlas/repository/graphdb/titan/query/NativeTitanMultiVertexQuery.java
>  PRE-CREATION 
>   
> graphdb/common/src/main/java/org/apache/atlas/repository/graphdb/titan/query/NativeTitanQueryFactory.java
>  ac7ff9e8 
>   graphdb/titan0/pom.xml ceda1bb2 
>   
> graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/Titan0Graph.java
>  2408287b 
>   
> graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/query/Titan0AtlasMultiVertexQuery.java
>  PRE-CREATION 
>   
> graphdb/titan1/src/main/java/org/apache/atlas/repository/graphdb/titan1/Titan1Graph.java
>  ffb6b37f 
>   
> graphdb/titan1/src/main/java/org/apache/atlas/repository/graphdb/titan1/query/Titan1AtlasMultiVertexQuery.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/discovery/ClassificationSearchProcessor.java
>  77b2c7c2 
> 
> 
> Diff: https://reviews.apache.org/r/60927/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Apoorv Naik
> 
>

[jira] [Updated] (ATLAS-1948) Importing hive_table in a database which is a CTAS of another table in different database throws exception due to export order.

[Ashutosh Mestry (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-1948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Mestry updated ATLAS-1948:
---
Attachment: ATLAS-1948-Residual-import.patch

> Importing hive_table in a database which is a CTAS of another table in 
> different database throws exception due to export order.
> ---
>
> Key: ATLAS-1948
> URL: https://issues.apache.org/jira/browse/ATLAS-1948
> Project: Atlas
>  Issue Type: Bug
>  Components:  atlas-core
>Affects Versions: 0.9-incubating
>Reporter: Sharmadha Sainath
>Assignee: Ashutosh Mestry
>Priority: Critical
> Fix For: 0.9-incubating
>
> Attachments: ATLAS-1948-Residual-import.patch, db1tb1.zip, 
> ImportTransformsErrorOnCTASonDiffDB.txt
>
>
> 1.Created 2 databases db1 , db2 in cluster1
> 2.Created 2 tables
>   1. db1.t1
>   2. db2.t2 as select * from db1.t1
> 3.Exported db1.t1 into zip file.
> 4.Imported zip file into cluster 2 with transforms option :
> {code}
> {
>   "options": {
>"transforms": "{ \"hive_column\": { \"qualifiedName\": [ 
> \"replace:cl1:cl2\" ]} }"
>   }
> }
> {code}
> 5. Import fails with 
> {code}
> {"errorCode":"ATLAS-500-00-001","errorMessage":"org.apache.atlas.exception.AtlasBaseException:
>  ObjectId is not valid 
> AtlasObjectId{guid='51c77c1e-265e-46ab-bbb5-5316cf80a53c', 
> typeName='hive_column', uniqueAttributes={}}"}
> {code}
> Only db1.t1 is imported into Atlas without any lineage. 
> Attached the exception stack trace.
> After this exporting db2.t2 and importing completes successfully.
> That is , first import ,either db1.t1 or db2.t1 is unsuccessful with 
> exception. Next import is successful.
> The exception *doesn't* happen and tables are successfully imported If both 
> the tables are in a single database. Export order if tables are in same db is 
> 1.table1, 
> 2.db,
> 3.table2, 
> 4.hive_process
> 5. hive_column_lineage
> If the tables are in different db , the order is ,
> 1.table1, 
> 2.db1,
> 3.hive_process,
> 4.hive_column_lineage
> 5.ctas table
> 6.db2  
> which is possibly causing the issue. 
> When cluster2  starts importing , it imports table1 , db1 and when it comes 
> to hive_column_lineage , it finds that column specified in 
> hive_column_lineage is not in cluster2 yet ,since ctas table comes after the 
> hive_column_lineage in import order and it throws "ObjectId is not valid 
> AtlasObjectId{guid='51c77c1e-265e-46ab-bbb5-5316cf80a53c', 
> typeName='hive_column' ".
> Thanks [~ayubkhan] for the analysis.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (ATLAS-1948) Importing hive_table in a database which is a CTAS of another table in different database throws exception due to export order.

[Ashutosh Mestry (JIRA)]

[ 
https://issues.apache.org/jira/browse/ATLAS-1948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16091983#comment-16091983
 ] 

Ashutosh Mestry commented on ATLAS-1948:


Patch contains: 
* Export: Ordering for process entities. 
* Import: Updated logic to handle export with incorrect ordering.

> Importing hive_table in a database which is a CTAS of another table in 
> different database throws exception due to export order.
> ---
>
> Key: ATLAS-1948
> URL: https://issues.apache.org/jira/browse/ATLAS-1948
> Project: Atlas
>  Issue Type: Bug
>  Components:  atlas-core
>Affects Versions: 0.9-incubating
>Reporter: Sharmadha Sainath
>Assignee: Ashutosh Mestry
>Priority: Critical
> Fix For: 0.9-incubating
>
> Attachments: ATLAS-1948-Residual-import.patch, db1tb1.zip, 
> ImportTransformsErrorOnCTASonDiffDB.txt
>
>
> 1.Created 2 databases db1 , db2 in cluster1
> 2.Created 2 tables
>   1. db1.t1
>   2. db2.t2 as select * from db1.t1
> 3.Exported db1.t1 into zip file.
> 4.Imported zip file into cluster 2 with transforms option :
> {code}
> {
>   "options": {
>"transforms": "{ \"hive_column\": { \"qualifiedName\": [ 
> \"replace:cl1:cl2\" ]} }"
>   }
> }
> {code}
> 5. Import fails with 
> {code}
> {"errorCode":"ATLAS-500-00-001","errorMessage":"org.apache.atlas.exception.AtlasBaseException:
>  ObjectId is not valid 
> AtlasObjectId{guid='51c77c1e-265e-46ab-bbb5-5316cf80a53c', 
> typeName='hive_column', uniqueAttributes={}}"}
> {code}
> Only db1.t1 is imported into Atlas without any lineage. 
> Attached the exception stack trace.
> After this exporting db2.t2 and importing completes successfully.
> That is , first import ,either db1.t1 or db2.t1 is unsuccessful with 
> exception. Next import is successful.
> The exception *doesn't* happen and tables are successfully imported If both 
> the tables are in a single database. Export order if tables are in same db is 
> 1.table1, 
> 2.db,
> 3.table2, 
> 4.hive_process
> 5. hive_column_lineage
> If the tables are in different db , the order is ,
> 1.table1, 
> 2.db1,
> 3.hive_process,
> 4.hive_column_lineage
> 5.ctas table
> 6.db2  
> which is possibly causing the issue. 
> When cluster2  starts importing , it imports table1 , db1 and when it comes 
> to hive_column_lineage , it finds that column specified in 
> hive_column_lineage is not in cluster2 yet ,since ctas table comes after the 
> hive_column_lineage in import order and it throws "ObjectId is not valid 
> AtlasObjectId{guid='51c77c1e-265e-46ab-bbb5-5316cf80a53c', 
> typeName='hive_column' ".
> Thanks [~ayubkhan] for the analysis.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (ATLAS-1963) Relationship instance creation does not result in the expected entity instance content

[Sarath Subramanian (JIRA)]

[ 
https://issues.apache.org/jira/browse/ATLAS-1963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16092043#comment-16092043
 ] 

Sarath Subramanian commented on ATLAS-1963:
---

[~davidrad], this issue is addressed in ATLAS-1959, this is in review 
currently. 

> Relationship instance creation does not result in the expected entity 
> instance content 
> ---
>
> Key: ATLAS-1963
> URL: https://issues.apache.org/jira/browse/ATLAS-1963
> Project: Atlas
>  Issue Type: Bug
>Reporter: David Radley
> Attachments: 0005LinkedMediaTypes.json, 0210Glossary.json, 
> 0220CategoryHierarchy.json
>
>
> I create types using the attached  json files. I then create 2 
> GlossaryCategory instances. 
> I then create a relationship from one category to another with 
> CategoryHierarchyLink.
> {
> "end1": {
> "guid" : "12bcfbd3-1574-452f-8139-4d14147fa56b",
> "typeName" : "GlossaryCategory"
> },
> "end2": {
> "guid" : "1b7b5834-abc6-4ea4-bb6e-b10ae066724e",
> "typeName" : "GlossaryCategory"
> },
> "typeName"  : "CategoryHierarchyLink"
> } 
> I get one of the categories and do not see the populated parent or children 
> category relationship in the relationship attributes.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (ATLAS-1919) Export of hive_table with fetchType "connected" fails with duplicate entry. Seems like, same edge(asset) is traversed twice.

[Ashutosh Mestry (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-1919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Mestry reassigned ATLAS-1919:
--

Assignee: Ashutosh Mestry

> Export of hive_table with fetchType "connected" fails with duplicate entry. 
> Seems like, same edge(asset) is traversed twice.
> 
>
> Key: ATLAS-1919
> URL: https://issues.apache.org/jira/browse/ATLAS-1919
> Project: Atlas
>  Issue Type: Bug
>  Components:  atlas-core
>Affects Versions: 0.9-incubating
>Reporter: Ayub Pathan
>Assignee: Ashutosh Mestry
>Priority: Critical
> Fix For: 0.9-incubating
>
> Attachments: ATLAS-1919.patch
>
>
> Export of hive_table with fetchType "connected" fails with duplicate entry. 
> Seems like, same edge(asset) is traversed twice. Possibly a loop condition.
> Export request payload
> {noformat}
> {
> "itemsToExport": [
> {
> "typeName": "hive_table",
> "uniqueAttributes": {
> "qualifiedName": "any_random_table@cluster"
> }
> }
> ],
> "options": {
>   "fetchType" : "connected"
>   }
> }
> {noformat}
> stacktrace
> {noformat}
> 2017-07-04 08:44:19,159 INFO  - [pool-2-thread-9 - 
> 8bc8d958-e650-4351-acf0-24c28d15627a:] ~ 
> export(item=AtlasObjectId{guid='null', typeName='hive_table', 
> uniqueAttributes={qualifiedName:db1.table1@cl1}}; matchType=null, 
> fetchType=CONNECTED): found 1 entities (ExportService:280)
> 2017-07-04 08:44:19,794 ERROR - [pool-2-thread-9 - 
> 8bc8d958-e650-4351-acf0-24c28d15627a:] ~ Fetching entity failed for: 
> AtlasObjectId{guid='null', typeName='hive_table', 
> uniqueAttributes={qualifiedName:db1.table1@cl1}} (ExportService:207)
> org.apache.atlas.exception.AtlasBaseException: Error writing file 
> 75a90231-3eca-4ba7-9e21-93e7ec2b0df2.
>   at org.apache.atlas.repository.impexp.ZipSink.saveToZip(ZipSink.java:91)
>   at org.apache.atlas.repository.impexp.ZipSink.add(ZipSink.java:50)
>   at 
> org.apache.atlas.repository.impexp.ExportService.addEntity(ExportService.java:433)
>   at 
> org.apache.atlas.repository.impexp.ExportService.processEntity(ExportService.java:297)
>   at 
> org.apache.atlas.repository.impexp.ExportService.processObjectId(ExportService.java:198)
>   at 
> org.apache.atlas.repository.impexp.ExportService.processItems(ExportService.java:162)
>   at 
> org.apache.atlas.repository.impexp.ExportService.run(ExportService.java:95)
>   at 
> org.apache.atlas.web.resources.AdminResource.export(AdminResource.java:325)
>   at sun.reflect.GeneratedMethodAccessor238.invoke(Unknown Source)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
>   at 
> com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205)
>   at 
> com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
>   at 
> com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)
>   at 
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
>   at 
> com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
>   at 
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
>   at 
> com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
>   at 
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542)
>   at 
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473)
>   at 
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419)
>   at 
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409)
>   at 
> com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409)
>   at 
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558)
>   at 
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733)
>   at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
>   at 
> com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:286)
>   at 
> com.google.inject.serv

Re: Review Request 60940: ATLAS-1961: Basic search improvement in use of Solr index for attribute filtering

[Madhan Neethiraj]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60940/
---

(Updated July 19, 2017, 12:02 a.m.)


Review request for atlas.


Changes
---

Updates to improve performance by:
- adding type and classification pre-filter in fulltext search
- having entity processor handle classification filter as well, when no 
classificaiton attribute filtering is specified


Bugs: ATLAS-1961
https://issues.apache.org/jira/browse/ATLAS-1961


Repository: atlas


Description
---

Basic search improvement in use of Solr index for attribute filtering


Diffs (updated)
-

  
repository/src/main/java/org/apache/atlas/discovery/EntitySearchProcessor.java 
3204ecf2 
  
repository/src/main/java/org/apache/atlas/discovery/FullTextSearchProcessor.java
 4ddd6425 
  repository/src/main/java/org/apache/atlas/discovery/SearchContext.java 
8dd7667e 
  repository/src/main/java/org/apache/atlas/discovery/SearchProcessor.java 
ff0bd2ee 


Diff: https://reviews.apache.org/r/60940/diff/2/

Changes: https://reviews.apache.org/r/60940/diff/1-2/


Testing
---

Validated that the search completes much quicker with this update.


Thanks,

Madhan Neethiraj

Re: Review Request 60938: [ATLAS-1959]: Enhance relationship attributes to support different cardinality mappings

[Madhan Neethiraj]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60938/#review180899
---




repository/src/main/java/org/apache/atlas/repository/store/graph/v1/EntityGraphRetriever.java
Line 466 (original), 468 (patched)


for relationshipAttributes, it is not neecessary to retrieve complete 
entity (and store it in entityExtInfo.referredEntities) - it could be expensive 
to retrieve and transmit when large number of entities are referenced (for 
example hive_db.tables referring to 1000s of tables). Storing AtlasObjectId  
(or List) is enough.

Same comment for line #474 below.


- Madhan Neethiraj


On July 18, 2017, 6:25 a.m., Sarath Subramanian wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60938/
> ---
> 
> (Updated July 18, 2017, 6:25 a.m.)
> 
> 
> Review request for atlas and Madhan Neethiraj.
> 
> 
> Bugs: ATLAS-1959
> https://issues.apache.org/jira/browse/ATLAS-1959
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> * Improve relationship model to support create/update operations and the 
> following cardinalities (previously supported using inverseReference):
> 1 to 1
> 1 to many
> many to 1
> many to many
> 
> 
> * Change legacyLabel flag in AtlasRelationshipEndDef to boolean flag.
> * Add unit tests for the above cases.
> 
> 
> Diffs
> -
> 
>   addons/models/0010-base_model.json 303f3796 
>   addons/models/0030-hive_model.json a795f0f3 
>   addons/models/0050-falcon_model.json 7755fa86 
>   addons/models/0060-hbase_model.json 1d264df4 
>   addons/models/0080-storm_model.json 25360ff0 
>   intg/src/main/java/org/apache/atlas/model/instance/AtlasEntity.java 
> 68da6af1 
>   intg/src/main/java/org/apache/atlas/model/instance/AtlasRelationship.java 
> 2de9bdf0 
>   intg/src/main/java/org/apache/atlas/model/typedef/AtlasRelationshipDef.java 
> fc820d49 
>   
> intg/src/main/java/org/apache/atlas/model/typedef/AtlasRelationshipEndDef.java
>  f80ea895 
>   intg/src/main/java/org/apache/atlas/type/AtlasEntityType.java e94dd190 
>   intg/src/main/java/org/apache/atlas/type/AtlasRelationshipType.java 
> 841b66f7 
>   intg/src/main/java/org/apache/atlas/type/AtlasStructType.java f97d7674 
>   intg/src/test/java/org/apache/atlas/TestRelationshipUtilsV2.java 
> PRE-CREATION 
>   intg/src/test/java/org/apache/atlas/TestUtilsV2.java 9774583d 
>   repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java 
> 6f6d74bc 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityGraphDiscoveryV1.java
>  12e8bb1f 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasGraphUtilsV1.java
>  cd9a47ad 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasRelationshipStoreV1.java
>  3ff6fbef 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v1/DeleteHandlerV1.java
>  d4fdc257 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v1/EntityGraphMapper.java
>  157f8cd2 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v1/EntityGraphRetriever.java
>  f4257be7 
>   
> repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java
>  de8e7ef3 
>   
> repository/src/test/java/org/apache/atlas/repository/impexp/ZipFileResourceTestUtils.java
>  d9017319 
>   
> repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasRelationshipStoreHardDeleteV1Test.java
>  PRE-CREATION 
>   
> repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasRelationshipStoreSoftDeleteV1Test.java
>  PRE-CREATION 
>   
> repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasRelationshipStoreV1Test.java
>  67702231 
> 
> 
> Diff: https://reviews.apache.org/r/60938/diff/2/
> 
> 
> Testing
> ---
> 
> added unit test - AtlasRelationshipStoreV1Test
> 
> mvn clean package - succeeded with no errors.
> 
> 
> Thanks,
> 
> Sarath Subramanian
> 
>

Review Request 60954: ATLAS-1948: Addressed Case Where Export Order List Did Not Add Process Guids At the End In All Cases

[Ashutosh Mestry]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60954/
---

Review request for atlas and Madhan Neethiraj.


Bugs: ATLAS-1948
https://issues.apache.org/jira/browse/ATLAS-1948


Repository: atlas


Description
---

**Background & Analysis**
Please see [ATLAS-1948](https://issues.apache.org/jira/browse/ATLAS-1948) for 
scenario details.

_Root Cause_
A case where lineage links 2 separate databases, the order of entity creation 
had process entities inserted between the 2 database entities. This caused 
process entities not to be created correctly. 

_Side Effect_
An export with the above problem would result in process entities not getting 
created, thereby causing entire Import operation to be aborted with "entitiy 
id" not found error.

**Implementation**

_Export API_
The export operation was addressed to maintain the process entitities to be 
created after all the other entities are created. 

This is a low-impact fix.

_Import API_
The import operation now maintains a separated list of failed entities. These 
entities are attempted to be imporated once all the non-process entities are 
imported.

This was achieved by implementing a _EntityImportStreamWithResidualList_. This 
private class accepts the _EntityImportStream_ and _List_ as 2 parameters. The 
_List_ has guids of entities that failed during regular import process. Once 
all the entities from the _EntitiyImportStream_ are exhausted, this list gets 
processed.


Diffs
-

  
repository/src/main/java/org/apache/atlas/repository/impexp/ExportService.java 
8f45e9fd 
  repository/src/main/java/org/apache/atlas/repository/impexp/ZipSource.java 
edb816fc 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStoreV1.java
 5ea4ff20 
  
repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java
 de8e7ef3 
  repository/src/test/resources/ctas.zip PRE-CREATION 


Diff: https://reviews.apache.org/r/60954/diff/1/


Testing
---

**Unit tests**

Additional unit tests for that uses this test case.

**Functional test**

Regular tests cases.


Thanks,

Ashutosh Mestry

Re: Review Request 60940: ATLAS-1961: Basic search improvement in use of Solr index for attribute filtering

[Apoorv Naik]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60940/#review180902
---


Ship it!




Ship It!

- Apoorv Naik


On July 19, 2017, 12:02 a.m., Madhan Neethiraj wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60940/
> ---
> 
> (Updated July 19, 2017, 12:02 a.m.)
> 
> 
> Review request for atlas.
> 
> 
> Bugs: ATLAS-1961
> https://issues.apache.org/jira/browse/ATLAS-1961
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Basic search improvement in use of Solr index for attribute filtering
> 
> 
> Diffs
> -
> 
>   
> repository/src/main/java/org/apache/atlas/discovery/EntitySearchProcessor.java
>  3204ecf2 
>   
> repository/src/main/java/org/apache/atlas/discovery/FullTextSearchProcessor.java
>  4ddd6425 
>   repository/src/main/java/org/apache/atlas/discovery/SearchContext.java 
> 8dd7667e 
>   repository/src/main/java/org/apache/atlas/discovery/SearchProcessor.java 
> ff0bd2ee 
> 
> 
> Diff: https://reviews.apache.org/r/60940/diff/2/
> 
> 
> Testing
> ---
> 
> Validated that the search completes much quicker with this update.
> 
> 
> Thanks,
> 
> Madhan Neethiraj
> 
>