[jira] [Reopened] (ATLAS-712) Support getTrait() API

[Keval Bhatt (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Keval Bhatt reopened ATLAS-712:
---

I have reverted the commit 
(http://git-wip-us.apache.org/repos/asf/incubator-atlas/commit/b6e0d60f) since 
there are some review comments open on review board 

> Support getTrait() API
> --
>
> Key: ATLAS-712
> URL: https://issues.apache.org/jira/browse/ATLAS-712
> Project: Atlas
>  Issue Type: Improvement
>Reporter: Suma Shivaprasad
>Assignee: Vimal Sharma
> Attachments: ATLAS-712-v2.patch, ATLAS-712-v3.patch, ATLAS-712.patch
>
>
> Given entity id and trait name, support rest API that returns the trait 
> instance for the entity. Currently, the other way of getting it is 
> getEntity() which returns full entity with all trait instances 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 51660: [ATLAS-1098] Added keyword checks while creating a new tag/trait

[Apoorv Naik]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51660/#review149172
---




repository/src/test/java/org/apache/atlas/service/DefaultMetadataServiceTest.java
 (line 202)


Need to delete this unwanted code. Doesn't correspond to the actual test

Please ignore.


- Apoorv Naik


On Sept. 16, 2016, 5:40 a.m., Apoorv Naik wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51660/
> ---
> 
> (Updated Sept. 16, 2016, 5:40 a.m.)
> 
> 
> Review request for atlas, Madhan Neethiraj, Shwetha GS, Suma Shivaprasad, and 
> Vimal Sharma.
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> [ATLAS-1098] Added keyword checks while creating a new tag/trait
> 
> 
> Diffs
> -
> 
>   
> repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java
>  6a937f4 
>   repository/src/main/scala/org/apache/atlas/query/QueryParser.scala 4d2429e 
>   repository/src/test/java/org/apache/atlas/BaseRepositoryTest.java 01c4bfa 
>   
> repository/src/test/java/org/apache/atlas/discovery/GraphBackedDiscoveryServiceTest.java
>  8a40110 
>   
> repository/src/test/java/org/apache/atlas/service/DefaultMetadataServiceTest.java
>  52dcfde 
> 
> Diff: https://reviews.apache.org/r/51660/diff/
> 
> 
> Testing
> ---
> 
> Tested with all possible keywords specified in the QueryParser, the UI shows 
> a red exception message stating that the type being created is a keyword
> 
> 
> Thanks,
> 
> Apoorv Naik
> 
>

Re: Review Request 51660: [ATLAS-1098] Added keyword checks while creating a new tag/trait

[Apoorv Naik]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51660/
---

(Updated Sept. 16, 2016, 5:40 a.m.)


Review request for atlas, Madhan Neethiraj, Shwetha GS, Suma Shivaprasad, and 
Vimal Sharma.


Changes
---

Incorporating Shwetha's patch and adding some Unit Tests for the same


Repository: atlas


Description
---

[ATLAS-1098] Added keyword checks while creating a new tag/trait


Diffs (updated)
-

  
repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java 
6a937f4 
  repository/src/main/scala/org/apache/atlas/query/QueryParser.scala 4d2429e 
  repository/src/test/java/org/apache/atlas/BaseRepositoryTest.java 01c4bfa 
  
repository/src/test/java/org/apache/atlas/discovery/GraphBackedDiscoveryServiceTest.java
 8a40110 
  
repository/src/test/java/org/apache/atlas/service/DefaultMetadataServiceTest.java
 52dcfde 

Diff: https://reviews.apache.org/r/51660/diff/


Testing
---

Tested with all possible keywords specified in the QueryParser, the UI shows a 
red exception message stating that the type being created is a keyword


Thanks,

Apoorv Naik

[jira] [Commented] (ATLAS-1098) Atlas allows creation of tag with name "isa" which causes exceptions during search

[Apoorv Naik (JIRA)]

[ 
https://issues.apache.org/jira/browse/ATLAS-1098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15495442#comment-15495442
 ] 

Apoorv Naik commented on ATLAS-1098:


Attaching the new patch with unit tests. They seem to be failing at the moment. 
Need a second pair of eyes to go through it.

> Atlas allows creation of tag with name "isa" which causes exceptions during 
> search
> --
>
> Key: ATLAS-1098
> URL: https://issues.apache.org/jira/browse/ATLAS-1098
> Project: Atlas
>  Issue Type: Bug
>Reporter: Sharmadha Sainath
>Assignee: Apoorv Naik
> Attachments: ATLAS-1098.patch, ATLAS-1098.patch
>
>
> Created a tag with name "isa". DSL query "Table where Table isa isa" throws 
> exception . Search in Tag tab also throws exception.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (ATLAS-1098) Atlas allows creation of tag with name "isa" which causes exceptions during search

[Apoorv Naik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-1098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Apoorv Naik updated ATLAS-1098:
---
Attachment: ATLAS-1098.patch

> Atlas allows creation of tag with name "isa" which causes exceptions during 
> search
> --
>
> Key: ATLAS-1098
> URL: https://issues.apache.org/jira/browse/ATLAS-1098
> Project: Atlas
>  Issue Type: Bug
>Reporter: Sharmadha Sainath
>Assignee: Apoorv Naik
> Attachments: ATLAS-1098.patch, ATLAS-1098.patch
>
>
> Created a tag with name "isa". DSL query "Table where Table isa isa" throws 
> exception . Search in Tag tab also throws exception.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 51939: Framework to apply updates to types in the type-system

[Apoorv Naik]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51939/#review149166
---




repository/src/main/java/org/apache/atlas/services/AtlasTypeUpdate.java (line 
79)


Can the list be null here ?



repository/src/main/java/org/apache/atlas/services/AtlasTypeUpdate.java (line 
87)


If found, the method will return false and vice-versa



repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java 
(line 85)


Can we preserve the specific imports instead of wildcards ?


- Apoorv Naik


On Sept. 15, 2016, 11:03 p.m., Sarath Kumar Subramanian wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51939/
> ---
> 
> (Updated Sept. 15, 2016, 11:03 p.m.)
> 
> 
> Review request for atlas, Madhan Neethiraj, Shwetha GS, and Suma Shivaprasad.
> 
> 
> Bugs: ATLAS-1174
> https://issues.apache.org/jira/browse/ATLAS-1174
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> 1. Introduce "version" attribute to all types in the type-system, this helps 
> to track changes made to the default types (hive, sqoop, falcon and storm 
> types) and user created types. If version is not mentioned during creation of 
> a type, default version "1.0" is assigned (optional attribute).
> 2. Using the version attributed for types, introduce a patch framework for 
> type system. This framework applies patches to a type using its version 
> number and can be used during upgrade - add new attributes to an existing 
> types and it will be run during atlas startup.
> The sequence of steps:
> a. During atlas startup, check $ATLAS_HOME/models/patches directory for any 
> available patch files (json files). If there any patch files handle them.
> b. Sample patch json file looks like:
> {
> "patches": [
> { 
> "action": "ADD_ATTRIBUTE",
> "typeName": "hive_column",
> "applyToVersion": "1.0",
> "updateToVersion": "2.0",
> "actionParams": [
> { "name": "position", "dataTypeName": "int", "multiplicity": "optional", 
> "isComposite": false, "isUnique": false, "isIndexable": false, 
> "reverseAttributeName": null }
> ]
> } ]
> }
> c. The framework updates the type in "typeName" for the matching version 
> number and after applying the patch, update the version to the one mentioned 
> in "updateToVersion"
> d. The json file can have more than one action (array of actions).
> e. There can be multiple patch json files in the directory and are applied in 
> the sort order of the filename. eg:
> 001-hive_column_add_position.json
> 002-hive_column_add_anotherattribute.json
> 
> 
> Diffs
> -
> 
>   common/src/main/java/org/apache/atlas/repository/Constants.java d7f9c89 
>   
> repository/src/main/java/org/apache/atlas/repository/typestore/GraphBackedTypeStore.java
>  a94d157 
>   repository/src/main/java/org/apache/atlas/services/AtlasTypeUpdate.java 
> PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java
>  6a937f4 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/AbstractDataType.java
>  fad091d 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/ClassType.java 
> c56987a 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/EnumType.java 
> bdd0a13 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/EnumTypeDefinition.java
>  6340615 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/HierarchicalType.java
>  7224699 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/HierarchicalTypeDefinition.java
>  9a299f0 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/IDataType.java 
> 85ddee7 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/StructType.java 
> 6f40c1d 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/StructTypeDefinition.java
>  f1ce1b7 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/TraitType.java 
> f23bf5b 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/TypeSystem.java 
> 70ba89b 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/utils/TypesUtil.java
>  ef8448d 
>   
> typesystem/src/main/scala/org/apache/atlas/typesystem/json/TypesSerialization.scala
>  5618938 
> 
> Diff: https://reviews.apache.org/r/51939/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sarath Kumar Subramanian
> 
>

Re: Review Request 51939: Framework to apply updates to types in the type-system

[Madhan Neethiraj]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51939/#review149157
---



Comments from partial review. I will complete the review later today.


repository/src/main/java/org/apache/atlas/services/AtlasTypeUpdate.java (line 
55)


Consider moving 'private' methods to end of the class - after public and 
protected methods. It makes it easier to read/review.



repository/src/main/java/org/apache/atlas/services/AtlasTypeUpdate.java (line 
56)


This method returns "false" when the given type has any of the attributes 
in the given array. checkIfAttributeExists() does not seem a good name.



repository/src/main/java/org/apache/atlas/services/AtlasTypeUpdate.java (line 
100)


handle patchArray == null. Also review other places that deal with contents 
of patch file.



repository/src/main/java/org/apache/atlas/services/AtlasTypeUpdate.java (line 
109)


"Invalid operation. Possible operation - ADD_ATTRIBUTE" ==>
  action + ": invalid action. Supported actions: ADD_ATTRIBUTE"



repository/src/main/java/org/apache/atlas/services/AtlasTypeUpdate.java (line 
114)


It will be helpful to add a comment showing a sample patch processed by 
this method.



repository/src/main/java/org/apache/atlas/services/AtlasTypeUpdate.java (line 
115)


metadataService is an attribute of this instance. Should this be passed as 
parameter?



repository/src/main/java/org/apache/atlas/services/AtlasTypeUpdate.java (line 
116)


It might be useful to isolate patch details to a separate little class. 
Consider the following class hierarchy:

class PatchData {
  String  typeName;
  String  applyToVersion;
  String  updateToVersion;
  Map params;
}

abstract class AtlasTypePatch {
  public enum PatchResult { SUCCESS, FAILED, SKIPPED };
 
  protected DefaultMetadataService metadataService;
  protected TypeSystem typeSystem;

  public void init(DefaultMetadataService metadataService, TypeSystem 
typeSystem) {
this.metadataService = metadataService;
this.typeSystem  = typeSystem;
  }
  
  public abstract PatchResult applyPatch(PatchData patch);
}

class AddAttributePatch extends AtlasTypePatch {
  public PatchResult applyPatch(PatchData patch) {
// ..
  }
}



repository/src/main/java/org/apache/atlas/services/AtlasTypeUpdate.java (line 
131)


currentTypeVersion.equalsIgnoreCase(applyVersion) || 
currentTypeVersion.startsWith(applyVersion)) ==> 
currentTypeVersion.equalsIgnoreCase(applyVersion) || 
currentTypeVersion.startsWith(applyVersion + "."))

..to avoid applying policies for these values: currentVersion="1.11" with 
appyVersion=""1.1"



repository/src/main/java/org/apache/atlas/services/AtlasTypeUpdate.java (line 
136)


LOG.error("failed to apply patch (typeName=" + typeName + "; 
applyToVersion=" + applyToVersion + "; updateToVersion=" + updateToVersion + 
"): type does not exist", e);



repository/src/main/java/org/apache/atlas/services/AtlasTypeUpdate.java (line 
141)


Replace with:
  LOG.error("failed to apply patch (typeName=" + typeName + "; 
applyToVersion=" + applyToVersion + "; updateToVersion=" + updateToVersion + 
")", e);
  
Review other error messages as well.



repository/src/main/java/org/apache/atlas/services/AtlasTypeUpdate.java (line 
148)


handle error and return null if the given type does not exist.



repository/src/main/java/org/apache/atlas/services/AtlasTypeUpdate.java (line 
157)


Is using JSON string the only way to update types in the system? There is 
got to be a better way than use of opaque strings..


- Madhan Neethiraj


On Sept. 15, 2016, 11:03 p.m., Sarath Kumar Subramanian wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51939/
> ---
> 
> (Updated Sept. 15, 2016, 11:03 p.m.)
> 
> 
> Review request for atlas, Madhan Neethiraj, Shwetha GS, and Suma Shivaprasad.
> 
> 
> Bugs: ATLAS-1174
> 

[jira] [Commented] (ATLAS-1145) Storm hook with kafka topic and hive bolt picks wrong hive table name

[Sarath Subramanian (JIRA)]

[ 
https://issues.apache.org/jira/browse/ATLAS-1145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15494879#comment-15494879
 ] 

Sarath Subramanian commented on ATLAS-1145:
---

Could you specify more info and reproduction steps?

> Storm hook with kafka topic and hive bolt picks wrong hive table name 
> --
>
> Key: ATLAS-1145
> URL: https://issues.apache.org/jira/browse/ATLAS-1145
> Project: Atlas
>  Issue Type: Bug
>Reporter: Shwetha G S
>Assignee: Sarath Subramanian
>
> Storm hook with kafka topic and hive bolt picks kafka topic name as hive 
> table name



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Review Request 51939: Framework to apply updates to types in the type-system

[Sarath Kumar Subramanian]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51939/
---

Review request for atlas, Madhan Neethiraj, Shwetha GS, and Suma Shivaprasad.


Bugs: ATLAS-1174
https://issues.apache.org/jira/browse/ATLAS-1174


Repository: atlas


Description
---

1. Introduce "version" attribute to all types in the type-system, this helps to 
track changes made to the default types (hive, sqoop, falcon and storm types) 
and user created types. If version is not mentioned during creation of a type, 
default version "1.0" is assigned (optional attribute).
2. Using the version attributed for types, introduce a patch framework for type 
system. This framework applies patches to a type using its version number and 
can be used during upgrade - add new attributes to an existing types and it 
will be run during atlas startup.
The sequence of steps:
a. During atlas startup, check $ATLAS_HOME/models/patches directory for any 
available patch files (json files). If there any patch files handle them.
b. Sample patch json file looks like:
{
"patches": [
{ 
"action": "ADD_ATTRIBUTE",
"typeName": "hive_column",
"applyToVersion": "1.0",
"updateToVersion": "2.0",
"actionParams": [
{ "name": "position", "dataTypeName": "int", "multiplicity": "optional", 
"isComposite": false, "isUnique": false, "isIndexable": false, 
"reverseAttributeName": null }
]
} ]
}
c. The framework updates the type in "typeName" for the matching version number 
and after applying the patch, update the version to the one mentioned in 
"updateToVersion"
d. The json file can have more than one action (array of actions).
e. There can be multiple patch json files in the directory and are applied in 
the sort order of the filename. eg:
001-hive_column_add_position.json
002-hive_column_add_anotherattribute.json


Diffs
-

  common/src/main/java/org/apache/atlas/repository/Constants.java d7f9c89 
  
repository/src/main/java/org/apache/atlas/repository/typestore/GraphBackedTypeStore.java
 a94d157 
  repository/src/main/java/org/apache/atlas/services/AtlasTypeUpdate.java 
PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java 
6a937f4 
  
typesystem/src/main/java/org/apache/atlas/typesystem/types/AbstractDataType.java
 fad091d 
  typesystem/src/main/java/org/apache/atlas/typesystem/types/ClassType.java 
c56987a 
  typesystem/src/main/java/org/apache/atlas/typesystem/types/EnumType.java 
bdd0a13 
  
typesystem/src/main/java/org/apache/atlas/typesystem/types/EnumTypeDefinition.java
 6340615 
  
typesystem/src/main/java/org/apache/atlas/typesystem/types/HierarchicalType.java
 7224699 
  
typesystem/src/main/java/org/apache/atlas/typesystem/types/HierarchicalTypeDefinition.java
 9a299f0 
  typesystem/src/main/java/org/apache/atlas/typesystem/types/IDataType.java 
85ddee7 
  typesystem/src/main/java/org/apache/atlas/typesystem/types/StructType.java 
6f40c1d 
  
typesystem/src/main/java/org/apache/atlas/typesystem/types/StructTypeDefinition.java
 f1ce1b7 
  typesystem/src/main/java/org/apache/atlas/typesystem/types/TraitType.java 
f23bf5b 
  typesystem/src/main/java/org/apache/atlas/typesystem/types/TypeSystem.java 
70ba89b 
  
typesystem/src/main/java/org/apache/atlas/typesystem/types/utils/TypesUtil.java 
ef8448d 
  
typesystem/src/main/scala/org/apache/atlas/typesystem/json/TypesSerialization.scala
 5618938 

Diff: https://reviews.apache.org/r/51939/diff/


Testing
---


Thanks,

Sarath Kumar Subramanian

Review Request 51936: Framework to apply updates to types in the type-system

[Sarath Kumar Subramanian]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51936/
---

Review request for atlas and Madhan Neethiraj.


Bugs: ATLAS-1174
https://issues.apache.org/jira/browse/ATLAS-1174


Repository: atlas


Description
---

1. Introduce "version" attribute to all types in the type-system, this helps to 
track changes made to the default types (hive, sqoop, falcon and storm types) 
and user created types. If version is not mentioned during creation of a type, 
default version "1.0" is assigned (optional attribute).
2. Using the version attributed for types, introduce a patch framework for type 
system. This framework applies patches to a type using its version number and 
can be used during upgrade - add new attributes to an existing types and it 
will be run during atlas startup.
The sequence of steps:
a. During atlas startup, check $ATLAS_HOME/models/patches directory for any 
available patch files (json files). If there any patch files handle them.
b. Sample patch json file looks like:
{
"patches": [
{ 
"action": "ADD_ATTRIBUTE",
"typeName": "hive_column",
"applyToVersion": "1.0",
"updateToVersion": "2.0",
"actionParams": [
{ "name": "position", "dataTypeName": "int", "multiplicity": "optional", 
"isComposite": false, "isUnique": false, "isIndexable": false, 
"reverseAttributeName": null }
]
} ]
}
c. The framework updates the type in "typeName" for the matching version number 
and after applying the patch, update the version to the one mentioned in 
"updateToVersion"
d. The json file can have more than one action (array of actions).
e. There can be multiple patch json files in the directory and are applied in 
the sort order of the filename. eg:
001-hive_column_add_position.json
002-hive_column_add_anotherattribute.json


Diffs
-

  repository/src/main/java/org/apache/atlas/services/AtlasTypeUpdate.java 
PRE-CREATION 

Diff: https://reviews.apache.org/r/51936/diff/


Testing
---


Thanks,

Sarath Kumar Subramanian

[jira] [Updated] (ATLAS-1174) Framework to apply updates to types in the type-system

[Sarath Subramanian (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-1174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sarath Subramanian updated ATLAS-1174:
--
Attachment: ATLAS-1174.patch

> Framework to apply updates to types in the type-system
> --
>
> Key: ATLAS-1174
> URL: https://issues.apache.org/jira/browse/ATLAS-1174
> Project: Atlas
>  Issue Type: New Feature
>Reporter: Sarath Subramanian
>Assignee: Sarath Subramanian
>  Labels: feature, patch
> Attachments: ATLAS-1174.patch
>
>
> 1. Introduce "version" attribute to all types in the type-system, this helps 
> to track changes made to the default types (hive, sqoop, falcon and storm 
> types) and user created types. If version is not mentioned during creation of 
> a type, default version "1.0" is assigned (optional attribute).
> 2. Using the version attributed for types, introduce a patch framework for 
> type system. This framework applies patches to a type using its version 
> number and can be used during upgrade - add new attributes to an existing 
> types and it will be run during atlas startup. 
> The sequence of steps:
> a. During atlas startup, check $ATLAS_HOME/models/patches directory for 
> any available patch files (json files). If there any patch files handle them.
> b. Sample patch json file looks like:
> {
>   "patches": [
> { 
>   "action": "ADD_ATTRIBUTE",
>   "typeName": "hive_column",
>   "applyToVersion": "1.0",
>   "updateToVersion": "2.0",
>   "actionParams": [
> { "name": "position",
>   "dataTypeName": "int",
>   "multiplicity": "optional",
>   "isComposite": false,
>   "isUnique": false,
>   "isIndexable": false,
>   "reverseAttributeName": null
> } ]
> } ]
> }
> c. The framework updates the type in "typeName" for the matching version 
> number and after applying the patch, update the version to the one mentioned 
> in "updateToVersion"
> d. The json file can have more than one action (array of actions).
> e. There can be multiple patch json files in the directory and are 
> applied in the sort order of the filename. eg:
> 001-hive_column_add_position.json
> 002-hive_column_add_anotherattribute.json



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (ATLAS-1174) Framework to apply updates to types in the type-system

[Sarath Subramanian (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-1174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sarath Subramanian updated ATLAS-1174:
--
Description: 
1. Introduce "version" attribute to all types in the type-system, this helps to 
track changes made to the default types (hive, sqoop, falcon and storm types) 
and user created types. If version is not mentioned during creation of a type, 
default version "1.0" is assigned (optional attribute).

2. Using the version attributed for types, introduce a patch framework for type 
system. This framework applies patches to a type using its version number and 
can be used during upgrade - add new attributes to an existing types and it 
will be run during atlas startup. 

The sequence of steps:

a. During atlas startup, check $ATLAS_HOME/models/patches directory for any 
available patch files (json files). If there any patch files handle them.
b. Sample patch json file looks like:
{
  "patches": [
{ 
  "action": "ADD_ATTRIBUTE",
  "typeName": "hive_column",
  "applyToVersion": "1.0",
  "updateToVersion": "2.0",
  "actionParams": [
{ "name": "position",
  "dataTypeName": "int",
  "multiplicity": "optional",
  "isComposite": false,
  "isUnique": false,
  "isIndexable": false,
  "reverseAttributeName": null
} ]
} ]
}

c. The framework updates the type in "typeName" for the matching version 
number and after applying the patch, update the version to the one mentioned in 
"updateToVersion"
d. The json file can have more than one action (array of actions).
e. There can be multiple patch json files in the directory and are applied 
in the sort order of the filename. eg:
001-hive_column_add_position.json
002-hive_column_add_anotherattribute.json

  was:
1. Introduce "typeVersion" attribute to all types in the type system, this 
helps to track the changes made to the default types out of the box (hive, 
sqoop, falcon and storm types) and user created types. If version is not 
mentioned during creation of a type, default version "1.0" is assigned 
(optional attribute).

2. Using the typeVersion attributed for types, introduce a patch framework for 
type system. This framework can be used during upgrade scenarios - like add new 
attributes to an existing types and will be run during startup of atlas. 

The sequence of steps:

a. During atlas startup, check $ATLAS_HOME/models/patches directory for any 
available patch files (json files). If there any patch files handle them.
b. Sample patch json file looks like:
{
  "patches": [
{ 
  "action": "ADD_ATTRIBUTE",
  "typeName": "hive_column",
  "applyToVersion": "1.0",
  "updateToVersion": "2.0",
  "actionParams": [
{ "name": "position",
  "dataTypeName": "int",
  "multiplicity": "optional",
  "isComposite": false,
  "isUnique": false,
  "isIndexable": false,
  "reverseAttributeName": null
} ]
} ]
}

c. The framework updates the type in "typeName" for the matching version 
number and after applying the patch "ADD_ATTRIBUTE" update the version to the 
one mentioned in "updateToVersion"
d. The json file can have more than one action (array of actions)
e. We can have multiple patch json files in the directory and they are 
applied in the sort order of the filename. eg:
001-hive_column_add_position.json
002-hive_column_add_anotherattribute.json


> Framework to apply updates to types in the type-system
> --
>
> Key: ATLAS-1174
> URL: https://issues.apache.org/jira/browse/ATLAS-1174
> Project: Atlas
>  Issue Type: New Feature
>Reporter: Sarath Subramanian
>Assignee: Sarath Subramanian
>
> 1. Introduce "version" attribute to all types in the type-system, this helps 
> to track changes made to the default types (hive, sqoop, falcon and storm 
> types) and user created types. If version is not mentioned during creation of 
> a type, default version "1.0" is assigned (optional attribute).
> 2. Using the version attributed for types, introduce a patch framework for 
> type system. This framework applies patches to a type using its version 
> number and can be used during upgrade - add new attributes to an existing 
> types and it will be run during atlas startup. 
> The sequence of steps:
> a. During atlas startup, check $ATLAS_HOME/models/patches directory for 
> any available patch files (json files). If there any patch files handle them.
> b. Sample patch json file looks like:
> {
>   "patches": [
> { 
>   

[jira] [Updated] (ATLAS-1174) Framework to apply updates to types in the type-system

[Sarath Subramanian (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-1174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sarath Subramanian updated ATLAS-1174:
--
Description: 
1. Introduce "typeVersion" attribute to all types in the type system, this 
helps to track the changes made to the default types out of the box (hive, 
sqoop, falcon and storm types) and user created types. If version is not 
mentioned during creation of a type, default version "1.0" is assigned 
(optional attribute).

2. Using the typeVersion attributed for types, introduce a patch framework for 
type system. This framework can be used during upgrade scenarios - like add new 
attributes to an existing types and will be run during startup of atlas. 

The sequence of steps:

a. During atlas startup, check $ATLAS_HOME/models/patches directory for any 
available patch files (json files). If there any patch files handle them.
b. Sample patch json file looks like:
{
  "patches": [
{ 
  "action": "ADD_ATTRIBUTE",
  "typeName": "hive_column",
  "applyToVersion": "1.0",
  "updateToVersion": "2.0",
  "actionParams": [
{ "name": "position",
  "dataTypeName": "int",
  "multiplicity": "optional",
  "isComposite": false,
  "isUnique": false,
  "isIndexable": false,
  "reverseAttributeName": null
} ]
} ]
}

c. The framework updates the type in "typeName" for the matching version 
number and after applying the patch "ADD_ATTRIBUTE" update the version to the 
one mentioned in "updateToVersion"
d. The json file can have more than one action (array of actions)
e. We can have multiple patch json files in the directory and they are 
applied in the sort order of the filename. eg:
001-hive_column_add_position.json
002-hive_column_add_anotherattribute.json

> Framework to apply updates to types in the type-system
> --
>
> Key: ATLAS-1174
> URL: https://issues.apache.org/jira/browse/ATLAS-1174
> Project: Atlas
>  Issue Type: New Feature
>Reporter: Sarath Subramanian
>Assignee: Sarath Subramanian
>
> 1. Introduce "typeVersion" attribute to all types in the type system, this 
> helps to track the changes made to the default types out of the box (hive, 
> sqoop, falcon and storm types) and user created types. If version is not 
> mentioned during creation of a type, default version "1.0" is assigned 
> (optional attribute).
> 2. Using the typeVersion attributed for types, introduce a patch framework 
> for type system. This framework can be used during upgrade scenarios - like 
> add new attributes to an existing types and will be run during startup of 
> atlas. 
> The sequence of steps:
> a. During atlas startup, check $ATLAS_HOME/models/patches directory for 
> any available patch files (json files). If there any patch files handle them.
> b. Sample patch json file looks like:
> {
>   "patches": [
> { 
>   "action": "ADD_ATTRIBUTE",
>   "typeName": "hive_column",
>   "applyToVersion": "1.0",
>   "updateToVersion": "2.0",
>   "actionParams": [
> { "name": "position",
>   "dataTypeName": "int",
>   "multiplicity": "optional",
>   "isComposite": false,
>   "isUnique": false,
>   "isIndexable": false,
>   "reverseAttributeName": null
> } ]
> } ]
> }
> c. The framework updates the type in "typeName" for the matching version 
> number and after applying the patch "ADD_ATTRIBUTE" update the version to the 
> one mentioned in "updateToVersion"
> d. The json file can have more than one action (array of actions)
> e. We can have multiple patch json files in the directory and they are 
> applied in the sort order of the filename. eg:
> 001-hive_column_add_position.json
> 002-hive_column_add_anotherattribute.json



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (ATLAS-1174) Framework to apply updates to types in the type-system

[Sarath Subramanian (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-1174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sarath Subramanian updated ATLAS-1174:
--
Summary: Framework to apply updates to types in the type-system  (was: 
Introduce patch application framework and create "typeVersion" attribute to all 
types in typesystem)

> Framework to apply updates to types in the type-system
> --
>
> Key: ATLAS-1174
> URL: https://issues.apache.org/jira/browse/ATLAS-1174
> Project: Atlas
>  Issue Type: New Feature
>Reporter: Sarath Subramanian
>Assignee: Sarath Subramanian
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (ATLAS-1174) Introduce patch application framework and create "typeVersion" attribute to all types in typesystem

[Sarath Subramanian (JIRA)]

Sarath Subramanian created ATLAS-1174:
-

 Summary: Introduce patch application framework and create 
"typeVersion" attribute to all types in typesystem
 Key: ATLAS-1174
 URL: https://issues.apache.org/jira/browse/ATLAS-1174
 Project: Atlas
  Issue Type: New Feature
Reporter: Sarath Subramanian
Assignee: Sarath Subramanian






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (ATLAS-1147) UI: column name doesn't show up in schema tab for hive table

[Andrew Ahn (JIRA)]

[ 
https://issues.apache.org/jira/browse/ATLAS-1147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15494017#comment-15494017
 ] 

Andrew Ahn commented on ATLAS-1147:
---

Sample file of patch attached as: SchemaLayoutView.js

> UI: column name doesn't show up in schema tab for hive table
> 
>
> Key: ATLAS-1147
> URL: https://issues.apache.org/jira/browse/ATLAS-1147
> Project: Atlas
>  Issue Type: Bug
>Reporter: Shwetha G S
>Assignee: Kalyani Kashikar
> Attachments: ATLAS-1147.patch, SchemaLayoutView.js, Screen Shot 
> 2016-08-30 at 9.21.07 PM.png
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (ATLAS-1147) UI: column name doesn't show up in schema tab for hive table

[Andrew Ahn (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-1147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Ahn updated ATLAS-1147:
--
Attachment: SchemaLayoutView.js

-In hosts that run Atlas server: copy attached SchemaLayoutView.js to 
/usr/hdp/current/atlas-server/server/webapp/atlas/js/views/schema/SchemaLayoutView.js
-Refresh the browser, so that its cache will be updated for the updated 
JavaScript file

> UI: column name doesn't show up in schema tab for hive table
> 
>
> Key: ATLAS-1147
> URL: https://issues.apache.org/jira/browse/ATLAS-1147
> Project: Atlas
>  Issue Type: Bug
>Reporter: Shwetha G S
>Assignee: Kalyani Kashikar
> Attachments: ATLAS-1147.patch, SchemaLayoutView.js, Screen Shot 
> 2016-08-30 at 9.21.07 PM.png
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 51896: ATLAS-1171: structured, high-level APIs

[Apoorv Naik]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51896/#review148935
---




common/pom.xml (line 38)


Same as above



pom.xml (line 424)


This is a deprecated version and is only in maintenance mode. Thew newer 
versions can be found here

https://mvnrepository.com/artifact/com.fasterxml.jackson.core


https://mvnrepository.com/artifact/com.fasterxml.jackson.jaxrs/jackson-jaxrs-json-provider


- Apoorv Naik


On Sept. 14, 2016, 5:33 p.m., Madhan Neethiraj wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51896/
> ---
> 
> (Updated Sept. 14, 2016, 5:33 p.m.)
> 
> 
> Review request for atlas.
> 
> 
> Bugs: ATLAS-1171
> https://issues.apache.org/jira/browse/ATLAS-1171
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> first-cut API for review
> 
> 
> Diffs
> -
> 
>   common/pom.xml e3b6465 
>   common/src/main/java/org/apache/atlas/api/AtlasEntities.java PRE-CREATION 
>   common/src/main/java/org/apache/atlas/api/AtlasTypes.java PRE-CREATION 
>   common/src/main/java/org/apache/atlas/api/SearchFilter.java PRE-CREATION 
>   common/src/main/java/org/apache/atlas/model/AtlasAttributeDef.java 
> PRE-CREATION 
>   common/src/main/java/org/apache/atlas/model/AtlasBaseModelObject.java 
> PRE-CREATION 
>   common/src/main/java/org/apache/atlas/model/AtlasBuiltInDatatypes.java 
> PRE-CREATION 
>   common/src/main/java/org/apache/atlas/model/AtlasEntity.java PRE-CREATION 
>   common/src/main/java/org/apache/atlas/model/AtlasEntityDef.java 
> PRE-CREATION 
>   common/src/main/java/org/apache/atlas/model/AtlasEnumDef.java PRE-CREATION 
>   common/src/main/java/org/apache/atlas/model/AtlasObjectId.java PRE-CREATION 
>   common/src/main/java/org/apache/atlas/model/AtlasProcess.java PRE-CREATION 
>   common/src/main/java/org/apache/atlas/model/AtlasStruct.java PRE-CREATION 
>   common/src/main/java/org/apache/atlas/model/AtlasStructDef.java 
> PRE-CREATION 
>   common/src/main/java/org/apache/atlas/model/AtlasTrait.java PRE-CREATION 
>   common/src/main/java/org/apache/atlas/model/AtlasTraitDef.java PRE-CREATION 
>   common/src/test/java/org/apache/atlas/model/TestAtlasEntity.java 
> PRE-CREATION 
>   pom.xml 95f28b0 
> 
> Diff: https://reviews.apache.org/r/51896/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Madhan Neethiraj
> 
>

Build failed in Jenkins: apache-atlas-nightly #426

[Apache Jenkins Server]

See 

Changes:

[kbhatt] ATLAS-712 Support getTrait() API (svimal2106 via kevalbhatt)

--
[...truncated 9084 lines...]
127.0.0.1 - - [15/Sep/2016:15:37:57 +] "GET 
/api/atlas/lineage/hive/table/default.tablezooyprhjcj@primary/inputs/graph 
HTTP/1.1" 200 - "-" "Java/1.7.0_80"
127.0.0.1 - - [15/Sep/2016:15:38:02 +] "GET 
/api/atlas/entities?type=hive_process=qualifiedName=QUERY:default.tablekimpykquum@primary:1473953881000-%3E:PATH_WRITE
 HTTP/1.1" 404 - "-" "Java/1.7.0_80"
127.0.0.1 - - [15/Sep/2016:15:38:07 +] "GET 
/api/atlas/entities?type=hive_process=qualifiedName=QUERY:default.tablekimpykquum@primary:1473953881000-%3E:PATH_WRITE
 HTTP/1.1" 200 - "-" "Java/1.7.0_80"
127.0.0.1 - - [15/Sep/2016:15:38:07 +] "GET 
/api/atlas/entities?type=hive_process=qualifiedName=QUERY:default.tablekimpykquum@primary:1473953881000-%3E:PATH_WRITE
 HTTP/1.1" 200 - "-" "Java/1.7.0_80"
127.0.0.1 - - [15/Sep/2016:15:38:07 +] "GET 
/api/atlas/entities/f6c1ce66-117b-4fe7-9931-6aaf948872a0 HTTP/1.1" 200 - "-" 
"Java/1.7.0_80"
127.0.0.1 - - [15/Sep/2016:15:38:07 +] "GET 
/api/atlas/entities/1599d4b8-b6c0-4558-99a9-3e945560ee09 HTTP/1.1" 200 - "-" 
"Java/1.7.0_80"
127.0.0.1 - - [15/Sep/2016:15:38:07 +] "GET 
/api/atlas/entities/b07ecee2-3925-4602-987e-f4a210dc4140 HTTP/1.1" 200 - "-" 
"Java/1.7.0_80"
127.0.0.1 - - [15/Sep/2016:15:38:07 +] "GET 
/api/atlas/entities?type=hdfs_path=qualifiedName=p
 HTTP/1.1" 200 - "-" "Java/1.7.0_80"
127.0.0.1 - - [15/Sep/2016:15:38:07 +] "GET 
/api/atlas/entities?type=hdfs_path=qualifiedName=p
 HTTP/1.1" 200 - "-" "Java/1.7.0_80"
127.0.0.1 - - [15/Sep/2016:15:38:07 +] "GET 
/api/atlas/entities/b07ecee2-3925-4602-987e-f4a210dc4140 HTTP/1.1" 200 - "-" 
"Java/1.7.0_80"
127.0.0.1 - - [15/Sep/2016:15:38:08 +] "GET 
/api/atlas/entities?type=hive_table=qualifiedName=default.tablekimpykquum@primary
 HTTP/1.1" 200 - "-" "Java/1.7.0_80"
127.0.0.1 - - [15/Sep/2016:15:38:08 +] "GET 
/api/atlas/entities?type=hive_table=qualifiedName=default.tablekimpykquum@primary
 HTTP/1.1" 200 - "-" "Java/1.7.0_80"
127.0.0.1 - - [15/Sep/2016:15:38:08 +] "GET 
/api/atlas/entities/1599d4b8-b6c0-4558-99a9-3e945560ee09 HTTP/1.1" 200 - "-" 
"Java/1.7.0_80"
127.0.0.1 - - [15/Sep/2016:15:38:10 +] "GET 
/api/atlas/entities?type=hive_table=qualifiedName=default.tablekimpykquum@primary
 HTTP/1.1" 200 - "-" "Java/1.7.0_80"
127.0.0.1 - - [15/Sep/2016:15:38:10 +] "GET 
/api/atlas/entities?type=hive_table=qualifiedName=default.tablekimpykquum@primary
 HTTP/1.1" 200 - "-" "Java/1.7.0_80"
127.0.0.1 - - [15/Sep/2016:15:38:10 +] "GET 
/api/atlas/entities?type=hive_process=qualifiedName=QUERY:default.tablekimpykquum@primary:1473953881000-%3E:PATH_WRITE
 HTTP/1.1" 200 - "-" "Java/1.7.0_80"
127.0.0.1 - - [15/Sep/2016:15:38:10 +] "GET 
/api/atlas/entities?type=hive_process=qualifiedName=QUERY:default.tablekimpykquum@primary:1473953881000-%3E:PATH_WRITE
 HTTP/1.1" 200 - "-" "Java/1.7.0_80"
127.0.0.1 - - [15/Sep/2016:15:38:10 +] "GET 
/api/atlas/entities/f6c1ce66-117b-4fe7-9931-6aaf948872a0 HTTP/1.1" 200 - "-" 
"Java/1.7.0_80"
127.0.0.1 - - [15/Sep/2016:15:38:10 +] "GET 
/api/atlas/entities/1599d4b8-b6c0-4558-99a9-3e945560ee09 HTTP/1.1" 200 - "-" 
"Java/1.7.0_80"
127.0.0.1 - - [15/Sep/2016:15:38:10 +] "GET 
/api/atlas/entities/b07ecee2-3925-4602-987e-f4a210dc4140 HTTP/1.1" 200 - "-" 
"Java/1.7.0_80"
127.0.0.1 - - [15/Sep/2016:15:38:10 +] "GET 
/api/atlas/entities?type=hdfs_path=qualifiedName=p
 HTTP/1.1" 200 - "-" "Java/1.7.0_80"
127.0.0.1 - - [15/Sep/2016:15:38:10 +] "GET 
/api/atlas/entities?type=hdfs_path=qualifiedName=p
 HTTP/1.1" 200 - "-" "Java/1.7.0_80"
127.0.0.1 - - [15/Sep/2016:15:38:10 +] "GET 
/api/atlas/entities/b07ecee2-3925-4602-987e-f4a210dc4140 HTTP/1.1" 200 - "-" 
"Java/1.7.0_80"
127.0.0.1 - - [15/Sep/2016:15:38:13 +] "GET 
/api/atlas/entities?type=hive_table=qualifiedName=default.tablekimpykquum@primary
 HTTP/1.1" 200 - "-" "Java/1.7.0_80"
127.0.0.1 - - [15/Sep/2016:15:38:13 +] "GET 
/api/atlas/entities?type=hive_table=qualifiedName=default.tablekimpykquum@primary
 HTTP/1.1" 200 - "-" "Java/1.7.0_80"
127.0.0.1 - - [15/Sep/2016:15:38:13 +] "GET 
/api/atlas/entities?type=hive_process=qualifiedName=QUERY:default.tablekimpykquum@primary:1473953881000-%3E:PATH_WRITE
 HTTP/1.1" 200 - "-" "Java/1.7.0_80"
127.0.0.1 - - [15/Sep/2016:15:38:13 +] "GET 

Re: Review Request 51896: ATLAS-1171: structured, high-level APIs

[David Radley]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51896/#review149064
---



Some thoughts on this change.

I notice AltasEntities is a new interface that you have introduced. I cannot 
see it being used anywhere. The methods in this and other interfaces all throw 
Exception, if this is an API interface then I think it needs specific 
Exceptions.
I think that an API change to help understanding needs to have comprehensive 
javadoc to explain the inputs and outputs - this is what I expected from the 
Jira description.
It looks like there is a only one test introduced. I think for an API change 
like this you should aim to add in comprehensive testing; normal paths, maximum 
and minimums of parameters (values and lengths), non-standard characters in any 
supplied strings and tests of the expected errors for error cases.

- David Radley


On Sept. 14, 2016, 5:33 p.m., Madhan Neethiraj wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51896/
> ---
> 
> (Updated Sept. 14, 2016, 5:33 p.m.)
> 
> 
> Review request for atlas.
> 
> 
> Bugs: ATLAS-1171
> https://issues.apache.org/jira/browse/ATLAS-1171
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> first-cut API for review
> 
> 
> Diffs
> -
> 
>   common/pom.xml e3b6465 
>   common/src/main/java/org/apache/atlas/api/AtlasEntities.java PRE-CREATION 
>   common/src/main/java/org/apache/atlas/api/AtlasTypes.java PRE-CREATION 
>   common/src/main/java/org/apache/atlas/api/SearchFilter.java PRE-CREATION 
>   common/src/main/java/org/apache/atlas/model/AtlasAttributeDef.java 
> PRE-CREATION 
>   common/src/main/java/org/apache/atlas/model/AtlasBaseModelObject.java 
> PRE-CREATION 
>   common/src/main/java/org/apache/atlas/model/AtlasBuiltInDatatypes.java 
> PRE-CREATION 
>   common/src/main/java/org/apache/atlas/model/AtlasEntity.java PRE-CREATION 
>   common/src/main/java/org/apache/atlas/model/AtlasEntityDef.java 
> PRE-CREATION 
>   common/src/main/java/org/apache/atlas/model/AtlasEnumDef.java PRE-CREATION 
>   common/src/main/java/org/apache/atlas/model/AtlasObjectId.java PRE-CREATION 
>   common/src/main/java/org/apache/atlas/model/AtlasProcess.java PRE-CREATION 
>   common/src/main/java/org/apache/atlas/model/AtlasStruct.java PRE-CREATION 
>   common/src/main/java/org/apache/atlas/model/AtlasStructDef.java 
> PRE-CREATION 
>   common/src/main/java/org/apache/atlas/model/AtlasTrait.java PRE-CREATION 
>   common/src/main/java/org/apache/atlas/model/AtlasTraitDef.java PRE-CREATION 
>   common/src/test/java/org/apache/atlas/model/TestAtlasEntity.java 
> PRE-CREATION 
>   pom.xml 95f28b0 
> 
> Diff: https://reviews.apache.org/r/51896/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Madhan Neethiraj
> 
>

Re: Thoughts on the first step to understand the Atlas data model internals

[David Radley]

Hi Hermanth,
Thanks for your response. I agree we should evolve the existing 
documentation. I was aware of the guide you pointed me to. This appears to 
be useful for a user of the REST API. This sort of document is sometimes 
known as an application programming guide. The motivation of this persona 
is to effectively use the Atlas REST APIs to manage their metadata.

I am interested in documenting the internal data model and motivation for 
the separations of concerns in the source code. This is for different 
personas, namely to document design consensus around the data model by and 
for the Atlas contributers and committers. 
 all the best David. 



From:   Hemanth Yamijala 
To: "dev@atlas.incubator.apache.org" 
Date:   15/09/2016 14:28
Subject:Re: Thoughts on the first step to understand the Atlas 
data model internals



David,

This might not directly address all the points you mention below, but 
regarding learning about Atlas from a core understanding perspective, you 
may want to go through the document here: 
http://atlas.incubator.apache.org/AtlasTechnicalUserGuide.pdf. This is 
something we created just after the 0.7 release timeframe and feel it 
represents core concepts to a reasonable extent. 

There are several improvements that it needs: 
Likely, it is not complete and missing pieces. 
Further, being in PDF form makes it hard to edit, so moving it to a more 
editable format and specifically adding it to source code itself should be 
a goal.

Etc...

But if it forms a starting point, then it would be great to base further 
improvements on top of it.

Thanks
Hemanth

From: David Radley 
Sent: Thursday, September 15, 2016 6:26 PM
To: atlas
Subject: Thoughts on the first step to understand the Atlas data model 
internals

Hi,
I am looking to understand the important pieces in the Atlas architecture
and the order that is useful to think about them. I wanted to check my
understanding and questions around the top level data model concept as
someone relatively new to the project. then update the docs. I am
interested in feedback / thoughts / things I may have misunderstood. I
think the next areas for a person looking to understand the internals to
understand is the type system and providers and maybe then tracking how
the code flows from the web to the graph.

It seems that the data model of Atlas is the where to start; the
fundamental interface is interface ResourceDefinition and the base object
is BaseResourceDefinition. As the top level data object, I think this
needs to be simple and intuitive and have a defined purpose.
I would expect top level metadata objects should have the ability to have
relationships and attributes, which it seems to have and a way to identify
them (names and guid - which I do not see in this object)

I do not think the validate*** request call methods should live in this
interface. I would separate out request logic from the core data model
object; as they are different concerns.

The baseResourceDefinition has :
 protected static final TypeSystem typeSystem = TypeSystem.getInstance
();

protected final Set instanceProperties = new HashSet<>();
protected final Set collectionProperties = new HashSet<>();
protected Map propertyDefs = new
HashMap<>();
protected Map properties = new HashMap<>();

protected final Map projections = new HashMap<>();
protected final Map relations = new HashMap<>();

protected final PropertyMapper propertyMapper;
protected final Map
propertyValueFormatters = new HashMap<>();


There is an implied concept of  Property here in the naming of these
fields. If this is important then I suggest we have a Property class /
Interface defining what we mean by it.
I see there are projections and relationships. do we need both of these
concepts in the top level object as the only implementation of a
Projection is a RelationshipProjection.
I see AttributeDefintion in the list, this class does not subclass
ResourceDefintion - I am left thinking that in some way these are both
Definitions which I would expect to be a super class / interface.   Also
given that this is in a systemtypes package it should be a system type.
I see AttributeInfo and a not sure how this related to
AttributeDefintions. It would be great to be able to add in javadoc to
help here.
EntityResourceDefinition implements ResourceDefinition but says typename
is not meaningful for it. This is confusing. We should have an interface
without a gettypename to be clean or at least an explanation.
In UML, we often view relationships to other objects as attributes. I
wonder how we think of attributes and relationships here - can you define
an attribute with typeA and have a relationship to TypeA. Can I assume

[jira] [Comment Edited] (ATLAS-1161) Tags should be bound to an object's name and remain bound to all incarnations of that name

[Dan Markwat (JIRA)]

[ 
https://issues.apache.org/jira/browse/ATLAS-1161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15493356#comment-15493356
 ] 

Dan Markwat edited comment on ATLAS-1161 at 9/15/16 1:46 PM:
-

[~davidrad], yes this is involved very much in large ETL pipelines.  Other 
times, it may be that someone simply needs to throw the table away - as part of 
debugging, trying new things, break-fix, etc.  In any event within a short span 
of time (days to months) - for anyone using Hive as a data warehouse or shared, 
structured storage tool in any way - what I've seen so far is that nobody will 
likely be creating *and tagging* a table within a database with the same name 
as a table a few days prior without the two tables being closely related in 
some way (but I could totally be wrong with this assertion).  This is certainly 
true for the use cases I've been exposed to in data warehousing type setups for 
Hive.  This may not be true in all cases, but it lends to the idea that we at 
least need some way of keeping these tags applied to named objects after the 
object has been dropped and recreated, and putting the responsibility of 
keeping these tags up to date in the hands of a user.  Does that sound right to 
you?

Having the ETL jobs be responsible for tagging appears to defeat the perceived 
purpose of Atlas which would be to persist the tags and become a book of record 
for them.  (This could also be a perception problem or misunderstanding of the 
purpose of tagging on our part!).  If a job is responsible for tagging, all 
jobs collectively become the definitive record of what is tagged whereas Atlas 
merely reports that at one time or another something was tagged which is not 
quite what we're looking for...but I'm open to suggestions about alternate ways 
to do this!

Trying to play devil's advocate for myself and maybe see if what we're doing is 
a better fit elsewhere, I had a tangential thought:
- the business catalog/taxonomy: how is that structured?  Do those remain 
tagged to new incarnations of objects after the originally-tagged object is 
dropped?  Do they get pushed to Ranger as "tags" (/ can they be if they do 
persist?)

[~shwethags], yup I know the tags are kept and there are only "soft deletes" 
with new entities and guids being created every time; but myself and the people 
I work with are interested in a notion of more existential tagging that isn't 
tied to a (potentially) volatile object.  We are also trying to find a unified 
the path to auditing anything that shares a name across multiple incarnations 
which Atlas currently doesn't appear to have.  We would query for all the 
objects named a certain thing then audit them individually.  Maybe that's the 
only way there ever will be to do it?  Do you have other suggestions?

And yes, rule-based classification (similar to Ranger) would likely be totally 
acceptable; whatever decouples the tagging from the lifecycle of the object.  I 
do expect there are some people who want to use tagging in a more volatile 
manner, but from what I've seen totally user-driven tagging/untagging might 
work for most use cases.  And could it be that anyone needing a fresh slate for 
every incarnation of an object could simply "reset" the tags applied to it 
(perhaps?) and then proceed as they currently would?


was (Author: dmarkwat):
[~davidrad], yes this is involved very much in large ETL pipelines.  Other 
times, it may be that someone simply needs to throw the table away - as part of 
debugging, trying new things, break-fix, etc.  In any event within a short span 
of time (days to months) - for anyone using Hive as a data warehouse or shared, 
structured storage tool in any way - nobody will likely be creating *and 
tagging* a table within a database with the same name as a table a few days 
prior without the two tables being closely related in some way (as far as I can 
tell).  This is certainly true for the use cases I've been exposed to in data 
warehousing type setups for Hive.  This may not be true in all cases (but maybe 
it mostly is?), but it lends to the idea that we at least need some way of 
keeping these tags applied to named objects after the object has been dropped 
and recreated, and putting the responsibility of keeping these tags up to date 
in the hands of a user.

Having the ETL jobs be responsible for tagging appears to defeat the perceived 
purpose of Atlas which would be to persist the tags and become a book of record 
for them.  (This could also be a perception problem or misunderstanding of the 
purpose of tagging on our part!).  If a job is responsible for tagging, all 
jobs collectively become the definitive record of what is tagged whereas Atlas 
merely reports that at one time or another something was tagged.

Trying to play devil's advocate for myself and maybe see if what we're doing is 
a better fit elsewhere, I had a tangential thought:

[jira] [Commented] (ATLAS-1161) Tags should be bound to an object's name and remain bound to all incarnations of that name

[Dan Markwat (JIRA)]

[ 
https://issues.apache.org/jira/browse/ATLAS-1161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15493356#comment-15493356
 ] 

Dan Markwat commented on ATLAS-1161:


[~davidrad], yes this is involved very much in large ETL pipelines.  Other 
times, it may be that someone simply needs to throw the table away - as part of 
debugging, trying new things, break-fix, etc.  In any event within a short span 
of time (days to months) - for anyone using Hive as a data warehouse or shared, 
structured storage tool in any way - nobody will likely be creating *and 
tagging* a table within a database with the same name as a table a few days 
prior without the two tables being closely related in some way (as far as I can 
tell).  This is certainly true for the use cases I've been exposed to in data 
warehousing type setups for Hive.  This may not be true in all cases (but maybe 
it mostly is?), but it lends to the idea that we at least need some way of 
keeping these tags applied to named objects after the object has been dropped 
and recreated, and putting the responsibility of keeping these tags up to date 
in the hands of a user.

Having the ETL jobs be responsible for tagging appears to defeat the perceived 
purpose of Atlas which would be to persist the tags and become a book of record 
for them.  (This could also be a perception problem or misunderstanding of the 
purpose of tagging on our part!).  If a job is responsible for tagging, all 
jobs collectively become the definitive record of what is tagged whereas Atlas 
merely reports that at one time or another something was tagged.

Trying to play devil's advocate for myself and maybe see if what we're doing is 
a better fit elsewhere, I had a tangential thought:
- the business catalog/taxonomy: how is that structured?  Do those remain 
tagged to new incarnations of objects after the originally-tagged object is 
dropped?  Do they get pushed to Ranger as "tags" (/ can they be if they do 
persist?)

[~shwethags], yup I know the tags are kept and there are only "soft deletes" 
with new entities and guids being created every time; but myself and the people 
I work with are interested in a notion of more existential tagging that isn't 
tied to a (potentially) volatile object.  We are also trying to find a unified 
the path to auditing anything that shares a name across multiple incarnations 
which Atlas currently doesn't appear to have.  We would query for all the 
objects named a certain thing then audit them individually.  Maybe that's the 
only way there ever will be to do it?  Do you have other suggestions?

And yes, rule-based classification (similar to Ranger) would likely be totally 
acceptable; whatever decouples the tagging from the lifecycle of the object.  I 
do expect there are some people who want to use tagging in a more volatile 
manner, but from what I've seen totally user-driven tagging/untagging might 
work for most use cases.  And could it be that anyone needing a fresh slate for 
every incarnation of an object could simply "reset" the tags applied to it 
(perhaps?) and then proceed as they currently would?

> Tags should be bound to an object's name and remain bound to all incarnations 
> of that name
> --
>
> Key: ATLAS-1161
> URL: https://issues.apache.org/jira/browse/ATLAS-1161
> Project: Atlas
>  Issue Type: Improvement
>Affects Versions: trunk, 0.7-incubating
>Reporter: Dan Markwat
>
> As a user I would like tags I ascribe to an object in Atlas carry to the next 
> incarnation of that object.  In effect, tags would be ascribed to a 
> fully-qualified object name and all incarnations of that name would have the 
> tags apply to it.  (Not unlike Ranger and the way it applies policies to 
> objects).
> Example: I create a Hive table, TableA.  I tag TableA with tags, Tag1 and 
> Tag2.  I drop TableA.
> In the current Atlas world, if I create TableA again, Tag1 and Tag2 need to 
> be re-applied to TableA.  In the ideal governance/security world, if I 
> re-create TableA I should not have to re-tag it with Tag1 and Tag2; instead, 
> I should be required to *untag* TableA if I desire TableA to be clean and 
> untagged.  This effectively functions like a light switch: user turns on 
> light, just because the bulb is swapped out doesn't mean the switch turned 
> off - the user must explicitly turn the switch off, just as they did to turn 
> it on.  Think also about Ranger: just because I deleted an object doesn't 
> mean that policy goes away.
> By effectively deleting the binding of Tag1 and Tag2 to the name TableA 
> whenever TableA is deleted, Atlas ceases to be a book of record for tags 
> associated with TableA, as those tags would need to be applied again.  This 
> is bad in a world where creating/dropping objects and tagging 

Re: Thoughts on the first step to understand the Atlas data model internals

[Hemanth Yamijala]

David,

This might not directly address all the points you mention below, but regarding 
learning about Atlas from a core understanding perspective, you may want to go 
through the document here: 
http://atlas.incubator.apache.org/AtlasTechnicalUserGuide.pdf. This is 
something we created just after the 0.7 release timeframe and feel it 
represents core concepts to a reasonable extent. 

There are several improvements that it needs: 
Likely, it is not complete and missing pieces. 
Further, being in PDF form makes it hard to edit, so moving it to a more 
editable format and specifically adding it to source code itself should be a 
goal.

Etc...

But if it forms a starting point, then it would be great to base further 
improvements on top of it.

Thanks
Hemanth

From: David Radley 
Sent: Thursday, September 15, 2016 6:26 PM
To: atlas
Subject: Thoughts on the first step to understand the Atlas data model internals

Hi,
I am looking to understand the important pieces in the Atlas architecture
and the order that is useful to think about them. I wanted to check my
understanding and questions around the top level data model concept as
someone relatively new to the project. then update the docs. I am
interested in feedback / thoughts / things I may have misunderstood. I
think the next areas for a person looking to understand the internals to
understand is the type system and providers and maybe then tracking how
the code flows from the web to the graph.

It seems that the data model of Atlas is the where to start; the
fundamental interface is interface ResourceDefinition and the base object
is BaseResourceDefinition. As the top level data object, I think this
needs to be simple and intuitive and have a defined purpose.
I would expect top level metadata objects should have the ability to have
relationships and attributes, which it seems to have and a way to identify
them (names and guid - which I do not see in this object)

I do not think the validate*** request call methods should live in this
interface. I would separate out request logic from the core data model
object; as they are different concerns.

The baseResourceDefinition has :
 protected static final TypeSystem typeSystem = TypeSystem.getInstance
();

protected final Set instanceProperties = new HashSet<>();
protected final Set collectionProperties = new HashSet<>();
protected Map propertyDefs = new
HashMap<>();
protected Map properties = new HashMap<>();

protected final Map projections = new HashMap<>();
protected final Map relations = new HashMap<>();

protected final PropertyMapper propertyMapper;
protected final Map
propertyValueFormatters = new HashMap<>();


There is an implied concept of  Property here in the naming of these
fields. If this is important then I suggest we have a Property class /
Interface defining what we mean by it.
I see there are projections and relationships. do we need both of these
concepts in the top level object as the only implementation of a
Projection is a RelationshipProjection.
I see AttributeDefintion in the list, this class does not subclass
ResourceDefintion - I am left thinking that in some way these are both
Definitions which I would expect to be a super class / interface.   Also
given that this is in a systemtypes package it should be a system type.
I see AttributeInfo and a not sure how this related to
AttributeDefintions. It would be great to be able to add in javadoc to
help here.
EntityResourceDefinition implements ResourceDefinition but says typename
is not meaningful for it. This is confusing. We should have an interface
without a gettypename to be clean or at least an explanation.
In UML, we often view relationships to other objects as attributes. I
wonder how we think of attributes and relationships here - can you define
an attribute with typeA and have a relationship to TypeA. Can I assume
that attributes are contained compositions with the same lifeycle as the
container. Relationships could be bidirectional ,directional or
aggregation containment relationships with an independent  lifecycle to
the resourcedfintion
PropertyValueFormatter I wonder why these are in the top level object. Do
we expect different formatters for a given type? What is the use case here
- I wonder if we are exposing a plug point, so that other code could
provide custom formatters. If this is the intent how would this work?
PropertyMapper I find this name misleading. It seems to be translating for
fully formatted name to a clean name. I suggest using a more descriptive
name like PropertyNameCleanser.  Do we expect different Mappers for a
given type? What is the use case here - I wonder if we are exposing a plug
point, so that other code could provide custom mappers. If this is the
intent how would this work?

I 

[jira] [Comment Edited] (ATLAS-1171) Structured, high-level public APIs

[David Radley (JIRA)]

[ 
https://issues.apache.org/jira/browse/ATLAS-1171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15493293#comment-15493293
 ] 

David Radley edited comment on ATLAS-1171 at 9/15/16 1:19 PM:
--

I wonder if you could include ATLAS-1054 in this Jira 


was (Author: davidrad):
I wonder if you are addressing ATLAS-1054 in this fix. 

> Structured, high-level public APIs
> --
>
> Key: ATLAS-1171
> URL: https://issues.apache.org/jira/browse/ATLAS-1171
> Project: Atlas
>  Issue Type: Bug
>Reporter: Madhan Neethiraj
>Assignee: Madhan Neethiraj
> Attachments: ATLAS-1171.1.patch
>
>
> Many Atlas APIs use opaque JSON strings (or HTTP request/response contents 
> that are not defined in the interface) to pass details of types/entities - 
> for example here are some APIs:
>  - TypesResource.submit(HttpServletRequest request)
>  - TypesResource.update(HttpServletRequest request)
>  - Response getDefinition(HttpServletRequest request, String typeName)
>  - Response.getTypesByFilter(HttpServletRequest request, String typeCategory, 
> String superType, String notSuperType)
>  - EntityResource.submit(HttpServletRequest request)
>  - EntityResource.updateEntities(HttpServletRequest request)
>  - Response LineageResource.inputsGraph(String guid)
>  - Response LineageResource.outputsGraph(String guid)
>  - Response LineageResource.schema(String guid)
>  - ..
> Also, some APIs expose unnecessary details in the interface, like the one 
> shown below. These look like implementation details which is better not 
> exposed in the interface.
> {code}
> "jsonClass": 
> "org.apache.atlas.typesystem.json.InstanceSerialization$_Reference"
> {code}
> A structured API, with explicit details of the parameters and return value 
> contents, would make it easier to understand the Atlas APIs and create 
> applications that use Atlas services.
> I will shortly add a patch that shows the first-cut of this API. Please 
> review and provide your feedback. The initial patch does not include bunch of 
> APIs - like lineage, search, REST, Java API, etc; but it should give a pretty 
> good idea of the data-structures that will be used in the APIs.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (ATLAS-1171) Structured, high-level public APIs

[David Radley (JIRA)]

[ 
https://issues.apache.org/jira/browse/ATLAS-1171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15493293#comment-15493293
 ] 

David Radley commented on ATLAS-1171:
-

I wonder if you are addressing ATLAS-1054 in this fix. 

> Structured, high-level public APIs
> --
>
> Key: ATLAS-1171
> URL: https://issues.apache.org/jira/browse/ATLAS-1171
> Project: Atlas
>  Issue Type: Bug
>Reporter: Madhan Neethiraj
>Assignee: Madhan Neethiraj
> Attachments: ATLAS-1171.1.patch
>
>
> Many Atlas APIs use opaque JSON strings (or HTTP request/response contents 
> that are not defined in the interface) to pass details of types/entities - 
> for example here are some APIs:
>  - TypesResource.submit(HttpServletRequest request)
>  - TypesResource.update(HttpServletRequest request)
>  - Response getDefinition(HttpServletRequest request, String typeName)
>  - Response.getTypesByFilter(HttpServletRequest request, String typeCategory, 
> String superType, String notSuperType)
>  - EntityResource.submit(HttpServletRequest request)
>  - EntityResource.updateEntities(HttpServletRequest request)
>  - Response LineageResource.inputsGraph(String guid)
>  - Response LineageResource.outputsGraph(String guid)
>  - Response LineageResource.schema(String guid)
>  - ..
> Also, some APIs expose unnecessary details in the interface, like the one 
> shown below. These look like implementation details which is better not 
> exposed in the interface.
> {code}
> "jsonClass": 
> "org.apache.atlas.typesystem.json.InstanceSerialization$_Reference"
> {code}
> A structured API, with explicit details of the parameters and return value 
> contents, would make it easier to understand the Atlas APIs and create 
> applications that use Atlas services.
> I will shortly add a patch that shows the first-cut of this API. Please 
> review and provide your feedback. The initial patch does not include bunch of 
> APIs - like lineage, search, REST, Java API, etc; but it should give a pretty 
> good idea of the data-structures that will be used in the APIs.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Thoughts on the first step to understand the Atlas data model internals

[David Radley]

Hi,
I am looking to understand the important pieces in the Atlas architecture 
and the order that is useful to think about them. I wanted to check my 
understanding and questions around the top level data model concept as 
someone relatively new to the project. then update the docs. I am 
interested in feedback / thoughts / things I may have misunderstood. I 
think the next areas for a person looking to understand the internals to 
understand is the type system and providers and maybe then tracking how 
the code flows from the web to the graph. 

It seems that the data model of Atlas is the where to start; the 
fundamental interface is interface ResourceDefinition and the base object 
is BaseResourceDefinition. As the top level data object, I think this 
needs to be simple and intuitive and have a defined purpose. 
I would expect top level metadata objects should have the ability to have 
relationships and attributes, which it seems to have and a way to identify 
them (names and guid - which I do not see in this object) 

I do not think the validate*** request call methods should live in this 
interface. I would separate out request logic from the core data model 
object; as they are different concerns.
 
The baseResourceDefinition has : 
 protected static final TypeSystem typeSystem = TypeSystem.getInstance
();

protected final Set instanceProperties = new HashSet<>();
protected final Set collectionProperties = new HashSet<>();
protected Map propertyDefs = new 
HashMap<>();
protected Map properties = new HashMap<>();

protected final Map projections = new HashMap<>();
protected final Map relations = new HashMap<>();

protected final PropertyMapper propertyMapper;
protected final Map 
propertyValueFormatters = new HashMap<>();


There is an implied concept of  Property here in the naming of these 
fields. If this is important then I suggest we have a Property class / 
Interface defining what we mean by it. 
I see there are projections and relationships. do we need both of these 
concepts in the top level object as the only implementation of a 
Projection is a RelationshipProjection. 
I see AttributeDefintion in the list, this class does not subclass 
ResourceDefintion - I am left thinking that in some way these are both 
Definitions which I would expect to be a super class / interface.   Also 
given that this is in a systemtypes package it should be a system type. 
I see AttributeInfo and a not sure how this related to 
AttributeDefintions. It would be great to be able to add in javadoc to 
help here.
EntityResourceDefinition implements ResourceDefinition but says typename 
is not meaningful for it. This is confusing. We should have an interface 
without a gettypename to be clean or at least an explanation.
In UML, we often view relationships to other objects as attributes. I 
wonder how we think of attributes and relationships here - can you define 
an attribute with typeA and have a relationship to TypeA. Can I assume 
that attributes are contained compositions with the same lifeycle as the 
container. Relationships could be bidirectional ,directional or 
aggregation containment relationships with an independent  lifecycle to 
the resourcedfintion 
PropertyValueFormatter I wonder why these are in the top level object. Do 
we expect different formatters for a given type? What is the use case here 
- I wonder if we are exposing a plug point, so that other code could 
provide custom formatters. If this is the intent how would this work?
PropertyMapper I find this name misleading. It seems to be translating for 
fully formatted name to a clean name. I suggest using a more descriptive 
name like PropertyNameCleanser.  Do we expect different Mappers for a 
given type? What is the use case here - I wonder if we are exposing a plug 
point, so that other code could provide custom mappers. If this is the 
intent how would this work? 

I notice in the interface is has  String getIdPropertyName(); This 
defaults to "id". This is confusing, from the above properties are 
attributes in the context of a ResourceDefinition. This seems to be 
referring to the hard coded id property in the Request REST object. 

all the best, David. 


  
Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number 
741598. 
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

Build failed in Jenkins: apache-atlas-nightly #425

[Apache Jenkins Server]

See 

Changes:

[sshivalingamurthy] ATLAS-1173 Doc: Minor editorial bug in the example given 
for property

--
[...truncated 9071 lines...]
127.0.0.1 - - [15/Sep/2016:10:37:14 +] "GET 
/api/atlas/lineage/hive/table/default.table4jvjhrbgoi@primary/inputs/graph 
HTTP/1.1" 200 - "-" "Java/1.7.0_80"
127.0.0.1 - - [15/Sep/2016:10:37:19 +] "GET 
/api/atlas/entities?type=hive_process=qualifiedName=QUERY:default.tabled2wzgw4uz1@primary:1473935837000-%3E:PATH_WRITE
 HTTP/1.1" 404 - "-" "Java/1.7.0_80"
127.0.0.1 - - [15/Sep/2016:10:37:24 +] "GET 
/api/atlas/entities?type=hive_process=qualifiedName=QUERY:default.tabled2wzgw4uz1@primary:1473935837000-%3E:PATH_WRITE
 HTTP/1.1" 200 - "-" "Java/1.7.0_80"
127.0.0.1 - - [15/Sep/2016:10:37:24 +] "GET 
/api/atlas/entities?type=hive_process=qualifiedName=QUERY:default.tabled2wzgw4uz1@primary:1473935837000-%3E:PATH_WRITE
 HTTP/1.1" 200 - "-" "Java/1.7.0_80"
127.0.0.1 - - [15/Sep/2016:10:37:24 +] "GET 
/api/atlas/entities/982dbfca-f1e8-454c-bd34-816f40955637 HTTP/1.1" 200 - "-" 
"Java/1.7.0_80"
127.0.0.1 - - [15/Sep/2016:10:37:24 +] "GET 
/api/atlas/entities/62813b41-1efa-4de9-a2b9-f73bdb1b852c HTTP/1.1" 200 - "-" 
"Java/1.7.0_80"
127.0.0.1 - - [15/Sep/2016:10:37:24 +] "GET 
/api/atlas/entities/27cec48c-1961-47f8-aeaa-1a15cf3f85af HTTP/1.1" 200 - "-" 
"Java/1.7.0_80"
127.0.0.1 - - [15/Sep/2016:10:37:24 +] "GET 
/api/atlas/entities?type=hdfs_path=qualifiedName=p
 HTTP/1.1" 200 - "-" "Java/1.7.0_80"
127.0.0.1 - - [15/Sep/2016:10:37:24 +] "GET 
/api/atlas/entities?type=hdfs_path=qualifiedName=p
 HTTP/1.1" 200 - "-" "Java/1.7.0_80"
127.0.0.1 - - [15/Sep/2016:10:37:24 +] "GET 
/api/atlas/entities/27cec48c-1961-47f8-aeaa-1a15cf3f85af HTTP/1.1" 200 - "-" 
"Java/1.7.0_80"
127.0.0.1 - - [15/Sep/2016:10:37:24 +] "GET 
/api/atlas/entities?type=hive_table=qualifiedName=default.tabled2wzgw4uz1@primary
 HTTP/1.1" 200 - "-" "Java/1.7.0_80"
127.0.0.1 - - [15/Sep/2016:10:37:24 +] "GET 
/api/atlas/entities?type=hive_table=qualifiedName=default.tabled2wzgw4uz1@primary
 HTTP/1.1" 200 - "-" "Java/1.7.0_80"
127.0.0.1 - - [15/Sep/2016:10:37:24 +] "GET 
/api/atlas/entities/62813b41-1efa-4de9-a2b9-f73bdb1b852c HTTP/1.1" 200 - "-" 
"Java/1.7.0_80"
127.0.0.1 - - [15/Sep/2016:10:37:26 +] "GET 
/api/atlas/entities?type=hive_table=qualifiedName=default.tabled2wzgw4uz1@primary
 HTTP/1.1" 200 - "-" "Java/1.7.0_80"
127.0.0.1 - - [15/Sep/2016:10:37:26 +] "GET 
/api/atlas/entities?type=hive_table=qualifiedName=default.tabled2wzgw4uz1@primary
 HTTP/1.1" 200 - "-" "Java/1.7.0_80"
127.0.0.1 - - [15/Sep/2016:10:37:26 +] "GET 
/api/atlas/entities?type=hive_process=qualifiedName=QUERY:default.tabled2wzgw4uz1@primary:1473935837000-%3E:PATH_WRITE
 HTTP/1.1" 200 - "-" "Java/1.7.0_80"
127.0.0.1 - - [15/Sep/2016:10:37:26 +] "GET 
/api/atlas/entities?type=hive_process=qualifiedName=QUERY:default.tabled2wzgw4uz1@primary:1473935837000-%3E:PATH_WRITE
 HTTP/1.1" 200 - "-" "Java/1.7.0_80"
127.0.0.1 - - [15/Sep/2016:10:37:26 +] "GET 
/api/atlas/entities/982dbfca-f1e8-454c-bd34-816f40955637 HTTP/1.1" 200 - "-" 
"Java/1.7.0_80"
127.0.0.1 - - [15/Sep/2016:10:37:26 +] "GET 
/api/atlas/entities/62813b41-1efa-4de9-a2b9-f73bdb1b852c HTTP/1.1" 200 - "-" 
"Java/1.7.0_80"
127.0.0.1 - - [15/Sep/2016:10:37:26 +] "GET 
/api/atlas/entities/27cec48c-1961-47f8-aeaa-1a15cf3f85af HTTP/1.1" 200 - "-" 
"Java/1.7.0_80"
127.0.0.1 - - [15/Sep/2016:10:37:26 +] "GET 
/api/atlas/entities?type=hdfs_path=qualifiedName=p
 HTTP/1.1" 200 - "-" "Java/1.7.0_80"
127.0.0.1 - - [15/Sep/2016:10:37:26 +] "GET 
/api/atlas/entities?type=hdfs_path=qualifiedName=p
 HTTP/1.1" 200 - "-" "Java/1.7.0_80"
127.0.0.1 - - [15/Sep/2016:10:37:26 +] "GET 
/api/atlas/entities/27cec48c-1961-47f8-aeaa-1a15cf3f85af HTTP/1.1" 200 - "-" 
"Java/1.7.0_80"
127.0.0.1 - - [15/Sep/2016:10:37:29 +] "GET 
/api/atlas/entities?type=hive_table=qualifiedName=default.tabled2wzgw4uz1@primary
 HTTP/1.1" 200 - "-" "Java/1.7.0_80"
127.0.0.1 - - [15/Sep/2016:10:37:29 +] "GET 
/api/atlas/entities?type=hive_table=qualifiedName=default.tabled2wzgw4uz1@primary
 HTTP/1.1" 200 - "-" "Java/1.7.0_80"
127.0.0.1 - - [15/Sep/2016:10:37:29 +] "GET 
/api/atlas/entities?type=hive_process=qualifiedName=QUERY:default.tabled2wzgw4uz1@primary:1473935837000-%3E:PATH_WRITE
 HTTP/1.1" 200 - "-" "Java/1.7.0_80"
127.0.0.1 - - [15/Sep/2016:10:37:29 +] "GET 

[jira] [Commented] (ATLAS-1134) Full text search for "id" shows the blank screen in some case

[Shwetha G S (JIRA)]

[ 
https://issues.apache.org/jira/browse/ATLAS-1134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15492818#comment-15492818
 ] 

Shwetha G S commented on ATLAS-1134:


I got the screen with 'No Record found!'. When does this issue happen?

> Full text search for "id" shows the blank screen in some case
> -
>
> Key: ATLAS-1134
> URL: https://issues.apache.org/jira/browse/ATLAS-1134
> Project: Atlas
>  Issue Type: Bug
>Affects Versions: 0.7-incubating
>Reporter: Kalyani Kashikar
>Assignee: Kalyani Kashikar
> Attachments: ATLAS-1134.patch
>
>
> Step to reproduce - 
> 1.Search for "id" in full text. 
> 2.In case of entity Id is null, it doesn't shows the result, loader is 
> display on screen.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (ATLAS-1142) Lineage UI Improvement.

[Shwetha G S (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-1142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shwetha G S updated ATLAS-1142:
---
Attachment: Screen Shot 2016-09-15 at 2.16.57 PM.png

[~aahn], the patch adds asset type with the asset name(see the attached 
screenshot). In the header and lineage, asset type like hive table, mysql 
table, hbase table is displayed along with the actual table name. Does that 
look fine? 

> Lineage UI Improvement.
> ---
>
> Key: ATLAS-1142
> URL: https://issues.apache.org/jira/browse/ATLAS-1142
> Project: Atlas
>  Issue Type: Improvement
>Reporter: Nixon Rodrigues
>Assignee: Keval Bhatt
>Priority: Minor
> Attachments: ATLAS-1142.patch, Screen Shot 2016-09-15 at 2.16.57 
> PM.png
>
>
> Following improvement to be done in Lineage UI & entity detail page
> * Lineage UI should be resizable.
> * All the nodes to be shown in format *name (typename)*



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 51800: Support getTrait() API

[Shwetha GS]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51800/#review149034
---




client/src/main/java/org/apache/atlas/AtlasClient.java (line 1000)


Return list



client/src/main/java/org/apache/atlas/AtlasClient.java (line 1012)


Return org.apache.atlas.typesystem.Struct



webapp/src/main/java/org/apache/atlas/web/resources/EntityResource.java (line 
658)


metadataService.getTraitDefinition() loads the whole entity everytime for 
every trait of the entity. This is in-efficient. Use getEntityDefinition() once 
and read all the traits from entity definition

You can also change metadataService.getTraitDefinition() to return 
org.apache.atlas.typesystem.Struct


- Shwetha GS


On Sept. 15, 2016, 6:53 a.m., Vimal Sharma wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51800/
> ---
> 
> (Updated Sept. 15, 2016, 6:53 a.m.)
> 
> 
> Review request for atlas.
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Given entity id and trait name, support rest API that returns the trait 
> instance for the entity. Currently, the other way of getting it is 
> getEntity() which returns full entity with all trait instances
> 
> 
> Diffs
> -
> 
>   client/src/main/java/org/apache/atlas/AtlasClient.java 5ed79bc 
>   webapp/src/main/java/org/apache/atlas/web/resources/EntityResource.java 
> 82016d0 
>   
> webapp/src/test/java/org/apache/atlas/web/resources/EntityJerseyResourceIT.java
>  a1988ef 
> 
> Diff: https://reviews.apache.org/r/51800/diff/
> 
> 
> Testing
> ---
> 
> 
> File Attachments
> 
> 
> ATLAS-712-v2.patch
>   
> https://reviews.apache.org/media/uploaded/files/2016/09/13/b5079426-19d0-48f2-88f7-08e4e645bb32__ATLAS-712-v2.patch
> 
> 
> Thanks,
> 
> Vimal Sharma
> 
>

Build failed in Jenkins: apache-atlas-nightly #424

[Apache Jenkins Server]

See 

Changes:

[sshivalingamurthy] Added Tom Beerbower as committer

--
[...truncated 9135 lines...]
127.0.0.1 - - [15/Sep/2016:07:40:43 +] "GET 
/api/atlas/lineage/hive/table/default.tablesrn1granko@primary/inputs/graph 
HTTP/1.1" 200 - "-" "Java/1.7.0_80"
127.0.0.1 - - [15/Sep/2016:07:40:48 +] "GET 
/api/atlas/entities?type=hive_process=qualifiedName=QUERY:default.tablex97gmjvfft@primary:1473925246000-%3E:PATH_WRITE
 HTTP/1.1" 404 - "-" "Java/1.7.0_80"
127.0.0.1 - - [15/Sep/2016:07:40:53 +] "GET 
/api/atlas/entities?type=hive_process=qualifiedName=QUERY:default.tablex97gmjvfft@primary:1473925246000-%3E:PATH_WRITE
 HTTP/1.1" 200 - "-" "Java/1.7.0_80"
127.0.0.1 - - [15/Sep/2016:07:40:53 +] "GET 
/api/atlas/entities?type=hive_process=qualifiedName=QUERY:default.tablex97gmjvfft@primary:1473925246000-%3E:PATH_WRITE
 HTTP/1.1" 200 - "-" "Java/1.7.0_80"
127.0.0.1 - - [15/Sep/2016:07:40:53 +] "GET 
/api/atlas/entities/899dc59c-913c-45b1-a708-e9be8788c495 HTTP/1.1" 200 - "-" 
"Java/1.7.0_80"
127.0.0.1 - - [15/Sep/2016:07:40:53 +] "GET 
/api/atlas/entities/3b96a9ec-ef24-4d7f-a616-f0d51724d6bb HTTP/1.1" 200 - "-" 
"Java/1.7.0_80"
127.0.0.1 - - [15/Sep/2016:07:40:53 +] "GET 
/api/atlas/entities/73b31d1b-e9ba-4c0a-b45d-98ba9e35e641 HTTP/1.1" 200 - "-" 
"Java/1.7.0_80"
127.0.0.1 - - [15/Sep/2016:07:40:53 +] "GET 
/api/atlas/entities?type=hdfs_path=qualifiedName=p
 HTTP/1.1" 200 - "-" "Java/1.7.0_80"
127.0.0.1 - - [15/Sep/2016:07:40:53 +] "GET 
/api/atlas/entities?type=hdfs_path=qualifiedName=p
 HTTP/1.1" 200 - "-" "Java/1.7.0_80"
127.0.0.1 - - [15/Sep/2016:07:40:53 +] "GET 
/api/atlas/entities/73b31d1b-e9ba-4c0a-b45d-98ba9e35e641 HTTP/1.1" 200 - "-" 
"Java/1.7.0_80"
127.0.0.1 - - [15/Sep/2016:07:40:53 +] "GET 
/api/atlas/entities?type=hive_table=qualifiedName=default.tablex97gmjvfft@primary
 HTTP/1.1" 200 - "-" "Java/1.7.0_80"
127.0.0.1 - - [15/Sep/2016:07:40:53 +] "GET 
/api/atlas/entities?type=hive_table=qualifiedName=default.tablex97gmjvfft@primary
 HTTP/1.1" 200 - "-" "Java/1.7.0_80"
127.0.0.1 - - [15/Sep/2016:07:40:53 +] "GET 
/api/atlas/entities/3b96a9ec-ef24-4d7f-a616-f0d51724d6bb HTTP/1.1" 200 - "-" 
"Java/1.7.0_80"
127.0.0.1 - - [15/Sep/2016:07:40:55 +] "GET 
/api/atlas/entities?type=hive_table=qualifiedName=default.tablex97gmjvfft@primary
 HTTP/1.1" 200 - "-" "Java/1.7.0_80"
127.0.0.1 - - [15/Sep/2016:07:40:55 +] "GET 
/api/atlas/entities?type=hive_table=qualifiedName=default.tablex97gmjvfft@primary
 HTTP/1.1" 200 - "-" "Java/1.7.0_80"
127.0.0.1 - - [15/Sep/2016:07:40:55 +] "GET 
/api/atlas/entities?type=hive_process=qualifiedName=QUERY:default.tablex97gmjvfft@primary:1473925246000-%3E:PATH_WRITE
 HTTP/1.1" 200 - "-" "Java/1.7.0_80"
127.0.0.1 - - [15/Sep/2016:07:40:55 +] "GET 
/api/atlas/entities?type=hive_process=qualifiedName=QUERY:default.tablex97gmjvfft@primary:1473925246000-%3E:PATH_WRITE
 HTTP/1.1" 200 - "-" "Java/1.7.0_80"
127.0.0.1 - - [15/Sep/2016:07:40:55 +] "GET 
/api/atlas/entities/899dc59c-913c-45b1-a708-e9be8788c495 HTTP/1.1" 200 - "-" 
"Java/1.7.0_80"
127.0.0.1 - - [15/Sep/2016:07:40:55 +] "GET 
/api/atlas/entities/3b96a9ec-ef24-4d7f-a616-f0d51724d6bb HTTP/1.1" 200 - "-" 
"Java/1.7.0_80"
127.0.0.1 - - [15/Sep/2016:07:40:55 +] "GET 
/api/atlas/entities/73b31d1b-e9ba-4c0a-b45d-98ba9e35e641 HTTP/1.1" 200 - "-" 
"Java/1.7.0_80"
127.0.0.1 - - [15/Sep/2016:07:40:55 +] "GET 
/api/atlas/entities?type=hdfs_path=qualifiedName=p
 HTTP/1.1" 200 - "-" "Java/1.7.0_80"
127.0.0.1 - - [15/Sep/2016:07:40:55 +] "GET 
/api/atlas/entities?type=hdfs_path=qualifiedName=p
 HTTP/1.1" 200 - "-" "Java/1.7.0_80"
127.0.0.1 - - [15/Sep/2016:07:40:55 +] "GET 
/api/atlas/entities/73b31d1b-e9ba-4c0a-b45d-98ba9e35e641 HTTP/1.1" 200 - "-" 
"Java/1.7.0_80"
127.0.0.1 - - [15/Sep/2016:07:40:58 +] "GET 
/api/atlas/entities?type=hive_table=qualifiedName=default.tablex97gmjvfft@primary
 HTTP/1.1" 200 - "-" "Java/1.7.0_80"
127.0.0.1 - - [15/Sep/2016:07:40:58 +] "GET 
/api/atlas/entities?type=hive_table=qualifiedName=default.tablex97gmjvfft@primary
 HTTP/1.1" 200 - "-" "Java/1.7.0_80"
127.0.0.1 - - [15/Sep/2016:07:40:58 +] "GET 
/api/atlas/entities?type=hive_process=qualifiedName=QUERY:default.tablex97gmjvfft@primary:1473925246000-%3E:PATH_WRITE
 HTTP/1.1" 200 - "-" "Java/1.7.0_80"
127.0.0.1 - - [15/Sep/2016:07:40:58 +] "GET 

[jira] [Commented] (ATLAS-1142) Lineage UI Improvement.

[Keval Bhatt (JIRA)]

[ 
https://issues.apache.org/jira/browse/ATLAS-1142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15492618#comment-15492618
 ] 

Keval Bhatt commented on ATLAS-1142:


Thanks [~shwethags] for the review.  I will update patch with excluding 
jquery-ui from pom.xml and update licence file as per your comment. 

> Lineage UI Improvement.
> ---
>
> Key: ATLAS-1142
> URL: https://issues.apache.org/jira/browse/ATLAS-1142
> Project: Atlas
>  Issue Type: Improvement
>Reporter: Nixon Rodrigues
>Assignee: Keval Bhatt
>Priority: Minor
> Attachments: ATLAS-1142.patch
>
>
> Following improvement to be done in Lineage UI & entity detail page
> * Lineage UI should be resizable.
> * All the nodes to be shown in format *name (typename)*



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (ATLAS-1142) Lineage UI Improvement.

[Shwetha G S (JIRA)]

[ 
https://issues.apache.org/jira/browse/ATLAS-1142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15492574#comment-15492574
 ] 

Shwetha G S commented on ATLAS-1142:


Build fails in rat check:
Unapproved licenses:

  
/Users/sshivalingamurthy/git/atlas-clone/dashboardv2/public/js/jquery-ui/jquery-ui.min.css
  
/Users/sshivalingamurthy/git/atlas-clone/dashboardv2/public/js/jquery-ui/jquery-ui.min.js

> Lineage UI Improvement.
> ---
>
> Key: ATLAS-1142
> URL: https://issues.apache.org/jira/browse/ATLAS-1142
> Project: Atlas
>  Issue Type: Improvement
>Reporter: Nixon Rodrigues
>Assignee: Keval Bhatt
>Priority: Minor
> Attachments: ATLAS-1142.patch
>
>
> Following improvement to be done in Lineage UI & entity detail page
> * Lineage UI should be resizable.
> * All the nodes to be shown in format *name (typename)*



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 51800: Support getTrait() API

[Vimal Sharma]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51800/
---

(Updated Sept. 15, 2016, 6:53 a.m.)


Review request for atlas.


Repository: atlas


Description
---

Given entity id and trait name, support rest API that returns the trait 
instance for the entity. Currently, the other way of getting it is getEntity() 
which returns full entity with all trait instances


Diffs (updated)
-

  client/src/main/java/org/apache/atlas/AtlasClient.java 5ed79bc 
  webapp/src/main/java/org/apache/atlas/web/resources/EntityResource.java 
82016d0 
  
webapp/src/test/java/org/apache/atlas/web/resources/EntityJerseyResourceIT.java 
a1988ef 

Diff: https://reviews.apache.org/r/51800/diff/


Testing
---


File Attachments


ATLAS-712-v2.patch
  
https://reviews.apache.org/media/uploaded/files/2016/09/13/b5079426-19d0-48f2-88f7-08e4e645bb32__ATLAS-712-v2.patch


Thanks,

Vimal Sharma

Re: Review Request 51800: Support getTrait() API

[Vimal Sharma]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51800/
---

(Updated Sept. 15, 2016, 6:52 a.m.)


Review request for atlas.


Repository: atlas


Description
---

Given entity id and trait name, support rest API that returns the trait 
instance for the entity. Currently, the other way of getting it is getEntity() 
which returns full entity with all trait instances


Diffs
-

  webapp/src/main/java/org/apache/atlas/web/resources/EntityResource.java 
82016d0 
  
webapp/src/test/java/org/apache/atlas/web/resources/EntityJerseyResourceIT.java 
a1988ef 

Diff: https://reviews.apache.org/r/51800/diff/


Testing
---


File Attachments (updated)


ATLAS-712-v2.patch
  
https://reviews.apache.org/media/uploaded/files/2016/09/13/b5079426-19d0-48f2-88f7-08e4e645bb32__ATLAS-712-v2.patch


Thanks,

Vimal Sharma

[jira] [Commented] (ATLAS-1142) Lineage UI Improvement.

[Shwetha G S (JIRA)]

[ 
https://issues.apache.org/jira/browse/ATLAS-1142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15492559#comment-15492559
 ] 

Shwetha G S commented on ATLAS-1142:


The js file mentions its MIT license. Can you replace jQuery Foundation License 
with MIT License in the following lines:
+This product bundles jQuery-ui.js, which is available under
+jQuery Foundation License.  For details, see 3party-licenses/jQuery-ui-LICENSE

> Lineage UI Improvement.
> ---
>
> Key: ATLAS-1142
> URL: https://issues.apache.org/jira/browse/ATLAS-1142
> Project: Atlas
>  Issue Type: Improvement
>Reporter: Nixon Rodrigues
>Assignee: Keval Bhatt
>Priority: Minor
> Attachments: ATLAS-1142.patch
>
>
> Following improvement to be done in Lineage UI & entity detail page
> * Lineage UI should be resizable.
> * All the nodes to be shown in format *name (typename)*



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (ATLAS-1173) Doc: Minor editorial bug in the example given for property atlas.server.ha.zookeeper.auth

[Shwetha G S (JIRA)]

[ 
https://issues.apache.org/jira/browse/ATLAS-1173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15492556#comment-15492556
 ] 

Shwetha G S commented on ATLAS-1173:


+1

> Doc: Minor editorial bug in the example given for property 
> atlas.server.ha.zookeeper.auth
> -
>
> Key: ATLAS-1173
> URL: https://issues.apache.org/jira/browse/ATLAS-1173
> Project: Atlas
>  Issue Type: Bug
>Affects Versions: 0.7-incubating
>Reporter: Hemanth Yamijala
>Assignee: Hemanth Yamijala
>Priority: Minor
> Attachments: ATLAS-1173.patch
>
>
> In {{Configuration.twiki}}, we have specified the example for Zookeeper 
> security configs as follows (under the High Availability Properties).
> {code}
> atlas.server.ha.zookeeper.acl=auth:sasl:cli...@comany.com
> {code}
> The auth: prefix in the value is incorrect and should be removed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)