[jira] [Commented] (ATLAS-1145) Storm hook with kafka topic and hive bolt picks wrong hive table name

2016-09-19 Thread Hemanth Yamijala (JIRA) 

[ 
https://issues.apache.org/jira/browse/ATLAS-1145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15505596#comment-15505596
 ] 

Hemanth Yamijala commented on ATLAS-1145:
-

[~sarath.ku...@gmail.com], Some time back I had a working version of Kafka to 
Hive storm topology capturing lineage into Atlas etc. Code is here: 
https://github.com/yhemanth/storm-samples/blob/master/src/main/java/com/dsinpractice/storm/samples/WordCountTopology.java
 However, we shifted focus from Hive to HDFS for later testing.

The main issue AFAIK is to setup Hive for streaming accurately. It took some 
time for me to figure that out.  Wish I had documented the steps more clearly 
:-(

These are properties I have in my hive-site which *I think* were used to get 
these to work. Please use these in case they are of help.

{code}

hive.support.concurrency
true


hive.enforce.bucketing
true


hive.exec.dynamic.partition.mode
nonstrict


hive.compactor.initiator.on
true


hive.txn.manager
org.apache.hadoop.hive.ql.lockmgr.DbTxnManager


hive.compactor.initiator.on
true 


hive.compactor.worker.threads
1 

{code}

> Storm hook with kafka topic and hive bolt picks wrong hive table name 
> --
>
> Key: ATLAS-1145
> URL: https://issues.apache.org/jira/browse/ATLAS-1145
> Project: Atlas
>  Issue Type: Bug
>Reporter: Shwetha G S
>Assignee: Sarath Subramanian
>
> Storm hook with kafka topic and hive bolt picks kafka topic name as hive 
> table name



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Thoughts on the first step to understand the Atlas data model internals

2016-09-19 Thread Shwetha Shivalingamurthy 
David,

EntityResourceDefinition was added as part of business catalog, and this
might change. 

org.apache.atlas.web.resources.EntityResource and
org.apache.atlas.web.resources.TypesResource are the initial APIs added,
these are used by AtlasClient and Atlas UI and is also documented in
Hemanth¹s doc

Regards,
Shwetha






On 15/09/16, 8:27 PM, "David Radley"  wrote:

>Hi Hermanth,
>Thanks for your response. I agree we should evolve the existing
>documentation. I was aware of the guide you pointed me to. This appears
>to 
>be useful for a user of the REST API. This sort of document is sometimes
>known as an application programming guide. The motivation of this persona
>is to effectively use the Atlas REST APIs to manage their metadata.
>
>I am interested in documenting the internal data model and motivation for
>the separations of concerns in the source code. This is for different
>personas, namely to document design consensus around the data model by
>and 
>for the Atlas contributers and committers.
> all the best David.
>
>
>
>From:   Hemanth Yamijala 
>To: "dev@atlas.incubator.apache.org" 
>Date:   15/09/2016 14:28
>Subject:Re: Thoughts on the first step to understand the Atlas
>data model internals
>
>
>
>David,
>
>This might not directly address all the points you mention below, but
>regarding learning about Atlas from a core understanding perspective, you
>may want to go through the document here:
>http://atlas.incubator.apache.org/AtlasTechnicalUserGuide.pdf. This is
>something we created just after the 0.7 release timeframe and feel it
>represents core concepts to a reasonable extent.
>
>There are several improvements that it needs:
>Likely, it is not complete and missing pieces.
>Further, being in PDF form makes it hard to edit, so moving it to a more
>editable format and specifically adding it to source code itself should
>be 
>a goal.
>
>Etc...
>
>But if it forms a starting point, then it would be great to base further
>improvements on top of it.
>
>Thanks
>Hemanth
>
>From: David Radley 
>Sent: Thursday, September 15, 2016 6:26 PM
>To: atlas
>Subject: Thoughts on the first step to understand the Atlas data model
>internals
>
>Hi,
>I am looking to understand the important pieces in the Atlas architecture
>and the order that is useful to think about them. I wanted to check my
>understanding and questions around the top level data model concept as
>someone relatively new to the project. then update the docs. I am
>interested in feedback / thoughts / things I may have misunderstood. I
>think the next areas for a person looking to understand the internals to
>understand is the type system and providers and maybe then tracking how
>the code flows from the web to the graph.
>
>It seems that the data model of Atlas is the where to start; the
>fundamental interface is interface ResourceDefinition and the base object
>is BaseResourceDefinition. As the top level data object, I think this
>needs to be simple and intuitive and have a defined purpose.
>I would expect top level metadata objects should have the ability to have
>relationships and attributes, which it seems to have and a way to identify
>them (names and guid - which I do not see in this object)
>
>I do not think the validate*** request call methods should live in this
>interface. I would separate out request logic from the core data model
>object; as they are different concerns.
>
>The baseResourceDefinition has :
> protected static final TypeSystem typeSystem = TypeSystem.getInstance
>();
>
>protected final Set instanceProperties = new HashSet<>();
>protected final Set collectionProperties = new HashSet<>();
>protected Map propertyDefs = new
>HashMap<>();
>protected Map properties = new HashMap<>();
>
>protected final Map projections = new HashMap<>();
>protected final Map relations = new HashMap<>();
>
>protected final PropertyMapper propertyMapper;
>protected final Map
>propertyValueFormatters = new HashMap<>();
>
>
>There is an implied concept of  Property here in the naming of these
>fields. If this is important then I suggest we have a Property class /
>Interface defining what we mean by it.
>I see there are projections and relationships. do we need both of these
>concepts in the top level object as the only implementation of a
>Projection is a RelationshipProjection.
>I see AttributeDefintion in the list, this class does not subclass
>ResourceDefintion - I am left thinking that in some way these are both
>Definitions which I would expect to be a super class / interface.   Also
>given that this is in a systemtypes package it should be a system type.
>I see AttributeInfo and a not sure how this related to

[jira] [Commented] (ATLAS-1145) Storm hook with kafka topic and hive bolt picks wrong hive table name

2016-09-19 Thread Shwetha G S (JIRA) 

[ 
https://issues.apache.org/jira/browse/ATLAS-1145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15505532#comment-15505532
 ] 

Shwetha G S commented on ATLAS-1145:


Hemanth has one - https://github.com/yhemanth/storm-samples

> Storm hook with kafka topic and hive bolt picks wrong hive table name 
> --
>
> Key: ATLAS-1145
> URL: https://issues.apache.org/jira/browse/ATLAS-1145
> Project: Atlas
>  Issue Type: Bug
>Reporter: Shwetha G S
>Assignee: Sarath Subramanian
>
> Storm hook with kafka topic and hive bolt picks kafka topic name as hive 
> table name



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 51939: Framework to apply updates to types in the type-system

2016-09-19 Thread Suma Shivaprasad 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51939/#review149608
---




repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java 
(line 191)


it would make more sense to sort by date instead of file name?


- Suma Shivaprasad


On Sept. 19, 2016, 6:10 p.m., Sarath Kumar Subramanian wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51939/
> ---
> 
> (Updated Sept. 19, 2016, 6:10 p.m.)
> 
> 
> Review request for atlas, Madhan Neethiraj, Shwetha GS, and Suma Shivaprasad.
> 
> 
> Bugs: ATLAS-1174
> https://issues.apache.org/jira/browse/ATLAS-1174
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> 1. Introduce "version" attribute to all types in the type-system, this helps 
> to track changes made to the default types (hive, sqoop, falcon and storm 
> types) and user created types. If version is not mentioned during creation of 
> a type, default version "1.0" is assigned (optional attribute).
> 2. Using the version attributed for types, introduce a patch framework for 
> type system. This framework applies patches to a type using its version 
> number and can be used during upgrade - add new attributes to an existing 
> types and it will be run during atlas startup.
> The sequence of steps:
> a. During atlas startup, check $ATLAS_HOME/models/patches directory for any 
> available patch files (json files). If there any patch files handle them.
> b. Sample patch json file looks like:
> {
> "patches": [
> { 
> "action": "ADD_ATTRIBUTE",
> "typeName": "hive_column",
> "applyToVersion": "1.0",
> "updateToVersion": "2.0",
> "actionParams": [
> { "name": "position", "dataTypeName": "int", "multiplicity": "optional", 
> "isComposite": false, "isUnique": false, "isIndexable": false, 
> "reverseAttributeName": null }
> ]
> } ]
> }
> c. The framework updates the type in "typeName" for the matching version 
> number and after applying the patch, update the version to the one mentioned 
> in "updateToVersion"
> d. The json file can have more than one action (array of actions).
> e. There can be multiple patch json files in the directory and are applied in 
> the sort order of the filename. eg:
> 001-hive_column_add_position.json
> 002-hive_column_add_anotherattribute.json
> 
> 
> Diffs
> -
> 
>   common/src/main/java/org/apache/atlas/repository/Constants.java d7f9c89 
>   
> repository/src/main/java/org/apache/atlas/repository/typestore/GraphBackedTypeStore.java
>  a94d157 
>   repository/src/main/java/org/apache/atlas/services/AddAttributePatch.java 
> PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/services/AtlasTypePatch.java 
> PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java
>  6a937f4 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/AbstractDataType.java
>  fad091d 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/ClassType.java 
> c56987a 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/EnumType.java 
> bdd0a13 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/EnumTypeDefinition.java
>  6340615 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/HierarchicalType.java
>  7224699 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/HierarchicalTypeDefinition.java
>  9a299f0 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/IDataType.java 
> 85ddee7 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/StructType.java 
> 6f40c1d 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/StructTypeDefinition.java
>  f1ce1b7 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/TraitType.java 
> f23bf5b 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/TypeSystem.java 
> 70ba89b 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/utils/TypesUtil.java
>  ef8448d 
>   
> typesystem/src/main/scala/org/apache/atlas/typesystem/json/TypesSerialization.scala
>  5618938 
> 
> Diff: https://reviews.apache.org/r/51939/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sarath Kumar Subramanian
> 
>

Re: Review Request 51939: Framework to apply updates to types in the type-system

2016-09-19 Thread Suma Shivaprasad 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51939/#review149607
---




repository/src/main/java/org/apache/atlas/services/AddAttributePatch.java (line 
102)


PatchResult should be modified to have the error message as to why the 
update failed



repository/src/main/java/org/apache/atlas/services/AddAttributePatch.java (line 
104)


Log instead of PTS


- Suma Shivaprasad


On Sept. 19, 2016, 6:10 p.m., Sarath Kumar Subramanian wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51939/
> ---
> 
> (Updated Sept. 19, 2016, 6:10 p.m.)
> 
> 
> Review request for atlas, Madhan Neethiraj, Shwetha GS, and Suma Shivaprasad.
> 
> 
> Bugs: ATLAS-1174
> https://issues.apache.org/jira/browse/ATLAS-1174
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> 1. Introduce "version" attribute to all types in the type-system, this helps 
> to track changes made to the default types (hive, sqoop, falcon and storm 
> types) and user created types. If version is not mentioned during creation of 
> a type, default version "1.0" is assigned (optional attribute).
> 2. Using the version attributed for types, introduce a patch framework for 
> type system. This framework applies patches to a type using its version 
> number and can be used during upgrade - add new attributes to an existing 
> types and it will be run during atlas startup.
> The sequence of steps:
> a. During atlas startup, check $ATLAS_HOME/models/patches directory for any 
> available patch files (json files). If there any patch files handle them.
> b. Sample patch json file looks like:
> {
> "patches": [
> { 
> "action": "ADD_ATTRIBUTE",
> "typeName": "hive_column",
> "applyToVersion": "1.0",
> "updateToVersion": "2.0",
> "actionParams": [
> { "name": "position", "dataTypeName": "int", "multiplicity": "optional", 
> "isComposite": false, "isUnique": false, "isIndexable": false, 
> "reverseAttributeName": null }
> ]
> } ]
> }
> c. The framework updates the type in "typeName" for the matching version 
> number and after applying the patch, update the version to the one mentioned 
> in "updateToVersion"
> d. The json file can have more than one action (array of actions).
> e. There can be multiple patch json files in the directory and are applied in 
> the sort order of the filename. eg:
> 001-hive_column_add_position.json
> 002-hive_column_add_anotherattribute.json
> 
> 
> Diffs
> -
> 
>   common/src/main/java/org/apache/atlas/repository/Constants.java d7f9c89 
>   
> repository/src/main/java/org/apache/atlas/repository/typestore/GraphBackedTypeStore.java
>  a94d157 
>   repository/src/main/java/org/apache/atlas/services/AddAttributePatch.java 
> PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/services/AtlasTypePatch.java 
> PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java
>  6a937f4 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/AbstractDataType.java
>  fad091d 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/ClassType.java 
> c56987a 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/EnumType.java 
> bdd0a13 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/EnumTypeDefinition.java
>  6340615 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/HierarchicalType.java
>  7224699 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/HierarchicalTypeDefinition.java
>  9a299f0 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/IDataType.java 
> 85ddee7 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/StructType.java 
> 6f40c1d 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/StructTypeDefinition.java
>  f1ce1b7 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/TraitType.java 
> f23bf5b 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/TypeSystem.java 
> 70ba89b 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/utils/TypesUtil.java
>  ef8448d 
>   
> typesystem/src/main/scala/org/apache/atlas/typesystem/json/TypesSerialization.scala
>  5618938 
> 
> Diff: https://reviews.apache.org/r/51939/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sarath Kumar Subramanian
> 
>

[jira] [Commented] (ATLAS-1145) Storm hook with kafka topic and hive bolt picks wrong hive table name

2016-09-19 Thread Sarath Subramanian (JIRA) 

[ 
https://issues.apache.org/jira/browse/ATLAS-1145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15505236#comment-15505236
 ] 

Sarath Subramanian commented on ATLAS-1145:
---

[~shwethags] Do you have a sample script/command to create a storm topology 
with kafka spout and hive bolt? I'm using  - 
storm jar 
/usr/hdp/current/storm-client/contrib/storm-starter/storm-starter-topologies-*.jar
 storm.starter.StatefulTopology test_topology_1

but it doesn't have any hive bolt or kafka spout.

> Storm hook with kafka topic and hive bolt picks wrong hive table name 
> --
>
> Key: ATLAS-1145
> URL: https://issues.apache.org/jira/browse/ATLAS-1145
> Project: Atlas
>  Issue Type: Bug
>Reporter: Shwetha G S
>Assignee: Sarath Subramanian
>
> Storm hook with kafka topic and hive bolt picks kafka topic name as hive 
> table name



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 51896: ATLAS-1171: structured, high-level APIs

2016-09-19 Thread Jeff Hagelberg 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51896/#review149121
---




common/src/main/java/org/apache/atlas/model/AtlasAttributeDef.java (line 55)


Generally, please add javadoc that describes the parameters and their 
intent.



common/src/main/java/org/apache/atlas/model/AtlasAttributeDef.java (line 58)


This seems redundant.  Isn't it automatically optional if valuesMinCount = 
0?



common/src/main/java/org/apache/atlas/model/AtlasAttributeDef.java (line 61)


Why is this needed?  It seems like if 'valuesMaxCount' > 1, it is 
automatically multi-valued.



common/src/main/java/org/apache/atlas/model/AtlasAttributeDef.java (line 62)


Should valuesMinCount/valuesMaxCount be changed to lowerBound/upperBound?  
Those seem like more intuitive names.



common/src/main/java/org/apache/atlas/model/AtlasAttributeDef.java (line 64)


How is this different from isUnique?



common/src/main/java/org/apache/atlas/model/AtlasBaseModelObject.java (line 45)


Could you change these to a java enumeration?  Most serialization 
frameworks have no trouble with enumerations.



common/src/main/java/org/apache/atlas/model/AtlasBaseModelObject.java (line 47)


It's not clear what "disabled" is and how it is different from "deleted".  
I also tend to think that the status should be an internal attribute that is 
not exposed.  It seems like by default, users should be able to assume that any 
object that comes back from a query has not been deleted.  I know that is not 
currently the case though.  The query behavior really should not depend on 
whether hard or soft delete is being used, though, it should be transparent.  I 
would prefer to have a separate set of apis for managing the entity status 
where you can query for deleted objects and change their state back to ACTIVE 
if desired.



common/src/main/java/org/apache/atlas/model/AtlasBuiltInDatatypes.java (line 22)


It seems like constants like this should be declared once in the code and 
referred to everyplace else.  For example, AtlasClient.DataSet, 
AtlasClient.Process, BooleanType.name (which could be made public).



common/src/main/java/org/apache/atlas/model/AtlasBuiltInDatatypes.java (line 33)


I'm not sure what this is.  Is it a data type?



common/src/main/java/org/apache/atlas/model/AtlasBuiltInDatatypes.java (line 34)


I think you are missing a few built-in types like dataset, etc.  The 
process type name also starts with a capital P.


- Jeff Hagelberg


On Sept. 19, 2016, 6:51 p.m., Madhan Neethiraj wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51896/
> ---
> 
> (Updated Sept. 19, 2016, 6:51 p.m.)
> 
> 
> Review request for atlas.
> 
> 
> Bugs: ATLAS-1171
> https://issues.apache.org/jira/browse/ATLAS-1171
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> first-cut API for review
> 
> 
> Diffs
> -
> 
>   common/pom.xml e3b6465 
>   common/src/main/java/org/apache/atlas/api/AtlasEntities.java PRE-CREATION 
>   common/src/main/java/org/apache/atlas/api/AtlasTypes.java PRE-CREATION 
>   common/src/main/java/org/apache/atlas/api/SearchFilter.java PRE-CREATION 
>   common/src/main/java/org/apache/atlas/model/instances/AtlasEntity.java 
> PRE-CREATION 
>   common/src/main/java/org/apache/atlas/model/instances/AtlasObjectId.java 
> PRE-CREATION 
>   common/src/main/java/org/apache/atlas/model/instances/AtlasProcess.java 
> PRE-CREATION 
>   common/src/main/java/org/apache/atlas/model/instances/AtlasStruct.java 
> PRE-CREATION 
>   common/src/main/java/org/apache/atlas/model/instances/AtlasTrait.java 
> PRE-CREATION 
>   common/src/main/java/org/apache/atlas/model/types/AtlasBaseTypeDef.java 
> PRE-CREATION 
>   common/src/main/java/org/apache/atlas/model/types/AtlasEntityDef.java 
> PRE-CREATION 
>   common/src/main/java/org/apache/atlas/model/types/AtlasEnumDef.java 
> PRE-CREATION 
>   common/src/main/java/org/apache/atlas/model/types/AtlasStructDef.java 
> PRE-CREATION 
>   common/src/main/java/org/apache/atlas/model/types/AtlasTraitDef.java 
> PRE-CREATION 
>   common/src/main/java/org/apache/atlas/model/types/AtlasTypeId.java 
> PRE-CREATION 
>

Re: Review Request 51939: Framework to apply updates to types in the type-system

2016-09-19 Thread Suma Shivaprasad 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51939/#review149553
---




repository/src/main/java/org/apache/atlas/repository/typestore/GraphBackedTypeStore.java
 (line 68)


is this the default type version. If so, pls rename as DEFAULT_TYPE_VERSION


- Suma Shivaprasad


On Sept. 19, 2016, 6:10 p.m., Sarath Kumar Subramanian wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51939/
> ---
> 
> (Updated Sept. 19, 2016, 6:10 p.m.)
> 
> 
> Review request for atlas, Madhan Neethiraj, Shwetha GS, and Suma Shivaprasad.
> 
> 
> Bugs: ATLAS-1174
> https://issues.apache.org/jira/browse/ATLAS-1174
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> 1. Introduce "version" attribute to all types in the type-system, this helps 
> to track changes made to the default types (hive, sqoop, falcon and storm 
> types) and user created types. If version is not mentioned during creation of 
> a type, default version "1.0" is assigned (optional attribute).
> 2. Using the version attributed for types, introduce a patch framework for 
> type system. This framework applies patches to a type using its version 
> number and can be used during upgrade - add new attributes to an existing 
> types and it will be run during atlas startup.
> The sequence of steps:
> a. During atlas startup, check $ATLAS_HOME/models/patches directory for any 
> available patch files (json files). If there any patch files handle them.
> b. Sample patch json file looks like:
> {
> "patches": [
> { 
> "action": "ADD_ATTRIBUTE",
> "typeName": "hive_column",
> "applyToVersion": "1.0",
> "updateToVersion": "2.0",
> "actionParams": [
> { "name": "position", "dataTypeName": "int", "multiplicity": "optional", 
> "isComposite": false, "isUnique": false, "isIndexable": false, 
> "reverseAttributeName": null }
> ]
> } ]
> }
> c. The framework updates the type in "typeName" for the matching version 
> number and after applying the patch, update the version to the one mentioned 
> in "updateToVersion"
> d. The json file can have more than one action (array of actions).
> e. There can be multiple patch json files in the directory and are applied in 
> the sort order of the filename. eg:
> 001-hive_column_add_position.json
> 002-hive_column_add_anotherattribute.json
> 
> 
> Diffs
> -
> 
>   common/src/main/java/org/apache/atlas/repository/Constants.java d7f9c89 
>   
> repository/src/main/java/org/apache/atlas/repository/typestore/GraphBackedTypeStore.java
>  a94d157 
>   repository/src/main/java/org/apache/atlas/services/AddAttributePatch.java 
> PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/services/AtlasTypePatch.java 
> PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java
>  6a937f4 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/AbstractDataType.java
>  fad091d 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/ClassType.java 
> c56987a 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/EnumType.java 
> bdd0a13 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/EnumTypeDefinition.java
>  6340615 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/HierarchicalType.java
>  7224699 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/HierarchicalTypeDefinition.java
>  9a299f0 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/IDataType.java 
> 85ddee7 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/StructType.java 
> 6f40c1d 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/StructTypeDefinition.java
>  f1ce1b7 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/TraitType.java 
> f23bf5b 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/TypeSystem.java 
> 70ba89b 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/utils/TypesUtil.java
>  ef8448d 
>   
> typesystem/src/main/scala/org/apache/atlas/typesystem/json/TypesSerialization.scala
>  5618938 
> 
> Diff: https://reviews.apache.org/r/51939/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sarath Kumar Subramanian
> 
>

Re: Review Request 51939: Framework to apply updates to types in the type-system

2016-09-19 Thread Suma Shivaprasad 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51939/#review149552
---




repository/src/main/java/org/apache/atlas/repository/typestore/GraphBackedTypeStore.java
 (line 351)


Log type version change at info level


- Suma Shivaprasad


On Sept. 19, 2016, 6:10 p.m., Sarath Kumar Subramanian wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51939/
> ---
> 
> (Updated Sept. 19, 2016, 6:10 p.m.)
> 
> 
> Review request for atlas, Madhan Neethiraj, Shwetha GS, and Suma Shivaprasad.
> 
> 
> Bugs: ATLAS-1174
> https://issues.apache.org/jira/browse/ATLAS-1174
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> 1. Introduce "version" attribute to all types in the type-system, this helps 
> to track changes made to the default types (hive, sqoop, falcon and storm 
> types) and user created types. If version is not mentioned during creation of 
> a type, default version "1.0" is assigned (optional attribute).
> 2. Using the version attributed for types, introduce a patch framework for 
> type system. This framework applies patches to a type using its version 
> number and can be used during upgrade - add new attributes to an existing 
> types and it will be run during atlas startup.
> The sequence of steps:
> a. During atlas startup, check $ATLAS_HOME/models/patches directory for any 
> available patch files (json files). If there any patch files handle them.
> b. Sample patch json file looks like:
> {
> "patches": [
> { 
> "action": "ADD_ATTRIBUTE",
> "typeName": "hive_column",
> "applyToVersion": "1.0",
> "updateToVersion": "2.0",
> "actionParams": [
> { "name": "position", "dataTypeName": "int", "multiplicity": "optional", 
> "isComposite": false, "isUnique": false, "isIndexable": false, 
> "reverseAttributeName": null }
> ]
> } ]
> }
> c. The framework updates the type in "typeName" for the matching version 
> number and after applying the patch, update the version to the one mentioned 
> in "updateToVersion"
> d. The json file can have more than one action (array of actions).
> e. There can be multiple patch json files in the directory and are applied in 
> the sort order of the filename. eg:
> 001-hive_column_add_position.json
> 002-hive_column_add_anotherattribute.json
> 
> 
> Diffs
> -
> 
>   common/src/main/java/org/apache/atlas/repository/Constants.java d7f9c89 
>   
> repository/src/main/java/org/apache/atlas/repository/typestore/GraphBackedTypeStore.java
>  a94d157 
>   repository/src/main/java/org/apache/atlas/services/AddAttributePatch.java 
> PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/services/AtlasTypePatch.java 
> PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java
>  6a937f4 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/AbstractDataType.java
>  fad091d 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/ClassType.java 
> c56987a 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/EnumType.java 
> bdd0a13 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/EnumTypeDefinition.java
>  6340615 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/HierarchicalType.java
>  7224699 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/HierarchicalTypeDefinition.java
>  9a299f0 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/IDataType.java 
> 85ddee7 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/StructType.java 
> 6f40c1d 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/StructTypeDefinition.java
>  f1ce1b7 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/TraitType.java 
> f23bf5b 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/TypeSystem.java 
> 70ba89b 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/utils/TypesUtil.java
>  ef8448d 
>   
> typesystem/src/main/scala/org/apache/atlas/typesystem/json/TypesSerialization.scala
>  5618938 
> 
> Diff: https://reviews.apache.org/r/51939/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sarath Kumar Subramanian
> 
>

Re: Review Request 51896: ATLAS-1171: structured, high-level APIs

2016-09-19 Thread Madhan Neethiraj 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51896/
---

(Updated Sept. 19, 2016, 6:51 p.m.)


Review request for atlas.


Changes
---

second draft version of the API. Classes under org.apache.atlas.model package 
are of the prime target for this review, as these classes capture the structure 
of the data being exchanged in the APIs. After reviewing this package, please 
look at the classes in org.apache.atlas.typesystem and the unit tests as well.

I will shortly update the review with details of the REST APIs.


Bugs: ATLAS-1171
https://issues.apache.org/jira/browse/ATLAS-1171


Repository: atlas


Description
---

first-cut API for review


Diffs (updated)
-

  common/pom.xml e3b6465 
  common/src/main/java/org/apache/atlas/api/AtlasEntities.java PRE-CREATION 
  common/src/main/java/org/apache/atlas/api/AtlasTypes.java PRE-CREATION 
  common/src/main/java/org/apache/atlas/api/SearchFilter.java PRE-CREATION 
  common/src/main/java/org/apache/atlas/model/instances/AtlasEntity.java 
PRE-CREATION 
  common/src/main/java/org/apache/atlas/model/instances/AtlasObjectId.java 
PRE-CREATION 
  common/src/main/java/org/apache/atlas/model/instances/AtlasProcess.java 
PRE-CREATION 
  common/src/main/java/org/apache/atlas/model/instances/AtlasStruct.java 
PRE-CREATION 
  common/src/main/java/org/apache/atlas/model/instances/AtlasTrait.java 
PRE-CREATION 
  common/src/main/java/org/apache/atlas/model/types/AtlasBaseTypeDef.java 
PRE-CREATION 
  common/src/main/java/org/apache/atlas/model/types/AtlasEntityDef.java 
PRE-CREATION 
  common/src/main/java/org/apache/atlas/model/types/AtlasEnumDef.java 
PRE-CREATION 
  common/src/main/java/org/apache/atlas/model/types/AtlasStructDef.java 
PRE-CREATION 
  common/src/main/java/org/apache/atlas/model/types/AtlasTraitDef.java 
PRE-CREATION 
  common/src/main/java/org/apache/atlas/model/types/AtlasTypeId.java 
PRE-CREATION 
  common/src/main/java/org/apache/atlas/typesystem/AtlasArrayType.java 
PRE-CREATION 
  common/src/main/java/org/apache/atlas/typesystem/AtlasBuiltInTypes.java 
PRE-CREATION 
  common/src/main/java/org/apache/atlas/typesystem/AtlasDataType.java 
PRE-CREATION 
  common/src/main/java/org/apache/atlas/typesystem/AtlasEntityType.java 
PRE-CREATION 
  common/src/main/java/org/apache/atlas/typesystem/AtlasEnumType.java 
PRE-CREATION 
  common/src/main/java/org/apache/atlas/typesystem/AtlasMapType.java 
PRE-CREATION 
  common/src/main/java/org/apache/atlas/typesystem/AtlasStructType.java 
PRE-CREATION 
  common/src/main/java/org/apache/atlas/typesystem/AtlasTraitType.java 
PRE-CREATION 
  common/src/main/java/org/apache/atlas/typesystem/AtlasTypeRegistry.java 
PRE-CREATION 
  common/src/test/java/org/apache/atlas/model/ModelTestUtil.java PRE-CREATION 
  common/src/test/java/org/apache/atlas/model/instances/TestAtlasEntity.java 
PRE-CREATION 
  common/src/test/java/org/apache/atlas/model/types/TestAtlasEntityDef.java 
PRE-CREATION 
  common/src/test/java/org/apache/atlas/model/types/TestAtlasEnumDef.java 
PRE-CREATION 
  common/src/test/java/org/apache/atlas/model/types/TestAtlasStructDef.java 
PRE-CREATION 
  common/src/test/java/org/apache/atlas/model/types/TestAtlasTraitDef.java 
PRE-CREATION 
  pom.xml ac5b042 

Diff: https://reviews.apache.org/r/51896/diff/


Testing
---


Thanks,

Madhan Neethiraj

Re: Review Request 51939: Framework to apply updates to types in the type-system

2016-09-19 Thread Sarath Kumar Subramanian 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51939/
---

(Updated Sept. 19, 2016, 6:10 p.m.)


Review request for atlas, Madhan Neethiraj, Shwetha GS, and Suma Shivaprasad.


Bugs: ATLAS-1174
https://issues.apache.org/jira/browse/ATLAS-1174


Repository: atlas


Description
---

1. Introduce "version" attribute to all types in the type-system, this helps to 
track changes made to the default types (hive, sqoop, falcon and storm types) 
and user created types. If version is not mentioned during creation of a type, 
default version "1.0" is assigned (optional attribute).
2. Using the version attributed for types, introduce a patch framework for type 
system. This framework applies patches to a type using its version number and 
can be used during upgrade - add new attributes to an existing types and it 
will be run during atlas startup.
The sequence of steps:
a. During atlas startup, check $ATLAS_HOME/models/patches directory for any 
available patch files (json files). If there any patch files handle them.
b. Sample patch json file looks like:
{
"patches": [
{ 
"action": "ADD_ATTRIBUTE",
"typeName": "hive_column",
"applyToVersion": "1.0",
"updateToVersion": "2.0",
"actionParams": [
{ "name": "position", "dataTypeName": "int", "multiplicity": "optional", 
"isComposite": false, "isUnique": false, "isIndexable": false, 
"reverseAttributeName": null }
]
} ]
}
c. The framework updates the type in "typeName" for the matching version number 
and after applying the patch, update the version to the one mentioned in 
"updateToVersion"
d. The json file can have more than one action (array of actions).
e. There can be multiple patch json files in the directory and are applied in 
the sort order of the filename. eg:
001-hive_column_add_position.json
002-hive_column_add_anotherattribute.json


Diffs (updated)
-

  common/src/main/java/org/apache/atlas/repository/Constants.java d7f9c89 
  
repository/src/main/java/org/apache/atlas/repository/typestore/GraphBackedTypeStore.java
 a94d157 
  repository/src/main/java/org/apache/atlas/services/AddAttributePatch.java 
PRE-CREATION 
  repository/src/main/java/org/apache/atlas/services/AtlasTypePatch.java 
PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java 
6a937f4 
  
typesystem/src/main/java/org/apache/atlas/typesystem/types/AbstractDataType.java
 fad091d 
  typesystem/src/main/java/org/apache/atlas/typesystem/types/ClassType.java 
c56987a 
  typesystem/src/main/java/org/apache/atlas/typesystem/types/EnumType.java 
bdd0a13 
  
typesystem/src/main/java/org/apache/atlas/typesystem/types/EnumTypeDefinition.java
 6340615 
  
typesystem/src/main/java/org/apache/atlas/typesystem/types/HierarchicalType.java
 7224699 
  
typesystem/src/main/java/org/apache/atlas/typesystem/types/HierarchicalTypeDefinition.java
 9a299f0 
  typesystem/src/main/java/org/apache/atlas/typesystem/types/IDataType.java 
85ddee7 
  typesystem/src/main/java/org/apache/atlas/typesystem/types/StructType.java 
6f40c1d 
  
typesystem/src/main/java/org/apache/atlas/typesystem/types/StructTypeDefinition.java
 f1ce1b7 
  typesystem/src/main/java/org/apache/atlas/typesystem/types/TraitType.java 
f23bf5b 
  typesystem/src/main/java/org/apache/atlas/typesystem/types/TypeSystem.java 
70ba89b 
  
typesystem/src/main/java/org/apache/atlas/typesystem/types/utils/TypesUtil.java 
ef8448d 
  
typesystem/src/main/scala/org/apache/atlas/typesystem/json/TypesSerialization.scala
 5618938 

Diff: https://reviews.apache.org/r/51939/diff/


Testing
---


Thanks,

Sarath Kumar Subramanian

[jira] [Resolved] (ATLAS-1180) Bug jira

2016-09-19 Thread Ayub Khan (JIRA) 

 [ 
https://issues.apache.org/jira/browse/ATLAS-1180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ayub Khan resolved ATLAS-1180.
--
Resolution: Invalid

The issue does not seem to be a part of Atlas project. Please assign against 
right project.

> Bug jira
> 
>
> Key: ATLAS-1180
> URL: https://issues.apache.org/jira/browse/ATLAS-1180
> Project: Atlas
>  Issue Type: Bug
>Reporter: Saritha
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (ATLAS-1180) Bug jira

2016-09-19 Thread Saritha (JIRA) 

[ 
https://issues.apache.org/jira/browse/ATLAS-1180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15504154#comment-15504154
 ] 

Saritha commented on ATLAS-1180:


cxf-core 3.1.7 is not validating the xml with xsd at appropriate location 

Exception says:  Could not generate the XML stream caused by: 
org.xml.sax.SAXParseException: cvc-elt.1: Cannot find the declaration of 
element 'ATT_CheckRS'..

Sample code is as below 

import org.apache.camel.Exchange; 
import org.apache.camel.Message; 
import org.apache.camel.Processor; 
  
public class TestProcess implements Processor { 
  
   public void process(Exchange exchange) throws Exception { 
  Message message =  exchange.getOut(); 
  
  message.setBody( bindMessage()); 
  exchange.setOut(message); 
  //java.lang.System.out.println("MIP of Http Endpoint is " + 
message); 
  java.lang.System.out.println("NEW MIP of Http Endpoint is " + 
exchange.getPattern()); 
  
   } 
  
  
   private String bindMessage(){ 
  return"http://www.w3.org/2001/XMLSchema-instance\; 
xmlns=\"http://www.travel.org/ABC/2003/05\;>" 
+"" 
+"" 
+"" 
+"" 
+"" 
+"" 
+"" 
+"" 
+""; 
  
  
  
   } 
} 
  


> Bug jira
> 
>
> Key: ATLAS-1180
> URL: https://issues.apache.org/jira/browse/ATLAS-1180
> Project: Atlas
>  Issue Type: Bug
>Reporter: Saritha
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (ATLAS-1180) Bug jira

2016-09-19 Thread Saritha (JIRA) 
Saritha created ATLAS-1180:
--

 Summary: Bug jira
 Key: ATLAS-1180
 URL: https://issues.apache.org/jira/browse/ATLAS-1180
 Project: Atlas
  Issue Type: Bug
Reporter: Saritha






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 51939: Framework to apply updates to types in the type-system

2016-09-19 Thread Sarath Kumar Subramanian 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51939/
---

(Updated Sept. 19, 2016, 4:56 p.m.)


Review request for atlas, Madhan Neethiraj, Shwetha GS, and Suma Shivaprasad.


Changes
---

new review request based on comments


Bugs: ATLAS-1174
https://issues.apache.org/jira/browse/ATLAS-1174


Repository: atlas


Description
---

1. Introduce "version" attribute to all types in the type-system, this helps to 
track changes made to the default types (hive, sqoop, falcon and storm types) 
and user created types. If version is not mentioned during creation of a type, 
default version "1.0" is assigned (optional attribute).
2. Using the version attributed for types, introduce a patch framework for type 
system. This framework applies patches to a type using its version number and 
can be used during upgrade - add new attributes to an existing types and it 
will be run during atlas startup.
The sequence of steps:
a. During atlas startup, check $ATLAS_HOME/models/patches directory for any 
available patch files (json files). If there any patch files handle them.
b. Sample patch json file looks like:
{
"patches": [
{ 
"action": "ADD_ATTRIBUTE",
"typeName": "hive_column",
"applyToVersion": "1.0",
"updateToVersion": "2.0",
"actionParams": [
{ "name": "position", "dataTypeName": "int", "multiplicity": "optional", 
"isComposite": false, "isUnique": false, "isIndexable": false, 
"reverseAttributeName": null }
]
} ]
}
c. The framework updates the type in "typeName" for the matching version number 
and after applying the patch, update the version to the one mentioned in 
"updateToVersion"
d. The json file can have more than one action (array of actions).
e. There can be multiple patch json files in the directory and are applied in 
the sort order of the filename. eg:
001-hive_column_add_position.json
002-hive_column_add_anotherattribute.json


Diffs (updated)
-

  common/src/main/java/org/apache/atlas/repository/Constants.java d7f9c89 
  
repository/src/main/java/org/apache/atlas/repository/typestore/GraphBackedTypeStore.java
 a94d157 
  repository/src/main/java/org/apache/atlas/services/AddAttributePatch.java 
PRE-CREATION 
  repository/src/main/java/org/apache/atlas/services/AtlasTypePatch.java 
PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java 
6a937f4 
  
typesystem/src/main/java/org/apache/atlas/typesystem/types/AbstractDataType.java
 fad091d 
  typesystem/src/main/java/org/apache/atlas/typesystem/types/ClassType.java 
c56987a 
  typesystem/src/main/java/org/apache/atlas/typesystem/types/EnumType.java 
bdd0a13 
  
typesystem/src/main/java/org/apache/atlas/typesystem/types/EnumTypeDefinition.java
 6340615 
  
typesystem/src/main/java/org/apache/atlas/typesystem/types/HierarchicalType.java
 7224699 
  
typesystem/src/main/java/org/apache/atlas/typesystem/types/HierarchicalTypeDefinition.java
 9a299f0 
  typesystem/src/main/java/org/apache/atlas/typesystem/types/IDataType.java 
85ddee7 
  typesystem/src/main/java/org/apache/atlas/typesystem/types/StructType.java 
6f40c1d 
  
typesystem/src/main/java/org/apache/atlas/typesystem/types/StructTypeDefinition.java
 f1ce1b7 
  typesystem/src/main/java/org/apache/atlas/typesystem/types/TraitType.java 
f23bf5b 
  typesystem/src/main/java/org/apache/atlas/typesystem/types/TypeSystem.java 
70ba89b 
  
typesystem/src/main/java/org/apache/atlas/typesystem/types/utils/TypesUtil.java 
ef8448d 
  
typesystem/src/main/scala/org/apache/atlas/typesystem/json/TypesSerialization.scala
 5618938 

Diff: https://reviews.apache.org/r/51939/diff/


Testing
---


Thanks,

Sarath Kumar Subramanian

Re: Review Request 51939: Framework to apply updates to types in the type-system

2016-09-19 Thread Sarath Kumar Subramanian 


> On Sept. 16, 2016, 1:19 a.m., Madhan Neethiraj wrote:
> > repository/src/main/java/org/apache/atlas/services/AtlasTypeUpdate.java, 
> > line 157
> > 
> >
> > Is using JSON string the only way to update types in the system? There 
> > is got to be a better way than use of opaque strings..

Currently creating a new type or updating an existing type in type system 
accepts only JSON String. This is by design.

JSONObject createType(String typeDefinition)
JSONObject createOrUpdateTypes(String typeDefinition, boolean isUpdate)


- Sarath Kumar


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51939/#review149157
---


On Sept. 15, 2016, 11:03 p.m., Sarath Kumar Subramanian wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51939/
> ---
> 
> (Updated Sept. 15, 2016, 11:03 p.m.)
> 
> 
> Review request for atlas, Madhan Neethiraj, Shwetha GS, and Suma Shivaprasad.
> 
> 
> Bugs: ATLAS-1174
> https://issues.apache.org/jira/browse/ATLAS-1174
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> 1. Introduce "version" attribute to all types in the type-system, this helps 
> to track changes made to the default types (hive, sqoop, falcon and storm 
> types) and user created types. If version is not mentioned during creation of 
> a type, default version "1.0" is assigned (optional attribute).
> 2. Using the version attributed for types, introduce a patch framework for 
> type system. This framework applies patches to a type using its version 
> number and can be used during upgrade - add new attributes to an existing 
> types and it will be run during atlas startup.
> The sequence of steps:
> a. During atlas startup, check $ATLAS_HOME/models/patches directory for any 
> available patch files (json files). If there any patch files handle them.
> b. Sample patch json file looks like:
> {
> "patches": [
> { 
> "action": "ADD_ATTRIBUTE",
> "typeName": "hive_column",
> "applyToVersion": "1.0",
> "updateToVersion": "2.0",
> "actionParams": [
> { "name": "position", "dataTypeName": "int", "multiplicity": "optional", 
> "isComposite": false, "isUnique": false, "isIndexable": false, 
> "reverseAttributeName": null }
> ]
> } ]
> }
> c. The framework updates the type in "typeName" for the matching version 
> number and after applying the patch, update the version to the one mentioned 
> in "updateToVersion"
> d. The json file can have more than one action (array of actions).
> e. There can be multiple patch json files in the directory and are applied in 
> the sort order of the filename. eg:
> 001-hive_column_add_position.json
> 002-hive_column_add_anotherattribute.json
> 
> 
> Diffs
> -
> 
>   common/src/main/java/org/apache/atlas/repository/Constants.java d7f9c89 
>   
> repository/src/main/java/org/apache/atlas/repository/typestore/GraphBackedTypeStore.java
>  a94d157 
>   repository/src/main/java/org/apache/atlas/services/AtlasTypeUpdate.java 
> PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java
>  6a937f4 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/AbstractDataType.java
>  fad091d 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/ClassType.java 
> c56987a 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/EnumType.java 
> bdd0a13 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/EnumTypeDefinition.java
>  6340615 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/HierarchicalType.java
>  7224699 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/HierarchicalTypeDefinition.java
>  9a299f0 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/IDataType.java 
> 85ddee7 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/StructType.java 
> 6f40c1d 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/StructTypeDefinition.java
>  f1ce1b7 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/TraitType.java 
> f23bf5b 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/TypeSystem.java 
> 70ba89b 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/utils/TypesUtil.java
>  ef8448d 
>   
> typesystem/src/main/scala/org/apache/atlas/typesystem/json/TypesSerialization.scala
>  5618938 
> 
> Diff: https://reviews.apache.org/r/51939/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sarath Kumar Subramanian
> 
>

[jira] [Commented] (ATLAS-1178) APIs in DefaultMetadataService should return POJOs

2016-09-19 Thread David Radley (JIRA) 

[ 
https://issues.apache.org/jira/browse/ATLAS-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15503547#comment-15503547
 ] 

David Radley commented on ATLAS-1178:
-

In looking at this Jira, I am looking to understand what this class with 
respect to others. 
  
I notice that this Interface and class has the name service in it, which 
implies it is dealing with json objects. It is in the ServerAPI project.
I suggest you consider renaming this class / Interface and move it out of the 
API project. I was thinking that a better name might be TypeSystemManager. 

Is the proposal here to remove all json awareness from the MetadataService 
interface and its default implementation and move this code into TypesResource 
and the like. I think this makes sense to avoid unnecessary cases of creaing 
json objects. Is the idea is that no json processing is done downstream of the 
Metadata interface? I notice AtlasClient.EntityResult is returned from some 
methods - are you proposing changing this to a POJO object and moving out any 
json creation as well?  

> APIs in DefaultMetadataService should return POJOs
> --
>
> Key: ATLAS-1178
> URL: https://issues.apache.org/jira/browse/ATLAS-1178
> Project: Atlas
>  Issue Type: Bug
>Reporter: Shwetha G S
>
> DefaultMetadataService APIs accept/return json strings and the rest APIs end 
> up serialising and deserialising everytime. DefaultMetadataService APIs are 
> internal APIs and should accept/return POJOs



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (ATLAS-247) hive_column leve lineage

2016-09-19 Thread Vimal Sharma (JIRA) 

 [ 
https://issues.apache.org/jira/browse/ATLAS-247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vimal Sharma updated ATLAS-247:
---
Attachment: ATLAS-247-v3.patch

> hive_column leve lineage
> 
>
> Key: ATLAS-247
> URL: https://issues.apache.org/jira/browse/ATLAS-247
> Project: Atlas
>  Issue Type: New Feature
>Affects Versions: 0.5-incubating
>Reporter: Herman Yu
>Assignee: Harish Butani
> Attachments: ATLAS-247-v3.patch, ATLAS-247.2.patch, ATLAS-247.patch
>
>
> hive_column is not inherited from DataSet, thus can't be using hive_process 
> to track column level lineages
> Is there specific reason that hive_column is not inheriting from Data Set? 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (ATLAS-247) hive_column leve lineage

2016-09-19 Thread Vimal Sharma (JIRA) 

 [ 
https://issues.apache.org/jira/browse/ATLAS-247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vimal Sharma updated ATLAS-247:
---
Attachment: (was: ATLAS-247.patch)

> hive_column leve lineage
> 
>
> Key: ATLAS-247
> URL: https://issues.apache.org/jira/browse/ATLAS-247
> Project: Atlas
>  Issue Type: New Feature
>Affects Versions: 0.5-incubating
>Reporter: Herman Yu
>Assignee: Harish Butani
> Attachments: ATLAS-247.2.patch, ATLAS-247.patch
>
>
> hive_column is not inherited from DataSet, thus can't be using hive_process 
> to track column level lineages
> Is there specific reason that hive_column is not inheriting from Data Set? 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Build failed in Jenkins: apache-atlas-nightly #428

2016-09-19 Thread Apache Jenkins Server 
See 

Changes:

[sshivalingamurthy] ATLAS-712 Support getTrait() API (svimal2106 via shwethags)

[sshivalingamurthy] ATLAS-1142 Lineage UI Improvement (kevalbhatt via shwethags)

[sshivalingamurthy] ATLAS-1098 Atlas allows creation of tag with name isa which 
causes

[sshivalingamurthy] ATLAS-1155 Errors in Eclipse when I bring in the latest 
code (davidrad

--
[...truncated 9151 lines...]
127.0.0.1 - - [19/Sep/2016:08:43:30 +] "GET 
/api/atlas/lineage/hive/table/default.tablej9oso0bwvz@primary/inputs/graph 
HTTP/1.1" 200 - "-" "Java/1.7.0_80"
127.0.0.1 - - [19/Sep/2016:08:43:35 +] "GET 
/api/atlas/entities?type=hive_process=qualifiedName=QUERY:default.tablesldlrk5sat@primary:1474274613000-%3E:PATH_WRITE
 HTTP/1.1" 404 - "-" "Java/1.7.0_80"
127.0.0.1 - - [19/Sep/2016:08:43:40 +] "GET 
/api/atlas/entities?type=hive_process=qualifiedName=QUERY:default.tablesldlrk5sat@primary:1474274613000-%3E:PATH_WRITE
 HTTP/1.1" 200 - "-" "Java/1.7.0_80"
127.0.0.1 - - [19/Sep/2016:08:43:40 +] "GET 
/api/atlas/entities?type=hive_process=qualifiedName=QUERY:default.tablesldlrk5sat@primary:1474274613000-%3E:PATH_WRITE
 HTTP/1.1" 200 - "-" "Java/1.7.0_80"
127.0.0.1 - - [19/Sep/2016:08:43:40 +] "GET 
/api/atlas/entities/13997f06-d995-47b9-8d7c-05fd36f4ad37 HTTP/1.1" 200 - "-" 
"Java/1.7.0_80"
127.0.0.1 - - [19/Sep/2016:08:43:40 +] "GET 
/api/atlas/entities/5f635669-955b-4d9c-9282-b3815b82e075 HTTP/1.1" 200 - "-" 
"Java/1.7.0_80"
127.0.0.1 - - [19/Sep/2016:08:43:40 +] "GET 
/api/atlas/entities/bd3fbc21-d70e-407b-8218-d7e3bc665f17 HTTP/1.1" 200 - "-" 
"Java/1.7.0_80"
127.0.0.1 - - [19/Sep/2016:08:43:40 +] "GET 
/api/atlas/entities?type=hdfs_path=qualifiedName=p
 HTTP/1.1" 200 - "-" "Java/1.7.0_80"
127.0.0.1 - - [19/Sep/2016:08:43:40 +] "GET 
/api/atlas/entities?type=hdfs_path=qualifiedName=p
 HTTP/1.1" 200 - "-" "Java/1.7.0_80"
127.0.0.1 - - [19/Sep/2016:08:43:40 +] "GET 
/api/atlas/entities/bd3fbc21-d70e-407b-8218-d7e3bc665f17 HTTP/1.1" 200 - "-" 
"Java/1.7.0_80"
127.0.0.1 - - [19/Sep/2016:08:43:40 +] "GET 
/api/atlas/entities?type=hive_table=qualifiedName=default.tablesldlrk5sat@primary
 HTTP/1.1" 200 - "-" "Java/1.7.0_80"
127.0.0.1 - - [19/Sep/2016:08:43:40 +] "GET 
/api/atlas/entities?type=hive_table=qualifiedName=default.tablesldlrk5sat@primary
 HTTP/1.1" 200 - "-" "Java/1.7.0_80"
127.0.0.1 - - [19/Sep/2016:08:43:40 +] "GET 
/api/atlas/entities/5f635669-955b-4d9c-9282-b3815b82e075 HTTP/1.1" 200 - "-" 
"Java/1.7.0_80"
127.0.0.1 - - [19/Sep/2016:08:43:43 +] "GET 
/api/atlas/entities?type=hive_table=qualifiedName=default.tablesldlrk5sat@primary
 HTTP/1.1" 200 - "-" "Java/1.7.0_80"
127.0.0.1 - - [19/Sep/2016:08:43:43 +] "GET 
/api/atlas/entities?type=hive_table=qualifiedName=default.tablesldlrk5sat@primary
 HTTP/1.1" 200 - "-" "Java/1.7.0_80"
127.0.0.1 - - [19/Sep/2016:08:43:43 +] "GET 
/api/atlas/entities?type=hive_process=qualifiedName=QUERY:default.tablesldlrk5sat@primary:1474274613000-%3E:PATH_WRITE
 HTTP/1.1" 200 - "-" "Java/1.7.0_80"
127.0.0.1 - - [19/Sep/2016:08:43:43 +] "GET 
/api/atlas/entities?type=hive_process=qualifiedName=QUERY:default.tablesldlrk5sat@primary:1474274613000-%3E:PATH_WRITE
 HTTP/1.1" 200 - "-" "Java/1.7.0_80"
127.0.0.1 - - [19/Sep/2016:08:43:43 +] "GET 
/api/atlas/entities/13997f06-d995-47b9-8d7c-05fd36f4ad37 HTTP/1.1" 200 - "-" 
"Java/1.7.0_80"
127.0.0.1 - - [19/Sep/2016:08:43:43 +] "GET 
/api/atlas/entities/5f635669-955b-4d9c-9282-b3815b82e075 HTTP/1.1" 200 - "-" 
"Java/1.7.0_80"
127.0.0.1 - - [19/Sep/2016:08:43:43 +] "GET 
/api/atlas/entities/bd3fbc21-d70e-407b-8218-d7e3bc665f17 HTTP/1.1" 200 - "-" 
"Java/1.7.0_80"
127.0.0.1 - - [19/Sep/2016:08:43:43 +] "GET 
/api/atlas/entities?type=hdfs_path=qualifiedName=p
 HTTP/1.1" 200 - "-" "Java/1.7.0_80"
127.0.0.1 - - [19/Sep/2016:08:43:43 +] "GET 
/api/atlas/entities?type=hdfs_path=qualifiedName=p
 HTTP/1.1" 200 - "-" "Java/1.7.0_80"
127.0.0.1 - - [19/Sep/2016:08:43:43 +] "GET 
/api/atlas/entities/bd3fbc21-d70e-407b-8218-d7e3bc665f17 HTTP/1.1" 200 - "-" 
"Java/1.7.0_80"
127.0.0.1 - - [19/Sep/2016:08:43:45 +] "GET 
/api/atlas/entities?type=hive_table=qualifiedName=default.tablesldlrk5sat@primary
 HTTP/1.1" 200 - "-" "Java/1.7.0_80"
127.0.0.1 - - [19/Sep/2016:08:43:45 +] "GET 
/api/atlas/entities?type=hive_table=qualifiedName=default.tablesldlrk5sat@primary
 HTTP/1.1" 200 - "-" "Java/1.7.0_80"
127.0.0.1 - -

[jira] [Comment Edited] (ATLAS-1142) Lineage UI Improvement.

2016-09-19 Thread Shwetha G S (JIRA) 

[ 
https://issues.apache.org/jira/browse/ATLAS-1142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15502454#comment-15502454
 ] 

Shwetha G S edited comment on ATLAS-1142 at 9/19/16 6:27 AM:
-

+1, Committed to master. Thanks Keval


was (Author: shwethags):
Committed to master. Thanks Keval

> Lineage UI Improvement.
> ---
>
> Key: ATLAS-1142
> URL: https://issues.apache.org/jira/browse/ATLAS-1142
> Project: Atlas
>  Issue Type: Improvement
>Reporter: Nixon Rodrigues
>Assignee: Keval Bhatt
>Priority: Minor
> Fix For: 0.8-incubating
>
> Attachments: ATLAS-1142.1.patch, ATLAS-1142.patch, Screen Shot 
> 2016-09-15 at 2.16.57 PM.png
>
>
> Following improvement to be done in Lineage UI & entity detail page
> * Lineage UI should be resizable.
> * All the nodes to be shown in format *name (typename)*



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (ATLAS-712) Support getTrait() API

2016-09-19 Thread Shwetha G S (JIRA) 

 [ 
https://issues.apache.org/jira/browse/ATLAS-712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shwetha G S resolved ATLAS-712.
---
   Resolution: Fixed
Fix Version/s: 0.8-incubating

Committed to master. Thanks Vimal for the patch, thanks Suma for the review

> Support getTrait() API
> --
>
> Key: ATLAS-712
> URL: https://issues.apache.org/jira/browse/ATLAS-712
> Project: Atlas
>  Issue Type: Improvement
>Reporter: Suma Shivaprasad
>Assignee: Vimal Sharma
> Fix For: 0.8-incubating
>
> Attachments: ATLAS-712-v2.patch, ATLAS-712-v3.patch, 
> ATLAS-712-v4.patch, ATLAS-712.patch
>
>
> Given entity id and trait name, support rest API that returns the trait 
> instance for the entity. Currently, the other way of getting it is 
> getEntity() which returns full entity with all trait instances 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (ATLAS-712) Support getTrait() API

2016-09-19 Thread Shwetha G S (JIRA) 

 [ 
https://issues.apache.org/jira/browse/ATLAS-712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shwetha G S updated ATLAS-712:
--
Attachment: ATLAS-712-v4.patch

Attached patch with the minor fix for commit

> Support getTrait() API
> --
>
> Key: ATLAS-712
> URL: https://issues.apache.org/jira/browse/ATLAS-712
> Project: Atlas
>  Issue Type: Improvement
>Reporter: Suma Shivaprasad
>Assignee: Vimal Sharma
> Fix For: 0.8-incubating
>
> Attachments: ATLAS-712-v2.patch, ATLAS-712-v3.patch, 
> ATLAS-712-v4.patch, ATLAS-712.patch
>
>
> Given entity id and trait name, support rest API that returns the trait 
> instance for the entity. Currently, the other way of getting it is 
> getEntity() which returns full entity with all trait instances 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (ATLAS-1179) Hive hook test failure

2016-09-19 Thread Shwetha G S (JIRA) 

 [ 
https://issues.apache.org/jira/browse/ATLAS-1179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shwetha G S reassigned ATLAS-1179:
--

Assignee: Shwetha G S

> Hive hook test failure
> --
>
> Key: ATLAS-1179
> URL: https://issues.apache.org/jira/browse/ATLAS-1179
> Project: Atlas
>  Issue Type: Bug
>Reporter: Shwetha G S
>Assignee: Shwetha G S
>Priority: Critical
>
> The build fails:
> {noformat}
>   HiveHookIT.testAlterTableDropColumn:996 expected:<1> but was:<2>
>   
> HiveHookIT.testCreateExternalTable:218->HiveITBase.assertTableIsRegistered:154->HiveITBase.assertEntityIsRegistered:160->HiveITBase.waitFor:204
>  Assertions failed. Failing after waiting for timeout 8 msecs
>   
> HiveHookIT.testInsertIntoTempTable:690->validateProcess:518->validateProcess:498->assertProcessIsRegistered:1662->HiveITBase.assertEntityIsRegistered:160->HiveITBase.waitFor:204
>  Assertions failed. Failing after waiting for timeout 8 msecs
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (ATLAS-1179) Hive hook test failure

2016-09-19 Thread Shwetha G S (JIRA) 
Shwetha G S created ATLAS-1179:
--

 Summary: Hive hook test failure
 Key: ATLAS-1179
 URL: https://issues.apache.org/jira/browse/ATLAS-1179
 Project: Atlas
  Issue Type: Bug
Reporter: Shwetha G S


The build fails:
{noformat}
  HiveHookIT.testAlterTableDropColumn:996 expected:<1> but was:<2>
  
HiveHookIT.testCreateExternalTable:218->HiveITBase.assertTableIsRegistered:154->HiveITBase.assertEntityIsRegistered:160->HiveITBase.waitFor:204
 Assertions failed. Failing after waiting for timeout 8 msecs
  
HiveHookIT.testInsertIntoTempTable:690->validateProcess:518->validateProcess:498->assertProcessIsRegistered:1662->HiveITBase.assertEntityIsRegistered:160->HiveITBase.waitFor:204
 Assertions failed. Failing after waiting for timeout 8 msecs
{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)