Re: Review Request 51939: Framework to apply updates to types in the type-system

[Sarath Kumar Subramanian]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51939/
---

(Updated Sept. 27, 2016, 4:08 p.m.)


Review request for atlas, Madhan Neethiraj, Shwetha GS, and Suma Shivaprasad.


Changes
---

throw exception when a patch fails - json deserialization, incorrect type or IO 
failures, similar to model behavior.


Bugs: ATLAS-1174
https://issues.apache.org/jira/browse/ATLAS-1174


Repository: atlas


Description
---

1. Introduce "version" attribute to all types in the type-system, this helps to 
track changes made to the default types (hive, sqoop, falcon and storm types) 
and user created types. If version is not mentioned during creation of a type, 
default version "1.0" is assigned (optional attribute).
2. Using the version attributed for types, introduce a patch framework for type 
system. This framework applies patches to a type using its version number and 
can be used during upgrade - add new attributes to an existing types and it 
will be run during atlas startup.
The sequence of steps:
a. During atlas startup, check $ATLAS_HOME/models/patches directory for any 
available patch files (json files). If there any patch files handle them.
b. Sample patch json file looks like:
{
"patches": [
{ 
"action": "ADD_ATTRIBUTE",
"typeName": "hive_column",
"applyToVersion": "1.0",
"updateToVersion": "2.0",
"actionParams": [
{ "name": "position", "dataTypeName": "int", "multiplicity": "optional", 
"isComposite": false, "isUnique": false, "isIndexable": false, 
"reverseAttributeName": null }
]
} ]
}
c. The framework updates the type in "typeName" for the matching version number 
and after applying the patch, update the version to the one mentioned in 
"updateToVersion"
d. The json file can have more than one action (array of actions).
e. There can be multiple patch json files in the directory and are applied in 
the sort order of the filename. eg:
001-hive_column_add_position.json
002-hive_column_add_anotherattribute.json


Diffs (updated)
-

  common/src/main/java/org/apache/atlas/AtlasConstants.java 17ffbd7 
  common/src/main/java/org/apache/atlas/repository/Constants.java d7f9c89 
  
repository/src/main/java/org/apache/atlas/repository/typestore/GraphBackedTypeStore.java
 a94d157 
  repository/src/main/java/org/apache/atlas/services/AtlasPatchHandler.java 
PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/services/AtlasTypeAttributePatch.java 
PRE-CREATION 
  repository/src/main/java/org/apache/atlas/services/AtlasTypePatch.java 
PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java 
3550492 
  
repository/src/test/java/org/apache/atlas/service/DefaultMetadataServiceTest.java
 6782970 
  
typesystem/src/main/java/org/apache/atlas/typesystem/types/AbstractDataType.java
 fad091d 
  typesystem/src/main/java/org/apache/atlas/typesystem/types/ClassType.java 
c56987a 
  typesystem/src/main/java/org/apache/atlas/typesystem/types/EnumType.java 
bdd0a13 
  
typesystem/src/main/java/org/apache/atlas/typesystem/types/EnumTypeDefinition.java
 6340615 
  
typesystem/src/main/java/org/apache/atlas/typesystem/types/HierarchicalType.java
 7224699 
  
typesystem/src/main/java/org/apache/atlas/typesystem/types/HierarchicalTypeDefinition.java
 9a299f0 
  typesystem/src/main/java/org/apache/atlas/typesystem/types/IDataType.java 
85ddee7 
  typesystem/src/main/java/org/apache/atlas/typesystem/types/StructType.java 
6f40c1d 
  
typesystem/src/main/java/org/apache/atlas/typesystem/types/StructTypeDefinition.java
 f1ce1b7 
  typesystem/src/main/java/org/apache/atlas/typesystem/types/TraitType.java 
f23bf5b 
  typesystem/src/main/java/org/apache/atlas/typesystem/types/TypeSystem.java 
70ba89b 
  
typesystem/src/main/java/org/apache/atlas/typesystem/types/utils/TypesUtil.java 
ef8448d 
  
typesystem/src/main/scala/org/apache/atlas/typesystem/json/TypesSerialization.scala
 5618938 

Diff: https://reviews.apache.org/r/51939/diff/


Testing
---


Thanks,

Sarath Kumar Subramanian

Re: Review Request 51939: Framework to apply updates to types in the type-system

[Madhan Neethiraj]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51939/#review150510
---


Ship it!




Ship It!

- Madhan Neethiraj


On Sept. 27, 2016, 12:25 a.m., Sarath Kumar Subramanian wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51939/
> ---
> 
> (Updated Sept. 27, 2016, 12:25 a.m.)
> 
> 
> Review request for atlas, Madhan Neethiraj, Shwetha GS, and Suma Shivaprasad.
> 
> 
> Bugs: ATLAS-1174
> https://issues.apache.org/jira/browse/ATLAS-1174
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> 1. Introduce "version" attribute to all types in the type-system, this helps 
> to track changes made to the default types (hive, sqoop, falcon and storm 
> types) and user created types. If version is not mentioned during creation of 
> a type, default version "1.0" is assigned (optional attribute).
> 2. Using the version attributed for types, introduce a patch framework for 
> type system. This framework applies patches to a type using its version 
> number and can be used during upgrade - add new attributes to an existing 
> types and it will be run during atlas startup.
> The sequence of steps:
> a. During atlas startup, check $ATLAS_HOME/models/patches directory for any 
> available patch files (json files). If there any patch files handle them.
> b. Sample patch json file looks like:
> {
> "patches": [
> { 
> "action": "ADD_ATTRIBUTE",
> "typeName": "hive_column",
> "applyToVersion": "1.0",
> "updateToVersion": "2.0",
> "actionParams": [
> { "name": "position", "dataTypeName": "int", "multiplicity": "optional", 
> "isComposite": false, "isUnique": false, "isIndexable": false, 
> "reverseAttributeName": null }
> ]
> } ]
> }
> c. The framework updates the type in "typeName" for the matching version 
> number and after applying the patch, update the version to the one mentioned 
> in "updateToVersion"
> d. The json file can have more than one action (array of actions).
> e. There can be multiple patch json files in the directory and are applied in 
> the sort order of the filename. eg:
> 001-hive_column_add_position.json
> 002-hive_column_add_anotherattribute.json
> 
> 
> Diffs
> -
> 
>   common/src/main/java/org/apache/atlas/AtlasConstants.java 17ffbd7 
>   common/src/main/java/org/apache/atlas/repository/Constants.java d7f9c89 
>   
> repository/src/main/java/org/apache/atlas/repository/typestore/GraphBackedTypeStore.java
>  a94d157 
>   repository/src/main/java/org/apache/atlas/services/AtlasPatchHandler.java 
> PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/services/AtlasTypeAttributePatch.java
>  PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/services/AtlasTypePatch.java 
> PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java
>  3550492 
>   
> repository/src/test/java/org/apache/atlas/service/DefaultMetadataServiceTest.java
>  6782970 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/AbstractDataType.java
>  fad091d 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/ClassType.java 
> c56987a 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/EnumType.java 
> bdd0a13 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/EnumTypeDefinition.java
>  6340615 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/HierarchicalType.java
>  7224699 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/HierarchicalTypeDefinition.java
>  9a299f0 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/IDataType.java 
> 85ddee7 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/StructType.java 
> 6f40c1d 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/StructTypeDefinition.java
>  f1ce1b7 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/TraitType.java 
> f23bf5b 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/TypeSystem.java 
> 70ba89b 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/utils/TypesUtil.java
>  ef8448d 
>   
> typesystem/src/main/scala/org/apache/atlas/typesystem/json/TypesSerialization.scala
>  5618938 
> 
> Diff: https://reviews.apache.org/r/51939/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sarath Kumar Subramanian
> 
>

Re: Review Request 51939: Framework to apply updates to types in the type-system

[Sarath Kumar Subramanian]

> On Sept. 20, 2016, 1:28 p.m., Suma Shivaprasad wrote:
> > typesystem/src/main/java/org/apache/atlas/typesystem/types/TypeSystem.java, 
> > line 314
> > 
> >
> > We would need to audit types as well to 
> > know what was the original type definition that was updated and the 
> > patch that was applied.

this will be addressed in separate jiira to track all type changes


- Sarath Kumar


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51939/#review149727
---


On Sept. 26, 2016, 5:25 p.m., Sarath Kumar Subramanian wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51939/
> ---
> 
> (Updated Sept. 26, 2016, 5:25 p.m.)
> 
> 
> Review request for atlas, Madhan Neethiraj, Shwetha GS, and Suma Shivaprasad.
> 
> 
> Bugs: ATLAS-1174
> https://issues.apache.org/jira/browse/ATLAS-1174
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> 1. Introduce "version" attribute to all types in the type-system, this helps 
> to track changes made to the default types (hive, sqoop, falcon and storm 
> types) and user created types. If version is not mentioned during creation of 
> a type, default version "1.0" is assigned (optional attribute).
> 2. Using the version attributed for types, introduce a patch framework for 
> type system. This framework applies patches to a type using its version 
> number and can be used during upgrade - add new attributes to an existing 
> types and it will be run during atlas startup.
> The sequence of steps:
> a. During atlas startup, check $ATLAS_HOME/models/patches directory for any 
> available patch files (json files). If there any patch files handle them.
> b. Sample patch json file looks like:
> {
> "patches": [
> { 
> "action": "ADD_ATTRIBUTE",
> "typeName": "hive_column",
> "applyToVersion": "1.0",
> "updateToVersion": "2.0",
> "actionParams": [
> { "name": "position", "dataTypeName": "int", "multiplicity": "optional", 
> "isComposite": false, "isUnique": false, "isIndexable": false, 
> "reverseAttributeName": null }
> ]
> } ]
> }
> c. The framework updates the type in "typeName" for the matching version 
> number and after applying the patch, update the version to the one mentioned 
> in "updateToVersion"
> d. The json file can have more than one action (array of actions).
> e. There can be multiple patch json files in the directory and are applied in 
> the sort order of the filename. eg:
> 001-hive_column_add_position.json
> 002-hive_column_add_anotherattribute.json
> 
> 
> Diffs
> -
> 
>   common/src/main/java/org/apache/atlas/AtlasConstants.java 17ffbd7 
>   common/src/main/java/org/apache/atlas/repository/Constants.java d7f9c89 
>   
> repository/src/main/java/org/apache/atlas/repository/typestore/GraphBackedTypeStore.java
>  a94d157 
>   repository/src/main/java/org/apache/atlas/services/AtlasPatchHandler.java 
> PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/services/AtlasTypeAttributePatch.java
>  PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/services/AtlasTypePatch.java 
> PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java
>  3550492 
>   
> repository/src/test/java/org/apache/atlas/service/DefaultMetadataServiceTest.java
>  6782970 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/AbstractDataType.java
>  fad091d 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/ClassType.java 
> c56987a 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/EnumType.java 
> bdd0a13 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/EnumTypeDefinition.java
>  6340615 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/HierarchicalType.java
>  7224699 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/HierarchicalTypeDefinition.java
>  9a299f0 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/IDataType.java 
> 85ddee7 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/StructType.java 
> 6f40c1d 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/StructTypeDefinition.java
>  f1ce1b7 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/TraitType.java 
> f23bf5b 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/TypeSystem.java 
> 70ba89b 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/utils/TypesUtil.java
>  ef8448d 
>   
> typesystem/src/main/scala/org/apache/atlas/typesystem/json/TypesSerialization.scala
>  5618938 
> 
> Diff: https://reviews.apache.org/r/51939/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sarath 

Re: Review Request 51939: Framework to apply updates to types in the type-system

[Sarath Kumar Subramanian]

> On Sept. 26, 2016, 4:43 p.m., Madhan Neethiraj wrote:
> > repository/src/main/java/org/apache/atlas/services/AtlasTypePatch.java, 
> > line 78
> > 
> >
> > Consider using type Map for parames.

fixed in latest diff


- Sarath Kumar


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51939/#review150489
---


On Sept. 26, 2016, 5:25 p.m., Sarath Kumar Subramanian wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51939/
> ---
> 
> (Updated Sept. 26, 2016, 5:25 p.m.)
> 
> 
> Review request for atlas, Madhan Neethiraj, Shwetha GS, and Suma Shivaprasad.
> 
> 
> Bugs: ATLAS-1174
> https://issues.apache.org/jira/browse/ATLAS-1174
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> 1. Introduce "version" attribute to all types in the type-system, this helps 
> to track changes made to the default types (hive, sqoop, falcon and storm 
> types) and user created types. If version is not mentioned during creation of 
> a type, default version "1.0" is assigned (optional attribute).
> 2. Using the version attributed for types, introduce a patch framework for 
> type system. This framework applies patches to a type using its version 
> number and can be used during upgrade - add new attributes to an existing 
> types and it will be run during atlas startup.
> The sequence of steps:
> a. During atlas startup, check $ATLAS_HOME/models/patches directory for any 
> available patch files (json files). If there any patch files handle them.
> b. Sample patch json file looks like:
> {
> "patches": [
> { 
> "action": "ADD_ATTRIBUTE",
> "typeName": "hive_column",
> "applyToVersion": "1.0",
> "updateToVersion": "2.0",
> "actionParams": [
> { "name": "position", "dataTypeName": "int", "multiplicity": "optional", 
> "isComposite": false, "isUnique": false, "isIndexable": false, 
> "reverseAttributeName": null }
> ]
> } ]
> }
> c. The framework updates the type in "typeName" for the matching version 
> number and after applying the patch, update the version to the one mentioned 
> in "updateToVersion"
> d. The json file can have more than one action (array of actions).
> e. There can be multiple patch json files in the directory and are applied in 
> the sort order of the filename. eg:
> 001-hive_column_add_position.json
> 002-hive_column_add_anotherattribute.json
> 
> 
> Diffs
> -
> 
>   common/src/main/java/org/apache/atlas/AtlasConstants.java 17ffbd7 
>   common/src/main/java/org/apache/atlas/repository/Constants.java d7f9c89 
>   
> repository/src/main/java/org/apache/atlas/repository/typestore/GraphBackedTypeStore.java
>  a94d157 
>   repository/src/main/java/org/apache/atlas/services/AtlasPatchHandler.java 
> PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/services/AtlasTypeAttributePatch.java
>  PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/services/AtlasTypePatch.java 
> PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java
>  3550492 
>   
> repository/src/test/java/org/apache/atlas/service/DefaultMetadataServiceTest.java
>  6782970 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/AbstractDataType.java
>  fad091d 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/ClassType.java 
> c56987a 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/EnumType.java 
> bdd0a13 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/EnumTypeDefinition.java
>  6340615 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/HierarchicalType.java
>  7224699 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/HierarchicalTypeDefinition.java
>  9a299f0 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/IDataType.java 
> 85ddee7 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/StructType.java 
> 6f40c1d 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/StructTypeDefinition.java
>  f1ce1b7 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/TraitType.java 
> f23bf5b 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/TypeSystem.java 
> 70ba89b 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/utils/TypesUtil.java
>  ef8448d 
>   
> typesystem/src/main/scala/org/apache/atlas/typesystem/json/TypesSerialization.scala
>  5618938 
> 
> Diff: https://reviews.apache.org/r/51939/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sarath Kumar Subramanian
> 
>

Re: Review Request 51939: Framework to apply updates to types in the type-system

[Sarath Kumar Subramanian]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51939/
---

(Updated Sept. 26, 2016, 5:25 p.m.)


Review request for atlas, Madhan Neethiraj, Shwetha GS, and Suma Shivaprasad.


Changes
---

changed params attribute from string to map


Bugs: ATLAS-1174
https://issues.apache.org/jira/browse/ATLAS-1174


Repository: atlas


Description
---

1. Introduce "version" attribute to all types in the type-system, this helps to 
track changes made to the default types (hive, sqoop, falcon and storm types) 
and user created types. If version is not mentioned during creation of a type, 
default version "1.0" is assigned (optional attribute).
2. Using the version attributed for types, introduce a patch framework for type 
system. This framework applies patches to a type using its version number and 
can be used during upgrade - add new attributes to an existing types and it 
will be run during atlas startup.
The sequence of steps:
a. During atlas startup, check $ATLAS_HOME/models/patches directory for any 
available patch files (json files). If there any patch files handle them.
b. Sample patch json file looks like:
{
"patches": [
{ 
"action": "ADD_ATTRIBUTE",
"typeName": "hive_column",
"applyToVersion": "1.0",
"updateToVersion": "2.0",
"actionParams": [
{ "name": "position", "dataTypeName": "int", "multiplicity": "optional", 
"isComposite": false, "isUnique": false, "isIndexable": false, 
"reverseAttributeName": null }
]
} ]
}
c. The framework updates the type in "typeName" for the matching version number 
and after applying the patch, update the version to the one mentioned in 
"updateToVersion"
d. The json file can have more than one action (array of actions).
e. There can be multiple patch json files in the directory and are applied in 
the sort order of the filename. eg:
001-hive_column_add_position.json
002-hive_column_add_anotherattribute.json


Diffs (updated)
-

  common/src/main/java/org/apache/atlas/AtlasConstants.java 17ffbd7 
  common/src/main/java/org/apache/atlas/repository/Constants.java d7f9c89 
  
repository/src/main/java/org/apache/atlas/repository/typestore/GraphBackedTypeStore.java
 a94d157 
  repository/src/main/java/org/apache/atlas/services/AtlasPatchHandler.java 
PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/services/AtlasTypeAttributePatch.java 
PRE-CREATION 
  repository/src/main/java/org/apache/atlas/services/AtlasTypePatch.java 
PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java 
3550492 
  
repository/src/test/java/org/apache/atlas/service/DefaultMetadataServiceTest.java
 6782970 
  
typesystem/src/main/java/org/apache/atlas/typesystem/types/AbstractDataType.java
 fad091d 
  typesystem/src/main/java/org/apache/atlas/typesystem/types/ClassType.java 
c56987a 
  typesystem/src/main/java/org/apache/atlas/typesystem/types/EnumType.java 
bdd0a13 
  
typesystem/src/main/java/org/apache/atlas/typesystem/types/EnumTypeDefinition.java
 6340615 
  
typesystem/src/main/java/org/apache/atlas/typesystem/types/HierarchicalType.java
 7224699 
  
typesystem/src/main/java/org/apache/atlas/typesystem/types/HierarchicalTypeDefinition.java
 9a299f0 
  typesystem/src/main/java/org/apache/atlas/typesystem/types/IDataType.java 
85ddee7 
  typesystem/src/main/java/org/apache/atlas/typesystem/types/StructType.java 
6f40c1d 
  
typesystem/src/main/java/org/apache/atlas/typesystem/types/StructTypeDefinition.java
 f1ce1b7 
  typesystem/src/main/java/org/apache/atlas/typesystem/types/TraitType.java 
f23bf5b 
  typesystem/src/main/java/org/apache/atlas/typesystem/types/TypeSystem.java 
70ba89b 
  
typesystem/src/main/java/org/apache/atlas/typesystem/types/utils/TypesUtil.java 
ef8448d 
  
typesystem/src/main/scala/org/apache/atlas/typesystem/json/TypesSerialization.scala
 5618938 

Diff: https://reviews.apache.org/r/51939/diff/


Testing
---


Thanks,

Sarath Kumar Subramanian

Re: Review Request 51939: Framework to apply updates to types in the type-system

[Madhan Neethiraj]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51939/#review150489
---




repository/src/main/java/org/apache/atlas/services/AtlasTypePatch.java (line 78)


Consider using type Map for parames.


- Madhan Neethiraj


On Sept. 26, 2016, 10:27 p.m., Sarath Kumar Subramanian wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51939/
> ---
> 
> (Updated Sept. 26, 2016, 10:27 p.m.)
> 
> 
> Review request for atlas, Madhan Neethiraj, Shwetha GS, and Suma Shivaprasad.
> 
> 
> Bugs: ATLAS-1174
> https://issues.apache.org/jira/browse/ATLAS-1174
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> 1. Introduce "version" attribute to all types in the type-system, this helps 
> to track changes made to the default types (hive, sqoop, falcon and storm 
> types) and user created types. If version is not mentioned during creation of 
> a type, default version "1.0" is assigned (optional attribute).
> 2. Using the version attributed for types, introduce a patch framework for 
> type system. This framework applies patches to a type using its version 
> number and can be used during upgrade - add new attributes to an existing 
> types and it will be run during atlas startup.
> The sequence of steps:
> a. During atlas startup, check $ATLAS_HOME/models/patches directory for any 
> available patch files (json files). If there any patch files handle them.
> b. Sample patch json file looks like:
> {
> "patches": [
> { 
> "action": "ADD_ATTRIBUTE",
> "typeName": "hive_column",
> "applyToVersion": "1.0",
> "updateToVersion": "2.0",
> "actionParams": [
> { "name": "position", "dataTypeName": "int", "multiplicity": "optional", 
> "isComposite": false, "isUnique": false, "isIndexable": false, 
> "reverseAttributeName": null }
> ]
> } ]
> }
> c. The framework updates the type in "typeName" for the matching version 
> number and after applying the patch, update the version to the one mentioned 
> in "updateToVersion"
> d. The json file can have more than one action (array of actions).
> e. There can be multiple patch json files in the directory and are applied in 
> the sort order of the filename. eg:
> 001-hive_column_add_position.json
> 002-hive_column_add_anotherattribute.json
> 
> 
> Diffs
> -
> 
>   common/src/main/java/org/apache/atlas/AtlasConstants.java 17ffbd7 
>   common/src/main/java/org/apache/atlas/repository/Constants.java d7f9c89 
>   
> repository/src/main/java/org/apache/atlas/repository/typestore/GraphBackedTypeStore.java
>  a94d157 
>   repository/src/main/java/org/apache/atlas/services/AtlasPatchHandler.java 
> PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/services/AtlasTypeAttributePatch.java
>  PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/services/AtlasTypePatch.java 
> PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java
>  3550492 
>   
> repository/src/test/java/org/apache/atlas/service/DefaultMetadataServiceTest.java
>  6782970 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/AbstractDataType.java
>  fad091d 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/ClassType.java 
> c56987a 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/EnumType.java 
> bdd0a13 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/EnumTypeDefinition.java
>  6340615 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/HierarchicalType.java
>  7224699 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/HierarchicalTypeDefinition.java
>  9a299f0 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/IDataType.java 
> 85ddee7 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/StructType.java 
> 6f40c1d 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/StructTypeDefinition.java
>  f1ce1b7 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/TraitType.java 
> f23bf5b 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/TypeSystem.java 
> 70ba89b 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/utils/TypesUtil.java
>  ef8448d 
>   
> typesystem/src/main/scala/org/apache/atlas/typesystem/json/TypesSerialization.scala
>  5618938 
> 
> Diff: https://reviews.apache.org/r/51939/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sarath Kumar Subramanian
> 
>

Re: Review Request 51939: Framework to apply updates to types in the type-system

[Sarath Kumar Subramanian]

On Sept. 21, 2016, 7:46 a.m., Sarath Kumar Subramanian wrote:
> > Add tests
> > 
> > 1. Currently, the model files(like hive_model.json) are auto generated from 
> > model definitions defined in java(like HiveDataModelGenerator). The patch 
> > files in this case has to be hand coded which is error prone
> > 2. For completeness, readability and debuggability, the type update has to 
> > be done in the corresponding model definitions like HiveDataModelGenerator. 
> > So, same data will be in two places and the model definitions and the patch 
> > files can go out of sync
> > 3. Since the model definitions(like HiveDataModelGenerator) will be updated 
> > anyways, if we modify ReservedTypesRegistrar to do type update instead of 
> > type create, the type updates will automatically be taken care with the 
> > same model json. So, model update patches are not necessary then
> > 3. This jira doesn't implement type versioning - doesn't have support for 
> > storing multiple versions of the type. But it maintains the version of the 
> > latest type definition which I think is useful for debugging, to know the 
> > version of type that the server knows. Can we maintain this info in 
> > HiveDataModelGenerator itself, and hence will be part of hive_model.json
> 
> Sarath Kumar Subramanian wrote:
> 1. patch json genarator can be addressed in a separate jiira
> 2. this is only for upgrade scenarios
> 3. lets address this in a separate jiira.
> 4. we dont want to maintain multiple versions of the same type in the 
> typesystem, will confuse users on which version to use to create entities.
> 
> Shwetha GS wrote:
> What benefit does this patch framework add when the type update can be 
> done with the existing functionality itself (Type versioning is useful and 
> can be done without the patch framework). The patch framework adds more 
> overhead, and I don't see the necessity
> 
> David Radley wrote:
> I agree - I think this capability would be better placed in separate 
> tooling that uses the Atlas REST APIs.
> 
> Madhan Neethiraj wrote:
> @Shwetha, @David - the rational for the patch framework is exactly same 
> as the need for reading contents of "models" directory during startup and 
> initializing the typesystem. This just makes it easier to deal with 
> updates/additions to typesystem during software upgrade/patch.
> 
> Sarath Kumar Subramanian wrote:
> The existing system to update types (using REST) has these limitations:
> 
> 1. Clients might add new attributes to a type and when we add a set of 
> new attributes in model.json -  the rest api model overwrites customer 
> changes.
> 
> 2. More granular type updates cannot be applied, if you add a new 
> attribute in hive_model for eg.,  During atlas startup it doesn't do a type 
> by model comparison and apply the diff. This patch can update to a specific 
> type based on its version - condition based update.
> 
> David Radley wrote:
> @Madhan @Sarath It seems to me that the model files - are system types 
> that we supply to aid integration with Hive and the like. As these do not 
> change, then I am happy these are in model files. If we put changes in patch 
> files, then patch files become the master. I think we need the Atlas 
> repository to be the master of all the types in all cases. It  should not be 
> a slave to patch files. I can see a place for patch files that tooling uses 
> at a one off upgrade of types to update the repository. I agree that care 
> needs to be taken on type shape changes, in practice I would expect that 
> these shape change operations could be restricted to admin roles.
> 
> Shwetha GS wrote:
> Sarath,
> >>Clients might add new attributes to a type and when we add a set of new 
> attributes in model.json - the rest api model overwrites customer changes
> Thats a valid point. 
> 
> The patch doesn't apply on trunk. Can you re-base the patch and add the 
> tests?
> 
> David,
> Registering these models/patches is part of set-up tool required for 
> hooks to work. These use internal methods directly which we should change to 
> make use of APIs. This set-up is required to be run before any hooks can 
> register the entities, hence its run as part of service start itself. But I 
> agree that we should move this out of repository.

Shwetha,
I have rebased the patch and added unit tests.

Thanks,
Sarath


- Sarath Kumar


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51939/#review149811
---


On Sept. 26, 2016, 3:27 p.m., Sarath Kumar Subramanian wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51939/
> ---
> 
> (Updated 

Re: Review Request 51939: Framework to apply updates to types in the type-system

[Sarath Kumar Subramanian]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51939/
---

(Updated Sept. 26, 2016, 3:27 p.m.)


Review request for atlas, Madhan Neethiraj, Shwetha GS, and Suma Shivaprasad.


Changes
---

added unit tests and rebased the patch - updated one


Bugs: ATLAS-1174
https://issues.apache.org/jira/browse/ATLAS-1174


Repository: atlas


Description
---

1. Introduce "version" attribute to all types in the type-system, this helps to 
track changes made to the default types (hive, sqoop, falcon and storm types) 
and user created types. If version is not mentioned during creation of a type, 
default version "1.0" is assigned (optional attribute).
2. Using the version attributed for types, introduce a patch framework for type 
system. This framework applies patches to a type using its version number and 
can be used during upgrade - add new attributes to an existing types and it 
will be run during atlas startup.
The sequence of steps:
a. During atlas startup, check $ATLAS_HOME/models/patches directory for any 
available patch files (json files). If there any patch files handle them.
b. Sample patch json file looks like:
{
"patches": [
{ 
"action": "ADD_ATTRIBUTE",
"typeName": "hive_column",
"applyToVersion": "1.0",
"updateToVersion": "2.0",
"actionParams": [
{ "name": "position", "dataTypeName": "int", "multiplicity": "optional", 
"isComposite": false, "isUnique": false, "isIndexable": false, 
"reverseAttributeName": null }
]
} ]
}
c. The framework updates the type in "typeName" for the matching version number 
and after applying the patch, update the version to the one mentioned in 
"updateToVersion"
d. The json file can have more than one action (array of actions).
e. There can be multiple patch json files in the directory and are applied in 
the sort order of the filename. eg:
001-hive_column_add_position.json
002-hive_column_add_anotherattribute.json


Diffs (updated)
-

  common/src/main/java/org/apache/atlas/AtlasConstants.java 17ffbd7 
  common/src/main/java/org/apache/atlas/repository/Constants.java d7f9c89 
  
repository/src/main/java/org/apache/atlas/repository/typestore/GraphBackedTypeStore.java
 a94d157 
  repository/src/main/java/org/apache/atlas/services/AtlasPatchHandler.java 
PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/services/AtlasTypeAttributePatch.java 
PRE-CREATION 
  repository/src/main/java/org/apache/atlas/services/AtlasTypePatch.java 
PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java 
3550492 
  
repository/src/test/java/org/apache/atlas/service/DefaultMetadataServiceTest.java
 6782970 
  
typesystem/src/main/java/org/apache/atlas/typesystem/types/AbstractDataType.java
 fad091d 
  typesystem/src/main/java/org/apache/atlas/typesystem/types/ClassType.java 
c56987a 
  typesystem/src/main/java/org/apache/atlas/typesystem/types/EnumType.java 
bdd0a13 
  
typesystem/src/main/java/org/apache/atlas/typesystem/types/EnumTypeDefinition.java
 6340615 
  
typesystem/src/main/java/org/apache/atlas/typesystem/types/HierarchicalType.java
 7224699 
  
typesystem/src/main/java/org/apache/atlas/typesystem/types/HierarchicalTypeDefinition.java
 9a299f0 
  typesystem/src/main/java/org/apache/atlas/typesystem/types/IDataType.java 
85ddee7 
  typesystem/src/main/java/org/apache/atlas/typesystem/types/StructType.java 
6f40c1d 
  
typesystem/src/main/java/org/apache/atlas/typesystem/types/StructTypeDefinition.java
 f1ce1b7 
  typesystem/src/main/java/org/apache/atlas/typesystem/types/TraitType.java 
f23bf5b 
  typesystem/src/main/java/org/apache/atlas/typesystem/types/TypeSystem.java 
70ba89b 
  
typesystem/src/main/java/org/apache/atlas/typesystem/types/utils/TypesUtil.java 
ef8448d 
  
typesystem/src/main/scala/org/apache/atlas/typesystem/json/TypesSerialization.scala
 5618938 

Diff: https://reviews.apache.org/r/51939/diff/


Testing
---


Thanks,

Sarath Kumar Subramanian

Re: Review Request 51939: Framework to apply updates to types in the type-system

[Sarath Kumar Subramanian]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51939/
---

(Updated Sept. 25, 2016, 11:13 p.m.)


Review request for atlas, Madhan Neethiraj, Shwetha GS, and Suma Shivaprasad.


Changes
---

added unit tests to test add/update attributes and code refactoring in patch 
files


Bugs: ATLAS-1174
https://issues.apache.org/jira/browse/ATLAS-1174


Repository: atlas


Description
---

1. Introduce "version" attribute to all types in the type-system, this helps to 
track changes made to the default types (hive, sqoop, falcon and storm types) 
and user created types. If version is not mentioned during creation of a type, 
default version "1.0" is assigned (optional attribute).
2. Using the version attributed for types, introduce a patch framework for type 
system. This framework applies patches to a type using its version number and 
can be used during upgrade - add new attributes to an existing types and it 
will be run during atlas startup.
The sequence of steps:
a. During atlas startup, check $ATLAS_HOME/models/patches directory for any 
available patch files (json files). If there any patch files handle them.
b. Sample patch json file looks like:
{
"patches": [
{ 
"action": "ADD_ATTRIBUTE",
"typeName": "hive_column",
"applyToVersion": "1.0",
"updateToVersion": "2.0",
"actionParams": [
{ "name": "position", "dataTypeName": "int", "multiplicity": "optional", 
"isComposite": false, "isUnique": false, "isIndexable": false, 
"reverseAttributeName": null }
]
} ]
}
c. The framework updates the type in "typeName" for the matching version number 
and after applying the patch, update the version to the one mentioned in 
"updateToVersion"
d. The json file can have more than one action (array of actions).
e. There can be multiple patch json files in the directory and are applied in 
the sort order of the filename. eg:
001-hive_column_add_position.json
002-hive_column_add_anotherattribute.json


Diffs (updated)
-

  common/src/main/java/org/apache/atlas/AtlasConstants.java 17ffbd7 
  common/src/main/java/org/apache/atlas/repository/Constants.java d7f9c89 
  
repository/src/main/java/org/apache/atlas/repository/typestore/GraphBackedTypeStore.java
 a94d157 
  repository/src/main/java/org/apache/atlas/services/AtlasPatchHandler.java 
PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/services/AtlasTypeAttributePatch.java 
PRE-CREATION 
  repository/src/main/java/org/apache/atlas/services/AtlasTypePatch.java 
PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java 
6a937f4 
  
repository/src/test/java/org/apache/atlas/service/DefaultMetadataServiceTest.java
 52dcfde 
  
typesystem/src/main/java/org/apache/atlas/typesystem/types/AbstractDataType.java
 fad091d 
  typesystem/src/main/java/org/apache/atlas/typesystem/types/ClassType.java 
c56987a 
  typesystem/src/main/java/org/apache/atlas/typesystem/types/EnumType.java 
bdd0a13 
  
typesystem/src/main/java/org/apache/atlas/typesystem/types/EnumTypeDefinition.java
 6340615 
  
typesystem/src/main/java/org/apache/atlas/typesystem/types/HierarchicalType.java
 7224699 
  
typesystem/src/main/java/org/apache/atlas/typesystem/types/HierarchicalTypeDefinition.java
 9a299f0 
  typesystem/src/main/java/org/apache/atlas/typesystem/types/IDataType.java 
85ddee7 
  typesystem/src/main/java/org/apache/atlas/typesystem/types/StructType.java 
6f40c1d 
  
typesystem/src/main/java/org/apache/atlas/typesystem/types/StructTypeDefinition.java
 f1ce1b7 
  typesystem/src/main/java/org/apache/atlas/typesystem/types/TraitType.java 
f23bf5b 
  typesystem/src/main/java/org/apache/atlas/typesystem/types/TypeSystem.java 
70ba89b 
  
typesystem/src/main/java/org/apache/atlas/typesystem/types/utils/TypesUtil.java 
ef8448d 
  
typesystem/src/main/scala/org/apache/atlas/typesystem/json/TypesSerialization.scala
 5618938 

Diff: https://reviews.apache.org/r/51939/diff/


Testing
---


Thanks,

Sarath Kumar Subramanian

Re: Review Request 51939: Framework to apply updates to types in the type-system

[Shwetha GS]

On Sept. 21, 2016, 2:46 p.m., Sarath Kumar Subramanian wrote:
> > Add tests
> > 
> > 1. Currently, the model files(like hive_model.json) are auto generated from 
> > model definitions defined in java(like HiveDataModelGenerator). The patch 
> > files in this case has to be hand coded which is error prone
> > 2. For completeness, readability and debuggability, the type update has to 
> > be done in the corresponding model definitions like HiveDataModelGenerator. 
> > So, same data will be in two places and the model definitions and the patch 
> > files can go out of sync
> > 3. Since the model definitions(like HiveDataModelGenerator) will be updated 
> > anyways, if we modify ReservedTypesRegistrar to do type update instead of 
> > type create, the type updates will automatically be taken care with the 
> > same model json. So, model update patches are not necessary then
> > 3. This jira doesn't implement type versioning - doesn't have support for 
> > storing multiple versions of the type. But it maintains the version of the 
> > latest type definition which I think is useful for debugging, to know the 
> > version of type that the server knows. Can we maintain this info in 
> > HiveDataModelGenerator itself, and hence will be part of hive_model.json
> 
> Sarath Kumar Subramanian wrote:
> 1. patch json genarator can be addressed in a separate jiira
> 2. this is only for upgrade scenarios
> 3. lets address this in a separate jiira.
> 4. we dont want to maintain multiple versions of the same type in the 
> typesystem, will confuse users on which version to use to create entities.
> 
> Shwetha GS wrote:
> What benefit does this patch framework add when the type update can be 
> done with the existing functionality itself (Type versioning is useful and 
> can be done without the patch framework). The patch framework adds more 
> overhead, and I don't see the necessity
> 
> David Radley wrote:
> I agree - I think this capability would be better placed in separate 
> tooling that uses the Atlas REST APIs.
> 
> Madhan Neethiraj wrote:
> @Shwetha, @David - the rational for the patch framework is exactly same 
> as the need for reading contents of "models" directory during startup and 
> initializing the typesystem. This just makes it easier to deal with 
> updates/additions to typesystem during software upgrade/patch.
> 
> Sarath Kumar Subramanian wrote:
> The existing system to update types (using REST) has these limitations:
> 
> 1. Clients might add new attributes to a type and when we add a set of 
> new attributes in model.json -  the rest api model overwrites customer 
> changes.
> 
> 2. More granular type updates cannot be applied, if you add a new 
> attribute in hive_model for eg.,  During atlas startup it doesn't do a type 
> by model comparison and apply the diff. This patch can update to a specific 
> type based on its version - condition based update.
> 
> David Radley wrote:
> @Madhan @Sarath It seems to me that the model files - are system types 
> that we supply to aid integration with Hive and the like. As these do not 
> change, then I am happy these are in model files. If we put changes in patch 
> files, then patch files become the master. I think we need the Atlas 
> repository to be the master of all the types in all cases. It  should not be 
> a slave to patch files. I can see a place for patch files that tooling uses 
> at a one off upgrade of types to update the repository. I agree that care 
> needs to be taken on type shape changes, in practice I would expect that 
> these shape change operations could be restricted to admin roles.

Sarath,
>>Clients might add new attributes to a type and when we add a set of new 
>>attributes in model.json - the rest api model overwrites customer changes
Thats a valid point. 

The patch doesn't apply on trunk. Can you re-base the patch and add the tests?

David,
Registering these models/patches is part of set-up tool required for hooks to 
work. These use internal methods directly which we should change to make use of 
APIs. This set-up is required to be run before any hooks can register the 
entities, hence its run as part of service start itself. But I agree that we 
should move this out of repository.


- Shwetha


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51939/#review149811
---


On Sept. 23, 2016, 3:02 a.m., Sarath Kumar Subramanian wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51939/
> ---
> 
> (Updated Sept. 23, 2016, 3:02 a.m.)
> 
> 
> Review request for atlas, Madhan Neethiraj, Shwetha GS, and Suma Shivaprasad.
> 
> 
> Bugs: ATLAS-1174
> 

Re: Review Request 51939: Framework to apply updates to types in the type-system

[Madhan Neethiraj]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51939/#review150153
---


Ship it!




Ship It!

- Madhan Neethiraj


On Sept. 23, 2016, 3:02 a.m., Sarath Kumar Subramanian wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51939/
> ---
> 
> (Updated Sept. 23, 2016, 3:02 a.m.)
> 
> 
> Review request for atlas, Madhan Neethiraj, Shwetha GS, and Suma Shivaprasad.
> 
> 
> Bugs: ATLAS-1174
> https://issues.apache.org/jira/browse/ATLAS-1174
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> 1. Introduce "version" attribute to all types in the type-system, this helps 
> to track changes made to the default types (hive, sqoop, falcon and storm 
> types) and user created types. If version is not mentioned during creation of 
> a type, default version "1.0" is assigned (optional attribute).
> 2. Using the version attributed for types, introduce a patch framework for 
> type system. This framework applies patches to a type using its version 
> number and can be used during upgrade - add new attributes to an existing 
> types and it will be run during atlas startup.
> The sequence of steps:
> a. During atlas startup, check $ATLAS_HOME/models/patches directory for any 
> available patch files (json files). If there any patch files handle them.
> b. Sample patch json file looks like:
> {
> "patches": [
> { 
> "action": "ADD_ATTRIBUTE",
> "typeName": "hive_column",
> "applyToVersion": "1.0",
> "updateToVersion": "2.0",
> "actionParams": [
> { "name": "position", "dataTypeName": "int", "multiplicity": "optional", 
> "isComposite": false, "isUnique": false, "isIndexable": false, 
> "reverseAttributeName": null }
> ]
> } ]
> }
> c. The framework updates the type in "typeName" for the matching version 
> number and after applying the patch, update the version to the one mentioned 
> in "updateToVersion"
> d. The json file can have more than one action (array of actions).
> e. There can be multiple patch json files in the directory and are applied in 
> the sort order of the filename. eg:
> 001-hive_column_add_position.json
> 002-hive_column_add_anotherattribute.json
> 
> 
> Diffs
> -
> 
>   common/src/main/java/org/apache/atlas/AtlasConstants.java 17ffbd7 
>   common/src/main/java/org/apache/atlas/repository/Constants.java d7f9c89 
>   
> repository/src/main/java/org/apache/atlas/repository/typestore/GraphBackedTypeStore.java
>  a94d157 
>   repository/src/main/java/org/apache/atlas/services/AtlasPatchHandler.java 
> PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/services/AtlasTypeAttributePatch.java
>  PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/services/AtlasTypePatch.java 
> PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java
>  6a937f4 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/AbstractDataType.java
>  fad091d 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/ClassType.java 
> c56987a 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/EnumType.java 
> bdd0a13 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/EnumTypeDefinition.java
>  6340615 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/HierarchicalType.java
>  7224699 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/HierarchicalTypeDefinition.java
>  9a299f0 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/IDataType.java 
> 85ddee7 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/StructType.java 
> 6f40c1d 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/StructTypeDefinition.java
>  f1ce1b7 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/TraitType.java 
> f23bf5b 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/TypeSystem.java 
> 70ba89b 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/utils/TypesUtil.java
>  ef8448d 
>   
> typesystem/src/main/scala/org/apache/atlas/typesystem/json/TypesSerialization.scala
>  5618938 
> 
> Diff: https://reviews.apache.org/r/51939/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sarath Kumar Subramanian
> 
>

Re: Review Request 51939: Framework to apply updates to types in the type-system

[Sarath Kumar Subramanian]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51939/
---

(Updated Sept. 22, 2016, 8:02 p.m.)


Review request for atlas, Madhan Neethiraj, Shwetha GS, and Suma Shivaprasad.


Changes
---

moved checkForPatches() method to a seperate class which can handle all patch 
updates


Bugs: ATLAS-1174
https://issues.apache.org/jira/browse/ATLAS-1174


Repository: atlas


Description
---

1. Introduce "version" attribute to all types in the type-system, this helps to 
track changes made to the default types (hive, sqoop, falcon and storm types) 
and user created types. If version is not mentioned during creation of a type, 
default version "1.0" is assigned (optional attribute).
2. Using the version attributed for types, introduce a patch framework for type 
system. This framework applies patches to a type using its version number and 
can be used during upgrade - add new attributes to an existing types and it 
will be run during atlas startup.
The sequence of steps:
a. During atlas startup, check $ATLAS_HOME/models/patches directory for any 
available patch files (json files). If there any patch files handle them.
b. Sample patch json file looks like:
{
"patches": [
{ 
"action": "ADD_ATTRIBUTE",
"typeName": "hive_column",
"applyToVersion": "1.0",
"updateToVersion": "2.0",
"actionParams": [
{ "name": "position", "dataTypeName": "int", "multiplicity": "optional", 
"isComposite": false, "isUnique": false, "isIndexable": false, 
"reverseAttributeName": null }
]
} ]
}
c. The framework updates the type in "typeName" for the matching version number 
and after applying the patch, update the version to the one mentioned in 
"updateToVersion"
d. The json file can have more than one action (array of actions).
e. There can be multiple patch json files in the directory and are applied in 
the sort order of the filename. eg:
001-hive_column_add_position.json
002-hive_column_add_anotherattribute.json


Diffs (updated)
-

  common/src/main/java/org/apache/atlas/AtlasConstants.java 17ffbd7 
  common/src/main/java/org/apache/atlas/repository/Constants.java d7f9c89 
  
repository/src/main/java/org/apache/atlas/repository/typestore/GraphBackedTypeStore.java
 a94d157 
  repository/src/main/java/org/apache/atlas/services/AtlasPatchHandler.java 
PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/services/AtlasTypeAttributePatch.java 
PRE-CREATION 
  repository/src/main/java/org/apache/atlas/services/AtlasTypePatch.java 
PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java 
6a937f4 
  
typesystem/src/main/java/org/apache/atlas/typesystem/types/AbstractDataType.java
 fad091d 
  typesystem/src/main/java/org/apache/atlas/typesystem/types/ClassType.java 
c56987a 
  typesystem/src/main/java/org/apache/atlas/typesystem/types/EnumType.java 
bdd0a13 
  
typesystem/src/main/java/org/apache/atlas/typesystem/types/EnumTypeDefinition.java
 6340615 
  
typesystem/src/main/java/org/apache/atlas/typesystem/types/HierarchicalType.java
 7224699 
  
typesystem/src/main/java/org/apache/atlas/typesystem/types/HierarchicalTypeDefinition.java
 9a299f0 
  typesystem/src/main/java/org/apache/atlas/typesystem/types/IDataType.java 
85ddee7 
  typesystem/src/main/java/org/apache/atlas/typesystem/types/StructType.java 
6f40c1d 
  
typesystem/src/main/java/org/apache/atlas/typesystem/types/StructTypeDefinition.java
 f1ce1b7 
  typesystem/src/main/java/org/apache/atlas/typesystem/types/TraitType.java 
f23bf5b 
  typesystem/src/main/java/org/apache/atlas/typesystem/types/TypeSystem.java 
70ba89b 
  
typesystem/src/main/java/org/apache/atlas/typesystem/types/utils/TypesUtil.java 
ef8448d 
  
typesystem/src/main/scala/org/apache/atlas/typesystem/json/TypesSerialization.scala
 5618938 

Diff: https://reviews.apache.org/r/51939/diff/


Testing
---


Thanks,

Sarath Kumar Subramanian

Re: Review Request 51939: Framework to apply updates to types in the type-system

[Sarath Kumar Subramanian]

On Sept. 21, 2016, 7:46 a.m., Sarath Kumar Subramanian wrote:
> > Add tests
> > 
> > 1. Currently, the model files(like hive_model.json) are auto generated from 
> > model definitions defined in java(like HiveDataModelGenerator). The patch 
> > files in this case has to be hand coded which is error prone
> > 2. For completeness, readability and debuggability, the type update has to 
> > be done in the corresponding model definitions like HiveDataModelGenerator. 
> > So, same data will be in two places and the model definitions and the patch 
> > files can go out of sync
> > 3. Since the model definitions(like HiveDataModelGenerator) will be updated 
> > anyways, if we modify ReservedTypesRegistrar to do type update instead of 
> > type create, the type updates will automatically be taken care with the 
> > same model json. So, model update patches are not necessary then
> > 3. This jira doesn't implement type versioning - doesn't have support for 
> > storing multiple versions of the type. But it maintains the version of the 
> > latest type definition which I think is useful for debugging, to know the 
> > version of type that the server knows. Can we maintain this info in 
> > HiveDataModelGenerator itself, and hence will be part of hive_model.json
> 
> Sarath Kumar Subramanian wrote:
> 1. patch json genarator can be addressed in a separate jiira
> 2. this is only for upgrade scenarios
> 3. lets address this in a separate jiira.
> 4. we dont want to maintain multiple versions of the same type in the 
> typesystem, will confuse users on which version to use to create entities.
> 
> Shwetha GS wrote:
> What benefit does this patch framework add when the type update can be 
> done with the existing functionality itself (Type versioning is useful and 
> can be done without the patch framework). The patch framework adds more 
> overhead, and I don't see the necessity
> 
> David Radley wrote:
> I agree - I think this capability would be better placed in separate 
> tooling that uses the Atlas REST APIs.
> 
> Madhan Neethiraj wrote:
> @Shwetha, @David - the rational for the patch framework is exactly same 
> as the need for reading contents of "models" directory during startup and 
> initializing the typesystem. This just makes it easier to deal with 
> updates/additions to typesystem during software upgrade/patch.

The existing system to update types (using REST) has these limitations:

1. Clients might add new attributes to a type and when we add a set of new 
attributes in model.json -  the rest api model overwrites customer changes.

2. More granular type updates cannot be applied, if you add a new attribute in 
hive_model for eg.,  During atlas startup it doesn't do a type by model 
comparison and apply the diff. This patch can update to a specific type based 
on its version - condition based update.


- Sarath Kumar


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51939/#review149811
---


On Sept. 22, 2016, 12:34 a.m., Sarath Kumar Subramanian wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51939/
> ---
> 
> (Updated Sept. 22, 2016, 12:34 a.m.)
> 
> 
> Review request for atlas, Madhan Neethiraj, Shwetha GS, and Suma Shivaprasad.
> 
> 
> Bugs: ATLAS-1174
> https://issues.apache.org/jira/browse/ATLAS-1174
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> 1. Introduce "version" attribute to all types in the type-system, this helps 
> to track changes made to the default types (hive, sqoop, falcon and storm 
> types) and user created types. If version is not mentioned during creation of 
> a type, default version "1.0" is assigned (optional attribute).
> 2. Using the version attributed for types, introduce a patch framework for 
> type system. This framework applies patches to a type using its version 
> number and can be used during upgrade - add new attributes to an existing 
> types and it will be run during atlas startup.
> The sequence of steps:
> a. During atlas startup, check $ATLAS_HOME/models/patches directory for any 
> available patch files (json files). If there any patch files handle them.
> b. Sample patch json file looks like:
> {
> "patches": [
> { 
> "action": "ADD_ATTRIBUTE",
> "typeName": "hive_column",
> "applyToVersion": "1.0",
> "updateToVersion": "2.0",
> "actionParams": [
> { "name": "position", "dataTypeName": "int", "multiplicity": "optional", 
> "isComposite": false, "isUnique": false, "isIndexable": false, 
> "reverseAttributeName": null }
> ]
> } ]
> }
> c. The framework updates the type in "typeName" for the matching version 
> number and after applying the patch, update the version to the one mentioned 
> in 

Re: Review Request 51939: Framework to apply updates to types in the type-system

[Madhan Neethiraj]

On Sept. 21, 2016, 2:46 p.m., Sarath Kumar Subramanian wrote:
> > Add tests
> > 
> > 1. Currently, the model files(like hive_model.json) are auto generated from 
> > model definitions defined in java(like HiveDataModelGenerator). The patch 
> > files in this case has to be hand coded which is error prone
> > 2. For completeness, readability and debuggability, the type update has to 
> > be done in the corresponding model definitions like HiveDataModelGenerator. 
> > So, same data will be in two places and the model definitions and the patch 
> > files can go out of sync
> > 3. Since the model definitions(like HiveDataModelGenerator) will be updated 
> > anyways, if we modify ReservedTypesRegistrar to do type update instead of 
> > type create, the type updates will automatically be taken care with the 
> > same model json. So, model update patches are not necessary then
> > 3. This jira doesn't implement type versioning - doesn't have support for 
> > storing multiple versions of the type. But it maintains the version of the 
> > latest type definition which I think is useful for debugging, to know the 
> > version of type that the server knows. Can we maintain this info in 
> > HiveDataModelGenerator itself, and hence will be part of hive_model.json
> 
> Sarath Kumar Subramanian wrote:
> 1. patch json genarator can be addressed in a separate jiira
> 2. this is only for upgrade scenarios
> 3. lets address this in a separate jiira.
> 4. we dont want to maintain multiple versions of the same type in the 
> typesystem, will confuse users on which version to use to create entities.
> 
> Shwetha GS wrote:
> What benefit does this patch framework add when the type update can be 
> done with the existing functionality itself (Type versioning is useful and 
> can be done without the patch framework). The patch framework adds more 
> overhead, and I don't see the necessity
> 
> David Radley wrote:
> I agree - I think this capability would be better placed in separate 
> tooling that uses the Atlas REST APIs.

@Shwetha, @David - the rational for the patch framework is exactly same as the 
need for reading contents of "models" directory during startup and initializing 
the typesystem. This just makes it easier to deal with updates/additions to 
typesystem during software upgrade/patch.


- Madhan


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51939/#review149811
---


On Sept. 22, 2016, 7:34 a.m., Sarath Kumar Subramanian wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51939/
> ---
> 
> (Updated Sept. 22, 2016, 7:34 a.m.)
> 
> 
> Review request for atlas, Madhan Neethiraj, Shwetha GS, and Suma Shivaprasad.
> 
> 
> Bugs: ATLAS-1174
> https://issues.apache.org/jira/browse/ATLAS-1174
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> 1. Introduce "version" attribute to all types in the type-system, this helps 
> to track changes made to the default types (hive, sqoop, falcon and storm 
> types) and user created types. If version is not mentioned during creation of 
> a type, default version "1.0" is assigned (optional attribute).
> 2. Using the version attributed for types, introduce a patch framework for 
> type system. This framework applies patches to a type using its version 
> number and can be used during upgrade - add new attributes to an existing 
> types and it will be run during atlas startup.
> The sequence of steps:
> a. During atlas startup, check $ATLAS_HOME/models/patches directory for any 
> available patch files (json files). If there any patch files handle them.
> b. Sample patch json file looks like:
> {
> "patches": [
> { 
> "action": "ADD_ATTRIBUTE",
> "typeName": "hive_column",
> "applyToVersion": "1.0",
> "updateToVersion": "2.0",
> "actionParams": [
> { "name": "position", "dataTypeName": "int", "multiplicity": "optional", 
> "isComposite": false, "isUnique": false, "isIndexable": false, 
> "reverseAttributeName": null }
> ]
> } ]
> }
> c. The framework updates the type in "typeName" for the matching version 
> number and after applying the patch, update the version to the one mentioned 
> in "updateToVersion"
> d. The json file can have more than one action (array of actions).
> e. There can be multiple patch json files in the directory and are applied in 
> the sort order of the filename. eg:
> 001-hive_column_add_position.json
> 002-hive_column_add_anotherattribute.json
> 
> 
> Diffs
> -
> 
>   common/src/main/java/org/apache/atlas/AtlasConstants.java 17ffbd7 
>   common/src/main/java/org/apache/atlas/repository/Constants.java d7f9c89 
>   
> 

Re: Review Request 51939: Framework to apply updates to types in the type-system

[David Radley]

On Sept. 21, 2016, 2:46 p.m., Sarath Kumar Subramanian wrote:
> > Add tests
> > 
> > 1. Currently, the model files(like hive_model.json) are auto generated from 
> > model definitions defined in java(like HiveDataModelGenerator). The patch 
> > files in this case has to be hand coded which is error prone
> > 2. For completeness, readability and debuggability, the type update has to 
> > be done in the corresponding model definitions like HiveDataModelGenerator. 
> > So, same data will be in two places and the model definitions and the patch 
> > files can go out of sync
> > 3. Since the model definitions(like HiveDataModelGenerator) will be updated 
> > anyways, if we modify ReservedTypesRegistrar to do type update instead of 
> > type create, the type updates will automatically be taken care with the 
> > same model json. So, model update patches are not necessary then
> > 3. This jira doesn't implement type versioning - doesn't have support for 
> > storing multiple versions of the type. But it maintains the version of the 
> > latest type definition which I think is useful for debugging, to know the 
> > version of type that the server knows. Can we maintain this info in 
> > HiveDataModelGenerator itself, and hence will be part of hive_model.json
> 
> Sarath Kumar Subramanian wrote:
> 1. patch json genarator can be addressed in a separate jiira
> 2. this is only for upgrade scenarios
> 3. lets address this in a separate jiira.
> 4. we dont want to maintain multiple versions of the same type in the 
> typesystem, will confuse users on which version to use to create entities.
> 
> Shwetha GS wrote:
> What benefit does this patch framework add when the type update can be 
> done with the existing functionality itself (Type versioning is useful and 
> can be done without the patch framework). The patch framework adds more 
> overhead, and I don't see the necessity

I agree - I think this capability would be better placed in separate tooling 
that uses the Atlas REST APIs.


- David


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51939/#review149811
---


On Sept. 22, 2016, 7:34 a.m., Sarath Kumar Subramanian wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51939/
> ---
> 
> (Updated Sept. 22, 2016, 7:34 a.m.)
> 
> 
> Review request for atlas, Madhan Neethiraj, Shwetha GS, and Suma Shivaprasad.
> 
> 
> Bugs: ATLAS-1174
> https://issues.apache.org/jira/browse/ATLAS-1174
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> 1. Introduce "version" attribute to all types in the type-system, this helps 
> to track changes made to the default types (hive, sqoop, falcon and storm 
> types) and user created types. If version is not mentioned during creation of 
> a type, default version "1.0" is assigned (optional attribute).
> 2. Using the version attributed for types, introduce a patch framework for 
> type system. This framework applies patches to a type using its version 
> number and can be used during upgrade - add new attributes to an existing 
> types and it will be run during atlas startup.
> The sequence of steps:
> a. During atlas startup, check $ATLAS_HOME/models/patches directory for any 
> available patch files (json files). If there any patch files handle them.
> b. Sample patch json file looks like:
> {
> "patches": [
> { 
> "action": "ADD_ATTRIBUTE",
> "typeName": "hive_column",
> "applyToVersion": "1.0",
> "updateToVersion": "2.0",
> "actionParams": [
> { "name": "position", "dataTypeName": "int", "multiplicity": "optional", 
> "isComposite": false, "isUnique": false, "isIndexable": false, 
> "reverseAttributeName": null }
> ]
> } ]
> }
> c. The framework updates the type in "typeName" for the matching version 
> number and after applying the patch, update the version to the one mentioned 
> in "updateToVersion"
> d. The json file can have more than one action (array of actions).
> e. There can be multiple patch json files in the directory and are applied in 
> the sort order of the filename. eg:
> 001-hive_column_add_position.json
> 002-hive_column_add_anotherattribute.json
> 
> 
> Diffs
> -
> 
>   common/src/main/java/org/apache/atlas/AtlasConstants.java 17ffbd7 
>   common/src/main/java/org/apache/atlas/repository/Constants.java d7f9c89 
>   
> repository/src/main/java/org/apache/atlas/repository/typestore/GraphBackedTypeStore.java
>  a94d157 
>   
> repository/src/main/java/org/apache/atlas/services/AtlasTypeAttributePatch.java
>  PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/services/AtlasTypePatch.java 
> PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java
>  6a937f4 

Re: Review Request 51939: Framework to apply updates to types in the type-system

[Sarath Kumar Subramanian]

On Sept. 21, 2016, 7:46 a.m., Sarath Kumar Subramanian wrote:
> > Add tests
> > 
> > 1. Currently, the model files(like hive_model.json) are auto generated from 
> > model definitions defined in java(like HiveDataModelGenerator). The patch 
> > files in this case has to be hand coded which is error prone
> > 2. For completeness, readability and debuggability, the type update has to 
> > be done in the corresponding model definitions like HiveDataModelGenerator. 
> > So, same data will be in two places and the model definitions and the patch 
> > files can go out of sync
> > 3. Since the model definitions(like HiveDataModelGenerator) will be updated 
> > anyways, if we modify ReservedTypesRegistrar to do type update instead of 
> > type create, the type updates will automatically be taken care with the 
> > same model json. So, model update patches are not necessary then
> > 3. This jira doesn't implement type versioning - doesn't have support for 
> > storing multiple versions of the type. But it maintains the version of the 
> > latest type definition which I think is useful for debugging, to know the 
> > version of type that the server knows. Can we maintain this info in 
> > HiveDataModelGenerator itself, and hence will be part of hive_model.json

1. patch json genarator can be addressed in a separate jiira
2. this is only for upgrade scenarios
3. lets address this in a separate jiira.
4. we dont want to maintain multiple versions of the same type in the 
typesystem, will confuse users on which version to use to create entities.


- Sarath Kumar


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51939/#review149811
---


On Sept. 22, 2016, 12:34 a.m., Sarath Kumar Subramanian wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51939/
> ---
> 
> (Updated Sept. 22, 2016, 12:34 a.m.)
> 
> 
> Review request for atlas, Madhan Neethiraj, Shwetha GS, and Suma Shivaprasad.
> 
> 
> Bugs: ATLAS-1174
> https://issues.apache.org/jira/browse/ATLAS-1174
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> 1. Introduce "version" attribute to all types in the type-system, this helps 
> to track changes made to the default types (hive, sqoop, falcon and storm 
> types) and user created types. If version is not mentioned during creation of 
> a type, default version "1.0" is assigned (optional attribute).
> 2. Using the version attributed for types, introduce a patch framework for 
> type system. This framework applies patches to a type using its version 
> number and can be used during upgrade - add new attributes to an existing 
> types and it will be run during atlas startup.
> The sequence of steps:
> a. During atlas startup, check $ATLAS_HOME/models/patches directory for any 
> available patch files (json files). If there any patch files handle them.
> b. Sample patch json file looks like:
> {
> "patches": [
> { 
> "action": "ADD_ATTRIBUTE",
> "typeName": "hive_column",
> "applyToVersion": "1.0",
> "updateToVersion": "2.0",
> "actionParams": [
> { "name": "position", "dataTypeName": "int", "multiplicity": "optional", 
> "isComposite": false, "isUnique": false, "isIndexable": false, 
> "reverseAttributeName": null }
> ]
> } ]
> }
> c. The framework updates the type in "typeName" for the matching version 
> number and after applying the patch, update the version to the one mentioned 
> in "updateToVersion"
> d. The json file can have more than one action (array of actions).
> e. There can be multiple patch json files in the directory and are applied in 
> the sort order of the filename. eg:
> 001-hive_column_add_position.json
> 002-hive_column_add_anotherattribute.json
> 
> 
> Diffs
> -
> 
>   common/src/main/java/org/apache/atlas/AtlasConstants.java 17ffbd7 
>   common/src/main/java/org/apache/atlas/repository/Constants.java d7f9c89 
>   
> repository/src/main/java/org/apache/atlas/repository/typestore/GraphBackedTypeStore.java
>  a94d157 
>   
> repository/src/main/java/org/apache/atlas/services/AtlasTypeAttributePatch.java
>  PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/services/AtlasTypePatch.java 
> PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java
>  6a937f4 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/AbstractDataType.java
>  fad091d 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/ClassType.java 
> c56987a 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/EnumType.java 
> bdd0a13 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/EnumTypeDefinition.java
>  6340615 
>   
> 

Re: Review Request 51939: Framework to apply updates to types in the type-system

[Sarath Kumar Subramanian]

> On Sept. 21, 2016, 1:55 p.m., Madhan Neethiraj wrote:
> > repository/src/main/java/org/apache/atlas/services/AtlasTypePatch.java, 
> > line 77
> > 
> >
> > Consider adding attributeDefinitions as a first class attribute - to 
> > save from having to tranform from/to JSON in many places. When we add 
> > support for other type of patches (like types..), appropraite members can 
> > be added here.
> > 
> > class PatchData {
> >   private String   action;
> >   private String   typeName;
> >   ..
> >   private List attributeDefinitions;
> >   private Map  params;
> > }

removed all json operations and now using pojo classes


> On Sept. 21, 2016, 1:55 p.m., Madhan Neethiraj wrote:
> > repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java,
> >  line 172
> > 
> >
> > checkForPatches() call will be missed in HA deployments. Make sure to 
> > call from instanceIsActive() - just as restoreTypeSystem() is called.

instanceIsActive() is not the right method for invoking checkPatches() method. 
Moved the method outside the non HA validation check.


- Sarath Kumar


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51939/#review149854
---


On Sept. 22, 2016, 12:34 a.m., Sarath Kumar Subramanian wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51939/
> ---
> 
> (Updated Sept. 22, 2016, 12:34 a.m.)
> 
> 
> Review request for atlas, Madhan Neethiraj, Shwetha GS, and Suma Shivaprasad.
> 
> 
> Bugs: ATLAS-1174
> https://issues.apache.org/jira/browse/ATLAS-1174
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> 1. Introduce "version" attribute to all types in the type-system, this helps 
> to track changes made to the default types (hive, sqoop, falcon and storm 
> types) and user created types. If version is not mentioned during creation of 
> a type, default version "1.0" is assigned (optional attribute).
> 2. Using the version attributed for types, introduce a patch framework for 
> type system. This framework applies patches to a type using its version 
> number and can be used during upgrade - add new attributes to an existing 
> types and it will be run during atlas startup.
> The sequence of steps:
> a. During atlas startup, check $ATLAS_HOME/models/patches directory for any 
> available patch files (json files). If there any patch files handle them.
> b. Sample patch json file looks like:
> {
> "patches": [
> { 
> "action": "ADD_ATTRIBUTE",
> "typeName": "hive_column",
> "applyToVersion": "1.0",
> "updateToVersion": "2.0",
> "actionParams": [
> { "name": "position", "dataTypeName": "int", "multiplicity": "optional", 
> "isComposite": false, "isUnique": false, "isIndexable": false, 
> "reverseAttributeName": null }
> ]
> } ]
> }
> c. The framework updates the type in "typeName" for the matching version 
> number and after applying the patch, update the version to the one mentioned 
> in "updateToVersion"
> d. The json file can have more than one action (array of actions).
> e. There can be multiple patch json files in the directory and are applied in 
> the sort order of the filename. eg:
> 001-hive_column_add_position.json
> 002-hive_column_add_anotherattribute.json
> 
> 
> Diffs
> -
> 
>   common/src/main/java/org/apache/atlas/AtlasConstants.java 17ffbd7 
>   common/src/main/java/org/apache/atlas/repository/Constants.java d7f9c89 
>   
> repository/src/main/java/org/apache/atlas/repository/typestore/GraphBackedTypeStore.java
>  a94d157 
>   
> repository/src/main/java/org/apache/atlas/services/AtlasTypeAttributePatch.java
>  PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/services/AtlasTypePatch.java 
> PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java
>  6a937f4 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/AbstractDataType.java
>  fad091d 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/ClassType.java 
> c56987a 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/EnumType.java 
> bdd0a13 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/EnumTypeDefinition.java
>  6340615 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/HierarchicalType.java
>  7224699 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/HierarchicalTypeDefinition.java
>  9a299f0 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/IDataType.java 
> 85ddee7 
>   

Re: Review Request 51939: Framework to apply updates to types in the type-system

[Sarath Kumar Subramanian]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51939/
---

(Updated Sept. 22, 2016, 12:34 a.m.)


Review request for atlas, Madhan Neethiraj, Shwetha GS, and Suma Shivaprasad.


Changes
---

removed all json operations to change typedef and used appropriate pojo 
classes, moved check for patches method in AtlasTypeAttributePatch, 
incorporated madhan and shwetha's comments.


Bugs: ATLAS-1174
https://issues.apache.org/jira/browse/ATLAS-1174


Repository: atlas


Description
---

1. Introduce "version" attribute to all types in the type-system, this helps to 
track changes made to the default types (hive, sqoop, falcon and storm types) 
and user created types. If version is not mentioned during creation of a type, 
default version "1.0" is assigned (optional attribute).
2. Using the version attributed for types, introduce a patch framework for type 
system. This framework applies patches to a type using its version number and 
can be used during upgrade - add new attributes to an existing types and it 
will be run during atlas startup.
The sequence of steps:
a. During atlas startup, check $ATLAS_HOME/models/patches directory for any 
available patch files (json files). If there any patch files handle them.
b. Sample patch json file looks like:
{
"patches": [
{ 
"action": "ADD_ATTRIBUTE",
"typeName": "hive_column",
"applyToVersion": "1.0",
"updateToVersion": "2.0",
"actionParams": [
{ "name": "position", "dataTypeName": "int", "multiplicity": "optional", 
"isComposite": false, "isUnique": false, "isIndexable": false, 
"reverseAttributeName": null }
]
} ]
}
c. The framework updates the type in "typeName" for the matching version number 
and after applying the patch, update the version to the one mentioned in 
"updateToVersion"
d. The json file can have more than one action (array of actions).
e. There can be multiple patch json files in the directory and are applied in 
the sort order of the filename. eg:
001-hive_column_add_position.json
002-hive_column_add_anotherattribute.json


Diffs (updated)
-

  common/src/main/java/org/apache/atlas/AtlasConstants.java 17ffbd7 
  common/src/main/java/org/apache/atlas/repository/Constants.java d7f9c89 
  
repository/src/main/java/org/apache/atlas/repository/typestore/GraphBackedTypeStore.java
 a94d157 
  
repository/src/main/java/org/apache/atlas/services/AtlasTypeAttributePatch.java 
PRE-CREATION 
  repository/src/main/java/org/apache/atlas/services/AtlasTypePatch.java 
PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java 
6a937f4 
  
typesystem/src/main/java/org/apache/atlas/typesystem/types/AbstractDataType.java
 fad091d 
  typesystem/src/main/java/org/apache/atlas/typesystem/types/ClassType.java 
c56987a 
  typesystem/src/main/java/org/apache/atlas/typesystem/types/EnumType.java 
bdd0a13 
  
typesystem/src/main/java/org/apache/atlas/typesystem/types/EnumTypeDefinition.java
 6340615 
  
typesystem/src/main/java/org/apache/atlas/typesystem/types/HierarchicalType.java
 7224699 
  
typesystem/src/main/java/org/apache/atlas/typesystem/types/HierarchicalTypeDefinition.java
 9a299f0 
  typesystem/src/main/java/org/apache/atlas/typesystem/types/IDataType.java 
85ddee7 
  typesystem/src/main/java/org/apache/atlas/typesystem/types/StructType.java 
6f40c1d 
  
typesystem/src/main/java/org/apache/atlas/typesystem/types/StructTypeDefinition.java
 f1ce1b7 
  typesystem/src/main/java/org/apache/atlas/typesystem/types/TraitType.java 
f23bf5b 
  typesystem/src/main/java/org/apache/atlas/typesystem/types/TypeSystem.java 
70ba89b 
  
typesystem/src/main/java/org/apache/atlas/typesystem/types/utils/TypesUtil.java 
ef8448d 
  
typesystem/src/main/scala/org/apache/atlas/typesystem/json/TypesSerialization.scala
 5618938 

Diff: https://reviews.apache.org/r/51939/diff/


Testing
---


Thanks,

Sarath Kumar Subramanian

Re: Review Request 51939: Framework to apply updates to types in the type-system

[Madhan Neethiraj]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51939/#review149854
---




repository/src/main/java/org/apache/atlas/services/AtlasTypeAttributePatch.java 
(line 61)


"attributeDefinitions" is an array in structType and classType. Lets keep 
this consistent here. Please refer to 
./addons/hive-bridge/target/models/hive_model.json.



repository/src/main/java/org/apache/atlas/services/AtlasTypeAttributePatch.java 
(line 87)


Consider adding "currentVersion == null ||".



repository/src/main/java/org/apache/atlas/services/AtlasTypeAttributePatch.java 
(line 103)


Consider returning PatchResult() with the error message from exception 
blocks.

It will be a good practice to minimize the number of returns from a method. 
I generally define a variable named "ret" at the entry of the method; set it to 
appropriate value in the body; and return from the end of the method. Make it 
easier to read and add debug tracing.



repository/src/main/java/org/apache/atlas/services/AtlasTypeAttributePatch.java 
(line 122)


Since this loop iterates for each attribute  in the patch, should this call 
only check for presense of the current patch attribute?



repository/src/main/java/org/apache/atlas/services/AtlasTypeAttributePatch.java 
(line 125)


Consider rewritting this 'if' as below, for better readability:

if(! typeContainsPatchAttrs) {
  if(action == PatchAction.ADD_ATTRIBUTES) {
// add the atttribute
  } else {
LOG.warn(action + ": invalid action for non-existing attribute. 
Ignored.");
  }
} else {
  if(action == PatchAction.UPDATE_ATTRIBUTES) {
// update the atttribute
  } else if(action == PatchAction.DELETE_ATTRIBUTES) {
// delete the atttribute
  } else {
LOG.warn(action + ": invalid action for already existing attribute. 
Ignored.");
  }
}



repository/src/main/java/org/apache/atlas/services/AtlasTypeAttributePatch.java 
(line 133)


"to the type" ==> "in type"



repository/src/main/java/org/apache/atlas/services/AtlasTypeAttributePatch.java 
(line 137)


"to the type" ==> "from type"



repository/src/main/java/org/apache/atlas/services/AtlasTypeAttributePatch.java 
(line 162)


Consider updating the error messagea to include more details (like 
typename): 

"error in getting type '" + patchData.getTypeName() + "' from typesystem"

Please review other error messages as well.



repository/src/main/java/org/apache/atlas/services/AtlasTypeAttributePatch.java 
(line 186)


Is it necessary to convert to JSON? Wouldn't iterating on currentAttributes 
work?

if(currentAttributes != null) {
  for(AttributeDefintion attributeDef : currentAttributes) {
s.add(attributeDef.name);
  }
}



repository/src/main/java/org/apache/atlas/services/AtlasTypePatch.java (line 77)


Consider adding attributeDefinitions as a first class attribute - to save 
from having to tranform from/to JSON in many places. When we add support for 
other type of patches (like types..), appropraite members can be added here.

class PatchData {
  private String   action;
  private String   typeName;
  ..
  private List attributeDefinitions;
  private Map  params;
}



repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java 
(line 171)


checkForPatches() call will be missed in HA deployments. Make sure to call 
from instanceIsActive() - just as restoreTypeSystem() is called.



repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java 
(line 190)


patchFile can be null. Please update to handle this condition.



repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java 
(line 220)


Supported actions: ADD_ATTRIBUTE, UPDATE_ATTRIBUTE, DELETE_ATTRIBUTE



repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java 
(line 225)


result could be null. Please update to handle this condition.

Re: Review Request 51939: Framework to apply updates to types in the type-system

[Shwetha GS]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51939/#review149811
---




repository/src/main/java/org/apache/atlas/services/AtlasTypeAttributePatch.java 
(line 42)


This class works on jsons directly which is error prone. Use POJOs instead 
- HierarchicalTypeDefinition, StructTypeDefinition



repository/src/main/java/org/apache/atlas/services/AtlasTypeAttributePatch.java 
(line 154)


== doesn't work on strings



repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java 
(line 177)


Move this to AtlasTypeAttributePatch


Add tests

1. Currently, the model files(like hive_model.json) are auto generated from 
model definitions defined in java(like HiveDataModelGenerator). The patch files 
in this case has to be hand coded which is error prone
2. For completeness, readability and debuggability, the type update has to be 
done in the corresponding model definitions like HiveDataModelGenerator. So, 
same data will be in two places and the model definitions and the patch files 
can go out of sync
3. Since the model definitions(like HiveDataModelGenerator) will be updated 
anyways, if we modify ReservedTypesRegistrar to do type update instead of type 
create, the type updates will automatically be taken care with the same model 
json. So, model update patches are not necessary then
3. This jira doesn't implement type versioning - doesn't have support for 
storing multiple versions of the type. But it maintains the version of the 
latest type definition which I think is useful for debugging, to know the 
version of type that the server knows. Can we maintain this info in 
HiveDataModelGenerator itself, and hence will be part of hive_model.json

- Shwetha GS


On Sept. 21, 2016, 7:51 a.m., Sarath Kumar Subramanian wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51939/
> ---
> 
> (Updated Sept. 21, 2016, 7:51 a.m.)
> 
> 
> Review request for atlas, Madhan Neethiraj, Shwetha GS, and Suma Shivaprasad.
> 
> 
> Bugs: ATLAS-1174
> https://issues.apache.org/jira/browse/ATLAS-1174
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> 1. Introduce "version" attribute to all types in the type-system, this helps 
> to track changes made to the default types (hive, sqoop, falcon and storm 
> types) and user created types. If version is not mentioned during creation of 
> a type, default version "1.0" is assigned (optional attribute).
> 2. Using the version attributed for types, introduce a patch framework for 
> type system. This framework applies patches to a type using its version 
> number and can be used during upgrade - add new attributes to an existing 
> types and it will be run during atlas startup.
> The sequence of steps:
> a. During atlas startup, check $ATLAS_HOME/models/patches directory for any 
> available patch files (json files). If there any patch files handle them.
> b. Sample patch json file looks like:
> {
> "patches": [
> { 
> "action": "ADD_ATTRIBUTE",
> "typeName": "hive_column",
> "applyToVersion": "1.0",
> "updateToVersion": "2.0",
> "actionParams": [
> { "name": "position", "dataTypeName": "int", "multiplicity": "optional", 
> "isComposite": false, "isUnique": false, "isIndexable": false, 
> "reverseAttributeName": null }
> ]
> } ]
> }
> c. The framework updates the type in "typeName" for the matching version 
> number and after applying the patch, update the version to the one mentioned 
> in "updateToVersion"
> d. The json file can have more than one action (array of actions).
> e. There can be multiple patch json files in the directory and are applied in 
> the sort order of the filename. eg:
> 001-hive_column_add_position.json
> 002-hive_column_add_anotherattribute.json
> 
> 
> Diffs
> -
> 
>   common/src/main/java/org/apache/atlas/AtlasConstants.java 17ffbd7 
>   common/src/main/java/org/apache/atlas/repository/Constants.java d7f9c89 
>   
> repository/src/main/java/org/apache/atlas/repository/typestore/GraphBackedTypeStore.java
>  a94d157 
>   
> repository/src/main/java/org/apache/atlas/services/AtlasTypeAttributePatch.java
>  PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/services/AtlasTypePatch.java 
> PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java
>  6a937f4 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/AbstractDataType.java
>  fad091d 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/ClassType.java 
> c56987a 
>   

Re: Review Request 51939: Framework to apply updates to types in the type-system

[Shwetha GS]

> On Sept. 20, 2016, 1:11 p.m., David Radley wrote:
> > It looks like this is a way of updating the versions of types using a local 
> > file on the Atlas server. I think we should be SaaS and expose this sort of 
> > functionality primarily as a REST API, otherwise this could become a 
> > barrier to integration.
> 
> Sarath Kumar Subramanian wrote:
> David, the goal of this framework is to apply patches during upgrade 
> scenarios - when user has changed the shape of a type (added new attributes, 
> new supertypes) or to update an existing type. The patch files will be 
> checked and applied during atlas startup, this will avoid dealing with 
> multiple patch scripts for upgrading multiple types. Currrently a type can be 
> updated using REST (PUT - /api/atlas/types) by including the type definition 
> in request payload. We can add only optional attributes, so instances 
> depending on old type will not be broken.
> 
> I can add a dedicated REST layer to this once we have ATLAS-1171 ready.
> 
> David Radley wrote:
> Hi Sumar, Thank you for explaining. I assume the patch files are used 
> once for the upgrade. This is a batch orientated way of changing the shapes 
> of types - which I can see could be of use in the short term, especially due 
> to the exising REST restrictions . My concern is that this approach, which 
> introduces versioning, might make introducing the SaaS runtime REST approach 
> more difficult. Also if we put effort into a batch process that we use to 
> upgrade type shapes - we may not put effort into the REST API that products 
> integrating with Atlas will use at runtime.  I assume the REST Implmentaiton 
> would be to relax the restriction to only allow updates of optional 
> attributes and to add the concept of versioning to the API. Are you planning 
> to do a dedicated REST implementation soon? all the best, David.

Type update is already available through REST API. The API takes the complete 
new type definition and automatically figures out what has changed and 
validates the change


- Shwetha


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51939/#review149652
---


On Sept. 21, 2016, 7:51 a.m., Sarath Kumar Subramanian wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51939/
> ---
> 
> (Updated Sept. 21, 2016, 7:51 a.m.)
> 
> 
> Review request for atlas, Madhan Neethiraj, Shwetha GS, and Suma Shivaprasad.
> 
> 
> Bugs: ATLAS-1174
> https://issues.apache.org/jira/browse/ATLAS-1174
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> 1. Introduce "version" attribute to all types in the type-system, this helps 
> to track changes made to the default types (hive, sqoop, falcon and storm 
> types) and user created types. If version is not mentioned during creation of 
> a type, default version "1.0" is assigned (optional attribute).
> 2. Using the version attributed for types, introduce a patch framework for 
> type system. This framework applies patches to a type using its version 
> number and can be used during upgrade - add new attributes to an existing 
> types and it will be run during atlas startup.
> The sequence of steps:
> a. During atlas startup, check $ATLAS_HOME/models/patches directory for any 
> available patch files (json files). If there any patch files handle them.
> b. Sample patch json file looks like:
> {
> "patches": [
> { 
> "action": "ADD_ATTRIBUTE",
> "typeName": "hive_column",
> "applyToVersion": "1.0",
> "updateToVersion": "2.0",
> "actionParams": [
> { "name": "position", "dataTypeName": "int", "multiplicity": "optional", 
> "isComposite": false, "isUnique": false, "isIndexable": false, 
> "reverseAttributeName": null }
> ]
> } ]
> }
> c. The framework updates the type in "typeName" for the matching version 
> number and after applying the patch, update the version to the one mentioned 
> in "updateToVersion"
> d. The json file can have more than one action (array of actions).
> e. There can be multiple patch json files in the directory and are applied in 
> the sort order of the filename. eg:
> 001-hive_column_add_position.json
> 002-hive_column_add_anotherattribute.json
> 
> 
> Diffs
> -
> 
>   common/src/main/java/org/apache/atlas/AtlasConstants.java 17ffbd7 
>   common/src/main/java/org/apache/atlas/repository/Constants.java d7f9c89 
>   
> repository/src/main/java/org/apache/atlas/repository/typestore/GraphBackedTypeStore.java
>  a94d157 
>   
> repository/src/main/java/org/apache/atlas/services/AtlasTypeAttributePatch.java
>  PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/services/AtlasTypePatch.java 
> PRE-CREATION 
>   
> 

Re: Review Request 51939: Framework to apply updates to types in the type-system

[David Radley]

> On Sept. 20, 2016, 1:11 p.m., David Radley wrote:
> > It looks like this is a way of updating the versions of types using a local 
> > file on the Atlas server. I think we should be SaaS and expose this sort of 
> > functionality primarily as a REST API, otherwise this could become a 
> > barrier to integration.
> 
> Sarath Kumar Subramanian wrote:
> David, the goal of this framework is to apply patches during upgrade 
> scenarios - when user has changed the shape of a type (added new attributes, 
> new supertypes) or to update an existing type. The patch files will be 
> checked and applied during atlas startup, this will avoid dealing with 
> multiple patch scripts for upgrading multiple types. Currrently a type can be 
> updated using REST (PUT - /api/atlas/types) by including the type definition 
> in request payload. We can add only optional attributes, so instances 
> depending on old type will not be broken.
> 
> I can add a dedicated REST layer to this once we have ATLAS-1171 ready.

Hi Sumar, Thank you for explaining. I assume the patch files are used once for 
the upgrade. This is a batch orientated way of changing the shapes of types - 
which I can see could be of use in the short term, especially due to the 
exising REST restrictions . My concern is that this approach, which introduces 
versioning, might make introducing the SaaS runtime REST approach more 
difficult. Also if we put effort into a batch process that we use to upgrade 
type shapes - we may not put effort into the REST API that products integrating 
with Atlas will use at runtime.  I assume the REST Implmentaiton would be to 
relax the restriction to only allow updates of optional attributes and to add 
the concept of versioning to the API. Are you planning to do a dedicated REST 
implementation soon? all the best, David.


- David


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51939/#review149652
---


On Sept. 21, 2016, 7:51 a.m., Sarath Kumar Subramanian wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51939/
> ---
> 
> (Updated Sept. 21, 2016, 7:51 a.m.)
> 
> 
> Review request for atlas, Madhan Neethiraj, Shwetha GS, and Suma Shivaprasad.
> 
> 
> Bugs: ATLAS-1174
> https://issues.apache.org/jira/browse/ATLAS-1174
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> 1. Introduce "version" attribute to all types in the type-system, this helps 
> to track changes made to the default types (hive, sqoop, falcon and storm 
> types) and user created types. If version is not mentioned during creation of 
> a type, default version "1.0" is assigned (optional attribute).
> 2. Using the version attributed for types, introduce a patch framework for 
> type system. This framework applies patches to a type using its version 
> number and can be used during upgrade - add new attributes to an existing 
> types and it will be run during atlas startup.
> The sequence of steps:
> a. During atlas startup, check $ATLAS_HOME/models/patches directory for any 
> available patch files (json files). If there any patch files handle them.
> b. Sample patch json file looks like:
> {
> "patches": [
> { 
> "action": "ADD_ATTRIBUTE",
> "typeName": "hive_column",
> "applyToVersion": "1.0",
> "updateToVersion": "2.0",
> "actionParams": [
> { "name": "position", "dataTypeName": "int", "multiplicity": "optional", 
> "isComposite": false, "isUnique": false, "isIndexable": false, 
> "reverseAttributeName": null }
> ]
> } ]
> }
> c. The framework updates the type in "typeName" for the matching version 
> number and after applying the patch, update the version to the one mentioned 
> in "updateToVersion"
> d. The json file can have more than one action (array of actions).
> e. There can be multiple patch json files in the directory and are applied in 
> the sort order of the filename. eg:
> 001-hive_column_add_position.json
> 002-hive_column_add_anotherattribute.json
> 
> 
> Diffs
> -
> 
>   common/src/main/java/org/apache/atlas/AtlasConstants.java 17ffbd7 
>   common/src/main/java/org/apache/atlas/repository/Constants.java d7f9c89 
>   
> repository/src/main/java/org/apache/atlas/repository/typestore/GraphBackedTypeStore.java
>  a94d157 
>   
> repository/src/main/java/org/apache/atlas/services/AtlasTypeAttributePatch.java
>  PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/services/AtlasTypePatch.java 
> PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java
>  6a937f4 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/AbstractDataType.java
>  fad091d 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/ClassType.java 
> 

Re: Review Request 51939: Framework to apply updates to types in the type-system

[Sarath Kumar Subramanian]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51939/
---

(Updated Sept. 21, 2016, 7:51 a.m.)


Review request for atlas, Madhan Neethiraj, Shwetha GS, and Suma Shivaprasad.


Changes
---

please ignore the previous diff(included patch file in the commit). Added 
support to update and delete attributes in addition to add, addressed suma's 
code review suggestions.


Bugs: ATLAS-1174
https://issues.apache.org/jira/browse/ATLAS-1174


Repository: atlas


Description
---

1. Introduce "version" attribute to all types in the type-system, this helps to 
track changes made to the default types (hive, sqoop, falcon and storm types) 
and user created types. If version is not mentioned during creation of a type, 
default version "1.0" is assigned (optional attribute).
2. Using the version attributed for types, introduce a patch framework for type 
system. This framework applies patches to a type using its version number and 
can be used during upgrade - add new attributes to an existing types and it 
will be run during atlas startup.
The sequence of steps:
a. During atlas startup, check $ATLAS_HOME/models/patches directory for any 
available patch files (json files). If there any patch files handle them.
b. Sample patch json file looks like:
{
"patches": [
{ 
"action": "ADD_ATTRIBUTE",
"typeName": "hive_column",
"applyToVersion": "1.0",
"updateToVersion": "2.0",
"actionParams": [
{ "name": "position", "dataTypeName": "int", "multiplicity": "optional", 
"isComposite": false, "isUnique": false, "isIndexable": false, 
"reverseAttributeName": null }
]
} ]
}
c. The framework updates the type in "typeName" for the matching version number 
and after applying the patch, update the version to the one mentioned in 
"updateToVersion"
d. The json file can have more than one action (array of actions).
e. There can be multiple patch json files in the directory and are applied in 
the sort order of the filename. eg:
001-hive_column_add_position.json
002-hive_column_add_anotherattribute.json


Diffs (updated)
-

  common/src/main/java/org/apache/atlas/AtlasConstants.java 17ffbd7 
  common/src/main/java/org/apache/atlas/repository/Constants.java d7f9c89 
  
repository/src/main/java/org/apache/atlas/repository/typestore/GraphBackedTypeStore.java
 a94d157 
  
repository/src/main/java/org/apache/atlas/services/AtlasTypeAttributePatch.java 
PRE-CREATION 
  repository/src/main/java/org/apache/atlas/services/AtlasTypePatch.java 
PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java 
6a937f4 
  
typesystem/src/main/java/org/apache/atlas/typesystem/types/AbstractDataType.java
 fad091d 
  typesystem/src/main/java/org/apache/atlas/typesystem/types/ClassType.java 
c56987a 
  typesystem/src/main/java/org/apache/atlas/typesystem/types/EnumType.java 
bdd0a13 
  
typesystem/src/main/java/org/apache/atlas/typesystem/types/EnumTypeDefinition.java
 6340615 
  
typesystem/src/main/java/org/apache/atlas/typesystem/types/HierarchicalType.java
 7224699 
  
typesystem/src/main/java/org/apache/atlas/typesystem/types/HierarchicalTypeDefinition.java
 9a299f0 
  typesystem/src/main/java/org/apache/atlas/typesystem/types/IDataType.java 
85ddee7 
  typesystem/src/main/java/org/apache/atlas/typesystem/types/StructType.java 
6f40c1d 
  
typesystem/src/main/java/org/apache/atlas/typesystem/types/StructTypeDefinition.java
 f1ce1b7 
  typesystem/src/main/java/org/apache/atlas/typesystem/types/TraitType.java 
f23bf5b 
  typesystem/src/main/java/org/apache/atlas/typesystem/types/TypeSystem.java 
70ba89b 
  
typesystem/src/main/java/org/apache/atlas/typesystem/types/utils/TypesUtil.java 
ef8448d 
  
typesystem/src/main/scala/org/apache/atlas/typesystem/json/TypesSerialization.scala
 5618938 

Diff: https://reviews.apache.org/r/51939/diff/


Testing
---


Thanks,

Sarath Kumar Subramanian

Re: Review Request 51939: Framework to apply updates to types in the type-system

[Sarath Kumar Subramanian]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51939/
---

(Updated Sept. 21, 2016, 7:21 a.m.)


Review request for atlas, Madhan Neethiraj, Shwetha GS, and Suma Shivaprasad.


Changes
---

added support to update and delete attributes in addition to add, addressed 
suma's code review suggestions.


Bugs: ATLAS-1174
https://issues.apache.org/jira/browse/ATLAS-1174


Repository: atlas


Description
---

1. Introduce "version" attribute to all types in the type-system, this helps to 
track changes made to the default types (hive, sqoop, falcon and storm types) 
and user created types. If version is not mentioned during creation of a type, 
default version "1.0" is assigned (optional attribute).
2. Using the version attributed for types, introduce a patch framework for type 
system. This framework applies patches to a type using its version number and 
can be used during upgrade - add new attributes to an existing types and it 
will be run during atlas startup.
The sequence of steps:
a. During atlas startup, check $ATLAS_HOME/models/patches directory for any 
available patch files (json files). If there any patch files handle them.
b. Sample patch json file looks like:
{
"patches": [
{ 
"action": "ADD_ATTRIBUTE",
"typeName": "hive_column",
"applyToVersion": "1.0",
"updateToVersion": "2.0",
"actionParams": [
{ "name": "position", "dataTypeName": "int", "multiplicity": "optional", 
"isComposite": false, "isUnique": false, "isIndexable": false, 
"reverseAttributeName": null }
]
} ]
}
c. The framework updates the type in "typeName" for the matching version number 
and after applying the patch, update the version to the one mentioned in 
"updateToVersion"
d. The json file can have more than one action (array of actions).
e. There can be multiple patch json files in the directory and are applied in 
the sort order of the filename. eg:
001-hive_column_add_position.json
002-hive_column_add_anotherattribute.json


Diffs (updated)
-

  ATLAS-1171.7.patch PRE-CREATION 
  common/src/main/java/org/apache/atlas/AtlasConstants.java 17ffbd7 
  common/src/main/java/org/apache/atlas/repository/Constants.java d7f9c89 
  
repository/src/main/java/org/apache/atlas/repository/typestore/GraphBackedTypeStore.java
 a94d157 
  
repository/src/main/java/org/apache/atlas/services/AtlasTypeAttributePatch.java 
PRE-CREATION 
  repository/src/main/java/org/apache/atlas/services/AtlasTypePatch.java 
PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java 
6a937f4 
  
typesystem/src/main/java/org/apache/atlas/typesystem/types/AbstractDataType.java
 fad091d 
  typesystem/src/main/java/org/apache/atlas/typesystem/types/ClassType.java 
c56987a 
  typesystem/src/main/java/org/apache/atlas/typesystem/types/EnumType.java 
bdd0a13 
  
typesystem/src/main/java/org/apache/atlas/typesystem/types/EnumTypeDefinition.java
 6340615 
  
typesystem/src/main/java/org/apache/atlas/typesystem/types/HierarchicalType.java
 7224699 
  
typesystem/src/main/java/org/apache/atlas/typesystem/types/HierarchicalTypeDefinition.java
 9a299f0 
  typesystem/src/main/java/org/apache/atlas/typesystem/types/IDataType.java 
85ddee7 
  typesystem/src/main/java/org/apache/atlas/typesystem/types/StructType.java 
6f40c1d 
  
typesystem/src/main/java/org/apache/atlas/typesystem/types/StructTypeDefinition.java
 f1ce1b7 
  typesystem/src/main/java/org/apache/atlas/typesystem/types/TraitType.java 
f23bf5b 
  typesystem/src/main/java/org/apache/atlas/typesystem/types/TypeSystem.java 
70ba89b 
  typesystem/src/main/java/org/apache/atlas/typesystem/types/TypeUtils.java 
f5c2ce9 
  
typesystem/src/main/java/org/apache/atlas/typesystem/types/utils/TypesUtil.java 
ef8448d 
  
typesystem/src/main/scala/org/apache/atlas/typesystem/json/TypesSerialization.scala
 5618938 

Diff: https://reviews.apache.org/r/51939/diff/


Testing
---


Thanks,

Sarath Kumar Subramanian

Re: Review Request 51939: Framework to apply updates to types in the type-system

[Suma Shivaprasad]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51939/#review149727
---




typesystem/src/main/java/org/apache/atlas/typesystem/types/TypeSystem.java 
(line 314)


We would need to audit types as well to 
know what was the original type definition that was updated and the patch 
that was applied.


- Suma Shivaprasad


On Sept. 19, 2016, 6:10 p.m., Sarath Kumar Subramanian wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51939/
> ---
> 
> (Updated Sept. 19, 2016, 6:10 p.m.)
> 
> 
> Review request for atlas, Madhan Neethiraj, Shwetha GS, and Suma Shivaprasad.
> 
> 
> Bugs: ATLAS-1174
> https://issues.apache.org/jira/browse/ATLAS-1174
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> 1. Introduce "version" attribute to all types in the type-system, this helps 
> to track changes made to the default types (hive, sqoop, falcon and storm 
> types) and user created types. If version is not mentioned during creation of 
> a type, default version "1.0" is assigned (optional attribute).
> 2. Using the version attributed for types, introduce a patch framework for 
> type system. This framework applies patches to a type using its version 
> number and can be used during upgrade - add new attributes to an existing 
> types and it will be run during atlas startup.
> The sequence of steps:
> a. During atlas startup, check $ATLAS_HOME/models/patches directory for any 
> available patch files (json files). If there any patch files handle them.
> b. Sample patch json file looks like:
> {
> "patches": [
> { 
> "action": "ADD_ATTRIBUTE",
> "typeName": "hive_column",
> "applyToVersion": "1.0",
> "updateToVersion": "2.0",
> "actionParams": [
> { "name": "position", "dataTypeName": "int", "multiplicity": "optional", 
> "isComposite": false, "isUnique": false, "isIndexable": false, 
> "reverseAttributeName": null }
> ]
> } ]
> }
> c. The framework updates the type in "typeName" for the matching version 
> number and after applying the patch, update the version to the one mentioned 
> in "updateToVersion"
> d. The json file can have more than one action (array of actions).
> e. There can be multiple patch json files in the directory and are applied in 
> the sort order of the filename. eg:
> 001-hive_column_add_position.json
> 002-hive_column_add_anotherattribute.json
> 
> 
> Diffs
> -
> 
>   common/src/main/java/org/apache/atlas/repository/Constants.java d7f9c89 
>   
> repository/src/main/java/org/apache/atlas/repository/typestore/GraphBackedTypeStore.java
>  a94d157 
>   repository/src/main/java/org/apache/atlas/services/AddAttributePatch.java 
> PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/services/AtlasTypePatch.java 
> PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java
>  6a937f4 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/AbstractDataType.java
>  fad091d 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/ClassType.java 
> c56987a 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/EnumType.java 
> bdd0a13 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/EnumTypeDefinition.java
>  6340615 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/HierarchicalType.java
>  7224699 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/HierarchicalTypeDefinition.java
>  9a299f0 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/IDataType.java 
> 85ddee7 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/StructType.java 
> 6f40c1d 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/StructTypeDefinition.java
>  f1ce1b7 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/TraitType.java 
> f23bf5b 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/TypeSystem.java 
> 70ba89b 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/utils/TypesUtil.java
>  ef8448d 
>   
> typesystem/src/main/scala/org/apache/atlas/typesystem/json/TypesSerialization.scala
>  5618938 
> 
> Diff: https://reviews.apache.org/r/51939/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sarath Kumar Subramanian
> 
>

Re: Review Request 51939: Framework to apply updates to types in the type-system

[Sarath Kumar Subramanian]

> On Sept. 20, 2016, 4:15 a.m., Suma Shivaprasad wrote:
> > repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java,
> >  line 191
> > 
> >
> > it would make more sense to sort by date instead of file name?

Suma, if the patch files are sorted my name (each patch file starting with 
001-xxx, 002-yyy), we can be sure the patch will be applied in order. Sorting 
by date can mess up the order if the files are not laid in order.


- Sarath Kumar


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51939/#review149608
---


On Sept. 19, 2016, 6:10 p.m., Sarath Kumar Subramanian wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51939/
> ---
> 
> (Updated Sept. 19, 2016, 6:10 p.m.)
> 
> 
> Review request for atlas, Madhan Neethiraj, Shwetha GS, and Suma Shivaprasad.
> 
> 
> Bugs: ATLAS-1174
> https://issues.apache.org/jira/browse/ATLAS-1174
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> 1. Introduce "version" attribute to all types in the type-system, this helps 
> to track changes made to the default types (hive, sqoop, falcon and storm 
> types) and user created types. If version is not mentioned during creation of 
> a type, default version "1.0" is assigned (optional attribute).
> 2. Using the version attributed for types, introduce a patch framework for 
> type system. This framework applies patches to a type using its version 
> number and can be used during upgrade - add new attributes to an existing 
> types and it will be run during atlas startup.
> The sequence of steps:
> a. During atlas startup, check $ATLAS_HOME/models/patches directory for any 
> available patch files (json files). If there any patch files handle them.
> b. Sample patch json file looks like:
> {
> "patches": [
> { 
> "action": "ADD_ATTRIBUTE",
> "typeName": "hive_column",
> "applyToVersion": "1.0",
> "updateToVersion": "2.0",
> "actionParams": [
> { "name": "position", "dataTypeName": "int", "multiplicity": "optional", 
> "isComposite": false, "isUnique": false, "isIndexable": false, 
> "reverseAttributeName": null }
> ]
> } ]
> }
> c. The framework updates the type in "typeName" for the matching version 
> number and after applying the patch, update the version to the one mentioned 
> in "updateToVersion"
> d. The json file can have more than one action (array of actions).
> e. There can be multiple patch json files in the directory and are applied in 
> the sort order of the filename. eg:
> 001-hive_column_add_position.json
> 002-hive_column_add_anotherattribute.json
> 
> 
> Diffs
> -
> 
>   common/src/main/java/org/apache/atlas/repository/Constants.java d7f9c89 
>   
> repository/src/main/java/org/apache/atlas/repository/typestore/GraphBackedTypeStore.java
>  a94d157 
>   repository/src/main/java/org/apache/atlas/services/AddAttributePatch.java 
> PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/services/AtlasTypePatch.java 
> PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java
>  6a937f4 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/AbstractDataType.java
>  fad091d 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/ClassType.java 
> c56987a 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/EnumType.java 
> bdd0a13 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/EnumTypeDefinition.java
>  6340615 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/HierarchicalType.java
>  7224699 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/HierarchicalTypeDefinition.java
>  9a299f0 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/IDataType.java 
> 85ddee7 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/StructType.java 
> 6f40c1d 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/StructTypeDefinition.java
>  f1ce1b7 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/TraitType.java 
> f23bf5b 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/TypeSystem.java 
> 70ba89b 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/utils/TypesUtil.java
>  ef8448d 
>   
> typesystem/src/main/scala/org/apache/atlas/typesystem/json/TypesSerialization.scala
>  5618938 
> 
> Diff: https://reviews.apache.org/r/51939/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sarath Kumar Subramanian
> 
>

Re: Review Request 51939: Framework to apply updates to types in the type-system

[Suma Shivaprasad]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51939/#review149703
---




typesystem/src/main/java/org/apache/atlas/typesystem/types/HierarchicalType.java
 (line 62)


DEFAULT_TYPE_VERSION seems to be defined in multiple places. should be 
defined at a single place  and could go to AtlasConstants


- Suma Shivaprasad


On Sept. 19, 2016, 6:10 p.m., Sarath Kumar Subramanian wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51939/
> ---
> 
> (Updated Sept. 19, 2016, 6:10 p.m.)
> 
> 
> Review request for atlas, Madhan Neethiraj, Shwetha GS, and Suma Shivaprasad.
> 
> 
> Bugs: ATLAS-1174
> https://issues.apache.org/jira/browse/ATLAS-1174
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> 1. Introduce "version" attribute to all types in the type-system, this helps 
> to track changes made to the default types (hive, sqoop, falcon and storm 
> types) and user created types. If version is not mentioned during creation of 
> a type, default version "1.0" is assigned (optional attribute).
> 2. Using the version attributed for types, introduce a patch framework for 
> type system. This framework applies patches to a type using its version 
> number and can be used during upgrade - add new attributes to an existing 
> types and it will be run during atlas startup.
> The sequence of steps:
> a. During atlas startup, check $ATLAS_HOME/models/patches directory for any 
> available patch files (json files). If there any patch files handle them.
> b. Sample patch json file looks like:
> {
> "patches": [
> { 
> "action": "ADD_ATTRIBUTE",
> "typeName": "hive_column",
> "applyToVersion": "1.0",
> "updateToVersion": "2.0",
> "actionParams": [
> { "name": "position", "dataTypeName": "int", "multiplicity": "optional", 
> "isComposite": false, "isUnique": false, "isIndexable": false, 
> "reverseAttributeName": null }
> ]
> } ]
> }
> c. The framework updates the type in "typeName" for the matching version 
> number and after applying the patch, update the version to the one mentioned 
> in "updateToVersion"
> d. The json file can have more than one action (array of actions).
> e. There can be multiple patch json files in the directory and are applied in 
> the sort order of the filename. eg:
> 001-hive_column_add_position.json
> 002-hive_column_add_anotherattribute.json
> 
> 
> Diffs
> -
> 
>   common/src/main/java/org/apache/atlas/repository/Constants.java d7f9c89 
>   
> repository/src/main/java/org/apache/atlas/repository/typestore/GraphBackedTypeStore.java
>  a94d157 
>   repository/src/main/java/org/apache/atlas/services/AddAttributePatch.java 
> PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/services/AtlasTypePatch.java 
> PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java
>  6a937f4 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/AbstractDataType.java
>  fad091d 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/ClassType.java 
> c56987a 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/EnumType.java 
> bdd0a13 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/EnumTypeDefinition.java
>  6340615 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/HierarchicalType.java
>  7224699 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/HierarchicalTypeDefinition.java
>  9a299f0 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/IDataType.java 
> 85ddee7 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/StructType.java 
> 6f40c1d 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/StructTypeDefinition.java
>  f1ce1b7 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/TraitType.java 
> f23bf5b 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/TypeSystem.java 
> 70ba89b 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/utils/TypesUtil.java
>  ef8448d 
>   
> typesystem/src/main/scala/org/apache/atlas/typesystem/json/TypesSerialization.scala
>  5618938 
> 
> Diff: https://reviews.apache.org/r/51939/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sarath Kumar Subramanian
> 
>

Re: Review Request 51939: Framework to apply updates to types in the type-system

[Suma Shivaprasad]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51939/#review149702
---




repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java 
(line 234)


we should log how many patch files were suceesfully applied out of the 
total count


- Suma Shivaprasad


On Sept. 19, 2016, 6:10 p.m., Sarath Kumar Subramanian wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51939/
> ---
> 
> (Updated Sept. 19, 2016, 6:10 p.m.)
> 
> 
> Review request for atlas, Madhan Neethiraj, Shwetha GS, and Suma Shivaprasad.
> 
> 
> Bugs: ATLAS-1174
> https://issues.apache.org/jira/browse/ATLAS-1174
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> 1. Introduce "version" attribute to all types in the type-system, this helps 
> to track changes made to the default types (hive, sqoop, falcon and storm 
> types) and user created types. If version is not mentioned during creation of 
> a type, default version "1.0" is assigned (optional attribute).
> 2. Using the version attributed for types, introduce a patch framework for 
> type system. This framework applies patches to a type using its version 
> number and can be used during upgrade - add new attributes to an existing 
> types and it will be run during atlas startup.
> The sequence of steps:
> a. During atlas startup, check $ATLAS_HOME/models/patches directory for any 
> available patch files (json files). If there any patch files handle them.
> b. Sample patch json file looks like:
> {
> "patches": [
> { 
> "action": "ADD_ATTRIBUTE",
> "typeName": "hive_column",
> "applyToVersion": "1.0",
> "updateToVersion": "2.0",
> "actionParams": [
> { "name": "position", "dataTypeName": "int", "multiplicity": "optional", 
> "isComposite": false, "isUnique": false, "isIndexable": false, 
> "reverseAttributeName": null }
> ]
> } ]
> }
> c. The framework updates the type in "typeName" for the matching version 
> number and after applying the patch, update the version to the one mentioned 
> in "updateToVersion"
> d. The json file can have more than one action (array of actions).
> e. There can be multiple patch json files in the directory and are applied in 
> the sort order of the filename. eg:
> 001-hive_column_add_position.json
> 002-hive_column_add_anotherattribute.json
> 
> 
> Diffs
> -
> 
>   common/src/main/java/org/apache/atlas/repository/Constants.java d7f9c89 
>   
> repository/src/main/java/org/apache/atlas/repository/typestore/GraphBackedTypeStore.java
>  a94d157 
>   repository/src/main/java/org/apache/atlas/services/AddAttributePatch.java 
> PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/services/AtlasTypePatch.java 
> PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java
>  6a937f4 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/AbstractDataType.java
>  fad091d 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/ClassType.java 
> c56987a 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/EnumType.java 
> bdd0a13 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/EnumTypeDefinition.java
>  6340615 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/HierarchicalType.java
>  7224699 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/HierarchicalTypeDefinition.java
>  9a299f0 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/IDataType.java 
> 85ddee7 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/StructType.java 
> 6f40c1d 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/StructTypeDefinition.java
>  f1ce1b7 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/TraitType.java 
> f23bf5b 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/TypeSystem.java 
> 70ba89b 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/utils/TypesUtil.java
>  ef8448d 
>   
> typesystem/src/main/scala/org/apache/atlas/typesystem/json/TypesSerialization.scala
>  5618938 
> 
> Diff: https://reviews.apache.org/r/51939/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sarath Kumar Subramanian
> 
>

Re: Review Request 51939: Framework to apply updates to types in the type-system

[Suma Shivaprasad]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51939/#review149700
---




repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java 
(line 231)


this should be ERROR level


- Suma Shivaprasad


On Sept. 19, 2016, 6:10 p.m., Sarath Kumar Subramanian wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51939/
> ---
> 
> (Updated Sept. 19, 2016, 6:10 p.m.)
> 
> 
> Review request for atlas, Madhan Neethiraj, Shwetha GS, and Suma Shivaprasad.
> 
> 
> Bugs: ATLAS-1174
> https://issues.apache.org/jira/browse/ATLAS-1174
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> 1. Introduce "version" attribute to all types in the type-system, this helps 
> to track changes made to the default types (hive, sqoop, falcon and storm 
> types) and user created types. If version is not mentioned during creation of 
> a type, default version "1.0" is assigned (optional attribute).
> 2. Using the version attributed for types, introduce a patch framework for 
> type system. This framework applies patches to a type using its version 
> number and can be used during upgrade - add new attributes to an existing 
> types and it will be run during atlas startup.
> The sequence of steps:
> a. During atlas startup, check $ATLAS_HOME/models/patches directory for any 
> available patch files (json files). If there any patch files handle them.
> b. Sample patch json file looks like:
> {
> "patches": [
> { 
> "action": "ADD_ATTRIBUTE",
> "typeName": "hive_column",
> "applyToVersion": "1.0",
> "updateToVersion": "2.0",
> "actionParams": [
> { "name": "position", "dataTypeName": "int", "multiplicity": "optional", 
> "isComposite": false, "isUnique": false, "isIndexable": false, 
> "reverseAttributeName": null }
> ]
> } ]
> }
> c. The framework updates the type in "typeName" for the matching version 
> number and after applying the patch, update the version to the one mentioned 
> in "updateToVersion"
> d. The json file can have more than one action (array of actions).
> e. There can be multiple patch json files in the directory and are applied in 
> the sort order of the filename. eg:
> 001-hive_column_add_position.json
> 002-hive_column_add_anotherattribute.json
> 
> 
> Diffs
> -
> 
>   common/src/main/java/org/apache/atlas/repository/Constants.java d7f9c89 
>   
> repository/src/main/java/org/apache/atlas/repository/typestore/GraphBackedTypeStore.java
>  a94d157 
>   repository/src/main/java/org/apache/atlas/services/AddAttributePatch.java 
> PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/services/AtlasTypePatch.java 
> PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java
>  6a937f4 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/AbstractDataType.java
>  fad091d 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/ClassType.java 
> c56987a 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/EnumType.java 
> bdd0a13 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/EnumTypeDefinition.java
>  6340615 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/HierarchicalType.java
>  7224699 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/HierarchicalTypeDefinition.java
>  9a299f0 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/IDataType.java 
> 85ddee7 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/StructType.java 
> 6f40c1d 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/StructTypeDefinition.java
>  f1ce1b7 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/TraitType.java 
> f23bf5b 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/TypeSystem.java 
> 70ba89b 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/utils/TypesUtil.java
>  ef8448d 
>   
> typesystem/src/main/scala/org/apache/atlas/typesystem/json/TypesSerialization.scala
>  5618938 
> 
> Diff: https://reviews.apache.org/r/51939/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sarath Kumar Subramanian
> 
>

Re: Review Request 51939: Framework to apply updates to types in the type-system

[Sarath Kumar Subramanian]

> On Sept. 20, 2016, 1:11 p.m., David Radley wrote:
> > It looks like this is a way of updating the versions of types using a local 
> > file on the Atlas server. I think we should be SaaS and expose this sort of 
> > functionality primarily as a REST API, otherwise this could become a 
> > barrier to integration.

David, the goal of this framework is to apply patches during upgrade scenarios 
- when user has changed the shape of a type (added new attributes, new 
supertypes) or to update an existing type. The patch files will be checked and 
applied during atlas startup, this will avoid dealing with multiple patch 
scripts for upgrading multiple types. Currrently a type can be updated using 
REST (PUT - /api/atlas/types) by including the type definition in request 
payload. We can add only optional attributes, so instances depending on old 
type will not be broken.

I can add a dedicated REST layer to this once we have ATLAS-1171 ready.


- Sarath Kumar


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51939/#review149652
---


On Sept. 19, 2016, 6:10 p.m., Sarath Kumar Subramanian wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51939/
> ---
> 
> (Updated Sept. 19, 2016, 6:10 p.m.)
> 
> 
> Review request for atlas, Madhan Neethiraj, Shwetha GS, and Suma Shivaprasad.
> 
> 
> Bugs: ATLAS-1174
> https://issues.apache.org/jira/browse/ATLAS-1174
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> 1. Introduce "version" attribute to all types in the type-system, this helps 
> to track changes made to the default types (hive, sqoop, falcon and storm 
> types) and user created types. If version is not mentioned during creation of 
> a type, default version "1.0" is assigned (optional attribute).
> 2. Using the version attributed for types, introduce a patch framework for 
> type system. This framework applies patches to a type using its version 
> number and can be used during upgrade - add new attributes to an existing 
> types and it will be run during atlas startup.
> The sequence of steps:
> a. During atlas startup, check $ATLAS_HOME/models/patches directory for any 
> available patch files (json files). If there any patch files handle them.
> b. Sample patch json file looks like:
> {
> "patches": [
> { 
> "action": "ADD_ATTRIBUTE",
> "typeName": "hive_column",
> "applyToVersion": "1.0",
> "updateToVersion": "2.0",
> "actionParams": [
> { "name": "position", "dataTypeName": "int", "multiplicity": "optional", 
> "isComposite": false, "isUnique": false, "isIndexable": false, 
> "reverseAttributeName": null }
> ]
> } ]
> }
> c. The framework updates the type in "typeName" for the matching version 
> number and after applying the patch, update the version to the one mentioned 
> in "updateToVersion"
> d. The json file can have more than one action (array of actions).
> e. There can be multiple patch json files in the directory and are applied in 
> the sort order of the filename. eg:
> 001-hive_column_add_position.json
> 002-hive_column_add_anotherattribute.json
> 
> 
> Diffs
> -
> 
>   common/src/main/java/org/apache/atlas/repository/Constants.java d7f9c89 
>   
> repository/src/main/java/org/apache/atlas/repository/typestore/GraphBackedTypeStore.java
>  a94d157 
>   repository/src/main/java/org/apache/atlas/services/AddAttributePatch.java 
> PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/services/AtlasTypePatch.java 
> PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java
>  6a937f4 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/AbstractDataType.java
>  fad091d 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/ClassType.java 
> c56987a 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/EnumType.java 
> bdd0a13 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/EnumTypeDefinition.java
>  6340615 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/HierarchicalType.java
>  7224699 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/HierarchicalTypeDefinition.java
>  9a299f0 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/IDataType.java 
> 85ddee7 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/StructType.java 
> 6f40c1d 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/StructTypeDefinition.java
>  f1ce1b7 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/TraitType.java 
> f23bf5b 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/TypeSystem.java 
> 70ba89b 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/utils/TypesUtil.java
>  

Re: Review Request 51939: Framework to apply updates to types in the type-system

[David Radley]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51939/#review149652
---



It looks like this is a way of updating the versions of types using a local 
file on the Atlas server. I think we should be SaaS and expose this sort of 
functionality primarily as a REST API, otherwise this could become a barrier to 
integration.

- David Radley


On Sept. 19, 2016, 6:10 p.m., Sarath Kumar Subramanian wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51939/
> ---
> 
> (Updated Sept. 19, 2016, 6:10 p.m.)
> 
> 
> Review request for atlas, Madhan Neethiraj, Shwetha GS, and Suma Shivaprasad.
> 
> 
> Bugs: ATLAS-1174
> https://issues.apache.org/jira/browse/ATLAS-1174
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> 1. Introduce "version" attribute to all types in the type-system, this helps 
> to track changes made to the default types (hive, sqoop, falcon and storm 
> types) and user created types. If version is not mentioned during creation of 
> a type, default version "1.0" is assigned (optional attribute).
> 2. Using the version attributed for types, introduce a patch framework for 
> type system. This framework applies patches to a type using its version 
> number and can be used during upgrade - add new attributes to an existing 
> types and it will be run during atlas startup.
> The sequence of steps:
> a. During atlas startup, check $ATLAS_HOME/models/patches directory for any 
> available patch files (json files). If there any patch files handle them.
> b. Sample patch json file looks like:
> {
> "patches": [
> { 
> "action": "ADD_ATTRIBUTE",
> "typeName": "hive_column",
> "applyToVersion": "1.0",
> "updateToVersion": "2.0",
> "actionParams": [
> { "name": "position", "dataTypeName": "int", "multiplicity": "optional", 
> "isComposite": false, "isUnique": false, "isIndexable": false, 
> "reverseAttributeName": null }
> ]
> } ]
> }
> c. The framework updates the type in "typeName" for the matching version 
> number and after applying the patch, update the version to the one mentioned 
> in "updateToVersion"
> d. The json file can have more than one action (array of actions).
> e. There can be multiple patch json files in the directory and are applied in 
> the sort order of the filename. eg:
> 001-hive_column_add_position.json
> 002-hive_column_add_anotherattribute.json
> 
> 
> Diffs
> -
> 
>   common/src/main/java/org/apache/atlas/repository/Constants.java d7f9c89 
>   
> repository/src/main/java/org/apache/atlas/repository/typestore/GraphBackedTypeStore.java
>  a94d157 
>   repository/src/main/java/org/apache/atlas/services/AddAttributePatch.java 
> PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/services/AtlasTypePatch.java 
> PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java
>  6a937f4 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/AbstractDataType.java
>  fad091d 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/ClassType.java 
> c56987a 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/EnumType.java 
> bdd0a13 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/EnumTypeDefinition.java
>  6340615 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/HierarchicalType.java
>  7224699 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/HierarchicalTypeDefinition.java
>  9a299f0 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/IDataType.java 
> 85ddee7 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/StructType.java 
> 6f40c1d 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/StructTypeDefinition.java
>  f1ce1b7 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/TraitType.java 
> f23bf5b 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/TypeSystem.java 
> 70ba89b 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/utils/TypesUtil.java
>  ef8448d 
>   
> typesystem/src/main/scala/org/apache/atlas/typesystem/json/TypesSerialization.scala
>  5618938 
> 
> Diff: https://reviews.apache.org/r/51939/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sarath Kumar Subramanian
> 
>

Re: Review Request 51939: Framework to apply updates to types in the type-system

[Suma Shivaprasad]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51939/#review149608
---




repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java 
(line 191)


it would make more sense to sort by date instead of file name?


- Suma Shivaprasad


On Sept. 19, 2016, 6:10 p.m., Sarath Kumar Subramanian wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51939/
> ---
> 
> (Updated Sept. 19, 2016, 6:10 p.m.)
> 
> 
> Review request for atlas, Madhan Neethiraj, Shwetha GS, and Suma Shivaprasad.
> 
> 
> Bugs: ATLAS-1174
> https://issues.apache.org/jira/browse/ATLAS-1174
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> 1. Introduce "version" attribute to all types in the type-system, this helps 
> to track changes made to the default types (hive, sqoop, falcon and storm 
> types) and user created types. If version is not mentioned during creation of 
> a type, default version "1.0" is assigned (optional attribute).
> 2. Using the version attributed for types, introduce a patch framework for 
> type system. This framework applies patches to a type using its version 
> number and can be used during upgrade - add new attributes to an existing 
> types and it will be run during atlas startup.
> The sequence of steps:
> a. During atlas startup, check $ATLAS_HOME/models/patches directory for any 
> available patch files (json files). If there any patch files handle them.
> b. Sample patch json file looks like:
> {
> "patches": [
> { 
> "action": "ADD_ATTRIBUTE",
> "typeName": "hive_column",
> "applyToVersion": "1.0",
> "updateToVersion": "2.0",
> "actionParams": [
> { "name": "position", "dataTypeName": "int", "multiplicity": "optional", 
> "isComposite": false, "isUnique": false, "isIndexable": false, 
> "reverseAttributeName": null }
> ]
> } ]
> }
> c. The framework updates the type in "typeName" for the matching version 
> number and after applying the patch, update the version to the one mentioned 
> in "updateToVersion"
> d. The json file can have more than one action (array of actions).
> e. There can be multiple patch json files in the directory and are applied in 
> the sort order of the filename. eg:
> 001-hive_column_add_position.json
> 002-hive_column_add_anotherattribute.json
> 
> 
> Diffs
> -
> 
>   common/src/main/java/org/apache/atlas/repository/Constants.java d7f9c89 
>   
> repository/src/main/java/org/apache/atlas/repository/typestore/GraphBackedTypeStore.java
>  a94d157 
>   repository/src/main/java/org/apache/atlas/services/AddAttributePatch.java 
> PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/services/AtlasTypePatch.java 
> PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java
>  6a937f4 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/AbstractDataType.java
>  fad091d 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/ClassType.java 
> c56987a 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/EnumType.java 
> bdd0a13 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/EnumTypeDefinition.java
>  6340615 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/HierarchicalType.java
>  7224699 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/HierarchicalTypeDefinition.java
>  9a299f0 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/IDataType.java 
> 85ddee7 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/StructType.java 
> 6f40c1d 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/StructTypeDefinition.java
>  f1ce1b7 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/TraitType.java 
> f23bf5b 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/TypeSystem.java 
> 70ba89b 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/utils/TypesUtil.java
>  ef8448d 
>   
> typesystem/src/main/scala/org/apache/atlas/typesystem/json/TypesSerialization.scala
>  5618938 
> 
> Diff: https://reviews.apache.org/r/51939/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sarath Kumar Subramanian
> 
>

Re: Review Request 51939: Framework to apply updates to types in the type-system

[Suma Shivaprasad]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51939/#review149607
---




repository/src/main/java/org/apache/atlas/services/AddAttributePatch.java (line 
102)


PatchResult should be modified to have the error message as to why the 
update failed



repository/src/main/java/org/apache/atlas/services/AddAttributePatch.java (line 
104)


Log instead of PTS


- Suma Shivaprasad


On Sept. 19, 2016, 6:10 p.m., Sarath Kumar Subramanian wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51939/
> ---
> 
> (Updated Sept. 19, 2016, 6:10 p.m.)
> 
> 
> Review request for atlas, Madhan Neethiraj, Shwetha GS, and Suma Shivaprasad.
> 
> 
> Bugs: ATLAS-1174
> https://issues.apache.org/jira/browse/ATLAS-1174
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> 1. Introduce "version" attribute to all types in the type-system, this helps 
> to track changes made to the default types (hive, sqoop, falcon and storm 
> types) and user created types. If version is not mentioned during creation of 
> a type, default version "1.0" is assigned (optional attribute).
> 2. Using the version attributed for types, introduce a patch framework for 
> type system. This framework applies patches to a type using its version 
> number and can be used during upgrade - add new attributes to an existing 
> types and it will be run during atlas startup.
> The sequence of steps:
> a. During atlas startup, check $ATLAS_HOME/models/patches directory for any 
> available patch files (json files). If there any patch files handle them.
> b. Sample patch json file looks like:
> {
> "patches": [
> { 
> "action": "ADD_ATTRIBUTE",
> "typeName": "hive_column",
> "applyToVersion": "1.0",
> "updateToVersion": "2.0",
> "actionParams": [
> { "name": "position", "dataTypeName": "int", "multiplicity": "optional", 
> "isComposite": false, "isUnique": false, "isIndexable": false, 
> "reverseAttributeName": null }
> ]
> } ]
> }
> c. The framework updates the type in "typeName" for the matching version 
> number and after applying the patch, update the version to the one mentioned 
> in "updateToVersion"
> d. The json file can have more than one action (array of actions).
> e. There can be multiple patch json files in the directory and are applied in 
> the sort order of the filename. eg:
> 001-hive_column_add_position.json
> 002-hive_column_add_anotherattribute.json
> 
> 
> Diffs
> -
> 
>   common/src/main/java/org/apache/atlas/repository/Constants.java d7f9c89 
>   
> repository/src/main/java/org/apache/atlas/repository/typestore/GraphBackedTypeStore.java
>  a94d157 
>   repository/src/main/java/org/apache/atlas/services/AddAttributePatch.java 
> PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/services/AtlasTypePatch.java 
> PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java
>  6a937f4 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/AbstractDataType.java
>  fad091d 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/ClassType.java 
> c56987a 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/EnumType.java 
> bdd0a13 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/EnumTypeDefinition.java
>  6340615 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/HierarchicalType.java
>  7224699 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/HierarchicalTypeDefinition.java
>  9a299f0 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/IDataType.java 
> 85ddee7 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/StructType.java 
> 6f40c1d 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/StructTypeDefinition.java
>  f1ce1b7 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/TraitType.java 
> f23bf5b 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/TypeSystem.java 
> 70ba89b 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/utils/TypesUtil.java
>  ef8448d 
>   
> typesystem/src/main/scala/org/apache/atlas/typesystem/json/TypesSerialization.scala
>  5618938 
> 
> Diff: https://reviews.apache.org/r/51939/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sarath Kumar Subramanian
> 
>

Re: Review Request 51939: Framework to apply updates to types in the type-system

[Suma Shivaprasad]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51939/#review149553
---




repository/src/main/java/org/apache/atlas/repository/typestore/GraphBackedTypeStore.java
 (line 68)


is this the default type version. If so, pls rename as DEFAULT_TYPE_VERSION


- Suma Shivaprasad


On Sept. 19, 2016, 6:10 p.m., Sarath Kumar Subramanian wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51939/
> ---
> 
> (Updated Sept. 19, 2016, 6:10 p.m.)
> 
> 
> Review request for atlas, Madhan Neethiraj, Shwetha GS, and Suma Shivaprasad.
> 
> 
> Bugs: ATLAS-1174
> https://issues.apache.org/jira/browse/ATLAS-1174
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> 1. Introduce "version" attribute to all types in the type-system, this helps 
> to track changes made to the default types (hive, sqoop, falcon and storm 
> types) and user created types. If version is not mentioned during creation of 
> a type, default version "1.0" is assigned (optional attribute).
> 2. Using the version attributed for types, introduce a patch framework for 
> type system. This framework applies patches to a type using its version 
> number and can be used during upgrade - add new attributes to an existing 
> types and it will be run during atlas startup.
> The sequence of steps:
> a. During atlas startup, check $ATLAS_HOME/models/patches directory for any 
> available patch files (json files). If there any patch files handle them.
> b. Sample patch json file looks like:
> {
> "patches": [
> { 
> "action": "ADD_ATTRIBUTE",
> "typeName": "hive_column",
> "applyToVersion": "1.0",
> "updateToVersion": "2.0",
> "actionParams": [
> { "name": "position", "dataTypeName": "int", "multiplicity": "optional", 
> "isComposite": false, "isUnique": false, "isIndexable": false, 
> "reverseAttributeName": null }
> ]
> } ]
> }
> c. The framework updates the type in "typeName" for the matching version 
> number and after applying the patch, update the version to the one mentioned 
> in "updateToVersion"
> d. The json file can have more than one action (array of actions).
> e. There can be multiple patch json files in the directory and are applied in 
> the sort order of the filename. eg:
> 001-hive_column_add_position.json
> 002-hive_column_add_anotherattribute.json
> 
> 
> Diffs
> -
> 
>   common/src/main/java/org/apache/atlas/repository/Constants.java d7f9c89 
>   
> repository/src/main/java/org/apache/atlas/repository/typestore/GraphBackedTypeStore.java
>  a94d157 
>   repository/src/main/java/org/apache/atlas/services/AddAttributePatch.java 
> PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/services/AtlasTypePatch.java 
> PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java
>  6a937f4 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/AbstractDataType.java
>  fad091d 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/ClassType.java 
> c56987a 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/EnumType.java 
> bdd0a13 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/EnumTypeDefinition.java
>  6340615 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/HierarchicalType.java
>  7224699 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/HierarchicalTypeDefinition.java
>  9a299f0 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/IDataType.java 
> 85ddee7 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/StructType.java 
> 6f40c1d 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/StructTypeDefinition.java
>  f1ce1b7 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/TraitType.java 
> f23bf5b 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/TypeSystem.java 
> 70ba89b 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/utils/TypesUtil.java
>  ef8448d 
>   
> typesystem/src/main/scala/org/apache/atlas/typesystem/json/TypesSerialization.scala
>  5618938 
> 
> Diff: https://reviews.apache.org/r/51939/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sarath Kumar Subramanian
> 
>

Re: Review Request 51939: Framework to apply updates to types in the type-system

[Suma Shivaprasad]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51939/#review149552
---




repository/src/main/java/org/apache/atlas/repository/typestore/GraphBackedTypeStore.java
 (line 351)


Log type version change at info level


- Suma Shivaprasad


On Sept. 19, 2016, 6:10 p.m., Sarath Kumar Subramanian wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51939/
> ---
> 
> (Updated Sept. 19, 2016, 6:10 p.m.)
> 
> 
> Review request for atlas, Madhan Neethiraj, Shwetha GS, and Suma Shivaprasad.
> 
> 
> Bugs: ATLAS-1174
> https://issues.apache.org/jira/browse/ATLAS-1174
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> 1. Introduce "version" attribute to all types in the type-system, this helps 
> to track changes made to the default types (hive, sqoop, falcon and storm 
> types) and user created types. If version is not mentioned during creation of 
> a type, default version "1.0" is assigned (optional attribute).
> 2. Using the version attributed for types, introduce a patch framework for 
> type system. This framework applies patches to a type using its version 
> number and can be used during upgrade - add new attributes to an existing 
> types and it will be run during atlas startup.
> The sequence of steps:
> a. During atlas startup, check $ATLAS_HOME/models/patches directory for any 
> available patch files (json files). If there any patch files handle them.
> b. Sample patch json file looks like:
> {
> "patches": [
> { 
> "action": "ADD_ATTRIBUTE",
> "typeName": "hive_column",
> "applyToVersion": "1.0",
> "updateToVersion": "2.0",
> "actionParams": [
> { "name": "position", "dataTypeName": "int", "multiplicity": "optional", 
> "isComposite": false, "isUnique": false, "isIndexable": false, 
> "reverseAttributeName": null }
> ]
> } ]
> }
> c. The framework updates the type in "typeName" for the matching version 
> number and after applying the patch, update the version to the one mentioned 
> in "updateToVersion"
> d. The json file can have more than one action (array of actions).
> e. There can be multiple patch json files in the directory and are applied in 
> the sort order of the filename. eg:
> 001-hive_column_add_position.json
> 002-hive_column_add_anotherattribute.json
> 
> 
> Diffs
> -
> 
>   common/src/main/java/org/apache/atlas/repository/Constants.java d7f9c89 
>   
> repository/src/main/java/org/apache/atlas/repository/typestore/GraphBackedTypeStore.java
>  a94d157 
>   repository/src/main/java/org/apache/atlas/services/AddAttributePatch.java 
> PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/services/AtlasTypePatch.java 
> PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java
>  6a937f4 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/AbstractDataType.java
>  fad091d 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/ClassType.java 
> c56987a 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/EnumType.java 
> bdd0a13 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/EnumTypeDefinition.java
>  6340615 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/HierarchicalType.java
>  7224699 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/HierarchicalTypeDefinition.java
>  9a299f0 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/IDataType.java 
> 85ddee7 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/StructType.java 
> 6f40c1d 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/StructTypeDefinition.java
>  f1ce1b7 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/TraitType.java 
> f23bf5b 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/TypeSystem.java 
> 70ba89b 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/utils/TypesUtil.java
>  ef8448d 
>   
> typesystem/src/main/scala/org/apache/atlas/typesystem/json/TypesSerialization.scala
>  5618938 
> 
> Diff: https://reviews.apache.org/r/51939/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sarath Kumar Subramanian
> 
>

Re: Review Request 51939: Framework to apply updates to types in the type-system

[Sarath Kumar Subramanian]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51939/
---

(Updated Sept. 19, 2016, 6:10 p.m.)


Review request for atlas, Madhan Neethiraj, Shwetha GS, and Suma Shivaprasad.


Bugs: ATLAS-1174
https://issues.apache.org/jira/browse/ATLAS-1174


Repository: atlas


Description
---

1. Introduce "version" attribute to all types in the type-system, this helps to 
track changes made to the default types (hive, sqoop, falcon and storm types) 
and user created types. If version is not mentioned during creation of a type, 
default version "1.0" is assigned (optional attribute).
2. Using the version attributed for types, introduce a patch framework for type 
system. This framework applies patches to a type using its version number and 
can be used during upgrade - add new attributes to an existing types and it 
will be run during atlas startup.
The sequence of steps:
a. During atlas startup, check $ATLAS_HOME/models/patches directory for any 
available patch files (json files). If there any patch files handle them.
b. Sample patch json file looks like:
{
"patches": [
{ 
"action": "ADD_ATTRIBUTE",
"typeName": "hive_column",
"applyToVersion": "1.0",
"updateToVersion": "2.0",
"actionParams": [
{ "name": "position", "dataTypeName": "int", "multiplicity": "optional", 
"isComposite": false, "isUnique": false, "isIndexable": false, 
"reverseAttributeName": null }
]
} ]
}
c. The framework updates the type in "typeName" for the matching version number 
and after applying the patch, update the version to the one mentioned in 
"updateToVersion"
d. The json file can have more than one action (array of actions).
e. There can be multiple patch json files in the directory and are applied in 
the sort order of the filename. eg:
001-hive_column_add_position.json
002-hive_column_add_anotherattribute.json


Diffs (updated)
-

  common/src/main/java/org/apache/atlas/repository/Constants.java d7f9c89 
  
repository/src/main/java/org/apache/atlas/repository/typestore/GraphBackedTypeStore.java
 a94d157 
  repository/src/main/java/org/apache/atlas/services/AddAttributePatch.java 
PRE-CREATION 
  repository/src/main/java/org/apache/atlas/services/AtlasTypePatch.java 
PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java 
6a937f4 
  
typesystem/src/main/java/org/apache/atlas/typesystem/types/AbstractDataType.java
 fad091d 
  typesystem/src/main/java/org/apache/atlas/typesystem/types/ClassType.java 
c56987a 
  typesystem/src/main/java/org/apache/atlas/typesystem/types/EnumType.java 
bdd0a13 
  
typesystem/src/main/java/org/apache/atlas/typesystem/types/EnumTypeDefinition.java
 6340615 
  
typesystem/src/main/java/org/apache/atlas/typesystem/types/HierarchicalType.java
 7224699 
  
typesystem/src/main/java/org/apache/atlas/typesystem/types/HierarchicalTypeDefinition.java
 9a299f0 
  typesystem/src/main/java/org/apache/atlas/typesystem/types/IDataType.java 
85ddee7 
  typesystem/src/main/java/org/apache/atlas/typesystem/types/StructType.java 
6f40c1d 
  
typesystem/src/main/java/org/apache/atlas/typesystem/types/StructTypeDefinition.java
 f1ce1b7 
  typesystem/src/main/java/org/apache/atlas/typesystem/types/TraitType.java 
f23bf5b 
  typesystem/src/main/java/org/apache/atlas/typesystem/types/TypeSystem.java 
70ba89b 
  
typesystem/src/main/java/org/apache/atlas/typesystem/types/utils/TypesUtil.java 
ef8448d 
  
typesystem/src/main/scala/org/apache/atlas/typesystem/json/TypesSerialization.scala
 5618938 

Diff: https://reviews.apache.org/r/51939/diff/


Testing
---


Thanks,

Sarath Kumar Subramanian

Re: Review Request 51939: Framework to apply updates to types in the type-system

[Sarath Kumar Subramanian]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51939/
---

(Updated Sept. 19, 2016, 4:56 p.m.)


Review request for atlas, Madhan Neethiraj, Shwetha GS, and Suma Shivaprasad.


Changes
---

new review request based on comments


Bugs: ATLAS-1174
https://issues.apache.org/jira/browse/ATLAS-1174


Repository: atlas


Description
---

1. Introduce "version" attribute to all types in the type-system, this helps to 
track changes made to the default types (hive, sqoop, falcon and storm types) 
and user created types. If version is not mentioned during creation of a type, 
default version "1.0" is assigned (optional attribute).
2. Using the version attributed for types, introduce a patch framework for type 
system. This framework applies patches to a type using its version number and 
can be used during upgrade - add new attributes to an existing types and it 
will be run during atlas startup.
The sequence of steps:
a. During atlas startup, check $ATLAS_HOME/models/patches directory for any 
available patch files (json files). If there any patch files handle them.
b. Sample patch json file looks like:
{
"patches": [
{ 
"action": "ADD_ATTRIBUTE",
"typeName": "hive_column",
"applyToVersion": "1.0",
"updateToVersion": "2.0",
"actionParams": [
{ "name": "position", "dataTypeName": "int", "multiplicity": "optional", 
"isComposite": false, "isUnique": false, "isIndexable": false, 
"reverseAttributeName": null }
]
} ]
}
c. The framework updates the type in "typeName" for the matching version number 
and after applying the patch, update the version to the one mentioned in 
"updateToVersion"
d. The json file can have more than one action (array of actions).
e. There can be multiple patch json files in the directory and are applied in 
the sort order of the filename. eg:
001-hive_column_add_position.json
002-hive_column_add_anotherattribute.json


Diffs (updated)
-

  common/src/main/java/org/apache/atlas/repository/Constants.java d7f9c89 
  
repository/src/main/java/org/apache/atlas/repository/typestore/GraphBackedTypeStore.java
 a94d157 
  repository/src/main/java/org/apache/atlas/services/AddAttributePatch.java 
PRE-CREATION 
  repository/src/main/java/org/apache/atlas/services/AtlasTypePatch.java 
PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java 
6a937f4 
  
typesystem/src/main/java/org/apache/atlas/typesystem/types/AbstractDataType.java
 fad091d 
  typesystem/src/main/java/org/apache/atlas/typesystem/types/ClassType.java 
c56987a 
  typesystem/src/main/java/org/apache/atlas/typesystem/types/EnumType.java 
bdd0a13 
  
typesystem/src/main/java/org/apache/atlas/typesystem/types/EnumTypeDefinition.java
 6340615 
  
typesystem/src/main/java/org/apache/atlas/typesystem/types/HierarchicalType.java
 7224699 
  
typesystem/src/main/java/org/apache/atlas/typesystem/types/HierarchicalTypeDefinition.java
 9a299f0 
  typesystem/src/main/java/org/apache/atlas/typesystem/types/IDataType.java 
85ddee7 
  typesystem/src/main/java/org/apache/atlas/typesystem/types/StructType.java 
6f40c1d 
  
typesystem/src/main/java/org/apache/atlas/typesystem/types/StructTypeDefinition.java
 f1ce1b7 
  typesystem/src/main/java/org/apache/atlas/typesystem/types/TraitType.java 
f23bf5b 
  typesystem/src/main/java/org/apache/atlas/typesystem/types/TypeSystem.java 
70ba89b 
  
typesystem/src/main/java/org/apache/atlas/typesystem/types/utils/TypesUtil.java 
ef8448d 
  
typesystem/src/main/scala/org/apache/atlas/typesystem/json/TypesSerialization.scala
 5618938 

Diff: https://reviews.apache.org/r/51939/diff/


Testing
---


Thanks,

Sarath Kumar Subramanian

Re: Review Request 51939: Framework to apply updates to types in the type-system

[Sarath Kumar Subramanian]

> On Sept. 16, 2016, 1:19 a.m., Madhan Neethiraj wrote:
> > repository/src/main/java/org/apache/atlas/services/AtlasTypeUpdate.java, 
> > line 157
> > 
> >
> > Is using JSON string the only way to update types in the system? There 
> > is got to be a better way than use of opaque strings..

Currently creating a new type or updating an existing type in type system 
accepts only JSON String. This is by design.

JSONObject createType(String typeDefinition)
JSONObject createOrUpdateTypes(String typeDefinition, boolean isUpdate)


- Sarath Kumar


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51939/#review149157
---


On Sept. 15, 2016, 11:03 p.m., Sarath Kumar Subramanian wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51939/
> ---
> 
> (Updated Sept. 15, 2016, 11:03 p.m.)
> 
> 
> Review request for atlas, Madhan Neethiraj, Shwetha GS, and Suma Shivaprasad.
> 
> 
> Bugs: ATLAS-1174
> https://issues.apache.org/jira/browse/ATLAS-1174
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> 1. Introduce "version" attribute to all types in the type-system, this helps 
> to track changes made to the default types (hive, sqoop, falcon and storm 
> types) and user created types. If version is not mentioned during creation of 
> a type, default version "1.0" is assigned (optional attribute).
> 2. Using the version attributed for types, introduce a patch framework for 
> type system. This framework applies patches to a type using its version 
> number and can be used during upgrade - add new attributes to an existing 
> types and it will be run during atlas startup.
> The sequence of steps:
> a. During atlas startup, check $ATLAS_HOME/models/patches directory for any 
> available patch files (json files). If there any patch files handle them.
> b. Sample patch json file looks like:
> {
> "patches": [
> { 
> "action": "ADD_ATTRIBUTE",
> "typeName": "hive_column",
> "applyToVersion": "1.0",
> "updateToVersion": "2.0",
> "actionParams": [
> { "name": "position", "dataTypeName": "int", "multiplicity": "optional", 
> "isComposite": false, "isUnique": false, "isIndexable": false, 
> "reverseAttributeName": null }
> ]
> } ]
> }
> c. The framework updates the type in "typeName" for the matching version 
> number and after applying the patch, update the version to the one mentioned 
> in "updateToVersion"
> d. The json file can have more than one action (array of actions).
> e. There can be multiple patch json files in the directory and are applied in 
> the sort order of the filename. eg:
> 001-hive_column_add_position.json
> 002-hive_column_add_anotherattribute.json
> 
> 
> Diffs
> -
> 
>   common/src/main/java/org/apache/atlas/repository/Constants.java d7f9c89 
>   
> repository/src/main/java/org/apache/atlas/repository/typestore/GraphBackedTypeStore.java
>  a94d157 
>   repository/src/main/java/org/apache/atlas/services/AtlasTypeUpdate.java 
> PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java
>  6a937f4 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/AbstractDataType.java
>  fad091d 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/ClassType.java 
> c56987a 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/EnumType.java 
> bdd0a13 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/EnumTypeDefinition.java
>  6340615 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/HierarchicalType.java
>  7224699 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/HierarchicalTypeDefinition.java
>  9a299f0 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/IDataType.java 
> 85ddee7 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/StructType.java 
> 6f40c1d 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/StructTypeDefinition.java
>  f1ce1b7 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/TraitType.java 
> f23bf5b 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/TypeSystem.java 
> 70ba89b 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/utils/TypesUtil.java
>  ef8448d 
>   
> typesystem/src/main/scala/org/apache/atlas/typesystem/json/TypesSerialization.scala
>  5618938 
> 
> Diff: https://reviews.apache.org/r/51939/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sarath Kumar Subramanian
> 
>

Re: Review Request 51939: Framework to apply updates to types in the type-system

[David Radley]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51939/#review149181
---



I like the idea of introducing versions for types. I would like it to be clear 
what the use cases are for this.

I do not fully understand the fix and have not applied it yet - apologises if I 
have misunderstood the context. I wonder :
1) when do you envisage versioning a default type vs subclassing it. It seems 
to me subclassing is more flexible in many use cases. 
2) Changing the shape of a type will effect any objects that depend on the old 
type shape - so all the instance data in Atlas and data in the users 
application that depends on the old shape amy break.  
3) What would a potential difference be between one version of a type and a 
higher version - what can you add, can you only add at the end of an object , 
what can you remove, how does this effect any children types? Is this purely 
for patch management or also exposed at runtime in the APIs.
4) This reminds me of previous thinking we have used. We created a system where 
we had types; you could only add attributes and adding optional attributes 
would not require a new version. Only when new mandatory attributes were added 
would we require a new version. Is this pattern relevant here?
4) Are we expecting versions to basically get bigger and better in a chain or 
are we expecting splits. If we are a chain, I would suggest there are use cases 
where creation of a new entity based on a type picks up the latest rather than 
the default. So example all the entities in a project.  
5) If we update a version in a type, presumable we need to keep the old version 
around so we can process entities and the like that depend on that shape. The  
updateToVersion implies that we have lost the old version - presumable existing 
data instances referring to the old type still need to work in queries and the 
like; presumably we are doing a soft delete to allow lineage to occur.

David Radley

- David Radley


On Sept. 15, 2016, 11:03 p.m., Sarath Kumar Subramanian wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51939/
> ---
> 
> (Updated Sept. 15, 2016, 11:03 p.m.)
> 
> 
> Review request for atlas, Madhan Neethiraj, Shwetha GS, and Suma Shivaprasad.
> 
> 
> Bugs: ATLAS-1174
> https://issues.apache.org/jira/browse/ATLAS-1174
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> 1. Introduce "version" attribute to all types in the type-system, this helps 
> to track changes made to the default types (hive, sqoop, falcon and storm 
> types) and user created types. If version is not mentioned during creation of 
> a type, default version "1.0" is assigned (optional attribute).
> 2. Using the version attributed for types, introduce a patch framework for 
> type system. This framework applies patches to a type using its version 
> number and can be used during upgrade - add new attributes to an existing 
> types and it will be run during atlas startup.
> The sequence of steps:
> a. During atlas startup, check $ATLAS_HOME/models/patches directory for any 
> available patch files (json files). If there any patch files handle them.
> b. Sample patch json file looks like:
> {
> "patches": [
> { 
> "action": "ADD_ATTRIBUTE",
> "typeName": "hive_column",
> "applyToVersion": "1.0",
> "updateToVersion": "2.0",
> "actionParams": [
> { "name": "position", "dataTypeName": "int", "multiplicity": "optional", 
> "isComposite": false, "isUnique": false, "isIndexable": false, 
> "reverseAttributeName": null }
> ]
> } ]
> }
> c. The framework updates the type in "typeName" for the matching version 
> number and after applying the patch, update the version to the one mentioned 
> in "updateToVersion"
> d. The json file can have more than one action (array of actions).
> e. There can be multiple patch json files in the directory and are applied in 
> the sort order of the filename. eg:
> 001-hive_column_add_position.json
> 002-hive_column_add_anotherattribute.json
> 
> 
> Diffs
> -
> 
>   common/src/main/java/org/apache/atlas/repository/Constants.java d7f9c89 
>   
> repository/src/main/java/org/apache/atlas/repository/typestore/GraphBackedTypeStore.java
>  a94d157 
>   repository/src/main/java/org/apache/atlas/services/AtlasTypeUpdate.java 
> PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java
>  6a937f4 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/AbstractDataType.java
>  fad091d 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/ClassType.java 
> c56987a 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/EnumType.java 
> bdd0a13 
>   
> 

Re: Review Request 51939: Framework to apply updates to types in the type-system

[Apoorv Naik]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51939/#review149166
---




repository/src/main/java/org/apache/atlas/services/AtlasTypeUpdate.java (line 
79)


Can the list be null here ?



repository/src/main/java/org/apache/atlas/services/AtlasTypeUpdate.java (line 
87)


If found, the method will return false and vice-versa



repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java 
(line 85)


Can we preserve the specific imports instead of wildcards ?


- Apoorv Naik


On Sept. 15, 2016, 11:03 p.m., Sarath Kumar Subramanian wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51939/
> ---
> 
> (Updated Sept. 15, 2016, 11:03 p.m.)
> 
> 
> Review request for atlas, Madhan Neethiraj, Shwetha GS, and Suma Shivaprasad.
> 
> 
> Bugs: ATLAS-1174
> https://issues.apache.org/jira/browse/ATLAS-1174
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> 1. Introduce "version" attribute to all types in the type-system, this helps 
> to track changes made to the default types (hive, sqoop, falcon and storm 
> types) and user created types. If version is not mentioned during creation of 
> a type, default version "1.0" is assigned (optional attribute).
> 2. Using the version attributed for types, introduce a patch framework for 
> type system. This framework applies patches to a type using its version 
> number and can be used during upgrade - add new attributes to an existing 
> types and it will be run during atlas startup.
> The sequence of steps:
> a. During atlas startup, check $ATLAS_HOME/models/patches directory for any 
> available patch files (json files). If there any patch files handle them.
> b. Sample patch json file looks like:
> {
> "patches": [
> { 
> "action": "ADD_ATTRIBUTE",
> "typeName": "hive_column",
> "applyToVersion": "1.0",
> "updateToVersion": "2.0",
> "actionParams": [
> { "name": "position", "dataTypeName": "int", "multiplicity": "optional", 
> "isComposite": false, "isUnique": false, "isIndexable": false, 
> "reverseAttributeName": null }
> ]
> } ]
> }
> c. The framework updates the type in "typeName" for the matching version 
> number and after applying the patch, update the version to the one mentioned 
> in "updateToVersion"
> d. The json file can have more than one action (array of actions).
> e. There can be multiple patch json files in the directory and are applied in 
> the sort order of the filename. eg:
> 001-hive_column_add_position.json
> 002-hive_column_add_anotherattribute.json
> 
> 
> Diffs
> -
> 
>   common/src/main/java/org/apache/atlas/repository/Constants.java d7f9c89 
>   
> repository/src/main/java/org/apache/atlas/repository/typestore/GraphBackedTypeStore.java
>  a94d157 
>   repository/src/main/java/org/apache/atlas/services/AtlasTypeUpdate.java 
> PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java
>  6a937f4 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/AbstractDataType.java
>  fad091d 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/ClassType.java 
> c56987a 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/EnumType.java 
> bdd0a13 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/EnumTypeDefinition.java
>  6340615 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/HierarchicalType.java
>  7224699 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/HierarchicalTypeDefinition.java
>  9a299f0 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/IDataType.java 
> 85ddee7 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/StructType.java 
> 6f40c1d 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/StructTypeDefinition.java
>  f1ce1b7 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/TraitType.java 
> f23bf5b 
>   typesystem/src/main/java/org/apache/atlas/typesystem/types/TypeSystem.java 
> 70ba89b 
>   
> typesystem/src/main/java/org/apache/atlas/typesystem/types/utils/TypesUtil.java
>  ef8448d 
>   
> typesystem/src/main/scala/org/apache/atlas/typesystem/json/TypesSerialization.scala
>  5618938 
> 
> Diff: https://reviews.apache.org/r/51939/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sarath Kumar Subramanian
> 
>

Re: Review Request 51939: Framework to apply updates to types in the type-system

[Madhan Neethiraj]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51939/#review149157
---



Comments from partial review. I will complete the review later today.


repository/src/main/java/org/apache/atlas/services/AtlasTypeUpdate.java (line 
55)


Consider moving 'private' methods to end of the class - after public and 
protected methods. It makes it easier to read/review.



repository/src/main/java/org/apache/atlas/services/AtlasTypeUpdate.java (line 
56)


This method returns "false" when the given type has any of the attributes 
in the given array. checkIfAttributeExists() does not seem a good name.



repository/src/main/java/org/apache/atlas/services/AtlasTypeUpdate.java (line 
100)


handle patchArray == null. Also review other places that deal with contents 
of patch file.



repository/src/main/java/org/apache/atlas/services/AtlasTypeUpdate.java (line 
109)


"Invalid operation. Possible operation - ADD_ATTRIBUTE" ==>
  action + ": invalid action. Supported actions: ADD_ATTRIBUTE"



repository/src/main/java/org/apache/atlas/services/AtlasTypeUpdate.java (line 
114)


It will be helpful to add a comment showing a sample patch processed by 
this method.



repository/src/main/java/org/apache/atlas/services/AtlasTypeUpdate.java (line 
115)


metadataService is an attribute of this instance. Should this be passed as 
parameter?



repository/src/main/java/org/apache/atlas/services/AtlasTypeUpdate.java (line 
116)


It might be useful to isolate patch details to a separate little class. 
Consider the following class hierarchy:

class PatchData {
  String  typeName;
  String  applyToVersion;
  String  updateToVersion;
  Map params;
}

abstract class AtlasTypePatch {
  public enum PatchResult { SUCCESS, FAILED, SKIPPED };
 
  protected DefaultMetadataService metadataService;
  protected TypeSystem typeSystem;

  public void init(DefaultMetadataService metadataService, TypeSystem 
typeSystem) {
this.metadataService = metadataService;
this.typeSystem  = typeSystem;
  }
  
  public abstract PatchResult applyPatch(PatchData patch);
}

class AddAttributePatch extends AtlasTypePatch {
  public PatchResult applyPatch(PatchData patch) {
// ..
  }
}



repository/src/main/java/org/apache/atlas/services/AtlasTypeUpdate.java (line 
131)


currentTypeVersion.equalsIgnoreCase(applyVersion) || 
currentTypeVersion.startsWith(applyVersion)) ==> 
currentTypeVersion.equalsIgnoreCase(applyVersion) || 
currentTypeVersion.startsWith(applyVersion + "."))

..to avoid applying policies for these values: currentVersion="1.11" with 
appyVersion=""1.1"



repository/src/main/java/org/apache/atlas/services/AtlasTypeUpdate.java (line 
136)


LOG.error("failed to apply patch (typeName=" + typeName + "; 
applyToVersion=" + applyToVersion + "; updateToVersion=" + updateToVersion + 
"): type does not exist", e);



repository/src/main/java/org/apache/atlas/services/AtlasTypeUpdate.java (line 
141)


Replace with:
  LOG.error("failed to apply patch (typeName=" + typeName + "; 
applyToVersion=" + applyToVersion + "; updateToVersion=" + updateToVersion + 
")", e);
  
Review other error messages as well.



repository/src/main/java/org/apache/atlas/services/AtlasTypeUpdate.java (line 
148)


handle error and return null if the given type does not exist.



repository/src/main/java/org/apache/atlas/services/AtlasTypeUpdate.java (line 
157)


Is using JSON string the only way to update types in the system? There is 
got to be a better way than use of opaque strings..


- Madhan Neethiraj


On Sept. 15, 2016, 11:03 p.m., Sarath Kumar Subramanian wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51939/
> ---
> 
> (Updated Sept. 15, 2016, 11:03 p.m.)
> 
> 
> Review request for atlas, Madhan Neethiraj, Shwetha GS, and Suma Shivaprasad.
> 
> 
> Bugs: ATLAS-1174
>