----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/51939/#review149811 -----------------------------------------------------------
repository/src/main/java/org/apache/atlas/services/AtlasTypeAttributePatch.java (line 42) <https://reviews.apache.org/r/51939/#comment217544> This class works on jsons directly which is error prone. Use POJOs instead - HierarchicalTypeDefinition, StructTypeDefinition repository/src/main/java/org/apache/atlas/services/AtlasTypeAttributePatch.java (line 154) <https://reviews.apache.org/r/51939/#comment217537> == doesn't work on strings repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java (line 177) <https://reviews.apache.org/r/51939/#comment217543> Move this to AtlasTypeAttributePatch Add tests 1. Currently, the model files(like hive_model.json) are auto generated from model definitions defined in java(like HiveDataModelGenerator). The patch files in this case has to be hand coded which is error prone 2. For completeness, readability and debuggability, the type update has to be done in the corresponding model definitions like HiveDataModelGenerator. So, same data will be in two places and the model definitions and the patch files can go out of sync 3. Since the model definitions(like HiveDataModelGenerator) will be updated anyways, if we modify ReservedTypesRegistrar to do type update instead of type create, the type updates will automatically be taken care with the same model json. So, model update patches are not necessary then 3. This jira doesn't implement type versioning - doesn't have support for storing multiple versions of the type. But it maintains the version of the latest type definition which I think is useful for debugging, to know the version of type that the server knows. Can we maintain this info in HiveDataModelGenerator itself, and hence will be part of hive_model.json - Shwetha GS On Sept. 21, 2016, 7:51 a.m., Sarath Kumar Subramanian wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/51939/ > ----------------------------------------------------------- > > (Updated Sept. 21, 2016, 7:51 a.m.) > > > Review request for atlas, Madhan Neethiraj, Shwetha GS, and Suma Shivaprasad. > > > Bugs: ATLAS-1174 > https://issues.apache.org/jira/browse/ATLAS-1174 > > > Repository: atlas > > > Description > ------- > > 1. Introduce "version" attribute to all types in the type-system, this helps > to track changes made to the default types (hive, sqoop, falcon and storm > types) and user created types. If version is not mentioned during creation of > a type, default version "1.0" is assigned (optional attribute). > 2. Using the version attributed for types, introduce a patch framework for > type system. This framework applies patches to a type using its version > number and can be used during upgrade - add new attributes to an existing > types and it will be run during atlas startup. > The sequence of steps: > a. During atlas startup, check $ATLAS_HOME/models/patches directory for any > available patch files (json files). If there any patch files handle them. > b. Sample patch json file looks like: > { > "patches": [ > { > "action": "ADD_ATTRIBUTE", > "typeName": "hive_column", > "applyToVersion": "1.0", > "updateToVersion": "2.0", > "actionParams": [ > { "name": "position", "dataTypeName": "int", "multiplicity": "optional", > "isComposite": false, "isUnique": false, "isIndexable": false, > "reverseAttributeName": null } > ] > } ] > } > c. The framework updates the type in "typeName" for the matching version > number and after applying the patch, update the version to the one mentioned > in "updateToVersion" > d. The json file can have more than one action (array of actions). > e. There can be multiple patch json files in the directory and are applied in > the sort order of the filename. eg: > 001-hive_column_add_position.json > 002-hive_column_add_anotherattribute.json > > > Diffs > ----- > > common/src/main/java/org/apache/atlas/AtlasConstants.java 17ffbd7 > common/src/main/java/org/apache/atlas/repository/Constants.java d7f9c89 > > repository/src/main/java/org/apache/atlas/repository/typestore/GraphBackedTypeStore.java > a94d157 > > repository/src/main/java/org/apache/atlas/services/AtlasTypeAttributePatch.java > PRE-CREATION > repository/src/main/java/org/apache/atlas/services/AtlasTypePatch.java > PRE-CREATION > > repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java > 6a937f4 > > typesystem/src/main/java/org/apache/atlas/typesystem/types/AbstractDataType.java > fad091d > typesystem/src/main/java/org/apache/atlas/typesystem/types/ClassType.java > c56987a > typesystem/src/main/java/org/apache/atlas/typesystem/types/EnumType.java > bdd0a13 > > typesystem/src/main/java/org/apache/atlas/typesystem/types/EnumTypeDefinition.java > 6340615 > > typesystem/src/main/java/org/apache/atlas/typesystem/types/HierarchicalType.java > 7224699 > > typesystem/src/main/java/org/apache/atlas/typesystem/types/HierarchicalTypeDefinition.java > 9a299f0 > typesystem/src/main/java/org/apache/atlas/typesystem/types/IDataType.java > 85ddee7 > typesystem/src/main/java/org/apache/atlas/typesystem/types/StructType.java > 6f40c1d > > typesystem/src/main/java/org/apache/atlas/typesystem/types/StructTypeDefinition.java > f1ce1b7 > typesystem/src/main/java/org/apache/atlas/typesystem/types/TraitType.java > f23bf5b > typesystem/src/main/java/org/apache/atlas/typesystem/types/TypeSystem.java > 70ba89b > > typesystem/src/main/java/org/apache/atlas/typesystem/types/utils/TypesUtil.java > ef8448d > > typesystem/src/main/scala/org/apache/atlas/typesystem/json/TypesSerialization.scala > 5618938 > > Diff: https://reviews.apache.org/r/51939/diff/ > > > Testing > ------- > > > Thanks, > > Sarath Kumar Subramanian > >
