subject:"Review Request 71649\: ATLAS\-3486 Define Data Models For Namespaces and Namespace Attributes in Atlas and also the corresponding type registry changes for the same"

Re: Review Request 71649: ATLAS-3486 Define Data Models For Namespaces and Namespace Attributes in Atlas and also the corresponding type registry changes for the same

2019-12-02 Thread Aadarsh Jajodia



> On Dec. 2, 2019, 5:06 a.m., Madhan Neethiraj wrote:
> > intg/src/main/java/org/apache/atlas/type/AtlasNamespaceType.java
> > Lines 126 (patched)
> > 
> >
> > Having AtlasNamespaceAttribute extend from AtlasAttribute can be 
> > helpful in many cases, like:
> >   - write atrribute values in EntityGraphMapper
> >   - read attribute values in EntityGraphRetriever
> >   - SolrIndexHelper
> >   - perhaps in search as well (SearchProcessor)
> > 
> > However, this would require:
> >   - AtlasNamespace extend from AtlasStructType (similr to 
> > AtlasClassificationType, AtlasEntityType)
> >   - AtlasNamespaceDef extend from AtlasStructDef
> > 
> > Please review the design for above.

Hi Madhan. Thanks for reviewing. The problem with AtlasNamespaceDef extending 
from AtlasStructDef, is that the attributes in a structDef are not tied to a 
specific entity type, like we want for the attributes in a Namespace Def. 
AtlasStructDef. AtlasStructDef has a list of List but those 
attributes are independent of any entity type, hence I was not able to extend 
AtlasNamespaceDef from AtlasStructDef.


- Aadarsh


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71649/#review218870
---


On Nov. 22, 2019, 2:03 a.m., Aadarsh Jajodia wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71649/
> ---
> 
> (Updated Nov. 22, 2019, 2:03 a.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Sridhar K, Le Ma, Madhan 
> Neethiraj, and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3486
> https://issues.apache.org/jira/browse/ATLAS-3486
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> This change is the first part of the bigger task defined as part of 
> ATLAS-3485.
> This adds the data model needed for supporting namespaces and namespace 
> attributes and also updates the type registry to include the applicable 
> namespace attributes for every entity type
> 
> 
> Diffs
> -
> 
>   intg/src/main/java/org/apache/atlas/AtlasErrorCode.java 
> 7a2aae2e9ae8174c5309164f3e41c940cbf3ddf8 
>   intg/src/main/java/org/apache/atlas/model/TypeCategory.java 
> f06f64f450f407e3f9a0e742726ff4dd12ccc695 
>   intg/src/main/java/org/apache/atlas/model/typedef/AtlasNamespaceDef.java 
> PRE-CREATION 
>   intg/src/main/java/org/apache/atlas/model/typedef/AtlasStructDef.java 
> bb7ead0f9f8bab3094eb82e9e286dd58e8a6e3de 
>   intg/src/main/java/org/apache/atlas/model/typedef/AtlasTypesDef.java 
> 3634fdfd313639eb97b3c4698e091487b0e44a80 
>   intg/src/main/java/org/apache/atlas/store/AtlasTypeDefStore.java 
> 4ee68a936f99bb4c819b5335da2cc8bf7d539397 
>   intg/src/main/java/org/apache/atlas/type/AtlasEntityType.java 
> 557ef74a95c2a939b4b89cd1db8fa4c73d52dd51 
>   intg/src/main/java/org/apache/atlas/type/AtlasNamespaceType.java 
> PRE-CREATION 
>   intg/src/main/java/org/apache/atlas/type/AtlasTypeRegistry.java 
> b071dc9d664cee9e1ffc54726ffbf15f4f602d30 
>   intg/src/main/java/org/apache/atlas/type/AtlasTypeUtil.java 
> 0883d54f490e22c6510e6fc0cb804b87713a7ecb 
>   intg/src/main/java/org/apache/atlas/typesystem/types/DataTypes.java 
> dba2d88146eff314191ae6bb24ad7337b0ea10ae 
>   intg/src/test/java/org/apache/atlas/TestRelationshipUtilsV2.java 
> 02613b5f7250b14324ed294c22de079b74d55b08 
>   intg/src/test/java/org/apache/atlas/TestUtilsV2.java 
> ff79994c519702e90b2e478d00cae0008889f956 
>   
> intg/src/test/java/org/apache/atlas/model/typedef/TestAtlasNamespaceDef.java 
> PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java
>  7c551304b2b65b90302f6e5fa9fc5b9f1b8e2c12 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/AtlasTypeDefGraphStore.java
>  2e2ab1a664171555c57560e1c0b4cbdbc20c0f6f 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasAbstractDefStoreV2.java
>  2cb2b47898ded4d6c5c84800ff93fa58b2c480da 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasNamespaceDefStoreV2.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasTypeDefGraphStoreV2.java
>  a5ccfb5b2055c88f596312f4033bc0034d3d165c 
>   
> repository/src/test/java/org/apache/atlas/repository/store/graph/v2/AtlasNamespaceDefStoreV2Test.java
>  PRE-CREATION 
>   webapp/src/main/java/org/apache/atlas/examples/QuickStartV2.java 
> 6cd0ee331b7ae24757b58e76ec47bf556106846a 
>   webapp/src/main/java/org/apache/atlas/web/rest/TypesREST.java 
> fb56fad6412079e20dd3e345b81a08d9e5ace657 
> 
> 
> Diff: https://reviews.apache.org/r/71649/diff/8/
> 
> 
> Testing
> ---
>

Re: Review Request 71649: ATLAS-3486 Define Data Models For Namespaces and Namespace Attributes in Atlas and also the corresponding type registry changes for the same

2019-12-01 Thread Madhan Neethiraj


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71649/#review218870
---




intg/src/main/java/org/apache/atlas/type/AtlasNamespaceType.java
Lines 126 (patched)


Having AtlasNamespaceAttribute extend from AtlasAttribute can be helpful in 
many cases, like:
  - write atrribute values in EntityGraphMapper
  - read attribute values in EntityGraphRetriever
  - SolrIndexHelper
  - perhaps in search as well (SearchProcessor)

However, this would require:
  - AtlasNamespace extend from AtlasStructType (similr to 
AtlasClassificationType, AtlasEntityType)
  - AtlasNamespaceDef extend from AtlasStructDef

Please review the design for above.


- Madhan Neethiraj


On Nov. 22, 2019, 2:03 a.m., Aadarsh Jajodia wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71649/
> ---
> 
> (Updated Nov. 22, 2019, 2:03 a.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Sridhar K, Le Ma, Madhan 
> Neethiraj, and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3486
> https://issues.apache.org/jira/browse/ATLAS-3486
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> This change is the first part of the bigger task defined as part of 
> ATLAS-3485.
> This adds the data model needed for supporting namespaces and namespace 
> attributes and also updates the type registry to include the applicable 
> namespace attributes for every entity type
> 
> 
> Diffs
> -
> 
>   intg/src/main/java/org/apache/atlas/AtlasErrorCode.java 
> 7a2aae2e9ae8174c5309164f3e41c940cbf3ddf8 
>   intg/src/main/java/org/apache/atlas/model/TypeCategory.java 
> f06f64f450f407e3f9a0e742726ff4dd12ccc695 
>   intg/src/main/java/org/apache/atlas/model/typedef/AtlasNamespaceDef.java 
> PRE-CREATION 
>   intg/src/main/java/org/apache/atlas/model/typedef/AtlasStructDef.java 
> bb7ead0f9f8bab3094eb82e9e286dd58e8a6e3de 
>   intg/src/main/java/org/apache/atlas/model/typedef/AtlasTypesDef.java 
> 3634fdfd313639eb97b3c4698e091487b0e44a80 
>   intg/src/main/java/org/apache/atlas/store/AtlasTypeDefStore.java 
> 4ee68a936f99bb4c819b5335da2cc8bf7d539397 
>   intg/src/main/java/org/apache/atlas/type/AtlasEntityType.java 
> 557ef74a95c2a939b4b89cd1db8fa4c73d52dd51 
>   intg/src/main/java/org/apache/atlas/type/AtlasNamespaceType.java 
> PRE-CREATION 
>   intg/src/main/java/org/apache/atlas/type/AtlasTypeRegistry.java 
> b071dc9d664cee9e1ffc54726ffbf15f4f602d30 
>   intg/src/main/java/org/apache/atlas/type/AtlasTypeUtil.java 
> 0883d54f490e22c6510e6fc0cb804b87713a7ecb 
>   intg/src/main/java/org/apache/atlas/typesystem/types/DataTypes.java 
> dba2d88146eff314191ae6bb24ad7337b0ea10ae 
>   intg/src/test/java/org/apache/atlas/TestRelationshipUtilsV2.java 
> 02613b5f7250b14324ed294c22de079b74d55b08 
>   intg/src/test/java/org/apache/atlas/TestUtilsV2.java 
> ff79994c519702e90b2e478d00cae0008889f956 
>   
> intg/src/test/java/org/apache/atlas/model/typedef/TestAtlasNamespaceDef.java 
> PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java
>  7c551304b2b65b90302f6e5fa9fc5b9f1b8e2c12 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/AtlasTypeDefGraphStore.java
>  2e2ab1a664171555c57560e1c0b4cbdbc20c0f6f 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasAbstractDefStoreV2.java
>  2cb2b47898ded4d6c5c84800ff93fa58b2c480da 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasNamespaceDefStoreV2.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasTypeDefGraphStoreV2.java
>  a5ccfb5b2055c88f596312f4033bc0034d3d165c 
>   
> repository/src/test/java/org/apache/atlas/repository/store/graph/v2/AtlasNamespaceDefStoreV2Test.java
>  PRE-CREATION 
>   webapp/src/main/java/org/apache/atlas/examples/QuickStartV2.java 
> 6cd0ee331b7ae24757b58e76ec47bf556106846a 
>   webapp/src/main/java/org/apache/atlas/web/rest/TypesREST.java 
> fb56fad6412079e20dd3e345b81a08d9e5ace657 
> 
> 
> Diff: https://reviews.apache.org/r/71649/diff/8/
> 
> 
> Testing
> ---
> 
> Added unit tests
> 
> 
> Thanks,
> 
> Aadarsh Jajodia
> 
>

Re: Review Request 71649: ATLAS-3486 Define Data Models For Namespaces and Namespace Attributes in Atlas and also the corresponding type registry changes for the same

2019-12-01 Thread Madhan Neethiraj


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71649/#review218867
---




intg/src/main/java/org/apache/atlas/model/typedef/AtlasNamespaceDef.java
Lines 75 (patched)


perhaps you meant to check if "this.attributeDefs" is null? Consider 
replacing lines #75 - #79 with:

  if (CollectionUtils.isNotEmpty(attributeDefs)) {
if (this.attributeDefs == null) {
  this.attributeDefs = new ArrayList<>();
}

this.attributeDefs.addAll(attributeDefs);
  }



intg/src/main/java/org/apache/atlas/model/typedef/AtlasNamespaceDef.java
Lines 198 (patched)


Move this method to #236, after all get/set methods



intg/src/main/java/org/apache/atlas/model/typedef/AtlasNamespaceDef.java
Lines 219 (patched)


Add '@Override' annotation



intg/src/main/java/org/apache/atlas/model/typedef/AtlasTypesDef.java
Lines 243 (patched)


insert following line here:
  sb.append("}");



intg/src/main/java/org/apache/atlas/type/AtlasEntityType.java
Lines 990 (patched)


Consider replacing populateNamespaceAttributes() by having 
AtlasNamespaceType.resolveReferences() call 
AtlasEntityType.addNamespaceAttribute() - as 
AtlasNamespaceType.resolveReferences() already iterates through namespace 
attributes and validates if entity-types specified are valid.



intg/src/main/java/org/apache/atlas/type/AtlasNamespaceType.java
Lines 83 (patched)


Comment in #83 should either be removed or moved to #114 below.



intg/src/main/java/org/apache/atlas/type/AtlasNamespaceType.java
Lines 103 (patched)


Only primitive or enum type attributes should be allowed in namespaces. 
Entity/Struct/Classification/Namespace types should not be allowed here.


- Madhan Neethiraj


On Nov. 22, 2019, 2:03 a.m., Aadarsh Jajodia wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71649/
> ---
> 
> (Updated Nov. 22, 2019, 2:03 a.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Sridhar K, Le Ma, Madhan 
> Neethiraj, and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3486
> https://issues.apache.org/jira/browse/ATLAS-3486
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> This change is the first part of the bigger task defined as part of 
> ATLAS-3485.
> This adds the data model needed for supporting namespaces and namespace 
> attributes and also updates the type registry to include the applicable 
> namespace attributes for every entity type
> 
> 
> Diffs
> -
> 
>   intg/src/main/java/org/apache/atlas/AtlasErrorCode.java 
> 7a2aae2e9ae8174c5309164f3e41c940cbf3ddf8 
>   intg/src/main/java/org/apache/atlas/model/TypeCategory.java 
> f06f64f450f407e3f9a0e742726ff4dd12ccc695 
>   intg/src/main/java/org/apache/atlas/model/typedef/AtlasNamespaceDef.java 
> PRE-CREATION 
>   intg/src/main/java/org/apache/atlas/model/typedef/AtlasStructDef.java 
> bb7ead0f9f8bab3094eb82e9e286dd58e8a6e3de 
>   intg/src/main/java/org/apache/atlas/model/typedef/AtlasTypesDef.java 
> 3634fdfd313639eb97b3c4698e091487b0e44a80 
>   intg/src/main/java/org/apache/atlas/store/AtlasTypeDefStore.java 
> 4ee68a936f99bb4c819b5335da2cc8bf7d539397 
>   intg/src/main/java/org/apache/atlas/type/AtlasEntityType.java 
> 557ef74a95c2a939b4b89cd1db8fa4c73d52dd51 
>   intg/src/main/java/org/apache/atlas/type/AtlasNamespaceType.java 
> PRE-CREATION 
>   intg/src/main/java/org/apache/atlas/type/AtlasTypeRegistry.java 
> b071dc9d664cee9e1ffc54726ffbf15f4f602d30 
>   intg/src/main/java/org/apache/atlas/type/AtlasTypeUtil.java 
> 0883d54f490e22c6510e6fc0cb804b87713a7ecb 
>   intg/src/main/java/org/apache/atlas/typesystem/types/DataTypes.java 
> dba2d88146eff314191ae6bb24ad7337b0ea10ae 
>   intg/src/test/java/org/apache/atlas/TestRelationshipUtilsV2.java 
> 02613b5f7250b14324ed294c22de079b74d55b08 
>   intg/src/test/java/org/apache/atlas/TestUtilsV2.java 
> ff79994c519702e90b2e478d00cae0008889f956 
>   
> intg/src/test/java/org/apache/atlas/model/typedef/TestAtlasNamespaceDef.java 
> PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java
>  7c551304b2b65b90302f6e5fa9fc5b9f1b8e2c12 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/AtlasTypeDefGraphStore.java
>  2e2ab1a664171555c57560e1c0b4cbdbc20c0f6f 
>   
>

Re: Review Request 71649: ATLAS-3486 Define Data Models For Namespaces and Namespace Attributes in Atlas and also the corresponding type registry changes for the same

2019-12-01 Thread Sarath Subramanian


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71649/#review218869
---




intg/src/main/java/org/apache/atlas/model/typedef/AtlasNamespaceDef.java
Lines 75 (patched)


this check is incorrect - adding null/empty 'attributeDefs' will reset the 
attributeDefs to emplty list. (old entries are removed). Please review

if (CollectionUtils.isNotEmpty(attributeDefs)) {
   this.attributeDefs.addAll(attributeDefs);
}



intg/src/main/java/org/apache/atlas/model/typedef/AtlasNamespaceDef.java
Lines 162 (patched)


'maxStrLength' and 'validPattern' are string attribute specific 
constraints, consider adding this constraints in AtlasAttributeDef.options map 
(define allowed string constraint enums). This will be extensible to add any 
condition in the future.



intg/src/main/java/org/apache/atlas/type/AtlasEntityType.java
Lines 49 (patched)


nit: remove new line



intg/src/main/java/org/apache/atlas/type/AtlasEntityType.java
Lines 1000 (patched)


consider refactoring this method:

private void addNamespaceAttribute(AtlasNamespaceAttribute nsAttribute) {
   String nsName = nsAttribute.getDefinedInType().getTypeName();

   if (!namespaceAttributes.containsKey(nsName)) {
  namespaceAttributes.put(nsName, new ArrayList<>());
   }

   namespaceAttributes.get(nsName).add(nsAttribute);
}



intg/src/main/java/org/apache/atlas/type/AtlasNamespaceType.java
Lines 31 (patched)


nit: remove unused import.



repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java
Lines 586 (patched)


this method is similar to method in line 505. consider refactoring to a 
single method and change behavior for AtlasNamespaceDef and AtlasStructDef



repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasAbstractDefStoreV2.java
Lines 67 (patched)


consider refactoring line 67-81 to avoid duplicates:

if (typeDef instanceof AtlasStructDef) {
   for (AtlasStructDef.AtlasAttributeDef attrDef : ((AtlasStructDef) 
typeDef).getAttributeDefs()) {
  validateAttributeName(typeDef, attrDef.getName());
   }
} else if (typeDef instanceof AtlasNamespaceDef) {
   for (AtlasNamespaceDef.AtlasNamespaceAttributeDef attrDef : 
((AtlasNamespaceDef) typeDef).getAttributeDefs()) {
  validateAttributeName(typeDef, attrDef.getName());
   }
}

private void validateAttributeName(AtlasBaseTypeDef typeDef, String 
attributeName) throws AtlasBaseException {
   if (AtlasDSL.Parser.isKeyword(attributeName)) {
  throw new AtlasBaseException(AtlasErrorCode.ATTRIBUTE_NAME_INVALID, 
attributeName, typeDef.getCategory().name());
   }
}



repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasAbstractDefStoreV2.java
Lines 93 (patched)


nit: remove new line



repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasNamespaceDefStoreV2.java
Lines 370 (patched)


encodedtypeNamePropertyKey => encodedTypeNamePropertyKey



repository/src/test/java/org/apache/atlas/repository/store/graph/v2/AtlasNamespaceDefStoreV2Test.java
Lines 32 (patched)


nit: remove unused imports in line 32-33


- Sarath Subramanian


On Nov. 21, 2019, 6:03 p.m., Aadarsh Jajodia wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71649/
> ---
> 
> (Updated Nov. 21, 2019, 6:03 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Sridhar K, Le Ma, Madhan 
> Neethiraj, and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3486
> https://issues.apache.org/jira/browse/ATLAS-3486
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> This change is the first part of the bigger task defined as part of 
> ATLAS-3485.
> This adds the data model needed for supporting namespaces and namespace 
> attributes and also updates the type registry to include the applicable 
> namespace attributes for every entity type
> 
> 
> Diffs
> -
> 
>   intg/src/main/java/org/apache/atlas/AtlasErrorCode.java 
> 7a2aae2e9ae8174c5309164f3e41c940cbf3ddf8 
>   intg/src/main/java/org/apache/atlas/model/TypeCategory.java 
>

Re: Review Request 71649: ATLAS-3486 Define Data Models For Namespaces and Namespace Attributes in Atlas and also the corresponding type registry changes for the same

2019-12-01 Thread Sarath Subramanian


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71649/#review218868
---



Patch fails to apply, can you please rebase.

- Sarath Subramanian


On Nov. 21, 2019, 6:03 p.m., Aadarsh Jajodia wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71649/
> ---
> 
> (Updated Nov. 21, 2019, 6:03 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Sridhar K, Le Ma, Madhan 
> Neethiraj, and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3486
> https://issues.apache.org/jira/browse/ATLAS-3486
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> This change is the first part of the bigger task defined as part of 
> ATLAS-3485.
> This adds the data model needed for supporting namespaces and namespace 
> attributes and also updates the type registry to include the applicable 
> namespace attributes for every entity type
> 
> 
> Diffs
> -
> 
>   intg/src/main/java/org/apache/atlas/AtlasErrorCode.java 
> 7a2aae2e9ae8174c5309164f3e41c940cbf3ddf8 
>   intg/src/main/java/org/apache/atlas/model/TypeCategory.java 
> f06f64f450f407e3f9a0e742726ff4dd12ccc695 
>   intg/src/main/java/org/apache/atlas/model/typedef/AtlasNamespaceDef.java 
> PRE-CREATION 
>   intg/src/main/java/org/apache/atlas/model/typedef/AtlasStructDef.java 
> bb7ead0f9f8bab3094eb82e9e286dd58e8a6e3de 
>   intg/src/main/java/org/apache/atlas/model/typedef/AtlasTypesDef.java 
> 3634fdfd313639eb97b3c4698e091487b0e44a80 
>   intg/src/main/java/org/apache/atlas/store/AtlasTypeDefStore.java 
> 4ee68a936f99bb4c819b5335da2cc8bf7d539397 
>   intg/src/main/java/org/apache/atlas/type/AtlasEntityType.java 
> 557ef74a95c2a939b4b89cd1db8fa4c73d52dd51 
>   intg/src/main/java/org/apache/atlas/type/AtlasNamespaceType.java 
> PRE-CREATION 
>   intg/src/main/java/org/apache/atlas/type/AtlasTypeRegistry.java 
> b071dc9d664cee9e1ffc54726ffbf15f4f602d30 
>   intg/src/main/java/org/apache/atlas/type/AtlasTypeUtil.java 
> 0883d54f490e22c6510e6fc0cb804b87713a7ecb 
>   intg/src/main/java/org/apache/atlas/typesystem/types/DataTypes.java 
> dba2d88146eff314191ae6bb24ad7337b0ea10ae 
>   intg/src/test/java/org/apache/atlas/TestRelationshipUtilsV2.java 
> 02613b5f7250b14324ed294c22de079b74d55b08 
>   intg/src/test/java/org/apache/atlas/TestUtilsV2.java 
> ff79994c519702e90b2e478d00cae0008889f956 
>   
> intg/src/test/java/org/apache/atlas/model/typedef/TestAtlasNamespaceDef.java 
> PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java
>  7c551304b2b65b90302f6e5fa9fc5b9f1b8e2c12 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/AtlasTypeDefGraphStore.java
>  2e2ab1a664171555c57560e1c0b4cbdbc20c0f6f 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasAbstractDefStoreV2.java
>  2cb2b47898ded4d6c5c84800ff93fa58b2c480da 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasNamespaceDefStoreV2.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasTypeDefGraphStoreV2.java
>  a5ccfb5b2055c88f596312f4033bc0034d3d165c 
>   
> repository/src/test/java/org/apache/atlas/repository/store/graph/v2/AtlasNamespaceDefStoreV2Test.java
>  PRE-CREATION 
>   webapp/src/main/java/org/apache/atlas/examples/QuickStartV2.java 
> 6cd0ee331b7ae24757b58e76ec47bf556106846a 
>   webapp/src/main/java/org/apache/atlas/web/rest/TypesREST.java 
> fb56fad6412079e20dd3e345b81a08d9e5ace657 
> 
> 
> Diff: https://reviews.apache.org/r/71649/diff/8/
> 
> 
> Testing
> ---
> 
> Added unit tests
> 
> 
> Thanks,
> 
> Aadarsh Jajodia
> 
>

Re: Review Request 71649: ATLAS-3486 Define Data Models For Namespaces and Namespace Attributes in Atlas and also the corresponding type registry changes for the same

2019-11-21 Thread Aadarsh Jajodia


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71649/
---

(Updated Nov. 22, 2019, 2:03 a.m.)


Review request for atlas, Ashutosh Mestry, Sridhar K, Le Ma, Madhan Neethiraj, 
and Sarath Subramanian.


Changes
---

Addressed Review comments


Bugs: ATLAS-3486
https://issues.apache.org/jira/browse/ATLAS-3486


Repository: atlas


Description
---

This change is the first part of the bigger task defined as part of ATLAS-3485.
This adds the data model needed for supporting namespaces and namespace 
attributes and also updates the type registry to include the applicable 
namespace attributes for every entity type


Diffs (updated)
-

  intg/src/main/java/org/apache/atlas/AtlasErrorCode.java 
7a2aae2e9ae8174c5309164f3e41c940cbf3ddf8 
  intg/src/main/java/org/apache/atlas/model/TypeCategory.java 
f06f64f450f407e3f9a0e742726ff4dd12ccc695 
  intg/src/main/java/org/apache/atlas/model/typedef/AtlasNamespaceDef.java 
PRE-CREATION 
  intg/src/main/java/org/apache/atlas/model/typedef/AtlasStructDef.java 
bb7ead0f9f8bab3094eb82e9e286dd58e8a6e3de 
  intg/src/main/java/org/apache/atlas/model/typedef/AtlasTypesDef.java 
3634fdfd313639eb97b3c4698e091487b0e44a80 
  intg/src/main/java/org/apache/atlas/store/AtlasTypeDefStore.java 
4ee68a936f99bb4c819b5335da2cc8bf7d539397 
  intg/src/main/java/org/apache/atlas/type/AtlasEntityType.java 
557ef74a95c2a939b4b89cd1db8fa4c73d52dd51 
  intg/src/main/java/org/apache/atlas/type/AtlasNamespaceType.java PRE-CREATION 
  intg/src/main/java/org/apache/atlas/type/AtlasTypeRegistry.java 
b071dc9d664cee9e1ffc54726ffbf15f4f602d30 
  intg/src/main/java/org/apache/atlas/type/AtlasTypeUtil.java 
0883d54f490e22c6510e6fc0cb804b87713a7ecb 
  intg/src/main/java/org/apache/atlas/typesystem/types/DataTypes.java 
dba2d88146eff314191ae6bb24ad7337b0ea10ae 
  intg/src/test/java/org/apache/atlas/TestRelationshipUtilsV2.java 
02613b5f7250b14324ed294c22de079b74d55b08 
  intg/src/test/java/org/apache/atlas/TestUtilsV2.java 
ff79994c519702e90b2e478d00cae0008889f956 
  intg/src/test/java/org/apache/atlas/model/typedef/TestAtlasNamespaceDef.java 
PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java
 7c551304b2b65b90302f6e5fa9fc5b9f1b8e2c12 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/AtlasTypeDefGraphStore.java
 2e2ab1a664171555c57560e1c0b4cbdbc20c0f6f 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasAbstractDefStoreV2.java
 2cb2b47898ded4d6c5c84800ff93fa58b2c480da 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasNamespaceDefStoreV2.java
 PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasTypeDefGraphStoreV2.java
 a5ccfb5b2055c88f596312f4033bc0034d3d165c 
  
repository/src/test/java/org/apache/atlas/repository/store/graph/v2/AtlasNamespaceDefStoreV2Test.java
 PRE-CREATION 
  webapp/src/main/java/org/apache/atlas/examples/QuickStartV2.java 
6cd0ee331b7ae24757b58e76ec47bf556106846a 
  webapp/src/main/java/org/apache/atlas/web/rest/TypesREST.java 
fb56fad6412079e20dd3e345b81a08d9e5ace657 


Diff: https://reviews.apache.org/r/71649/diff/8/

Changes: https://reviews.apache.org/r/71649/diff/7-8/


Testing
---

Added unit tests


Thanks,

Aadarsh Jajodia

Re: Review Request 71649: ATLAS-3486 Define Data Models For Namespaces and Namespace Attributes in Atlas and also the corresponding type registry changes for the same

2019-11-21 Thread Sarath Subramanian


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71649/#review218727
---



I don't see namespace attribuites indexed. This should be indexed. 

Check GraphBackSearchIndexer.addIndexForType() method


intg/src/main/java/org/apache/atlas/model/typedef/AtlasNamespaceDef.java
Lines 174 (patched)


removed unused constructors in line 174, 179, 193 and 203



intg/src/main/java/org/apache/atlas/model/typedef/AtlasNamespaceDef.java
Lines 162 (patched)


'maxStrLength' and 'validPattern' are string specific attribute options. 
Consider adding a enum - namespaceAttributeOptions {MAX_STRING_LENGTH, 
VALID_PATTERN} and use 'options' map in AtlasAttributeDef to set the namespace 
attribute options. This will be easier to extend for any int, date options as 
well.



intg/src/main/java/org/apache/atlas/store/AtlasTypeDefStore.java
Lines 83 (patched)


add REST endpoints in TypesREST for:

/namespacedef/name/{name}
/namespacedef/guid/{guid}



repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasNamespaceDefStoreV2.java
Lines 22 (patched)


nit: remove unused imports in line 22,42



repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasNamespaceDefStoreV2.java
Lines 59 (patched)


validateType() checks for invalid types only for structDef attruibutes, add 
check in validateType() to check for namespace attributes as well.



repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasNamespaceDefStoreV2.java
Lines 133 (patched)


is this line needed? please review.



repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasNamespaceDefStoreV2.java
Lines 226 (patched)


"update classification-def " => "update namespace-def "



repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasNamespaceDefStoreV2.java
Lines 260 (patched)


"delete struct-def " => "delete namespace-def "



repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasNamespaceDefStoreV2.java
Lines 297 (patched)


AtlasBaseException is never thrown



repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasNamespaceDefStoreV2.java
Lines 349 (patched)


'if' condition in line 349 and 353 is duplicate. please review.



repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasNamespaceDefStoreV2.java
Lines 357 (patched)


check for null as well for 'applicableEntityTypes'. => 
CollectionUtils.isNotEmpty(attributeDef.getApplicableEntityTypes())



repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasNamespaceDefStoreV2.java
Lines 382 (patched)


=> CollectionUtils.isNotEmpty(attributeDef.getApplicableEntityTypes())



repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasNamespaceDefStoreV2.java
Lines 470 (patched)


refactor to:

if (!currAttrNames.containsAll(attrNames)) {
 //throw exception
}



repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasNamespaceDefStoreV2.java
Lines 496 (patched)


check for null attributeDef.getApplicableEntityTypes() as well ?



repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasNamespaceDefStoreV2.java
Lines 515 (patched)


=> 
if(!updatedApplicableEntityTypes.containsAll(existingAttribute.getApplicableEntityTypes()))
 {
// throw exception
}



repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasNamespaceDefStoreV2.java
Lines 526 (patched)


set using encoded property key here:

AtlasGraphUtilsV2.encodePropertyKey(propertyKey);



repository/src/test/java/org/apache/atlas/repository/store/graph/AtlasTypeDefGraphStoreTest.java
Line 694 (original), 694 (patched)


contains only whitespace changes. revert changes to this file


- Sarath Subramanian


On Nov. 20, 2019, 7:03 p.m., Aadarsh Jajodia wrote:
> 
> ---
> This is

Re: Review Request 71649: ATLAS-3486 Define Data Models For Namespaces and Namespace Attributes in Atlas and also the corresponding type registry changes for the same

2019-11-20 Thread Aadarsh Jajodia


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71649/
---

(Updated Nov. 21, 2019, 3:03 a.m.)


Review request for atlas, Ashutosh Mestry, Sridhar K, Le Ma, Madhan Neethiraj, 
and Sarath Subramanian.


Changes
---

Rebased


Bugs: ATLAS-3486
https://issues.apache.org/jira/browse/ATLAS-3486


Repository: atlas


Description
---

This change is the first part of the bigger task defined as part of ATLAS-3485.
This adds the data model needed for supporting namespaces and namespace 
attributes and also updates the type registry to include the applicable 
namespace attributes for every entity type


Diffs (updated)
-

  intg/src/main/java/org/apache/atlas/AtlasErrorCode.java 
7a2aae2e9ae8174c5309164f3e41c940cbf3ddf8 
  intg/src/main/java/org/apache/atlas/model/TypeCategory.java 
f06f64f450f407e3f9a0e742726ff4dd12ccc695 
  intg/src/main/java/org/apache/atlas/model/typedef/AtlasNamespaceDef.java 
PRE-CREATION 
  intg/src/main/java/org/apache/atlas/model/typedef/AtlasStructDef.java 
bb7ead0f9f8bab3094eb82e9e286dd58e8a6e3de 
  intg/src/main/java/org/apache/atlas/model/typedef/AtlasTypesDef.java 
3634fdfd313639eb97b3c4698e091487b0e44a80 
  intg/src/main/java/org/apache/atlas/store/AtlasTypeDefStore.java 
4ee68a936f99bb4c819b5335da2cc8bf7d539397 
  intg/src/main/java/org/apache/atlas/type/AtlasEntityType.java 
557ef74a95c2a939b4b89cd1db8fa4c73d52dd51 
  intg/src/main/java/org/apache/atlas/type/AtlasNamespaceType.java PRE-CREATION 
  intg/src/main/java/org/apache/atlas/type/AtlasTypeRegistry.java 
b071dc9d664cee9e1ffc54726ffbf15f4f602d30 
  intg/src/main/java/org/apache/atlas/type/AtlasTypeUtil.java 
0883d54f490e22c6510e6fc0cb804b87713a7ecb 
  intg/src/main/java/org/apache/atlas/typesystem/types/DataTypes.java 
dba2d88146eff314191ae6bb24ad7337b0ea10ae 
  intg/src/test/java/org/apache/atlas/TestRelationshipUtilsV2.java 
02613b5f7250b14324ed294c22de079b74d55b08 
  intg/src/test/java/org/apache/atlas/TestUtilsV2.java 
ff79994c519702e90b2e478d00cae0008889f956 
  intg/src/test/java/org/apache/atlas/model/typedef/TestAtlasNamespaceDef.java 
PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/AtlasTypeDefGraphStore.java
 2e2ab1a664171555c57560e1c0b4cbdbc20c0f6f 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasNamespaceDefStoreV2.java
 PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasTypeDefGraphStoreV2.java
 a5ccfb5b2055c88f596312f4033bc0034d3d165c 
  
repository/src/test/java/org/apache/atlas/repository/store/graph/AtlasTypeDefGraphStoreTest.java
 51dd16b8518c9a16088547f3e95c0ef401695895 
  
repository/src/test/java/org/apache/atlas/repository/store/graph/v2/AtlasNamespaceDefStoreV2Test.java
 PRE-CREATION 
  webapp/src/main/java/org/apache/atlas/examples/QuickStartV2.java 
6cd0ee331b7ae24757b58e76ec47bf556106846a 


Diff: https://reviews.apache.org/r/71649/diff/7/

Changes: https://reviews.apache.org/r/71649/diff/6-7/


Testing
---

Added unit tests


Thanks,

Aadarsh Jajodia

Re: Review Request 71649: ATLAS-3486 Define Data Models For Namespaces and Namespace Attributes in Atlas and also the corresponding type registry changes for the same

2019-11-19 Thread Aadarsh Jajodia


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71649/
---

(Updated Nov. 20, 2019, 2:24 a.m.)


Review request for atlas, Ashutosh Mestry, Sridhar K, Le Ma, Madhan Neethiraj, 
and Sarath Subramanian.


Changes
---

Updated patch to handle cases when applicable entity types cannot be removed as 
part of a namespaceDef update call


Bugs: ATLAS-3486
https://issues.apache.org/jira/browse/ATLAS-3486


Repository: atlas


Description
---

This change is the first part of the bigger task defined as part of ATLAS-3485.
This adds the data model needed for supporting namespaces and namespace 
attributes and also updates the type registry to include the applicable 
namespace attributes for every entity type


Diffs (updated)
-

  intg/src/main/java/org/apache/atlas/AtlasErrorCode.java 
7a2aae2e9ae8174c5309164f3e41c940cbf3ddf8 
  intg/src/main/java/org/apache/atlas/model/TypeCategory.java 
f06f64f450f407e3f9a0e742726ff4dd12ccc695 
  intg/src/main/java/org/apache/atlas/model/typedef/AtlasNamespaceDef.java 
PRE-CREATION 
  intg/src/main/java/org/apache/atlas/model/typedef/AtlasStructDef.java 
e10965b870c4a300b41e93ee046b5f6d6b722728 
  intg/src/main/java/org/apache/atlas/model/typedef/AtlasTypesDef.java 
3634fdfd313639eb97b3c4698e091487b0e44a80 
  intg/src/main/java/org/apache/atlas/store/AtlasTypeDefStore.java 
4ee68a936f99bb4c819b5335da2cc8bf7d539397 
  intg/src/main/java/org/apache/atlas/type/AtlasEntityType.java 
884447f81c57faf917a8d0565fc0a0c7ebbd99f0 
  intg/src/main/java/org/apache/atlas/type/AtlasNamespaceType.java PRE-CREATION 
  intg/src/main/java/org/apache/atlas/type/AtlasTypeRegistry.java 
8b4fd1c3b9005d0a8852f2828475b4ad6a806822 
  intg/src/main/java/org/apache/atlas/type/AtlasTypeUtil.java 
0883d54f490e22c6510e6fc0cb804b87713a7ecb 
  intg/src/main/java/org/apache/atlas/typesystem/types/DataTypes.java 
dba2d88146eff314191ae6bb24ad7337b0ea10ae 
  intg/src/test/java/org/apache/atlas/TestRelationshipUtilsV2.java 
02613b5f7250b14324ed294c22de079b74d55b08 
  intg/src/test/java/org/apache/atlas/TestUtilsV2.java 
530d5cda44b7d3746f5cb8dd5a23b3c68b254cac 
  intg/src/test/java/org/apache/atlas/model/typedef/TestAtlasNamespaceDef.java 
PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/AtlasTypeDefGraphStore.java
 2e2ab1a664171555c57560e1c0b4cbdbc20c0f6f 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasNamespaceDefStoreV2.java
 PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasTypeDefGraphStoreV2.java
 a5ccfb5b2055c88f596312f4033bc0034d3d165c 
  
repository/src/test/java/org/apache/atlas/repository/store/graph/AtlasTypeDefGraphStoreTest.java
 51dd16b8518c9a16088547f3e95c0ef401695895 
  
repository/src/test/java/org/apache/atlas/repository/store/graph/v2/AtlasNamespaceDefStoreV2Test.java
 PRE-CREATION 
  webapp/src/main/java/org/apache/atlas/examples/QuickStartV2.java 
6cd0ee331b7ae24757b58e76ec47bf556106846a 


Diff: https://reviews.apache.org/r/71649/diff/6/

Changes: https://reviews.apache.org/r/71649/diff/5-6/


Testing
---

Added unit tests


Thanks,

Aadarsh Jajodia

Re: Review Request 71649: ATLAS-3486 Define Data Models For Namespaces and Namespace Attributes in Atlas and also the corresponding type registry changes for the same

2019-11-19 Thread Aadarsh Jajodia


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71649/
---

(Updated Nov. 19, 2019, 11:07 p.m.)


Review request for atlas, Ashutosh Mestry, Sridhar K, Le Ma, Madhan Neethiraj, 
and Sarath Subramanian.


Changes
---

Addressed review comments. Also added unit tests to validate applicable entity 
types


Bugs: ATLAS-3486
https://issues.apache.org/jira/browse/ATLAS-3486


Repository: atlas


Description
---

This change is the first part of the bigger task defined as part of ATLAS-3485.
This adds the data model needed for supporting namespaces and namespace 
attributes and also updates the type registry to include the applicable 
namespace attributes for every entity type


Diffs (updated)
-

  intg/src/main/java/org/apache/atlas/AtlasErrorCode.java 
7a2aae2e9ae8174c5309164f3e41c940cbf3ddf8 
  intg/src/main/java/org/apache/atlas/model/TypeCategory.java 
f06f64f450f407e3f9a0e742726ff4dd12ccc695 
  intg/src/main/java/org/apache/atlas/model/typedef/AtlasNamespaceDef.java 
PRE-CREATION 
  intg/src/main/java/org/apache/atlas/model/typedef/AtlasStructDef.java 
e10965b870c4a300b41e93ee046b5f6d6b722728 
  intg/src/main/java/org/apache/atlas/model/typedef/AtlasTypesDef.java 
3634fdfd313639eb97b3c4698e091487b0e44a80 
  intg/src/main/java/org/apache/atlas/store/AtlasTypeDefStore.java 
4ee68a936f99bb4c819b5335da2cc8bf7d539397 
  intg/src/main/java/org/apache/atlas/type/AtlasEntityType.java 
884447f81c57faf917a8d0565fc0a0c7ebbd99f0 
  intg/src/main/java/org/apache/atlas/type/AtlasNamespaceType.java PRE-CREATION 
  intg/src/main/java/org/apache/atlas/type/AtlasTypeRegistry.java 
8b4fd1c3b9005d0a8852f2828475b4ad6a806822 
  intg/src/main/java/org/apache/atlas/type/AtlasTypeUtil.java 
0883d54f490e22c6510e6fc0cb804b87713a7ecb 
  intg/src/main/java/org/apache/atlas/typesystem/types/DataTypes.java 
dba2d88146eff314191ae6bb24ad7337b0ea10ae 
  intg/src/test/java/org/apache/atlas/TestRelationshipUtilsV2.java 
02613b5f7250b14324ed294c22de079b74d55b08 
  intg/src/test/java/org/apache/atlas/TestUtilsV2.java 
530d5cda44b7d3746f5cb8dd5a23b3c68b254cac 
  intg/src/test/java/org/apache/atlas/model/typedef/TestAtlasNamespaceDef.java 
PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/AtlasTypeDefGraphStore.java
 2e2ab1a664171555c57560e1c0b4cbdbc20c0f6f 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasNamespaceDefStoreV2.java
 PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasTypeDefGraphStoreV2.java
 a5ccfb5b2055c88f596312f4033bc0034d3d165c 
  
repository/src/test/java/org/apache/atlas/repository/store/graph/AtlasTypeDefGraphStoreTest.java
 51dd16b8518c9a16088547f3e95c0ef401695895 
  
repository/src/test/java/org/apache/atlas/repository/store/graph/v2/AtlasNamespaceDefStoreV2Test.java
 PRE-CREATION 
  webapp/src/main/java/org/apache/atlas/examples/QuickStartV2.java 
6cd0ee331b7ae24757b58e76ec47bf556106846a 


Diff: https://reviews.apache.org/r/71649/diff/5/

Changes: https://reviews.apache.org/r/71649/diff/4-5/


Testing
---

Added unit tests


Thanks,

Aadarsh Jajodia

Re: Review Request 71649: ATLAS-3486 Define Data Models For Namespaces and Namespace Attributes in Atlas and also the corresponding type registry changes for the same

2019-11-19 Thread Aadarsh Jajodia



> On Nov. 15, 2019, 1:03 a.m., Sarath Subramanian wrote:
> > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasNamespaceDefStoreV2.java
> > Lines 264 (patched)
> > 
> >
> > this is not valid. NamespaceDef should be deleted only when no entity 
> > has reference to that namespace name.
> > 
> > Maybe check if there are any entity referring this namespace (using 
> > index query?) before deleting.
> > 
> > Same for line #294

So i can remove this check from this patch and when i submit the patch for the 
entity changes I can make this change for the delete method?


- Aadarsh


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71649/#review218640
---


On Nov. 14, 2019, 12:08 a.m., Aadarsh Jajodia wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71649/
> ---
> 
> (Updated Nov. 14, 2019, 12:08 a.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Sridhar K, Le Ma, Madhan 
> Neethiraj, and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3486
> https://issues.apache.org/jira/browse/ATLAS-3486
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> This change is the first part of the bigger task defined as part of 
> ATLAS-3485.
> This adds the data model needed for supporting namespaces and namespace 
> attributes and also updates the type registry to include the applicable 
> namespace attributes for every entity type
> 
> 
> Diffs
> -
> 
>   intg/src/main/java/org/apache/atlas/AtlasErrorCode.java 
> 7a2aae2e9ae8174c5309164f3e41c940cbf3ddf8 
>   intg/src/main/java/org/apache/atlas/model/TypeCategory.java 
> f06f64f450f407e3f9a0e742726ff4dd12ccc695 
>   intg/src/main/java/org/apache/atlas/model/typedef/AtlasNamespaceDef.java 
> PRE-CREATION 
>   intg/src/main/java/org/apache/atlas/model/typedef/AtlasStructDef.java 
> e10965b870c4a300b41e93ee046b5f6d6b722728 
>   intg/src/main/java/org/apache/atlas/model/typedef/AtlasTypesDef.java 
> 3634fdfd313639eb97b3c4698e091487b0e44a80 
>   intg/src/main/java/org/apache/atlas/store/AtlasTypeDefStore.java 
> 4ee68a936f99bb4c819b5335da2cc8bf7d539397 
>   intg/src/main/java/org/apache/atlas/type/AtlasEntityType.java 
> 884447f81c57faf917a8d0565fc0a0c7ebbd99f0 
>   intg/src/main/java/org/apache/atlas/type/AtlasNamespaceType.java 
> PRE-CREATION 
>   intg/src/main/java/org/apache/atlas/type/AtlasTypeRegistry.java 
> 8b4fd1c3b9005d0a8852f2828475b4ad6a806822 
>   intg/src/main/java/org/apache/atlas/type/AtlasTypeUtil.java 
> 0883d54f490e22c6510e6fc0cb804b87713a7ecb 
>   intg/src/main/java/org/apache/atlas/typesystem/types/DataTypes.java 
> dba2d88146eff314191ae6bb24ad7337b0ea10ae 
>   intg/src/test/java/org/apache/atlas/TestRelationshipUtilsV2.java 
> 02613b5f7250b14324ed294c22de079b74d55b08 
>   intg/src/test/java/org/apache/atlas/TestUtilsV2.java 
> 530d5cda44b7d3746f5cb8dd5a23b3c68b254cac 
>   
> intg/src/test/java/org/apache/atlas/model/typedef/TestAtlasNamespaceDef.java 
> PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/AtlasTypeDefGraphStore.java
>  2e2ab1a664171555c57560e1c0b4cbdbc20c0f6f 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasNamespaceDefStoreV2.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasTypeDefGraphStoreV2.java
>  a5ccfb5b2055c88f596312f4033bc0034d3d165c 
>   
> repository/src/test/java/org/apache/atlas/repository/store/graph/AtlasTypeDefGraphStoreTest.java
>  51dd16b8518c9a16088547f3e95c0ef401695895 
>   
> repository/src/test/java/org/apache/atlas/repository/store/graph/v2/AtlasNamespaceDefStoreV2Test.java
>  PRE-CREATION 
>   webapp/src/main/java/org/apache/atlas/examples/QuickStartV2.java 
> 6cd0ee331b7ae24757b58e76ec47bf556106846a 
> 
> 
> Diff: https://reviews.apache.org/r/71649/diff/4/
> 
> 
> Testing
> ---
> 
> Added unit tests
> 
> 
> Thanks,
> 
> Aadarsh Jajodia
> 
>

Re: Review Request 71649: ATLAS-3486 Define Data Models For Namespaces and Namespace Attributes in Atlas and also the corresponding type registry changes for the same

2019-11-14 Thread Sarath Subramanian


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71649/#review218640
---




intg/src/main/java/org/apache/atlas/AtlasErrorCode.java
Lines 163 (patched)


it will be useful to include namespace name and attribute name in the error 
message.



intg/src/main/java/org/apache/atlas/type/AtlasEntityType.java
Lines 27 (patched)


nit: unused import



intg/src/test/java/org/apache/atlas/TestUtilsV2.java
Lines 1466 (patched)


defineNamespaceTypesDef() is never used. please review



repository/src/main/java/org/apache/atlas/repository/store/graph/AtlasTypeDefGraphStore.java
Lines 559 (patched)


preDeleteByGuid() and preDeleteByName() may not be needed. Please review.



repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasNamespaceDefStoreV2.java
Lines 253 (patched)


preDeleteByName() and preDeleteByGuid() may not be needed. We will not be 
deleting any reference edges from the namespaceDef vertex (since no edges 
exists).



repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasNamespaceDefStoreV2.java
Lines 264 (patched)


this is not valid. NamespaceDef should be deleted only when no entity has 
reference to that namespace name.

Maybe check if there are any entity referring this namespace (using index 
query?) before deleting.

Same for line #294



repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasNamespaceDefStoreV2.java
Lines 311 (patched)


toNamespaceDef => toAtlasNamespaceDef



repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasNamespaceDefStoreV2.java
Lines 321 (patched)


toNamespaceDef => toAtlasNamespaceDef



repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasNamespaceDefStoreV2.java
Lines 382 (patched)


toJsonFromAttribute => toJsonFromNamespaceAttribute



repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasNamespaceDefStoreV2.java
Lines 420 (patched)


toAttributeDefFromJson => toNamespaceAttributeDefFromJson



repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasNamespaceDefStoreV2.java
Lines 421 (patched)


AtlasBaseException is never thrown



repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasNamespaceDefStoreV2.java
Lines 524 (patched)


we may need to check for NamespaceAttributeDef with empty/null 
'applicableTypes' => 
CollectionUtils.isEmpty(attributeDef.getApplicableEntityTypes())



repository/src/test/java/org/apache/atlas/repository/store/graph/AtlasTypeDefGraphStoreTest.java
Lines 697 (patched)


empty test?



webapp/src/main/java/org/apache/atlas/examples/QuickStartV2.java
Lines 22 (patched)


nit: review unused imports


- Sarath Subramanian


On Nov. 13, 2019, 4:08 p.m., Aadarsh Jajodia wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71649/
> ---
> 
> (Updated Nov. 13, 2019, 4:08 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Sridhar K, Le Ma, Madhan 
> Neethiraj, and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3486
> https://issues.apache.org/jira/browse/ATLAS-3486
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> This change is the first part of the bigger task defined as part of 
> ATLAS-3485.
> This adds the data model needed for supporting namespaces and namespace 
> attributes and also updates the type registry to include the applicable 
> namespace attributes for every entity type
> 
> 
> Diffs
> -
> 
>   intg/src/main/java/org/apache/atlas/AtlasErrorCode.java 
> 7a2aae2e9ae8174c5309164f3e41c940cbf3ddf8 
>   intg/src/main/java/org/apache/atlas/model/TypeCategory.java 
> f06f64f450f407e3f9a0e742726ff4dd12ccc695 
>   intg/src/main/java/org/apache/atlas/model/typedef/AtlasNamespaceDef.java 
> PRE-CREATION 
>   intg/src/main/java/org/apache/atlas/model/typedef/AtlasStructDef.java 
> e10965b870c4a300b41e93ee046b5f6d6b722728 
>

Re: Review Request 71649: ATLAS-3486 Define Data Models For Namespaces and Namespace Attributes in Atlas and also the corresponding type registry changes for the same

2019-11-13 Thread Aadarsh Jajodia


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71649/
---

(Updated Nov. 14, 2019, 12:08 a.m.)


Review request for atlas, Ashutosh Mestry, Sridhar K, Le Ma, Madhan Neethiraj, 
and Sarath Subramanian.


Changes
---

Addressed review comments


Bugs: ATLAS-3486
https://issues.apache.org/jira/browse/ATLAS-3486


Repository: atlas


Description
---

This change is the first part of the bigger task defined as part of ATLAS-3485.
This adds the data model needed for supporting namespaces and namespace 
attributes and also updates the type registry to include the applicable 
namespace attributes for every entity type


Diffs (updated)
-

  intg/src/main/java/org/apache/atlas/AtlasErrorCode.java 
7a2aae2e9ae8174c5309164f3e41c940cbf3ddf8 
  intg/src/main/java/org/apache/atlas/model/TypeCategory.java 
f06f64f450f407e3f9a0e742726ff4dd12ccc695 
  intg/src/main/java/org/apache/atlas/model/typedef/AtlasNamespaceDef.java 
PRE-CREATION 
  intg/src/main/java/org/apache/atlas/model/typedef/AtlasStructDef.java 
e10965b870c4a300b41e93ee046b5f6d6b722728 
  intg/src/main/java/org/apache/atlas/model/typedef/AtlasTypesDef.java 
3634fdfd313639eb97b3c4698e091487b0e44a80 
  intg/src/main/java/org/apache/atlas/store/AtlasTypeDefStore.java 
4ee68a936f99bb4c819b5335da2cc8bf7d539397 
  intg/src/main/java/org/apache/atlas/type/AtlasEntityType.java 
884447f81c57faf917a8d0565fc0a0c7ebbd99f0 
  intg/src/main/java/org/apache/atlas/type/AtlasNamespaceType.java PRE-CREATION 
  intg/src/main/java/org/apache/atlas/type/AtlasTypeRegistry.java 
8b4fd1c3b9005d0a8852f2828475b4ad6a806822 
  intg/src/main/java/org/apache/atlas/type/AtlasTypeUtil.java 
0883d54f490e22c6510e6fc0cb804b87713a7ecb 
  intg/src/main/java/org/apache/atlas/typesystem/types/DataTypes.java 
dba2d88146eff314191ae6bb24ad7337b0ea10ae 
  intg/src/test/java/org/apache/atlas/TestRelationshipUtilsV2.java 
02613b5f7250b14324ed294c22de079b74d55b08 
  intg/src/test/java/org/apache/atlas/TestUtilsV2.java 
530d5cda44b7d3746f5cb8dd5a23b3c68b254cac 
  intg/src/test/java/org/apache/atlas/model/typedef/TestAtlasNamespaceDef.java 
PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/AtlasTypeDefGraphStore.java
 2e2ab1a664171555c57560e1c0b4cbdbc20c0f6f 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasNamespaceDefStoreV2.java
 PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasTypeDefGraphStoreV2.java
 a5ccfb5b2055c88f596312f4033bc0034d3d165c 
  
repository/src/test/java/org/apache/atlas/repository/store/graph/AtlasTypeDefGraphStoreTest.java
 51dd16b8518c9a16088547f3e95c0ef401695895 
  
repository/src/test/java/org/apache/atlas/repository/store/graph/v2/AtlasNamespaceDefStoreV2Test.java
 PRE-CREATION 
  webapp/src/main/java/org/apache/atlas/examples/QuickStartV2.java 
6cd0ee331b7ae24757b58e76ec47bf556106846a 


Diff: https://reviews.apache.org/r/71649/diff/4/

Changes: https://reviews.apache.org/r/71649/diff/3-4/


Testing
---

Added unit tests


Thanks,

Aadarsh Jajodia

Re: Review Request 71649: ATLAS-3486 Define Data Models For Namespaces and Namespace Attributes in Atlas and also the corresponding type registry changes for the same

2019-11-09 Thread Madhan Neethiraj


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71649/#review218589
---




intg/src/main/java/org/apache/atlas/model/typedef/AtlasNamespaceDef.java
Lines 46 (patched)


I suggest to use List instead  of Set, to be consistent with other 
attributeDef lists - in AtlasStructDef 
(AtlasEntityDef/AtlasClassificationDef/AtlasRelationshipDef).



intg/src/main/java/org/apache/atlas/model/typedef/AtlasTypesDef.java
Line 83 (original), 86 (patched)


Consider adding a constructor, instead of updating existing one.



intg/src/main/java/org/apache/atlas/type/AtlasEntityType.java
Lines 271 (patched)


Consider marking populateNamespaceAttributes() method as private, as this 
is called only from this class.  Also, move this impl to later in the class, 
after all public and protected methods.



intg/src/main/java/org/apache/atlas/type/AtlasEntityType.java
Lines 272 (patched)


Instead of iteraring namespaceDef, consider iterating through 
namespaceTypes:

  private void populateNamespaceAttributes(AtlasTypeRegistry typeRegistry) {
for (AtlasNamespaceType nsType : typeRegistry.getAllNamespaceTypes()) {
  for (AtlasNamespaceAttribute nsAttribute : 
namespaceType.getAttributes()) {
if 
(nsAttribute.getApplicableEntityTypes().contains(this.getTypeName())) {
  addNamespaceAttribute(nsAttribute);
}
  }
}
  }

  private void addNamespaceAttribute(AtlasNamespaceAttribute nsAttribute) {
StringnsName = 
nsAttribute.getDefinedInType().getTypeName();
List attributes = 
namespaceAttributes.get(nsName);

if (attributes == null) {
  attributes = new ArrayList<>();

  namespaceAttributes.put(nsName, attributes);
}

attributes.add(nsAttribute);
  }



intg/src/main/java/org/apache/atlas/type/AtlasEntityType.java
Lines 272 (patched)


Instead of iteraring namespaceDef, consider iterating through 
namespaceTypes:

  private void populateNamespaceAttributes(AtlasTypeRegistry typeRegistry) {
for (AtlasNamespaceType nsType : typeRegistry.getAllNamespaceTypes()) {
  for (AtlasNamespaceAttribute nsAttribute : 
namespaceType.getAttributes()) {
if 
(nsAttribute.getApplicableEntityTypes().contains(this.getTypeName())) {
  addNamespaceAttribute(nsAttribute);
}
  }
}
  }

  private void addNamespaceAttribute(AtlasNamespaceAttribute nsAttribute) {
StringnsName = 
nsAttribute.getDefinedInType().getTypeName();
List attributes = 
namespaceAttributes.get(nsName);

if (attributes == null) {
  attributes = new ArrayList<>();

  namespaceAttributes.put(nsName, attributes);
}

attributes.add(nsAttribute);
  }



intg/src/main/java/org/apache/atlas/type/AtlasTypeUtil.java
Line 310 (original), 302 (patched)


retain existing getTypesDef() method, to avoid breaking exising callers of 
this method. Add a new method that takes 'namespaces' parameter.



repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasNamespaceDefStoreV2.java
Lines 191 (patched)


"struct-def" => "namespace-def"



repository/src/test/java/org/apache/atlas/repository/tagpropagation/ClassificationPropagationTest.java
Line 619 (original), 619 (patched)


existing constructor for AtlasTypesDef should be retained; changes to this 
file can be reverted.



tools/classification-updater/src/main/java/org/apache/atlas/tools/BulkFetchAndUpdate.java
Line 524 (original), 524 (patched)


existing constructor for AtlasTypesDef should be retained; changes to this 
file can be reverted.


- Madhan Neethiraj


On Nov. 8, 2019, 9:42 p.m., Aadarsh Jajodia wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71649/
> ---
> 
> (Updated Nov. 8, 2019, 9:42 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Sridhar K, Le Ma, Madhan 
> Neethiraj, and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3486
> https://issues.apache.org/jira/browse/ATLAS-3486
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
>

Re: Review Request 71649: ATLAS-3486 Define Data Models For Namespaces and Namespace Attributes in Atlas and also the corresponding type registry changes for the same

2019-11-08 Thread Aadarsh Jajodia


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71649/
---

(Updated Nov. 8, 2019, 9:42 p.m.)


Review request for atlas, Ashutosh Mestry, Sridhar K, Le Ma, Madhan Neethiraj, 
and Sarath Subramanian.


Summary (updated)
-

ATLAS-3486 Define Data Models For Namespaces and Namespace Attributes in Atlas 
and also the corresponding type registry changes for the same


Bugs: ATLAS-3486
https://issues.apache.org/jira/browse/ATLAS-3486


Repository: atlas


Description (updated)
---

This change is the first part of the bigger task defined as part of ATLAS-3485.
This adds the data model needed for supporting namespaces and namespace 
attributes and also updates the type registry to include the applicable 
namespace attributes for every entity type


Diffs
-

  intg/src/main/java/org/apache/atlas/model/TypeCategory.java 
f06f64f450f407e3f9a0e742726ff4dd12ccc695 
  intg/src/main/java/org/apache/atlas/model/typedef/AtlasNamespaceDef.java 
PRE-CREATION 
  intg/src/main/java/org/apache/atlas/model/typedef/AtlasStructDef.java 
e10965b870c4a300b41e93ee046b5f6d6b722728 
  intg/src/main/java/org/apache/atlas/model/typedef/AtlasTypesDef.java 
3634fdfd313639eb97b3c4698e091487b0e44a80 
  intg/src/main/java/org/apache/atlas/store/AtlasTypeDefStore.java 
4ee68a936f99bb4c819b5335da2cc8bf7d539397 
  intg/src/main/java/org/apache/atlas/type/AtlasEntityType.java 
884447f81c57faf917a8d0565fc0a0c7ebbd99f0 
  intg/src/main/java/org/apache/atlas/type/AtlasNamespaceType.java PRE-CREATION 
  intg/src/main/java/org/apache/atlas/type/AtlasTypeRegistry.java 
8b4fd1c3b9005d0a8852f2828475b4ad6a806822 
  intg/src/main/java/org/apache/atlas/type/AtlasTypeUtil.java 
0883d54f490e22c6510e6fc0cb804b87713a7ecb 
  intg/src/main/java/org/apache/atlas/typesystem/types/DataTypes.java 
dba2d88146eff314191ae6bb24ad7337b0ea10ae 
  intg/src/test/java/org/apache/atlas/TestRelationshipUtilsV2.java 
02613b5f7250b14324ed294c22de079b74d55b08 
  intg/src/test/java/org/apache/atlas/TestUtilsV2.java 
530d5cda44b7d3746f5cb8dd5a23b3c68b254cac 
  intg/src/test/java/org/apache/atlas/model/typedef/TestAtlasNamespaceDef.java 
PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/AtlasTypeDefGraphStore.java
 2e2ab1a664171555c57560e1c0b4cbdbc20c0f6f 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasNamespaceDefStoreV2.java
 PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasTypeDefGraphStoreV2.java
 a5ccfb5b2055c88f596312f4033bc0034d3d165c 
  
repository/src/test/java/org/apache/atlas/repository/store/graph/AtlasTypeDefGraphStoreTest.java
 51dd16b8518c9a16088547f3e95c0ef401695895 
  
repository/src/test/java/org/apache/atlas/repository/store/graph/v2/AtlasNamespaceDefStoreV2Test.java
 PRE-CREATION 
  
repository/src/test/java/org/apache/atlas/repository/tagpropagation/ClassificationPropagationTest.java
 6f9c05e7a7a30c678a0a376d25402c6026d0b391 
  
tools/classification-updater/src/main/java/org/apache/atlas/tools/BulkFetchAndUpdate.java
 91ff89a00d779af9437ba5f6d129bd4595e5f036 
  webapp/src/main/java/org/apache/atlas/examples/QuickStartV2.java 
6cd0ee331b7ae24757b58e76ec47bf556106846a 


Diff: https://reviews.apache.org/r/71649/diff/3/


Testing
---

Added unit tests


Thanks,

Aadarsh Jajodia

Re: Review Request 71649: ATLAS-3486 Define Data Models For Namespaces and Namespace Attributes in Atlas and also the corresponding type registry changes for the same

Re: Review Request 71649: ATLAS-3486 Define Data Models For Namespaces and Namespace Attributes in Atlas and also the corresponding type registry changes for the same

Re: Review Request 71649: ATLAS-3486 Define Data Models For Namespaces and Namespace Attributes in Atlas and also the corresponding type registry changes for the same

Re: Review Request 71649: ATLAS-3486 Define Data Models For Namespaces and Namespace Attributes in Atlas and also the corresponding type registry changes for the same

Re: Review Request 71649: ATLAS-3486 Define Data Models For Namespaces and Namespace Attributes in Atlas and also the corresponding type registry changes for the same

Re: Review Request 71649: ATLAS-3486 Define Data Models For Namespaces and Namespace Attributes in Atlas and also the corresponding type registry changes for the same

Re: Review Request 71649: ATLAS-3486 Define Data Models For Namespaces and Namespace Attributes in Atlas and also the corresponding type registry changes for the same

Re: Review Request 71649: ATLAS-3486 Define Data Models For Namespaces and Namespace Attributes in Atlas and also the corresponding type registry changes for the same

Re: Review Request 71649: ATLAS-3486 Define Data Models For Namespaces and Namespace Attributes in Atlas and also the corresponding type registry changes for the same

Re: Review Request 71649: ATLAS-3486 Define Data Models For Namespaces and Namespace Attributes in Atlas and also the corresponding type registry changes for the same

Re: Review Request 71649: ATLAS-3486 Define Data Models For Namespaces and Namespace Attributes in Atlas and also the corresponding type registry changes for the same

Re: Review Request 71649: ATLAS-3486 Define Data Models For Namespaces and Namespace Attributes in Atlas and also the corresponding type registry changes for the same

Re: Review Request 71649: ATLAS-3486 Define Data Models For Namespaces and Namespace Attributes in Atlas and also the corresponding type registry changes for the same

Re: Review Request 71649: ATLAS-3486 Define Data Models For Namespaces and Namespace Attributes in Atlas and also the corresponding type registry changes for the same

Re: Review Request 71649: ATLAS-3486 Define Data Models For Namespaces and Namespace Attributes in Atlas and also the corresponding type registry changes for the same

15 matches

Site Navigation

Mail list logo

Footer information