Re: Review Request 71922: ATLAS-3564: New version of AWS S3 model addition

[Sidharth Mishra]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71922/
---

(Updated Jan. 11, 2020, 1:50 a.m.)


Review request for atlas, Ashutosh Mestry, Sridhar K, Madhan Neethiraj, and 
Sarath Subramanian.


Bugs: ATLAS-3564
https://issues.apache.org/jira/browse/ATLAS-3564


Repository: atlas


Description
---

This is the new version) of aws s3 model at atlas to allow the hierarchical 
structure and attributes same as AWS S3 Console. The existing aws s3 models has 
limitations like pseudo directory not containing another pseudo directory, 
pseudo directory doesn't have attributes of aws object (as some case it gets 
populated), both object and bucket doesn't containing all the attributes of AWS 
S3 etc.


Diffs (updated)
-

  addons/models/3000-Cloud/3030-aws_s3_v2_typedefs.json PRE-CREATION 


Diff: https://reviews.apache.org/r/71922/diff/5/

Changes: https://reviews.apache.org/r/71922/diff/4-5/


Testing
---


Thanks,

Sidharth Mishra

[jira] [Commented] (ATLAS-3486) Define Data Models For Namespaces and Namespace Attributes in Atlas

[ASF subversion and git services (Jira)]

[ 
https://issues.apache.org/jira/browse/ATLAS-3486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17013259#comment-17013259
 ] 

ASF subversion and git services commented on ATLAS-3486:


Commit 926a198c228928d278d1b336eb42c8f22d4bce3b in atlas's branch 
refs/heads/branch-2.0 from Aadarsh Jajodia
[ https://gitbox.apache.org/repos/asf?p=atlas.git;h=926a198 ]

ATLAS-3486: Introduce Atlas NamespaceDef

Signed-off-by: Sarath Subramanian 
(cherry picked from commit 6d94f39cf3b6207bc095310eebbe3292071be68b)


> Define Data Models For Namespaces and Namespace Attributes in Atlas
> ---
>
> Key: ATLAS-3486
> URL: https://issues.apache.org/jira/browse/ATLAS-3486
> Project: Atlas
>  Issue Type: Sub-task
>  Components:  atlas-core
>Affects Versions: 2.0.0
>Reporter: Aadarsh Jajodia
>Assignee: Aadarsh Jajodia
>Priority: Major
> Fix For: 2.1.0
>
>
> This change defines the data models needed for namespace definition and 
> namespace attribute definitions



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (ATLAS-3486) Define Data Models For Namespaces and Namespace Attributes in Atlas

[ASF subversion and git services (Jira)]

[ 
https://issues.apache.org/jira/browse/ATLAS-3486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17013258#comment-17013258
 ] 

ASF subversion and git services commented on ATLAS-3486:


Commit 6d94f39cf3b6207bc095310eebbe3292071be68b in atlas's branch 
refs/heads/master from Aadarsh Jajodia
[ https://gitbox.apache.org/repos/asf?p=atlas.git;h=6d94f39 ]

ATLAS-3486: Introduce Atlas NamespaceDef

Signed-off-by: Sarath Subramanian 


> Define Data Models For Namespaces and Namespace Attributes in Atlas
> ---
>
> Key: ATLAS-3486
> URL: https://issues.apache.org/jira/browse/ATLAS-3486
> Project: Atlas
>  Issue Type: Sub-task
>  Components:  atlas-core
>Affects Versions: 2.0.0
>Reporter: Aadarsh Jajodia
>Assignee: Aadarsh Jajodia
>Priority: Major
> Fix For: 2.1.0
>
>
> This change defines the data models needed for namespace definition and 
> namespace attribute definitions



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Review Request 71982: ATLAS-3581:HBase hook converts qualifiedName to lowercase for namespace, table, column-family and column

[Abhay Kulkarni]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71982/#review219220
---


Ship it!




Ship It!

- Abhay Kulkarni


On Jan. 10, 2020, 9:56 p.m., Ramesh Mani wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71982/
> ---
> 
> (Updated Jan. 10, 2020, 9:56 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Abhay Kulkarni, Madhan Neethiraj, 
> and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3581
> https://issues.apache.org/jira/browse/ATLAS-3581
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> ATLAS-3581:HBase hook converts qualifiedName to lowercase for namespace, 
> table, column-family and column
> 
> 
> Diffs
> -
> 
>   
> addons/hbase-bridge/src/main/java/org/apache/atlas/hbase/bridge/HBaseAtlasHook.java
>  26be9b3 
>   
> addons/hbase-bridge/src/main/java/org/apache/atlas/hbase/bridge/HBaseBridge.java
>  fde70f1 
> 
> 
> Diff: https://reviews.apache.org/r/71982/diff/1/
> 
> 
> Testing
> ---
> 
> Verified in local VM
> 
> 
> Thanks,
> 
> Ramesh Mani
> 
>

Re: Review Request 71982: ATLAS-3581:HBase hook converts qualifiedName to lowercase for namespace, table, column-family and column

[Ramesh Mani]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71982/
---

(Updated Jan. 10, 2020, 9:56 p.m.)


Review request for atlas, Ashutosh Mestry, Abhay Kulkarni, Madhan Neethiraj, 
and Sarath Subramanian.


Changes
---

Added BUG ID


Bugs: ATLAS-3581
https://issues.apache.org/jira/browse/ATLAS-3581


Repository: atlas


Description
---

ATLAS-3581:HBase hook converts qualifiedName to lowercase for namespace, table, 
column-family and column


Diffs
-

  
addons/hbase-bridge/src/main/java/org/apache/atlas/hbase/bridge/HBaseAtlasHook.java
 26be9b3 
  
addons/hbase-bridge/src/main/java/org/apache/atlas/hbase/bridge/HBaseBridge.java
 fde70f1 


Diff: https://reviews.apache.org/r/71982/diff/1/


Testing
---

Verified in local VM


Thanks,

Ramesh Mani

Review Request 71982: ATLAS-3581:HBase hook converts qualifiedName to lowercase for namespace, table, column-family and column

[Ramesh Mani]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71982/
---

Review request for atlas, Ashutosh Mestry, Abhay Kulkarni, Madhan Neethiraj, 
and Sarath Subramanian.


Repository: atlas


Description
---

ATLAS-3581:HBase hook converts qualifiedName to lowercase for namespace, table, 
column-family and column


Diffs
-

  
addons/hbase-bridge/src/main/java/org/apache/atlas/hbase/bridge/HBaseAtlasHook.java
 26be9b3 
  
addons/hbase-bridge/src/main/java/org/apache/atlas/hbase/bridge/HBaseBridge.java
 fde70f1 


Diff: https://reviews.apache.org/r/71982/diff/1/


Testing
---

Verified in local VM


Thanks,

Ramesh Mani

Re: Review Request 71922: ATLAS-3564: New version of AWS S3 model addition

[Sidharth Mishra]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71922/
---

(Updated Jan. 10, 2020, 7:13 p.m.)


Review request for atlas, Ashutosh Mestry, Sridhar K, Madhan Neethiraj, and 
Sarath Subramanian.


Bugs: ATLAS-3564
https://issues.apache.org/jira/browse/ATLAS-3564


Repository: atlas


Description
---

This is the new version) of aws s3 model at atlas to allow the hierarchical 
structure and attributes same as AWS S3 Console. The existing aws s3 models has 
limitations like pseudo directory not containing another pseudo directory, 
pseudo directory doesn't have attributes of aws object (as some case it gets 
populated), both object and bucket doesn't containing all the attributes of AWS 
S3 etc.


Diffs (updated)
-

  addons/models/3000-Cloud/3030-aws_s3_typedefs_v2.json PRE-CREATION 


Diff: https://reviews.apache.org/r/71922/diff/4/

Changes: https://reviews.apache.org/r/71922/diff/3-4/


Testing
---


Thanks,

Sidharth Mishra

Re: Review Request 71922: ATLAS-3564: New version of AWS S3 model addition

[Sidharth Mishra]

> On Jan. 10, 2020, 6:26 p.m., Ashutosh Mestry wrote:
> > addons/models/3000-Cloud/3030-aws_s3_typedefs_v2.json
> > Lines 15 (patched)
> > 
> >
> > Rename: aws_s3_v2_contained (since containee does not exist in english 
> > language)

What will be relationship name - aws_s3_v2_container_contained?


> On Jan. 10, 2020, 6:26 p.m., Ashutosh Mestry wrote:
> > addons/models/3000-Cloud/3030-aws_s3_typedefs_v2.json
> > Lines 20 (patched)
> > 
> >
> > Should contain ownerId.

We have OwnerName which will be usefull for Atlas. Do you think OwnerId like 
f9fca4893bbb1d52d88edbcf22a42cc5b81c29496c8b881fc160faab93c73945 which is 
specific to aws it worth adding to atals model?


> On Jan. 10, 2020, 6:26 p.m., Ashutosh Mestry wrote:
> > addons/models/3000-Cloud/3030-aws_s3_typedefs_v2.json
> > Lines 21 (patched)
> > 
> >
> > Add bucketName, since all objects have it.

The qualified name in each object will have bucketName. Should we add another 
item for it. That will be redundant.


> On Jan. 10, 2020, 6:26 p.m., Ashutosh Mestry wrote:
> > addons/models/3000-Cloud/3030-aws_s3_typedefs_v2.json
> > Lines 23 (patched)
> > 
> >
> > Considering adding: bucketName.

same as above.


> On Jan. 10, 2020, 6:26 p.m., Ashutosh Mestry wrote:
> > addons/models/3000-Cloud/3030-aws_s3_typedefs_v2.json
> > Lines 58 (patched)
> > 
> >
> > Should be moved to base class. Since every s3 object has this.

This is the object base type and directory is derived from this


> On Jan. 10, 2020, 6:26 p.m., Ashutosh Mestry wrote:
> > addons/models/3000-Cloud/3030-aws_s3_typedefs_v2.json
> > Lines 111 (patched)
> > 
> >
> > Consider adding: Region

The qualified name in each object will have Region as well. Should we add 
another item for it. That will be redundant.


> On Jan. 10, 2020, 6:26 p.m., Ashutosh Mestry wrote:
> > addons/models/3000-Cloud/3030-aws_s3_typedefs_v2.json
> > Lines 121 (patched)
> > 
> >
> > What is the purpose of this?

This is being added for backward compatibility i.e. if we can eventually move 
from older version to new one completely.


> On Jan. 10, 2020, 6:26 p.m., Ashutosh Mestry wrote:
> > addons/models/3000-Cloud/3030-aws_s3_typedefs_v2.json
> > Lines 129 (patched)
> > 
> >
> > What is the purpose of this?

same as above


> On Jan. 10, 2020, 6:26 p.m., Ashutosh Mestry wrote:
> > addons/models/3000-Cloud/3030-aws_s3_typedefs_v2.json
> > Lines 137 (patched)
> > 
> >
> > Rename: currentStorageLocation to location.

rename to storageLocation as only location may be confusing with path or key. 
Please suggest.


> On Jan. 10, 2020, 6:26 p.m., Ashutosh Mestry wrote:
> > addons/models/3000-Cloud/3030-aws_s3_typedefs_v2.json
> > Lines 190 (patched)
> > 
> >
> > What is the purpose of this?

This is being added for backward compatibility i.e. if we can eventually move 
from older version to new one completely.


> On Jan. 10, 2020, 6:26 p.m., Ashutosh Mestry wrote:
> > addons/models/3000-Cloud/3030-aws_s3_typedefs_v2.json
> > Lines 214 (patched)
> > 
> >
> > I think this attribute is at object level.

I have removed isEncrypted.


> On Jan. 10, 2020, 6:26 p.m., Ashutosh Mestry wrote:
> > addons/models/3000-Cloud/3030-aws_s3_typedefs_v2.json
> > Lines 222 (patched)
> > 
> >
> > This is at object level.

Bucket is not derived from object and thats the reason we have this at both the 
places. Only this and tag is common property among object and bucket. Do you 
think extracting these two to another basetype will make sense?


> On Jan. 10, 2020, 6:26 p.m., Ashutosh Mestry wrote:
> > addons/models/3000-Cloud/3030-aws_s3_typedefs_v2.json
> > Lines 238 (patched)
> > 
> >
> > This should be part of base class.

same as above.


- Sidharth


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71922/#review219215
---


On Jan. 8, 2020, 11:12 p.m., Sidharth Mishra

[jira] [Created] (ATLAS-3581) HBase hook converts qualifiedName to lowercase for namespace, table, column-family and column

[Ramesh Mani (Jira)]

Ramesh Mani created ATLAS-3581:
--

 Summary: HBase hook converts qualifiedName to lowercase for 
namespace, table, column-family and column
 Key: ATLAS-3581
 URL: https://issues.apache.org/jira/browse/ATLAS-3581
 Project: Atlas
  Issue Type: Bug
  Components: atlas-intg
Reporter: Ramesh Mani


HBase hook converts qualifiedName to lowercase for namespace, table, 
column-family and column



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Assigned] (ATLAS-3581) HBase hook converts qualifiedName to lowercase for namespace, table, column-family and column

[Ramesh Mani (Jira)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-3581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ramesh Mani reassigned ATLAS-3581:
--

Assignee: Ramesh Mani

> HBase hook converts qualifiedName to lowercase for namespace, table, 
> column-family and column
> -
>
> Key: ATLAS-3581
> URL: https://issues.apache.org/jira/browse/ATLAS-3581
> Project: Atlas
>  Issue Type: Bug
>  Components: atlas-intg
>Reporter: Ramesh Mani
>Assignee: Ramesh Mani
>Priority: Major
>
> HBase hook converts qualifiedName to lowercase for namespace, table, 
> column-family and column



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Review Request 71922: ATLAS-3564: New version of AWS S3 model addition

[Ashutosh Mestry via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71922/#review219215
---




addons/models/3000-Cloud/3030-aws_s3_typedefs_v2.json
Lines 15 (patched)


Rename: aws_s3_v2_contained (since containee does not exist in english 
language)



addons/models/3000-Cloud/3030-aws_s3_typedefs_v2.json
Lines 20 (patched)


Should contain ownerId.



addons/models/3000-Cloud/3030-aws_s3_typedefs_v2.json
Lines 21 (patched)


Add bucketName, since all objects have it.



addons/models/3000-Cloud/3030-aws_s3_typedefs_v2.json
Lines 23 (patched)


Considering adding: bucketName.



addons/models/3000-Cloud/3030-aws_s3_typedefs_v2.json
Lines 24 (patched)


Consider adding: encryption



addons/models/3000-Cloud/3030-aws_s3_typedefs_v2.json
Lines 58 (patched)


Should be moved to base class. Since every s3 object has this.



addons/models/3000-Cloud/3030-aws_s3_typedefs_v2.json
Lines 102 (patched)


Rename: aws_s3_v2_directory



addons/models/3000-Cloud/3030-aws_s3_typedefs_v2.json
Lines 111 (patched)


Consider adding: Region



addons/models/3000-Cloud/3030-aws_s3_typedefs_v2.json
Lines 121 (patched)


What is the purpose of this?



addons/models/3000-Cloud/3030-aws_s3_typedefs_v2.json
Lines 129 (patched)


What is the purpose of this?



addons/models/3000-Cloud/3030-aws_s3_typedefs_v2.json
Lines 137 (patched)


Rename: currentStorageLocation to location.



addons/models/3000-Cloud/3030-aws_s3_typedefs_v2.json
Lines 168 (patched)


Need not be indexable.



addons/models/3000-Cloud/3030-aws_s3_typedefs_v2.json
Lines 190 (patched)


What is the purpose of this?



addons/models/3000-Cloud/3030-aws_s3_typedefs_v2.json
Lines 214 (patched)


I think this attribute is at object level.



addons/models/3000-Cloud/3030-aws_s3_typedefs_v2.json
Lines 222 (patched)


This is at object level.



addons/models/3000-Cloud/3030-aws_s3_typedefs_v2.json
Lines 238 (patched)


This should be part of base class.


- Ashutosh Mestry


On Jan. 8, 2020, 11:12 p.m., Sidharth Mishra wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71922/
> ---
> 
> (Updated Jan. 8, 2020, 11:12 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Sridhar K, Madhan Neethiraj, and 
> Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3564
> https://issues.apache.org/jira/browse/ATLAS-3564
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> This is the new version) of aws s3 model at atlas to allow the hierarchical 
> structure and attributes same as AWS S3 Console. The existing aws s3 models 
> has limitations like pseudo directory not containing another pseudo 
> directory, pseudo directory doesn't have attributes of aws object (as some 
> case it gets populated), both object and bucket doesn't containing all the 
> attributes of AWS S3 etc.
> 
> 
> Diffs
> -
> 
>   addons/models/3000-Cloud/3030-aws_s3_typedefs_v2.json PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/71922/diff/3/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Sidharth Mishra
> 
>

[jira] [Commented] (ATLAS-3576) UI: Improve system attributes presentation

[Keval Bhatt (Jira)]

[ 
https://issues.apache.org/jira/browse/ATLAS-3576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17012867#comment-17012867
 ] 

Keval Bhatt commented on ATLAS-3576:


Please use the updated patch [^ATLAS-3576-3.patch]

> UI: Improve system attributes presentation 
> ---
>
> Key: ATLAS-3576
> URL: https://issues.apache.org/jira/browse/ATLAS-3576
> Project: Atlas
>  Issue Type: Improvement
>Reporter: Keval Bhatt
>Assignee: Keval Bhatt
>Priority: Major
> Attachments: ATLAS-3576-3.patch, ATLAS-3576-v1.patch, 
> ATLAS-3576.patch, Screen Shot 2020-01-08 at 9.19.00 PM.png, Screen Shot 
> 2020-01-08 at 9.19.20 PM.png
>
>
> Before Patch:
> !Screen Shot 2020-01-08 at 9.19.00 PM.png|width=618,height=284!
>  
> After Patch:
> !Screen Shot 2020-01-08 at 9.19.20 PM.png|width=622,height=271!
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (ATLAS-3576) UI: Improve system attributes presentation

[Keval Bhatt (Jira)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-3576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Keval Bhatt updated ATLAS-3576:
---
Attachment: ATLAS-3576-3.patch

> UI: Improve system attributes presentation 
> ---
>
> Key: ATLAS-3576
> URL: https://issues.apache.org/jira/browse/ATLAS-3576
> Project: Atlas
>  Issue Type: Improvement
>Reporter: Keval Bhatt
>Assignee: Keval Bhatt
>Priority: Major
> Attachments: ATLAS-3576-3.patch, ATLAS-3576-v1.patch, 
> ATLAS-3576.patch, Screen Shot 2020-01-08 at 9.19.00 PM.png, Screen Shot 
> 2020-01-08 at 9.19.20 PM.png
>
>
> Before Patch:
> !Screen Shot 2020-01-08 at 9.19.00 PM.png|width=618,height=284!
>  
> After Patch:
> !Screen Shot 2020-01-08 at 9.19.20 PM.png|width=622,height=271!
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (ATLAS-2916) Unnecessary entity update causes by different AtlasObjectId

[chaitali borole (Jira)]

[ 
https://issues.apache.org/jira/browse/ATLAS-2916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17012583#comment-17012583
 ] 

chaitali borole commented on ATLAS-2916:


Hi,

Can you please provide steps to reproduce this bug.

> Unnecessary entity update causes by different AtlasObjectId
> ---
>
> Key: ATLAS-2916
> URL: https://issues.apache.org/jira/browse/ATLAS-2916
> Project: Atlas
>  Issue Type: Improvement
>  Components:  atlas-core
>Affects Versions: 1.0.0
>Reporter: Jiaqi Shan
>Priority: Minor
>
>     When creating a Hive process,  ColumnEntity will update with the value of 
> TableEntity's AtlasObjectId.  There is an example to show the differences 
> bwtween CurrentEntity and EntityInStore.
> {panel:title=CurrentEntity}
> AtlasEntity{AtlasStruct{typeName='hive_column', attributes=[owner:bi_sh, 
> qualifiedName:bi.yingbang.t@bdyf, name:t, comment:t, position:1, type:int, 
> table:{color:#d04437}AtlasObjectId\{guid='-17020754238878791', 
> typeName='hive_table', 
> uniqueAttributes={qualifiedName:b.yingbang@bdyf}}{color}]}guid='431c8847-8fd2-454d-b77a-19aeef0d6b9b',
>  status=null, createdBy='null', updatedBy='null', createTime=null, 
> updateTime=null, version=0, relationshipAttributes=[], classifications=[], 
> meanings=[]}
> {panel}
> {panel:title=EntityInStore}
> AtlasEntity{AtlasStruct{typeName='hive_column', attributes=[owner:bi_sh, 
> qualifiedName:bi.yingbang.t@bdyf, name:t, description:null, comment:t, 
> position:1, type:int, 
> {color:#d04437}table:AtlasObjectId\{guid='da35aff2-9851-499d-99cf-f1fbafb6e92b',
>  typeName='hive_table', 
> uniqueAttributes={}}{color}]}guid='431c8847-8fd2-454d-b77a-19aeef0d6b9b', 
> status=ACTIVE, createdBy='bi_sh', updatedBy='bi_sh', 
> createTime=2018-10-09T11:26:51.685Z, updateTime=2018-10-09T11:26:51.685Z, 
> version=0, relationshipAttributes=[], classifications=[], meanings=[]}
> {panel}
>      Actually there is no metadata changed in ColumnEntity, the difference of 
> table's AtlasObjectId is caused by Hive Hook setting a new guid for 
> TableEntity. So maybe it's not necessary to update Hive column entity in this 
> instance.
>      We propose to add a LRU cache to skip updating the same entitiy which 
> was sent in an earlier notification. But in situation deleting and 
> re-creating the entity with the same uniqueAttributes, this solution goes 
> wrong.
>      Is there any other good solution to aviod this problem?
>  
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)