[jira] [Updated] (HDFS-12502) nntop should support a category based on FilesInGetListingOps

2017-10-24 Thread Zhe Zhang (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HDFS-12502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhe Zhang updated HDFS-12502:
-
Fix Version/s: (was: 3.1.0)
   (was: 3.0.0)
   (was: 2.8.3)
   (was: 2.9.0)

> nntop should support a category based on FilesInGetListingOps
> -
>
> Key: HDFS-12502
> URL: https://issues.apache.org/jira/browse/HDFS-12502
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: metrics
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
> Attachments: HDFS-12502.00.patch, HDFS-12502.01.patch, 
> HDFS-12502.02.patch, HDFS-12502.03.patch, HDFS-12502.04.patch
>
>
> Large listing ops can oftentimes be the main contributor to NameNode 
> slowness. The aggregate cost of listing ops is proportional to the 
> {{FilesInGetListingOps}} rather than the number of listing ops. Therefore 
> it'd be very useful for nntop to support this category.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-12502) nntop should support a category based on FilesInGetListingOps

2017-10-24 Thread Zhe Zhang (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HDFS-12502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhe Zhang updated HDFS-12502:
-
Fix Version/s: (was: 2.7.5)

> nntop should support a category based on FilesInGetListingOps
> -
>
> Key: HDFS-12502
> URL: https://issues.apache.org/jira/browse/HDFS-12502
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: metrics
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
> Fix For: 2.9.0, 2.8.3, 3.0.0, 3.1.0
>
> Attachments: HDFS-12502.00.patch, HDFS-12502.01.patch, 
> HDFS-12502.02.patch, HDFS-12502.03.patch, HDFS-12502.04.patch
>
>
> Large listing ops can oftentimes be the main contributor to NameNode 
> slowness. The aggregate cost of listing ops is proportional to the 
> {{FilesInGetListingOps}} rather than the number of listing ops. Therefore 
> it'd be very useful for nntop to support this category.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-12502) nntop should support a category based on FilesInGetListingOps

2017-10-19 Thread Zhe Zhang (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HDFS-12502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhe Zhang updated HDFS-12502:
-
   Resolution: Fixed
 Hadoop Flags: Reviewed
Fix Version/s: 3.1.0
   3.0.0
   2.7.5
   2.8.3
   2.9.0
   Status: Resolved  (was: Patch Available)

Thanks for the review [~shv]. I just committed the patch to trunk~branch-2.7.

> nntop should support a category based on FilesInGetListingOps
> -
>
> Key: HDFS-12502
> URL: https://issues.apache.org/jira/browse/HDFS-12502
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: metrics
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
> Fix For: 2.9.0, 2.8.3, 2.7.5, 3.0.0, 3.1.0
>
> Attachments: HDFS-12502.00.patch, HDFS-12502.01.patch, 
> HDFS-12502.02.patch, HDFS-12502.03.patch, HDFS-12502.04.patch
>
>
> Large listing ops can oftentimes be the main contributor to NameNode 
> slowness. The aggregate cost of listing ops is proportional to the 
> {{FilesInGetListingOps}} rather than the number of listing ops. Therefore 
> it'd be very useful for nntop to support this category.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-12502) nntop should support a category based on FilesInGetListingOps

2017-10-16 Thread Zhe Zhang (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HDFS-12502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhe Zhang updated HDFS-12502:
-
Attachment: HDFS-12502.04.patch

Thanks [~ywskycn]! Attaching new patch to address the comment.

> nntop should support a category based on FilesInGetListingOps
> -
>
> Key: HDFS-12502
> URL: https://issues.apache.org/jira/browse/HDFS-12502
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: metrics
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
> Attachments: HDFS-12502.00.patch, HDFS-12502.01.patch, 
> HDFS-12502.02.patch, HDFS-12502.03.patch, HDFS-12502.04.patch
>
>
> Large listing ops can oftentimes be the main contributor to NameNode 
> slowness. The aggregate cost of listing ops is proportional to the 
> {{FilesInGetListingOps}} rather than the number of listing ops. Therefore 
> it'd be very useful for nntop to support this category.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-12502) nntop should support a category based on FilesInGetListingOps

2017-10-05 Thread Zhe Zhang (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HDFS-12502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhe Zhang updated HDFS-12502:
-
Attachment: (was: HDFS-12502.03.patch)

> nntop should support a category based on FilesInGetListingOps
> -
>
> Key: HDFS-12502
> URL: https://issues.apache.org/jira/browse/HDFS-12502
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: metrics
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
> Attachments: HDFS-12502.00.patch, HDFS-12502.01.patch, 
> HDFS-12502.02.patch, HDFS-12502.03.patch
>
>
> Large listing ops can oftentimes be the main contributor to NameNode 
> slowness. The aggregate cost of listing ops is proportional to the 
> {{FilesInGetListingOps}} rather than the number of listing ops. Therefore 
> it'd be very useful for nntop to support this category.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-12502) nntop should support a category based on FilesInGetListingOps

2017-10-05 Thread Zhe Zhang (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HDFS-12502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhe Zhang updated HDFS-12502:
-
Attachment: HDFS-12502.03.patch

> nntop should support a category based on FilesInGetListingOps
> -
>
> Key: HDFS-12502
> URL: https://issues.apache.org/jira/browse/HDFS-12502
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: metrics
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
> Attachments: HDFS-12502.00.patch, HDFS-12502.01.patch, 
> HDFS-12502.02.patch, HDFS-12502.03.patch
>
>
> Large listing ops can oftentimes be the main contributor to NameNode 
> slowness. The aggregate cost of listing ops is proportional to the 
> {{FilesInGetListingOps}} rather than the number of listing ops. Therefore 
> it'd be very useful for nntop to support this category.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-12502) nntop should support a category based on FilesInGetListingOps

2017-10-05 Thread Zhe Zhang (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HDFS-12502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhe Zhang updated HDFS-12502:
-
Attachment: HDFS-12502.03.patch

Thanks [~xkrogen], good points. Updating the patch to address.

> nntop should support a category based on FilesInGetListingOps
> -
>
> Key: HDFS-12502
> URL: https://issues.apache.org/jira/browse/HDFS-12502
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: metrics
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
> Attachments: HDFS-12502.00.patch, HDFS-12502.01.patch, 
> HDFS-12502.02.patch, HDFS-12502.03.patch
>
>
> Large listing ops can oftentimes be the main contributor to NameNode 
> slowness. The aggregate cost of listing ops is proportional to the 
> {{FilesInGetListingOps}} rather than the number of listing ops. Therefore 
> it'd be very useful for nntop to support this category.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-12502) nntop should support a category based on FilesInGetListingOps

2017-10-05 Thread Zhe Zhang (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HDFS-12502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhe Zhang updated HDFS-12502:
-
Attachment: HDFS-12502.02.patch

Thanks [~shv]! Updating the patch to address comments. Actually the reported UT 
failures were due to the NPE. Also added JavaDoc and comments.

> nntop should support a category based on FilesInGetListingOps
> -
>
> Key: HDFS-12502
> URL: https://issues.apache.org/jira/browse/HDFS-12502
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: metrics
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
> Attachments: HDFS-12502.00.patch, HDFS-12502.01.patch, 
> HDFS-12502.02.patch
>
>
> Large listing ops can oftentimes be the main contributor to NameNode 
> slowness. The aggregate cost of listing ops is proportional to the 
> {{FilesInGetListingOps}} rather than the number of listing ops. Therefore 
> it'd be very useful for nntop to support this category.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-12502) nntop should support a category based on FilesInGetListingOps

2017-09-21 Thread Zhe Zhang (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HDFS-12502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhe Zhang updated HDFS-12502:
-
Attachment: HDFS-12502.01.patch

v00 patch actually doesn't work, because {{logAuditEvent}} for {{getListing}} 
doesn't pass the FileStatus of the directory as a parameter.

I see two options here:
# Extend the format of audit logs to add an optional field representing "number 
of files in recursive operations". This applies to listing and deletes. Adding 
the info to audit logs would help more accurately identify abusive users 
issuing large listings. E.g. we will be able to tell which large dirs were 
listed. But considering the field is only applicable to a small number of 
operations, it's a little awkward to add to the schema.
# Add a method in {{TopMetrics}} to report {{filesInGetListing}} and directly 
call it from {{getListing}}. The attached v01 patch uses this approach. I 
verified with {{MiniDFSClusterManager}} that it does report the correct 
metrics. If we decide to use this approach I'll add a unit test.

Pinging a few folks based on previous discussions on RPC scalability: 
[~andrew.wang] [~drankye] [~ywskycn] [~shv]. Any thoughts?


> nntop should support a category based on FilesInGetListingOps
> -
>
> Key: HDFS-12502
> URL: https://issues.apache.org/jira/browse/HDFS-12502
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: metrics
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
> Attachments: HDFS-12502.00.patch, HDFS-12502.01.patch
>
>
> Large listing ops can oftentimes be the main contributor to NameNode 
> slowness. The aggregate cost of listing ops is proportional to the 
> {{FilesInGetListingOps}} rather than the number of listing ops. Therefore 
> it'd be very useful for nntop to support this category.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-12502) nntop should support a category based on FilesInGetListingOps

2017-09-20 Thread Zhe Zhang (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HDFS-12502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhe Zhang updated HDFS-12502:
-
Status: Patch Available  (was: Open)

> nntop should support a category based on FilesInGetListingOps
> -
>
> Key: HDFS-12502
> URL: https://issues.apache.org/jira/browse/HDFS-12502
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: metrics
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
> Attachments: HDFS-12502.00.patch
>
>
> Large listing ops can oftentimes be the main contributor to NameNode 
> slowness. The aggregate cost of listing ops is proportional to the 
> {{FilesInGetListingOps}} rather than the number of listing ops. Therefore 
> it'd be very useful for nntop to support this category.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-12502) nntop should support a category based on FilesInGetListingOps

2017-09-20 Thread Zhe Zhang (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HDFS-12502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhe Zhang updated HDFS-12502:
-
Attachment: HDFS-12502.00.patch

Initial patch attached.

> nntop should support a category based on FilesInGetListingOps
> -
>
> Key: HDFS-12502
> URL: https://issues.apache.org/jira/browse/HDFS-12502
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: metrics
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
> Attachments: HDFS-12502.00.patch
>
>
> Large listing ops can oftentimes be the main contributor to NameNode 
> slowness. The aggregate cost of listing ops is proportional to the 
> {{FilesInGetListingOps}} rather than the number of listing ops. Therefore 
> it'd be very useful for nntop to support this category.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-12502) nntop should support a category based on FilesInGetListingOps

2017-09-19 Thread Zhe Zhang (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HDFS-12502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhe Zhang updated HDFS-12502:
-
Description: Large listing ops can oftentimes be the main contributor to 
NameNode slowness. The aggregate cost of listing ops is proportional to the 

> nntop should support a category based on FilesInGetListingOps
> -
>
> Key: HDFS-12502
> URL: https://issues.apache.org/jira/browse/HDFS-12502
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: metrics
>Reporter: Zhe Zhang
>
> Large listing ops can oftentimes be the main contributor to NameNode 
> slowness. The aggregate cost of listing ops is proportional to the 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-12502) nntop should support a category based on FilesInGetListingOps

2017-09-19 Thread Zhe Zhang (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HDFS-12502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhe Zhang updated HDFS-12502:
-
Description: Large listing ops can oftentimes be the main contributor to 
NameNode slowness. The aggregate cost of listing ops is proportional to the 
{{FilesInGetListingOps}} rather than the number of listing ops. Therefore it'd 
be very useful for nntop to support this category.  (was: Large listing ops can 
oftentimes be the main contributor to NameNode slowness. The aggregate cost of 
listing ops is proportional to the )

> nntop should support a category based on FilesInGetListingOps
> -
>
> Key: HDFS-12502
> URL: https://issues.apache.org/jira/browse/HDFS-12502
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: metrics
>Reporter: Zhe Zhang
>
> Large listing ops can oftentimes be the main contributor to NameNode 
> slowness. The aggregate cost of listing ops is proportional to the 
> {{FilesInGetListingOps}} rather than the number of listing ops. Therefore 
> it'd be very useful for nntop to support this category.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-12502) nntop should support a category based on FilesInGetListingOps

2017-09-19 Thread Zhe Zhang (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HDFS-12502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhe Zhang updated HDFS-12502:
-
Summary: nntop should support a category based on FilesInGetListingOps  
(was: nntop should support category based on FilesInGetListingOps)

> nntop should support a category based on FilesInGetListingOps
> -
>
> Key: HDFS-12502
> URL: https://issues.apache.org/jira/browse/HDFS-12502
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: metrics
>Reporter: Zhe Zhang
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org