[jira] [Commented] (HDFS-12703) Exceptions are fatal to decommissioning monitor
[
https://issues.apache.org/jira/browse/HDFS-12703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16882293#comment-16882293
]
Xue Liu commented on HDFS-12703:
+1 on the latest patch
Looks great to me! Thanks [~hexiaoqiao] for working on the patch and guys for
reviewing!
> Exceptions are fatal to decommissioning monitor
> ---
>
> Key: HDFS-12703
> URL: https://issues.apache.org/jira/browse/HDFS-12703
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.7.0
>Reporter: Daryn Sharp
>Assignee: He Xiaoqiao
>Priority: Critical
> Attachments: HDFS-12703.001.patch, HDFS-12703.002.patch,
> HDFS-12703.003.patch, HDFS-12703.004.patch, HDFS-12703.005.patch,
> HDFS-12703.006.patch, HDFS-12703.007.patch, HDFS-12703.008.patch,
> HDFS-12703.009.patch, HDFS-12703.010.patch, HDFS-12703.011.patch,
> HDFS-12703.012.patch, HDFS-12703.013.patch
>
>
> The {{DecommissionManager.Monitor}} runs as an executor scheduled task. If
> an exception occurs, all decommissioning ceases until the NN is restarted.
> Per javadoc for {{executor#scheduleAtFixedRate}}: *If any execution of the
> task encounters an exception, subsequent executions are suppressed*. The
> monitor thread is alive but blocked waiting for an executor task that will
> never come. The code currently disposes of the future so the actual
> exception that aborted the task is gone.
> Failover is insufficient since the task is also likely dead on the standby.
> Replication queue init after the transition to active will fix the under
> replication of blocks on currently decommissioning nodes but future nodes
> never decommission. The standby must be bounced prior to failover – and
> hopefully the error condition does not reoccur.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12703) Exceptions are fatal to decommissioning monitor
[
https://issues.apache.org/jira/browse/HDFS-12703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16874376#comment-16874376
]
Xue Liu commented on HDFS-12703:
[~elgoiri] Makes sense to me, we should add this.
[~hexiaoqiao] Thanks for the work on this JIRA! As I mentioned in the previous
comment, we saw this erroneous DN state causing exception:
{code:java}
Preconditions.checkState(false, "A node is in an invalid state!")
{code}
It would be great if we can add a test case that tests that for some DN not in
isDecommissionInProgress() or isEnteringMaintenance().
> Exceptions are fatal to decommissioning monitor
> ---
>
> Key: HDFS-12703
> URL: https://issues.apache.org/jira/browse/HDFS-12703
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.7.0
>Reporter: Daryn Sharp
>Assignee: Xue Liu
>Priority: Critical
> Attachments: HDFS-12703.001.patch, HDFS-12703.002.patch,
> HDFS-12703.003.patch, HDFS-12703.004.patch
>
>
> The {{DecommissionManager.Monitor}} runs as an executor scheduled task. If
> an exception occurs, all decommissioning ceases until the NN is restarted.
> Per javadoc for {{executor#scheduleAtFixedRate}}: *If any execution of the
> task encounters an exception, subsequent executions are suppressed*. The
> monitor thread is alive but blocked waiting for an executor task that will
> never come. The code currently disposes of the future so the actual
> exception that aborted the task is gone.
> Failover is insufficient since the task is also likely dead on the standby.
> Replication queue init after the transition to active will fix the under
> replication of blocks on currently decommissioning nodes but future nodes
> never decommission. The standby must be bounced prior to failover – and
> hopefully the error condition does not reoccur.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-8631) WebHDFS : Support get/setQuota
[ https://issues.apache.org/jira/browse/HDFS-8631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16871534#comment-16871534 ] Xue Liu commented on HDFS-8631: --- Sure [~csun] feel free to work on this! > WebHDFS : Support get/setQuota > -- > > Key: HDFS-8631 > URL: https://issues.apache.org/jira/browse/HDFS-8631 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 2.7.2 >Reporter: nijel >Assignee: Xue Liu >Priority: Major > Attachments: HDFS-8631-001.patch, HDFS-8631-002.patch, > HDFS-8631-003.patch, HDFS-8631-004.patch, HDFS-8631-005.patch, > HDFS-8631-006.patch > > > User is able do quota management from filesystem object. Same operation can > be allowed trough REST API. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12703) Exceptions are fatal to decommissioning monitor
[
https://issues.apache.org/jira/browse/HDFS-12703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16852532#comment-16852532
]
Xue Liu commented on HDFS-12703:
Thanks [~goiri] for reviewing! I think TestDecommission already covers the
scenarios pretty well.
Test fails all due to TestWebHdfsTimeouts, should be unrelated?
> Exceptions are fatal to decommissioning monitor
> ---
>
> Key: HDFS-12703
> URL: https://issues.apache.org/jira/browse/HDFS-12703
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.7.0
>Reporter: Daryn Sharp
>Assignee: Xue Liu
>Priority: Critical
> Attachments: HDFS-12703.001.patch
>
>
> The {{DecommissionManager.Monitor}} runs as an executor scheduled task. If
> an exception occurs, all decommissioning ceases until the NN is restarted.
> Per javadoc for {{executor#scheduleAtFixedRate}}: *If any execution of the
> task encounters an exception, subsequent executions are suppressed*. The
> monitor thread is alive but blocked waiting for an executor task that will
> never come. The code currently disposes of the future so the actual
> exception that aborted the task is gone.
> Failover is insufficient since the task is also likely dead on the standby.
> Replication queue init after the transition to active will fix the under
> replication of blocks on currently decommissioning nodes but future nodes
> never decommission. The standby must be bounced prior to failover – and
> hopefully the error condition does not reoccur.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12703) Exceptions are fatal to decommissioning monitor
[
https://issues.apache.org/jira/browse/HDFS-12703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16850261#comment-16850261
]
Xue Liu commented on HDFS-12703:
Hi the exception I found come from check():
{code:java}
Preconditions.checkState(false, "A node is in an invalid state!")
{code}
This will cause exception and the way executor#scheduleAtFixedRate works, the
thread's further execution will be suppressed.
So I just add exception handle to prevent exception killing the decommission,
and add a few logs to help with debugging.
It is not clear why sometimes DNs will be in a erroneous state, but this change
will unblock the thread and allow us dig further.
> Exceptions are fatal to decommissioning monitor
> ---
>
> Key: HDFS-12703
> URL: https://issues.apache.org/jira/browse/HDFS-12703
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.7.0
>Reporter: Daryn Sharp
>Assignee: Xue Liu
>Priority: Critical
> Attachments: HDFS-12703.001.patch
>
>
> The {{DecommissionManager.Monitor}} runs as an executor scheduled task. If
> an exception occurs, all decommissioning ceases until the NN is restarted.
> Per javadoc for {{executor#scheduleAtFixedRate}}: *If any execution of the
> task encounters an exception, subsequent executions are suppressed*. The
> monitor thread is alive but blocked waiting for an executor task that will
> never come. The code currently disposes of the future so the actual
> exception that aborted the task is gone.
> Failover is insufficient since the task is also likely dead on the standby.
> Replication queue init after the transition to active will fix the under
> replication of blocks on currently decommissioning nodes but future nodes
> never decommission. The standby must be bounced prior to failover – and
> hopefully the error condition does not reoccur.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12703) Exceptions are fatal to decommissioning monitor
[
https://issues.apache.org/jira/browse/HDFS-12703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xue Liu updated HDFS-12703:
---
Attachment: HDFS-12703.001.patch
> Exceptions are fatal to decommissioning monitor
> ---
>
> Key: HDFS-12703
> URL: https://issues.apache.org/jira/browse/HDFS-12703
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.7.0
>Reporter: Daryn Sharp
>Assignee: Xue Liu
>Priority: Critical
> Attachments: HDFS-12703.001.patch
>
>
> The {{DecommissionManager.Monitor}} runs as an executor scheduled task. If
> an exception occurs, all decommissioning ceases until the NN is restarted.
> Per javadoc for {{executor#scheduleAtFixedRate}}: *If any execution of the
> task encounters an exception, subsequent executions are suppressed*. The
> monitor thread is alive but blocked waiting for an executor task that will
> never come. The code currently disposes of the future so the actual
> exception that aborted the task is gone.
> Failover is insufficient since the task is also likely dead on the standby.
> Replication queue init after the transition to active will fix the under
> replication of blocks on currently decommissioning nodes but future nodes
> never decommission. The standby must be bounced prior to failover – and
> hopefully the error condition does not reoccur.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-12703) Exceptions are fatal to decommissioning monitor
[
https://issues.apache.org/jira/browse/HDFS-12703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xue Liu reassigned HDFS-12703:
--
Assignee: Xue Liu
> Exceptions are fatal to decommissioning monitor
> ---
>
> Key: HDFS-12703
> URL: https://issues.apache.org/jira/browse/HDFS-12703
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.7.0
>Reporter: Daryn Sharp
>Assignee: Xue Liu
>Priority: Critical
>
> The {{DecommissionManager.Monitor}} runs as an executor scheduled task. If
> an exception occurs, all decommissioning ceases until the NN is restarted.
> Per javadoc for {{executor#scheduleAtFixedRate}}: *If any execution of the
> task encounters an exception, subsequent executions are suppressed*. The
> monitor thread is alive but blocked waiting for an executor task that will
> never come. The code currently disposes of the future so the actual
> exception that aborted the task is gone.
> Failover is insufficient since the task is also likely dead on the standby.
> Replication queue init after the transition to active will fix the under
> replication of blocks on currently decommissioning nodes but future nodes
> never decommission. The standby must be bounced prior to failover – and
> hopefully the error condition does not reoccur.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12703) Exceptions are fatal to decommissioning monitor
[
https://issues.apache.org/jira/browse/HDFS-12703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16845252#comment-16845252
]
Xue Liu commented on HDFS-12703:
Hi folks,
We observed this issue on our prod cluster recently, basically there are some
exceptions on DatanodeAdminMonitor thread that are unhandled, and this caused
decommission to stop. The thread execution is suppressed as the Jira described.
We are adding some error handling to catch the specific exception. Will update
what exception we have once running in prod.
I will provide a patch with exception handling, and if possible, fix the root
cause of the exception.
> Exceptions are fatal to decommissioning monitor
> ---
>
> Key: HDFS-12703
> URL: https://issues.apache.org/jira/browse/HDFS-12703
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.7.0
>Reporter: Daryn Sharp
>Priority: Critical
>
> The {{DecommissionManager.Monitor}} runs as an executor scheduled task. If
> an exception occurs, all decommissioning ceases until the NN is restarted.
> Per javadoc for {{executor#scheduleAtFixedRate}}: *If any execution of the
> task encounters an exception, subsequent executions are suppressed*. The
> monitor thread is alive but blocked waiting for an executor task that will
> never come. The code currently disposes of the future so the actual
> exception that aborted the task is gone.
> Failover is insufficient since the task is also likely dead on the standby.
> Replication queue init after the transition to active will fix the under
> replication of blocks on currently decommissioning nodes but future nodes
> never decommission. The standby must be bounced prior to failover – and
> hopefully the error condition does not reoccur.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-8631) WebHDFS : Support get/setQuota
[ https://issues.apache.org/jira/browse/HDFS-8631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16831784#comment-16831784 ] Xue Liu commented on HDFS-8631: --- Hi [~surendrasingh] Are you still actively working on this issue? If no, I would be glad to take this JIRA, as one of our production tool would benefit a lot from it! > WebHDFS : Support get/setQuota > -- > > Key: HDFS-8631 > URL: https://issues.apache.org/jira/browse/HDFS-8631 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 2.7.2 >Reporter: nijel >Assignee: Surendra Singh Lilhore >Priority: Major > Attachments: HDFS-8631-001.patch, HDFS-8631-002.patch, > HDFS-8631-003.patch, HDFS-8631-004.patch, HDFS-8631-005.patch, > HDFS-8631-006.patch > > > User is able do quota management from filesystem object. Same operation can > be allowed trough REST API. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-14406) Add per user RPC Processing time
[ https://issues.apache.org/jira/browse/HDFS-14406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16822267#comment-16822267 ] Xue Liu edited comment on HDFS-14406 at 4/20/19 12:05 AM: -- [~xkrogen] Makes sense to me, I will go back to per user metrics for this jira. Though I don't believe volume will be a blocking factor. We have metrics like NameNodeInfo via jmx give us all info on LiveNode and DeadNode, which is in the scale of thousands. And in a large system we would have parsing wherever we collect these metrics, so I am not worried about readability either. [~csun] Since we are only talking about per user metrics here, which approach you think is better? we can pass a flag to RpcDetailedMetrics, and add different prefix accordingly, or we just create another metrics class? was (Author: xuel1): [~xkrogen] Makes sense to me, I will go back to per user metrics for this jira. Though I don't believe volume will be a blocking factor. We have metrics like NameNodeInfo via jmx give us all info on LiveNode and DeadNode, which is in the scale of thousands. And in a large system we would have parsing wherever we collect them, so I am not worried about readability either. [~csun] Since we are only talking about per user metrics here, which approach you think is better? we can pass a flag to RpcDetailedMetrics, and add different prefix accordingly, or we just create another metrics class? > Add per user RPC Processing time > > > Key: HDFS-14406 > URL: https://issues.apache.org/jira/browse/HDFS-14406 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.2.0 >Reporter: Xue Liu >Assignee: Xue Liu >Priority: Minor > Fix For: 3.2.0 > > Attachments: HDFS-14406.001.patch, HDFS-14406.002.patch, > HDFS-14406.003.patch, HDFS-14406.004.patch, HDFS-14406.005.patch, > HDFS-14406.006.patch > > > For a shared cluster we would want to separate users' resources, as well as > having our metrics reflecting on the usage, latency, etc, for each user. > This JIRA aims to add per user RPC processing time metrics and expose it via > JMX. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14406) Add per user RPC Processing time
[ https://issues.apache.org/jira/browse/HDFS-14406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16822267#comment-16822267 ] Xue Liu commented on HDFS-14406: [~xkrogen] Makes sense to me, I will go back to per user metrics for this jira. Though I don't believe volume will be a blocking factor. We have metrics like NameNodeInfo via jmx give us all info on LiveNode and DeadNode, which is in the scale of thousands. And in a large system we would have parsing wherever we collect them, so I am not worried about readability either. [~csun] Since we are only talking about per user metrics here, which approach you think is better? we can pass a flag to RpcDetailedMetrics, and add different prefix accordingly, or we just create another metrics class? > Add per user RPC Processing time > > > Key: HDFS-14406 > URL: https://issues.apache.org/jira/browse/HDFS-14406 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.2.0 >Reporter: Xue Liu >Assignee: Xue Liu >Priority: Minor > Fix For: 3.2.0 > > Attachments: HDFS-14406.001.patch, HDFS-14406.002.patch, > HDFS-14406.003.patch, HDFS-14406.004.patch, HDFS-14406.005.patch, > HDFS-14406.006.patch > > > For a shared cluster we would want to separate users' resources, as well as > having our metrics reflecting on the usage, latency, etc, for each user. > This JIRA aims to add per user RPC processing time metrics and expose it via > JMX. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-14406) Add per user RPC Processing time
[ https://issues.apache.org/jira/browse/HDFS-14406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16822235#comment-16822235 ] Xue Liu edited comment on HDFS-14406 at 4/19/19 10:32 PM: -- Hi guys thanks for the comments! Reflected on [~xkrogen]'s comment of extending, as in patch 5, I made these changes: # Make RpcUserMetrics a subclass of RpcDetailedMetrics. Only keep 2 metrics rpcMetrics and rpcDetailedMetrics, and determine whether we create RpcUserMetrics or RpcDetailedMetrics based on enable flag at runtime. # Introduce per user per method metrics, as I am receiving suggestion that this would be even more useful because it gives us an idea on how the RPC cost of each user looks. In patch 6: 1. I found out that that this dynamic type doesn't necessarily benefit our performance, but make the code look more complicated than it should be. Now I am just creating rpcUserMetrics regardless, and only update it when there is enable flag. 2. Kept this per user per user per method metrics. What do you guys think of this idea? [~csun] [~linyiqun] [~xkrogen] [~elgoiri] Some other fixes: 1. [~csun] I changed String.format to String plus for better performance. 2. Docs in core-default, Metrics.md are still as user metrics. I will update them once we have impl worked out. 3. A few checkstyles, I will fix them the last. was (Author: xuel1): Hi guys thanks for the comments! Reflected on [~xkrogen]'s comment of extending, as in patch 5, I made these changes: # Make RpcUserMetrics a subclass of RpcDetailedMetrics. Only keep 2 metrics rpcMetrics and rpcDetailedMetrics, and determine whether we create RpcUserMetrics or RpcDetailedMetrics based on enable flag at runtime. # Introduce per user per method metrics, as I am receiving suggestion that this would be even more useful because it gives us an idea on how the RPC cost of each user looks. In patch 6: 1. I found out that that this dynamic type doesn't necessarily benefit our performance, but make the code look more complicated than it should be. Now I am just creating rpcUserMetrics regardless, and only update it when there is enable flag. 2. Kept this per user per user per method metrics. What do you guys think of this idea? [~thefacts] Some other fixes: 1. [~csun] I changed String.format to String plus for better performance. 2. Docs in core-default, Metrics.md are still as user metrics. I will update them once we have impl worked out. 3. A few checkstyles, I will fix them the last. > Add per user RPC Processing time > > > Key: HDFS-14406 > URL: https://issues.apache.org/jira/browse/HDFS-14406 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.2.0 >Reporter: Xue Liu >Assignee: Xue Liu >Priority: Minor > Fix For: 3.2.0 > > Attachments: HDFS-14406.001.patch, HDFS-14406.002.patch, > HDFS-14406.003.patch, HDFS-14406.004.patch, HDFS-14406.005.patch, > HDFS-14406.006.patch > > > For a shared cluster we would want to separate users' resources, as well as > having our metrics reflecting on the usage, latency, etc, for each user. > This JIRA aims to add per user RPC processing time metrics and expose it via > JMX. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-14406) Add per user RPC Processing time
[ https://issues.apache.org/jira/browse/HDFS-14406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16822235#comment-16822235 ] Xue Liu edited comment on HDFS-14406 at 4/19/19 10:30 PM: -- Hi guys thanks for the comments! Reflected on [~xkrogen]'s comment of extending, as in patch 5, I made these changes: # Make RpcUserMetrics a subclass of RpcDetailedMetrics. Only keep 2 metrics rpcMetrics and rpcDetailedMetrics, and determine whether we create RpcUserMetrics or RpcDetailedMetrics based on enable flag at runtime. # Introduce per user per method metrics, as I am receiving suggestion that this would be even more useful because it gives us an idea on how the RPC cost of each user looks. In patch 6: 1. I found out that that this dynamic type doesn't necessarily benefit our performance, but make the code look more complicated than it should be. Now I am just creating rpcUserMetrics regardless, and only update it when there is enable flag. 2. Kept this per user per user per method metrics. What do you guys think of this idea? [~thefacts] Some other fixes: 1. [~csun] I changed String.format to String plus for better performance. 2. Docs in core-default, Metrics.md are still as user metrics. I will update them once we have impl worked out. 3. A few checkstyles, I will fix them the last. was (Author: xuel1): Hi guys thanks for the comments! Reflected on [~xkrogen]'s comment of extending, as in patch 5, I made these changes: # Make RpcUserMetrics a subclass of RpcDetailedMetrics. Only keep 2 metrics rpcMetrics and rpcDetailedMetrics, and determine whether we create RpcUserMetrics or RpcDetailedMetrics based on enable flag at runtime. # Introduce per user per method metrics, as I am receiving suggestion that this would be even more useful because it gives us an idea on how the RPC cost of each user looks. In patch 6: 1. I found out that that this dynamic type doesn't necessarily benefit our performance, but make the code look more complicated than it should be. Now I am just creating rpcUserMetrics regardless, and only update it when there is enable flag. 2. Kept this per user per user per method metrics. What do you guys think of this idea? [~thefacts] Some other fixes: 1. [~csun] I changed String.format to String plus for better performance. 2. Docs in core-default, Metrics.md are still as user metrics. I will update them once we have impl worked out. 3. A few checkstyles, I will fix them the last. > Add per user RPC Processing time > > > Key: HDFS-14406 > URL: https://issues.apache.org/jira/browse/HDFS-14406 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.2.0 >Reporter: Xue Liu >Assignee: Xue Liu >Priority: Minor > Fix For: 3.2.0 > > Attachments: HDFS-14406.001.patch, HDFS-14406.002.patch, > HDFS-14406.003.patch, HDFS-14406.004.patch, HDFS-14406.005.patch, > HDFS-14406.006.patch > > > For a shared cluster we would want to separate users' resources, as well as > having our metrics reflecting on the usage, latency, etc, for each user. > This JIRA aims to add per user RPC processing time metrics and expose it via > JMX. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14406) Add per user RPC Processing time
[ https://issues.apache.org/jira/browse/HDFS-14406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16822235#comment-16822235 ] Xue Liu commented on HDFS-14406: Hi guys thanks for the comments! Reflected on [~xkrogen]'s comment of extending, as in patch 5, I made these changes: # Make RpcUserMetrics a subclass of RpcDetailedMetrics. Only keep 2 metrics rpcMetrics and rpcDetailedMetrics, and determine whether we create RpcUserMetrics or RpcDetailedMetrics based on enable flag at runtime. # Introduce per user per method metrics, as I am receiving suggestion that this would be even more useful because it gives us an idea on how the RPC cost of each user looks. In patch 6: 1. I found out that that this dynamic type doesn't necessarily benefit our performance, but make the code look more complicated than it should be. Now I am just creating rpcUserMetrics regardless, and only update it when there is enable flag. 2. Kept this per user per user per method metrics. What do you guys think of this idea? [~thefacts] Some other fixes: 1. [~csun] I changed String.format to String plus for better performance. 2. Docs in core-default, Metrics.md are still as user metrics. I will update them once we have impl worked out. 3. A few checkstyles, I will fix them the last. > Add per user RPC Processing time > > > Key: HDFS-14406 > URL: https://issues.apache.org/jira/browse/HDFS-14406 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.2.0 >Reporter: Xue Liu >Assignee: Xue Liu >Priority: Minor > Fix For: 3.2.0 > > Attachments: HDFS-14406.001.patch, HDFS-14406.002.patch, > HDFS-14406.003.patch, HDFS-14406.004.patch, HDFS-14406.005.patch, > HDFS-14406.006.patch > > > For a shared cluster we would want to separate users' resources, as well as > having our metrics reflecting on the usage, latency, etc, for each user. > This JIRA aims to add per user RPC processing time metrics and expose it via > JMX. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14406) Add per user RPC Processing time
[ https://issues.apache.org/jira/browse/HDFS-14406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xue Liu updated HDFS-14406: --- Attachment: HDFS-14406.006.patch > Add per user RPC Processing time > > > Key: HDFS-14406 > URL: https://issues.apache.org/jira/browse/HDFS-14406 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.2.0 >Reporter: Xue Liu >Assignee: Xue Liu >Priority: Minor > Fix For: 3.2.0 > > Attachments: HDFS-14406.001.patch, HDFS-14406.002.patch, > HDFS-14406.003.patch, HDFS-14406.004.patch, HDFS-14406.005.patch, > HDFS-14406.006.patch > > > For a shared cluster we would want to separate users' resources, as well as > having our metrics reflecting on the usage, latency, etc, for each user. > This JIRA aims to add per user RPC processing time metrics and expose it via > JMX. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14406) Add per user RPC Processing time
[ https://issues.apache.org/jira/browse/HDFS-14406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xue Liu updated HDFS-14406: --- Attachment: HDFS-14406.005.patch > Add per user RPC Processing time > > > Key: HDFS-14406 > URL: https://issues.apache.org/jira/browse/HDFS-14406 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.2.0 >Reporter: Xue Liu >Assignee: Xue Liu >Priority: Minor > Fix For: 3.2.0 > > Attachments: HDFS-14406.001.patch, HDFS-14406.002.patch, > HDFS-14406.003.patch, HDFS-14406.004.patch, HDFS-14406.005.patch > > > For a shared cluster we would want to separate users' resources, as well as > having our metrics reflecting on the usage, latency, etc, for each user. > This JIRA aims to add per user RPC processing time metrics and expose it via > JMX. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14406) Add per user RPC Processing time
[ https://issues.apache.org/jira/browse/HDFS-14406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16818463#comment-16818463 ] Xue Liu commented on HDFS-14406: [~linyiqun] Thanks for the comment! Checkstyles and others has been fixed. Regarding unit test, I think all metrics registered using MetricsRegistry are exposed via JMX, do I still need to add a single JMX test for it? > Add per user RPC Processing time > > > Key: HDFS-14406 > URL: https://issues.apache.org/jira/browse/HDFS-14406 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.2.0 >Reporter: Xue Liu >Assignee: Xue Liu >Priority: Minor > Fix For: 3.2.0 > > Attachments: HDFS-14406.001.patch, HDFS-14406.002.patch, > HDFS-14406.003.patch, HDFS-14406.004.patch > > > For a shared cluster we would want to separate users' resources, as well as > having our metrics reflecting on the usage, latency, etc, for each user. > This JIRA aims to add per user RPC processing time metrics and expose it via > JMX. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14406) Add per user RPC Processing time
[ https://issues.apache.org/jira/browse/HDFS-14406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xue Liu updated HDFS-14406: --- Attachment: HDFS-14406.004.patch > Add per user RPC Processing time > > > Key: HDFS-14406 > URL: https://issues.apache.org/jira/browse/HDFS-14406 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.2.0 >Reporter: Xue Liu >Assignee: Xue Liu >Priority: Minor > Fix For: 3.2.0 > > Attachments: HDFS-14406.001.patch, HDFS-14406.002.patch, > HDFS-14406.003.patch, HDFS-14406.004.patch > > > For a shared cluster we would want to separate users' resources, as well as > having our metrics reflecting on the usage, latency, etc, for each user. > This JIRA aims to add per user RPC processing time metrics and expose it via > JMX. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14406) Add per user RPC Processing time
[ https://issues.apache.org/jira/browse/HDFS-14406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xue Liu updated HDFS-14406: --- Attachment: (was: HDFS-14406.004.patch) > Add per user RPC Processing time > > > Key: HDFS-14406 > URL: https://issues.apache.org/jira/browse/HDFS-14406 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.2.0 >Reporter: Xue Liu >Assignee: Xue Liu >Priority: Minor > Fix For: 3.2.0 > > Attachments: HDFS-14406.001.patch, HDFS-14406.002.patch, > HDFS-14406.003.patch, HDFS-14406.004.patch > > > For a shared cluster we would want to separate users' resources, as well as > having our metrics reflecting on the usage, latency, etc, for each user. > This JIRA aims to add per user RPC processing time metrics and expose it via > JMX. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14406) Add per user RPC Processing time
[ https://issues.apache.org/jira/browse/HDFS-14406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xue Liu updated HDFS-14406: --- Attachment: HDFS-14406.004.patch > Add per user RPC Processing time > > > Key: HDFS-14406 > URL: https://issues.apache.org/jira/browse/HDFS-14406 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.2.0 >Reporter: Xue Liu >Assignee: Xue Liu >Priority: Minor > Fix For: 3.2.0 > > Attachments: HDFS-14406.001.patch, HDFS-14406.002.patch, > HDFS-14406.003.patch, HDFS-14406.004.patch > > > For a shared cluster we would want to separate users' resources, as well as > having our metrics reflecting on the usage, latency, etc, for each user. > This JIRA aims to add per user RPC processing time metrics and expose it via > JMX. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14406) Add per user RPC Processing time
[
https://issues.apache.org/jira/browse/HDFS-14406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16815829#comment-16815829
]
Xue Liu commented on HDFS-14406:
Fixed some checkstyles. Some is left unfixed, such as:
{quote}./hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/metrics/RpcUserMetrics.java:41:
final MetricsRegistry registry;:25: Variable 'registry' must be private and
have accessor methods. [VisibilityModifier]
{quote}
to maintain consistency with other rpc metrics.
> Add per user RPC Processing time
>
>
> Key: HDFS-14406
> URL: https://issues.apache.org/jira/browse/HDFS-14406
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 3.2.0
>Reporter: Xue Liu
>Assignee: Xue Liu
>Priority: Minor
> Fix For: 3.2.0
>
> Attachments: HDFS-14406.001.patch, HDFS-14406.002.patch,
> HDFS-14406.003.patch
>
>
> For a shared cluster we would want to separate users' resources, as well as
> having our metrics reflecting on the usage, latency, etc, for each user.
> This JIRA aims to add per user RPC processing time metrics and expose it via
> JMX.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14406) Add per user RPC Processing time
[ https://issues.apache.org/jira/browse/HDFS-14406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xue Liu updated HDFS-14406: --- Attachment: HDFS-14406.003.patch > Add per user RPC Processing time > > > Key: HDFS-14406 > URL: https://issues.apache.org/jira/browse/HDFS-14406 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.2.0 >Reporter: Xue Liu >Assignee: Xue Liu >Priority: Minor > Fix For: 3.2.0 > > Attachments: HDFS-14406.001.patch, HDFS-14406.002.patch, > HDFS-14406.003.patch > > > For a shared cluster we would want to separate users' resources, as well as > having our metrics reflecting on the usage, latency, etc, for each user. > This JIRA aims to add per user RPC processing time metrics and expose it via > JMX. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14406) Add per user RPC Processing time
[ https://issues.apache.org/jira/browse/HDFS-14406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16814938#comment-16814938 ] Xue Liu commented on HDFS-14406: Thanks for the comments and suggestions! New patch available, makes the user RPC metrics optional. Agree on that we are now facing the bottleneck of lock waiting time, that would have an unfair effect on per user RPC processing time. In this case, the difference of RPC processing time may not be that big. However I do like to think of this metric as general indicator, that we can probably expose to users. Also, it might be good we have some generosity here, so that other ipc like Yarn may also use it. [~elgoiri][~daryn] Thanks for the suggestion, I have fixed checkstyles and made it optional. [~xkrogen], the work on cost-based FCQ seems interesting, which is something we also want in our prod cluster in the future. From your perspective, would it be interesting if we can have a more detailed metrics, say per user per method rpc processing time? > Add per user RPC Processing time > > > Key: HDFS-14406 > URL: https://issues.apache.org/jira/browse/HDFS-14406 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.2.0 >Reporter: Xue Liu >Assignee: Xue Liu >Priority: Minor > Fix For: 3.2.0 > > Attachments: HDFS-14406.001.patch, HDFS-14406.002.patch > > > For a shared cluster we would want to separate users' resources, as well as > having our metrics reflecting on the usage, latency, etc, for each user. > This JIRA aims to add per user RPC processing time metrics and expose it via > JMX. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14406) Add per user RPC Processing time
[ https://issues.apache.org/jira/browse/HDFS-14406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xue Liu updated HDFS-14406: --- Attachment: HDFS-14406.002.patch > Add per user RPC Processing time > > > Key: HDFS-14406 > URL: https://issues.apache.org/jira/browse/HDFS-14406 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.2.0 >Reporter: Xue Liu >Assignee: Xue Liu >Priority: Minor > Fix For: 3.2.0 > > Attachments: HDFS-14406.001.patch, HDFS-14406.002.patch > > > For a shared cluster we would want to separate users' resources, as well as > having our metrics reflecting on the usage, latency, etc, for each user. > This JIRA aims to add per user RPC processing time metrics and expose it via > JMX. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-8409) HDFS client RPC call throws "java.lang.IllegalStateException"
[
https://issues.apache.org/jira/browse/HDFS-8409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16814726#comment-16814726
]
Xue Liu commented on HDFS-8409:
---
We are also seeing a lot of this in our prod cluster. I think that this
exception will be thrown if callId is not reset and then retry happens. Can
someone take a further look at this? Thanks!
> HDFS client RPC call throws "java.lang.IllegalStateException"
> -
>
> Key: HDFS-8409
> URL: https://issues.apache.org/jira/browse/HDFS-8409
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
>Reporter: Juan Yu
>Assignee: Juan Yu
>Priority: Major
> Attachments: HDFS-8409.001.patch, HDFS-8409.002.patch,
> HDFS-8409.003.patch
>
>
> When the HDFS client RPC calls need to retry, it sometimes throws
> "java.lang.IllegalStateException" and retry is aborted and cause the client
> call will fail.
> {code}
> Caused by: java.lang.IllegalStateException
> at
> com.google.common.base.Preconditions.checkState(Preconditions.java:129)
> at org.apache.hadoop.ipc.Client.setCallIdAndRetryCount(Client.java:116)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:99)
> at com.sun.proxy.$Proxy16.getFileInfo(Unknown Source)
> at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1912)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1089)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1085)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1085)
> at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1400)
> {code}
> Here is the check that throws exception
> {code}
> public static void setCallIdAndRetryCount(int cid, int rc) {
> ...
> Preconditions.checkState(callId.get() == null);
> }
> {code}
> The RetryInvocationHandler tries to call it with not null callId and causes
> exception.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-14406) Add per user RPC Processing time
[ https://issues.apache.org/jira/browse/HDFS-14406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16811341#comment-16811341 ] Xue Liu edited comment on HDFS-14406 at 4/5/19 11:13 PM: - Patch available. Just want to say being a new contributor and all, thank you in advance for reviewing and commenting :) was (Author: xuel1): Patch available. Just want to say being a new contributor and all, thank you advance for reviewing and commenting :) > Add per user RPC Processing time > > > Key: HDFS-14406 > URL: https://issues.apache.org/jira/browse/HDFS-14406 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.2.0 >Reporter: Xue Liu >Assignee: Xue Liu >Priority: Minor > Fix For: 3.2.0 > > Attachments: HDFS-14406.001.patch > > > For a shared cluster we would want to separate users' resources, as well as > having our metrics reflecting on the usage, latency, etc, for each user. > This JIRA aims to add per user RPC processing time metrics and expose it via > JMX. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14406) Add per user RPC Processing time
[ https://issues.apache.org/jira/browse/HDFS-14406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xue Liu updated HDFS-14406: --- Attachment: (was: HDFS-14406.001.patch) > Add per user RPC Processing time > > > Key: HDFS-14406 > URL: https://issues.apache.org/jira/browse/HDFS-14406 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.2.0 >Reporter: Xue Liu >Assignee: Xue Liu >Priority: Minor > Fix For: 3.2.0 > > Attachments: HDFS-14406.001.patch > > > For a shared cluster we would want to separate users' resources, as well as > having our metrics reflecting on the usage, latency, etc, for each user. > This JIRA aims to add per user RPC processing time metrics and expose it via > JMX. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14406) Add per user RPC Processing time
[ https://issues.apache.org/jira/browse/HDFS-14406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xue Liu updated HDFS-14406: --- Attachment: HDFS-14406.001.patch Status: Patch Available (was: Open) > Add per user RPC Processing time > > > Key: HDFS-14406 > URL: https://issues.apache.org/jira/browse/HDFS-14406 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.2.0 >Reporter: Xue Liu >Assignee: Xue Liu >Priority: Minor > Fix For: 3.2.0 > > Attachments: HDFS-14406.001.patch > > > For a shared cluster we would want to separate users' resources, as well as > having our metrics reflecting on the usage, latency, etc, for each user. > This JIRA aims to add per user RPC processing time metrics and expose it via > JMX. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14406) Add per user RPC Processing time
[ https://issues.apache.org/jira/browse/HDFS-14406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16811341#comment-16811341 ] Xue Liu commented on HDFS-14406: Patch available. Just want to say being a new contributor and all, thank you advance for reviewing and commenting :) > Add per user RPC Processing time > > > Key: HDFS-14406 > URL: https://issues.apache.org/jira/browse/HDFS-14406 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.2.0 >Reporter: Xue Liu >Assignee: Xue Liu >Priority: Minor > Fix For: 3.2.0 > > Attachments: HDFS-14406.001.patch > > > For a shared cluster we would want to separate users' resources, as well as > having our metrics reflecting on the usage, latency, etc, for each user. > This JIRA aims to add per user RPC processing time metrics and expose it via > JMX. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14406) Add per user RPC Processing time
[ https://issues.apache.org/jira/browse/HDFS-14406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xue Liu updated HDFS-14406: --- Attachment: HDFS-14406.001.patch > Add per user RPC Processing time > > > Key: HDFS-14406 > URL: https://issues.apache.org/jira/browse/HDFS-14406 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.2.0 >Reporter: Xue Liu >Assignee: Xue Liu >Priority: Minor > Fix For: 3.2.0 > > Attachments: HDFS-14406.001.patch > > > For a shared cluster we would want to separate users' resources, as well as > having our metrics reflecting on the usage, latency, etc, for each user. > This JIRA aims to add per user RPC processing time metrics and expose it via > JMX. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14406) Add per user RPC Processing time
[ https://issues.apache.org/jira/browse/HDFS-14406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xue Liu updated HDFS-14406: --- Description: For a shared cluster we would want to separate users' resources, as well as having our metrics reflecting on the usage, latency, etc, for each user. This JIRA aims to add per user RPC processing time metrics and expose it via JMX. was: For a shared cluster we would want to separate users' resources, as well as having our metrics reflecting on the usage, latency, etc, for each user. This JIRA aims to add per user RPC response time metrics and expose it via JMX. > Add per user RPC Processing time > > > Key: HDFS-14406 > URL: https://issues.apache.org/jira/browse/HDFS-14406 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.2.0 >Reporter: Xue Liu >Assignee: Xue Liu >Priority: Minor > Fix For: 3.2.0 > > > For a shared cluster we would want to separate users' resources, as well as > having our metrics reflecting on the usage, latency, etc, for each user. > This JIRA aims to add per user RPC processing time metrics and expose it via > JMX. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14406) Add per user RPC Processing time
[ https://issues.apache.org/jira/browse/HDFS-14406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xue Liu updated HDFS-14406: --- Description: For a shared cluster we would want to separate users' resources, as well as having our metrics reflecting on the usage, latency, etc, for each user. This Jira aims to add per user RPC response time metrics and expose it via jmx. was: For a shared cluster we would want to separate users' resources, as well as having our metrics reflecting on the usage, latency, etc, for each user. This Jira aims to add per user RPC response time metrics and export it via jmx. > Add per user RPC Processing time > > > Key: HDFS-14406 > URL: https://issues.apache.org/jira/browse/HDFS-14406 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.2.0 >Reporter: Xue Liu >Priority: Minor > Fix For: 3.2.0 > > > For a shared cluster we would want to separate users' resources, as well as > having our metrics reflecting on the usage, latency, etc, for each user. > This Jira aims to add per user RPC response time metrics and expose it via > jmx. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-14406) Add per user RPC Processing time
Xue Liu created HDFS-14406: -- Summary: Add per user RPC Processing time Key: HDFS-14406 URL: https://issues.apache.org/jira/browse/HDFS-14406 Project: Hadoop HDFS Issue Type: Improvement Affects Versions: 3.2.0 Reporter: Xue Liu Fix For: 3.2.0 For a shared cluster we would want to separate users' resources, as well as having our metrics reflecting on the usage, latency, etc, for each user. This Jira aims to add per user RPC response time metrics and export it via jmx. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
