[jira] [Updated] (HDFS-13055) Aggregate usage statistics from datanodes
[ https://issues.apache.org/jira/browse/HDFS-13055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDFS-13055: - Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 3.2.0 Status: Resolved (was: Patch Available) +1. I've committed this. Thanks for the contribution [~ajayydv]! > Aggregate usage statistics from datanodes > - > > Key: HDFS-13055 > URL: https://issues.apache.org/jira/browse/HDFS-13055 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > Fix For: 3.2.0 > > Attachments: HDFS-13055.001.patch, HDFS-13055.002.patch, > HDFS-13055.003.patch, HDFS-13055.004.patch, HDFS-13055.005.patch, > HDFS-13055.006.patch, HDFS-13055.007.patch, HDFS-13055.008.patch, > HDFS-13055.009.patch, HDFS-13055.010.patch > > > We collect variety of statistics in DataNodes and expose them via JMX. > Aggregating some of the high level statistics which we are already collecting > in {{DataNodeMetrics}} (like bytesRead,bytesWritten etc) over a configurable > time window will create a central repository accessible via JMX and UI. > Breaking this into 3 parts as suggested by [~arpitagarwal] > # Generate usage report on DN. (will handle in this jira) > # Pass usage report to NN periodically. > # Process report on NN and expose stats via JMX. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13055) Aggregate usage statistics from datanodes
[ https://issues.apache.org/jira/browse/HDFS-13055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Kumar updated HDFS-13055: -- Attachment: HDFS-13055.010.patch > Aggregate usage statistics from datanodes > - > > Key: HDFS-13055 > URL: https://issues.apache.org/jira/browse/HDFS-13055 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > Attachments: HDFS-13055.001.patch, HDFS-13055.002.patch, > HDFS-13055.003.patch, HDFS-13055.004.patch, HDFS-13055.005.patch, > HDFS-13055.006.patch, HDFS-13055.007.patch, HDFS-13055.008.patch, > HDFS-13055.009.patch, HDFS-13055.010.patch > > > We collect variety of statistics in DataNodes and expose them via JMX. > Aggregating some of the high level statistics which we are already collecting > in {{DataNodeMetrics}} (like bytesRead,bytesWritten etc) over a configurable > time window will create a central repository accessible via JMX and UI. > Breaking this into 3 parts as suggested by [~arpitagarwal] > # Generate usage report on DN. (will handle in this jira) > # Pass usage report to NN periodically. > # Process report on NN and expose stats via JMX. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13055) Aggregate usage statistics from datanodes
[ https://issues.apache.org/jira/browse/HDFS-13055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Kumar updated HDFS-13055: -- Attachment: HDFS-13055.009.patch > Aggregate usage statistics from datanodes > - > > Key: HDFS-13055 > URL: https://issues.apache.org/jira/browse/HDFS-13055 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > Attachments: HDFS-13055.001.patch, HDFS-13055.002.patch, > HDFS-13055.003.patch, HDFS-13055.004.patch, HDFS-13055.005.patch, > HDFS-13055.006.patch, HDFS-13055.007.patch, HDFS-13055.008.patch, > HDFS-13055.009.patch > > > We collect variety of statistics in DataNodes and expose them via JMX. > Aggregating some of the high level statistics which we are already collecting > in {{DataNodeMetrics}} (like bytesRead,bytesWritten etc) over a configurable > time window will create a central repository accessible via JMX and UI. > Breaking this into 3 parts as suggested by [~arpitagarwal] > # Generate usage report on DN. (will handle in this jira) > # Pass usage report to NN periodically. > # Process report on NN and expose stats via JMX. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13055) Aggregate usage statistics from datanodes
[ https://issues.apache.org/jira/browse/HDFS-13055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Kumar updated HDFS-13055: -- Description: We collect variety of statistics in DataNodes and expose them via JMX. Aggregating some of the high level statistics which we are already collecting in {{DataNodeMetrics}} (like bytesRead,bytesWritten etc) over a configurable time window will create a central repository accessible via JMX and UI. Breaking this into 3 parts as suggested by [~arpitagarwal] # Generate usage report on DN. (will handle in this jira) # Pass usage report to NN periodically. # Process report on NN and expose stats via JMX. was:We collect variety of statistics in DataNodes and expose them via JMX. Aggregating some of the high level statistics which we are already collecting in {{DataNodeMetrics}} (like bytesRead,bytesWritten etc) over a configurable time window will create a central repository accessible via JMX and UI. > Aggregate usage statistics from datanodes > - > > Key: HDFS-13055 > URL: https://issues.apache.org/jira/browse/HDFS-13055 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > Attachments: HDFS-13055.001.patch, HDFS-13055.002.patch, > HDFS-13055.003.patch, HDFS-13055.004.patch, HDFS-13055.005.patch, > HDFS-13055.006.patch, HDFS-13055.007.patch, HDFS-13055.008.patch > > > We collect variety of statistics in DataNodes and expose them via JMX. > Aggregating some of the high level statistics which we are already collecting > in {{DataNodeMetrics}} (like bytesRead,bytesWritten etc) over a configurable > time window will create a central repository accessible via JMX and UI. > Breaking this into 3 parts as suggested by [~arpitagarwal] > # Generate usage report on DN. (will handle in this jira) > # Pass usage report to NN periodically. > # Process report on NN and expose stats via JMX. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13055) Aggregate usage statistics from datanodes
[ https://issues.apache.org/jira/browse/HDFS-13055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Kumar updated HDFS-13055: -- Attachment: HDFS-13055.008.patch > Aggregate usage statistics from datanodes > - > > Key: HDFS-13055 > URL: https://issues.apache.org/jira/browse/HDFS-13055 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > Attachments: HDFS-13055.001.patch, HDFS-13055.002.patch, > HDFS-13055.003.patch, HDFS-13055.004.patch, HDFS-13055.005.patch, > HDFS-13055.006.patch, HDFS-13055.007.patch, HDFS-13055.008.patch > > > We collect variety of statistics in DataNodes and expose them via JMX. > Aggregating some of the high level statistics which we are already collecting > in {{DataNodeMetrics}} (like bytesRead,bytesWritten etc) over a configurable > time window will create a central repository accessible via JMX and UI. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13055) Aggregate usage statistics from datanodes
[ https://issues.apache.org/jira/browse/HDFS-13055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Kumar updated HDFS-13055: -- Attachment: HDFS-13055.007.patch > Aggregate usage statistics from datanodes > - > > Key: HDFS-13055 > URL: https://issues.apache.org/jira/browse/HDFS-13055 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > Attachments: HDFS-13055.001.patch, HDFS-13055.002.patch, > HDFS-13055.003.patch, HDFS-13055.004.patch, HDFS-13055.005.patch, > HDFS-13055.006.patch, HDFS-13055.007.patch > > > We collect variety of statistics in DataNodes and expose them via JMX. > Aggregating some of the high level statistics which we are already collecting > in {{DataNodeMetrics}} (like bytesRead,bytesWritten etc) over a configurable > time window will create a central repository accessible via JMX and UI. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13055) Aggregate usage statistics from datanodes
[ https://issues.apache.org/jira/browse/HDFS-13055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Kumar updated HDFS-13055: -- Attachment: HDFS-13055.006.patch > Aggregate usage statistics from datanodes > - > > Key: HDFS-13055 > URL: https://issues.apache.org/jira/browse/HDFS-13055 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > Attachments: HDFS-13055.001.patch, HDFS-13055.002.patch, > HDFS-13055.003.patch, HDFS-13055.004.patch, HDFS-13055.005.patch, > HDFS-13055.006.patch > > > We collect variety of statistics in DataNodes and expose them via JMX. > Aggregating some of the high level statistics which we are already collecting > in {{DataNodeMetrics}} (like bytesRead,bytesWritten etc) over a configurable > time window will create a central repository accessible via JMX and UI. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13055) Aggregate usage statistics from datanodes
[ https://issues.apache.org/jira/browse/HDFS-13055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Kumar updated HDFS-13055: -- Attachment: HDFS-13055.005.patch > Aggregate usage statistics from datanodes > - > > Key: HDFS-13055 > URL: https://issues.apache.org/jira/browse/HDFS-13055 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > Attachments: HDFS-13055.001.patch, HDFS-13055.002.patch, > HDFS-13055.003.patch, HDFS-13055.004.patch, HDFS-13055.005.patch > > > We collect variety of statistics in DataNodes and expose them via JMX. > Aggregating some of the high level statistics which we are already collecting > in {{DataNodeMetrics}} (like bytesRead,bytesWritten etc) over a configurable > time window will create a central repository accessible via JMX and UI. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13055) Aggregate usage statistics from datanodes
[ https://issues.apache.org/jira/browse/HDFS-13055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Kumar updated HDFS-13055: -- Attachment: HDFS-13055.004.patch > Aggregate usage statistics from datanodes > - > > Key: HDFS-13055 > URL: https://issues.apache.org/jira/browse/HDFS-13055 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > Attachments: HDFS-13055.001.patch, HDFS-13055.002.patch, > HDFS-13055.003.patch, HDFS-13055.004.patch > > > We collect variety of statistics in DataNodes and expose them via JMX. > Aggregating some of the high level statistics which we are already collecting > in {{DataNodeMetrics}} (like bytesRead,bytesWritten etc) over a configurable > time window will create a central repository accessible via JMX and UI. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13055) Aggregate usage statistics from datanodes
[ https://issues.apache.org/jira/browse/HDFS-13055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Kumar updated HDFS-13055: -- Attachment: HDFS-13055.003.patch > Aggregate usage statistics from datanodes > - > > Key: HDFS-13055 > URL: https://issues.apache.org/jira/browse/HDFS-13055 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > Attachments: HDFS-13055.001.patch, HDFS-13055.002.patch, > HDFS-13055.003.patch > > > We collect variety of statistics in DataNodes and expose them via JMX. > Aggregating some of the high level statistics which we are already collecting > in {{DataNodeMetrics}} (like bytesRead,bytesWritten etc) over a configurable > time window will create a central repository accessible via JMX and UI. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13055) Aggregate usage statistics from datanodes
[ https://issues.apache.org/jira/browse/HDFS-13055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Kumar updated HDFS-13055: -- Attachment: (was: HDFS-13055.003.patch) > Aggregate usage statistics from datanodes > - > > Key: HDFS-13055 > URL: https://issues.apache.org/jira/browse/HDFS-13055 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > Attachments: HDFS-13055.001.patch, HDFS-13055.002.patch > > > We collect variety of statistics in DataNodes and expose them via JMX. > Aggregating some of the high level statistics which we are already collecting > in {{DataNodeMetrics}} (like bytesRead,bytesWritten etc) over a configurable > time window will create a central repository accessible via JMX and UI. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13055) Aggregate usage statistics from datanodes
[ https://issues.apache.org/jira/browse/HDFS-13055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Kumar updated HDFS-13055: -- Attachment: HDFS-13055.003.patch > Aggregate usage statistics from datanodes > - > > Key: HDFS-13055 > URL: https://issues.apache.org/jira/browse/HDFS-13055 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > Attachments: HDFS-13055.001.patch, HDFS-13055.002.patch, > HDFS-13055.003.patch > > > We collect variety of statistics in DataNodes and expose them via JMX. > Aggregating some of the high level statistics which we are already collecting > in {{DataNodeMetrics}} (like bytesRead,bytesWritten etc) over a configurable > time window will create a central repository accessible via JMX and UI. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13055) Aggregate usage statistics from datanodes
[ https://issues.apache.org/jira/browse/HDFS-13055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Kumar updated HDFS-13055: -- Attachment: HDFS-13055.002.patch > Aggregate usage statistics from datanodes > - > > Key: HDFS-13055 > URL: https://issues.apache.org/jira/browse/HDFS-13055 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > Attachments: HDFS-13055.001.patch, HDFS-13055.002.patch > > > We collect variety of statistics in DataNodes and expose them via JMX. > Aggregating some of the high level statistics which we are already collecting > in {{DataNodeMetrics}} (like bytesRead,bytesWritten etc) over a configurable > time window will create a central repository accessible via JMX and UI. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13055) Aggregate usage statistics from datanodes
[ https://issues.apache.org/jira/browse/HDFS-13055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Kumar updated HDFS-13055: -- Status: Patch Available (was: Open) > Aggregate usage statistics from datanodes > - > > Key: HDFS-13055 > URL: https://issues.apache.org/jira/browse/HDFS-13055 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > Attachments: HDFS-13055.001.patch > > > We collect variety of statistics in DataNodes and expose them via JMX. > Aggregating some of the high level statistics which we are already collecting > in {{DataNodeMetrics}} (like bytesRead,bytesWritten etc) over a configurable > time window will create a central repository accessible via JMX and UI. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13055) Aggregate usage statistics from datanodes
[ https://issues.apache.org/jira/browse/HDFS-13055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Kumar updated HDFS-13055: -- Attachment: HDFS-13055.001.patch > Aggregate usage statistics from datanodes > - > > Key: HDFS-13055 > URL: https://issues.apache.org/jira/browse/HDFS-13055 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > Attachments: HDFS-13055.001.patch > > > We collect variety of statistics in DataNodes and expose them via JMX. > Aggregating some of the high level statistics which we are already collecting > in {{DataNodeMetrics}} (like bytesRead,bytesWritten etc) over a configurable > time window will create a central repository accessible via JMX and UI. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13055) Aggregate usage statistics from datanodes
[ https://issues.apache.org/jira/browse/HDFS-13055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Kumar updated HDFS-13055: -- Attachment: (was: HDFS-13055.001.patch) > Aggregate usage statistics from datanodes > - > > Key: HDFS-13055 > URL: https://issues.apache.org/jira/browse/HDFS-13055 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > > We collect variety of statistics in DataNodes and expose them via JMX. > Aggregating some of the high level statistics which we are already collecting > in {{DataNodeMetrics}} (like bytesRead,bytesWritten etc) over a configurable > time window will create a central repository accessible via JMX and UI. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13055) Aggregate usage statistics from datanodes
[ https://issues.apache.org/jira/browse/HDFS-13055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Kumar updated HDFS-13055: -- Attachment: HDFS-13055.001.patch > Aggregate usage statistics from datanodes > - > > Key: HDFS-13055 > URL: https://issues.apache.org/jira/browse/HDFS-13055 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > Attachments: HDFS-13055.001.patch > > > We collect variety of statistics in DataNodes and expose them via JMX. > Aggregating some of the high level statistics which we are already collecting > in {{DataNodeMetrics}} (like bytesRead,bytesWritten etc) over a configurable > time window will create a central repository accessible via JMX and UI. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org