[jira] [Commented] (HDFS-13541) NameNode Port based selective encryption
[
https://issues.apache.org/jira/browse/HDFS-13541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16491147#comment-16491147
]
Benoy Antony commented on HDFS-13541:
-
{quote}
in our environment, cross data center traffic actually compose a small
fraction of all traffic, having additional DataXceiverServer thread sitting and
listening on every single data node, but being idle most of time does not seem
to be ideal.
{quote}
If its a small fraction, then one option would be to use (1) swebhdfs or (2)
HTTPFS gateway which uses RPC. They work seamlessly for clients via the
FileSystem implementation and swebhdfs scheme.
(1) introduces HTTP protocol overhead and lack of QOS on namenode.
(2) introduces HTTP protocol overhead and an additional hop.
But these are the common solutions to this problem for small fractions of data
which needs to be treated differently. But your use case may be different
which warrants secure and non secure RPC and data transfer.
> NameNode Port based selective encryption
>
>
> Key: HDFS-13541
> URL: https://issues.apache.org/jira/browse/HDFS-13541
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode, namenode, security
>Reporter: Chen Liang
>Assignee: Chen Liang
>Priority: Major
> Attachments: NameNode Port based selective encryption-v1.pdf
>
>
> Here at LinkedIn, one issue we face is that we need to enforce different
> security requirement based on the location of client and the cluster.
> Specifically, for clients from outside of the data center, it is required by
> regulation that all traffic must be encrypted. But for clients within the
> same data center, unencrypted connections are more desired to avoid the high
> encryption overhead.
> HADOOP-10221 introduced pluggable SASL resolver, based on which HADOOP-10335
> introduced WhitelistBasedResolver which solves the same problem. However we
> found it difficult to fit into our environment for several reasons. In this
> JIRA, on top of pluggable SASL resolver, *we propose a different approach of
> running RPC two ports on NameNode, and the two ports will be enforcing
> encrypted and unencrypted connections respectively, and the following
> DataNode access will simply follow the same behaviour of
> encryption/unencryption*. Then by blocking unencrypted port on datacenter
> firewall, we can completely block unencrypted external access.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13541) NameNode Port based selective encryption
[ https://issues.apache.org/jira/browse/HDFS-13541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16470684#comment-16470684 ] Benoy Antony commented on HDFS-13541: - Thanks [~vagarychen]. I have reviewed the document. Overall, I believe, this approach makes the administration of selective encryption simpler. The selection can be done using firewall rules. I have a few comments based on my understanding of the design. # Currently the input to _SaslPropertiesResolver_ methods is ip address. This feature adds ingress port as an input. I believe, it should be made generic so that we do not have to change it in future. We could also think of passing additional parameters based on the connection even though they may not be used in the current implementations of _SaslPropertiesResolver._ # _The_ _selective data transfer protection_ design needs more scrutiny as it's protection is derived from _selective RPC protection_. Based on what I understood, the protection is dictated by the value of _encrypted message_ sent by the client when it handshakes with datanode just before the data transfer. The _encrypted message is either_ _auth_ or _auth_conf_ encrypted by the secret shared between namenode and the datanode. If so, what prevents the external client from replaying the _encrypted message_ from a different connection between an internal client and datanode ? # Another side effect of derived QOP for data transfer protection is that one cannot enable RPC protection alone with this approach. # As mentioned in the document, Encrypting the entire data pipeline is not necessary. I believe, it should be optimized. # All things equal, I prefer the approach where datanode also listens on two ports, as it makes the entire approach easy to understand. It will also solve issues specified in #2 and #3 above. Those issues are the result of QOP of data transfer operation becoming a derivative of RPC operation. > NameNode Port based selective encryption > > > Key: HDFS-13541 > URL: https://issues.apache.org/jira/browse/HDFS-13541 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode, namenode, security >Reporter: Chen Liang >Assignee: Chen Liang >Priority: Major > Attachments: NameNode Port based selective encryption-v1.pdf > > > Here at LinkedIn, one issue we face is that we need to enforce different > security requirement based on the location of client and the cluster. > Specifically, for clients from outside of the data center, it is required by > regulation that all traffic must be encrypted. But for clients within the > same data center, unencrypted connections are more desired to avoid the high > encryption overhead. > HADOOP-10221 introduced pluggable SASL resolver, based on which HADOOP-10335 > introduced WhitelistBasedResolver which solves the same problem. However we > found it difficult to fit into our environment for several reasons. In this > JIRA, on top of pluggable SASL resolver, *we propose a different approach of > running RPC two ports on NameNode, and the two ports will be enforcing > encrypted and unencrypted connections respectively, and the following > DataNode access will simply follow the same behaviour of > encryption/unencryption*. Then by blocking unencrypted port on datacenter > firewall, we can completely block unencrypted external access. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-7466) Allow different values for dfs.datanode.balance.max.concurrent.moves per datanode
[ https://issues.apache.org/jira/browse/HDFS-7466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16206818#comment-16206818 ] Benoy Antony commented on HDFS-7466: I think [~dan...@cloudera.com]'s suggestion makes a lot of sense. I got some time now to implement this scheme. I'll first submit a brief design spec on this. > Allow different values for dfs.datanode.balance.max.concurrent.moves per > datanode > - > > Key: HDFS-7466 > URL: https://issues.apache.org/jira/browse/HDFS-7466 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: balancer & mover >Affects Versions: 2.6.0 >Reporter: Benoy Antony >Assignee: Benoy Antony > > It is possible to configure different values for > _dfs.datanode.balance.max.concurrent.moves_ per datanode. But the value will > be used by balancer/mover which obtains the value from its own configuration. > The correct approach will be to obtain the value from the datanode itself. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work started] (HDFS-7466) Allow different values for dfs.datanode.balance.max.concurrent.moves per datanode
[ https://issues.apache.org/jira/browse/HDFS-7466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HDFS-7466 started by Benoy Antony. -- > Allow different values for dfs.datanode.balance.max.concurrent.moves per > datanode > - > > Key: HDFS-7466 > URL: https://issues.apache.org/jira/browse/HDFS-7466 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: balancer & mover >Affects Versions: 2.6.0 >Reporter: Benoy Antony >Assignee: Benoy Antony > > It is possible to configure different values for > _dfs.datanode.balance.max.concurrent.moves_ per datanode. But the value will > be used by balancer/mover which obtains the value from its own configuration. > The correct approach will be to obtain the value from the datanode itself. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-5290) Port Encryption Support in DataTransfer protocol to Hadoop 1
[ https://issues.apache.org/jira/browse/HDFS-5290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benoy Antony resolved HDFS-5290. Resolution: Won't Do Cleaning up jiras which is not relevant anymore. > Port Encryption Support in DataTransfer protocol to Hadoop 1 > > > Key: HDFS-5290 > URL: https://issues.apache.org/jira/browse/HDFS-5290 > Project: Hadoop HDFS > Issue Type: Improvement > Components: security >Affects Versions: 1.2.1 >Reporter: Benoy Antony >Assignee: Benoy Antony > Fix For: 1.3.0 > > Attachments: HDFS-5290.patch > > > Currently all HDFS RPCs performed by NNs/DNs/clients can be optionally > encrypted. However, actual data read or written between DNs and clients (or > DNs to DNs) is sent in the clear. When processing sensitive data on a shared > cluster, confidentiality of the data read/written from/to HDFS may be desired. > This change is already done in Hadoop 2 and trunk. > The task is to port this change to Hadoop 1. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-4823) Inode.toString () should return the full path
[ https://issues.apache.org/jira/browse/HDFS-4823?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benoy Antony resolved HDFS-4823. Resolution: Won't Do Cleaning up jiras which is not relevant anymore. > Inode.toString () should return the full path > --- > > Key: HDFS-4823 > URL: https://issues.apache.org/jira/browse/HDFS-4823 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 1.1.2 >Reporter: Benoy Antony >Assignee: Benoy Antony >Priority: Minor > Attachments: HDFS-4823.patch > > > Indoe.ToString() is used in many error messages. This gives the name of the > file / directory, but not the fullpath. > org.apache.hadoop.security.AccessControlException > org.apache.hadoop.security.AccessControlException: Permission denied: > user=user1, access=WRITE, inode="warehouse":user2:supergroup:rwxrwxr-x) > The fix is to provide the full path n line with Hadoop 2. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11384) Add option for balancer to disperse getBlocks calls to avoid NameNode's rpc.CallQueueLength spike
[ https://issues.apache.org/jira/browse/HDFS-11384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15903651#comment-15903651 ] Benoy Antony commented on HDFS-11384: - [~zhaoyunjiong], Could you please check to see if sleeping outside the sync block with a potentially longer delay is sufficient to avoid spike ? > Add option for balancer to disperse getBlocks calls to avoid NameNode's > rpc.CallQueueLength spike > - > > Key: HDFS-11384 > URL: https://issues.apache.org/jira/browse/HDFS-11384 > Project: Hadoop HDFS > Issue Type: Improvement > Components: balancer & mover >Affects Versions: 2.7.3 >Reporter: yunjiong zhao >Assignee: yunjiong zhao > Attachments: balancer.day.png, balancer.week.png, HDFS-11384.001.patch > > > When running balancer on hadoop cluster which have more than 3000 Datanodes > will cause NameNode's rpc.CallQueueLength spike. We observed this situation > could cause Hbase cluster failure due to RegionServer's WAL timeout. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11384) Add option for balancer to disperse getBlocks calls to avoid NameNode's rpc.CallQueueLength spike
[ https://issues.apache.org/jira/browse/HDFS-11384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15891587#comment-15891587 ] Benoy Antony commented on HDFS-11384: - Thanks for the explanation, [~zhaoyunjiong]. The patch looks good. I will commit this tomorrow if there are no other comments. > Add option for balancer to disperse getBlocks calls to avoid NameNode's > rpc.CallQueueLength spike > - > > Key: HDFS-11384 > URL: https://issues.apache.org/jira/browse/HDFS-11384 > Project: Hadoop HDFS > Issue Type: Improvement > Components: balancer & mover >Affects Versions: 2.7.3 >Reporter: yunjiong zhao >Assignee: yunjiong zhao > Attachments: balancer.day.png, balancer.week.png, HDFS-11384.001.patch > > > When running balancer on hadoop cluster which have more than 3000 Datanodes > will cause NameNode's rpc.CallQueueLength spike. We observed this situation > could cause Hbase cluster failure due to RegionServer's WAL timeout. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11384) Add option for balancer to disperse getBlocks calls to avoid NameNode's rpc.CallQueueLength spike
[ https://issues.apache.org/jira/browse/HDFS-11384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15890957#comment-15890957 ] Benoy Antony commented on HDFS-11384: - Sleeping inside the *Synchronized* block should be avoided as it will lock prevent other threads from obtaining the lock while the thread is sleeping. One tradeoff in sleeping fixed vs variable time is that code gets complicated. Since by default, the delay is not applied, it is okay to sleep for a fixed interval after getBlocks(). > Add option for balancer to disperse getBlocks calls to avoid NameNode's > rpc.CallQueueLength spike > - > > Key: HDFS-11384 > URL: https://issues.apache.org/jira/browse/HDFS-11384 > Project: Hadoop HDFS > Issue Type: Improvement > Components: balancer & mover >Affects Versions: 2.7.3 >Reporter: yunjiong zhao >Assignee: yunjiong zhao > Attachments: balancer.day.png, balancer.week.png, HDFS-11384.001.patch > > > When running balancer on hadoop cluster which have more than 3000 Datanodes > will cause NameNode's rpc.CallQueueLength spike. We observed this situation > could cause Hbase cluster failure due to RegionServer's WAL timeout. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-11384) Add option for balancer to disperse getBlocks calls to avoid NameNode's rpc.CallQueueLength spike
[ https://issues.apache.org/jira/browse/HDFS-11384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15890957#comment-15890957 ] Benoy Antony edited comment on HDFS-11384 at 3/1/17 8:17 PM: - Sleeping inside the *Synchronized* block should be avoided as it will prevent other threads from obtaining the lock while the thread is sleeping. One tradeoff in sleeping fixed vs variable time is that code gets complicated. Since by default, the delay is not applied, it is okay to sleep for a fixed interval after getBlocks(). was (Author: benoyantony): Sleeping inside the *Synchronized* block should be avoided as it will lock prevent other threads from obtaining the lock while the thread is sleeping. One tradeoff in sleeping fixed vs variable time is that code gets complicated. Since by default, the delay is not applied, it is okay to sleep for a fixed interval after getBlocks(). > Add option for balancer to disperse getBlocks calls to avoid NameNode's > rpc.CallQueueLength spike > - > > Key: HDFS-11384 > URL: https://issues.apache.org/jira/browse/HDFS-11384 > Project: Hadoop HDFS > Issue Type: Improvement > Components: balancer & mover >Affects Versions: 2.7.3 >Reporter: yunjiong zhao >Assignee: yunjiong zhao > Attachments: balancer.day.png, balancer.week.png, HDFS-11384.001.patch > > > When running balancer on hadoop cluster which have more than 3000 Datanodes > will cause NameNode's rpc.CallQueueLength spike. We observed this situation > could cause Hbase cluster failure due to RegionServer's WAL timeout. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11384) Add option for balancer to disperse getBlocks calls to avoid NameNode's rpc.CallQueueLength spike
[ https://issues.apache.org/jira/browse/HDFS-11384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15888766#comment-15888766 ] Benoy Antony commented on HDFS-11384: - [~zhaoyunjiong], If there are blocks to balance, then there will be sufficient delays between successive getBlocks. In such cases, we do not have to sleep. It will be better to keep track of the interval between successive getBlocks and sleep only for the required time. Can you also write a unit test to cover this change ? > Add option for balancer to disperse getBlocks calls to avoid NameNode's > rpc.CallQueueLength spike > - > > Key: HDFS-11384 > URL: https://issues.apache.org/jira/browse/HDFS-11384 > Project: Hadoop HDFS > Issue Type: Improvement > Components: balancer & mover >Affects Versions: 2.7.3 >Reporter: yunjiong zhao >Assignee: yunjiong zhao > Attachments: balancer.day.png, balancer.week.png, HDFS-11384.001.patch > > > When running balancer on hadoop cluster which have more than 3000 Datanodes > will cause NameNode's rpc.CallQueueLength spike. We observed this situation > could cause Hbase cluster failure due to RegionServer's WAL timeout. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10735) Distcp using webhdfs on secure HA clusters fails with StandbyException
[ https://issues.apache.org/jira/browse/HDFS-10735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benoy Antony updated HDFS-10735: Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Committed to trunk , branch-2 and branch-2.8 > Distcp using webhdfs on secure HA clusters fails with StandbyException > -- > > Key: HDFS-10735 > URL: https://issues.apache.org/jira/browse/HDFS-10735 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 2.7.2 >Reporter: Benoy Antony >Assignee: Benoy Antony > Fix For: 2.8.0 > > Attachments: HDFS-10735-001.patch, exception.txt > > > A distcp running on HADOOP 2.7.2 cluster , copying from HADOOP 2.4.1 cluster > fails. > Both are secure clusters. > The logs indicates that task contacts the StandBy name node on the remote > cluster i first and gets a standByException. > Instead of retrying with the active name node on the remote cluster, the > execution errors out throwing InvalidToken > The exception is attached -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10735) Distcp using webhdfs on secure HA clusters fails with StandbyException
[ https://issues.apache.org/jira/browse/HDFS-10735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benoy Antony updated HDFS-10735: Description: A distcp running on HADOOP 2.7.2 cluster , copying from HADOOP 2.4.1 cluster fails. Both are secure clusters. The logs indicates that task contacts the StandBy name node on the remote cluster i first and gets a standByException. Instead of retrying with the active name node on the remote cluster, the execution errors out throwing InvalidToken The exception is attached was: A distcp running on HADOOP 2.7.2 cluster , copying from HADOOP 2.4.1 cluster fails. Both are secure clusters. The logs indicates that task contacts the StandBy name node on the remote cluster is contacted first and gets a standByException. Instead of retrying with the active name node on the remote cluster, the execution errors out throwing InvalidToken The exception is attached > Distcp using webhdfs on secure HA clusters fails with StandbyException > -- > > Key: HDFS-10735 > URL: https://issues.apache.org/jira/browse/HDFS-10735 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 2.7.2 >Reporter: Benoy Antony >Assignee: Benoy Antony > Fix For: 2.8.0 > > Attachments: HDFS-10735-001.patch, exception.txt > > > A distcp running on HADOOP 2.7.2 cluster , copying from HADOOP 2.4.1 cluster > fails. > Both are secure clusters. > The logs indicates that task contacts the StandBy name node on the remote > cluster i first and gets a standByException. > Instead of retrying with the active name node on the remote cluster, the > execution errors out throwing InvalidToken > The exception is attached -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10735) Distcp using webhdfs on secure HA clusters fails with StandbyException
[ https://issues.apache.org/jira/browse/HDFS-10735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15570939#comment-15570939 ] Benoy Antony commented on HDFS-10735: - [~arpitagarwal], Thanks for the review. Since the issue happens when interacting with an older version of HDFS, it will be difficult to test. If there are no further comments, I'll commit this in a couple of days. > Distcp using webhdfs on secure HA clusters fails with StandbyException > -- > > Key: HDFS-10735 > URL: https://issues.apache.org/jira/browse/HDFS-10735 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 2.7.2 >Reporter: Benoy Antony >Assignee: Benoy Antony > Fix For: 2.8.0 > > Attachments: HDFS-10735-001.patch, exception.txt > > > A distcp running on HADOOP 2.7.2 cluster , copying from HADOOP 2.4.1 cluster > fails. > Both are secure clusters. > The logs indicates that task contacts the StandBy name node on the remote > cluster is contacted first and gets a standByException. > Instead of retrying with the active name node on the remote cluster, the > execution errors out throwing InvalidToken > The exception is attached -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10735) Distcp using webhdfs on secure HA clusters fails with StandbyException
[ https://issues.apache.org/jira/browse/HDFS-10735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benoy Antony updated HDFS-10735: Attachment: HDFS-10735-001.patch > Distcp using webhdfs on secure HA clusters fails with StandbyException > -- > > Key: HDFS-10735 > URL: https://issues.apache.org/jira/browse/HDFS-10735 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 2.7.2 >Reporter: Benoy Antony >Assignee: Benoy Antony > Fix For: 2.8.0 > > Attachments: HDFS-10735-001.patch, exception.txt > > > A distcp running on HADOOP 2.7.2 cluster , copying from HADOOP 2.4.1 cluster > fails. > Both are secure clusters. > The logs indicates that task contacts the StandBy name node on the remote > cluster is contacted first and gets a standByException. > Instead of retrying with the active name node on the remote cluster, the > execution errors out throwing InvalidToken > The exception is attached -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10735) Distcp using webhdfs on secure HA clusters fails with StandbyException
[ https://issues.apache.org/jira/browse/HDFS-10735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benoy Antony updated HDFS-10735: Attachment: (was: HDFS-10735-001.patch) > Distcp using webhdfs on secure HA clusters fails with StandbyException > -- > > Key: HDFS-10735 > URL: https://issues.apache.org/jira/browse/HDFS-10735 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 2.7.2 >Reporter: Benoy Antony >Assignee: Benoy Antony > Fix For: 2.8.0 > > Attachments: exception.txt > > > A distcp running on HADOOP 2.7.2 cluster , copying from HADOOP 2.4.1 cluster > fails. > Both are secure clusters. > The logs indicates that task contacts the StandBy name node on the remote > cluster is contacted first and gets a standByException. > Instead of retrying with the active name node on the remote cluster, the > execution errors out throwing InvalidToken > The exception is attached -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Issue Comment Deleted] (HDFS-10735) Distcp using webhdfs on secure HA clusters fails with StandbyException
[
https://issues.apache.org/jira/browse/HDFS-10735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Benoy Antony updated HDFS-10735:
Comment: was deleted
(was: | (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 4s{color}
| {color:red} HDFS-10735 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12822708/HDFS-10735-001.patch |
| JIRA Issue | HDFS-10735 |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/16349/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
)
> Distcp using webhdfs on secure HA clusters fails with StandbyException
> --
>
> Key: HDFS-10735
> URL: https://issues.apache.org/jira/browse/HDFS-10735
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
>Affects Versions: 2.7.2
>Reporter: Benoy Antony
>Assignee: Benoy Antony
> Fix For: 2.8.0
>
> Attachments: exception.txt
>
>
> A distcp running on HADOOP 2.7.2 cluster , copying from HADOOP 2.4.1 cluster
> fails.
> Both are secure clusters.
> The logs indicates that task contacts the StandBy name node on the remote
> cluster is contacted first and gets a standByException.
> Instead of retrying with the active name node on the remote cluster, the
> execution errors out throwing InvalidToken
> The exception is attached
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10735) Distcp using webhdfs on secure HA clusters fails with StandbyException
[ https://issues.apache.org/jira/browse/HDFS-10735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benoy Antony updated HDFS-10735: Status: Patch Available (was: Open) > Distcp using webhdfs on secure HA clusters fails with StandbyException > -- > > Key: HDFS-10735 > URL: https://issues.apache.org/jira/browse/HDFS-10735 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 2.7.2 >Reporter: Benoy Antony >Assignee: Benoy Antony > Fix For: 2.8.0 > > Attachments: HDFS-10735-001.patch, exception.txt > > > A distcp running on HADOOP 2.7.2 cluster , copying from HADOOP 2.4.1 cluster > fails. > Both are secure clusters. > The logs indicates that task contacts the StandBy name node on the remote > cluster is contacted first and gets a standByException. > Instead of retrying with the active name node on the remote cluster, the > execution errors out throwing InvalidToken > The exception is attached -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10735) Distcp using webhdfs on secure HA clusters fails with StandbyException
[ https://issues.apache.org/jira/browse/HDFS-10735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benoy Antony updated HDFS-10735: Attachment: HDFS-10735-001.patch > Distcp using webhdfs on secure HA clusters fails with StandbyException > -- > > Key: HDFS-10735 > URL: https://issues.apache.org/jira/browse/HDFS-10735 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 2.7.2 >Reporter: Benoy Antony >Assignee: Benoy Antony > Fix For: 2.8.0 > > Attachments: HDFS-10735-001.patch, exception.txt > > > A distcp running on HADOOP 2.7.2 cluster , copying from HADOOP 2.4.1 cluster > fails. > Both are secure clusters. > The logs indicates that task contacts the StandBy name node on the remote > cluster is contacted first and gets a standByException. > Instead of retrying with the active name node on the remote cluster, the > execution errors out throwing InvalidToken > The exception is attached -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10735) Distcp using webhdfs on secure HA clusters fails with StandbyException
[ https://issues.apache.org/jira/browse/HDFS-10735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15412749#comment-15412749 ] Benoy Antony commented on HDFS-10735: - Attaching the patch. The patch is tested on our clusters. > Distcp using webhdfs on secure HA clusters fails with StandbyException > -- > > Key: HDFS-10735 > URL: https://issues.apache.org/jira/browse/HDFS-10735 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 2.7.2 >Reporter: Benoy Antony >Assignee: Benoy Antony > Fix For: 2.8.0 > > Attachments: exception.txt > > > A distcp running on HADOOP 2.7.2 cluster , copying from HADOOP 2.4.1 cluster > fails. > Both are secure clusters. > The logs indicates that task contacts the StandBy name node on the remote > cluster is contacted first and gets a standByException. > Instead of retrying with the active name node on the remote cluster, the > execution errors out throwing InvalidToken > The exception is attached -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10735) Distcp using webhdfs on secure HA clusters fails with StandbyException
[ https://issues.apache.org/jira/browse/HDFS-10735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benoy Antony updated HDFS-10735: Attachment: exception.txt > Distcp using webhdfs on secure HA clusters fails with StandbyException > -- > > Key: HDFS-10735 > URL: https://issues.apache.org/jira/browse/HDFS-10735 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 2.7.2 >Reporter: Benoy Antony >Assignee: Benoy Antony > Fix For: 2.8.0 > > Attachments: exception.txt > > > A distcp running on HADOOP 2.7.2 cluster , copying from HADOOP 2.4.1 cluster > fails. > Both are secure clusters. > The logs indicates that task contacts the StandBy name node on the remote > cluster is contacted first and gets a standByException. > Instead of retrying with the active name node on the remote cluster, the > execution errors out throwing InvalidToken > The exception is attached -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-10735) Distcp using webhdfs on secure HA clusters fails with StandbyException
Benoy Antony created HDFS-10735: --- Summary: Distcp using webhdfs on secure HA clusters fails with StandbyException Key: HDFS-10735 URL: https://issues.apache.org/jira/browse/HDFS-10735 Project: Hadoop HDFS Issue Type: Bug Components: webhdfs Affects Versions: 2.7.2 Reporter: Benoy Antony Assignee: Benoy Antony Fix For: 2.8.0 A distcp running on HADOOP 2.7.2 cluster , copying from HADOOP 2.4.1 cluster fails. Both are secure clusters. The logs indicates that task contacts the StandBy name node on the remote cluster is contacted first and gets a standByException. Instead of retrying with the active name node on the remote cluster, the execution errors out throwing InvalidToken The exception is attached -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10477) Stop decommission a rack of DataNodes caused NameNode fail over to standby
[
https://issues.apache.org/jira/browse/HDFS-10477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15404918#comment-15404918
]
Benoy Antony commented on HDFS-10477:
-
Didn't see Arpit's previous comments. If the recommended approach is set to
dfs.namenode.fslock.fair to false, then it may be good to sleep for a
millisecond between lock release and re-acquisition.
The patch looks good to me , +1
> Stop decommission a rack of DataNodes caused NameNode fail over to standby
> --
>
> Key: HDFS-10477
> URL: https://issues.apache.org/jira/browse/HDFS-10477
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.7.2
>Reporter: yunjiong zhao
>Assignee: yunjiong zhao
> Attachments: HDFS-10477.002.patch, HDFS-10477.003.patch,
> HDFS-10477.004.patch, HDFS-10477.005.patch, HDFS-10477.patch
>
>
> In our cluster, when we stop decommissioning a rack which have 46 DataNodes,
> it locked Namesystem for about 7 minutes as below log shows:
> {code}
> 2016-05-26 20:11:41,697 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.27:1004
> 2016-05-26 20:11:51,171 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 285258 over-replicated blocks on 10.142.27.27:1004 during recommissioning
> 2016-05-26 20:11:51,171 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.118:1004
> 2016-05-26 20:11:59,972 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 279923 over-replicated blocks on 10.142.27.118:1004 during recommissioning
> 2016-05-26 20:11:59,972 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.113:1004
> 2016-05-26 20:12:09,007 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 294307 over-replicated blocks on 10.142.27.113:1004 during recommissioning
> 2016-05-26 20:12:09,008 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.117:1004
> 2016-05-26 20:12:18,055 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 314381 over-replicated blocks on 10.142.27.117:1004 during recommissioning
> 2016-05-26 20:12:18,056 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.130:1004
> 2016-05-26 20:12:25,938 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 272779 over-replicated blocks on 10.142.27.130:1004 during recommissioning
> 2016-05-26 20:12:25,939 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.121:1004
> 2016-05-26 20:12:34,134 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 287248 over-replicated blocks on 10.142.27.121:1004 during recommissioning
> 2016-05-26 20:12:34,134 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.33:1004
> 2016-05-26 20:12:43,020 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 299868 over-replicated blocks on 10.142.27.33:1004 during recommissioning
> 2016-05-26 20:12:43,020 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.137:1004
> 2016-05-26 20:12:52,220 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 303914 over-replicated blocks on 10.142.27.137:1004 during recommissioning
> 2016-05-26 20:12:52,220 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.51:1004
> 2016-05-26 20:13:00,362 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 281175 over-replicated blocks on 10.142.27.51:1004 during recommissioning
> 2016-05-26 20:13:00,362 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.12:1004
> 2016-05-26 20:13:08,756 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 274880 over-replicated blocks on 10.142.27.12:1004 during recommissioning
> 2016-05-26 20:13:08,757 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.15:1004
> 2016-05-26 20:13:17,185 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 286334 over-replicated blocks on 10.142.27.15:1004 during recommissioning
> 2016-05-26 20:13:17,185 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.14:1004
> 2016-05-26 20:13:25,369 INFO
> org.apache.hadoop.hdfs.server.blockma
[jira] [Commented] (HDFS-10477) Stop decommission a rack of DataNodes caused NameNode fail over to standby
[
https://issues.apache.org/jira/browse/HDFS-10477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15404912#comment-15404912
]
Benoy Antony commented on HDFS-10477:
-
The patch looks good.
There is no need to sleep as the admin can control the lock fairness via
dfs.namenode.fslock.fair. The default value is true which means that the
earlier threads will acquire the lock.
> Stop decommission a rack of DataNodes caused NameNode fail over to standby
> --
>
> Key: HDFS-10477
> URL: https://issues.apache.org/jira/browse/HDFS-10477
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.7.2
>Reporter: yunjiong zhao
>Assignee: yunjiong zhao
> Attachments: HDFS-10477.002.patch, HDFS-10477.003.patch,
> HDFS-10477.004.patch, HDFS-10477.005.patch, HDFS-10477.patch
>
>
> In our cluster, when we stop decommissioning a rack which have 46 DataNodes,
> it locked Namesystem for about 7 minutes as below log shows:
> {code}
> 2016-05-26 20:11:41,697 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.27:1004
> 2016-05-26 20:11:51,171 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 285258 over-replicated blocks on 10.142.27.27:1004 during recommissioning
> 2016-05-26 20:11:51,171 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.118:1004
> 2016-05-26 20:11:59,972 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 279923 over-replicated blocks on 10.142.27.118:1004 during recommissioning
> 2016-05-26 20:11:59,972 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.113:1004
> 2016-05-26 20:12:09,007 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 294307 over-replicated blocks on 10.142.27.113:1004 during recommissioning
> 2016-05-26 20:12:09,008 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.117:1004
> 2016-05-26 20:12:18,055 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 314381 over-replicated blocks on 10.142.27.117:1004 during recommissioning
> 2016-05-26 20:12:18,056 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.130:1004
> 2016-05-26 20:12:25,938 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 272779 over-replicated blocks on 10.142.27.130:1004 during recommissioning
> 2016-05-26 20:12:25,939 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.121:1004
> 2016-05-26 20:12:34,134 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 287248 over-replicated blocks on 10.142.27.121:1004 during recommissioning
> 2016-05-26 20:12:34,134 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.33:1004
> 2016-05-26 20:12:43,020 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 299868 over-replicated blocks on 10.142.27.33:1004 during recommissioning
> 2016-05-26 20:12:43,020 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.137:1004
> 2016-05-26 20:12:52,220 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 303914 over-replicated blocks on 10.142.27.137:1004 during recommissioning
> 2016-05-26 20:12:52,220 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.51:1004
> 2016-05-26 20:13:00,362 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 281175 over-replicated blocks on 10.142.27.51:1004 during recommissioning
> 2016-05-26 20:13:00,362 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.12:1004
> 2016-05-26 20:13:08,756 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 274880 over-replicated blocks on 10.142.27.12:1004 during recommissioning
> 2016-05-26 20:13:08,757 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.15:1004
> 2016-05-26 20:13:17,185 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 286334 over-replicated blocks on 10.142.27.15:1004 during recommissioning
> 2016-05-26 20:13:17,185 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.14:1004
> 2016-05-26 20:13:25,369 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: In
[jira] [Commented] (HDFS-10477) Stop decommission a rack of DataNodes caused NameNode fail over to standby
[
https://issues.apache.org/jira/browse/HDFS-10477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15364835#comment-15364835
]
Benoy Antony commented on HDFS-10477:
-
Thanks for the info, Akira.
> Stop decommission a rack of DataNodes caused NameNode fail over to standby
> --
>
> Key: HDFS-10477
> URL: https://issues.apache.org/jira/browse/HDFS-10477
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.7.2
>Reporter: yunjiong zhao
>Assignee: yunjiong zhao
> Attachments: HDFS-10477.002.patch, HDFS-10477.003.patch,
> HDFS-10477.patch
>
>
> In our cluster, when we stop decommissioning a rack which have 46 DataNodes,
> it locked Namesystem for about 7 minutes as below log shows:
> {code}
> 2016-05-26 20:11:41,697 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.27:1004
> 2016-05-26 20:11:51,171 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 285258 over-replicated blocks on 10.142.27.27:1004 during recommissioning
> 2016-05-26 20:11:51,171 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.118:1004
> 2016-05-26 20:11:59,972 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 279923 over-replicated blocks on 10.142.27.118:1004 during recommissioning
> 2016-05-26 20:11:59,972 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.113:1004
> 2016-05-26 20:12:09,007 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 294307 over-replicated blocks on 10.142.27.113:1004 during recommissioning
> 2016-05-26 20:12:09,008 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.117:1004
> 2016-05-26 20:12:18,055 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 314381 over-replicated blocks on 10.142.27.117:1004 during recommissioning
> 2016-05-26 20:12:18,056 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.130:1004
> 2016-05-26 20:12:25,938 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 272779 over-replicated blocks on 10.142.27.130:1004 during recommissioning
> 2016-05-26 20:12:25,939 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.121:1004
> 2016-05-26 20:12:34,134 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 287248 over-replicated blocks on 10.142.27.121:1004 during recommissioning
> 2016-05-26 20:12:34,134 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.33:1004
> 2016-05-26 20:12:43,020 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 299868 over-replicated blocks on 10.142.27.33:1004 during recommissioning
> 2016-05-26 20:12:43,020 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.137:1004
> 2016-05-26 20:12:52,220 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 303914 over-replicated blocks on 10.142.27.137:1004 during recommissioning
> 2016-05-26 20:12:52,220 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.51:1004
> 2016-05-26 20:13:00,362 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 281175 over-replicated blocks on 10.142.27.51:1004 during recommissioning
> 2016-05-26 20:13:00,362 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.12:1004
> 2016-05-26 20:13:08,756 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 274880 over-replicated blocks on 10.142.27.12:1004 during recommissioning
> 2016-05-26 20:13:08,757 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.15:1004
> 2016-05-26 20:13:17,185 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 286334 over-replicated blocks on 10.142.27.15:1004 during recommissioning
> 2016-05-26 20:13:17,185 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.14:1004
> 2016-05-26 20:13:25,369 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 280219 over-replicated blocks on 10.142.27.14:1004 during recommissioning
> 2016-05-26 20:13:25,370 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.28:1004
>
[jira] [Commented] (HDFS-10477) Stop decommission a rack of DataNodes caused NameNode fail over to standby
[
https://issues.apache.org/jira/browse/HDFS-10477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15364805#comment-15364805
]
Benoy Antony commented on HDFS-10477:
-
Looks good. Could you please re-trigger the build to make sure the tests pass ?
> Stop decommission a rack of DataNodes caused NameNode fail over to standby
> --
>
> Key: HDFS-10477
> URL: https://issues.apache.org/jira/browse/HDFS-10477
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.7.2
>Reporter: yunjiong zhao
>Assignee: yunjiong zhao
> Attachments: HDFS-10477.002.patch, HDFS-10477.003.patch,
> HDFS-10477.patch
>
>
> In our cluster, when we stop decommissioning a rack which have 46 DataNodes,
> it locked Namesystem for about 7 minutes as below log shows:
> {code}
> 2016-05-26 20:11:41,697 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.27:1004
> 2016-05-26 20:11:51,171 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 285258 over-replicated blocks on 10.142.27.27:1004 during recommissioning
> 2016-05-26 20:11:51,171 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.118:1004
> 2016-05-26 20:11:59,972 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 279923 over-replicated blocks on 10.142.27.118:1004 during recommissioning
> 2016-05-26 20:11:59,972 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.113:1004
> 2016-05-26 20:12:09,007 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 294307 over-replicated blocks on 10.142.27.113:1004 during recommissioning
> 2016-05-26 20:12:09,008 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.117:1004
> 2016-05-26 20:12:18,055 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 314381 over-replicated blocks on 10.142.27.117:1004 during recommissioning
> 2016-05-26 20:12:18,056 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.130:1004
> 2016-05-26 20:12:25,938 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 272779 over-replicated blocks on 10.142.27.130:1004 during recommissioning
> 2016-05-26 20:12:25,939 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.121:1004
> 2016-05-26 20:12:34,134 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 287248 over-replicated blocks on 10.142.27.121:1004 during recommissioning
> 2016-05-26 20:12:34,134 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.33:1004
> 2016-05-26 20:12:43,020 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 299868 over-replicated blocks on 10.142.27.33:1004 during recommissioning
> 2016-05-26 20:12:43,020 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.137:1004
> 2016-05-26 20:12:52,220 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 303914 over-replicated blocks on 10.142.27.137:1004 during recommissioning
> 2016-05-26 20:12:52,220 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.51:1004
> 2016-05-26 20:13:00,362 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 281175 over-replicated blocks on 10.142.27.51:1004 during recommissioning
> 2016-05-26 20:13:00,362 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.12:1004
> 2016-05-26 20:13:08,756 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 274880 over-replicated blocks on 10.142.27.12:1004 during recommissioning
> 2016-05-26 20:13:08,757 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.15:1004
> 2016-05-26 20:13:17,185 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 286334 over-replicated blocks on 10.142.27.15:1004 during recommissioning
> 2016-05-26 20:13:17,185 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.14:1004
> 2016-05-26 20:13:25,369 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 280219 over-replicated blocks on 10.142.27.14:1004 during recommissioning
> 2016-05-26 20:13:25,370 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.Datanod
[jira] [Commented] (HDFS-10477) Stop decommission a rack of DataNodes caused NameNode fail over to standby
[
https://issues.apache.org/jira/browse/HDFS-10477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15325033#comment-15325033
]
Benoy Antony commented on HDFS-10477:
-
[~kihwal], Your comments regarding starvation makes sense.
[~zhaoyunjiong], I think its a good idea to combine these two patches.
> Stop decommission a rack of DataNodes caused NameNode fail over to standby
> --
>
> Key: HDFS-10477
> URL: https://issues.apache.org/jira/browse/HDFS-10477
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.7.2
>Reporter: yunjiong zhao
>Assignee: yunjiong zhao
> Attachments: HDFS-10477.002.patch, HDFS-10477.patch
>
>
> In our cluster, when we stop decommissioning a rack which have 46 DataNodes,
> it locked Namesystem for about 7 minutes as below log shows:
> {code}
> 2016-05-26 20:11:41,697 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.27:1004
> 2016-05-26 20:11:51,171 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 285258 over-replicated blocks on 10.142.27.27:1004 during recommissioning
> 2016-05-26 20:11:51,171 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.118:1004
> 2016-05-26 20:11:59,972 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 279923 over-replicated blocks on 10.142.27.118:1004 during recommissioning
> 2016-05-26 20:11:59,972 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.113:1004
> 2016-05-26 20:12:09,007 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 294307 over-replicated blocks on 10.142.27.113:1004 during recommissioning
> 2016-05-26 20:12:09,008 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.117:1004
> 2016-05-26 20:12:18,055 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 314381 over-replicated blocks on 10.142.27.117:1004 during recommissioning
> 2016-05-26 20:12:18,056 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.130:1004
> 2016-05-26 20:12:25,938 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 272779 over-replicated blocks on 10.142.27.130:1004 during recommissioning
> 2016-05-26 20:12:25,939 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.121:1004
> 2016-05-26 20:12:34,134 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 287248 over-replicated blocks on 10.142.27.121:1004 during recommissioning
> 2016-05-26 20:12:34,134 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.33:1004
> 2016-05-26 20:12:43,020 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 299868 over-replicated blocks on 10.142.27.33:1004 during recommissioning
> 2016-05-26 20:12:43,020 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.137:1004
> 2016-05-26 20:12:52,220 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 303914 over-replicated blocks on 10.142.27.137:1004 during recommissioning
> 2016-05-26 20:12:52,220 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.51:1004
> 2016-05-26 20:13:00,362 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 281175 over-replicated blocks on 10.142.27.51:1004 during recommissioning
> 2016-05-26 20:13:00,362 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.12:1004
> 2016-05-26 20:13:08,756 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 274880 over-replicated blocks on 10.142.27.12:1004 during recommissioning
> 2016-05-26 20:13:08,757 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.15:1004
> 2016-05-26 20:13:17,185 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 286334 over-replicated blocks on 10.142.27.15:1004 during recommissioning
> 2016-05-26 20:13:17,185 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.14:1004
> 2016-05-26 20:13:25,369 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 280219 over-replicated blocks on 10.142.27.14:1004 during recommissioning
> 2016-05-26 20:13:25,370 INFO
> org.apache.hadoop.hd
[jira] [Commented] (HDFS-10477) Stop decommission a rack of DataNodes caused NameNode fail over to standby
[
https://issues.apache.org/jira/browse/HDFS-10477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15313282#comment-15313282
]
Benoy Antony commented on HDFS-10477:
-
If its possible to release lock per storage, then that's better.
If not , I prefer the first version which does releases the lock per each
datanode without the additional processing.
The logs show that the each node is processed in around 10 seconds.
> Stop decommission a rack of DataNodes caused NameNode fail over to standby
> --
>
> Key: HDFS-10477
> URL: https://issues.apache.org/jira/browse/HDFS-10477
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.7.2
>Reporter: yunjiong zhao
>Assignee: yunjiong zhao
> Attachments: HDFS-10477.002.patch, HDFS-10477.patch
>
>
> In our cluster, when we stop decommissioning a rack which have 46 DataNodes,
> it locked Namesystem for about 7 minutes as below log shows:
> {code}
> 2016-05-26 20:11:41,697 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.27:1004
> 2016-05-26 20:11:51,171 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 285258 over-replicated blocks on 10.142.27.27:1004 during recommissioning
> 2016-05-26 20:11:51,171 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.118:1004
> 2016-05-26 20:11:59,972 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 279923 over-replicated blocks on 10.142.27.118:1004 during recommissioning
> 2016-05-26 20:11:59,972 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.113:1004
> 2016-05-26 20:12:09,007 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 294307 over-replicated blocks on 10.142.27.113:1004 during recommissioning
> 2016-05-26 20:12:09,008 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.117:1004
> 2016-05-26 20:12:18,055 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 314381 over-replicated blocks on 10.142.27.117:1004 during recommissioning
> 2016-05-26 20:12:18,056 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.130:1004
> 2016-05-26 20:12:25,938 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 272779 over-replicated blocks on 10.142.27.130:1004 during recommissioning
> 2016-05-26 20:12:25,939 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.121:1004
> 2016-05-26 20:12:34,134 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 287248 over-replicated blocks on 10.142.27.121:1004 during recommissioning
> 2016-05-26 20:12:34,134 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.33:1004
> 2016-05-26 20:12:43,020 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 299868 over-replicated blocks on 10.142.27.33:1004 during recommissioning
> 2016-05-26 20:12:43,020 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.137:1004
> 2016-05-26 20:12:52,220 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 303914 over-replicated blocks on 10.142.27.137:1004 during recommissioning
> 2016-05-26 20:12:52,220 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.51:1004
> 2016-05-26 20:13:00,362 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 281175 over-replicated blocks on 10.142.27.51:1004 during recommissioning
> 2016-05-26 20:13:00,362 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.12:1004
> 2016-05-26 20:13:08,756 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 274880 over-replicated blocks on 10.142.27.12:1004 during recommissioning
> 2016-05-26 20:13:08,757 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.15:1004
> 2016-05-26 20:13:17,185 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 286334 over-replicated blocks on 10.142.27.15:1004 during recommissioning
> 2016-05-26 20:13:17,185 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.14:1004
> 2016-05-26 20:13:25,369 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 280219 over
[jira] [Comment Edited] (HDFS-10477) Stop decommission a rack of DataNodes caused NameNode fail over to standby
[
https://issues.apache.org/jira/browse/HDFS-10477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15313282#comment-15313282
]
Benoy Antony edited comment on HDFS-10477 at 6/2/16 11:21 PM:
--
If its possible to release lock per storage, then that's better.
If not , I prefer the first version which releases the lock per each datanode
without the additional processing.
The logs show that the each node is processed in around 10 seconds.
was (Author: benoyantony):
If its possible to release lock per storage, then that's better.
If not , I prefer the first version which does releases the lock per each
datanode without the additional processing.
The logs show that the each node is processed in around 10 seconds.
> Stop decommission a rack of DataNodes caused NameNode fail over to standby
> --
>
> Key: HDFS-10477
> URL: https://issues.apache.org/jira/browse/HDFS-10477
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.7.2
>Reporter: yunjiong zhao
>Assignee: yunjiong zhao
> Attachments: HDFS-10477.002.patch, HDFS-10477.patch
>
>
> In our cluster, when we stop decommissioning a rack which have 46 DataNodes,
> it locked Namesystem for about 7 minutes as below log shows:
> {code}
> 2016-05-26 20:11:41,697 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.27:1004
> 2016-05-26 20:11:51,171 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 285258 over-replicated blocks on 10.142.27.27:1004 during recommissioning
> 2016-05-26 20:11:51,171 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.118:1004
> 2016-05-26 20:11:59,972 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 279923 over-replicated blocks on 10.142.27.118:1004 during recommissioning
> 2016-05-26 20:11:59,972 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.113:1004
> 2016-05-26 20:12:09,007 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 294307 over-replicated blocks on 10.142.27.113:1004 during recommissioning
> 2016-05-26 20:12:09,008 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.117:1004
> 2016-05-26 20:12:18,055 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 314381 over-replicated blocks on 10.142.27.117:1004 during recommissioning
> 2016-05-26 20:12:18,056 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.130:1004
> 2016-05-26 20:12:25,938 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 272779 over-replicated blocks on 10.142.27.130:1004 during recommissioning
> 2016-05-26 20:12:25,939 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.121:1004
> 2016-05-26 20:12:34,134 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 287248 over-replicated blocks on 10.142.27.121:1004 during recommissioning
> 2016-05-26 20:12:34,134 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.33:1004
> 2016-05-26 20:12:43,020 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 299868 over-replicated blocks on 10.142.27.33:1004 during recommissioning
> 2016-05-26 20:12:43,020 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.137:1004
> 2016-05-26 20:12:52,220 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 303914 over-replicated blocks on 10.142.27.137:1004 during recommissioning
> 2016-05-26 20:12:52,220 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.51:1004
> 2016-05-26 20:13:00,362 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 281175 over-replicated blocks on 10.142.27.51:1004 during recommissioning
> 2016-05-26 20:13:00,362 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.12:1004
> 2016-05-26 20:13:08,756 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 274880 over-replicated blocks on 10.142.27.12:1004 during recommissioning
> 2016-05-26 20:13:08,757 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.15:1004
> 2016-05-26 20:13:17,185 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 2863
[jira] [Commented] (HDFS-9597) BaseReplicationPolicyTest should update data node stats after adding a data node
[ https://issues.apache.org/jira/browse/HDFS-9597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15099063#comment-15099063 ] Benoy Antony commented on HDFS-9597: committed to branch-2.8 also. > BaseReplicationPolicyTest should update data node stats after adding a data > node > > > Key: HDFS-9597 > URL: https://issues.apache.org/jira/browse/HDFS-9597 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 3.0.0, 2.8.0 >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Blocker > Fix For: 2.8.0 > > Attachments: HDFS-9597.001.patch > > > Looks like HDFS-9034 broke > TestReplicationPolicyConsiderLoad#testChooseTargetWithDecomNodes. > This test has been failing since yesterday. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9034) "StorageTypeStats" Metric should not count failed storage.
[ https://issues.apache.org/jira/browse/HDFS-9034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15099064#comment-15099064 ] Benoy Antony commented on HDFS-9034: committed to branch-2.8. > "StorageTypeStats" Metric should not count failed storage. > -- > > Key: HDFS-9034 > URL: https://issues.apache.org/jira/browse/HDFS-9034 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Reporter: Archana T >Assignee: Surendra Singh Lilhore > Attachments: HDFS-9034.01.patch, HDFS-9034.02.patch, > HDFS-9034.03.patch, HDFS-9034.04.patch, dfsStorage_NN_UI2.png > > > When we remove one storage type from all the DNs, still NN UI shows entry of > those storage type -- > Ex:for ARCHIVE > Steps-- > 1. ARCHIVE Storage type was added for all DNs > 2. Stop DNs > 3. Removed ARCHIVE Storages from all DNs > 4. Restarted DNs > NN UI shows below -- > DFS Storage Types > Storage Type Configured Capacity Capacity Used Capacity Remaining > ARCHIVE 57.18 GB64 KB (0%) 39.82 GB (69.64%) 64 KB > 1 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-9597) BaseReplicationPolicyTest should update data node stats after adding a data node
[ https://issues.apache.org/jira/browse/HDFS-9597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benoy Antony updated HDFS-9597: --- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 2.8.0 Status: Resolved (was: Patch Available) Committed in trunk and branch-2. Thanks [~jojochuang] for the fix and [~yangzhe1991] for the review. > BaseReplicationPolicyTest should update data node stats after adding a data > node > > > Key: HDFS-9597 > URL: https://issues.apache.org/jira/browse/HDFS-9597 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 3.0.0, 2.8.0 >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang > Fix For: 2.8.0 > > Attachments: HDFS-9597.001.patch > > > Looks like HDFS-9034 broke > TestReplicationPolicyConsiderLoad#testChooseTargetWithDecomNodes. > This test has been failing since yesterday. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-9597) BaseReplicationPolicyTest should update data node stats after adding a data node
[ https://issues.apache.org/jira/browse/HDFS-9597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benoy Antony updated HDFS-9597: --- Priority: Blocker (was: Major) > BaseReplicationPolicyTest should update data node stats after adding a data > node > > > Key: HDFS-9597 > URL: https://issues.apache.org/jira/browse/HDFS-9597 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 3.0.0, 2.8.0 >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Blocker > Fix For: 2.8.0 > > Attachments: HDFS-9597.001.patch > > > Looks like HDFS-9034 broke > TestReplicationPolicyConsiderLoad#testChooseTargetWithDecomNodes. > This test has been failing since yesterday. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9597) BaseReplicationPolicyTest should update data node stats after adding a data node
[ https://issues.apache.org/jira/browse/HDFS-9597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15071664#comment-15071664 ] Benoy Antony commented on HDFS-9597: +1. The fix looks good to me. Thanks for taking care of it. Will commit the patch after the tests. > BaseReplicationPolicyTest should update data node stats after adding a data > node > > > Key: HDFS-9597 > URL: https://issues.apache.org/jira/browse/HDFS-9597 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 3.0.0, 2.8.0 >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang > Attachments: HDFS-9597.001.patch > > > Looks like HDFS-9034 broke > TestReplicationPolicyConsiderLoad#testChooseTargetWithDecomNodes. > This test has been failing since yesterday. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-9034) "StorageTypeStats" Metric should not count failed storage.
[ https://issues.apache.org/jira/browse/HDFS-9034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benoy Antony updated HDFS-9034: --- Resolution: Fixed Hadoop Flags: Reviewed Target Version/s: 2.8.0 Status: Resolved (was: Patch Available) Committed in trunk and branch-2. Thanks [~surendrasingh] for the contribution. Thanks [~archanat] for reporting the issue. > "StorageTypeStats" Metric should not count failed storage. > -- > > Key: HDFS-9034 > URL: https://issues.apache.org/jira/browse/HDFS-9034 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Reporter: Archana T >Assignee: Surendra Singh Lilhore > Attachments: HDFS-9034.01.patch, HDFS-9034.02.patch, > HDFS-9034.03.patch, HDFS-9034.04.patch, dfsStorage_NN_UI2.png > > > When we remove one storage type from all the DNs, still NN UI shows entry of > those storage type -- > Ex:for ARCHIVE > Steps-- > 1. ARCHIVE Storage type was added for all DNs > 2. Stop DNs > 3. Removed ARCHIVE Storages from all DNs > 4. Restarted DNs > NN UI shows below -- > DFS Storage Types > Storage Type Configured Capacity Capacity Used Capacity Remaining > ARCHIVE 57.18 GB64 KB (0%) 39.82 GB (69.64%) 64 KB > 1 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9034) "StorageTypeStats" Metric should not count failed storage.
[ https://issues.apache.org/jira/browse/HDFS-9034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15066868#comment-15066868 ] Benoy Antony commented on HDFS-9034: Thanks for the clarification, Surendra. The patch looks good . +1. If there are no other comments, I'll commit this, tomorrow. > "StorageTypeStats" Metric should not count failed storage. > -- > > Key: HDFS-9034 > URL: https://issues.apache.org/jira/browse/HDFS-9034 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Reporter: Archana T >Assignee: Surendra Singh Lilhore > Attachments: HDFS-9034.01.patch, HDFS-9034.02.patch, > HDFS-9034.03.patch, HDFS-9034.04.patch, dfsStorage_NN_UI2.png > > > When we remove one storage type from all the DNs, still NN UI shows entry of > those storage type -- > Ex:for ARCHIVE > Steps-- > 1. ARCHIVE Storage type was added for all DNs > 2. Stop DNs > 3. Removed ARCHIVE Storages from all DNs > 4. Restarted DNs > NN UI shows below -- > DFS Storage Types > Storage Type Configured Capacity Capacity Used Capacity Remaining > ARCHIVE 57.18 GB64 KB (0%) 39.82 GB (69.64%) 64 KB > 1 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9034) "StorageTypeStats" Metric should not count failed storage.
[ https://issues.apache.org/jira/browse/HDFS-9034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15064719#comment-15064719 ] Benoy Antony commented on HDFS-9034: Let say, node registers with 10 Storages of 2 TB. Assume none of these failed. Assume 9 are DISK and 1 is ARCHIVE stats.add(d) is called via updateDnStat . StorageTypes will be recorded as DISK and ARCHIVE. For the next hearBeat , the single ARCHIVE is reported as FAILED. stats.subtract(node) will be called, but it will not remove the FAILED storage. StorageTypes will remain as DISK and ARCHIVE. stats.add(node) will be called, but will skip the FAILED storage. StorageTypes will remain as DISK and ARCHIVE. Wouldn't this happen ? > "StorageTypeStats" Metric should not count failed storage. > -- > > Key: HDFS-9034 > URL: https://issues.apache.org/jira/browse/HDFS-9034 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Reporter: Archana T >Assignee: Surendra Singh Lilhore > Attachments: HDFS-9034.01.patch, HDFS-9034.02.patch, > HDFS-9034.03.patch, HDFS-9034.04.patch, dfsStorage_NN_UI2.png > > > When we remove one storage type from all the DNs, still NN UI shows entry of > those storage type -- > Ex:for ARCHIVE > Steps-- > 1. ARCHIVE Storage type was added for all DNs > 2. Stop DNs > 3. Removed ARCHIVE Storages from all DNs > 4. Restarted DNs > NN UI shows below -- > DFS Storage Types > Storage Type Configured Capacity Capacity Used Capacity Remaining > ARCHIVE 57.18 GB64 KB (0%) 39.82 GB (69.64%) 64 KB > 1 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9034) "StorageTypeStats" Metric should not count failed storage.
[ https://issues.apache.org/jira/browse/HDFS-9034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15063140#comment-15063140 ] Benoy Antony commented on HDFS-9034: Reviewed the patch. Sorry for the delay. Patch looks good. I have following comment: 1. In DatanodeStats.substract, we subtract the storage capacity only if it is NOT FAILED. But suppose it was good when we added it, then not subtracting will be a mistake. But if it was bad during add, we should not subtract it. So we need some additional information to correctly determine whether we should subtract it or not. We also need test cases to make sure that these cases are handled correctly. > "StorageTypeStats" Metric should not count failed storage. > -- > > Key: HDFS-9034 > URL: https://issues.apache.org/jira/browse/HDFS-9034 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Reporter: Archana T >Assignee: Surendra Singh Lilhore > Attachments: HDFS-9034.01.patch, HDFS-9034.02.patch, > HDFS-9034.03.patch, HDFS-9034.04.patch, dfsStorage_NN_UI2.png > > > When we remove one storage type from all the DNs, still NN UI shows entry of > those storage type -- > Ex:for ARCHIVE > Steps-- > 1. ARCHIVE Storage type was added for all DNs > 2. Stop DNs > 3. Removed ARCHIVE Storages from all DNs > 4. Restarted DNs > NN UI shows below -- > DFS Storage Types > Storage Type Configured Capacity Capacity Used Capacity Remaining > ARCHIVE 57.18 GB64 KB (0%) 39.82 GB (69.64%) 64 KB > 1 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9034) "StorageTypeStats" Metric should not count failed storage.
[ https://issues.apache.org/jira/browse/HDFS-9034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15049781#comment-15049781 ] Benoy Antony commented on HDFS-9034: I'll review this tomorrow. > "StorageTypeStats" Metric should not count failed storage. > -- > > Key: HDFS-9034 > URL: https://issues.apache.org/jira/browse/HDFS-9034 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Reporter: Archana T >Assignee: Surendra Singh Lilhore > Attachments: HDFS-9034.01.patch, HDFS-9034.02.patch, > HDFS-9034.03.patch, HDFS-9034.04.patch, dfsStorage_NN_UI2.png > > > When we remove one storage type from all the DNs, still NN UI shows entry of > those storage type -- > Ex:for ARCHIVE > Steps-- > 1. ARCHIVE Storage type was added for all DNs > 2. Stop DNs > 3. Removed ARCHIVE Storages from all DNs > 4. Restarted DNs > NN UI shows below -- > DFS Storage Types > Storage Type Configured Capacity Capacity Used Capacity Remaining > ARCHIVE 57.18 GB64 KB (0%) 39.82 GB (69.64%) 64 KB > 1 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-6363) Improve concurrency while checking inclusion and exclusion of datanodes
[ https://issues.apache.org/jira/browse/HDFS-6363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benoy Antony updated HDFS-6363: --- Attachment: HDFS-6363-004.patch > Improve concurrency while checking inclusion and exclusion of datanodes > --- > > Key: HDFS-6363 > URL: https://issues.apache.org/jira/browse/HDFS-6363 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Benoy Antony >Assignee: Benoy Antony > Labels: BB2015-05-TBR > Attachments: HDFS-6363-002.patch, HDFS-6363-003.patch, > HDFS-6363-004.patch, HDFS-6363.patch > > > HostFileManager holds two effectively immutable objects - includes and > excludes. These two objects can be safely published together using a volatile > container instead of synchronizing for all mutators and accessors. > This improves the concurrency while using HostFileManager. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-6363) Improve concurrency while checking inclusion and exclusion of datanodes
[ https://issues.apache.org/jira/browse/HDFS-6363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benoy Antony updated HDFS-6363: --- Attachment: HDFS-6363-003.patch Attaching the patch after fixing checkstyle issues. > Improve concurrency while checking inclusion and exclusion of datanodes > --- > > Key: HDFS-6363 > URL: https://issues.apache.org/jira/browse/HDFS-6363 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Benoy Antony >Assignee: Benoy Antony > Labels: BB2015-05-TBR > Attachments: HDFS-6363-002.patch, HDFS-6363-003.patch, HDFS-6363.patch > > > HostFileManager holds two effectively immutable objects - includes and > excludes. These two objects can be safely published together using a volatile > container instead of synchronizing for all mutators and accessors. > This improves the concurrency while using HostFileManager. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-6363) Improve concurrency while checking inclusion and exclusion of datanodes
[ https://issues.apache.org/jira/browse/HDFS-6363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benoy Antony updated HDFS-6363: --- Attachment: HDFS-6363-002.patch Thanks for the review , [~wheat9]. Attaching a patch using _AtomicReference_. > Improve concurrency while checking inclusion and exclusion of datanodes > --- > > Key: HDFS-6363 > URL: https://issues.apache.org/jira/browse/HDFS-6363 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Benoy Antony >Assignee: Benoy Antony > Labels: BB2015-05-TBR > Attachments: HDFS-6363-002.patch, HDFS-6363.patch > > > HostFileManager holds two effectively immutable objects - includes and > excludes. These two objects can be safely published together using a volatile > container instead of synchronizing for all mutators and accessors. > This improves the concurrency while using HostFileManager. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9034) "StorageTypeStats" Metric should not count failed storage.
[ https://issues.apache.org/jira/browse/HDFS-9034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15027483#comment-15027483 ] Benoy Antony commented on HDFS-9034: Patch generally looks good. I cannot apply it to the current code base. Could you please rebase the patch ? > "StorageTypeStats" Metric should not count failed storage. > -- > > Key: HDFS-9034 > URL: https://issues.apache.org/jira/browse/HDFS-9034 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Reporter: Archana T >Assignee: Surendra Singh Lilhore > Attachments: HDFS-9034.01.patch, HDFS-9034.02.patch, > dfsStorage_NN_UI2.png > > > When we remove one storage type from all the DNs, still NN UI shows entry of > those storage type -- > Ex:for ARCHIVE > Steps-- > 1. ARCHIVE Storage type was added for all DNs > 2. Stop DNs > 3. Removed ARCHIVE Storages from all DNs > 4. Restarted DNs > NN UI shows below -- > DFS Storage Types > Storage Type Configured Capacity Capacity Used Capacity Remaining > ARCHIVE 57.18 GB64 KB (0%) 39.82 GB (69.64%) 64 KB > 1 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9034) "StorageTypeStats" Metric should not count failed storage.
[ https://issues.apache.org/jira/browse/HDFS-9034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15026044#comment-15026044 ] Benoy Antony commented on HDFS-9034: Sure, I will review tomorrow. > "StorageTypeStats" Metric should not count failed storage. > -- > > Key: HDFS-9034 > URL: https://issues.apache.org/jira/browse/HDFS-9034 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Reporter: Archana T >Assignee: Surendra Singh Lilhore > Attachments: HDFS-9034.01.patch, HDFS-9034.02.patch, > dfsStorage_NN_UI2.png > > > When we remove one storage type from all the DNs, still NN UI shows entry of > those storage type -- > Ex:for ARCHIVE > Steps-- > 1. ARCHIVE Storage type was added for all DNs > 2. Stop DNs > 3. Removed ARCHIVE Storages from all DNs > 4. Restarted DNs > NN UI shows below -- > DFS Storage Types > Storage Type Configured Capacity Capacity Used Capacity Remaining > ARCHIVE 57.18 GB64 KB (0%) 39.82 GB (69.64%) 64 KB > 1 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6407) new namenode UI, lost ability to sort columns in datanode tab
[ https://issues.apache.org/jira/browse/HDFS-6407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14699893#comment-14699893 ] Benoy Antony commented on HDFS-6407: It will be good to specify the version information of the datatables component. This will help in maintaining this functionality. For other Js components, the version information is included in the file name. > new namenode UI, lost ability to sort columns in datanode tab > - > > Key: HDFS-6407 > URL: https://issues.apache.org/jira/browse/HDFS-6407 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.4.0 >Reporter: Nathan Roberts >Assignee: Haohui Mai >Priority: Critical > Labels: BB2015-05-TBR > Attachments: 002-datanodes-sorted-capacityUsed.png, > 002-datanodes.png, 002-filebrowser.png, 002-snapshots.png, > HDFS-6407-002.patch, HDFS-6407-003.patch, HDFS-6407.008.patch, > HDFS-6407.009.patch, HDFS-6407.010.patch, HDFS-6407.011.patch, > HDFS-6407.4.patch, HDFS-6407.5.patch, HDFS-6407.6.patch, HDFS-6407.7.patch, > HDFS-6407.patch, browse_directory.png, datanodes.png, snapshots.png, sorting > 2.png, sorting table.png > > > old ui supported clicking on column header to sort on that column. The new ui > seems to have dropped this very useful feature. > There are a few tables in the Namenode UI to display datanodes information, > directory listings and snapshots. > When there are many items in the tables, it is useful to have ability to sort > on the different columns. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7483) Display information per tier on the Namenode UI
[ https://issues.apache.org/jira/browse/HDFS-7483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14634243#comment-14634243 ] Benoy Antony commented on HDFS-7483: Nice. Thanks for the snippet. I like this approach. +1 pending jenkins. > Display information per tier on the Namenode UI > --- > > Key: HDFS-7483 > URL: https://issues.apache.org/jira/browse/HDFS-7483 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Benoy Antony >Assignee: Benoy Antony > Attachments: HDFS-7483-001.patch, HDFS-7483-002.patch, > HDFS-7483.003.patch, overview.png, storagetypes.png, > storagetypes_withnostorage.png, withOneStorageType.png, withTwoStorageType.png > > > If cluster has different types of storage, it is useful to display the > storage information per type. > The information will be available via JMX (HDFS-7390) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7483) Display information per tier on the Namenode UI
[ https://issues.apache.org/jira/browse/HDFS-7483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14630563#comment-14630563 ] Benoy Antony commented on HDFS-7483: In the existing use cases of fmt_percentage, the percentage is already generated on the server side. I do not understand the approach that you are suggesting. Could you please provide me with a working snippet ? > Display information per tier on the Namenode UI > --- > > Key: HDFS-7483 > URL: https://issues.apache.org/jira/browse/HDFS-7483 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Benoy Antony >Assignee: Benoy Antony > Attachments: HDFS-7483-001.patch, HDFS-7483-002.patch, overview.png, > storagetypes.png, storagetypes_withnostorage.png, withOneStorageType.png, > withTwoStorageType.png > > > If cluster has different types of storage, it is useful to display the > storage information per type. > The information will be available via JMX (HDFS-7390) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7483) Display information per tier on the Namenode UI
[ https://issues.apache.org/jira/browse/HDFS-7483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14629045#comment-14629045 ] Benoy Antony commented on HDFS-7483: [~wheat9], As I mentioned, there is no good way of displaying percentage using math helper and fmt_percentage filter. If you have no further comments, I'll commit this patch by end of day tomorrow. > Display information per tier on the Namenode UI > --- > > Key: HDFS-7483 > URL: https://issues.apache.org/jira/browse/HDFS-7483 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Benoy Antony >Assignee: Benoy Antony > Attachments: HDFS-7483-001.patch, HDFS-7483-002.patch, overview.png, > storagetypes.png, storagetypes_withnostorage.png, withOneStorageType.png, > withTwoStorageType.png > > > If cluster has different types of storage, it is useful to display the > storage information per type. > The information will be available via JMX (HDFS-7390) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7483) Display information per tier on the Namenode UI
[ https://issues.apache.org/jira/browse/HDFS-7483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14628406#comment-14628406 ] Benoy Antony commented on HDFS-7483: To add on, I had tried the approach while working on the patch. _math_ is a helper whereas fmt_percentage is a filter. We cannot do something like "helper | filter" . Some helpers support a "filters" attribute. But math helper does not support "filters" attribute. So I could not reuse math helper and fmt_percentage filter. That's why I wrote a a new percentage helper. > Display information per tier on the Namenode UI > --- > > Key: HDFS-7483 > URL: https://issues.apache.org/jira/browse/HDFS-7483 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Benoy Antony >Assignee: Benoy Antony > Attachments: HDFS-7483-001.patch, HDFS-7483-002.patch, overview.png, > storagetypes.png, storagetypes_withnostorage.png, withOneStorageType.png, > withTwoStorageType.png > > > If cluster has different types of storage, it is useful to display the > storage information per type. > The information will be available via JMX (HDFS-7390) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7483) Display information per tier on the Namenode UI
[ https://issues.apache.org/jira/browse/HDFS-7483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14627342#comment-14627342 ] Benoy Antony commented on HDFS-7483: I don't think the suggested approach will work. But I could be wrong. [~wheat9], Could you please provide a working code snippet for this approach ? > Display information per tier on the Namenode UI > --- > > Key: HDFS-7483 > URL: https://issues.apache.org/jira/browse/HDFS-7483 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Benoy Antony >Assignee: Benoy Antony > Attachments: HDFS-7483-001.patch, HDFS-7483-002.patch, overview.png, > storagetypes.png, storagetypes_withnostorage.png, withOneStorageType.png, > withTwoStorageType.png > > > If cluster has different types of storage, it is useful to display the > storage information per type. > The information will be available via JMX (HDFS-7390) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7483) Display information per tier on the Namenode UI
[ https://issues.apache.org/jira/browse/HDFS-7483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benoy Antony updated HDFS-7483: --- Attachment: HDFS-7483-002.patch > Display information per tier on the Namenode UI > --- > > Key: HDFS-7483 > URL: https://issues.apache.org/jira/browse/HDFS-7483 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Benoy Antony >Assignee: Benoy Antony > Attachments: HDFS-7483-001.patch, HDFS-7483-002.patch, overview.png, > storagetypes.png, storagetypes_withnostorage.png, withOneStorageType.png, > withTwoStorageType.png > > > If cluster has different types of storage, it is useful to display the > storage information per type. > The information will be available via JMX (HDFS-7390) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7483) Display information per tier on the Namenode UI
[ https://issues.apache.org/jira/browse/HDFS-7483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benoy Antony updated HDFS-7483: --- Attachment: (was: HDFS-7483-002.patch) > Display information per tier on the Namenode UI > --- > > Key: HDFS-7483 > URL: https://issues.apache.org/jira/browse/HDFS-7483 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Benoy Antony >Assignee: Benoy Antony > Attachments: HDFS-7483-001.patch, HDFS-7483-002.patch, overview.png, > storagetypes.png, storagetypes_withnostorage.png, withOneStorageType.png, > withTwoStorageType.png > > > If cluster has different types of storage, it is useful to display the > storage information per type. > The information will be available via JMX (HDFS-7390) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7483) Display information per tier on the Namenode UI
[ https://issues.apache.org/jira/browse/HDFS-7483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benoy Antony updated HDFS-7483: --- Status: Patch Available (was: In Progress) > Display information per tier on the Namenode UI > --- > > Key: HDFS-7483 > URL: https://issues.apache.org/jira/browse/HDFS-7483 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Benoy Antony >Assignee: Benoy Antony > Attachments: HDFS-7483-001.patch, HDFS-7483-002.patch, overview.png, > storagetypes.png, storagetypes_withnostorage.png, withOneStorageType.png, > withTwoStorageType.png > > > If cluster has different types of storage, it is useful to display the > storage information per type. > The information will be available via JMX (HDFS-7390) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7483) Display information per tier on the Namenode UI
[ https://issues.apache.org/jira/browse/HDFS-7483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benoy Antony updated HDFS-7483: --- Attachment: withTwoStorageType.png withOneStorageType.png HDFS-7483-002.patch Thanks for the review and comments , [~wheat9] and [~arpitagarwal]. Updated the patch based on the comments. # Displays Storage type information on the overview screen. The table is displayed only if there are more than one storage types. Following screenshots are attached: #* Overview page when there is only one storage type. #* Overview page when there are two storage types. # Added dust.js helper to calculate percentage on the client side. # Removed an unnecessary _System.out.println_ added by HDFS-7390. > Display information per tier on the Namenode UI > --- > > Key: HDFS-7483 > URL: https://issues.apache.org/jira/browse/HDFS-7483 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Benoy Antony >Assignee: Benoy Antony > Attachments: HDFS-7483-001.patch, HDFS-7483-002.patch, overview.png, > storagetypes.png, storagetypes_withnostorage.png, withOneStorageType.png, > withTwoStorageType.png > > > If cluster has different types of storage, it is useful to display the > storage information per type. > The information will be available via JMX (HDFS-7390) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7483) Display information per tier on the Namenode UI
[ https://issues.apache.org/jira/browse/HDFS-7483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benoy Antony updated HDFS-7483: --- Attachment: (was: HDFS-7483-001.patch) > Display information per tier on the Namenode UI > --- > > Key: HDFS-7483 > URL: https://issues.apache.org/jira/browse/HDFS-7483 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Benoy Antony >Assignee: Benoy Antony > Attachments: HDFS-7483-001.patch, overview.png, storagetypes.png, > storagetypes_withnostorage.png > > > If cluster has different types of storage, it is useful to display the > storage information per type. > The information will be available via JMX (HDFS-7390) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7483) Display information per tier on the Namenode UI
[ https://issues.apache.org/jira/browse/HDFS-7483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benoy Antony updated HDFS-7483: --- Attachment: storagetypes.png storagetypes_withnostorage.png HDFS-7483-001.patch overview.png > Display information per tier on the Namenode UI > --- > > Key: HDFS-7483 > URL: https://issues.apache.org/jira/browse/HDFS-7483 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Benoy Antony >Assignee: Benoy Antony > Attachments: HDFS-7483-001.patch, overview.png, storagetypes.png, > storagetypes_withnostorage.png > > > If cluster has different types of storage, it is useful to display the > storage information per type. > The information will be available via JMX (HDFS-7390) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7483) Display information per tier on the Namenode UI
[ https://issues.apache.org/jira/browse/HDFS-7483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benoy Antony updated HDFS-7483: --- Attachment: HDFS-7483-001.patch Attaching a patch which displays storage stats by storage type on the name node ui. The screen shots are attached. The patch has the following changes. 1. The jmx for Blocktstats is enhanced by adding PercentRemaining and PercentUsed. 2. Refactored header part of name node ui to header.html 3. Added a page for storage types as shown in the screen shots and tested on a local cluster. > Display information per tier on the Namenode UI > --- > > Key: HDFS-7483 > URL: https://issues.apache.org/jira/browse/HDFS-7483 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Benoy Antony >Assignee: Benoy Antony > Attachments: HDFS-7483-001.patch > > > If cluster has different types of storage, it is useful to display the > storage information per type. > The information will be available via JMX (HDFS-7390) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7483) Display information per tier on the Namenode UI
[ https://issues.apache.org/jira/browse/HDFS-7483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benoy Antony updated HDFS-7483: --- Description: If cluster has different types of storage, it is useful to display the storage information per type. The information will be available via JMX (HDFS-7390) was: If cluster has different types of storage, it is useful to display the storage information per tier. The information will be available via JMX (HDFS-7390) > Display information per tier on the Namenode UI > --- > > Key: HDFS-7483 > URL: https://issues.apache.org/jira/browse/HDFS-7483 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Benoy Antony >Assignee: Benoy Antony > > If cluster has different types of storage, it is useful to display the > storage information per type. > The information will be available via JMX (HDFS-7390) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Work started] (HDFS-7483) Display information per tier on the Namenode UI
[ https://issues.apache.org/jira/browse/HDFS-7483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HDFS-7483 started by Benoy Antony. -- > Display information per tier on the Namenode UI > --- > > Key: HDFS-7483 > URL: https://issues.apache.org/jira/browse/HDFS-7483 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Benoy Antony >Assignee: Benoy Antony > > If cluster has different tiers of storage, it is useful to display the > storage information per tier. > The information will be available via JMX (HDFS-7390) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7483) Display information per tier on the Namenode UI
[ https://issues.apache.org/jira/browse/HDFS-7483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benoy Antony updated HDFS-7483: --- Description: If cluster has different types of storage, it is useful to display the storage information per tier. The information will be available via JMX (HDFS-7390) was: If cluster has different tiers of storage, it is useful to display the storage information per tier. The information will be available via JMX (HDFS-7390) > Display information per tier on the Namenode UI > --- > > Key: HDFS-7483 > URL: https://issues.apache.org/jira/browse/HDFS-7483 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Benoy Antony >Assignee: Benoy Antony > > If cluster has different types of storage, it is useful to display the > storage information per tier. > The information will be available via JMX (HDFS-7390) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7390) Provide JMX metrics per storage type
[ https://issues.apache.org/jira/browse/HDFS-7390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benoy Antony updated HDFS-7390: --- Resolution: Fixed Fix Version/s: 2.8.0 Status: Resolved (was: Patch Available) > Provide JMX metrics per storage type > > > Key: HDFS-7390 > URL: https://issues.apache.org/jira/browse/HDFS-7390 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 2.5.2 >Reporter: Benoy Antony >Assignee: Benoy Antony > Labels: BB2015-05-TBR > Fix For: 2.8.0 > > Attachments: HDFS-7390-003.patch, HDFS-7390-004.patch, > HDFS-7390-005.patch, HDFS-7390-006.patch, HDFS-7390-007.patch, > HDFS-7390-008.patch, HDFS-7390.patch, HDFS-7390.patch > > > HDFS-2832 added heterogeneous support. In a cluster with different storage > types, it is useful to have metrics per storage type. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7390) Provide JMX metrics per storage type
[ https://issues.apache.org/jira/browse/HDFS-7390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14606020#comment-14606020 ] Benoy Antony commented on HDFS-7390: committed to trunk and branch-2. > Provide JMX metrics per storage type > > > Key: HDFS-7390 > URL: https://issues.apache.org/jira/browse/HDFS-7390 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 2.5.2 >Reporter: Benoy Antony >Assignee: Benoy Antony > Labels: BB2015-05-TBR > Fix For: 2.8.0 > > Attachments: HDFS-7390-003.patch, HDFS-7390-004.patch, > HDFS-7390-005.patch, HDFS-7390-006.patch, HDFS-7390-007.patch, > HDFS-7390-008.patch, HDFS-7390.patch, HDFS-7390.patch > > > HDFS-2832 added heterogeneous support. In a cluster with different storage > types, it is useful to have metrics per storage type. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7390) Provide JMX metrics per storage type
[ https://issues.apache.org/jira/browse/HDFS-7390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14605987#comment-14605987 ] Benoy Antony commented on HDFS-7390: The test failure on _ TestWebHDFS_ is unrelated. Thanks for the review, [~arpitagarwal]. I'll commit this shortly. > Provide JMX metrics per storage type > > > Key: HDFS-7390 > URL: https://issues.apache.org/jira/browse/HDFS-7390 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 2.5.2 >Reporter: Benoy Antony >Assignee: Benoy Antony > Labels: BB2015-05-TBR > Attachments: HDFS-7390-003.patch, HDFS-7390-004.patch, > HDFS-7390-005.patch, HDFS-7390-006.patch, HDFS-7390-007.patch, > HDFS-7390-008.patch, HDFS-7390.patch, HDFS-7390.patch > > > HDFS-2832 added heterogeneous support. In a cluster with different storage > types, it is useful to have metrics per storage type. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7390) Provide JMX metrics per storage type
[ https://issues.apache.org/jira/browse/HDFS-7390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benoy Antony updated HDFS-7390: --- Attachment: HDFS-7390-008.patch Attaching the patch which addresses [~arpitagarwal]'s comments. > Provide JMX metrics per storage type > > > Key: HDFS-7390 > URL: https://issues.apache.org/jira/browse/HDFS-7390 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 2.5.2 >Reporter: Benoy Antony >Assignee: Benoy Antony > Labels: BB2015-05-TBR > Attachments: HDFS-7390-003.patch, HDFS-7390-004.patch, > HDFS-7390-005.patch, HDFS-7390-006.patch, HDFS-7390-007.patch, > HDFS-7390-008.patch, HDFS-7390.patch, HDFS-7390.patch > > > HDFS-2832 added heterogeneous support. In a cluster with different storage > types, it is useful to have metrics per storage type. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7390) Provide JMX metrics per storage type
[ https://issues.apache.org/jira/browse/HDFS-7390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14601772#comment-14601772 ] Benoy Antony commented on HDFS-7390: The checkstyle issue is due to the length of _BlockManager_ which was already above 2000 lines. Not some that was introduced by the patch > Provide JMX metrics per storage type > > > Key: HDFS-7390 > URL: https://issues.apache.org/jira/browse/HDFS-7390 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 2.5.2 >Reporter: Benoy Antony >Assignee: Benoy Antony > Labels: BB2015-05-TBR > Attachments: HDFS-7390-003.patch, HDFS-7390-004.patch, > HDFS-7390-005.patch, HDFS-7390-006.patch, HDFS-7390-007.patch, > HDFS-7390.patch, HDFS-7390.patch > > > HDFS-2832 added heterogeneous support. In a cluster with different storage > types, it is useful to have metrics per storage type. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7390) Provide JMX metrics per storage type
[ https://issues.apache.org/jira/browse/HDFS-7390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benoy Antony updated HDFS-7390: --- Attachment: HDFS-7390-007.patch Attaching a patch after fixing the compilation issues. > Provide JMX metrics per storage type > > > Key: HDFS-7390 > URL: https://issues.apache.org/jira/browse/HDFS-7390 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 2.5.2 >Reporter: Benoy Antony >Assignee: Benoy Antony > Labels: BB2015-05-TBR > Attachments: HDFS-7390-003.patch, HDFS-7390-004.patch, > HDFS-7390-005.patch, HDFS-7390-006.patch, HDFS-7390-007.patch, > HDFS-7390.patch, HDFS-7390.patch > > > HDFS-2832 added heterogeneous support. In a cluster with different storage > types, it is useful to have metrics per storage type. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7390) Provide JMX metrics per storage type
[ https://issues.apache.org/jira/browse/HDFS-7390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benoy Antony updated HDFS-7390: --- Attachment: HDFS-7390-006.patch Thanks for the review [~arpitagarwal] Attaching a patch which addresses the comments. > Provide JMX metrics per storage type > > > Key: HDFS-7390 > URL: https://issues.apache.org/jira/browse/HDFS-7390 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 2.5.2 >Reporter: Benoy Antony >Assignee: Benoy Antony > Labels: BB2015-05-TBR > Attachments: HDFS-7390-003.patch, HDFS-7390-004.patch, > HDFS-7390-005.patch, HDFS-7390-006.patch, HDFS-7390.patch, HDFS-7390.patch > > > HDFS-2832 added heterogeneous support. In a cluster with different storage > types, it is useful to have metrics per storage type. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7390) Provide JMX metrics per storage type
[ https://issues.apache.org/jira/browse/HDFS-7390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benoy Antony updated HDFS-7390: --- Attachment: (was: HDFS-7390-005.patch) > Provide JMX metrics per storage type > > > Key: HDFS-7390 > URL: https://issues.apache.org/jira/browse/HDFS-7390 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 2.5.2 >Reporter: Benoy Antony >Assignee: Benoy Antony > Labels: BB2015-05-TBR > Attachments: HDFS-7390-003.patch, HDFS-7390-004.patch, > HDFS-7390-005.patch, HDFS-7390.patch, HDFS-7390.patch > > > HDFS-2832 added heterogeneous support. In a cluster with different storage > types, it is useful to have metrics per storage type. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7390) Provide JMX metrics per storage type
[ https://issues.apache.org/jira/browse/HDFS-7390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benoy Antony updated HDFS-7390: --- Attachment: HDFS-7390-005.patch > Provide JMX metrics per storage type > > > Key: HDFS-7390 > URL: https://issues.apache.org/jira/browse/HDFS-7390 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 2.5.2 >Reporter: Benoy Antony >Assignee: Benoy Antony > Labels: BB2015-05-TBR > Attachments: HDFS-7390-003.patch, HDFS-7390-004.patch, > HDFS-7390-005.patch, HDFS-7390-005.patch, HDFS-7390.patch, HDFS-7390.patch > > > HDFS-2832 added heterogeneous support. In a cluster with different storage > types, it is useful to have metrics per storage type. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7390) Provide JMX metrics per storage type
[ https://issues.apache.org/jira/browse/HDFS-7390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benoy Antony updated HDFS-7390: --- Attachment: HDFS-7390-005.patch fixing checkstyle warning > Provide JMX metrics per storage type > > > Key: HDFS-7390 > URL: https://issues.apache.org/jira/browse/HDFS-7390 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 2.5.2 >Reporter: Benoy Antony >Assignee: Benoy Antony > Labels: BB2015-05-TBR > Attachments: HDFS-7390-003.patch, HDFS-7390-004.patch, > HDFS-7390-005.patch, HDFS-7390.patch, HDFS-7390.patch > > > HDFS-2832 added heterogeneous support. In a cluster with different storage > types, it is useful to have metrics per storage type. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7390) Provide JMX metrics per storage type
[ https://issues.apache.org/jira/browse/HDFS-7390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benoy Antony updated HDFS-7390: --- Attachment: HDFS-7390-004.patch Attaching a patch after fixing checkstyle warnings. > Provide JMX metrics per storage type > > > Key: HDFS-7390 > URL: https://issues.apache.org/jira/browse/HDFS-7390 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 2.5.2 >Reporter: Benoy Antony >Assignee: Benoy Antony > Labels: BB2015-05-TBR > Attachments: HDFS-7390-003.patch, HDFS-7390-004.patch, > HDFS-7390.patch, HDFS-7390.patch > > > HDFS-2832 added heterogeneous support. In a cluster with different storage > types, it is useful to have metrics per storage type. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7390) Provide JMX metrics per storage type
[
https://issues.apache.org/jira/browse/HDFS-7390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Benoy Antony updated HDFS-7390:
---
Attachment: HDFS-7390-003.patch
Updated the patch. A new MXBean _BlockStats_ is defined and is implemented by
_BlockManager_.
Used standard MXBean approach (based on CompositeDataSupport) to output Json.
Testcases are added.
Dependency on HADOOP-11308 is no longer needed and it is closed.
The sample Json output is as below:
{code}
{
"name" : "Hadoop:service=NameNode,name=BlockStats",
"modelerType" :
"org.apache.hadoop.hdfs.server.blockmanagement.BlockManager",
"TieredStorageStats" : [ {
"key" : "RAM_DISK",
"value" : {
"blockPoolUsed" : 24576,
"capacityRemaining" : 1634963226624,
"capacityTotal" : 2994494914560,
"capacityUsed" : 24576,
"nodesInService" : 6
}
}, {
"key" : "ARCHIVE",
"value" : {
"blockPoolUsed" : 12288,
"capacityRemaining" : 817481576448,
"capacityTotal" : 1497247457280,
"capacityUsed" : 12288,
"nodesInService" : 3
}
}, {
"key" : "DISK",
"value" : {
"blockPoolUsed" : 12288,
"capacityRemaining" : 817481650176,
"capacityTotal" : 1497247457280,
"capacityUsed" : 12288,
"nodesInService" : 3
}
} ]
}
{code}
> Provide JMX metrics per storage type
>
>
> Key: HDFS-7390
> URL: https://issues.apache.org/jira/browse/HDFS-7390
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: 2.5.2
>Reporter: Benoy Antony
>Assignee: Benoy Antony
> Labels: BB2015-05-TBR
> Attachments: HDFS-7390-003.patch, HDFS-7390.patch, HDFS-7390.patch
>
>
> HDFS-2832 added heterogeneous support. In a cluster with different storage
> types, it is useful to have metrics per storage type.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8291) Modify NN WebUI to display correct unit
[ https://issues.apache.org/jira/browse/HDFS-8291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14529751#comment-14529751 ] Benoy Antony commented on HDFS-8291: Terabyte (TB) is more widely used than Tebibyte(TiB). I suggest fixing the calculation than changing from TB to TiB. > Modify NN WebUI to display correct unit > > > Key: HDFS-8291 > URL: https://issues.apache.org/jira/browse/HDFS-8291 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Zhongyi Xie >Assignee: Zhongyi Xie >Priority: Minor > Attachments: HDFS-8291.001.patch, HDFS-8291.002.patch > > > NN Web UI displays its capacity and usage in TB, but it is actually TiB. We > should either change the unit name or the calculation to ensure it follows > standards. > http://en.wikipedia.org/wiki/Tebibyte > http://en.wikipedia.org/wiki/Terabyte -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-6407) new namenode UI, lost ability to sort columns in datanode tab
[ https://issues.apache.org/jira/browse/HDFS-6407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benoy Antony updated HDFS-6407: --- Attachment: HDFS-6407-003.patch Adjusted the page size to a lower number (instead of All) to ensure that pages get painted fast when there are lots of items. This is an existing problem when there are thousands of items to display. Sorting thousands of items has no noticeable delay to require any optimization. Tested with 8500 files in one directory. > new namenode UI, lost ability to sort columns in datanode tab > - > > Key: HDFS-6407 > URL: https://issues.apache.org/jira/browse/HDFS-6407 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.4.0 >Reporter: Nathan Roberts >Assignee: Benoy Antony >Priority: Minor > Attachments: 002-datanodes-sorted-capacityUsed.png, > 002-datanodes.png, 002-filebrowser.png, 002-snapshots.png, > HDFS-6407-002.patch, HDFS-6407-003.patch, HDFS-6407.patch, > browse_directory.png, datanodes.png, snapshots.png > > > old ui supported clicking on column header to sort on that column. The new ui > seems to have dropped this very useful feature. > There are a few tables in the Namenode UI to display datanodes information, > directory listings and snapshots. > When there are many items in the tables, it is useful to have ability to sort > on the different columns. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6407) new namenode UI, lost ability to sort columns in datanode tab
[ https://issues.apache.org/jira/browse/HDFS-6407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14524415#comment-14524415 ] Benoy Antony commented on HDFS-6407: The sorting seems to be correct for all the fields. Please let me know what makes you think otherwise and I can fix it. The purpose of this jira is to put back sorting on data nodes tab. Since it was easy to enable sorting on other tables using the plugin, sorting and pagination was added to those tables as well. The processing is done completely on the client side and so there is no impact on the server side. At this point, the process of enabling sorting pagination for a table is simple and uniform. To give up this simplicity, there should be some noticeable delay with the current approach. Tested with 6,000 files. It took time to render 6,000 items irrespective of sorting is enabled or not. If pagination is set , then items are painted instantly. In any case, no noticeable delay is caused by sorting. So I am not sure whether there is any point in optimizing this client side logic. One enhancement which enables the page to render fast , will be to set the default pagination to reasonably higher number like 200. In cases where there are 10,000+ items to paint, the pages gets rendered fast if page size is 200 items. If the user wants all items, then he can select "All" display all files. Or User can search for the file in other pages.I will update the patch with page size set to 200 items for all tables. > new namenode UI, lost ability to sort columns in datanode tab > - > > Key: HDFS-6407 > URL: https://issues.apache.org/jira/browse/HDFS-6407 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.4.0 >Reporter: Nathan Roberts >Assignee: Benoy Antony >Priority: Minor > Attachments: 002-datanodes-sorted-capacityUsed.png, > 002-datanodes.png, 002-filebrowser.png, 002-snapshots.png, > HDFS-6407-002.patch, HDFS-6407.patch, browse_directory.png, datanodes.png, > snapshots.png > > > old ui supported clicking on column header to sort on that column. The new ui > seems to have dropped this very useful feature. > There are a few tables in the Namenode UI to display datanodes information, > directory listings and snapshots. > When there are many items in the tables, it is useful to have ability to sort > on the different columns. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6407) new namenode UI, lost ability to sort columns in datanode tab
[ https://issues.apache.org/jira/browse/HDFS-6407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14523563#comment-14523563 ] Benoy Antony commented on HDFS-6407: That will be a good optimization. Can we please do it as an enhancement in a separate jira ? > new namenode UI, lost ability to sort columns in datanode tab > - > > Key: HDFS-6407 > URL: https://issues.apache.org/jira/browse/HDFS-6407 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.4.0 >Reporter: Nathan Roberts >Assignee: Benoy Antony >Priority: Minor > Attachments: 002-datanodes-sorted-capacityUsed.png, > 002-datanodes.png, 002-filebrowser.png, 002-snapshots.png, > HDFS-6407-002.patch, HDFS-6407.patch, browse_directory.png, datanodes.png, > snapshots.png > > > old ui supported clicking on column header to sort on that column. The new ui > seems to have dropped this very useful feature. > There are a few tables in the Namenode UI to display datanodes information, > directory listings and snapshots. > When there are many items in the tables, it is useful to have ability to sort > on the different columns. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HDFS-6407) new namenode UI, lost ability to sort columns in datanode tab
[ https://issues.apache.org/jira/browse/HDFS-6407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14523438#comment-14523438 ] Benoy Antony edited comment on HDFS-6407 at 5/1/15 4:52 PM: The patch is updated to use the data tables plugin (http://www.datatables.net/). The same plugin is used in the Resource Manager UI also. Please check the attached screenshots to see the sorting in action. The patch takes care of sorting based on file sizes. Also supports pagination on the client side. Notes about the sorting and pagination features added to the different tables on Namenode UI: # Uses data tables plugin (http://www.datatables.net/). # Uses latest version 1.10.7 (Yarn uses 1.9.4 and Yarn can potentially migrate to the latest version) # Added logic to detect columns showing file sizes and sort based on file sizes. # Added custom logic detect columns showing file size and percentage and sort based on file sizes. This is used to detect and sort “BlockPool Used” column which displays value like 7.74 MB (10%). # The default pagination is to display all the rows to match the existing behavior. The user can further select smaller page sizes using the drop down. # The following tables are set up to be sortable and pageable : datanodes, decommissioning nodes, datanode volume failures, snapshot table directories, snapshotted directories and file/dir listing. # For file/dir listings , default sort happens on the name column (last column). For snapshotted directories, the default sort happens on the snapshot directory (second column). For all other tables, the sort happens on the first column. was (Author: benoyantony): The patch is updated to use the data tables plugin (http://www.datatables.net/). The same plugin is used in the Resource Manager UI also. The patch takes care of sorting based on file sizes. Also supports pagination on the client side. Notes about the sorting and pagination features added to the different tables on Namenode UI: # Uses data tables plugin (http://www.datatables.net/). # Uses latest version 1.10.7 (Yarn uses 1.9.4 and Yarn can potentially migrate to the latest version) # Added logic to detect columns showing file sizes and sort based on file sizes. # Added custom logic detect columns showing file size and percentage and sort based on file sizes. This is used to detect and sort “BlockPool Used” column which displays value like 7.74 MB (10%). # The default pagination is to display all the rows to match the existing behavior. The user can further select smaller page sizes using the drop down. # The following tables are set up to be sortable and pageable : datanodes, decommissioning nodes, datanode volume failures, snapshot table directories, snapshotted directories and file/dir listing. # For file/dir listings , default sort happens on the name column (last column). For snapshotted directories, the default sort happens on the snapshot directory (second column). For all other tables, the sort happens on the first column. > new namenode UI, lost ability to sort columns in datanode tab > - > > Key: HDFS-6407 > URL: https://issues.apache.org/jira/browse/HDFS-6407 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.4.0 >Reporter: Nathan Roberts >Assignee: Benoy Antony >Priority: Minor > Attachments: 002-datanodes-sorted-capacityUsed.png, > 002-datanodes.png, 002-filebrowser.png, 002-snapshots.png, > HDFS-6407-002.patch, HDFS-6407.patch, browse_directory.png, datanodes.png, > snapshots.png > > > old ui supported clicking on column header to sort on that column. The new ui > seems to have dropped this very useful feature. > There are a few tables in the Namenode UI to display datanodes information, > directory listings and snapshots. > When there are many items in the tables, it is useful to have ability to sort > on the different columns. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HDFS-6407) new namenode UI, lost ability to sort columns in datanode tab
[ https://issues.apache.org/jira/browse/HDFS-6407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14523438#comment-14523438 ] Benoy Antony edited comment on HDFS-6407 at 5/1/15 4:51 PM: The patch is updated to use the data tables plugin (http://www.datatables.net/). The same plugin is used in the Resource Manager UI also. The patch takes care of sorting based on file sizes. Also supports pagination on the client side. Notes about the sorting and pagination features added to the different tables on Namenode UI: # Uses data tables plugin (http://www.datatables.net/). # Uses latest version 1.10.7 (Yarn uses 1.9.4 and Yarn can potentially migrate to the latest version) # Added logic to detect columns showing file sizes and sort based on file sizes. # Added custom logic detect columns showing file size and percentage and sort based on file sizes. This is used to detect and sort “BlockPool Used” column which displays value like 7.74 MB (10%). # The default pagination is to display all the rows to match the existing behavior. The user can further select smaller page sizes using the drop down. # The following tables are set up to be sortable and pageable : datanodes, decommissioning nodes, datanode volume failures, snapshot table directories, snapshotted directories and file/dir listing. # For file/dir listings , default sort happens on the name column (last column). For snapshotted directories, the default sort happens on the snapshot directory (second column). For all other tables, the sort happens on the first column. was (Author: benoyantony): The patch is update using the data tables plugin (http://www.datatables.net/). The same plugin is used in the Resource Manager UI also. The patch takes care of sorting based on file sizes. Also supports pagination on the client side. Notes about the sorting and pagination features added to the different tables on Namenode UI: # Uses data tables plugin (http://www.datatables.net/). # Uses latest version 1.10.7 (Yarn uses 1.9.4 and Yarn can potentially migrate to the latest version) # Added logic to detect columns showing file sizes and sort based on file sizes. # Added custom logic detect columns showing file size and percentage and sort based on file sizes. This is used to detect and sort “BlockPool Used” column which displays value like 7.74 MB (10%). # The default pagination is to display all the rows to match the existing behavior. The user can further select smaller page sizes using the drop down. # The following tables are set up to be sortable and pageable : datanodes, decommissioning nodes, datanode volume failures, snapshot table directories, snapshotted directories and file/dir listing. # For file/dir listings , default sort happens on the name column (last column). For snapshotted directories, the default sort happens on the snapshot directory (second column). For all other tables, the sort happens on the first column. > new namenode UI, lost ability to sort columns in datanode tab > - > > Key: HDFS-6407 > URL: https://issues.apache.org/jira/browse/HDFS-6407 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.4.0 >Reporter: Nathan Roberts >Assignee: Benoy Antony >Priority: Minor > Attachments: 002-datanodes-sorted-capacityUsed.png, > 002-datanodes.png, 002-filebrowser.png, 002-snapshots.png, > HDFS-6407-002.patch, HDFS-6407.patch, browse_directory.png, datanodes.png, > snapshots.png > > > old ui supported clicking on column header to sort on that column. The new ui > seems to have dropped this very useful feature. > There are a few tables in the Namenode UI to display datanodes information, > directory listings and snapshots. > When there are many items in the tables, it is useful to have ability to sort > on the different columns. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-6407) new namenode UI, lost ability to sort columns in datanode tab
[ https://issues.apache.org/jira/browse/HDFS-6407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benoy Antony updated HDFS-6407: --- Attachment: 002-snapshots.png 002-filebrowser.png 002-datanodes-sorted-capacityUsed.png 002-datanodes.png > new namenode UI, lost ability to sort columns in datanode tab > - > > Key: HDFS-6407 > URL: https://issues.apache.org/jira/browse/HDFS-6407 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.4.0 >Reporter: Nathan Roberts >Assignee: Benoy Antony >Priority: Minor > Attachments: 002-datanodes-sorted-capacityUsed.png, > 002-datanodes.png, 002-filebrowser.png, 002-snapshots.png, HDFS-6407.patch, > browse_directory.png, datanodes.png, snapshots.png > > > old ui supported clicking on column header to sort on that column. The new ui > seems to have dropped this very useful feature. > There are a few tables in the Namenode UI to display datanodes information, > directory listings and snapshots. > When there are many items in the tables, it is useful to have ability to sort > on the different columns. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-6407) new namenode UI, lost ability to sort columns in datanode tab
[ https://issues.apache.org/jira/browse/HDFS-6407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benoy Antony updated HDFS-6407: --- Attachment: HDFS-6407-002.patch The patch is update using the data tables plugin (http://www.datatables.net/). The same plugin is used in the Resource Manager UI also. The patch takes care of sorting based on file sizes. Also supports pagination on the client side. Notes about the sorting and pagination features added to the different tables on Namenode UI: # Uses data tables plugin (http://www.datatables.net/). # Uses latest version 1.10.7 (Yarn uses 1.9.4 and Yarn can potentially migrate to the latest version) # Added logic to detect columns showing file sizes and sort based on file sizes. # Added custom logic detect columns showing file size and percentage and sort based on file sizes. This is used to detect and sort “BlockPool Used” column which displays value like 7.74 MB (10%). # The default pagination is to display all the rows to match the existing behavior. The user can further select smaller page sizes using the drop down. # The following tables are set up to be sortable and pageable : datanodes, decommissioning nodes, datanode volume failures, snapshot table directories, snapshotted directories and file/dir listing. # For file/dir listings , default sort happens on the name column (last column). For snapshotted directories, the default sort happens on the snapshot directory (second column). For all other tables, the sort happens on the first column. > new namenode UI, lost ability to sort columns in datanode tab > - > > Key: HDFS-6407 > URL: https://issues.apache.org/jira/browse/HDFS-6407 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.4.0 >Reporter: Nathan Roberts >Assignee: Benoy Antony >Priority: Minor > Attachments: 002-datanodes-sorted-capacityUsed.png, > 002-datanodes.png, 002-filebrowser.png, 002-snapshots.png, > HDFS-6407-002.patch, HDFS-6407.patch, browse_directory.png, datanodes.png, > snapshots.png > > > old ui supported clicking on column header to sort on that column. The new ui > seems to have dropped this very useful feature. > There are a few tables in the Namenode UI to display datanodes information, > directory listings and snapshots. > When there are many items in the tables, it is useful to have ability to sort > on the different columns. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-5823) Document async audit logging
[
https://issues.apache.org/jira/browse/HDFS-5823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14357338#comment-14357338
]
Benoy Antony commented on HDFS-5823:
[~daryn]
Can't I achieve the same by configuring an AsyncAppender as below :
Do you see any downside in doing this ?
{code}
{code}
> Document async audit logging
>
>
> Key: HDFS-5823
> URL: https://issues.apache.org/jira/browse/HDFS-5823
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Affects Versions: 2.0.0-alpha, 3.0.0
>Reporter: Daryn Sharp
>Assignee: Daryn Sharp
>
> HDFS-5241 added an option for async log4j audit logging. The option is
> considered semi-experimental and should be documented in hdfs-defaults.xml
> after it's stability under stress is proven.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-5823) Document async audit logging
[ https://issues.apache.org/jira/browse/HDFS-5823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14353731#comment-14353731 ] Benoy Antony commented on HDFS-5823: [~daryn] Is the async audit log feature feature stable? > Document async audit logging > > > Key: HDFS-5823 > URL: https://issues.apache.org/jira/browse/HDFS-5823 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 2.0.0-alpha, 3.0.0 >Reporter: Daryn Sharp >Assignee: Daryn Sharp > > HDFS-5241 added an option for async log4j audit logging. The option is > considered semi-experimental and should be documented in hdfs-defaults.xml > after it's stability under stress is proven. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7467) Provide storage tier information for a directory via fsck
[
https://issues.apache.org/jira/browse/HDFS-7467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Benoy Antony updated HDFS-7467:
---
Fix Version/s: 2.7.0
> Provide storage tier information for a directory via fsck
> -
>
> Key: HDFS-7467
> URL: https://issues.apache.org/jira/browse/HDFS-7467
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: balancer & mover
>Affects Versions: 2.6.0
>Reporter: Benoy Antony
>Assignee: Benoy Antony
> Fix For: 2.7.0
>
> Attachments: HDFS-7467-002.patch, HDFS-7467-003.patch,
> HDFS-7467-004.patch, HDFS-7467.patch, storagepolicydisplay.pdf
>
>
> Currently _fsck_ provides information regarding blocks for a directory.
> It should be augmented to provide storage tier information (optionally).
> The sample report could be as follows :
> {code}
> Storage Tier Combination# of blocks % of blocks
> DISK:1,ARCHIVE:2 340730 97.7393%
>
> ARCHIVE:3 39281.1268%
>
> DISK:2,ARCHIVE:231220.8956%
>
> DISK:2,ARCHIVE:1 7480.2146%
>
> DISK:1,ARCHIVE:3 440.0126%
>
> DISK:3,ARCHIVE:2 300.0086%
>
> DISK:3,ARCHIVE:1 90.0026%
> {code}
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7467) Provide storage tier information for a directory via fsck
[
https://issues.apache.org/jira/browse/HDFS-7467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14337588#comment-14337588
]
Benoy Antony commented on HDFS-7467:
committed to trunk and branch-2.
> Provide storage tier information for a directory via fsck
> -
>
> Key: HDFS-7467
> URL: https://issues.apache.org/jira/browse/HDFS-7467
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: balancer & mover
>Affects Versions: 2.6.0
>Reporter: Benoy Antony
>Assignee: Benoy Antony
> Fix For: 2.7.0
>
> Attachments: HDFS-7467-002.patch, HDFS-7467-003.patch,
> HDFS-7467-004.patch, HDFS-7467.patch, storagepolicydisplay.pdf
>
>
> Currently _fsck_ provides information regarding blocks for a directory.
> It should be augmented to provide storage tier information (optionally).
> The sample report could be as follows :
> {code}
> Storage Tier Combination# of blocks % of blocks
> DISK:1,ARCHIVE:2 340730 97.7393%
>
> ARCHIVE:3 39281.1268%
>
> DISK:2,ARCHIVE:231220.8956%
>
> DISK:2,ARCHIVE:1 7480.2146%
>
> DISK:1,ARCHIVE:3 440.0126%
>
> DISK:3,ARCHIVE:2 300.0086%
>
> DISK:3,ARCHIVE:1 90.0026%
> {code}
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7467) Provide storage tier information for a directory via fsck
[
https://issues.apache.org/jira/browse/HDFS-7467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Benoy Antony updated HDFS-7467:
---
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Provide storage tier information for a directory via fsck
> -
>
> Key: HDFS-7467
> URL: https://issues.apache.org/jira/browse/HDFS-7467
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: balancer & mover
>Affects Versions: 2.6.0
>Reporter: Benoy Antony
>Assignee: Benoy Antony
> Attachments: HDFS-7467-002.patch, HDFS-7467-003.patch,
> HDFS-7467-004.patch, HDFS-7467.patch, storagepolicydisplay.pdf
>
>
> Currently _fsck_ provides information regarding blocks for a directory.
> It should be augmented to provide storage tier information (optionally).
> The sample report could be as follows :
> {code}
> Storage Tier Combination# of blocks % of blocks
> DISK:1,ARCHIVE:2 340730 97.7393%
>
> ARCHIVE:3 39281.1268%
>
> DISK:2,ARCHIVE:231220.8956%
>
> DISK:2,ARCHIVE:1 7480.2146%
>
> DISK:1,ARCHIVE:3 440.0126%
>
> DISK:3,ARCHIVE:2 300.0086%
>
> DISK:3,ARCHIVE:1 90.0026%
> {code}
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7467) Provide storage tier information for a directory via fsck
[
https://issues.apache.org/jira/browse/HDFS-7467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14335606#comment-14335606
]
Benoy Antony commented on HDFS-7467:
Thanks for the review [~szetszwo].
If there are no further comments, I'll commit the patch tomorrow.
> Provide storage tier information for a directory via fsck
> -
>
> Key: HDFS-7467
> URL: https://issues.apache.org/jira/browse/HDFS-7467
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: balancer & mover
>Affects Versions: 2.6.0
>Reporter: Benoy Antony
>Assignee: Benoy Antony
> Attachments: HDFS-7467-002.patch, HDFS-7467-003.patch,
> HDFS-7467-004.patch, HDFS-7467.patch, storagepolicydisplay.pdf
>
>
> Currently _fsck_ provides information regarding blocks for a directory.
> It should be augmented to provide storage tier information (optionally).
> The sample report could be as follows :
> {code}
> Storage Tier Combination# of blocks % of blocks
> DISK:1,ARCHIVE:2 340730 97.7393%
>
> ARCHIVE:3 39281.1268%
>
> DISK:2,ARCHIVE:231220.8956%
>
> DISK:2,ARCHIVE:1 7480.2146%
>
> DISK:1,ARCHIVE:3 440.0126%
>
> DISK:3,ARCHIVE:2 300.0086%
>
> DISK:3,ARCHIVE:1 90.0026%
> {code}
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7467) Provide storage tier information for a directory via fsck
[
https://issues.apache.org/jira/browse/HDFS-7467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Benoy Antony updated HDFS-7467:
---
Attachment: HDFS-7467-004.patch
Attaching the patch which takes care of the recommended changes.
Removed the method StoragePolicySummary.process() as the statements in that
method could be done via other methods.
> Provide storage tier information for a directory via fsck
> -
>
> Key: HDFS-7467
> URL: https://issues.apache.org/jira/browse/HDFS-7467
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: balancer & mover
>Affects Versions: 2.6.0
>Reporter: Benoy Antony
>Assignee: Benoy Antony
> Attachments: HDFS-7467-002.patch, HDFS-7467-003.patch,
> HDFS-7467-004.patch, HDFS-7467.patch, storagepolicydisplay.pdf
>
>
> Currently _fsck_ provides information regarding blocks for a directory.
> It should be augmented to provide storage tier information (optionally).
> The sample report could be as follows :
> {code}
> Storage Tier Combination# of blocks % of blocks
> DISK:1,ARCHIVE:2 340730 97.7393%
>
> ARCHIVE:3 39281.1268%
>
> DISK:2,ARCHIVE:231220.8956%
>
> DISK:2,ARCHIVE:1 7480.2146%
>
> DISK:1,ARCHIVE:3 440.0126%
>
> DISK:3,ARCHIVE:2 300.0086%
>
> DISK:3,ARCHIVE:1 90.0026%
> {code}
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7467) Provide storage tier information for a directory via fsck
[
https://issues.apache.org/jira/browse/HDFS-7467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14312484#comment-14312484
]
Benoy Antony commented on HDFS-7467:
The findbugs warning and test failure are unrelated.
> Provide storage tier information for a directory via fsck
> -
>
> Key: HDFS-7467
> URL: https://issues.apache.org/jira/browse/HDFS-7467
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: balancer & mover
>Affects Versions: 2.6.0
>Reporter: Benoy Antony
>Assignee: Benoy Antony
> Attachments: HDFS-7467-002.patch, HDFS-7467-003.patch,
> HDFS-7467.patch, storagepolicydisplay.pdf
>
>
> Currently _fsck_ provides information regarding blocks for a directory.
> It should be augmented to provide storage tier information (optionally).
> The sample report could be as follows :
> {code}
> Storage Tier Combination# of blocks % of blocks
> DISK:1,ARCHIVE:2 340730 97.7393%
>
> ARCHIVE:3 39281.1268%
>
> DISK:2,ARCHIVE:231220.8956%
>
> DISK:2,ARCHIVE:1 7480.2146%
>
> DISK:1,ARCHIVE:3 440.0126%
>
> DISK:3,ARCHIVE:2 300.0086%
>
> DISK:3,ARCHIVE:1 90.0026%
> {code}
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HDFS-7467) Provide storage tier information for a directory via fsck
[
https://issues.apache.org/jira/browse/HDFS-7467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14311609#comment-14311609
]
Benoy Antony edited comment on HDFS-7467 at 2/8/15 10:57 PM:
-
Attaching the patch which fixes the findbugs
was (Author: benoyantony):
Fixing the patch which fixes the findbugs
> Provide storage tier information for a directory via fsck
> -
>
> Key: HDFS-7467
> URL: https://issues.apache.org/jira/browse/HDFS-7467
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: balancer & mover
>Affects Versions: 2.6.0
>Reporter: Benoy Antony
>Assignee: Benoy Antony
> Attachments: HDFS-7467-002.patch, HDFS-7467-003.patch,
> HDFS-7467.patch, storagepolicydisplay.pdf
>
>
> Currently _fsck_ provides information regarding blocks for a directory.
> It should be augmented to provide storage tier information (optionally).
> The sample report could be as follows :
> {code}
> Storage Tier Combination# of blocks % of blocks
> DISK:1,ARCHIVE:2 340730 97.7393%
>
> ARCHIVE:3 39281.1268%
>
> DISK:2,ARCHIVE:231220.8956%
>
> DISK:2,ARCHIVE:1 7480.2146%
>
> DISK:1,ARCHIVE:3 440.0126%
>
> DISK:3,ARCHIVE:2 300.0086%
>
> DISK:3,ARCHIVE:1 90.0026%
> {code}
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7467) Provide storage tier information for a directory via fsck
[
https://issues.apache.org/jira/browse/HDFS-7467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Benoy Antony updated HDFS-7467:
---
Attachment: HDFS-7467-003.patch
Fixing the patch which fixes the findbugs
> Provide storage tier information for a directory via fsck
> -
>
> Key: HDFS-7467
> URL: https://issues.apache.org/jira/browse/HDFS-7467
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: balancer & mover
>Affects Versions: 2.6.0
>Reporter: Benoy Antony
>Assignee: Benoy Antony
> Attachments: HDFS-7467-002.patch, HDFS-7467-003.patch,
> HDFS-7467.patch, storagepolicydisplay.pdf
>
>
> Currently _fsck_ provides information regarding blocks for a directory.
> It should be augmented to provide storage tier information (optionally).
> The sample report could be as follows :
> {code}
> Storage Tier Combination# of blocks % of blocks
> DISK:1,ARCHIVE:2 340730 97.7393%
>
> ARCHIVE:3 39281.1268%
>
> DISK:2,ARCHIVE:231220.8956%
>
> DISK:2,ARCHIVE:1 7480.2146%
>
> DISK:1,ARCHIVE:3 440.0126%
>
> DISK:3,ARCHIVE:2 300.0086%
>
> DISK:3,ARCHIVE:1 90.0026%
> {code}
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7467) Provide storage tier information for a directory via fsck
[
https://issues.apache.org/jira/browse/HDFS-7467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Benoy Antony updated HDFS-7467:
---
Status: Patch Available (was: In Progress)
> Provide storage tier information for a directory via fsck
> -
>
> Key: HDFS-7467
> URL: https://issues.apache.org/jira/browse/HDFS-7467
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: balancer & mover
>Affects Versions: 2.6.0
>Reporter: Benoy Antony
>Assignee: Benoy Antony
> Attachments: HDFS-7467-002.patch, HDFS-7467.patch,
> storagepolicydisplay.pdf
>
>
> Currently _fsck_ provides information regarding blocks for a directory.
> It should be augmented to provide storage tier information (optionally).
> The sample report could be as follows :
> {code}
> Storage Tier Combination# of blocks % of blocks
> DISK:1,ARCHIVE:2 340730 97.7393%
>
> ARCHIVE:3 39281.1268%
>
> DISK:2,ARCHIVE:231220.8956%
>
> DISK:2,ARCHIVE:1 7480.2146%
>
> DISK:1,ARCHIVE:3 440.0126%
>
> DISK:3,ARCHIVE:2 300.0086%
>
> DISK:3,ARCHIVE:1 90.0026%
> {code}
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7467) Provide storage tier information for a directory via fsck
[
https://issues.apache.org/jira/browse/HDFS-7467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Benoy Antony updated HDFS-7467:
---
Attachment: HDFS-7467-002.patch
Attaching the patch which provides the _storagepolicy_ information via _fsck_.
The above comments are considered.
Sample outputs are shown the pdf attached.
> Provide storage tier information for a directory via fsck
> -
>
> Key: HDFS-7467
> URL: https://issues.apache.org/jira/browse/HDFS-7467
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: balancer & mover
>Affects Versions: 2.6.0
>Reporter: Benoy Antony
>Assignee: Benoy Antony
> Attachments: HDFS-7467-002.patch, HDFS-7467.patch,
> storagepolicydisplay.pdf
>
>
> Currently _fsck_ provides information regarding blocks for a directory.
> It should be augmented to provide storage tier information (optionally).
> The sample report could be as follows :
> {code}
> Storage Tier Combination# of blocks % of blocks
> DISK:1,ARCHIVE:2 340730 97.7393%
>
> ARCHIVE:3 39281.1268%
>
> DISK:2,ARCHIVE:231220.8956%
>
> DISK:2,ARCHIVE:1 7480.2146%
>
> DISK:1,ARCHIVE:3 440.0126%
>
> DISK:3,ARCHIVE:2 300.0086%
>
> DISK:3,ARCHIVE:1 90.0026%
> {code}
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7467) Provide storage tier information for a directory via fsck
[
https://issues.apache.org/jira/browse/HDFS-7467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Benoy Antony updated HDFS-7467:
---
Attachment: storagepolicydisplay.pdf
Attaching the document containing sample fsck outputs when invoked with
_storagepolicies_ option
> Provide storage tier information for a directory via fsck
> -
>
> Key: HDFS-7467
> URL: https://issues.apache.org/jira/browse/HDFS-7467
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: balancer & mover
>Affects Versions: 2.6.0
>Reporter: Benoy Antony
>Assignee: Benoy Antony
> Attachments: HDFS-7467.patch, storagepolicydisplay.pdf
>
>
> Currently _fsck_ provides information regarding blocks for a directory.
> It should be augmented to provide storage tier information (optionally).
> The sample report could be as follows :
> {code}
> Storage Tier Combination# of blocks % of blocks
> DISK:1,ARCHIVE:2 340730 97.7393%
>
> ARCHIVE:3 39281.1268%
>
> DISK:2,ARCHIVE:231220.8956%
>
> DISK:2,ARCHIVE:1 7480.2146%
>
> DISK:1,ARCHIVE:3 440.0126%
>
> DISK:3,ARCHIVE:2 300.0086%
>
> DISK:3,ARCHIVE:1 90.0026%
> {code}
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6133) Make Balancer support exclude specified path
[ https://issues.apache.org/jira/browse/HDFS-6133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14310564#comment-14310564 ] Benoy Antony commented on HDFS-6133: I know that there are different ways to accomplish the desired functionality of pinning the blocks under a directory to a set of nodes. I want to suggest yet another approach using storage policies. This is based on the fact that the set of favored nodes can be considered as a tier. Let’s say blocks under /hbase needs to be pinned to a set of nodes. 1. Mark the data.dirs on those set of datanodes as a special storage type , say HBASE_TIER. [HBASE_TIER]/hadoop01/data 2. Define a storage policy - HBASE_POLICY which insists that all blocks following the policy, reside in HBASE_TIER 3. Set storage policy of /hbase to HBASE_POLICY Since the balancer(and mover) considers the storage policy while moving/balancing, balancer will not move these blocks out of the favored nodes(HBASE_TIER) Now if we want to keep other blocks ( non hbase blocks) on these datanodes as well(may not be a great idea), then we can model it by having multiple storage types on a single directory like [HBASE_TIER,DISK]/hadoop01/data. This make sense since one want to have blocks having multiple storage characteristics on the same node. This will need a (minor) code modification. > Make Balancer support exclude specified path > > > Key: HDFS-6133 > URL: https://issues.apache.org/jira/browse/HDFS-6133 > Project: Hadoop HDFS > Issue Type: Improvement > Components: balancer & mover, namenode >Reporter: zhaoyunjiong >Assignee: zhaoyunjiong > Attachments: HDFS-6133-1.patch, HDFS-6133-2.patch, HDFS-6133-3.patch, > HDFS-6133-4.patch, HDFS-6133-5.patch, HDFS-6133-6.patch, HDFS-6133-7.patch, > HDFS-6133-8.patch, HDFS-6133.patch > > > Currently, run Balancer will destroying Regionserver's data locality. > If getBlocks could exclude blocks belongs to files which have specific path > prefix, like "/hbase", then we can run Balancer without destroying > Regionserver's data locality. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7467) Provide storage tier information for a directory via fsck
[
https://issues.apache.org/jira/browse/HDFS-7467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14268598#comment-14268598
]
Benoy Antony commented on HDFS-7467:
{quote}
If a file does not satisfies the specified policy, fsck should show such
information.
{quote}
That's good for a file. What do we do for directories which can potentially
have children with different policies ?
> Provide storage tier information for a directory via fsck
> -
>
> Key: HDFS-7467
> URL: https://issues.apache.org/jira/browse/HDFS-7467
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: balancer & mover
>Affects Versions: 2.6.0
>Reporter: Benoy Antony
>Assignee: Benoy Antony
> Attachments: HDFS-7467.patch
>
>
> Currently _fsck_ provides information regarding blocks for a directory.
> It should be augmented to provide storage tier information (optionally).
> The sample report could be as follows :
> {code}
> Storage Tier Combination# of blocks % of blocks
> DISK:1,ARCHIVE:2 340730 97.7393%
>
> ARCHIVE:3 39281.1268%
>
> DISK:2,ARCHIVE:231220.8956%
>
> DISK:2,ARCHIVE:1 7480.2146%
>
> DISK:1,ARCHIVE:3 440.0126%
>
> DISK:3,ARCHIVE:2 300.0086%
>
> DISK:3,ARCHIVE:1 90.0026%
> {code}
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HDFS-7467) Provide storage tier information for a directory via fsck
[
https://issues.apache.org/jira/browse/HDFS-7467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14266871#comment-14266871
]
Benoy Antony edited comment on HDFS-7467 at 1/6/15 10:22 PM:
-
1.
{quote}
Are all storage policies in fallback storage equivalent to other storage
policies that this output can always be fully described by the percentages that
Tsz has suggested?
{quote}
There is a possibility that some storage tier combination may not belong to a
storage policy.
My recommendation is to display the policy along with the combination if
possible. If not, display the combination. Lowercase for policy name is
intentional.
{code}
Storage Policy # of blocks % of blocks
cold(DISK:1,ARCHIVE:2) 340730 97.7393%
frozen(ARCHIVE:3) 39281.1268%
DISK:2,ARCHIVE:2 31220.8956%
warm(DISK:2,ARCHIVE:1) 7480.2146%
DISK:1,ARCHIVE:3440.0126%
DISK:3,ARCHIVE:2300.0086%
DISK:3,ARCHIVE:1 90.0026%
{code}
2.
{quote}
There should also be some warning messages as well in fsck for all files that
are unable to meet the requested ideal for their storage policy and are using
fallback storage, perhaps with a switch since that could become overly volumous
output.
{quote}
This is a nice feature. Will look into that .
was (Author: benoyantony):
1.
{quote}
Are all storage policies in fallback storage equivalent to other storage
policies that this output can always be fully described by the percentages that
Tsz has suggested?
{quote}
There is a possibility that some storage tier combination may not belong to a
storage policy.
My recommendation is to display the policy along with the combination if
possible. If not, display the combination. Lowercase for policy name is
intentional.
{code}
Storage Policy # of blocks % of blocks
cold(DISK:1,ARCHIVE:2) 340730 97.7393%
frozen(ARCHIVE:3) 39281.1268%
DISK:2,ARCHIVE:2 31220.8956%
warm(DISK:2,ARCHIVE:1) 7480.2146%
DISK:1,ARCHIVE:3 440.0126%
DISK:3,ARCHIVE:2 300.0086%
DISK:3,ARCHIVE:190.0026%
{code}
2.
{quote}
There should also be some warning messages as well in fsck for all files that
are unable to meet the requested ideal for their storage policy and are using
fallback storage, perhaps with a switch since that could become overly volumous
output.
{quote}
This is a nice feature. Will look into that .
> Provide storage tier information for a directory via fsck
> -
>
> Key: HDFS-7467
> URL: https://issues.apache.org/jira/browse/HDFS-7467
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: balancer & mover
>Affects Versions: 2.6.0
>Reporter: Benoy Antony
>Assignee: Benoy Antony
> Attachments: HDFS-7467.patch
>
>
> Currently _fsck_ provides information regarding blocks for a directory.
> It should be augmented to provide storage tier information (optionally).
> The sample report could be as follows :
> {code}
> Storage Tier Combination# of blocks % of blocks
> DISK:1,ARCHIVE:2 340730 97.7393%
>
> ARCHIVE:3 39281.1268%
>
> DISK:2,ARCHIVE:231220.8956%
>
> DISK:2,ARCHIVE:1 7480.2146%
>
> DISK:1,ARCHIVE:3 440.0126%
>
> DISK:3,ARCHIVE:2 300.0086%
>
> DISK:3,ARCHIVE:1 90.0026%
> {code}
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HDFS-7467) Provide storage tier information for a directory via fsck
[
https://issues.apache.org/jira/browse/HDFS-7467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14266871#comment-14266871
]
Benoy Antony edited comment on HDFS-7467 at 1/6/15 10:18 PM:
-
1.
{quote}
Are all storage policies in fallback storage equivalent to other storage
policies that this output can always be fully described by the percentages that
Tsz has suggested?
{quote}
There is a possibility that some storage tier combination may not belong to a
storage policy.
My recommendation is to display the policy along with the combination if
possible. If not, display the combination. Lowercase for policy name is
intentional.
{code}
Storage Policy # of blocks % of blocks
cold(DISK:1,ARCHIVE:2) 340730 97.7393%
frozen(ARCHIVE:3) 39281.1268%
DISK:2,ARCHIVE:2 31220.8956%
warm(DISK:2,ARCHIVE:1) 7480.2146%
DISK:1,ARCHIVE:3 440.0126%
DISK:3,ARCHIVE:2 300.0086%
DISK:3,ARCHIVE:190.0026%
{code}
2.
{quote}
There should also be some warning messages as well in fsck for all files that
are unable to meet the requested ideal for their storage policy and are using
fallback storage, perhaps with a switch since that could become overly volumous
output.
{quote}
This is a nice feature. Will look into that .
was (Author: benoyantony):
1.
{quote}
Are all storage policies in fallback storage equivalent to other storage
policies that this output can always be fully described by the percentages that
Tsz has suggested?
{quote}
There is a possibility that some storage tier combination may not belong to a
storage policy.
My recommendation is to display the policy along with the combination if
possible. If not, display the combination. Lowercase for policy name is
intentional.
{code}
Storage Policy # of blocks % of blocks
cold(DISK:1,ARCHIVE:2) 340730 97.7393%
frozen(ARCHIVE:3) 39281.1268%
DISK:2,ARCHIVE:2 31220.8956%
warm(DISK:2,ARCHIVE:1) 7480.2146%
DISK:1,ARCHIVE:3 440.0126%
DISK:3,ARCHIVE:2 300.0086%
DISK:3,ARCHIVE:190.0026%
{code}
2.
{quote}
There should also be some warning messages as well in fsck for all files that
are unable to meet the requested ideal for their storage policy and are using
fallback storage, perhaps with a switch since that could become overly volumous
output.
{quote}
This is a nice feature. Will look into that .
> Provide storage tier information for a directory via fsck
> -
>
> Key: HDFS-7467
> URL: https://issues.apache.org/jira/browse/HDFS-7467
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: balancer & mover
>Affects Versions: 2.6.0
>Reporter: Benoy Antony
>Assignee: Benoy Antony
> Attachments: HDFS-7467.patch
>
>
> Currently _fsck_ provides information regarding blocks for a directory.
> It should be augmented to provide storage tier information (optionally).
> The sample report could be as follows :
> {code}
> Storage Tier Combination# of blocks % of blocks
> DISK:1,ARCHIVE:2 340730 97.7393%
>
> ARCHIVE:3 39281.1268%
>
> DISK:2,ARCHIVE:231220.8956%
>
> DISK:2,ARCHIVE:1 7480.2146%
>
> DISK:1,ARCHIVE:3 440.0126%
>
> DISK:3,ARCHIVE:2 300.0086%
>
> DISK:3,ARCHIVE:1 90.0026%
> {code}
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HDFS-7467) Provide storage tier information for a directory via fsck
[
https://issues.apache.org/jira/browse/HDFS-7467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14266871#comment-14266871
]
Benoy Antony edited comment on HDFS-7467 at 1/6/15 10:17 PM:
-
1.
{quote}
Are all storage policies in fallback storage equivalent to other storage
policies that this output can always be fully described by the percentages that
Tsz has suggested?
{quote}
There is a possibility that some storage tier combination may not belong to a
storage policy.
My recommendation is to display the policy along with the combination if
possible. If not, display the combination. Lowercase for policy name is
intentional.
{code}
Storage Policy # of blocks % of blocks
cold(DISK:1,ARCHIVE:2) 340730 97.7393%
frozen(ARCHIVE:3) 39281.1268%
DISK:2,ARCHIVE:2 31220.8956%
warm(DISK:2,ARCHIVE:1) 7480.2146%
DISK:1,ARCHIVE:3 440.0126%
DISK:3,ARCHIVE:2 300.0086%
DISK:3,ARCHIVE:190.0026%
{code}
2.
{quote}
There should also be some warning messages as well in fsck for all files that
are unable to meet the requested ideal for their storage policy and are using
fallback storage, perhaps with a switch since that could become overly volumous
output.
{quote}
This is a nice feature. Will look into that .
was (Author: benoyantony):
1.
{quote}
Are all storage policies in fallback storage equivalent to other storage
policies that this output can always be fully described by the percentages that
Tsz has suggested?
{quote}
There is a possibility that some storage tier combination may not belong to a
storage policy.
My recommendation is to display the policy along with the combination if
possible. If not, display the combination. Lowercase for policy name is
intentional.
{code}
Storage Policy # of blocks % of blocks
cold (DISK:1,ARCHIVE:2)340730 97.7393%
frozen (ARCHIVE:3) 39281.1268%
DISK:2,ARCHIVE:2 31220.8956%
warm (DISK:2,ARCHIVE:1)7480.2146%
DISK:1,ARCHIVE:3 440.0126%
DISK:3,ARCHIVE:2 300.0086%
DISK:3,ARCHIVE:190.0026%
{code}
2.
{quote}
There should also be some warning messages as well in fsck for all files that
are unable to meet the requested ideal for their storage policy and are using
fallback storage, perhaps with a switch since that could become overly volumous
output.
{quote}
This is a nice feature. Will look into that .
> Provide storage tier information for a directory via fsck
> -
>
> Key: HDFS-7467
> URL: https://issues.apache.org/jira/browse/HDFS-7467
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: balancer & mover
>Affects Versions: 2.6.0
>Reporter: Benoy Antony
>Assignee: Benoy Antony
> Attachments: HDFS-7467.patch
>
>
> Currently _fsck_ provides information regarding blocks for a directory.
> It should be augmented to provide storage tier information (optionally).
> The sample report could be as follows :
> {code}
> Storage Tier Combination# of blocks % of blocks
> DISK:1,ARCHIVE:2 340730 97.7393%
>
> ARCHIVE:3 39281.1268%
>
> DISK:2,ARCHIVE:231220.8956%
>
> DISK:2,ARCHIVE:1 7480.2146%
>
> DISK:1,ARCHIVE:3 440.0126%
>
> DISK:3,ARCHIVE:2 300.0086%
>
> DISK:3,ARCHIVE:1 90.0026%
> {code}
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HDFS-7467) Provide storage tier information for a directory via fsck
[
https://issues.apache.org/jira/browse/HDFS-7467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14266871#comment-14266871
]
Benoy Antony edited comment on HDFS-7467 at 1/6/15 10:16 PM:
-
1.
{quote}
Are all storage policies in fallback storage equivalent to other storage
policies that this output can always be fully described by the percentages that
Tsz has suggested?
{quote}
There is a possibility that some storage tier combination may not belong to a
storage policy.
My recommendation is to display the policy along with the combination if
possible. If not, display the combination. Lowercase for policy name is
intentional.
{code}
Storage Policy # of blocks % of blocks
cold (DISK:1,ARCHIVE:2)340730 97.7393%
frozen (ARCHIVE:3) 39281.1268%
DISK:2,ARCHIVE:2 31220.8956%
warm (DISK:2,ARCHIVE:1)7480.2146%
DISK:1,ARCHIVE:3 440.0126%
DISK:3,ARCHIVE:2 300.0086%
DISK:3,ARCHIVE:190.0026%
{code}
2.
{quote}
There should also be some warning messages as well in fsck for all files that
are unable to meet the requested ideal for their storage policy and are using
fallback storage, perhaps with a switch since that could become overly volumous
output.
{quote}
This is a nice feature. Will look into that .
was (Author: benoyantony):
1.
{quote}
Are all storage policies in fallback storage equivalent to other storage
policies that this output can always be fully described by the percentages that
Tsz has suggested?
{quote}
There is a possibility that some storage tier combination may not belong to a
storage policy.
My recommendation is to display the policy along with the combination if
possible. If not, display the combination. Lowercase for policy name is
intentional.
{code}
Storage Policy # of blocks % of blocks
cold (DISK:1,ARCHIVE:2) 340730 97.7393%
frozen (ARCHIVE:3) 39281.1268%
DISK:2,ARCHIVE:231220.8956%
warm (DISK:2,ARCHIVE:1) 7480.2146%
DISK:1,ARCHIVE:3 440.0126%
DISK:3,ARCHIVE:2 300.0086%
DISK:3,ARCHIVE:1 90.0026%
{code}
2.
{quote}
There should also be some warning messages as well in fsck for all files that
are unable to meet the requested ideal for their storage policy and are using
fallback storage, perhaps with a switch since that could become overly volumous
output.
{quote}
This is a nice feature. Will look into that .
> Provide storage tier information for a directory via fsck
> -
>
> Key: HDFS-7467
> URL: https://issues.apache.org/jira/browse/HDFS-7467
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: balancer & mover
>Affects Versions: 2.6.0
>Reporter: Benoy Antony
>Assignee: Benoy Antony
> Attachments: HDFS-7467.patch
>
>
> Currently _fsck_ provides information regarding blocks for a directory.
> It should be augmented to provide storage tier information (optionally).
> The sample report could be as follows :
> {code}
> Storage Tier Combination# of blocks % of blocks
> DISK:1,ARCHIVE:2 340730 97.7393%
>
> ARCHIVE:3 39281.1268%
>
> DISK:2,ARCHIVE:231220.8956%
>
> DISK:2,ARCHIVE:1 7480.2146%
>
> DISK:1,ARCHIVE:3 440.0126%
>
> DISK:3,ARCHIVE:2 300.0086%
>
> DISK:3,ARCHIVE:1 90.0026%
> {code}
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7467) Provide storage tier information for a directory via fsck
[
https://issues.apache.org/jira/browse/HDFS-7467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14266871#comment-14266871
]
Benoy Antony commented on HDFS-7467:
1.
{quote}
Are all storage policies in fallback storage equivalent to other storage
policies that this output can always be fully described by the percentages that
Tsz has suggested?
{quote}
There is a possibility that some storage tier combination may not belong to a
storage policy.
My recommendation is to display the policy along with the combination if
possible. If not, display the combination. Lowercase for policy name is
intentional.
{code}
Storage Policy # of blocks % of blocks
cold (DISK:1,ARCHIVE:2) 340730 97.7393%
frozen (ARCHIVE:3) 39281.1268%
DISK:2,ARCHIVE:231220.8956%
warm (DISK:2,ARCHIVE:1) 7480.2146%
DISK:1,ARCHIVE:3 440.0126%
DISK:3,ARCHIVE:2 300.0086%
DISK:3,ARCHIVE:1 90.0026%
{code}
2.
{quote}
There should also be some warning messages as well in fsck for all files that
are unable to meet the requested ideal for their storage policy and are using
fallback storage, perhaps with a switch since that could become overly volumous
output.
{quote}
This is a nice feature. Will look into that .
> Provide storage tier information for a directory via fsck
> -
>
> Key: HDFS-7467
> URL: https://issues.apache.org/jira/browse/HDFS-7467
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: balancer & mover
>Affects Versions: 2.6.0
>Reporter: Benoy Antony
>Assignee: Benoy Antony
> Attachments: HDFS-7467.patch
>
>
> Currently _fsck_ provides information regarding blocks for a directory.
> It should be augmented to provide storage tier information (optionally).
> The sample report could be as follows :
> {code}
> Storage Tier Combination# of blocks % of blocks
> DISK:1,ARCHIVE:2 340730 97.7393%
>
> ARCHIVE:3 39281.1268%
>
> DISK:2,ARCHIVE:231220.8956%
>
> DISK:2,ARCHIVE:1 7480.2146%
>
> DISK:1,ARCHIVE:3 440.0126%
>
> DISK:3,ARCHIVE:2 300.0086%
>
> DISK:3,ARCHIVE:1 90.0026%
> {code}
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
