[jira] [Commented] (HDFS-8241) Remove unused Namenode startup option FINALIZE
[ https://issues.apache.org/jira/browse/HDFS-8241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14512569#comment-14512569 ] Charles Wimmer commented on HDFS-8241: -- A fatal error would obviously be prefereable to a silent failure. It is also no longer obvious when a filesystem requires finalization. (I'll skip my rant about how HDFS-5138 was probably the cause for me losing blocks on an upgrade from 2.2 to 2.4.1) Remove unused Namenode startup option FINALIZE - Key: HDFS-8241 URL: https://issues.apache.org/jira/browse/HDFS-8241 Project: Hadoop HDFS Issue Type: Bug Components: namenode Affects Versions: 2.7.0 Reporter: Brahma Reddy Battula Assignee: Brahma Reddy Battula Command : hdfs namenode -finalize 15/04/24 22:26:23 INFO namenode.NameNode: createNameNode [-finalize] *Use of the argument 'FINALIZE' is no longer supported.* To finalize an upgrade, start the NN and then run `hdfs dfsadmin -finalizeUpgrade' -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6075) Introducing non-replication mode
[
https://issues.apache.org/jira/browse/HDFS-6075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13925274#comment-13925274
]
Charles Wimmer commented on HDFS-6075:
--
dfs.datanode.balance.bandwidthPerSec may be set dynamically while the cluster
is running. We requested this feature for exactly the type of operational
situation you describe. You may not be able to eliminate replication, but you
can minimize the impact by temporarily setting the bandwidth extremely low.
From hdfs dfsadmin -help:
{noformat}
-setBalancerBandwidth bandwidth:
Changes the network bandwidth used by each datanode during
HDFS block balancing.
bandwidth is the maximum number of bytes per second
that will be used by each datanode. This value overrides
the dfs.balance.bandwidthPerSec parameter.
--- NOTE: The new value is not persistent on the DataNode.---
{noformat}
Introducing non-replication mode
--
Key: HDFS-6075
URL: https://issues.apache.org/jira/browse/HDFS-6075
Project: Hadoop HDFS
Issue Type: New Feature
Components: datanode, namenode
Reporter: Adam Kawa
Priority: Minor
Afaik, HDFS does not provide an easy way to temporarily disable the
replication of missing blocks.
If you would like to temporarily disable the replication, you would have to
* set dfs.namenode.replication.interval (_The periodicity in seconds with
which the namenode computes repliaction work for datanodes_ Default 3) to
something very high. *Disadvantage*: you have to restart the NN
* go into the safe-mode. *Disadvantage*: all write operations will fail
We have the situation that we need to replace our top-of-rack switches for
each rack. Replacing a switch should take around 30 minutes. Each rack has
around 0.6 PB of data. We would like to avoid an expensive replication, since
we know that we will put this rack online quickly. To avoid any downtime, or
excessive network transfer, we think that temporarily disabling the
replication could fit us.
The default block placement policy puts blocks into two racks, so when one
rack temporarily goes offline, we still have an access to at least replica of
each block. Of course, if we lose this replica, then we would have to wait
until the rack goes back online. This is what the administrator should be
aware of.
This feature could disable the replication
* globally - for a whole cluster
* partially - e.g. only for missing blocks that come from a specified set of
DataNodes. So a file like we_will_be_back_soon :) could be introduced,
similar to include and exclude.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (HDFS-2203) Add additional fields to the JMX output on NameNode
Add additional fields to the JMX output on NameNode --- Key: HDFS-2203 URL: https://issues.apache.org/jira/browse/HDFS-2203 Project: Hadoop HDFS Issue Type: Improvement Components: name-node Affects Versions: 0.20.204.0, 0.20.205.0, 0.23.0 Reporter: Charles Wimmer Priority: Minor When accessing the JMX data via http (http://axoniteblue-nn1.blue.ygrid.yahoo.com:50070/jmx) there are a couple of useful fields missing from this bean: name : Hadoop:service=NameNode,name=NameNodeInfo, modelerType : org.apache.hadoop.hdfs.server.namenode.FSNamesystem, Please add the number of blocks and the configured capacity. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2203) Add additional fields to the JMX output on NameNode
[ https://issues.apache.org/jira/browse/HDFS-2203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Charles Wimmer updated HDFS-2203: - Description: When accessing the JMX data via http (http://namenode:50070/jmx) there are a couple of useful fields missing from this bean: name : Hadoop:service=NameNode,name=NameNodeInfo, modelerType : org.apache.hadoop.hdfs.server.namenode.FSNamesystem, Please add the number of blocks and the configured capacity. was: When accessing the JMX data via http (http://axoniteblue-nn1.blue.ygrid.yahoo.com:50070/jmx) there are a couple of useful fields missing from this bean: name : Hadoop:service=NameNode,name=NameNodeInfo, modelerType : org.apache.hadoop.hdfs.server.namenode.FSNamesystem, Please add the number of blocks and the configured capacity. Add additional fields to the JMX output on NameNode --- Key: HDFS-2203 URL: https://issues.apache.org/jira/browse/HDFS-2203 Project: Hadoop HDFS Issue Type: Improvement Components: name-node Affects Versions: 0.20.204.0, 0.20.205.0, 0.23.0 Reporter: Charles Wimmer Priority: Minor When accessing the JMX data via http (http://namenode:50070/jmx) there are a couple of useful fields missing from this bean: name : Hadoop:service=NameNode,name=NameNodeInfo, modelerType : org.apache.hadoop.hdfs.server.namenode.FSNamesystem, Please add the number of blocks and the configured capacity. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2203) Add additional fields to the JMX output on NameNode
[
https://issues.apache.org/jira/browse/HDFS-2203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13071293#comment-13071293
]
Charles Wimmer commented on HDFS-2203:
--
I meant per DataNode blocks and configured capacity:
LiveNodes:
{hostA:{usedSpace:8264583897088,lastContact:0},hostB:{usedSpace:7914785189533,lastContact:0},hostC:{usedSpace:4715591979008,lastContact:2},hostD:{usedSpace:6168919373255,lastContact:1},hostE:{usedSpace:8315075813376,lastContact:1},hostF:
snip
hostY:{usedSpace:8272411804419,lastContact:1},hostZ:{usedSpace:8359994176231,lastContact:0}},
I would like the same data that shows up on the live nodes page:
http://namenode:50070/dfsnodelist.jsp?whatNodes=LIVE
Add additional fields to the JMX output on NameNode
---
Key: HDFS-2203
URL: https://issues.apache.org/jira/browse/HDFS-2203
Project: Hadoop HDFS
Issue Type: Improvement
Components: name-node
Affects Versions: 0.20.204.0, 0.20.205.0, 0.23.0
Reporter: Charles Wimmer
Priority: Minor
Labels: newbie
When accessing the JMX data via http (http://namenode:50070/jmx) there are a
couple of useful fields missing from this bean:
name : Hadoop:service=NameNode,name=NameNodeInfo,
modelerType : org.apache.hadoop.hdfs.server.namenode.FSNamesystem,
Please add the number of blocks and the configured capacity.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
