[jira] [Commented] (HDFS-7789) DFsck should resolve the path to support cross-FS symlinks
[ https://issues.apache.org/jira/browse/HDFS-7789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14339079#comment-14339079 ] Lohit Vijayarenu commented on HDFS-7789: Looks good to me. +1 > DFsck should resolve the path to support cross-FS symlinks > -- > > Key: HDFS-7789 > URL: https://issues.apache.org/jira/browse/HDFS-7789 > Project: Hadoop HDFS > Issue Type: Improvement > Components: tools >Affects Versions: 2.6.0 >Reporter: Gera Shegalov >Assignee: Gera Shegalov > Attachments: HDFS-7789.001.patch > > > DFsck should resolve the specified path such that it can be used in with > viewfs and other cross-filesystem symlinks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7281) Missing block is marked as corrupted block
[
https://issues.apache.org/jira/browse/HDFS-7281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14238143#comment-14238143
]
Lohit Vijayarenu commented on HDFS-7281:
Hi [~mingma], These changes look good. This changes format of fsck output,
which should be ok to change (unless someone scrapes output of fsck command).
May be mark it as incompatible in release notes?
> Missing block is marked as corrupted block
> --
>
> Key: HDFS-7281
> URL: https://issues.apache.org/jira/browse/HDFS-7281
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Ming Ma
>Assignee: Ming Ma
> Labels: supportability
> Attachments: HDFS-7281-2.patch, HDFS-7281.patch
>
>
> In the situation where the block lost all its replicas, fsck shows the block
> is missing as well as corrupted. Perhaps it is better not to mark the block
> corrupted in this case. The reason it is marked as corrupted is
> numCorruptNodes == numNodes == 0 in the following code.
> {noformat}
> BlockManager
> final boolean isCorrupt = numCorruptNodes == numNodes;
> {noformat}
> Would like to clarify if it is the intent to mark missing block as corrupted
> or it is just a bug.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7123) Run legacy fsimage checkpoint in parallel with PB fsimage checkpoint
[ https://issues.apache.org/jira/browse/HDFS-7123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14145065#comment-14145065 ] Lohit Vijayarenu commented on HDFS-7123: If we drop the lock then users of these images will have to have the knowledge that both are of them could be out of sync. Parallel checkpoint takes more CPU, but that that point it is holding big lock where no other CPU intensive task is going on anyways. Changes look good. +1 on the patch. > Run legacy fsimage checkpoint in parallel with PB fsimage checkpoint > > > Key: HDFS-7123 > URL: https://issues.apache.org/jira/browse/HDFS-7123 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Ming Ma >Assignee: Ming Ma > Attachments: HDFS-7123.patch > > > HDFS-7097 will address the checkpoint and BR issue. In addition, it might > still be useful to reduce the overall checkpoint duration, given it blocks > edit log replay. If there is large volume of edit log to catch up and NN fail > overs, it will impact the availability. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6982) nntop: top-like tool for name node users
[ https://issues.apache.org/jira/browse/HDFS-6982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14132282#comment-14132282 ] Lohit Vijayarenu commented on HDFS-6982: To add to what [~maysamyabandeh] already added, our initial though was that if we can aggregate this kind of information within NameNode, then we could do throttling kind of feature on per user/rpc basis. This would be a feedback tool for auto-tuning within NameNode. There was also thoughts around having hot paths on similar lines which could be used by Replication manager possibly. So, having this within NameNode makes lot of sense. > nntop: top-like tool for name node users > - > > Key: HDFS-6982 > URL: https://issues.apache.org/jira/browse/HDFS-6982 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Maysam Yabandeh >Assignee: Maysam Yabandeh > Attachments: HDFS-6982.patch, HDFS-6982.v2.patch, nntop-design-v1.pdf > > > In this jira we motivate the need for nntop, a tool that, similarly to what > top does in Linux, gives the list of top users of the HDFS name node and > gives insight about which users are sending majority of each traffic type to > the name node. This information turns out to be the most critical when the > name node is under pressure and the HDFS admin needs to know which user is > hammering the name node and with what kind of requests. Here we present the > design of nntop which has been in production at Twitter in the past 10 > months. nntop proved to have low cpu overhead (< 2% in a cluster of 4K > nodes), low memory footprint (less than a few MB), and quite efficient for > the write path (only two hash lookup for updating a metric). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6469) Coordinated replication of the namespace using ConsensusNode
[ https://issues.apache.org/jira/browse/HDFS-6469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14020875#comment-14020875 ] Lohit Vijayarenu commented on HDFS-6469: Still reading design document. One question I had was about this bq. A CNode can fail or be taken down intentionally for maintenance. The other replicated CNodes will continue operating as long as they form a quorum sufficient for the Coordination Engine to generate agreements. If the quorum is lost, then the cluster will freeze waiting for the quorum to be restored. Can you explain little bit more about if CNodes really have to have a quorum. Can something of that be handled by Coordination Engine already? > Coordinated replication of the namespace using ConsensusNode > > > Key: HDFS-6469 > URL: https://issues.apache.org/jira/browse/HDFS-6469 > Project: Hadoop HDFS > Issue Type: New Feature > Components: namenode >Affects Versions: 3.0.0 >Reporter: Konstantin Shvachko >Assignee: Konstantin Shvachko > Attachments: CNodeDesign.pdf > > > This is a proposal to introduce ConsensusNode - an evolution of the NameNode, > which enables replication of the namespace on multiple nodes of an HDFS > cluster by means of a Coordination Engine. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6452) ConfiguredFailoverProxyProvider should randomize currentProxyIndex on initialization
[
https://issues.apache.org/jira/browse/HDFS-6452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14010660#comment-14010660
]
Lohit Vijayarenu commented on HDFS-6452:
On our clusters we are seeing about 70-75% of load is readonly (getFileInfo,
listStatus, getBlockLocation). With this we have been thinking about enabling
namenode stale reads. If we do that, then having clients pick random NameNode
would distribute clients across both NameNode. How about we have an option to
randomize which NN to talk to by the client. By default
ConfiguredFailoverProxyProvider would behave like today, but an option to
randomize would be useful.
> ConfiguredFailoverProxyProvider should randomize currentProxyIndex on
> initialization
>
>
> Key: HDFS-6452
> URL: https://issues.apache.org/jira/browse/HDFS-6452
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: ha, hdfs-client
>Affects Versions: 2.4.0
>Reporter: Gera Shegalov
>Assignee: Gera Shegalov
>
> We observe that the clients iterate proxies in the fixed order. Depending on
> the order of namenodes in dfs.ha.namenodes. (e.g. 'nn1,nn2') and
> the current standby (nn1), all the clients will hit nn1 first, and then
> failover to nn2. Chatting with [~lohit] we think we can simply select the
> initial value of {{currentProxyIndex}} randomly, and keep the logic of
> {{performFailover}} of iterating from left-to-right. This should halve the
> unnecessary load on standby NN.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6248) SNN crash during replay of FSEditLog of files inside directories having QuotaExceeded directories
[ https://issues.apache.org/jira/browse/HDFS-6248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13971484#comment-13971484 ] Lohit Vijayarenu commented on HDFS-6248: Thanks [~kihwal] > SNN crash during replay of FSEditLog of files inside directories having > QuotaExceeded directories > -- > > Key: HDFS-6248 > URL: https://issues.apache.org/jira/browse/HDFS-6248 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.0.6-alpha, 2.4.0 > Environment: NameNode HA setup with Active/Standby using QJM >Reporter: Lohit Vijayarenu > Attachments: HDFS-6248.1.patch > > > We are seeing cases when Secondary NameNode crashes without recovery when it > tries to replay edit log of files which are part of directories which have > exceeded Quota. While debugging we got stack trace but we are still trying to > reproduce this and wanted to note this to see if anyone else had seen this > issue already. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6248) SNN crash during replay of FSEditLog of files inside directories having QuotaExceeded directories
[ https://issues.apache.org/jira/browse/HDFS-6248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lohit Vijayarenu updated HDFS-6248: --- Attachment: HDFS-6248.1.patch Not verifying Quota when in standby could be way to solve this. Attaching potential patch for inputs from others. > SNN crash during replay of FSEditLog of files inside directories having > QuotaExceeded directories > -- > > Key: HDFS-6248 > URL: https://issues.apache.org/jira/browse/HDFS-6248 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.0.6-alpha, 2.4.0 > Environment: NameNode HA setup with Active/Standby using QJM >Reporter: Lohit Vijayarenu > Attachments: HDFS-6248.1.patch > > > We are seeing cases when Secondary NameNode crashes without recovery when it > tries to replay edit log of files which are part of directories which have > exceeded Quota. While debugging we got stack trace but we are still trying to > reproduce this and wanted to note this to see if anyone else had seen this > issue already. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6248) SNN crash during replay of FSEditLog of files inside directories having QuotaExceeded directories
[ https://issues.apache.org/jira/browse/HDFS-6248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13970372#comment-13970372 ] Lohit Vijayarenu commented on HDFS-6248: One corner case I suspect is QuotaCheck done FSDirectory::addChild in active vs standby namenodes. When a file is created by active namenode and synced to edits, active NN's quota check might be close to its max, by the time standby NN replays this edit log space quota could have increased because of other files in a directory and valid edit log might hit QuotaExceededException. I feel when Standby namenode replays edits, it should ignore quota check since it is already controlled by Active Namenode anyways. This should solve the race condition and prevent Standby namenode from crashing. What do other think about this approach? > SNN crash during replay of FSEditLog of files inside directories having > QuotaExceeded directories > -- > > Key: HDFS-6248 > URL: https://issues.apache.org/jira/browse/HDFS-6248 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.0.6-alpha, 2.4.0 > Environment: NameNode HA setup with Active/Standby using QJM >Reporter: Lohit Vijayarenu > > We are seeing cases when Secondary NameNode crashes without recovery when it > tries to replay edit log of files which are part of directories which have > exceeded Quota. While debugging we got stack trace but we are still trying to > reproduce this and wanted to note this to see if anyone else had seen this > issue already. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6248) SNN crash during replay of FSEditLog of files inside directories having QuotaExceeded directories
[
https://issues.apache.org/jira/browse/HDFS-6248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13970222#comment-13970222
]
Lohit Vijayarenu commented on HDFS-6248:
Looking at FSEditLogLoader::applyEditLogOp
{code}
// add to the file tree
newFile = (INodeFile)fsDir.unprotectedAddFile(
addCloseOp.path, addCloseOp.permissions,
replication, addCloseOp.mtime,
addCloseOp.atime, addCloseOp.blockSize,
true, addCloseOp.clientName, addCloseOp.clientMachine);
fsNamesys.leaseManager.addLease(addCloseOp.clientName, addCloseOp.path);
{code}
could return newFile as null because of QuotaExceededExcetion in
uprotectedAddFile
NPE exception happens further down in same function at
{code}
// Update the salient file attributes.
newFile.setAccessTime(addCloseOp.atime);
newFile.setModificationTimeForce(addCloseOp.mtime);
updateBlocks(fsDir, addCloseOp, newFile);
{code}
Even though stack trace points to line number of 2.0.6 release, I could not
find any changes in trunk source code.
> SNN crash during replay of FSEditLog of files inside directories having
> QuotaExceeded directories
> --
>
> Key: HDFS-6248
> URL: https://issues.apache.org/jira/browse/HDFS-6248
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.0.6-alpha, 2.4.0
> Environment: NameNode HA setup with Active/Standby using QJM
>Reporter: Lohit Vijayarenu
>
> We are seeing cases when Secondary NameNode crashes without recovery when it
> tries to replay edit log of files which are part of directories which have
> exceeded Quota. While debugging we got stack trace but we are still trying to
> reproduce this and wanted to note this to see if anyone else had seen this
> issue already.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6248) SNN crash during replay of FSEditLog of files inside directories having QuotaExceeded directories
[
https://issues.apache.org/jira/browse/HDFS-6248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13970218#comment-13970218
]
Lohit Vijayarenu commented on HDFS-6248:
Here is stack trace of SNN before crash.
{noformat}
2014-03-29 18:07:35,380 ERROR
org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader: Encountered exception
on operation AddOp [length=0, path=/user/foo/bar.txt, replication=3,
mtime=1396116335071, atime=1396116335071, blockSize=536870912, blocks=[],
permissions=foo:supergroup:rw-r--r--,
clientName=DFSClient_attempt_1395346107078_146938_m_41_1_1098354233_1,
clientMachine=10.10.10.10, opCode=OP_ADD, txid=487688396]
java.lang.NullPointerException
at
org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.applyEditLogOp(FSEditLogLoader.java:281)
at
org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:171)
at
org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:90)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:708)
at
org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.doTailEdits(EditLogTailer.java:227)
at
org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.doWork(EditLogTailer.java:321)
at
org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.access$200(EditLogTailer.java:279)
at
org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread$1.run(EditLogTailer.java:296)
at
org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:456)
at
org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.run(EditLogTailer.java:292)
2014-03-29 18:07:35,622 WARN org.apache.hadoop.hdfs.server.namenode.NameNode:
Quota violation in image for //user/foo (Namespace quota : 1445052 consumed :
1304943) (Diskspace quota : 219902322000 consumed : 2199023483200164).
2014-03-29 18:07:36,429 FATAL
org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer: Unknown error
encountered while tailing edits. Shutting down standby NN.
java.io.IOException: Failed to apply edit log operation AddOp [length=0,
path=/user/foo/bar.txt, replication=3, mtime=1396116335071,
atime=1396116335071, blockSize=536870912, blocks=[],
permissions=foo:supergroup:rw-r--r--,
clientName=DFSClient_attempt_1395346107078_146938_m_41_1_1098354233_1,
clientMachine=10.10.10.10, opCode=OP_ADD, txid=487688396]: error null
at
org.apache.hadoop.hdfs.server.namenode.MetaRecoveryContext.editLogLoaderPrompt(MetaRecoveryContext.java:94)
at
org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:174)
at
org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:90)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:708)
at
org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.doTailEdits(EditLogTailer.java:227)
at
org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.doWork(EditLogTailer.java:321)
at
org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.access$200(EditLogTailer.java:279)
at
org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread$1.run(EditLogTailer.java:296)
at
org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:456)
at
org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.run(EditLogTailer.java:292)
2014-03-29 18:07:36,431 INFO org.apache.hadoop.util.ExitUtil: Exiting with
status 1
2014-03-29 18:07:36,433 INFO org.apache.hadoop.hdfs.server.namenode.NameNode:
SHUTDOWN_MSG:
{noformat}
> SNN crash during replay of FSEditLog of files inside directories having
> QuotaExceeded directories
> --
>
> Key: HDFS-6248
> URL: https://issues.apache.org/jira/browse/HDFS-6248
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.0.6-alpha, 2.4.0
> Environment: NameNode HA setup with Active/Standby using QJM
>Reporter: Lohit Vijayarenu
>
> We are seeing cases when Secondary NameNode crashes without recovery when it
> tries to replay edit log of files which are part of directories which have
> exceeded Quota. While debugging we got stack trace but we are still trying to
> reproduce this and wanted to note this to see if anyone else had seen this
> issue already.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (HDFS-6248) SNN crash during replay of FSEditLog of files inside directories having QuotaExceeded directories
Lohit Vijayarenu created HDFS-6248: -- Summary: SNN crash during replay of FSEditLog of files inside directories having QuotaExceeded directories Key: HDFS-6248 URL: https://issues.apache.org/jira/browse/HDFS-6248 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.4.0, 2.0.6-alpha Environment: NameNode HA setup with Active/Standby using QJM Reporter: Lohit Vijayarenu We are seeing cases when Secondary NameNode crashes without recovery when it tries to replay edit log of files which are part of directories which have exceeded Quota. While debugging we got stack trace but we are still trying to reproduce this and wanted to note this to see if anyone else had seen this issue already. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-5442) Zero loss HDFS data replication for multiple datacenters
[ https://issues.apache.org/jira/browse/HDFS-5442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13849350#comment-13849350 ] Lohit Vijayarenu commented on HDFS-5442: Thanks for sharing the design document. This looks to be very good start. Few initial comments. It might be good to break up the work into two major features. 1. BlockAllocation policy for cross datacenter (which I understand is synchronous replication from design document) 2. Asynchronous replication This would give flexibility for users to chose either of the features based on their use case and infrastructure support. Few more very high level comments - There seems to be assumption of replication of entire namespace at few places. This might not be desirable in many cases. Enabling this feature per directory or list of directories would be very useful. - There seems to be assumption of primary cluster and secondary cluster. Can this be chained to having something A->B and B->C. Or even the use case of A->B or B->A. Calling out those with configuration options would be very useful for cluster admins. - Another place which would need more information is about primary cluster NN tracking datanode information from secondary cluster (via secondary cluster NN). This needs to be thought to see if this is really scalable. This I assume would mean DataNode would have globally unique identifies now. How are failures of DataNodes handles and communicated back to Primary NN. How are DataNodes allocated for reads. How is space accounted for within clusters. Unique block ids across different clusters and such. Having more details on them will be very useful. - Minor: It might be worth changing Primary/Secondary to Source/Destination cluster. It is little confusing when also thinking about Primary/Secondary NameNodes in same document. - Adding few cases of failure and recover would be useful. For example in synchronous replication, what happens when secondary cluster is slow or down. How would data be re-replicated. - How would ReplicationManager or changing replication of files work in general with this policy? > Zero loss HDFS data replication for multiple datacenters > > > Key: HDFS-5442 > URL: https://issues.apache.org/jira/browse/HDFS-5442 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Avik Dey > Attachments: Disaster Recovery Solution for Hadoop.pdf > > > Hadoop is architected to operate efficiently at scale for normal hardware > failures within a datacenter. Hadoop is not designed today to handle > datacenter failures. Although HDFS is not designed for nor deployed in > configurations spanning multiple datacenters, replicating data from one > location to another is common practice for disaster recovery and global > service availability. There are current solutions available for batch > replication using data copy/export tools. However, while providing some > backup capability for HDFS data, they do not provide the capability to > recover all your HDFS data from a datacenter failure and be up and running > again with a fully operational Hadoop cluster in another datacenter in a > matter of minutes. For disaster recovery from a datacenter failure, we should > provide a fully distributed, zero data loss, low latency, high throughput and > secure HDFS data replication solution for multiple datacenter setup. > Design and code for Phase-1 to follow soon. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HDFS-5499) Provide way to throttle per FileSystem read/write bandwidth
[ https://issues.apache.org/jira/browse/HDFS-5499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13819611#comment-13819611 ] Lohit Vijayarenu commented on HDFS-5499: [~wheat9] Thanks. Yes, that is one option as well. But concern of other hidden components getting throttled would still remain for this if this was set as a clusterwide config, right? > Provide way to throttle per FileSystem read/write bandwidth > --- > > Key: HDFS-5499 > URL: https://issues.apache.org/jira/browse/HDFS-5499 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Lohit Vijayarenu > Attachments: HDFS-5499.1.patch > > > In some cases it might be worth to throttle read/writer bandwidth on per JVM > basis so that clients do not spawn too many thread and start data movement > causing other JVMs to starve. Ability to throttle read/write bandwidth on per > FileSystem would help avoid such issues. > Challenge seems to be how well this can be fit into FileSystem code. If one > enables throttling around FileSystem APIs, then any hidden data transfer > within cluster using them might also be affected. Eg. copying job jar during > job submission, localizing resources for distributed cache and such. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HDFS-5499) Provide way to throttle per FileSystem read/write bandwidth
[ https://issues.apache.org/jira/browse/HDFS-5499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lohit Vijayarenu updated HDFS-5499: --- Attachment: HDFS-5499.1.patch We have been thinking something along this line in the patch, which can throttle HDFS read/write and Hftp reads. Posting here for any inputs from people who might have thought about this use case and what they think about this. > Provide way to throttle per FileSystem read/write bandwidth > --- > > Key: HDFS-5499 > URL: https://issues.apache.org/jira/browse/HDFS-5499 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Lohit Vijayarenu > Attachments: HDFS-5499.1.patch > > > In some cases it might be worth to throttle read/writer bandwidth on per JVM > basis so that clients do not spawn too many thread and start data movement > causing other JVMs to starve. Ability to throttle read/write bandwidth on per > FileSystem would help avoid such issues. > Challenge seems to be how well this can be fit into FileSystem code. If one > enables throttling around FileSystem APIs, then any hidden data transfer > within cluster using them might also be affected. Eg. copying job jar during > job submission, localizing resources for distributed cache and such. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (HDFS-5499) Provide way to throttle per FileSystem read/write bandwidth
Lohit Vijayarenu created HDFS-5499: -- Summary: Provide way to throttle per FileSystem read/write bandwidth Key: HDFS-5499 URL: https://issues.apache.org/jira/browse/HDFS-5499 Project: Hadoop HDFS Issue Type: Improvement Reporter: Lohit Vijayarenu In some cases it might be worth to throttle read/writer bandwidth on per JVM basis so that clients do not spawn too many thread and start data movement causing other JVMs to starve. Ability to throttle read/write bandwidth on per FileSystem would help avoid such issues. Challenge seems to be how well this can be fit into FileSystem code. If one enables throttling around FileSystem APIs, then any hidden data transfer within cluster using them might also be affected. Eg. copying job jar during job submission, localizing resources for distributed cache and such. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5250) Enable one url to browse the entire federated hdfs file system on dfsclusterhealth summary page
[ https://issues.apache.org/jira/browse/HDFS-5250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13776694#comment-13776694 ] Lohit Vijayarenu commented on HDFS-5250: Agreed that ViewFileSystem is client side mount, but that was a way of getting this solved. If that is really ugly, is there any other clean way to get this fixed? Many people use browse filesystem and it would be good to know if there is someone who have thought about this in a cleaner way. Have you thought about any higher level projects like Amabari implementing this in future may be? > Enable one url to browse the entire federated hdfs file system on > dfsclusterhealth summary page > --- > > Key: HDFS-5250 > URL: https://issues.apache.org/jira/browse/HDFS-5250 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Vrushali C > > Ideally, we should have one url to browse the entire federated file system on > the main dfsclusterhealth summary page along with the list of namenodes in > the cluster -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-5175) Provide clients a way to set IP header bits on connections
Lohit Vijayarenu created HDFS-5175: -- Summary: Provide clients a way to set IP header bits on connections Key: HDFS-5175 URL: https://issues.apache.org/jira/browse/HDFS-5175 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.0.5-alpha Reporter: Lohit Vijayarenu It would be very helpful if we had ability for clients to set IP headers when they make socket connections for data transfers. We were looking into setting up QoS using DSCP bit and saw that there is no easy way to let clients pass down a specific value when clients make connection to DataNode. As a quick fix we did something similar to io.file.buffer.size where client could pass down DSCP integer value and when DFSClient opens a stream, it could set the value on socket using setTrafficClass Opening this JIRA to get more inputs from others who have had experience and might have already thought about this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4924) Show NameNode state on dfsclusterhealth page
[ https://issues.apache.org/jira/browse/HDFS-4924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13689752#comment-13689752 ] Lohit Vijayarenu commented on HDFS-4924: [~cnauroth] I see dfshealth.jsp constructs that info, but not via MBean. Hence I did not reuse those function in dfsclusterhealth page which constructs everything via Http JXM output. [~atm] From what I see dfsclusterhealth does not have this info. dfshealth page has it for individual NN page, but not at cluster summary page. > Show NameNode state on dfsclusterhealth page > > > Key: HDFS-4924 > URL: https://issues.apache.org/jira/browse/HDFS-4924 > Project: Hadoop HDFS > Issue Type: Improvement > Components: federation >Affects Versions: 2.1.0-beta >Reporter: Lohit Vijayarenu >Priority: Minor > Attachments: HDFS-4924.trunk.1.patch > > > dfsclusterhealth.jsp shows summary of multiple namenodes in cluster. With > federation combined with HA it becomes difficult to quickly know the state of > NameNodes in the cluster. It would be good to show if NameNode is > Active/Standy on summary page. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4924) Show NameNode state on dfsclusterhealth page
[ https://issues.apache.org/jira/browse/HDFS-4924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lohit Vijayarenu updated HDFS-4924: --- Status: Patch Available (was: Open) > Show NameNode state on dfsclusterhealth page > > > Key: HDFS-4924 > URL: https://issues.apache.org/jira/browse/HDFS-4924 > Project: Hadoop HDFS > Issue Type: Improvement > Components: federation >Affects Versions: 2.1.0-beta >Reporter: Lohit Vijayarenu >Priority: Minor > Attachments: HDFS-4924.trunk.1.patch > > > dfsclusterhealth.jsp shows summary of multiple namenodes in cluster. With > federation combined with HA it becomes difficult to quickly know the state of > NameNodes in the cluster. It would be good to show if NameNode is > Active/Standy on summary page. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4924) Show NameNode state on dfsclusterhealth page
[ https://issues.apache.org/jira/browse/HDFS-4924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lohit Vijayarenu updated HDFS-4924: --- Attachment: HDFS-4924.trunk.1.patch Attached is first draft where state of NameNode is shown on dfsclusterhealth.jsp page beside NameNode. 1. If NameNode is in SafeMode, state column shows this with text returned from JMX 2. If HA enabled, state is shown as Active/Standby 3. If HA not enabled and NN is up then state is shown as Active 4. If State is unkown because dfsclusterhealth pull data from olderversion of NameNode, then state is shown as Unkown because assuming a state is confusing. > Show NameNode state on dfsclusterhealth page > > > Key: HDFS-4924 > URL: https://issues.apache.org/jira/browse/HDFS-4924 > Project: Hadoop HDFS > Issue Type: Improvement > Components: federation >Affects Versions: 2.1.0-beta >Reporter: Lohit Vijayarenu >Priority: Minor > Attachments: HDFS-4924.trunk.1.patch > > > dfsclusterhealth.jsp shows summary of multiple namenodes in cluster. With > federation combined with HA it becomes difficult to quickly know the state of > NameNodes in the cluster. It would be good to show if NameNode is > Active/Standy on summary page. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-4924) Show NameNode state on dfsclusterhealth page
Lohit Vijayarenu created HDFS-4924: -- Summary: Show NameNode state on dfsclusterhealth page Key: HDFS-4924 URL: https://issues.apache.org/jira/browse/HDFS-4924 Project: Hadoop HDFS Issue Type: Improvement Components: federation Affects Versions: 2.1.0-beta Reporter: Lohit Vijayarenu Priority: Minor dfsclusterhealth.jsp shows summary of multiple namenodes in cluster. With federation combined with HA it becomes difficult to quickly know the state of NameNodes in the cluster. It would be good to show if NameNode is Active/Standy on summary page. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3875) Issue handling checksum errors in write pipeline
[ https://issues.apache.org/jira/browse/HDFS-3875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13659229#comment-13659229 ] Lohit Vijayarenu commented on HDFS-3875: Could this be targeted for 2.0.5 release? We are seeing this exact same issue on one of our clusters. We are running hadoop-2.0.3-alpha release. > Issue handling checksum errors in write pipeline > > > Key: HDFS-3875 > URL: https://issues.apache.org/jira/browse/HDFS-3875 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode, hdfs-client >Affects Versions: 2.0.2-alpha >Reporter: Todd Lipcon >Assignee: Kihwal Lee >Priority: Critical > Attachments: hdfs-3875.branch-0.23.no.test.patch.txt, > hdfs-3875.branch-0.23.patch.txt, hdfs-3875.branch-0.23.with.test.patch.txt, > hdfs-3875.patch.txt, hdfs-3875.trunk.no.test.patch.txt, > hdfs-3875.trunk.no.test.patch.txt, hdfs-3875.trunk.patch.txt, > hdfs-3875.trunk.patch.txt, hdfs-3875.trunk.with.test.patch.txt, > hdfs-3875.trunk.with.test.patch.txt, hdfs-3875-wip.patch > > > We saw this issue with one block in a large test cluster. The client is > storing the data with replication level 2, and we saw the following: > - the second node in the pipeline detects a checksum error on the data it > received from the first node. We don't know if the client sent a bad > checksum, or if it got corrupted between node 1 and node 2 in the pipeline. > - this caused the second node to get kicked out of the pipeline, since it > threw an exception. The pipeline started up again with only one replica (the > first node in the pipeline) > - this replica was later determined to be corrupt by the block scanner, and > unrecoverable since it is the only replica -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2843) Rename protobuf message StorageInfoProto to NodeInfoProto
[
https://issues.apache.org/jira/browse/HDFS-2843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13617547#comment-13617547
]
Lohit Vijayarenu commented on HDFS-2843:
Isn't this considered incompatible change?
{code}
- required StorageInfoProto storageInfo = 3; // Node information
+ required NodeInfoProto nodeInfo = 3; // Node information
{code}
Protobuf suggest you add newer elements, but I also see that StorageInfoProto
was already declared as required. I do not understand how would this affect on
disk format vs newer code reading it, but it sure looks like incompatible
change.
> Rename protobuf message StorageInfoProto to NodeInfoProto
> -
>
> Key: HDFS-2843
> URL: https://issues.apache.org/jira/browse/HDFS-2843
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Reporter: Suresh Srinivas
>Assignee: Suresh Srinivas
> Attachments: HDFS-2843.patch
>
>
> StorageInfoProto has cTime, layoutVersion, namespaceID and clusterID. This is
> really information of a node that is part of the cluster, such as Namenode,
> Standby/Secondary/Backup/Checkpointer and datanodes. To reflect this, I want
> to rename it as NodeInfoProto from StorageInfoProto.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4497) commons-daemon 1.0.3 dependency has bad group id causing build issues
[
https://issues.apache.org/jira/browse/HDFS-4497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13593600#comment-13593600
]
Lohit Vijayarenu commented on HDFS-4497:
+1 Patch looks good
> commons-daemon 1.0.3 dependency has bad group id causing build issues
> -
>
> Key: HDFS-4497
> URL: https://issues.apache.org/jira/browse/HDFS-4497
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: build
>Affects Versions: 2.0.3-alpha
>Reporter: Sangjin Lee
> Attachments: HDFS-4497.patch
>
>
> The commons-daemon dependency of the hadoop-hdfs module has been at version
> 1.0.3 for a while. However, 1.0.3 has a pretty well-known groupId error in
> its pom ("org.apache.commons" as opposed to "commons-daemon"). This problem
> has since been corrected on commons-daemon starting 1.0.4.
> This causes build problems for many who depend on hadoop-hdfs directly and
> indirectly, however. Maven can skip over this metadata inconsistency. But
> other less forgiving build systems such as ivy and gradle have much harder
> time working around this problem. For example, in gradle, pretty much the
> only obvious way to work around this is to override this dependency version.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4452) getAdditionalBlock() can create multiple blocks if the client times out and retries.
[
https://issues.apache.org/jira/browse/HDFS-4452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13565749#comment-13565749
]
Lohit Vijayarenu commented on HDFS-4452:
Running few benchmarks on 2.0.2-alpha I did hit this once. Should this be
marked blocker?
> getAdditionalBlock() can create multiple blocks if the client times out and
> retries.
>
>
> Key: HDFS-4452
> URL: https://issues.apache.org/jira/browse/HDFS-4452
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.0.2-alpha
>Reporter: Konstantin Shvachko
>Assignee: Konstantin Shvachko
>Priority: Critical
> Attachments: TestAddBlockRetry.java
>
>
> HDFS client tries to addBlock() to a file. If NameNode is busy the client can
> timeout and will reissue the same request again. The two requests will race
> with each other in {{FSNamesystem.getAdditionalBlock()}}, which can result in
> creating two new blocks on the NameNode while the client will know of only
> one of them. This eventually results in {{NotReplicatedYetException}} because
> the extra block is never reported by any DataNode, which stalls file creation
> and puts it in invalid state with an empty block in the middle.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1451) Extend the pluggable block placement policy interface to handle/allow file-level intelligence
[ https://issues.apache.org/jira/browse/HDFS-1451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13542495#comment-13542495 ] Lohit Vijayarenu commented on HDFS-1451: This would be useful. With this kind of feature once could decide how to place replicas of files based on performance vs reliability. If one has multi level topology (like power domain followed by racks), once can chose to place replicas across power domain so that even losing few power outlets will not cause outage to important files. > Extend the pluggable block placement policy interface to handle/allow > file-level intelligence > - > > Key: HDFS-1451 > URL: https://issues.apache.org/jira/browse/HDFS-1451 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 0.21.0 >Reporter: Harold Lim > > The current pluggable block placement policy interface > (https://issues.apache.org/jira/browse/HDFS-385) sets the policy globally. I > propose to have this interface be extended to allow setting the policy on a > file-level basis. For example, there may be cases where certain files may > simply need the default policy and certain other files need more complex > block placement policies. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HDFS-854) Datanode should scan devices in parallel to generate block report
[ https://issues.apache.org/jira/browse/HDFS-854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12803501#action_12803501 ] Lohit Vijayarenu commented on HDFS-854: --- Or probably keep last added/deleted blocks in memory and send block report and doing disk scan once in a while? > Datanode should scan devices in parallel to generate block report > - > > Key: HDFS-854 > URL: https://issues.apache.org/jira/browse/HDFS-854 > Project: Hadoop HDFS > Issue Type: Improvement > Components: data-node >Reporter: dhruba borthakur >Assignee: Dmytro Molkov > > A Datanode should scan its disk devices in parallel so that the time to > generate a block report is reduced. This will reduce the startup time of a > cluster. > A datanode has 12 disk (each of 1 TB) to store HDFS blocks. There is a total > of 150K blocks on these 12 disks. It takes the datanode upto 20 minutes to > scan these devices to generate the first block report. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
