[jira] [Created] (HDFS-6248) SNN crash during replay of FSEditLog of files inside directories having QuotaExceeded directories
Lohit Vijayarenu created HDFS-6248: -- Summary: SNN crash during replay of FSEditLog of files inside directories having QuotaExceeded directories Key: HDFS-6248 URL: https://issues.apache.org/jira/browse/HDFS-6248 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.4.0, 2.0.6-alpha Environment: NameNode HA setup with Active/Standby using QJM Reporter: Lohit Vijayarenu We are seeing cases when Secondary NameNode crashes without recovery when it tries to replay edit log of files which are part of directories which have exceeded Quota. While debugging we got stack trace but we are still trying to reproduce this and wanted to note this to see if anyone else had seen this issue already. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HDFS-5499) Provide way to throttle per FileSystem read/write bandwidth
Lohit Vijayarenu created HDFS-5499: -- Summary: Provide way to throttle per FileSystem read/write bandwidth Key: HDFS-5499 URL: https://issues.apache.org/jira/browse/HDFS-5499 Project: Hadoop HDFS Issue Type: Improvement Reporter: Lohit Vijayarenu In some cases it might be worth to throttle read/writer bandwidth on per JVM basis so that clients do not spawn too many thread and start data movement causing other JVMs to starve. Ability to throttle read/write bandwidth on per FileSystem would help avoid such issues. Challenge seems to be how well this can be fit into FileSystem code. If one enables throttling around FileSystem APIs, then any hidden data transfer within cluster using them might also be affected. Eg. copying job jar during job submission, localizing resources for distributed cache and such. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (HDFS-5175) Provide clients a way to set IP header bits on connections
Lohit Vijayarenu created HDFS-5175: -- Summary: Provide clients a way to set IP header bits on connections Key: HDFS-5175 URL: https://issues.apache.org/jira/browse/HDFS-5175 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.0.5-alpha Reporter: Lohit Vijayarenu It would be very helpful if we had ability for clients to set IP headers when they make socket connections for data transfers. We were looking into setting up QoS using DSCP bit and saw that there is no easy way to let clients pass down a specific value when clients make connection to DataNode. As a quick fix we did something similar to io.file.buffer.size where client could pass down DSCP integer value and when DFSClient opens a stream, it could set the value on socket using setTrafficClass Opening this JIRA to get more inputs from others who have had experience and might have already thought about this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-4924) Show NameNode state on dfsclusterhealth page
Lohit Vijayarenu created HDFS-4924: -- Summary: Show NameNode state on dfsclusterhealth page Key: HDFS-4924 URL: https://issues.apache.org/jira/browse/HDFS-4924 Project: Hadoop HDFS Issue Type: Improvement Components: federation Affects Versions: 2.1.0-beta Reporter: Lohit Vijayarenu Priority: Minor dfsclusterhealth.jsp shows summary of multiple namenodes in cluster. With federation combined with HA it becomes difficult to quickly know the state of NameNodes in the cluster. It would be good to show if NameNode is Active/Standy on summary page. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
