[jira] Commented: (HDFS-1580) Add interface for generic Write Ahead Logging mechanisms
[ https://issues.apache.org/jira/browse/HDFS-1580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13003127#comment-13003127 ] Ivan Kelly commented on HDFS-1580: -- To read in a log you pass in a URI to the getInputStream method. For BK, this will allow us to have no explicit transfer stage. Files are a different matter. I haven't thought much about it yet, but file URI do have a space for host. file://host/path. The FileJournalManager could return a URI with the host part set for getLogs. This could be passed to the getInputStream call, which would download it from the remote host if nescessary. So, the steps would be as you outlined except for the iterates over each of the RemoteLogs, and fetches them. Instead you would just iterate over them and load them using the JournalManager.getInputStream method. The implementation of the method would take care of whether the file is remote or local. Add interface for generic Write Ahead Logging mechanisms Key: HDFS-1580 URL: https://issues.apache.org/jira/browse/HDFS-1580 Project: Hadoop HDFS Issue Type: Improvement Reporter: Ivan Kelly Attachments: HDFS-1580+1521.diff, HDFS-1580.diff, generic_wal_iface.pdf, generic_wal_iface.pdf, generic_wal_iface.txt -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HDFS-442) dfsthroughput in test.jar throws NPE
[ https://issues.apache.org/jira/browse/HDFS-442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13003222#comment-13003222 ] Greg Poirier commented on HDFS-442: --- This can be fixed by modifying src/test/org/apache/hadoop/hdfs/BenchmarkThroughput.java:198 From: 198 String localDir = conf.get(mapred.temp.dir); 199 dir = new LocalDirAllocator(mapred.temp.dir); To: 198 String localDir = conf.get(hadoop.tmp.dir); 199 dir = new LocalDirAllocator(hadoop.tmp.dir); It looks as though this has been removed from subversion trunk though... but it still exists in 0.20.2/3, which I'm running currently. dfsthroughput in test.jar throws NPE Key: HDFS-442 URL: https://issues.apache.org/jira/browse/HDFS-442 Project: Hadoop HDFS Issue Type: Bug Components: test Affects Versions: 0.20.1 Reporter: Ramya R Priority: Minor On running hadoop jar hadoop-test.jar dfsthroughput OR hadoop org.apache.hadoop.hdfs.BenchmarkThroughput, we get NullPointerException. Below is the stacktrace: {noformat} Exception in thread main java.lang.NullPointerException at java.util.Hashtable.put(Hashtable.java:394) at java.util.Properties.setProperty(Properties.java:143) at java.lang.System.setProperty(System.java:731) at org.apache.hadoop.hdfs.BenchmarkThroughput.run(BenchmarkThroughput.java:198) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) at org.apache.hadoop.hdfs.BenchmarkThroughput.main(BenchmarkThroughput.java:229) {noformat} -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HDFS-442) dfsthroughput in test.jar throws NPE
[ https://issues.apache.org/jira/browse/HDFS-442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13003225#comment-13003225 ] Greg Poirier commented on HDFS-442: --- Sorry, I should have noted that this assumes you have a hadoop.tmp.dir property configured in conf/core-site.xml. I was surprised that a Configuration object wasn't all-inclusive. dfsthroughput in test.jar throws NPE Key: HDFS-442 URL: https://issues.apache.org/jira/browse/HDFS-442 Project: Hadoop HDFS Issue Type: Bug Components: test Affects Versions: 0.20.1 Reporter: Ramya R Priority: Minor On running hadoop jar hadoop-test.jar dfsthroughput OR hadoop org.apache.hadoop.hdfs.BenchmarkThroughput, we get NullPointerException. Below is the stacktrace: {noformat} Exception in thread main java.lang.NullPointerException at java.util.Hashtable.put(Hashtable.java:394) at java.util.Properties.setProperty(Properties.java:143) at java.lang.System.setProperty(System.java:731) at org.apache.hadoop.hdfs.BenchmarkThroughput.run(BenchmarkThroughput.java:198) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) at org.apache.hadoop.hdfs.BenchmarkThroughput.main(BenchmarkThroughput.java:229) {noformat} -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HDFS-1073) Simpler model for Namenode's fs Image and edit Logs
[ https://issues.apache.org/jira/browse/HDFS-1073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13003236#comment-13003236 ] Todd Lipcon commented on HDFS-1073: --- Progress is coming along on this issue. I've pushed a git branch for the work-in-progress here: https://github.com/toddlipcon/hadoop-hdfs/tree/hdfs-1073-march I've based this branch on top of HDFS-1521 and HDFS-1538 Simpler model for Namenode's fs Image and edit Logs Key: HDFS-1073 URL: https://issues.apache.org/jira/browse/HDFS-1073 Project: Hadoop HDFS Issue Type: Improvement Reporter: Sanjay Radia Assignee: Todd Lipcon Attachments: hdfs-1073-editloading-algos.txt, hdfs-1073.txt, hdfs1073.pdf The naming and handling of NN's fsImage and edit logs can be significantly improved resulting simpler and more robust code. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HDFS-1611) Some logical issues need to address.
[ https://issues.apache.org/jira/browse/HDFS-1611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated HDFS-1611: -- Status: Open (was: Patch Available) Some logical issues need to address. Key: HDFS-1611 URL: https://issues.apache.org/jira/browse/HDFS-1611 Project: Hadoop HDFS Issue Type: Bug Components: hdfs client, name-node Affects Versions: 0.20.2, 0.20.1 Reporter: Uma Maheswara Rao G Priority: Minor Attachments: HDFS-1611.patch Title: Some code level logical issues. Description: 1. DFSClient: Consider the below case, if we enable only info, then below log will never be logged. if (ClientDatanodeProtocol.LOG.isDebugEnabled()) { ClientDatanodeProtocol.LOG.info(ClientDatanodeProtocol addr= + addr); } 2.org.apache.hadoop.hdfs.server.namenode.FSNamesystem.registerMBean() catch (NotCompliantMBeanException e) { e.printStackTrace(); } We can avoid using stackTace(). Better to add log message. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HDFS-1611) Some logical issues need to address.
[ https://issues.apache.org/jira/browse/HDFS-1611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13003247#comment-13003247 ] Todd Lipcon commented on HDFS-1611: --- For the log message for ClientDatanodeProtocol, I think it's better to change the log to Debug level, rather than change the guard. It's not that useful of a log message to be set at INFO level. Some logical issues need to address. Key: HDFS-1611 URL: https://issues.apache.org/jira/browse/HDFS-1611 Project: Hadoop HDFS Issue Type: Bug Components: hdfs client, name-node Affects Versions: 0.20.1, 0.20.2 Reporter: Uma Maheswara Rao G Priority: Minor Attachments: HDFS-1611.patch Title: Some code level logical issues. Description: 1. DFSClient: Consider the below case, if we enable only info, then below log will never be logged. if (ClientDatanodeProtocol.LOG.isDebugEnabled()) { ClientDatanodeProtocol.LOG.info(ClientDatanodeProtocol addr= + addr); } 2.org.apache.hadoop.hdfs.server.namenode.FSNamesystem.registerMBean() catch (NotCompliantMBeanException e) { e.printStackTrace(); } We can avoid using stackTace(). Better to add log message. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Assigned: (HDFS-1611) Some logical issues need to address.
[ https://issues.apache.org/jira/browse/HDFS-1611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon reassigned HDFS-1611: - Assignee: Uma Maheswara Rao G Some logical issues need to address. Key: HDFS-1611 URL: https://issues.apache.org/jira/browse/HDFS-1611 Project: Hadoop HDFS Issue Type: Bug Components: hdfs client, name-node Affects Versions: 0.20.1, 0.20.2 Reporter: Uma Maheswara Rao G Assignee: Uma Maheswara Rao G Priority: Minor Attachments: HDFS-1611.patch Title: Some code level logical issues. Description: 1. DFSClient: Consider the below case, if we enable only info, then below log will never be logged. if (ClientDatanodeProtocol.LOG.isDebugEnabled()) { ClientDatanodeProtocol.LOG.info(ClientDatanodeProtocol addr= + addr); } 2.org.apache.hadoop.hdfs.server.namenode.FSNamesystem.registerMBean() catch (NotCompliantMBeanException e) { e.printStackTrace(); } We can avoid using stackTace(). Better to add log message. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HDFS-1665) Balancer sleeps inadequately
[ https://issues.apache.org/jira/browse/HDFS-1665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13003256#comment-13003256 ] Todd Lipcon commented on HDFS-1665: --- +1, good catch Balancer sleeps inadequately Key: HDFS-1665 URL: https://issues.apache.org/jira/browse/HDFS-1665 Project: Hadoop HDFS Issue Type: Bug Components: balancer Reporter: Tsz Wo (Nicholas), SZE Assignee: Tsz Wo (Nicholas), SZE Priority: Minor Attachments: h1665_20110225.patch, h1665_20110225b.patch The value of {{dfs.heartbeat.interval}} is in milliseconds. Balancer seems misused it. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Assigned: (HDFS-1690) Name Node should not process format command while it is running.
[ https://issues.apache.org/jira/browse/HDFS-1690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G reassigned HDFS-1690: - Assignee: Uma Maheswara Rao G Name Node should not process format command while it is running. Key: HDFS-1690 URL: https://issues.apache.org/jira/browse/HDFS-1690 Project: Hadoop HDFS Issue Type: Bug Components: name-node Affects Versions: 0.20.1, 0.20.2, 0.21.0 Reporter: Uma Maheswara Rao G Assignee: Uma Maheswara Rao G Currently NameNode allows format command while it running. In this case the command is executed partially (Lock file is deleted) and an exception thrown. Because of this Name Node should be formatted after restart. This sort of cases can happen accidentally. To prevent such cases Name Node should not execute the format command partially while it is running. It can stright away throw exception/log saying, Name Node is running. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira