[jira] Commented: (HDFS-1580) Add interface for generic Write Ahead Logging mechanisms
[ https://issues.apache.org/jira/browse/HDFS-1580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13003127#comment-13003127 ] Ivan Kelly commented on HDFS-1580: -- To read in a log you pass in a URI to the getInputStream method. For BK, this will allow us to have no explicit transfer stage. Files are a different matter. I haven't thought much about it yet, but file URI do have a space for host. file://host/path. The FileJournalManager could return a URI with the host part set for getLogs. This could be passed to the getInputStream call, which would download it from the remote host if nescessary. So, the steps would be as you outlined except for the iterates over each of the RemoteLogs, and fetches them. Instead you would just iterate over them and load them using the JournalManager.getInputStream method. The implementation of the method would take care of whether the file is remote or local. Add interface for generic Write Ahead Logging mechanisms Key: HDFS-1580 URL: https://issues.apache.org/jira/browse/HDFS-1580 Project: Hadoop HDFS Issue Type: Improvement Reporter: Ivan Kelly Attachments: HDFS-1580+1521.diff, HDFS-1580.diff, generic_wal_iface.pdf, generic_wal_iface.pdf, generic_wal_iface.txt -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HDFS-442) dfsthroughput in test.jar throws NPE
[
https://issues.apache.org/jira/browse/HDFS-442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13003222#comment-13003222
]
Greg Poirier commented on HDFS-442:
---
This can be fixed by modifying
src/test/org/apache/hadoop/hdfs/BenchmarkThroughput.java:198
From:
198 String localDir = conf.get(mapred.temp.dir);
199 dir = new LocalDirAllocator(mapred.temp.dir);
To:
198 String localDir = conf.get(hadoop.tmp.dir);
199 dir = new LocalDirAllocator(hadoop.tmp.dir);
It looks as though this has been removed from subversion trunk though... but it
still exists in 0.20.2/3, which I'm running currently.
dfsthroughput in test.jar throws NPE
Key: HDFS-442
URL: https://issues.apache.org/jira/browse/HDFS-442
Project: Hadoop HDFS
Issue Type: Bug
Components: test
Affects Versions: 0.20.1
Reporter: Ramya R
Priority: Minor
On running hadoop jar hadoop-test.jar dfsthroughput OR hadoop
org.apache.hadoop.hdfs.BenchmarkThroughput, we get NullPointerException.
Below is the stacktrace:
{noformat}
Exception in thread main java.lang.NullPointerException
at java.util.Hashtable.put(Hashtable.java:394)
at java.util.Properties.setProperty(Properties.java:143)
at java.lang.System.setProperty(System.java:731)
at
org.apache.hadoop.hdfs.BenchmarkThroughput.run(BenchmarkThroughput.java:198)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at
org.apache.hadoop.hdfs.BenchmarkThroughput.main(BenchmarkThroughput.java:229)
{noformat}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HDFS-442) dfsthroughput in test.jar throws NPE
[
https://issues.apache.org/jira/browse/HDFS-442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13003225#comment-13003225
]
Greg Poirier commented on HDFS-442:
---
Sorry, I should have noted that this assumes you have a hadoop.tmp.dir property
configured in conf/core-site.xml. I was surprised that a Configuration object
wasn't all-inclusive.
dfsthroughput in test.jar throws NPE
Key: HDFS-442
URL: https://issues.apache.org/jira/browse/HDFS-442
Project: Hadoop HDFS
Issue Type: Bug
Components: test
Affects Versions: 0.20.1
Reporter: Ramya R
Priority: Minor
On running hadoop jar hadoop-test.jar dfsthroughput OR hadoop
org.apache.hadoop.hdfs.BenchmarkThroughput, we get NullPointerException.
Below is the stacktrace:
{noformat}
Exception in thread main java.lang.NullPointerException
at java.util.Hashtable.put(Hashtable.java:394)
at java.util.Properties.setProperty(Properties.java:143)
at java.lang.System.setProperty(System.java:731)
at
org.apache.hadoop.hdfs.BenchmarkThroughput.run(BenchmarkThroughput.java:198)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at
org.apache.hadoop.hdfs.BenchmarkThroughput.main(BenchmarkThroughput.java:229)
{noformat}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HDFS-1073) Simpler model for Namenode's fs Image and edit Logs
[ https://issues.apache.org/jira/browse/HDFS-1073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13003236#comment-13003236 ] Todd Lipcon commented on HDFS-1073: --- Progress is coming along on this issue. I've pushed a git branch for the work-in-progress here: https://github.com/toddlipcon/hadoop-hdfs/tree/hdfs-1073-march I've based this branch on top of HDFS-1521 and HDFS-1538 Simpler model for Namenode's fs Image and edit Logs Key: HDFS-1073 URL: https://issues.apache.org/jira/browse/HDFS-1073 Project: Hadoop HDFS Issue Type: Improvement Reporter: Sanjay Radia Assignee: Todd Lipcon Attachments: hdfs-1073-editloading-algos.txt, hdfs-1073.txt, hdfs1073.pdf The naming and handling of NN's fsImage and edit logs can be significantly improved resulting simpler and more robust code. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HDFS-1611) Some logical issues need to address.
[
https://issues.apache.org/jira/browse/HDFS-1611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Todd Lipcon updated HDFS-1611:
--
Status: Open (was: Patch Available)
Some logical issues need to address.
Key: HDFS-1611
URL: https://issues.apache.org/jira/browse/HDFS-1611
Project: Hadoop HDFS
Issue Type: Bug
Components: hdfs client, name-node
Affects Versions: 0.20.2, 0.20.1
Reporter: Uma Maheswara Rao G
Priority: Minor
Attachments: HDFS-1611.patch
Title: Some code level logical issues.
Description:
1. DFSClient:
Consider the below case, if we enable only info, then below log will never
be logged.
if (ClientDatanodeProtocol.LOG.isDebugEnabled()) {
ClientDatanodeProtocol.LOG.info(ClientDatanodeProtocol addr= + addr);
}
2.org.apache.hadoop.hdfs.server.namenode.FSNamesystem.registerMBean()
catch (NotCompliantMBeanException e) {
e.printStackTrace();
}
We can avoid using stackTace(). Better to add log message.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HDFS-1611) Some logical issues need to address.
[
https://issues.apache.org/jira/browse/HDFS-1611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13003247#comment-13003247
]
Todd Lipcon commented on HDFS-1611:
---
For the log message for ClientDatanodeProtocol, I think it's better to change
the log to Debug level, rather than change the guard. It's not that useful of a
log message to be set at INFO level.
Some logical issues need to address.
Key: HDFS-1611
URL: https://issues.apache.org/jira/browse/HDFS-1611
Project: Hadoop HDFS
Issue Type: Bug
Components: hdfs client, name-node
Affects Versions: 0.20.1, 0.20.2
Reporter: Uma Maheswara Rao G
Priority: Minor
Attachments: HDFS-1611.patch
Title: Some code level logical issues.
Description:
1. DFSClient:
Consider the below case, if we enable only info, then below log will never
be logged.
if (ClientDatanodeProtocol.LOG.isDebugEnabled()) {
ClientDatanodeProtocol.LOG.info(ClientDatanodeProtocol addr= + addr);
}
2.org.apache.hadoop.hdfs.server.namenode.FSNamesystem.registerMBean()
catch (NotCompliantMBeanException e) {
e.printStackTrace();
}
We can avoid using stackTace(). Better to add log message.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Assigned: (HDFS-1611) Some logical issues need to address.
[
https://issues.apache.org/jira/browse/HDFS-1611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Todd Lipcon reassigned HDFS-1611:
-
Assignee: Uma Maheswara Rao G
Some logical issues need to address.
Key: HDFS-1611
URL: https://issues.apache.org/jira/browse/HDFS-1611
Project: Hadoop HDFS
Issue Type: Bug
Components: hdfs client, name-node
Affects Versions: 0.20.1, 0.20.2
Reporter: Uma Maheswara Rao G
Assignee: Uma Maheswara Rao G
Priority: Minor
Attachments: HDFS-1611.patch
Title: Some code level logical issues.
Description:
1. DFSClient:
Consider the below case, if we enable only info, then below log will never
be logged.
if (ClientDatanodeProtocol.LOG.isDebugEnabled()) {
ClientDatanodeProtocol.LOG.info(ClientDatanodeProtocol addr= + addr);
}
2.org.apache.hadoop.hdfs.server.namenode.FSNamesystem.registerMBean()
catch (NotCompliantMBeanException e) {
e.printStackTrace();
}
We can avoid using stackTace(). Better to add log message.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HDFS-1665) Balancer sleeps inadequately
[
https://issues.apache.org/jira/browse/HDFS-1665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13003256#comment-13003256
]
Todd Lipcon commented on HDFS-1665:
---
+1, good catch
Balancer sleeps inadequately
Key: HDFS-1665
URL: https://issues.apache.org/jira/browse/HDFS-1665
Project: Hadoop HDFS
Issue Type: Bug
Components: balancer
Reporter: Tsz Wo (Nicholas), SZE
Assignee: Tsz Wo (Nicholas), SZE
Priority: Minor
Attachments: h1665_20110225.patch, h1665_20110225b.patch
The value of {{dfs.heartbeat.interval}} is in milliseconds. Balancer seems
misused it.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Assigned: (HDFS-1690) Name Node should not process format command while it is running.
[ https://issues.apache.org/jira/browse/HDFS-1690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G reassigned HDFS-1690: - Assignee: Uma Maheswara Rao G Name Node should not process format command while it is running. Key: HDFS-1690 URL: https://issues.apache.org/jira/browse/HDFS-1690 Project: Hadoop HDFS Issue Type: Bug Components: name-node Affects Versions: 0.20.1, 0.20.2, 0.21.0 Reporter: Uma Maheswara Rao G Assignee: Uma Maheswara Rao G Currently NameNode allows format command while it running. In this case the command is executed partially (Lock file is deleted) and an exception thrown. Because of this Name Node should be formatted after restart. This sort of cases can happen accidentally. To prevent such cases Name Node should not execute the format command partially while it is running. It can stright away throw exception/log saying, Name Node is running. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
