[jira] Updated: (HDFS-1572) Checkpointer should trigger checkpoint with specified period.
[
https://issues.apache.org/jira/browse/HDFS-1572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Liyin Liang updated HDFS-1572:
--
Attachment: 1572-2.diff
Hi Jakob,
Thanks for you work and advice. We can move some logic to static functions
to make it more testable. I'll attach a patch with a simple test case. Any
thoughts?
Checkpointer should trigger checkpoint with specified period.
-
Key: HDFS-1572
URL: https://issues.apache.org/jira/browse/HDFS-1572
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 0.22.0
Reporter: Liyin Liang
Priority: Blocker
Fix For: 0.21.0
Attachments: 1527-1.diff, 1572-2.diff, HDFS-1572.patch
{code:}
long now = now();
boolean shouldCheckpoint = false;
if(now = lastCheckpointTime + periodMSec) {
shouldCheckpoint = true;
} else {
long size = getJournalSize();
if(size = checkpointSize)
shouldCheckpoint = true;
}
{code}
{dfs.namenode.checkpoint.period} in configuration determines the period of
checkpoint. However, with above code, the Checkpointer triggers a checkpoint
every 5 minutes (periodMSec=5*60*1000). According to SecondaryNameNode.java,
the first *if* statement should be:
{code:}
if(now = lastCheckpointTime + 1000 * checkpointPeriod) {
{code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1496) TestStorageRestore is failing after HDFS-903 fix
[ https://issues.apache.org/jira/browse/HDFS-1496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Shvachko updated HDFS-1496: -- Fix Version/s: 0.22.0 TestStorageRestore is failing after HDFS-903 fix Key: HDFS-1496 URL: https://issues.apache.org/jira/browse/HDFS-1496 Project: Hadoop HDFS Issue Type: Bug Components: test Affects Versions: 0.22.0, 0.23.0 Reporter: Konstantin Boudnik Assignee: Hairong Kuang Priority: Blocker Fix For: 0.22.0 TestStorageRestore seems to be failing after HDFS-903 commit. Running git bisect confirms it. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-884) DataNode makeInstance should report the directory list when failing to start up
[
https://issues.apache.org/jira/browse/HDFS-884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Konstantin Shvachko updated HDFS-884:
-
Status: Open (was: Patch Available)
DataNode makeInstance should report the directory list when failing to start
up
---
Key: HDFS-884
URL: https://issues.apache.org/jira/browse/HDFS-884
Project: Hadoop HDFS
Issue Type: Improvement
Components: data-node
Affects Versions: 0.22.0
Reporter: Steve Loughran
Assignee: Steve Loughran
Priority: Minor
Fix For: 0.22.0
Attachments: HDFS-884.patch, HDFS-884.patch, InvalidDirs.patch
When {{Datanode.makeInstance()}} cannot work with one of the directories in
dfs.data.dir, it logs this at warn level (while losing the stack trace).
It should include the nested exception for better troubleshooting. Then, when
all dirs in the list fail, an exception is thrown, but this exception does
not include the list of directories. It should list the absolute path of
every missing/failing directory, so that whoever sees the exception can see
where to start looking for problems: either the filesystem or the
configuration.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-884) DataNode makeInstance should report the directory list when failing to start up
[
https://issues.apache.org/jira/browse/HDFS-884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Konstantin Shvachko updated HDFS-884:
-
Attachment: InvalidDirs.patch
Missed one case where TestDatanodeConfig relies on that createDataNode()
returns null. Fixed it.
DataNode makeInstance should report the directory list when failing to start
up
---
Key: HDFS-884
URL: https://issues.apache.org/jira/browse/HDFS-884
Project: Hadoop HDFS
Issue Type: Improvement
Components: data-node
Affects Versions: 0.22.0
Reporter: Steve Loughran
Assignee: Steve Loughran
Priority: Minor
Fix For: 0.22.0
Attachments: HDFS-884.patch, HDFS-884.patch, InvalidDirs.patch,
InvalidDirs.patch
When {{Datanode.makeInstance()}} cannot work with one of the directories in
dfs.data.dir, it logs this at warn level (while losing the stack trace).
It should include the nested exception for better troubleshooting. Then, when
all dirs in the list fail, an exception is thrown, but this exception does
not include the list of directories. It should list the absolute path of
every missing/failing directory, so that whoever sees the exception can see
where to start looking for problems: either the filesystem or the
configuration.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-884) DataNode makeInstance should report the directory list when failing to start up
[
https://issues.apache.org/jira/browse/HDFS-884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Konstantin Shvachko updated HDFS-884:
-
Status: Patch Available (was: Open)
DataNode makeInstance should report the directory list when failing to start
up
---
Key: HDFS-884
URL: https://issues.apache.org/jira/browse/HDFS-884
Project: Hadoop HDFS
Issue Type: Improvement
Components: data-node
Affects Versions: 0.22.0
Reporter: Steve Loughran
Assignee: Steve Loughran
Priority: Minor
Fix For: 0.22.0
Attachments: HDFS-884.patch, HDFS-884.patch, InvalidDirs.patch,
InvalidDirs.patch
When {{Datanode.makeInstance()}} cannot work with one of the directories in
dfs.data.dir, it logs this at warn level (while losing the stack trace).
It should include the nested exception for better troubleshooting. Then, when
all dirs in the list fail, an exception is thrown, but this exception does
not include the list of directories. It should list the absolute path of
every missing/failing directory, so that whoever sees the exception can see
where to start looking for problems: either the filesystem or the
configuration.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1561) BackupNode listens on default host
[
https://issues.apache.org/jira/browse/HDFS-1561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12979434#action_12979434
]
Konstantin Shvachko commented on HDFS-1561:
---
TestStorageRestore is HDFS-1496.
The rest is related to IPv6 problems. TestBackupNode and TestHDFSServerPorts
reuse configuration after it has been updated by NN or BN. But the updated
config contains IPv6-style address 0:0:0:0:0:0:0:0:50100 instead of
0.0.0.0:50100, which results in the following exception in NetUtils.
{code}
java.lang.NumberFormatException: For input string: 0:0:0:0:0:0:0:50100
at
java.lang.NumberFormatException.forInputString(NumberFormatException.java:48)
at java.lang.Integer.parseInt(Integer.java:456)
at java.lang.Integer.parseInt(Integer.java:497)
at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:155)
at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:129)
at
org.apache.hadoop.hdfs.server.namenode.BackupNode.getRpcServerAddress(BackupNode.java:81)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:350)
at
org.apache.hadoop.hdfs.server.namenode.BackupNode.initialize(BackupNode.java:129)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.init(NameNode.java:578)
at
org.apache.hadoop.hdfs.server.namenode.BackupNode.init(BackupNode.java:72)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1532)
at
org.apache.hadoop.hdfs.TestHDFSServerPorts.startBackupNode(TestHDFSServerPorts.java:137)
at
org.apache.hadoop.hdfs.TestHDFSServerPorts.canStartBackupNode(TestHDFSServerPorts.java:236)
at
org.apache.hadoop.hdfs.TestHDFSServerPorts.testBackupNodePorts(TestHDFSServerPorts.java:382)
{code}
I can fix it by resetting config parameters, but may it is better to run Hudson
with IPv4 flag as suggested in HADOOP-6056.
What people think?
BackupNode listens on default host
--
Key: HDFS-1561
URL: https://issues.apache.org/jira/browse/HDFS-1561
Project: Hadoop HDFS
Issue Type: Bug
Components: name-node
Affects Versions: 0.21.0
Reporter: Konstantin Shvachko
Fix For: 0.22.0
Attachments: BNAddress.patch
Currently BackupNode uses DNS to find its default host name, and then starts
RPC server listening on that address ignoring the address specified in the
configuration. Therefore, there is no way to start BackupNode on a particular
ip or host address. BackupNode should use the address specified in the
configuration instead.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1561) BackupNode listens on default host
[ https://issues.apache.org/jira/browse/HDFS-1561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Shvachko updated HDFS-1561: -- Status: Open (was: Patch Available) BackupNode listens on default host -- Key: HDFS-1561 URL: https://issues.apache.org/jira/browse/HDFS-1561 Project: Hadoop HDFS Issue Type: Bug Components: name-node Affects Versions: 0.21.0 Reporter: Konstantin Shvachko Fix For: 0.22.0 Attachments: BNAddress.patch Currently BackupNode uses DNS to find its default host name, and then starts RPC server listening on that address ignoring the address specified in the configuration. Therefore, there is no way to start BackupNode on a particular ip or host address. BackupNode should use the address specified in the configuration instead. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1561) BackupNode listens on default host
[ https://issues.apache.org/jira/browse/HDFS-1561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Shvachko updated HDFS-1561: -- Attachment: BNAddress.patch I replaced default addresses with real ones in order to avoid IPv6 conversions. BackupNode listens on default host -- Key: HDFS-1561 URL: https://issues.apache.org/jira/browse/HDFS-1561 Project: Hadoop HDFS Issue Type: Bug Components: name-node Affects Versions: 0.21.0 Reporter: Konstantin Shvachko Fix For: 0.22.0 Attachments: BNAddress.patch Currently BackupNode uses DNS to find its default host name, and then starts RPC server listening on that address ignoring the address specified in the configuration. Therefore, there is no way to start BackupNode on a particular ip or host address. BackupNode should use the address specified in the configuration instead. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1561) BackupNode listens on default host
[ https://issues.apache.org/jira/browse/HDFS-1561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Shvachko updated HDFS-1561: -- Attachment: (was: BNAddress.patch) BackupNode listens on default host -- Key: HDFS-1561 URL: https://issues.apache.org/jira/browse/HDFS-1561 Project: Hadoop HDFS Issue Type: Bug Components: name-node Affects Versions: 0.21.0 Reporter: Konstantin Shvachko Fix For: 0.22.0 Attachments: BNAddress.patch Currently BackupNode uses DNS to find its default host name, and then starts RPC server listening on that address ignoring the address specified in the configuration. Therefore, there is no way to start BackupNode on a particular ip or host address. BackupNode should use the address specified in the configuration instead. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1561) BackupNode listens on default host
[ https://issues.apache.org/jira/browse/HDFS-1561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Shvachko updated HDFS-1561: -- Attachment: BNAddress.patch BackupNode listens on default host -- Key: HDFS-1561 URL: https://issues.apache.org/jira/browse/HDFS-1561 Project: Hadoop HDFS Issue Type: Bug Components: name-node Affects Versions: 0.21.0 Reporter: Konstantin Shvachko Fix For: 0.22.0 Attachments: BNAddress.patch, BNAddress.patch Currently BackupNode uses DNS to find its default host name, and then starts RPC server listening on that address ignoring the address specified in the configuration. Therefore, there is no way to start BackupNode on a particular ip or host address. BackupNode should use the address specified in the configuration instead. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1561) BackupNode listens on default host
[ https://issues.apache.org/jira/browse/HDFS-1561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Shvachko updated HDFS-1561: -- Assignee: Konstantin Shvachko Status: Patch Available (was: Open) BackupNode listens on default host -- Key: HDFS-1561 URL: https://issues.apache.org/jira/browse/HDFS-1561 Project: Hadoop HDFS Issue Type: Bug Components: name-node Affects Versions: 0.21.0 Reporter: Konstantin Shvachko Assignee: Konstantin Shvachko Fix For: 0.22.0 Attachments: BNAddress.patch, BNAddress.patch Currently BackupNode uses DNS to find its default host name, and then starts RPC server listening on that address ignoring the address specified in the configuration. Therefore, there is no way to start BackupNode on a particular ip or host address. BackupNode should use the address specified in the configuration instead. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
