[jira] [Updated] (HDFS-2191) Move datanodeMap from FSNamesystem to DatanodeManager
[ https://issues.apache.org/jira/browse/HDFS-2191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated HDFS-2191: - Resolution: Fixed Fix Version/s: 0.23.0 Hadoop Flags: [Reviewed] Status: Resolved (was: Patch Available) I have committed this. > Move datanodeMap from FSNamesystem to DatanodeManager > - > > Key: HDFS-2191 > URL: https://issues.apache.org/jira/browse/HDFS-2191 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: name-node >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Tsz Wo (Nicholas), SZE > Fix For: 0.23.0 > > Attachments: h2191_20110723.patch, h2191_20110723b.patch, > h2191_20110726.patch > > -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2191) Move datanodeMap from FSNamesystem to DatanodeManager
[
https://issues.apache.org/jira/browse/HDFS-2191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tsz Wo (Nicholas), SZE updated HDFS-2191:
-
Attachment: h2191_20110726.patch
Thanks Suresh for the comments.
h2191_20110726.patch: addressed everything except for #2.
1. Is this correct? 1 * heartbeatIntervalSeconds Why 1? Should you be
multiplying it by 1000 when getting it from conf and not 1 *?
It is correct. In the original code, we have
{code}
-long heartbeatInterval = conf.getLong(
-DFSConfigKeys.DFS_HEARTBEAT_INTERVAL_KEY,
-DFSConfigKeys.DFS_HEARTBEAT_INTERVAL_DEFAULT) * 1000;
this.heartbeatRecheckInterval = conf.getInt(
DFSConfigKeys.DFS_NAMENODE_HEARTBEAT_RECHECK_INTERVAL_KEY,
DFSConfigKeys.DFS_NAMENODE_HEARTBEAT_RECHECK_INTERVAL_DEFAULT); // 5
minutes
-this.heartbeatExpireInterval = 2 * heartbeatRecheckInterval +
- 10 * heartbeatInterval;
{code}
So, 1 = 10*1000 is correct. I changed heartbeatInterval (in ms) to
heartbeatInterval to heartbeatIntervalSeconds since we will convert
heartbeatInterval back to seconds in calculating blockInvalidateLimit below.
{code}
-this.blockInvalidateLimit = Math.max(this.blockInvalidateLimit,
- 20*(int)(heartbeatInterval/1000));
{code}
2. blockManager.getDatanodeManager().getDatanode() - we could add a method
blockManager.getDataNode() - I prefer this instead of exposing DatanodeManager
to FSNamesystem.
FSNamesystem currently requires quite a few methods in DatanodeMnaager, e.g.
getNetworkTopology(), registerDatanode(..), etc. The getDatanode(..) methods
are only two of those methods. Let's think about how to change the APIs after
the code separation is done. Okay?
3. Earlier getDatanode() used to lock using datanodeMap, do we need that with
your patch? How is it synchronized now? Also datanodeMap access may not be
synchronized correctly - for example datanodeDump synchronizes it but other
instances such as DatanodeManager#getDatanode() does not.
I think you are talking about FSNamesystem.heartbeatCheck(). I incorrectly
removed the synchronization. Fixed it.
4. In a future patch FSNamesystem#heartbeats and its synchronization should
move to blockmanager
That's correct.
5. Please change javadoc of getDatanode() as throws UnregisteredNodeException.
Done.
6. I am sure you have already planned for removing
FSNamesystem#blockInvalidateLimit in future patch.
Removed.
7. How is DatanodeManager#getDatanodeCyclicIteration() synchronized? It cannot
be synchronized using FSNamesystem.writeLock() as done currently?
It should accquire the write lock of DatanodeManager when we have read-write
lock in DatanodeManager in the future.
> Move datanodeMap from FSNamesystem to DatanodeManager
> -
>
> Key: HDFS-2191
> URL: https://issues.apache.org/jira/browse/HDFS-2191
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: name-node
>Reporter: Tsz Wo (Nicholas), SZE
>Assignee: Tsz Wo (Nicholas), SZE
> Attachments: h2191_20110723.patch, h2191_20110723b.patch,
> h2191_20110726.patch
>
>
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2191) Move datanodeMap from FSNamesystem to DatanodeManager
[ https://issues.apache.org/jira/browse/HDFS-2191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated HDFS-2191: - Attachment: h2191_20110723b.patch h2191_20110723b.patch: fixed the javadoc warnings. > Move datanodeMap from FSNamesystem to DatanodeManager > - > > Key: HDFS-2191 > URL: https://issues.apache.org/jira/browse/HDFS-2191 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: name-node >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Tsz Wo (Nicholas), SZE > Attachments: h2191_20110723.patch, h2191_20110723b.patch > > -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2191) Move datanodeMap from FSNamesystem to DatanodeManager
[ https://issues.apache.org/jira/browse/HDFS-2191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated HDFS-2191: - Attachment: h2191_20110723.patch h2191_20110723.patch: 1st patch > Move datanodeMap from FSNamesystem to DatanodeManager > - > > Key: HDFS-2191 > URL: https://issues.apache.org/jira/browse/HDFS-2191 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: name-node >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Tsz Wo (Nicholas), SZE > Attachments: h2191_20110723.patch > > -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2191) Move datanodeMap from FSNamesystem to DatanodeManager
[ https://issues.apache.org/jira/browse/HDFS-2191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated HDFS-2191: - Status: Patch Available (was: Open) > Move datanodeMap from FSNamesystem to DatanodeManager > - > > Key: HDFS-2191 > URL: https://issues.apache.org/jira/browse/HDFS-2191 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: name-node >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Tsz Wo (Nicholas), SZE > Attachments: h2191_20110723.patch > > -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
