[
https://issues.apache.org/jira/browse/HDFS-13098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16348992#comment-16348992
]
Wei Yan commented on HDFS-13098:
--------------------------------
{quote}Currently, we do the assignment of DNs to subclusters using external
tools that generate {{hdfs-site.xml}}. These tools could be moved into the RBF
infrastructure.
To solve this, we could make the DNs to just register the first time through
the Router and afterwards switch to heartbeating into the actual Namenodes.
{quote}
If I understand correctly, here the first hearbeat to Routers is to determine
which subcluster this new DN will join. Afterwards, the DN will only talk to
NNs in that subcluster. If this's correct, the first heartbeat to Routers is
just like an "external" tool to generate the hdfs-site.xml, right?
{quote} * Policies to assign datanodes to subcluters
* Datanodes to make blockpool configuration dynamic{quote}
Do u have some example policies here, to determine the DN assignment? I'm
thinking about data assignment policies, like allocating data to subcluter to
balance subcluster workload, moving "warm" data to cheaper hardware subcluster,
etc. Maybe I miss sth here, but do we need to relocate DNs from clusters to
clusters quite often?
> RBF: Datanodes interacting with Routers
> ---------------------------------------
>
> Key: HDFS-13098
> URL: https://issues.apache.org/jira/browse/HDFS-13098
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Reporter: Íñigo Goiri
> Priority: Major
>
> Datanodes talk to particular Namenodes. We could use the Router
> infrastructure for the Datanodes to register/heartbeating into them and the
> Routers would forward this to particular Namenodes. This would make the
> assignment of Datanodes to subclusters potentially more dynamic.
> The implementation would potentially include:
> * Router to implement part of DatanodeProtocol
> * Forwarding DN messages into Routers
> * Policies to assign datanodes to subclusters
> * Datanodes to make blockpool configuration dynamic
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
