[
https://issues.apache.org/jira/browse/HDFS-10285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16283102#comment-16283102
]
Uma Maheswara Rao G edited comment on HDFS-10285 at 12/8/17 8:44 AM:
---------------------------------------------------------------------
{quote}
Here's a rhetorical question: If managing multiple services is hard, why not
bundle oozie, spark, storm, sqoop, kafka, ranger, knox, hive server, etc in the
same process? Or ZK so HA is easier to deploy/manage?
{quote}
Few of my thoughts on this question, each of these projects build for their own
purpose, with its own spec, not for just for helping HDFS or any other single
project. And none of that projects need to access other project internal data
structures. Where as SPS is only functions for HDFS and access internal data
structures. Even forcibly separated out, we need to expose ‘for SPS only’ RPC
APIs. This strikes me to put a question in other way as well, is it make sense
to separate ReplicationMonitor as one separate process? is it fine to start
EDEK as one separate? is it ok to start other threads (like decommissioning
task) as separate processes and co-ordinate via RPC? so that NameSystem class
may become very light weight? I think its the value vs cost will decide whether
to separate or merge into single.
Coming to ZK part, As ZK is not build only for HDFS, I don’t think to have any
such thoughts. Its general purpose co-ordination system. Technically we can’t
keep monitoring services inside NN, because the worry itself is, NN may die,
need failover and so need external process to monitor. Anyway. I think the
whole discussion is about services inside a project, but not cross projects
itself, IMHO.
Here SPS providing only the missing functionality of HSM, that is end-end
policy satisfaction. So, IMV, for users it may not worth to manage additional
process to achieve that missing functionality for particular feature.
{quote}
Today, I looked at the code more closely. It can hold the lock (read lock, but
still) way too long. Notably, but not limited to, you can’t hold the lock while
doing block placement.
{quote}
Appreciate your review Daryn. I think it should be easy to address. We will
make sure to address the comment before merge? is that make sense.
{quote}
I should start sending bills to everyone who makes this fraudulent claim. .
FSDirectory#addToInodeMap imposes a nontrivial performance penalty even when
SPS is not enabled. We had to hack out the similar EZ check because it had a
noticeable performance impact esp. on startup. However now that we support EZ,
I need to revisit optimizing it.
{quote}
Thanks for review!. Nice find. Fundamentally, if sps disabled we don't even
need to load the things into Qs as no one will process them. So, adding
condition of checking enabled, can avoid even that enqueuing calls, in disabled
case. So, it will end up having one extra bool if check if disabled. With this
change, impact should be negligible IIUC. we will take this comment. Thanks
{quote}
I’m curious why it isn’t just part of the standard replication monitoring. If
the DN is told to replicate to itself, it just does the storage movement.
{quote}
That's a good question. Overall approach is exactly same as RM. RM is has its
own q build up for redundancy blocks, and Underreplication scan/check happens
at block level, it make sense. Where as in SPS, policy changes for file, so all
blocks in that file needs movement and policy check should happen
in-co-ordination with replication blocks where they stored currently. So, we
track the queues at file level here and scan/check all block for that files
together at once. Also , We wanted to provide, on the fly reconfigure feature
and we carefully thought that, we don’t want to interfere replication logic
should be given more priority than SPS work. While scheduling blocks, we
respect xmits counts, they are shared between, RM, SPS for controlling DN load.
Assignment priority given to replication/EC blocks, then SPS blocks, when
sending tasks to DN. So, as part of impact analysis, we thought of keeping SPS
in it's own thread than RM thread would be clean and safer than running in that
same loop of RM.
was (Author: umamaheswararao):
{quote}
Here's a rhetorical question: If managing multiple services is hard, why not
bundle oozie, spark, storm, sqoop, kafka, ranger, knox, hive server, etc in the
same process? Or ZK so HA is easier to deploy/manage?
{quote}
Few of my thoughts on this question, each of these projects build for their own
purpose, with its own spec, not for just for helping HDFS or any other single
project. And none of that projects need to access other project internal data
structures. Where as SPS is only functions for HDFS and access internal data
structures. Even forcibly separated out, we need to expose ‘for SPS only’ RPC
APIs. This strikes me to put a question in other way as well, is it make sense
to separate ReplicationMonitor as one separate process? is it fine to start
EDEK as one separate? is it ok to start other threads (like decommissioning
task) as separate processes and co-ordinate via RPC? so that NameSystem class
may become very light weight? I think its the value vs cost will decide whether
to separate or merge into single.
Coming to ZK part, As ZK is not build only for HDFS, I don’t think to have any
such thoughts. Its general purpose co-ordination system. Technically we can’t
keep monitoring services inside NN, because the worry itself is, NN may die,
need failover and so need external process to monitor. Anyway. I think the
whole discussion is about services inside a project, but not cross projects
itself, IMHO.
Here SPS providing only the missing functionality of HSM, that is end-end
policy satisfaction. So, IMV, for users it may not worth to manage additional
process to achieve that missing functionality for particular feature.
{quote}
Today, I looked at the code more closely. It can hold the lock (read lock, but
still) way too long. Notably, but not limited to, you can’t hold the lock while
doing block placement.
{quote}
Appreciate your review Daryn. I think it should be easy to address. We will
make sure to address the comment before merge? is that make sense.
{quote}
I’m curious why it isn’t just part of the standard replication monitoring. If
the DN is told to replicate to itself, it just does the storage movement.
{quote}
That's a good question. Overall approach is exactly same as RM. RM is has its
own q build up for redundancy blocks, and Underreplication scan/check happens
at block level, it make sense. Where as in SPS, policy changes for file, so all
blocks in that file needs movement and policy check should happen
in-co-ordination with replication blocks where they stored currently. So, we
track the queues at file level here and scan/check all block for that files
together at once. Also , We wanted to provide, on the fly reconfigure feature
and we carefully thought that, we don’t want to interfere replication logic
should be given more priority than SPS work. While scheduling blocks, we
respect xmits counts, they are shared between, RM, SPS for controlling DN load.
Assignment priority given to replication/EC blocks, then SPS blocks, when
sending tasks to DN. So, as part of impact analysis, we thought of keeping SPS
in it's own thread than RM thread would be clean and safer than running in that
same loop of RM.
> Storage Policy Satisfier in Namenode
> ------------------------------------
>
> Key: HDFS-10285
> URL: https://issues.apache.org/jira/browse/HDFS-10285
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: datanode, namenode
> Affects Versions: HDFS-10285
> Reporter: Uma Maheswara Rao G
> Assignee: Uma Maheswara Rao G
> Attachments: HDFS-10285-consolidated-merge-patch-00.patch,
> HDFS-10285-consolidated-merge-patch-01.patch,
> HDFS-10285-consolidated-merge-patch-02.patch,
> HDFS-10285-consolidated-merge-patch-03.patch,
> HDFS-SPS-TestReport-20170708.pdf,
> Storage-Policy-Satisfier-in-HDFS-June-20-2017.pdf,
> Storage-Policy-Satisfier-in-HDFS-May10.pdf,
> Storage-Policy-Satisfier-in-HDFS-Oct-26-2017.pdf
>
>
> Heterogeneous storage in HDFS introduced the concept of storage policy. These
> policies can be set on directory/file to specify the user preference, where
> to store the physical block. When user set the storage policy before writing
> data, then the blocks could take advantage of storage policy preferences and
> stores physical block accordingly.
> If user set the storage policy after writing and completing the file, then
> the blocks would have been written with default storage policy (nothing but
> DISK). User has to run the ‘Mover tool’ explicitly by specifying all such
> file names as a list. In some distributed system scenarios (ex: HBase) it
> would be difficult to collect all the files and run the tool as different
> nodes can write files separately and file can have different paths.
> Another scenarios is, when user rename the files from one effected storage
> policy file (inherited policy from parent directory) to another storage
> policy effected directory, it will not copy inherited storage policy from
> source. So it will take effect from destination file/dir parent storage
> policy. This rename operation is just a metadata change in Namenode. The
> physical blocks still remain with source storage policy.
> So, Tracking all such business logic based file names could be difficult for
> admins from distributed nodes(ex: region servers) and running the Mover tool.
> Here the proposal is to provide an API from Namenode itself for trigger the
> storage policy satisfaction. A Daemon thread inside Namenode should track
> such calls and process to DN as movement commands.
> Will post the detailed design thoughts document soon.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
