[
https://issues.apache.org/jira/browse/HDFS-7343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15602545#comment-15602545
]
Wei Zhou commented on HDFS-7343:
--------------------------------
Hi, [~andrew.wang], thanks for the replies!
{quote}
Do you also plan to capturing strace or other information? What's the
performance impact?
{quote}
I did some experiments on the overheads of strace before, it slows down the
performance for over 2X, so it's may not applicable. How about get the file
range data by hook into DataXceiver or DFSInputStream?
{quote}
and if it's a blackbox, it's much harder for admins to use it effectively.
{quote}
Yes, you are right, so we won't make it a black box. The conditions to take an
action or not will be clearly defined. Reasons will be logged when a rule is
executed or skipped. SSM will provide full info for admin to insure that SSM is
still under control.
{quote}
Have we done any prototyping and simulation of the rules engine? There are
workload generators like SWIM which could be useful here.
{quote}
We haven't done prototyping or simulation yet, it's still in the early stage.
Thanks for your advice, I will investigate it.
{quote}
What is the comparative cost-per-byte of SSD vs. HDD? I'm pretty sure it's
greater than 1.36x, meaning we might be better off buying more HDD to get more
throughput. Alternatively, buying more RAM depending on the dataset size.
{quote}
Yes, it's pretty larger than 1.36x, but it may not applicable to using more
HDDs to get the same performance:
The server may not have enough disk slots to accommodate more HDDs. So we keep
the disk number constant (the maximum slots) in the study.
HDD performance (both throughput and latency) decays dramatically compared with
SSD as the load increases. The 1.36X throughput and 1/3 latency over HDD are
measured under a proper load for HDD. The improvement can be much higher.
Big RAM does not help too much for IO intensive workloads just like cases in
the study.
{quote}
This is an example of application-specific tuning for HSM, which is a best
case. If the SSM doesn't correctly recognize the workload pattern, we won't
achieve the full 1.36x improvement.
{quote}
As above, it's a very specific case with static storage policy setting.
{quote}
I'm also unable to find the "com.yahoo.ycsb.workloads.CareWorkload" mentioned,
do you have a reference?
{quote}
Really sorry for that, it's a mistake! It should be
"com.yahoo.ycsb.workloads.CoreWorkload".
{quote}
The NameNode already does this checking, so it seems better for the enforcement
of quotas to be done in one place for consistency.
{quote}
Yes, great suggestion! Thanks!
{quote}
Maybe a user changes some files from ALL_SSD to DISK ...
{quote}
That's a good question! SSM can't handle the case automatically and
transparently, maybe we have to expose an interface for user to notify SSM to
reserve certain amount quota.
{quote}
Sorry to be this direct, but is improving average cluster throughput useful to
end users?
{quote}
Apologize for the misleading caused to you! what I supposed to mean is that
both the efficiency of the whole cluster and workloads. SSM tries to take
actions to optimize the performance of the workload because SSM can not know
whether it's profitable or not in advance. If turned out to be no improvement,
suppose the action taken won't hurts the performance as well or the overhead is
acceptable.
{quote}
I'm sure there's work to be done in the I/O path too, particularly the write
path
{quote}
SSM allows clients to use fast storage actively for write IO as shown in use
case 4. Besides, we are also investigating to improve the write performance
outside SSM. In your opinion, is there anything else for SSM to do to improve
write performance?
{quote}
I'm also not convinced that our average I/O utilization is that high to begin
with.
{quote}
Agreed that average I/O utilization is that high to begin with. Sorry for the
confusing, I'd like to clarify that for utilization we emphasize on the
utilization of fast storage instead of average I/O utilization.
{quote}
Optimistic scheduling and better resource isolation might lead to big
improvements here.
{quote}
Interesting to know this info, I will think about it as well.
{quote}
Adding a metrics collection system for OS-level metrics which needs to be
operated, managed, and deployed.
{quote}
Yes, OS-level metrics will bring difficulties for SSM. To some extent HDFS
statistic info can be used instead, so OS-level metrics is discarded.
{quote}
Potentially a Kafka dependency
{quote}
Based on the discussion with you and [~anu], it's preferred to use the polling
model (poll data directly from NNs and DNs). Do you have any
suggestions/comments on this? I can start the implementation in this way.
{quote}
I recommend stripping down the system to focus on the most important usecases
to start and growing it from there. My preference is for archival usecases.
{quote}
Yes, totally agree with this! I'll get start with this usecase. Thanks for your
suggestion!
> HDFS smart storage management
> -----------------------------
>
> Key: HDFS-7343
> URL: https://issues.apache.org/jira/browse/HDFS-7343
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Kai Zheng
> Assignee: Wei Zhou
> Attachments: HDFS-Smart-Storage-Management.pdf
>
>
> As discussed in HDFS-7285, it would be better to have a comprehensive and
> flexible storage policy engine considering file attributes, metadata, data
> temperature, storage type, EC codec, available hardware capabilities,
> user/application preference and etc.
> Modified the title for re-purpose.
> We'd extend this effort some bit and aim to work on a comprehensive solution
> to provide smart storage management service in order for convenient,
> intelligent and effective utilizing of erasure coding or replicas, HDFS cache
> facility, HSM offering, and all kinds of tools (balancer, mover, disk
> balancer and so on) in a large cluster.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
