[
https://issues.apache.org/jira/browse/HDFS-7343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15609772#comment-15609772
]
Andrew Wang commented on HDFS-7343:
-----------------------------------
I'm not opposed to a branch, but I'd like to see a design doc rev that
clarifies what usecases are being targeted for the first iteration of this
work, and the corresponding implementation plan. Like I said in an earlier
comment, I think archival usecases are the most important for end users, and
can be handled with a pretty simple system by looking at atimes / ctimes for
paths.
The SSD stuff I'm less convinced of, due to the difficulties of providing
reliable application-level SLOs. I think the best solutions here need to
leverage application-level information about working sets and priorities from
YARN and YARN apps. This is much more accurate than trying to determine the
working sets via HDFS or OS-level information.
> HDFS smart storage management
> -----------------------------
>
> Key: HDFS-7343
> URL: https://issues.apache.org/jira/browse/HDFS-7343
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Kai Zheng
> Assignee: Wei Zhou
> Attachments: HDFS-Smart-Storage-Management.pdf
>
>
> As discussed in HDFS-7285, it would be better to have a comprehensive and
> flexible storage policy engine considering file attributes, metadata, data
> temperature, storage type, EC codec, available hardware capabilities,
> user/application preference and etc.
> Modified the title for re-purpose.
> We'd extend this effort some bit and aim to work on a comprehensive solution
> to provide smart storage management service in order for convenient,
> intelligent and effective utilizing of erasure coding or replicas, HDFS cache
> facility, HSM offering, and all kinds of tools (balancer, mover, disk
> balancer and so on) in a large cluster.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
