[jira] [Commented] (HDFS-7343) HDFS smart storage management
[
https://issues.apache.org/jira/browse/HDFS-7343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17738021#comment-17738021
]
Brahma Reddy Battula commented on HDFS-7343:
[~PhiloHe] thanks.
{quote}iii) This project is under maintenance phase. We have no plan to move it
into HDFS or somewhere as subproject, or make it become an apache incubation
project.
{quote}
ok.. Not sure, we can move as apache incubation project or not.
I am in favour of moving to incubation or subproject. Let's see if anybody else
have any other thoughts on this.
> HDFS smart storage management
> -
>
> Key: HDFS-7343
> URL: https://issues.apache.org/jira/browse/HDFS-7343
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Kai Zheng
>Assignee: Wei Zhou
>Priority: Major
> Attachments: HDFS-Smart-Storage-Management-update.pdf,
> HDFS-Smart-Storage-Management.pdf,
> HDFSSmartStorageManagement-General-20170315.pdf,
> HDFSSmartStorageManagement-Phase1-20170315.pdf, access_count_tables.jpg,
> move.jpg, tables_in_ssm.xlsx
>
>
> As discussed in HDFS-7285, it would be better to have a comprehensive and
> flexible storage policy engine considering file attributes, metadata, data
> temperature, storage type, EC codec, available hardware capabilities,
> user/application preference and etc.
> Modified the title for re-purpose.
> We'd extend this effort some bit and aim to work on a comprehensive solution
> to provide smart storage management service in order for convenient,
> intelligent and effective utilizing of erasure coding or replicas, HDFS cache
> facility, HSM offering, and all kinds of tools (balancer, mover, disk
> balancer and so on) in a large cluster.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-7343) HDFS smart storage management
[ https://issues.apache.org/jira/browse/HDFS-7343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17691365#comment-17691365 ] ruiliang commented on HDFS-7343: https://github.com/Intel-bigdata/SSM This repository has been archived by the owner on Jan 4, 2023. It is now read-only. Is this item still available? Why not develop it? Or did something else take over? thank you > HDFS smart storage management > - > > Key: HDFS-7343 > URL: https://issues.apache.org/jira/browse/HDFS-7343 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Kai Zheng >Assignee: Wei Zhou >Priority: Major > Attachments: HDFS-Smart-Storage-Management-update.pdf, > HDFS-Smart-Storage-Management.pdf, > HDFSSmartStorageManagement-General-20170315.pdf, > HDFSSmartStorageManagement-Phase1-20170315.pdf, access_count_tables.jpg, > move.jpg, tables_in_ssm.xlsx > > > As discussed in HDFS-7285, it would be better to have a comprehensive and > flexible storage policy engine considering file attributes, metadata, data > temperature, storage type, EC codec, available hardware capabilities, > user/application preference and etc. > Modified the title for re-purpose. > We'd extend this effort some bit and aim to work on a comprehensive solution > to provide smart storage management service in order for convenient, > intelligent and effective utilizing of erasure coding or replicas, HDFS cache > facility, HSM offering, and all kinds of tools (balancer, mover, disk > balancer and so on) in a large cluster. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-7343) HDFS smart storage management
[ https://issues.apache.org/jira/browse/HDFS-7343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17642239#comment-17642239 ] Feilong He commented on HDFS-7343: -- [~brahmareddy], thanks for comment. i) No feature is pending. As you may know, we have made an independent project called SSM based on this Jira's design. It is basically production ready except some experimental features, like data sync, HA, etc. ii) No, kafka and ZK are not required. It is recommended to deploy SSM in HDFS cluster. The only prerequisite is user needs to deploy mysql for maintaining SSM metadata. iii) This project is under maintenance phase. We have no plan to move it into HDFS or somewhere as subproject, or make it become an apache incubation project. > HDFS smart storage management > - > > Key: HDFS-7343 > URL: https://issues.apache.org/jira/browse/HDFS-7343 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Kai Zheng >Assignee: Wei Zhou >Priority: Major > Attachments: HDFS-Smart-Storage-Management-update.pdf, > HDFS-Smart-Storage-Management.pdf, > HDFSSmartStorageManagement-General-20170315.pdf, > HDFSSmartStorageManagement-Phase1-20170315.pdf, access_count_tables.jpg, > move.jpg, tables_in_ssm.xlsx > > > As discussed in HDFS-7285, it would be better to have a comprehensive and > flexible storage policy engine considering file attributes, metadata, data > temperature, storage type, EC codec, available hardware capabilities, > user/application preference and etc. > Modified the title for re-purpose. > We'd extend this effort some bit and aim to work on a comprehensive solution > to provide smart storage management service in order for convenient, > intelligent and effective utilizing of erasure coding or replicas, HDFS cache > facility, HSM offering, and all kinds of tools (balancer, mover, disk > balancer and so on) in a large cluster. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-7343) HDFS smart storage management
[
https://issues.apache.org/jira/browse/HDFS-7343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17641615#comment-17641615
]
Brahma Reddy Battula commented on HDFS-7343:
{quote}Hi Brahma, currently we have no plan to merge this feature to upstream.
We have a repo to maintain this project. See
[https://github.com/Intel-bigdata/SSM]
{quote}
Ok. thanks.
[~zhouwei] /[~PhiloHe]
i) Any features are pending.? is it production ready..?
ii) kafka and ZK are required to deploy this.?
iii) any chance to move this as subproject or incubation to apache..?
> HDFS smart storage management
> -
>
> Key: HDFS-7343
> URL: https://issues.apache.org/jira/browse/HDFS-7343
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Kai Zheng
>Assignee: Wei Zhou
>Priority: Major
> Attachments: HDFS-Smart-Storage-Management-update.pdf,
> HDFS-Smart-Storage-Management.pdf,
> HDFSSmartStorageManagement-General-20170315.pdf,
> HDFSSmartStorageManagement-Phase1-20170315.pdf, access_count_tables.jpg,
> move.jpg, tables_in_ssm.xlsx
>
>
> As discussed in HDFS-7285, it would be better to have a comprehensive and
> flexible storage policy engine considering file attributes, metadata, data
> temperature, storage type, EC codec, available hardware capabilities,
> user/application preference and etc.
> Modified the title for re-purpose.
> We'd extend this effort some bit and aim to work on a comprehensive solution
> to provide smart storage management service in order for convenient,
> intelligent and effective utilizing of erasure coding or replicas, HDFS cache
> facility, HSM offering, and all kinds of tools (balancer, mover, disk
> balancer and so on) in a large cluster.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-7343) HDFS smart storage management
[ https://issues.apache.org/jira/browse/HDFS-7343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17210639#comment-17210639 ] Feilong He commented on HDFS-7343: -- Hi Brahma, currently we have no plan to merge this feature to upstream. We have a repo to maintain this project. See https://github.com/Intel-bigdata/SSM > HDFS smart storage management > - > > Key: HDFS-7343 > URL: https://issues.apache.org/jira/browse/HDFS-7343 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Kai Zheng >Assignee: Wei Zhou >Priority: Major > Attachments: HDFS-Smart-Storage-Management-update.pdf, > HDFS-Smart-Storage-Management.pdf, > HDFSSmartStorageManagement-General-20170315.pdf, > HDFSSmartStorageManagement-Phase1-20170315.pdf, access_count_tables.jpg, > move.jpg, tables_in_ssm.xlsx > > > As discussed in HDFS-7285, it would be better to have a comprehensive and > flexible storage policy engine considering file attributes, metadata, data > temperature, storage type, EC codec, available hardware capabilities, > user/application preference and etc. > Modified the title for re-purpose. > We'd extend this effort some bit and aim to work on a comprehensive solution > to provide smart storage management service in order for convenient, > intelligent and effective utilizing of erasure coding or replicas, HDFS cache > facility, HSM offering, and all kinds of tools (balancer, mover, disk > balancer and so on) in a large cluster. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-7343) HDFS smart storage management
[ https://issues.apache.org/jira/browse/HDFS-7343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17209992#comment-17209992 ] Brahma Reddy Battula commented on HDFS-7343: Any Update on this feature..? > HDFS smart storage management > - > > Key: HDFS-7343 > URL: https://issues.apache.org/jira/browse/HDFS-7343 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Kai Zheng >Assignee: Wei Zhou >Priority: Major > Attachments: HDFS-Smart-Storage-Management-update.pdf, > HDFS-Smart-Storage-Management.pdf, > HDFSSmartStorageManagement-General-20170315.pdf, > HDFSSmartStorageManagement-Phase1-20170315.pdf, access_count_tables.jpg, > move.jpg, tables_in_ssm.xlsx > > > As discussed in HDFS-7285, it would be better to have a comprehensive and > flexible storage policy engine considering file attributes, metadata, data > temperature, storage type, EC codec, available hardware capabilities, > user/application preference and etc. > Modified the title for re-purpose. > We'd extend this effort some bit and aim to work on a comprehensive solution > to provide smart storage management service in order for convenient, > intelligent and effective utilizing of erasure coding or replicas, HDFS cache > facility, HSM offering, and all kinds of tools (balancer, mover, disk > balancer and so on) in a large cluster. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-7343) HDFS smart storage management
[ https://issues.apache.org/jira/browse/HDFS-7343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16918144#comment-16918144 ] David Mollitor commented on HDFS-7343: -- I'd like to see all of the data placement rules condensed down into a series of discrete rules in a rules engine. https://www.baeldung.com/java-rule-engines > HDFS smart storage management > - > > Key: HDFS-7343 > URL: https://issues.apache.org/jira/browse/HDFS-7343 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Kai Zheng >Assignee: Wei Zhou >Priority: Major > Attachments: HDFS-Smart-Storage-Management-update.pdf, > HDFS-Smart-Storage-Management.pdf, > HDFSSmartStorageManagement-General-20170315.pdf, > HDFSSmartStorageManagement-Phase1-20170315.pdf, access_count_tables.jpg, > move.jpg, tables_in_ssm.xlsx > > > As discussed in HDFS-7285, it would be better to have a comprehensive and > flexible storage policy engine considering file attributes, metadata, data > temperature, storage type, EC codec, available hardware capabilities, > user/application preference and etc. > Modified the title for re-purpose. > We'd extend this effort some bit and aim to work on a comprehensive solution > to provide smart storage management service in order for convenient, > intelligent and effective utilizing of erasure coding or replicas, HDFS cache > facility, HSM offering, and all kinds of tools (balancer, mover, disk > balancer and so on) in a large cluster. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-7343) HDFS smart storage management
[ https://issues.apache.org/jira/browse/HDFS-7343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15970253#comment-15970253 ] Rakesh R commented on HDFS-7343: bq. It also can bring performance issue. For example, if a rule only cares the last 5s data, then the whole large table will be scanned in order to filter out the records needed. Thanks [~zhouwei] for the example. I understand, SSM will have the data aggregation function to keep data volume under control and the aggregation window plays an important role in this module. Since this is a performance-centric area, just a suggestion to keep the database module interface(store/retrieve/aggregation functions) pluggable during implementation so that one can do necessary changes(later) based on the test results. > HDFS smart storage management > - > > Key: HDFS-7343 > URL: https://issues.apache.org/jira/browse/HDFS-7343 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Kai Zheng >Assignee: Wei Zhou > Attachments: access_count_tables.jpg, > HDFSSmartStorageManagement-General-20170315.pdf, > HDFS-Smart-Storage-Management.pdf, > HDFSSmartStorageManagement-Phase1-20170315.pdf, > HDFS-Smart-Storage-Management-update.pdf, move.jpg, tables_in_ssm.xlsx > > > As discussed in HDFS-7285, it would be better to have a comprehensive and > flexible storage policy engine considering file attributes, metadata, data > temperature, storage type, EC codec, available hardware capabilities, > user/application preference and etc. > Modified the title for re-purpose. > We'd extend this effort some bit and aim to work on a comprehensive solution > to provide smart storage management service in order for convenient, > intelligent and effective utilizing of erasure coding or replicas, HDFS cache > facility, HSM offering, and all kinds of tools (balancer, mover, disk > balancer and so on) in a large cluster. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-7343) HDFS smart storage management
[
https://issues.apache.org/jira/browse/HDFS-7343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15960830#comment-15960830
]
Wei Zhou commented on HDFS-7343:
Thanks [~rakeshr] for reviewing the design and thoughtful comments.
{quote}
I hope these tables will be deleted after performing the aggregation functions
{quote}
Sorry for not making it clear that tables will be deleted after performing
aggregation. Through aggregation, the total number of tables in SSM are
controllable.
{quote}
minimize the number of time spec tables, how about capturing...
{quote}
Yes, it can reduce the number of tables and the costs of across table query
operations. It can be good for situations, but there can be some issues either:
- It increases the memory to store these data. For every polling from NN and
each file, you have to store an {{access_time}}, so the table looks like in the
following one (assume 5s for next polling). In fact, the {{access_time}} for
files returned from the same polling are the same, so about 1/3 memory
consumption can be saved. Also, only {{access_time}} may be not enough for
determine whether it is in the given time interval, an {{end_time}} is also
needed, so totally it will bring 100% memory overhead.
||acess_time||fid||count||
|sec-2017-03-31-12-59-05|1|3|
|sec-...|...|...|
|sec-2017-03-31-12-59-35|1|2|
|sec-2017-03-31-12-59-35|3|7|
|sec-2017-03-31-12-59-40|3|1|
|sec-2017-03-31-12-59-45|3|1|
|sec-2017-03-31-12-59-45|2|1|
|sec-2017-03-31-12-59-50|3|3|
- It also can bring performance issue. For example, if a rule only cares the
last 5s data, then the whole large table will be scanned in order to filter out
the records needed.
{quote}
maintain separate tables per units of time
{quote}
Yes, it's much better to do this if data are tracked in the way mentioned above.
> HDFS smart storage management
> -
>
> Key: HDFS-7343
> URL: https://issues.apache.org/jira/browse/HDFS-7343
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Kai Zheng
>Assignee: Wei Zhou
> Attachments: access_count_tables.jpg,
> HDFSSmartStorageManagement-General-20170315.pdf,
> HDFS-Smart-Storage-Management.pdf,
> HDFSSmartStorageManagement-Phase1-20170315.pdf,
> HDFS-Smart-Storage-Management-update.pdf, move.jpg, tables_in_ssm.xlsx
>
>
> As discussed in HDFS-7285, it would be better to have a comprehensive and
> flexible storage policy engine considering file attributes, metadata, data
> temperature, storage type, EC codec, available hardware capabilities,
> user/application preference and etc.
> Modified the title for re-purpose.
> We'd extend this effort some bit and aim to work on a comprehensive solution
> to provide smart storage management service in order for convenient,
> intelligent and effective utilizing of erasure coding or replicas, HDFS cache
> facility, HSM offering, and all kinds of tools (balancer, mover, disk
> balancer and so on) in a large cluster.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-7343) HDFS smart storage management
[
https://issues.apache.org/jira/browse/HDFS-7343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15958923#comment-15958923
]
Rakesh R commented on HDFS-7343:
Thanks [~zhouwei] for more details about the data points.
bq. Create a table to store the info and insert the table name into table
access_count_table.
It looks like lot of tables will be created to capture time period details,
sec_1...sec_n, min_1...min_n, hour_1...hour_n, day_1day_n,
month_1...month_12 etc. I hope these tables will be deleted after performing
the aggregation functions. Again, it may exhaust DB by growing the number of
tables if the aggregation time is longer, right?. Just a plain thought to
minimize the number of time spec tables, how about capturing {{access_time}} as
a column field and update {{access_time}} of respective {{fid}}? I think, using
the {{access_time}} attribute, we would be able to filter out specific
{{fid_access_count}} between a certain {{start_time}} and {{end_time}}.
Table {{seconds_level}} => composite key {{acess_time}} and {{fid}} to uniquely
identify each row in the table.
||acess_time||fid||count||
|sec-2017-03-31-12-59-45|3|1|
|sec-2017-03-31-12-59-45|2|1|
Again, for faster aggregation function probably we could maintain separate
{{tables per units of time}} like below. After the aggregate function, we could
delete those rows used for aggregation.
(1) seconds_level
(2) minutes_level
(3) hours_level
(4) days_level
(5) weeks_level
(6) months_level
(7) years_level
> HDFS smart storage management
> -
>
> Key: HDFS-7343
> URL: https://issues.apache.org/jira/browse/HDFS-7343
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Kai Zheng
>Assignee: Wei Zhou
> Attachments: access_count_tables.jpg,
> HDFSSmartStorageManagement-General-20170315.pdf,
> HDFS-Smart-Storage-Management.pdf,
> HDFSSmartStorageManagement-Phase1-20170315.pdf,
> HDFS-Smart-Storage-Management-update.pdf, move.jpg, tables_in_ssm.xlsx
>
>
> As discussed in HDFS-7285, it would be better to have a comprehensive and
> flexible storage policy engine considering file attributes, metadata, data
> temperature, storage type, EC codec, available hardware capabilities,
> user/application preference and etc.
> Modified the title for re-purpose.
> We'd extend this effort some bit and aim to work on a comprehensive solution
> to provide smart storage management service in order for convenient,
> intelligent and effective utilizing of erasure coding or replicas, HDFS cache
> facility, HSM offering, and all kinds of tools (balancer, mover, disk
> balancer and so on) in a large cluster.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-7343) HDFS smart storage management
[
https://issues.apache.org/jira/browse/HDFS-7343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15952175#comment-15952175
]
Wei Zhou commented on HDFS-7343:
Thanks for [~eddyxu] and [~drankye] for the discussion on SSM design. On file
{{accessCount}}, we plan to use SQL database to store and track these data. As
showed in the chart,
- SSM polls {{accessCount}} data from NN to get file access count info happened
in the time interval (for example, 5s).
- Create a table to store the info and insert the table name into table
{{access_count_table}}.
- Then file access count of last time interval can be calculated by
accumulating data in tables that their {{start time}} and {{end time}} falls in
the interval.
- To control the total amount of data, second-level of {{accessCount}} tables
will be aggregated into minute-level, hour-level, day-level, month-level and
year-level. The longer the time from now, the larger the granularity for
aggregation. More accurate data kept for near now than long ago.
The excel file attached demos more info about the data and tables maintained in
SSM.
!access_count_tables.jpg!
> HDFS smart storage management
> -
>
> Key: HDFS-7343
> URL: https://issues.apache.org/jira/browse/HDFS-7343
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Kai Zheng
>Assignee: Wei Zhou
> Attachments: access_count_tables.jpg,
> HDFSSmartStorageManagement-General-20170315.pdf,
> HDFS-Smart-Storage-Management.pdf,
> HDFSSmartStorageManagement-Phase1-20170315.pdf,
> HDFS-Smart-Storage-Management-update.pdf, move.jpg, tables_in_ssm.xlsx
>
>
> As discussed in HDFS-7285, it would be better to have a comprehensive and
> flexible storage policy engine considering file attributes, metadata, data
> temperature, storage type, EC codec, available hardware capabilities,
> user/application preference and etc.
> Modified the title for re-purpose.
> We'd extend this effort some bit and aim to work on a comprehensive solution
> to provide smart storage management service in order for convenient,
> intelligent and effective utilizing of erasure coding or replicas, HDFS cache
> facility, HSM offering, and all kinds of tools (balancer, mover, disk
> balancer and so on) in a large cluster.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-7343) HDFS smart storage management
[
https://issues.apache.org/jira/browse/HDFS-7343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15945026#comment-15945026
]
Kai Zheng commented on HDFS-7343:
-
Thanks [~eddyxu] for your time reviewing this and reading the docs! Very good
questions. :)
bq. SSM needs to maintain stats of each file / block in NN ...
This needs to clarify some bit. SSM server needs to maintain states of each
file in SSM side. In NN side, it just maintains access count records happened
for some files recently. For example, for a time period if 100 files are read,
then in the NN AccessCounterMap there're only 100 entries; then when SSM
retrieves and polls to SSM, NN will remove the 100 entries if necessary (given
a threshold). It's SSM side to maintain and aggregate all the access counts for
all files.
We want SSM to maintain states and access count aggregates for all files, but
in a compact and concise approach, avoiding the memory consumption like O(n).
It's possible because SSM not like NN, the stored states info can be grouped
and shared based on common folders and access patterns.
bq. when SSM pulling these stats from NN, what kind of overhead we are
expecting?
An approach mentioned above is to let SSM fetch file meta lists by many
iterations across the inode tree, at a speed that NN can affords.
bq. I think that in the general design, we should work on define a good
interface time series store for metrics, instead of specifying RocksDB. RocksDB
might be a good implementation for now.
I thought a generic interface to abstract away concrete database impl is a good
idea. RocksDB may be not a good fit, you're right we're still thinking about a
better one. Initially we want to use a embedded SQL database to store the
collected file states for flexible query, but looks like no good one for Apache
space. The needed time series feature (like continuous aggregation/compaction,
window sliding count) would be easy to come up for our purpose given a SQL
database. Do you have any SQL or time series database to recommend? Thanks!
bq. it is not clear to me why both SSM and NN need to persist metrics in two
separated rocksdb? ...
We know that current HDFS NameNode now supports to collect many kinds of
metrics from DNs and we also want NN to record certain file access counts. Such
metrics can suffer from loss during NN switch or restart. To minimize the
memory consumption on this we suggest enhancing NN to allow it to spill metircs
to a local RocksDB. But I think we can consider this separately to avoid the
confusion here in SSM design. Sounds good?
bq. How stale or extensible the syntax will be? would the syntax be easy for
other applications to generate / parse / validate and etc?
We want to make the rule extensible, concise and flexible. The Json and YAML
are good as data formats, but could be very verbose to describe a rule, which
is why I haven't yet seen a DSL well written in them. You have a good
suggestion here, to support applications to generate/validate rules, to do
that, a Java library would be needed. Sounds good?
bq. How to update the accessCount ? how many samples of accessCount to be
maintained the accessCount during an interval? What if SSM failed to pull for
the the accessCount?
Regarding how to implement {{accessCount}} is worth to be addressed separately
and we'll follow here later. One thing to be noted now is, SSM or its
applications should be able to suffer from access count loss. In case some
files lose some read counts, the impact would be that SSM may not adjust
storage options for them in first time, which should be fine and no data hurts.
Anyhow we need good trade-off between access count accuracy and system
complexity/cost.
bq. Maybe we can define SSM rules as soft rules, while make HSM/EC and etc
rules as hard rules?
I might catch you so agree SSM rules are soft rules that may not have to
happen, but not very well to understand HSM/EC rules as hard rules. I thought
SSM aims to ease the deployment and admin operations for HSM/EC facilities.
> HDFS smart storage management
> -
>
> Key: HDFS-7343
> URL: https://issues.apache.org/jira/browse/HDFS-7343
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Kai Zheng
>Assignee: Wei Zhou
> Attachments: HDFSSmartStorageManagement-General-20170315.pdf,
> HDFS-Smart-Storage-Management.pdf,
> HDFSSmartStorageManagement-Phase1-20170315.pdf,
> HDFS-Smart-Storage-Management-update.pdf, move.jpg
>
>
> As discussed in HDFS-7285, it would be better to have a comprehensive and
> flexible storage policy engine considering file attributes, metadata, data
> temperature, storage type, EC codec, available hardware capabilities,
> user/application preference and etc.
> Modified the title for re-purpose.
> We'd extend this effor
[jira] [Commented] (HDFS-7343) HDFS smart storage management
[
https://issues.apache.org/jira/browse/HDFS-7343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15941287#comment-15941287
]
Lei (Eddy) Xu commented on HDFS-7343:
-
Hi, [~drankye] and [~zhouwei]
Thanks for the updated design docs. It is a very good write up.
I have a few questions.
* IIUC, SSM needs to maintain stats of each file / block in NN, which adds O(n)
overhead to JVM heap, where n is {# of blocks or # of files}. Additionally,
when SSM pulling these stats from NN, what kind of overhead we are expecting?
If SCM pulls the full block map / file namespace, would that be a performance
hit to NN?
* I think that in the general design, we should work on define a good interface
time series store for metrics, instead of specifying RocksDB. RocksDB might be
a good implementation for now.
* In Figure 1 of general design, it is not clear to me why both SSM and NN need
to persist metrics in two _separated_ rocksdb? If NN needs to persist metrics
to rocksdb, does that mean both ANN and SNN in a HA setup need to persist them?
What about the HA of SSM?
* Rules. How stale or extensible the syntax will be? would the syntax be easy
for other applications to generate / parse / validate and etc? What would be
your opinion of using json or YAML for the syntax? Would it be possible that
when adding a rule, SSM can verify the rule based on HSM/EC/SPS policies and
etc?
* in Phase 1 design, {{file.accessCount(interval)}}. How to update the
accessCount ? how many samples of accessCount to be maintained the accessCount
during an interval? What if SSM failed to pull for the the accessCount?
* Maybe we can define SSM rules as _soft_ rules, while make HSM/EC and etc
rules as hard rules?
Thanks!
> HDFS smart storage management
> -
>
> Key: HDFS-7343
> URL: https://issues.apache.org/jira/browse/HDFS-7343
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Kai Zheng
>Assignee: Wei Zhou
> Attachments: HDFSSmartStorageManagement-General-20170315.pdf,
> HDFS-Smart-Storage-Management.pdf,
> HDFSSmartStorageManagement-Phase1-20170315.pdf,
> HDFS-Smart-Storage-Management-update.pdf, move.jpg
>
>
> As discussed in HDFS-7285, it would be better to have a comprehensive and
> flexible storage policy engine considering file attributes, metadata, data
> temperature, storage type, EC codec, available hardware capabilities,
> user/application preference and etc.
> Modified the title for re-purpose.
> We'd extend this effort some bit and aim to work on a comprehensive solution
> to provide smart storage management service in order for convenient,
> intelligent and effective utilizing of erasure coding or replicas, HDFS cache
> facility, HSM offering, and all kinds of tools (balancer, mover, disk
> balancer and so on) in a large cluster.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-7343) HDFS smart storage management
[ https://issues.apache.org/jira/browse/HDFS-7343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15940055#comment-15940055 ] Kai Zheng commented on HDFS-7343: - Thanks Anoop for the thoughts and good questions! bq. Tracking of data hotness and movement is at what level? Block level? Or only file level? We want to support block level, but not sure how would it be useful, since for modern HDFS data files, they may be mostly single block files. If it's typical that there are many large files of many blocks of different hotness in typical clusters, we may add to consider block level support. Wonder if [~andrew.wang] could give some comments about this. Thanks. bq. HBase, being a user, we will compact our HFiles into one ... After discussed with Anoop offline, I got the point. HBase itself does fine level cache stuffs so it won't need the help of HDFS cache, therefore SSM can't help HBase in the cache path. In other cases, it's possible that in HBase there are cold tables and even cold regions so in underlying HDFS there could be HDFS blocks of different temperatures, then HDFS HSM could help. SSM aims to ease HDFS-HSM deployment and usage, so SSM can help HBase in such cases. For HBase, I thought [~anu] has some considerations. Anu could you cast your points about this? Thanks! > HDFS smart storage management > - > > Key: HDFS-7343 > URL: https://issues.apache.org/jira/browse/HDFS-7343 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Kai Zheng >Assignee: Wei Zhou > Attachments: HDFSSmartStorageManagement-General-20170315.pdf, > HDFS-Smart-Storage-Management.pdf, > HDFSSmartStorageManagement-Phase1-20170315.pdf, > HDFS-Smart-Storage-Management-update.pdf, move.jpg > > > As discussed in HDFS-7285, it would be better to have a comprehensive and > flexible storage policy engine considering file attributes, metadata, data > temperature, storage type, EC codec, available hardware capabilities, > user/application preference and etc. > Modified the title for re-purpose. > We'd extend this effort some bit and aim to work on a comprehensive solution > to provide smart storage management service in order for convenient, > intelligent and effective utilizing of erasure coding or replicas, HDFS cache > facility, HSM offering, and all kinds of tools (balancer, mover, disk > balancer and so on) in a large cluster. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-7343) HDFS smart storage management
[ https://issues.apache.org/jira/browse/HDFS-7343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15939992#comment-15939992 ] Anoop Sam John commented on HDFS-7343: -- Tracking of data hotness and movement is at what level? Block level? Or only file level? HBase, being a user, we will compact our HFiles into one (major compaction) and if the tracking is at file level, it might not be that useful for HBase. Just saying > HDFS smart storage management > - > > Key: HDFS-7343 > URL: https://issues.apache.org/jira/browse/HDFS-7343 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Kai Zheng >Assignee: Wei Zhou > Attachments: HDFSSmartStorageManagement-General-20170315.pdf, > HDFS-Smart-Storage-Management.pdf, > HDFSSmartStorageManagement-Phase1-20170315.pdf, > HDFS-Smart-Storage-Management-update.pdf, move.jpg > > > As discussed in HDFS-7285, it would be better to have a comprehensive and > flexible storage policy engine considering file attributes, metadata, data > temperature, storage type, EC codec, available hardware capabilities, > user/application preference and etc. > Modified the title for re-purpose. > We'd extend this effort some bit and aim to work on a comprehensive solution > to provide smart storage management service in order for convenient, > intelligent and effective utilizing of erasure coding or replicas, HDFS cache > facility, HSM offering, and all kinds of tools (balancer, mover, disk > balancer and so on) in a large cluster. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-7343) HDFS smart storage management
[
https://issues.apache.org/jira/browse/HDFS-7343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15939969#comment-15939969
]
Kai Zheng commented on HDFS-7343:
-
Thanks [~zhouwei] for updating the docs and addressing the questions.
[~andrew.wang] has a good asking:
bq. How often does the SSM need to poll the NN to get information? How much
information each time? Some triggers might require listing a lot of the
namespace.
I had an offline discussion about this with Wei, [~HuafengWang] and [~Sammi].
To avoid listing a lot of the namespace, one way is to let SSM server have a
copy of namespace. This would allow to evolve SSM towards really powerful and
doing a lot of things without affecting NameNode very much. So the questions
now comes to:
* What kinds of file meta info are needed to make SSM function;
* How to fetch the namespace to prepare for the base;
* Given the namespace base, how to incrementally get updated time by time.
h4. What file meta info are needed
Roughly file meta info like filename, xattributes, permissions, atime/mtime,
length and etc. but not including block list info. So it's part of the
namespace, very compacted, not the whole, should be much smaller than the whole.
h4. How to fetch the namespace to prepare for the base
There're two ways for SSM to do this:
* Fetch the fsimage and load the needed info by itself.
* Get file status list from NameNode page by page at a NN affordable speed.
There doesn't urgent need that requires SSM to complete the work in short time.
It can take some days in a large cluster and the work can happen in idle time.
In either way, SSM can take quite much time to finish the work so a {{safe
mode}} would be needed to allow the period. During safe mode time, SSM won't
allow relevant operations.
h4. Given the namespace base, how to incrementally get updated time by time
I thought both [~andrew.wang] and [~eddyxu] (by offline) suggested we can use
the inotify support. It's a good chance to exercise the function.
Note this will consider NameNode federation. And the maintained files info will
be checkpointed to HDFS ensure it's reliability. When SSM is restarted on other
nodes, the data can be reloaded from HDFS other than from NameNodes again.
Any comment? Thanks!
> HDFS smart storage management
> -
>
> Key: HDFS-7343
> URL: https://issues.apache.org/jira/browse/HDFS-7343
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Kai Zheng
>Assignee: Wei Zhou
> Attachments: HDFSSmartStorageManagement-General-20170315.pdf,
> HDFS-Smart-Storage-Management.pdf,
> HDFSSmartStorageManagement-Phase1-20170315.pdf,
> HDFS-Smart-Storage-Management-update.pdf, move.jpg
>
>
> As discussed in HDFS-7285, it would be better to have a comprehensive and
> flexible storage policy engine considering file attributes, metadata, data
> temperature, storage type, EC codec, available hardware capabilities,
> user/application preference and etc.
> Modified the title for re-purpose.
> We'd extend this effort some bit and aim to work on a comprehensive solution
> to provide smart storage management service in order for convenient,
> intelligent and effective utilizing of erasure coding or replicas, HDFS cache
> facility, HSM offering, and all kinds of tools (balancer, mover, disk
> balancer and so on) in a large cluster.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-7343) HDFS smart storage management
[
https://issues.apache.org/jira/browse/HDFS-7343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15925928#comment-15925928
]
Wei Zhou commented on HDFS-7343:
Thanks [~andrew.wang] for these great questions. These questions are answered
based on the Phase 1 design.
{quote}
One possible staging:
{quote}
Agreed, that's indeed the core parts of this phase. I updated it in the new doc
and added in some other items.
{quote}
Could you describe how you will satisfy usecases 4 and 5 in more detail?
{quote}
These two cases will not be supported in Phase 1. They are based on the
assumption that enough metrics been collected from HDFS.
{quote}
the complete set of NameNode changes required
{quote}
The change mainly includes:
# Implement metrics collecting logical. Only file access count info need
special treatment for Phase 1.
# Provide supporting/API for SSM to get these metrics
# Implement a RocksDB. It mainly used for store metrics data during SSM
crashed, reboots. During these cases metrics data accumulates in NN for a
while. It does not required to be checkpoint into HDFS. Though NN can stores
all the data into it for other purposes.
It won't bring large change to NN logical existed.
{quote}
The lack of HA means this will be a non-starter
{quote}
Sorry for not making it clear. Hadoop HA is supported in Phase 1.
{quote}
Why are ChangeStoragePolicy and EnforceStoragePolicy separate actions?
{quote}
I'm trying to make it as basic as possible that high-level operations can be
realized by using these simple actions.
{quote}
What is the set of metrics do you plan to collect from HDFS?
{quote}
Listed in Section "Objects" phase 1 design document, their attributes are the
metrics planned to collect.
{quote}
centralized read statistics... Is there a plan to implement this?
{quote}
Yes, we implement a statistics (file access count) for file level cache.
{quote}
description of the trigger syntax
{quote}
It is an event-based mechanism, SSM will do the rule condition check when the
corresponding event happened. The event can be any events happened in HDFS, but
in phase 1 we are going to support timer events and file level events like
{{file crate}} through INotify.
{quote}
• How often does the SSM wake up to check rules?
{quote}
If a rule has some kind of trigger then SSM will check the rule when the
corresponding event happens. If the rule specifies no trigger then it will be
checked periodically. By default, we have a configurable time interval to check
these rules. An optimized way for rules that have connect with time is to check
the rule based on the time set, for example, if a rule is to check file last
modify time exceeds 30 days then, the rule only needed to be checked in day
level.
{quote}
• Could you provide a complete list of conditions that are planned?
{quote}
The condition can be any boolean expression. You can use the metrics/internal
variables provided to set up the expression you need.
{quote}
How do you plan to implement accessCount over a time range?
{quote}
Please reference the Section "Rule/Execution flow" in phase 1 design document.
{quote}
Any other new metrics or information you plan to add to HDFS as part of this
work?
{quote}
No metrics will be added into HDFS besides file access count according to
current phase 1 design.
{quote}
Prefer we use atime or ctime rather than "age", since they're more specific.
{quote}
Yes, we'll use these instead of 'age'. We thought 'age' would be more user
(those who have no idea of the low-level implementation) friendly.
{quote}
• Could you provide a complete definition of the object matching syntax?
{quote}
Listed in Section "Objects" phase 1 design document
{quote}
• Do rules support basic boolean operators like AND, OR, NOT for objects
and conditions?
{quote}
Yes, they are supported.
{quote}
• Aren't many of these matches going to require listing the complete
filesystem? Or are you planning to use HDFS inotify?
{quote}
First, the rule itself should be specific instead of over general.
Second, in SSM we can do many optimizations to decrease the needs of listing
the filesystem. From example, if a rule is 'cache file under directory /foo if
the file access count in 3 mins larger than 10', then there is no need to list
all the files under /foo, only take the files that been accessed into account.
We also use HDFS inotify to track filesystem changes and update it in SSM local
info library, this also reduces the needes to listing the filesystem for latest
states.
{quote}
• The "cache" action is underspecified, what cache pool is used?
{quote}
For simple of demonstration, we use just 'cache' (a default pool is used).
Parameters have to be specified in order to customize the behavior.
{quote}
• Can actions happen concurrently? Is there a way of limiting concurrency?
{quote}
Yes, it can happen concurrent
[jira] [Commented] (HDFS-7343) HDFS smart storage management
[ https://issues.apache.org/jira/browse/HDFS-7343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15867046#comment-15867046 ] Wei Zhou commented on HDFS-7343: After discussed with [~andrew.wang] and [~drankye], in order to make the design doc more friendly to read and maintain, I'll divide the previous doc into two: one focus on high-level design and the other one focus on phase 1. Phase 1 targets at three scenarios: * Cache hot data * Move hot data to faster storage * Archive cold data Based on these cases, enough details will be provided to answer Andrew's great questions. Thanks! > HDFS smart storage management > - > > Key: HDFS-7343 > URL: https://issues.apache.org/jira/browse/HDFS-7343 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Kai Zheng >Assignee: Wei Zhou > Attachments: HDFS-Smart-Storage-Management.pdf, > HDFS-Smart-Storage-Management-update.pdf, move.jpg > > > As discussed in HDFS-7285, it would be better to have a comprehensive and > flexible storage policy engine considering file attributes, metadata, data > temperature, storage type, EC codec, available hardware capabilities, > user/application preference and etc. > Modified the title for re-purpose. > We'd extend this effort some bit and aim to work on a comprehensive solution > to provide smart storage management service in order for convenient, > intelligent and effective utilizing of erasure coding or replicas, HDFS cache > facility, HSM offering, and all kinds of tools (balancer, mover, disk > balancer and so on) in a large cluster. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-7343) HDFS smart storage management
[
https://issues.apache.org/jira/browse/HDFS-7343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15849385#comment-15849385
]
Andrew Wang commented on HDFS-7343:
---
Hi Wei, thanks for posting the new doc. At a high-level I think the scope of
this project is really big. The doc doesn't go into sufficient implementation
detail for me to really understand what's involved in the first development
phase. Splitting this into further phases would help. One possible staging:
* Define triggers and the data required to implement them
* Data collection from HDFS
* Implement the different actions
* The rules syntax and definition
Other comments:
* Could you describe how you will satisfy usecases 4 and 5 in more detail?
* Please describe the complete set of NameNode changes required, particularly
the use of LevelDB and additional state
* The lack of HA means this will be a non-starter for many production
deployments, could you comment on the difficulty of implementing HA? This
should really be covered in the initial design.
* Why are the StorageManager and CacheManager treated as separate components?
If the StorageManager incorporates storage policies, EC, S3, etc, it already
seems quite general
* Why are ChangeStoragePolicy and EnforceStoragePolicy separate actions? Is
there a usecase to changing the SP but not moving the data?
Metric collection:
* What is the set of metrics do you plan to collect from HDFS?
* Right now we don't have centralized read statistics which would be obviously
useful to implement a caching policy. Is there a plan to implement this?
Triggers:
* Could you provide a complete description of the trigger syntax? Notably, I
don't see a way to "hash" the time in the examples.
* How often does the SSM wake up to check rules?
Conditions:
* Could you provide a complete list of conditions that are planned?
* How do you plan to implement accessCount over a time range?
* Any other new metrics or information you plan to add to HDFS as part of this
work?
* Prefer we use atime or ctime rather than "age", since they're more specific
Object matching:
* Could you provide a complete definition of the object matching syntax?
* Do rules support basic boolean operators like AND, OR, NOT for objects and
conditions?
* Is there a reason you chose to implement regex matches rather than file
globbing for path matching? Are these regexs on the full path, or per path
component?
* Aren't many of these matches going to require listing the complete
filesystem? Or are you planning to use HDFS inotify?
Actions:
* The "cache" action is underspecified, what cache pool is used?
* How often does the SSM need to poll the NN to get information? How much
information each time? Some triggers might require listing a lot of the
namespace.
* Can actions happen concurrently? Is there a way of limiting concurrency?
* Can you run multiple actions in a rule? Is there a syntax for defining
"functions"?
* Are there substitutions that can be used to reference the filename, e.g.
"${file}"? Same for DN objects, the diskbalancer needs the DN host:port.
Operational questions:
* Is there an audit log for actions taken by the SSM?
* Is there a way to see when each action started, stopped, and its status?
* How are errors and logs from actions exposed?
* What metrics are exposed by the SSM?
* Why are there configuration options to enable individual actions? Isn't this
behavior already defined by the rules file?
* Why does the SSM need a "dfs.ssm.enabled" config? Is there a usecase for
having an SSM service started, but not enabled?
* Is the rules file dynamically refreshable?
* What do we do if the rules file is malformed? What do we do if there are
conflicting rules or multiple matches?
* dfs.ssm.msg.datanode.interval is described as the polling interval for the
NN, typo?
* What happens if multiple SSMs are accidentally started?
> HDFS smart storage management
> -
>
> Key: HDFS-7343
> URL: https://issues.apache.org/jira/browse/HDFS-7343
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Kai Zheng
>Assignee: Wei Zhou
> Attachments: HDFS-Smart-Storage-Management.pdf,
> HDFS-Smart-Storage-Management-update.pdf, move.jpg
>
>
> As discussed in HDFS-7285, it would be better to have a comprehensive and
> flexible storage policy engine considering file attributes, metadata, data
> temperature, storage type, EC codec, available hardware capabilities,
> user/application preference and etc.
> Modified the title for re-purpose.
> We'd extend this effort some bit and aim to work on a comprehensive solution
> to provide smart storage management service in order for convenient,
> intelligent and effective utilizing of erasure coding or replicas, HDFS cache
> facility, HSM offering, and all kinds of tools (balancer, mover, dis
[jira] [Commented] (HDFS-7343) HDFS smart storage management
[
https://issues.apache.org/jira/browse/HDFS-7343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15814478#comment-15814478
]
Wei Zhou commented on HDFS-7343:
Thanks [~anu] for these insightful questions!
{quote}
Would you be able to count how many times a particular rule was triggered in a
given time window ?
{quote}
Sure, it's a very useful feature, we will implement this.
{quote}
store them in SSM instead of storing it in NN, or feel free to store it as a
file on HDFS.
{quote}
Yes, it also makes HA supporting much easier by store rule in HDFS. We will
implement it in this way. Thanks!
{quote}
Are you now saying we will support HA ?
{quote}
Sorry for not making it clear. Phase 1 supports HA.
{quote}
when the rule gets written may be there is no issue, but as the file count
increases this becomes a problem.
{quote}
Yes, I agree. So it followed by a {{Second}}. We continuing tracks the state of
rules and give feedbacks when it becomes a problem.
{quote}
I thought we did not want to restrict the number of times a rule fires since
that would introduce uncertainty.
{quote}
Agreed. I tend to not implement it either, even it could be a potential dirty
solution to anti rules that out of control.
{quote}
Why not just rely on background SSM logic and rely on the rules doing the right
thing ?
{quote}
HDFS client talks to SSM only (at least for now) when it wants to query a
recommended file storage policy from SSM before creating a file. The storage
policy that SSM would return is controlled by rule. HDFS Client then set the
file storage policy to the recommended one explicitly. If HDFS client can not
connect to SSM, then client just create the file with system default policy. So
there is no connection between SSM and the file writing IO. Sorry, I'm not very
clear about your question.
Thanks again!
> HDFS smart storage management
> -
>
> Key: HDFS-7343
> URL: https://issues.apache.org/jira/browse/HDFS-7343
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Kai Zheng
>Assignee: Wei Zhou
> Attachments: HDFS-Smart-Storage-Management-update.pdf,
> HDFS-Smart-Storage-Management.pdf, move.jpg
>
>
> As discussed in HDFS-7285, it would be better to have a comprehensive and
> flexible storage policy engine considering file attributes, metadata, data
> temperature, storage type, EC codec, available hardware capabilities,
> user/application preference and etc.
> Modified the title for re-purpose.
> We'd extend this effort some bit and aim to work on a comprehensive solution
> to provide smart storage management service in order for convenient,
> intelligent and effective utilizing of erasure coding or replicas, HDFS cache
> facility, HSM offering, and all kinds of tools (balancer, mover, disk
> balancer and so on) in a large cluster.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-7343) HDFS smart storage management
[ https://issues.apache.org/jira/browse/HDFS-7343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15813615#comment-15813615 ] Anu Engineer commented on HDFS-7343: bq. Also, it makes SSM more stable as these data stored in NN. When SSM node failure happened, we can simply launch another instance on another node. I do see now where this thought came from. However I think that SSM should be able to stand independent and not rely on Namenode. Here are some reasons I can think of. 1. SSM can be implemented with no changes to NN, that makes for easier and faster development 2. No added complexity in Namenode 3. Moving State from SSM to Namenode just makes SSM simpler, but makes Namenode to that degree more complicated. So while I have a better understanding of the motivation, making NN store rules and metrics which are needed by SSM feels like a wrong choice. As I said earlier, if you want to run this in other scenarios then this dependency on NN makes it hard. For example, someone is running SSM in cloud and is using a cloud native file system instead of Namenode. bq. This brings in uncertainty to users. We can implement automatical rule-engine level throttle in Phase 2. if a rule is misbehaving then NN will become slow, thus it brings uncertainty. But I am fine with the choice of postponing this to a later stage. Would you be able to count how many times a particular rule was triggered in a given time window ? That would be useful to debug this issue. bq. Rule is the core part for SSM to function. For convenient and reliable consideration, it's better to store it in NN to keep SSM simple and stateless as suggested. Rules are core part of SSM.So let us store them in SSM instead of storing it in NN, or feel free to store it as a file on HDFS. Modifying Namenode to store config of some other service will make Namenode a dumping ground of config for all other services. bq. Yes, good question. We can support HA by many ways, for example, periodically checkpoint the data to HDFS or store the data in the same way as edit log. Sorry, I am not unable to understand this response clearly. Are you now saying we will support HA ? bq. First, we provide some verification mechanism when adding some rule. For example, we can give the user some warning when the candidate files of an action (such as move) exceeding some certain value. This is a classic time of check to time of use problem. When the rule gets written may be there is no issue, but as the file count increases this becomes a problem. bq. Second, the execution state and other info related info can also be showed in the dashboard or queried. It's convenient for users to track the status and take actions accordingly. It's also very good to implement a timeout mechanism. Agreed, but now have we not introduced the uncertainty issue back into the solution ? I thought we did not want to restrict the number of times a rule fires since that would introduce uncertainty. bq. HDFS client will bypass SSM when the query fails, then the client goes back to the original working flow. It has almost no effect on the existing I/O. So then the SSM rules are violated ? How does it deal with that issue ? since you have to deal with SSM being down why have the HDFS client even talk to SSM in an I/O path ? Why not just rely on background SSM logic and rely on the rules doing the right thing ? Thanks for sharing the graph, I appreciate it. > HDFS smart storage management > - > > Key: HDFS-7343 > URL: https://issues.apache.org/jira/browse/HDFS-7343 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Kai Zheng >Assignee: Wei Zhou > Attachments: HDFS-Smart-Storage-Management-update.pdf, > HDFS-Smart-Storage-Management.pdf, move.jpg > > > As discussed in HDFS-7285, it would be better to have a comprehensive and > flexible storage policy engine considering file attributes, metadata, data > temperature, storage type, EC codec, available hardware capabilities, > user/application preference and etc. > Modified the title for re-purpose. > We'd extend this effort some bit and aim to work on a comprehensive solution > to provide smart storage management service in order for convenient, > intelligent and effective utilizing of erasure coding or replicas, HDFS cache > facility, HSM offering, and all kinds of tools (balancer, mover, disk > balancer and so on) in a large cluster. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-7343) HDFS smart storage management
[
https://issues.apache.org/jira/browse/HDFS-7343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15813506#comment-15813506
]
Wei Zhou commented on HDFS-7343:
{quote}
though I'd encourage you to think about how far we can get with a stateless
system (possibly by pushing more work into the NN and DN).
{quote}
>From this words, I myself thought it's better to store these data in NN to
>approach the stateless system suggested by Andrew. I did not mean that Andrew
>said it's better to do it in this way. Sorry for my poor English.
> HDFS smart storage management
> -
>
> Key: HDFS-7343
> URL: https://issues.apache.org/jira/browse/HDFS-7343
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Kai Zheng
>Assignee: Wei Zhou
> Attachments: HDFS-Smart-Storage-Management-update.pdf,
> HDFS-Smart-Storage-Management.pdf, move.jpg
>
>
> As discussed in HDFS-7285, it would be better to have a comprehensive and
> flexible storage policy engine considering file attributes, metadata, data
> temperature, storage type, EC codec, available hardware capabilities,
> user/application preference and etc.
> Modified the title for re-purpose.
> We'd extend this effort some bit and aim to work on a comprehensive solution
> to provide smart storage management service in order for convenient,
> intelligent and effective utilizing of erasure coding or replicas, HDFS cache
> facility, HSM offering, and all kinds of tools (balancer, mover, disk
> balancer and so on) in a large cluster.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-7343) HDFS smart storage management
[ https://issues.apache.org/jira/browse/HDFS-7343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15812285#comment-15812285 ] Anu Engineer commented on HDFS-7343: bq. Some methodologies existed in NN to collect metrics and events from DNs, so it's better to store these data in NN to make SSM stateless as suggested by Andrew. Could you please point me to this suggestion from Andrew. I would like to understand what he is saying better. > HDFS smart storage management > - > > Key: HDFS-7343 > URL: https://issues.apache.org/jira/browse/HDFS-7343 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Kai Zheng >Assignee: Wei Zhou > Attachments: HDFS-Smart-Storage-Management-update.pdf, > HDFS-Smart-Storage-Management.pdf, move.jpg > > > As discussed in HDFS-7285, it would be better to have a comprehensive and > flexible storage policy engine considering file attributes, metadata, data > temperature, storage type, EC codec, available hardware capabilities, > user/application preference and etc. > Modified the title for re-purpose. > We'd extend this effort some bit and aim to work on a comprehensive solution > to provide smart storage management service in order for convenient, > intelligent and effective utilizing of erasure coding or replicas, HDFS cache > facility, HSM offering, and all kinds of tools (balancer, mover, disk > balancer and so on) in a large cluster. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-7343) HDFS smart storage management
[
https://issues.apache.org/jira/browse/HDFS-7343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15811374#comment-15811374
]
Wei Zhou commented on HDFS-7343:
Thanks [~anu] for reviewing the design document and the great comments.
{code}
1. I would like to understand the technical trade-offs that was considered in
making this choice.
{code}
Some methodologies existed in NN to collect metrics and events from DNs, so
it's better to store these data in NN to make SSM stateless as suggested by
Andrew. Also, it makes SSM more stable as these data stored in NN. When SSM
node failure happened, we can simply launch another instance on another node.
{code}
2. throttle the number of times a particular rule is executed in a time window?
{code}
Yes, good suggestion, I think we can make it a part of rule (for example,
provide a keyword). For now, it's better to provide a predictable SSM that a
rule getting executed when the condition fulfilled. If a throttle added in
rule-engine level then it's hard for users to predict the execution of the
rule. This brings in uncertainty to users. We can implement automatical
rule-engine level throttle in Phase 2.
{code}
3. Do we need to store the rules inside Namenode ?
{code}
Rule is the core part for SSM to function. For convenient and reliable
consideration, it's better to store it in NN to keep SSM simple and stateless
as suggested. Also the size of rule is very small (pure text) and suppose it
should never be a burden to NN.
{code}
4. HA support
{code}
Yes, good question. We can support HA by many ways, for example, periodically
checkpoint the data to HDFS or store the data in the same way as edit log.
{code}
5. but how do you intend to protect this end point?
{code}
Yes, if the cluster implements the Kerberos protocol, then web interface,
consoles and other parts of SSM are all works with Kerberos enabled.
{code}
6. How do we prevent a run-away rule?
{code}
This is a very good question.
First, we provide some verification mechanism when adding some rule. For
example, we can give the user some warning when the candidate files of an
action (such as move) exceeding some certain value.
Second, the execution state and other info related info can also be showed in
the dashboard or queried. It's convenient for users to track the status and
take actions accordingly. It's also very good to implement a timeout mechanism.
{code}
7. On the HDFS client querying SSM before writing, what happens if the SSM is
down?
{code}
Sorry for not making it clearly. Client queries SSM only once just before
creating the file, SSM does not need to participate in write procedure. So,
HDFS client will bypass SSM when the query fails, then the client goes back to
the original working flow. It has almost no effect on the existing I/O.
{code}
I would love to learn how this is working out in real world clusters.
{code}
We did some prototypes for POC. Three typical cases implemented with some
extent simplification:
# Move data to SSD based on the access count
# Cache data based on the access count
# Archive data based on file's age
The following chart shows the testing result of the first case. The rule is "if
a file been read for more than 2 times within 10 mins then move the file to
SSD". As we can see the time used for read decreases after the rule been
executed.
!move.jpg!
{code}
I think we have accidentally omitted reference to our classic balancer here.
{code}
Yes, thanks for your reminder.
> HDFS smart storage management
> -
>
> Key: HDFS-7343
> URL: https://issues.apache.org/jira/browse/HDFS-7343
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Kai Zheng
>Assignee: Wei Zhou
> Attachments: HDFS-Smart-Storage-Management-update.pdf,
> HDFS-Smart-Storage-Management.pdf, move.jpg
>
>
> As discussed in HDFS-7285, it would be better to have a comprehensive and
> flexible storage policy engine considering file attributes, metadata, data
> temperature, storage type, EC codec, available hardware capabilities,
> user/application preference and etc.
> Modified the title for re-purpose.
> We'd extend this effort some bit and aim to work on a comprehensive solution
> to provide smart storage management service in order for convenient,
> intelligent and effective utilizing of erasure coding or replicas, HDFS cache
> facility, HSM offering, and all kinds of tools (balancer, mover, disk
> balancer and so on) in a large cluster.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-7343) HDFS smart storage management
[
https://issues.apache.org/jira/browse/HDFS-7343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15805521#comment-15805521
]
Anu Engineer commented on HDFS-7343:
[~zhouwei] Thank you for addressing all the issues I had. The updated design
doc looks excellent. if there are any JIRAs that you need help with, please let
me know and I will be happy to chip in. I am almost at a +1. However, I had
some questions and comments. Please treat the following sections as Questions
(Things that I don’t understand and would like to know), comments (some
subjective comments, please feel free to ignore them), nitpick (completely
ignore them, written down to avoid someone else asking the same question later).
h4. Questions
{noformat}
NameNode then stores the data into database. In this way, SSM has no need to
maintain state checkpoints.
{noformat}
1. I would like to understand the technical trade-offs that was considered in
making this choice. Other applications that do this, like Amabri chooses to
store this data in a database maintained within the application. However, for
SSM you are choosing to store them in Namenode. In fact when I look at the
architecture diagram (it is very well drawn, thank you) It looks to me that it
is trivial to have the LevelDB on the SSM side instead of Namenode. So I am
wondering what advantage we have by maintaining more metadata on namenode side.
2. 1. In Rule Engine section, can we throttle the number of times a
particular rule is executed in a time window? What I am trying to prevent is
some kind of flapping where two opposite rules are triggered continuously.
3. Also there is a reference {noformat} SSM also stores some data (for example,
rules) into database through NameNode. {noformat} Do we need to store the rules
inside Namenode ? Would it make more sense to store it in SSM itself. The
reason why I am asking is that in future I see that this platform could be
leveraged by Hive or HBase. If that is that case, having an independent rule
store might be more interesting than a pure HDFS one.
4. {noformat} HA supporting will be considered later {noformat} I am all for
postponing HA support, but I am not able to understand how we are going to
store rules in namenode and ignore HA. Are we going to say SSM will not work if
HA is enabled? Most clusters we see are HA enabled. However, if we avoid
dependencies on Namenode, SSM might work with HA enabled cluster. I cannot see
anything in SSM that inherently prevents it from working with an HA enabled
cluster.
5. {noformat} SSM also exports interface (for example, through RESTful API) for
administrator and client to manage SSM or query information.{noformat} May be
something for later, but how do you intend to protect this end point? Is it
going to be Kerberos? Usually I would never ask a service to consider security
till you have to core modules, but in this case the potential for abuse is very
high. I understand that we might not have it in the initial releases of SSM,
but it might be good to think about it.
6. How do we prevent a run-away rule? Let me give you an example, recently I
was cleaning a cluster and decided to run “rm –rf “ on the data directories.
Each datanode had more than 12 million files and just a normal file system
operation was taking forever. So a rule like *file.path matches "/fooA/*.dat"*,
might run forever (I am looking at you , Hive). Are you planning to provide a
timeout on the execution of a rule or is it that rule will run until we reach
the end of processing? if we don’t have timeouts, it might be hard to honor
other rules which want to run at a specific time. Even with multiple threads,
you might not be able to make too much progress and since most of these rules
are going to be run against Namenode and you would have a limited bandwidth to
work with.
7. On the HDFS client querying SSM before writing, what happens if the SSM is
down? Will client wait and retry , potentially making I/O slower and eventually
bypassing SSM ? Have you considered using Storage Policy Satisfier,
HDFS-10285. Even if SSM is down or a client does not talk to SSM, we could rely
on SPS to move the data to the right location. Some of your storage manager
functionality can leverage what is being done in Storage Policy Satisfier. So
can you please clarify how you will handle clients that does not talk to SSM
and what happens to I/O when SSM is down.
h4. Comments
1. I still share some of the concerns voiced by [~andrew.wang]. It is going to
be challenging to create a set of static rules for changing conditions of the
cluster, especially when works loads are different. But sometimes we learn
surprising things by doing rather than talking. I would love to learn how this
is working out in real world clusters. If you have any data to share I would l
appreciate it.
h4. Nitpic
{noformat}
HSM, Cach
[jira] [Commented] (HDFS-7343) HDFS smart storage management
[ https://issues.apache.org/jira/browse/HDFS-7343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15610058#comment-15610058 ] Anu Engineer commented on HDFS-7343: [~drankye] Thanks for the confirmation on Kafka. It does address large part of my concern. I look forward to seeing an updated design doc that contains the final design and what we are solving in the first iteration and how. > HDFS smart storage management > - > > Key: HDFS-7343 > URL: https://issues.apache.org/jira/browse/HDFS-7343 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Kai Zheng >Assignee: Wei Zhou > Attachments: HDFS-Smart-Storage-Management.pdf > > > As discussed in HDFS-7285, it would be better to have a comprehensive and > flexible storage policy engine considering file attributes, metadata, data > temperature, storage type, EC codec, available hardware capabilities, > user/application preference and etc. > Modified the title for re-purpose. > We'd extend this effort some bit and aim to work on a comprehensive solution > to provide smart storage management service in order for convenient, > intelligent and effective utilizing of erasure coding or replicas, HDFS cache > facility, HSM offering, and all kinds of tools (balancer, mover, disk > balancer and so on) in a large cluster. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-7343) HDFS smart storage management
[
https://issues.apache.org/jira/browse/HDFS-7343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15610047#comment-15610047
]
Kai Zheng commented on HDFS-7343:
-
bq. I am still concerned with introduction of Kafka and injecting a cluster
wide dependency graph.
Hi Anu, looks like according to latest discussion with Andrew, the
{{KafkaService}} isn't a must or depended, so hopefully this solves your
concern.
> HDFS smart storage management
> -
>
> Key: HDFS-7343
> URL: https://issues.apache.org/jira/browse/HDFS-7343
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Kai Zheng
>Assignee: Wei Zhou
> Attachments: HDFS-Smart-Storage-Management.pdf
>
>
> As discussed in HDFS-7285, it would be better to have a comprehensive and
> flexible storage policy engine considering file attributes, metadata, data
> temperature, storage type, EC codec, available hardware capabilities,
> user/application preference and etc.
> Modified the title for re-purpose.
> We'd extend this effort some bit and aim to work on a comprehensive solution
> to provide smart storage management service in order for convenient,
> intelligent and effective utilizing of erasure coding or replicas, HDFS cache
> facility, HSM offering, and all kinds of tools (balancer, mover, disk
> balancer and so on) in a large cluster.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-7343) HDFS smart storage management
[ https://issues.apache.org/jira/browse/HDFS-7343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15610007#comment-15610007 ] Kai Zheng commented on HDFS-7343: - Hi [~andrew.wang], About SSD cases, I thought of an original input for the motivation from a large Hadoop deployment in China. The user evaluated how to deploy certain amounts of SSDs via HSM to see if any help to speed up some workloads. The overall pain mentioned was they don't want to maintain by their operators manually what data should be kept in SSDs and then when to move out as needed or better according to some condition change. I agree fixed SLOs are important but I'm not sure that's all the cases. In interactive queries, for example, data miners may try different queries adjusting the conditions, combinations or the like, against some same data sets. We would expect the later runnings should be faster though understand earlier runnings are slow. For repeatedly running queries like daily jobs, it may be natural to expect them to be faster given there are enough SSDs during that time. > HDFS smart storage management > - > > Key: HDFS-7343 > URL: https://issues.apache.org/jira/browse/HDFS-7343 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Kai Zheng >Assignee: Wei Zhou > Attachments: HDFS-Smart-Storage-Management.pdf > > > As discussed in HDFS-7285, it would be better to have a comprehensive and > flexible storage policy engine considering file attributes, metadata, data > temperature, storage type, EC codec, available hardware capabilities, > user/application preference and etc. > Modified the title for re-purpose. > We'd extend this effort some bit and aim to work on a comprehensive solution > to provide smart storage management service in order for convenient, > intelligent and effective utilizing of erasure coding or replicas, HDFS cache > facility, HSM offering, and all kinds of tools (balancer, mover, disk > balancer and so on) in a large cluster. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-7343) HDFS smart storage management
[ https://issues.apache.org/jira/browse/HDFS-7343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15609821#comment-15609821 ] Anu Engineer commented on HDFS-7343: +1 on [~andrew.wang]'s comment. I am still concerned with introduction of Kafka and injecting a cluster wide dependency graph. As I said earlier, I do think this is a good effort, but I concur with Andrew that we should focus our efforts, prove that it is useful and then make overarching changes. > HDFS smart storage management > - > > Key: HDFS-7343 > URL: https://issues.apache.org/jira/browse/HDFS-7343 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Kai Zheng >Assignee: Wei Zhou > Attachments: HDFS-Smart-Storage-Management.pdf > > > As discussed in HDFS-7285, it would be better to have a comprehensive and > flexible storage policy engine considering file attributes, metadata, data > temperature, storage type, EC codec, available hardware capabilities, > user/application preference and etc. > Modified the title for re-purpose. > We'd extend this effort some bit and aim to work on a comprehensive solution > to provide smart storage management service in order for convenient, > intelligent and effective utilizing of erasure coding or replicas, HDFS cache > facility, HSM offering, and all kinds of tools (balancer, mover, disk > balancer and so on) in a large cluster. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-7343) HDFS smart storage management
[ https://issues.apache.org/jira/browse/HDFS-7343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15609772#comment-15609772 ] Andrew Wang commented on HDFS-7343: --- I'm not opposed to a branch, but I'd like to see a design doc rev that clarifies what usecases are being targeted for the first iteration of this work, and the corresponding implementation plan. Like I said in an earlier comment, I think archival usecases are the most important for end users, and can be handled with a pretty simple system by looking at atimes / ctimes for paths. The SSD stuff I'm less convinced of, due to the difficulties of providing reliable application-level SLOs. I think the best solutions here need to leverage application-level information about working sets and priorities from YARN and YARN apps. This is much more accurate than trying to determine the working sets via HDFS or OS-level information. > HDFS smart storage management > - > > Key: HDFS-7343 > URL: https://issues.apache.org/jira/browse/HDFS-7343 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Kai Zheng >Assignee: Wei Zhou > Attachments: HDFS-Smart-Storage-Management.pdf > > > As discussed in HDFS-7285, it would be better to have a comprehensive and > flexible storage policy engine considering file attributes, metadata, data > temperature, storage type, EC codec, available hardware capabilities, > user/application preference and etc. > Modified the title for re-purpose. > We'd extend this effort some bit and aim to work on a comprehensive solution > to provide smart storage management service in order for convenient, > intelligent and effective utilizing of erasure coding or replicas, HDFS cache > facility, HSM offering, and all kinds of tools (balancer, mover, disk > balancer and so on) in a large cluster. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-7343) HDFS smart storage management
[ https://issues.apache.org/jira/browse/HDFS-7343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15604535#comment-15604535 ] Wei Zhou commented on HDFS-7343: Thanks [~andrew.wang] for the replies! I totally agree with your comments and will refine the design document. What about open a branch for this work? Some guys here are writing codes for determinate parts of SSM, it would be more convenient to work with a branch. Thanks again for your great suggestions! > HDFS smart storage management > - > > Key: HDFS-7343 > URL: https://issues.apache.org/jira/browse/HDFS-7343 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Kai Zheng >Assignee: Wei Zhou > Attachments: HDFS-Smart-Storage-Management.pdf > > > As discussed in HDFS-7285, it would be better to have a comprehensive and > flexible storage policy engine considering file attributes, metadata, data > temperature, storage type, EC codec, available hardware capabilities, > user/application preference and etc. > Modified the title for re-purpose. > We'd extend this effort some bit and aim to work on a comprehensive solution > to provide smart storage management service in order for convenient, > intelligent and effective utilizing of erasure coding or replicas, HDFS cache > facility, HSM offering, and all kinds of tools (balancer, mover, disk > balancer and so on) in a large cluster. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-7343) HDFS smart storage management
[ https://issues.apache.org/jira/browse/HDFS-7343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15603629#comment-15603629 ] Andrew Wang commented on HDFS-7343: --- Hi Wei Zhou, thanks for the replies, inline: bq. How about get the file range data by hook into DataXceiver or DFSInputStream? The issue here is that DFSInputStream is client-side, and clients might lie to game the system. It's also more challenging to ingest client metrics, since you can't poll a client. We'd probably have to build some DN functionality to aggregate client metrics. bq. Yes, you are right, so we won't make it a black box... I'm looking forward to the next revision of the design doc :) bq. The server may not have enough disk slots to accommodate more HDDs. Sure, but you could also buy more servers to get more disk slots or RAM capacity. I like to look at this at the cluster-level. bq. HDD performance...decays dramatically compared with SSD... The 1.36X throughput and 1/3 latency over HDD are measured under a proper load for HDD. The improvement can be much higher. Yea, it's tricky to setup the correct baseline for these comparisons. If the HDDs are overloaded, then SSD looks comparatively better, but no one who cares about latency would overload their HDDs. bq. SSM tries to take actions to optimize the performance of the workload because SSM can not know whether it's profitable or not in advance. If turned out to be no improvement, suppose the action taken won't hurts the performance as well or the overhead is acceptable. This will make it difficult for end users since they want reliable performance, particularly when it comes to SLOs. Even if a job goes at SSD-level performance 95% of the time, the 5% it doesn't will violate the SLO. This means the SLO still needs to be set to HDD-level performance, meaning that we spent extra money for SSDs for the cluster but weren't able to improve the guarantees to end users. If SSM guesses wrong, it might even do worse than HDD-level performance due to additional disk and network I/O from data movement. bq. In your opinion, is there anything else for SSM to do to improve write performance? No, not directly. If you're already looking at write performance, that's fine with me. bq. I don't have much else to add, though I'd encourage you to think about how far we can get with a stateless system (possibly by pushing more work into the NN and DN). As soon as a service is stateful, high-availability becomes harder. This relates to SLOs as well; if a service isn't highly-available, then users can't set SLOs based on the service being up. > HDFS smart storage management > - > > Key: HDFS-7343 > URL: https://issues.apache.org/jira/browse/HDFS-7343 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Kai Zheng >Assignee: Wei Zhou > Attachments: HDFS-Smart-Storage-Management.pdf > > > As discussed in HDFS-7285, it would be better to have a comprehensive and > flexible storage policy engine considering file attributes, metadata, data > temperature, storage type, EC codec, available hardware capabilities, > user/application preference and etc. > Modified the title for re-purpose. > We'd extend this effort some bit and aim to work on a comprehensive solution > to provide smart storage management service in order for convenient, > intelligent and effective utilizing of erasure coding or replicas, HDFS cache > facility, HSM offering, and all kinds of tools (balancer, mover, disk > balancer and so on) in a large cluster. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-7343) HDFS smart storage management
[
https://issues.apache.org/jira/browse/HDFS-7343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15602545#comment-15602545
]
Wei Zhou commented on HDFS-7343:
Hi, [~andrew.wang], thanks for the replies!
{quote}
Do you also plan to capturing strace or other information? What's the
performance impact?
{quote}
I did some experiments on the overheads of strace before, it slows down the
performance for over 2X, so it's may not applicable. How about get the file
range data by hook into DataXceiver or DFSInputStream?
{quote}
and if it's a blackbox, it's much harder for admins to use it effectively.
{quote}
Yes, you are right, so we won't make it a black box. The conditions to take an
action or not will be clearly defined. Reasons will be logged when a rule is
executed or skipped. SSM will provide full info for admin to insure that SSM is
still under control.
{quote}
Have we done any prototyping and simulation of the rules engine? There are
workload generators like SWIM which could be useful here.
{quote}
We haven't done prototyping or simulation yet, it's still in the early stage.
Thanks for your advice, I will investigate it.
{quote}
What is the comparative cost-per-byte of SSD vs. HDD? I'm pretty sure it's
greater than 1.36x, meaning we might be better off buying more HDD to get more
throughput. Alternatively, buying more RAM depending on the dataset size.
{quote}
Yes, it's pretty larger than 1.36x, but it may not applicable to using more
HDDs to get the same performance:
The server may not have enough disk slots to accommodate more HDDs. So we keep
the disk number constant (the maximum slots) in the study.
HDD performance (both throughput and latency) decays dramatically compared with
SSD as the load increases. The 1.36X throughput and 1/3 latency over HDD are
measured under a proper load for HDD. The improvement can be much higher.
Big RAM does not help too much for IO intensive workloads just like cases in
the study.
{quote}
This is an example of application-specific tuning for HSM, which is a best
case. If the SSM doesn't correctly recognize the workload pattern, we won't
achieve the full 1.36x improvement.
{quote}
As above, it's a very specific case with static storage policy setting.
{quote}
I'm also unable to find the "com.yahoo.ycsb.workloads.CareWorkload" mentioned,
do you have a reference?
{quote}
Really sorry for that, it's a mistake! It should be
"com.yahoo.ycsb.workloads.CoreWorkload".
{quote}
The NameNode already does this checking, so it seems better for the enforcement
of quotas to be done in one place for consistency.
{quote}
Yes, great suggestion! Thanks!
{quote}
Maybe a user changes some files from ALL_SSD to DISK ...
{quote}
That's a good question! SSM can't handle the case automatically and
transparently, maybe we have to expose an interface for user to notify SSM to
reserve certain amount quota.
{quote}
Sorry to be this direct, but is improving average cluster throughput useful to
end users?
{quote}
Apologize for the misleading caused to you! what I supposed to mean is that
both the efficiency of the whole cluster and workloads. SSM tries to take
actions to optimize the performance of the workload because SSM can not know
whether it's profitable or not in advance. If turned out to be no improvement,
suppose the action taken won't hurts the performance as well or the overhead is
acceptable.
{quote}
I'm sure there's work to be done in the I/O path too, particularly the write
path
{quote}
SSM allows clients to use fast storage actively for write IO as shown in use
case 4. Besides, we are also investigating to improve the write performance
outside SSM. In your opinion, is there anything else for SSM to do to improve
write performance?
{quote}
I'm also not convinced that our average I/O utilization is that high to begin
with.
{quote}
Agreed that average I/O utilization is that high to begin with. Sorry for the
confusing, I'd like to clarify that for utilization we emphasize on the
utilization of fast storage instead of average I/O utilization.
{quote}
Optimistic scheduling and better resource isolation might lead to big
improvements here.
{quote}
Interesting to know this info, I will think about it as well.
{quote}
Adding a metrics collection system for OS-level metrics which needs to be
operated, managed, and deployed.
{quote}
Yes, OS-level metrics will bring difficulties for SSM. To some extent HDFS
statistic info can be used instead, so OS-level metrics is discarded.
{quote}
Potentially a Kafka dependency
{quote}
Based on the discussion with you and [~anu], it's preferred to use the polling
model (poll data directly from NNs and DNs). Do you have any
suggestions/comments on this? I can start the implementation in this way.
{quote}
I recommend stripping down the system to focus on the most important usecases
to start and growing it from there. My preference is for archi
[jira] [Commented] (HDFS-7343) HDFS smart storage management
[
https://issues.apache.org/jira/browse/HDFS-7343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15601016#comment-15601016
]
Wei Zhou commented on HDFS-7343:
Thanks [~xiaochen] for reviewing and the great comments. I'm busy with some
other issues, sorry for the latency!
{quote}
May be too early to ask: in order to do HDFS management work, the SSM has to
run as hdfs superuser, right?
{quote}
SSM will follow the current settings on permissions and use the lowest possible
permission to do the work, that is, SSM uses different permissions for
different actions.
{quote}
And related to Andrew's question on performance-based decisions, is it manual
or automatic (or both)?
{quote}
Yes, it's true. The automatic part is very important to the rule system. It's
not convenient for user to set rules for all the cases, sometimes it may also
impossible. For files that not given rules we can use it as a default rule. For
files that been given rules, SSM respects these rules but may also add some
automatic rules for these files if needed. For examples, for a file with a
simple rule (every 1d at 0:00 | age lt 30d | cache) specified by user, SSM may
add another rule (every 1d at 6:00 | age lt 30d | cache) to the file
automatically based on the file's access history.
{quote}
If the query is not latency-sensitive, the caching-uncaching in the 'automatic'
way may be unnecessary. Is it possible to not have the automatic way happen for
some workloads? I can think of similar cases where converting between EC <->
replica may not be necessary.
{quote}
yes. These are conditions to be fulfilled before 'the automatic way'. For the
operation converting between EC <-> replica in your case, SSM will mainly
depends on rules specified by users instead of making aggressive assumptions.
Please also reference the response above to Andrew's question:
{quote}
Also on the rules solver, how do we quantify the cost of executing an action?
It's important to avoid unnecessarily migrating data back and forth.
{quote}
Thanks!
> HDFS smart storage management
> -
>
> Key: HDFS-7343
> URL: https://issues.apache.org/jira/browse/HDFS-7343
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Kai Zheng
>Assignee: Wei Zhou
> Attachments: HDFS-Smart-Storage-Management.pdf
>
>
> As discussed in HDFS-7285, it would be better to have a comprehensive and
> flexible storage policy engine considering file attributes, metadata, data
> temperature, storage type, EC codec, available hardware capabilities,
> user/application preference and etc.
> Modified the title for re-purpose.
> We'd extend this effort some bit and aim to work on a comprehensive solution
> to provide smart storage management service in order for convenient,
> intelligent and effective utilizing of erasure coding or replicas, HDFS cache
> facility, HSM offering, and all kinds of tools (balancer, mover, disk
> balancer and so on) in a large cluster.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-7343) HDFS smart storage management
[
https://issues.apache.org/jira/browse/HDFS-7343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15592678#comment-15592678
]
Andrew Wang commented on HDFS-7343:
---
Thanks for the replies Wei Zhou, inline:
{quote}
To solve this, we make IO statistics from both HDFS level and system level
(system wide, like data given by system tool ‘iostat’). IO caused by SCR can be
measured from system level data.
{quote}
Just iostat-level data won't tell you what file or file range is being read
though. Do you also plan to capturing strace or other information? What's the
performance impact?
{quote}
For SSM it have to consider the factors like file length, access history and
memory availability before making a decision to cache a file. It tries to
minimize the possibility to cache a file that not needed to.
{quote}
{quote}
For performance consideration, SSM makes the first action have higher priority
than the second one. It depends and not always the case.
{quote}
Unfortunately this doesn't really clarify things for me. The rules engine is
the most important part of this work, and if it's a blackbox, it's much harder
for admins to use it effectively. This is especially true in debugging
scenarios when the rules engine isn't doing what the admin wants.
Have we done any prototyping and simulation of the rules engine? There are
workload generators like SWIM which could be useful here.
{quote}
For example, Jingcheng Du and I did a study on HSM last year, we found that the
throughput of cluster with 4 SSDs + 4 HDDs on each DN is 1.36x larger than
cluster with 8 HDDs on each DN, it’s almost as good as cluster with 8 SSDs on
each DN.
{quote}
Thanks for the reference. Some questions about this study though:
* What is the comparative cost-per-byte of SSD vs. HDD? I'm pretty sure it's
greater than 1.36x, meaning we might be better off buying more HDD to get more
throughput. Alternatively, buying more RAM depending on the dataset size.
* This is an example of application-specific tuning for HSM, which is a best
case. If the SSM doesn't correctly recognize the workload pattern, we won't
achieve the full 1.36x improvement.
* I'm also unable to find the "com.yahoo.ycsb.workloads.CareWorkload"
mentioned, do you have a reference?
{quote}
For example, the amount of memory available has to be checked before caching a
file, if not enough memory available then the action will be canceled.
{quote}
The NameNode already does this checking, so it seems better for the enforcement
of quotas to be done in one place for consistency.
A related question, how does this interact with user-generated actions? Maybe a
user changes some files from ALL_SSD to DISK since they want to free up SSD
quota for an important job they're going to run later. The SSM then sees there
is available SSD and uses it. Then the user is out of SSD quota and their
important job runs slowly.
{quote}
SSM pays more attention on the efficiency of the whole cluster than a
particular workload, it may not improve the end-to-end execution time of one
workload but it may improve another workload in the cluster.
{quote}
Sorry to be this direct, but is improving average cluster throughput useful to
end users? Broadly speaking, for ad-hoc user-submitted jobs, you care about
end-to-end latency. For large batch jobs, they aren't that performance
sensitive, and their working sets are unlikely to fit in memory/SSD anyway. In
this case, we care very much about improving a particular workload.
I'll end with some overall comments:
If the goal is improving performance, would our time be better spent on the I/O
paths, HSM and caching? I mentioned sub-block caching and client-side metrics
as potential improvements for in-memory caching. Integrating it with the
storage policies API and YARN resource management would also be useful. I'm
sure there's work to be done in the I/O path too, particularly the write path
which hasn't seen as much love as reads. This means we'd get more upside from
fast storage like SSD.
I'm also not convinced that our average I/O utilization is that high to begin
with. Typical YARN CPU utilization is <50%, and that's with many jobs being CPU
bound. On the storage side, most clusters are capacity-bound. Optimistic
scheduling and better resource isolation might lead to big improvements here.
I'm also concerned about scope creep, particularly since the replies to my
comments indicate a system even bigger than the one described in the design
document. It involves:
* A policy engine that can understand a wide variety of OS, HDFS, and
application-level hints and performance metrics, as well as additional
constraints from user-provided rules, system quotas, and data movement costs.
* Adding a metrics collection system for OS-level metrics which needs to be
operated, managed, and deployed.
* The SSM itself, which is a stateful service, w
[jira] [Commented] (HDFS-7343) HDFS smart storage management
[
https://issues.apache.org/jira/browse/HDFS-7343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15592342#comment-15592342
]
Wei Zhou commented on HDFS-7343:
Continue on comments from [~andrew.wang],
{quote}
Could you talk a little bit more about the rules solver? What happens when a
rule cannot be satisfied?
{quote}
A rule is a declaration which defines actions to be implied on some objects
under certain condition. It’s a guideline for SSM to function. Rule solver
parses a rule and takes the action specified if its predefined condition
fulfilled. But this does not mean that the action will be executed physically.
It depends on many factors. For example, the amount of memory available has to
be checked before caching a file, if not enough memory available then the
action will be canceled.
>From above a rule is essentially a hint for SSM. When a rule cannot be
>satisfied SSM can write some logs with the reason to log
>file/console/dashboard. Admin can check those info for further processing.
{quote}
improve average throughput, but not improve end-to-end execution time (the SLO
metric).
{quote}
SSM pays more attention on the efficiency of the whole cluster than a
particular workload, it may not improve the end-to-end execution time of one
workload but it may improve another workload in the cluster. Another case is
that it won’t help for a CPU intensive workload though we do make some
optimization to IO. To make SSM work better, we could expose some interface for
workloads to provide hints to SSM.
{quote}
Also on the rules solver, how do we quantify the cost of executing an action?
It's important to avoid unnecessarily migrating data back and forth.
{quote}
It’s very hard to quantify the cost generally in a dynamic environment. Moving
hot data to faster storage may impact the performance now but may boost it
later. What we do now is trying to minimize the cost based on access history,
current status of the cluster, rules and other mechanisms like hints from user.
Restrict conditions have to be full filled (rules, cluster states, history,
hints etc.) before actually executing an action. Generally, the greater the
cost, the harder the conditions. For example, Actions like archive file and
balance the cluster may depends higher on the rules or user’s hint compared
with actions like cache a file. Yes, it's very important to avoid unnecessarily
migrating data back and forth and SSM tries to minimize it at the very
beginning.
{quote}
Could you talk some more about the value of Kafka in this architecture,
compared to a naive implementation that just polls the NN and DN for
information?
HDFS's inotify mechanism might also be interesting here.
{quote}
Please reference the reply to #3 question from [~anu] also. For SSM:
1. It’s a message collector for SSM. It provides a high efficiency and reliable
way for nodes to send messages out. If all the nodes send out messages to SSM
directly then it’s very hard for SSM to handling issues such as message
buffering, persist to avoid losing messages, unstable service time due to too
many nodes and etc. It decouples the SSM from the cluster and let it can focus
on the message processing logic.
2. It’s a message recorder for SSM. If SSM stopped by user or crashed while the
HDFS cluster is still working, then messages from nodes can be stored in Kafka.
These messages are good material for SSM to warm up quickly. Without Kafka then
these precious data will be lost. It makes SSM more robust.
{quote}
Also wondering if with Kafka we still need a periodic snapshot of state, since
Kafka is just a log.
{quote}
SSM snapshots the data digested from those raw logs and other info managed, but
raw logs themselves are not stored. The data to be snapshotted is essential for
SSM to function better.
{quote}
The doc talks a lot about improving performance, but I think the more important
usecase is actually saving cost by migrating data to archival or EC storage.
This is because of the above difficulties surrounding actually understanding
application-level performance with just FS-level information.
{quote}
Agreed that it’s an important use case and impossible for SSM itself to improve
the performance for all cases as you mentioned. But it’s a trend that DNs will
have larger memory and faster storage. How to make use of these hardwares to
improve the performance is also an important issue to solve. For example,
[~jingcheng...@intel.com] and I did a [study on HSM |
http://blog.cloudera.com/blog/2016/06/new-study-evaluating-apache-hbase-performance-on-modern-storage-media/]
last year, we found that the throughput of cluster with 4 SSDs + 4 HDDs on
each DN is 1.36x larger than cluster with 8 HDDs on each DN, it’s almost as
good as cluster with 8 SSDs on each DN. It’s also the same case for latency. So
it should improve the performance by using the fast storage efficie
[jira] [Commented] (HDFS-7343) HDFS smart storage management
[
https://issues.apache.org/jira/browse/HDFS-7343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15590949#comment-15590949
]
Wei Zhou commented on HDFS-7343:
Hi [~andrew.wang], thank you for reviewing the document and great comments.
Sorry for the delay!
{quote}
First was that most reads happen via SCR, thus we do not have reliable IO
statistics.
{quote}
IO statistics is indeed a point. To solve this, we make IO statistics from both
HDFS level and system level (system wide, like data given by system tool
‘iostat’). IO caused by SCR can be measured from system level data.
{quote}
Do you plan to address these issues as part of this work?
{quote}
Agreed that it’s a waste of precious resources.
For SSM it have to consider the factors like file length, access history and
memory availability before making a decision to cache a file. It tries to
minimize the possibility to cache a file that not needed to.
It’s a great ideal to extend HDFS cache to cache partial block instead of the
whole file. This should help for many scenarios.
SSM can support the extended feature by collecting block read offset info to
justify the part to be cached.
{quote}
It's difficult to prioritize at the HDFS-level since performance is measured at
the app-level.
{quote}
Agreed that it's difficult to prioritize at the HDFS-level. Prioritize is not a
general and static concept in SSM. It’s used to for schedule some specific
cases. For example, one rule triggers an action of moving hot file into faster
storage while another rule triggers an ‘archive cold file’ action at the same
time. For performance consideration, SSM makes the first action have higher
priority than the second one. It depends and not always the case.
{quote}
If you're looking at purely HDFS-level information, without awareness of users,
jobs, and their corresponding priorities, admins will have a hard time mapping
rules to their actual SLOs.
{quote}
Yes, it’s not possible to handle all issues just from purely HDFS-level
information. It’s a great plus for SSM to collect high-level application info
for better efficiency. Also, we could expose some APIs for admin/user to
provide some hint to SSM to make it work better. Take your case for example,
user could hint the operation is for a time sensitive job then SSM can cache it
in memory or move the file to faster storage.
Your following questions will be replied in comments followed. Thanks!
> HDFS smart storage management
> -
>
> Key: HDFS-7343
> URL: https://issues.apache.org/jira/browse/HDFS-7343
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Kai Zheng
>Assignee: Wei Zhou
> Attachments: HDFS-Smart-Storage-Management.pdf
>
>
> As discussed in HDFS-7285, it would be better to have a comprehensive and
> flexible storage policy engine considering file attributes, metadata, data
> temperature, storage type, EC codec, available hardware capabilities,
> user/application preference and etc.
> Modified the title for re-purpose.
> We'd extend this effort some bit and aim to work on a comprehensive solution
> to provide smart storage management service in order for convenient,
> intelligent and effective utilizing of erasure coding or replicas, HDFS cache
> facility, HSM offering, and all kinds of tools (balancer, mover, disk
> balancer and so on) in a large cluster.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-7343) HDFS smart storage management
[
https://issues.apache.org/jira/browse/HDFS-7343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15587704#comment-15587704
]
Wei Zhou commented on HDFS-7343:
Thanks [~anu] for reviewing the design document and great comments!
For your comments:
{quote}
1. Is this service attempting to become the answer for all administrative
problems of HDFS? In other words, Is this service is trying to be a catch all
service?
I am not able to see what is common between caching and file placement and
between running distcp for remote replication and running balancer and disk
balancer.
{quote}
For long run, SSM is going to provide user an end-to-end storage management
automation solution. Any facility can be used in this project towards the
solution. The use cases and examples listed in the document just give examples
of the possible scenarios where SSM can be used and what can SSM do. SSM can
help from different angles by using these facilities.
{quote}
But then in the latter parts of the document we drag in distcp, disk balancer
and balancer issues. SSM might be the right place for these in the long run,
but my suggestion is to focus on the core parts of the service and then extend
it to other things once the core is stabilized.
{quote}
You are absolutely right, we have to be focused on implementing the core
part/module now instead of involving too much beyond at the same time, it's the
basis of other functions.
{quote}
2. Do we need a new rules language – Would you please consider using a language
which admins will already know, for example, if we can write these rules in
python or even JavaScript, you don’t need to invent a whole new language. Every
time I have to configure Kerberos rules, I have to lookup the mini-regex
meanings. I am worried that this little rule language will blow up and once it
is in, this is something that we will need to support for the long term.
{quote}
Yes, it's a very good question and we do have thought about it before. We aimed
at providing administrator/user a simple and specific rule language without
touching too much besides the rule logic itself. In fact, a rule is very simple
that only have to declare when and which action to be implied on some objects
(can be a file, node, etc.). A general and systematic language like python or
java script maybe too heavy for defining a rule.
{quote}
3. In this model we are proposing a push model where the datanode and Namenode
pushes data to some kafka endpoint. I would prefer if namenode and datanode was
not aware of this service at all. This service can easily connect to namenode
and read almost all the data which is needed. If you need extra RPC to be added
in datanode and namenode that would an option too. But creating a dependency
from namenode and all datanodes in a cluster seems to be something that you
want to do after very careful consideration. If we move to a pull model, you
might not even need kafka service to be running in the initial versions.
{quote}
Good point! This is also a very good way to implement SSM.
If using pull model, the advantage are:
(1) No dependency on Kafka service, and it’s indeed much easier for
development, testing and deployment.
(2) Closer relationship with HDFS which may be able to support features that
cannot be done in the model described in the design document.
The disadvantage are:
(1) It may have potential performance issue. SSM have to know the messages
timely in order to work effectively. In order to decrease the overhead of
getting messages, SSM have to query NameNodes for the messages on a very high
frequency all the time. It’s also very hard for SSM to query DataNodes one by
one to get messages in a large scale cluster.
(2) It simplifies the process of message collecting and management. If SSM
stopped by user or crashed while the HDFS cluster is still working, then
messages from nodes shall be lost without Kafka, and it’s not friendly for SSM
to collect historical data.
Above, both of the models are workable and we may need more discussion on it.
What’s your opinion?
> HDFS smart storage management
> -
>
> Key: HDFS-7343
> URL: https://issues.apache.org/jira/browse/HDFS-7343
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Kai Zheng
>Assignee: Wei Zhou
> Attachments: HDFS-Smart-Storage-Management.pdf
>
>
> As discussed in HDFS-7285, it would be better to have a comprehensive and
> flexible storage policy engine considering file attributes, metadata, data
> temperature, storage type, EC codec, available hardware capabilities,
> user/application preference and etc.
> Modified the title for re-purpose.
> We'd extend this effort some bit and aim to work on a comprehensive solution
> to provide smart storage management service in order for convenie
[jira] [Commented] (HDFS-7343) HDFS smart storage management
[ https://issues.apache.org/jira/browse/HDFS-7343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15585033#comment-15585033 ] Wei Zhou commented on HDFS-7343: Thanks all for reviewing and great comments. I'm thinking about this and will reply in detail later. Thanks! > HDFS smart storage management > - > > Key: HDFS-7343 > URL: https://issues.apache.org/jira/browse/HDFS-7343 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Kai Zheng >Assignee: Wei Zhou > Attachments: HDFS-Smart-Storage-Management.pdf > > > As discussed in HDFS-7285, it would be better to have a comprehensive and > flexible storage policy engine considering file attributes, metadata, data > temperature, storage type, EC codec, available hardware capabilities, > user/application preference and etc. > Modified the title for re-purpose. > We'd extend this effort some bit and aim to work on a comprehensive solution > to provide smart storage management service in order for convenient, > intelligent and effective utilizing of erasure coding or replicas, HDFS cache > facility, HSM offering, and all kinds of tools (balancer, mover, disk > balancer and so on) in a large cluster. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-7343) HDFS smart storage management
[
https://issues.apache.org/jira/browse/HDFS-7343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15583292#comment-15583292
]
Xiao Chen commented on HDFS-7343:
-
Thanks all for the great documentation and discussions. It will be an
interesting undertaking. :)
May be too early to ask: in order to do HDFS management work, the SSM has to
run as hdfs superuser, right?
And related to Andrew's question on performance-based decisions, is it manual
or automatic (or both)?
The doc says {{SSM can make prediction on a file’s read based on read
historical information and cache the file automatically before the read
operation happens}}, and later gives an example of a similar rule ({{every 1d
at 0:00 | age lt 30d | cache}}). I think that means both: the description
indicating the automatic part, and the rule showing a same example for a manual
control. Is it true?
If the query is not latency-sensitive, the caching-uncaching in the 'automatic'
way may be unnecessary. Is it possible to not have the automatic way happen for
some workloads? I can think of similar cases where converting between EC <->
replica may not be necessary.
> HDFS smart storage management
> -
>
> Key: HDFS-7343
> URL: https://issues.apache.org/jira/browse/HDFS-7343
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Kai Zheng
>Assignee: Wei Zhou
> Attachments: HDFS-Smart-Storage-Management.pdf
>
>
> As discussed in HDFS-7285, it would be better to have a comprehensive and
> flexible storage policy engine considering file attributes, metadata, data
> temperature, storage type, EC codec, available hardware capabilities,
> user/application preference and etc.
> Modified the title for re-purpose.
> We'd extend this effort some bit and aim to work on a comprehensive solution
> to provide smart storage management service in order for convenient,
> intelligent and effective utilizing of erasure coding or replicas, HDFS cache
> facility, HSM offering, and all kinds of tools (balancer, mover, disk
> balancer and so on) in a large cluster.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-7343) HDFS smart storage management
[ https://issues.apache.org/jira/browse/HDFS-7343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15576893#comment-15576893 ] Andrew Wang commented on HDFS-7343: --- Hi everyone, thanks for the great document. This is a very ambitious project, and would be a great addition to HDFS. Some comments: * We'd thought about doing automatic cache management as a follow-on for HDFS-4949, but there were a couple issues. First was that most reads happen via SCR, thus we do not have reliable IO statistics. Second, performance-sensitive apps are often reading structured columnar data like Parquet, and thus only ranges of a file. Whole-file or whole-block caching is thus very coarse, and wastes precious memory or SSD. Do you plan to address these issues as part of this work? * It's difficult to prioritize at the HDFS-level since performance is measured at the app-level. Typically clusters have two broad categories of jobs running: time-sensitive queries issued by users, and low-priority batch work. These two categories can also be accessing the same data, though with different access patterns. If you're looking at purely HDFS-level information, without awareness of users, jobs, and their corresponding priorities, admins will have a hard time mapping rules to their actual SLOs. * Could you talk a little bit more about the rules solver? What happens when a rule cannot be satisfied? This also ties back into app-level performance, since there are "all-or-nothing" properties where caching half a dataset might improve average throughput, but not improve end-to-end execution time (the SLO metric). * Also on the rules solver, how do we quantify the cost of executing an action? It's important to avoid unnecessarily migrating data back and forth. * Could you talk some more about the value of Kafka in this architecture, compared to a naive implementation that just polls the NN and DN for information? Also wondering if with Kafka we still need a periodic snapshot of state, since Kafka is just a log. * HDFS's inotify mechanism might also be interesting here. The doc talks a lot about improving performance, but I think the more important usecase is actually saving cost by migrating data to archival or EC storage. This is because of the above difficulties surrounding actually understanding application-level performance with just FS-level information. FWIW, we've had reasonable success with time-based policies for aging out data to colder storage with HDFS-4949. This is because many workloads have an access distribution that heavily skews toward newer data. So, some simple rules with time-based triggers or looking at file atimes might get us 80% of what users want. > HDFS smart storage management > - > > Key: HDFS-7343 > URL: https://issues.apache.org/jira/browse/HDFS-7343 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Kai Zheng >Assignee: Wei Zhou > Attachments: HDFS-Smart-Storage-Management.pdf > > > As discussed in HDFS-7285, it would be better to have a comprehensive and > flexible storage policy engine considering file attributes, metadata, data > temperature, storage type, EC codec, available hardware capabilities, > user/application preference and etc. > Modified the title for re-purpose. > We'd extend this effort some bit and aim to work on a comprehensive solution > to provide smart storage management service in order for convenient, > intelligent and effective utilizing of erasure coding or replicas, HDFS cache > facility, HSM offering, and all kinds of tools (balancer, mover, disk > balancer and so on) in a large cluster. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-7343) HDFS smart storage management
[ https://issues.apache.org/jira/browse/HDFS-7343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15576163#comment-15576163 ] Anu Engineer commented on HDFS-7343: [~zhouwei] Thanks for posting a detailed design document. I was not part of the earlier discussions so I may not have the right context for many of my comments. Please do feel free to provide me the right context if I am missing any. I think this proposal addresses the one of the biggest issue in running an HDFS cluster. We rely on lots of operator interactions to keep the cluster running while these should be automated. I completely agree with the sentiment. Couple of minor comments: 1. Is this service attempting to become the answer for all administrative problems of HDFS? In other words, Is this service is trying to be a catch all service? I am not able to see what is common between caching and file placement and between running distcp for remote replication and running balancer and disk balancer. My personal view is that this service should pick the right focus and focus on those issues. For example, the first 5 uses cases are all related to performance and file placement. But then in the latter parts of the document we drag in distcp, disk balancer and balancer issues. SSM might be the right place for these in the long run, but my suggestion is to focus on the core parts of the service and then extend it to other things once the core is stabilized. 2. Do we need a new rules language – Would you please consider using a language which admins will already know, for example, if we can write these rules in python or even JavaScript, you don’t need to invent a whole new language. Every time I have to configure Kerberos rules, I have to look the mini-regex meanings. I am worried that this little rule language will blow up and once it is in, this is something that we will need to support for the long term. 3. In this model we are proposing a push model where the datanode and Namenode pushes data to some kafka endpoint. I would prefer if namenode and datanode was not aware of this service at all. This service can easily connect to namenode and read almost all the data which is needed. If you need extra RPC to be added in datanode and namenode that would an option too. But creating a dependency from namenode and all datanodes in a cluster seems to be something that you want to do after very careful consideration. If we move to a pull model, you might not even need kafka service to be running in the initial versions. > HDFS smart storage management > - > > Key: HDFS-7343 > URL: https://issues.apache.org/jira/browse/HDFS-7343 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Kai Zheng >Assignee: Wei Zhou > Attachments: HDFS-Smart-Storage-Management.pdf > > > As discussed in HDFS-7285, it would be better to have a comprehensive and > flexible storage policy engine considering file attributes, metadata, data > temperature, storage type, EC codec, available hardware capabilities, > user/application preference and etc. > Modified the title for re-purpose. > We'd extend this effort some bit and aim to work on a comprehensive solution > to provide smart storage management service in order for convenient, > intelligent and effective utilizing of erasure coding or replicas, HDFS cache > facility, HSM offering, and all kinds of tools (balancer, mover, disk > balancer and so on) in a large cluster. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
