[ https://issues.apache.org/jira/browse/HDFS-7343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Wei Zhou updated HDFS-7343: --------------------------- Attachment: HDFS-Smart-Storage-Management-update.pdf Based on the discussion and feedbacks collected, we updated the design document. There are many changes compared with the previous version: # The ultimate target is separated into two phases, and in phase 1, we focus on implement a rule-based automation engine that integrates the facilities in HDFS. We will make it an end-to-end intelligent solution in phase 2. # Kafka service dependency removed, SSM gets info directly from NN. # DNs report metrics and events to NN instead of been polled by SSM directly. # Metrics, events and some other info are maintained in NN off-heap memory and stored in NN. # SSM is stateless and polling info from NN when starts up. [~andrew.wang], [~anu], [~xiaochen], [~eddyxu] and anybody else, please help review it. Your suggestion or comment is appreciated. Thanks! > HDFS smart storage management > ----------------------------- > > Key: HDFS-7343 > URL: https://issues.apache.org/jira/browse/HDFS-7343 > Project: Hadoop HDFS > Issue Type: Improvement > Reporter: Kai Zheng > Assignee: Wei Zhou > Attachments: HDFS-Smart-Storage-Management-update.pdf, > HDFS-Smart-Storage-Management.pdf > > > As discussed in HDFS-7285, it would be better to have a comprehensive and > flexible storage policy engine considering file attributes, metadata, data > temperature, storage type, EC codec, available hardware capabilities, > user/application preference and etc. > Modified the title for re-purpose. > We'd extend this effort some bit and aim to work on a comprehensive solution > to provide smart storage management service in order for convenient, > intelligent and effective utilizing of erasure coding or replicas, HDFS cache > facility, HSM offering, and all kinds of tools (balancer, mover, disk > balancer and so on) in a large cluster. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org