[ https://issues.apache.org/jira/browse/HDFS-7240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15285948#comment-15285948 ]
Anu Engineer commented on HDFS-7240: ------------------------------------ [~andrew.wang] Thank you for your comments, They are well thought out and extremely valuable questions. I will make sure that all areas that you are asking about is discussed in the next update of design doc. bq. Anu said he'd be posting a new design doc soon to address my questions. I am working on that, but just to make sure your questions are not lost in the big picture of the design doc, I am answering them individually here. bq. My biggest concern is that erasure coding is not a first-class consideration in this system. Nothing in ozone prevents a chunk being EC encoded. In fact ozone makes no assumptions about the location or the types of chunks at all. So it is quite trivial to create a new chunk type and write them into containers. We are focused on overall picture of ozone right now, and I would welcome any contribution you can make on EC and ozone chunks if that is a concern that you would like us to address earlier. From the architecture point of view I do not see any issues. bq. Since LevelDB is being used for metadata storage and separately being replicated via Raft, are there concerns about metadata write amplification? Metadata is such a small slice of information of a block – really what you are saying is that block Name, hash for the block gets written twice, once thru RAFT log and second time when RAFT commits this information. Since the data we are talking about is so small I am not worried about it at all. bq. Can we re-use the QJM code instead of writing a new replicated log implementation? QJM is battle-tested, and bq.sensus is a known hard problem to get right. We considered this, however the consensus is to write a *consensus protocol* that is easier to understand and make it easy for more contributors to work on it. The fact that QJM was not written as a library makes it very hard for us to pull it out in a clean fashion. Again if you feel very strongly about it, please feel free to move QJM to a library which can be reused and all of us will benefit from it. bq. Are there concerns about storing millions of chunk files per disk? Writing each chunk as a separate file requires more metadata ops and fsyncs than appending to a file. We also need to be very careful to never require a full scan of the filesystem. The HDFS DN does full scans right now (DU, volume scanner). Nothing in the chunk architecture assumes that chunk files are separate files. The fact that a chunk is a triplet \{FileName, Offset, Length\} gives you the flexibility to store 1000s of chunks in a physical file. bq. Any thoughts about how we go about packing multiple chunks into a larger file? Yes, write the first chunk and then write the second chunk to the same file. In fact, chunks are specifically designed to address the small file problem. So two keys can point to a same file. For example KeyA -> \{File,0, 100\} KeyB -> \{File,101, 1000\} Is a perfectly valid layout under container architecture bq. Merges and splits of containers. We need nice large 5GB containers to hit the SCM scalability targets. Together, these factors mean Ozone will likely be doing many more merges and splits than HBase to keep the container size high Ozone actively tries to avoid merges and tries to split only when needed. A container can be thought of as a really large block, so I am not sure if I am going to see anything other than standard block workload on containers. The fact that containers can be split, is something that allows us to avoid pre-allocation of container space. That is merely a convenience and if you think of these as blocks, you will see that it is very similar. Ozone will never try to do merges and splits at HBase level. From the container and ozone perspective we are more focused on a good data distribution on the cluster – aka what the balancer does today, and containers are a flat namespace – just like blocks which we allocate when needed. So once more – just make sure we are on the same page – Merges are rare(not required generally) and splits happen if we want to re-distribute data on a same machine. bq. What kind of sharing do we get with HDFS, considering that HDFS doesn't use block containers, and the metadata services are separate from the NN? not shared? Great question. We initially started off by attacking the scalability question of ozone and soon realized that HDFS scalability and ozone scalability has to solve the same problems. So the container infrastructure that we have built is something that can be used by both ozone and HDFS. Currently we are focused on ozone and containers will co-exist on datanodes with blockpools. That is ozone should be and will be deployable on a vanilla HDFS cluster. In future, if we want to scale HDFS, containers might be an easy way to do it. bq. Any thoughts on how we will transition applications like Hive and HBase to Ozone? These apps use rename and directories for synchronization, which are not possible on Ozone. These applications are written with the assumption of a Posix file system, so migrating them to Ozone does not make much sense. The work we are doing in ozone, especially container layer is useful in HDFS and these applications might benefit indirectly. bq. Have you experienced data loss from independent node failures, thus motivating the need for copysets? Yes, we have seen this issue in real world. Since we had to pick an algorithm for allocation and copysets offered a set of advantages with very minimal work. Hence the allocation choice. If you look at the paper you will see this was done on HDFS itself with very minimal cost, and also this work originally came from facebook. So it is useful in both normal HDFS and in RAMCloud. bq. It's also not clear how this type of data placement meshes with EC, or the other quite sophisticated types of block placement we currently support in HDFS The chunks will support remote blocks. That is a notion that we did not discuss due to time constraints in the presentation. In the presentation we just showed chunks as a files on the same machine, but a chunk could exist on a EC block and key could point to it. bq. How do you plan to handle files larger than 5GB? Large files will live on a set of containers, again the easiest model to reason about containers is to think of them as blocks. In other words, just like normal HDFS. bq. Large files right now are also not spread across multiple nodes and disks, limiting IO performance. Are you saying that when you write a file of say 256 MB size you would be constrained to same machine since the block size is much larger? That is why we have chunks. With chunks we can easily distribute this information and leave pointers to those chunks. I agree with the concern and we would might have to tune the placement algorithms once we see more real world workloads. bq. Are all reads and writes served by the container's Raft master? IIUC that's how you get strong consistency, but it means we don't have the same performance benefits we have now in HDFS from 3-node replication. Yes, and I don't think it will be a bottleneck. Since we are only reading and writing *metadata* – which is very small – and data can be read from any machine. bq. I also ask that more of this information and decision making be shared on public mailing lists and JIRA. Absolutely, we have been working actively and posting code patches to ozone branch. Feel free to ask us anything. It is kind of sad that we have to wait for an apachecon for someone to ask me these questions. I would encourage you to follow the JIRAs tagged with ozone to keep up with what is happening in the ozone world. I can assure you all the work we are doing is always tagged with _ozone:_` so that when you get the mail from JIRA it is immediately visible. bq. The KSM is not mentioned in the architecture document. The namespace management component did not have separate name, it was called SCM in the original arch. doc. Thanks to [~arpitagarwal] – he came up with this name (just like ozone) and argued that we should make it separate, which will make it easy to understand and maintain. If you look at code you will see KSM functionality is currently in SCM. I am planning to move it out to KSM. bq. the fact that the Ozone metadata is being replicated via Raft rather than stored in containers. Metadata is stored in containers, but containers need replication. Replication via RAFT was something that design evolved to, and we are surfacing that to the community. Like all proposals under consideration, we will update the design doc and listen to community feedback. You just had a chance to listen to our design (little earlier) in person since we got a chance meet. Writing documents take far more energy and time than chatting face to face. We will soon update the design docs. bq. I not aware that there is already progress internally at Hortonworks on a Raft implementation. We have been prototyping various parts of the system. Including KSM, SCM and other parts of Ozone. bq. We've previously expressed interest in being involved in the design and implementation of Ozone, but we can't meaningfully contribute if this work is being done privately. I feel that this is a completely uncalled for statement / misplaced sentiment here. We have 54 JIRAs on ozone so far. You are always welcome to ask questions or comment. I have personally posted a 43 page document explaining how ozone would look like to users, how they can manage it and REST interfaces offered by Ozone. Unfortunately I have not seen much engagement. While I am sorry that you feel left out, I do not see how you can attribute it to engineers who work on ozone. We have been more than liberal about the sharing internals including this specific presentation we are discussing here. So I am completely lost when you say that this work is being done privately. In fact, I would request you to be active on ozone JIRAs, start by asking questions, make comments (just like you did with this one except for this last comment) and I would love working with you on ozone. It is never our intent to abandon our fellow contributors in this journey, but unless you participate in JIRAs it is very difficult for us to know that you have more than a fleeting interest. > Object store in HDFS > -------------------- > > Key: HDFS-7240 > URL: https://issues.apache.org/jira/browse/HDFS-7240 > Project: Hadoop HDFS > Issue Type: New Feature > Reporter: Jitendra Nath Pandey > Assignee: Jitendra Nath Pandey > Attachments: Ozone-architecture-v1.pdf, ozone_user_v0.pdf > > > This jira proposes to add object store capabilities into HDFS. > As part of the federation work (HDFS-1052) we separated block storage as a > generic storage layer. Using the Block Pool abstraction, new kinds of > namespaces can be built on top of the storage layer i.e. datanodes. > In this jira I will explore building an object store using the datanode > storage, but independent of namespace metadata. > I will soon update with a detailed design document. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org