[ https://issues.apache.org/jira/browse/HDFS-7240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15317035#comment-15317035 ]
Anu Engineer commented on HDFS-7240: ------------------------------------ Hi [~eddyxu] , Thank you for reviewing the design doc and comments. Please see my comments below. bq. Since Ozone is decided to use range partition, how would key / data distribution achieve balancing from initial state? For example, a user Foo runs Hive and creates 10GB of data, these data are distributed to up to 6 (containers) DNs? You bring up a very valid point. This was the most contentious issue in ozone world for a while. We originally went with hash partition schemes and secondary index because of these concerns. The issue (and very rightly so) with that approach was that secondary index is eventually consistent and makes it hard to use. So we switched over to this scheme. So our current thought is this, each of the containers will report -- size, number of operations and number of keys to SCM. This will allow SCM to balance the allocation of the key space. So if you have a large number of reads and writes, which are completely independent, then they will fill up the cluster/container space evenly. But we have an opposing requirement here, generally there is a locality of access in the namespace. So for most cases if you are reading and writing to a bucket, then it is most efficient to keep that data together. Now let us look at this specific case, if you have containers configured to say 2GB, then 10GB of data will map to 5 containers. So the model works out to 5 containers. These containers will be spread across a set of machines due to the SCM’s location choosing algorithms. bq. Would you explain what is the benefit of recovering failure pipeline by using a parallel writes to all 3 containers? It is not very clear in the design. The point I was trying to make is that pipeline relies on Quorum as defined by RSM. So if we decide to use this pipeline with RAFT, then I was just trying to make a point that pipeline can be broken, and we will not attempt to heal it. Please let me know if this makes sense. bq. How does ozone differentiate a recover write from a malicious (or buggy) re-write? Thanks for flagging this, right now we do not. We can always prevent it in the container layer. It is small extension to make, we can write to a temporary file and replace the original if and only if the hashes match. I will file a work item to fix this. bq. You mentioned that KMS/SCM separation is for future scalability. Do KMS / SCM maintains 1:1, 1:n or n:m relationship? Though it is not in this phase. I'd like to know whether it is considered. Btw, they are also Raft replicated? KSM:SCM has a n:m relationship. Even though in easiest deployment configuration it is 1:1. So yes it is defined that way. They are always Raft replicated. bq. The raft ring / leader is per-container? Yes, and No. Let me explain this a little more. If you think only in terms of RAFT protocol, then we have a RAFT leader is per machine set. That is, we are going to have a leader for 3 machines (assuming a 3 machine RAFT ring). Now let us switch over to a developer’s point of view. Someone like me who is writing code against containers thinks strictly in terms of containers. So from an ozone developers point of view, we have a Raft leader for a container. In other words, containers provide an abstraction that makes you think that RAFT protocol is for the container, whereas in reality it is a shared ring that is used by many containers that share those 3 machines. This might be something that we want to explore in greater depth during the call. bq. For pipeline, say if we have a pipeline A->B->C, if the data writes successfully on A->B, and the metadata Raft writes are succeed on B,C, IIUC, that is a What would be the result for a read request sent to A or C? I am going to walk thru this with little more details, so that we are all on the same page. What you are describing is a situation where the RAFT leader is either B or C (Since RAFT is an active leader protocol) and for the sake of this illustration let us assume that we are talking about 2 scenarios. One where data is written to leader and another datanode and scenario two, where data is written to followers but not to the leader. Let us look at both in greater detail. Case 1: Data is written to machines B (leader) and Machine A. But when RAFT commit happens, machine A is off-line and RAFT data is written to Machine B and Machine C. So we have situation where B is the only machine with metadata as well as data. We deal with this issue in two ways, one when the commit callback happens in C, C will check if it has the data block and since it does not, it will attempt to copy that block from either B or A. Also when A's RAFT ring comes back up it will catch up with the RAFT log and the data is already available on Machine A. So in both cases, we are replicating data/metadata as soon as we can. Now let us look at the case where a client goes to C and says I want this data block, before the copy is done -- the client will feel that read is a bit slow, since Machine C will copy the data from Machine B or A, write to its local storage and then return the data. Case 2: Data is written to 2 followers and leader does not have the data block. The work flow is identical; leader will copy block from another machine before returning the block. Case 3: I also want to illustrate an extreme case, let us say a client did NOT write any data blocks, and attempted to write a key, a key will get committed, but container will not be able to find the data blocks at all. Since no data blocks were written by the client, the copy attempt will fail, and RAFT leader will learn that this is Block with No replicas. This would be similar in nature to HDFS. bq. How to handle split (merge, migrate) container during writes? I have made an eloquent argument about why we don't need to do merge in the first release of Ozone. When split is happening, the easiest way to deal with it is to pause the writes. if you don't mind, could you please take a look at http://schd.ws/hosted_files/apachebigdata2016/fc/Hadoop%20Object%20Store%20-%20Ozone.pdf - slides 38-42. I avoided repeating that in the design doc, since it was already quite large. We can go over this in detail if you like during the call. bq. Since container size is determined by the space usage instead of # of keys, would that result large performance variants on listing operation. You are absolutely right; it can have variation in performance. The alternative we have is to use hash partition with secondary indices. if you like we can revisit hash/range partition in the conf, call. Last time, we decided to have range as the primary method, but reserved the option of bringing hash partition back at a later stage. > Object store in HDFS > -------------------- > > Key: HDFS-7240 > URL: https://issues.apache.org/jira/browse/HDFS-7240 > Project: Hadoop HDFS > Issue Type: New Feature > Reporter: Jitendra Nath Pandey > Assignee: Jitendra Nath Pandey > Attachments: Ozone-architecture-v1.pdf, Ozonedesignupdate.pdf, > ozone_user_v0.pdf > > > This jira proposes to add object store capabilities into HDFS. > As part of the federation work (HDFS-1052) we separated block storage as a > generic storage layer. Using the Block Pool abstraction, new kinds of > namespaces can be built on top of the storage layer i.e. datanodes. > In this jira I will explore building an object store using the datanode > storage, but independent of namespace metadata. > I will soon update with a detailed design document. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org