Re: Block placement in HDFS
Hi All, I try to divide some data into partitions explicitly (like regions of Hbase). I wonder the following way to do is the best method. For example, when we assume a block size 64MB, a file potion corresponding to 0~63MB is allocated to first block? I have three questions as follows: Is the above method valid? Is it the best method? Is there alternative method? Thank in advance. -- Hyunsik Choi Database & Information Systems Group Dept. of Computer Science & Engineering, Korea University On Mon, 2008-11-24 at 20:44 -0800, Mahadev Konar wrote: > Hi Dennis, > I don't think that is possible to do. The block placement is determined > by HDFS internally (which is local, rack local and off rack). > > > mahadev > > > On 11/24/08 6:59 PM, "dennis81" <[EMAIL PROTECTED]> wrote: > > > > > Hi everyone, > > > > I was wondering whether it is possible to control the placement of the > > blocks of a file in HDFS. Is it possible to instruct HDFS about which nodes > > will hold the block replicas? > > > > Thanks!
Re: Block placement in HDFS
Fyi - Owen is referring to: https://issues.apache.org/jira/browse/HADOOP-2559 On 11/24/08 10:22 PM, "Owen O'Malley" <[EMAIL PROTECTED]> wrote: On Nov 24, 2008, at 8:44 PM, Mahadev Konar wrote: > Hi Dennis, > I don't think that is possible to do. No, it is not possible. > The block placement is determined > by HDFS internally (which is local, rack local and off rack). Actually, it was changed in 0.17 or so to be node-local, off-rack, and a second node off rack. -- Owen
Re: Block placement in HDFS
Hi Dennis, There were some discussions on this topic earlier: http://issues.apache.org/jira/browse/HADOOP-3799 Do you have any specific use-case for this feature? thanks, dhruba On Mon, Nov 24, 2008 at 10:22 PM, Owen O'Malley <[EMAIL PROTECTED]> wrote: > > On Nov 24, 2008, at 8:44 PM, Mahadev Konar wrote: > > Hi Dennis, >> I don't think that is possible to do. >> > > No, it is not possible. > > The block placement is determined >> by HDFS internally (which is local, rack local and off rack). >> > > Actually, it was changed in 0.17 or so to be node-local, off-rack, and a > second node off rack. > > -- Owen >
Re: Block placement in HDFS
On Nov 24, 2008, at 8:44 PM, Mahadev Konar wrote: Hi Dennis, I don't think that is possible to do. No, it is not possible. The block placement is determined by HDFS internally (which is local, rack local and off rack). Actually, it was changed in 0.17 or so to be node-local, off-rack, and a second node off rack. -- Owen
Re: Block placement in HDFS
Hi Dennis, I don't think that is possible to do. The block placement is determined by HDFS internally (which is local, rack local and off rack). mahadev On 11/24/08 6:59 PM, "dennis81" <[EMAIL PROTECTED]> wrote: > > Hi everyone, > > I was wondering whether it is possible to control the placement of the > blocks of a file in HDFS. Is it possible to instruct HDFS about which nodes > will hold the block replicas? > > Thanks!
