Hello everyone, I am trying to devise my own block placement policy in
hadoop 3. My query is that when I choose favored nodes is that choice valid
fora particular file or a particular block? What if I wish to send
different blocks of the same file to different nodes. How can I control
that? Any
>> datanodes and corresponding rack numbers.
>> 2. I have gone through the default block placement policy code multiple
>> times and I can see that a parameter called favored nodes has been used. I
>> believe this parameter is of interest to me.
>>
>> The trouble is t
t; datanodes and corresponding rack numbers.
> 2. I have gone through the default block placement policy code multiple
> times and I can see that a parameter called favored nodes has been used. I
> believe this parameter is of interest to me.
>
> The trouble is that I have a set of no
block placement policy code multiple
times and I can see that a parameter called favored nodes has been used. I
believe this parameter is of interest to me.
The trouble is that I have a set of nodes with known ip addresses but I
cant see any specific ip addresses being used in the placement policy
Hi Shuubham,
You can simply create your own block placement class by extending
BlockPlacementPolicy.
As a starting point, you may want to have a look at Block Placement
Default Policy Source Code
<https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/
Can anyone give me some idea as to how I can write my own block placement
strategy in hadoop 3.
Shuubham Ojha
wrote:
Hi all,
I have a question related to HDFS Block placement policy. The default,
"The default block placement policy is as follows: Place the first
replica somewhere – either a random node (if the HDFS client is
outside the Hadoop/DataNode cluster) or on the local node (if the HDFS
clie
Hi all,
I have a question related to HDFS Block placement policy. The default,
"The default block placement policy is as follows: Place the first replica
somewhere – either a random node (if the HDFS client is outside the
Hadoop/DataNode cluster) or on the local node (if the HDFS clie
Hello All,
I was wondering if the following issues can be solved by extending hdfs
classes with custom implementations if possible.
Here are my requirements :
1. Is there a way to control that all file blocks belonging to a particular
hdfs directory & file go to the same physical datanode ( an
cality-in-hdfs.html
>
> On Thu, Oct 2, 2014 at 4:12 PM, SF Hadoop wrote:
>
>> What is the block placement policy hadoop follows when rack aware is not
>> enabled?
>>
>> Does it just round robin?
>>
>> Thanks.
>>
>
>
It appears to be randomly chosen. I just came across this blog post from
Lars George about HBase file locality in HDFS
http://www.larsgeorge.com/2010/05/hbase-file-locality-in-hdfs.html
On Thu, Oct 2, 2014 at 4:12 PM, SF Hadoop wrote:
> What is the block placement policy hadoop follows w
It’s still random.
If rack aware is not enabled, all nodes are in “default-rack”, and random nodes
are chosen for block replications.
Regards,
Yi Liu
From: SF Hadoop [mailto:sfhad...@gmail.com]
Sent: Friday, October 03, 2014 7:12 AM
To: user@hadoop.apache.org
Subject: Block placement without
What is the block placement policy hadoop follows when rack aware is not
enabled?
Does it just round robin?
Thanks.
Hi there,
According to the code, the current implement of HDFS only supports one
specific type of block placement policy, which is
BlockPlacementPolicyDefault by default.
The default policy is enough for most of the circumstances, but under some
special circumstances, it works not so well.
For
Hi,
I've made some changed to the default block placement policy and want to
see how if affects a cluster. Any suggestions on how I can test the
before and after of a cluster after making these changes?
I read up a bit on Rumen and GridMix in my search for tools that would
help me benc
>
>
>
>
> On 07/24/2014 02:12 PM, Harsh J wrote:
>>
>> Hello,
>>
>> (Inline)
>>
>> On Thu, Jul 24, 2014 at 11:11 PM, Arjun Bakshi
>> wrote:
>>>
>>> Hi,
>>>
>>> I want to write a block placement policy that takes the
4 02:12 PM, Harsh J wrote:
Hello,
(Inline)
On Thu, Jul 24, 2014 at 11:11 PM, Arjun Bakshi wrote:
Hi,
I want to write a block placement policy that takes the size of the file
being placed into account. Something like what is done in CoHadoop or BEEMR
paper. I have the following question
Hello,
(Inline)
On Thu, Jul 24, 2014 at 11:11 PM, Arjun Bakshi wrote:
> Hi,
>
> I want to write a block placement policy that takes the size of the file
> being placed into account. Something like what is done in CoHadoop or BEEMR
> paper. I have the following questions:
>
&g
Hi,
I want to write a block placement policy that takes the size of the file
being placed into account. Something like what is done in CoHadoop or
BEEMR paper. I have the following questions:
1- What is srcPath in chooseTarget? Is it the path to the original
un-chunked file, or it is a path
]
Sent: Friday, July 26, 2013 6:45 AM
To: user@hadoop.apache.org
Subject: HDFS block placement
Hey,
I am a bit confused about the block placement in Hadoop. Assume that there
is no replication and a task (map or reduce) writes a file to HDFS, will be
all blocks stored on the same local node (the no
Your thought is correct. If space is available locally, then it is
automatically stored locally.
On Fri, Jul 26, 2013 at 5:14 PM, Lukas Kairies
wrote:
> Hey,
>
> I am a bit confused about the block placement in Hadoop. Assume that there
> is no replication and a task (map or redu
Hey,
I am a bit confused about the block placement in Hadoop. Assume that
there is no replication and a task (map or reduce) writes a file to
HDFS, will be all blocks stored on the same local node (the node on
which the task runs)? I think yes but I am node sure.
Kind Regards,
Lukas Kairies
ething more specific you are looking for?
>
>
> On Mon, May 20, 2013 at 12:30 PM, Mohammad Mustaqeem <
> 3m.mustaq...@gmail.com> wrote:
>
>> Has anybody used pluggable block placement policy?
>> If yes then give direction to me, how to use this feature in hadoop
>&g
> wrote:
> Has anybody used pluggable block placement policy?
> If yes then give direction to me, how to use this feature in hadoop
> 2.0.3-alpha and I also need the code of "pluggable block placement policy".
>
> Thanks in advance.
>
> --
> *With regards ---*
&g
Has anybody used pluggable block placement policy?
If yes then give direction to me, how to use this feature in hadoop
2.0.3-alpha and I also need the code of "pluggable block placement policy".
Thanks in advance.
--
*With regards ---*
*Mohammad Mustaqeem*,
M.Tech (CSE)
MNNIT
Have a look at
org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.
This is default placement policy, you should be able to extend this to
implement your own policy and set config parameter
"dfs.block.replicator.classname" to point to your class
On Sat, May 4, 2013 at 8:57 P
topology.script.file.name in branch-1).
Hope this helps,
Ivan
From: Mohammad Tariq [mailto:donta...@gmail.com]
Sent: Saturday, May 04, 2013 8:35 AM
To: user@hadoop.apache.org
Subject: Re: Block placement Policy
You might find this useful : https://issues.apache.org/jira/browse/HDFS-385
Warm Regards,
Tariq
Do you know, how to use it?
Sorry, I am very new :(
--
*With regards ---*
*Mohammad Mustaqeem*,
M.Tech (CSE)
MNNIT Allahabad
9026604270
You might find this useful : https://issues.apache.org/jira/browse/HDFS-385
Warm Regards,
Tariq
https://mtariq.jux.com/
cloudfront.blogspot.com
On Sat, May 4, 2013 at 8:57 PM, Mohammad Mustaqeem
<3m.mustaq...@gmail.com>wrote:
> I have read somewhere that a user can specified his own
> ReplicaPl
I have read somewhere that a user can specified his own ReplicaPlacementPolicy.
How can I specify my own ReplicaPlacementPolicy?
I you have any sample ReplicaPlacementPolicy, then please share it..
--
*With regards ---*
*Mohammad Mustaqeem*,
M.Tech (CSE)
MNNIT Allahabad
9026604270
30 matches
Mail list logo
