we have a problem.
What's the unit for "dfs.balance.bandwidthPerSec" ?
-
On Mar 24, 2013, at 1:23 PM, Balaji Narayanan (பாலாஜி நாராயணன்)
wrote:
> Are you running balancer? If balancer is running and if it is slow, try
> increasing the balancer bandwidth
>
>
&g
as completed the balancing.
>
>
> On 24 March 2013 11:32, Tapas Sarangi wrote:
> Yes, we are running balancer, though a balancer process runs for almost a day
> or more before exiting and starting over.
> Current dfs.balance.bandwidthPerSec value is set to 2x10^9. I assum
nd
> drive? If it's set to multiple folders on the same drives, it is probably
> multiplying the amount of "available capacity" incorrectly in that it assumes
> a 1-1 relationship between folder and total capacity of the drive.
>
>
> On Sun, Mar 24, 2013 at 3:0
complete, bring back in the smaller nodes, but maybe only after you
> tweak the rack topology to match your disk layout more than network layout to
> compensate for the unbalanced nodes.
>
> Just my 2 cents
>
>
> On Sun, Mar 24, 2013 at 4:31 PM, Tapas Sarangi
&g
.
Why do you suggest such steps need to be tested for balancer ? Please explain.
I guess we had a discussion earlier on this thread and came to the conclusion
that the threshold will not help in this situation.
-
> On Sun, Mar 24, 2013 at 11:01 PM, Tapas Sarangi
> wrote:
&g
>
Thanks. Can you please specify which are the dfs properties that we can set or
modify to force the flow of blocks directed towards the larger nodes than the
smaller nodes ?
-
>
> On Mar 24, 2013 4:45 PM, "Tapas Sarangi" wrote:
> Hi,
>
> Thanks for the
t;
>
>
>
>
>
> -
>
>
>
>
>> On Sun, Mar 24, 2013 at 11:01 PM, Tapas Sarangi
>> wrote:
>> Yes, thanks for pointing, but I already know that it is completing the
>> balancing when exiting otherwise it shouldn't exit.
>&g
A short term hack at best, but should help the situation a
> bit.
>
> On Mar 24, 2013 7:09 PM, "Tapas Sarangi" wrote:
>
> On Mar 24, 2013, at 4:34 PM, Jamal B wrote:
>
>> It shouldn't cause further problems since most of your small nodes are
>>
Did you check if you have any disk that is "read-only" for the nodes that has
the missing blocks ? If you know which are the blocks, you can manually copy
the blocks and the corresponding '.meta' file to another node. Hadoop will
re-read those blocks and replicate them.
-
On Mar 28, 2013
On Mar 28, 2013, at 7:13 PM, Felix GV wrote:
> I'm using the version of hadoop in CDH 4.2, which is a version of Hadoop 2.0
> with a bunch of patches on top...
>
> I've tried copying one block and its .meta file to one of my new DN, then
> restarted the DN service, and it did pick up the miss
Hello,
I am using one of the old legacy version (0.20) of hadoop for our cluster. We
have scheduled for an upgrade to the newer version within a couple of months,
but I would like to understand a couple of things before moving towards the
upgrade plan.
We have about 200 datanodes and some of t
/docs/r1.0.4/commands_manual.html#balancer
>
> Regards
>
> Bertrand
>
> On Mon, Mar 18, 2013 at 10:01 PM, Tapas Sarangi
> wrote:
> Hello,
>
> I am using one of the old legacy version (0.20) of hadoop for our cluster. We
> have scheduled for an upgrade to the newe
Thanks
-Tapas
> Bertrand
>
> On Mon, Mar 18, 2013 at 10:46 PM, Tapas Sarangi
> wrote:
> Hi,
>
> Sorry about that, had it written, but thought it was obvious.
> Yes, balancer is active and running on the namenode.
>
> -Tapas
>
> On Mar 18, 2013, at 4:4
s to
>> racks?
>> About files, how is used the replication factor(s) and block size(s)?
>>
>> Surely trivial questions again.
>>
>> Bertrand
>>
>> On Mon, Mar 18, 2013 at 10:46 PM, Tapas Sarangi
>> wrote:
>> Hi,
>>
>> Sorr
. How does the balancer look for the usage percentage ?
-Tapas
> On Tue, Mar 19, 2013 at 6:56 AM, Tapas Sarangi
> wrote:
>> Hi,
>>
>> On Mar 18, 2013, at 8:21 PM, 李洪忠 wrote:
>>
>> Maybe you need to modify the rackware script to make the rack balance, i
s a file system not as a scheduler. How
multiple racks are going to help in balancing the disk-usage across datanodes ?
-Tapas
>
> 2013/3/19 Tapas Sarangi
> Hello,
>
> I am using one of the old legacy version (0.20) of hadoop for our cluster. We
> have scheduled for an upgrade
Any more follow ups ?
Thanks
-Tapas
On Mar 19, 2013, at 9:55 AM, Tapas Sarangi wrote:
>
> On Mar 18, 2013, at 11:50 PM, Harsh J wrote:
>
>> What do you mean that the balancer is always active?
>
> meaning, the same process is active for a long time. The process that
y be slow unless you raise the allowed
> bandwidth.
>
> On Wed, Mar 20, 2013 at 7:37 AM, Tapas Sarangi
> wrote:
>> Any more follow ups ?
>>
>> Thanks
>> -Tapas
>>
>> On Mar 19, 2013, at 9:55 AM, Tapas Sarangi wrote:
>>
>>>
&g
Thanks for your reply. Some follow up questions below :
On Mar 20, 2013, at 5:35 AM, Алексей Бабутин wrote:
>
>
> dfs.balance.bandwidthPerSec in hdfs-site.xml.I think balancer cant help
> you,because it makes all the nodes equal.They can differ only on balancer
> threshold.Threshold =10 by d
19 matches
Mail list logo
