Re: Max Connect retries

That is for client connect retry in ipc level. You can decrease the max.retries by configuring ipc.client.connect.max.retries.on.timeouts in core-site.xml Thanks Xuan Gong From: Telles Nobrega mailto:tellesnobr...@gmail.com>> Reply-To: "user@hadoop.apache.org"

Stopping ntpd signals SIGTERM, then causes namenode exit

A shell script is deployed on every node of HDFS cluster, the script is invoked hourly by crontab, and its content is as follows: #!/bin/bash service ntpd stop ntpdate 192.168.0.1 #it's a valid ntpd server in LAN service ntpd start chkconfig ntpd on After several days, NameNode crashed suddenly,

Re: Name Node format error

check hadoop version across the cluster, include the client machine. Drake 민영근 Ph.D kt NexR On Sun, Feb 8, 2015 at 8:04 AM, SP wrote: > Hi All, > > > I see these error in my JN logs. when I am trying to setup HA. can any one > help. > > 2015-02-07 14:32:41,220 WARN org.apache.hadoop.ipc.Server:

Re: Home for Apache Big Data Solutions?

Dear Amir, Thank you for your interest in contributing these projects to the ASF! Sincerely appreciate it. My suggestion would be to look into the Apache Incubator, which is the home for incoming projects at the ASF. The TL;DR answer is: 1. You’ll need to create a proposal for each project that

Re: Home for Apache Big Data Solutions?

Hi Amir, do you already have a kind of proposal/description of your projects ? Is it like distribution, or tooling ? What's the current license ? (by the way, I know Juju as I created some charms for it). Thanks, Regards JB On 02/08/2015 05:36 PM, MrAsanjar . wrote: Hi all, My name is Amir Sa

Home for Apache Big Data Solutions?

Hi all, My name is Amir Sanjar, Big Data Solution Development Lead at Canonical. My team has been developing various Big Data solutions build on top of Apache Hadoop projects (i.e. Hadoop, Hive, Pig,..) . We would like to contribute these pure open source solutions to the Apache community. I wish t

Re: Adding datanodes to Hadoop cluster - Will data redistribute?

Hi Manoj, Existing data is not automatically redistributed when you add new DataNodes. Take a look at the 'hdfs balancer' command which can be run as a separate administrative tool to rebalance data distribution across DataNodes. From: Manoj Venkatesh mailto:manove...@gmail.com>> Reply-To: "us

Re: Adding datanodes to Hadoop cluster - Will data redistribute?

Hi Manoj, Pls try http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HdfsUserGuide.html#Balancer Rg: Vikas Parashar (Vicky) On Sat, Feb 7, 2015 at 1:04 AM, Manoj Venkatesh wrote: > Dear Hadoop experts, > > I have a Hadoop cluster of 8 nodes, 6 were added during cluster cre

Re: Adding datanodes to Hadoop cluster - Will data redistribute?

Look at hdfs balancer Artem Ervits On Feb 6, 2015 5:54 PM, "Manoj Venkatesh" wrote: > Dear Hadoop experts, > > I have a Hadoop cluster of 8 nodes, 6 were added during cluster creation > and 2 additional nodes were added later to increase disk and CPU capacity. > What i see is that processing is

Re: Adding datanodes to Hadoop cluster - Will data redistribute?

First confirm if new nodes are added into cluster or not. You can use "hadoop dfsadmin -report" command to check per node hdfs usage. If new nodes are listed in this command then you can run hadoop balancer to manually redistribute some of the data. Regards, Chandrashekhar On 07-Feb-2015 4:24 AM,

Re: Adding datanodes to Hadoop cluster - Will data redistribute?

Hi, Have you tried; $ hdfs balancer On 02/06/2015 09:34 PM, Manoj Venkatesh wrote: Dear Hadoop experts, I have a Hadoop cluster of 8 nodes, 6 were added during cluster creation and 2 additional nodes were added later to increase disk and CPU capacity. What i see is that processing is shared

Name Node format error

Hi All, I see these error in my JN logs. when I am trying to setup HA. can any one help. 2015-02-07 14:32:41,220 WARN org.apache.hadoop.ipc.Server: Incorrect header or version mismatch from 192.168.1.100:45535 got version 7 expected version 9 2015-02-07 14:35:35,244 WARN org.apache.hadoop.ipc.Se

jobcache directories data retention

Hi all, Our hadoop nodes suffer from high utilization of inodes, which probably eventually brings us to blacklisted job trackers. We found that a lot of the inodes are used under the jobcache library as directories (most empty, some are not) of what it seems as a long finished jobs. It does looks

Re: Adding datanodes to Hadoop cluster - Will data redistribute?

Have you tried hdfs balancer? http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HDFSCommands.html#balancer On Fri, Feb 6, 2015 at 11:34 AM, Manoj Venkatesh wrote: > Dear Hadoop experts, > > I have a Hadoop cluster of 8 nodes, 6 were added during cluster creation > and 2 addi

Re: Adding datanodes to Hadoop cluster - Will data redistribute?

Hi Manoj, You need to use balancer to re-balance data between nodes. http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HdfsUserGuide.html#Balancer > *dfs.datanode.fsdataset.volume.choosing.policy* have options 'Round > Robin' or 'Available Space', are there any other configur

Max Connect retries

Hi, I changed my cluster config so a failed nodemanager can be detected in about 30 seconds. When I'm running a wordcount the reduce gets stuck in 25% for a quite while and logs show nodes trying to connect to the failed node: org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-telles

Re: Adding datanodes to Hadoop cluster - Will data redistribute?

Look at the hadoop balancer - it will live data around and balance it across the nodes. Sent from the wilds on my BlackBerry smartphone. From: Manoj Venkatesh Sent: Friday, February 6, 2015 5:54 PM To: user@hadoop.apache.org Reply To: user@hadoop.apache.org Subject: Adding datanodes to Hadoop clu