RE: Any reason Hadoop logs cant be directed to a separate filesystem?
Yes, and its called using cron and writing a simple ksh script to clear out any
files that are older than 15 days.
There may be another way, but that's really the easiest.
Date: Thu, 23 Jun 2011 02:44:48 +0530
From: jagaran_...@yahoo.co.in
Subject: Re: Any reason Hadoop logs cant be directed to a separate filesystem?
To: common-user@hadoop.apache.org
Hi,
Can I limit the log file duration ?
I want to keep files for last 15 days only.
Regards,
Jagaran
From: Jack Craig jcr...@carrieriq.com
To: common-user@hadoop.apache.org common-user@hadoop.apache.org
Sent: Wed, 22 June, 2011 2:00:23 PM
Subject: Re: Any reason Hadoop logs cant be directed to a separate filesystem?
Thx to both respondents.
Note i've not tried this redirection as I have only production grids
available.
Our grids are growing and with them, log volume.
As until now that log location has been in the same fs as the grid data,
so running out of space due log bloat is a growing problem.
From your replies, sounds like I can relocate my logs, Cool!
But now the tough question, if i set up a too small partition and it runs out
of
space,
will my grid become unstable if hadoop can no longer write to its logs?
Thx again, jackc...
Jack Craig, Operations
CarrierIQ.comhttp://CarrierIQ.com
1200 Villa Ct, Suite 200
Mountain View, CA. 94041
650-625-5456
On Jun 22, 2011, at 1:09 PM, Harsh J wrote:
Jack,
I believe the location can definitely be set to any desired path.
Could you tell us the issues you face when you change it?
P.s. The env var is used to set the config property hadoop.log.dir
internally. So as long as you use the regular scripts (bin/ or init.d/
ones) to start daemons, it would apply fine.
On Thu, Jun 23, 2011 at 1:32 AM, Jack Craig
jcr...@carrieriq.commailto:jcr...@carrieriq.com wrote:
Hi Folks,
In the hadoop-env.sh, we find, ...
# Where log files are stored. $HADOOP_HOME/logs by default.
# export HADOOP_LOG_DIR=${HADOOP_HOME}/logs
is there any reason this location could not be a separate filesystem on the
name
node?
Thx, jackc...
Jack Craig, Operations
CarrierIQ.comhttp://CarrierIQ.com
1200 Villa Ct, Suite 200
Mountain View, CA. 94041
650-625-5456
--
Harsh J
Re: Any reason Hadoop logs cant be directed to a separate filesystem?
yeah, tats what we do.
But its again an extra process, if hadoop had an ability, then it would be
great.
it uses log4j, i tired to tweak it, but it is throwing error.
Regards,
Jagaran
From: Michael Segel michael_se...@hotmail.com
To: common-user@hadoop.apache.org
Sent: Sat, 25 June, 2011 3:58:19 AM
Subject: RE: Any reason Hadoop logs cant be directed to a separate filesystem?
Yes, and its called using cron and writing a simple ksh script to clear out any
files that are older than 15 days.
There may be another way, but that's really the easiest.
Date: Thu, 23 Jun 2011 02:44:48 +0530
From: jagaran_...@yahoo.co.in
Subject: Re: Any reason Hadoop logs cant be directed to a separate filesystem?
To: common-user@hadoop.apache.org
Hi,
Can I limit the log file duration ?
I want to keep files for last 15 days only.
Regards,
Jagaran
From: Jack Craig jcr...@carrieriq.com
To: common-user@hadoop.apache.org common-user@hadoop.apache.org
Sent: Wed, 22 June, 2011 2:00:23 PM
Subject: Re: Any reason Hadoop logs cant be directed to a separate filesystem?
Thx to both respondents.
Note i've not tried this redirection as I have only production grids
available.
Our grids are growing and with them, log volume.
As until now that log location has been in the same fs as the grid data,
so running out of space due log bloat is a growing problem.
From your replies, sounds like I can relocate my logs, Cool!
But now the tough question, if i set up a too small partition and it runs out
of
space,
will my grid become unstable if hadoop can no longer write to its logs?
Thx again, jackc...
Jack Craig, Operations
CarrierIQ.comhttp://CarrierIQ.com
1200 Villa Ct, Suite 200
Mountain View, CA. 94041
650-625-5456
On Jun 22, 2011, at 1:09 PM, Harsh J wrote:
Jack,
I believe the location can definitely be set to any desired path.
Could you tell us the issues you face when you change it?
P.s. The env var is used to set the config property hadoop.log.dir
internally. So as long as you use the regular scripts (bin/ or init.d/
ones) to start daemons, it would apply fine.
On Thu, Jun 23, 2011 at 1:32 AM, Jack Craig
jcr...@carrieriq.commailto:jcr...@carrieriq.com wrote:
Hi Folks,
In the hadoop-env.sh, we find, ...
# Where log files are stored. $HADOOP_HOME/logs by default.
# export HADOOP_LOG_DIR=${HADOOP_HOME}/logs
is there any reason this location could not be a separate filesystem on the
name
node?
Thx, jackc...
Jack Craig, Operations
CarrierIQ.comhttp://CarrierIQ.com
1200 Villa Ct, Suite 200
Mountain View, CA. 94041
650-625-5456
--
Harsh J
Re: Any reason Hadoop logs cant be directed to a separate filesystem?
Looks like you missed the '#' in line beginning
Feel free to set HADOOP_LOG_DIR in that script or elsewhere
On 6/22/11 1:02 PM, Jack Craig jcr...@carrieriq.com wrote:
Hi Folks,
In the hadoop-env.sh, we find, ...
# Where log files are stored. $HADOOP_HOME/logs by default.
# export HADOOP_LOG_DIR=${HADOOP_HOME}/logs
is there any reason this location could not be a separate filesystem on
the name node?
Thx, jackc...
Jack Craig, Operations
CarrierIQ.comhttp://CarrierIQ.com
1200 Villa Ct, Suite 200
Mountain View, CA. 94041
650-625-5456
Re: Any reason Hadoop logs cant be directed to a separate filesystem?
Jack,
I believe the location can definitely be set to any desired path.
Could you tell us the issues you face when you change it?
P.s. The env var is used to set the config property hadoop.log.dir
internally. So as long as you use the regular scripts (bin/ or init.d/
ones) to start daemons, it would apply fine.
On Thu, Jun 23, 2011 at 1:32 AM, Jack Craig jcr...@carrieriq.com wrote:
Hi Folks,
In the hadoop-env.sh, we find, ...
# Where log files are stored. $HADOOP_HOME/logs by default.
# export HADOOP_LOG_DIR=${HADOOP_HOME}/logs
is there any reason this location could not be a separate filesystem on the
name node?
Thx, jackc...
Jack Craig, Operations
CarrierIQ.comhttp://CarrierIQ.com
1200 Villa Ct, Suite 200
Mountain View, CA. 94041
650-625-5456
--
Harsh J
Re: Any reason Hadoop logs cant be directed to a separate filesystem?
Thx to both respondents.
Note i've not tried this redirection as I have only production grids available.
Our grids are growing and with them, log volume.
As until now that log location has been in the same fs as the grid data,
so running out of space due log bloat is a growing problem.
From your replies, sounds like I can relocate my logs, Cool!
But now the tough question, if i set up a too small partition and it runs out
of space,
will my grid become unstable if hadoop can no longer write to its logs?
Thx again, jackc...
Jack Craig, Operations
CarrierIQ.comhttp://CarrierIQ.com
1200 Villa Ct, Suite 200
Mountain View, CA. 94041
650-625-5456
On Jun 22, 2011, at 1:09 PM, Harsh J wrote:
Jack,
I believe the location can definitely be set to any desired path.
Could you tell us the issues you face when you change it?
P.s. The env var is used to set the config property hadoop.log.dir
internally. So as long as you use the regular scripts (bin/ or init.d/
ones) to start daemons, it would apply fine.
On Thu, Jun 23, 2011 at 1:32 AM, Jack Craig
jcr...@carrieriq.commailto:jcr...@carrieriq.com wrote:
Hi Folks,
In the hadoop-env.sh, we find, ...
# Where log files are stored. $HADOOP_HOME/logs by default.
# export HADOOP_LOG_DIR=${HADOOP_HOME}/logs
is there any reason this location could not be a separate filesystem on the
name node?
Thx, jackc...
Jack Craig, Operations
CarrierIQ.comhttp://CarrierIQ.com
1200 Villa Ct, Suite 200
Mountain View, CA. 94041
650-625-5456
--
Harsh J
Re: Any reason Hadoop logs cant be directed to a separate filesystem?
Hi,
Can I limit the log file duration ?
I want to keep files for last 15 days only.
Regards,
Jagaran
From: Jack Craig jcr...@carrieriq.com
To: common-user@hadoop.apache.org common-user@hadoop.apache.org
Sent: Wed, 22 June, 2011 2:00:23 PM
Subject: Re: Any reason Hadoop logs cant be directed to a separate filesystem?
Thx to both respondents.
Note i've not tried this redirection as I have only production grids available.
Our grids are growing and with them, log volume.
As until now that log location has been in the same fs as the grid data,
so running out of space due log bloat is a growing problem.
From your replies, sounds like I can relocate my logs, Cool!
But now the tough question, if i set up a too small partition and it runs out
of
space,
will my grid become unstable if hadoop can no longer write to its logs?
Thx again, jackc...
Jack Craig, Operations
CarrierIQ.comhttp://CarrierIQ.com
1200 Villa Ct, Suite 200
Mountain View, CA. 94041
650-625-5456
On Jun 22, 2011, at 1:09 PM, Harsh J wrote:
Jack,
I believe the location can definitely be set to any desired path.
Could you tell us the issues you face when you change it?
P.s. The env var is used to set the config property hadoop.log.dir
internally. So as long as you use the regular scripts (bin/ or init.d/
ones) to start daemons, it would apply fine.
On Thu, Jun 23, 2011 at 1:32 AM, Jack Craig
jcr...@carrieriq.commailto:jcr...@carrieriq.com wrote:
Hi Folks,
In the hadoop-env.sh, we find, ...
# Where log files are stored. $HADOOP_HOME/logs by default.
# export HADOOP_LOG_DIR=${HADOOP_HOME}/logs
is there any reason this location could not be a separate filesystem on the
name
node?
Thx, jackc...
Jack Craig, Operations
CarrierIQ.comhttp://CarrierIQ.com
1200 Villa Ct, Suite 200
Mountain View, CA. 94041
650-625-5456
--
Harsh J
