Re: Logging from the job
Hi Alexander,
Where are you looking for the logs? The output of the tasks should be in
$HADOOP_LOG_DIR/userlogs/attempt*/{stdout,stderr,syslog}.
Could you provide the exact java command line your tasks are running with
(do 'ps -ef | grep Child' on one of the nodes when the job is running).
Alex K
On Tue, Apr 27, 2010 at 1:52 AM, Alexander Semenov wrote:
> Hi all.
>
> I'm not sure if I'm posting to correct mail list, please, suggest the
> correct one if so.
>
> I need to log statements from the running job, e.g. use Apache commons
> logging to print debug messages on map and reduce operations. I've tuned
> the conf/log4.properties file for my logging domain but log statements
> are still missing in the log files and on the console. I start the job
> like this:
>
> hadoop jar jar_file.jar input_dir output_dir
>
> The job finishes gracefully but I see no logging.
>
> Any suggestions?
> Thanks.
>
>
Re: Logging from the job
Alexander Semenov wrote:
Ok, thanks. Unfortunately ant is currently not installed on machine
running hadoop. What if I use slf4j + logback just in job's jar?
depends on the classpath. You can do it in your own code by getting
whatever classloader you use then call getResource("log4.properties") to
get the URL, which you can print to the console, something like:
System.err.println("Log4J properties at
"+this.getClass().getClassloader().getResource("log4.properties"));
BTW, is hadoop planning to migrate on this stack from the deprecated
Apache commons and log4j?
Deprecated? That's very much a point of view.
1. commons-log is a thin front end to anything; I have my own that can
be switched in when I desire to get the logs from >1 machine in one
place. Where it is great for is in libraries which can be embedded in
other things -Hadoop does get used this way- as it stops the library
saying "here is the logging tool you must use". you get to choose.
2. Log4J is a fantastic logging API with some really good back end
implementations. In Hadoop, I'd recommend the rolling Logs. There is
good support in Hadoop for changing logging levels as you go along.
3. slf4j was meant to be a log api that avoided all the problems of
layering, but it actually introduces a new one: the risk of >1 SLF4J on
the classpath. And, as it's an extra JAR that Jetty requires, you have
to make sure your job's SLF4j doesn't clash with the one used for Jetty.
And its back-ends aren't on a par with Log4J.
I'm happy with commons-logging and log4j, just wish that jetty had
stayed with commons-logging. The alternative would be the
java.util.logging APIs, which are themselves a dog to configure. You
need to point to the right config file using JVM system properties which
really need to be set on the command line to get picked up early enough,
and as the docs say:
"By default, the LogManager reads its initial configuration from a
properties file "lib/logging.properties" in the JRE directory. "
This isn't as bad as having a commons-logging or log4.properties in the
JAR of a third party library, but it means that by default, one logging
setup per JVM unless you deliberately configure each app differently.
Which is a PITA, and its probably one of the reasons that the java
logging API never took off, and doesn't have as good back end loggers as
Log4J.
Summary: try and find the properties file, it may just be classpath/JVM
quirks, learn to use the rolling/daily log4j logs, don't keep the logs
on your root drive either.
-Steve
Re: Logging from the job
Ok, thanks. Unfortunately ant is currently not installed on machine running hadoop. What if I use slf4j + logback just in job's jar? BTW, is hadoop planning to migrate on this stack from the deprecated Apache commons and log4j? On Tue, 2010-04-27 at 10:57 +0100, Steve Loughran wrote: > Jeff Zhang wrote: > > you should use the log4j rather than the apache common logging > > > > Not necessarily. Hadoop code uses commons logging which defaults to going > to log4j, but lets you plug in different back ends via JVM properties. > This lets you do some tricks with log aggregation. > > If changes to log4.properties don't seem to get picked up, it could be > some JAR has a copy of the file which is getting in the way. You can use > Ant's task to maybe track this down. > > classpathref="some classpath reference">
Re: Logging from the job
I'm expecting to see the logs on the console since the root logger is configured to do so. On Tue, 2010-04-27 at 14:28 +0530, Amareshwari Sri Ramadasu wrote: > Where are you looking for the logs? > They will be available in Tasklogs. You can view them from web ui from > taskdetails.jsp page. > > -Amareshwari > > On 4/27/10 2:22 PM, "Alexander Semenov" wrote: > > Hi all. > > I'm not sure if I'm posting to correct mail list, please, suggest the > correct one if so. > > I need to log statements from the running job, e.g. use Apache commons > logging to print debug messages on map and reduce operations. I've tuned > the conf/log4.properties file for my logging domain but log statements > are still missing in the log files and on the console. I start the job > like this: > > hadoop jar jar_file.jar input_dir output_dir > > The job finishes gracefully but I see no logging. > > Any suggestions? > Thanks. > >
Re: Logging from the job
Jeff Zhang wrote: you should use the log4j rather than the apache common logging Not necessarily. Hadoop code uses commons logging which defaults to going to log4j, but lets you plug in different back ends via JVM properties. This lets you do some tricks with log aggregation. If changes to log4.properties don't seem to get picked up, it could be some JAR has a copy of the file which is getting in the way. You can use Ant's task to maybe track this down. classpathref="some classpath reference">
Re: Logging from the job
you should use the log4j rather than the apache common logging On Tue, Apr 27, 2010 at 4:52 PM, Alexander Semenov wrote: > Hi all. > > I'm not sure if I'm posting to correct mail list, please, suggest the > correct one if so. > > I need to log statements from the running job, e.g. use Apache commons > logging to print debug messages on map and reduce operations. I've tuned > the conf/log4.properties file for my logging domain but log statements > are still missing in the log files and on the console. I start the job > like this: > > hadoop jar jar_file.jar input_dir output_dir > > The job finishes gracefully but I see no logging. > > Any suggestions? > Thanks. > > -- Best Regards Jeff Zhang
Re: Logging from the job
Where are you looking for the logs? They will be available in Tasklogs. You can view them from web ui from taskdetails.jsp page. -Amareshwari On 4/27/10 2:22 PM, "Alexander Semenov" wrote: Hi all. I'm not sure if I'm posting to correct mail list, please, suggest the correct one if so. I need to log statements from the running job, e.g. use Apache commons logging to print debug messages on map and reduce operations. I've tuned the conf/log4.properties file for my logging domain but log statements are still missing in the log files and on the console. I start the job like this: hadoop jar jar_file.jar input_dir output_dir The job finishes gracefully but I see no logging. Any suggestions? Thanks.
Logging from the job
Hi all. I'm not sure if I'm posting to correct mail list, please, suggest the correct one if so. I need to log statements from the running job, e.g. use Apache commons logging to print debug messages on map and reduce operations. I've tuned the conf/log4.properties file for my logging domain but log statements are still missing in the log files and on the console. I start the job like this: hadoop jar jar_file.jar input_dir output_dir The job finishes gracefully but I see no logging. Any suggestions? Thanks.
