Re: Unable to redirect Spark logs to slf4j

2014-03-05 Thread Patrick Wendell
Hey, Maybe I don't understand the slf4j model completely, but I think you need to add a concrete implementation of a logger. So in your case you'd the logback-classic binding in place of the log4j binding at compile time: http://mvnrepository.com/artifact/ch.qos.logback/logback-classic/1.1.1 - P

Re: Unable to redirect Spark logs to slf4j

2014-03-05 Thread Sergey Parhomenko
Hi Patrick, Thanks for the patch. I tried building a patched version of spark-core_2.10-0.9.0-incubating.jar but the Maven build fails: *[ERROR] /home/das/Work/thx/incubator-spark/core/src/main/scala/org/apache/spark/Logging.scala:22: object impl is not a member of package org.slf4j* *[ERROR] impo

Re: Unable to redirect Spark logs to slf4j

2014-03-05 Thread Patrick Wendell
Hey All, We have a fix for this but it didn't get merged yet. I'll put it as a blocker for Spark 0.9.1. https://github.com/pwendell/incubator-spark/commit/66594e88e5be50fca073a7ef38fa62db4082b3c8 https://spark-project.atlassian.net/browse/SPARK-1190 Sergey if you could try compiling Spark with

Re: Unable to redirect Spark logs to slf4j

2014-03-05 Thread Paul Brown
Hi, Sergey -- Here's my recipe, implemented via Maven; YMMV if you need to do it via sbt, etc., but it should be equivalent: 1) Replace org.apache.spark.Logging trait with this: https://gist.github.com/prb/bc239b1616f5ac40b4e5 (supplied by Patrick during the discussion on the dev list) 2) Amend y

Re: Unable to redirect Spark logs to slf4j

2014-03-05 Thread Sergey Parhomenko
Hi Sean, We're not using log4j actually, we're trying to redirect all logging to slf4j which then uses logback as the logging implementation. The fix you mentioned - am I right to assume it is not part of the latest released Spark version (0.9.0)? If so, are there any workarounds or advices on ho

Re: Unable to redirect Spark logs to slf4j

2014-03-05 Thread Sean Owen
Yes I think that issue is fixed (Patrick you had the last eyes on it IIRC?) If you are using log4j, in general, do not redirect log4j to slf4j. Stuff using log4j is already using log4j, done. -- Sean Owen | Director, Data Science | London On Wed, Mar 5, 2014 at 1:12 PM, Sergey Parhomenko wrote:

Unable to redirect Spark logs to slf4j

2014-03-05 Thread Sergey Parhomenko
Hi, I'm trying to redirect Spark logs to slf4j. Spark seem to be using Log4J, so I did the typical steps of forcing a Log4J-based framework to use slf4j - manually excluded slf4j-log4j12 and log4j, and included log4j-over-slf4j. When doing that however Spark starts failing on initialization with: