Re: SparkSubmit.scala and stderr

2015-02-03 Thread Marcelo Vanzin
Hi Jay, On Tue, Feb 3, 2015 at 6:28 AM, jayhutfles jayhutf...@gmail.com wrote: // Exposed for testing private[spark] var printStream: PrintStream = System.err But as the comment states that it's for testing, maybe I'm misunderstanding its intent... The comment is there to tell

Re: SparkSubmit.scala and stderr

2015-02-03 Thread Reynold Xin
We can use ScalaTest's privateMethodTester also instead of exposing that. On Tue, Feb 3, 2015 at 2:18 PM, Marcelo Vanzin van...@cloudera.com wrote: Hi Jay, On Tue, Feb 3, 2015 at 6:28 AM, jayhutfles jayhutf...@gmail.com wrote: // Exposed for testing private[spark] var printStream:

SparkSubmit.scala and stderr

2015-02-03 Thread jayhutfles
Hi all, I just saw that the SparkSubmit.scala class has the following lines: object SparkSubmit { ... // Exposed for testing private[spark] var printStream: PrintStream = System.err ... } This causes all verbose logging messages elsewhere in SparkSubmit to go to stderr, not

Re: SparkSubmit.scala and stderr

2015-02-03 Thread Evan Chan
Why not just use SLF4J? On Tue, Feb 3, 2015 at 2:22 PM, Reynold Xin r...@databricks.com wrote: We can use ScalaTest's privateMethodTester also instead of exposing that. On Tue, Feb 3, 2015 at 2:18 PM, Marcelo Vanzin van...@cloudera.com wrote: Hi Jay, On Tue, Feb 3, 2015 at 6:28 AM,

Re: SparkSubmit.scala and stderr

2015-02-03 Thread Sean Owen
Despite its name, stderr is frequently used as the destination for anything that's not the output of the program, which includes log messages. That way, for example, you can redirect the output of such a program to capture its result without also capturing log or error messages, which will still