Hmm, clearly the parameter is not passed to the program. This should be an
activator issue. I wonder how do you specify the other parameters, like
driver memory, num cores, etc.? Just out of curiosity, can you run a
program:
import org.apache.spark.SparkConf
val out=new SparkConf(true).get("spark.driver.extraJavaOptions")
in your env and see what the output is?
Also, make sure spark-defaults.conf is on your classpath.
On Tue, Oct 6, 2015 at 11:19 AM, Jeff Jones <jjo...@adaptivebiotech.com>
wrote:
> Here’s an example. I echoed JAVA_OPTS so that you can see what I’ve got.
> Then I call ‘activator run’ in the project directory.
>
>
> jjones-mac:analyzer-perf jjones$ echo $JAVA_OPTS
>
> -Xmx4g -Xmx4g
> -Dlog4j.configuration=file:/Users/jjones/src/adaptive/adaptiveobjects/analyzer-perf/conf/log4j.properties
>
> jjones-mac:analyzer-perf jjones$ activator run
>
> [info] Loading project definition from
> /Users/jjones/src/adaptive/adaptiveobjects/analyzer-perf/project
>
> [info] Set current project to analyzer-perf (in build
> file:/Users/jjones/src/adaptive/adaptiveobjects/analyzer-perf/)
>
> [info] Running com.adaptive.analyzer.perf.AnalyzerPerf
>
> 11:15:24.066 [run-main-0] INFO org.apache.spark.SparkContext - Running
> Spark version 1.4.1
>
> 11:15:24.150 [run-main-0] DEBUG o.a.h.m.lib.MutableMetricsFactory - field
> org.apache.hadoop.metrics2.lib.MutableRate
> org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginSuccess
> with annotation @org.apache.hadoop.metrics2.annotation.Metric(about=,
> always=false, sampleName=Ops, type=DEFAULT, value=[Rate of successful
> kerberos logins and latency (milliseconds)], valueName=Time)
>
> 11:15:24.156 [run-main-0] DEBUG o.a.h.m.lib.MutableMetricsFactory - field
> org.apache.hadoop.metrics2.lib.MutableRate
> org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginFailure
> with annotation @org.apache.hadoop.metrics2.annotation.Metric(about=,
> always=false, sampleName=Ops, type=DEFAULT, value=[Rate of failed kerberos
> logins and latency (milliseconds)], valueName=Time)
>
> As I mentioned below but repeated for completeness, I also have this in my
> code.
>
> import org.apache.log4j.PropertyConfigurator
>
> PropertyConfigurator.configure("conf/log4j.properties")
> Logger.getRootLogger().setLevel(Level.OFF)
> Logger.getLogger("org").setLevel(Level.OFF)
> Logger.getLogger("akka").setLevel(Level.OFF)
>
> And here’s my log4j.properties (note, I’ve also tried setting the level to
> OFF):
>
> # Set everything to be logged to the console
>
> log4j.rootCategory=WARN
>
> log4j.appender.console=org.apache.log4j.ConsoleAppender
>
> log4j.appender.console.layout=org.apache.log4j.PatternLayout
>
> log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p
> %c{1}: %m%n
>
>
> # Change this to set Spark log level
>
> log4j.logger.org.apache.spark=WARN
>
>
> # Silence akka remoting
>
> log4j.logger.Remoting=WARN
>
>
> # Ignore messages below warning level from Jetty, because it's a bit
> verbose
>
> log4j.logger.org.eclipse.jetty=WARN
>
>
> spark.log.threshold=OFF
>
> spark.root.logger=OFF,DRFA
>
>
> From: Alex Kozlov
> Date: Tuesday, October 6, 2015 at 10:50 AM
>
> To: Jeff Jones
> Cc: "user@spark.apache.org"
> Subject: Re: How can I disable logging when running local[*]?
>
> Try
>
> JAVA_OPTS='-Dlog4j.configuration=file:/<path-to-log4j.properties>'
>
> Internally, this is just spark.driver.extraJavaOptions, which you should
> be able to set in conf/spark-defaults.conf
>
> Can you provide more details how you invoke the driver?
>
> On Tue, Oct 6, 2015 at 9:48 AM, Jeff Jones <jjo...@adaptivebiotech.com>
> wrote:
>
>> Thanks. Any chance you know how to pass this to a Scala app that is run
>> via TypeSafe activator?
>>
>> I tried putting it $JAVA_OPTS but I get:
>>
>> Unrecognized option: --driver-java-options
>>
>> Error: Could not create the Java Virtual Machine.
>>
>> Error: A fatal exception has occurred. Program will exit.
>>
>>
>> I tried a bunch of different quoting but nothing produced a good result.
>> I also tried passing it directly to activator using –jvm but it still
>> produces the same results with verbose logging. Is there a way I can tell
>> if it’s picking up my file?
>>
>>
>>
>> From: Alex Kozlov
>> Date: Monday, October 5, 2015 at 8:34 PM
>> To: Jeff Jones
>> Cc: "user@spark.apache.org"
>> Subject: Re: How can I disable logging when running local[*]?
>>
>> Did you try “--driver-java-options
>> '-Dlog4j.configuration=file:/<path-to-log4j.properties>'” and setting the
>> log4j.rootLogger=FATAL,console?
>>
>> On Mon, Oct 5, 2015 at 8:19 PM, Jeff Jones <jjo...@adaptivebiotech.com>
>> wrote:
>>
>>> I’ve written an application that hosts the Spark driver in-process using
>>> “local[*]”. I’ve turned off logging in my conf/log4j.properties file. I’ve
>>> also tried putting the following code prior to creating my SparkContext.
>>> These were coupled together from various posts I’ve. None of these steps
>>> have worked. I’m still getting a ton of logging to the console. Anything
>>> else I can try?
>>>
>>> Thanks,
>>> Jeff
>>>
>>> private def disableLogging(): Unit = {
>>> import org.apache.log4j.PropertyConfigurator
>>>
>>> PropertyConfigurator.configure("conf/log4j.properties")
>>> Logger.getRootLogger().setLevel(Level.OFF)
>>> Logger.getLogger("org").setLevel(Level.OFF)
>>> Logger.getLogger("akka").setLevel(Level.OFF)
>>> }
>>>
>>>
>>>
>>> This message (and any attachments) is intended only for the designated
>>> recipient(s). It
>>> may contain confidential or proprietary information, or have other
>>> limitations on use as
>>> indicated by the sender. If you are not a designated recipient, you may
>>> not review, use,
>>> copy or distribute this message. If you received this in error, please
>>> notify the sender by
>>> reply e-mail and delete this message.
>>>
>>
>>
>>
>> --
>> Alex Kozlov
>> (408) 507-4987
>> (408) 830-9982 fax
>> (650) 887-2135 efax
>> ale...@gmail.com
>>
>>
>> This message (and any attachments) is intended only for the designated
>> recipient(s). It
>> may contain confidential or proprietary information, or have other
>> limitations on use as
>> indicated by the sender. If you are not a designated recipient, you may
>> not review, use,
>> copy or distribute this message. If you received this in error, please
>> notify the sender by
>> reply e-mail and delete this message.
>>
>
>
>
> This message (and any attachments) is intended only for the designated
> recipient(s). It
> may contain confidential or proprietary information, or have other
> limitations on use as
> indicated by the sender. If you are not a designated recipient, you may
> not review, use,
> copy or distribute this message. If you received this in error, please
> notify the sender by
> reply e-mail and delete this message.
>
--
Alex Kozlov
(408) 507-4987
(408) 830-9982 fax
(650) 887-2135 efax
ale...@gmail.com
