I changed the place of --files and works.
( IT DOESN'T WORK)
spark-submit --conf spark.metrics.conf=metrics.properties --name
"myProject" --master yarn-cluster --class myCompany.spark.MyClass *--files
/opt/myProject/conf/log4j.properties* --jars $SPARK_CLASSPATH
--executor-memory 1024m
I'm trying to configure log4j in Spark.
spark-submit --conf spark.metrics.conf=metrics.properties --name
"myProject" --master yarn-cluster --class myCompany.spark.MyClass *--files
/opt/myProject/conf/log4j.properties* --jars $SPARK_CLASSPATH
--executor-memory 1024m --num-executors 5
Hi,
Am trying to configure log4j on an AWS EMR 4.2 Spark cluster for a streaming
job set in client mode.
I changed
/etc/spark/conf/log4j.properties
to use a FileAppender. However the INFO logging still goes to console.
Thanks for any suggestions,
--
Nick
>From the console:
Found the issue, a conflict between setting Java options in both
spark-defaults.conf and in the spark-submit.
--
Nick
From: Afshartous, Nick <nafshart...@turbine.com>
Sent: Friday, December 18, 2015 11:46 AM
To: user@spark.apache.org
Subject: Confi
> To: Afshartous, Nick
> Cc: user@spark.apache.org
> Subject: Re: Configuring Log4J (Spark 1.5 on EMR 4.1)
>
> This file only exists on the master and not the slave nodes, so you are
> probably running into https://issues.apache.org/jira/browse/SPARK-11105,
> which has al
Hi,
On Spark 1.5 on EMR 4.1 the message below appears in stderr in the Yarn UI.
ERROR StatusLogger No log4j2 configuration file found. Using default
configuration: logging only errors to the console.
I do see that there is
/usr/lib/spark/conf/log4j.properties
Can someone please advise
ser@spark.apache.org
Subject: Re: Configuring Log4J (Spark 1.5 on EMR 4.1)
This file only exists on the master and not the slave nodes, so you are
probably running into https://issues.apache.org/jira/browse/SPARK-11105, which
has already been fixed in the not-yet-released Spark 1.6.0. EMR will upgrade to
Sp
