I changed the place of --files and works.
( IT DOESN'T WORK)
spark-submit --conf spark.metrics.conf=metrics.properties --name
"myProject" --master yarn-cluster --class myCompany.spark.MyClass *--files
/opt/myProject/conf/log4j.properties* --jars $SPARK_CLASSPATH
--executor-memory 1024m
I'm trying to configure log4j in Spark.
spark-submit --conf spark.metrics.conf=metrics.properties --name
"myProject" --master yarn-cluster --class myCompany.spark.MyClass *--files
/opt/myProject/conf/log4j.properties* --jars $SPARK_CLASSPATH
--executor-memory 1024m --num-executors 5
> To: Afshartous, Nick
> Cc: user@spark.apache.org
> Subject: Re: Configuring Log4J (Spark 1.5 on EMR 4.1)
>
> This file only exists on the master and not the slave nodes, so you are
> probably running into https://issues.apache.org/jira/browse/SPARK-11105,
> which has al
Hi,
On Spark 1.5 on EMR 4.1 the message below appears in stderr in the Yarn UI.
ERROR StatusLogger No log4j2 configuration file found. Using default
configuration: logging only errors to the console.
I do see that there is
/usr/lib/spark/conf/log4j.properties
Can someone please advise
ser@spark.apache.org
Subject: Re: Configuring Log4J (Spark 1.5 on EMR 4.1)
This file only exists on the master and not the slave nodes, so you are
probably running into https://issues.apache.org/jira/browse/SPARK-11105, which
has already been fixed in the not-yet-released Spark 1.6.0. EMR will upgrade to
Sp
