[
https://issues.apache.org/jira/browse/SPARK-30597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon updated SPARK-30597:
---------------------------------
Description:
We run the spark application in Yarn HDFS/NFS/WebHDFS and standalone HDFS/NFS
mode.
when the application is submitted in standalone HDFS mode the configuration
jar(properties file) is not read when the application is started, and this make
the logger to fall back to default log file and log level.
So when application is submitted to standalone HDFS, configuration files are
not read.
STD OUT LOGS from Standalone HDFS - properties file is not found
{code}
log4j: Trying to find [osa-log.properties] using context classloader
sun.misc.Launcher$AppClassLoader@4cdf35a9.
log4j: Trying to find [osa-log.properties] using
sun.misc.Launcher$AppClassLoader@4cdf35a9 class loader.
log4j: Trying to find [osa-log.properties] using
ClassLoader.getSystemResource().
log4j: Could not find resource: [osa-log.properties].
log4j: Reading configuration from URL
jar:[file:/osa/spark-2.4.3-bin-hadoop2.7/jars/spark-core_2.11-2.4.3.jar!/org/apache/spark/log4j-defaults.properties|file:///osa/spark-2.4.3-bin-hadoop2.7/jars/spark-core_2.11-2.4.3.jar!/org/apache/spark/log4j-defaults.properties]
{code}
STD out from Standalone NFS - properties files are found and able to load it
{code}
log4j: Trying to find [osa-log.properties] using context classloader
sun.misc.Launcher$AppClassLoader@4cdf35a9.
log4j: Using URL
[jar:file:/scratch/xxxx/gitlocal/soa-osa/out/configdir/ux6m3UQZ/app/sx_DatePipeline_A44C5337_B0D6_4A67_9D60_6BE629DABADA_x0jR7lg5_public/__config__.jar!/osa-log.properties]
for automatic log4j configuration.
log4j: Preferred configurator class: OSALogPropertyConfigurator
{code}
STD out from YARN HDFS — properties files are found and able to load it
{code}
log4j: Trying to find [osa-log.properties] using context classloader
sun.misc.Launcher$AppClassLoader@2626b418.
log4j: Using URL
[file:/tmp/hadoop-yarn/nm-local-dir/usercache/osa/filecache/16/__osa_config__8272345390991627491.jar/osa-log.properties|file:///tmp/hadoop-yarn/nm-local-dir/usercache/osa/filecache/16/__osa_config__8272345390991627491.jar/osa-log.properties]
for automatic log4j configuration.
log4j: Preferred configurator class: OSALogPropertyConfigurator
log4j: configuration
[file:/tmp/hadoop-yarn/nm-local-dir/usercache/osa/filecache/16/__osa_config__8272345390991627491.jar/osa-log.properties|file:///tmp/hadoop-yarn/nm-local-dir/usercache/osa/filecache/16/__osa_config__8272345390991627491.jar/osa-log.properties]
{code}
was:
We run the spark application in Yarn HDFS/NFS/WebHDFS and standalone HDFS/NFS
mode.
when the application is submitted in standalone HDFS mode the configuration
jar(properties file) is not read when the application is started, and this make
the logger to fall back to default log file and log level.
So when application is submitted to standalone HDFS, configuration files are
not read.
STD OUT LOGS from Standalone HDFS - properties file is not found
==========
log4j: Trying to find [osa-log.properties] using context classloader
sun.misc.Launcher$AppClassLoader@4cdf35a9.
log4j: Trying to find [osa-log.properties] using
sun.misc.Launcher$AppClassLoader@4cdf35a9 class loader.
log4j: Trying to find [osa-log.properties] using
ClassLoader.getSystemResource().
log4j: Could not find resource: [osa-log.properties].
log4j: Reading configuration from URL
jar:file:/osa/spark-2.4.3-bin-hadoop2.7/jars/spark-core_2.11-2.4.3.jar!/org/apache/spark/log4j-defaults.properties
STD out from Standalone NFS - properties files are found and able to load it
===================
log4j: Trying to find [osa-log.properties] using context classloader
sun.misc.Launcher$AppClassLoader@4cdf35a9.
log4j: Using URL
[jar:file:/scratch/xxxx/gitlocal/soa-osa/out/configdir/ux6m3UQZ/app/sx_DatePipeline_A44C5337_B0D6_4A67_9D60_6BE629DABADA_x0jR7lg5_public/__config__.jar!/osa-log.properties]
for automatic log4j configuration.
log4j: Preferred configurator class: OSALogPropertyConfigurator
STD out from YARN HDFS — properties files are found and able to load it
===============
log4j: Trying to find [osa-log.properties] using context classloader
sun.misc.Launcher$AppClassLoader@2626b418.
log4j: Using URL
[file:/tmp/hadoop-yarn/nm-local-dir/usercache/osa/filecache/16/__osa_config__8272345390991627491.jar/osa-log.properties]
for automatic log4j configuration.
log4j: Preferred configurator class: OSALogPropertyConfigurator
log4j: configuration
file:/tmp/hadoop-yarn/nm-local-dir/usercache/osa/filecache/16/__osa_config__8272345390991627491.jar/osa-log.properties
> Unable to load properties fine in SparkStandalone HDFS mode
> -----------------------------------------------------------
>
> Key: SPARK-30597
> URL: https://issues.apache.org/jira/browse/SPARK-30597
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 2.4.3
> Environment: 2.4.3
> standalone with HDFS
> Reporter: Gajanan Hebbar
> Priority: Major
>
> We run the spark application in Yarn HDFS/NFS/WebHDFS and standalone HDFS/NFS
> mode.
> when the application is submitted in standalone HDFS mode the configuration
> jar(properties file) is not read when the application is started, and this
> make the logger to fall back to default log file and log level.
> So when application is submitted to standalone HDFS, configuration files are
> not read.
> STD OUT LOGS from Standalone HDFS - properties file is not found
> {code}
> log4j: Trying to find [osa-log.properties] using context classloader
> sun.misc.Launcher$AppClassLoader@4cdf35a9.
> log4j: Trying to find [osa-log.properties] using
> sun.misc.Launcher$AppClassLoader@4cdf35a9 class loader.
> log4j: Trying to find [osa-log.properties] using
> ClassLoader.getSystemResource().
> log4j: Could not find resource: [osa-log.properties].
> log4j: Reading configuration from URL
> jar:[file:/osa/spark-2.4.3-bin-hadoop2.7/jars/spark-core_2.11-2.4.3.jar!/org/apache/spark/log4j-defaults.properties|file:///osa/spark-2.4.3-bin-hadoop2.7/jars/spark-core_2.11-2.4.3.jar!/org/apache/spark/log4j-defaults.properties]
> {code}
>
>
> STD out from Standalone NFS - properties files are found and able to load it
> {code}
> log4j: Trying to find [osa-log.properties] using context classloader
> sun.misc.Launcher$AppClassLoader@4cdf35a9.
> log4j: Using URL
> [jar:file:/scratch/xxxx/gitlocal/soa-osa/out/configdir/ux6m3UQZ/app/sx_DatePipeline_A44C5337_B0D6_4A67_9D60_6BE629DABADA_x0jR7lg5_public/__config__.jar!/osa-log.properties]
> for automatic log4j configuration.
> log4j: Preferred configurator class: OSALogPropertyConfigurator
> {code}
>
> STD out from YARN HDFS — properties files are found and able to load it
> {code}
> log4j: Trying to find [osa-log.properties] using context classloader
> sun.misc.Launcher$AppClassLoader@2626b418.
> log4j: Using URL
> [file:/tmp/hadoop-yarn/nm-local-dir/usercache/osa/filecache/16/__osa_config__8272345390991627491.jar/osa-log.properties|file:///tmp/hadoop-yarn/nm-local-dir/usercache/osa/filecache/16/__osa_config__8272345390991627491.jar/osa-log.properties]
> for automatic log4j configuration.
> log4j: Preferred configurator class: OSALogPropertyConfigurator
> log4j: configuration
> [file:/tmp/hadoop-yarn/nm-local-dir/usercache/osa/filecache/16/__osa_config__8272345390991627491.jar/osa-log.properties|file:///tmp/hadoop-yarn/nm-local-dir/usercache/osa/filecache/16/__osa_config__8272345390991627491.jar/osa-log.properties]
> {code}
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
