Hi Debu, you need to define spark config properties before the jar file path at spark-submit. Everything after the jar path will be passed as arguments to your application.
Best Regards Debabrata Ghosh <[email protected]> schrieb am Do. 31. Okt. 2019 um 03:26: > Greetings All ! > > I needed some help in obtaining the application logs but I am really > confused where it's currently located. Please allow me to explain my > problem: > 1. I am running the Spark application (written in Java) in a Hortonworks > Data Platform Hadoop cluster > > 2. My spark-submit command is the following: > spark-submit --class com.ibm.spark.sparkkafkamapper.ServiceMapperMain > --master yarn --packages > org.apache.kafka:kafka-clients:0.10.0.1,org.apache.spark:spark-streaming-kafka-0-10_2.11:2.3.0 > --deploy-mode client --files > job-configuration.properties,log4j.properties,/usr/hdp/current/spark2-client/conf/hive-site.xml > ./spark-job.jar --settings=job-configuration.properties > log4j-spark.properties --conf > "spark.driver.extraJavaOptions=-Dlog4j.configuration=log4j.properties" > --conf > "spark.executor.extraJavaOptions=-Dlog4j.configuration=log4j.properties" > > 3. Content of the log4.properties is the following: > # Set everything to be logged to the console > log4j.rootCategory=ALL, console > log4j.appender.console=org.apache.log4j.ConsoleAppender > log4j.appender.console.target=System.err > log4j.appender.console.layout=org.apache.log4j.PatternLayout > log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p > %c{1}: %m%n > > # Set the default spark-shell log level to WARN. When running the > spark-shell, the > # log level for this class is used to overwrite the root logger's log > level, so that > # the user can have different defaults for the shell and regular Spark > apps. > log4j.logger.org.apache.spark.repl.Main=WARN > > # Settings to quiet third party logs that are too verbose > log4j.logger.org.spark_project.jetty=ALL > log4j.logger.org.spark_project.jetty.util.component.AbstractLifeCycle=ERROR > log4j.logger.org.apache.spark.repl.SparkIMain$exprTyper=INFO > log4j.logger.org.apache.spark.repl.SparkILoop$SparkILoopInterpreter=INFO > log4j.logger.org.apache.parquet=ERROR > log4j.logger.parquet=ERROR > > # SPARK-9183: Settings to avoid annoying messages when looking up > nonexistent UDFs in SparkSQL with Hive support > log4j.logger.org.apache.hadoop.hive.metastore.RetryingHMSHandler=FATAL > log4j.logger.org.apache.hadoop.hive.ql.exec.FunctionRegistry=ERROR > > #Any custom class debug > log4j.logger.com.vmeg.code=DEBUG > > 4. Inside the spark application, I am having plenty of logger functions > and one example is like this: > logger.info("No dataset recieved for this batch"); > > Question: My question is where is the application log files being written > ? I have checked in the yarn logs but couldn't find the messages I have > written in the java file. > Request your help please as I am little confused and know that there is > something very silly which I am missing. > > Thanks in advance ! > > Debu > -- *Roland Johann*Software Developer/Data Engineer *phenetic GmbH* Lütticher Straße 10, 50674 Köln, Germany Mobil: +49 172 365 26 46 Mail: [email protected] Web: phenetic.io Handelsregister: Amtsgericht Köln (HRB 92595) Geschäftsführer: Roland Johann, Uwe Reimann
