[ https://issues.apache.org/jira/browse/SPARK-48217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Noam Shemesh updated SPARK-48217: --------------------------------- Priority: Critical (was: Major) > Spark stdout and stderr getting removed at end of spark job triggered from > cloudera hue workflow > ------------------------------------------------------------------------------------------------ > > Key: SPARK-48217 > URL: https://issues.apache.org/jira/browse/SPARK-48217 > Project: Spark > Issue Type: Improvement > Components: Spark Submit > Affects Versions: 1.6.0 > Reporter: Noam Shemesh > Priority: Critical > Attachments: workflow_running_logs_printed.png, > workflow_succeeded_logs_cleaned.png > > > Hello, > we are running spark job triggered from cloudera hue workflow > and spark printing stdout and stderr logs during execution as expected: > e.g. - !workflow_running_logs_printed.png! > > *But stdout and stderr logs getting cleaned when workflows finished/status > succeeded* > !workflow_succeeded_logs_cleaned.png! > > following is spark-submit command workflow is triggering: > _/usr/bin/spark-submit _ > _--master yarn-client _ > _--driver-memory 4g _ > _--executor-memory 16g _ > _--executor-cores 4 _ > _--class tst _ > _--files `ls -m *.conf | tr -d '\n '` _ > _--conf "spark.dynamicAllocation.maxExecutors=4" _ > _--conf "spark.kryoserializer.buffer.max=1024" _ > _tst.jar $*_ > > > does someone familiar with this spark job behavior or can advise ideas to fix > it? > > Thanks in advance -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org