Hi, I am currently running pyspark jobs against Spark 1.1.0 on YARN. When I run example Java jobs such as spark-pi, the following files get created:
bash-4.1$ tree spark-pi-1420624364958 spark-pi-1420624364958 âââ APPLICATION_COMPLETE âââ EVENT_LOG_1 âââ SPARK_VERSION_1.1.0 0 directories, 3 files However, when I run my pyspark job, no APPLICATION_COMPLETE file gets created. bash-4.1$ tree pyspark-1420628130353 pyspark -1420628130353 âââ EVENT_LOG_1 âââ SPARK_VERSION_1.1.0 0 directories, 2 files If I touch the file into this directory, it just appears as <not started> in the history server UI. I am submitting jobs using spark-submit for now: bin/spark-submit --master yarn-client --executor-memory 4G --executor-cores 12 --num-executors 10 –queue highpriority <path to python file> Is there a setting I am missing for this APPLICATION_COMPLETE file to be created when a pyspark job completes? Thanks, Michael This e-mail (including any attachments) is private and confidential, may contain proprietary or privileged information and is intended for the named recipient(s) only. Unintended recipients are strictly prohibited from taking action on the basis of information in this e-mail and must contact the sender immediately, delete this e-mail (and all attachments) and destroy any hard copies. Nomura will not accept responsibility or liability for the accuracy or completeness of, or the presence of any virus or disabling code in, this e-mail. If verification is sought please request a hard copy. Any reference to the terms of executed transactions should be treated as preliminary only and subject to formal written confirmation by Nomura. Nomura reserves the right to retain, monitor and intercept e-mail communications through its networks (subject to and in accordance with applicable laws). No confidentiality or privilege is waived or lost by Nomura by any mistransmission of this e-mail. Any reference to "Nomura" is a reference to any entity in the Nomura Holdings, Inc. group. Please read our Electronic Communications Legal Notice which forms part of this e-mail: http://www.Nomura.com/email_disclaimer.htm
