Hi Michael, I think you need to explicitly call sc.stop() on the spark context for it to close down properly (this doesn't happen automatically). See https://issues.apache.org/jira/browse/SPARK-2972 for more details
Andrew On Wed, Jan 7, 2015 at 3:38 AM, <[email protected]> wrote: > Hi, > > > > I am currently running pyspark jobs against Spark 1.1.0 on YARN. When I > run example Java jobs such as spark-pi, the following files get created: > > > > bash-4.1$ tree spark-pi-1420624364958 > > spark-pi-1420624364958 > > âââ APPLICATION_COMPLETE > > âââ EVENT_LOG_1 > > âââ SPARK_VERSION_1.1.0 > > > > 0 directories, 3 files > > > > However, when I run my pyspark job, no APPLICATION_COMPLETE file gets > created. > > > > bash-4.1$ tree pyspark-1420628130353 > > pyspark -1420628130353 > > âââ EVENT_LOG_1 > > âââ SPARK_VERSION_1.1.0 > > > > 0 directories, 2 files > > > > If I touch the file into this directory, it just appears as <not started> > in the history server UI. > > > > I am submitting jobs using spark-submit for now: > > > > *bin/spark-submit --master yarn-client --executor-memory 4G > --executor-cores 12 --num-executors 10 –queue highpriority <path to python > file>* > > > > > > Is there a setting I am missing for this APPLICATION_COMPLETE file to be > created when a pyspark job completes? > > > > Thanks, > > Michael > > This e-mail (including any attachments) is private and confidential, may > contain proprietary or privileged information and is intended for the named > recipient(s) only. Unintended recipients are strictly prohibited from > taking action on the basis of information in this e-mail and must contact > the sender immediately, delete this e-mail (and all attachments) and > destroy any hard copies. Nomura will not accept responsibility or liability > for the accuracy or completeness of, or the presence of any virus or > disabling code in, this e-mail. If verification is sought please request a > hard copy. Any reference to the terms of executed transactions should be > treated as preliminary only and subject to formal written confirmation by > Nomura. Nomura reserves the right to retain, monitor and intercept e-mail > communications through its networks (subject to and in accordance with > applicable laws). No confidentiality or privilege is waived or lost by > Nomura by any mistransmission of this e-mail. Any reference to "Nomura" is > a reference to any entity in the Nomura Holdings, Inc. group. Please read > our Electronic Communications Legal Notice which forms part of this e-mail: > http://www.Nomura.com/email_disclaimer.htm >
