That will kill an entire Spark application, not a batch Job.
On Wed, Dec 5, 2018 at 3:07 PM Priya Matpadi wrote:
> if you are deploying your spark application on YARN cluster,
> 1. ssh into master node
> 2. List the currently running application and retreive the application_id
> yarn
if you are deploying your spark application on YARN cluster,
1. ssh into master node
2. List the currently running application and retreive the application_id
yarn application --list
3. Kill the application using application_id of the form
application_x_ from output of list command
Hi All,
How to track batch jobs in spark? For example, is there some id or token i
can get after I spawn a batch job and use it to track the progress or to
kill the batch job itself?
For Streaming, we have StreamingQuery.id()
Thanks!
Directory to use for "scratch" space in Spark, including map output files
and RDDs that get stored on disk.
It seems a directory on my local disk. But currently the temp directory is
on hdfs, under the path of df.write.parquet(path)
Regard,
Junfeng Chen
On Wed, Dec 5, 2018 at 6:28 PM Sandip
I was involved in a project like that and we decided to deploy the data in
https://ckan.org/. We used Spark for the data pipeline and transformation. Hih.
jg
> On Dec 4, 2018, at 21:14, Affan Syed wrote:
>
> All,
>
> We have been thinking about exposing our platform for analytics an OData
tryspark.local.dir property.
On Wed, Dec 5, 2018 at 1:42 PM JF Chen wrote:
> I have two spark apps writing data to one directory. I notice they share
> one temp directory, and the spark fist finish writing will clear the temp
> directory and the slower one may throw "No lease on *** File does
I have two spark apps writing data to one directory. I notice they share
one temp directory, and the spark fist finish writing will clear the temp
directory and the slower one may throw "No lease on *** File does not
exist" error
So how to specify the temp directory?
Thanks!
Regard,
Junfeng Chen
