Re: Spark on k8s - map persistentStorage for data spilling

2019-03-01 Thread Tomasz Krol
wonder if your organization can consider modifying your Kubernetes > setup to make your emptyDir volumes larger and faster? > > > > -Matt Cheah > > > > *From: *Tomasz Krol > *Date: *Friday, March 1, 2019 at 10:53 AM > *To: *Matt Cheah > *Cc: *"user

Re: Spark on k8s - map persistentStorage for data spilling

2019-03-01 Thread Matt Cheah
and faster? -Matt Cheah From: Tomasz Krol Date: Friday, March 1, 2019 at 10:53 AM To: Matt Cheah Cc: "user@spark.apache.org" Subject: Re: Spark on k8s - map persistentStorage for data spilling Hi Matt, Thanks for coming back to me. Yeah that doesn't work. Basically in the pr

Re: Spark on k8s - map persistentStorage for data spilling

2019-03-01 Thread Tomasz Krol
let us know if that moves > the spills as expected? > > > > -Matt Cheah > > > > *From: *Tomasz Krol > *Date: *Wednesday, February 27, 2019 at 3:41 AM > *To: *"user@spark.apache.org" > *Subject: *Spark on k8s - map persistentStorage for data spilling > &

Re: Spark on k8s - map persistentStorage for data spilling

2019-02-28 Thread Matt Cheah
; Subject: Spark on k8s - map persistentStorage for data spilling Hey Guys, I hope someone will be able to help me, as I've stuck with this for a while:) Basically I am running some jobs on kubernetes as per documentation https://spark.apache.org/docs/latest/running-on-kuber

Spark on k8s - map persistentStorage for data spilling

2019-02-27 Thread Tomasz Krol
Hey Guys, I hope someone will be able to help me, as I've stuck with this for a while:) Basically I am running some jobs on kubernetes as per documentation https://spark.apache.org/docs/latest/running-on-kubernetes.html All works fine, however if I run queries on bigger data volume, then jobs