Re: [VOTE] SPIP: An Official Kubernetes Operator for Apache Spark

+1 (non-binding) On Wed, Nov 15, 2023 at 12:57 PM Xiao Li wrote: > +1 > > bo yang 于2023年11月15日周三 05:55写道： > >> +1 >> >> On Tue, Nov 14, 2023 at 7:18 PM huaxin gao >> wrote: >> >>> +1 >>> >>> On Tue, Nov 14, 2023 at 10:45 AM Holden Karau >>> wrote: >>> +1 On Tue, Nov 14, 2023

Re: [DISCUSSION] SPIP: An Official Kubernetes Operator for Apache Spark

+1 On Thu, Nov 9, 2023 at 7:43 PM Ryan Blue wrote: > +1 > > On Thu, Nov 9, 2023 at 4:23 PM Hussein Awala wrote: > >> +1 for creating an official Kubernetes operator for Apache Spark >> >> On Fri, Nov 10, 2023 at 12:38 AM huaxin gao >> wrote: >> >>> +1 >>> >>> On Thu, Nov 9, 2023 at 3:14 PM DB

Re: Spark Issue with Istio in Distributed Mode

Can we set this in spark conf or does it need to goto istio / envoy conf? > > > > On Sat, 3 Sept 2022 at 10:28, Ilan Filonenko wrote: > >> This might be a result of the idle_timeout that is configured in envoy. >> The default is an hour. >> >> On Sat, Se

Re: Spark Issue with Istio in Distributed Mode

This might be a result of the idle_timeout that is configured in envoy. The default is an hour. On Sat, Sep 3, 2022 at 12:17 AM Deepak Sharma wrote: > Hi All, > In 1 of our cluster , we enabled Istio where spark is running in > distributed mode. > Spark works fine when we run it with Istio in

Re: Thoughts on Spark 3 release, or a preview release

+1 for preview release On Fri, Sep 13, 2019 at 9:58 AM Thomas Graves wrote: > +1, I think having preview release would be great. > > Tom > > On Fri, Sep 13, 2019 at 4:55 AM Stavros Kontopoulos < > stavros.kontopou...@lightbend.com> wrote: > >> +1 as a contributor and as a user. Given the amount

Re: [VOTE][SPARK-25299] SPIP: Shuffle Storage API

+1 (non-binding). This API is versatile and flexible enough to handle Bloomberg's internal use-cases. The ability for us to vary implementation strategies is quite appealing. It is also worth to note the minimal changes to Spark core in order to make it work. This is a very much needed addition

Re: Spark-optimized Shuffle (SOS) any update?

Recently, the community has actively been working on this. The JIRA to follow is: https://issues.apache.org/jira/browse/SPARK-25299. A group of various companies including Bloomberg and Palantir are in the works of a WIP solution that implements a varied version of Option #5 (which is elaborated

Re: [VOTE] SPARK 2.4.0 (RC4)

+1 (non-binding) in reference to all k8s tests for 2.11 (including SparkR Tests with R version being 3.4.1) *[INFO] --- scalatest-maven-plugin:1.0:test (integration-test) @ spark-kubernetes-integration-tests_2.11 ---* *Discovery starting.* *Discovery completed in 202 milliseconds.* *Run starting.

Re: [DISCUSS][K8S][TESTS] Include Kerberos integration tests for Spark 2.4

On Erik's note, would SPARK-23257 be included in, say, 2.4.1? When would the next RC be? I would like to propose the inclusion of the Kerberos feature sooner rather than later as it would increase Spark-on-K8S adoption in production workloads while bringing greater feature parity with Yarn and

Re: Python kubernetes spark 2.4 branch

Is this in reference to: https://issues.apache.org/jira/browse/SPARK-24736 ? On Tue, Sep 25, 2018 at 12:38 PM Yinan Li wrote: > Can you give more details on how you ran your app, did you build your own > image, and which image are you using? > > On Tue, Sep 25, 2018 at 10:23 AM Garlapati,

Re: no logging in pyspark code?

A JIRA has been opened up on this exact topic: SPARK-25236 , a few days ago, after seeing another case of print(_, file=sys.stderr) in a most recent review. I agree that we should include logging for PySpark workers. On Mon, Aug 27, 2018 at 1:29

Re: [DISCUSS] SparkR support on k8s back-end for Spark 2.4

Okay, if there is a consensus in merging the SparkR feature and its respective tests separately, I will split up the current PR to allow for the feature to be merged while the e2e tests to be used for locally testing until we are ready to merge (similar to PySpark). Will give it another day to

Re: [DISCUSS] SparkR support on k8s back-end for Spark 2.4

Correct, the OS change and updates would require more testing, from what Shane has told me, and could potentially surface some issue that could delay a major release. So yes, the release manager would need to run the tests manually and after the release we would switch to a fully integrated

Re: [DISCUSS] SparkR support on k8s back-end for Spark 2.4

ore, I am wondering if there is any desire to ensure SparkR support (for k8s) is merged by 2.4 by the community, and if so, then maybe merging this without the Jenkins build seems even more appropriate. Best, Ilan Filonenko On Wed, Aug 15, 2018 at 3:33 PM, Erik Erlandson wrote: > The SparkR s

Build timeout -- continuous-integration/appveyor/pr — AppVeyor build failed

be increased. Thanks! Best, Ilan Filonenko

Re: Welcoming some new committers

Congrats to everyone! :) On Fri, Mar 2, 2018 at 7:34 PM Felix Cheung wrote: > Congrats and welcome! > > -- > *From:* Dongjoon Hyun > *Sent:* Friday, March 2, 2018 4:27:10 PM > *To:* Spark dev list > *Subject:* Re: