Re: [DISCUSS] Build error message guideline

I've created a PR to add the error message guidelines to the Spark contributing guide. Would appreciate some eyes on it! https://github.com/apache/spark-website/pull/332 On Wed, Apr 14, 2021 at 5:34 PM Yuming Wang wrote: > +1 LGTM. > > On Thu, Apr 15, 2021 at 1:50 AM Karen wrote: > >> That

Re: [SPARK-34738] issues w/k8s+minikube and PV tests

i'm all for that... and once they're turned off, we can finish the minikube/k8s/move-to-docker project in a couple of hours max. On Thu, Apr 15, 2021 at 3:00 PM Holden Karau wrote: > What about if we just turn off the PV tests for now? > I'd be happy to help with the debugging/upgrading. > >

Re: [SPARK-34738] issues w/k8s+minikube and PV tests

What about if we just turn off the PV tests for now? I'd be happy to help with the debugging/upgrading. On Thu, Apr 15, 2021 at 2:28 AM Rob Vesse wrote: > > There’s at least one test (the persistent volumes one) that relies on some > Minikube functionality because we run integration tests for

Re: [DISCUSS] Add error IDs

We could leave space in the numbering system, but a more flexible method may be to have the severity as a field associated with the error class - the same way we would associate error ID with SQLSTATE, or with whether an error is user-facing or internal. As you noted, I don't believe there is a

Production results of push-based shuffle after rolling out to 100% of Spark workloads at LinkedIn

Hi, We previously raised the SPIP for push-based shuffle in SPARK-30602 . Thanks for the reviews from the community, a significant portion of the code has already been merged. In the meantime, we have been continuing to improve the solution

Re: [VOTE] Release Spark 2.4.8 (RC2)

Thanks all for voting. Unfortunately, we found a long-standing correctness bug SPARK-35080 and 2.4 was affected too. That is said we need to drop RC2 in favor of RC3. The fix is ready for merging at https://github.com/apache/spark/pull/32179. -- Sent from:

Re: [DISCUSS] Add error IDs

Machine readable logs are always good, especially if you can read the entire logs into an SQL query. It might be good to use some specific differentiation between hint/warn/fatal error in the numbering so that any automated analysis of the logs can identify the class of an error even if its an

Re: UserGroupInformation.doAS is working well in Spark Executors?

If are using kerberized HDFS the spark principal (or whoever is running the cluster) has to be declared as a proxy user. https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/Superusers.html Once done, you call the val ugi = UserGroupInformation.createProxyUser("joe",

Re: [SPARK-34738] issues w/k8s+minikube and PV tests

There’s at least one test (the persistent volumes one) that relies on some Minikube functionality because we run integration tests for our $dayjob Spark image builds using Docker for Desktop instead and that one test fails because it relies on some minikube specific functionality.  That test

Re: please read: current state and the future of the apache spark build system

Much thanks for your work on infra @Shane. Especially, we (I and @huangtianhua) got really much help from you when make Arm CI work. [1] > prepare jenkins worker ansible configs and stick in the spark repo https://github.com/apache/spark/pull/32178 I take a quick glance on it, it seems it