Re: DataFrameReader bottleneck in DataSource#checkAndGlobPathIfNecessary when reading S3 files

Hello, I have a ticket/PR out for this issue: https://issues.apache.org/jira/browse/SPARK-29089 https://github.com/apache/spark/pull/25899 Can somebody please take a look/anything else I can do to get this through the door? Thanks, Arwin From: Steve Loughran

Re: Is it feasible to build and run Spark on Windows?

Super. Thanks Deepak! On Mon, Dec 9, 2019 at 6:58 PM Deepak Vohra wrote: > Please install Apache Spark on Windows as discussed in Apache Spark on > Windows - DZone Open Source > > > Apache Spark on Windows - DZone Open Source > >

Re: Spark 3.0 preview release 2?

Yup, it would be great to release these more often. > On Dec 9, 2019, at 4:25 PM, Takeshi Yamamuro wrote: > > +1; Looks great if we can in terms of user's feedbacks. > > Bests, > Takeshi > > On Tue, Dec 10, 2019 at 3:14 AM Dongjoon Hyun > wrote: > Thank you,

Re: Spark 3.0 preview release 2?

+1; Looks great if we can in terms of user's feedbacks. Bests, Takeshi On Tue, Dec 10, 2019 at 3:14 AM Dongjoon Hyun wrote: > Thank you, All. > > +1 for another `3.0-preview`. > > Also, thank you Yuming for volunteering for that! > > Bests, > Dongjoon. > > > On Mon, Dec 9, 2019 at 9:39 AM Xiao

Re: Is it feasible to build and run Spark on Windows?

Thanks Deepak! Yes, I want to try it with Docker. But my AWS account ran out of free period. Is there a shared EC2 for Spark that we can use for free? Ping On Monday, December 9, 2019, Deepak Vohra wrote: > Haven't tested but the general procedure is to exclude all guava dependencies that

Re: SQL test failures in PR builder?

yeah, totally weird. i'm actually going to take this moment and clean up the build scripts for both of these jobs. there's a lot of years-old cruft that i'll delete and make things more readable. On Sun, Dec 8, 2019 at 7:50 PM Sean Owen wrote: > > Hm, so they look pretty similar except for

Re: Is it feasible to build and run Spark on Windows?

Hi Deepak, I tried it. Unfortunately, it still doesn't work. 28.1-jre isn't downloaded for somehow. I'll try something else. Thank you very much for your help! Ping On Fri, Dec 6, 2019 at 5:28 PM Deepak Vohra wrote: > As multiple guava versions are found exclude guava from all the >

Re: Release Apache Spark 2.4.5 and 2.4.6

Sure, seems fine. The release cadence slows down in a branch over time as there is probably less to fix, so Jan-Feb 2020 for 2.4.5 and something like middle or Q3 2020 for 2.4.6 is a reasonable expectation. It might plausibly be the last 2.4.x release but who knows. On Mon, Dec 9, 2019 at 12:29

Release Apache Spark 2.4.5 and 2.4.6

Hi, All. Along with the discussion on 3.0.0, I'd like to discuss about the next releases on `branch-2.4`. As we know, `branch-2.4` is our LTS branch and also there exists some questions on the release plans. More releases are important not only for the latest K8s version support, but also for

Re: Spark 3.0 preview release 2?

Thank you, All. +1 for another `3.0-preview`. Also, thank you Yuming for volunteering for that! Bests, Dongjoon. On Mon, Dec 9, 2019 at 9:39 AM Xiao Li wrote: > When entering the official release candidates, the new features have to be > disabled or even reverted [if the conf is not

Re: Spark 3.0 preview release 2?

When entering the official release candidates, the new features have to be disabled or even reverted [if the conf is not available] if the fixes are not trivial; otherwise, we might need 10+ RCs to make the final release. The new features should not block the release based on the previous

Re: Next DSv2 sync date

Actually, my conflict was cancelled so I'll send out the usual invite for Wednesday. Sorry for the noise. On Sun, Dec 8, 2019 at 3:15 PM Ryan Blue wrote: > Hi everyone, > > I have a conflict with the normal DSv2 sync time this Wednesday and I'd > like to attend to talk about the TableProvider

Re: Spark 3.0 preview release 2?

Seems fine to me of course. Honestly that wouldn't be a bad result for a release candidate, though we would probably roll another one now. How about simply moving to a release candidate? If not now then at least move to code freeze from the start of 2020. There is also some downside in pushing out