Re: [DISCUSS] Deprecate Python < 3.6 in Spark 3.0

+1 to testing the absolute minimum number of python variants as possible. ;) On Mon, Oct 28, 2019 at 7:46 PM Hyukjin Kwon wrote: > +1 from me as well. > > 2019년 10월 29일 (화) 오전 5:34, Xiangrui Meng 님이 작성: > >> +1. And we should start testing 3.7 and maybe 3.8 in Jenkins. >> >> On Thu, Oct 24,

Re: [DISCUSS] Deprecate Python < 3.6 in Spark 3.0

+1 from me as well. 2019년 10월 29일 (화) 오전 5:34, Xiangrui Meng 님이 작성: > +1. And we should start testing 3.7 and maybe 3.8 in Jenkins. > > On Thu, Oct 24, 2019 at 9:34 AM Dongjoon Hyun > wrote: > >> Thank you for starting the thread. >> >> In addition to that, we currently are testing Python 3.6

Re: [DISCUSS] Deprecate Python < 3.6 in Spark 3.0

+1. And we should start testing 3.7 and maybe 3.8 in Jenkins. On Thu, Oct 24, 2019 at 9:34 AM Dongjoon Hyun wrote: > Thank you for starting the thread. > > In addition to that, we currently are testing Python 3.6 only in Apache > Spark Jenkins environment. > > Given that Python 3.8 is already

Re: Use Hadoop-3.2 as a default Hadoop profile in 3.0.0?

Thank you for the feedback, Sean and Xiao. Bests, Dongjoon. On Mon, Oct 28, 2019 at 12:52 PM Xiao Li wrote: > The stability and quality of Hadoop 3.2 profile are unknown. The changes > are massive, including Hive execution and a new version of Hive > thriftserver. > > To reduce the risk, I

Re: Use Hadoop-3.2 as a default Hadoop profile in 3.0.0?

The stability and quality of Hadoop 3.2 profile are unknown. The changes are massive, including Hive execution and a new version of Hive thriftserver. To reduce the risk, I would like to keep the current default version unchanged. When it becomes stable, we can change the default profile to

Re: Use Hadoop-3.2 as a default Hadoop profile in 3.0.0?

I'm OK with that, but don't have a strong opinion nor info about the implications. That said my guess is we're close to the point where we don't need to support Hadoop 2.x anyway, so, yeah. On Mon, Oct 28, 2019 at 2:33 PM Dongjoon Hyun wrote: > > Hi, All. > > There was a discussion on publishing

Use Hadoop-3.2 as a default Hadoop profile in 3.0.0?

Hi, All. There was a discussion on publishing artifacts built with Hadoop 3 . But, we are still publishing with Hadoop 2.7.3 and `3.0-preview` will be the same because we didn't change anything yet. Technically, we need to change two places for publishing. 1. Jenkins Snapshot Publishing

Re: [build system] intermittent network issues + potential power shutoff over the weekend

we're back up and building! On Mon, Oct 28, 2019 at 8:35 AM Shane Knapp wrote: > ok, it looks like the colo will have power until monday morning, and >> it will be shut down from 8am to noon to perform some maintenance. >> >> this means jenkins will be up all weekend, but down monday morning.

Re: [build system] intermittent network issues + potential power shutoff over the weekend

Thank you for fixing the worker ENVs, Shane. Bests, Dongjoon. On Mon, Oct 28, 2019 at 10:47 AM Shane Knapp wrote: > i will need to restart jenkins -- the worker's ENV vars got borked when > they came back up. > > this is happening NOW. > > shane > > On Mon, Oct 28, 2019 at 10:37 AM Shane Knapp

Re: [build system] intermittent network issues + potential power shutoff over the weekend

i will need to restart jenkins -- the worker's ENV vars got borked when they came back up. this is happening NOW. shane On Mon, Oct 28, 2019 at 10:37 AM Shane Knapp wrote: > we're back up and building! > > On Mon, Oct 28, 2019 at 8:35 AM Shane Knapp wrote: > >> ok, it looks like the colo

Re: [build system] intermittent network issues + potential power shutoff over the weekend

> > ok, it looks like the colo will have power until monday morning, and > it will be shut down from 8am to noon to perform some maintenance. > > this means jenkins will be up all weekend, but down monday morning. > > jenkins is currently down due to colo maintenance. expect it to return in ~3.5

Re: Spark 3.0 and S3A

There will be a "Hadoop 3.x" version of 3.0, as it's essential to get a JDK 11-compatible build. you can see the hadoop-3.2 profile. hadoop-aws is pulled in in the hadoop-cloud module I believe, so bears checking whether the profile updates the versions there too. On Mon, Oct 28, 2019 at 10:34 AM

Spark 3.0 and S3A

Howdy folks, I have a question about what is happening with the 3.0 release in relation to Hadoop and hadoop-aws . Today, among other builds, we release a build of Spark built against Hadoop 2.7 and another one built