Re: [VOTE] SPARK 3.0.0-preview (RC1)

Thanks Dongjoon for reporting this issue, I checked and it seems like the release script would use the `SPARK_VERSION` instead of that specified in `version.py`, thus it's not able to generate a pyspark release tarball. I submitted a PR (https://github.com/apache/spark/pull/26306) to propose a

Re: [VOTE] SPARK 3.0.0-preview (RC1)

Hi, Xingbo. PySpark seems to fail to build. There is only `sha512`. SparkR_3.0.0-preview.tar.gz SparkR_3.0.0-preview.tar.gz.asc SparkR_3.0.0-preview.tar.gz.sha512 *pyspark-3.0.0.preview.tar.gz.sha512* spark-3.0.0-preview-bin-hadoop2.7.tgz spark-3.0.0-preview-bin-hadoop2.7.tgz.asc

Re: [VOTE] SPARK 3.0.0-preview (RC1)

Thanks for the correction, we shall remove the statement > > Everything else please retarget to an appropriate release. > Reynold Xin 于2019年10月29日周二 下午7:09写道： > Does the description make sense? This is a preview release so there is no > need to retarget versions. > > On Tue, Oct 29, 2019 at

Re: [VOTE] SPARK 3.0.0-preview (RC1)

Does the description make sense? This is a preview release so there is no need to retarget versions. On Tue, Oct 29, 2019 at 7:01 PM Xingbo Jiang wrote: > Please vote on releasing the following candidate as Apache Spark version > 3.0.0-preview. > > The vote is open until November 2 PST and

[VOTE] SPARK 3.0.0-preview (RC1)

Please vote on releasing the following candidate as Apache Spark version 3.0.0-preview. The vote is open until November 2 PST and passes if a majority +1 PMC votes are cast, with a minimum of 3 +1 votes. [ ] +1 Release this package as Apache Spark 3.0.0-preview [ ] -1 Do not release this package

[DISCUSS] writing structured streaming dataframe to custom S3 buckets?

Hello, I have a local S3 service that is writable and readable using AWS sdk APIs. I created the spark session and then set the hadoop configurations as follows - // Create Spark Session val spark = SparkSession .builder() .master("local[*]") .appName("S3Loaders")

Re: [DISCUSS] Deprecate Python < 3.6 in Spark 3.0

+1, too. On Tue, Oct 29, 2019 at 4:16 PM Holden Karau wrote: > +1 to deprecating but not yet removing support for 3.6 > > On Tue, Oct 29, 2019 at 3:47 AM Shane Knapp wrote: > >> +1 to testing the absolute minimum number of python variants as >> possible. ;) >> >> On Mon, Oct 28, 2019 at 7:46

Re: Packages to release in 3.0.0-preview

Thank you Dongjoon. Please check out the latest code from test-spark-jdk11 . It works with JDK 1.8. One workaround is to install the Spark packages to local maven repository using hadoop-3.2 profile and JDK 1.8. On Mon, Oct 28, 2019 at 5:03 AM

Re: [DISCUSS] Deprecate Python < 3.6 in Spark 3.0

+1 to deprecating but not yet removing support for 3.6 On Tue, Oct 29, 2019 at 3:47 AM Shane Knapp wrote: > +1 to testing the absolute minimum number of python variants as possible. > ;) > > On Mon, Oct 28, 2019 at 7:46 PM Hyukjin Kwon wrote: > >> +1 from me as well. >> >> 2019년 10월 29일 (화) 오전