Re: [DISCUSS] Support decimals with negative scale in decimal operation

2019-01-06 Thread Wenchen Fan
I think we need to do this for backward compatibility, and according to the discussion in the doc, SQL standard allows negative scale. To do this, I think the PR should also include a doc for the decimal type, like the definition of precision and scale(this one

Re: Spark Packaging Jenkins

2019-01-06 Thread Dongjoon Hyun
Thank you, Felix and Shane. +1 for packaging except signing in Jenkins. That sounds perfect for the next releases (2.2.3, 2.3.3, 3.0.0). BTW, thanks to Shane, the followings are recovered from today. 1. For all live branches (master to branch-2.2), Jenkins Packaging jobs are recovered to build

Re: Spark Packaging Jenkins

2019-01-06 Thread Felix Cheung
Awesome Shane! From: shane knapp Sent: Sunday, January 6, 2019 11:38 AM To: Felix Cheung Cc: Dongjoon Hyun; Wenchen Fan; dev Subject: Re: Spark Packaging Jenkins noted. i like the idea of building (but not signing) the release and will update the job(s) this

Re: Spark Packaging Jenkins

2019-01-06 Thread shane knapp
noted. i like the idea of building (but not signing) the release and will update the job(s) this week. On Sun, Jan 6, 2019 at 11:22 AM Felix Cheung wrote: > https://spark.apache.org/release-process.html > > Look for do-release-docker.sh script > > > -- > *From:*

Re: Spark Packaging Jenkins

2019-01-06 Thread Felix Cheung
https://spark.apache.org/release-process.html Look for do-release-docker.sh script From: Felix Cheung Sent: Sunday, January 6, 2019 11:17 AM To: Dongjoon Hyun; Wenchen Fan Cc: dev; shane knapp Subject: Re: Spark Packaging Jenkins The release process doc should

Re: Spark Packaging Jenkins

2019-01-06 Thread Felix Cheung
The release process doc should have been updated on this - as mentioned we do not use Jenkins for release signing (take this offline if further discussion is needed) The release build on Jenkins can still be useful for pre-validating the release build process (without actually signing it)

Spark Scheduler - Task and job levels - How it Works?

2019-01-06 Thread Miguel F. S. Vasconcelos
Hello, I'm new to Spark and trying to understand how exactly spark scheduler works. In the article /"Resilient Distributed Datasets: A Fault-Tolerant Abstraction for In-Memory Cluster Computing"/ in section 5.1 Job Scheduling" its said that: / "Whenever a user runs an action (e.g., count or