Re: [RESULT][VOTE] Spark 2.2.1 (RC2)

Hi, dev, the version of latest spark doc is still 2.2.0, when to publish the 2.2.1 doc ? -- Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/ - To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

Re: Spark error while trying to spark.read.json()

- dev java.lang.AbstractMethodError almost always means that you have different libraries on the classpath than at compilation time. In this case I would check to make sure you have the correct version of Scala (and only have one version of scala) on the classpath. On Tue, Dec 19, 2017 at 5:42

Spark error while trying to spark.read.json()

Hi All, Can anyone help me with below error, Exception in thread "main" java.lang.AbstractMethodError at scala.collection.TraversableLike$class.filterNot(TraversableLike.scala:278) at org.apache.spark.sql.types.StructType.filterNot(StructType.scala:98) at

Re: Timeline for Spark 2.3

Do people really need to be around for the branch cut (modulo the person cutting the branch)? 1st or 2nd doesn't really matter to me, but I am +1 kicking this off as soon as we enter the new year :) Michael On Tue, Dec 19, 2017 at 4:39 PM, Holden Karau wrote: > Sounds

Re: Timeline for Spark 2.3

Sounds reasonable, although I'd choose the 2nd perhaps just since lots of folks are off on the 1st? On Tue, Dec 19, 2017 at 4:36 PM, Sameer Agarwal wrote: > Let's aim for the 2.3 branch cut on 1st Jan and RC1 a week after that > (i.e., week of 8th Jan)? > > > On Fri, Dec

Re: Timeline for Spark 2.3

Let's aim for the 2.3 branch cut on 1st Jan and RC1 a week after that (i.e., week of 8th Jan)? On Fri, Dec 15, 2017 at 12:54 AM, Holden Karau wrote: > So personally I’d be in favour or pushing to early January, doing a > release over the holidays is a little rough with

Re: Publishing official docker images for KubernetesSchedulerBackend

I'd follow LEGAL-270, yes. The best resource on licensing is https://www.apache.org/legal/resolved.html ; it doesn't all have to be AL2, but needs to be compatible (sometimes with additional conditions). Auditing is basically entrusted to the PMC when voting on releases. I'll look at it with you.

Fwd: Publishing official docker images for KubernetesSchedulerBackend

Here are some specific questions I'd recommend for the Apache Spark PMC to bring to ASF legal counsel: 1) Does the philosophy described on LEGAL-270 still represent a sanctioned approach to publishing releases via container image? 2) If the transitive closure of pulled-in licenses on each of

Re: Publishing official docker images for KubernetesSchedulerBackend

Reasoning by analogy to other Apache projects is generally not sufficient when it come to securing legally permissible form or behavior -- that another project is doing something is not a guarantee that they are doing it right. If we have issues or legal questions, we need to formulate them and

Re: Publishing official docker images for KubernetesSchedulerBackend

Unfortunately you'll need to chase down the license of all the bits that are distributed directly by the project. This was a big job back in the day for the Maven artifacts and some work to maintain. Most of the work is one-time, at least. On Tue, Dec 19, 2017 at 12:53 PM Erik Erlandson

Re: Publishing official docker images for KubernetesSchedulerBackend

Agreed that the GPL family would be "toxic." The current images have been at least informally confirmed to use licenses that are ASF compatible. Is there an officially sanctioned method of license auditing that can be applied here? On Tue, Dec 19, 2017 at 11:45 AM, Sean Owen

Re: Publishing official docker images for KubernetesSchedulerBackend

I think that's all correct, though the license of third party dependencies is actually a difficult and sticky part. The ASF couldn't make a software release including any GPL software for example, and it's not just a matter of adding a disclaimer. Any actual bits distributed by the PMC would have

Re: Publishing official docker images for KubernetesSchedulerBackend

I've been looking a bit more into ASF legal posture on licensing and container images. What I have found indicates that ASF considers container images to be just another variety of distribution channel. As such, it is acceptable to publish official releases; for example an image such as

Re: Decimals

Hello everybody, I did some further researches and now I am sharing my findings. I am sorry, it is going to be a quite long e-mail, but I'd really appreciate some feedbacks when you have time to read it. Spark's current implementation of arithmetic operations on decimals was "copied" from Hive.