Re: [ANNOUNCE] Nightly maven and package builds for Spark

2015-07-24 Thread Bharath Ravi Kumar
I noticed the last (1.5) build has a timestamp of 16th July. Have nightly builds been discontinued since then? Thanks, Bharath On Sun, May 24, 2015 at 1:11 PM, Patrick Wendell pwend...@gmail.com wrote: Hi All, This week I got around to setting up nightly builds for Spark on Jenkins. I'd

review SPARK-8730

2015-07-24 Thread Eugen Cepoi
Hey, I've opened a PR to fix ser/de issue of primitives classes in the java serializer. I already encountered this problem in different scenarios, so I am bringing it up. Would be great if someone wants to have a look at it! :) https://issues.apache.org/jira/browse/SPARK-8730

Re: non-deprecation compiler warnings are upgraded to build errors now

2015-07-24 Thread Punyashloka Biswal
Would it make sense to isolate the use of deprecated warnings to a subset of projects? That way we could turn on more stringent checks for the other ones. Punya On Thu, Jul 23, 2015 at 12:08 AM Reynold Xin r...@databricks.com wrote: Hi all, FYI, we just merged a patch that fails a build if

Re: PySpark on PyPi

2015-07-24 Thread Jeremy Freeman
Hey all, great discussion, just wanted to +1 that I see a lot of value in steps that make it easier to use PySpark as an ordinary python library. You might want to check out this (https://github.com/minrk/findspark https://github.com/minrk/findspark), started by Jupyter project devs, that

Re: non-deprecation compiler warnings are upgraded to build errors now

2015-07-24 Thread Reynold Xin
You can give it a shot, but we will have to revert it for a project as soon as a project uses a deprecated API somewhere. On Fri, Jul 24, 2015 at 7:43 AM, Punyashloka Biswal punya.bis...@gmail.com wrote: Would it make sense to isolate the use of deprecated warnings to a subset of projects?

Re: non-deprecation compiler warnings are upgraded to build errors now

2015-07-24 Thread Iulian Dragoș
On Thu, Jul 23, 2015 at 6:08 AM, Reynold Xin r...@databricks.com wrote: Hi all, FYI, we just merged a patch that fails a build if there is a scala compiler warning (if it is not deprecation warning). I’m a bit confused, since I see quite a lot of warnings in semi-legitimate code. For

Re: [ANNOUNCE] Nightly maven and package builds for Spark

2015-07-24 Thread Patrick Wendell
Hey Bharath, There was actually an incompatible change to the build process that broke several of the Jenkins builds. This should be patched up in the next day or two and nightly builds will resume. - Patrick On Fri, Jul 24, 2015 at 12:51 AM, Bharath Ravi Kumar reachb...@gmail.com wrote: I

Policy around backporting bug fixes

2015-07-24 Thread Patrick Wendell
Hi All, A few times I've been asked about backporting and when to backport and not backport fix patches. Since I have managed this for many of the past releases, I wanted to point out the way I have been thinking about it. If we have some consensus I can put it on the wiki. The trade off when

Re: non-deprecation compiler warnings are upgraded to build errors now

2015-07-24 Thread Reynold Xin
Jenkins only run Scala 2.10. I'm actually not sure what the behavior is with 2.11 for that patch. iulian - can you take a look into it and see if it is working as expected? On Fri, Jul 24, 2015 at 10:24 AM, Iulian Dragoș iulian.dra...@typesafe.com wrote: On Thu, Jul 23, 2015 at 6:08 AM,

jenkins failing on Kinesis shard limits

2015-07-24 Thread Steve Loughran
Looks like Jenkins is hitting some AWS limits https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/38396/testReport/org.apache.spark.streaming.kinesis/KinesisBackedBlockRDDSuite/_It_is_not_a_test_/

Re: review SPARK-8730

2015-07-24 Thread Sean Owen
It looks like you have a number of review comments on the PR that you have not replied to. The PR does not merge at the moment either. On Fri, Jul 24, 2015 at 12:03 PM, Eugen Cepoi cepoi.eu...@gmail.com wrote: Hey, I've opened a PR to fix ser/de issue of primitives classes in the java

Re: review SPARK-8730

2015-07-24 Thread Eugen Cepoi
I just got those comments and it doesn't merge since few time, the code evolved since I opened the pr 2015-07-24 14:12 GMT+02:00 Sean Owen so...@cloudera.com: It looks like you have a number of review comments on the PR that you have not replied to. The PR does not merge at the moment either.