Re: Should spark-ec2 get its own repo?

2015-08-01 Thread Matt Goodman
I think that is a good idea, and slated to happen. At the very least a README or some such. Is this a use case for git submodules? I am considering porting some of this to a more general spark-cloud launcher, including google/aliyun/rackspace. It shouldn't be hard at all given the current

Re: Should spark-ec2 get its own repo?

2015-08-01 Thread Josh Rosen
I don't think that using git submodules is a good idea here: - The extra `git submodule init git submodule update` step can lead to confusing problems in certain workflows. - We'd wind up with many commits that serve only to bump the submodule SHA; these commits will be hard to

Re: FrequentItems in spark-sql-execution-stat

2015-08-01 Thread Burak Yavuz
Hi Yucheng, Thanks for pointing out the issue. You are correct, in the case that the final map is completely empty after the merge, we do need to add the final element to the map, with the correct count (decrement the count with the max count that was already in the map). I'll submit a fix for

Re: [ANNOUNCE] Nightly maven and package builds for Spark

2015-08-01 Thread Bharath Ravi Kumar
Thanks for fixing it. On Sun, Aug 2, 2015 at 3:17 AM, Patrick Wendell pwend...@gmail.com wrote: Hey All, I got it up and running - it was a newly surfaced bug in the build scripts. - Patrick On Wed, Jul 29, 2015 at 6:05 AM, Bharath Ravi Kumar reachb...@gmail.com wrote: Hey Patrick,

Re: [ANNOUNCE] Nightly maven and package builds for Spark

2015-08-01 Thread Patrick Wendell
Hey All, I got it up and running - it was a newly surfaced bug in the build scripts. - Patrick On Wed, Jul 29, 2015 at 6:05 AM, Bharath Ravi Kumar reachb...@gmail.com wrote: Hey Patrick, Any update on this front please? Thanks, Bharath On Fri, Jul 24, 2015 at 8:38 PM, Patrick Wendell