Re: Using Travis for JDK7/8 compilation and lint-java.

Thank you, Sean! On Mon, May 23, 2016 at 2:09 PM, Sean Owen wrote: > No, because then none of the Java 8 support can build. Marcelo has a JIRA > for handling that the right way with bootstrap class path config. > > Ideally it can be rolled into Jenkins though there are

Re: Using Travis for JDK7/8 compilation and lint-java.

No, because then none of the Java 8 support can build. Marcelo has a JIRA for handling that the right way with bootstrap class path config. Ideally it can be rolled into Jenkins though there are possibly historical reasons it was not enabled before. Best to fix those if possible but if not I'd

Re: Using Travis for JDK7/8 compilation and lint-java.

For #1 below, currently Jenkins uses Java 8: JAVA_HOME=/usr/java/jdk1.8.0_60 How about switching to Java 7 ? My two cents. On Mon, May 23, 2016 at 1:24 PM, Dongjoon Hyun wrote: > Thank you for your opinion! > > Sure. I know that history and totally agree with all your

Re: Using Travis for JDK7/8 compilation and lint-java.

I want to clarify something here. For Travis CI, it's free for open source projects and there is only one management point, `.travis.xml`, for Spark community. It's not some like physical Jenkins cluster farm. It's just a cloud service like Github. PS. I'm also not an employee of Travis(or

Re: Using Travis for JDK7/8 compilation and lint-java.

Thank you, Shane! I really hope that SparkPullRequestBuilder handle them if possible. Dongjoon. On Mon, May 23, 2016 at 1:24 PM, Dongjoon Hyun wrote: > Thank you for your opinion! > > Sure. I know that history and totally agree with all your concerns. > I indeed has

Re: Using Travis for JDK7/8 compilation and lint-java.

Thank you for your opinion! Sure. I know that history and totally agree with all your concerns. I indeed has hesitated about sending this kind of suggestion for a while. If Travis CI cannot handle those simple jobs at this time again, we must turn off from Spark PR queue. We can see the result

Re: Using Travis for JDK7/8 compilation and lint-java.

chiming in, as i'm the one who currently maintains the CI infrastructure... :) +1 on not having more than one CI system... there's no way i can commit to keeping an eye on anything else other than jenkins. and i agree wholeheartedly w/michael: if it's this important, let's add it to the

Re: Using Travis for JDK7/8 compilation and lint-java.

We did turn on travis a few years ago, but ended up turning it off because it was failing (I believe because of insufficient resources) which was confusing for developers. I wouldn't be opposed to turning it on if it provides more/faster signal, but its not obvious to me that it would. In

Re: spark on kubernetes

OK created this issue https://issues.apache.org/jira/browse/SPARK-15487 please comment on this and also let me know if anyone want to collaborate on implementing it. Its my first contribution to Spark so will be exciting. - Gurvinder On 05/23/2016 07:55 PM, Gurvinder Singh wrote: > On 05/23/2016

Re: [VOTE] Removing module maintainer process

+1 non-binding (as a contributor anything which speed things up is worth a try, and git blame is a good enough substitute for the list when figuring out who to ping on a PR). On Monday, May 23, 2016, Imran Rashid wrote: > +1 (binding) > > On Mon, May 23, 2016 at 8:13 AM,

Re: spark on kubernetes

Sounds surprisingly close to this: https://github.com/apache/spark/pull/9608 I can ressurect the work on the bridge mode for Spark 2. The reason why the work on the old one was suspended was because Spark was going through so many changes at that time that a lot of work done, was wiped out by

Re: Running TPCDSQueryBenchmark results in java.lang.OutOfMemoryError

Yes, git log commit dafcb05c2ef8e09f45edfb7eabf58116c23975a0 Author: Sameer Agarwal Date: Sun May 22 23:32:39 2016 -0700 for #2 see my comments in https://issues.apache.org/jira/browse/SPARK-15078 > On 23 May 2016,

Re: spark on kubernetes

This will also simplify Mesos users as well, DCOS has to work around this with our own proxying. Tim On Sun, May 22, 2016 at 11:53 PM, Gurvinder Singh wrote: > Hi Reynold, > > So if that's OK with you, can I go ahead and create JIRA for this. As it > seems this

How to map values read from test file to 2 different RDDs

Hi I am reading a text file with 16 fields. All the place holders for the values of this text file has been defined in say 2 different case classes: Case1 and Case2 How do i map values read from text file , so my function in scala should be able to return 2 different RDDs , with each each RDD of

Re: Using Travis for JDK7/8 compilation and lint-java.

Thank you, Steve and Hyukjin. And, don't worry, Ted. Travis launches new VMs for every PR. Apache Spark repository uses the following setting. VM: Google Compute Engine OS: Ubuntu 14.04.3 LTS Server Edition 64bit CPU: ~2 CORE RAM: 7.5GB FYI, you can find more information about this here.

Re: Running TPCDSQueryBenchmark results in java.lang.OutOfMemoryError

Can you tell us the commit hash using which the test was run ? For #2, if you can give full stack trace, that would be nice. Thanks On Mon, May 23, 2016 at 8:58 AM, Ovidiu-Cristian MARCU < ovidiu-cristian.ma...@inria.fr> wrote: > Hi > > 1) Using latest spark 2.0 I've managed to run

Running TPCDSQueryBenchmark results in java.lang.OutOfMemoryError

Hi 1) Using latest spark 2.0 I've managed to run TPCDSQueryBenchmark first 9 queries and then it ends in the OutOfMemoryError [1]. What was the configuration used for running this benchmark? Can you explain the meaning of 4 shuffle partitions? Thanks! On my local system I use:

Re: [VOTE] Removing module maintainer process

+1 (binding) On Mon, May 23, 2016 at 8:13 AM, Tom Graves wrote: > +1 (binding) > > Tom > > > On Sunday, May 22, 2016 7:34 PM, Matei Zaharia > wrote: > > > It looks like the discussion thread on this has only had positive replies, > so I'm

Re: Using Travis for JDK7/8 compilation and lint-java.

Do you know if more than one PR would be verified on the same machine ? I wonder whether the 'mvn install' from two simultaneous PR builds may have conflict. On Sun, May 22, 2016 at 9:21 PM, Dongjoon Hyun wrote: > Thank you for feedback. Sure, correctly, that's the reason

Re: [VOTE] Removing module maintainer process

+1 (binding) Tom On Sunday, May 22, 2016 7:34 PM, Matei Zaharia wrote: It looks like the discussion thread on this has only had positive replies, so I'm going to call a VOTE. The proposal is to remove the maintainer process in

Re: I will fix SPARK-15477

You don't need it assigned but I do not know if this is a valid change. If in doubt try to get confirmation from someone familiar with the implications that it is correct On Mon, May 23, 2016, 04:27 马骉 wrote: > Hi > I will fix Spark 15477 > Can you please assign the

Re: Using Travis for JDK7/8 compilation and lint-java.

+1 - I wouldn't be bothered if a build becomes longer if I can write cleaner codes without manually running it. I have just looked though the related PRs and JIRAs and it looks generally okay and reasonable to me. 2016-05-23 18:54 GMT+09:00 Steve Loughran : > > On 23

Re: Using Travis for JDK7/8 compilation and lint-java.

On 23 May 2016, at 05:21, Dongjoon Hyun > wrote: Thank you for feedback. Sure, correctly, that's the reason why the current SparkPullRequestBuilder do not run `lint-java`. :-) In addition, that's the same reason why contributors are reluctant to

Re: Building spark master failed

You’re right, I tought latest will only compile against Java8. Thanks > On 23 May 2016, at 11:35, Dongjoon Hyun wrote: > > Hi, > > That is not the latest. > > The bug was fixed 5 days ago. > > Regards, > Dongjoon. > > > On Mon, May 23, 2016 at 2:16 AM,

Re: Building spark master failed

Hi, That is not the latest. The bug was fixed 5 days ago. Regards, Dongjoon. On Mon, May 23, 2016 at 2:16 AM, Ovidiu-Cristian MARCU < ovidiu-cristian.ma...@inria.fr> wrote: > Hi > > I have the following issue when trying to build the latest spark source > code on master: > >

I will fix SPARK-15477

Hi I will fix Spark 15477 Can you please assign the contributor permission to me? I've start using spark since two years ago, and familiar with spark source code and coding style. Warmest regards~ From: Biao Ma

Building spark master failed

Hi I have the following issue when trying to build the latest spark source code on master: /spark/common/network-common/src/main/java/org/apache/spark/network/util/JavaUtils.java:147: error: cannot find symbol [error] if (process != null && process.isAlive()) { [error]

Re: spark on kubernetes

Hi Reynold, So if that's OK with you, can I go ahead and create JIRA for this. As it seems this feature is missing currently and can benefit not just for kubernetes users but in general Spark standalone mode users too. - Gurvinder On 05/22/2016 12:49 PM, Gurvinder Singh wrote: > On 05/22/2016

Re: [VOTE] Removing module maintainer process

+1 (binding) On Mon, 23 May 2016 at 04:19, Matei Zaharia wrote: > Correction, let's run this for 72 hours, so until 9 PM EST May 25th. > > > On May 22, 2016, at 8:34 PM, Matei Zaharia > wrote: > > > > It looks like the discussion thread on this