Re: Jenkins upgrade/Test Parallelization & Containerization

olden.ka...@gmail.com> on behalf of >>> Holden Karau <hol...@pigscanfly.ca> >>> *Sent:* Tuesday, November 7, 2017 2:14:18 PM >>> *To:* Sean Owen >>> *Cc:* Xin Lu; dev@spark.apache.org >>> *Subject:* Re: Jenkins upgrade/Test Parallelization &a

Re: Jenkins upgrade/Test Parallelization & Containerization

>> *From:* holden.ka...@gmail.com <holden.ka...@gmail.com> on behalf of >> Holden Karau <hol...@pigscanfly.ca> >> *Sent:* Tuesday, November 7, 2017 2:14:18 PM >> *To:* Sean Owen >> *Cc:* Xin Lu; dev@spark.apache.org >> *Subject:* Re: Jenkins upgrade/Test

Re: Jenkins upgrade/Test Parallelization & Containerization

u <hol...@pigscanfly.ca> > *Sent:* Tuesday, November 7, 2017 2:14:18 PM > *To:* Sean Owen > *Cc:* Xin Lu; dev@spark.apache.org > *Subject:* Re: Jenkins upgrade/Test Parallelization & Containerization > > True, I think we've seen that the Amp Lab Jenkins needs to be more focus

Re: Jenkins upgrade/Test Parallelization & Containerization

That makes sense, in that case do we know how hard it would be to make the necessary hands to the AMP Lab Jenkins to support this? On Tue, Nov 7, 2017 at 4:04 PM Reynold Xin wrote: > My understanding is that AMP actually can provide more resources or adapt > changes, while

Re: Jenkins upgrade/Test Parallelization & Containerization

My understanding is that AMP actually can provide more resources or adapt changes, while ASF needs to manage 200+ projects and it's hard to accommodate much. I could be wrong though. On Tue, Nov 7, 2017 at 2:14 PM, Holden Karau wrote: > True, I think we've seen that the

Re: Jenkins upgrade/Test Parallelization & Containerization

True, I think we've seen that the Amp Lab Jenkins needs to be more focused on running AMP Lab projects, and while I don't know how difficult the ASF Jenkins is I assume it might be an easier place to make changes going forward? (Of course this could be the grass is greener on the other side and I

Re: Jenkins upgrade/Test Parallelization & Containerization

Faster tests would be great. I recall that the straightforward ways to parallelize via Maven haven't worked because many tests collide with one another. Is this about running each module's tests in a container? that should work. I can see how this is becoming essential for repeatable and reliable

Re: Jenkins upgrade/Test Parallelization & Containerization

Just wanting to +1 this idea. One potential option would be to look at migrating away from the AMP Lab Jenkins infra into the ASF infra. I've added Josh, Shane, and Sean to the CC line explicitly since I think they might have opinions about this. On Tue, Oct 31, 2017 at 11:05 PM, Xin Lu

Re: Jenkins upgrade/Test Parallelization & Containerization

Oh and to be clear part of my +1 is dockerized Spark builds would simplify a lot of headaches we face trying to coordinate changes on the PySpark side (it's not just oooh shiny faster build times, although that's pretty compelling in its self). On Tue, Nov 7, 2017 at 11:20 AM, Holden Karau

Jenkins upgrade/Test Parallelization & Containerization

Hi everyone, I tried sending emails to this list and I'm not sure if it went through so I'm trying again. Anyway, a couple months ago before I left Databricks I was working on a proof of concept that parallelized Spark tests on jenkins. The way it worked was basically it build the spark jars