Re: [rails-oceania] How to run CI tests in parallel

[Adam Boas]

Hi Tim,

It seems to me that AR is pretty well tested already. I'm not really all that 
keen on putting database access tests all through my test base. Accessing the 
database from a unit test actually makes it an integration test. I'm not saying 
that is wrong, but it does cause me to ask myself exactly what I am trying to 
test.

I'm not suggesting that people shouldn't use state based testing, or even that 
they shouldn't use factories or fixtures. I am suggesting that tests need as 
much (or more) refactoring love as production code. Too often bad (slow) 
patterns evolve and are not addressed. When the app is small it doesn't seem to 
matter. New developers come along and copy the poor patterns they see in the 
test suite and the problem gets bigger. Eventually parallelising the build 
seems like the only way forward.

If you're lucky, like someone on this thread, you already anticipated that your 
problem was way too important for serial testing from the get go and have a 
build rockstar on your team writing complicated build stuff for you. For the 
rest of us, we now have to hive off valuable developer resources to start 
working through all the problems involved with parallel building.

I'm not saying people shouldn't go parallel. I'm just saying that for many of 
us, fixing our crappy tests is a much better return on investment. And that is 
really my point; tests should be seen as a Return On investment problem. I feel 
certain that all developers are aware that there is no such thing as total 
certainty with tests. You can only approximate and the principal of diminishing 
returns will apply. You can put your resources into solving the parallelising 
problem or you can put them into making your test suite hum, you may eventually 
need to do both.


Adam Boas
e: adam.b...@gmail.com
m: +61 457 741 117



On 27/11/2012, at 7:40 AM, Tim Uckun  wrote:

> On Mon, Nov 26, 2012 at 12:18 PM, Adam Boas  wrote:
>> Lots of Factory based (database dependent) unit-level specs
> 
> Most rails apps exists solely to shuttle data between the database and
> the browser.  If you are mocking out these interactions you are not
> really testing the most important part of your application. This is
> especially true if you have split up your app into multiple classes
> all of which are interacting with each other and the database.
> 
> It's expensive sure but it has to be done.
> 
> BTW once I tried putting the database in a RAM disk for testing
> purposes, it didn't really help that much so the problem seems to be
> AR rather than the database itself.
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Ruby or Rails Oceania" group.
> To post to this group, send email to rails-oceania@googlegroups.com.
> To unsubscribe from this group, send email to 
> rails-oceania+unsubscr...@googlegroups.com.
> For more options, visit this group at 
> http://groups.google.com/group/rails-oceania?hl=en.
> 

-- 
You received this message because you are subscribed to the Google Groups "Ruby 
or Rails Oceania" group.
To post to this group, send email to rails-oceania@googlegroups.com.
To unsubscribe from this group, send email to 
rails-oceania+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/rails-oceania?hl=en.

Re: [rails-oceania] How to run CI tests in parallel

[Tim Uckun]

On Mon, Nov 26, 2012 at 12:18 PM, Adam Boas  wrote:
> Lots of Factory based (database dependent) unit-level specs

Most rails apps exists solely to shuttle data between the database and
the browser.  If you are mocking out these interactions you are not
really testing the most important part of your application. This is
especially true if you have split up your app into multiple classes
all of which are interacting with each other and the database.

It's expensive sure but it has to be done.

BTW once I tried putting the database in a RAM disk for testing
purposes, it didn't really help that much so the problem seems to be
AR rather than the database itself.

-- 
You received this message because you are subscribed to the Google Groups "Ruby 
or Rails Oceania" group.
To post to this group, send email to rails-oceania@googlegroups.com.
To unsubscribe from this group, send email to 
rails-oceania+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/rails-oceania?hl=en.

Re: [rails-oceania] How to run CI tests in parallel

[Ian Tinsley]

I like what you're saying Adam, you pretty much describe our environment 
and our solution.

We are working to move more features into unit tests, increase use of stubs 
within features and to change the architecture to reduce close-coupling 
between components to get our test time down and reduce our crippling 
dependencies.

In the meantime we have hit it with pragmatic and unsophisticated brute 
force - we have broken our builds up into many smaller builds (broken up by 
application or context) and build using a feature branch on a build server 
which kicks off all necessary builds in parallel on different servers. Each 
server is completely independent and deterministic, creating the database 
from scratch and running bundle install before every build. We then work on 
any broken builds in isolation locally and re-submit when ready. It's not 
cheap (we have 15 VM's) but it's very effective and devs that were once 
browsing the web while 'raking' can now be productive while waiting for 
relatively fast feedback.

I still think that if you need to maintain the ability to refactor 
aggressively and confidently you need excellent coverage at the feature or 
integration level. Unit tests with mocked or stubbed dependencies will not 
really help you find regressions when you are refactoring across many 
classes and relying on manual testing to find regressions is not an option 
if you want to run continuous delivery so i think build time is always 
likely to be an issue and simple parallelisation solutions will always be 
necessary for big projects.



On Monday, November 26, 2012 10:28:38 AM UTC+11, Adam wrote:
>
> I know this opinion is probably going to be unpopular, but I just want to 
> put out it there: 
>
> If you need to parallellize your build, then your problems getting 
> parallel building working are certainly not your biggest problem. 
>
> Even if you get a parallel build working, and it is certainly doable. 
> Where is that going to go? 
>
> Step 1 people will stop doing a pre-commit build - The fact that you need 
> a parallel build means that it is almost certainly prohibitively time 
> consuming for a dev to do one. 
>
> Step 2, people start caring less about build efficiency. Why care, you 
> can't run it locally anyway. The net effect of this is that your parallel 
> build will eventually start taking as long or longer than your old serial 
> one.
>
> Step 3 big refactorings become super scary, prohibitively time consuming 
> or simply not doable. This is because you can't quickly re-run your specs 
> anymore. A core refactor to be tested will have to go back and forth to the 
> build server to be verified (probably many times). That means you are going 
> to need another set of parallel builders focussing on another branch so 
> that you don't go screwing up the deployable master.
>
> Step 4, the super smart guy/gal who built your parallel build system 
> leaves the team and now you are in a world of pain :-) But seriously, a lot 
> of effort will go into understanding, maintaining and expanding your build.
>
> Every project I have worked on that required a parallel build had 3 common 
> threads:
>
>
>1. Lots of Factory based (database dependent) unit-level specs
>2. Large amounts of what should really be orthogonal, unit level specs 
>being done in acceptance tests
>3. A large application with more than one candidate application 
>extraction
>
>
> I have been privileged to work with some super smart people, and the 
> projects suffering from the problems above were no exception, so it wasn't 
> that they were poor developers or bad designers. Generally early 
> extractions don't get done when there are tight deadlines. By the time you 
> are feeling that your build is too slow and it is starting to get in the 
> way of productivity, your application is already probably too big. At this 
> point I recommend trying to identify groups of functionality that are 
> candidates for an extraction. If they can operate completely independently 
> and be deployed completely independently then that is great. Worst case 
> scenario look to extract large pieces out as mountable engines.
>
> There are certainly non-trivial issues with maintaining multiple, 
> interdependent apps, but all-in-all I think teams will maintain greater 
> flexibility and development speed if effort is put into this kind of 
> modularisation rather than ever more complicated build structures designed 
> to stop the team needing to address the design issue.
>
>
> Adam Boas
> e: adam...@gmail.com 
> m: +61 457 741 117
>
>
>  
> On 26/11/2012, at 9:37 AM, Thomas Egret > 
> wrote:
>
> I have been using parallel_tests as well and it's very awesome however 
> sometimes the tests suite just run smoothly and sometimes just fails so we 
> had to deactivate it for the moment (wonder if you use it as well on 
> Travis).
>
> Wish some people could give some hints regarding how to ensure my tests to 
> pass using paral

Re: [rails-oceania] How to run CI tests in parallel

[Andy Newport]

While I agree with the "your test suite should be fast or you're doing it 
wrong" crew to a certain extent, we've found any problem worth solving 
needs cucumber and integration tests anyway so parallelizing CI is a must 
regardless of how fast you can get your unit tests.

Like most people we've done parallel_tests but it has issues when you run 
both rspec and cucumber (or need to run across more than 1 box).

We're currently using an in house gem called Nitra (look at github rather 
than rubygems: https://github.com/newporta/nitra) to parallelize our build 
across a bunch of commodity hardware and use Jenkins to manage the queuing 
and CI. That gives us easy access to running the full suite on our unmerged 
git branches using Jenkins' copy job feature.

We've also tried Hydra but it had some fatal design flaws that kept us from 
using it in anger.

It's in full time usage running a few dozen builds a day. It's based on 
decent unix conventions (abuses fork and pipe semantics for speed/memory 
reasons) and we're just about to get the upgrade work done to get it RSpec 
2.12. compatible. Feel free to ask questions if you're interested, it's 
currently a pretty steep integration curve since we're the only ones 
currently using it and all the support stuff is buried in our ci rake tasks.

On Monday, November 26, 2012 1:01:56 PM UTC+13, Sebastian Porto wrote:
>
> Adam, I agree on most of what you are saying, you have described many of 
> our problems with create accuracy (lots of factories, many, many candidates 
> for extraction) and we are certainly guilty of not running the full test 
> locally before sending to CI. It is really easy to end up in that place. We 
> have been looking into ways to split our app, but we haven't done any of 
> this yet. We'll get there.
>
> Sebastian
>
>   Adam Boas 
>  26 November 2012 10:18 AM
> I know this opinion is probably going to be unpopular, but I just want to 
> put out it there: 
>
> If you need to parallellize your build, then your problems getting 
> parallel building working are certainly not your biggest problem. 
>
> Even if you get a parallel build working, and it is certainly doable. 
> Where is that going to go? 
>
> Step 1 people will stop doing a pre-commit build - The fact that you need 
> a parallel build means that it is almost certainly prohibitively time 
> consuming for a dev to do one. 
>
> Step 2, people start caring less about build efficiency. Why care, you 
> can't run it locally anyway. The net effect of this is that your parallel 
> build will eventually start taking as long or longer than your old serial 
> one.
>
> Step 3 big refactorings become super scary, prohibitively time consuming 
> or simply not doable. This is because you can't quickly re-run your specs 
> anymore. A core refactor to be tested will have to go back and forth to the 
> build server to be verified (probably many times). That means you are going 
> to need another set of parallel builders focussing on another branch so 
> that you don't go screwing up the deployable master.
>
> Step 4, the super smart guy/gal who built your parallel build system 
> leaves the team and now you are in a world of pain :-) But seriously, a lot 
> of effort will go into understanding, maintaining and expanding your build.
>
> Every project I have worked on that required a parallel build had 3 common 
> threads:
>
>
>1. Lots of Factory based (database dependent) unit-level specs
>2. Large amounts of what should really be orthogonal, unit level specs 
>being done in acceptance tests
>3. A large application with more than one candidate application 
>extraction
>
>
> I have been privileged to work with some super smart people, and the 
> projects suffering from the problems above were no exception, so it wasn't 
> that they were poor developers or bad designers. Generally early 
> extractions don't get done when there are tight deadlines. By the time you 
> are feeling that your build is too slow and it is starting to get in the 
> way of productivity, your application is already probably too big. At this 
> point I recommend trying to identify groups of functionality that are 
> candidates for an extraction. If they can operate completely independently 
> and be deployed completely independently then that is great. Worst case 
> scenario look to extract large pieces out as mountable engines.
>
> There are certainly non-trivial issues with maintaining multiple, 
> interdependent apps, but all-in-all I think teams will maintain greater 
> flexibility and development speed if effort is put into this kind of 
> modularisation rather than ever more complicated build structures designed 
> to stop the team needing to address the design issue.
>
>
> Adam Boas
> e: adam...@gmail.com 
> m: +61 457 741 117
>
>
>  
>
>
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Ruby or Rails Oceania" group.
> To post to this group, send email to rails-...@goog

Re: [rails-oceania] How to run CI tests in parallel

[Sebastian Porto]

Adam, I agree on most of what you are saying, you have described many of 
our problems with create accuracy (lots of factories, many, many 
candidates for extraction) and we are certainly guilty of not running 
the full test locally before sending to CI. It is really easy to end up 
in that place. We have been looking into ways to split our app, but we 
haven't done any of this yet. We'll get there.


Sebastian


Adam Boas 
26 November 2012 10:18 AM
I know this opinion is probably going to be unpopular, but I just want 
to put out it there:


If you need to parallellize your build, then your problems getting 
parallel building working are certainly not your biggest problem.


Even if you get a parallel build working, and it is certainly doable. 
Where is that going to go?


Step 1 people will stop doing a pre-commit build - The fact that you 
need a parallel build means that it is almost certainly prohibitively 
time consuming for a dev to do one.


Step 2, people start caring less about build efficiency. Why care, you 
can't run it locally anyway. The net effect of this is that your 
parallel build will eventually start taking as long or longer than 
your old serial one.


Step 3 big refactorings become super scary, prohibitively time 
consuming or simply not doable. This is because you can't quickly 
re-run your specs anymore. A core refactor to be tested will have to 
go back and forth to the build server to be verified (probably many 
times). That means you are going to need another set of parallel 
builders focussing on another branch so that you don't go screwing up 
the deployable master.


Step 4, the super smart guy/gal who built your parallel build system 
leaves the team and now you are in a world of pain :-) But seriously, 
a lot of effort will go into understanding, maintaining and expanding 
your build.


Every project I have worked on that required a parallel build had 3 
common threads:


 1. Lots of Factory based (database dependent) unit-level specs
 2. Large amounts of what should really be orthogonal, unit level
specs being done in acceptance tests
 3. A large application with more than one candidate application
extraction


I have been privileged to work with some super smart people, and the 
projects suffering from the problems above were no exception, so it 
wasn't that they were poor developers or bad designers. Generally 
early extractions don't get done when there are tight deadlines. By 
the time you are feeling that your build is too slow and it is 
starting to get in the way of productivity, your application is 
already probably too big. At this point I recommend trying to identify 
groups of functionality that are candidates for an extraction. If they 
can operate completely independently and be deployed completely 
independently then that is great. Worst case scenario look to extract 
large pieces out as mountable engines.


There are certainly non-trivial issues with maintaining multiple, 
interdependent apps, but all-in-all I think teams will maintain 
greater flexibility and development speed if effort is put into this 
kind of modularisation rather than ever more complicated build 
structures designed to stop the team needing to address the design issue.



Adam Boas
e: adam.b...@gmail.com 
m: +61 457 741 117





--
You received this message because you are subscribed to the Google 
Groups "Ruby or Rails Oceania" group.

To post to this group, send email to rails-oceania@googlegroups.com.
To unsubscribe from this group, send email to 
rails-oceania+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/rails-oceania?hl=en.

Thomas Egret 
26 November 2012 9:37 AM
I have been using parallel_tests as well and it's very awesome however 
sometimes the tests suite just run smoothly and sometimes just fails 
so we had to deactivate it for the moment (wonder if you use it as 
well on Travis).


Wish some people could give some hints regarding how to ensure my 
tests to pass using parallel_tests.




--
You received this message because you are subscribed to the Google 
Groups "Ruby or Rails Oceania" group.

To post to this group, send email to rails-oceania@googlegroups.com.
To unsubscribe from this group, send email to 
rails-oceania+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/rails-oceania?hl=en.

James Healy 
26 November 2012 9:28 AM
A previous project I worked on used Jenkins with a vagrant plugin to 
run concurrent tests in isolated VMs.


It took a bit of setting up, but once it got going it was amazing and 
dropped our build time from >20 minutes to <5 minutes.


The downside was that it became tricker to understand the "state of 
the build" at a glance. Instead of checking 1 project, you had to 
check 6 (unit tests, acceptance tests 1, acceptance tests 2,

Re: [rails-oceania] How to run CI tests in parallel

[Jack Chen]

Teamcity will display the failing tests for a build chain in the one place, 
rather than having you look in build logs in separate places. Which is great, 
cause our build logs are about 7MB of text.

With regards to parallel acceptance tests, you can either make sure each 
process has its own instances of DB/search/etc or run those in serial. We've 
added the ability for only one serial test to be running while running tests in 
parallel.

On 26/11/2012, at 10:20 AM, Ben Hoskings  wrote:

> Mm, vagrant is great for this sort of stuff.
> 
> FYI this isn't a problem any more: there's a master build that triggers the 6 
> (now 7) parallel builds, and it only passes if all the sub-builds pass.
> 
> The one bit of extra work is that to find each failure, you have to check 
> multiple build logs, but in practice it hasn't been an issue.
> 
> - Ben
> 
> 
> 
> On 26/11/2012, at 9:28 AM, James Healy  wrote:
> 
>> A previous project I worked on used Jenkins with a vagrant plugin to run 
>> concurrent tests in isolated VMs.
>> 
>> It took a bit of setting up, but once it got going it was amazing and 
>> dropped our build time from >20 minutes to <5 minutes.
>> 
>> The downside was that it became tricker to understand the "state of the 
>> build" at a glance. Instead of checking 1 project, you had to check 6 (unit 
>> tests, acceptance tests 1, acceptance tests 2, etc).
>> 
>> We also tried parallel_tests for a while and it mostly worked, but we ran 
>> into a few issues with parallel acceptance specs stepping on each others 
>> toes.
>> 
>> James
>> 
>> 
>> On 22 November 2012 13:12, Sebastian Porto  wrote:
>> Hi Guys
>> 
>> There was a discussion the other day about CI server and it was mention 
>> several times how good it is to run the tests in parallel. I am intrigued by 
>> this and will like to implement this. But I don't know where to start. 
>> 
>> What is your approach for running CI test in parallel? We are using TeamCity 
>> at the moment, were using Jenkins before. And Rspec.
>> 
>> Thanks
>> Sebastian
>> 
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Ruby or Rails Oceania" group.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msg/rails-oceania/-/HqhReVKyNx0J.
>> To post to this group, send email to rails-oceania@googlegroups.com.
>> To unsubscribe from this group, send email to 
>> rails-oceania+unsubscr...@googlegroups.com.
>> For more options, visit this group at 
>> http://groups.google.com/group/rails-oceania?hl=en.
>> 
>> 
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Ruby or Rails Oceania" group.
>> To post to this group, send email to rails-oceania@googlegroups.com.
>> To unsubscribe from this group, send email to 
>> rails-oceania+unsubscr...@googlegroups.com.
>> For more options, visit this group at 
>> http://groups.google.com/group/rails-oceania?hl=en.
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Ruby or Rails Oceania" group.
> To post to this group, send email to rails-oceania@googlegroups.com.
> To unsubscribe from this group, send email to 
> rails-oceania+unsubscr...@googlegroups.com.
> For more options, visit this group at 
> http://groups.google.com/group/rails-oceania?hl=en.
> 

-- 
You received this message because you are subscribed to the Google Groups "Ruby 
or Rails Oceania" group.
To post to this group, send email to rails-oceania@googlegroups.com.
To unsubscribe from this group, send email to 
rails-oceania+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/rails-oceania?hl=en.

Re: [rails-oceania] How to run CI tests in parallel

[Adam Boas]

I know this opinion is probably going to be unpopular, but I just want to put 
out it there: 

If you need to parallellize your build, then your problems getting parallel 
building working are certainly not your biggest problem. 

Even if you get a parallel build working, and it is certainly doable. Where is 
that going to go? 

Step 1 people will stop doing a pre-commit build - The fact that you need a 
parallel build means that it is almost certainly prohibitively time consuming 
for a dev to do one. 

Step 2, people start caring less about build efficiency. Why care, you can't 
run it locally anyway. The net effect of this is that your parallel build will 
eventually start taking as long or longer than your old serial one.

Step 3 big refactorings become super scary, prohibitively time consuming or 
simply not doable. This is because you can't quickly re-run your specs anymore. 
A core refactor to be tested will have to go back and forth to the build server 
to be verified (probably many times). That means you are going to need another 
set of parallel builders focussing on another branch so that you don't go 
screwing up the deployable master.

Step 4, the super smart guy/gal who built your parallel build system leaves the 
team and now you are in a world of pain :-) But seriously, a lot of effort will 
go into understanding, maintaining and expanding your build.

Every project I have worked on that required a parallel build had 3 common 
threads:

Lots of Factory based (database dependent) unit-level specs
Large amounts of what should really be orthogonal, unit level specs being done 
in acceptance tests
A large application with more than one candidate application extraction

I have been privileged to work with some super smart people, and the projects 
suffering from the problems above were no exception, so it wasn't that they 
were poor developers or bad designers. Generally early extractions don't get 
done when there are tight deadlines. By the time you are feeling that your 
build is too slow and it is starting to get in the way of productivity, your 
application is already probably too big. At this point I recommend trying to 
identify groups of functionality that are candidates for an extraction. If they 
can operate completely independently and be deployed completely independently 
then that is great. Worst case scenario look to extract large pieces out as 
mountable engines.

There are certainly non-trivial issues with maintaining multiple, 
interdependent apps, but all-in-all I think teams will maintain greater 
flexibility and development speed if effort is put into this kind of 
modularisation rather than ever more complicated build structures designed to 
stop the team needing to address the design issue.


Adam Boas
e: adam.b...@gmail.com
m: +61 457 741 117



On 26/11/2012, at 9:37 AM, Thomas Egret  wrote:

> I have been using parallel_tests as well and it's very awesome however 
> sometimes the tests suite just run smoothly and sometimes just fails so we 
> had to deactivate it for the moment (wonder if you use it as well on Travis).
> 
> Wish some people could give some hints regarding how to ensure my tests to 
> pass using parallel_tests.
> 
> 
> 2012/11/26 James Healy 
> A previous project I worked on used Jenkins with a vagrant plugin to run 
> concurrent tests in isolated VMs.
> 
> It took a bit of setting up, but once it got going it was amazing and dropped 
> our build time from >20 minutes to <5 minutes.
> 
> The downside was that it became tricker to understand the "state of the 
> build" at a glance. Instead of checking 1 project, you had to check 6 (unit 
> tests, acceptance tests 1, acceptance tests 2, etc).
> 
> We also tried parallel_tests for a while and it mostly worked, but we ran 
> into a few issues with parallel acceptance specs stepping on each others toes.
> 
> James
> 
> 
> 
> On 22 November 2012 13:12, Sebastian Porto  wrote:
> Hi Guys
> 
> There was a discussion the other day about CI server and it was mention 
> several times how good it is to run the tests in parallel. I am intrigued by 
> this and will like to implement this. But I don't know where to start. 
> 
> What is your approach for running CI test in parallel? We are using TeamCity 
> at the moment, were using Jenkins before. And Rspec.
> 
> Thanks
> Sebastian
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Ruby or Rails Oceania" group.
> To view this discussion on the web visit 
> https://groups.google.com/d/msg/rails-oceania/-/HqhReVKyNx0J.
> To post to this group, send email to rails-oceania@googlegroups.com.
> To unsubscribe from this group, send email to 
> rails-oceania+unsubscr...@googlegroups.com.
> For more options, visit this group at 
> http://groups.google.com/group/rails-oceania?hl=en.
> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Ruby or Rails Oceania" group.
> To post to this group, send email to ra

Re: [rails-oceania] How to run CI tests in parallel

[Ben Hoskings]

Mm, vagrant is great for this sort of stuff.

FYI this isn't a problem any more: there's a master build that triggers the 6 
(now 7) parallel builds, and it only passes if all the sub-builds pass.

The one bit of extra work is that to find each failure, you have to check 
multiple build logs, but in practice it hasn't been an issue.

- Ben



On 26/11/2012, at 9:28 AM, James Healy  wrote:

> A previous project I worked on used Jenkins with a vagrant plugin to run 
> concurrent tests in isolated VMs.
> 
> It took a bit of setting up, but once it got going it was amazing and dropped 
> our build time from >20 minutes to <5 minutes.
> 
> The downside was that it became tricker to understand the "state of the 
> build" at a glance. Instead of checking 1 project, you had to check 6 (unit 
> tests, acceptance tests 1, acceptance tests 2, etc).
> 
> We also tried parallel_tests for a while and it mostly worked, but we ran 
> into a few issues with parallel acceptance specs stepping on each others toes.
> 
> James
> 
> 
> On 22 November 2012 13:12, Sebastian Porto  wrote:
> Hi Guys
> 
> There was a discussion the other day about CI server and it was mention 
> several times how good it is to run the tests in parallel. I am intrigued by 
> this and will like to implement this. But I don't know where to start. 
> 
> What is your approach for running CI test in parallel? We are using TeamCity 
> at the moment, were using Jenkins before. And Rspec.
> 
> Thanks
> Sebastian
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Ruby or Rails Oceania" group.
> To view this discussion on the web visit 
> https://groups.google.com/d/msg/rails-oceania/-/HqhReVKyNx0J.
> To post to this group, send email to rails-oceania@googlegroups.com.
> To unsubscribe from this group, send email to 
> rails-oceania+unsubscr...@googlegroups.com.
> For more options, visit this group at 
> http://groups.google.com/group/rails-oceania?hl=en.
> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Ruby or Rails Oceania" group.
> To post to this group, send email to rails-oceania@googlegroups.com.
> To unsubscribe from this group, send email to 
> rails-oceania+unsubscr...@googlegroups.com.
> For more options, visit this group at 
> http://groups.google.com/group/rails-oceania?hl=en.

-- 
You received this message because you are subscribed to the Google Groups "Ruby 
or Rails Oceania" group.
To post to this group, send email to rails-oceania@googlegroups.com.
To unsubscribe from this group, send email to 
rails-oceania+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/rails-oceania?hl=en.

Re: [rails-oceania] How to run CI tests in parallel

[Thomas Egret]

I have been using parallel_tests as well and it's very awesome however
sometimes the tests suite just run smoothly and sometimes just fails so we
had to deactivate it for the moment (wonder if you use it as well on
Travis).

Wish some people could give some hints regarding how to ensure my tests to
pass using parallel_tests.


2012/11/26 James Healy 

> A previous project I worked on used Jenkins with a vagrant plugin to run
> concurrent tests in isolated VMs.
>
> It took a bit of setting up, but once it got going it was amazing and
> dropped our build time from >20 minutes to <5 minutes.
>
> The downside was that it became tricker to understand the "state of the
> build" at a glance. Instead of checking 1 project, you had to check 6 (unit
> tests, acceptance tests 1, acceptance tests 2, etc).
>
> We also tried parallel_tests for a while and it mostly worked, but we ran
> into a few issues with parallel acceptance specs stepping on each others
> toes.
>
> James
>
>
>
> On 22 November 2012 13:12, Sebastian Porto  wrote:
>
>> Hi Guys
>>
>> There was a discussion the other day about CI server and it was mention
>> several times how good it is to run the tests in parallel. I am intrigued
>> by this and will like to implement this. But I don't know where to start.
>>
>> What is your approach for running CI test in parallel? We are using
>> TeamCity at the moment, were using Jenkins before. And Rspec.
>>
>> Thanks
>> Sebastian
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Ruby or Rails Oceania" group.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msg/rails-oceania/-/HqhReVKyNx0J.
>> To post to this group, send email to rails-oceania@googlegroups.com.
>> To unsubscribe from this group, send email to
>> rails-oceania+unsubscr...@googlegroups.com.
>> For more options, visit this group at
>> http://groups.google.com/group/rails-oceania?hl=en.
>>
>
>  --
> You received this message because you are subscribed to the Google Groups
> "Ruby or Rails Oceania" group.
> To post to this group, send email to rails-oceania@googlegroups.com.
> To unsubscribe from this group, send email to
> rails-oceania+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/rails-oceania?hl=en.
>

-- 
You received this message because you are subscribed to the Google Groups "Ruby 
or Rails Oceania" group.
To post to this group, send email to rails-oceania@googlegroups.com.
To unsubscribe from this group, send email to 
rails-oceania+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/rails-oceania?hl=en.

Re: [rails-oceania] How to run CI tests in parallel

[James Healy]

A previous project I worked on used Jenkins with a vagrant plugin to run
concurrent tests in isolated VMs.

It took a bit of setting up, but once it got going it was amazing and
dropped our build time from >20 minutes to <5 minutes.

The downside was that it became tricker to understand the "state of the
build" at a glance. Instead of checking 1 project, you had to check 6 (unit
tests, acceptance tests 1, acceptance tests 2, etc).

We also tried parallel_tests for a while and it mostly worked, but we ran
into a few issues with parallel acceptance specs stepping on each others
toes.

James


On 22 November 2012 13:12, Sebastian Porto  wrote:

> Hi Guys
>
> There was a discussion the other day about CI server and it was mention
> several times how good it is to run the tests in parallel. I am intrigued
> by this and will like to implement this. But I don't know where to start.
>
> What is your approach for running CI test in parallel? We are using
> TeamCity at the moment, were using Jenkins before. And Rspec.
>
> Thanks
> Sebastian
>
> --
> You received this message because you are subscribed to the Google Groups
> "Ruby or Rails Oceania" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/rails-oceania/-/HqhReVKyNx0J.
> To post to this group, send email to rails-oceania@googlegroups.com.
> To unsubscribe from this group, send email to
> rails-oceania+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/rails-oceania?hl=en.
>

-- 
You received this message because you are subscribed to the Google Groups "Ruby 
or Rails Oceania" group.
To post to this group, send email to rails-oceania@googlegroups.com.
To unsubscribe from this group, send email to 
rails-oceania+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/rails-oceania?hl=en.

Re: [rails-oceania] How to run CI tests in parallel

[Iain Beeston]

Does anyone have any idea what kind of extra resources a CI server needs to
run tests in parallel? Would it be significant? (Our CI server is a bit
anaemic, so I've been putting off implementing parallel test runs)



Iain



On 25 November 2012 22:02, William Madden  wrote:

> We've been using parallel_tests
>
>
> On Thursday, 22 November 2012 13:48:22 UTC+1, Jack Chen wrote:
>
>> We use a heavily modified version of Hydra for our Teamcity setup.
>> Supports the flowId part of the TC formatter stuff, retrying flakey tests
>> and build time sorting to maximise CPU usage. May be able to clean it up
>> and release it, but I'd need to get permission.
>>
>> On 22/11/2012, at 11:12 PM, Sebastian Porto  wrote:
>>
>> Hi Guys
>>
>> There was a discussion the other day about CI server and it was mention
>> several times how good it is to run the tests in parallel. I am intrigued
>> by this and will like to implement this. But I don't know where to start.
>>
>> What is your approach for running CI test in parallel? We are using
>> TeamCity at the moment, were using Jenkins before. And Rspec.
>>
>> Thanks
>> Sebastian
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Ruby or Rails Oceania" group.
>> To view this discussion on the web visit https://groups.google.com/d/**
>> msg/rails-oceania/-/**HqhReVKyNx0J
>> .
>> To post to this group, send email to rails-...@googlegroups.com**.
>> To unsubscribe from this group, send email to rails-oceani...@**
>> googlegroups.com.
>>
>> For more options, visit this group at http://groups.google.com/**
>> group/rails-oceania?hl=en
>> .
>>
>>  --
> You received this message because you are subscribed to the Google Groups
> "Ruby or Rails Oceania" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/rails-oceania/-/Wdk--X6QvGIJ.
>
> To post to this group, send email to rails-oceania@googlegroups.com.
> To unsubscribe from this group, send email to
> rails-oceania+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/rails-oceania?hl=en.
>

-- 
You received this message because you are subscribed to the Google Groups "Ruby 
or Rails Oceania" group.
To post to this group, send email to rails-oceania@googlegroups.com.
To unsubscribe from this group, send email to 
rails-oceania+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/rails-oceania?hl=en.

Re: [rails-oceania] How to run CI tests in parallel

[William Madden]

We've been using parallel_tests

On Thursday, 22 November 2012 13:48:22 UTC+1, Jack Chen wrote:
>
> We use a heavily modified version of Hydra for our Teamcity setup. 
> Supports the flowId part of the TC formatter stuff, retrying flakey tests 
> and build time sorting to maximise CPU usage. May be able to clean it up 
> and release it, but I'd need to get permission. 
>
> On 22/11/2012, at 11:12 PM, Sebastian Porto > 
> wrote:
>
> Hi Guys
>
> There was a discussion the other day about CI server and it was mention 
> several times how good it is to run the tests in parallel. I am intrigued 
> by this and will like to implement this. But I don't know where to start. 
>
> What is your approach for running CI test in parallel? We are using 
> TeamCity at the moment, were using Jenkins before. And Rspec.
>
> Thanks
> Sebastian
>
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Ruby or Rails Oceania" group.
> To view this discussion on the web visit 
> https://groups.google.com/d/msg/rails-oceania/-/HqhReVKyNx0J.
> To post to this group, send email to rails-...@googlegroups.com
> .
> To unsubscribe from this group, send email to 
> rails-oceani...@googlegroups.com .
> For more options, visit this group at 
> http://groups.google.com/group/rails-oceania?hl=en.
>
>

-- 
You received this message because you are subscribed to the Google Groups "Ruby 
or Rails Oceania" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/rails-oceania/-/Wdk--X6QvGIJ.
To post to this group, send email to rails-oceania@googlegroups.com.
To unsubscribe from this group, send email to 
rails-oceania+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/rails-oceania?hl=en.

Re: [rails-oceania] How to run CI tests in parallel

[Jack Chen]

We use a heavily modified version of Hydra for our Teamcity setup. Supports the 
flowId part of the TC formatter stuff, retrying flakey tests and build time 
sorting to maximise CPU usage. May be able to clean it up and release it, but 
I'd need to get permission. 

On 22/11/2012, at 11:12 PM, Sebastian Porto  wrote:

> Hi Guys
> 
> There was a discussion the other day about CI server and it was mention 
> several times how good it is to run the tests in parallel. I am intrigued by 
> this and will like to implement this. But I don't know where to start. 
> 
> What is your approach for running CI test in parallel? We are using TeamCity 
> at the moment, were using Jenkins before. And Rspec.
> 
> Thanks
> Sebastian
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Ruby or Rails Oceania" group.
> To view this discussion on the web visit 
> https://groups.google.com/d/msg/rails-oceania/-/HqhReVKyNx0J.
> To post to this group, send email to rails-oceania@googlegroups.com.
> To unsubscribe from this group, send email to 
> rails-oceania+unsubscr...@googlegroups.com.
> For more options, visit this group at 
> http://groups.google.com/group/rails-oceania?hl=en.

-- 
You received this message because you are subscribed to the Google Groups "Ruby 
or Rails Oceania" group.
To post to this group, send email to rails-oceania@googlegroups.com.
To unsubscribe from this group, send email to 
rails-oceania+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/rails-oceania?hl=en.

[rails-oceania] How to run CI tests in parallel

[Sebastian Porto]

Hi Guys

There was a discussion the other day about CI server and it was mention 
several times how good it is to run the tests in parallel. I am intrigued 
by this and will like to implement this. But I don't know where to start. 

What is your approach for running CI test in parallel? We are using 
TeamCity at the moment, were using Jenkins before. And Rspec.

Thanks
Sebastian

-- 
You received this message because you are subscribed to the Google Groups "Ruby 
or Rails Oceania" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/rails-oceania/-/HqhReVKyNx0J.
To post to this group, send email to rails-oceania@googlegroups.com.
To unsubscribe from this group, send email to 
rails-oceania+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/rails-oceania?hl=en.