That's an interesting use case.
Can you add a last step to your tests a copy operation back to the master
node workspace? This way each slave is responsible for copying it's test
results back to the master node. In essence you are extending your parallel
jobs one step further.
Do share what best works for you in the end. The parallel groovy script
idea also sounds interesting.
Ioannis
On Thursday, October 15, 2015 at 6:30:44 AM UTC-4, Azul Inho wrote:
>
> Hey there,
>
> I have a master job which runs all the different tests of a particular
> project. Let me call it 'masterjob'
> This masterjob has a multijob parallel phase which fans-out roughly 60-80
> test jobs, these are run in parallel and spread out across different AWS
> regions.
>
> The parallelism is there in order to achieve quick feedback and making
> sure the feedback loop times don't increase as the number of tests
> increases.
> It also gives me a quick point to check in how long it took to run all the
> tests as the total time is the time the masterjob took to complete.
>
> The problem I have is after the parallel job phase, where I need to
> aggregate all the downstream artifacts from all the jobs across every AWS
> region.
> This step is done sequencially and is currently taking 20% of the total
> duration of the masterjob.
> I am looking to run the aggregation of artifacts in parallel, and mantain
> the feedback for the total duration of all the jobs mapped to the main
> masterjob as I have today.
>
> I am thinking the best approach might be to build a piece of groovy code
> that retrieves the artifacts from the current jobs in parallel and discard
> the aggregate downstream artifacts plugins.
>
> Or as an alternative pushing those artifacts to a proper repository as
> part of the job execution, and aggregate them again in the main masterjob
> but in parallel through some shell/python/groovy step.
>
> Who else has this problem?
>
>
> Anyone
>
>
--
You received this message because you are subscribed to the Google Groups
"Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to jenkinsci-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/jenkinsci-users/337834b7-ecab-4db9-aa1e-51e226e94dc3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
