Re: [racket-dev] CI improved for Racket

Thu, 11 Apr 2019 06:33:18 -0700 


On 11/04/2019 11:56, Jack Firth wrote:
> So about 30-40 total cores for that second category?

That would be awesome! :)

> About how much
> total RAM is needed? 

Someone might correct me here but from what I can see it would be great
to have something like 2G/core - but if not it shouldn't be a problem.
One can always force certain jobs to certain hosts through the use of
tags so you can run 2 RAM hungry jobs or 1 RAM hungry job and 4 where
RAM doesn't matter so much.

> I'm assuming Gitlab owns all persisted data so
> there's not much need for these machines to have non-transient disk
> storage. 

No, currently I have a central cache for the gitlab machines locally.
However, I have in place an S3 bucket (Google would also be possible but
would need to check) I will sponsor to hold the cache between the machines.

> How much importance does resource density play, in terms of
> ideal cores-and-RAM-per-machine? Can you run the CI tasks effectively
> across several (>=8 nodes) small machines (<=4 cores, 4GB RAM) instead
> of a few big ones? How bursty is the CI workload, and would the machines
> spend a lot of time sitting idle?
> 

So here it depends. There are 14 jobs at the moment that cannot go to
the free gitlab machines. These are building Racket in one way or
another. I would say best is to build Racket with at least 4 cores, so
depending on the available hardware I would split the jobs the best
possible way. With a single 20 cores machines with 64G RAM I could ran 5
builds/jobs in parallel but however if I only had 16G RAM than it might
be better not to do 5 in parallel. If RAM is limited per machine, it's
better to have more machines. The machines will sit idle sometimes but
not often because some jobs just take a long time. So if Matthew pushes
something every two hours, jobs start to queue up. They might sit idle
during weekends though when people don't generally work.

If you have a specific suggestion then I could tell you what would work
and what wouldn't but in any case, I need to already say thank you for
your interest in helping the CI process.

> It might be feasible for me to donate a Kubernetes cluster to the cause
> (but I make no promises).
> 

Thanks. I don't know much about kubernetes (still at the docker level...
:)) but I saw the name thrown around in the gitlab docs so it should be
fine integrating such a cluster with CI.

Paulo

> On Thu, Apr 11, 2019 at 2:21 AM Paulo Matos <[email protected]> wrote:
> 
> 
> 
>     On 11/04/2019 11:01, [email protected]
>     <mailto:[email protected]> wrote:
>     > On Wednesday, April 10, 2019 at 12:15:02 AM UTC-7, Paulo Matos wrote:
>     >
>     >     Currently I don't have enough machines or AWS time to dedicate to
>     >     Racket builds
>     >
>     >
>     > How much do you need?
>     >
> 
>     There are really two types of needs here.
>      - Different architecture machines: ppc64, mips32, mips64, arm
>      - x86_64 with several cores - speaking in terms of cores, having 16
>     more cores would be great. Also the machines wouldn't need to be on all
>     the time if not possible. Lets say you have a spare machines, you could
>     start the gitlab-runner only during certain periods and gitlab would
>     schedule the jobs appropriately.
> 
>     With regards to the number of cores, the more, the merrier. Running qemu
>     to emulate the architectures above is slow and a lot of the jobs have to
>     wait because I only have 2 machines (totalling 20 cores) assigned to
>     this and there are lots of jobs that need to run.
> 
>     Regards,
> 
>     -- 
>     Paulo Matos
> 
> -- 
> You received this message because you are subscribed to the Google
> Groups "Racket Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to [email protected]
> <mailto:[email protected]>.
> To post to this group, send email to [email protected]
> <mailto:[email protected]>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/racket-dev/CAAXAoJWJJJb39Ec1MN78MzKL0SDgJ7TEAj9QkV%3DXVRxodfP6Uw%40mail.gmail.com
> <https://groups.google.com/d/msgid/racket-dev/CAAXAoJWJJJb39Ec1MN78MzKL0SDgJ7TEAj9QkV%3DXVRxodfP6Uw%40mail.gmail.com?utm_medium=email&utm_source=footer>.
> For more options, visit https://groups.google.com/d/optout.

-- 
Paulo Matos

-- 
You received this message because you are subscribed to the Google Groups 
"Racket Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to [email protected].
To post to this group, send email to [email protected].
To view this discussion on the web visit 
https://groups.google.com/d/msgid/racket-dev/7a136b88-a9e8-f30f-9f04-6306efc489be%40linki.tools.
For more options, visit https://groups.google.com/d/optout.






















					Reply via email to