Re: [prometheus-developers] Prometheus builds: Try Earthly?

[Vlad A. Ionescu]

👍

On Fri, Mar 19, 2021 at 1:41 AM Ben Kochie  wrote:

> Sorry, but this doesn't seem like any real improvement over the tooling we
> already have. We're already CI vendor portable. We converted from Travis to
> Circle long ago, our build steps just aren't that complicated. Everything
> is in a standardized Makefile system. Adding more abstraction wrappers
> isn't an improvement over GNU make.
>
> On Fri, Mar 19, 2021 at 2:21 AM Vlad A. Ionescu  wrote:
>
>> Hi Bjoern,
>>
>> Thanks for the pointers! I don’t think Earthly needs to necessarily
>> replace promu. Or at least not entirely, and not all at the same time.
>> Conceivably, it’s possible to set up Earthly to work kinda like a
>> pass-through for the promu commands.
>>
>> I’m not familiar with the challenges of the Prometheus build, so I’ll try
>> to list a few *possible* benefits. I’m shooting in the dark here -
>> emphasis on “possible” :-)
>>
>>- In general, build scripts not captured by promu run containerized
>>when ran through Earthly. This helps with keeping everything reproducible.
>>(Earthly can also run things natively
>>,
>>if that is needed - but it’s not the default). If promu is ran in Earthly,
>>then it can optionally run containerized, eliminating any
>>environment-specific variance.
>>- Changes to the CI scripts can be tested locally, thus helping with
>>the iteration speed when changing the build in any way.
>>- Integration tests for client libraries can execute in a
>>containerized environment. This helps with putting together the right
>>language-specific dependencies, and also possibly set up matrix testing, 
>> to
>>test against multiple versions of the client’s language. Such tests can
>>execute isolated from one-another and in parallel.
>>- For setups where many languages and repos interact with each other,
>>Earthly has very powerful cross-repo import mechanisms if needed to bring
>>everything together for integration testing. For example, if another
>>project needs promu, it can be as easy as something like COPY
>>github.com/prometheus/promu+promu/promu ./. You can also control
>>versioning, tags, branches, etc. through build args.
>>- Multi-platform support allows testing against other architectures
>>(e.g. arm64). In Earthly, this works really well via QEMU under the hood.
>>- If migrating from one CI vendor to another, migrating to Earthly
>>first helps abstract away the CI. This helps reduce CI vendor lock-in, in
>>case Circle decides to do what Travis did.
>>- Compute intensive steps (if there are any) can be cached and not
>>repeated if the inputs have not changed. In Earthly, this can work with a 
>> remote
>>cache  too.
>>
>> If any of this seems interesting, I’d be happy to put together a demo PR
>> to POC a particular benefit. Also open to hearing of any specific
>> challenges the community has with the build.
>>
>> Cheers,
>> Vlad.
>>
>> On Thu, Mar 18, 2021 at 8:43 AM Bjoern Rabenstein 
>> wrote:
>>
>>> On 17.03.21 11:43, Vlad A. Ionescu wrote:
>>> >
>>> > Not sure if this is the right place for this question. Wondering if
>>> anyone
>>> > is interested in trying https://github.com/earthly/earthly for the
>>> > Prometheus build.
>>> >
>>> > Earthly could help with reproducing CI failures locally (via
>>> containers)
>>> > and for performing multiple isolated integration tests in parallel.
>>> >
>>> > It works well on top of Circle CI.
>>>
>>> The Prometheus project has its own organically grown build system, see
>>> https://github.com/prometheus/promu , and rather elaborate CircleCI
>>> setup. Not saying that's a perfect solution, but any new solution
>>> needs to meet that bar, plus justify the effort and friction of
>>> changing by some added relevant value.
>>>
>>> From that perspective, perhaps the first step should be to clarify
>>> what would improve and what would change, and then convince
>>> stakeholders that they actually want and need the improvements.
>>>
>>> --
>>> Björn Rabenstein
>>> [PGP-ID] 0x851C3DA17D748D03
>>> [email] bjo...@rabenste.in
>>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Prometheus Developers" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to prometheus-developers+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/prometheus-developers/CAO_OUmMq9h4HAWxqOCnoWQjM5Eqb753iEpL%2B%3D%3DATfmx6mdHzBQ%40mail.gmail.com
>> 
>> .
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Developers" group.
To unsubscribe from

Re: [prometheus-developers] Prometheus builds: Try Earthly?

[Ben Kochie]

Sorry, but this doesn't seem like any real improvement over the tooling we
already have. We're already CI vendor portable. We converted from Travis to
Circle long ago, our build steps just aren't that complicated. Everything
is in a standardized Makefile system. Adding more abstraction wrappers
isn't an improvement over GNU make.

On Fri, Mar 19, 2021 at 2:21 AM Vlad A. Ionescu  wrote:

> Hi Bjoern,
>
> Thanks for the pointers! I don’t think Earthly needs to necessarily
> replace promu. Or at least not entirely, and not all at the same time.
> Conceivably, it’s possible to set up Earthly to work kinda like a
> pass-through for the promu commands.
>
> I’m not familiar with the challenges of the Prometheus build, so I’ll try
> to list a few *possible* benefits. I’m shooting in the dark here -
> emphasis on “possible” :-)
>
>- In general, build scripts not captured by promu run containerized
>when ran through Earthly. This helps with keeping everything reproducible.
>(Earthly can also run things natively
>,
>if that is needed - but it’s not the default). If promu is ran in Earthly,
>then it can optionally run containerized, eliminating any
>environment-specific variance.
>- Changes to the CI scripts can be tested locally, thus helping with
>the iteration speed when changing the build in any way.
>- Integration tests for client libraries can execute in a
>containerized environment. This helps with putting together the right
>language-specific dependencies, and also possibly set up matrix testing, to
>test against multiple versions of the client’s language. Such tests can
>execute isolated from one-another and in parallel.
>- For setups where many languages and repos interact with each other,
>Earthly has very powerful cross-repo import mechanisms if needed to bring
>everything together for integration testing. For example, if another
>project needs promu, it can be as easy as something like COPY
>github.com/prometheus/promu+promu/promu ./. You can also control
>versioning, tags, branches, etc. through build args.
>- Multi-platform support allows testing against other architectures
>(e.g. arm64). In Earthly, this works really well via QEMU under the hood.
>- If migrating from one CI vendor to another, migrating to Earthly
>first helps abstract away the CI. This helps reduce CI vendor lock-in, in
>case Circle decides to do what Travis did.
>- Compute intensive steps (if there are any) can be cached and not
>repeated if the inputs have not changed. In Earthly, this can work with a 
> remote
>cache  too.
>
> If any of this seems interesting, I’d be happy to put together a demo PR
> to POC a particular benefit. Also open to hearing of any specific
> challenges the community has with the build.
>
> Cheers,
> Vlad.
>
> On Thu, Mar 18, 2021 at 8:43 AM Bjoern Rabenstein 
> wrote:
>
>> On 17.03.21 11:43, Vlad A. Ionescu wrote:
>> >
>> > Not sure if this is the right place for this question. Wondering if
>> anyone
>> > is interested in trying https://github.com/earthly/earthly for the
>> > Prometheus build.
>> >
>> > Earthly could help with reproducing CI failures locally (via containers)
>> > and for performing multiple isolated integration tests in parallel.
>> >
>> > It works well on top of Circle CI.
>>
>> The Prometheus project has its own organically grown build system, see
>> https://github.com/prometheus/promu , and rather elaborate CircleCI
>> setup. Not saying that's a perfect solution, but any new solution
>> needs to meet that bar, plus justify the effort and friction of
>> changing by some added relevant value.
>>
>> From that perspective, perhaps the first step should be to clarify
>> what would improve and what would change, and then convince
>> stakeholders that they actually want and need the improvements.
>>
>> --
>> Björn Rabenstein
>> [PGP-ID] 0x851C3DA17D748D03
>> [email] bjo...@rabenste.in
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-developers+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-developers/CAO_OUmMq9h4HAWxqOCnoWQjM5Eqb753iEpL%2B%3D%3DATfmx6mdHzBQ%40mail.gmail.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://