[gentoo-dev] Re: Flow's Manifesto and questions for nominees (was: Re: [gentoo-project] Gentoo Council Election 202306 ... Nominations Open in Just Over 24 Hours.)
On 14/07/2023 09.33, Sam James wrote:
Florian Schmaus writes:
[[PGP Signed Part:Undecided]]
Posted to gentoo-dev@ since we are now entering a technical discussion
again.
For those who did not follow gentoo-project@, the previous posts include:
https://marc.info/?l=gentoo-project=168918875000738=2
https://marc.info/?l=gentoo-project=168881103930591=2
On 12/07/2023 21.28, Alec Warner wrote:
On Wed, Jul 12, 2023 at 12:07 PM Florian Schmaus wrote:
Apologies for not replying to everyone individually.
I thank my fellow council candidates who took the time to reply to this
sensitive and obviously controversial matter. I understand that not
everyone feels comfortable taking a stance in this discussion.
I asked the other council candidates about their opinion on EGO_SUM.
Unfortunately, some replies included only a rather shallow answer. A few
focused on criticism of my actions and how I approach the issue. Which
is obviously fine. I read it all and have empathy for everyone who feels
aggravated. You may or may not share the complaints. But let us focus on
the actual matter for a moment.
Even the voices raised for a restricted reintroduction of EGO_SUM just
mention an abstract limit [1]. A concrete limit is not mentioned,
although I asked for it and provided my idea including specific limits.
Not knowing the concrete figures others have in mind makes it difficult
to find a compromise. For example, a fellow council candidate postulated
that it would be quicker for me to implement a limit-check in pkgcheck
than discuss EGO_SUM. I wish that were the case. Unfortunately it is
I think this misrepresents my point. All I said was that a bound should
be added matching what's in Portage right now.
Please in future respond to me directly if you're going to claim something
about what I've said.
[...]
EGO_SUM affects two dimensions that could be limited/restricted:
A) the process environment, which may run into the Linux kernel
environment limit on exec(3)
B) the size of the package directory, where EGO_SUM affects the size of
ebuilds and the Manifest
[...]
A), however, is a different beast. There is undeniably a
kernel-enforced limit that we could hit due to an extremely large
EGO_SUM (among other things). However, the only bug report I know that
runs into this kernel limit was with texlive (bug #719202). The low
number of recorded bugs caused by the environment limit matches with
the fact that even the ebuild with the most EGO_SUM entries that I
ever analyzed, app-containers/cri-o-1.23.1 (2022-02-16) with 2052
EGO_SUM entries, does *not* run into the environment limit.
I thought I'd gave you a list before, but maybe it was someone else.
Anyway, a non-exhaustive list (I remember maybe two more but I got bored):
* https://bugs.gentoo.org/829545 ("app-admin/vault-1.9.1 - find: The environment is
too large for exec().")
* https://bugs.gentoo.org/829684 ("app-metrics/prometheus-2.31.1 - find: The
environment is too large for exec().")
* https://bugs.gentoo.org/830187 (you're CC'd on this) ("go lang ebuild: SRC_URI too long that
it causes "Argument list too long" error")
* https://bugs.gentoo.org/831265 ("sys-cluster/minikube-1.24.0 - find: The
environment is too large for exec().")
* a0be89b772474e3336d3de699d71482aa89d2444 ("app-emulation/nerdctl: drop
0.14.0")
Thanks for providing this valuable information, Sam. I was indeed not
aware of those bugs. They all seem to be fixed before 2022-02-16, that
is the date of the ::gentoo tree I mostly analyzed (which was selected
because it was just before EGO_SUM was deprecated).
Limiting the process environment to 90% of the kernel-enforced limit, as
antarus also suggested (potentially by approximating the EGO_SUM
entries) would have probably prevented those bugs. As I previously
wrote, I would be happy to work on a pkgcheck for that, if the limit is
only about the kernel's process environment limit (A).
However this still leaves us with some that seem to also demand a limit
with regard to the package-directory size (B).
Other related bugs (as it's useful as a summary of where we are):
* https://bugs.gentoo.org/540146 ("sys-apps/portage: limit no of exported variables
in EAPI 6")
* https://bugs.gentoo.org/720180 ("sys-apps/portage: add support to delay export of
"A" variable until last moment")
* https://bugs.gentoo.org/721088 ("[Future EAPI] Don't export A")
* https://bugs.gentoo.org/833567 ("[Future EAPI] src_fetch_extra phase the runs
after src_unpack")
I am not aware of a bug (yet?) for radhermit's suggestion wrt external
helpers which is related but different to exporting fewer variables.
Improving, that is, reducing, what portage exports to child processes of
the ebuild is sensible. But it is only indirectly related to EGO_SUM and
not a strict prerequisite to re-introduce EGO_SUM.
- Flow
OpenPGP_0x8CAC2A9678548E35.asc
Description: OpenPGP public key
OpenPGP_signature
Description: OpenPGP digital signature
[gentoo-dev] Re: Flow's Manifesto and questions for nominees (was: Re: [gentoo-project] Gentoo Council Election 202306 ... Nominations Open in Just Over 24 Hours.)
Sam James writes:
> Florian Schmaus writes:
>
>> [[PGP Signed Part:Undecided]]
>> Posted to gentoo-dev@ since we are now entering a technical discussion
>> again.
>>
>> For those who did not follow gentoo-project@, the previous posts include:
>>
>> https://marc.info/?l=gentoo-project=168918875000738=2
>> https://marc.info/?l=gentoo-project=168881103930591=2
>>
>> On 12/07/2023 21.28, Alec Warner wrote:
>>> On Wed, Jul 12, 2023 at 12:07 PM Florian Schmaus wrote:
Apologies for not replying to everyone individually.
I thank my fellow council candidates who took the time to reply to this
sensitive and obviously controversial matter. I understand that not
everyone feels comfortable taking a stance in this discussion.
I asked the other council candidates about their opinion on EGO_SUM.
Unfortunately, some replies included only a rather shallow answer. A few
focused on criticism of my actions and how I approach the issue. Which
is obviously fine. I read it all and have empathy for everyone who feels
aggravated. You may or may not share the complaints. But let us focus on
the actual matter for a moment.
Even the voices raised for a restricted reintroduction of EGO_SUM just
mention an abstract limit [1]. A concrete limit is not mentioned,
although I asked for it and provided my idea including specific limits.
Not knowing the concrete figures others have in mind makes it difficult
to find a compromise. For example, a fellow council candidate postulated
that it would be quicker for me to implement a limit-check in pkgcheck
than discuss EGO_SUM. I wish that were the case. Unfortunately it is
>
> I think this misrepresents my point. All I said was that a bound should
> be added matching what's in Portage right now.
>
> Please in future respond to me directly if you're going to claim something
> about what I've said.
>
>> [...]
>> EGO_SUM affects two dimensions that could be limited/restricted:
>> A) the process environment, which may run into the Linux kernel
>>environment limit on exec(3)
>> B) the size of the package directory, where EGO_SUM affects the size of
>>ebuilds and the Manifest
>>
>> [...]
>>
>> A), however, is a different beast. There is undeniably a
>> kernel-enforced limit that we could hit due to an extremely large
>> EGO_SUM (among other things). However, the only bug report I know that
>> runs into this kernel limit was with texlive (bug #719202). The low
>> number of recorded bugs caused by the environment limit matches with
>> the fact that even the ebuild with the most EGO_SUM entries that I
>> ever analyzed, app-containers/cri-o-1.23.1 (2022-02-16) with 2052
>> EGO_SUM entries, does *not* run into the environment limit.
>>
>
> I thought I'd gave you a list before, but maybe it was someone else.
>
> Anyway, a non-exhaustive list (I remember maybe two more but I got bored):
> * https://bugs.gentoo.org/829545 ("app-admin/vault-1.9.1 - find: The
> environment is too large for exec().")
> * https://bugs.gentoo.org/829684 ("app-metrics/prometheus-2.31.1 - find: The
> environment is too large for exec().")
> * https://bugs.gentoo.org/830187 (you're CC'd on this) ("go lang ebuild:
> SRC_URI too long that it causes "Argument list too long" error")
> * https://bugs.gentoo.org/831265 ("sys-cluster/minikube-1.24.0 - find: The
> environment is too large for exec().")
> * a0be89b772474e3336d3de699d71482aa89d2444 ("app-emulation/nerdctl: drop
> 0.14.0")
>
Sorry, as I said this, I came across some more. These are the ones I was
thinking of:
* https://bugs.gentoo.org/830266 ("app-admin/filebeat-7.16.2 fails to compile:
Assertion failed: bc_ctl.arg_max >= LINE_MAX (xargs.c: main: 511)")
* https://bugs.gentoo.org/832964 ("sys-cluster/kops-1.21.0 fails to compile:
Assertion failed: bc_ctl.arg_max >= LINE_MAX (xargs.c: main: 511)")
* https://bugs.gentoo.org/833961 ("net-p2p/go-ipfs-0.11.0 - Assertion failed:
bc_ctl.arg_max >= LINE_MAX (xargs.c: main: 511)")
* https://bugs.gentoo.org/835712 ("dev-util/packer-1.7.9 fails to compile:
Assertion failed: bc_ctl.arg_max >= LINE_MAX (xargs.c: main: 511)")
> Other related bugs (as it's useful as a summary of where we are):
> * https://bugs.gentoo.org/540146 ("sys-apps/portage: limit no of exported
> variables in EAPI 6")
> * https://bugs.gentoo.org/720180 ("sys-apps/portage: add support to delay
> export of "A" variable until last moment")
> * https://bugs.gentoo.org/721088 ("[Future EAPI] Don't export A")
> * https://bugs.gentoo.org/833567 ("[Future EAPI] src_fetch_extra phase the
> runs after src_unpack")
>
> I am not aware of a bug (yet?) for radhermit's suggestion wrt external
> helpers which is related but different to exporting fewer variables.
>
> thanks,
> sam
signature.asc
Description: PGP signature
[gentoo-dev] Re: Flow's Manifesto and questions for nominees (was: Re: [gentoo-project] Gentoo Council Election 202306 ... Nominations Open in Just Over 24 Hours.)
Florian Schmaus writes:
> [[PGP Signed Part:Undecided]]
> Posted to gentoo-dev@ since we are now entering a technical discussion
> again.
>
> For those who did not follow gentoo-project@, the previous posts include:
>
> https://marc.info/?l=gentoo-project=168918875000738=2
> https://marc.info/?l=gentoo-project=168881103930591=2
>
> On 12/07/2023 21.28, Alec Warner wrote:
>> On Wed, Jul 12, 2023 at 12:07 PM Florian Schmaus wrote:
>>> Apologies for not replying to everyone individually.
>>>
>>> I thank my fellow council candidates who took the time to reply to this
>>> sensitive and obviously controversial matter. I understand that not
>>> everyone feels comfortable taking a stance in this discussion.
>>>
>>> I asked the other council candidates about their opinion on EGO_SUM.
>>> Unfortunately, some replies included only a rather shallow answer. A few
>>> focused on criticism of my actions and how I approach the issue. Which
>>> is obviously fine. I read it all and have empathy for everyone who feels
>>> aggravated. You may or may not share the complaints. But let us focus on
>>> the actual matter for a moment.
>>>
>>> Even the voices raised for a restricted reintroduction of EGO_SUM just
>>> mention an abstract limit [1]. A concrete limit is not mentioned,
>>> although I asked for it and provided my idea including specific limits.
>>> Not knowing the concrete figures others have in mind makes it difficult
>>> to find a compromise. For example, a fellow council candidate postulated
>>> that it would be quicker for me to implement a limit-check in pkgcheck
>>> than discuss EGO_SUM. I wish that were the case. Unfortunately it is
I think this misrepresents my point. All I said was that a bound should
be added matching what's in Portage right now.
Please in future respond to me directly if you're going to claim something
about what I've said.
> [...]
> EGO_SUM affects two dimensions that could be limited/restricted:
> A) the process environment, which may run into the Linux kernel
>environment limit on exec(3)
> B) the size of the package directory, where EGO_SUM affects the size of
>ebuilds and the Manifest
>
> [...]
>
> A), however, is a different beast. There is undeniably a
> kernel-enforced limit that we could hit due to an extremely large
> EGO_SUM (among other things). However, the only bug report I know that
> runs into this kernel limit was with texlive (bug #719202). The low
> number of recorded bugs caused by the environment limit matches with
> the fact that even the ebuild with the most EGO_SUM entries that I
> ever analyzed, app-containers/cri-o-1.23.1 (2022-02-16) with 2052
> EGO_SUM entries, does *not* run into the environment limit.
>
I thought I'd gave you a list before, but maybe it was someone else.
Anyway, a non-exhaustive list (I remember maybe two more but I got bored):
* https://bugs.gentoo.org/829545 ("app-admin/vault-1.9.1 - find: The
environment is too large for exec().")
* https://bugs.gentoo.org/829684 ("app-metrics/prometheus-2.31.1 - find: The
environment is too large for exec().")
* https://bugs.gentoo.org/830187 (you're CC'd on this) ("go lang ebuild:
SRC_URI too long that it causes "Argument list too long" error")
* https://bugs.gentoo.org/831265 ("sys-cluster/minikube-1.24.0 - find: The
environment is too large for exec().")
* a0be89b772474e3336d3de699d71482aa89d2444 ("app-emulation/nerdctl: drop
0.14.0")
Other related bugs (as it's useful as a summary of where we are):
* https://bugs.gentoo.org/540146 ("sys-apps/portage: limit no of exported
variables in EAPI 6")
* https://bugs.gentoo.org/720180 ("sys-apps/portage: add support to delay
export of "A" variable until last moment")
* https://bugs.gentoo.org/721088 ("[Future EAPI] Don't export A")
* https://bugs.gentoo.org/833567 ("[Future EAPI] src_fetch_extra phase the runs
after src_unpack")
I am not aware of a bug (yet?) for radhermit's suggestion wrt external
helpers which is related but different to exporting fewer variables.
thanks,
sam
[gentoo-dev] Re: Flow's Manifesto and questions for nominees (was: Re: [gentoo-project] Gentoo Council Election 202306 ... Nominations Open in Just Over 24 Hours.)
Posted to gentoo-dev@ since we are now entering a technical discussion again. For those who did not follow gentoo-project@, the previous posts include: https://marc.info/?l=gentoo-project=168918875000738=2 https://marc.info/?l=gentoo-project=168881103930591=2 On 12/07/2023 21.28, Alec Warner wrote: On Wed, Jul 12, 2023 at 12:07 PM Florian Schmaus wrote: Apologies for not replying to everyone individually. I thank my fellow council candidates who took the time to reply to this sensitive and obviously controversial matter. I understand that not everyone feels comfortable taking a stance in this discussion. I asked the other council candidates about their opinion on EGO_SUM. Unfortunately, some replies included only a rather shallow answer. A few focused on criticism of my actions and how I approach the issue. Which is obviously fine. I read it all and have empathy for everyone who feels aggravated. You may or may not share the complaints. But let us focus on the actual matter for a moment. Even the voices raised for a restricted reintroduction of EGO_SUM just mention an abstract limit [1]. A concrete limit is not mentioned, although I asked for it and provided my idea including specific limits. Not knowing the concrete figures others have in mind makes it difficult to find a compromise. For example, a fellow council candidate postulated that it would be quicker for me to implement a limit-check in pkgcheck than discuss EGO_SUM. I wish that were the case. Unfortunately it is potentially not trivial to implement if we want such a check to be robust. But even worse, a specific limit must be known before implementing such a check. And we currently have none. I think my concern here is that I don't expect the Council to really 'vote on a specific limit.' The limit is an implementation detail, it can change, it shouldn't require a council vote to change. So my advice is "pick something reasonable that you think holds up to scrutiny, and implement with that" and "expect the limit to change, either because of the scrutiny, or because it might change in the future" and implement your check accordingly (so e.g. the limit is easily changeable.) Please find below why this may not be enough. But the real crux of an EGO_SUM reintroduction with a limit is the following. Either the limit is too restrictive, and most packages are affected by it and can not use EGO_SUM, which ultimately only corresponds to the current state. Or the limit only affects a fraction of the packages, so you should not bother having a limit. Again the idea is there is already a limit ( the aforementioned environment limit ) and one of the goals is to have a QA check that says your ebuild is approaching that limit so you can do something productive about it, as well as to avoid ebuilds that are not installable. So just implement that. If you need a number, I think "90% of the env limit" is defensible (but again, any reasonable number will do fine.) EGO_SUM affects two dimensions that could be limited/restricted: A) the process environment, which may run into the Linux kernel environment limit on exec(3) B) the size of the package directory, where EGO_SUM affects the size of ebuilds and the Manifest I would be happy to put in any effort required to implement A) in pkgcheck, as I did for portage, if this check is the only thing that keeps us from reintroducing EGO_SUM. Unfortunately, some argue that we need to limit B). Much of the effort I put into researching the EGO_SUM situation was analyzing how EGO_SUM's impact on package-directory size affects Gentoo. The result of the analysis strongly indicates that rather large package-directories can be sustained by ::gentoo in the foreseeable future. Especially since we are only talking about ~250 EGO_SUM packages currently, and a significant fraction of those packages will not have enormous package directories. And I also suggested that the policy is reconsidered at least every two years or once the number of EGO_SUM packages has doubled (whatever comes first). My investigation of the history of EGO_SUM's deprecation has not surfaced any technical issue which justified EGO_SUM's deprecation with regard to B). It appears that technical issues do not drive the desire to limit B), but by esthetic preferences, which are highly subjective. A), however, is a different beast. There is undeniably a kernel-enforced limit that we could hit due to an extremely large EGO_SUM (among other things). However, the only bug report I know that runs into this kernel limit was with texlive (bug #719202). The low number of recorded bugs caused by the environment limit matches with the fact that even the ebuild with the most EGO_SUM entries that I ever analyzed, app-containers/cri-o-1.23.1 (2022-02-16) with 2052 EGO_SUM entries, does *not* run into the environment limit. The deprecation of EGO_SUM was and is unnecessary, a security issue, and was almost wholly *not*
