Re: [gentoo-dev] Current unavoidable use of xz utils in Gentoo

[Eddie Chapman]

On 02/04/2024 20:46, Eli Schwartz wrote:

On 4/2/24 4:43 AM, Eddie Chapman wrote:

Well, they change one thing. It's hard for the security professionals at
work to deal with things when they are constantly having to respond to the
three-ring circus.


This is a complaint I hear very often from the people working at the heart
of things. Stop making noise, shut up, we're overworked here and dealing
with your "complaints" just adds to our stress. I do understand and
sympathise with those feelings, believe me I do, I feel them myself in
other contexts.

But I hope you understand this is not finding things to nitpick about for
the sake of it. Does the Gentoo dev community want people on the "outside"
to raise their concerns on their mailing list if those persons feel like
said community have got something very wrong, yes or no? If not then put a
note on the mailing list page saying "please don't bother us, we're too
overworked, just post patches" or something to that effect.



I would be delighted to hear reasonable concerns. That's not what I'm
referring to by "three-ring circus".



What does one have to do with the other? Why is it necessary to claim
that based on some sort of vibe check "there is too much compassion going
around in our communities, and this must mean that not enough effort is
being expended on the technical and cleanup aspects"?


I have not made such a claim, I've said I see lots of technical and
cleanup aspects. I've only stated the things that *are* happening versus
what is not happening at all (literally zilch) and which should be
happening, which is efforts towards a solution *not* involving the xz
utilities.



You say that as though a solution *not* involving the xz utilities is a
reasonable takeaway from this scenario.

In order to demonstrate that such efforts deserve discussion at all, let
alone efforts towards solving it, you first need to prove that:

- the xz utilities as shipped by Gentoo are something that should be
   moved away from

- the xz utilities as released in 2022, when the backdoor author had
   just as much access as you or I -- that is, none, aside for the right
   to submit patches as suggestions -- are something that should be moved
   away from

You have made no effort to justify either approach aside for claiming
that for unidentified reasons you believe this scenario demonstrates
that the "apparently innocent maintainer" now has an incentive to
"downplay the involvement of the bad actor".

If you had, I would be infinitely more interested in what you have to
say on the topic.

Also, if you had started with such.



Reading in between the lines, e.g. "trying desperately to salvage the
situation with xz-utils", I suspect you are trying to subtly suggest that
any second of time where gentoo hasn't yet removed xz-utils from gentoo as
a dead end is "cavalier".


Not quite, I've never advocated removing xz-utils at all, more than happy
for it to remain for whoever wants to use it. The only reason I started
this thread is I'm very unhappy about that fact that it is currently
impossible to NOT execute xz utilities on the Gentoo systems I'm
responsible for, without heavy customisation.

I'm also not demanding anything, let alone demanding anything instantly.
If I have please point out where.



Thank you for clarifying.

I am now just as convinced as I was yesterday, that you are trying to
subtly suggest it, but I'm newly convinced that you're also being
mendacious about it.

"It is cavalier for Gentoo to allow xz-utils as a package in the @system
set" is not meaningfully distinct from "it is cavalier for Gentoo to not
work to allow me to depclean xz-utils".



I understand that you are passionate about your suggestion to make
portage not validate distfile hashes.


That's incorrect, I've never suggested Portage should not validate
distfile hashes. The current behaviour is that validating distfile hashes
is something that can be disabled if a user wishes to, and I have no
problem with that at all, would not change a thing. I've said that, in
order to implement what I have suggested, a user would have to disable it,
which is not ideal, but acceptable if the user controls the distfile
distribution. And I only suggested that in order to try and make the idea
more acceptable by not requiring impractical infra changes that would be
needed to generate uncompressed hashes for the Manifests).



In other words, you didn't care to find a robust solution, you just want
something that you personally can use, and which requires being less
secure than using xz-utils.

But it's okay! It's not harassing portage devs with unreasonable
demands! Because it's *optional*, and by default people would just...
use xz-utils.

Even though the ***entire premise*** of changing anything here is that
xz-utils as shipped by Gentoo is somehow dangerous and users have a
valid reason to want to avoid it entirely.

If you're going to recommend a solution for users who consider xz-utils
to be 

Re: [gentoo-dev] Current unavoidable use of xz utils in Gentoo

[Eli Schwartz]

On 4/2/24 4:43 AM, Eddie Chapman wrote:
>> Well, they change one thing. It's hard for the security professionals at
>> work to deal with things when they are constantly having to respond to the
>> three-ring circus.
> 
> This is a complaint I hear very often from the people working at the heart
> of things. Stop making noise, shut up, we're overworked here and dealing
> with your "complaints" just adds to our stress. I do understand and
> sympathise with those feelings, believe me I do, I feel them myself in
> other contexts.
> 
> But I hope you understand this is not finding things to nitpick about for
> the sake of it. Does the Gentoo dev community want people on the "outside"
> to raise their concerns on their mailing list if those persons feel like
> said community have got something very wrong, yes or no? If not then put a
> note on the mailing list page saying "please don't bother us, we're too
> overworked, just post patches" or something to that effect.


I would be delighted to hear reasonable concerns. That's not what I'm
referring to by "three-ring circus".


>> What does one have to do with the other? Why is it necessary to claim
>> that based on some sort of vibe check "there is too much compassion going
>> around in our communities, and this must mean that not enough effort is
>> being expended on the technical and cleanup aspects"?
> 
> I have not made such a claim, I've said I see lots of technical and
> cleanup aspects. I've only stated the things that *are* happening versus
> what is not happening at all (literally zilch) and which should be
> happening, which is efforts towards a solution *not* involving the xz
> utilities.


You say that as though a solution *not* involving the xz utilities is a
reasonable takeaway from this scenario.

In order to demonstrate that such efforts deserve discussion at all, let
alone efforts towards solving it, you first need to prove that:

- the xz utilities as shipped by Gentoo are something that should be
  moved away from

- the xz utilities as released in 2022, when the backdoor author had
  just as much access as you or I -- that is, none, aside for the right
  to submit patches as suggestions -- are something that should be moved
  away from

You have made no effort to justify either approach aside for claiming
that for unidentified reasons you believe this scenario demonstrates
that the "apparently innocent maintainer" now has an incentive to
"downplay the involvement of the bad actor".

If you had, I would be infinitely more interested in what you have to
say on the topic.

Also, if you had started with such.


>> Reading in between the lines, e.g. "trying desperately to salvage the
>> situation with xz-utils", I suspect you are trying to subtly suggest that
>> any second of time where gentoo hasn't yet removed xz-utils from gentoo as
>> a dead end is "cavalier".
> 
> Not quite, I've never advocated removing xz-utils at all, more than happy
> for it to remain for whoever wants to use it. The only reason I started
> this thread is I'm very unhappy about that fact that it is currently
> impossible to NOT execute xz utilities on the Gentoo systems I'm
> responsible for, without heavy customisation.
> 
> I'm also not demanding anything, let alone demanding anything instantly.
> If I have please point out where.


Thank you for clarifying.

I am now just as convinced as I was yesterday, that you are trying to
subtly suggest it, but I'm newly convinced that you're also being
mendacious about it.

"It is cavalier for Gentoo to allow xz-utils as a package in the @system
set" is not meaningfully distinct from "it is cavalier for Gentoo to not
work to allow me to depclean xz-utils".


>> I understand that you are passionate about your suggestion to make
>> portage not validate distfile hashes.
> 
> That's incorrect, I've never suggested Portage should not validate
> distfile hashes. The current behaviour is that validating distfile hashes
> is something that can be disabled if a user wishes to, and I have no
> problem with that at all, would not change a thing. I've said that, in
> order to implement what I have suggested, a user would have to disable it,
> which is not ideal, but acceptable if the user controls the distfile
> distribution. And I only suggested that in order to try and make the idea
> more acceptable by not requiring impractical infra changes that would be
> needed to generate uncompressed hashes for the Manifests).


In other words, you didn't care to find a robust solution, you just want
something that you personally can use, and which requires being less
secure than using xz-utils.

But it's okay! It's not harassing portage devs with unreasonable
demands! Because it's *optional*, and by default people would just...
use xz-utils.

Even though the ***entire premise*** of changing anything here is that
xz-utils as shipped by Gentoo is somehow dangerous and users have a
valid reason to want to avoid it entirely.

If you're going to 

Re: [gentoo-dev] Current unavoidable use of xz utils in Gentoo

[Eddie Chapman]

On 01/04/2024 15:56, Azamat Hackimov wrote:

There is no problem in the XZ/LZMA format itself as the reference
algorithm is not compromised. It's all about trust between developers
of application and developers of distribution. If you lost trust to
xz-utils's developers, you may use alternatives like app-arch/pxz or
app-arch/pixz. I don't see reasons why we should change format instead
of changing a tool.



Hello Azamat,

Yes, I have no issue with the format at all, just with the xz utils 
project.  But I was suggesting to have support for two compression 
algorithms as an improvement for the future, in case of some unknown 
other major problem with one of them emerges in future. I suppose kind 
of a similar reasoning, but not quite the same, that we have BLAKE2B and 
SHA512 hashes. But there are severe practical problems for Gentoo infra 
resources with having two of course.


Thank you for the pointer to app-arch/pxz and app-arch/pixz. I've had a 
close look at them both but sadly they are not suitable as they both 
rely on the xz utils project to do the main work. Once calls the xz exe 
and the other links against liblzma.


I have been looking around a bit since Friday at what true alternatives 
(no relying on liblzma) there are to just decompress existing XZ/LZMA 
binaries. There is p7zip which is a command line fork of 7zip that's 
been around a good while. However in the years since that fork was 
created the 7zip project themselves have begun doing source code 
releases with build instructions, with the command line tool apparently 
working fine.


On balance the upstream 7zip actually looks like a better option than 
p7zip now since p7zip maintenance has stagnated somewhat. On the one 
hand 7zip is actively developed, of course because of the large Windows 
userbase, and security fixes would be available immediately when a new 
release comes about (there were sec issues fixed in 7zip last year for 
example which didn't make it into p7zip in a timely fashion).  But on 
the other hand most distros have used p7zip and I've only seen Arch and 
Debian that currently package the 7zip releases, so the latest 7zip 
releases have had only minimum real world testing and code scrutiny in 
the Linux world (although it's likely much of the code will still be the 
same as what it was when p7zip was forked, so in that sense at least a 
significant portion of the code has had wider testing, in a manner of 
speaking). Still, I'm not sure about 7zip, doesn't seem ideal.


Thomas Gall, elsewhere in this thread, pointed out a pure Rust 
implementation which is interesting.

https://github.com/gendx/lzma-rs

The GH page says it supports decompression of "LZMA, LZMA2 and a subset 
of the .xz file format".


If anyone else knows of any other true alternatives please do let me 
know. I'm currently looking into the feasibility of hacking my Gentoo 
installations so that .xz distfiles are decompressed during the ebuild 
process using an alternative implementation, allowing me to get rid of 
xz utils.


Thanks,
Eddie

Re: [gentoo-dev] [PATCH 2/2] texlive-module.eclass: add texlive-module_update_tlpdb

[Ulrich Mueller]

> On Tue, 02 Apr 2024, Florian Schmaus wrote:

> + # If we are updating this package, then there is no need to update
> + # the tlpdb in postrm, as it will be again updated in postinst.
> + [[ -n ${REPLACING_VERSIONS} && ${EBUILD_PHASE} == postrm ]] && return

Sorry for having missed this before. REPLACING_VERSIONS isn't defined
in pkg_postrm, so the test should be for REPLACED_BY_VERSION.

Also it would be cleaner to test for the phase first, to make sure that
the variable is valid in this phase:

[[ ${EBUILD_PHASE} == postrm && -n ${REPLACED_BY_VERSION} ]] && return

Ulrich


signature.asc
Description: PGP signature

[gentoo-dev] [PATCH 2/2] texlive-module.eclass: add texlive-module_update_tlpdb

[Florian Schmaus]

Update (or create) the tlpdb based on the contents of
/usr/share/tlpkg/tlpobj.

Closes: https://bugs.gentoo.org/928162
Signed-off-by: Florian Schmaus 
---
 eclass/texlive-module.eclass | 57 
 1 file changed, 57 insertions(+)

diff --git a/eclass/texlive-module.eclass b/eclass/texlive-module.eclass
index d19e02f02647..15346a3535eb 100644
--- a/eclass/texlive-module.eclass
+++ b/eclass/texlive-module.eclass
@@ -420,6 +420,61 @@ texlive-module_src_install() {
texlive-common_handle_config_files
 }
 
+# @FUNCTION: texlive-module_update_tlpdb
+# @DESCRIPTION:
+# Update the TexLive package database at /usr/share/tlpkg/texlive.tlpdb.
+
+texlive-module_update_tlpdb() {
+   [[ ${TL_PV} -lt 2023 ]] && return
+
+   # If we are updating this package, then there is no need to update
+   # the tlpdb in postrm, as it will be again updated in postinst.
+   [[ -n ${REPLACING_VERSIONS} && ${EBUILD_PHASE} == postrm ]] && return
+
+   local tlpkg="${EROOT}"/usr/share/tlpkg
+   local tlpobj="${tlpkg}"/tlpobj
+   local tlpdb="${tlpkg}"/texlive.tlpdb
+
+   ebegin "Regenerating TexLive package database (${tlpdb}, 
${EBUILD_PHASE})"
+
+   local new_tlpdb="${T}"/texlive.tlpdb
+
+   touch "${new_tlpdb}" || die
+
+   find "${tlpobj}" -maxdepth 1 -type f -name "*.tlpobj" -print0 |
+   sort -z |
+   xargs -0 --no-run-if-empty cat >> "${new_tlpdb}"
+   assert "generating tlpdb failed"
+
+   if [[ -f ${tlpdb} ]]; then
+   cmp -s "${new_tlpdb}" "${tlpdb}"
+   local ret=$?
+   case ${ret} in
+   # content equal
+   0)
+   # Nothing to do, return.
+   eend 0
+   return
+   ;;
+   # content differs
+   1)
+   ;;
+   # cmp failed with an error
+   *)
+   eend ${ret} "comparing new and existing tlpdb 
failed (exit status: ${ret})"
+   die
+   ;;
+   esac
+   fi
+
+   mv "${new_tlpdb}" "${tlpdb}"
+   eend $? "moving tlpdb into position failed (exit status: ${?})" || die
+
+   if [[ ! -s ${tlpdb} ]]; then
+   rm "${tlpdb}" || die
+   fi
+}
+
 # @FUNCTION: texlive-module_pkg_postinst
 # @DESCRIPTION:
 # exported function:
@@ -428,6 +483,7 @@ texlive-module_src_install() {
 
 texlive-module_pkg_postinst() {
etexmf-update
+   texlive-module_update_tlpdb
[[ -n ${TL_MODULE_INFORMATION} ]] && elog "${TL_MODULE_INFORMATION}"
 }
 
@@ -439,6 +495,7 @@ texlive-module_pkg_postinst() {
 
 texlive-module_pkg_postrm() {
[[ -z ${REPLACING_VERSIONS} ]] && etexmf-update
+   texlive-module_update_tlpdb
 }
 
 fi
-- 
2.43.2

[gentoo-dev] [PATCH 1/2] texlive-module.eclass: only invoke etexmf-update in postinst if not replacing versions

[Florian Schmaus]

Signed-off-by: Florian Schmaus 
---
 eclass/texlive-module.eclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/eclass/texlive-module.eclass b/eclass/texlive-module.eclass
index 9fc4e619ff9b..d19e02f02647 100644
--- a/eclass/texlive-module.eclass
+++ b/eclass/texlive-module.eclass
@@ -438,7 +438,7 @@ texlive-module_pkg_postinst() {
 # installed texmf trees.
 
 texlive-module_pkg_postrm() {
-   etexmf-update
+   [[ -z ${REPLACING_VERSIONS} ]] && etexmf-update
 }
 
 fi
-- 
2.43.2

Re: [gentoo-dev] Current unavoidable use of xz utils in Gentoo

[Eddie Chapman]

Michał Górny wrote:
> On Mon, 2024-04-01 at 08:57 +0100, Eddie Chapman wrote:
>
>> I stand by and reiterate my view that there is far too much of a
>> cavalier attitude towards the matter in general out there including here
>> in Gentoo. But not in particular here, it is everywhere where this is
>> being discussed at the moment.
>
> I would like to point out that the xz/sshd issue was primarily a social
> one, not a technical one.
>
> The primary problem in open source today isn't bad code.  It's projects
> relying on overburdened, burned out maintainers.  And on top of that, users
> who are complaining, demanding, outright hostile or primarily contributing
> by walls of text on a mailing lists, that bring nothing to discussion
> except for furthering the burnout of open source developers who are
> actually trying to do something.
>
> Think about that.
>
>
> --
> Best regards,
> Michał Górny
>

I'm sorry for having contributed to your burnout. I have a lot of respect
for you personally Michał, the quality of your contributions to Gentoo are
outstanding, and have to admit I've often felt a little worried for you
with the amount of work you do. I don't know you at all, I hope you don't
mind me saying that. Don't worry I think it's quite unlikely I'll bring
any new concerns to this list again in future, I'll certainly think twice
about it.

regards,
Eddie

Re: [gentoo-dev] Current unavoidable use of xz utils in Gentoo

[Eddie Chapman]

OK, I said I was done and this is a waste of time for everyone, but if
people want to keep the discussion going I'll bite :-)

Eli Schwartz wrote:
> But also, please keep in mind that 98% of all people on the internet can
> do whatever they want and it simply doesn't matter. They are public
> commentators at a three-ring circus and their cavalier or panicked
> attitudes change nothing.

I disagree, think it is very important to have discussions about what the
oss/linux community thinks, not just what they do. And I think those
discussions do significantly influence what is actually done, whether the
"doers" actually realise it or not.

> Well, they change one thing. It's hard for the security professionals at
> work to deal with things when they are constantly having to respond to the
> three-ring circus.

This is a complaint I hear very often from the people working at the heart
of things. Stop making noise, shut up, we're overworked here and dealing
with your "complaints" just adds to our stress. I do understand and
sympathise with those feelings, believe me I do, I feel them myself in
other contexts.

But I hope you understand this is not finding things to nitpick about for
the sake of it. Does the Gentoo dev community want people on the "outside"
to raise their concerns on their mailing list if those persons feel like
said community have got something very wrong, yes or no? If not then put a
note on the mailing list page saying "please don't bother us, we're too
overworked, just post patches" or something to that effect.

> Please stop insulting the work of the people who are working very hard
> to analyze and learn about this issue and taking steps to try to mitigate
> it...

I'm certainly not trying to insult anyone. I've expressed a lot of
appreciation for their work. I have *criticised* the prevailing view
though.

> What does one have to do with the other? Why is it necessary to claim
> that based on some sort of vibe check "there is too much compassion going
> around in our communities, and this must mean that not enough effort is
> being expended on the technical and cleanup aspects"?

I have not made such a claim, I've said I see lots of technical and
cleanup aspects. I've only stated the things that *are* happening versus
what is not happening at all (literally zilch) and which should be
happening, which is efforts towards a solution *not* involving the xz
utilities.

> Reading in between the lines, e.g. "trying desperately to salvage the
> situation with xz-utils", I suspect you are trying to subtly suggest that
> any second of time where gentoo hasn't yet removed xz-utils from gentoo as
> a dead end is "cavalier".

Not quite, I've never advocated removing xz-utils at all, more than happy
for it to remain for whoever wants to use it. The only reason I started
this thread is I'm very unhappy about that fact that it is currently
impossible to NOT execute xz utilities on the Gentoo systems I'm
responsible for, without heavy customisation.

I'm also not demanding anything, let alone demanding anything instantly.
If I have please point out where.

> I understand that you are passionate about your suggestion to make
> portage not validate distfile hashes.

That's incorrect, I've never suggested Portage should not validate
distfile hashes. The current behaviour is that validating distfile hashes
is something that can be disabled if a user wishes to, and I have no
problem with that at all, would not change a thing. I've said that, in
order to implement what I have suggested, a user would have to disable it,
which is not ideal, but acceptable if the user controls the distfile
distribution. And I only suggested that in order to try and make the idea
more acceptable by not requiring impractical infra changes that would be
needed to generate uncompressed hashes for the Manifests).

> But I don't understand how you think
> it's a solution to the xz-utils problem. For a wide variety of reasons,
> but the simplest one is that your proposal has zero plan of action for
> solving this at the community level and is entirely designed to allow
> "lone wolf" users to use throwaway systems performing
> security-sensitive actions (decompressing and hosting distfiles) in a
> networked environment that has the xz-utils installed, to feed into other
> security-sensitive systems (daily drivers etc.) that don't, but do have to
> trust the artifacts produced by the former.

I'm not entirely clear what you're trying to say in this paragraph. But
what I will say is I've tried very hard in any suggestions I've made to
only suggest things which will NOT change any default behaviour or require
big changes. The average user would not see any change from my revised
suggestions at all. I accepted after the first responses in this thread
that there was no appetite here to stop using xz utils. I then asked the
list about an idea I had just to see how palatable it might be. It was not
supposed to be a concrete plan, I was seeking 

Re: [gentoo-dev] Current unavoidable use of xz utils in Gentoo

[Joonas Niilola]

On 1.4.2024 23.07, James Le Cuirot wrote:
> 
> That's not stupid at all, I'd been thinking exactly the same thing. I raised
> this whole issue during a discussion at FOSDEM 2019, where I admitted that I
> didn't check the code changes for packages I was bumping, knowing that few to
> none of the other people in the room did so either. Despite speaking up then,
> I still didn't do it because it's a heavy a burden and I'm not paid to do it.
> Now I'm thinking I really should, but I could really use some help. I'll raise
> this idea at work. You could say that we specialise in these things. :)
> 
> Regards,
> Chewi

Offtopic but I'll just throw this out there: "pkgdiff-mg -b" from
mgorny-dev-scripts does wonders when bumping packages. Not everyone
knows about this so posting for awareness.

(Maybe slightly related after all since it would've shown the suspicious
CmakeLists.txt change at least)

-- juippis


OpenPGP_signature.asc
Description: OpenPGP digital signature