Re: avoid wrapper scripts when possible

[Maxime Devos]

Hi,

Attila Lendvai schreef op di 07-09-2021 om 18:52 [+]:
> pardon me for reviving a years old discussion, but i'm facing the same 
> problem once again.
> 
> i have updated trezor support, and i wanted to test generating a new gpg key:
> 
> $ trezor-gpg init "foobar "
> 
> leading to: OSError: Cannot find '.trezor-gpg-gpg-agent' in $PATH
> 
> upstream relies on sys.argv[0] in a non-trivial way:
> 
> https://github.com/romanz/trezor-agent/blob/338a075ed500f210a707cb3adf90104602e35c48/libagent/gpg/__init__.py#L124

You could patch trezor-agent with something like

  # Python syntax might be incorrect
  if sys.argv[0] == ".trezor-gpg'
sys.argv[0] = "trezor-gpg"
  device_name = os.path.basename(sys.argv[0]).rsplit('-',1)[0]

Would that work?

> is there any chance to revive this effort?
> 
> original discussion: 
> https://lists.gnu.org/archive/html/guix-devel/2017-11/msg00017.html
> 
> shall i open an issue? for this specific problem, or the wider situation 
> discussed in the mailing list thread?

FWIW, there's 'wrap-program' and there is 'wrap-script'.
wrap-script is less likely to cause errors here (it doesn't rename the 
executable)
but less general.  Maybe wrap-script can be used instead of wrap-program?

Greetings,
Maime


signature.asc
Description: This is a digitally signed message part

Re: Rethinking propagated inputs?

[Sarah Morgensen]

Maxim Cournoyer  writes:

> It seems the original issue pointed at was somewhat misguided; is there
> still something to be fixed about propagated inputs?  It seems the
> discussion has moved toward handling propagated inputs for the use of
> pkg-config.
>
> What are the current problems with it, and what would the advantages be
> to move away from the status quo?  If there aren't clear benefits, I'd
> prefer the status quo, abstaining from the added complexity.

As I mentioned up-thread:

Sarah Morgensen  writes:

>> Does anyone have an idea how we should handle propagations for the sake
>> of pkg-config?  Perhaps we could add "linked-inputs", which are added
>> when building packages and environments when not using --ad-hoc, but
>> not when union-building profiles.  WDYT?
>
> I know nothing about pkg-config, but such an input would help
> simplify things for Go (and I think for Rust) since many inputs need to
> be propagated only at build-time.

In Go(/Rust), the dependency packages' output contains only source, and
for a package to build it must have the source of all its transitive
inputs available.

However, IMO it doesn't make sense to automatically install these in a
profile when installing that package.  If you're installing a
source-only package, you likely don't want all the transitive
dependencies automatically installed--what if you're just inspecting the
source, or want to try building it with different dependencies?  If you
want all the build dependencies as well, you would just do

  guix environment go-foo --ad-hoc go-foo

like you would do for other Guix packages.

In the context of this discussion, I can think of three types of
solution:

1. Add a modifier like #:profile? (defaulting to #f) to
propagated-inputs entries
  (propagated-inputs
`(("foo" #:profile? #t)))

2. Rename "propagated-inputs" to e.g. "profile-propagated-inputs", and
then introduce a new "propagated-inputs", which only propagates into the
inputs of dependents, and not into built profiles.

3. Introduce a way to prevent an input from propagating anything into
the current package/profile
  (inputs
`(("foo" #:propagate? #f)))
and then provide CLI options (transformations) to control this.

I think the first two are clear and make the packager consider exactly
what they're going to propagate into user profiles.  Of course, both
also rely on the packager always knowing what should be propagated to a
user's profile or not.  The third option allows more control over that
(and could be combined with one of the first two), but feels a bit rough
to use...

WDYT?

--
Sarah

Re: avoid wrapper scripts when possible

[Attila Lendvai]

pardon me for reviving a years old discussion, but i'm facing the same problem 
once again.

i have updated trezor support, and i wanted to test generating a new gpg key:

$ trezor-gpg init "foobar "

leading to: OSError: Cannot find '.trezor-gpg-gpg-agent' in $PATH

upstream relies on sys.argv[0] in a non-trivial way:

https://github.com/romanz/trezor-agent/blob/338a075ed500f210a707cb3adf90104602e35c48/libagent/gpg/__init__.py#L124

is there any chance to revive this effort?

original discussion: 
https://lists.gnu.org/archive/html/guix-devel/2017-11/msg00017.html

shall i open an issue? for this specific problem, or the wider situation 
discussed in the mailing list thread?

- attila
PGP: 5D5F 45C7 DFCD 0A39

Python Site Package Syntax Runtime Error

[Antwane Mason]

Has anyone come across the following runtime error when running a
python-based application in guix? I'm wondering if something went wrong
during the package build for python-onlykey-agent that messed up the
PYTHONPATH export for the runtime.  I should note that python-onlykey-agent
is a program that I have packaged to eventually submit to guix.

*--8<---cut here---start->8---*

export
PYTHONPATH="...:/gnu/store/wand0zrwwnds6x636746116cfh3sy50k-python-pyopenssl-20.0.0/lib/python3.8/site-packages${PYTHONPATH:+:}$PYTHONPATH"

SyntaxError: invalid syntax

*--8<---cut here---end--->8---*

Here is the entire stack trace for reference.

*--8<---cut here---start->8---*

antwane@wmguix ~$ onlykey-agent g...@github.com
Traceback (most recent call last):
  File
"/gnu/store/s2w1lq80x9vcwp5382kn98f5pi2k4b7b-python-onlykey-agent-1.1.12/bin/.onlykey-agent-real",
line 11, in 
load_entry_point('onlykey-agent==1.1.12', 'console_scripts',
'onlykey-agent')()
  File
"/gnu/store/sd2ic7bpv8fx3imy1j8xcjclx71sv6q0-python-3.8.2/lib/python3.8/site-packages/pkg_resources/__init__.py",
line 489, in load_entry_point
return get_distribution(dist).load_entry_point(group, name)
  File
"/gnu/store/sd2ic7bpv8fx3imy1j8xcjclx71sv6q0-python-3.8.2/lib/python3.8/site-packages/pkg_resources/__init__.py",
line 2852, in load_entry_point
return ep.load()
  File
"/gnu/store/sd2ic7bpv8fx3imy1j8xcjclx71sv6q0-python-3.8.2/lib/python3.8/site-packages/pkg_resources/__init__.py",
line 2443, in load
return self.resolve()
  File
"/gnu/store/sd2ic7bpv8fx3imy1j8xcjclx71sv6q0-python-3.8.2/lib/python3.8/site-packages/pkg_resources/__init__.py",
line 2449, in resolve
module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File
"/gnu/store/s2w1lq80x9vcwp5382kn98f5pi2k4b7b-python-onlykey-agent-1.1.12/bin/onlykey_agent.py",
line 2
export
PYTHONPATH="/gnu/store/s2w1lq80x9vcwp5382kn98f5pi2k4b7b-python-onlykey-agent-1.1.12/lib/python3.8/site-packages:/gnu/store/ahbp2qnrx7m6m5yrxcfsf37gqmgkm13c-python-lib-agent-1.0.3/lib/python3.8/site-packages:/gnu/store/xq0bpzrxy461igy3vwymgmlfg85ghfgv-python-onlykey-1.2.6/lib/python3.8/site-packages:/gnu/store/sd2ic7bpv8fx3imy1j8xcjclx71sv6q0-python-3.8.2/lib/python3.8/site-packages:/gnu/store/gb0xnb6vldlmwiky77w7hxpkjbvp1560-python-wheel-0.36.2/lib/python3.8/site-packages:/gnu/store/7c48ls8c4xddx22yy7rlnszs0fjl56k7-python-unidecode-1.1.1/lib/python3.8/site-packages:/gnu/store/hi8yk6yzivzwpxak2fcggd6zl887n7zj-python-semver-2.9.0/lib/python3.8/site-packages:/gnu/store/gvka9y88b5i6h1fbwb5qxi8vv0ajh6d3-python-pynacl-1.4.0/lib/python3.8/site-packages:/gnu/store/4153n7v3hy0pn2df0c8rag7v4gmgg0ki-python-pymsgbox-1.0.6/lib/python3.8/site-packages:/gnu/store/ppdbw2sl5filj79rl6k0wqy7xkz2603n-python-pycryptodome-3.9.9/lib/python3.8/site-packages:/gnu/store/b2pf54i477iby8f17am9p0x2bz96gn63-python-mnemonic-0.20/lib/python3.8/site-packages:/gnu/store/3vl9a6y7d9r3w3diy2bj15jxd0qgzycv-python-ecdsa-0.14.1/lib/python3.8/site-packages:/gnu/store/1qz6j51mmn2z1za41fb9vs9dgmwisng3-python-docutils-0.16/lib/python3.8/site-packages:/gnu/store/v74q9niaprwv0l25hb1hcyhvyzwp7zmf-python-daemon-2.2.3/lib/python3.8/site-packages:/gnu/store/jvm96kifzns968g0l115chfdp3n5ksba-python-configargparse-1.2.3/lib/python3.8/site-packages:/gnu/store/j3w4j8x76n7wdbjrh377fwx4hwgmfwmy-python-backports.shutil-which-3.5.2/lib/python3.8/site-packages:/gnu/store/hjmz8ymac939ribn7g3jkgms4dk2az3a-python-six-1.14.0/lib/python3.8/site-packages:/gnu/store/bxbhdka9x8pnl7jnpkvz1nvgg2m8xqfs-python-prompt-toolkit-3.0.18/lib/python3.8/site-packages:/gnu/store/2lrpayrb6rrfmscfkfii65jdd328v945-python-onlykey-solo-0.0.28/lib/python3.8/site-packages:/gnu/store/1xmngywbzxx8lb539z4zcggm12gw6lzm-python-hidapi-0.7.99.post21/lib/python3.8/site-packages:/gnu/store/j1qn1pjvpyljmrb7hh1pjsgz44b13y5i-python-cython-0.29.22/lib/python3.8/site-packages:/gnu/store/d5zp1z8vz42c8b69fszck1vqh59npms4-python-aenum-2.2.4/lib/python3.8/site-packages:/gnu/store/sfy8vc8da10appy093izjz4byzd7sfqp-python-cffi-1.14.4/lib/python3.8/site-packages:/gnu/store/wi6ihkh2dq1sj6pq6g28f5vw157kfy8x-python-lockfile-0.12.2/lib/python3.8/site-packages:/gnu/store/xjiplzqxr084ximm4wdhyx3wzf8nxvv0-python-wcwidth-0.1.8/lib/python3.8/site-packages:/gnu/store/kffam5lzshdhb0ygq0r76gqvavs0190c-python-requests-2.25.0/lib/python3.8/site-packages:/gnu/store/qkj3ssg0y3ya0d450pp0kaa7zbg9i0x6-python-pyusb-1.0.2/lib/python3.8/site-packages:/gnu/store/b3k1kmx7rzkykb4ik5nz0jdabqr86vvv-python-pyserial-3.5/lib/python3.8/site-packages:/gnu/store/w3q52n0pc5zvblr8qdsgdfiw33psj6pz-python-intelhex-2.2.1/lib/python3.8/site-packages:/gnu/store/fpk2f2jwqah0s340hdhmr734vvxq3z3j-python-fido2-0.9.1/lib/python3.8/site-packages:/gnu/store/sjiiqs605sfsqxjdkcm3m8rx0ahv5wvv-python-cryptography-3.3.1/lib/python3.8/site-packages:/gnu/store/5a12m99wdfqydzwx46nlfqqvi9q3pq49-python-click-7.1.2/lib/python3.8/site-packages:/gnu/store

Re: [core-updates-frozen] Bug in binutils 2.37

[Guillaume Le Vaillant]

Leo Famulari  skribis:

> On Mon, Sep 06, 2021 at 03:39:52PM +, Guillaume Le Vaillant wrote:
>> There's a bug in binutils 1.37, which we are using on the
>> core-updates-frozen branch.
>
> 2.37, right? :)
>

Indeed. :)


>> It's a file descriptor leak that can lead to 'malformed archive' errors
>> when linking libraries. We get this problem at least when building
>> qtwekbit and qtwebengine. A workaround allows us to compile qtwebkit
>> (see [1]), but it doesn't work for qtwebengine.
>> 
>> The bug was discussed at [2] and upstream has a patch to fix it at [3].
>> However, adding this patch to our binutils rebuilds the world.
>> I'm currently trying to build things with the patched binutils.
>> If everything works, should I push this fix on core-updates-frozen, or
>> does someone have an idea that would lead to less rebuilds?
>
> There are a few options:
>
> 1) Apply the patch in the normal way and rebuild the world. If the
> changes in the patch are limited to fixing this bug, then we can be
> confident that changing binutils will not break other packages, which is
> the main goal behind "freezing" the core-updates branch. Do you think
> that expectation is reasonable? Otherwise, the branch is frozen except
> for bug fixes; we'd like to avoid rebuilding the world but it's not a
> problem if we have to.
> 2) Create a binutils-fixed package and only use it for qtwebkit and
> qtwebengine
> 3) Try to work around the bug in the qtwebkit and qtwebengine packages
>
> 2 and 3 are not great because maybe the bug affects other packages in
> some situations. Do you know if it manifests deterministically?

The problem happens when linking a library with a lot of members (I
don't know exactly how many), which is the case at least for qtwebkit
and qtwebengine. Users creating such libraries in their projects will
also have the problem.

Increasing the maximum number of open file descriptors seems not be
a very robust workaround. Setting it at 4096 for qtwebkit works,
but I tried 4096, 8192 and 16384 for qtwebengine and it didn't work.

I have built many packages with the patched binutils and I didn't get
any new failure so far. I'm not yet at qtwebkit and qtwebengine though
(compiling the 20 versions of rust took a while).
So when I get there, and if the patch really solves the issue, pushing
it looks like the best solution.

Do you know if there are other world-rebuilding pending fixes that could
go in at the same time?


signature.asc
Description: PGP signature

Re: [core-updates-frozen] Bug in binutils 1.37

[Leo Famulari]

On Mon, Sep 06, 2021 at 03:39:52PM +, Guillaume Le Vaillant wrote:
> There's a bug in binutils 1.37, which we are using on the
> core-updates-frozen branch.

2.37, right? :)

> It's a file descriptor leak that can lead to 'malformed archive' errors
> when linking libraries. We get this problem at least when building
> qtwekbit and qtwebengine. A workaround allows us to compile qtwebkit
> (see [1]), but it doesn't work for qtwebengine.
> 
> The bug was discussed at [2] and upstream has a patch to fix it at [3].
> However, adding this patch to our binutils rebuilds the world.
> I'm currently trying to build things with the patched binutils.
> If everything works, should I push this fix on core-updates-frozen, or
> does someone have an idea that would lead to less rebuilds?

There are a few options:

1) Apply the patch in the normal way and rebuild the world. If the
changes in the patch are limited to fixing this bug, then we can be
confident that changing binutils will not break other packages, which is
the main goal behind "freezing" the core-updates branch. Do you think
that expectation is reasonable? Otherwise, the branch is frozen except
for bug fixes; we'd like to avoid rebuilding the world but it's not a
problem if we have to.
2) Create a binutils-fixed package and only use it for qtwebkit and
qtwebengine
3) Try to work around the bug in the qtwebkit and qtwebengine packages

2 and 3 are not great because maybe the bug affects other packages in
some situations. Do you know if it manifests deterministically?

Re: PEP 668 -- Graceful cooperation between external and Python package managers

[Leo Famulari]

On Tue, Sep 07, 2021 at 04:39:28PM +0200, Maxime Devos wrote:
> See .
> I haven't looked closely into this myself.
> It might be relevant to Guix.
> 
> For LWN subscribers, there is an article about the PEP:
> .

Here is a link for everyone to read:

https://lwn.net/SubscriberLink/867657/c4e77bb70e27c910/

PEP 668 -- Graceful cooperation between external and Python package managers

[Maxime Devos]

See .
I haven't looked closely into this myself.
It might be relevant to Guix.

For LWN subscribers, there is an article about the PEP:
.

Greetings,
Maxime.


signature.asc
Description: This is a digitally signed message part

Re: Rethinking propagated inputs?

[宋文武]

Hello, good plan!


Liliana Marie Prikler  writes:

> At this point the question then becomes what to name this "build"
> output and what to put into it besides pkg-config stuff.
In Debian, it's a "dev" package, includes .h (C headers), .pc, .a
(static libraries).


> Particularly in the land of glib, there also exist typelibs, which can
> be used as "build" inputs in the case of Vala or as runtime inputs in
> the case of pygobject and other language bindings.
It's "gir1.2-glib-2.0" in debian, we can add a "gir" output.


> Perhaps this is
> early bikeshedding when we'd actually need to code up #:by, but what
> would be the better approach here?  Separate "build" into
> "pkg-config", "cmake" (for CMake-based package discovery), "typelib"
> etc. or have one more or less anonymous blob called "build"?
I think we should have "build" (or "dev") and "typelib" (or "gir"), but
not "pkg-config" and "cmake" in most cases.

With a "dev" output, we don't need 'propagated-inputs' in other outputs
most cases, so reduce the mess for profiles.

With a "typelib" output, we can reduce the runtime closure size for
packages which don't depends on them.

The build time clojure size reduced by "pkg-config" and "cmake" outputs is
small.

Re: Rethinking propagated inputs?

[Maxime Devos]

Liliana Marie Prikler schreef op zo 05-09-2021 om 23:10 [+0200]:
> Am Sonntag, den 05.09.2021, 22:27 +0200 schrieb Maxime Devos:
> > Liliana Marie Prikler schreef op zo 05-09-2021 om 21:37 [+0200]:
> > > > > I must admit that this solution appears to have some surface
> > > > > elegance, but what exactly would go in the "build" output of a
> > > > > package?  You mentioned pkg-config files (obviously), but those
> > > > > don't suffice to actually build a package, do they?
> > > > 
> > > > Sometimes they do suffice.  The .pc files contain the "-
> > > > L/.../LIB", "-I/.../include" and "-lstuff" flags needed for
> > > > compilation.  If the build system of the package uses pkg-config, 
> > > > it will use those flags, so the compiler will find the library in
> > > > that case.
> > > > 
> > > > Not sure if they always do suffice.
> > > 
> > > Is that so?  I would think the build process needs to see stuff
> > > outside of its inputs for that to work, e.g. the actual header it
> > > wants to include, which isn't part of "build".  Am I
> > > misunderstanding our sandbox requirements?
> > 
> > The .pc file includes the absolute path to the library and include
> > directories, so the output "build" with the .pc file has a reference
> > to the output "out" with the libraries and include headers.  More
> > concretely, take the .pc from the glib package:
> > 
> > prefix=/gnu/store/98hgv3i6hdqgiq98ldy7rkpdwhah8iq2-glib-2.62.6
> > libdir=${prefix}/lib
> > includedir=${prefix}/include
> > [more stuff]
> > Requires.private: libpcre >=  8.31
> > Libs: -L${libdir} -lglib-2.0
> > Libs.private: -pthread
> > Cflags: -I${includedir}/glib-2.0 -I${libdir}/glib-2.0/include
> > 
> > The (transitive) references of all inputs to the build process are
> > included in the sandbox.  In this case, if the package has the
> > hypothetical glib:build among its inputs, the daemon will
> > automatically make glib:out available as well in the sandbox.
> And IIUC if glib had a "lib" output, glib:lib would be available in the
> sandbox instead of glib:out due to the reference by glib:build?

If a package has the proposed 'glib:build' in its (propagated-)inputs,
and if 'glib' had a 'lib' output, then in the build sandbox of the package,
'glib:lib' will be available.

>   If
> that's the case using #:by and "build" outputs might be a preferable
> solution, if not for all packages then at least for some.

> At this point the question then becomes what to name this "build"
> output and what to put into it besides pkg-config stuff.  Particularly
> in the land of glib, there also exist typelibs, which can be used as
> "build" inputs in the case of Vala or as runtime inputs in the case of
> pygobject and other language bindings.  Perhaps this is early
> bikeshedding when we'd actually need to code up #:by, but what would be
> the better approach here?  Separate "build" into "pkg-config", "cmake"
> (for CMake-based package discovery), "typelib" etc. or have one more or
> less anonymous blob called "build"?

I don't know (-:.  Maybe let's just start with pkg-config and #:by?
for now?  The name can always be changed later.

Greetings,
Maxime.


signature.asc
Description: This is a digitally signed message part