Re: avoid wrapper scripts when possible
Related is #50798 which I just filed https://debbugs.gnu.org/cgi/bugreport.cgi?bug=50789 This is a similar issue in the executable path picked up by a program (finding the -real one) and using that for writing a .desktop file. Not sure if there is anything more general to do here, other than having to find these cases and patch manually? Is there a preferred way of making these patches? Thanks, John
Re: avoid wrapper scripts when possible
Attila Lendvai schreef op wo 08-09-2021 om 09:20 [+]: > thanks for the ideas Maxime! > > > > You could patch trezor-agent with something like > > > > Python syntax might be incorrect > > > > > > if sys.argv[0] == ".trezor-gpg' > > sys.argv[0] = "trezor-gpg" > > device_name = os.path.basename(sys.argv[0]).rsplit('-',1)[0] > > > > Would that work? > > most probably, but i thought first i'll pursue a more general fix for > this. i will fall back to just patching upstream if i fail. > > > > FWIW, there's 'wrap-program' and there is 'wrap-script'. > > > > wrap-script is less likely to cause errors here (it doesn't rename the > > executable) > > but less general. Maybe wrap-script can be used instead of wrap-program? > > not sure how to do this. looking at python-build-system.scm it looks > like there is no way to tell it to use wrap-script, so i went ahead > and just changed the wrap-program call to wrap-script globally, and > gave it a try. Are you referring to the lines (for-each (lambda (dir) (let ((files (list-of-files dir))) (for-each (cut wrap-program <> var) files))) bindirs) in the procedure 'wrap' from (guix build python-build-system)? > the first error i encountered was in the package called scons. it has > .bat files that wrap-script fails on. if i delete them in a snippet, > then scons' own build system errors out. The .bat are only for Windows, so you could remove them after the 'install' phase or before the 'wrap' phase. They seem to be useless on GNU/{Linux,Hurd}. > any hints on how to proceed from here? Instead of modifying 'wrap', you could replace the 'wrap' phase with something like `(#:phases (modify-phases %standard-phases (replace 'wrap (lambda _ ... customised wrapping code ... Also, replacing wrap-program with wrap-script globally won't work because wrap-script is less general, it only spports python, R, perl and bash. So something like (define (wrap-script-or-program ...) (if it-uses-python-or-ruby-or... (wrap-script ...) (wrap-program ...))) may be required. > was this what you meant at all? > > would it be a worthwhile improvement to hack my way through this path > at all? There were some changes on core-updates to reduce double-wrapping, so possibly it's already solved. > shall i try to make it configurable from the package's definition, > change the default, and force it back to wrap-program for packages > that fail with wrap-script? Perhaps try something like 'wrap-script-or-program' first? That should reduce the number of package definitions that need to be adjusted. Greetings, Maxime. signature.asc Description: This is a digitally signed message part
Re: avoid wrapper scripts when possible
thanks for the ideas Maxime! > You could patch trezor-agent with something like > > Python syntax might be incorrect > > > if sys.argv[0] == ".trezor-gpg' > sys.argv[0] = "trezor-gpg" > device_name = os.path.basename(sys.argv[0]).rsplit('-',1)[0] > > Would that work? most probably, but i thought first i'll pursue a more general fix for this. i will fall back to just patching upstream if i fail. > FWIW, there's 'wrap-program' and there is 'wrap-script'. > > wrap-script is less likely to cause errors here (it doesn't rename the > executable) > but less general. Maybe wrap-script can be used instead of wrap-program? not sure how to do this. looking at python-build-system.scm it looks like there is no way to tell it to use wrap-script, so i went ahead and just changed the wrap-program call to wrap-script globally, and gave it a try. the first error i encountered was in the package called scons. it has .bat files that wrap-script fails on. if i delete them in a snippet, then scons' own build system errors out. any hints on how to proceed from here? was this what you meant at all? would it be a worthwhile improvement to hack my way through this path at all? shall i try to make it configurable from the package's definition, change the default, and force it back to wrap-program for packages that fail with wrap-script? - attila
Re: avoid wrapper scripts when possible
Hi, Attila Lendvai schreef op di 07-09-2021 om 18:52 [+]: > pardon me for reviving a years old discussion, but i'm facing the same > problem once again. > > i have updated trezor support, and i wanted to test generating a new gpg key: > > $ trezor-gpg init "foobar " > > leading to: OSError: Cannot find '.trezor-gpg-gpg-agent' in $PATH > > upstream relies on sys.argv[0] in a non-trivial way: > > https://github.com/romanz/trezor-agent/blob/338a075ed500f210a707cb3adf90104602e35c48/libagent/gpg/__init__.py#L124 You could patch trezor-agent with something like # Python syntax might be incorrect if sys.argv[0] == ".trezor-gpg' sys.argv[0] = "trezor-gpg" device_name = os.path.basename(sys.argv[0]).rsplit('-',1)[0] Would that work? > is there any chance to revive this effort? > > original discussion: > https://lists.gnu.org/archive/html/guix-devel/2017-11/msg00017.html > > shall i open an issue? for this specific problem, or the wider situation > discussed in the mailing list thread? FWIW, there's 'wrap-program' and there is 'wrap-script'. wrap-script is less likely to cause errors here (it doesn't rename the executable) but less general. Maybe wrap-script can be used instead of wrap-program? Greetings, Maime signature.asc Description: This is a digitally signed message part
Re: avoid wrapper scripts when possible
pardon me for reviving a years old discussion, but i'm facing the same problem once again. i have updated trezor support, and i wanted to test generating a new gpg key: $ trezor-gpg init "foobar " leading to: OSError: Cannot find '.trezor-gpg-gpg-agent' in $PATH upstream relies on sys.argv[0] in a non-trivial way: https://github.com/romanz/trezor-agent/blob/338a075ed500f210a707cb3adf90104602e35c48/libagent/gpg/__init__.py#L124 is there any chance to revive this effort? original discussion: https://lists.gnu.org/archive/html/guix-devel/2017-11/msg00017.html shall i open an issue? for this specific problem, or the wider situation discussed in the mailing list thread? - attila PGP: 5D5F 45C7 DFCD 0A39
Re: avoid wrapper scripts when possible
Ricardo Wurmus writes: > Ricardo Wurmus writes: > >> I wonder if there’s a way to have a script header that starts out as a >> shell script and then calls itself again but skipping over the header, >> thus calling the original script. >> >> Take this Python script as an example: >> >> --8<---cut here---start->8--- >> #!/home/rekado/.guix-profile/bin/python >> print("hello from python") >> --8<---cut here---end--->8--- >> >> We then prepend this header: >> >> --8<---cut here---start->8--- >> #!/run/current-system/profile/bin/bash >> echo "hello from bash" >> exec /home/rekado/.guix-profile/bin/python >> <(/run/current-system/profile/bin/tail -n +4 "$0") >> --8<---cut here---end--->8--- >> >> When we execute the resulting thing it will be a shell script, and do >> shell things printing “hello from bash”. Then it scrubs itself off the >> script and removes the original script’s Python shebang using tail (with >> “-n +4” where 4 is the number of lines of the shell script header + 1), >> and replaces itself with Python calling the original script. >> >> This seems to work just fine […] > > Erm, it does not. “$0” becomes a file descriptor like “/dev/fd/63”. > “exec -a $0” also doesn’t help here. How about this: --8<---cut here---start->8--- #!/home/rekado/.guix-profile/bin/guile --no-auto-compile #!# (let (($0 (car (command-line (execl "/home/rekado/.guix-profile/bin/python3" $0 $0)) #!/home/rekado/.guix-profile/bin/python3 import sys; print("hello from python: "+sys.argv[0]) --8<---cut here---end--->8--- The first two lines are Guile code, but they are also line comments in shell, Perl, Python, Ruby, and R. The Guile code in this example calls the script again as a Python script. Before doing that it can set environment variables, like so: --8<---cut here---start->8--- #!/home/rekado/.guix-profile/bin/guile --no-auto-compile #!# #\- (setenv "PYTHONPATH" (string-append "/gnu/store/foo:/gnu/store/bar:" (getenv "PYTHONPATH"))) #\- (let (($0 (car (command-line (execl "/home/rekado/.guix-profile/bin/python3" $0 $0)) #!/home/rekado/.guix-profile/bin/python3 import sys; print("hello from python: "+sys.argv[0]) print(sys.path) --8<---cut here---end--->8--- Did I overlook something? Or could we use this hack to remove separate wrapper scripts for Perl, Python, Ruby, and R? -- Ricardo GPG: BCA6 89B6 3655 3801 C3C6 2150 197A 5888 235F ACAC https://elephly.net
Re: avoid wrapper scripts when possible
Ricardo Wurmus writes: > For binaries (like emacs) we’d still create shell wrappers where needed, > because it’s harder to do this natively. You could link in a constructor function to replace a wrapper that just sets the environment, couldn't you? I've done that at times and it allows you to run gdb directly, for instance. How easy it is presumably depends on how the program is built, e.g. autoconf LIBS. If it's difficult to separate configure and build, you can use a dummy linker script at configure time.
Re: avoid wrapper scripts when possible
Hi! Hartmut Goebel skribis: > answering on a mail which Ricardo CCed to me and which did not make it > to the list yet. Thus I full-quote. > > Am 04.11.2017 um 23:30 schrieb Ricardo Wurmus: >> How about this: >> >> --8<---cut here---start->8--- >> #!/home/rekado/.guix-profile/bin/guile --no-auto-compile >> #!# (let (($0 (car (command-line (execl >> "/home/rekado/.guix-profile/bin/python3" $0 $0)) >> #!/home/rekado/.guix-profile/bin/python3 >> import sys; print("hello from python: "+sys.argv[0]) >> --8<---cut here---end--->8--- >> >> The first two lines are Guile code, but they are also line comments in >> shell, Perl, Python, Ruby, and R. The Guile code in this example calls >> the script again as a Python script. Before doing that it can set >> environment variables, like so: >> >> --8<---cut here---start->8--- >> #!/home/rekado/.guix-profile/bin/guile --no-auto-compile >> #!# >> #\- (setenv "PYTHONPATH" (string-append "/gnu/store/foo:/gnu/store/bar:" >> (getenv "PYTHONPATH"))) >> #\- (let (($0 (car (command-line (execl >> "/home/rekado/.guix-profile/bin/python3" $0 $0)) >> #!/home/rekado/.guix-profile/bin/python3 >> import sys; print("hello from python: "+sys.argv[0]) >> print(sys.path) >> --8<---cut here---end--->8--- >> >> Did I overlook something? Or could we use this hack to remove separate >> wrapper scripts for Perl, Python, Ruby, and R? Overall this looks good to me (with the adjustments Hartmut mentions). Now, we should have a white list of languages for which this can be done. ‘wrap-program’ would grab the shebang and check if the basename of the interpreter is in that white list. Thoughts? Ludo’.
Re: avoid wrapper scripts when possible
Hi, Ricado proposed a new and better solution already. Nevertheless I want to comment on this proposal, sint it includes a pitfall and would not work: Am 03.11.2017 um 22:17 schrieb Ricardo Wurmus: > exec /home/rekado/.guix-profile/bin/python > <(/run/current-system/profile/bin/tail -n +4 "$0") [...] > This seems to work just fine and off the top of my head I can’t think of > a situation where this would fail — unless the wrapped script performed > the same kind of trick of reading its contents through $0. (How likely > is that?) Indeed for Python programs it is quite common to revert to the filename using the variable "__file__". And if feeding in the script from stdin would set __file__ to "". Using __file__ is more often used in package modules than in scripts (which could use $0 (sys.argv[0]). Anyway this is a pitfall we should avoid. Please not that Ricardo's new solution, based on guile and execl does not suffer from this problem (I verified). -- Regards Hartmut Goebel | Hartmut Goebel | h.goe...@crazy-compilers.com | | www.crazy-compilers.com | compilers which you thought are impossible |
Re: avoid wrapper scripts when possible
Hi, answering on a mail which Ricardo CCed to me and which did not make it to the list yet. Thus I full-quote. Am 04.11.2017 um 23:30 schrieb Ricardo Wurmus: > How about this: > > --8<---cut here---start->8--- > #!/home/rekado/.guix-profile/bin/guile --no-auto-compile > #!# (let (($0 (car (command-line (execl > "/home/rekado/.guix-profile/bin/python3" $0 $0)) > #!/home/rekado/.guix-profile/bin/python3 > import sys; print("hello from python: "+sys.argv[0]) > --8<---cut here---end--->8--- > > The first two lines are Guile code, but they are also line comments in > shell, Perl, Python, Ruby, and R. The Guile code in this example calls > the script again as a Python script. Before doing that it can set > environment variables, like so: > > --8<---cut here---start->8--- > #!/home/rekado/.guix-profile/bin/guile --no-auto-compile > #!# > #\- (setenv "PYTHONPATH" (string-append "/gnu/store/foo:/gnu/store/bar:" > (getenv "PYTHONPATH"))) > #\- (let (($0 (car (command-line (execl > "/home/rekado/.guix-profile/bin/python3" $0 $0)) > #!/home/rekado/.guix-profile/bin/python3 > import sys; print("hello from python: "+sys.argv[0]) > print(sys.path) > --8<---cut here---end--->8--- > > Did I overlook something? Or could we use this hack to remove separate > wrapper scripts for Perl, Python, Ruby, and R? I testes this and (beside some necessary adjustments) this works. Indeed this a very elegant hack :-). My respect! Compared to my idea this is more elegant since it does not require a separate file. Some fine-tuning needs to be done, though: 1) The setenv-line should only append the existing env-var, if this was set. (At least in Python, an empty PYTHONPATH part is interpreted as "current directory", thus adding an empty part would change the desired behavior.) 2) Arguments need to be passed to the script, too. Since I was eager to learn how this can be done, here is my draft fixing these two issues: --8<---cut here---start->8--- #!/bin/guile --no-auto-compile #!#; Guix wrapping header #\- (define (append-env name path) (let ((var (getenv name))) (setenv name (if var (string-append path ":" var) path #\- (append-env "PYTHONPATH" "/gnu/store/foo:/gnu/store/bar") #\- (apply execl "/bin/python3" (car (command-line)) (command-line)) #!/home/rekado/.guix-profile/bin/python3 print("This is the real program") import sys, os print("sys.path[:4]", sys.path[:4]) print("argumentes:", sys.argv) print("ppid", os.getppid()) --8<---cut here---end--->8--- -- Regards Hartmut Goebel | Hartmut Goebel | h.goe...@crazy-compilers.com | | www.crazy-compilers.com | compilers which you thought are impossible |
Re: avoid wrapper scripts when possible
Christopher Baines writes: > However, I think that the file wrapping approach has advantages for > visibility. Maybe it could be tweaked to keep ensure the wrapper script > has the same name as the script its wrapping, e.g. when wrapping foo, > replace foo with a bash script, and move the real script > to .wrapped-bin/foo. I agree that having a separate wrapper is certainly nicer than patching upstream scripts automatically. The idea of moving original scripts out of the way sounds intriguing, but I fear that we would have a similar set of problems. In the case of conda’s “activate” script there would be an obvious problem: the script tries to determine the directory that contains it. If that were a different directory than the one the developers intended, it would fail in unexpected ways. I wonder if there’s a way to have a script header that starts out as a shell script and then calls itself again but skipping over the header, thus calling the original script. Take this Python script as an example: --8<---cut here---start->8--- #!/home/rekado/.guix-profile/bin/python print("hello from python") --8<---cut here---end--->8--- We then prepend this header: --8<---cut here---start->8--- #!/run/current-system/profile/bin/bash echo "hello from bash" exec /home/rekado/.guix-profile/bin/python <(/run/current-system/profile/bin/tail -n +4 "$0") --8<---cut here---end--->8--- When we execute the resulting thing it will be a shell script, and do shell things printing “hello from bash”. Then it scrubs itself off the script and removes the original script’s Python shebang using tail (with “-n +4” where 4 is the number of lines of the shell script header + 1), and replaces itself with Python calling the original script. This seems to work just fine and off the top of my head I can’t think of a situation where this would fail — unless the wrapped script performed the same kind of trick of reading its contents through $0. (How likely is that?) What do you think? Is this too weird to consider? Weirder than renaming binaries and replacing them with scripts? One side effect would be that “file” would be rather confused about these scripts, but maybe that’s acceptable. -- Ricardo GPG: BCA6 89B6 3655 3801 C3C6 2150 197A 5888 235F ACAC https://elephly.net
Re: avoid wrapper scripts when possible
On Fri, 3 Nov 2017 18:54:16 + Christopher Baines wrote: > However, I think that the file wrapping approach has advantages for > visibility. Maybe it could be tweaked to keep ensure the wrapper script > has the same name as the script its wrapping, e.g. when wrapping foo, > replace foo with a bash script, and move the real script > to .wrapped-bin/foo. We already try to do just that, by using the -a flag when exec'ing the real program from the wrapper. But... > the environment activation feature of our “conda” package currently > fails. This is because the “deactivate” shell script is wrapped in > another shell script. This leads to the actual shell script to be > called “.deactivate-real”. The script compares the value of “$0” with > the expected name “deactivate”. This fails so conda misbehaves. ... it doesn't behave correctly when the "real" program is shebang-interpreted. It works fine for compiled programs. E.g. our "emacs-25.3" is a wrapper that sets XDG and GTK env vars, but top/htop/ps show the proper "emacs-25.3". OTOH we have the "deactivate" example from Ricardo, which is a shell script, and e.g. youtube-dl, whose --help output includes "Usage: .youtube-dl-real ...". So the interpreters are not passing on the intended zeroth argument. IMHO it'd be nice to find some way to fix that. A while ago I had looked into patching Perl libraries/programs. But like Hartmut mentioned for python, I noticed after some research that patching perl paths would be nontrivial, and would require a perl parser to determine where and how to patch the library paths correctly. 2c, `~Eric pgp92UJXH00k6.pgp Description: OpenPGP digital signature
Re: avoid wrapper scripts when possible
Hallo Ricardo, Am 02.11.2017 um 22:31 schrieb Ricardo Wurmus: > This made me wonder if we could avoid shell wrappers in more cases. This is an interesting challenge :-) > If > the target script is a Python script we could set the environment > variables right after the Python shebang with Python code. If the > target script is a shell script we can set environment variables right > there after the shebang. Basically thsi would be a good solution IMO. But I'm afraid it might be complicated to insert the code "right after the […] shebang". For example Python has "doc-strings" [1], this is: if the first statement of the script is a string (expression statement), this string is put into the variable __doc__. So for at least for Python we might need a language-aware scanner to insert the code at the correct position. We might need similar tor other languages. But taking your challenge :-) I had another idea which should work for all script languages: The real script's she-bang delegates to the wrapper (instead of /bin/sh), which execs "interpreter $0". Here is a proof on concept: 8<--- real.py #!/tmp/wrapper.sh print("This is real") import sys, os print("sys.path[:2]", sys.path[:2]) print("argumentes:", sys.argv) print("ppid", os.getppid()) 8<--- 8<--- wrapper.sh #!/bin/sh echo "This is the wrapper" echo "arguments in wrapper:" "$@" echo "ppid:" $PPID export PYTHONPATH=/foo/bar exec python3 /tmp/real.py "${@:2}" # <- replace first argument by full path 8<--- I have to admit that this still adds another script and is less elegant, but should be easier to implement. []1] https://www.python.org/dev/peps/pep-0257/#what-is-a-docstring -- Regards Hartmut Goebel | Hartmut Goebel | h.goe...@crazy-compilers.com | | www.crazy-compilers.com | compilers which you thought are impossible |
Re: avoid wrapper scripts when possible
Christopher Baines writes: > However, I think that the file wrapping approach has advantages for > visibility. Maybe it could be tweaked to keep ensure the wrapper script > has the same name as the script its wrapping, e.g. when wrapping foo, > replace foo with a bash script, and move the real script > to .wrapped-bin/foo. To probably state the obvious, yes it would be great if top/ps would show `foo' instead of .foo-x.y-real; also great for `killall'. Greetings, janneke -- Jan Nieuwenhuizen | GNU LilyPond http://lilypond.org Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com
Re: avoid wrapper scripts when possible
On Thu, 02 Nov 2017 22:31:16 +0100 Ricardo Wurmus wrote: > Hi Guix, > > the environment activation feature of our “conda” package currently > fails. This is because the “deactivate” shell script is wrapped in > another shell script. This leads to the actual shell script to be > called “.deactivate-real”. The script compares the value of “$0” with > the expected name “deactivate”. This fails so conda misbehaves. > > Since “deactivate” is really just a shell script I think we could avoid > the renaming and the external wrapper by setting the environment > variables in the “deactivate” script itself. > > This made me wonder if we could avoid shell wrappers in more cases. If > the target script is a Python script we could set the environment > variables right after the Python shebang with Python code. If the > target script is a shell script we can set environment variables right > there after the shebang. > > For binaries (like emacs) we’d still create shell wrappers where needed, > because it’s harder to do this natively. > > What do you think? I recently tried to implement automatic wrapping for the ruby build system [1]. I ended up implementing a ruby specific wrapping function, that generated a ruby wrapper, rather than a bash wrapper. I first tried using a bash wrapper, but it broke using the -S flag with ruby to execute the script, as when -S is passed to ruby, it expects the script on the PATH to use ruby in the shebang, and not bash. 1: https://debbugs.gnu.org/cgi/bugreport.cgi?bug=28773 After I wrote this, I also stumbled upon #27003: Generalized wrap phase for perl, python. I haven't read through the thread enough to work out what the state of it is. 2: https://debbugs.gnu.org/cgi/bugreport.cgi?bug=27003 I've also found some other bugs relating to wrapping, [3] is about the issue you mention, and [4] relates to the native inputs problem, that I came across when I was working on the ruby-build-system wrapping. 3: https://debbugs.gnu.org/cgi/bugreport.cgi?bug=26752 4: https://debbugs.gnu.org/cgi/bugreport.cgi?bug=25235 I haven't read through all of these in much detail though. Back to your specific question about avoiding shell wrappers. For ruby, I found this necessary to avoid breaking one way of invoking the scripts. However, I think that the file wrapping approach has advantages for visibility. Maybe it could be tweaked to keep ensure the wrapper script has the same name as the script its wrapping, e.g. when wrapping foo, replace foo with a bash script, and move the real script to .wrapped-bin/foo. pgpaurpUVVpec.pgp Description: OpenPGP digital signature
avoid wrapper scripts when possible
Hi Guix, the environment activation feature of our “conda” package currently fails. This is because the “deactivate” shell script is wrapped in another shell script. This leads to the actual shell script to be called “.deactivate-real”. The script compares the value of “$0” with the expected name “deactivate”. This fails so conda misbehaves. Since “deactivate” is really just a shell script I think we could avoid the renaming and the external wrapper by setting the environment variables in the “deactivate” script itself. This made me wonder if we could avoid shell wrappers in more cases. If the target script is a Python script we could set the environment variables right after the Python shebang with Python code. If the target script is a shell script we can set environment variables right there after the shebang. For binaries (like emacs) we’d still create shell wrappers where needed, because it’s harder to do this natively. What do you think? -- Ricardo GPG: BCA6 89B6 3655 3801 C3C6 2150 197A 5888 235F ACAC https://elephly.net