Re: Proper shebang for python3

[Eli the Bearded]

In comp.lang.python, Thomas 'PointedEars' Lahn   wrote:
> Michael Torrie wrote:
>> On 7/24/19 4:20 PM, Cameron Simpson wrote:
>>> That is some progress, hooray. Then there's just sbin -> bin to go.
>> I suppose in the olden days sbin was for static binaries, […]
> No, “sbin” is short for “*system* binaries” which in general only the 
> superuser should be able to execute.

I think Michael is confusing "sbin" with the statically linked utilities
some systems (particularly older ones, but also FreeBSD in /rescue/)
have for repairing the system when things start to go bad. You'd want
a shell (sh is great), a basic editor (eg, eg), and a smattering of
other tools, akin to the ones listed as "must be in /sbin" in your
linuxfoundation link.

But more than a few utilities in /sbin are useful for non-superusers.
Eg ip or ifconfig for informational purposes like identifying current
IP address and getting MAC.

> Which is why the above is a Very Bad Idea[tm].

Why? Programs that can *only* be usefully run by a privileged user
or in a system context (eg halt or getty) already *must* prevent non
privileged use. So why would it be a Very Bad Idea[tm] to have them in
a common directory like /bin/?

(Feel free to crosspost and set follow-ups to another group if you like.
But I would suggest *not* a Linux group, since this is something general
to all Unix-likes.)

> 

Elijah
--
uses both netbsd and linux regularly
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Proper shebang for python3

[Cameron Simpson]

On 24Jul2019 22:57, Dan Sommers <2qdxy4rzwzuui...@potatochowder.com> wrote:

On 7/24/19 10:24 PM, Michael Torrie wrote:

... In more recent times, binaries that are mostly applicable to the
super user go there.  I don't see why you would want to merge those.
A normal user rarely has need of much in /sbin.  Already /bin has way
too much stuff in it (although I don't see any other way to
practically do it without ridiculous PATHs searching all over the
disk).


On ancient file systems (SysV?), ISTR something like 1400 files in a
directory being some sort of limit, or perhaps a major performance
issue.  Separate directories (including /usr/X11/bin) mitigated that,
too.


I don't recall there being any specific hard limit though. What does 
happen in decent modern filesystems is that directories have a hash 
table in them for lookups, making a file name access constant time 
instead of a linear search.


Also, shells like zsh keep a hash table of the available commands to 
speed invocation, avoiding a linear $PATH search.


I don't recall stuff like /usr/X11/bin being aimed at performance (but I 
would not necessarily know); it has always seemed more like partitioning 
out various software groups - in this case the X11 Window System stuff.  
If nothing else, this makes separate development easier.



Having said that, I note that on my CentOS 7 workstation, sbin seems
to be in the path by default. So that negates my argument I suppose.
Although I might have made that change myself.


My .shrc file has all sorts of leftovers from the old days, but my
current Linux PATH is just $HOME/bin, $HOME/local/bin, and /usr/bin.


Hmm. 21 componetent in my $PATH here :-)

Cheers,
Cameron Simpson 
--
https://mail.python.org/mailman/listinfo/python-list

Re: Proper shebang for python3

[MRAB]

On 2019-07-25 03:57, Dan Sommers wrote:

On 7/24/19 10:24 PM, Michael Torrie wrote:

  > ... In more recent times, binaries that are mostly applicable to the
  > super user go there.  I don't see why you would want to merge those.
  > A normal user rarely has need of much in /sbin.  Already /bin has way
  > too much stuff in it (although I don't see any other way to
  > practically do it without ridiculous PATHs searching all over the
  > disk).

On ancient file systems (SysV?), ISTR something like 1400 files in a
directory being some sort of limit, or perhaps a major performance
issue.  Separate directories (including /usr/X11/bin) mitigated that,
too.


[snip]

At one time I used an MS-DOS application suite whose database component 
stored predefined queries in individual files. There were _many_ 
predefined queries, so _many_ small files.


This took up a lot of disk space, and running a predefined query was 
slow because of the need to search for the file (the FAT layout used a 
linked list).


I put all of the predefined queries into a single file and called an 
external program to look up the query on demand, copy it into a file, 
and then performed the query.


That saved a _lot_ of disk space, and, despite the overhead of calling 
an external program, was _much_ faster!

--
https://mail.python.org/mailman/listinfo/python-list

Re: Proper shebang for python3

[Dan Sommers]

On 7/24/19 10:24 PM, Michael Torrie wrote:

> ... In more recent times, binaries that are mostly applicable to the
> super user go there.  I don't see why you would want to merge those.
> A normal user rarely has need of much in /sbin.  Already /bin has way
> too much stuff in it (although I don't see any other way to
> practically do it without ridiculous PATHs searching all over the
> disk).

On ancient file systems (SysV?), ISTR something like 1400 files in a
directory being some sort of limit, or perhaps a major performance
issue.  Separate directories (including /usr/X11/bin) mitigated that,
too.

> Having said that, I note that on my CentOS 7 workstation, sbin seems
> to be in the path by default. So that negates my argument I suppose.
> Although I might have made that change myself.

My .shrc file has all sorts of leftovers from the old days, but my
current Linux PATH is just $HOME/bin, $HOME/local/bin, and /usr/bin.

Get Off My Lawn,
Dan
--
https://mail.python.org/mailman/listinfo/python-list

Re: Proper shebang for python3

[Cameron Simpson]

On 24Jul2019 20:24, Michael Torrie  wrote:

On 7/24/19 4:20 PM, Cameron Simpson wrote:

That is some progress, hooray. Then there's just sbin -> bin to go.


I suppose in the olden days sbin was for static binaries, usable in
single user mode for recovering the system without the main drive
mounted.


Yep. Happy days.


In more recent times, binaries that are mostly applicable to
the super user go there.  I don't see why you would want to merge those.
A normal user rarely has need of much in /sbin.


I say unto to you "ifconfig". And, frankly, _any_ sbin command which can 
be meaningfully run as nonroot, particularly for reporting.


I have always found this "oh its for root" distinction pretty vacuous, 
and outstandingly annoying when basic system querying stuff isn't in the 
default $PATH because of this.  Maybe it is because I've been a sysadmin 
for many years, but most physical machines are personal machines these 
days anyway - we're our own sysadmins.



Already /bin has way>too much stuff in it (although I don't see any other way 
to practically
do it without ridiculous PATHs searching all over the disk).


Like modules with many names, the number of things in /bin or /usr/bin 
is generally irrelevant. Nobody does an "ls" in there without expecting 
a fair amount of stuff - like imports, we invoke commands by name. Who 
_cares_ how many names there are?



Having said that, I note that on my CentOS 7 workstation, sbin seems to
be in the path by default. So that negates my argument I suppose.
Although I might have made that change myself.


I have historically had to add it to my $PATH on most platforms.

Cheers,
Cameron Simpson 
--
https://mail.python.org/mailman/listinfo/python-list

Re: Proper shebang for python3

[Michael Torrie]

On 7/24/19 4:20 PM, Cameron Simpson wrote:
> That is some progress, hooray. Then there's just sbin -> bin to go. 

I suppose in the olden days sbin was for static binaries, usable in
single user mode for recovering the system without the main drive
mounted.  In more recent times, binaries that are mostly applicable to
the super user go there.  I don't see why you would want to merge those.
 A normal user rarely has need of much in /sbin.  Already /bin has way
too much stuff in it (although I don't see any other way to practically
do it without ridiculous PATHs searching all over the disk).

Having said that, I note that on my CentOS 7 workstation, sbin seems to
be in the path by default. So that negates my argument I suppose.
Although I might have made that change myself.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Proper shebang for python3

[Cameron Simpson]

On 24Jul2019 21:36, Barry Scott  wrote:

On 23 Jul 2019, at 00:13, Cameron Simpson  wrote:
Why _any_ modern system has anything other than /bin in the base 
install escapes me.  In the distant past /sbin and a distinct /usr 
with its own bin had their values, but these days? Bah!


On fedora its all in /usr these days with symlinks to the old 
locations.


$ ls -l / | grep usr
lrwxrwxrwx1 root root 7 Feb 11 13:47 bin -> usr/bin/
lrwxrwxrwx1 root root 7 Feb 11 13:47 lib -> usr/lib/
lrwxrwxrwx1 root root 9 Feb 11 13:47 lib64 -> usr/lib64/
lrwxrwxrwx1 root root 8 Feb 11 13:47 sbin -> usr/sbin/
drwxr-xr-x.  13 root root  4096 May  5 17:22 usr/


That is some progress, hooray. Then there's just sbin -> bin to go. They 
could merge lib and lib64 too if they embedded an architecture signature 
in library filenames.


You can read about why here 
https://www.freedesktop.org/wiki/Software/systemd/TheCaseForTheUsrMerge/ 
and also https://fedoraproject.org/wiki/Features/UsrMove 


Thanks for these references.

Cheers,
Cameron Simpson 
--
https://mail.python.org/mailman/listinfo/python-list

Re: Proper shebang for python3

[Barry Scott]

> On 23 Jul 2019, at 00:13, Cameron Simpson  wrote:
> 
> Why _any_ modern system has anything other than /bin in the base install 
> escapes me.  In the distant past /sbin and a distinct /usr with its own bin 
> had their values, but these days? Bah!


On fedora its all in /usr these days with symlinks to the old locations.

$ ls -l / | grep usr
lrwxrwxrwx1 root root 7 Feb 11 13:47 bin -> usr/bin/
lrwxrwxrwx1 root root 7 Feb 11 13:47 lib -> usr/lib/
lrwxrwxrwx1 root root 9 Feb 11 13:47 lib64 -> usr/lib64/
lrwxrwxrwx1 root root 8 Feb 11 13:47 sbin -> usr/sbin/
drwxr-xr-x.  13 root root  4096 May  5 17:22 usr/

You can read about why here 
https://www.freedesktop.org/wiki/Software/systemd/TheCaseForTheUsrMerge/ 
 and 
also https://fedoraproject.org/wiki/Features/UsrMove 


Barry

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Proper shebang for python3

[Cameron Simpson]

On 23Jul2019 00:19, אורי  wrote:

We are using `#!/usr/bin/env python`, for example on
https://github.com/speedy-net/speedy-net/blob/master/speedy/core/manage.py

For bash we are using `#!/usr/bin/env bash`. I don't know if those are the
best but they work.


Worthwhile. Plenty of platforms do not install bash in /bin.

Though personally I only VERY RARELY want bash for a shell script.  
/bin/sh is portable and the extras in bash have relatively little value 
in scripting; by the time your script really wants them it is usually 
worth moving to a more expressive language. Like Python.


Cheers,
Cameron Simpson 
--
https://mail.python.org/mailman/listinfo/python-list

Re: Proper shebang for python3

[Cameron Simpson]

On 22Jul2019 21:14, Eli the Bearded <*@eli.users.panix.com> wrote:

In comp.lang.python, Tim Daneliuk   wrote:

On 7/20/19 1:20 PM, Chris Angelico wrote:
> On Sun, Jul 21, 2019 at 4:13 AM Michael Speer  wrote:
>> You may want to use `#!/usr/bin/env python3` instead.


I no longer have one to verify, but I recall Solaris boxen used /bin/env
not /usr/bin/env.


That is my recollection too. A pain point. To the point that I'd hand 
make a /usr/bin/env symlink.


Why _any_ modern system has anything other than /bin in the base install 
escapes me.  In the distant past /sbin and a distinct /usr with its own 
bin had their values, but these days? Bah!


(I'm not complaining about /usr/local/bin here - keeping added stuff 
distinct from the vendor/distributor stuff is very valuable.)



So, no, do NOT encode the hard location - ever.  Always use env to
discover the one that the user has specified.


But wait, you just hard coded the location of env...


Yeah. Too many boots, too many straps.


   The only exception is
/bin/sh which - for a variety of reasons - can reliably counted upon.


B! Fully half of my work porting trn4 to my cellphone was fixing all
the places that ancient build system believed /bin/sh was the name of
sh. In that environment (Termux shell on an Android phone) the location
is /data/data/com.termux/files/usr/bin/sh (and env is also in
/data/data/com.termux/files/usr/bin hahaha).


I'd do the symlink thing there too, if feasible. (Counterpoint: I've a 
DVR here where changes to / don't survive a reboot and there's no 
/etc/rc.local like file which survives a reboot either.)


POSIX systems have a /bin/sh. I would move heaven itself to ensure this.


Even on more traditional environments -cough-Solaris-cough- /bin/sh may
exist but be so ancient as to break things that work elsewhere. "^" as
a synonym for "|", is a noteworthy gotcha.


Aye. I still quote ^ in my scripts to this day for this reason. But 
then, I learnt shell programming on V7 UNIX, well older than Solaris.



Figuring out where things are on the user's path is a laudable goal, but
do it only at install time, not run time, for consistent runs.


Perhaps. Consistent runs require consistent environments. That often 
includes executables.


Cheers,
Cameron Simpson 
--
https://mail.python.org/mailman/listinfo/python-list

Re: Proper shebang for python3

[אורי]

We are using `#!/usr/bin/env python`, for example on
https://github.com/speedy-net/speedy-net/blob/master/speedy/core/manage.py

For bash we are using `#!/usr/bin/env bash`. I don't know if those are the
best but they work.

אורי
u...@speedy.net


On Sat, Jul 20, 2019 at 9:12 PM Michael Speer  wrote:

> You may want to use `#!/usr/bin/env python3` instead.
>
> There is a concept in python called the virtual environment. This used to
> be done with a tool called virtualenv in python2, and is now done mainly
> through a venv module in python3.
>
> A virtual environment goes into a directory of your choosing and will have
> its own python3 executable, and pip3 executable, and when you add
> dependencies, they are also placed into the directory structure under your
> chosen directory.
>
> When you do a `. /bin/activate` the included source will places
> the virtual environment's bin/ folder at the beginning of your PATH
> environment variable, making it the default python3 when you type it
> without a full path.
>
> This allows you to run scripts that need different, or even conflicting,
> sets of dependencies without bothering with the underlying linux
> distribution's python installation's modules.
>
> If you use `#!/usr/bin/python3`, it will always use exactly the system
> version that is installed, and the system's installed modules.
>
> Your scripts will still default to the system installation if a virtual
> environment is not activated. So you lose nothing by doing it this way, but
> gain a little control from it.
>
>
> On Sat, Jul 20, 2019 at 1:41 PM Manfred Lotz  wrote:
>
> > Hi there,
> > Pretty new to python I've got a question regarding the proper shebang
> > for Python 3.
> >
> > I use
> >#!/usr/bin/python3
> >
> > which works fine.
> >
> > Today I saw
> >#!/usr/bin/python3 -tt
> >
> > and was wondering what -tt means.
> >
> > Being on Fedora 30, Python 3.7.3 the man page of python3 doesn't even
> > mention -t.
> >
> > python 2 man page mentions
> >
> >-t Issue a warning when a source file mixes tabs and spaces
> >for indentation in a way that makes  it  depend  on  the worth
> >of a tab expressed in spaces.  Issue an error when the option is
> >given twice.
> >
> > I guess that -t has the same meaning with python 3.7.3.
> >
> >
> > My questions:
> >
> > 1. Is my guess correct?
> >
> > 2. Is it a bug that it is not mentioned? python3 --help doesn't mention
> > it either.
> >
> >
> > --
> > Manfred
> >
> >
> > --
> > https://mail.python.org/mailman/listinfo/python-list
> >
> --
> https://mail.python.org/mailman/listinfo/python-list
>
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Proper shebang for python3

[Eli the Bearded]

In comp.lang.python, Tim Daneliuk   wrote:
> On 7/20/19 1:20 PM, Chris Angelico wrote:
> > On Sun, Jul 21, 2019 at 4:13 AM Michael Speer  wrote:
> >> You may want to use `#!/usr/bin/env python3` instead.

I no longer have one to verify, but I recall Solaris boxen used /bin/env
not /usr/bin/env.

> So, no, do NOT encode the hard location - ever.  Always use env to
> discover the one that the user has specified. 

But wait, you just hard coded the location of env...

>The only exception is
> /bin/sh which - for a variety of reasons - can reliably counted upon.

B! Fully half of my work porting trn4 to my cellphone was fixing all
the places that ancient build system believed /bin/sh was the name of
sh. In that environment (Termux shell on an Android phone) the location
is /data/data/com.termux/files/usr/bin/sh (and env is also in
/data/data/com.termux/files/usr/bin hahaha).

Even on more traditional environments -cough-Solaris-cough- /bin/sh may
exist but be so ancient as to break things that work elsewhere. "^" as
a synonym for "|", is a noteworthy gotcha.

Figuring out where things are on the user's path is a laudable goal, but
do it only at install time, not run time, for consistent runs.

Elijah
--
pathological edge cases -r- us
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Proper shebang for python3

[Tim Daneliuk]

On 7/20/19 4:28 PM, Brian Oney wrote:
> Why not make a compromise? What would be a potential pitfall of the
> following spitbang?
> 
> #!python

Not sure this really changes the discussion.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Proper shebang for python3

[Cameron Simpson]

On 21Jul2019 15:47, Peter J. Holzer  wrote:

On 2019-07-21 09:04:43 +1000, Cameron Simpson wrote:
I'm with Tim Daneliuk. The environment matters and should be honoured 
except

in extremely weird cases.


I don't think that all the scripts in /usr/bin are extremely weird
cases.


I think I'd better add some nuance to my stance.

I've no problem with all the scripts shipped from an OS vendor having 
#!/usr/bin/python (or whatever fixed path) in them. They have been 
released tested against the system python and should _expect_ to run 
against it. My position here is that the entire OS distribution 
constitutes a working (and via the #! controlled) environment.


But consider third party tools. Including personal tools, but basicly 
anything from _outside_ the local system in terms of authorship.


Particularly with a language like Python which is strongly backwards 
compatible, they should generally use "#!/usr/bin/env python" (or 
python2 or python3 as appropriate, when that matters) so that they can 
run in the environment they find themselves in.


1: You can can always execute a script via a specific interpreter 
explicitly.


2: If you want to test a script it is easier to provide an environment 
that exercises the script in a particular way than to hand patch the 
shebang lines on every run/reconfig.


3: If the script (per one of Chris' examples) requires a specific python 
such as 3.6, you can always go "#!usr/bin/env python3.6" in the script 
to express the target version and provide a executable "python3.6" name 
in you environment. I keep a personal ~/bin-local directory for just 
this kind of per-host stuff myself, and of course one can do the same 
thing in places like venvs or /usr/local/bin etc. And thus _still_ leave 
the script itself without a hardwired path.


[...]
If you require a specific outcoming, set a specific environment. It 
is under

your control. Control it.

For your specific example, "man youtube-dl" _is_ affected by the
environment. It honours the $MANPATH variable.


MANPATH is explicitely intended to control man.

But man doesn't fail if you set your PATH to something weird. It will
still invoke /usr/bin/groff even if that isn't in the PATH.
(I expected that there is also an environment variable to control that
but the manpage doesn't mention one).


Heh.

I wrote my own "man" yonks ago for various reasons. Guess what? I expect 
to type "man" and get mine most of the time, but type "man" when not me 
and get /usr/bin/man (absent weirdness). That applies interactively and 
also in scripts.


Same philosophy. Use the command name to express intent and the 
environment to choose the implementation of the intent. And so also in 
the shebang lines.


For Peter J. Holzer, if we must play the "I've been doing this 
forever" game: I've been sysadmining etc longer than your 25 years and disagree with

you.


That's fine. Unlike Tim I don't claim that anybody who disagrees with me
must be a newbie.


Aye; sorry for the snarkiness. Which is why I'm disagreeing on some 
things instead of asserting that you're wrong, because you're not 
"wrong".


Cheers,
Cameron Simpson 
--
https://mail.python.org/mailman/listinfo/python-list

Re: Proper shebang for python3

[Peter J. Holzer]

On 2019-07-21 10:26:17 -0500, Tim Daneliuk wrote:
> On 7/21/19 8:47 AM, Peter J. Holzer wrote:
> > That's fine. Unlike Tim I don't claim that anybody who disagrees with me
> > must be a newbie.
> 
> Peter, that's ad hominem and unfair.

No, it isn't. Please read the first paragraph of
https://en.wikipedia.org/wiki/Ad_hominem

> I never said anything close to that.

Well, let's see what you wrote:

| We don't need to bikeshed this.

Translation: This is not a matter of opinion or taste. There is one
objectively true answer.

| All we need is people who disagree with this view to spend a year in
| software packaging, operations, deployment and DevOps ... then get back
| to us...

Translation: People who disagree with your opinion obviously haven't
worked even a single year in this field because otherwise they would
have discovered the one objectively true answer.

| Grr..

I don't have to translate that :-)


> What I said is that if someone were to spend an extended period of time
> in devops and systems engineering at large scale, they'd quickly come to
> hate hardwired paths.

You are just stating the same point again: Everybody who has experience
in the field agrees with you, therefore anybody who disagrees can't have
experience.


> The truth is that there is no single answer to this problem until you
> hermetically seal an environment.

If you can hermetically seal the environment, the argument is moot. I
thought (and still think) we are talking about the case where you can't.
Therefore I took exception to your insistence that there is in fact one
single answer and that this single answer is one which I know from
bitter experience to be quite fragile.


> But short of that, the ability to decide to use something other than
> system-provided tools absolutely IS a requirement in systems of any
> scale.

I completely agree with that. I just don't think that using
#!/usr/bin/env is a good way to achieve that.

hp

-- 
   _  | Peter J. Holzer| we build much bigger, better disasters now
|_|_) || because we have much more sophisticated
| |   | h...@hjp.at | management tools.
__/   | http://www.hjp.at/ | -- Ross Anderson 


signature.asc
Description: PGP signature
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Proper shebang for python3

[Chris Angelico]

On Mon, Jul 22, 2019 at 1:36 AM Tim Daneliuk  wrote:
>
> On 7/21/19 8:47 AM, Peter J. Holzer wrote:
> > That's fine. Unlike Tim I don't claim that anybody who disagrees with me
> > must be a newbie.
>
> Peter, that's ad hominem and unfair.  I never said anything close to that.
> What I said is that if someone were to spend an extended period of time
> in devops and systems engineering at large scale, they'd quickly come to
> hate hardwired paths.
>
> The truth is that there is no single answer to this problem until you
> hermetically seal an environment.  Docker comes close, a freestanding
> VM does this most completely.   But short of that, the ability to decide
> to use something other than system-provided tools absolutely IS a requirement
> in systems of any scale.

Your final paragraph does not justify your preceding. Yes, you need
the ability to decide to use something other than the system-provided
tool; that's why "/usr/bin/env python3" is a valid shebang. No, that
does NOT mean that people with any decent experience will "hate
hardwired paths". The hardwired path ("#!/usr/bin/python3") has its
own value. Thanks to an absolute path in its shebang, I can run "sudo
iotop" regardless of whether I have a venv active. I cannot run "sudo
python3 `which iotop`" reliably, because that command uses the
environment to look things up.

You're absolutely right that there's no single answer to the problem.
That's why script authors have the option to use any path they like,
including /usr/bin/env.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Proper shebang for python3

[Tim Daneliuk]

On 7/21/19 8:47 AM, Peter J. Holzer wrote:
> That's fine. Unlike Tim I don't claim that anybody who disagrees with me
> must be a newbie.

Peter, that's ad hominem and unfair.  I never said anything close to that.
What I said is that if someone were to spend an extended period of time
in devops and systems engineering at large scale, they'd quickly come to
hate hardwired paths.

The truth is that there is no single answer to this problem until you
hermetically seal an environment.  Docker comes close, a freestanding
VM does this most completely.   But short of that, the ability to decide
to use something other than system-provided tools absolutely IS a requirement
in systems of any scale.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Proper shebang for python3

[Peter J. Holzer]

On 2019-07-21 09:04:43 +1000, Cameron Simpson wrote:
> On 21Jul2019 08:14, Chris Angelico  wrote:
> > On Sun, Jul 21, 2019 at 5:26 AM Tim Daneliuk  wrote:
> > > So, no, do NOT encode the hard location - ever.  Always use env to
> > > discover the one that the user has specified.  The only exception
> > > is /bin/sh which - for a variety of reasons - can reliably counted
> > > upon.
> > 
> > A quick grep through my $PATH shows that there are a number of
> > executable Python scripts there, including add-apt-repository,
> > calibre, some lilypond stuff, trash-can management, samba-tool, iotop,
> > and youtube-dl. If I have a venv active with, say, Python 3.9, then
> > `/usr/bin/env python` is going to point to Python 3.9. What are the
> > odds that all those scripts will work with Python 3.9 with no
> > libraries installed? Why should typing "youtube-dl B7xai5u_tnk" be
> > affected by a virtual environment, when typing "man youtube-dl"
> > wouldn't be?? Using env for everything is a terrible idea and one that
> > will basically make virtual environments useless.
> 
> I'm with Tim Daneliuk. The environment matters and should be honoured except
> in extremely weird cases.

I don't think that all the scripts in /usr/bin are extremely weird
cases.


> If you require a specific outcoming, set a specific environment. It is under
> your control. Control it.
> 
> For your specific example, "man youtube-dl" _is_ affected by the
> environment. It honours the $MANPATH variable.

MANPATH is explicitely intended to control man.

But man doesn't fail if you set your PATH to something weird. It will
still invoke /usr/bin/groff even if that isn't in the PATH.
(I expected that there is also an environment variable to control that
but the manpage doesn't mention one).


> For Peter J. Holzer, if we must play the "I've been doing this forever"
> game: I've been sysadmining etc longer than your 25 years and disagree with
> you.

That's fine. Unlike Tim I don't claim that anybody who disagrees with me
must be a newbie.

hp

-- 
   _  | Peter J. Holzer| we build much bigger, better disasters now
|_|_) || because we have much more sophisticated
| |   | h...@hjp.at | management tools.
__/   | http://www.hjp.at/ | -- Ross Anderson 


signature.asc
Description: PGP signature
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Proper shebang for python3

[Peter J. Holzer]

On 2019-07-20 15:26:46 -0500, Tim Daneliuk wrote:
> On 7/20/19 2:56 PM, Peter J. Holzer wrote:
> > On 2019-07-20 14:11:44 -0500, Tim Daneliuk wrote:
> >> So, no, do NOT encode the hard location - ever.  Always use env to
> >> discover the one that the user has specified.  The only exception is
> >> /bin/sh which - for a variety of reasons - can reliably counted upon.
> >>
> >> We don't need to bikeshed this.  All we need is people who disagree
> >> with this view to spend a year in software packaging, operations,
> >> deployment and DevOps ... then get back to us...
> >
> > After 25 years in software packaging, operations, deployment and DevOps
> > I disagree: A program should not behave differently because of a
> > different path, #!/usr/bin/env is a total no-no.
> >
> > There is a nice way to achieve this: Just use the interpreter of the
> > virtual environment in the shebang.
> > (That requires rewriting the shebang during installation, but that's a
> > minor inconvenience)
> 
> 
> And what happens with programs that have no virtenv equivalent?
> perl, go, ruby, awk, sed, grep, etc. have no simple way to
> get installed virtual short of insisting the everything live in a
> docker container or VM?

Perl is an excellent example: When you install a Perl script which has
been packaged with one of the usual tools (e.g., Makemaker or
Module::Build), the shebang is set to the interpreter used for
installation. The user of the script doesn't have to know what
environment to use. (Perl also has perlbrew which is similar to virtual
environments, but I haven't used that much).

Go is a compiled language: The executables are binaries (no shebang) and
even statically linked. Again, the user doesn't have to care about a
language environment.

awk, grep, sed, etc. are often used in shell scripts, and they were
probably the reason why I first developed a "a program's behaviour must
not depend on the environment /except as documented/" policy: Back then
in the 1990s one usually wanted to have GNU awk, sed, grep etc. in
addition to the system utilities. Having scripts randomly fail or
(worse) produce wrong results depending on the order of /usr/bin and
/usr/local/bin in the user's PATH and the exact contents of
/usr/local/bin wasn't fun (oh, and don't forget cron).


> The fact is that most large compute environments are slow to upgrade
> the OS.  That means core tools also lag considerably behind as well.
> Being able to install newer versions along side the OS' own and then
> use them by default is manifestly necessary.

I agree. But (for me at least) it is important to do that robustly, and
#!/usr/bin/env is not robust. 

> That's why users have the ability to modify $PATH to suit their own
> needs.  All /usr/bin/env does is to tell the interpreter, "honor the
> intent of the spawning shell".

You probably just phrased that badly, but as written this is completely
wrong: /usr/bin/env doesn't tell the interpreter anything. It *chooses*
the interpreter to invoke.


> If you want really big fun, try going into an older CentOS or RedHat 
> instances and, say,
> upgrading system python to python3.

Don't do that. Install python3 somewhere else, e.g. into /usr/local.

> It's super fun.  Yes, in that case, you COULD use a venv.  But there
> are tons of other tools for which this is not an option - gcc,
> autoconf, perl, go awk, sed, bash, ad infinitum, ad nauseum are
> invariably backleveled on production OS instances.  My way fixes that
> problem.

Your way only fixes your problem only if environments where your script
may be called. You may know that /usr/local/bin/foo is a python script
that requires /usr/local/python3.8/bin/python to be in the path before
any other python interpreter and you may rememmber that in every
instance where that script is called directly or indirectly. But do your
colleagues? Especially if they aren't programmers?

And what happens if /usr/local/bin/bar requires
/usr/local/python3.4/bin/python? Do you explicitely set the PATH before
invoking each script?

By putting #!/usr/local/python3.8/bin/python at the top of
/usr/local/bin/foo and #!/usr/local/python3.4/bin/python at the top of
/usr/local/bin/bar I can just invoke both scripts. Even better, my users
can invoke both scripts and they don't even have to know they are
written in Python.


> You may have 25 years at this but I have 40 - does that make me nearly twice
> as right?  Arguments from authority are silly.

No, but it invalidates your notion that anybody with a bit of experience
would obviously use #!/usr/bin/env. I do have a bit more than one year
of experience and I think #!/usr/bin/env is a terrible idea. (I use it
for throwaway scripts which need a venv, but not for anything in
production.)

hp

-- 
   _  | Peter J. Holzer| we build much bigger, better disasters now
|_|_) || because we have much more sophisticated
| |   | h...@hjp.at | management tools.
__/   | http://www.hjp.at/ | -- Ross An

Re: Proper shebang for python3

[eryk sun]

On 7/20/19, Michael Speer  wrote:
>
> You may want to use `#!/usr/bin/env python3` instead.

This is partially supported in Windows, but only if .py files are
associated with the py.exe launcher and the shebang runs "python"
instead of "python3".

py.exe supports four builtin virtual commands: "/usr/bin/env python",
"/usr/bin/python", "/usr/local/bin/python", and "python". The "python"
command can be qualified with a version specification of the form
"X[.Y][-32|-64]", such as "python3". The "/usr/bin/env python" virtual
command searches PATH, but only if the command matches "python"
exactly. If there's a version specified, or if it's not an "env"
virtual command, the launcher looks for a registered Python
installation instead of searching PATH.

For all other commands, the launcher skips a virtual Unix prefix (i.e.
"/usr/bin/env", "/usr/bin", and "/usr/local/bin"), if present, and
searches PATH. For example, given "/usr/bin/pypy", it searches PATH
for "pypy" plus the file extensions in PATHEXT (i.e. .COM, .EXE, etc).
If the command isn't found in PATH, it checks in the "[commands]"
section of "%LocalAppData%\py.ini". For example, py.ini could define
"pypy=C:\PyPy71\pypy.exe".

A virtual environment should have a "python" executable, so shebangs
that use "/usr/bin/env python" will prefer an active virtual
environment. But a lot of scripts use "python3" instead of "python".
This is a stumbling block in Windows since we don't install versioned
"pythonX[.Y].exe" binaries. Thus the "env" virtual command is special
cased to find a qualified Python version in the list of registered
Python installations instead of searching PATH. (The new store app
distribution does install versioned app links, but, even in this case,
virtual environments lack versioned binaries in the "Scripts"
directory.)

That said, some developers manually create versioned binaries as
symlinks, hardlinks, or copies. Also, maybe a future version of the
full (non-app) Python installation will do the same. Thus I think the
launcher should search PATH for all "/usr/bin/env python*" shebangs,
but without the binary-type specification (i.e. "-32" or "-64") if
present. In this case, instead of a regular search based on WINAPI
SearchPath, we'll need a custom search that tokenizes and walks PATH
and checks the binary type via GetBinaryTypeW. If no matching version
is found in PATH, the "env" search should fall back on the list of
registered versions.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Proper shebang for python3

[Brian Oney via Python-list]

On July 21, 2019 10:04:47 AM GMT+02:00, Manfred Lotz  wrote:
>On Sun, 21 Jul 2019 10:21:55 +1000
>Cameron Simpson  wrote:
>
>> On 21Jul2019 09:31, Chris Angelico  wrote:
>> >On Sun, Jul 21, 2019 at 9:15 AM Cameron Simpson 
>> >wrote: So you mean that a tool that depends on running on a
>> >consistent environment, it should use a shebang of
>> >"/usr/bin/python3.6" instead of "/usr/bin/env python3"?  
>> 
>> Jeez. No. That is the _opposite_ of what I'm saying.
>> 
>> >Because, wow, that would be exactly what is
>> >already happening on my system. Why use /usr/bin/env and then wrap
>> >something around it to force the environment, when you could just
>set
>> >a correct shebang so it properly defines its execution environment? 
>
>> 
>> Because the shebang is hardwired and inflexible.
>> 
>> Because it means hand patching (even scripted) a bazillion scripts to
>
>> that they know their physical install.
>> 
>> Because it presumes detailed hardwired knowledge of the target system
>> in a script which should work anywhere.
>> 
>> Instead a tiny _common_ shell script resembling this:
>> 
>>   #!/bin/sh
>>   # Run command in the official environment.
>>   exec env - PATH=/path/to/3.6venv/bin:/usr/sbin:/bin exec ${1+"$@"}
>> 
>> arranges things. The "env -" is aimed at "clean" daemon or install 
>> environments.  You can do subtler or less intrusive things in other 
>> settings.
>> 
>
>I took a look and found that Fedora 30 and Debian Jessie both use
>hard-wired paths for python in the rpm resp. deb packages.
>
>I'm being new to Python and I am not acquainted in any way with
>virtualenv resp. venv so cannot currently judge its pro and cons.
>
>So I will stick to:
>   #!/usr/bin/env python3 
>
>as shebang for my scripts.

I think that's a good decision. Most of the conversation applies to sysadmins 
concerned with answering your question in absolute terms. When you start 
writing scripts which are restricted to a specific environment and are intended 
to be distributed, you may revisit this thread.

Be blissful until then :). Chris et al have "fixed" things for you.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Proper shebang for python3

[Manfred Lotz]

On Sun, 21 Jul 2019 10:21:55 +1000
Cameron Simpson  wrote:

> On 21Jul2019 09:31, Chris Angelico  wrote:
> >On Sun, Jul 21, 2019 at 9:15 AM Cameron Simpson 
> >wrote: So you mean that a tool that depends on running on a
> >consistent environment, it should use a shebang of
> >"/usr/bin/python3.6" instead of "/usr/bin/env python3"?  
> 
> Jeez. No. That is the _opposite_ of what I'm saying.
> 
> >Because, wow, that would be exactly what is
> >already happening on my system. Why use /usr/bin/env and then wrap
> >something around it to force the environment, when you could just set
> >a correct shebang so it properly defines its execution environment?  
> 
> Because the shebang is hardwired and inflexible.
> 
> Because it means hand patching (even scripted) a bazillion scripts to 
> that they know their physical install.
> 
> Because it presumes detailed hardwired knowledge of the target system
> in a script which should work anywhere.
> 
> Instead a tiny _common_ shell script resembling this:
> 
>   #!/bin/sh
>   # Run command in the official environment.
>   exec env - PATH=/path/to/3.6venv/bin:/usr/sbin:/bin exec ${1+"$@"}
> 
> arranges things. The "env -" is aimed at "clean" daemon or install 
> environments.  You can do subtler or less intrusive things in other 
> settings.
> 

I took a look and found that Fedora 30 and Debian Jessie both use
hard-wired paths for python in the rpm resp. deb packages.

I'm being new to Python and I am not acquainted in any way with
virtualenv resp. venv so cannot currently judge its pro and cons.

So I will stick to:
   #!/usr/bin/env python3 

as shebang for my scripts.


-- 
Manfred


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Proper shebang for python3

[Barry Scott]

> On 21 Jul 2019, at 08:29, Christian Gollwitzer  wrote:
> 
> Am 21.07.19 um 07:31 schrieb Tim Daneliuk:
>> On 7/20/19 6:04 PM, Chris Angelico wrote:
>>> Are you aware of every systemwide command that happens to be
>>> implemented in Python, such that you won't execute any of them while
>>> you have the venv active?
>> No, but this has never been a problem because the newer versions of
>> python tend to be pretty good - within a major release tree - of being
>> backward compatible.  Certainly, if I were running, say, RedHat 4 with
>> a very new version of python in my path first, this could theoretically
>> be an issue.  I've just not run into it.
> 
> It's not about the core language, but these system tools depend on packages, 
> sometimes specialist packages, which might not be available in your venv.

Well said.

The fedora packaging system replaces all shebang lines with the full path to 
either python2 or python3 on the system
to make sure that scripts installed in the system run as intended and do not 
break because of the users PATH.

How you think about the shebang lines depends on where the scripts will be used.

If its as an installed system component the exact path is usually the right 
thing to do.

If it you personal scripts then you may well find /usr/bin/env python3 makes 
you life
simpler.

For code I'm testing I usual do not depend no shebang lines at all. Often I'm 
testing
on many python versions. And end up with something like this to test on all 
interesting python
versions.

for PTYHON in python3.6 python3.7 python3.8
do
$PYTHON my_script.py
done

Barry
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Proper shebang for python3

[Christian Gollwitzer]

Am 21.07.19 um 07:31 schrieb Tim Daneliuk:

On 7/20/19 6:04 PM, Chris Angelico wrote:

Are you aware of every systemwide command that happens to be
implemented in Python, such that you won't execute any of them while
you have the venv active?


No, but this has never been a problem because the newer versions of
python tend to be pretty good - within a major release tree - of being
backward compatible.  Certainly, if I were running, say, RedHat 4 with
a very new version of python in my path first, this could theoretically
be an issue.  I've just not run into it.



It's not about the core language, but these system tools depend on 
packages, sometimes specialist packages, which might not be available in 
your venv.


Christian

--
https://mail.python.org/mailman/listinfo/python-list

Re: Proper shebang for python3

[Chris Angelico]

On Sun, Jul 21, 2019 at 3:41 PM Tim Daneliuk  wrote:
>
> On 7/20/19 6:04 PM, Chris Angelico wrote:
> > Are you aware of every systemwide command that happens to be
> > implemented in Python, such that you won't execute any of them while
> > you have the venv active?
>
> No, but this has never been a problem because the newer versions of
> python tend to be pretty good - within a major release tree - of being
> backward compatible.  Certainly, if I were running, say, RedHat 4 with
> a very new version of python in my path first, this could theoretically
> be an issue.  I've just not run into it.
>

Many Python packages are incompatible with CPython 3.9 at the moment. MANY.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Proper shebang for python3

[Tim Daneliuk]

On 7/20/19 6:04 PM, Cameron Simpson wrote:
> If you require a specific outcoming, set a specific environment. It is under 
> your control. Control it.

Exactly right.  I have just had the REALLY irritating experience of trying to 
bootstrap a
location insensitive version of linuxbrew that mostly works, but is crippled by 
brain damage
that insists that the best version of perl will always be found in /usr/bin.

I have been a hardware engineer (analog and digital), software implementor, 
systems designer,
devops person, and large scale (physical data center) platform engineer and 
this kind of
bad thinking just kills site reliability.

I stipulate that there are corner cases where Chris A. is correct - it is 
certainly
possible to clobber a system with incorrect /usr/bin/env findings.   But I'd 
argue
that there is a simple work around - run another terminal session (terminator, 
tmux,
screen, ... whatever floats your boat) that has $PATH set up to find things in 
/usr/bin
first.  Voila! Problem solved and everyone can use env the way they want to.

In some respects, this problem does get a little simpler when you docker-ize 
your
software distributions until ... you need newer versions of tools than even the 
latest
docker OS implementations support.  Then you're back to the same old noise ...
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Proper shebang for python3

[Tim Daneliuk]

On 7/20/19 6:04 PM, Chris Angelico wrote:
> Are you aware of every systemwide command that happens to be
> implemented in Python, such that you won't execute any of them while
> you have the venv active?

No, but this has never been a problem because the newer versions of
python tend to be pretty good - within a major release tree - of being
backward compatible.  Certainly, if I were running, say, RedHat 4 with
a very new version of python in my path first, this could theoretically
be an issue.  I've just not run into it.


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Proper shebang for python3

[Manfred Lotz]

On Sat, 20 Jul 2019 23:28:35 +0200
Brian Oney  wrote:

> Why not make a compromise? What would be a potential pitfall of the
> following spitbang?
> 
> #!python

I think that per definition a path in a shebang has to be absolute.

Actually, your suggestion won't work for people who use the fish
shell (just for the given reason).


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Proper shebang for python3

[Cameron Simpson]

On 21Jul2019 09:31, Chris Angelico  wrote:

On Sun, Jul 21, 2019 at 9:15 AM Cameron Simpson  wrote:
So you mean that a tool that depends on running on a consistent
environment, it should use a shebang of "/usr/bin/python3.6" instead
of "/usr/bin/env python3"?


Jeez. No. That is the _opposite_ of what I'm saying.


Because, wow, that would be exactly what is
already happening on my system. Why use /usr/bin/env and then wrap
something around it to force the environment, when you could just set
a correct shebang so it properly defines its execution environment?


Because the shebang is hardwired and inflexible.

Because it means hand patching (even scripted) a bazillion scripts to 
that they know their physical install.


Because it presumes detailed hardwired knowledge of the target system in 
a script which should work anywhere.


Instead a tiny _common_ shell script resembling this:

 #!/bin/sh
 # Run command in the official environment.
 exec env - PATH=/path/to/3.6venv/bin:/usr/sbin:/bin exec ${1+"$@"}

arranges things. The "env -" is aimed at "clean" daemon or install 
environments.  You can do subtler or less intrusive things in other 
settings.


There are any number of variations available here that don't hardwire 
the environment into the various _scripts_, but into the invocation 
setting. Or just start an environment (eg a shell) with the desired 
environment.


Cheers,
Cameron Simpson 
--
https://mail.python.org/mailman/listinfo/python-list

Re: Proper shebang for python3

[Chris Angelico]

On Sun, Jul 21, 2019 at 9:15 AM Cameron Simpson  wrote:
>
> On 21Jul2019 08:14, Chris Angelico  wrote:
> >On Sun, Jul 21, 2019 at 5:26 AM Tim Daneliuk  wrote:
> >> So, no, do NOT encode the hard location - ever.  Always use env to 
> >> discover the one that
> >> the user has specified.  The only exception is /bin/sh which - for a 
> >> variety of reasons -
> >> can reliably counted upon.
> >
> >A quick grep through my $PATH shows that there are a number of
> >executable Python scripts there, including add-apt-repository,
> >calibre, some lilypond stuff, trash-can management, samba-tool, iotop,
> >and youtube-dl. If I have a venv active with, say, Python 3.9, then
> >`/usr/bin/env python` is going to point to Python 3.9. What are the
> >odds that all those scripts will work with Python 3.9 with no
> >libraries installed? Why should typing "youtube-dl B7xai5u_tnk" be
> >affected by a virtual environment, when typing "man youtube-dl"
> >wouldn't be?? Using env for everything is a terrible idea and one that
> >will basically make virtual environments useless.
>
> I'm with Tim Daneliuk. The environment matters and should be honoured
> except in extremely weird cases.
>
> If you require a specific outcoming, set a specific environment. It is
> under your control. Control it.
>
> For your specific example, "man youtube-dl" _is_ affected by the
> environment. It honours the $MANPATH variable.
>
> For Peter J. Holzer, if we must play the "I've been doing this forever"
> game: I've been sysadmining etc longer than your 25 years and disagree
> with you. If your tools are strongly affected by version, run them with
> an environment that finds the right version.
>
> Installers, particularly those run as root or in other low level setup
> situations, should have a WELL DEFINED environment.
>
> If you have a tool that isn't portable, put it in a wrapper that coddles
> its behaviour.

So you mean that a tool that depends on running on a consistent
environment, it should use a shebang of "/usr/bin/python3.6" instead
of "/usr/bin/env python3"? Because, wow, that would be exactly what is
already happening on my system. Why use /usr/bin/env and then wrap
something around it to force the environment, when you could just set
a correct shebang so it properly defines its execution environment?

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Proper shebang for python3

[Cameron Simpson]

On 21Jul2019 08:14, Chris Angelico  wrote:

On Sun, Jul 21, 2019 at 5:26 AM Tim Daneliuk  wrote:

So, no, do NOT encode the hard location - ever.  Always use env to discover the 
one that
the user has specified.  The only exception is /bin/sh which - for a variety of 
reasons -
can reliably counted upon.


A quick grep through my $PATH shows that there are a number of
executable Python scripts there, including add-apt-repository,
calibre, some lilypond stuff, trash-can management, samba-tool, iotop,
and youtube-dl. If I have a venv active with, say, Python 3.9, then
`/usr/bin/env python` is going to point to Python 3.9. What are the
odds that all those scripts will work with Python 3.9 with no
libraries installed? Why should typing "youtube-dl B7xai5u_tnk" be
affected by a virtual environment, when typing "man youtube-dl"
wouldn't be?? Using env for everything is a terrible idea and one that
will basically make virtual environments useless.


I'm with Tim Daneliuk. The environment matters and should be honoured 
except in extremely weird cases.


If you require a specific outcoming, set a specific environment. It is 
under your control. Control it.


For your specific example, "man youtube-dl" _is_ affected by the 
environment. It honours the $MANPATH variable.


For Peter J. Holzer, if we must play the "I've been doing this forever" 
game: I've been sysadmining etc longer than your 25 years and disagree 
with you. If your tools are strongly affected by version, run them with 
an environment that finds the right version.


Installers, particularly those run as root or in other low level setup 
situations, should have a WELL DEFINED environment.


If you have a tool that isn't portable, put it in a wrapper that coddles 
its behaviour.


Cheers,
Cameron Simpson 
--
https://mail.python.org/mailman/listinfo/python-list

Re: Proper shebang for python3

[Tim Daneliuk]

On 7/20/19 5:47 PM, Tim Daneliuk wrote:
> On 7/20/19 5:14 PM, Chris Angelico wrote:
>> Using env for everything is a terrible idea and one that
>> will basically make virtual environments useless.
> 
> Not if you manage them properly.
> 
> Everyone's mileage is different, but when I enter a venv, I ensure everything 
> I do
> there is packaged to work together.  I pip install things in there and ensure
> that the tools I invoke will work in that environment.
> 
> In the example you cite, I would simply run the tools in question outside any
> environment where I know they don't work.
> 

I guess I should clarify something:

1) I pip install everything locally inside my own $HOME for that set of things 
I need
   generally and will not be using in a venv.

2) I pip install everything I need in each venv, even if it already exists 
under 1) above.

I effectively treat venvs as fully self-contained environments to the degree 
possible.


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Proper shebang for python3

[Chris Angelico]

On Sun, Jul 21, 2019 at 8:56 AM Tim Daneliuk  wrote:
>
> On 7/20/19 5:14 PM, Chris Angelico wrote:
> > Using env for everything is a terrible idea and one that
> > will basically make virtual environments useless.
>
> Not if you manage them properly.
>
> Everyone's mileage is different, but when I enter a venv, I ensure everything 
> I do
> there is packaged to work together.  I pip install things in there and ensure
> that the tools I invoke will work in that environment.
>
> In the example you cite, I would simply run the tools in question outside any
> environment where I know they don't work.

Are you aware of every systemwide command that happens to be
implemented in Python, such that you won't execute any of them while
you have the venv active? The ones I mentioned are all installed into
/usr/local/bin or /usr/bin or some other directory on my main $PATH. I
only know they're Python scripts because I just ran "file
/usr/local/bin/*" and had it check them all. If they all reacted to
the activation of a venv, I would have to be extremely careful of
which commands I ever run while a venv is active. Which basically
means I would never activate a venv ever again - not worth it.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Proper shebang for python3

[Tim Daneliuk]

On 7/20/19 5:14 PM, Chris Angelico wrote:
> Using env for everything is a terrible idea and one that
> will basically make virtual environments useless.

Not if you manage them properly.

Everyone's mileage is different, but when I enter a venv, I ensure everything I 
do
there is packaged to work together.  I pip install things in there and ensure
that the tools I invoke will work in that environment.

In the example you cite, I would simply run the tools in question outside any
environment where I know they don't work.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Proper shebang for python3

[Chris Angelico]

On Sun, Jul 21, 2019 at 5:26 AM Tim Daneliuk  wrote:
> So, no, do NOT encode the hard location - ever.  Always use env to discover 
> the one that
> the user has specified.  The only exception is /bin/sh which - for a variety 
> of reasons -
> can reliably counted upon.

A quick grep through my $PATH shows that there are a number of
executable Python scripts there, including add-apt-repository,
calibre, some lilypond stuff, trash-can management, samba-tool, iotop,
and youtube-dl. If I have a venv active with, say, Python 3.9, then
`/usr/bin/env python` is going to point to Python 3.9. What are the
odds that all those scripts will work with Python 3.9 with no
libraries installed? Why should typing "youtube-dl B7xai5u_tnk" be
affected by a virtual environment, when typing "man youtube-dl"
wouldn't be?? Using env for everything is a terrible idea and one that
will basically make virtual environments useless.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Proper shebang for python3

[Brian Oney via Python-list]

On Sat, 2019-07-20 at 15:26 -0500, Tim Daneliuk wrote:
> On 7/20/19 2:56 PM, Peter J. Holzer wrote:
> > On 2019-07-20 14:11:44 -0500, Tim Daneliuk wrote:
> > > So, no, do NOT encode the hard location - ever.  Always use env to
> > > discover the one that the user has specified.  The only exception is
> > > /bin/sh which - for a variety of reasons - can reliably counted upon.
> > > 
> > > We don't need to bikeshed this.  All we need is people who disagree
> > > with this view to spend a year in software packaging, operations,
> > > deployment and DevOps ... then get back to us...
> > 
> > After 25 years in software packaging, operations, deployment and DevOps
> > I disagree: A program should not behave differently because of a
> > different path, #!/usr/bin/env is a total no-no.
> > 
> > There is a nice way to achieve this: Just use the interpreter of the
> > virtual environment in the shebang.
> > (That requires rewriting the shebang during installation, but that's a
> > minor inconvenience)
> > 
> > hp
> > 
> 
> 
> And what happens with programs that have no virtenv equivalent?
> perl, go, ruby, awk, sed, grep, etc. have no simple way to
> get installed virtual short of insisting the everything live in a
> docker container or VM?
> 
> The fact is that most large compute environments are slow to upgrade
> the OS.  That means core tools also lag considerably behind as well.
> Being able to install newer versions along side the OS' own and then
> use them by default is manifestly necessary.  That's why users have
> the ability to modify $PATH to suit their own needs.  All /usr/bin/env
> does is to tell the interpreter, "honor the intent of the spawning shell".
> This shouldn't even be a question ... and it's why so much garbage continues
> to live forever.  Having to constantly code around old systems versions of
> tools with not other recourse is just crazy.
> 
> In actual fact, the very best way to write portable, reliable, and operable
> systems of code is to divorce them entirely (or as a nearly as possible) for
> the OS tools as you can.  That's why docker works so well.  That's why go
> avoids dynamic linking.
> 
> In my case, that's why I compile my own version of
> languages and daily use utilities to live outside the OS.  I get absolutely
> predicable behavior irrespective of my distro and whatever backleveled cruft
> it has laying around.   I _know_ every version of every important tool my code
> will be using ... all by setting up $PATH properly and using /usr/bin/env in
> my interpreters.
> 
> If you want really big fun, try going into an older CentOS or RedHat 
> instances and, say,
> upgrading system python to python3.  It's super fun.  Yes, in that case, you 
> COULD
> use a venv.  But there are tons of other tools for which this is not an 
> option -
> gcc, autoconf, perl, go awk, sed, bash, ad infinitum, ad nauseum are 
> invariably
> backleveled on production OS instances.  My way fixes that problem.  Your way
> forces me to code around it ... which is a way worse solution.
> 
> You may have 25 years at this but I have 40 - does that make me nearly twice
> as right?  Arguments from authority are silly.
> 
> P.S. https://www.tundraware.com/TechnicalNotes/Divorce-Your-Linux-Admin/
Nice.

Emacs (well spacemacs) is my authority.

The key sequence 'SPC i !' inserts

#!/usr/bin/env python

as specified in the insert-shebang package 
https://github.com/psachin/insert-shebang

My limited experience tells me to expect the user (me) to know what
they're doing, hence env.

Why not make a compromise? What would be a potential pitfall of the
following spitbang?

#!python








-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Proper shebang for python3

[Tim Daneliuk]

On 7/20/19 2:56 PM, Peter J. Holzer wrote:
> On 2019-07-20 14:11:44 -0500, Tim Daneliuk wrote:
>> So, no, do NOT encode the hard location - ever.  Always use env to
>> discover the one that the user has specified.  The only exception is
>> /bin/sh which - for a variety of reasons - can reliably counted upon.
>>
>> We don't need to bikeshed this.  All we need is people who disagree
>> with this view to spend a year in software packaging, operations,
>> deployment and DevOps ... then get back to us...
>
> After 25 years in software packaging, operations, deployment and DevOps
> I disagree: A program should not behave differently because of a
> different path, #!/usr/bin/env is a total no-no.
>
> There is a nice way to achieve this: Just use the interpreter of the
> virtual environment in the shebang.
> (That requires rewriting the shebang during installation, but that's a
> minor inconvenience)
>
> hp
>



And what happens with programs that have no virtenv equivalent?
perl, go, ruby, awk, sed, grep, etc. have no simple way to
get installed virtual short of insisting the everything live in a
docker container or VM?

The fact is that most large compute environments are slow to upgrade
the OS.  That means core tools also lag considerably behind as well.
Being able to install newer versions along side the OS' own and then
use them by default is manifestly necessary.  That's why users have
the ability to modify $PATH to suit their own needs.  All /usr/bin/env
does is to tell the interpreter, "honor the intent of the spawning shell".
This shouldn't even be a question ... and it's why so much garbage continues
to live forever.  Having to constantly code around old systems versions of
tools with not other recourse is just crazy.

In actual fact, the very best way to write portable, reliable, and operable
systems of code is to divorce them entirely (or as a nearly as possible) for
the OS tools as you can.  That's why docker works so well.  That's why go
avoids dynamic linking.

In my case, that's why I compile my own version of
languages and daily use utilities to live outside the OS.  I get absolutely
predicable behavior irrespective of my distro and whatever backleveled cruft
it has laying around.   I _know_ every version of every important tool my code
will be using ... all by setting up $PATH properly and using /usr/bin/env in
my interpreters.

If you want really big fun, try going into an older CentOS or RedHat instances 
and, say,
upgrading system python to python3.  It's super fun.  Yes, in that case, you 
COULD
use a venv.  But there are tons of other tools for which this is not an option -
gcc, autoconf, perl, go awk, sed, bash, ad infinitum, ad nauseum are invariably
backleveled on production OS instances.  My way fixes that problem.  Your way
forces me to code around it ... which is a way worse solution.

You may have 25 years at this but I have 40 - does that make me nearly twice
as right?  Arguments from authority are silly.

P.S. https://www.tundraware.com/TechnicalNotes/Divorce-Your-Linux-Admin/
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Proper shebang for python3

[Peter J. Holzer]

On 2019-07-20 14:11:44 -0500, Tim Daneliuk wrote:
> So, no, do NOT encode the hard location - ever.  Always use env to
> discover the one that the user has specified.  The only exception is
> /bin/sh which - for a variety of reasons - can reliably counted upon.
> 
> We don't need to bikeshed this.  All we need is people who disagree
> with this view to spend a year in software packaging, operations,
> deployment and DevOps ... then get back to us...

After 25 years in software packaging, operations, deployment and DevOps
I disagree: A program should not behave differently because of a
different path, #!/usr/bin/env is a total no-no.

There is a nice way to achieve this: Just use the interpreter of the
virtual environment in the shebang.
(That requires rewriting the shebang during installation, but that's a
minor inconvenience)

hp

-- 
   _  | Peter J. Holzer| we build much bigger, better disasters now
|_|_) || because we have much more sophisticated
| |   | h...@hjp.at | management tools.
__/   | http://www.hjp.at/ | -- Ross Anderson 


signature.asc
Description: PGP signature
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Proper shebang for python3

[Manfred Lotz]

On Sat, 20 Jul 2019 14:11:21 -0400
Michael Speer  wrote:

> You may want to use `#!/usr/bin/env python3` instead.
> 

In my case it doesn't matter. However, I agree that your suggestion is
usually preferable.

> There is a concept in python called the virtual environment. This
> used to be done with a tool called virtualenv in python2, and is now
> done mainly through a venv module in python3.
> 
> A virtual environment goes into a directory of your choosing and will
> have its own python3 executable, and pip3 executable, and when you add
> dependencies, they are also placed into the directory structure under
> your chosen directory.
> 
> When you do a `. /bin/activate` the included source will
> places the virtual environment's bin/ folder at the beginning of your
> PATH environment variable, making it the default python3 when you
> type it without a full path.
> 
> This allows you to run scripts that need different, or even
> conflicting, sets of dependencies without bothering with the
> underlying linux distribution's python installation's modules.
> 

New to me. Interesting. 

> If you use `#!/usr/bin/python3`, it will always use exactly the system
> version that is installed, and the system's installed modules.
> 
> Your scripts will still default to the system installation if a
> virtual environment is not activated. So you lose nothing by doing it
> this way, but gain a little control from it.
> 

Ok, understood.


-- 
Manfred


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Proper shebang for python3

[Tim Daneliuk]

On 7/20/19 1:20 PM, Chris Angelico wrote:
> On Sun, Jul 21, 2019 at 4:13 AM Michael Speer  wrote:
>>
>> You may want to use `#!/usr/bin/env python3` instead.
>>
>> There is a concept in python called the virtual environment. This used to
>> be done with a tool called virtualenv in python2, and is now done mainly
>> through a venv module in python3.
>>
>> A virtual environment goes into a directory of your choosing and will have
>> its own python3 executable, and pip3 executable, and when you add
>> dependencies, they are also placed into the directory structure under your
>> chosen directory.
>>
>> When you do a `. /bin/activate` the included source will places
>> the virtual environment's bin/ folder at the beginning of your PATH
>> environment variable, making it the default python3 when you type it
>> without a full path.
>>
>> This allows you to run scripts that need different, or even conflicting,
>> sets of dependencies without bothering with the underlying linux
>> distribution's python installation's modules.
>>
>> If you use `#!/usr/bin/python3`, it will always use exactly the system
>> version that is installed, and the system's installed modules.
>>
>> Your scripts will still default to the system installation if a virtual
>> environment is not activated. So you lose nothing by doing it this way, but
>> gain a little control from it.
>>
> 
> ONLY if you actually want this script to behave differently inside a
> venv. When you're setting a shebang on a script, you often do not want
> that. You potentially LOSE a lot of control.
> 
> ChrisA

Disagree strongly.  The environment should always define where you want to find 
key
binaries, whether in a venv or not.  There are many use cases where you want to
override system binaries.  For example, you may be running on an older OS 
release
and want to point to a newer one installed elsewhere.  Similarly, the DevOps 
workflow
may demand particular configurations for the pipelines to run properly.

I have spent way too much time in my career trying to undo this kind of 
imperious
programming that thinks the coder knows best, when in the actual runtime 
environment,
they actually don't.  Most recently, I have been working to produce a 
configuration
for linuxbrew that can be installed by an unprivileged user at any location 
they like.
It is maddening to spend hours getting things working only to find out that 
some genius
decided that the only version of, say, perl or autoconf, is to be found 
exclusively in
/usr/bin.

So, no, do NOT encode the hard location - ever.  Always use env to discover the 
one that
the user has specified.  The only exception is /bin/sh which - for a variety of 
reasons -
can reliably counted upon.

We don't need to bikeshed this.  All we need is people who disagree with this 
view to spend
a year in software packaging, operations, deployment and DevOps ... then get 
back to us...

Grr..

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Proper shebang for python3

[Chris Angelico]

On Sun, Jul 21, 2019 at 4:13 AM Michael Speer  wrote:
>
> You may want to use `#!/usr/bin/env python3` instead.
>
> There is a concept in python called the virtual environment. This used to
> be done with a tool called virtualenv in python2, and is now done mainly
> through a venv module in python3.
>
> A virtual environment goes into a directory of your choosing and will have
> its own python3 executable, and pip3 executable, and when you add
> dependencies, they are also placed into the directory structure under your
> chosen directory.
>
> When you do a `. /bin/activate` the included source will places
> the virtual environment's bin/ folder at the beginning of your PATH
> environment variable, making it the default python3 when you type it
> without a full path.
>
> This allows you to run scripts that need different, or even conflicting,
> sets of dependencies without bothering with the underlying linux
> distribution's python installation's modules.
>
> If you use `#!/usr/bin/python3`, it will always use exactly the system
> version that is installed, and the system's installed modules.
>
> Your scripts will still default to the system installation if a virtual
> environment is not activated. So you lose nothing by doing it this way, but
> gain a little control from it.
>

ONLY if you actually want this script to behave differently inside a
venv. When you're setting a shebang on a script, you often do not want
that. You potentially LOSE a lot of control.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Proper shebang for python3

[Michael Speer]

You may want to use `#!/usr/bin/env python3` instead.

There is a concept in python called the virtual environment. This used to
be done with a tool called virtualenv in python2, and is now done mainly
through a venv module in python3.

A virtual environment goes into a directory of your choosing and will have
its own python3 executable, and pip3 executable, and when you add
dependencies, they are also placed into the directory structure under your
chosen directory.

When you do a `. /bin/activate` the included source will places
the virtual environment's bin/ folder at the beginning of your PATH
environment variable, making it the default python3 when you type it
without a full path.

This allows you to run scripts that need different, or even conflicting,
sets of dependencies without bothering with the underlying linux
distribution's python installation's modules.

If you use `#!/usr/bin/python3`, it will always use exactly the system
version that is installed, and the system's installed modules.

Your scripts will still default to the system installation if a virtual
environment is not activated. So you lose nothing by doing it this way, but
gain a little control from it.


On Sat, Jul 20, 2019 at 1:41 PM Manfred Lotz  wrote:

> Hi there,
> Pretty new to python I've got a question regarding the proper shebang
> for Python 3.
>
> I use
>#!/usr/bin/python3
>
> which works fine.
>
> Today I saw
>#!/usr/bin/python3 -tt
>
> and was wondering what -tt means.
>
> Being on Fedora 30, Python 3.7.3 the man page of python3 doesn't even
> mention -t.
>
> python 2 man page mentions
>
>-t Issue a warning when a source file mixes tabs and spaces
>for indentation in a way that makes  it  depend  on  the worth
>of a tab expressed in spaces.  Issue an error when the option is
>given twice.
>
> I guess that -t has the same meaning with python 3.7.3.
>
>
> My questions:
>
> 1. Is my guess correct?
>
> 2. Is it a bug that it is not mentioned? python3 --help doesn't mention
> it either.
>
>
> --
> Manfred
>
>
> --
> https://mail.python.org/mailman/listinfo/python-list
>
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Proper shebang for python3

[Manfred Lotz]

On Sun, 21 Jul 2019 03:44:24 +1000
Chris Angelico  wrote:

> On Sun, Jul 21, 2019 at 3:41 AM Manfred Lotz 
> wrote:
> >
> > Hi there,
> > Pretty new to python I've got a question regarding the proper
> > shebang for Python 3.
> >
> > I use
> >#!/usr/bin/python3
> >
> > which works fine.
> >
> > Today I saw
> >#!/usr/bin/python3 -tt
> >
> > and was wondering what -tt means.
> >
> > Being on Fedora 30, Python 3.7.3 the man page of python3 doesn't
> > even mention -t.
> >
> > python 2 man page mentions
> >
> >-t Issue a warning when a source file mixes tabs and
> > spaces for indentation in a way that makes  it  depend  on  the
> > worth of a tab expressed in spaces.  Issue an error when the option
> > is given twice.
> >
> > I guess that -t has the same meaning with python 3.7.3.  
> 
> Far as I know, the -tt option to Python 3 is accepted for 2/3
> compatibility, but does absolutely nothing. Having it on the shebang
> probably means that it used to be /usr/bin/python -tt, and got
> migrated to python3 without removing it.
> 

This is a plausible explanation.

Thanks, Manfred




-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Proper shebang for python3

[Chris Angelico]

On Sun, Jul 21, 2019 at 3:41 AM Manfred Lotz  wrote:
>
> Hi there,
> Pretty new to python I've got a question regarding the proper shebang
> for Python 3.
>
> I use
>#!/usr/bin/python3
>
> which works fine.
>
> Today I saw
>#!/usr/bin/python3 -tt
>
> and was wondering what -tt means.
>
> Being on Fedora 30, Python 3.7.3 the man page of python3 doesn't even
> mention -t.
>
> python 2 man page mentions
>
>-t Issue a warning when a source file mixes tabs and spaces
>for indentation in a way that makes  it  depend  on  the worth
>of a tab expressed in spaces.  Issue an error when the option is
>given twice.
>
> I guess that -t has the same meaning with python 3.7.3.

Far as I know, the -tt option to Python 3 is accepted for 2/3
compatibility, but does absolutely nothing. Having it on the shebang
probably means that it used to be /usr/bin/python -tt, and got
migrated to python3 without removing it.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Proper shebang for python3

[Manfred Lotz]

Hi there,
Pretty new to python I've got a question regarding the proper shebang
for Python 3.

I use 
   #!/usr/bin/python3

which works fine.

Today I saw 
   #!/usr/bin/python3 -tt

and was wondering what -tt means.

Being on Fedora 30, Python 3.7.3 the man page of python3 doesn't even
mention -t. 

python 2 man page mentions

   -t Issue a warning when a source file mixes tabs and spaces
   for indentation in a way that makes  it  depend  on  the worth
   of a tab expressed in spaces.  Issue an error when the option is
   given twice.

I guess that -t has the same meaning with python 3.7.3. 


My questions:

1. Is my guess correct?

2. Is it a bug that it is not mentioned? python3 --help doesn't mention
it either.


-- 
Manfred


-- 
https://mail.python.org/mailman/listinfo/python-list