Re: Friday Filosofical Finking: Import protections
On Fri, Apr 19, 2019 at 3:27 AM Akkana Peck wrote: > > Chris Angelico writes: > > Actually, only the Python interpreter has to be able to do those > > steps. That's why I put the comment *on the same line* as the import > > statement (not immediately above it, for instance). It comes out like > > this: > > > > (env) rosuav@sikorsky:~/shed$ python3 BL2_find_items.py > > Traceback (most recent call last): > > File "BL2_find_items.py", line 19, in > > import lzo # ImportError? pip install python-lzo > > ModuleNotFoundError: No module named 'lzo' > > (env) rosuav@sikorsky:~/shed$ > > Oh, I see: because it prints the full line that caused the exception. > Clever! Yes - provided the .py file is available. If you delete the .py file and just run the .pyc, this doesn't work. (Another reason not to do that, honestly.) > > > for regular users, when you know an error is both likely and unclear > > > to read, it might make sense to catch the exception and print > > > a clearer message. > > > > Define "clearer", though. Given that many MANY users won't read *any* > > error message, the clarity becomes largely moot, and only a handful of > > people will (a) read what you print out, (b) be able to resolve the > > problem, and (c) not be able to figure it out from four lines of > > output. > > When writing programs for general use (which this admittedly wasn't), > it seems sad to accept unclear errors on the assumption that some > users don't read error messages. Even most of the nontechnical > users I know will read a one-line error message, though they > certainly wouldn't try to read a 4-line stack trace. I usually try > to catch errors that I expect will be common, and print something > clearer than the default traceback. TBH, I don't think the default message is all that unclear. It does lack any sort of "how to solve this" information, but that can be provided by the comment. Something that frequently annoys me in terms of debugging is something that "knows better" - it absorbs a nice, useful, helpful message, and spits out one bland, generic message, based on what the developer THINKS is the cause. Remember: You will *never* think of everything that can go wrong. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Friday Filosofical Finking: Import protections
Chris Angelico writes: > Actually, only the Python interpreter has to be able to do those > steps. That's why I put the comment *on the same line* as the import > statement (not immediately above it, for instance). It comes out like > this: > > (env) rosuav@sikorsky:~/shed$ python3 BL2_find_items.py > Traceback (most recent call last): > File "BL2_find_items.py", line 19, in > import lzo # ImportError? pip install python-lzo > ModuleNotFoundError: No module named 'lzo' > (env) rosuav@sikorsky:~/shed$ Oh, I see: because it prints the full line that caused the exception. Clever! > > for regular users, when you know an error is both likely and unclear > > to read, it might make sense to catch the exception and print > > a clearer message. > > Define "clearer", though. Given that many MANY users won't read *any* > error message, the clarity becomes largely moot, and only a handful of > people will (a) read what you print out, (b) be able to resolve the > problem, and (c) not be able to figure it out from four lines of > output. When writing programs for general use (which this admittedly wasn't), it seems sad to accept unclear errors on the assumption that some users don't read error messages. Even most of the nontechnical users I know will read a one-line error message, though they certainly wouldn't try to read a 4-line stack trace. I usually try to catch errors that I expect will be common, and print something clearer than the default traceback. > Indeed. But the biggest argument in favour of this style of thing is > that it requires almost zero effort and has almost zero code > readability cost. Imagine having half a dozen dependencies and tagging > each one with a comment like mine... and now imagine having to bracket > each one of them with a try/except and an appropriate message. Certainly. I don't use try/except on imports in general. And I like your short form, and there are definitely places where I'll use that now where a try/except would have been overkill. ...Akkana -- https://mail.python.org/mailman/listinfo/python-list
Re: Friday Filosofical Finking: Import protections
On Fri, Apr 19, 2019 at 2:46 AM Akkana Peck wrote: > > Chris Angelico writes: > > I write this as: > > > > import whois # ImportError? pip install python-whois > [ ... ] > > it means that normal exception handling is still > > happening (which might be important if I import this into something > > else), plus it's printing the message to stderr rather than stdout > [ ... ] > > About the only downside is that it assumes the .py file is available - > > this won't work with a .pyc-only setup. > > It also assumes the user is capable of (1) finding the .py file > (maybe not so easy if it's imported from another program) and > (2) knowing Python well enough to find and understand the line with > the comment. Actually, only the Python interpreter has to be able to do those steps. That's why I put the comment *on the same line* as the import statement (not immediately above it, for instance). It comes out like this: (env) rosuav@sikorsky:~/shed$ python3 BL2_find_items.py Traceback (most recent call last): File "BL2_find_items.py", line 19, in import lzo # ImportError? pip install python-lzo ModuleNotFoundError: No module named 'lzo' (env) rosuav@sikorsky:~/shed$ > Which in my example is not a problem since I'm probably > the only user of my domaincheck script; but when writing programs > for regular users, when you know an error is both likely and unclear > to read, it might make sense to catch the exception and print > a clearer message. Define "clearer", though. Given that many MANY users won't read *any* error message, the clarity becomes largely moot, and only a handful of people will (a) read what you print out, (b) be able to resolve the problem, and (c) not be able to figure it out from four lines of output. > Your counterarguments are quite valid, though. It's a trade-off. Indeed. But the biggest argument in favour of this style of thing is that it requires almost zero effort and has almost zero code readability cost. Imagine having half a dozen dependencies and tagging each one with a comment like mine... and now imagine having to bracket each one of them with a try/except and an appropriate message. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Friday Filosofical Finking: Import protections
Chris Angelico writes: > I write this as: > > import whois # ImportError? pip install python-whois [ ... ] > it means that normal exception handling is still > happening (which might be important if I import this into something > else), plus it's printing the message to stderr rather than stdout [ ... ] > About the only downside is that it assumes the .py file is available - > this won't work with a .pyc-only setup. It also assumes the user is capable of (1) finding the .py file (maybe not so easy if it's imported from another program) and (2) knowing Python well enough to find and understand the line with the comment. Which in my example is not a problem since I'm probably the only user of my domaincheck script; but when writing programs for regular users, when you know an error is both likely and unclear to read, it might make sense to catch the exception and print a clearer message. Your counterarguments are quite valid, though. It's a trade-off. ...Akkana -- https://mail.python.org/mailman/listinfo/python-list
Re: Friday Filosofical Finking: Import protections
On 18/04/2019 17:10, Manolo Martínez wrote: On 2019-04-17, DL Neil wrote: 2. When the program can still do something useful (if perhaps feature-limited) without the imported module by substituting something else in its place. Isn't this a very common scenario, similar to what package management systems call "optional dependencies"? I wouldn't have said "very common." I maintain a small podcast aggregator that tags podcasts using an external tagging library as an optional dependency---people can choose not to install it if they don't care about tags. That library is imported within a try/except block. Most imports I've seen have been for mandatory functionality; while my current code could run without its CRC library, everything it tried to talk to would reject its messages, for example! -- Rhodri James *-* Kynesim Ltd -- https://mail.python.org/mailman/listinfo/python-list
Re: Friday Filosofical Finking: Import protections
On 2019-04-17, DL Neil wrote: > 2. When the program can still do something useful (if perhaps > feature-limited) without the imported module by substituting > something else in its place. Isn't this a very common scenario, similar to what package management systems call "optional dependencies"? I maintain a small podcast aggregator that tags podcasts using an external tagging library as an optional dependency---people can choose not to install it if they don't care about tags. That library is imported within a try/except block. Manolo signature.asc Description: PGP signature -- https://mail.python.org/mailman/listinfo/python-list
Re: Friday Filosofical Finking: Import protections
On Fri, Apr 19, 2019 at 1:36 AM Akkana Peck wrote:
> One example: there are a gazillion whois modules, and when I run my
> domaincheck program on a machine where I haven't yet installed
> whois, it's helpful to have a reminder of which one I should
> install. (Of course, if you have one of the other whois modules
> installed, the program will fail somewhere else.)
>
> try:
> import whois
> # python-whois from pypi, not whois from pypi or python-whois from debian
> # https://bitbucket.org/richardpenman/pywhois
> except ImportError:
> print("Couldn't import whois. Try: pip3 install python-whois")
> sys.exit(1)
I write this as:
import whois # ImportError? pip install python-whois
Way easier, AND it means that normal exception handling is still
happening (which might be important if I import this into something
else), plus it's printing the message to stderr rather than stdout
(not usually significant, but when it is, I'd usually rather the
errors go to stderr).
About the only downside is that it assumes the .py file is available -
this won't work with a .pyc-only setup.
ChrisA
--
https://mail.python.org/mailman/listinfo/python-list
Re: Friday Filosofical Finking: Import protections
DL Neil writes:
> On 18/04/19 8:44 AM, Grant Edwards wrote:
> > 2. When the program can still do something useful (if perhaps
> > feature-limited) without the imported module by substituting
> > something else in its place.
>
> Any (publishable) examples?
One of the targets my RSS fetching program supports is Plucker on
PalmOS, which used to be how I read RSS feeds back in the day
(nowadays I use simplified HTML on Android). Plucker on Palm had
problems with accented characters, so I added a little module called
ununicode to translate strings to plain ASCII (so á would become a).
For most target platforms, it was a nonissue.
try:
import ununicode
has_ununicode = True
except ImportError as e:
has_ununicode = False
def output_encode(s, encoding):
if encoding == 'ascii' and has_ununicode:
return ununicode.toascii(s, in_encoding=encoding)
else:
return s
The program still worked fine if the module wasn't there, it just
wrote accented characters because it couldn't simplify them.
And yes, it could have tested "if 'ununicode' in sys.modules" instead
of setting a variable; I didn't know about that at the time.
> but... what of the third inherent assumption: that the user(s) will be able
> to handle the situation (discussed in another msg 'here')?
One example: there are a gazillion whois modules, and when I run my
domaincheck program on a machine where I haven't yet installed
whois, it's helpful to have a reminder of which one I should
install. (Of course, if you have one of the other whois modules
installed, the program will fail somewhere else.)
try:
import whois
# python-whois from pypi, not whois from pypi or python-whois from debian
# https://bitbucket.org/richardpenman/pywhois
except ImportError:
print("Couldn't import whois. Try: pip3 install python-whois")
sys.exit(1)
The third case has already been mentioned: modules that change name
but don't change their APIs much.
# Tkinter changed capitalization from Python 2 to Python 3.
try:
import tkinter
except ModuleNotFoundError:
import Tkinter as tkinter
I thought I had examples for gtk and qt -- GUI libraries change
their import syntax a lot -- but I can't find them.
...Akkana
--
https://mail.python.org/mailman/listinfo/python-list
Re: Friday Filosofical Finking: Import protections
On 2019-04-18, DL Neil wrote: > On 18/04/19 8:44 AM, Grant Edwards wrote: >> On 2019-04-17, DL Neil wrote: >> >>> Do you bother with exception handling for import statements? >> >> Sometimes. There are two cases when I do that: >> >> 1. When the module has different names under Python2 and Python3 and >> the program tries first one, then the other. > > Excellent example - and a lot easier than interrogating os.environ (for > example), ie permission cf forgiveness. > > >> 2. When the program can still do something useful (if perhaps >> feature-limited) without the imported module by substituting >> something else in its place. > > Any (publishable) examples? I can describe one example, but the source doesn't belong to me so I can't publish it. I wrote an application that dissects a proprietary Ethernet protocol and prints out what's going on as a human-readable transcript. In the "normal" case, it uses pylibpcap to either capture packets live or read them from a saved capture file. If the import of pylibpcap fails, I replace it with a built-in class which can only read packets from a one particular type/version of capture file. If you try to do a live capture with the built-in class (or read an unsupported capture file format), it prints an error message saying that's only possible with pylibpcap and exits. I can recall one or two other similar cases where a built-in class handles a limited set of the functionality provided by the missing module, but they're too obscure to describe succinctly... >> You've omitted the second thing assumed by the authors: without numpy, >> scipy, pandas, et alia the program can do nothing useful. > > but... what of the third inherent assumption: that the user(s) will be > able to handle the situation (discussed in another msg 'here')? Or they notify somebody who can. The probability of being able to programmatically download and properly install a missing module in order to automagically recover from a failed import is, in practice, zero. You could try to format the error in a prettier way, I suppose, but that's just going to confuse experienced users and admins, and it isn't going to help the user who doesn't know what to do anyway. -- Grant Edwards grant.b.edwardsYow! Don't hit me!! I'm in at the Twilight Zone!!! gmail.com -- https://mail.python.org/mailman/listinfo/python-list
Re: Friday Filosofical Finking: Import protections
DL Neil wrote: Thus the basic question: why do we (apparently) so seldom consider the possibility of an ImportError? Because the cases in which we can do something useful about it are relatively rare. -- Greg -- https://mail.python.org/mailman/listinfo/python-list
Re: Friday Filosofical Finking: Import protections
On Thu, Apr 18, 2019 at 2:32 PM DL Neil wrote: > > On 18/04/19 8:29 AM, Chris Angelico wrote: > > On Thu, Apr 18, 2019 at 6:21 AM DL Neil > > wrote: > >> Do you bother with exception handling for import statements? > >> Can we assume that if such a catastrophic error occurs, it is quite > >> acceptable for the code to fall-over in a tumbling-fumble? > > > > I try/except around import statements only if it's possible for the > > program to recover from the exception. For instance, something that > ... > > User reactions have been covered elsewhere 'here'. You've given an example of how a user might respond to the exception traceback. But can you actually offer anything that your program could do that's BETTER than a traceback? If not, what's the point in catching the exception? > > For something that is absolutely required for the program to continue, > > what would be in your exception handler? Print a message to stderr and > > exit? That's exactly what not-catching-the-exception is for. I do > > sometimes annotate the imports, though: > > We disagree. Although I hasten to add, if the error is trapped, then > there should be some added-value over what is system-provided. > > Application's log? SysLog? Particularly in a centralised or server-based > application especially web-services. Or, you could stick with stderr, which means that the message will go whereever the message ought to be sent. If you're running a web service, you should be catching stderr and putting it somewhere. > > from dataclasses import dataclass # ImportError? Upgrade to Python 3.7 > > or pip install dataclasses > > > > If that bombs out, the entire line will get printed, comment and all, > > and there isn't really anything else that I would want to do with the > > exception. > > So I guess the best way to answer your question is with another > > question: If such a catastrophic error occurs, what ELSE would your > > code do than fall over? If there's an answer to that question, then > > sure, catch the ImportError. Otherwise don't. > > Kudos! To whom are you addressing such annotations? > Whoever might be running the script. I don't particularly discriminate :) ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Friday Filosofical Finking: Import protections
On 18Apr2019 16:05, DL Neil wrote: On 18/04/19 8:45 AM, MRAB wrote: On 2019-04-17 21:20, DL Neil wrote: Do you bother with exception handling for import statements? Can we assume that if such a catastrophic error occurs, it is quite acceptable for the code to fall-over in a tumbling-fumble? [snip] Catch only what you (well, the script) can fix. If it needs numpy, but can't import numpy, then when can it do? Might as well just let it fail. I suppose an alternative might be to try to download and install numpy and then retry, but what if it can't be downloaded, or the installation fails? No, you have to give up at some point. It's better just to report the problem and leave it to the user to fix it. Ah, but is that not the point - the user cannot fix it (and neither should he be allowed to do so: see later comments about Operations functions). I'm missing something here. To me there are usually 2 scenarios where a failed import obstructs the user: - the user is someone like you or I - a person using a programme from source they're fetched - trying to run some programme with a module dependency; here we can reasonably assuming some responsibility for obtaining the missing module; ideally the place we got the programme from might have some pointers - the user is running some kind of packaged app. With python that tends to be either something obtained via pip from PyPI or a downloaded package, such as a bundle Mac .app or a Windows .msi file or a package from a linux distro. Here the responsibility is with the person who made the package/bundle. In the latter case the source _should_ have resolvable dependencies. For example pip packages can name the modules they require and pip itself will fetch those in turn. Bundles apps (.app, .msi etc) tend to include the modules within the bundle. Linux distro packages should include dependencies as well, and the package programme should fetch those. All of these should fulfil the requirements at programme install time, not at programme run time. Do you have a third scenario you could detail? What the user faces (unless we more properly handle the exception) is: import davids_brains Traceback (most recent call last): File "", line 1, in ModuleNotFoundError: No module named 'davids_brains' You and I will think this fine - and 'here' on the list will even beg OPs to give us this much data. However, despite users who have been known to make other comments about my (?lack of) brains, they will tend to offer such helpful advice as: "we need the missing module"...! I think this case is usually the former of the 2 above: the user has fetched from somewhere random. That somewhere should at least identify what is required, even if the user needs to do some further legwork themselves. I think my stance here is that it is the person installing the app/module who needs to sort this, because they know the context in which the app/module is being installed. Do we use pip to get stuff? Do we use the OS package manager to get stuff? Are we in a managed environment where we have to ask the local sysadmins to do this? The point is that the app/module can't know how to obtain missing components. If they didn't come with it, there are many unknowns about the environment and plenty of legitimate circumstances where an automatic install is infeasible or actually undesirable. I think that _most_ ImportErrors should not be caught unless the recovery is extremely well defined. While I'm not a fan of the "import settings" scenario (executable settings files? scary), I do catch ImportErrors in several places in my personal kit. Like Chris I think this is only appropriate where there's a very well defined coping mechanism, in particular _not_ assuming one can fetch more software. So: Python 2/3 imports: personally I try the python 3 import and fall back to the python 2 if that fails. But if the python 2 import fails, _that_ import error gets out. No "install" style recovery. Optional modules: I've got a module which supports several possible index backends. So I've got some classes along these lines: class LMDBIndex(_Index): [...] @classmethod def is_supported(cls): ''' Test whether this index class is supported by the Python environment. ''' try: import lmdb except ImportError: return False return True and some code which looks for a working index class: for indexname, indexclass in indexclasses: if not indexclass.is_supported(): continue ... some other checks ... return indexclass ... raise ValueError( "no supported index classes available: tried %r" % (indexclasses,)) So: no recovery or autoinstall. It just chooses the first (== most preferred) index class and returns it, and raises an exception if none is available. Cheers, Cameron Simpson -- https://mail.python.org/mailman/listinfo/python-list
Re: Friday Filosofical Finking: Import protections
On 18/04/19 1:24 PM, Cameron Simpson wrote: On 17Apr2019 21:45, MRAB wrote: On 2019-04-17 21:20, DL Neil wrote: Do you bother with exception handling for import statements? [...] Catch only what you (well, the script) can fix. If it needs numpy, but can't import numpy, then when can it do? Might as well just let it fail. I'm of this mind too, but... I suppose an alternative might be to try to download and install numpy and then retry, but what if it can't be downloaded, or the installation fails? As an example of what an open ended can of worms attempts recovery might be, yeah. How hard do you try? But also, "installation fails": that isn't always a clean situation: it can litter the install area with partial junk. But this is also a bad example: it is something an _invoked_ programme should never try to do. Except by specific deliberate design and request, a running application shouldn't presume it has rights to install additional things, or even to try. I have personally (though metaphorically) clipped devs across the ear for doing themselves the moral equivalent of the above: try thing, then just "sudo try thing" when it was forbidden. ... +1 Installing additional packages is the same as self modifying code: as a rule, the admins install packages, not the app. +1 Sorry, ranting now over. Not at all, an excellent point which needs to be taken out and dusted-off, every now-and-again - particularly for such ear-clipping training exercises! -- Regards =dn -- https://mail.python.org/mailman/listinfo/python-list
Re: Friday Filosofical Finking: Import protections
On 18/04/19 8:29 AM, Chris Angelico wrote: On Thu, Apr 18, 2019 at 6:21 AM DL Neil wrote: Do you bother with exception handling for import statements? Can we assume that if such a catastrophic error occurs, it is quite acceptable for the code to fall-over in a tumbling-fumble? I try/except around import statements only if it's possible for the program to recover from the exception. For instance, something that ... User reactions have been covered elsewhere 'here'. For something that is absolutely required for the program to continue, what would be in your exception handler? Print a message to stderr and exit? That's exactly what not-catching-the-exception is for. I do sometimes annotate the imports, though: We disagree. Although I hasten to add, if the error is trapped, then there should be some added-value over what is system-provided. Application's log? SysLog? Particularly in a centralised or server-based application especially web-services. from dataclasses import dataclass # ImportError? Upgrade to Python 3.7 or pip install dataclasses If that bombs out, the entire line will get printed, comment and all, and there isn't really anything else that I would want to do with the exception. So I guess the best way to answer your question is with another question: If such a catastrophic error occurs, what ELSE would your code do than fall over? If there's an answer to that question, then sure, catch the ImportError. Otherwise don't. Kudos! To whom are you addressing such annotations? -- Regards =dn -- https://mail.python.org/mailman/listinfo/python-list
Re: Friday Filosofical Finking: Import protections
On 18/04/19 8:44 AM, Grant Edwards wrote: On 2019-04-17, DL Neil wrote: Do you bother with exception handling for import statements? Sometimes. There are two cases when I do that: 1. When the module has different names under Python2 and Python3 and the program tries first one, then the other. Excellent example - and a lot easier than interrogating os.environ (for example), ie permission cf forgiveness. 2. When the program can still do something useful (if perhaps feature-limited) without the imported module by substituting something else in its place. Any (publishable) examples? Most of the code I read, both in books and during code review, eschews any form of ImportError check. Even data science people who 'clean' every data field towards inclusion/exclusion in the analysis, take for granted that numpy, scipy, pandas, et al, will be available to their code. You've omitted the second thing assumed by the authors: without numpy, scipy, pandas, et alia the program can do nothing useful. but... what of the third inherent assumption: that the user(s) will be able to handle the situation (discussed in another msg 'here')? Does such a check seem un-pythonic? [sto] (maybe 'forgiveness cf permission'?) It's probably rather unpythonic if you're not going to anything useful in the exception handler. Indisputable! Can we assume that if such a catastrophic error occurs, it is quite acceptable for the code to fall-over in a tumbling-fumble? It's certainly OK with me. I'm not sure why you refer to raising an exception as "fall-over in a tumbling fumble". Raising an exception is the normal way to indicate failure in Python. Apologies! Evidently a cultural reference that did not export well. To understand "tumbling fumble" perhaps think of dropping something, attempting to catch it with one hand, then having to make a second try with the other, and probably failing to intercede before the floor... I could have used other terms, but likely to be considered 'politically-INcorrect'. No, "tumbling fumble" describes having no exception handling and merely allowing the program to "crash". Thus the basic question: why do we (apparently) so seldom consider the possibility of an ImportError? Does it make a difference if operating in/out of a dev-ops environment? I have no idea what "a dev-ops environment means", and I plan on keeping it that way. :) I see that you value your sanity. On the other hand you don't seem so worried about your physical safety - please note that I'll be hiding behind you, should the (ever-so nice) dev-ops lady on one of my projects happen to read this... -- Regards =dn -- https://mail.python.org/mailman/listinfo/python-list
Re: Friday Filosofical Finking: Import protections
On 18/04/19 8:45 AM, MRAB wrote: On 2019-04-17 21:20, DL Neil wrote: Do you bother with exception handling for import statements? Can we assume that if such a catastrophic error occurs, it is quite acceptable for the code to fall-over in a tumbling-fumble? [snip] Catch only what you (well, the script) can fix. If it needs numpy, but can't import numpy, then when can it do? Might as well just let it fail. I suppose an alternative might be to try to download and install numpy and then retry, but what if it can't be downloaded, or the installation fails? No, you have to give up at some point. It's better just to report the problem and leave it to the user to fix it. Ah, but is that not the point - the user cannot fix it (and neither should he be allowed to do so: see later comments about Operations functions). What the user faces (unless we more properly handle the exception) is: import davids_brains Traceback (most recent call last): File "", line 1, in ModuleNotFoundError: No module named 'davids_brains' You and I will think this fine - and 'here' on the list will even beg OPs to give us this much data. However, despite users who have been known to make other comments about my (?lack of) brains, they will tend to offer such helpful advice as: "we need the missing module"...! -- Regards =dn -- https://mail.python.org/mailman/listinfo/python-list
Re: Friday Filosofical Finking: Import protections
On 18/04/19 8:53 AM, Larry Martell wrote: On 2019-04-17 21:20, DL Neil wrote: Do you bother with exception handling for import statements? I often have to do something like this: try: from settings import SITE_WAFER_DIAMETER except ImportError: SITE_WAFER_DIAMETER = 300 That's an interesting application. Most of the multiple input configuration options examples (I've seen) start with a base of the least-prioritised values, and then 'add' higher-priority configuration option-values to that. In this case we: import the config file, but if it's not available use the stated fall-back value(s). Good one! -- Regards =dn -- https://mail.python.org/mailman/listinfo/python-list
Re: Friday Filosofical Finking: Import protections
On 17Apr2019 21:45, MRAB wrote: On 2019-04-17 21:20, DL Neil wrote: Do you bother with exception handling for import statements? [...] Catch only what you (well, the script) can fix. If it needs numpy, but can't import numpy, then when can it do? Might as well just let it fail. I'm of this mind too, but... I suppose an alternative might be to try to download and install numpy and then retry, but what if it can't be downloaded, or the installation fails? As an example of what an open ended can of worms attempts recovery might be, yeah. How hard do you try? But also, "installation fails": that isn't always a clean situation: it can litter the install area with partial junk. But this is also a bad example: it is something an _invoked_ programme should never try to do. Except by specific deliberate design and request, a running application shouldn't presume it has rights to install additional things, or even to try. I have personally (though metaphorically) clipped devs across the ear for doing themselves the moral equivalent of the above: try thing, then just "sudo try thing" when it was forbidden. Particularly in managed environments, the setup is often deliberately designed to not permit this. Consider the app behind a web service: those which are able to install code are in theory open to being manipulated from the outside to install and run code -malicious code. For this reason such enivoronments are deliberately designed so that an app has the barest minimum privileges to perform its task. So: the app _can't_ write to its code area or to the htdocs tree (in whatever form that may be) - that way lies site defacement and application subversion. It can't create tables in the database or modify schemas. It can't modify data it should not touch, or read data it should never see (think reading credential tables or modifying role definitions as some examples). Installing additional packages is the same as self modifying code: as a rule, the admins install packages, not the app. Sorry, ranting now over. Cheers, Cameron Simpson -- https://mail.python.org/mailman/listinfo/python-list
Re: Friday Filosofical Finking: Import protections
On 2019-04-17 21:20, DL Neil wrote: > Do you bother with exception handling for import statements? I often have to do something like this: try: from settings import SITE_WAFER_DIAMETER except ImportError: SITE_WAFER_DIAMETER = 300 -- https://mail.python.org/mailman/listinfo/python-list
Re: Friday Filosofical Finking: Import protections
On 2019-04-17 21:20, DL Neil wrote: (I know it's not Friday [exp], and after personal apologies[apo]) Do you bother with exception handling for import statements? Most of the code I read, both in books and during code review, eschews any form of ImportError check. Even data science people who 'clean' every data field towards inclusion/exclusion in the analysis, take for granted that numpy, scipy, pandas, et al, will be available to their code. Does such a check seem un-pythonic? [sto] (maybe 'forgiveness cf permission'?) Can we assume that if such a catastrophic error occurs, it is quite acceptable for the code to fall-over in a tumbling-fumble? Does it make a difference if operating in/out of a dev-ops environment? Might such only occur once, because once the environment has been correctly-built, it will/can *never* happen again? Can we assume Built-in and PSL modules *must* be present, so we only need to talk about covering in-house code and pip-ed (or similar) modules? Is it more/less important in __main__ than in an imported module? Do you handle each import separately, or all in a single try..except block? Do you try to anticipate and cover every import in the system at the top of __main__, eg imports inside imported modules? What about OpSys-specific imports (etc)? Do modules import-ed only in specific situations, deserve more, or less, attention? [snip] Catch only what you (well, the script) can fix. If it needs numpy, but can't import numpy, then when can it do? Might as well just let it fail. I suppose an alternative might be to try to download and install numpy and then retry, but what if it can't be downloaded, or the installation fails? No, you have to give up at some point. It's better just to report the problem and leave it to the user to fix it. -- https://mail.python.org/mailman/listinfo/python-list
Re: Friday Filosofical Finking: Import protections
On 2019-04-17, DL Neil wrote: > Do you bother with exception handling for import statements? Sometimes. There are two cases when I do that: 1. When the module has different names under Python2 and Python3 and the program tries first one, then the other. 2. When the program can still do something useful (if perhaps feature-limited) without the imported module by substituting something else in its place. > Most of the code I read, both in books and during code review, > eschews any form of ImportError check. Even data science people who > 'clean' every data field towards inclusion/exclusion in the > analysis, take for granted that numpy, scipy, pandas, et al, will be > available to their code. You've omitted the second thing assumed by the authors: without numpy, scipy, pandas, et alia the program can do nothing useful. > Does such a check seem un-pythonic? [sto] (maybe 'forgiveness cf > permission'?) It's probably rather unpythonic if you're not going to anything useful in the exception handler. > Can we assume that if such a catastrophic error occurs, it is quite > acceptable for the code to fall-over in a tumbling-fumble? It's certainly OK with me. I'm not sure why you refer to raising an exception as "fall-over in a tumbling fumble". Raising an exception is the normal way to indicate failure in Python. > Does it make a difference if operating in/out of a dev-ops > environment? I have no idea what "a dev-ops environment means", and I plan on keeping it that way. :) -- Grant Edwards grant.b.edwardsYow! WHOA!! Ken and Barbie at are having TOO MUCH FUN!! gmail.comIt must be the NEGATIVE IONS!! -- https://mail.python.org/mailman/listinfo/python-list
Re: Friday Filosofical Finking: Import protections
On Thu, Apr 18, 2019 at 6:21 AM DL Neil wrote: > > (I know it's not Friday [exp], and after personal apologies[apo]) > > > Do you bother with exception handling for import statements? > > > Most of the code I read, both in books and during code review, eschews > any form of ImportError check. Even data science people who 'clean' > every data field towards inclusion/exclusion in the analysis, take for > granted that numpy, scipy, pandas, et al, will be available to their code. > > > Does such a check seem un-pythonic? [sto] (maybe 'forgiveness cf > permission'?) > > Can we assume that if such a catastrophic error occurs, it is quite > acceptable for the code to fall-over in a tumbling-fumble? I try/except around import statements only if it's possible for the program to recover from the exception. For instance, something that runs on Py2 and Py3 might attempt to import from a Py3 name (urllib comes to mind), and if it fails, redo the import from the Py2 name; or something might try to import an optional subroutine, and if it isn't present, set something to None, or to a null function. For something that is absolutely required for the program to continue, what would be in your exception handler? Print a message to stderr and exit? That's exactly what not-catching-the-exception is for. I do sometimes annotate the imports, though: from dataclasses import dataclass # ImportError? Upgrade to Python 3.7 or pip install dataclasses If that bombs out, the entire line will get printed, comment and all, and there isn't really anything else that I would want to do with the exception. So I guess the best way to answer your question is with another question: If such a catastrophic error occurs, what ELSE would your code do than fall over? If there's an answer to that question, then sure, catch the ImportError. Otherwise don't. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
