Re: PyWart: Python's import statement and the history of external dependencies
On 24/11/2014 06:25, Rick Johnson wrote: On Sunday, November 23, 2014 4:37:53 PM UTC-6, Gregory Ewing wrote: Chris Angelico wrote: Just out of curiosity, why does the stdlib need modules for manipulating .wav and other sound files, but we have to go to PyPI to get a PostgreSQL client? I suspect it's mainly for historical reasons. The wave module has been around since the very early days of Python when the stldib was smaller and the barrier to entry was lower. By today's standards, it probably wouldn't make it in. Immigration policies evolve basically in the same manner. When a country is founded in a land which is sparsely populated, naturally the state will welcome any and all immigrants (as noted in the New Colossus), but as the population becomes more and more dense, the natural (and wise) reaction is to limit, or even prohibit, the influx of migrants. The Native Americans are no doubt regretting their decision to welcome any and all immigrants. Would they have made the same decision using Python which obviously wouldn't have been available at that time? -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence -- https://mail.python.org/mailman/listinfo/python-list
Re: PyWart: Python's import statement and the history of external dependencies
On Monday, November 24, 2014 4:35:13 AM UTC-6, Mark Lawrence wrote: The Native Americans are no doubt regretting their decision to welcome any and all immigrants. Would they have made the same decision using Python which obviously wouldn't have been available at that time? FINDERS KEEPERS; LOSERS WEEPERS! Contrary to your naive interpretation of history, the native Americans did not *welcome* anyone. The demise of the native american society is a direct result of sins they committed against their *OWN* evolution -- dancing mindlessly around camp fires and worshiping spirits and rocks is the path to evolutionary abyss! On the other hand, the Europeans recognized the potential of the land, *seized it*, and created the most powerful nation this world has ever seen. EVOLUTION LOVES A WINNER! This universe does not give a flying fig for your individual delusions, nor your collective societal delusions. Religion and spiritualism are nothing more than coping mechanisms fostered by emotional feeble minds who are incapable of facing the grim realities of their own mortality! Both have done more harm to human evolution and happiness than almost anything else. I'm disgusted to a member of a species for which a vast majority of my fellows have cast aside intelligence for the mind numbing repetitive rituals and idol worship of shiny things or the maternal comfort of a father/mother figure. WHERE'S WALDO? Where are the monuments or the vast civilizations of the these natives? All we have today are a few arrowheads and some clay pots. Is pup tenting and bareback horse riding something to aspire to? WHERE ARE ALL ANIMAL RIGHTS ACTIVIST? There is reason why they are called natives, and it has more to do with their lack of any levels of organization and less to do with their birth- right. Native Americans were a nomadic people, and they were not a nomadic people simply because they enjoyed hauling their houses on their backs every three months (actually, the women did most of the hauling) WHERE ARE ALL THE FEMINIST? They were nomadic because they lacked the collective intelligence to appreciate the intrinsic value of large centralized civilizations, and as such, they were forced to constantly chase the food sources. A MIND IS A TERRIBLE THING TO WASTE! Their failure to thrive is a direct result of choosing to utilize every muscle in the body except for the most important muscle of all, that is, the brain. Instead of trying to live with the earth, they should of been trying to exploit it. These resources exist for human exploitation, not simply because we have been ordained by some benevolent creature, no, but because long ago we won the battle of intelligence against our rivals, and the spoils of that war are the rights to exploit this earth, and all it's inhabitants. TO THE VICTOR GO THE SPOILS! The native Americans lost the battle of evolution, and as a result, they also forfeited the rights of historical narrative. HISTORY IS WRITTEN BY THE WINNERS! So go ahead and take your children on another scouting trip, so they can build a tee-pee and cook Indian bread, and you can tell them *fantastical* stories about how the Indians lived in harmony with the earth, and you can all sing Kumbaya to the strumming sounds of a guitar. But at some point in their lives, maybe around the time you shatter their innocence with the truth about Santa Claus and the Easter Bunny, you might want to also reveal to them the results of a societal rejection of intelligence -- lest they grow up and become like our dear friend Mark, who is *still* writing letters to Santa. ALL I WANT FOR CHRISTMAS IS TO SURROUND BY INTELLIGENT BEINGS! -- https://mail.python.org/mailman/listinfo/python-list
Re: PyWart: Python's import statement and the history of external dependencies
On 11/24/14 11:51 AM, Rick Johnson wrote: ... lots of off-topic ranting ... Everyone: as tempting as it is to respond, the best course of action will be to completely ignore Rick's rants. Thanks, -- Ned Batchelder, http://nedbatchelder.com -- https://mail.python.org/mailman/listinfo/python-list
Re: PyWart: Python's import statement and the history of external dependencies
On 24/11/2014 16:51, Rick Johnson wrote: EVOLUTION LOVES A WINNER! I think evolution actually requires losers. Clearly there are more extinct species, peoples, languages etc etc than there are existing ones so perhaps if evolution 'loves' anything it 'loves' a loser. -- Robin Becker -- https://mail.python.org/mailman/listinfo/python-list
Re: PyWart: Python's import statement and the history of external dependencies
Rick Johnson rantingrickjohn...@gmail.com: FINDERS KEEPERS; LOSERS WEEPERS! Contrary to your naive interpretation of history, the native Americans [...] Was this rant at least produced by a Python generator? Where's the StopIteration exception when you need one? Marko -- https://mail.python.org/mailman/listinfo/python-list
Re: PyWart: Python's import statement and the history of external dependencies
On Monday, November 24, 2014 10:38:52 PM UTC+5:30, Marko Rauhamaa wrote: Rick Johnson : FINDERS KEEPERS; LOSERS WEEPERS! Contrary to your naive interpretation of history, the native Americans [...] Was this rant at least produced by a Python generator? Where's the StopIteration exception when you need one? Real culprit is Ian who just segfaulted python with an infinite generator -- https://mail.python.org/mailman/listinfo/python-list
Re: PyWart: Python's import statement and the history of external dependencies
On Nov 23, 2014 4:41 PM, Gregory Ewing greg.ew...@canterbury.ac.nz wrote: Chris Angelico wrote: Just out of curiosity, why does the stdlib need modules for manipulating .wav and other sound files, but we have to go to PyPI to get a PostgreSQL client? I suspect it's mainly for historical reasons. The wave module has been around since the very early days of Python when the stldib was smaller and the barrier to entry was lower. By today's standards, it probably wouldn't make it in. Size may also be a factor. The wave module is around 300 lines of Python code and isn't growing. The typical dbapi module is probably a bit larger. -- https://mail.python.org/mailman/listinfo/python-list
Re: PyWart: Python's import statement and the history of external dependencies
On Mon, Nov 24, 2014 at 3:23 AM, Dennis Lee Bieber wlfr...@ix.netcom.com wrote: On Sun, 23 Nov 2014 18:43:40 +1100, Ben Finney ben+pyt...@benfinney.id.au declaimed the following: PostgreSQL is a full-blown system that is itself under continual development, and its APIs continually change to match. Whatever Python API for PostgreSQL gets put into the standard library today is likely to be obsolete long before today's version of Python gets close to ending support. That makes it a poor candidate for inclusion in the standard library. I'd also consider that such a module is highly dependent upon having the related server available. If PostgreSQL access were to become a standard module I'd argue that the library should also include MySQL, Firebird/Interbase, M$ SQL Server, at a minimum. I don't know how many ought to be included, but definitely MySQL would be wanted. Including clients for proprietary protocols may be considered less important. It would also be logical to include some of the meta-protocols, though, like ODBC. SQLite3, as a file-server database, made sense as the actual database engine is accessed as part of the Python process when invoked -- not something that had to be configured/administered at a system-wide level. This is true, and I've seen plenty of explanations as to why SQLite3 but no other database. I just found it odd that, against the exclusion of several modules which would be of significant value to a very common Python use-case (namely, servers (eg web) that need back-end databases) is the inclusion of something equally specific but less commonly useful (.WAV file manipulation). But, that's what PyPI's for. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: PyWart: Python's import statement and the history of external dependencies
Chris Angelico wrote: Just out of curiosity, why does the stdlib need modules for manipulating .wav and other sound files, but we have to go to PyPI to get a PostgreSQL client? I suspect it's mainly for historical reasons. The wave module has been around since the very early days of Python when the stldib was smaller and the barrier to entry was lower. By today's standards, it probably wouldn't make it in. -- Greg -- https://mail.python.org/mailman/listinfo/python-list
Re: PyWart: Python's import statement and the history of external dependencies
On Sunday, November 23, 2014 4:37:53 PM UTC-6, Gregory Ewing wrote: Chris Angelico wrote: Just out of curiosity, why does the stdlib need modules for manipulating .wav and other sound files, but we have to go to PyPI to get a PostgreSQL client? I suspect it's mainly for historical reasons. The wave module has been around since the very early days of Python when the stldib was smaller and the barrier to entry was lower. By today's standards, it probably wouldn't make it in. Immigration policies evolve basically in the same manner. When a country is founded in a land which is sparsely populated, naturally the state will welcome any and all immigrants (as noted in the New Colossus), but as the population becomes more and more dense, the natural (and wise) reaction is to limit, or even prohibit, the influx of migrants. PSYCHOANALYSIS OF MIGRANT/STATE SYMBIOSIS: The majority my readers would probably agree that any state which projected such liberal immigration policies would be doing so based on pure selfishness, and they would be correct, however, most would fail to realize that both the migrant *AND* the state are basing their decisions on pure selfishness. THERE IS NO SUCH THING AS A PURELY SELFISH ACT! For the state, gaining citizens means expanding it's military power (you know, the whole power-in-numbers thing), and expanding it's economic power (money, money, money!), and last but not least, expanding it's tax base (hey, did i mention money yet?). GREED IS PROPAGATED BY THE SYNERGY OF SYMBIOTIC SELFISHNESS! For the migrant, he seeks to gain or benefit by some means, because if he did not gain something *positive* from the interaction, then he would not be a migrant, no, by definition he would be a *masochist*! Like the old saying goes: When your at the bottom, the only direction you can go is up!. So by this definition, the migrant seeks to utilize the resources in his new land for he failed to utilize in his home land. FLIES DON'T CARE MUCH FOR VINEGAR, BUT THEY SURE LOVE HONEY! In the balance of this relationship, the state is more venerable than the migrant. The state only survives *if* it can maintain its population (but more importantly *IF* it can maintain a quality population!), but the migrant can survive under *any* state that is willing to provide for him, he need not pledge allegiance to any one state, no, no,,, and if at any point the relationship becomes untenable for him, he has proven *already* his willingness to betray his responsibility as an equal member of this symbiosis, by simply re-locating! But is the migrant *really* an equal member? No. The migrant does not share equally the burdens of the symbiosis. Since the migrant is free to utilize the resources of the state, and then simply pack-up and leave when the resources are exhausted, the migrant is in-effect the master, and the state is the slave. And although the state does extract a tiny amount of power from the migrant, this power is useless when one party can nullify the contract just because the honey pot ran empty. CONCLUSION: The migrant/state symbiosis is a fine example of how we (as *weak* emotional beings) fall into these emotional traps set by forked tongued propagandist in hopes of diverting our attention away from reality. No being in this universe makes any choice based on altruistic motives, no, altruism is a mask worn by the most cunning of con-artist -- a true weapon of deceit! -- https://mail.python.org/mailman/listinfo/python-list
Re: PyWart: Python's import statement and the history of external dependencies
On Mon, Nov 24, 2014 at 5:25 PM, Rick Johnson rantingrickjohn...@gmail.com wrote: The migrant/state symbiosis is a fine example of how we (as *weak* emotional beings) fall into these emotional traps set by forked tongued propagandist in hopes of diverting our attention away from reality. No being in this universe makes any choice based on altruistic motives, no, altruism is a mask worn by the most cunning of con-artist -- a true weapon of deceit! As CS Lewis wrote in various places, notably The Screwtape Letters, people who are fundamentally driven by pure selfishness are utterly unable to even comprehend love and genuine altruism. Fortunately for us, your email is both off-topic and useless, so we can cheerfully ignore it. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: PyWart: Python's import statement and the history of external dependencies
On Fri, Nov 21, 2014 at 6:07 PM, Rick Johnson rantingrickjohn...@gmail.com wrote: On Friday, November 21, 2014 5:59:44 PM UTC-6, Chris Angelico wrote: In other words, what you want is: # today's method, import based on search path import sys # new explicit path method import '/usr/local/lib/python3.5/lib-dynload/math.cpython-35m.so' [...] There are a VERY VERY few cases where you really do want to import a module from a specific file path. For those situations, there are already techniques you can use. Why change the behaviour of import itself? Chris, either take the time to read and understand my posts *FULLY*, or don't bother to reply. I have in no way suggested that we import via filepaths. Stop lying. I see very little substantive difference between what Chris posted and your suggestion of: __search_path__ = ['/usr/local/lib/python3.5/lib-dynload'] import math His comment about non-portability stands. -- https://mail.python.org/mailman/listinfo/python-list
Re: PyWart: Python's import statement and the history of external dependencies
Ian Kelly wrote: On Thu, Nov 20, 2014 at 8:53 PM, Rick Johnson rantingrickjohn...@gmail.com wrote: FOR INSTANCE: Let's say i write a module that presents a reusable GUI calendar widget, and then i name the module calender.py. Then Later, when i try to import *MY* GUI widget named calendar, i will not get *MY* calendar widget, no, i will get the Python calendar module. Because like a fool you created a local module with a totally generic name like calendar and dumped it into the top-level namespace. Firstly, let me say that I am amazed that people have waded through Rick's self-aggrandising purple prose, insults, demands and utter confusion to find something worth engaging him on. Having said that, it's not fair to blame the user for shadowing standard library modules: - All users start off as beginners, who may not be aware that this is even a possibility; - It's hard to keep track of what modules are in the standard library. Which of the following is *not* in Python 3.3's std lib? No cheating by looking them up.) os2emxpath, wave, sndheader, statslib, poplist, plist, pickletools, picklelib, path, cgi, cgitb, copylib, xpath - By default, sys.path is set up so that user modules have priority over std lib modules. Given a name clash, shadowing in inevitable. The Python import system gives you the ability to create packages, so use them -- create a package to contain all the widgets you create. Packages come with their own challenges and complexities, especially if you're trying to support both Python 2 and 3 from the same code base. Besides, for simple uses, packages are overkill. [...] # Contrary to popular belief, sys.path is *NOT* a module, # # no, it's a global! # I really doubt that this is a popular belief. I'm not aware of anyone who believes that sys.path is a module. If they did, a simple import would prove that it is not: py import sys.path Traceback (most recent call last): File stdin, line 1, in module ImportError: No module named path But yes, sys.path is not just global, but process-wide global. *All* modules share the same sys.path. This is another confusing fundamental of Python, it seems *MORE* intuitive that changes to the import search path would only affect the *CURRENT* module's import search path, but that's is not the case! This just seems like it would create a maintenance nightmare. Before you could import anything, you'd have to figure out what the search path is for the particular module you're working and whether it happens to include the particular package you want. You find it doesn't, so you make a local change to the search path. Later, you make the same change to other modules that need to import it. Later still, the package moves to a different location in the file system, and now you get to go through and update every one of those modules with the new directory. Lucky you. That would be horrible. But here's an alternative which is less horrible and maybe even useful. There's still a single process-wide search path, but there's a second per-module search path which is searched first. By default it's empty. So a module can define it's own extra search path: __path__ = ['look/here', 'and/here'] import something without affecting any other modules. And after all that, it would still fail if you happened to want to import both calendar modules into the same module. __path__ = [] import calendar __path__ = ['my/python/modules'] import calendar as mycalendar -- Steven -- https://mail.python.org/mailman/listinfo/python-list
Re: PyWart: Python's import statement and the history of external dependencies
On Sat, Nov 22, 2014 at 11:25 PM, Steven D'Aprano steve+comp.lang.pyt...@pearwood.info wrote: Ian Kelly wrote: - It's hard to keep track of what modules are in the standard library. Which of the following is *not* in Python 3.3's std lib? No cheating by looking them up.) os2emxpath, wave, sndheader, statslib, poplist, plist, pickletools, picklelib, path, cgi, cgitb, copylib, xpath Okay, here's my guesses. os2emxpath: In the stdlib, but more often accessed as os.path while running under OS/2 wave: Not in the stdlib, though I'd avoid the name anyway. sndheader: Not in the stdlib - probably on PyPI though poplist, plist, pickletools, picklelib: I suspect PyPI, not stdlib, but could be wrong path: Not in the stdlib (there's os.path and I doubt there'd be both) cgi, cgitb: In the stdlib copylib: No idea, could be either way. xpath: I'll guess this as not being present. I'm probably pretty wrong, though. # Contrary to popular belief, sys.path is *NOT* a module, # # no, it's a global! # I really doubt that this is a popular belief. I'm not aware of anyone who believes that sys.path is a module. But yes, sys.path is not just global, but process-wide global. *All* modules share the same sys.path. Even leaving aside Rick's sloppy language, I still doubt that it's popular belief that sys.path be module-specific. You're modifying something in a different module, and Python's always maintained that two instances of import sys will give two references to the exact same module object. That would be horrible. But here's an alternative which is less horrible and maybe even useful. There's still a single process-wide search path, but there's a second per-module search path which is searched first. By default it's empty. So a module can define it's own extra search path: __path__ = ['look/here', 'and/here'] import something without affecting any other modules. That's what Rick said first, and then said that if you're going to be explicit, you should do the job properly and not have _any_ implicit paths. Thing is, though, it still breaks the sys.modules concept. Either __path__ is ignored if the module was found in sys.modules, or it's possible to have multiple entries with the same name (which would make it hard to have a module replace itself in sys.modules, currently a supported thing). Although I suppose all it'd require is that sys.modules be keyed by __file__ rather than __name__, so they're identified by fully qualified path and file name. (What does that do in the face of .pyc files?) And after all that, it would still fail if you happened to want to import both calendar modules into the same module. __path__ = [] import calendar __path__ = ['my/python/modules'] import calendar as mycalendar Frankly, if you actually want this, I think it's time to turn to an uglier-but-more-flexible method.like poking around in importlib. (I'm not sure off-hand how you'd go about it, it's not instantly obvious from help(importlib).) I'm more concerned about the possibility of your import succeeding or failing depending on the order of other imports: # foo.py import calendar # bar.py __path__ = ['my/python/modules'] import foo import calendar How's that one to be resolved? That's what I don't like. So long as sys.modules is (a) process-wide and (b) keyed by module name rather than file name, sys.path MUST be process-wide too, and MUST be set on startup, or as soon as possible afterwards. Any module imported prior to altering sys.path will be fetched based on the previous search path - and you have to import sys to change sys.path, which means the minimum set of unalterable modules is, on Python 3.5: rosuav@sikorsky:~$ cat showmods.py import sys print(, .join(sorted(sys.modules.keys( rosuav@sikorsky:~$ python3 showmods.py __main__, _codecs, _collections_abc, _frozen_importlib, _imp, _io, _signal, _sitebuiltins, _stat, _sysconfigdata, _thread, _warnings, _weakref, _weakrefset, abc, builtins, codecs, encodings, encodings.aliases, encodings.latin_1, encodings.utf_8, errno, genericpath, io, marshal, os, os.path, posix, posixpath, site, stat, sys, sysconfig, zipimport ... that's a decent lot of modules you can't fiddle with. Hence PYTHONPATH, which presumably is processed by the interpreter prior to loading any modules. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: PyWart: Python's import statement and the history of external dependencies
On 2014-11-22 23:25, Steven D'Aprano wrote: Having said that, it's not fair to blame the user for shadowing standard library modules: - All users start off as beginners, who may not be aware that this is even a possibility; While it's one thing to explicitly shadow a module (creating your own module named calendar; if you need the stdlib module, you know it and have to rename your local one), the implicit shadowing has stung me a number of times. I'll unthinkingly create an email.py and then need to send email so I'll import smtplib (which internally imports stdlib's email.py but it finds my local email.py) and everything falls over. The error messages don't make it obvious what happened. I've been burned enough times that I now know not to do this. But it was certainly burned into my mind after the 3rd time it happened. And after all that, it would still fail if you happened to want to import both calendar modules into the same module. __path__ = [] import calendar __path__ = ['my/python/modules'] import calendar as mycalendar Hrm. Never knew this. Could you point me at some docs on that? (searching for __path__ on python.org isn't turning up much that jumps out at me as being correlated) -tkc -- https://mail.python.org/mailman/listinfo/python-list
Re: PyWart: Python's import statement and the history of external dependencies
Tim Chase wrote: On 2014-11-22 23:25, Steven D'Aprano wrote: And after all that, it would still fail if you happened to want to import both calendar modules into the same module. __path__ = [] import calendar __path__ = ['my/python/modules'] import calendar as mycalendar Hrm. Never knew this. Could you point me at some docs on that? (searching for __path__ on python.org isn't turning up much that jumps out at me as being correlated) You've misunderstood what I said, I was proposing it as a hypothetical feature, not something existing. Sorry for any confusion. -- Steven -- https://mail.python.org/mailman/listinfo/python-list
Re: PyWart: Python's import statement and the history of external dependencies
On 2014-11-23 12:00, Steven D'Aprano wrote: And after all that, it would still fail if you happened to want to import both calendar modules into the same module. __path__ = [] import calendar __path__ = ['my/python/modules'] import calendar as mycalendar Hrm. Never knew this. Could you point me at some docs on that? (searching for __path__ on python.org isn't turning up much that jumps out at me as being correlated) You've misunderstood what I said, I was proposing it as a hypothetical feature, not something existing. Sorry for any confusion. Ah, thanks. I wondered but should have tried it out to see that it wasn't an existing feature. I'm not sure if/when I would use such a feature, but it was at least an interesting idea. -tkc -- https://mail.python.org/mailman/listinfo/python-list
Re: PyWart: Python's import statement and the history of external dependencies
Chris Angelico wrote: On Sat, Nov 22, 2014 at 11:25 PM, Steven D'Aprano steve+comp.lang.pyt...@pearwood.info wrote: Ian Kelly wrote: - It's hard to keep track of what modules are in the standard library. Which of the following is *not* in Python 3.3's std lib? No cheating by looking them up.) os2emxpath, wave, sndheader, statslib, poplist, plist, pickletools, picklelib, path, cgi, cgitb, copylib, xpath Okay, here's my guesses. os2emxpath: In the stdlib, but more often accessed as os.path while running under OS/2 Correct. wave: Not in the stdlib, though I'd avoid the name anyway. Incorrect. The wave module is for manipulating .wav files. sndheader: Not in the stdlib - probably on PyPI though Correct. It is actually spelled sndhdr. poplist, plist, pickletools, picklelib: I suspect PyPI, not stdlib, but could be wrong There's a pickletools, but not picklelib. plist is actually spelled plistlib, and there is poplib, not poplist. path: Not in the stdlib (there's os.path and I doubt there'd be both) cgi, cgitb: In the stdlib Correct. copylib: No idea, could be either way. Spelled copy, not copylist. xpath: I'll guess this as not being present. Correct. Not bad, but as you can see, the naming convensions are all over the place. It's easy to accidentally shadow the std lib. [...] That would be horrible. But here's an alternative which is less horrible and maybe even useful. There's still a single process-wide search path, but there's a second per-module search path which is searched first. By default it's empty. So a module can define it's own extra search path: __path__ = ['look/here', 'and/here'] import something without affecting any other modules. That's what Rick said first, and then said that if you're going to be explicit, you should do the job properly and not have _any_ implicit paths. Rick should ask himself why virtually every single language, from compiled languages like Ada, C, Pascal and Java, to interpreted languages like bash, all use search paths instead of explicit paths. Hint: the answer is not that every single language designer in the world is a moron. Thing is, though, it still breaks the sys.modules concept. Either __path__ is ignored if the module was found in sys.modules Good point. -- Steven -- https://mail.python.org/mailman/listinfo/python-list
Re: PyWart: Python's import statement and the history of external dependencies
On Sunday, November 23, 2014 12:00:15 PM UTC+5:30, Steven D'Aprano wrote: Rick should ask himself why virtually every single language, from compiled languages like Ada, C, Pascal and Java, to interpreted languages like bash, all use search paths instead of explicit paths. Hint: the answer is not that every single language designer in the world is a moron. In all fairness to Rick I can think of two languages -- Erlang and Emacs-lisp. They probably dont do what he wants (I dont really know what he wants -- his verbiage is too much for me to get through) however they come close. Erlang warns about shadowing system-names Elisp has a function -- list-load-path-shadows -- that informs about all shadowings in the system. In a way this makes sense because for a system to warn about shadowing is a trivial cost. For a programmer to keep track of which toes are being stepped on is an error-prone headache. -- https://mail.python.org/mailman/listinfo/python-list
Re: PyWart: Python's import statement and the history of external dependencies
On Sun, Nov 23, 2014 at 5:30 PM, Steven D'Aprano steve+comp.lang.pyt...@pearwood.info wrote: Chris Angelico wrote: Okay, here's my guesses. os2emxpath: In the stdlib, but more often accessed as os.path while running under OS/2 Correct. I'm in a special position here, as I actually have an OS/2 system, and have used Python on it. So I know about the particular obscure system that you happened to pick there. Now, if you'd said amigapath, I wouldn't have known. [chomp a few correct and a few wrong, here and there] Not bad, but as you can see, the naming convensions are all over the place. It's easy to accidentally shadow the std lib. Agreed. There's no way to guarantee that you aren't shadowing someone else's module, except by namespacing your own - either putting your name at the beginning of every file name (fredfoobar_calendar) or by making a package (fredfoobar.calendar). It would be nice, though, to be able to *detect* shadowing. Is there a way (maybe through importlib) to ask Python to enumerate every file that could be found when attempting to import calendar? Something like this: import importlib importlib.find_all_modules(calendar) ['calendar.py', '/usr/local/lib/python3.5/calendar.py'] That would be a useful feature, if it doesn't already exist. If it does, it could do with a bit more visibility - poking around on Google and interactively with help(importlib) etc didn't reveal it to me. That's what Rick said first, and then said that if you're going to be explicit, you should do the job properly and not have _any_ implicit paths. Rick should ask himself why virtually every single language, from compiled languages like Ada, C, Pascal and Java, to interpreted languages like bash, all use search paths instead of explicit paths. Hint: the answer is not that every single language designer in the world is a moron. Maybe they are. Maybe Rick's just so amazingly brilliant that he, alone among all language designers, has figured this out. Maybe it's time for his fork of Python to sweep the world with its incredible solutions to massively prevalent problems! While it *is* possible for every language in the world to get something wrong, there's usually a few around the place that get it right. Until Python 3.3, most popular languages got Unicode support either wrong (using UTF-16 being the most common wrongness) or inefficient (using UTF-32), or just didn't have any at all (bytes everywhere). Could someone look at the state of world programming languages and say Everyone uses UTF-16, and not every language designer in the world is a moron, therefore UTF-16 must be the right thing to do? No, because Haskell, Scheme, Pike, and bash all had it right, already. Sure, they're not the most-used languages in the world, but they certainly are used (bash plenty, though to be honest, not many people use bash scripts for Unicode heavy-lifting), and they prove that UTF-16 isn't the only way to go. So, where are the languages that use explicit paths? Show us why they're so great, using actual production code. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: PyWart: Python's import statement and the history of external dependencies
On Sun, Nov 23, 2014 at 5:30 PM, Steven D'Aprano steve+comp.lang.pyt...@pearwood.info wrote: wave: Not in the stdlib, though I'd avoid the name anyway. Incorrect. The wave module is for manipulating .wav files. sndheader: Not in the stdlib - probably on PyPI though Correct. It is actually spelled sndhdr. Just out of curiosity, why does the stdlib need modules for manipulating .wav and other sound files, but we have to go to PyPI to get a PostgreSQL client? It's a queer world... ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: PyWart: Python's import statement and the history of external dependencies
Chris Angelico ros...@gmail.com writes: Just out of curiosity, why does the stdlib need modules for manipulating .wav and other sound files, but we have to go to PyPI to get a PostgreSQL client? It's a queer world... I would venture the follow two reasons, either of which is sufficient to explain the difference: * Modules enter the standard library only if they are judged both to be good quality *and* have a credible promise to continue to maintain them in the standard library on an ongoing basis. If one module has reliable maintainers offering to do the work and another does not, the latter will not be a candidate for inclusion in the Python standard library. * Modules enter the standard library only if their feature set is stable enough that they can remain essentially unchanged, feature-wise, for many years and still be widely useful. For a data stream format (like WAV and other mature formats), a module working well today is likely to work just as well for the same purpose in several years's time, long enough for today's Python to go through its full life cycle of support. PostgreSQL is a full-blown system that is itself under continual development, and its APIs continually change to match. Whatever Python API for PostgreSQL gets put into the standard library today is likely to be obsolete long before today's version of Python gets close to ending support. That makes it a poor candidate for inclusion in the standard library. -- \ “The generation of random numbers is too important to be left | `\to chance.” —Robert R. Coveyou | _o__) | Ben Finney -- https://mail.python.org/mailman/listinfo/python-list
Re: PyWart: Python's import statement and the history of external dependencies
On Sun, Nov 23, 2014 at 6:43 PM, Ben Finney ben+pyt...@benfinney.id.au wrote: For a data stream format (like WAV and other mature formats), a module working well today is likely to work just as well for the same purpose in several years's time, long enough for today's Python to go through its full life cycle of support. PostgreSQL is a full-blown system that is itself under continual development, and its APIs continually change to match. Whatever Python API for PostgreSQL gets put into the standard library today is likely to be obsolete long before today's version of Python gets close to ending support. That makes it a poor candidate for inclusion in the standard library. That makes sense, as differences go. Though if wave and sndhdr were currently PyPI-only and someone came to python-ideas/python-dev saying These modules ought to be in the standard library, I doubt they'd get a huge amount of support. Of course, since they're currently in the stdlib, there's absolutely no reason to *remove* them, but any similar modules are likely to stay pip-gettable rather than autoincluded. In contrast, something like Lib/email.py is providing functionality that heaps of Python scripts want - sending or parsing emails is pretty common, so it makes good sense to have that in the stdlib. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: PyWart: Python's import statement and the history of external dependencies
On 2014-11-20 19:53, Rick Johnson wrote: FOR INSTANCE: Let's say i write a module that presents a reusable GUI calendar widget, and then i name the module calender.py. Then Later, when i try to import *MY* GUI widget named calendar, i will not get *MY* calendar widget, no, i will get the Python calendar module. The reason Python finds the library module instead of my module is because the Python library is higher in search path than my module. What messed-up version of Python are you running? Or did you fail to test your conjecture? $ cat calendar.py print(This is my local calendar.py) x=1 $ cat importtest.py import calendar print(dir(calendar)) $ python2 importtest.py This is my local calendar.py ['__builtins__', '__doc__', '__file__', '__name__', '__package__', 'x'] $ python3 importtest.py This is my local calendar.py ['__builtins__', '__cached__', '__doc__', '__file__', '__name__', '__package__', 'x'] It finds my local module, not the system calendar module. -tkc -- https://mail.python.org/mailman/listinfo/python-list
Re: PyWart: Python's import statement and the history of external dependencies
On 11/20/14 10:53 PM, Rick Johnson wrote: If you had taken the time to read my example utilizing a lobby boy, then you might have understood what i was talking about. Rick, if you are frustrated that people don't know what you are talking about, you should try writing shorter messages, with less bombast. I don't know if you are aiming for humor with your messages, but to my ears, you are verging on self-parody. I'm surprised that people are taking you seriously. -- Ned Batchelder, http://nedbatchelder.com -- https://mail.python.org/mailman/listinfo/python-list
Re: PyWart: Python's import statement and the history of external dependencies
On Thu, Nov 20, 2014 at 8:53 PM, Rick Johnson rantingrickjohn...@gmail.com wrote: FOR INSTANCE: Let's say i write a module that presents a reusable GUI calendar widget, and then i name the module calender.py. Then Later, when i try to import *MY* GUI widget named calendar, i will not get *MY* calendar widget, no, i will get the Python calendar module. Because like a fool you created a local module with a totally generic name like calendar and dumped it into the top-level namespace. The Python import system gives you the ability to create packages, so use them -- create a package to contain all the widgets you create. If they're only for your own use, then you can just call it ricklib or anything else you want. Then your import becomes: import ricklib.calendar # or from ricklib import calendar Now you can drop as much stuff in there as you like, and none of it will ever conflict with the standard library (unless a standard ricklib module is added, which is unlikely). The reason Python finds the library module instead of my module is because the Python library is higher in search path than my module. Your PYTHONPATH is added to the sys.path before the Python library, not after, so to the extent that there is an issue, it's actually the other way around. You would get your local calendar module when trying to import the system calendar module. # Contrary to popular belief, sys.path is *NOT* a module, # # no, it's a global! # I really doubt that this is a popular belief. This is another confusing fundamental of Python, it seems *MORE* intuitive that changes to the import search path would only affect the *CURRENT* module's import search path, but that's is not the case! This just seems like it would create a maintenance nightmare. Before you could import anything, you'd have to figure out what the search path is for the particular module you're working and whether it happens to include the particular package you want. You find it doesn't, so you make a local change to the search path. Later, you make the same change to other modules that need to import it. Later still, the package moves to a different location in the file system, and now you get to go through and update every one of those modules with the new directory. Lucky you. And after all that, it would still fail if you happened to want to import both calendar modules into the same module. in the 25% of cases where *NAME SHADOWING* or where the author knows the location and can specify the location to save lobby boy from knocking on doors until his knuckles bleed, the author should have an option defining the search path *LOCALLY* (in a manner that will *NOT* affect the import search path further down the import chain), WITHOUT needing to import one of the many discombobulated and confusing import tools that Python has tortured this community with. By doing so, the author would also make the module non-portable. This is a major problem if they intend to distribute it, but still a problem even if they don't. Any time the code is moved to a new system and placed in a new location (or worse: moved to a new operating system) the code will have to be updated. And you can pretty much forget about hosting it on a network share. 3) Filepath imports (deprecated in 3.4 but under discussion for reprecation or replacement) import importlib.machinery loader = importlib.machinery.SourceFileLoader(module.name, /path /to/file.py) foo = loader.load_module() foo.MyClass() That's ridiculously noisy. I have an idea, if you *REALLY* want to introduce boilerplate hell then Python should adopt import header files, NOT! It's three lines of code to replace one. Two if you exclude the importlib.machinery import that doesn't need to be repeated. Is this really any worse than something like: local_search_path.insert(0, /path/to/local/module) import my_local_module that you are proposing? -- https://mail.python.org/mailman/listinfo/python-list
Re: PyWart: Python's import statement and the history of external dependencies
On Friday, November 21, 2014 4:29:48 AM UTC-6, Tim Chase wrote: What messed-up version of Python are you running? Or did you fail to test your conjecture? $ cat calendar.py print(This is my local calendar.py) x=1 $ cat importtest.py import calendar print(dir(calendar)) $ python2 importtest.py This is my local calendar.py ['__builtins__', '__doc__', '__file__', '__name__', '__package__', 'x'] $ python3 importtest.py This is my local calendar.py ['__builtins__', '__cached__', '__doc__', '__file__', '__name__', '__package__', 'x'] It finds my local module, not the system calendar module. You failed to provide the needed information! 1. We need to see the values in sys.path. So print them directly before making the call to import. 2. Where did you store your custom calendar.py script? Remember, import is an implicit mechanism, it works by searching the values of 'sys.path' one-by-one (if the module was not already loaded!) until the *FIRST* module matching the name you requested is found. If the folder that holds *your* custom calendar.py script is listed *before* the folder containing the python calendar modules then of course your module will be correctly loaded. -- https://mail.python.org/mailman/listinfo/python-list
Re: PyWart: Python's import statement and the history of external dependencies
On 2014-11-21 07:52, Rick Johnson wrote: On Friday, November 21, 2014 4:29:48 AM UTC-6, Tim Chase wrote: What messed-up version of Python are you running? Or did you fail to test your conjecture? $ cat calendar.py print(This is my local calendar.py) x=1 $ cat importtest.py import calendar print(dir(calendar)) $ python2 importtest.py This is my local calendar.py ['__builtins__', '__doc__', '__file__', '__name__', '__package__', 'x'] $ python3 importtest.py This is my local calendar.py ['__builtins__', '__cached__', '__doc__', '__file__', '__name__', '__package__', 'x'] It finds my local module, not the system calendar module. You failed to provide the needed information! 1. We need to see the values in sys.path. So print them directly before making the call to import. 2. Where did you store your custom calendar.py script? Had you paid attention, having watched the creation of the two files and the invocation of stock python, you would know that they were in the same folder, and there's no modification of sys.path in the process. So if you're seeing different behaviors due to a modified sys.path, then you should be aware that *you* modified sys.path and thus broke things. The above was tested on Python2 and Python3, both on Linux (Debian in this case) and Win32 (using copy con instead of cat ) with the out-of-box install. The only time I've been stung by name overloading is in the indirect case of creating a local email.py file and then importing smtplib only to have things break in unforeseen ways. If smtplib used relative imports for $STDLIB/email.py I suspect it would ameliorate that particular issue for me. -tkc -- https://mail.python.org/mailman/listinfo/python-list
Re: PyWart: Python's import statement and the history of external dependencies
On Fri, Nov 21, 2014 at 9:12 AM, Tim Chase python.l...@tim.thechases.com wrote: The only time I've been stung by name overloading is in the indirect case of creating a local email.py file and then importing smtplib only to have things break in unforeseen ways. If smtplib used relative imports for $STDLIB/email.py I suspect it would ameliorate that particular issue for me. Relative imports are based on package namespaces and can only be done within a package. There's no way to do a relative import from a separate top-level package, whether they happen to be found in the same directory or not. $ mkdir p1 $ touch p1/__init__.py $ echo 'from ..p2 import m2' p1/m1.py $ mkdir p2 $ touch p2/__init__.py $ touch p2/m2.py $ python3 -c 'import p1.m1' Traceback (most recent call last): File string, line 1, in module File /home/ikelly/p1/m1.py, line 1, in module from ..p2 import m2 ValueError: attempted relative import beyond top-level package -- https://mail.python.org/mailman/listinfo/python-list
Re: PyWart: Python's import statement and the history of external dependencies
On Friday, November 21, 2014 9:34:55 AM UTC-6, Ian wrote: On Thu, Nov 20, 2014 at 8:53 PM, Rick Johnson FOR INSTANCE: Let's say i write a module that presents a reusable GUI calendar widget, and then i name the module calender.py. Then Later, when i try to import *MY* GUI widget named calendar, i will not get *MY* calendar widget, no, i will get the Python calendar module. Because like a fool you created a local module with a totally generic name like calendar and dumped it into the top-level namespace. Are you also going to call drivers fools because they bought a certain brand of car only to have the airbag explode in their face? What's next, are you going to blame them for not wearing a mask? Why am *i* the fool when it's obvious that the creators of Python were either shortsighted and/or careless with the designs? The only people who suffer are those who put their trust in the designer, and not the designer himself -- something is wrong with this picture! The Python import system gives you the ability to create packages, so use them -- create a package to contain all the widgets you create. If they're only for your own use, then you can just call it ricklib or anything else you want. Of course, I did that a long time ago! But like everything in Python, when your try to be cleaver and create a workaround for the design flaws of this language (and there are many!) you end up with shards of jagged metal stuck in your face! GvR MUST BE A HUGE FAN OF THE JIGSAW MOVIES! Now you can drop as much stuff in there as you like, and none of it will ever conflict with the standard library (unless a standard ricklib module is added, which is unlikely). Yes, and now we've solved one problem by replacing it with it's inverse -- try importing the *python lib* calendar module and all you will get is your local intra-package version. Now, the only way to get to the lib module is by mutilating sys.path, or using an import utility module to import by filepath. THANKS PYTHON! You are correct though, Python's packaging system CAN BE used to isolate intra-package scripts from clashing with outside scripts, HOWEVER, there are many caveats that one must know (which are NOT intuitive!) to use Python's packaging system without requiring a plastic surgeon on retainer! Lets say i take your advice, and i want to use python packages to protect a set of modules from the global import scope. # Explosive Trap 1 # # Any attempt to import a stdlib module, who's name# # matches one of the modules within the current package, # # will result in importing the local package module, # # forcing you to: (1) inject search paths into sys.path# # manually (2) import a module to correctly import the # # dependency # YUCK! # Explosive Trap 2 # # Any code in the __init__ file will be executed NOT ONLY # # when the package is imported DIRECTLY, but even when the # # package name is referenced as part of a larger absolute # # import path # OUCH! Anyone would expect that when *DIRECTLY* importing a package, if the __init__ file has code, then THAT code should be executed, HOWEVER, not many would expect that merely referencing the package name (in order to import a more deeply nested package) would cause ALL the intermediate __init__ files to execute -- this is madness, and it prevents using an __init__ file as an import hub (without side-effects)! Yeah rick, but why would you want to use an __init__ file as an import hub Because the alternative is messy. If i have a collection of modules under a package, sometimes i would like to import all the exportable objects into the __init__ file and use the package as an import hub. Imagine a package layout like this: + ricklib __init__.py + subpkg1 (ricklib.subpkg1) __init__.py module1.py module2.py module3.py + subpkg1a (ricklib.subpkg1.subpkg1a) And the contents of ricklib.subpkg1.__init__.py are as follows: from module1 import Class1 from module2 import Class2 from module3 import Class3 Now, when i want to import a number of modules from subpkg1 i can do: from ricklib.subpkg1 import Class1, Class2, ... Instead of this: from ricklib.subpkg1.module1 import Class1 from ricklib.subpkg1.module2 import Class2 And everything will work fine, UNTIL, i try to access subpkg1a from ricklib.subpkg1.subpkg1a import
Re: PyWart: Python's import statement and the history of external dependencies
On Sat, Nov 22, 2014 at 5:24 AM, Rick Johnson rantingrickjohn...@gmail.com wrote: Of course, I did that a long time ago! But like everything in Python, when your try to be cleaver... Just so you know, I never try to be one of these. http://img1.wikia.nocookie.net/__cb20110615074214/americanmcgeesalice/images/1/1a/Vorpal_Cleaver.png Along with misuse of your when you mean you, plus a few of the all-too-common misplaced (or omitted) apostrophes, this kind of errant English leaves me wondering if you take as much care over your Python code as you do when writing c.l.py posts, and then blame Python for the results. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: PyWart: Python's import statement and the history of external dependencies
On 11/21/2014 11:24 AM, Rick Johnson wrote: Why am *i* the fool when it's obvious that the creators of Python were either shortsighted and/or careless with the designs? The only people who suffer are those who put their trust in the designer, and not the designer himself -- something is wrong with this picture! So if you know how to create the perfect language that has a solution to these very difficult problems, please come forward with it. Until such time, it's okay to identify problems with the language as you see them, but to cast aspersion on the developers is out of line. Do you know Guido personally to know that he's short-sighted or careless? I certainly don't, but from what I can see he is neither. Not perfect, but pretty darn smart. Much smarter than I am (despite the fact I can also recognize several of Python's flaws). The same goes for many other core developers. They tend to be smart, articulate, and know how to communicate with people. Everything is about trade-offs and the ones Python makes work very well for most people. Perhaps the problem with the picture is that you are unable to see it clearly. I assure you that Python developers not only create Python, they use it too, for their own purposes. Why would think they are some sort of cabal getting kicks from leading the poor masses of Python programmers at their whim? -- https://mail.python.org/mailman/listinfo/python-list
Re: PyWart: Python's import statement and the history of external dependencies
On Fri, Nov 21, 2014 at 11:24 AM, Rick Johnson rantingrickjohn...@gmail.com wrote: Are you also going to call drivers fools because they bought a certain brand of car only to have the airbag explode in their face? No, but I'll call them fools if they buy a car and the engine catches fire because they never bothered to change the oil. If you don't want to have module name collisions, then don't create modules with names that are likely to collide when Python gives you an excellent tool for avoiding collisions (namespaces). Don't go blaming Python for bad design when you couldn't even be bothered to use the tools made available to you. Now you can drop as much stuff in there as you like, and none of it will ever conflict with the standard library (unless a standard ricklib module is added, which is unlikely). Yes, and now we've solved one problem by replacing it with it's inverse -- try importing the *python lib* calendar module and all you will get is your local intra-package version. Now, the only way to get to the lib module is by mutilating sys.path, or using an import utility module to import by filepath. Um, no. If your calendar module is named ricklib.calendar, then importing just calendar will import the standard library calendar. The only exception is if you're doing import calendar from inside the ricklib package, and you're using Python 2, and you don't have from __future__ import absolute_import at the top of your module. The solution to this is easy: just add that __future__ import to the top of your module, and poof, implicit relative imports don't happen. This is also fixed entirely in Python 3. This is the second rather blatant error you've made about Python's import mechanism, which makes me suspect that you don't really know very much about it. Anyone would expect that when *DIRECTLY* importing a package, if the __init__ file has code, then THAT code should be executed, HOWEVER, not many would expect that merely referencing the package name (in order to import a more deeply nested package) would cause ALL the intermediate __init__ files to execute -- this is madness, and it prevents using an __init__ file as an import hub (without side-effects)! The whole point of the __init__.py file, in case you didn't intuit it from the name, is to host any initialization code for the package. Why on earth would you expect to import a module from a package without initializing the package? Because the alternative is messy. If i have a collection of modules under a package, sometimes i would like to import all the exportable objects into the __init__ file and use the package as an import hub. What is the point of putting things into a hierarchical namespace in the first place if you're just going to turn around and subvert it like this? But the current global import search path injections are just the inverse. You make changes to sys.path in one module, and if you fail to reset the changes before execution moves to the next module in the import chain, then that module's import search path will be affected in implicit ways that could result in importing the wrong module. No, because the trick you describe doesn't even work. If you edit sys.path in one file in order to import the coconut module: sys.path.insert(0, '/path/to/island') import coconut And then in another module change the sys.path file and try to import a different coconut module: sys.path[0] = '/path/to/other/island') import coconut You think the second import will produce the second coconut module? It won't, because the sys.modules cache will already contain an entry for 'coconut' that points to the first module imported. In order to make this work, you would have to not only modify sys.path but also clear the sys.modules cache. Hopefully by the time you've done that you will have realized that you're abusing the import system by having multiple modules with the same name, and that as a general rule modules shouldn't be manipulating the value of sys.path *at all*. Instead, set your sys.path correctly from the PYTHONPATH environment variable, and if you really must modify sys.path, try to do it only from the main script. It's three lines of code to replace one. Two if you exclude the importlib.machinery import that doesn't need to be repeated. Is this really any worse than something like: local_search_path.insert(0, /path/to/local/module) import my_local_module that you are proposing? If the changes were LOCAL, then i would have no problem to this type of mutation, But they are not. The example of a direct loader call that described as boilerplate hell, to which I was responding, doesn't change anything, locally or globally. All it does is import a module from an arbitrary location. -- https://mail.python.org/mailman/listinfo/python-list
Re: PyWart: Python's import statement and the history of external dependencies
On Friday, November 21, 2014 1:06:18 PM UTC-6, Michael Torrie wrote: On 11/21/2014 11:24 AM, Rick Johnson wrote: Why am *i* the fool when it's obvious that the creators of Python were either shortsighted and/or careless with the designs? The only people who suffer are those who put their trust in the designer, and not the designer himself -- something is wrong with this picture! So if you know how to create the perfect language that has a solution to these very difficult problems, please come forward with it. Until such time, it's okay to identify problems with the language as you see them, but to cast aspersion on the developers is out of line. Do you know Guido personally to know that he's short-sighted or careless? Not personally. But how will we *ever* know if he refuses to participate in these discussions? YE SHALL *KNOW* THEM BY THEIR FRUITS! Although GvR has created one of the best scripting languages known to man, he has utterly failed to foster a sense of community between the members, and no matter how great Python might be, the fact that our community is nothing more than a *privileged* white-boys club is nothing less than self-destructive. But even ignoring the atrocious sexual and racial diversity of this community, outside of Python-dev and Python-ideas, GvR has refuses to engage with the mainstream alter boys. And even WITHIN those moderated groups, he's very *selective* about who he will respond to. THIS COMMUNITY IS *ONLY* SUPERFICIALLY FREE! I know we just *LOVE* to advertise our commitment to diversity[1] but the truth is, our commitment is only a friendly mask covering the evil truth. The truth is, there exists a STRONG undercurrent of hostility towards ANY dissent or outside influence within the upper echelons of this community. This is NOT a free and open society that prides itself on building heterogeneous relationships to solve complex problems, no, this is a *Theocracy*, where one man is raised above all as though his phallus has been plated with gold! And although he has not taken hammer and chisel to stone tablets and carved malevolent laws by his own hand, by virtue of his silence, he HAS implicitly defined a religious creed which forbids all free thought and expression -- and those who *dare* to dissent will be excommunicated. That's just what they did over at Python- ideas not to long ago! WHY WOULD THEY DO THAT? GvR has effectively walled himself off from the community with the exception of a few advisers which he allows to enter his inner circle. This is NOT how you treat people who are part of *YOUR* community, that is, IF you want to call yourself benevolent! SO DROP THE BENEVOLENT ACT ALREADY, IT'S PATRONIZING! If he *REALLY* wants to earn that benevolent badge of his, then he needs to demand that all these forums become free and open to all opinions and ideas. But more importantly, he needs to start participating with the community at all levels, not just hiding behind the coat tails of his attack dogs and brown-nosers. START BEARING SOME PALATABLE FRUITS! [1]: https://www.python.org/community/diversity/ -- https://mail.python.org/mailman/listinfo/python-list
Re: PyWart: Python's import statement and the history of external dependencies
On Friday, November 21, 2014 1:24:53 PM UTC-6, Ian wrote: On Fri, Nov 21, 2014 at 11:24 AM, Rick Johnson Are you also going to call drivers fools because they bought a certain brand of car only to have the airbag explode in their face? No, but I'll call them fools if they buy a car and the engine catches fire because they never bothered to change the oil. As Dennis pointed out that's highly unlikely. If you don't want to have module name collisions, then don't create modules with names that are likely to collide when Python gives you an excellent tool for avoiding collisions (namespaces). Don't go blaming Python for bad design when you couldn't even be bothered to use the tools made available to you. And by namespaces you must be talking about packages here? Okay, hold that thought... Now you can drop as much stuff in there as you like, and none of it will ever conflict with the standard library (unless a standard ricklib module is added, which is unlikely). Yes, and now we've solved one problem by replacing it with it's inverse -- try importing the *python lib* calendar module and all you will get is your local intra-package version. Now, the only way to get to the lib module is by mutilating sys.path, or using an import utility module to import by filepath. Um, no. If your calendar module is named ricklib.calendar, then importing just calendar will import the standard library calendar. The only exception is if you're doing import calendar from inside the ricklib package, and you're using Python 2, and you don't have from __future__ import absolute_import at the top of your module. The solution to this is easy: just add that __future__ import to the top of your module, and poof, implicit relative imports don't happen. This is also fixed entirely in Python 3. I wish the irony of needing to know an implicit rule to solve an implicit problem could be funny, but it's really just sad. I can't help but be reminded of the Python zen. If it's difficult to explain it's probably a bad idea. What's more difficult to explain than an implicit rule you have no knowledge of? NOW THERE'S SOME IRONY FOR YOU! Anyone would expect that when *DIRECTLY* importing a package, if the __init__ file has code, then THAT code should be executed, HOWEVER, not many would expect that merely referencing the package name (in order to import a more deeply nested package) would cause ALL the intermediate __init__ files to execute -- this is madness, and it prevents using an __init__ file as an import hub (without side-effects)! The whole point of the __init__.py file, in case you didn't intuit it from the name, is to host any initialization code for the package. Why on earth would you expect to import a module from a package without initializing the package? Because you failed to notice that i was NOT importing the package which contained the __init__file, no, i was importing a *SUB PACKAGE* of the package that contained the __init__ file. Why does the code in the main package need to run when i *explicitly* and *directly* fetched a nested resource within the package? Nothing within the __init__ file is affecting the code within the subpackage i wanted, and inversely, the package i wanted does not depend on anything within the __init__ file. There exists no symbiotic relationship between these two machinery, yet, by referencing one of them, the other starts doing unnecessary things! There is a workaround, but it's even more of a mess. In order to maintain a unique import hub without the chance of side effects from __init__ files, i can move all the code in ricklib.subpkg1.__init__.py (the code that does all the imports) into a normal file named ricklib.subpkg1._subpkg1.py. + ricklib __init__.py + subpkg1 (ricklib.subpkg1) __init__.py _subpkg1.py module1.py module2.py module3.py + subpkg1a (ricklib.subpkg1.subpkg1a) Now, since the __init__ file has no global code, i can import ricklib.subpkg1.subpkg1s without side effect -- but of course, at a cost! Advanced Python packages are a zero sum game. You cannot remove the problems, all you can do is move them to new locations. so instead of the former (with side effects): from ricklib.subpkg1.subpkg1a import something I can do this (without side effect): from ricklib.subpkg1.subpkg1a._subpkg1a import something But at the cost of my sanity. Because the alternative is messy. If i have a collection of modules under a package, sometimes i would like to import all the exportable objects into the __init__ file and use the package as an import hub. What is the point of putting things into a hierarchical namespace in the first place if you're just going to turn around and subvert it like this? I'm not subverting it, i'm merely trying to organize my *VAST* code libraries utilizing the ONLY tools that have been
Re: PyWart: Python's import statement and the history of external dependencies
On 11/21/2014 01:29 PM, Rick Johnson wrote: Not personally. But how will we *ever* know if he refuses to participate in these discussions? Why should he participate in these discussions? Why should you be in charge of said discussions? By the way, Python has more than certainly borne fruit, and the vast majority of it is very good indeed. -- https://mail.python.org/mailman/listinfo/python-list
Re: PyWart: Python's import statement and the history of external dependencies
On Friday, November 21, 2014 4:25:49 PM UTC-6, Rick Johnson wrote: # STEP 3 # # Make the following changes to the import machinery: # # Before Python reads a module file, Python will clear the # # values in sys.path_extra, OR, query the# # __search_paths__ variable, if any paths exists in this # # list, THEN THESE PATHS MUST BE SEARCHED, AND THEY MUST # # BE SEARCHED BEFORE ANY PATHS IN sys.path, AND NO # # PEEKING IN sys.modules IS ALLOWED! # Actually, to be more specific, not only should the __search_path__ variable be unique to each file, Python should query the value of __search_path__ variable each time it encounters an import statement. So, if we wanted to import multiple modules who share the same name, but who exist in different namespaces, we would could do so by changing the values of __search_path__ before calling the import: # # import the first coconut module Python can find using # the default search path, if not raise an error # import coconut # # import the first coconut module python can find using # *ONLY* the explicit paths provided, or raise an error. # __search_path__ = [ 'path/to/a/specific/coconut/tree', 'might/be/here/also' ] import coconut At first i was thinking that *IF* any explicit paths were defined via __search_path__, then Python should look in all those paths first, then if nothing is found look in sys.modules, then if nothing is found, exhaust sys.modules, BUT NOW I AM THINKING THAT IS A BAD IDEA! When __search_path__ *IS* defined, Python should look *ONLY* in the provided paths, and if not module is found, raise an ExplictImportError. Because if we're going to be explicit then we need to also be consistent. If the programmer is requesting that Python search in a predefined directories, then obviously he doing so to avoid name clashes, so if Python went on and returned a module OUTSIDE of those predefined paths, Python would be circumventing (IMPLICITLY AGAIN!) the wishes of the programmer. Succinct Summary Of My New Proposal: 1. Use the historical implicit import mechanism for most day to day imports, and let Python do all the heavy lifting. 2. Use the new explicit import mechanism for advanced name resolutions, but realize that since you are now taking control of import, so you must do more work, and you must be exhaustively explicit about *where* Python searches. -- https://mail.python.org/mailman/listinfo/python-list
Re: PyWart: Python's import statement and the history of external dependencies
On Sat, Nov 22, 2014 at 10:21 AM, Rick Johnson rantingrickjohn...@gmail.com wrote: 1. Use the historical implicit import mechanism for most day to day imports, and let Python do all the heavy lifting. 2. Use the new explicit import mechanism for advanced name resolutions, but realize that since you are now taking control of import, so you must do more work, and you must be exhaustively explicit about *where* Python searches. In other words, what you want is: # today's method, import based on search path import sys # new explicit path method import '/usr/local/lib/python3.5/lib-dynload/math.cpython-35m.so' Can you tell me, please, how this is going to work for *any* system other than the one it was built on? One of the reasons I write Python code is because I expect it to work unchanged on Windows, Linux, Mac OS, BSD, OS/2, uPy, and any other platform I care to throw it onto. If I want to hard-code all the details, I might as well write C code and at least take advantage of ./configure to work out the right paths for me. There are a VERY VERY few cases where you really do want to import a module from a specific file path. For those situations, there are already techniques you can use. Why change the behaviour of import itself? ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: PyWart: Python's import statement and the history of external dependencies
On Fri, Nov 21, 2014 at 3:25 PM, Rick Johnson rantingrickjohn...@gmail.com wrote: The only exception is if you're doing import calendar from inside the ricklib package, and you're using Python 2, and you don't have from __future__ import absolute_import at the top of your module. The solution to this is easy: just add that __future__ import to the top of your module, and poof, implicit relative imports don't happen. This is also fixed entirely in Python 3. I wish the irony of needing to know an implicit rule to solve an implicit problem could be funny, but it's really just sad. I can't help but be reminded of the Python zen. If it's difficult to explain it's probably a bad idea. What's more difficult to explain than an implicit rule you have no knowledge of? It's not so much implicit as an old wart that was fixed in a backward-compatible way. As I said it's not an issue in Python 3, so it's kind of pointless to be griping about it now. Anyone would expect that when *DIRECTLY* importing a package, if the __init__ file has code, then THAT code should be executed, HOWEVER, not many would expect that merely referencing the package name (in order to import a more deeply nested package) would cause ALL the intermediate __init__ files to execute -- this is madness, and it prevents using an __init__ file as an import hub (without side-effects)! The whole point of the __init__.py file, in case you didn't intuit it from the name, is to host any initialization code for the package. Why on earth would you expect to import a module from a package without initializing the package? Because you failed to notice that i was NOT importing the package which contained the __init__file, no, i was importing a *SUB PACKAGE* of the package that contained the __init__ file. No, I understood that. My question stands. Why does the code in the main package need to run when i *explicitly* and *directly* fetched a nested resource within the package? Nothing within the __init__ file is affecting the code within the subpackage i wanted, and inversely, the package i wanted does not depend on anything within the __init__ file. There exists no symbiotic relationship between these two machinery, yet, by referencing one of them, the other starts doing unnecessary things! It has nothing to do with the __init__ file specifically. When you import ham.spam.eggs, Python first imports ham, then ham.spam, then ham.spam.eggs. The reason for this should be obvious: the module is imported as a local variable in whatever context the code is running in. ham.spam.eggs is not a legal variable name. Instead it has to use the variable ham, and populate it with the module ham. The module ham then gets an attribute (i.e. module-level global) named spam that is populated with the module ham.spam. Finally that module gets an attribute eggs that is populated with the ham.spam.eggs module. Because of this, it is impossible to import the module ham.spam.eggs (under that name, anyway) without first importing ham and ham.spam. Another reason is that a package can affect the way its contents are imported, e.g. by setting a package-level __path__ variable. But also, the __init__.py is executed because it is considered to be the initializer for the entire package. Would you expect to be able to create a class instance and call one of its methods without having the __init__ method called? The __init__.py file is called that because it's meant to be analogous. What you seem to be asking for is a way lump a subpackage into a package without treating that higher-level package as a package. There is a way to do this, using namespace packages, which have no __init__.py files. There is a workaround, but it's even more of a mess. In order to maintain a unique import hub without the chance of side effects from __init__ files, i can move all the code in ricklib.subpkg1.__init__.py (the code that does all the imports) into a normal file named ricklib.subpkg1._subpkg1.py. Or you could just accept that if you don't want importing ham to automatically import ham.spam, then you shouldn't put that import in ham/__init__.py. Most __init__.py files *should* be empty. If Python expects me to use packages to protect my module names from clashing, but it gives me no method by which to import packages without causing side effects, then what is a boy to do (besides creating workarounds for workarounds)? If you don't want any initialization to happen when you import your modules, then don't put any initialization code in your __init__.py. Or use namespace packages as mentioned above. -- https://mail.python.org/mailman/listinfo/python-list
Re: PyWart: Python's import statement and the history of external dependencies
On Friday, November 21, 2014 3:21:31 PM UTC-8, Rick Johnson wrote: On Friday, November 21, 2014 4:25:49 PM UTC-6, Rick Johnson wrote: # STEP 3 # # Make the following changes to the import machinery: # # Before Python reads a module file, Python will clear the # # values in sys.path_extra, OR, query the# # __search_paths__ variable, if any paths exists in this # # list, THEN THESE PATHS MUST BE SEARCHED, AND THEY MUST # # BE SEARCHED BEFORE ANY PATHS IN sys.path, AND NO # # PEEKING IN sys.modules IS ALLOWED! # Actually, to be more specific, not only should the __search_path__ variable be unique to each file, Python should query the value of __search_path__ variable each time it encounters an import statement. So, if we wanted to import multiple modules who share the same name, but who exist in different namespaces, we would could do so by changing the values of __search_path__ before calling the import: # # import the first coconut module Python can find using # the default search path, if not raise an error # import coconut # # import the first coconut module python can find using # *ONLY* the explicit paths provided, or raise an error. # __search_path__ = [ 'path/to/a/specific/coconut/tree', 'might/be/here/also' ] import coconut At first i was thinking that *IF* any explicit paths were defined via __search_path__, then Python should look in all those paths first, then if nothing is found look in sys.modules, then if nothing is found, exhaust sys.modules, BUT NOW I AM THINKING THAT IS A BAD IDEA! When __search_path__ *IS* defined, Python should look *ONLY* in the provided paths, and if not module is found, raise an ExplictImportError. Because if we're going to be explicit then we need to also be consistent. If the programmer is requesting that Python search in a predefined directories, then obviously he doing so to avoid name clashes, so if Python went on and returned a module OUTSIDE of those predefined paths, Python would be circumventing (IMPLICITLY AGAIN!) the wishes of the programmer. Succinct Summary Of My New Proposal: 1. Use the historical implicit import mechanism for most day to day imports, and let Python do all the heavy lifting. 2. Use the new explicit import mechanism for advanced name resolutions, but realize that since you are now taking control of import, so you must do more work, and you must be exhaustively explicit about *where* Python searches. I have a better idea. Don't give modules the same name as built-in or very popular modules. You seem like the type of person that would stick their hair into an electric egg beater and then blame the manufacturer because it ripped out their hair. You'd probably say that the egg beater should recognize that there's hair in the beaters and turn off. -- https://mail.python.org/mailman/listinfo/python-list
Import a module from a specific file path (was: PyWart: Python's import statement and the history of external dependencies)
Chris Angelico ros...@gmail.com writes: In other words, what you want is: # today's method, import based on search path import sys # new explicit path method import '/usr/local/lib/python3.5/lib-dynload/math.cpython-35m.so' I don't think I'd ever want to specify an absolute file path for the module. But it would make my Python life immeasurably better if I could specify *relative* file paths for importing a module. import '../foo/bar/beans.py' as beans Can you tell me, please, how this is going to work for *any* system other than the one it was built on? Allowing relative paths makes this portable, so long as the application's relative tree structure is maintained. There are a VERY VERY few cases where you really do want to import a module from a specific file path. For those situations, there are already techniques you can use. Why change the behaviour of import itself? An example: * I write a program, ‘fooprog’, using the hypothetical “import a module at a specified file path” feature:: #! /usr/bin/python3 import './bar/beans.py' as beans beans.do_stuff() * The program ‘fooprog’ and the module ‘beans.py’ are in a sensible directory structure:: foo-proj-1.0/ foo/ __init__.py fooprog bar/ __init__.py beans.py tests/ __init__.py test_fooprog.py test_bar.py * Running the program ‘fooprog’ invokes Python (via its standard shebang), which invokes the file ‘fooprog’ as the ‘__main__’ module *with no package awareness*. Or, equivalently, ‘python foo/fooprog’ skips the shebang step. * Regardless of package considerations, specifying ‘./bar/beans.py’ locates the module ‘beans’ for import, just fine. That example will work in the presence of this “import from a named file location” feature. But it will fail with the standard Python import feature today. How would you expect this example to change so it will work with current Python? Solutions usually seem to entail contortions of cluttering the import block by discovering the current path, and fussing around with ‘sys.path’, before finally doing the import:: #! /usr/bin/python3 import sys import os.path program_dir = os.path.dirname(__file__) parent_dir = os.path.dirname(program_dir) if not parent_dir in sys.path: sys.path.insert(1, parent_dir) import foo.bar.beans as beans beans.do_stuff() Maybe you can suggest a better portable method to do this today in Python. Importantly, once you've come up with how to do it today in Python: Do you really consider that superior to simply specifying a filesystem path for a file containing the module? -- \“I'd take the awe of understanding over the awe of ignorance | `\ any day.” —Douglas Adams | _o__) | Ben Finney -- https://mail.python.org/mailman/listinfo/python-list
Re: PyWart: Python's import statement and the history of external dependencies
On Friday, November 21, 2014 5:59:44 PM UTC-6, Chris Angelico wrote: On Sat, Nov 22, 2014 at 10:21 AM, Rick Johnson rantingrickjohn...@gmail.com wrote: 1. Use the historical implicit import mechanism for most day to day imports, and let Python do all the heavy lifting. 2. Use the new explicit import mechanism for advanced name resolutions, but realize that since you are now taking control of import, so you must do more work, and you must be exhaustively explicit about *where* Python searches. In other words, what you want is: # today's method, import based on search path import sys # new explicit path method import '/usr/local/lib/python3.5/lib-dynload/math.cpython-35m.so' [...] There are a VERY VERY few cases where you really do want to import a module from a specific file path. For those situations, there are already techniques you can use. Why change the behaviour of import itself? Chris, either take the time to read and understand my posts *FULLY*, or don't bother to reply. I have in no way suggested that we import via filepaths. Stop lying. -- https://mail.python.org/mailman/listinfo/python-list
Re: PyWart: Python's import statement and the history of external dependencies
On Friday, November 21, 2014 6:33:32 PM UTC-6, Ian wrote: On Fri, Nov 21, 2014 at 3:25 PM, Rick Johnson Why does the code in the main package need to run when i *explicitly* and *directly* fetched a nested resource within the package?[...] It has nothing to do with the __init__ file specifically. When you import ham.spam.eggs [...] Would you expect to be able to create a class instance and call one of its methods without having the __init__ method called? Not an instance method, but a class method, yes! ;-) Just as Objects have both class level *AND* instance level scoping for methods, so too should packages -- in a different sort of way of course. If you don't want any initialization to happen when you import your modules, then don't put any initialization code in your __init__.py. Or use namespace packages as mentioned above. Ha, that's what i realized after being caught in the __init__ nightmare not too long ago :). But now I have my deeply nested module namespaces working just fine under Python 2.x. although i feel like the system is too clunky and in need of refining. Maybe namespace packages are what i need, although i see they are Python 3.x only. :-( In any event, thanks for taking the time to explain the details. I feel i understand the systems much better now. Your coconut example was the key to my understanding. And now, i can stand on the top of the mountain and see how the system works -- I no longer have this damn mountain blocking my view. I think Python's import and packaging systems are both very good ideas that just happen to be covered in a blanket of dust, and all we need to do it give then a good cleaning to bring out that showroom shine -- and detailing is my specialty! -- https://mail.python.org/mailman/listinfo/python-list
Re: Import a module from a specific file path (was: PyWart: Python's import statement and the history of external dependencies)
On Sat, Nov 22, 2014 at 11:37 AM, Ben Finney ben+pyt...@benfinney.id.au wrote: I don't think I'd ever want to specify an absolute file path for the module. But it would make my Python life immeasurably better if I could specify *relative* file paths for importing a module. Allowing relative paths makes this portable, so long as the application's relative tree structure is maintained. Maybe you can suggest a better portable method to do this today in Python. Ah, didn't think of relative paths. Yes, I can see that'd be both more useful and less problematic. What you suggest looks very much like you're running something from a package, though. * The program ‘fooprog’ and the module ‘beans.py’ are in a sensible directory structure:: foo-proj-1.0/ foo/ __init__.py fooprog bar/ __init__.py beans.py tests/ __init__.py test_fooprog.py test_bar.py So can you simply: $ python -m foo.fooprog ? (or 'python3', either way) ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: PyWart: Python's import statement and the history of external dependencies
On Sat, Nov 22, 2014 at 12:07 PM, Rick Johnson rantingrickjohn...@gmail.com wrote: On Friday, November 21, 2014 5:59:44 PM UTC-6, Chris Angelico wrote: On Sat, Nov 22, 2014 at 10:21 AM, Rick Johnson rantingrickjohn...@gmail.com wrote: 1. Use the historical implicit import mechanism for most day to day imports, and let Python do all the heavy lifting. 2. Use the new explicit import mechanism for advanced name resolutions, but realize that since you are now taking control of import, so you must do more work, and you must be exhaustively explicit about *where* Python searches. In other words, what you want is: # today's method, import based on search path import sys # new explicit path method import '/usr/local/lib/python3.5/lib-dynload/math.cpython-35m.so' [...] There are a VERY VERY few cases where you really do want to import a module from a specific file path. For those situations, there are already techniques you can use. Why change the behaviour of import itself? Chris, either take the time to read and understand my posts *FULLY*, or don't bother to reply. I have in no way suggested that we import via filepaths. Stop lying. Okay. Explain to me the difference between your proposed explicit import mechanism, where you specifically name the path where Python should import something from, and importing via filepaths. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: PyWart: Python's import statement and the history of external dependencies
On 16/11/2014 3:01 PM, Rick Johnson wrote: Python's attempt to solve the external dependencies problem has yet to produce the results that many people, including myself, would like. I'd say this was an argumentum ad populum, only you didn't cite anything that shows the many you claim you speak for. Actually, Python is not alone in this deficiency, no, Python is just *ANOTHER* language in a *STRING* of languages over the years who has *YET AGAIN* implemented the same old crusty design patterns, packaged them in a shiny metallic wrapping paper with a big red bow on top, and hoped that no one would notice the stench... Are you sure that's not just you? 1. Name clashes! 2. Smaller name pool! Just off the top of my head, we have several solutions for this: 1) Rebinding imports import foo as foo2 2) Namespace packages import mypackage.foo 3) Filepath imports (deprecated in 3.4 but under discussion for reprecation or replacement) import importlib.machinery loader = importlib.machinery.SourceFileLoader(module.name, /path /to/file.py) foo = loader.load_module() foo.MyClass() 4) Virtual environments In before but WHY so MANY different WAYS OMG PYTHON you M.F.P.O.S.: because they offer flexibility. 3. Machinery is too implicit! See: importlib 4. Circular imports are inevitable! They're also clear indication of bad design. That's not something Python should explicitly set out to support. 5. Much too difficult to use and/or explain! Well sure, if your reaction to things you don't understand is to rant against them instead of asking questions, then I can see how that would be true. Also: citation needed. 6. Too many gotchas! Is this the sequel to Too Many Cooks? Awesome. In closing, there is only one thing you need to know about Python's import statement: it is deceptively easy to underestimate it's ability to *EFF UP* your expectations! There's your problem right there: a language conforms to its specification, not your random expectation. I'm eager to see how you resolve your concerns in RickPython3000. -- https://mail.python.org/mailman/listinfo/python-list
Re: PyWart: Python's import statement and the history of external dependencies
On Fri, Nov 21, 2014 at 11:14 AM, alex23 wuwe...@gmail.com wrote: 1. Name clashes! 2. Smaller name pool! Just off the top of my head, we have several solutions for this: 1) Rebinding imports import foo as foo2 To be fair to Rick, this doesn't actually solve anything. If you have two modules called foo, you can't import one of them this way. But part of the problem is actually before you even get as far as importing: it's the nature of open source, forking, and the internet. There's a module called remix, which one Peter Sobot forked and tweaked for his own use, and then distributed a slightly changed version of. (I believe the original intention was for the patches to be incorporated into trunk, but that hasn't happened as yet.) If you want to use his app, you need to use his tweaked remix module; if you want to use someone else's, you might need to use the original from upstream. What do you do about that? They're both called remix. How do you import one or the other? How do you install one or the other? There is fundamentally no solution to this. As soon as there are two incompatible (even just slightly incompatible) versions of a project in the wild, people need to cope with it. It's the same with avconv vs ffmpeg, it's the same with wodim vs cdrecord, it's the same with all of them - until one of them undergoes a name change, everyone's stuck. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: PyWart: Python's import statement and the history of external dependencies
On Thursday, November 20, 2014 6:15:03 PM UTC-6, alex23 wrote: On 16/11/2014 3:01 PM, Rick Johnson wrote: [...] Actually, Python is not alone in this deficiency, no, Python is just *ANOTHER* language in a *STRING* of languages over the years who has *YET AGAIN* implemented the same old crusty design patterns, packaged them in a shiny metallic wrapping paper with a big red bow on top, and hoped that no one would notice the stench... Are you sure that's not just you? 1. Name clashes! 2. Smaller name pool! Just off the top of my head, we have several solutions for this: 1) Rebinding imports import foo as foo2 2) Namespace packages import mypackage.foo I was not talking about name clashes within the script (which are the problem of the programmer, because he and only he has *DIRECT* control over the contents of any scripts he writes!) no, i was talking about two or more modules who share the same name who then, in effect, *SHADOW* each other (this is problem for which the programmer *CANNOT* solve *INTUITIVELY* because he cannot *EASILY* control the manner in which the underlying import mechanism finds modules). If you had taken the time to read my example utilizing a lobby boy, then you might have understood what i was talking about. FOR INSTANCE: Let's say i write a module that presents a reusable GUI calendar widget, and then i name the module calender.py. Then Later, when i try to import *MY* GUI widget named calendar, i will not get *MY* calendar widget, no, i will get the Python calendar module. The reason Python finds the library module instead of my module is because the Python library is higher in search path than my module. OKAY, FAIR ENOUGH... Now, i could *TRY* to be cleaver and inject the directory that holds *MY* calendar module as the *FIRST* directory of sys.path, and this will work, but it's a TOTAL FREAKING HACK, and unbeknownst to most first-time import hackers, the changes to sys.path does not only affect the contents of sys.path within the current module, but *ALL* modules further down the import chain will suffer the changes! #BE WARNED:# # Contrary to popular belief, sys.path is *NOT* a module, # # no, it's a global! # This is another confusing fundamental of Python, it seems *MORE* intuitive that changes to the import search path would only affect the *CURRENT* module's import search path, but that's is not the case! #Lesson Of The Day # # Try to be intuitive, and python will *break* you!# And you know why, it's because our dear leaders have this irrational phobia of global variables. But the joke is on them, because they fail to realize that importing a module into more than one namespace is creating globals! YOU CAN PUT LIPSTICK ON A PIG, BUT IT'S STILL A PIG! This is what happens when your emperor stops wearing clothes, and more proof that Python's import mechanism sucks dingo balls! Now some people will say: Don't name your modules the same as a Python library modules and problem solved! Sorry, but that's not good enough for me! I believe that the current import mechanism is only *HALF* of what it should be. Currently, import uses an implicit name resolution via predefined paths in sys.path, and for 75% of the time, this implicit import mechanism will work just fine -- it's a slow algorithm, but fast enough for humans not to notice. HOWEVER, in the 25% of cases where *NAME SHADOWING* or where the author knows the location and can specify the location to save lobby boy from knocking on doors until his knuckles bleed, the author should have an option defining the search path *LOCALLY* (in a manner that will *NOT* affect the import search path further down the import chain), WITHOUT needing to import one of the many discombobulated and confusing import tools that Python has tortured this community with. 3) Filepath imports (deprecated in 3.4 but under discussion for reprecation or replacement) import importlib.machinery loader = importlib.machinery.SourceFileLoader(module.name, /path /to/file.py) foo = loader.load_module() foo.MyClass() That's ridiculously noisy. I have an idea, if you *REALLY* want to introduce boilerplate hell then Python should adopt import header files, NOT! 4) Virtual environments In before but WHY so MANY different WAYS OMG PYTHON you M.F.P.O.S.: because they offer flexibility. 3. Machinery is too implicit! See: importlib 4. Circular
PyWart: Python's import statement and the history of external dependencies
Python's attempt to solve the external dependencies problem has yet to produce the results that many people, including myself, would like. Actually, Python is not alone in this deficiency, no, Python is just *ANOTHER* language in a *STRING* of languages over the years who has *YET AGAIN* implemented the same old crusty design patterns, packaged them in a shiny metallic wrapping paper with a big red bow on top, and hoped that no one would notice the stench... WELL I NOTICED M.F.! Before we can formulate a solution to this mess, we must first obtain an inside perspective of the world in which a Python script lives during run-time. Welcome to Python Gardens: a master planned community Well... urm, sort of. @_@ I like to analyze problems, when possible, in relation to real world tangible examples. So for me, i like to think of the main script (aka: __main__) as an apartment building, and each module that runs under the main script as a single apartment, and finally, the inhabitants of the apartments as objects. Furthermore, we can define the apartment building as being global space, and each individual apartment as being local space. The first law we encounter is that global space is reserved for all tenants/guest, but local space is by *invitation* only! You can think of import as similar to sending out an invitation, and requesting that a friend join you inside your apartment (we'll get back to import later). And of course, with this being the 21st century and all, every apartment has *local* access to *global* resources like water and electrical. In Python, these global resources are available locally in every module via the implicit namespace of __builtins__. You can think of built-ins as a house keeper robot that lives in every apartment (I call mine Rosie). It was there when you moved in --a house warming gift of sorts-- and it helps you with chores and makes your life a little easier. Sometimes i abuse my robot but i always apologize afterwards. Now that we know how the global and local spaces are defined (in the context of modules), and what implicit/explicit gifts (are supplied), and rules (are demanded) in our little Python world, we have a good basis to start understanding why Python's import mechanism is at best a helpful failure that only *naively* attempts to streamline the management of such an important language feature as external dependencies! Up until this point, our apartment example has not mapped the actual *mechanics* of import to a real world example, but now that we have the correct perspective, we can tread into the dark and damp underworld that is import. Remember when i said: import is like sending out an invitation? Well, in actuality, import is only *superficially* similar to sending out an invitation. You see, when you send an invitation in real life, the most important two things you'll need are a *name* and an *address* --a name so you'll know *who* to send the invitation to, and an address so you'll know *where* to send the invitation-- but Python's import mechanism does not work that way. When you import an external dependency into a Python module all you need is the name of a module -- addresses are neither required *OR* allowed! Like almost all modern languages, Python has adopted the ubiquitous practice of internalizing a mapping of known directories from which to search for modules (called a search path), so when we import somemodule the following happens (Note: I'm not going into too many details for the sake of topic): 1. Python checks if the external resource has already been loaded if not 1, then 2. Python looks in a few special places if not 2, then 3. Python searches one-by-one the directories in sys.path RESULT: Python finds the resource or barfs an exception. I have no problem with step 1, however, step 2 and step 3 can be redundant, excessive, and even unreliable. I can explain better why i levee such harsh words for import by going back to our apartment building example. Let's imagine that Python is the lobby boy of our apartment building. And in our little example, one of the duties of the lobby boy is to manage invitations between tenants and guests. Each time a tenant wants to invite someone into their apartment, they must pick up the phone, press the import button (which connects them to the lobby boy via a voice connection) and they say the name of the person they want to invite. But *EVEN IF* they know the address of the recipient, they are not allowed to say the address to the lobby boy, no, because *THIS* building is owned by a evil tyrant, who has declared that any mention of an address when calling import is punishable by the fatal exception! So, being a good tenant who does not want to *DIE*, they simply mention
Re: PyWart: Python's import statement and the history of external dependencies
On Sun, Nov 16, 2014 at 4:01 PM, Rick Johnson rantingrickjohn...@gmail.com wrote: Creating an implicit name resolution system (aka: import) to abstract away an explicit name resolution system (file-paths) has resulted in more problems that it can solve: 1. Name clashes! 2. Smaller name pool! 3. Machinery is too implicit! 4. Circular imports are inevitable! 5. Much too difficult to use and/or explain! 6. Too many gotchas! And the ability to write cross-platform code! ChrisA -- https://mail.python.org/mailman/listinfo/python-list
