Re: Python 3.12.1, Windows 11: shebang line #!/usr/bin/env python3 doesn't work any more
On 1/15/2024 7:24 PM, Thomas Passin wrote: On 1/15/2024 6:27 PM, Greg Ewing via Python-list wrote: On 16/01/24 11:55 am, Mats Wichmann wrote: Windows natively has something called python.exe and python3.exe which is interfering here I'm wondering whether py.exe should be taught to recognise these stubs and ignore them. This sounds like something that could trip a lot of people up. There are registry entries that say where all the python.org install locations are. I suppose, but don't know, that py.exe checks them. The registry entries are in Computer\HKEY_CURRENT_USER\SOFTWARE\Python\PythonCore For python.org installs that are installed for all users, the entries are in Computer\HKEY_CURRENT_USER\SOFTWARE\Python\PythonCore -- https://mail.python.org/mailman/listinfo/python-list
Re: Question about garbage collection
On 1/15/2024 9:47 PM, Akkana Peck via Python-list wrote: I wrote: Also be warned that some modules (particularly if they're based on libraries not written in Python) might not garbage collect, so you may need to use other methods of cleaning up after those objects. Chris Angelico writes: Got any examples of that? The big one for me was gdk-pixbuf, part of GTK. When you do something like gtk.gdk.pixbuf_new_from_file(), there's a Python object that gets created, but there's also the underlying C code that allocates memory for the pixbuf. When the object went out of scope, the Python object was automatically garbage collected, but the pixbuf data leaked. This kind of thing can happen with PyQt, also. There are ways to minimize it but I don't know if you can ever be sure all Qt C++ objects will get deleted. It depends on the type of object and the circumstances. Calling gc.collect() caused the pixbuf data to be garbage collected too. There used to be a post explaining this on the pygtk mailing list: the link was http://www.daa.com.au/pipermail/pygtk/2003-December/006499.html but that page is gone now and I can't seem to find any other archives of that list (it's not on archive.org either). And this was from GTK2; I never checked whether the extra gc.collect() is still necessary in GTK3, but I figure leaving it in doesn't hurt anything. I use pixbufs in a tiled map application, so there are a lot of small pixbufs being repeatedly read and then deallocated. ...Akkana -- https://mail.python.org/mailman/listinfo/python-list
Re: Question about garbage collection
On Tue, 16 Jan 2024 at 13:49, Akkana Peck via Python-list wrote: > > I wrote: > > > Also be warned that some modules (particularly if they're based on > > > libraries not written in Python) might not garbage collect, so you may > > > need to use other methods of cleaning up after those objects. > > Chris Angelico writes: > > Got any examples of that? > > The big one for me was gdk-pixbuf, part of GTK. When you do something like > gtk.gdk.pixbuf_new_from_file(), there's a Python object that gets created, > but there's also the underlying C code that allocates memory for the pixbuf. > When the object went out of scope, the Python object was automatically > garbage collected, but the pixbuf data leaked. Calling gc.collect() caused > the pixbuf data to be garbage collected too. > > There used to be a post explaining this on the pygtk mailing list: the link > was > http://www.daa.com.au/pipermail/pygtk/2003-December/006499.html > but that page is gone now and I can't seem to find any other archives of that > list (it's not on archive.org either). And this was from GTK2; I never > checked whether the extra gc.collect() is still necessary in GTK3, but I > figure leaving it in doesn't hurt anything. I use pixbufs in a tiled map > application, so there are a lot of small pixbufs being repeatedly read and > then deallocated. > Okay, so to clarify: the Python object will always be garbage collected correctly, but a buggy third-party module might have *external* resources (in that case, the pixbuf) that aren't properly released. Either that, or there is a reference loop, which doesn't necessarily mean you NEED to call gc.collect(), but it can help if you want to get rid of them more promptly. (Python will detect such loops at some point, but not always immediately.) But these are bugs in the module, particularly the first case, and should be considered as such. 2003 is fully two decades ago now, and I would not expect that a serious bug like that has been copied into PyGObject (the newer way of using GTK from Python). So, Python's garbage collection CAN be assumed to "just work", unless you find evidence to the contrary. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Question about garbage collection
I wrote: > > Also be warned that some modules (particularly if they're based on > > libraries not written in Python) might not garbage collect, so you may need > > to use other methods of cleaning up after those objects. Chris Angelico writes: > Got any examples of that? The big one for me was gdk-pixbuf, part of GTK. When you do something like gtk.gdk.pixbuf_new_from_file(), there's a Python object that gets created, but there's also the underlying C code that allocates memory for the pixbuf. When the object went out of scope, the Python object was automatically garbage collected, but the pixbuf data leaked. Calling gc.collect() caused the pixbuf data to be garbage collected too. There used to be a post explaining this on the pygtk mailing list: the link was http://www.daa.com.au/pipermail/pygtk/2003-December/006499.html but that page is gone now and I can't seem to find any other archives of that list (it's not on archive.org either). And this was from GTK2; I never checked whether the extra gc.collect() is still necessary in GTK3, but I figure leaving it in doesn't hurt anything. I use pixbufs in a tiled map application, so there are a lot of small pixbufs being repeatedly read and then deallocated. ...Akkana -- https://mail.python.org/mailman/listinfo/python-list
Re: Python 3.12.1, Windows 11: shebang line #!/usr/bin/env python3 doesn't work any more
On 1/15/2024 6:27 PM, Greg Ewing via Python-list wrote: On 16/01/24 11:55 am, Mats Wichmann wrote: Windows natively has something called python.exe and python3.exe which is interfering here I'm wondering whether py.exe should be taught to recognise these stubs and ignore them. This sounds like something that could trip a lot of people up. There are registry entries that say where all the python.org install locations are. I suppose, but don't know, that py.exe checks them. The registry entries are inComputer\HKEY_CURRENT_USER\SOFTWARE\Python\PythonCore -- https://mail.python.org/mailman/listinfo/python-list
Re: Python 3.12.1, Windows 11: shebang line #!/usr/bin/env python3 doesn't work any more
On 16/01/24 11:55 am, Mats Wichmann wrote: Windows natively has something called python.exe and python3.exe which is interfering here I'm wondering whether py.exe should be taught to recognise these stubs and ignore them. This sounds like something that could trip a lot of people up. -- Greg -- https://mail.python.org/mailman/listinfo/python-list
Re: Python 3.12.1, Windows 11: shebang line #!/usr/bin/env python3 doesn't work any more
On 1/15/24 12:01, Thomas Passin via Python-list wrote: On 1/15/2024 1:26 PM, Mats Wichmann via Python-list wrote: On 1/15/24 09:44, Sibylle Koczian via Python-list wrote: First and foremost I want to understand why I'm seeing this: - Python scripts with "/usr/bin/env python3" as shebang line work as expected on a computer with Windows 10 and Python 3.11.5. They have worked for years on this machine, using either the latest Python or one version before (depending on availability of some packages). There is a virtual machine with ArchLinux on the same machine and some of the scripts are copies from that. - I've got a second computer with Windows 11 and I installed Python 3.12.1 on it. After copying some scripts from my first computer I found that I couldn't start them: not by entering the script name in a console, not using py.exe, not double clicking in the explorer. Entering \python probably worked - I think I tried that too, but I'm not really sure, because that's really not practical. In the Python documentation for versions 3.11 and 3.12 I found no differences regarding py.exe and shebang lines. Then I removed the "/env" from the shebang lines and could start the scripts from the second computer. That certainly is a solution, but why??? It's because of Windows itself. The default nowadays is that irritating little stub that prompts you to go install Python from the WIndows store. When you use the "env" form, it looks for python (or python3 in your case) in the PATH *first* and you'll get a hit. Mine looks like: C:\Users\mats\AppData\Local\Microsoft\WindwsApps\python.exe and python3.exe you can check what it's doing for you by using the "where" command in a windows shell. On your older Windows 10 machine you either never had that stub - I don't know when it was added, maybe someone from Microsoft listening here knows - or it's been superseded by changes to the PATH, or something. On my fairly new Win 11 box the base of that path is early in the user portion of PATH, so that must be a default. py.exe without the "/usr/bin/env" magic doesn't put PATH searching first, according to that snip from the docs that's been posted here several times., so you shouldn't fall down that particular rathole. Python from the App Store is not the same as Python from python.org: yes. this question is about the python.org distribution. but, Windows natively has something called python.exe and python3.exe which is interfering here, IF the python.org install isn't directed to put itself into the path, AND if the "#!/usr/bin/env python3" form is used, causing a search in PATH, which is the setup Sibylle has described, unless I've misunderstood details. -- https://mail.python.org/mailman/listinfo/python-list
Re: Question about garbage collection
On Tue, 16 Jan 2024 at 06:32, Akkana Peck via Python-list wrote: > > > Frank Millman wrote at 2024-1-15 15:51 +0200: > > >I have read that one should not have to worry about garbage collection > > >in modern versions of Python - it 'just works'. > > Dieter Maurer via Python-list writes: > > There are still some isolated cases when not all objects > > in an unreachable cycle are destroyed > > (see e.g. step 2 of > > "https://devguide.python.org/internals/garbage-collector/index.html#destroying-unreachable-objects;). > > Also be warned that some modules (particularly if they're based on libraries > not written in Python) might not garbage collect, so you may need to use > other methods of cleaning up after those objects. > Got any examples of that? ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Question about garbage collection
> Frank Millman wrote at 2024-1-15 15:51 +0200: > >I have read that one should not have to worry about garbage collection > >in modern versions of Python - it 'just works'. Dieter Maurer via Python-list writes: > There are still some isolated cases when not all objects > in an unreachable cycle are destroyed > (see e.g. step 2 of > "https://devguide.python.org/internals/garbage-collector/index.html#destroying-unreachable-objects;). Also be warned that some modules (particularly if they're based on libraries not written in Python) might not garbage collect, so you may need to use other methods of cleaning up after those objects. ...Akkana -- https://mail.python.org/mailman/listinfo/python-list
Re: Python 3.12.1, Windows 11: shebang line #!/usr/bin/env python3 doesn't work any more
On 1/15/2024 1:26 PM, Mats Wichmann via Python-list wrote: On 1/15/24 09:44, Sibylle Koczian via Python-list wrote: First and foremost I want to understand why I'm seeing this: - Python scripts with "/usr/bin/env python3" as shebang line work as expected on a computer with Windows 10 and Python 3.11.5. They have worked for years on this machine, using either the latest Python or one version before (depending on availability of some packages). There is a virtual machine with ArchLinux on the same machine and some of the scripts are copies from that. - I've got a second computer with Windows 11 and I installed Python 3.12.1 on it. After copying some scripts from my first computer I found that I couldn't start them: not by entering the script name in a console, not using py.exe, not double clicking in the explorer. Entering \python probably worked - I think I tried that too, but I'm not really sure, because that's really not practical. In the Python documentation for versions 3.11 and 3.12 I found no differences regarding py.exe and shebang lines. Then I removed the "/env" from the shebang lines and could start the scripts from the second computer. That certainly is a solution, but why??? It's because of Windows itself. The default nowadays is that irritating little stub that prompts you to go install Python from the WIndows store. When you use the "env" form, it looks for python (or python3 in your case) in the PATH *first* and you'll get a hit. Mine looks like: C:\Users\mats\AppData\Local\Microsoft\WindwsApps\python.exe and python3.exe you can check what it's doing for you by using the "where" command in a windows shell. On your older Windows 10 machine you either never had that stub - I don't know when it was added, maybe someone from Microsoft listening here knows - or it's been superseded by changes to the PATH, or something. On my fairly new Win 11 box the base of that path is early in the user portion of PATH, so that must be a default. py.exe without the "/usr/bin/env" magic doesn't put PATH searching first, according to that snip from the docs that's been posted here several times., so you shouldn't fall down that particular rathole. Python from the App Store is not the same as Python from python.org: "The Microsoft Store package is a simple installation of Python that is suitable for running scripts and packages, and using IDLE or other development environments. It requires Windows 10 and above, but can be safely installed without corrupting other programs. It also provides many convenient commands for launching Python and its tools." - https://docs.python.org/3/using/windows.html Also: "The Windows Store distribution of Python is a sandboxed application ... The internal components of Windows Store apps are protected from being accessed from other applications, and so the PyXLL add-in cannot use the Python DLLs and packages that are installed as part of the Windows Store Python app." From the PyXLL support site - https://support.pyxll.com/hc/en-gb/articles/4417634326675-Python-installed-via-the-Windows-Store-cannot-be-used-with-PyXLL The "py" launcher is installed by the installer from python.org. -- https://mail.python.org/mailman/listinfo/python-list
Re: Question about garbage collection
Frank Millman wrote at 2024-1-15 15:51 +0200: >I have read that one should not have to worry about garbage collection >in modern versions of Python - it 'just works'. There are still some isolated cases when not all objects in an unreachable cycle are destroyed (see e.g. step 2 of "https://devguide.python.org/internals/garbage-collector/index.html#destroying-unreachable-objects;). But Python's own objects (e.g. traceback cycles) or instances of classes implemented in Python should no longer be affected. Thus, unless you use extensions implemented in C (with "legacy finalizer"s), garbage collection should not make problems. On the other hand, your application, too, must avoid memory leaks. Caches of various forms (with data for several sessions) might introduce them. -- https://mail.python.org/mailman/listinfo/python-list
Re: Python 3.12.1, Windows 11: shebang line #!/usr/bin/env python3 doesn't work any more
On 1/15/24 09:44, Sibylle Koczian via Python-list wrote: In the Python documentation for versions 3.11 and 3.12 I found no differences regarding py.exe and shebang lines. Then I removed the "/env" from the shebang lines and could start the scripts from the second computer. That certainly is a solution, but why??? Sibylle also, it looks like you can disable the PATH-searching behavior of the /usr/bin/env virtual path: > The environment variable PYLAUNCHER_NO_SEARCH_PATH may be set (to any value) to skip this search of PATH. -- https://mail.python.org/mailman/listinfo/python-list
Re: Python 3.12.1, Windows 11: shebang line #!/usr/bin/env python3 doesn't work any more
On 1/15/24 09:44, Sibylle Koczian via Python-list wrote: First and foremost I want to understand why I'm seeing this: - Python scripts with "/usr/bin/env python3" as shebang line work as expected on a computer with Windows 10 and Python 3.11.5. They have worked for years on this machine, using either the latest Python or one version before (depending on availability of some packages). There is a virtual machine with ArchLinux on the same machine and some of the scripts are copies from that. - I've got a second computer with Windows 11 and I installed Python 3.12.1 on it. After copying some scripts from my first computer I found that I couldn't start them: not by entering the script name in a console, not using py.exe, not double clicking in the explorer. Entering \python probably worked - I think I tried that too, but I'm not really sure, because that's really not practical. In the Python documentation for versions 3.11 and 3.12 I found no differences regarding py.exe and shebang lines. Then I removed the "/env" from the shebang lines and could start the scripts from the second computer. That certainly is a solution, but why??? It's because of Windows itself. The default nowadays is that irritating little stub that prompts you to go install Python from the WIndows store. When you use the "env" form, it looks for python (or python3 in your case) in the PATH *first* and you'll get a hit. Mine looks like: C:\Users\mats\AppData\Local\Microsoft\WindwsApps\python.exe and python3.exe you can check what it's doing for you by using the "where" command in a windows shell. On your older Windows 10 machine you either never had that stub - I don't know when it was added, maybe someone from Microsoft listening here knows - or it's been superseded by changes to the PATH, or something. On my fairly new Win 11 box the base of that path is early in the user portion of PATH, so that must be a default. py.exe without the "/usr/bin/env" magic doesn't put PATH searching first, according to that snip from the docs that's been posted here several times., so you shouldn't fall down that particular rathole. -- https://mail.python.org/mailman/listinfo/python-list
Re: Python 3.12.1, Windows 11: shebang line #!/usr/bin/env python3 doesn't work any more
Am 15.01.2024 um 00:46 schrieb Mike Dewhirst via Python-list: In Windows the provided methods for running complex command lines are either a batch file or a shortcut.Someone very kindly pointed out to me in this thread that there is a PEP for py.exe. I don't use py.exe originally because I didn't trust it believing it was a new-fangled Microsoft trick. I did read that PEP but it has no relevance for my mixed Windows/Linux environments. On reflection I now believe I won't use py.exe because it introduces an unnecessary layer of indirection.The bottom line is that you still need to know which Python a particular set of circumstances demands and if you use py.exe you then need to also understand how it chooses and how it interprets shebang lines written for your Linux environment. And if that isn't your situation I have jumped to the wrong conclusion.I have found no problem in Windows when I use shebang lines in scripts intended for execution in both Linux and Windows. They are ignored unless you use py.exe.My advice is to give up py.exe unless your use case mandates shebang lines in Windows.M--(Unsigned mail from my phone) First and foremost I want to understand why I'm seeing this: - Python scripts with "/usr/bin/env python3" as shebang line work as expected on a computer with Windows 10 and Python 3.11.5. They have worked for years on this machine, using either the latest Python or one version before (depending on availability of some packages). There is a virtual machine with ArchLinux on the same machine and some of the scripts are copies from that. - I've got a second computer with Windows 11 and I installed Python 3.12.1 on it. After copying some scripts from my first computer I found that I couldn't start them: not by entering the script name in a console, not using py.exe, not double clicking in the explorer. Entering \python probably worked - I think I tried that too, but I'm not really sure, because that's really not practical. In the Python documentation for versions 3.11 and 3.12 I found no differences regarding py.exe and shebang lines. Then I removed the "/env" from the shebang lines and could start the scripts from the second computer. That certainly is a solution, but why??? Sibylle -- https://mail.python.org/mailman/listinfo/python-list
Re: Question about garbage collection
> I do have several circular references. My experience is that if I do not > take some action to break the references when closing the session, the > objects remain alive. Below is a very simple program to illustrate this. > > Am I missing something? All comments appreciated. Python has normal reference counting, but also has a cyclic garbage collector. Here's plenty of detail about how it works: https://devguide.python.org/internals/garbage-collector/index.html Skip -- https://mail.python.org/mailman/listinfo/python-list
Question about garbage collection
Hi all I have read that one should not have to worry about garbage collection in modern versions of Python - it 'just works'. I don't want to rely on that. My app is a long-running server, with multiple clients logging on, doing stuff, and logging off. They can create many objects, some of them long-lasting. I want to be sure that all objects created are gc'd when the session ends. I do have several circular references. My experience is that if I do not take some action to break the references when closing the session, the objects remain alive. Below is a very simple program to illustrate this. Am I missing something? All comments appreciated. Frank Millman == import gc class delwatcher: # This stores enough information to identify the object being watched. # It does not store a reference to the object itself. def __init__(self, obj): self.id = (obj.type, obj.name, id(obj)) print('***', *self.id, 'created ***') def __del__(self): print('***', *self.id, 'deleted ***') class Parent: def __init__(self, name): self.type = 'parent' self.name = name self.children = [] self._del = delwatcher(self) class Child: def __init__(self, parent, name): self.type = 'child' self.parent = parent self.name = name parent.children.append(self) self._del = delwatcher(self) p1 = Parent('P1') p2 = Parent('P2') c1_1 = Child(p1, 'C1_1') c1_2 = Child(p1, 'C1_2') c2_1 = Child(p2, 'C2_1') c2_2 = Child(p2, 'C2_2') input('waiting ...') # if next 2 lines are included, parent and child can be gc'd # for ch in p1.children: # ch.parent = None # if next line is included, child can be gc'd, but not parent # p1.children = None del c1_1 del p1 gc.collect() input('wait some more ...') -- https://mail.python.org/mailman/listinfo/python-list
Re: Extract lines from file, add to new files
Op 14/01/2024 om 13:28 schreef Left Right via Python-list: Python isn't a context-free language, so the grammar that is used to describe it doesn't actually describe the language... so, it's a "pretend grammar" that ignores indentation. No it doesn't. Here is the definition of a block, it clearly mentions indentation: block: | NEWLINE INDENTstatements DEDENT | simple_stmts But you are correct that python in not a context-free language. But so is any programming language. Yet a lot of those non context-free language designers thought it helpful to have a modified/extended BNF description of a superset of the intended language and used other means to further check whether the code in question was valid or not. -- Antoon Pardon -- https://mail.python.org/mailman/listinfo/python-list
Re: Extract lines from file, add to new files
On 15/01/24 21:13, Greg Ewing via Python-list wrote: On 15/01/24 1:54 pm, dn wrote: Soon after, Wirth simplified rather than expanded, and developed Pascal. Before Pascal there was Algol-W, which Wirth invented as a rebellion against how complicated Algol 68 was becoming. When I first saw this I was stunned, then attracted to its simplicity, but then steered-away once realised that it needed 'more' to cope with 'the outside world'. Pascal was intended as a teaching language, and as such it was lacking in practicality in a few spots. But it didn't need much tweaking to make it a very useful language. UCSD Pascal, Turbo Pascal, Delphi, etc. enjoyed a lot of popularity. A variant of UCSD was the main language for Macintosh application development for a number of years. Ironically, I didn't come across Pascal as a teaching-language. Borland were trying to turn Turbo Pascal into a practical development environment, beyond teaching (as with others of their 'Turbo' series). As I say, it didn't float my business-world boat. - not before I won a case a wine from Philippe Kahn's own vineyard for solving some 'interview question' and exchanging jokes with him, in French, at some Trade Show in London. Two surprises: one, that it actually turned-up a few weeks later, and two, that I (suddenly) had so many friends! Ah, the good, old days... -- Regards, =dn -- https://mail.python.org/mailman/listinfo/python-list
Re: Extract lines from file, add to new files
On Mon, 15 Jan 2024 at 19:26, Greg Ewing via Python-list wrote: > > On 15/01/24 9:07 pm, Chris Angelico wrote: > > The grammar *can't* specify everything. If it did, it would have to > > have rules for combining individual letters into a NAME and individual > > characters into a string literal. > > The lexical level of a grammar can be, and often is, described > formally using regular expressions. > > Although some might consider that this doesn't contradict > your statement about readability. :-) > Not even slightly :) ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Extract lines from file, add to new files
On 15/01/24 9:07 pm, Chris Angelico wrote: The grammar *can't* specify everything. If it did, it would have to have rules for combining individual letters into a NAME and individual characters into a string literal. The lexical level of a grammar can be, and often is, described formally using regular expressions. Although some might consider that this doesn't contradict your statement about readability. :-) -- Greg -- https://mail.python.org/mailman/listinfo/python-list
Re: Extract lines from file, add to new files
On 15/01/24 1:54 pm, dn wrote: Soon after, Wirth simplified rather than expanded, and developed Pascal. Before Pascal there was Algol-W, which Wirth invented as a rebellion against how complicated Algol 68 was becoming. When I first saw this I was stunned, then attracted to its simplicity, but then steered-away once realised that it needed 'more' to cope with 'the outside world'. Pascal was intended as a teaching language, and as such it was lacking in practicality in a few spots. But it didn't need much tweaking to make it a very useful language. UCSD Pascal, Turbo Pascal, Delphi, etc. enjoyed a lot of popularity. A variant of UCSD was the main language for Macintosh application development for a number of years. -- Greg -- https://mail.python.org/mailman/listinfo/python-list
Re: Extract lines from file, add to new files
On Mon, 15 Jan 2024 at 18:56, Greg Ewing via Python-list wrote: > > On 15/01/24 1:28 am, Left Right wrote: > > Python isn't a context-free language, so the grammar that is used to > > describe it doesn't actually describe the language > > Very few languages have a formal grammar that *fully* describes > the set of strings that constitute valid programs, including all > the rules about things having to be declared, types matching up, > etc. The only one I know of which attempted that is Algol 68, > and it seems to be regarded as a technical success but a practical > failure. > > > ... so, it's a "pretend grammar" that ignores indentation. > > Indentation isn't ignored, it appears in the grammar by means of > INDENT and DEDENT lexical tokens. > > It's true that the meaning of these tokens is described informally > elsewhere, but that's true of all the lexical features. > I've recently been doing a bit of work with grammar parsers, and to be quite honest, the grammar is only about one third of the overall parser. There are three sections with roughly equal importance: 1. Tokenizer 2. Grammar 3. What to DO with that grammar (actions) INDENT and DEDENT are being handled at the tokenizer stage, and so are a lot of other rules like backslashes in quoted strings. On the flip side, string prefixes (like b"...") seem to be handled in the third phase, and the grammar actually doesn't concern itself with those either. The grammar *can't* specify everything. If it did, it would have to have rules for combining individual letters into a NAME and individual characters into a string literal. The grammar would be completely unreadable. (I tried, and even just building up a decimal literal in that style was quite a pain.) ChrisA -- https://mail.python.org/mailman/listinfo/python-list