[issue43799] OpenSSL 3.0.0: define OPENSSL_API_COMPAT 1.1.1
Ivan Pozdeev added the comment: @christian.heimes https://github.com/python/cpython/pull/25481 also needs backporting to 3.9 and 3.8. A Pyenv user has been affected by the "implicit declaration of function ‘SSLv3_method’" compliation error that it fixes in 3.9.9: https://github.com/pyenv/pyenv/issues/2181 -- nosy: +ivan.pozdeev.gm ___ Python tracker <https://bugs.python.org/issue43799> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43878] ./configure fails on Apple Silicon with coreutils uname
Ivan Pozdeev added the comment: > Someone nonchalantly updated these in > https://github.com/python/cpython/commit/2fc857a5721a5b42bcb696c9cae1bbcc82a91b17 > so this bug is now fixed That PR only goes into 3.11. While this ticket claims to have fixed the problem for 3.10 as well. Should that PR be backported? Alternatively, https://github.com/python/cpython/pull/25450 should be merged, but into `3.10` instead of `master`. -- nosy: +ivan.pozdeev.gm ___ Python tracker <https://bugs.python.org/issue43878> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39035] Travis CI fail on backports: pyvenv not installed
Ivan Pozdeev added the comment: While we are at it, shall we enable build config validation (https://docs.travis-ci.com/user/build-config-validation)? It would produce warnings for outdated keys like this. -- ___ Python tracker <https://bugs.python.org/issue39035> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39035] Travis CI fail on backports: pyvenv not installed
Ivan Pozdeev added the comment: > New changeset 94d2c8df1a7657015a2fcdb4c4d43392f91f8348 by Inada Naoki in > branch 'master': bpo-39035: travis: Don't use beta group (GH-17602) > INADA-san: do you want to close this issue since you pushed changes? This may actually fix the problem. `group: beta` is undocumented. It was required long ago once to try out a new feature but I've absolutely no idea what it does (and if it does anything) now. It's quite possible that it can give us some beta unsupported VM image. -- ___ Python tracker <https://bugs.python.org/issue39035> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39035] Travis CI fail on backports: pyvenv not installed
Ivan Pozdeev added the comment: I already diagnosed this in https://mail.python.org/archives/list/python-...@python.org/message/3Z4FNPEFTFTYDX6RYOQ54UKOVLQAWUEU/: * https://travis-ci.org/python/cpython/jobs/616384134 looks for `pythonX.Y` but doesn't account for the fact that it could be Pyenv's stub * https://travis-ci.org/python/cpython/builds/616384157 is trying to use Python version(s) that are not guaranteed to be preinstalled in the VM for all `language:`s. I could fix this but I need to know what the hard requirements are to pick an optimal configuration. * Is the (preinstalled only for `language: c`) `clang 7.0.0` required? Would 3.4-4.0 (available from xenial distro) do? * Is `xenial` required? Would `bionic` do? In bionic, clang 7 is available from distro. * Which Python version needs to be preinstalled? Tools/ssl/multissltests.py says it's supposed to be compatible with 2.7 and 3.4-3.7. Options are, by decreasing "sum of flexibility+reliability": * use 3.5.1 or 2.7.12 from distro * hard require `language:python; python: 3.7` (then we cannot use custom `clang`) * use another pyenv-preinstalled version (there are also 3.6 and 2.7 that are supposed to be preinstalled but there might be none) -- nosy: +__Vano ___ Python tracker <https://bugs.python.org/issue39035> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11871] test_default_timeout() of test_threading.BarrierTests failure: BrokenBarrierError
Ivan Pozdeev added the comment: Got this issue today in AppVeyor's PR check: https://ci.appveyor.com/project/python/cpython/builds/24945165, so it's not local to David's worker. (At rerun, the test succeeeded, so the check status was not affected.) -- nosy: +Ivan.Pozdeev versions: +Python 3.7, Python 3.8, Python 3.9 -Python 3.2, Python 3.3 ___ Python tracker <https://bugs.python.org/issue11871> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35723] Add "time zone index" cache to datetime objects
Ivan Pozdeev added the comment: > which unfortunately use `is` to determine whether two datetimes are in the > same zone or not This sounds like a bug. Whether a tzinfo is a constant from a predefined set or something with a smart comparison semantic is none of datetime's business. -- nosy: +Ivan.Pozdeev ___ Python tracker <https://bugs.python.org/issue35723> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36501] Remove POSIX.1e ACLs in tests that rely on default permissions behavior
Ivan Pozdeev added the comment: Given the downsides, I think the proposed solution as it's now is too hacky to be a net improvement. "Skip mode_t checks" looks like the only way to go (since I've no idea how to "create temporary dirs" transparently for arbitrary test logic). But skipping tests defeats the purpose of a buildbot, so it's not a solution for the problem at hand. -- resolution: -> rejected stage: patch review -> resolved status: open -> closed ___ Python tracker <https://bugs.python.org/issue36501> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36501] Remove POSIX.1e ACLs in tests that rely on default permissions behavior
Ivan Pozdeev added the comment: Seeing during PR composition how basically every mode check and every `test.support.temp_umask` use is broken by ACLs, I'm starting to doubt that fixing individual test cases is the way to go. Though we can simply not worry about supporting everything imaginable and solve problems as they come and declare highly unusual cases unsupported. For the record, other potential problems: * All open(O_CREAT) and umask uses anywhere in the tests are broken by default. * Though only 4 test files were affected for now -- test_pathlib, test_tarfile, test_os, test_import . * There are other overlay security frameworks that override POSIX permissions like NFSv4 and SELinux. And possible solutions: * Skip mode_t checks outright if overlay security is detected like it's already done in Windows * Embed cleanup code into test.support's temp_* and such. In the POSIX.1e case, will have to create temporary dirs or change permissions for the current dir. * I don't think regrtest's temporary dir is a good place for this 'cuz tests are supposed to be runnable directly with `unittest`, too. -- ___ Python tracker <https://bugs.python.org/issue36501> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36501] Remove POSIX.1e ACLs in tests that rely on default permissions behavior
Change by Ivan Pozdeev : -- keywords: +patch pull_requests: +12584 stage: -> patch review ___ Python tracker <https://bugs.python.org/issue36501> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36501] Remove POSIX.1e ACLs in tests that rely on default permissions behavior
New submission from Ivan Pozdeev : In Linuxes with ACLs enabled, the following tests fail, as Steve Dower discovered in https://mail.python.org/pipermail/python-dev/2019-March/156929.html: == FAIL: test_mode (test.test_os.MakedirTests) -- Traceback (most recent call last): File "/home/osboxes/Documents/cpython/Lib/test/test_os.py", line 1157, in test_mode self.assertEqual(os.stat(parent).st_mode & 0o777, 0o775) AssertionError: 493 != 509 == FAIL: test_open_mode (test.test_pathlib.PosixPathTest) -- Traceback (most recent call last): File "/home/osboxes/Documents/cpython/Lib/test/test_pathlib.py", line 2104, in test_open_mode self.assertEqual(stat.S_IMODE(st.st_mode), 0o666) AssertionError: 420 != 438 == FAIL: test_touch_mode (test.test_pathlib.PosixPathTest) -- Traceback (most recent call last): File "/home/osboxes/Documents/cpython/Lib/test/test_pathlib.py", line 2117, in test_touch_mode self.assertEqual(stat.S_IMODE(st.st_mode), 0o666) AssertionError: 420 != 438 POSIX.1e is supported by major distros even though it's officially withdrawn; see https://en.wikipedia.org/wiki/Access_control_list#Filesystem_ACLs . -- components: Tests messages: 339313 nosy: Ivan.Pozdeev priority: normal severity: normal status: open title: Remove POSIX.1e ACLs in tests that rely on default permissions behavior type: behavior versions: Python 2.7, Python 3.7, Python 3.8, Python 3.9 ___ Python tracker <https://bugs.python.org/issue36501> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33944] Deprecate and remove pth files
Ivan Pozdeev added the comment: On 02.03.2019 9:01, Barry A. Warsaw In all the cases you've described, Python is no different from any other Linux software. E.g. I can install something into /etc/profile.d that would break the shell or set an envvar that would change the behavior of standard utilities. This is by design: Linux is designed for maximum interoperability, so there's only one of each component in the system and everything uses it whenever it needs that kind of functionality. It does support multiple versions of the same software, but it's a compromise that significantly complicates maintenance (primarily how to disambiguate them when something requests just "component X"), so they strive to avoid it whenever possible. Likewise, complete freedom for root to wreak havoc in the system is also by design: distro maintainers only test and support official packages; anything else you use is either your responsibility or an app supplier's if they provide official support (and are within their right to deny support if you tweak the environment beyond their support promise) -- same as for any other software as well. This is not even specific to .pth files, either, so you won't really eliminate the problem by removing them. You can break any other part of Python in subtle ways just as well -- e.g. overwrite or override binary files with incompatible ones, causing segfaults in random places (https://stackoverflow.com/q/51816639/648265 ). Now, Linux does have "lower tier environments" that don't automatically affect "higher tiers". 1) Software installed into /usr/local doesn't hijack system scripts thanks to absolute paths in their shebangs; software in /opt is not on PATH at all; 2) /etc/profile* and bashrc are only executed by login shells and interactive shells, not by scripts, limiting their effect to processes created within a user session; 3) anything within a user's profile or run as a regular user (including ~/.bash*) doesn't affect system-wide settings and processes run as root. Blindly replicating 2) won't do for Python, however. Unlike Bash which has all the functionality compiled in, Python has an external standard library and arbitrary additional packages. They both are essential for its operation as a system component that other software can use without additional manipulations, AND Python gives the user freedom on how to arrange them in the system. So there must be a way to provide any "additional manipulations" that may be needed that the built-in startup logic doesn't have. From administration POV, any such startup logic is a part of the core offer to the system: core files+libraries+connecting logic = Python system component, so it must be invoked whenever Python is invoked. And we do already have ways to apply startup code only to a "lower-tier environment" if such a need arises: user-specific -- user site; interactive-specific -- PYTHONSTARTUP. There's no such thing as a "login shell" for Python but there's Python run in a user session; /etc/profile* can set envvars that would apply only there. So it seems to me that what you are asking for is "/etc/profile.d for Python". When designing such a feature, note, however, that the concept of login sessions is completely alien to Python. I believe a way to provide an additional site-packages directory will do (I can't readily see an already available way to do so in https://docs.python.org/3/using/cmdline.html ). -- ___ Python tracker <https://bugs.python.org/issue33944> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33944] Deprecate and remove pth files
Ivan Pozdeev added the comment:
On 02.03.2019 2:25, Barry A. Warsaw wrote:
> The fact that .pth files are global and affect the entire Python
> installation. <...> Right now, there’s no control over the scope of such
> environmental customizations; it’s all or nothing.
That's the entire purpose of "customizing the environment in which the
program specified by the user would run". A customization can very well
be implemented to be application-specific but it doesn't have to. Python
was never designed to isolate modules from each other (an "application"
as you say it is just another module) -- on the contrary, the amount of
power it gives the user over the code that they don't control is one of
its key appeals. A Python installation acts as a unit where anything can
affect anything else, and the order is maintained with
https://en.wikipedia.org/wiki/Soft_security .
So, if you need a compartmentalized application, a regular Python
installation is a wrong tool for the job.
Compartmentalization comes at the price of:
* rampant code duplication ('cuz if you actively distrust external
code, you have to bring all the code you need with you) and all its
corollaries (no automatic security fixes and modernized semantics;
no memory and disk space economy from shared library reuse)
o so compartmentalization is absolutely impossible within a shared
environment: anything that you use can be changed by the user
(e.g. to satisfy the requirements of something else, too)
* lack of interoperability (how many Android apps do you know that can
use each other's functionality?).
Venv does a pretty good job of providing you with a private copy of any
3rd-party modules you require but not the envvars, the interpreter and
the standard library (and any OS facilities they depend on). If you
require a harder barrier between your app and the rest of the system
and/or wish to actively prevent users from altering your application,
you'll have to use a private Python installation (e.g. in /opt), or hide
it from everyone with the likes of Pyinstaller, or an OS-level
container, or a VM... or just drop the pretense and go SaaS(S) (that'll
teach those sneaky bastards to mess with my code!).
> Applications should be able to opt in or out of them, just like they can with
> individual packages (which must be imported in order to affect the
> interpreter state).
Right on the contrary. To decide what environment an application shall
be run in is the user's prerogative. The application itself has
absolutely no business meddling in this. All it can do is declare some
requirements for the environment (either explicitly or implicitly by
making assumptions) and refuse to work or malfunction if they are not
met (and the user is still fully within their right to say: "screw you,
I know what I am doing" -- and fool the app into thinking they are met
and assume responsibility for any breakages).
With "individual packages", it's actually completely the same: the app
can decide which ones it wants to use, but it's the user who decides
which ones are available for use!
--
___
Python tracker
<https://bugs.python.org/issue33944>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36160] Multiple errors in test_site.py on sysconfig._CONFIG_VARS.clear() if run on its own
Change by Ivan Pozdeev : -- keywords: +patch pull_requests: +12134 stage: -> patch review ___ Python tracker <https://bugs.python.org/issue36160> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36160] Multiple errors in test_site.py on sysconfig._CONFIG_VARS.clear() if run on its own
New submission from Ivan Pozdeev : Sample failure: > cpython\branches\3.7>python.bat -m test.test_site Running Debug|x64 interpreter... E.s. == ERROR: test_addpackage (__main__.HelperFunctionsTests) -- Traceback (most recent call last): File "C:\Users\Sasha\Documents\cpython\branches\3.7\lib\test\test_site.py", li ne 77, in tearDown sysconfig._CONFIG_VARS.clear() AttributeError: 'NoneType' object has no attribute 'clear' == ERROR: test_addpackage_import_bad_exec (__main__.HelperFunctionsTests) -- Traceback (most recent call last): File "C:\Users\Sasha\Documents\cpython\branches\3.7\lib\test\test_site.py", li ne 77, in tearDown sysconfig._CONFIG_VARS.clear() AttributeError: 'NoneType' object has no attribute 'clear' The reason is that `sysconfig._CONFIG_VARS' is None until the first call to `sysconfig.get_config_vars()'. When the suite is used in conjunction with the others, other tests have already called it by the time test_site.py gets control. -- components: Tests messages: 336947 nosy: Ivan.Pozdeev priority: normal severity: normal status: open title: Multiple errors in test_site.py on sysconfig._CONFIG_VARS.clear() if run on its own type: behavior versions: Python 3.7, Python 3.8 ___ Python tracker <https://bugs.python.org/issue36160> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33944] Deprecate and remove pth files
Ivan Pozdeev added the comment: On 01.03.2019 20:27, Ivan Pozdeev wrote: > The fact that this feature is mixed up with and often supplements > "real packages" that a program would explicitly use is actually > incidental: a package with a .pth does not need to have any > functionality intended for explicit use. > Eureka! So, there are actually two kinds of packages: "functional packages" to be used explicitly and "environment packages" to customize the execution environment. The infrastructure just doesn't distinguish between them and allows a package to combine both types of functionality for convenience. By this logic, pywin32's .pth is effectively a private import hook to allow for its nonstandard structure. It could be in a separate "environment package" that would be a dependency but that would complicate things for no real gain. The caveat with "environment packages" is that there are no predefined dependencies between them and between them and "functional packages". Their required execution order rather depends on user's needs. E.g. the order of import hooks' registration would matter if more than one can serve a specific name, and the user may prefer any of the options; whether some import hook is required to import some installed packages depends on the way they are installed. This is the same with any other plugin functionality, too. And I'm not aware of any general solution because a solution is very situational. The best we can do here that I see is to allow the user (or, you guessed it, yet another "environment package" for manageability) to specify load order dependencies between .pth's. -- ___ Python tracker <https://bugs.python.org/issue33944> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33944] Deprecate and remove pth files
Ivan Pozdeev added the comment: On 01.03.2019 3:58, Steve Dower wrote > Import hooks can always be injected by a package __init__.py before the > importer will try and resolve the module, so nothing is needed there. I thought the flaw in this reasoning in https://bugs.python.org/issue33944#msg320277 was obvious and didn't want to bother people refuting it. Apparently not. To do anything in __init__.py, that __init__.py itself needs to be already importable. This very well may not be the case -- in fact, import hooks were designed specifically for the scenarios where this is not the case. Imagine e.g. loading modules from a cloud storage (why not?) -- so nothing on the system at all except the hook. Or, suggested earlier in this ticket, a union namespace where the code to import needs to be constructed on the fly. > .pth files really only satisfy the "run at startup because I'm a dependency > of something that my user wants and don't make them opt-in to my changed > behaviour" Startup code (custom or not) is not a dependency of anything. It rather customizes the environment in which the program specified by the user would run, _before_ any user code could be allowed to get control. It is not a part of the program to be run but rather of the environment that the user wants, and it needs to be implicit so the user can use the same commands and code (compare venv). This is a required feature because the stock Python startup logic cannot possibly provide all the customizations that a user may need (compare initrd). .pth's are equivalent to sitecustomize but allow the user to manage the set of code chunks automatically using the packaging infrastructure (compare .d directories in UNIX). The fact that this feature is mixed up with and often supplements "real packages" that a program would explicitly use is actually incidental: a package with a .pth does not need to have any functionality intended for explicit use. > which I don't like If you don't like something, there's always a specific reason -- though you may not understand it consciously. So the way to go is dig into it, find out what specific speck is putting you off -- only then can you be sure that you are concentrating on the right thing and won't throw the baby out with the bathwater. Try to change one trait in your mind's eye leaving all else intact -- will the feeling go away? If it will, you are on the right track; can the trait you chose be split further? You know you found it when you can't change any further part and change the feeling and you can say with confidence how exactly what it's doing misaligns with your moral compass. We already identified a few real reasons: hard to see, hard to debug, encourages unreadable code, run in arbitrary order when the order matters (and IIRC I provided fixes for all). What else? -- ___ Python tracker <https://bugs.python.org/issue33944> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33944] Deprecate and remove pth files
Change by Ivan Pozdeev : -- pull_requests: +12117 ___ Python tracker <https://bugs.python.org/issue33944> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33944] Deprecate and remove pth files
Ivan Pozdeev added the comment: On second thought, the inability to debug code that runs at startup, before user code ever gets control, is a fundamental issue (this problem arises for any software that has startup code), so such a facility in stock codebase has a merit. -- ___ Python tracker <https://bugs.python.org/issue33944> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33944] Deprecate and remove pth files
Change by Ivan Pozdeev : -- pull_requests: +12114 ___ Python tracker <https://bugs.python.org/issue33944> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33944] Deprecate and remove pth files
Ivan Pozdeev added the comment: On 26.02.2019 23:37, Barry A. Warsaw wrote: > My opposition would lessen if there were clear ways to debug, and preferably > also prevent, pth interpretation. Easy. Insert a chunk into site.py that would call pdb.set_trace() if an envvar (e.g. `PYSITEDEBUG') or a command line switch is set. Actually, why can't whoever has this problem add such a chunk themselves? Is this really such a frequent and ubiquitous problem that this needs to be in the stock codebase? I suspect we're dealing with a vocal minority here. -- ___ Python tracker <https://bugs.python.org/issue33944> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33944] Deprecate and remove pth files
Ivan Pozdeev added the comment: > This brings to mind the transition of many programs from using a single > config file or startup script to using a directory of config/startup files > parsed/executed in alphabetical order. Would a sitecustomize.d/ directory > (with files within it executed in alphabetical order) as a replacement for > executable code in .pth files be an improvement on the status quo? No, because the required execution order is governed by package interdependencies rather than names. SysVInit went around this by hand-picking number prefixes to files in rcN.d/ but this proved unmaintainable in the long run. -- ___ Python tracker <https://bugs.python.org/issue33944> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33944] Deprecate and remove pth files
Ivan Pozdeev added the comment: > I’m a little concerned about this approach because it means random third > party modules can affect the global environment for your application, without > knowing it. Since the hook installation happens at import time, and just > depending on a library that has such a .pth file will install it, the end > application will not have control over its global state. But "affecting the global environment for your application" is exactly what is intended here. You want multiple packages to all load their code into the same namespaces (aka module objects), thus of course potentially affecting/overriding each other's functionality. That's what you get when you have plugins -- a badly-written/incompatible plugin can and will break your app. It doesn't have to "just depend on a library that has such a .pth file", it's up to the import hook's implementation. I just gave as example the simplest solution that requires zero effort on the main package maintainer's part. E.g. you can only allow adding a new submodule by default, or require the "parent" package to "allow" insertions into itself, or move registration into the parent's configuration file (so the user needs to enable the plugin manually), or provide some more granular code injection techniques like e.g. event handler lists that certain plugins' functions will be added into. All that matters here is that the hook is going to automagically assemble the resulting namespaces from parts upon import. Finally, Python applications don't have full control over their global state anyway. Any module can monkey-patch or override any other module via a variety of means. So, this risk is not something new or unexpected. -- nosy: +__Vano ___ Python tracker <https://bugs.python.org/issue33944> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33944] Deprecate and remove pth files
Ivan Pozdeev added the comment: @barry > Interdependent namespace packages. If they get loaded in the wrong order, > they can mess up __path__ settings Actually, when writing the PR, I had a revelation how this could be implemented. Via an import hook that would work like a union FS! In its .pth file, each such package will import the hook's module (which will cause the hook to be installed on the first import) and "register" its namespaces and/or dependencies with it. The hook will then calculate the required load order and enforce it upon import of any of the registered namespaces. -- ___ Python tracker <https://bugs.python.org/issue33944> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12920] inspect.getsource only works for objects loaded from files, not interactive session
Ivan Pozdeev added the comment: See https://bugs.python.org/issue33826?@ok_message=msg%20328824%20created%0Aissue%2033826%20message_count%2C%20messages%20edited%20ok&@template=item#msg319692 how IPython stores source from interactive input and why it's not appropriate for vanilla REPL IMO. -- nosy: +Ivan.Pozdeev ___ Python tracker <https://bugs.python.org/issue12920> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33826] enable discovery of class source code in IPython interactively defined classes
Ivan Pozdeev added the comment: > which, as I explained, is already possible without the patch Sorry, I myself explained in https://bugs.python.org/msg319692 that it's not possible. The rest still stands though. -- ___ Python tracker <https://bugs.python.org/issue33826> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33826] enable discovery of class source code in IPython interactively defined classes
Ivan Pozdeev added the comment: @Thomas Viehmann , as it's currently formulated, this is a duplicate because it strives to allow getting class source from past interactive input -- which, as I explained, is already possible without the patch and seems to be inappropriate for vanilla console anyway. Your patch is rather a new class feature that provides a more robust link to its source, wherever it's located. If you reformulate your ticket to propose that instead, it will no longer be a duplicate. I'm not sure if the UI allows that, it may be better to create a new ticket. -- ___ Python tracker <https://bugs.python.org/issue33826> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33944] Deprecate and remove pth files
Ivan Pozdeev added the comment: The primary motivation behind the suggestion seems to be the fact that the feature is abused. However, the documentation has no info whatsoever on what is the intended use -- thus what constitutes abuse. Without that, the accusations are kind of baseless -- how can we blame package authors for having to figure it out for themselves? I've made a PR with the corresponding note. Since the discussion has revealed a number of valid use cases for the feature for which there are no adequate alternatives currently, I hope it will diminish the discontent and be grounds to incite package authors to remove unnecessary logic from there. -- ___ Python tracker <https://bugs.python.org/issue33944> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33944] Deprecate and remove pth files
Change by Ivan Pozdeev : -- keywords: +patch pull_requests: +9463 stage: -> patch review ___ Python tracker <https://bugs.python.org/issue33944> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue34360] urllib.parse doesn't fully comply to RFC 3986
Ivan Pozdeev added the comment: I confirm violation of https://tools.ietf.org/html/rfc3986#section-3.2.2 . URLs are now covered by RFC 3986 which obsoletes RFC 1808 that `urllib's documentation refers to. This new URL RFC adds [] to 'reserved' characters, so them being present unquoted anywhere where reserved characters are not allowed shall be a parsing error. -- nosy: +Ivan.Pozdeev title: urllib.parse doesn't fail with multiple unmatching square brackets -> urllib.parse doesn't fully comply to RFC 3986 versions: +Python 3.6, Python 3.7, Python 3.8 ___ Python tracker <https://bugs.python.org/issue34360> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33944] Deprecate and remove pth files
Ivan Pozdeev added the comment: > They are very difficult to debug because they're processed too early. .pth's are processed by site.py, so no more difficult than site/sitecustomize. You can e.g. run `site.addpackage(,,None)' to debug the logic. > They usually contain globs of inscrutable code. An ability to contain code is there for a reason: to allow a module do something more intelligent than adding hardcoded paths if needed (e.g. pywin32 adds a subdir with .dll dependencies to PATH). A chunk of code is limited to a single line -- a conscious limitation to deter misuse 'cuz search path setup code is presumed to be small. If someone needs something other than path setup, they should do it upon the module's import instead. If they insist on misusing the feature, Python's design does what it's supposed to do in such cases: "make right things easy, make wrong things hard". If there's a valid reason to allow larger code chunks, we can introduce a different syntax -- e.g. something like bash here-documents. > Exceptions in pth files can get swallowed in some cases. If this happens, it's a bug. A line from .pth is executed with "exec line", any exceptions should propagate up the stack as usual. > They are loaded in indeterminate order. Present a use case justifying a specific order. I can see a probable use case: a package needs to do something using its dependencies, so any .pth for the dependencies should run before the one for the package. But I can't see why that package can't do this upon its import instead (saves unnecessary work if the user won't be using that package in that session, too). The only valid case I can see is if the package is using some 3rd-party import system (e.g. a .7z archive or some module repository) that needs to be loaded first for its search path to make sense. -- ___ Python tracker <https://bugs.python.org/issue33944> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33944] Deprecate and remove pth files
Ivan Pozdeev added the comment: I *think* we need to ask maintainers of packages who use .pth -- at least, Mark Hammond (pywin32) -- to find out the impact and if everything can be done with other means. AFAICS it at least allows pywin32 to have many top-level modules without cluttering `site-packages'. pywin32 e.g. also copies some files to %windir%\system32 for some reason. And last time I checked, distutils had no functionality that involved symlinks, regardless of the OS. -- nosy: +Ivan.Pozdeev, mhammond type: -> enhancement ___ Python tracker <https://bugs.python.org/issue33944> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33826] enable discovery of class source code in IPython interactively defined classes
Ivan Pozdeev added the comment: Oh, I see, you want to be able to get the source for code entered into the interactive session. IPython does this by creating a separate fake "file name" for every input and adding corresponding entries for them into `linecache.cache'. This doesn't work for classes 'cuz for them, inspect.getfile() rather looks at __module__.__file__ , and there's no single __file__ for __main__. Now, there are two concerns here: * This will lead to linecache being polluted with old input, effectively a memory leak. Unlike vanilla's readline cache which has a maximum length. Not much concern for IPython which only needs to be good enough for its target audience and can be dropped if it doesn't work for a specific scenario. But not for vanilla which needs to work reliably in all cases. * There indeed is a discrepancy between functions that link to source directly and classes that rely on heuristics for that (inspect.findsource() searches the file with a regex for the class' definition, how absurd is that?). Adding __file__ and __firstlineno__ _could_ help here, the `class' directive will need to add them. * In any case, the returned source is prone to "not necessarily what the interpreter uses", but that's the problem with how the returned source is generated, not where it's taken from. -- ___ Python tracker <https://bugs.python.org/issue33826> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33826] enable discovery of class source code in IPython interactively defined classes
Ivan Pozdeev added the comment: I got that part, thank you. I can't get 1)what you're expecting as an output and 2)why this should work the way you're suggesting 'cuz functions and classes are very different. In particular, classes can be modified dynamically, unlike functions. So if you just print out the lines from the file with the definition (which is what getsource() does), it may not be what the interpreter is actually using. -- ___ Python tracker <https://bugs.python.org/issue33826> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33826] enable discovery of class source code in IPython interactively defined classes
Ivan Pozdeev added the comment: What do you mean by "class source code" anyway? A class doesn't actually contain any executable code (and there's no code object associated with it). -- ___ Python tracker <https://bugs.python.org/issue33826> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33826] enable discovery of class source code in IPython interactively defined classes
Ivan Pozdeev added the comment: In [1]: import logging In [2]: logging.Logger.__module__ Out[2]: 'logging' In [4]: import sys In [6]: sys.modules[logging.Logger.__module__].__file__ Out[6]: 'C:\\Program Files\\Python36\\lib\\logging\\__init__.py' -- nosy: +Ivan.Pozdeev ___ Python tracker <https://bugs.python.org/issue33826> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33709] test.support.FS_NONASCII returns incorrect result in Windows with non-US locale
Ivan Pozdeev added the comment: In 3.x, it turns out, this doesn't result in test failures in stock configuration. It does though if PYTHONLEGACYWINDOWSFSENCODING is in system environment. I was diagnosing failures in 2.x and saw that 3.x has the same logic, so it was a no-brainer to replicate the change... -- ___ Python tracker <https://bugs.python.org/issue33709> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33661] urllib may leak sensitive HTTP headers to a third-party web site
Ivan Pozdeev added the comment: It's not about "convincing" me or anyone else. It's about showing how this will be a strict improvement. I showed that the HTTP RFC allows apps to rely on the fact that they are receiving all the headers. So filtering them arbitrarily violates the HTTP standard -- while the whole purpose of `urllib` is to conform to it (or it would not be able to reliably talk to HTTP servers). So, your suggestion is a disaster rather than improvement. -- ___ Python tracker <https://bugs.python.org/issue33661> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33711] Could not find externals/db-* in msi.py on license generation
Ivan Pozdeev added the comment: ... can be built with such a glaring mistake in the script. -- ___ Python tracker <https://bugs.python.org/issue33711> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33711] Could not find externals/db-* in msi.py on license generation
New submission from Ivan Pozdeev : c:\Users\Sasha\Documents\cpython\Tools\msi>python msi.py Traceback (most recent call last): File "msi.py", line 1372, in add_files(db) File "msi.py", line 956, in add_files generate_license() File "msi.py", line 928, in generate_license raise ValueError, "Could not find "+srcdir+"/externals/"+pat ValueError: Could not find c:\Users\Sasha\Documents\cpython/externals/db-* The dir in externals is actually called bsddb-. I've no idea how they can official releases -- components: Build messages: 318284 nosy: Ivan.Pozdeev priority: normal pull_requests: 6907 severity: normal status: open title: Could not find externals/db-* in msi.py on license generation type: behavior versions: Python 2.7 ___ Python tracker <https://bugs.python.org/issue33711> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33709] test.support.FS_NONASCII returns incorrect result in Windows with non-US locale
Change by Ivan Pozdeev : -- pull_requests: +6904 ___ Python tracker <https://bugs.python.org/issue33709> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33709] test.support.FS_NONASCII returns incorrect result in Windows with non-US locale
Change by Ivan Pozdeev : -- keywords: +patch pull_requests: +6903 stage: -> patch review ___ Python tracker <https://bugs.python.org/issue33709> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33709] test.support.FS_NONASCII returns incorrect result in Windows with non-US locale
New submission from Ivan Pozdeev : This causes test_ntpath and test_posixpath to fail in subj. Sample failure: == FAIL: test_expandvars_nonascii (__main__.NtCommonTest) -- Traceback (most recent call last): File "c:\Users\Sasha\Documents\cpython\lib\test\test_genericpath.py", line 230 , in test_expandvars_nonascii check(u'$spam bar', u'%s bar' % unonascii) File "c:\Users\Sasha\Documents\cpython\lib\test\test_genericpath.py", line 214 , in check self.assertEqual(expandvars(value), expected) AssertionError: u'? bar' != u'\xe6 bar' - ? bar ? ^ + \xe6 bar ? ^ Cause: if sys.getfilesystemencoding()=='mbcs', encoding Unicode characters that are missing in the current locale succeeds but produces '?'. So, test.support.FS_NONASCII's test fails to detect if a character is present in the current locale. -- components: Tests messages: 318271 nosy: Ivan.Pozdeev priority: normal severity: normal status: open title: test.support.FS_NONASCII returns incorrect result in Windows with non-US locale type: behavior versions: Python 2.7, Python 3.6, Python 3.7, Python 3.8 ___ Python tracker <https://bugs.python.org/issue33709> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33479] Document tkinter and threads
Ivan Pozdeev added the comment: @markroseman I'm about 50% okay with your changes. Could you create a PR against my branch so we can work out the specifics? -- ___ Python tracker <https://bugs.python.org/issue33479> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33479] Document tkinter and threads
Ivan Pozdeev added the comment:
On 29.05.2018 23:20, Mark Roseman wrote:
> Mark Roseman added the comment:
>
> Ivan, thanks for making a good first pass of this. The thread section still
> feels a lot like 'fighting' with the model. Do you mind if I take a crack at
> it? Won't get to it for a few days, but in case you have any stuff you're in
> the middle of.
>
> I should clarify too that Tk apps almost universally do use a blocking event
> loop (i.e. 'vwait forever' at the end of a script). Application-level event
> handlers are supposed to respond quickly so control goes back to the event
> loop.
>
> It's when control doesn't return there that things like the 'update
> idletasks' hacks are needed. In practice, I've noticed that's what seems to
> trip people up when they first start, as they try to emulate the flow of
> their non-GUI code, which frequently blocks. Far better that the program is
> restructured so that the event handler completes quickly. It's actually worse
> than it looks, because you can end up having nested event loops if you start
> randomly throwing this stuff in. That's conceptually hard for most people and
> a good source of bugs.
>
I'm pretty much done with the threading section but if you think it
could use more clarification, sure. You can make a PR against my branch
to integrate changes (or vice versa).
In line with the aforementioned plan, the "Threading model" section
needs to tell these things critical to interstanding the module's
behavior and usage patterns:
* There are two basic GUI execution models (stricty speaking, these are
event-driven models in general, but no need to go that deep): single
thread with pumping messages by hand (read: cooperative multitasking),
and UI thread + worker threads (read: preemptive mulitasking). The
latter is prevalent now so the reader is more likely to be familiar with it.
* Tcl/Tk implements the former model (which is unusual), Tkinter
emulates the latter with its own logic (so Tcl/Tk docs won't have info
on this) but supports the former one as well. (So update() is not a
"hack" at all, it's just intended for a different use case that doesn't
come up often.)
* Tkinter calls can and should be made from any threads (this is also
unusual), but there are user-visible limitations.
* Tcl event loop is shared (another unusual gimmick), which is also
user-visible.
This section is not the place to showcase concrete usage patterns,
that's what tutorials are for. But it can make references to relevant
Tkinter functions as long as this doesn't garble the narration.
I imagine tutorial as a separate page (see the plan how it should be
linked to), with the following sections, each illustrated with code.
It's not meant to be an essential part of this ticket because it's of
lower priority.
* Create initial UI, then run mainloop(). All the rest is done with ui
commands and events. (a hello world example)
* Start a worker thread for any action that may take more than a
fraction of a second. Make Tkinter calls from the worker thread to pass
back info on its progress.
* Collect worker threads and do other cleanup at exit via a cleanup
function. Call it from both a special exit command, if any, _and_ from
.protocol("WM_DELETE_WINDOW"). Lengthy/perpetual worker threads' logic
must be interruptable for this.
* For more complex logic, use the Model-View-Presenter pattern.
* ?Something about exception handling? (Propagating exceptions? Making
unhandled exceptions visible to the user? I dunno atm)
* An example of using Tcl's execution model, i.e. with
dooneevent()/update()/update_idletasks() instead of mainloop(), like a
Tcl program would do.
--
nosy: +__Vano
___
Python tracker
<https://bugs.python.org/issue33479>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33257] Race conditions in Tkinter with non-threaded Tcl
Ivan Pozdeev <ivan_pozd...@mail.ru> added the comment: Found yet another race condition -- when creating multiple Tk()s. Attached example (loops indefinitely inside Tcl: some internal list becomes circular for some reason). The obvious fix by wrapping Tcl calls with locks in Tkapp_New doesn't fix it -- must be something else. Since creatng multiple Tk's is an uncommon use case, this can wait. -- Added file: https://bugs.python.org/file47620/test_threads_tk.py ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue33257> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33661] urllib may leak sensitive HTTP headers to a third-party web site
Ivan Pozdeev <ivan_pozd...@mail.ru> added the comment: According to https://stackoverflow.com/questions/1969709/how-to-forward-headers-on-http-redirect , there's nothing in the specs that mention (even the possibility) of any special request header processing. According to https://tools.ietf.org/html/rfc7231#section-6.4 , redirection targets are to be treated as effectively equal to the original URL. So, there aren't any grounds for the proposed filtering from web standards' POV. Neither are there from security POV: once you have given your credentials to a server, it is free to do whatever it wants with them. So, by giving them, you have effectively put down your signature that you trust the server with your data -- which implies trusting its advice where to resend it. The server could as well do that resending itself and passed you the end result. So, your proposed filtering does not actually achieve anything meaningful.1 -- nosy: +Ivan.Pozdeev ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue33661> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33637] pip cannot build extensions for debug Python
Ivan Pozdeev <ivan_pozd...@mail.ru> added the comment: >one will see all the variables and call hierarchy and be able to step through the code. With optimizations, the current line jumps unpredictably, and some lines can't be visited because they're optimized away. -- ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue33637> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33637] pip cannot build extensions for debug Python
New submission from Ivan Pozdeev <ivan_pozd...@mail.ru>:
(the output is for 2.7 head; for master, it's the same save for paths, library
versions and DeprecationWarning's)
>python.bat -m ensurepip
Running Debug|x64 interpreter...
Looking in links: c:\users\sasha\appdata\local\temp\tmpiysoby
Collecting setuptools
Collecting pip
Installing collected packages: setuptools, pip
Successfully installed pip-10.0.1 setuptools-39.0.1
>python.bat -m pip install scandir
Running Debug|x64 interpreter...
Collecting scandir
Using cached
https://files.pythonhosted.org/packages/13/bb/e541b74230bbf7a20a3949a2ee6631be299378a784f5445aa5d0047c192b/scandir-1.7.tar.gz
Installing collected packages: scandir
Running setup.py install for scandir ... error
Complete output from command
C:\Users\Sasha\Documents\cpython\PCBuild\amd64\python_d.exe -u -c "import
setuptools,
tokenize;__file__='c:\\users\\sasha\\appdata\\local\\temp\\pip-install-xafnbn\\scandir\\setup.py';f=getattr(tokenize,
'open', open)(__file__);code=f.read().replace('\r\n',
'\n');f.close();exec(compile(code, __file__, 'exec'))" install --record
c:\users\sasha\appdata\local\temp\pip-record-gbe0kk\install-record.txt
--single-version-externally-managed --compile:
running install
running build
running build_py
creating build
creating build\lib.win-amd64-2.7-pydebug
copying scandir.py -> build\lib.win-amd64-2.7-pydebug
running build_ext
building '_scandir' extension
creating build\temp.win-amd64-2.7-pydebug
creating build\temp.win-amd64-2.7-pydebug\Release
c:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\BIN\amd64\cl.exe /c
/nologo /Ox /MD /W3 /GS- /DNDEBUG -IC:\Users\Sasha\Documents\cpython\include
-IC:\Users\Sasha\Documents\cpython\PC /Tc_scandir.c
/Fobuild\temp.win-amd64-2.7-pydebug\Release\_scandir.obj
_scandir.c
c:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\BIN\amd64\link.exe
/DLL /nologo /INCREMENTAL:NO /LIBPATH:C:\Users\Sasha\Documents\cpython\libs
/LIBPATH:C:\Users\Sasha\Documents\cpython\PCbuild\amd64
/LIBPATH:C:\Users\Sasha\Documents\cpython\PC\VS9.0\amd64 /EXPORT:init_scandir
build\temp.win-amd64-2.7-pydebug\Release\_scandir.obj
/OUT:build\lib.win-amd64-2.7-pydebug\_scandir.pyd
/IMPLIB:build\temp.win-amd64-2.7-pydebug\Release\_scandir.lib
/MANIFESTFILE:build\temp.win-amd64-2.7-pydebug\Release\_scandir.pyd.manifest
LINK : fatal error LNK1104: cannot open file 'python27.lib'
error: command 'c:\\Program Files (x86)\\Microsoft Visual Studio
9.0\\VC\\BIN\\amd64\\link.exe' failed with exit status 1104
Command "C:\Users\Sasha\Documents\cpython\PCBuild\amd64\python_d.exe -u -c
"import setuptools,
tokenize;__file__='c:\\users\\sasha\\appdata\\local\\temp\\pip-install-xafnbn\\scandir\\setup.py';f=getattr(tokenize,
'open', open)(__file__);code=f.read().replace('\r\n',
'\n');f.close();exec(compile(code, __file__, 'exec'))" install --record
c:\users\sasha\appdata\local\temp\pip-record-gbe0kk\install-record.txt
--single-version-externally-managed --compile" failed with error code
1 in c:\users\sasha\appdata\local\temp\pip-install-xafnbn\scandir\
The cause is that distutils' `build_ext` looks at the --debug switch instead of
the running Python to decide which Python runtime to link against.
A workaround is thus to run setup.py by hand with the --debug switch.
Suggested fix is to split compiler options into optimization options and
linking options and:
* for debug Python, build without optimizations and against debug C runtime +
Python runtime regardless of --debug
* for release Python,
* build without optimizations but against release C runtime + Python
runtime with --debug
Rationale:
* It doesn't make sense to try to build against a different runtime than the
running Python -- this will always fail because include/lib paths, preprocessor
directives and such that build_ext generates is for the running Python.
* Must build against the same C runtime that Python uses in any case since
using two different runtimes will break things, especially in 2.7 that relies
on stdio.
* Building an extension without optimizations will allow to diagnose problems
in it. Debug facilities of the runtime like malloc tracking and probably
tracing will not work, but at least one will see all the variables and call
hierarchy.
--
components: Distutils
messages: 317589
nosy: Ivan.Pozdeev, dstufft, eric.araujo
priority: normal
severity: normal
status: open
title: pip cannot build extensions for debug Python
type: behavior
versions: Python 2.7, Python 3.6, Python 3.7, Python 3.8
___
Python tracker <rep...@bugs.python.org>
<https://bugs.python.org/issue33637>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33316] Windows: PyThread_release_lock always fails
Ivan Pozdeev <ivan_pozd...@mail.ru> added the comment: > So, nothing more to do here. In case that was cryptic, it means: the PR can be acted upon, no other changes are needed. -- ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue33316> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33479] Document tkinter and threads
Ivan Pozdeev <ivan_pozd...@mail.ru> added the comment: I'm currently rewriting the docs, too, according to the plan @ #msg316492. WIP @ https://github.com/native-api/cpython/tree/tkinter_docs . You PR lines up fine though is made redundant by https://github.com/native-api/cpython/commit/79b195a9028fd7bf6e8186dfced0fad6a41e87fa -- instead of removing Doc\library\tk.rst, I reduced it to a summary of chapter content without any claims about it whatsoever, like other chapter head pages. -- ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue33479> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33257] Race conditions in Tkinter with non-threaded Tcl
Change by Ivan Pozdeev <ivan_pozd...@mail.ru>: -- pull_requests: +6627 ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue33257> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22552] ctypes.CDLL returns singleton objects, resulting in usage conflicts
Change by Ivan Pozdeev <ivan_pozd...@mail.ru>: -- resolution: -> rejected stage: -> resolved status: open -> closed ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue22552> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue31645] openssl build fails in win32 if .pl extension is not associated with Perl
Change by Ivan Pozdeev <ivan_pozd...@mail.ru>: -- resolution: -> wont fix stage: -> resolved status: open -> closed ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue31645> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33257] Race conditions in Tkinter with non-threaded Tcl
Ivan Pozdeev <ivan_pozd...@mail.ru> added the comment: 2nd fix ready. @devteam The issue is fixed completely now, the fix can be merged. Pushed to the same PR, see https://github.com/python/cpython/pull/6444#issuecomment-389844872 why. Used the option "Hold the Tcl lock for the entire duration of a PythonCmd. To allow "downstream" calls, make the Tcl lock reentrant." from https://bugs.python.org/issue33257#msg316087 as it's the cleanest fix. @Scott M, you can help here by doing a code review. -- ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue33257> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33479] Document tkinter and threads
Ivan Pozdeev <ivan_pozd...@mail.ru> added the comment: @markroseman replied to python-dev since those perspectives are off topic for this ticket. -- ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue33479> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33316] Windows: PyThread_release_lock always fails
Ivan Pozdeev <ivan_pozd...@mail.ru> added the comment: > So I would drop the pretense and make these fns void. Scratch that. In POSIX, they are #define'd as pthread fns which do return an error code. So, nothing more to do here. -- ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue33316> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33479] Document tkinter and threads
Ivan Pozdeev <ivan_pozd...@mail.ru> added the comment:
> This seems very complicated. The official line on threads for Tk has always
> been to make all Tk calls from one thread, which is at least predictable and
> comprehensible. Is there any reason for Tkinter to suggest anything different?
Tcl/Tk doesn't have a notion of a blocking event loop like other GUI toolkits
do. Any code using Tcl must essentially call Tcl_DoOneEvent/`update` regularly
at strategic points.
This allows to work completely in one thread -- i.e. even if the OS has no
threads whatsoever. Tcl/Tk is very old, and this model made perfect sense back
then (and still does in some scenarios -- e.g.
https://stackoverflow.com/questions/4083796/how-do-i-run-unittest-on-a-tkinter-app
-- so there's no point in dropping it).
If we'll be updating tutorials (the reference is a priority though), we
definitly need to demonstrate this option.
The current best practice for GUI programming is different. There's one "GUI"
thread that runs just the event loop constantly, and other threads submit
GUI-related work items into its queue (whatever they are called - "messages",
"events", "futures"...). Likewise, for any lengthy task, the GUI thread spawns
worker threads that report back on their progress via the same queue.
All more or less modern GUI toolkits implement and advertize this model as the
primary one -- as does Tkinter. So, at least the work item submitting API must
be thread-safe (and in other GUI tooltikts, it is -- see
https://mail.python.org/pipermail/python-dev/2018-May/153359.html ).
For programmer's convenience, Tkinter does this transparently: whenever and
whatever Python thread a call is made from, it makes it look for the Tcl
interpreter as if all calls are sequential, and it enforces correct order for
interdependent calls.
A great deal of complexity comes from the fact that Tcl's threading model is
very unothodox. Tcl's team seem to only have accepted threads reluctantly and
to have been campaigning against threads for years before that
(https://wiki.tcl.tk/38128 is arguably the most egregious case).
So, what they did is tie a Tcl interpreter to an OS thread (by using thread
local storage for its data). Long story short, no-one else does it like this
(at least, I've never seen or heard of anything of the kind), and this is not
what thread-local storage is meant for
(http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2014/n4324.html lists some
use cases for TLS). The best practice is to use locks to ensure orderly access
to the shared state instead. My guess for the decision is it was the easiest
way to migrate the code base (yet still tough as https://wiki.tcl.tk/1370 seems
to hint), and it kinda lines up with that "single thread" mindset.
Tkinter thus has to jump through hoops for calls coming from other threads
(since for Python, it absolutely doesn't matter which OS thread makes a call).
All the limitations when using threaded Tcl (see the letter attached to the
ticket for details) come solely from this tying. With nonthreaded Tcl (bugs
nonwithstanding), it's free-for-all, everything can be called from everywhere.
The only upside is that with threaded Tcl, calls to different interpreters can
run in parallel.
> This ignores the compilation issue of course. FYI, the Tcl core group will
> probably eliminate the possibility of doing non-threaded builds in the
> future, though with backwards compatibility issues, that's neither here nor
> there.
I know. Me asking them for clarifications from Tcl/Tk's side seems to have
triggered it, in fact. Since the sole offender is their threading model, the
way is to show them how it's defective and work towards improving it. We have
at least a few years with old versions, and the limitations seem tolerable at
first glance anyway, so that's not a priority.
> do you know how to identify the tcl/tk build on MacOS or Linux?
The same way Tkinter does @_tkinter.c:624 . The equivalent Python is
`tkinter.Tk().tk.call("array","get","tcl_platform","threaded")`
--
___
Python tracker <rep...@bugs.python.org>
<https://bugs.python.org/issue33479>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33479] Document tkinter and threads
Ivan Pozdeev <ivan_pozd...@mail.ru> added the comment: I was composing a letter to python-dev with all I know of how tkinter works with regard to threads and Tcl, and the fixing plans so that we're all on the same page if you wish to participate. I'm no longer sure if it belongs in the mailing list so see it in the attachment. The plan I have for fixing the documentation is there towards the end. It includes what you suggested but is more detailed. I'll recite it below: --- * Document the current behaviour by rewriting https://docs.python.org/3/library/tkinter.html . Since I've recently learned Tkinter, I can say which critical information is missing and which existing one proved useless. Principles: * include fundamental information critical to understand the module's behaviour * concentrate on reference documentation for the module because it's more important to have than a user guide ( https://meta.serverfault.com/questions/8934/what-to-do-with-questions-when-the-answer-is-in-a-man-page#comment22241_8938 ) * split off/drop anything unrelated to the above two Resulting scheme: 0. Intro: +supported Tcl/Tk versions and the two flavors; the fact that Tkinter adds its own logic, and the Tkinter doc concentrates on that logic and refers to Tcl/Tk docs for details that are taken from there without changes. * Move external links section to be bottom. 1. Architecture. "Unlike most other GUI toolkits, Tcl/Tk consists of a few separate modules with a clear distinction between them, and this is non-transparect to the user:..." Base on https://docs.python.org/3/library/tkinter.html#how-tk-and-tkinter-are-related , but focus on what implements what rather than what _calls_ what; drop Xlib entry (transparent implementation detail, not required for understanding). 2. Threading model. The above-described general idea and user-visible limitations what can be called where and when. (the letter has details that this refers to, too long to recite) 3. References for `tkinter`, `tkinter.Tk()`, `_tkinter.Tkapp` (public interface only -- `call()` at least). Mention which functions cannot be called from other threads. Do not mention the two issue33257 bugs. * Move widget reference to another page. * Drop Python-Tcl mappings unless the reference sections need them as supplemental reference. * Drop tutorial: too primitive to be useful. Move tutorials to another page like https://docs.python.org/3/library/logging.html does. * Drop https://docs.python.org/3/library/tk.html -- proved hard to find. Make https://docs.python.org/3/library/tkinter.html the head page instead. * Discuss which of the described behaviour is as intended and which is a bug. Proceed with fixing. -- assignee: -> docs@python components: +Documentation nosy: +Ivan.Pozdeev, docs@python Added file: https://bugs.python.org/file47586/[Python-Dev] Tkinter threading model description and fix plan.eml ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue33479> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33480] Improvement suggestions for urllib.parse.urlparser
Ivan Pozdeev <ivan_pozd...@mail.ru> added the comment: Such drastic changes of uncertain usefulness are best discussed at python-ideas first. What you're really asking for seems to be to parse all "levels" at the same time. Try to think of a use case that would make that help anything practical and bring that to the list. I fail to see any use case 'cuz you never need query parameters and things like username/port at the same time. All else that you suggest is either already being done (username/port parsing, read the docs) or likewise has no use cases I can think of where it would make things more convenient than they already are (dict emulation, None). -- nosy: +Ivan.Pozdeev ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue33480> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33257] Race conditions in Tkinter with non-threaded Tcl
Ivan Pozdeev <ivan_pozd...@mail.ru> added the comment: The 1st PR is ready. I'll continue working on the 2nd one in this ticket, too, if there are no objections. -- ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue33257> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33412] Tkinter hangs if using multiple threads and event handlers
Ivan Pozdeev <ivan_pozd...@mail.ru> added the comment: > Without thread support, event generation from multiple threads fails > immediately. This ticket is for threaded Tcl only, so this is off topic. In nonthreaded Tcl, this script crashes rather than freezes, for an entire ly different reason that I already explained in https://bugs.python.org/issue33257 . This ticket is solved if you ask me. The only remaining matter is that there's no documentation: * on Tkinter threading model: https://docs.python.org/3/library/tk.html claims full thread safety which is untrue. * on best practices with Tkinter: as you could see, all the more or less obvious solutions are flawed. (That includes my solution: the program doesn't terminate gracefully if you close the window by hand.) I'm going to cover at least the first item as part of executing Guido's suggestion to "add a warning to the docs". -- ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue33412> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33412] Tkinter hangs if using multiple threads and event handlers
Ivan Pozdeev <ivan_pozd...@mail.ru> added the comment: Attached a fixed script. `Tk.after()` works from a worker thread, while `Tk.destroy()` doesn't. That's because Tkinter implements Tcl calls (_tkinter.c:Tkapp_Call) from another thread by posting an event to the interpreter's queue (Tcl_ThreadQueueEvent) and waiting for result. So a call normally works, but would hang if the interpreter's event loop is not running. `destroy()`'s Python part (Lib\tkinter\__init__.py:2055) stops the event loop, then makes more Tcl calls -- which hang for the aforementioned reason if made from another thread. -- Added file: https://bugs.python.org/file47570/TkinterHanders32.py ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue33412> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33412] Tkinter hangs if using multiple threads and event handlers
Ivan Pozdeev <ivan_pozd...@mail.ru> added the comment: > Another possibility is for stop() to change conditions so that > 'self.target.event_generate(c)' fails with an exception Could you elaborate? Since there're no docs on event_generate(), I can't look up how to make it "fail with an exception" without actually posting an event. > The only problem is that the first t.join() hangs because of a thread > deadlock bug. t.join() blocks until t.run exits. t.run does not exit until > the last event_generate, with running=False, returns. But that blocks until > dummy_handler runs. > I suppose there is a teeny possibility that 'running' could be flipped > between the test and the call. Can that be prevented with a lock? The idea is to let the worker threads finish their work, not terminate them forcibly. So the real problem is that stop() blocks the event loop. It should rather run asynchronously, wait for threads, then trigger `self.root.destroy()` in the main thread... somehow. -- resolution: wont fix -> not a bug ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue33412> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33257] Race conditions in Tkinter with non-threaded Tcl
Change by Ivan Pozdeev <ivan_pozd...@mail.ru>: Added file: https://bugs.python.org/file47566/0001-build-2.7-with-threaded-Tcl.patch ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue33257> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33257] Race conditions in Tkinter with non-threaded Tcl
Change by Ivan Pozdeev <ivan_pozd...@mail.ru>: Added file: https://bugs.python.org/file47565/0001-Build-non-threaded-debug-Tcl.zip ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue33257> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33257] Race conditions in Tkinter with non-threaded Tcl
Ivan Pozdeev <ivan_pozd...@mail.ru> added the comment: ... Finally, since https://bugs.python.org/issue30916, the 3.x Windows build uses a binary external and needs even more patching to build and link against a debug and/or custom Tcl/Tk (too long to describe, can give a patch). -- ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue33257> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33257] Race conditions in Tkinter with non-threaded Tcl
Ivan Pozdeev <ivan_pozd...@mail.ru> added the comment: > Is threaded tcl just a compile switch, as with CPython? Yes. In tcl.vcxproj, tk.vcxproj and tix.vcxproj, there are parameters named like TclOpts that are passed as OPTS arg to the lib's makefiles. For Tcl 8.5, you need to pass "threads" to build with threads; for 8.6, "nothreads" to build without. Also, in tcltk.props, the resulting DLL/LIB names are hardcoded with or without the 't' suffix. You'll need to fix that if you wish to switch the Tcl flavor. -- ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue33257> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33257] Race conditions in Tkinter with non-threaded Tcl
Ivan Pozdeev <ivan_pozd...@mail.ru> added the comment: > An automated test would need to fail more consistently, without needing to > click. Running faster would also be good (and perhaps increase failure > rate). The the test would be that the file runs in subprocess without an > error (perhaps multiple times). There may be a test.support function for > this. Thanks for the ideas. I'm on it. Was thinking of just wrapping the logic with unittest machinery but for a subprocess, it's easier to test stderr for exceptions. > You also claimed that tkinter is so broken, with either threaded or > non-threaded tcl/tk https://bugs.python.org/issue33412 > CPython's current 2.7 windows installer is installing non-threaded 8.5.18. > (I have no idea if or how I could check.) You can check this by seeing that the Tcl/Tk DLLs lack the 't' suffix. > It would seem then, that we should change to threaded 8.5.18 before 2.7.16. > (2.7.15 was just released). https://bugs.python.org/issue33257#msg316092 -- ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue33257> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33412] Tkinter hangs if using multiple threads and event handlers
Ivan Pozdeev <ivan_pozd...@mail.ru> added the comment: > Do you have a suggestion for what to do short of dropping Tkinter support? Didn't really look into this. At first glance, from the trace log, the main thread seems to grab a lock at some initial point, and then tries to grab it again when running an event handler. So making the lock reentrant may help. -- ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue33412> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33257] Race conditions in Tkinter with non-threaded Tcl
Ivan Pozdeev <ivan_pozd...@mail.ru> added the comment: > Ivan, as I also said there, it is not clear to me, given your subsequent > comments, what you consider to be the status of the PR. The PR fixes the problem exposed by TkinterCrash2-2.py and TkinterCrash3-2-2.py and only lacks an autotest (I asked for any ideas for it and got no response). Then I found another related bug (https://bugs.python.org/issue33257#msg315286 ) and though to fix it as well -- but it turned out to be deeper than expected. I'd be fine to commit the current fix and work on the second bug separately. -- ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue33257> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33257] Race conditions in Tkinter with non-threaded Tcl
Ivan Pozdeev <ivan_pozd...@mail.ru> added the comment: > I ran TkinterCrash3-2-2.py 15 times on 64-bit Win10, installed x64 3.6.5, > with tk 8.6.8, and experienced no crashes. I wrote in the initial message that this bug only happens with nonthreaded Tcl, regardless of Python version and Tcl version. I've built Py2 with threaded Tcl 8.5.19.0 just to make sure, and indeed, it went away. -- ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue33257> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33257] Race conditions in Tkinter with non-threaded Tcl
Ivan Pozdeev <ivan_pozd...@mail.ru> added the comment: > So this issue doesn't occur when linked with a thread-aware Tcl/Tk, right? Maybe we should just make sure that's the only configuration we ensure? This would break compatibility, including some usage patterns (see the "Reuse the threaded versions" option above). Can this actually be considered for anything short of the next major release? While official Windows releases use a pocket version, POSIX ones would probably link against whatever a distribution provides. -- nosy: +__Vano ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue33257> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33257] Race conditions in Tkinter with non-threaded Tcl
Ivan Pozdeev <ivan_pozd...@mail.ru> added the comment: Here are the possible courses of action to fix this. All require design changes, thus can be shot down on a whim. So I'll ask for advice and approval at python-dev first, to be sure that my efforts won't be scrapped. First, some terminology I'll be using: * A "(Tcl interpreter) call" (in quotes) is a sequence of interdependent calls into Tcl that a Tkinter routine makes in the course of its action. * Outstanding "calls" are those from other threads that have not started yet and are currently waiting for the Tcl lock. * A "downstream" "call" is a call made by a running PythonCmd's Python payload. * Each call to PythonCmd has a "start" -- before releasing the Tcl lock -- and an "end" -- when it reacquires it to Tcl_SetObjResult() and return * A PythonCmd is "active" when it's before its "start" and "end" Now, the fix options proper: * At the "end", rearrange the Tcl interpreter's stack frames so that the current PythonCmd's one is on top (effectively, make it look as if independent "calls" use independent Tcl stacks. Theoretically possible.) * Confirmed this to be impossible with Tcl's public API * Prohibit other `PythonCmd's while one is already "active", except its "downstream" ones. Options: * At the "end", the old PythonCmd checks if there are others "calls" on top of it (will only happen if such "call" has another "active" PythonCmd on top: all other Tkinter routines hold the Tcl lock for their entire duration after the 33257 fix), wait if there are. * Problematic: this wait algorithm is biased against the old PythonCmd. Will lead to a potentially infinite wait if there are many threads making Tcl calls that result in PythonCmds. * Hold the Tcl lock for the entire duration of a PythonCmd. To allow "downstream" calls, make the Tcl lock reentrant. * Requires redesign of about two dozen functions in thread_*.c and pycond.h: there are only non-reentrant locks currently. * Behaviour change: wouldn't allow outstanding "calls" while a PythonCmd is "active". This change is transparent to Python code though. * Reuse the threaded versions of Tkapp_* routines: bind a nonthreaded interpreter to a thread and forward calls to it as events like in the threaded case * Problematic: visible behaviour change: `tk.call()`s would now hang without a running `mainloop()`. Currently, only event handlers don't fire. Rejected options: * The new PythonCmd somehow checks at "start" if another one is "active" and waits if it's not a descendant of it. * The new PythonCmd's Tcl interpreter frame has already been allocated when it gets control, so the old one cannot return before the new one in any case. -- ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue33257> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33412] Tkinter hangs if using multiple threads and event handlers
Ivan Pozdeev <ivan_pozd...@mail.ru> added the comment: > worker threads are waiting for the Tcl lock Pardon. They are waiting for Tkapp_ThreadSend()s into the main thread to return. The effect is still the same. -- ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue33412> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33412] Tkinter hangs if using multiple threads and event handlers
Change by Ivan Pozdeev <ivan_pozd...@mail.ru>: Added file: https://bugs.python.org/file47564/trace.py ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue33412> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33412] Tkinter hangs if using multiple threads and event handlers
New submission from Ivan Pozdeev <ivan_pozd...@mail.ru>: With threaded Tkinter, TkinterHanders3.py from https://bugs.python.org/issue33257 (attached) hangs. Tracing with thread_debug and a modified trace.py (to show TIDs, attached) shows that worker threads are waiting for the Tcl lock while the main thread that holds it keeps waiting for some other lock with a strange timeout: 19000: PyThread_acquire_lock_timed(001B0F80, 0) called 19000: PyThread_acquire_lock(001B0F80, 0) -> 0 19000: PyThread_acquire_lock_timed(001B0F80, -100) called Tested on 3.6 head, win7 x64, debug build. -- components: Tkinter files: TkinterHanders3.py messages: 316082 nosy: Ivan.Pozdeev priority: normal severity: normal status: open title: Tkinter hangs if using multiple threads and event handlers type: crash versions: Python 3.6, Python 3.7, Python 3.8 Added file: https://bugs.python.org/file47562/TkinterHanders3.py ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue33412> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33412] Tkinter hangs if using multiple threads and event handlers
Change by Ivan Pozdeev <ivan_pozd...@mail.ru>: Added file: https://bugs.python.org/file47563/trace.zip ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue33412> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33257] Race conditions in Tkinter with non-threaded Tcl
Change by Ivan Pozdeev <ivan_pozd...@mail.ru>: Added file: https://bugs.python.org/file47561/TkinterHanders3.py ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue33257> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33257] Race conditions in Tkinter with non-threaded Tcl
Ivan Pozdeev <ivan_pozd...@mail.ru> added the comment: > Look like the only way is to rearrange Tcl stack frames so that the right one > is on top. Scratch that. Tkinter allows to execute entire scripts where the order of the frames is important. -- ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue33257> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33257] Race conditions in Tkinter with non-threaded Tcl
Ivan Pozdeev <ivan_pozd...@mail.ru> added the comment: Finally dug up the reason but unsure how to fix it. Here's the trace. The format is `$TID: $FUNCTION [some of the args]' 0x1A34: Tkapp_Call(_object *, _object *) 0x1A34: PythonCmd(void *, Tcl_Interp *, int, const char * *) 0x0230f430 0x02d297d0 "47566760dummy_handler" 0x03a28580 "5" 0x03a29140 "??" 0x03a29140 "??" 0x1D7C: Tkapp_Call(_object *, _object *) 0x1D7C: PythonCmd(void *, Tcl_Interp *, int, const char * *) 0x0230f430 0x02d297d0 "47566760dummy_handler" 0x03a28580 "5" 0x03a29140 "??" 0x03a29140 "??" 0x1A34: PythonCmd(void *, Tcl_Interp *, int, const char * *) setting result 0x1A34: PythonCmd(void *, Tcl_Interp *, int, const char * *) exit: 0 TclStackFree: incorrect freePtr (02BA6430 != 02BA6A20). Call out of sequence?python_d.exe has triggered a breakpoint. A second PythonCmd is started when the 1st one is still in progress, then the 1st one returns, the Tcl interpreter tries to unwind the topmost stack frame, and it's the wrong one. Denying other PythonCmd is out of question (a nested cmd deadlocks). Waiting for them to finish, either (would wait for potentially unlimited time). Look like the only way is to rearrange Tcl stack frames so that the right one is on top. (Since Tkinter's "interpreter calls" (sequences of Tcl calls done while holding the lock) are supposed to be independent from one another, it doesn't really matter which order the frames are in.) I seriously doubt that's possible with just the public interface though. -- ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue33257> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33316] PyThread_release_lock always fails
Ivan Pozdeev <ivan_pozd...@mail.ru> added the comment: In all PyMUTEX_* and most PyCOND_* functions, all the underlying APIs are void. So there's no point for the functions to return anything other than as convention. A comment at the start of condvar.h says: "APIs generally return 0 on success and non-zero on error". In Python\ceval_gil.h:57, there are boilerplace checker macros for them (which will never fire). So I guess this is done for uniformity. Regardless, PyMUTEX_UNLOCK and PyMUTEX_LOCK at least are called both with and without checking the return value. So I would drop the pretense and make these fns void. --- That said, the reported error still persists after the fix. The code confusing about which value LeaveNonRecursiveMutex should return on success. Pushed another commit, confirmed to work now. -- ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue33316> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33316] PyThread_release_lock always fails
Change by Ivan Pozdeev <ivan_pozd...@mail.ru>: -- keywords: +patch pull_requests: +6237 stage: -> patch review ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue33316> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33316] PyThread_release_lock always fails
New submission from Ivan Pozdeev <ivan_pozd...@mail.ru>: In win7 x64 debug mode with thread_debug=1, every PyThread_release_lock() is accompanied with a message on stderr: Could not PyThread_release_lock() error: 0 -- components: Interpreter Core, Windows messages: 315497 nosy: Ivan.Pozdeev, paul.moore, steve.dower, tim.golden, zach.ware priority: normal severity: normal status: open title: PyThread_release_lock always fails type: behavior versions: Python 3.6, Python 3.7, Python 3.8 ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue33316> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33257] Race conditions in Tkinter with non-threaded Tcl
Change by Ivan Pozdeev <ivan_pozd...@mail.ru>: Removed file: https://bugs.python.org/file47535/TkinterHanders.py ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue33257> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33257] Race conditions in Tkinter with non-threaded Tcl
Change by Ivan Pozdeev <ivan_pozd...@mail.ru>: Added file: https://bugs.python.org/file47536/TkinterHanders.py ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue33257> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33257] Race conditions in Tkinter with non-threaded Tcl
Ivan Pozdeev <ivan_pozd...@mail.ru> added the comment: Attached reproducing example for event handlers. If launching more than one EventThread, it abort()'s immediately. -- Added file: https://bugs.python.org/file47535/TkinterHanders.py ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue33257> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33257] Race conditions in Tkinter with non-threaded Tcl
Ivan Pozdeev <ivan_pozd...@mail.ru> added the comment: Wait a second... I think I noticed another similar bug. In Tkapp_CallProc. Same case, Tkapp_CallArgs and Tkapp_CallDeallocArgs are called when not holding both locks. The attached example doesn't use Python event handlers, so it didn't manifest itself in my tests. -- ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue33257> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33257] Race conditions in Tkinter with non-threaded Tcl
Ivan Pozdeev <ivan_pozd...@mail.ru> added the comment: @terry.reedy Apparently, this bug has gone unnoticed for years (to be precise, since https://github.com/python/cpython/commit/b5bfb9f38c786c3330b2d52d93b664588c42283d in 2002), and there are all kinds of weird rumors circulating about whether tkinter is thread-safe or not, and what can be run where ( see e.g. https://stackoverflow.com/search?q=tkinter+thread+safe ). Having run into it in my project, I set out to get to the bottom of this ( https://github.com/RedFantom/mtTkinter/issues/5 ) ...and looks like I did. -- ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue33257> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33257] Race conditions in Tkinter with non-threaded Tcl
Ivan Pozdeev <ivan_pozd...@mail.ru> added the comment: My best idea for a test as of now is to run the example ~20 times in a loop (or continuously for a comparable amount of time & threads) and catch and register any exceptions in all threads as per https://stackoverflow.com/questions/18349614/check-if-a-python-thread-threw-an-exception . -- ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue33257> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33255] json.dumps has different behaviour if encoding='utf-8' or encoding='utf8'
Ivan Pozdeev <ivan_pozd...@mail.ru> added the comment: Treating 'utf-8' and its aliases differently (when they specifically mean the Python's, rather than something else's, encoding) is definitely as issue. You shouldn't hardcode a list of aliases though; rather use existing facilities to resolve them. From quick googling, e.g. `codecs.lookup().name` can get the canonical name. Make sure to follow https://devguide.python.org/pullrequest when doing the PR; a test case will likely be needed, too. -- nosy: +Ivan.Pozdeev ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue33255> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33257] Race conditions in Tkinter with non-threaded Tcl
Change by Ivan Pozdeev <ivan_pozd...@mail.ru>: -- keywords: +patch pull_requests: +6139 stage: -> patch review ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue33257> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33257] Race conditions in Tkinter with non-threaded Tcl
Change by Ivan Pozdeev <ivan_pozd...@mail.ru>: Added file: https://bugs.python.org/file47530/TkinterCrash3-2-2.py ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue33257> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33257] Race conditions in Tkinter with non-threaded Tcl
New submission from Ivan Pozdeev <ivan_pozd...@mail.ru>:
(Marked only 2.7 as affected but this would affect any branch if built with
nonthreaded Tcl.)
When running the attached TkinterCrash2-2.py repeatedly with 2.7.14 and 2.7
head, win7 x64, two kinds of errors pop up randomly:
1. Crashes and freezes.
2. Exceptions on the console like:
Exception in thread Thread-14:
Traceback (most recent call last):
File "C:\Users\Sasha\Documents\cpython\lib\threading.py", line 801, in __boots
trap_inner
self.run()
File "../tkt/TkinterCrash2-2.py", line 50, in run
self.deliverToqueue((self.target, z, y))
File "../tkt/TkinterCrash2-2.py", line 133, in arrival_122
new_yz[1])
File "C:\Users\Sasha\Documents\cpython\lib\lib-tk\Tkinter.py", line 2328, in
create_line
return self._create('line', args, kw)
File "C:\Users\Sasha\Documents\cpython\lib\lib-tk\Tkinter.py", line 2310, in
_create
*(args + self._options(cnf, kw
ValueError: invalid literal for int() with base 10: 'None'
Running the same code with the minimal required changes (attached as
TkinterCrash3-2-2.py) under 3.6 (same platform) goes without any errors.
Diagnostics showed:
1. Under debug Python, a crash became an MSVC double-free assertion error. The
stacktrace is:
Tkapp_Call->Tkapp_CallDeallocArgs->Tcl_DecrRefCount->TclFreeObj->free
2. The exceptions are caused by a ` create line` Tk call randomly
returning "None" (a string) instead of an integer -- which it should never do
according to its doc. Since it happens inconsistently, this also suggests a
race condition.
3. In Tkapp_Call and SetVar, Tcl lock isn't held when creating/destroying Tcl
objects for the call. The underlying fns modify the global state (free objects
list, reference counters), so it should be held.
These two are the only such places in the code.
I've fixed this, will file the PR shortly. Holding the lock when calling
Tkapp_CallDeallocArgs in Tkapp_Call eliminated the crashes and freezes, locking
objects' creation in the same fn decreased the number of exceptions greatly,
and locking creation in SetVar, too, eliminated them completely.
I did not check if objects created with AsObj in SetVar need to be disposed of
like in Tkapp_Call.
Also have no idea how to autotest the fix. If someone does, I'm all ears.
--
components: Tkinter
files: TkinterCrash2-2.py
messages: 315172
nosy: Ivan.Pozdeev
priority: normal
severity: normal
status: open
title: Race conditions in Tkinter with non-threaded Tcl
type: crash
versions: Python 2.7
Added file: https://bugs.python.org/file47529/TkinterCrash2-2.py
___
Python tracker <rep...@bugs.python.org>
<https://bugs.python.org/issue33257>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22552] ctypes.CDLL returns singleton objects, resulting in usage conflicts
Change by Ivan Pozdeev <ivan_pozd...@mail.ru>: -- type: crash -> behavior ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue22552> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue31643] test_uuid: test_getnode and test_windll_getnode fail if connected to the Internet via an Android phone
Ivan Pozdeev <ivan_pozd...@mail.ru> added the comment: > Xavier de Gaye <xdeg...@gmail.com> added the comment: > > On a Samsung device with Android 5.1 and tethering enabled, only one network > device has a MAC address and it is not an all-zero MAC address. > > Ivan can you please provide more information about this problem: which > device, what Android version, what is the output of 'ip link' ? C:\> ipconfig -all <...> Samsung Galaxy S7 - Ethernet адаптер: DNS-суффикс этого подключения . . : Описание . . . . . . . . . . . . : Android USB Ethernet/RNDIS Физический адрес. . . . . . . . . : 00-00-00-00-00-00 Dhcp включен. . . . . . . . . . . : да Автонастройка включена . . . . . : да IP-адрес . . . . . . . . . . . . : 192.168.42.2 Маска подсети . . . . . . . . . . : 255.255.255.0 Основной шлюз . . . . . . . . . . : 192.168.42.129 DHCP-сервер . . . . . . . . . . . : 192.168.42.129 DNS-серверы . . . . . . . . . . . : 192.168.42.129 Device name: Samsung Galaxy S7 Model number: SM-G930F Android version: 6.0.1 Firmware version: G930FXXU1APGJ Baseband version: G930VVRS4APH1 Build number: MMB29M.G930VVRU2APG5 > Serhiy Storchaka<storchaka+cpyt...@gmail.com> added the comment: > > I'm not sure that is better than a random address. Even if an > all-zero MAC address is not explicitly prohibited it is used in a multiple > independent devices. A random address looks more appropriate fore the purpose > of generating a UUID. The question is not whether it's good UUID source material. The question, for the purpose of the test, is whether it's valid getnode() output -- which it is for the current logic. -- nosy: +__Vano ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue31643> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue31643] test_uuid: test_getnode and test_windll_getnode fail if connected to the Internet via an Android phone
Change by Ivan Pozdeev <ivan_pozd...@mail.ru>: Removed file: https://bugs.python.org/file47178/0001-Allow-for-all-zero-MAC-based-node-ID-e.g.-mobile-mod.patch ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue31643> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue31643] test_uuid: test_getnode and test_windll_getnode fail if connected to the Internet via an Android phone
Change by Ivan Pozdeev <ivan_pozd...@mail.ru>: -- pull_requests: +4701 ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue31643> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue32266] test_pathlib fails if current path has junctions
Change by Ivan Pozdeev <ivan_pozd...@mail.ru>: -- keywords: +patch pull_requests: +4679 stage: -> patch review ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue32266> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue32266] test_pathlib fails if current path has junctions
New submission from Ivan Pozdeev <ivan_pozd...@mail.ru>:
On this machine, C:\Users is a junction to D:\Users .
Sample failure:
Running Release|x64 interpreter...
FFFs...s..s.s.ss..FF...FFFs...s..s.s.ss..FF...
==
FAIL: test_complex_symlinks_absolute (__main__.PathTest)
--
Traceback (most recent call last):
File "Lib\test\test_pathlib.py", line 2027, in test_complex_symlinks_absolute
self._check_complex_symlinks(BASE)
File "Lib\test\test_pathlib.py", line 1994, in _check_complex_symlinks
self.assertEqual(p, P)
AssertionError:
WindowsPath('D:/Users/Ivan/Documents/cpython_all/@test_2376_tmp') !=
WindowsPath('C:/Users/Ivan/Documents/cpython_all/@test_2376_tmp')
==
FAIL: test_complex_symlinks_relative (__main__.PathTest)
--
Traceback (most recent call last):
File "Lib\test\test_pathlib.py", line 2031, in test_complex_symlinks_relative
self._check_complex_symlinks('.')
File "Lib\test\test_pathlib.py", line 1994, in _check_complex_symlinks
self.assertEqual(p, P)
AssertionError:
WindowsPath('D:/Users/Ivan/Documents/cpython_all/@test_2376_tmp') !=
WindowsPath('C:/Users/Ivan/Documents/cpython_all/@test_2376_tmp')
==
FAIL: test_complex_symlinks_relative_dot_dot (__main__.PathTest)
--
Traceback (most recent call last):
File "Lib\test\test_pathlib.py", line 2035, in
test_complex_symlinks_relative_dot_dot
self._check_complex_symlinks(os.path.join('dirA', '..'))
File "Lib\test\test_pathlib.py", line 1994, in _check_complex_symlinks
self.assertEqual(p, P)
AssertionError:
WindowsPath('D:/Users/Ivan/Documents/cpython_all/@test_2376_tmp') !=
WindowsPath('C:/Users/Ivan/Documents/cpython_all/@test_2376_tmp')
==
FAIL: test_resolve_common (__main__.PathTest)
--
Traceback (most recent call last):
File "Lib\test\test_pathlib.py", line 1491, in test_resolve_common
os.path.join(BASE, 'foo'))
AssertionError: 'D:\\Users\\Ivan\\Documents\\cpython_all\\@test_2376_tmp\\foo'
!= 'C:\\Users\\Ivan\\Documents\\cpython_all\\@test_2376_tmp\\foo'
- D:\Users\Ivan\Documents\cpython_all\@test_2376_tmp\foo
? ^
+ C:\Users\Ivan\Documents\cpython_all\@test_2376_tmp\foo
? ^
==
FAIL: test_resolve_dot (__main__.PathTest)
--
Traceback (most recent call last):
File "Lib\test\test_pathlib.py", line 1550, in test_resolve_dot
self.assertEqual(q.resolve(strict=True), p)
AssertionError:
WindowsPath('D:/Users/Ivan/Documents/cpython_all/@test_2376_tmp') !=
WindowsPath('C:/Users/Ivan/Documents/cpython_all/@test_2376_tmp')
==
FAIL: test_complex_symlinks_absolute (__main__.WindowsPathTest)
--
Traceback (most recent call last):
File "Lib\test\test_pathlib.py", line 2027, in test_complex_symlinks_absolute
self._check_complex_symlinks(BASE)
File "Lib\test\test_pathlib.py", line 1994, in _check_complex_symlinks
self.assertEqual(p, P)
AssertionError:
WindowsPath('D:/Users/Ivan/Documents/cpython_all/@test_2376_tmp') !=
WindowsPath('C:/Users/Ivan/Documents/cpython_all/@test_2376_tmp')
==
FAIL: test_complex_symlinks_relative (__main__.WindowsPathTest)
--
Traceback (most recent call last):
File "Lib\test\test_pathlib.py", line 2031, in test_complex_symlinks_relative
self._check_complex_symlinks('.')
File "Lib\test\test_pathlib.py", line 1994, in _check_complex_symlinks
self.assertEqual(p, P)
AssertionError:
WindowsPath('D:/Users/Ivan/Documents/cpython_all/@test_2376_tmp') !=
WindowsPath('C:/Users/Ivan/Documents/cpython_all/@test_2376_tmp')
==
FAIL: test_complex_symlinks_relative_dot_dot (__main__.WindowsPathTe
