[issue16487] Allow ssl certificates to be specified from memory rather than files.
Jan-Philip Gehrcke added the comment: I am not too attached to "my" patch, but because I love Python I really would like us to land on a solution. > However I want all changes and new additions to the SSL module to follow PEP > 543 so I can provide a PEP 543-compatible interface in the near future (3.8 > or 3.9). Christian, I wonder: did we make progress on this front? Will the stdlib in the 3.9 release make it easy to load cert and key material from memory? > Mainly to not crush the will to help and contribute Thanks Anton for your words, truly appreciated. I do not want to sound too negative, but I would like to confirm that emotionally this issue here was a bit of an unpleasant experience for me, and it certainly inhibited my intentions, energy, motivation to (try to) contribute more. Thanks, though, to all supporters here. -- versions: +Python 3.8, Python 3.9 ___ Python tracker <https://bugs.python.org/issue16487> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16487] Allow ssl certificates to be specified from memory rather than files.
Jan-Philip Gehrcke added the comment: Thanks for the discussion. Since I tried to join the efforts here in 2016 two years ago I was (and still am) enthusiastic, and willing to invest quite a bit of energy. Still, we have missed the 3.6 and 3.7 releases to change something about the fact that there is no simple way to load certificate data from memory rather than files. I really appreciate the work that as been done on https://www.python.org/dev/peps/pep-0543/. It's a truly great effort. But I think the fact that we did not land a solution since 2012 is somewhat an indicator showing that building smaller chunks and showing the willingness to iterate on a solution would have led to a more tangible outcome, something that I could use right now when building Python applications (how great would that be?). Honestly, the small API changes that were discussed here for years and that I think I implemented do not seem problematic to me from a maintenance and deprecation perspective. Yes, PEP 543 proposes a more idealistic way of doing things. But I do not see how implementing PEP 543 would have been slowed down or negatively affected in any way by the last patch I proposed. https://github.com/python/cpython/pull/5162 looks very big, and C is not as much my comfort zone as is Python. Still, please let me know if you would like me to contribute! By the way, I would have _really_ liked to see the following feedback around the first time I submitted the patch: > For example it's missing several GIL releases calls. The password callback > doesn't look correct in some places. I remember that this aspect of writing the patch consumed quite a bit of my time and I operated with care, knowing the risks. I very much would have appreciated specific feedback on this. To learn, and to actually address problems. Jan-Philip -- ___ Python tracker <https://bugs.python.org/issue16487> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16487] Allow ssl certificates to be specified from memory rather than files.
Jan-Philip Gehrcke <jgehr...@gmail.com> added the comment: Hello! I would very much appreciate if we can find a way for us to get another review for the last patch. I did most of the work in August 2016 and got a review from Senthil and Christian which I processed. When I got back to the patch for converting it to the GitHub workflow about a year later I realized that the details faded from my mind. 4 months later, after re-digging into the details, I went into the patch again and resolved the merge conflicts that accumulated over time. Antoine kindly left another round of comments which I replied to. Now, 2 months after that last effort I'd like to not forget about the details again, but instead be able to respond to review feedback while still being on top of the matter. So, any kind and level of feedback would be greatly appreciated. Thank you! -- ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue16487> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16487] Allow ssl certificates to be specified from memory rather than files.
Jan-Philip Gehrcke <jgehr...@gmail.com> added the comment: Hey Antoine, Christian, Senthil! I have invested quite a bit more time to double-check my responses to the questions asked so far, clarified where appropriate, and updated the pull request on GitHub after manually resolving the merge conflicts that accumulated over time. I would very much appreciate to get another round of feedback. Can we somehow expedite this? I am available for pushing this further, so that maybe we can still land this in 3.7 after we missed 3.6 last year. Based on the diff this change _appears_ to be risky but I would like to stress again that I approached this issue pretty conservatively. I tried to make this a conceptually backwards-compatible change by keeping the old code path intact (the old tests pass, too, but this is not the best criterion here). Cheers and thanks! Jan-Philip -- ___ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue16487> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16487] Allow ssl certificates to be specified from memory rather than files.
Jan-Philip Gehrcke added the comment: Hey Senthil and Christian! > Could you convert your latest patch into PR against > https://github.com/python/cpython That was fun. There we go: https://github.com/python/cpython/pull/2449 I hope I was not too late with that for the 3.7 development. -- ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue16487> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16487] Allow ssl certificates to be specified from memory rather than files.
Changes by Jan-Philip Gehrcke <jgehr...@gmail.com>: -- pull_requests: +2506 ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue16487> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16487] Allow ssl certificates to be specified from memory rather than files.
Jan-Philip Gehrcke added the comment: Christian, Senthil, would appreciate if I got another round of feedback (in the review thread) :-) -- nosy: +orsenthil ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue16487> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16487] Allow ssl certificates to be specified from memory rather than files.
Jan-Philip Gehrcke added the comment: Thanks Christian, much appreciated. Just responded to your review. -- ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue16487> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16487] Allow ssl certificates to be specified from memory rather than files.
Jan-Philip Gehrcke added the comment: Hello! Like everybody in this thread I would love to see this land and have prepared a new patch, hoping that we can process this still for 3.6. Antoine summarized the core task here very well: > Let's stay focused on what is > necessary to solve this use case, IMO (i.e. specify non-file-backed data > as a certification chain or private key). What follows is what I believe is the consensus in this thread about a minimum viable solution (points mentioned earlier, enriched with some detail): - The `certfile` and `keyfile` arguments to `SSLContext.load_cert_chain()` can both be either a file path or a file object. - `str` or `bytes` values to both, `certfile` and `keyfile`, are interpreted as paths. - The `password` behavior must stay as is, independent of `certfile`/`keyfile` being file object or path. - When handing file paths to `load_cert_chain()`, the behavior must stay as is, in all detail. - When handing file objects to `load_cert_chain()`, - use the OpenSSL API call `PEM_read_bio_PrivateKey()` for loading the key data (reading PEM data is what's currently supported). - support the case where the `certfile` file object first presents a private key and then (a) certificate(s), as currently documented: """ Often the private key is stored in the same file as the certificate; in this case, only the certfile parameter to SSLContext.load_cert_chain() and wrap_socket() needs to be passed. If the private key is stored with the certificate, it should come before the first certificate in the certificate chain: """ Ref: - https://docs.python.org/3/library/ssl.html#combined-key-and-certificate - resemble the OpenSSL API call `SSL_CTX_use_certificate_chain_file()` behavior for loading the certificate data (as this is what's currently used in the `SSLContext.load_cert_chain()` implementation in _ssl.c. There is no direct correspondence for this function reading the data directly from memory BIOs. Relevant documentation for resembling that behavior: """ SSL_CTX_use_certificate_chain_file() adds the first certificate found in the file to the certificate store. The other certificates are added to the store of chain certificates using SSL_CTX_add1_chain_cert. """ """ For a longer chain, the client must send the complete chain [...]. This can only be accomplished by either adding the intermediate CA certificates into the trusted certificate store for the SSL_CTX object [...], or by adding the chain certificates using the SSL_CTX_add_extra_chain_cert function, """ """ When building its own certificate chain, an OpenSSL client/server will try to fill in missing certificates from CAfile/CApath, if the certificate chain was not explicitly specified (see SSL_CTX_add_extra_chain_cert, SSL_CTX_use_certificate. """ Refs: - https://github.com/python/cpython/blob/0852878a81edd5c16776a68ce34c45cca233deae/Modules/_ssl.c#L2824 - https://www.openssl.org/docs/manmaster/ssl/SSL_CTX_use_certificate.html - https://www.openssl.org/docs/manmaster/ssl/SSL_CTX_set_client_cert_cb.html - https://www.openssl.org/docs/manmaster/ssl/SSL_CTX_load_verify_locations.html When inspecting the OpenSSL code for SSL_CTX_use_certificate_chain_file(), some of the above statements are not entirely correct. After inspection, to me the safest bet seems to keep the memory/BIO-based load_cert_chain() implementation as close as possible to SSL_CTX_use_certificate_chain_file() from OpenSSL 1.0.2h: https://fossies.org/dox/openssl-1.0.2h/ssl__rsa_8c_source.html#l00685 use_certificate_chain_file() in OpenSSL 1.1.0-pre6: https://fossies.org/dox/openssl-1.1.0-pre6/ssl__rsa_8c_source.html#l00599 Using this as a template (instead of coming up with a solution purely based on OpenSSL docs) ensures that this part of the current `SSLContext.load_cert_chain()` docs will be remain correct: """ The certfile [...] in PEM format containing the certificate as well as any number of CA certificates needed to establish the certificate’s authenticity. """ More behavioral detail that I planned with: - `certfile` and `keyfile`, if given both, must both be a file path or both be a file object. A mixture is undefined behavior and leads to an error depending on the specifics of the mixture. - If `certfile` and `keyfile` are not file paths, expect them to have a read() method and just call it. Let a TypeError or AttributeError be handled by the caller (as done elsewhere in the stdlib). - Use the existing Python MemoryBIO abstractions introduced by Geert Jansen in this beautiful patch: https://gith
[issue6634] sys.exit() called from threads other than the main one: undocumented behaviour
Jan-Philip Gehrcke added the comment: Martin, I very much like the order you suggested, thanks. I did not feel confident enough for re-structuring the entire entry. So, can we agree on using that for Python 2.7? Is there a consensus regarding the approach to take for Python 3.5? Except from Martin there was no feedback on the patch. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6634 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6634] sys.exit() called from threads other than the main one: undocumented behaviour
Changes by Jan-Philip Gehrcke jgehr...@gmail.com: Added file: http://bugs.python.org/file38096/issue6634_py27.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6634 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23427] Python should expose command when invoked with -c
Jan-Philip Gehrcke added the comment: Victor, I support the idea of sys.command. However, it would be unpopulated most of the time (e.g. set to None by default). Now, is that something we should push forward or not? I would work on a patch, but we should have an agreement first, I guess. Mihail, the original argv becomes modified in the very early bootstrap phase, and the command gets lost within that process: it gets *overwritten* with -c, which is exactly why you are observing two -c. This happens here: https://hg.python.org/cpython/file/default/Modules/main.c#l684 So, no, without a code change in main.c there will be no way to retain the command for later usage. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23427 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23427] Python should expose command when invoked with -c
New submission from Jan-Philip Gehrcke: When Python is invoked with the `-c command` switch, the command string does not get exposed in sys.argv: $ python -c import sys; print(sys.argv) ['-c'] $ python -c import sys; print(sys.argv) arg1 ['-c', 'arg1'] The command string does not get exposed anywhere, AFAIK, so it is inaccessible from within Python programs. There might be application scenarios in which it is useful to access the command string, such as for debugging purposes. One scenario is when a Python session should be able to re-spawn itself in a subprocess (I came across this question on StackOverflow: http://stackoverflow.com/q/28412903/145400) I propose to make the command string accessible. If you agree that it might make sense, the question is *how/where* to expose it. One possible way is to retain it in sys.argv, as in this example: $ python -c import sys; print(sys.argv) arg1 ['-c', 'import sys; print(sys.argv)', 'arg1'] The current sys.argv docs say If the command was executed using the -c command line option to the interpreter, argv[0] is set to the string '-c'. This sentence could then be adjusted to [...], argv[0] is set to the string '-c', and argv[1] contains the command. This method breaks existing applications that are started with the -c method and that consume command line arguments in a sys.argv[1:] fashion. The tests in Lib/test/test_cmd_line.py all pass, however. A second method would be to change sys.argv[0] from '-c' to '-c command'. This would break existing applications that check for sys.argv[0] == 'c'. A third method would be to leave sys.argv as it is, and expose the command with a new attribute in the sys module. I have attached a patch for variant 1 (passes all tests in Lib/test/test_cmd_line.py), to demonstrate which code is affected: the translation from the real argv to sys' argv is triggered in Modules/main.c. The patch does not change behavior of '-m' (it's funny, however, that the current version of main.c at first replaces the module string with '-m', whereas the runpy module later on replaces '-m' with the path to the module file anyway.). As a side node, I figure that the sys.argv documentation should be adjusted to properly reflect the -m behavior, which is: $ ./python -m testmodule foo testmodule sys.argv: ['/data/local/pythondev/pythontip/cpython/testmodule.py', 'foo'] Let me hear your comments, and I am willing to work on code and doc patches, thanks! -- assignee: docs@python components: Documentation, Interpreter Core files: sys_argv_cmd.patch keywords: patch messages: 235633 nosy: docs@python, georg.brandl, haypo, jgehrcke, pitrou priority: normal severity: normal status: open title: Python should expose command when invoked with -c type: enhancement versions: Python 3.5, Python 3.6 Added file: http://bugs.python.org/file38065/sys_argv_cmd.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23427 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6634] sys.exit() called from threads other than the main one: undocumented behaviour
Jan-Philip Gehrcke added the comment: I'd love to find an agreement here. I think we are quite close to getting this closed, so further input is very welcome. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6634 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20709] os.utime(path_to_directory): wrong documentation for Windows.
Jan-Philip Gehrcke added the comment: Can these super-small doc patches get applied or should we change something? Thanks! -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20709 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6634] sys.exit() called from threads other than the main one: undocumented behaviour
Jan-Philip Gehrcke added the comment: Thanks for your feedback Antoine. I'm not sure what the doc patch achieves. Let me try to bring things in order. It should achieve two things: 1. Properly describe the stderr-writing behavior of sys.exit(). === Current 2.7 docs: ..., and any other object is printed to stderr. This is wrong in its generality and requires clarification. Agreed? How would you improve the docs in this regard? This is the reasoning I had in mind: We could change the original sentence, but IMO it then contains too many pieces of important information and becomes difficult to digest, e.g.: If another type of object is passed, None is equivalent to passing zero, and any other object is printed to stderr (only when called in the main thread) and results in an exit code of 1.. Difficult, right? I thought about removing the stderr part from this sentence and discussing this topic separately, in simpler sentences. But: I think the stderr part needs to stay in the original sentence, because it clarifies how different argument types are dealt with. Then I took another point of view: we have acknowledged that the behavior is problematic, we just cannot change it anymore for 2.7. The direct consequence from this point of view is to warn about current behavior. 6 years ago, I took the docs literally and relied on getting proper error messages printed to stderr. And I didn't get those. I would have appreciated a warning, I guess. With this being said, I'd love to look at an alternative proposal of how we could change the docs in this regard. 2. Clarify that when called from a thread, the thread exits silently Current 2.7 docs: Since exit ultimately only raises an exception, it will only exit the process when called from the main thread. This sentence is problematic. It relates a cause to an effect, and this relation is wrong or at least incomplete. The only valuable information left in the sentence is the *effect*, without providing an explanation. I would rather want to take the inverse approach, and explain the *cause* as correct as possible. In other words: the above sentence misses to explain what actually happens when sys.exit() is called in a non-primary thread. The actual behavior and cause for mentioned effect is, as far as I am aware: If raised in a non-primary thread, SystemExit is caught automatically and causes the calling thread to exit silently. This is what I added as a note. And I think that we agree that this cause may have many effects, whereas only one of them is that sys.exit() called from a non-primary thread can not terminate the process. It only states more verbosely what is already said in the current version. I hope to have convinced you that this is not true. Also, warnings are really for important issues (such as security issues); using them too liberally is a disservice to the reader. I agree. I really think that both points need to be addressed in the docs, and I have now clarified my reasoning, but am very open to further suggestions. And I really do not want to make a strong point about whether things should end up in red or blue boxes :-). -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6634 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18454] distutils crashes when uploading to PyPI having only the username (no pw) defined
Changes by Jan-Philip Gehrcke jgehr...@gmail.com: Added file: http://bugs.python.org/file38006/issue18454_py27_prompt_test.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18454 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18454] distutils crashes when uploading to PyPI having only the username (no pw) defined
Jan-Philip Gehrcke added the comment: First, I want to address the situation in 2.7. Please have a look at my patch and my reasoning. This is my setup.py test file content: from distutils.core import setup setup(name='foo', version='1.0', packages=['testpackage']) This is the current situation for 2.7 head, in summary: 1) Without a proper .pypirc file in place, uploading fails with Upload failed (401): You must be identified to edit package information 2) With a valid .pypirc in place which does not define a password this exception is raised: TypeError: cannot concatenate 'str' and 'NoneType' objects (1) happens due to the fact that the default username/password strings are empty, but the request is still fired against PyPI. This obviously fails. This could be fixed by prompting for both, username and password or by aborting before sending the request. However, this would be quite a significant change in behavior. I think we should not do this, because AFAIK the 2.7 documentation does not specify what the behavior should be *without* existing .pypirc file. So, I think there might not be enough reason to consider such a change a bug fix. What do you think? (2) violates the docs and I guess this is what Nick meant when he said that the stdlib should be updated. I propose a patch so that what the docs say (If omitted, the user will be prompt to type it when needed) is what actually happens. The patch consists of two parts: Part A: issue18454_py27_prompt.patch This uses getpass/sys.stdin.readline() for retrieving the password, depending on whether the process is attached to a tty or not, respectively. The config parsing and value propagation is a little messy in the command/upload.py module, so I have also added comments for clarity and simplified the logic a little. Part B: issue18454_py27_prompt_test.patch - Here it gets messy. Normally, we want to have a new test case that fails before applying patch A, and succeeds after applying it. However, here we actually have to deal with an already existing bogus test case which succeeds before and fails after A gets applied. A very bad sign, usually. But in this case I am of the strong opinion that we can and have to change the test implementation, because: - the test actually makes sure that `cmd.password` is set to None when `cmd.finalize_options()` has done its work. However,`cmd.password` being None leads to the TypeError exception described above. That is, this test case basically ensures the very existance of the bug that we want to fix. - it has a suspiciously useless name. It is called `test_saved_password`, but actually tests the case when no password is saved in .pypirc. - this test actually has quite low code coverage. AFAIK it *only* tests the behavior of `cmd.finalize_options()`. We want to change this behavior, so we have to consider changing the test. My patch modifies this test case so that it has a proper name and makes sure that `cmd.finalize_options()` obtains a password from stdin. The involvement of stdin makes testing a little complicated. In a proper testing environment, we'd probably spawn a subprocess and provide it with some real stdin data. The distutils upload command test structure seems not to be prepared for this endeavor, so I have decided to temporarily/locally patch sys.stdin, and ensure restoration with a try/finally clause. With A and B in place, all distutils unit tests validate. I used two methods of invocation, for covering the two cases (with and without attached tty): $ nohup python test_distutils.py 21 out.log $ python test_distutils.py -- keywords: +patch Added file: http://bugs.python.org/file38005/issue18454_py27_prompt.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18454 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6634] sys.exit() called from threads other than the main one: undocumented behaviour
Changes by Jan-Philip Gehrcke jgehr...@gmail.com: Added file: http://bugs.python.org/file37988/issue6634_py35.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6634 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6634] sys.exit() called from threads other than the main one: undocumented behaviour
Changes by Jan-Philip Gehrcke jgehr...@gmail.com: Removed file: http://bugs.python.org/file37986/issue6634_py35.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6634 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6634] sys.exit() called from threads other than the main one: undocumented behaviour
Changes by Jan-Philip Gehrcke jgehr...@gmail.com: Removed file: http://bugs.python.org/file19006/thread_sys_exit_test.py ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6634 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6634] sys.exit() called from threads other than the main one: undocumented behaviour
Jan-Philip Gehrcke added the comment: For Python 3.5, I have attached a patch that - adds relevant test cases to test_threading.py which probe the interpreter's stderr output for compliance with what the docs state. - makes sys.exit(msg) write msg to stderr, even if called from a non-primary thread, so that the tests succeed. If we take this path, the documentation for 3.5 does not need to be adjusted. The discussion in this thread diversified itself a bit: Why don't threads have their own ThreadExit exception, rather than overloading the use, and therefore, the meaning, of the SystemExit exception? sys.exit and the SystemExit exception should *only* be used to exit the entire system, not just a thread! While I absolutely agree that this would be conceptually cleaner, implementing this would be a larger refactoring task. Deciding whether this should be done or not slows this issue down, and I think this discussion should probably be taken elsewhere. -- versions: +Python 3.5 -Python 3.1, Python 3.2 Added file: http://bugs.python.org/file37986/issue6634_py35.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6634 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6634] sys.exit() called from threads other than the main one: undocumented behaviour
Changes by Jan-Philip Gehrcke jgehr...@gmail.com: Removed file: http://bugs.python.org/file37988/issue6634_py35.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6634 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6634] sys.exit() called from threads other than the main one: undocumented behaviour
Changes by Jan-Philip Gehrcke jgehr...@gmail.com: Removed file: http://bugs.python.org/file37983/issue6634_py27.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6634 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6634] sys.exit() called from threads other than the main one: undocumented behaviour
Jan-Philip Gehrcke added the comment: Regarding the documentation patch: I like to start sentences with a capital letter. Perhaps change it to start “Calling :func:`exit` only terminates . . .”. Thanks for feedback. Have now used Invocation of to not repeat call* in the sentence, because I left the when called from the main thread part unchanged. Okay? With the code change patch, it might be neater to use the SystemExit.code attribute rather than e.args[0]. Oh, thanks. Was not aware of the existence of the `code` attribute. If anyone else was wondering: existence and behavior are defined in Objects/exceptions.c via `static PyMemberDef SystemExit_members[]` and via `static int SystemExit_init()`. It is populated upon construction of a SystemExit instance: Py_CLEAR(self-code); if (size == 1) self-code = PyTuple_GET_ITEM(args, 0); else /* size 1 */ self-code = args; Hence, the translation from arguments to exit code considers *all* arguments. I adjusted the patch to use the `code` attribute. -- Added file: http://bugs.python.org/file37989/issue6634_py35.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6634 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6634] sys.exit() called from threads other than the main one: undocumented behaviour
Changes by Jan-Philip Gehrcke jgehr...@gmail.com: Added file: http://bugs.python.org/file37990/issue6634_py27.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6634 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20709] os.utime(path_to_directory): wrong documentation for Windows.
Jan-Philip Gehrcke added the comment: Windows is the only Python-supported platform where utime did not work for directories, ages ago, right? If that is the case, I support Larry Hastings' approach of removing the entire sentence: Whether a directory can be given for path depends on whether the operating system implements directories as files (for example, *Windows does not*) Agreed? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20709 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20709] os.utime(path_to_directory): wrong documentation for Windows.
Changes by Jan-Philip Gehrcke jgehr...@gmail.com: -- keywords: +patch Added file: http://bugs.python.org/file37979/issue20709_py27.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20709 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20709] os.utime(path_to_directory): wrong documentation for Windows.
Changes by Jan-Philip Gehrcke jgehr...@gmail.com: Added file: http://bugs.python.org/file37980/issue20709_tip.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20709 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20709] os.utime(path_to_directory): wrong documentation for Windows.
Jan-Philip Gehrcke added the comment: I have added patches for the documentation, where I removed the sentence in question and re-wrapped the paragraph. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20709 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6634] sys.exit() called from threads other than the main one: undocumented behaviour
Jan-Philip Gehrcke added the comment: For Python 2.7, we will not change behavior, even if unexpected. Instead, the sys.exit-docs should be adjusted and - warn about the fact that nothing is written to stderr if sys.exit(msg) gets called from a non-primary thread, and - note that SystemExit raised in a non-primary thread lets the thread exit silently. I have attached a corresponding patch, please review my wording! -- keywords: +patch Added file: http://bugs.python.org/file37983/issue6634_py27.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6634 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21575] list.sort() should show arguments in tutorial
Jan-Philip Gehrcke added the comment: I have updated the patch with a cross-reference to the sorted() built-in, which explains the arguments. W.r.t. to Éric's suggestion: the sorted() doc refers to the sorting howto in the wiki. Now everything is connected. -- Added file: http://bugs.python.org/file35365/sort_tut_02.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue21575 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21575] list.sort() should show arguments in tutorial
New submission from Jan-Philip Gehrcke: Currently, the tutorial for the list sort method does not show allowed arguments: list.sort() Sort the items of the list in place. (see e.g. https://docs.python.org/3.4/tutorial/datastructures.html) Is there a reason why we do not show the arguments there? For simplicity? One should note that a web search for Python's list methods ranks that page pretty high. We could list the defaults, as in the built-in help: L.sort(cmp=None, key=None, reverse=False) And could link to https://docs.python.org/2/library/functions.html#sorted for an explanation. -- assignee: docs@python components: Documentation messages: 219085 nosy: docs@python, eric.araujo, ezio.melotti, georg.brandl, jgehrcke priority: normal severity: normal status: open title: list.sort() should show arguments in tutorial versions: Python 2.7, Python 3.4, Python 3.5 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue21575 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20584] On FreeBSD, signal.NSIG is smaller than biggest signal value
Jan-Philip Gehrcke added the comment: We should match the unit test with the documentation for signal.NSIG. Either the code or the docs or both need to change. Currently the docs say that signal.NSIG is One more than the number of the highest signal number. (https://docs.python.org/3.4/library/signal.html#signal.NSIG). In case of FreeBSD's _SIG_MAXSIG (128) the documentation is still wrong: the highest signal value MAX is 126 (see http://bugs.python.org/issue20584#msg210892). According to the docs, NSIG should then be 127. In signal_nsig_freebsd-2.patch the test `self.assertLess(max(signals), signal.NSIG)` tests for NSIG MAX instead of for NSIG = MAX+1. So, either - we count signals by ourselves and build our own value of NSIG, in which case we can guarantee that NSIG = MAX+1 or - we rely on the platform header files for extracting NSIG, but must note in the docs that NSIG is not always MAX+1. The current patch + a documentation change would implement the latter case. What is the exact meaning of _SIG_MAXSIG, where is that meaning defined? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20584 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20584] On FreeBSD, signal.NSIG is smaller than biggest signal value
Jan-Philip Gehrcke added the comment: If you are thinking TL;DR: This fails on FreeBSD: signal.signal(signal.SIGRTMAX, lambda *a: None) Traceback (most recent call last): File stdin, line 1, in module ValueError: signal number out of range Although of infrequent use, I doubt that this is expected or desired behavior. -- nosy: +haypo ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20584 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20709] os.utime(path_to_directory): wrong documentation for Windows.
New submission from Jan-Philip Gehrcke: The os.utime() docs for Python 2 (http://docs.python.org/2/library/os.html#os.utime) and 3 (http://docs.python.org/3/library/os.html#os.utime) both contain the sentence Whether a directory can be given for path depends on whether the operating system implements directories as files (for example, *Windows does not*) Windoes does not is wrong. CPython 2.7 on Windows 7: os.utime(., (100, 100)) os.stat(.).st_mtime 100.0 We should - either name a specific system for which this does not work - or remove this example. Windows XP and newer support this operation via SetFileTime (http://msdn.microsoft.com/en-us/library/windows/desktop/ms724933%28v=vs.85%29.aspx): Sets the date and time that the specified file or directory was created, last accessed, or last modified. I have grepped myself through a couple of CPython source trees and found posixmodule.c:if (!SetFileTime(hFile, NULL, atime, mtime) in 2.5, 2.6, 2.7. I guess the statement comes from 2.4 times, where `SetFileTime` only appears in `PC/bdist_wininst/extract.c`. So, do we just remove the hint or does someone have an example at hand for which combination of Python and platform this does not work for directories? Once we have a decision, I will be happy to provide the mini patch. -- assignee: docs@python components: Documentation messages: 211760 nosy: docs@python, jgehrcke priority: normal severity: normal status: open title: os.utime(path_to_directory): wrong documentation for Windows. versions: Python 2.7, Python 3.1, Python 3.2, Python 3.3, Python 3.4, Python 3.5 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20709 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10728] argparse.ArgumentParser.print_help uses sys.stdout
Jan-Philip Gehrcke added the comment: The version action currently writes to stderr. The _VersionAction(Action)'s __call__() method finishes off with parser.exit(message=formatter.format_help()) and parser.exit() by default writes to stderr. Here, Steven says Help is definitely intended to print to stdout. and I expected the same to happen regarding the version information. Why should it go to stderr? I see that this is a minor problem and only if someone here agrees I would open an issue for that. -- nosy: +jgehrcke ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10728 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20584] On FreeBSD, signal.NSIG is smaller than biggest signal value
New submission from Jan-Philip Gehrcke: On FreeBSD, signal.NSIG is smaller than what the documentation promises: One more than the number of the highest signal number. On Linux, the highest numerical signal value is smaller/equal signal.NSIG (expected behavior): import signal signals = [s for s in dir(signal) if s.startswith(SIG)] max([(getattr(signal, s), s) for s in signals]) (64, 'SIGRTMAX') signal.NSIG 65 On FreeBSD (since version 7, when SIGRTMIN/MAX have been introduced), Python's signal.NSIG is either 32 (if defined by the system, depending on __BSD_VISIBLE, see http://svnweb.freebsd.org/base/head/sys/sys/signal.h?revision=233519view=markup#l331) or 64 (if chosen by signalmodule.c as a fallback). In any case, on FreeBSD the numerical values of SIGRTMIN/MAX are 65 and 126 and therefore both greater than Python's signal.NSIG: http://svnweb.freebsd.org/base/head/sys/sys/signal.h?revision=233519view=markup#l117 Consequently, Python's signal module exposes a number NSIG which is not 'true'. Two disadvantages: - signal.NSIG is just not meaningful on FreeBSD. It is part of the signal module's public interface, and should do what its documentation says: One more than the number of the highest signal number. - this might lead to unexpected behavior when for instance calling signal.signal(signal.SIGRTMAX, signal.SIG_DFL). This works on Linux, but fails with a ValueError on FreeBSD: raised directly by signalmodule.c, because sig_num = NSIG, i.e. sig_num seemingly is an invalid signal number, although it is not (https://github.com/python/cpython/blob/3.3/Modules/signalmodule.c#L323). This is the reason why I became aware of this topic. I see three arguments here: - if the system does not provide NSIG via signal.h and Python's signalvalue makes the wrong guess (i.e. fallback to 64), then signalmodule.c would be to blame. - if the system provides NSIG via signal.h and this is not the true maximum, then one could say that Python is not to blame. - on the other hand, signalmodule.c is aware of all signals that it actively checked for and therefore could derive One more than the number of the highest signal number on its own via something in the lines of max(signal_values)+1. Regarding the latter point: if Python misses to check for a valid signal on a certain platform, then this actively derived NSIG value would not be entirely correct, either, seen from the platform's perspective. But the signal module would then at least be consistent with itself. In case of FreeBSD, I am actually not sure if signal.NSIG *is* provided by the system or determined by the fallback method in signalmodule.c (I can't get my hands on a FreeBSD machine at the moment). What do you think? Btw, parts of this have already been mentioned here: http://bugs.python.org/issue12060 -- components: Library (Lib) messages: 210854 nosy: jgehrcke, neologix, sdaoden priority: normal severity: normal status: open title: On FreeBSD, signal.NSIG is smaller than biggest signal value type: behavior versions: Python 2.7, Python 3.1, Python 3.2, Python 3.3 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20584 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20584] On FreeBSD, signal.NSIG is smaller than biggest signal value
Jan-Philip Gehrcke added the comment: As a follow-up, relevant output from FreeBSD 9: $ python Python 2.7.5 (default, Dec 20 2013, 21:12:37) [GCC 4.2.1 20070831 patched [FreeBSD]] on freebsd9 Type help, copyright, credits or license for more information. import signal signals = [s for s in dir(signal) if s.startswith(SIG)] max((getattr(signal, s), s) for s in signals) (126, 'SIGRTMAX') signal.NSIG 32 signal.signal(signal.SIGRTMAX, lambda *a: None) Traceback (most recent call last): File stdin, line 1, in module ValueError: signal number out of range Hence, it's not the fallback to 64, it's FreeBSD's signal.h telling that NSIG is 32. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20584 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18454] distutils crashes when uploading to PyPI having only the username (no pw) defined
New submission from Jan-Philip Gehrcke: When updating an existing project on PyPI via distutils using the upload command, I observe erroneous behavior regarding the credentials when I do not want to store my password in clear text in the pypirc file: (1) When running python setup.py sdist upload without having the pypirc file in place, I get the error Upload failed (401): You must be identified to edit package information (2) When running the same command as above with the pypirc file in place but without having the 'password' option in the 'pypi' section defined, I get a TypeError exception. In both cases and at least in the second case I expect to be prompted for my credentials. This is what the 2.7.5 docs are saying about the contents of the pypirc file (http://docs.python.org/2.7/distutils/packageindex.html#the-pypirc-file): password, that will be used to authenticate. If omitted the user will be prompt to type it when needed. I have seen http://bugs.python.org/issue5187 saying distutils is feature frozen but the current situation is buggy. Either there is a documentation mistake (it clearly says that the user is prompted for the password) or there is an error in the code (*, see below), or both. * Regarding the TypeError mentioned above: In distutils/command/upload.py, finalize_options(), the configuration dictionary is retrieved from _read_pypirc() (distutils/config.py). There, the value for the password key in the config dictionary is set to None if not defined in the pypirc configuration file. The password value is not modified/updated in finalize_options() if self.distribution.password is not set. I think the latter is only set when the 'register' command is used. Hence, when the user for example simply runs python setup.py sdist upload and did not set the password in the pypirc file, the password value stays None. Nevertheless, in distutils/command/upload.py, upload_file(), password is treated as string: auth = Basic + standard_b64encode(self.username + : + self.password) This obviously leads to TypeError: cannot concatenate 'str' and 'NoneType' objects I would be happy to work on a patch if we agree on what the proper behavior should be. Me, as a user of PyPI, would vote for being prompted in both cases outlined above. I do not like to store my PyPI password in clear text in the file system. And after specifying how distutils should behave in case (2) I think we all agree that distutils/tests/test_upload.py should provide a test for this case. An example configuration file with username but without password is already defined via PYPIRC_NOPASSWORD. Currently, this config is only tested within an edge-case in test_saved_password() with dist.password = 'xxx', simulating the simultaneous usage of 'register' with 'upload' if I understood correctly. Register probably is used less frequently than upload alone. Looking forward to your input, Jan-Philip -- assignee: eric.araujo components: Distutils messages: 193062 nosy: eric.araujo, jason.coombs, jgehrcke, loewis, tarek, techtonik priority: normal severity: normal status: open title: distutils crashes when uploading to PyPI having only the username (no pw) defined type: crash versions: Python 2.7 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18454 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6634] sys.exit() called from threads other than the main one: undocumented behaviour
Jan-Philip Gehrcke jgehr...@gmail.com added the comment: Sorry for the delay. Before suggesting a doc change to correct/complete the description of the *current* situation, we actually should consider changing this situation. I think this is reasonable and I feel encouraged by Gabriel Genellina: I see no reason for sys.exit(msg) NOT to write to stderr inside a child thread. This patch enables printing to stderr from child threads and clones the behavior of `sys.exit(arg)` called from the main thread: # PATCH BEGIN --- C:/Python27/Lib/threading.pySat Apr 10 18:55:48 2010 +++ C:/python_sys_exit_issue/threading.pySat Sep 25 14:50:24 2010 @@ -531,6 +531,15 @@ except SystemExit: if __debug__: self._note(%s.__bootstrap(): raised SystemExit, self) +# Now get and handle the exit code, given by the user via +# the second expression after `raise` or via the argument of +# sys.exit(). +code = self.__exc_info()[1].code +# Ignore None and integer exit code. Print any other object +# to stderr as it is the behavior of sys.exit(arg) called +# from the main thread. +if code is not None and not isinstance(code, int): +_sys.stderr.write(%s\n % code) except: if __debug__: self._note(%s.__bootstrap(): unhandled exception, self) # PATCH END A script with different testcases including output is attached. What do you think? All the best, Jan-Philip Gehrcke -- Added file: http://bugs.python.org/file19006/thread_sys_exit_test.py ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6634 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6634] sys.exit() called from threads other than the main one: undocumented behaviour
New submission from Jan-Philip Gehrcke jgehr...@gmail.com:
Hey there,
hopefully I fill out this form in an adequate way!
I ran into some problems while using sys.exit('msg') together with
threads, which could have been avoided with slightly more information in
the docs here: http://docs.python.org/library/sys.html#sys.exit
Maybe the following two statements should not stay as they are:
(1) Exit from Python.
---
This is not true when called from a thread other than the main one. We
could add a hint, saying that sys.exit() then actually behaves like
thread.exit(), which causes only the calling thread to exit, but not the
main program.
2) [...] and any other object is printed to sys.stderr
This is also not true when called from a thread other than the main one.
Calling sys.exit('msg') then doesn't print anything to stderr. That was
annoying in my case and required debugging a bug that would have
discovered itself via stderr, if the message would have been printed..
:-) After some research, I think this behaviour is described in the
documentation for thread.exit(): [...] this will cause the thread to
exit *silently*.
Okay, now that I am aware of this behaviour, I won't run into these
problems again. But the next one?
I think (1) is clearly a documentation thing. Regarding (2): first of
all, the documentation should say that the message is suppressed in
special cases (child threads). But: what argues against printing to
stderr here? I don't get the point and only see a lost feature,
affording a quick way to kill a thread while dropping an error message.
Was this kicked out intentionally? Maybe someone could help me with a
good argument here :-)
Thank you for your work,
Jan-Philip Gehrcke
--
assignee: georg.brandl
components: Documentation
messages: 91237
nosy: georg.brandl, jgehrcke
severity: normal
status: open
title: sys.exit() called from threads other than the main one: undocumented
behaviour
versions: Python 2.4, Python 2.5, Python 2.6, Python 2.7, Python 3.0, Python 3.1
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue6634
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
