[issue16487] Allow ssl certificates to be specified from memory rather than files.

Kristján Valur Jónsson added the comment: I gave up contributing a long time ago now because it was too emotionally exhausting to me. This issue was one that helped tip the scales. I hope things have become easier now because good projects like Python need the enthusiasm and spirit of

[issue16487] Allow ssl certificates to be specified from memory rather than files.

Jan-Philip Gehrcke added the comment: I am not too attached to "my" patch, but because I love Python I really would like us to land on a solution. > However I want all changes and new additions to the SSL module to follow PEP > 543 so I can provide a PEP 543-compatible interface in the near

[issue16487] Allow ssl certificates to be specified from memory rather than files.

Lord Anton Hvornum added the comment: I agree with Jan-Philip Gehrcke, would have been nice to have had this. Pretty pissed reading through this ancient issue. In regards to how people treat other volunteers: We're all working for free, and I think expectations from people with

[issue16487] Allow ssl certificates to be specified from memory rather than files.

Jan-Philip Gehrcke added the comment: Thanks for the discussion. Since I tried to join the efforts here in 2016 two years ago I was (and still am) enthusiastic, and willing to invest quite a bit of energy. Still, we have missed the 3.6 and 3.7 releases to change something about the fact

[issue16487] Allow ssl certificates to be specified from memory rather than files.

Christian Heimes added the comment: Senthil, I'm not a fan of PR 2449 because it provides yet another way to load certificates and keys from memory. It's a clever idea to use MemoryBIO here. But the approach is *not* compatible with PEP 543. The PEP requires an API that

[issue16487] Allow ssl certificates to be specified from memory rather than files.

Senthil Kumaran added the comment: Hi Cristian, > I don't want to have three ways to load certificates, especially when it > involves more C code. I think this (more C code) is the primary and the *only* negative point against the current patch. And that seems

[issue16487] Allow ssl certificates to be specified from memory rather than files.

Antoine Pitrou added the comment: Le 01/12/2017 à 10:47, Christian Heimes a écrit : > > It doesn't matter for certificates. I guess you also want to load the key > from a memory buffer, don't you? That's going to be less secure. Yes, that's what I meant. --

[issue16487] Allow ssl certificates to be specified from memory rather than files.

Christian Heimes added the comment: It doesn't matter for certificates. I guess you also want to load the key from a memory buffer, don't you? That's going to be less secure. -- ___ Python tracker

[issue16487] Allow ssl certificates to be specified from memory rather than files.

Antoine Pitrou added the comment: Is it possible to have a single C function that loads the certificates from memory, or would that make the file-loading case less secure? Le 01/12/2017 à 10:25, Christian Heimes a écrit : > > Christian Heimes added the

[issue16487] Allow ssl certificates to be specified from memory rather than files.

Christian Heimes added the comment: Correct, PEP 543 won't land in Python 3.7 because neither me nor Cory have the resources to land it. However I want all changes and new additions to the SSL module to follow PEP 543 so I can provide a PEP 543-compatible interface in the

[issue16487] Allow ssl certificates to be specified from memory rather than files.

Nathaniel Smith added the comment: My impression was that progress on PEP 543 is temporarily stalled and not going to be finished for 3.7. Is that wrong? There's going to need to be a significant amount of shimming to implement the PEP 543 interfaces on top of ssl no matter

[issue16487] Allow ssl certificates to be specified from memory rather than files.

Martin Richard added the comment: FWIW, PyOpenSSL allows to load certificates and keys from a memory buffer and much more. It's also fairly easy to switch from ssl to PyOpenSSL. It's probably a viable alternative in many cases. --

[issue16487] Allow ssl certificates to be specified from memory rather than files.

Kristján Valur Jónsson added the comment: OP here, lurking. The need to load server certificates from memory is quite real. Some seven years ago I wrote custom code to handle that for CCPs python branch, and contributed patches to that effect. It's always dismaying to see

[issue16487] Allow ssl certificates to be specified from memory rather than files.

Christian Heimes added the comment: I'm working on a PEP that builds on top of PEP 543 and addresses some issues like IDNA #28414, OpenSSL/LibreSSL compatibility, hostname verification, verification chain, and TLS 1.3. As part of the PEP implementation, I'll add a

[issue16487] Allow ssl certificates to be specified from memory rather than files.

Antoine Pitrou added the comment: Christian, what is the status of PEP 543? Is someone actively working on it? Should we hope for a resolution (i.e. an implementation accepted in the stdlib) in the near future? I'm worried about such purity arguments. It reminds me of when

[issue16487] Allow ssl certificates to be specified from memory rather than files.

Christian Heimes added the comment: I'm -1 on the patch. Any new feature and API shall follow https://www.python.org/dev/peps/pep-0543/. I have some code that adds a proper Certificate class that wraps OpenSSL's X509* type. --

[issue16487] Allow ssl certificates to be specified from memory rather than files.

Senthil Kumaran added the comment: Hello JP, I had been little inactive for a while (>2 years). However, I think, I can take now some load in CPython world and I will take this ticket/feature forward this Sunday (1-Dec-2017). If Christian (or currently active committers)

[issue16487] Allow ssl certificates to be specified from memory rather than files.

Jan-Philip Gehrcke added the comment: Hello! I would very much appreciate if we can find a way for us to get another review for the last patch. I did most of the work in August 2016 and got a review from Senthil and Christian which I processed. When I got back to the

[issue16487] Allow ssl certificates to be specified from memory rather than files.

Change by STINNER Victor : -- components: -Library (Lib) nosy: -haypo ___ Python tracker ___

[issue16487] Allow ssl certificates to be specified from memory rather than files.

Jan-Philip Gehrcke added the comment: Hey Antoine, Christian, Senthil! I have invested quite a bit more time to double-check my responses to the questions asked so far, clarified where appropriate, and updated the pull request on GitHub after manually resolving the merge

[issue16487] Allow ssl certificates to be specified from memory rather than files.

Changes by Nathaniel Smith : -- nosy: +njs ___ Python tracker ___ ___ Python-bugs-list

[issue16487] Allow ssl certificates to be specified from memory rather than files.

Jan-Philip Gehrcke added the comment: Hey Senthil and Christian! > Could you convert your latest patch into PR against > https://github.com/python/cpython That was fun. There we go: https://github.com/python/cpython/pull/2449 I hope I was not too late with that for the 3.7 development.

[issue16487] Allow ssl certificates to be specified from memory rather than files.

Changes by Jan-Philip Gehrcke : -- pull_requests: +2506 ___ Python tracker ___ ___

[issue16487] Allow ssl certificates to be specified from memory rather than files.

Changes by Senthil Kumaran : -- versions: -Python 3.6 ___ Python tracker ___ ___

[issue16487] Allow ssl certificates to be specified from memory rather than files.

Senthil Kumaran added the comment: Hi Jan-Philip, we might be close on this one. Could you convert your latest patch into PR against (https://github.com/python/cpython). Is will help to record you as the author as we can discuss the patch in python sprints and get this in. :-) Thanks!

[issue16487] Allow ssl certificates to be specified from memory rather than files.

Jan-Philip Gehrcke added the comment: Christian, Senthil, would appreciate if I got another round of feedback (in the review thread) :-) -- nosy: +orsenthil ___ Python tracker

[issue16487] Allow ssl certificates to be specified from memory rather than files.

Jan-Philip Gehrcke added the comment: Thanks Christian, much appreciated. Just responded to your review. -- ___ Python tracker ___

[issue16487] Allow ssl certificates to be specified from memory rather than files.

Changes by Christian Heimes : -- assignee: -> christian.heimes components: +SSL ___ Python tracker ___

[issue16487] Allow ssl certificates to be specified from memory rather than files.

Christian Heimes added the comment: Thanks for your patch. I have left some comments (sorry for the delay). -- versions: +Python 3.7 -Python 3.4 ___ Python tracker

[issue16487] Allow ssl certificates to be specified from memory rather than files.

Jan-Philip Gehrcke added the comment: Hello! Like everybody in this thread I would love to see this land and have prepared a new patch, hoping that we can process this still for 3.6. Antoine summarized the core task here very well: > Let's stay focused on what is > necessary to solve this

[issue16487] Allow ssl certificates to be specified from memory rather than files.

Changes by Christian Heimes : -- assignee: christian.heimes -> ___ Python tracker ___ ___

[issue16487] Allow ssl certificates to be specified from memory rather than files.

Martin Richard added the comment: I'm not sure I know how to do this correctly: I lack of experience both with openssl C API and writing python modules in C. It may be more flexible, but unless the key is protected/crypted somehow, one would need a string or bytes buffer to hold the key when

[issue16487] Allow ssl certificates to be specified from memory rather than files.

Martin Richard added the comment: Hi, I would like to update this patch so it can finally land in cpython, hopefully 3.6. tl;dr of the thread: In a nutshell, the latest patch from Kristján Valur Jónsson updates SSLContext.load_cert_chain(certfile, keyfile=None, password=None) and

[issue16487] Allow ssl certificates to be specified from memory rather than files.

Antoine Pitrou added the comment: I thing adding keydata and certdata makes things more complicated, on the contrary. You start having an API with many optional arguments but some of them are exclusive with each other (because you can only specify a single key and cert chain). The cafile,

[issue16487] Allow ssl certificates to be specified from memory rather than files.

Martin Richard added the comment: You are right. And if certfile and keyfile (args of load_cert_chain()) accept file-like objects, we agree that cafile (load_verify_location()) should accept them too? -- ___ Python tracker rep...@bugs.python.org

[issue16487] Allow ssl certificates to be specified from memory rather than files.

Antoine Pitrou added the comment: Le 09/07/2015 15:40, Martin Richard a écrit : And if certfile and keyfile (args of load_cert_chain()) accept file-like objects, we agree that cafile (load_verify_location()) should accept them too? It could, but that's a separate issue. Let's stay focused on

[issue16487] Allow ssl certificates to be specified from memory rather than files.

STINNER Victor added the comment: Sorry, I didn't take time to read the whole discussion. For me, it's a good idea to accept a filename or a file object in the same parameter. Having two exclusive parameters for the same thing (ex: CA) doesn't smell like a great API. --

[issue16487] Allow ssl certificates to be specified from memory rather than files.

Christian Heimes added the comment: I'd rather introduce new types and have the function accept either a string (for path to fiel) or a X509 object and a PKey object. It's more flexible and secure. With a private key type we can properly support crypto ENGINEs and wipe memory when the object

[issue16487] Allow ssl certificates to be specified from memory rather than files.

Changes by Christian Heimes li...@cheimes.de: -- assignee: - christian.heimes ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16487 ___ ___

[issue16487] Allow ssl certificates to be specified from memory rather than files.

Kristján Valur Jónsson added the comment: Update the latest patch to the current state of python. -- Added file: http://bugs.python.org/file32611/ssl2.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16487

[issue16487] Allow ssl certificates to be specified from memory rather than files.

Kristján Valur Jónsson added the comment: The patch should be valid, please try it out. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16487 ___

[issue16487] Allow ssl certificates to be specified from memory rather than files.

Christian Heimes added the comment: Your patch looks like Benjamin's fix for issue #17828 and not like a SSL improvement. :) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16487 ___

[issue16487] Allow ssl certificates to be specified from memory rather than files.

Changes by Kristján Valur Jónsson krist...@ccpgames.com: Removed file: http://bugs.python.org/file32611/ssl2.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16487 ___

[issue16487] Allow ssl certificates to be specified from memory rather than files.

Kristján Valur Jónsson added the comment: Haha, indeed. what nonsense. Here is the correct one. -- Added file: http://bugs.python.org/file32612/ssl2.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16487

[issue16487] Allow ssl certificates to be specified from memory rather than files.

Changes by STINNER Victor victor.stin...@gmail.com: -- nosy: +haypo ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16487 ___ ___ Python-bugs-list

[issue16487] Allow ssl certificates to be specified from memory rather than files.

Christian Heimes added the comment: What's the status of this patch? I need a way to load CA certs from memory for another patch of mine. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16487

[issue16487] Allow ssl certificates to be specified from memory rather than files.

Changes by Donald Stufft donald.stu...@gmail.com: -- nosy: +dstufft ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16487 ___ ___ Python-bugs-list

[issue16487] Allow ssl certificates to be specified from memory rather than files.

Kristján Valur Jónsson added the comment: Thanks for your comments Christian. You don't check ERR_GET_LIB() in some places. Do you have a particular place in mind? About DER. As I understand, currently _ssl only supports PEM. If that is the case, then supporting DER should, IMHO, be a

[issue16487] Allow ssl certificates to be specified from memory rather than files.

Christian Heimes added the comment: I found two places: if (ERR_GET_REASON(err) == X509_R_CERT_ALREADY_IN_HASH_TABLE) { if (ERR_GET_REASON(err) == PEM_R_BAD_BASE64_DECODE) AFAIK the _ssl module only supports PEM certs for loading. On the other hands cert data can only be retrieved as dict

[issue16487] Allow ssl certificates to be specified from memory rather than files.

Antoine Pitrou added the comment: It feels a bit strange to treat PEM certs as binary data, especially since the SSL module treats PEM as ASCII unicode. For example DER_cert_to_PEM_cert() accepts bytes and returns str, PEM_cert_to_DER_cert() converts str to bytes. I agree that PEM is

[issue16487] Allow ssl certificates to be specified from memory rather than files.

Kristján Valur Jónsson added the comment: Ok, thanks. The consistency argument is strong, also Antoine's suggestion to use the return type of read() as a discriminant. also please excuse me because I am not a habitual user of Python 3 and haven't become used to the str/binary dichotomy yet.

[issue16487] Allow ssl certificates to be specified from memory rather than files.

Christian Heimes added the comment: EVE Online is still using Python 2.7? You gotta hurry up or Guido will beat you with Dropbox's 3.x port. :) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16487

[issue16487] Allow ssl certificates to be specified from memory rather than files.

Kristján Valur Jónsson added the comment: 2.7 is the pinnacle of pythonic achievement. Particularly our branch of it :) One day we'll move, I'm sure, when there is an opportune moment. For example, if we were to start supporting a new game, a new platform. But for now, if it ain't broke, we

[issue16487] Allow ssl certificates to be specified from memory rather than files.

Kristján Valur Jónsson added the comment: Okay, I have updated the patch as suggested. string mode means PEM encoding, binary mode DER encoding. Note that DER encoding is more limited, there is no way to concatentate private keys and certificates into one file (the PEM decoder searches the file

[issue16487] Allow ssl certificates to be specified from memory rather than files.

Kristján Valur Jónsson added the comment: Here is an updated patch. We now support file-like objects. New helper functions try to turn file arguments into either Py_buffer objects containing the read data, or PyBytesObject argument with the file system encoding of the path. A file-like object

[issue16487] Allow ssl certificates to be specified from memory rather than files.

Christian Heimes added the comment: Thx Kristján! My patch maps PyUnicode to PEM encoded cert data and objects with Py_Buffer support to DER encoded cert data. Perhaps you like to you the same concept in your patch to support TextIO and BytesIO read() methods. Feel free to reuse as much of

[issue16487] Allow ssl certificates to be specified from memory rather than files.

Christian Heimes added the comment: PS: I like to have DER cert support for #17134. I'd rather not convert DER to PEM. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16487 ___

[issue16487] Allow ssl certificates to be specified from memory rather than files.

Kristján Valur Jónsson added the comment: ok, I concede that a file-like object makes sense from a polymorphism point of view. It makes no sense from a streaming point of view. A caller can then wrap their data into a StringIO instance. I'll rework the patch in the manner you suggest.

[issue16487] Allow ssl certificates to be specified from memory rather than files.

Antoine Pitrou added the comment: Agreed, a file-like object is the way to go. I don't think you need to write the logic in C, by the way. You can write a high-level function and defer to a low-level C func for the basic API wrapping. -- stage: patch review - needs patch

[issue16487] Allow ssl certificates to be specified from memory rather than files.

Changes by Christian Heimes li...@cheimes.de: -- nosy: +christian.heimes ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16487 ___ ___

[issue16487] Allow ssl certificates to be specified from memory rather than files.

Kristján Valur Jónsson added the comment: Antoine, are you suggesting that we remove the current c-level capability to use file system files (using open()) and just go with raw bytes data at the C api level, and then do the 'filename or filelike object' in Python land? --

[issue16487] Allow ssl certificates to be specified from memory rather than files.

Antoine Pitrou added the comment: Antoine, are you suggesting that we remove the current c-level capability to use file system files (using open()) and just go with raw bytes data at the C api level, and then do the 'filename or filelike object' in Python land? Yes, I think that's

[issue16487] Allow ssl certificates to be specified from memory rather than files.

Christian Heimes added the comment: I didn't know about this issue and have worked on a similar feature in #18138. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16487 ___

[issue16487] Allow ssl certificates to be specified from memory rather than files.

Antoine Pitrou added the comment: Ha, funny. Now it's time to reconciliate your respective patches :) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16487 ___

[issue16487] Allow ssl certificates to be specified from memory rather than files.

Brandon Craig Rhodes added the comment: Kristján, you are certainly correct that a single-argument that can be either a filename or a cert is inappropriate; we should not be peeking inside of strings to guess what they contain. And I think you also have a good point about Pythonic-ness when

[issue16487] Allow ssl certificates to be specified from memory rather than files.

Kristján Valur Jónsson added the comment: Hi there. Thanks for your comments. This is the kind of discussion I was hoping to have about my draft patch. I too have reservations about adding arguments. In the version of this that we have in house, we actually don't use a certdata argument,

[issue16487] Allow ssl certificates to be specified from memory rather than files.

Brandon Craig Rhodes added the comment: Kristján, your patch is a wonderful idea—I am about to commit production code that will have to create tens of thousands of temporary files during operation, one file each time SSL is started up on a socket, which could be avoided if something like this

[issue16487] Allow ssl certificates to be specified from memory rather than files.

Jesús Cea Avión added the comment: Kristján, are you pursuing this yet?. Can we move on? -- nosy: +jcea title: Allow ssl certificates to be speficfied from memory rather than files. - Allow ssl certificates to be specified from memory rather than files.

[issue16487] Allow ssl certificates to be specified from memory rather than files.

Kristján Valur Jónsson added the comment: I wasn't pushing it anymore because there seemed to be no interest and Antoine attitude made me lose interest in contributing this. No one liked to comment on my approach except Antoine and I had grown tired of his persistent negativity and aloof

[issue16487] Allow ssl certificates to be specified from memory rather than files.

Kristján Valur Jónsson added the comment: Manually edited the .patch file The build changes are gone but should probably go in separately in some form later. -- Added file: http://bugs.python.org/file28866/sslpatch2.patch ___ Python tracker