Re: credentials in memory
On Sun, 20 Nov 2022, Howard Chu via curl-library wrote: Here are some possible mitigations we could implement in curl: Store sensitive keys in a dedicated mmap'd region, mprotect the region to remove read access whenever the key isn't actively being used. As we want to support lots of systems without mmap, that would just be one solution to how to protect credentials. I think that's the smaller problem. The bigger work I think is to make sure that we properly limit the scope/lifetimes so that we can encrypt/protect/clear credentials immediately after use and only have them readable in memory as short a moment in time as possible. But: I don't see anyone stepping up to the challenge of actually making this happen so this is all hypothetical for now. -- / daniel.haxx.se | Commercial curl support up to 24x7 is available! | Private help, bug fixes, support, ports, new features | https://curl.se/support.html -- Unsubscribe: https://lists.haxx.se/listinfo/curl-library Etiquette: https://curl.se/mail/etiquette.html
Re: credentials in memory
Dan Fandrich via curl-library wrote: > However, if the attacker somehow only has access to the memory and not the > rest > of the process' assets (case 2.), then use of a hardware security device can > protect the > keys from directly being stolen, But, there will be some times that curl needs > the raw secrets in order to pass them to other dependencies or write them into > buffers, and when they're in memory, the attacker can still get them. And if > they're in memory, then can end up on disk in a core or hibernation file where > the attacker can read them. > > The 3. case is the most interesting one that this proposal could help > mitigate. > A Heartbleed-style bug that gives arbitrary memory to an attacker could return > memory containing a secret. If secrets are not stored in plaintext in RAM, > then > it becomes much harder to obtain those secrets. But it's still not perfect. > > Here are some possible mitigations we could implement in curl: Store sensitive keys in a dedicated mmap'd region, mprotect the region to remove read access whenever the key isn't actively being used. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/ -- Unsubscribe: https://lists.haxx.se/listinfo/curl-library Etiquette: https://curl.se/mail/etiquette.html
Re: credentials in memory
On Fri, Sep 30, 2022 at 09:43:39AM +0200, Daniel Stenberg via curl-library wrote: > libcurl hold credentials (passwords for servers and proxies) in memory in > clear text, potentially for a long time. If something goes wrong and that > memory is accessed by an external party, things would be bad. > > Is it worth doing something about? It's important to define what exactly the threat is we want to protect against. These are the main ones I can think of: 1. local attackers with access to the process & its memory 2. local attackers with access to memory, core dumps, swap space or hibernation files 3. remote attackers tricking curl into returning secrets from memory (via a Heartbleed-style attack or by returning uninitialized stack or heap space, for example) In the 1. case, if an attacker can access the process' resources and memory, there's not much left to protect. If the attacker has the same rights as the process, then everything that curl can access, including passwords, the attacker can, too. Even if secrets are encrypted, the attacker has access to the same decryption keys as the real process so he can just decrypt them. I don't see any way to protect against this in the general case. However, if the attacker somehow only has access to the memory and not the rest of the process' assets (case 2.), then use of a hardware security device can protect the keys from directly being stolen, But, there will be some times that curl needs the raw secrets in order to pass them to other dependencies or write them into buffers, and when they're in memory, the attacker can still get them. And if they're in memory, then can end up on disk in a core or hibernation file where the attacker can read them. The 3. case is the most interesting one that this proposal could help mitigate. A Heartbleed-style bug that gives arbitrary memory to an attacker could return memory containing a secret. If secrets are not stored in plaintext in RAM, then it becomes much harder to obtain those secrets. But it's still not perfect. Here are some possible mitigations we could implement in curl: 1. Clearing secrets with memset once their need is over. If the secrets are out of RAM, they won't end up in core files, swap files or process debuggers. However, curl needs some secrets for the entire lifetime of the process (e.g. proxy credentials) so it can't clear all of them all the time. Also, curl needs them in RAM for a short time to use them before they're cleared, so an attacker could just grab them at that time. 2. Creating a random session key at startup and encrypting secrets as soon as possible using that key. This suffers from the same problem mentioned before, in that the secrets have to be decrypted to use them, even if only for a short time. Also, encrypting many secrets using the same key makes it easier for an attacker to guess that key. So, a Heartbleed-style attack that is able to obtain many encrypted secrets could still obtain the decryption key. 3. Creating a new random key for each secret. This avoids the key reuse problem in mitigation 2. by creating a new, random key for each and every new secret that needs to be protected. The overhead is greater but it's more secure. However, those random keys are still stored in RAM so a Heartbleed-style attack that obtains the encrypted secrets might at some point also obtain the decryption keys as well, exposing those secrets. 4. Storing the random keys from 3. in a hardware security module. This avoids the possibility that the attacker might obtain the decryption keys by storing them in an HSM. Unfortunately, curl still needs to decrypt them eventually and they'll be in RAM for at a least a short time while that happens, during which time they're still subject to being obtained by an attacker. So, no matter how much we are able to protect those keys, they'll still be vulnerable to an attacker. All we can do is reduce the time that they're vulnerable, which just increases the time an attacker needs to get them, not whether or not they *can* get them. Still, reducing the time could turn a practical, but slow, attack into an impractical one. And, unless applications do the same that curl does, these mitigations are of limited use since the application itself becomes the weak link in the chain. But, if libcurl does something then we can at least point our fingers elsewhere if the application doesn't take the same amount of care. We also need to consider the costs of implementing a solution: implementation costs, maintenance costs, increased difficulty in debugging, performance degradation, portability, etc. This isn't my area of expertise, and I'm sure there have been many PhDs theses written on exactly what I'm speculating about above. It's a complicated topic, and we should have a clear idea of the benefits of retrofitting some kind of protection into curl before doing it. Dan -- Unsubscribe: https://lists.haxx.se/listinfo/curl-library Etiquette:
Re: credentials in memory
Daniel Stenberg via curl-library wrote: > Hi, > > I bring back an old discussion as I think it might be worth having it again. > > libcurl hold credentials (passwords for servers and proxies) in memory in > clear text, potentially for a long time. If something goes wrong and that > memory is > accessed by an external party, things would be bad. > > We work really hard on preventing and avoiding the "something goes wrong" > part, but every once in a while we come back to the fact that we still keep > credentials in the clear in memory. This is also something that our good > friends at Trail of Bits have commented on during the ongoing security audit > they are > performing on curl. > > Is it worth doing something about? > > If it is, what's a reasonable mitigation? We need to be able to use them in > the clear, so whatever we do we cannot just hash them. We would need to have > a way > to store them encrypted and decrypt them on demand when they are needed and > then only use them "in the clear" for as a short period as possible. > > This would mostly remove them from being readable as-is, like if there's a > stack reveal or heap leak as a result of a vulnerability perhaps, the risk for > credential leaks would be reduced. > > The "encryption" then wouldn't have to be complicated and could use a > randomly generated "key", probably created when the handle is created. > > Of course, since the passwords are passed in to libcurl from applications, > this dance is less effective if they then keep the credentials around in the > clear in > memory anyway, but I think maybe they typically keep them around for a > shorter time in general. > > Thoughts? Pointless? Improvements? Long ago when I was developing PC/Mac Kerberos client libraries, my solution was to encrypt secrets in an in-memory credential cache using the contents of the stack as the encryption key. As such, the key could only be constructed by calling the same sequence of functions with the same parameters as the original call, and the key would generally be different for every app/ program invocation. (My in-memory credential cache is still part of MIT Kerberos5 libraries today, but I guess not in the same form as 25 years ago...) Today I would try something similar, with an explicitly managed mmap'd buffer region for the credential cache. On some platforms you can mark a mapping as non-swappable, so there's no danger of it persisting to disk. Also you can mprotect to remove its read access until you actually need to retrieve a value. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/ -- Unsubscribe: https://lists.haxx.se/listinfo/curl-library Etiquette: https://curl.se/mail/etiquette.html
Re: credentials in memory
On Fri, Sep 30, 2022 at 01:49:40PM +0200, Stefan Eissing via curl-library wrote: > > > > Am 30.09.2022 um 13:41 schrieb Daniel Stenberg : > > > > On Fri, 30 Sep 2022, Stefan Eissing wrote: > > > >> I know of threee patterns to solve this problem (and increase usability > >> as a side effect): > > > > Those methods transfer the data to another process, and that is certainly > > even more safe since then the sensitive data is not even present in the > > heap of the first process. > > > > But: introducing a second process or a daemon or something for this > > purpose, while safer, would be a significant new factor and complication > > that would basically prevent a huge portion of our users from using it. > > > > I think a simpler first step could be to just "scramble" the data while > > "long-term stored" in memory. > > It's certainly simpler and it makes leaking the "interesting" parts of > memory easier. But for cases where someone gets access to all the memory or > a core dump, it will not make things more secure, just obscure. > > One thing I have seen for memory scanning protection is to put protected > pages around the location where sensitive data is. So someone scanning > memory from above or below will run into a segfault. Hi, Plus (on platforms supporting this) lock the pages into memory and exclude them from dump. Further question as I did exactly this for libcurl data with custom memory management; Does libcurl use the custom memory management also for the passwords? - Ville -- "I want to move to Theory, everything works in theory." [l...@iki.fi][+358-50-386 6269] -- Unsubscribe: https://lists.haxx.se/listinfo/curl-library Etiquette: https://curl.se/mail/etiquette.html
Re: credentials in memory
> Am 30.09.2022 um 13:49 schrieb Stefan Eissing via curl-library > : > > > >> Am 30.09.2022 um 13:41 schrieb Daniel Stenberg : >> >> On Fri, 30 Sep 2022, Stefan Eissing wrote: >> >>> I know of threee patterns to solve this problem (and increase usability as >>> a side effect): >> >> Those methods transfer the data to another process, and that is certainly >> even more safe since then the sensitive data is not even present in the heap >> of the first process. >> >> But: introducing a second process or a daemon or something for this purpose, >> while safer, would be a significant new factor and complication that would >> basically prevent a huge portion of our users from using it. >> >> I think a simpler first step could be to just "scramble" the data while >> "long-term stored" in memory. > > It's certainly simpler and it makes leaking the "interesting" parts of memory > easier. But for cases where someone gets access to all the memory or a core > dump, it will not make things more secure, just obscure. > makes it more *difficult*. > One thing I have seen for memory scanning protection is to put protected > pages around the location where sensitive data is. So someone scanning memory > from above or below will run into a segfault. > > -Stefan > -- > Unsubscribe: https://lists.haxx.se/listinfo/curl-library > Etiquette: https://curl.se/mail/etiquette.html -- Unsubscribe: https://lists.haxx.se/listinfo/curl-library Etiquette: https://curl.se/mail/etiquette.html
Re: credentials in memory
> Am 30.09.2022 um 13:41 schrieb Daniel Stenberg : > > On Fri, 30 Sep 2022, Stefan Eissing wrote: > >> I know of threee patterns to solve this problem (and increase usability as a >> side effect): > > Those methods transfer the data to another process, and that is certainly > even more safe since then the sensitive data is not even present in the heap > of the first process. > > But: introducing a second process or a daemon or something for this purpose, > while safer, would be a significant new factor and complication that would > basically prevent a huge portion of our users from using it. > > I think a simpler first step could be to just "scramble" the data while > "long-term stored" in memory. It's certainly simpler and it makes leaking the "interesting" parts of memory easier. But for cases where someone gets access to all the memory or a core dump, it will not make things more secure, just obscure. One thing I have seen for memory scanning protection is to put protected pages around the location where sensitive data is. So someone scanning memory from above or below will run into a segfault. -Stefan -- Unsubscribe: https://lists.haxx.se/listinfo/curl-library Etiquette: https://curl.se/mail/etiquette.html
Re: credentials in memory
On Fri, 30 Sep 2022, Stefan Eissing wrote: I know of threee patterns to solve this problem (and increase usability as a side effect): Those methods transfer the data to another process, and that is certainly even more safe since then the sensitive data is not even present in the heap of the first process. But: introducing a second process or a daemon or something for this purpose, while safer, would be a significant new factor and complication that would basically prevent a huge portion of our users from using it. I think a simpler first step could be to just "scramble" the data while "long-term stored" in memory. -- / daniel.haxx.se | Commercial curl support up to 24x7 is available! | Private help, bug fixes, support, ports, new features | https://curl.se/support.html -- Unsubscribe: https://lists.haxx.se/listinfo/curl-library Etiquette: https://curl.se/mail/etiquette.html
Re: credentials in memory
On Fri, 30 Sep 2022, David Woodhouse wrote: Don't forget to ensure that all *transitional* storage is securely wiped, including request buffers in which the password has been (decrypted and) sent. The buffers we use for transport are all used temporary and are never kept around for long until they are overwritten again. I suppose that if an error occurs exactly when the block of data is meant to get sent off and the buffer then contains the password (for FTP, or HTTP basic or similar), it risks linger around for a longer time. -- / daniel.haxx.se | Commercial curl support up to 24x7 is available! | Private help, bug fixes, support, ports, new features | https://curl.se/support.html -- Unsubscribe: https://lists.haxx.se/listinfo/curl-library Etiquette: https://curl.se/mail/etiquette.html
Re: credentials in memory
On Fri, 2022-09-30 at 09:43 +0200, Daniel Stenberg via curl-library wrote: > Hi, > > I bring back an old discussion as I think it might be worth having it again. > > libcurl hold credentials (passwords for servers and proxies) in memory in > clear text, potentially for a long time. If something goes wrong and that > memory is accessed by an external party, things would be bad. > > We work really hard on preventing and avoiding the "something goes wrong" > part, but every once in a while we come back to the fact that we still keep > credentials in the clear in memory. This is also something that our good > friends at Trail of Bits have commented on during the ongoing security audit > they are performing on curl. > > Is it worth doing something about? > > If it is, what's a reasonable mitigation? We need to be able to use them in > the clear, so whatever we do we cannot just hash them. We would need to have > a > way to store them encrypted and decrypt them on demand when they are needed > and then only use them "in the clear" for as a short period as possible. > > This would mostly remove them from being readable as-is, like if there's a > stack reveal or heap leak as a result of a vulnerability perhaps, the risk > for > credential leaks would be reduced. > > The "encryption" then wouldn't have to be complicated and could use a > randomly > generated "key", probably created when the handle is created. > > Of course, since the passwords are passed in to libcurl from applications, > this dance is less effective if they then keep the credentials around in the > clear in memory anyway, but I think maybe they typically keep them around for > a shorter time in general. > > Thoughts? Pointless? Improvements? Don't forget to ensure that all *transitional* storage is securely wiped, including request buffers in which the password has been (decrypted and) sent. smime.p7s Description: S/MIME cryptographic signature -- Unsubscribe: https://lists.haxx.se/listinfo/curl-library Etiquette: https://curl.se/mail/etiquette.html
Re: credentials in memory
> Am 30.09.2022 um 09:43 schrieb Daniel Stenberg via curl-library > : > > Hi, > > Is it worth doing something about? > Well, if you like to prevent picking passwords easily from memory dumps while a transaction is running, e.g. longer download, you may just do a bit of xor for the long term storage. Like get a random 16 byte string at start and then xor values with it. This way you won't need a crypto library as a reference for a non SSL enabled curl. On the other side, when our applications passes a password to curl, it is still in some object property or even the textfield of the GUI. Best regards, Christian -- Read our blog about news on our plugins: http://www.mbsplugins.de/ -- Unsubscribe: https://lists.haxx.se/listinfo/curl-library Etiquette: https://curl.se/mail/etiquette.html
Re: credentials in memory
> The "encryption" then wouldn't have to be complicated and could use a randomly > generated "key", probably created when the handle is created. That looks reasonable. Random key is harder to find in any memory dump. Especially if not base64-encoded or something like that. > Of course, since the passwords are passed in to libcurl from applications, > this dance is less effective if they then keep the credentials around in the > clear in memory anyway, but I think maybe they typically keep them around for > a shorter time in general. Yep, but what the application does is not our concern. If curl / libcurl can be made "safe", it's only to its advantage. Whether the application take advantage of that or not is their problem. > Thoughts? Pointless? Improvements? I'd still put it behind a CURLOPT for the sake of all low-powered devices that cannot really afford any additional load due to encryption. -- Unsubscribe: https://lists.haxx.se/listinfo/curl-library Etiquette: https://curl.se/mail/etiquette.html
Re: credentials in memory
On Fri, 30 Sept 2022 at 10:12, Stefan Eissing via curl-library wrote: > > > > > Am 30.09.2022 um 09:43 schrieb Daniel Stenberg via curl-library > > : > > > > Hi, > > > > I bring back an old discussion as I think it might be worth having it again. > > > > libcurl hold credentials (passwords for servers and proxies) in memory in > > clear text, potentially for a long time. If something goes wrong and that > > memory is accessed by an external party, things would be bad. > > > > We work really hard on preventing and avoiding the "something goes wrong" > > part, but every once in a while we come back to the fact that we still keep > > credentials in the clear in memory. This is also something that our good > > friends at Trail of Bits have commented on during the ongoing security > > audit they are performing on curl. > > > > Is it worth doing something about? > > > > If it is, what's a reasonable mitigation? We need to be able to use them in > > the clear, so whatever we do we cannot just hash them. We would need to > > have a way to store them encrypted and decrypt them on demand when they are > > needed and then only use them "in the clear" for as a short period as > > possible. > > > > This would mostly remove them from being readable as-is, like if there's a > > stack reveal or heap leak as a result of a vulnerability perhaps, the risk > > for credential leaks would be reduced. > > > > The "encryption" then wouldn't have to be complicated and could use a > > randomly generated "key", probably created when the handle is created. > > > > Of course, since the passwords are passed in to libcurl from applications, > > this dance is less effective if they then keep the credentials around in > > the clear in memory anyway, but I think maybe they typically keep them > > around for a shorter time in general. > > > > Thoughts? Pointless? Improvements? > > > I know of threee patterns to solve this problem (and increase usability as a > side effect): > > 1. macOS keychain which persists credentials and makes them available to > authorized applications > 2. ssh-agent > 3. neverbleed (https://github.com/h2o/neverbleed) that spawns a process, > using pipes to handle keys in out-of-process memory > > Maybe a "curl-agent" could be a portable approach? interesting, a different approach might be smoothing out integration with the various vaults/secrets (ex. hashicorp, bitwarden, etc). Jim -- Unsubscribe: https://lists.haxx.se/listinfo/curl-library Etiquette: https://curl.se/mail/etiquette.html
Re: credentials in memory
> Am 30.09.2022 um 09:43 schrieb Daniel Stenberg via curl-library > : > > Hi, > > I bring back an old discussion as I think it might be worth having it again. > > libcurl hold credentials (passwords for servers and proxies) in memory in > clear text, potentially for a long time. If something goes wrong and that > memory is accessed by an external party, things would be bad. > > We work really hard on preventing and avoiding the "something goes wrong" > part, but every once in a while we come back to the fact that we still keep > credentials in the clear in memory. This is also something that our good > friends at Trail of Bits have commented on during the ongoing security audit > they are performing on curl. > > Is it worth doing something about? > > If it is, what's a reasonable mitigation? We need to be able to use them in > the clear, so whatever we do we cannot just hash them. We would need to have > a way to store them encrypted and decrypt them on demand when they are needed > and then only use them "in the clear" for as a short period as possible. > > This would mostly remove them from being readable as-is, like if there's a > stack reveal or heap leak as a result of a vulnerability perhaps, the risk > for credential leaks would be reduced. > > The "encryption" then wouldn't have to be complicated and could use a > randomly generated "key", probably created when the handle is created. > > Of course, since the passwords are passed in to libcurl from applications, > this dance is less effective if they then keep the credentials around in the > clear in memory anyway, but I think maybe they typically keep them around for > a shorter time in general. > > Thoughts? Pointless? Improvements? I know of threee patterns to solve this problem (and increase usability as a side effect): 1. macOS keychain which persists credentials and makes them available to authorized applications 2. ssh-agent 3. neverbleed (https://github.com/h2o/neverbleed) that spawns a process, using pipes to handle keys in out-of-process memory Maybe a "curl-agent" could be a portable approach? Kind Regards, Stefan -- Unsubscribe: https://lists.haxx.se/listinfo/curl-library Etiquette: https://curl.se/mail/etiquette.html
