Re: SecureD Futures (v2.0)
On 05/29/2018 11:29 AM, Brad Roberts wrote: On 5/29/2018 1:57 AM, Adam Wilson via Digitalmars-d wrote: One of the pillars of SecureD is that ONLY safe, well-known, algorithms are presented. If reasonable we will only present one algorithm for a specific purpose. If there is a good reason to add more than one algorithm, we will. One of the pillars of crypto is that eventually a problem will be found with every algorithm, it's just a matter of time. So, having just one available means that eventually the library will be horribly broken. The corollary here is that having a fallback is pretty essential. Sure. For example, TLS 1.3 supports AES, Camellia, ARIA, and ChaCha20-Poly1305 for symmetric ciphers. However, when you look at the best-practice recommendations, AES is the clear winner. And when you look at what actually gets used, it's AES. AES is the Gold Standard and it would take a fundamental breakthrough in cryptanalysis for that to change. But what happens to TLS1.3 if there is a fundamental breakthrough? - ARIA uses an AES derived substitution-permutation network. - Camellia uses the same S-box that AES does. (It was an AES contender) - Poly1305 for ChaCha20 directly uses AES in it's PRF. Which algorithm do you switch to? Anything that breaks AES can also be used against these other algorithms. In this case, ChaCha20 is probably the best answer, but only if you apply a different MAC to it. It's a good thing that a fundamental breakthrough in cryptanalysis is unlikely in the foreseeable future. Of course, that's what we told ourselves before 3DES and Differential Cryptanalysis. I don't disagree with you. But the overwhelming cacophony of choice presents it's own issues. (See: The Paradox of Choice) SecureD v2 could easily support these ciphers, and I don't really have a problem adding them for the purposes of compatibility. But the default is going to be AES for the forseeable future. The design premise of SecureD is to make the the safe defaults easy and everything else hard. In cases where there are known weaknesses SecureD does intend to support multiple algorithms, but usually only two, for example SHA2 potentially suffers from the same weaknesses as SHA1, and as a result SecureD v2 will be getting SHA3 support whenever OpenSSL 1.1.1 drops. But it's not going to add things like Whirlpool or Skein. Or RSA versus ECC. I am considering adding Curve25519 support to v2, but with NSA/NIST deprecating ECC altogether (as it is less resistant to quantum computing than RSA, see: https://en.wikipedia.org/wiki/Elliptic-curve_cryptography#Quantum_computing_attacks) it may be better to wait-and-see what comes of the work being done on quantum-resistant public-key cryptography. The new Supersingular Isogeny Diffie-Hellman (SIDH) looks like a strong contender, but nothing implements it, and it's way too new to put any trust in it yet. -- Adam Wilson IRC: LightBender import quiet.dlang.dev;
Re: SecureD Futures (v2.0)
On 5/29/2018 1:57 AM, Adam Wilson via Digitalmars-d wrote: One of the pillars of SecureD is that ONLY safe, well-known, algorithms are presented. If reasonable we will only present one algorithm for a specific purpose. If there is a good reason to add more than one algorithm, we will. One of the pillars of crypto is that eventually a problem will be found with every algorithm, it's just a matter of time. So, having just one available means that eventually the library will be horribly broken. The corollary here is that having a fallback is pretty essential.
Re: SecureD Futures (v2.0)
On 05/28/2018 04:02 PM, sarn wrote: On Monday, 28 May 2018 at 07:52:43 UTC, Adam Wilson wrote: I understand that. Sorry, not for nothing, but you obviously don't. For starters, if you were familiar with the key derivation tools available 24hrs ago, you wouldn't have come up with PBKDF2 on PBKDF2. I suggest slowing down a little, and asking people on a crypto forum if you're still not sure. If you explain your problem from the start, they might even have some better ideas. When that RFC (correctly) recommends using a salt, it's talking about HKDF-Extract, which is a tool for taking a large amount of mostly-random data and turning it into an appropriate KDK. That's not the problem you have here. The problem you have is generating keys for different purposes from a KDK. That's a problem HKDF-Expand addresses, and it doesn't use a salt. Let me ask the question a different way. What is the reason NOT to use 2 different salts for the MAC and KEY generation steps? You might choose to not use extra salts for the same reason you've already chosen to not encrypt three times, or add extra HMACs for each individual block of ciphertext: it's not solving any problems. Ok, but I am not harming anything either. I asked specifically what would be reasons not to do something, you replied with "it's not solving any problems". In a technical sense you are correct. I am not solving any problems. However. I am not creating any either. Adding a salt to an HKDF does not hurt anything. Ever. As to the PBKDF2 comment. I'll concede that I am not perfect. Which is why I asked for input and I obviously "slowed down" enough to bother to ask. And when my mistake was pointed out I corrected it without further comment. We make all make mistakes sometimes, and cryptography is much harder than most to get right. So when we aren't sure we ask. And my statement was posed as a question. I was inviting feedback. One of the pillars of SecureD is that ONLY safe, well-known, algorithms are presented. If reasonable we will only present one algorithm for a specific purpose. If there is a good reason to add more than one algorithm, we will. For example, I am well aware of the BCrypt/SCrypt family as well as Argon2. At this point Argon2 appears to be the heir-apparent. Sadly, Argon2 is so new that neither Botan or OpenSSL implement it. So that one is ruled out for practical purposes, at least for now. I would also like to point out the OpenSSL will never implement BCrypt, but does implement SCrypt, and an Argon2 implementation is an open question due to it's use of threads. It turns out the key-stretching field is very active right now, and precisely what will become the de-facto standard is impossible to say. I don't feel comfortable trying to pick a winner to implement. For the moment, PBKDF2 is stable, broadly implemented, and most importantly, well-understood. SecureD v2 will default to 1,000,000 iterations which yields roughly 650ms on an Intel 8700K. As older hardware is unlikely to be as fast, times of over one second could be seen in production while still providing some buffer against the future. That said the default PBKDF2 method will be tune-able. I have long hoped for a reasonable replacement for PBKDF2, and with all the recent activity it is likely that something will shake out. But the attacks against Argon2 coming out so soon after it's release do not bode well for it in the future. And SCrypt requiring 16MB of RAM to achieve BCrypt's security makes it challenging to recommend for use on servers, especially in today's cloud environments where memory costs money. I know PBKDF2 is the conservative choice, but SecureD is all about conservative. For the moment, there aren't any great options, so conservative wins by default. SCrypt Memory Math: Assuming a 4GB VM in the cloud and we are willing to dedicate 1GB to SCrypt per second at BCrypt equivalent security. 1024/16=64 That's 64 connections per second, that is ... not great. And dedicating 1/4 of your systems RAM just for password hashing is being very generous. You're not going to see that in production because it is terribly expensive. As PBKDF2 and SCrypt are the only key-stretching algorithms available in both OpenSSL and Botan, and I cannot recommend SCrypt for cloud/server scenarios. PBKDF2 it is. To be perfectly honest, key-stretching is a mitigation. Weak passwords will be easily guessed no matter what algorithm is used. The real fix is to force better passwords. -- Adam Wilson IRC: LightBender import quiet.dlang.dev;
Re: SecureD Futures (v2.0)
On Monday, 28 May 2018 at 07:52:43 UTC, Adam Wilson wrote: I understand that. Sorry, not for nothing, but you obviously don't. For starters, if you were familiar with the key derivation tools available 24hrs ago, you wouldn't have come up with PBKDF2 on PBKDF2. I suggest slowing down a little, and asking people on a crypto forum if you're still not sure. If you explain your problem from the start, they might even have some better ideas. When that RFC (correctly) recommends using a salt, it's talking about HKDF-Extract, which is a tool for taking a large amount of mostly-random data and turning it into an appropriate KDK. That's not the problem you have here. The problem you have is generating keys for different purposes from a KDK. That's a problem HKDF-Expand addresses, and it doesn't use a salt. Let me ask the question a different way. What is the reason NOT to use 2 different salts for the MAC and KEY generation steps? You might choose to not use extra salts for the same reason you've already chosen to not encrypt three times, or add extra HMACs for each individual block of ciphertext: it's not solving any problems.
Re: SecureD Futures (v2.0)
On 05/28/2018 12:14 AM, sarn wrote: On Monday, 28 May 2018 at 06:22:02 UTC, Adam Wilson wrote: On 05/27/2018 08:52 PM, sarn wrote: On Monday, 28 May 2018 at 02:25:20 UTC, Adam Wilson wrote: I like it. But it does require more space. We need three salts and three lengths in the header. One for the PBKDF2 KDK, one for the MAC key, and one for the encryption key. HKDF-Expand doesn't need a salt. You just need one salt to make the KDK (whether you use PBKDF2 or HKDF-Extract for that) and no extra salts for deriving the encryption and MAC key. Strictly speaking, it's is Optional but Strongly Recommended per RFC5869-3.1 There's HKDF-Expand and there's HKDF-Extract. HKDF-Extract takes an optional salt and HKDF-Expand doesn't use a salt at all (take a closer look at that RFC). That OpenSSL routine is the combination of the two, so that's why it takes the salt. I understand that. But the way I read the first paragraph of 3.1 is that yes, it can work without a Salt, but HKDF, in general, is strongly suggested to be used with a salt. If we are really concerned about bytes we could reuse the salt for all three steps (KDK/MAC/KEY), but TBH, I'm not worried about disk usage. Let me ask the question a different way. What is the reason NOT to use 2 different salts for the MAC and KEY generation steps? -- Adam Wilson IRC: LightBender import quiet.dlang.dev;
Re: SecureD Futures (v2.0)
On Monday, 28 May 2018 at 06:22:02 UTC, Adam Wilson wrote: On 05/27/2018 08:52 PM, sarn wrote: On Monday, 28 May 2018 at 02:25:20 UTC, Adam Wilson wrote: I like it. But it does require more space. We need three salts and three lengths in the header. One for the PBKDF2 KDK, one for the MAC key, and one for the encryption key. HKDF-Expand doesn't need a salt. You just need one salt to make the KDK (whether you use PBKDF2 or HKDF-Extract for that) and no extra salts for deriving the encryption and MAC key. Strictly speaking, it's is Optional but Strongly Recommended per RFC5869-3.1 There's HKDF-Expand and there's HKDF-Extract. HKDF-Extract takes an optional salt and HKDF-Expand doesn't use a salt at all (take a closer look at that RFC). That OpenSSL routine is the combination of the two, so that's why it takes the salt.
Re: SecureD Futures (v2.0)
On 05/27/2018 08:52 PM, sarn wrote: On Monday, 28 May 2018 at 02:25:20 UTC, Adam Wilson wrote: I like it. But it does require more space. We need three salts and three lengths in the header. One for the PBKDF2 KDK, one for the MAC key, and one for the encryption key. HKDF-Expand doesn't need a salt. You just need one salt to make the KDK (whether you use PBKDF2 or HKDF-Extract for that) and no extra salts for deriving the encryption and MAC key. Strictly speaking, it's is Optional but Strongly Recommended per RFC5869-3.1 The use case here is that this data is going into storage and that storage is cheap. We don't have to be strict on our byte budget. :) https://tools.ietf.org/html/rfc5869 https://www.openssl.org/docs/man1.1.0/crypto/EVP_PKEY_CTX_set_hkdf_md.html SecureD is supposed to be "Crypto done right." So I might as well do it right and follow the RFC. -- Adam Wilson IRC: LightBender import quiet.dlang.dev;
Re: SecureD Futures (v2.0)
On Monday, 28 May 2018 at 02:25:20 UTC, Adam Wilson wrote: I like it. But it does require more space. We need three salts and three lengths in the header. One for the PBKDF2 KDK, one for the MAC key, and one for the encryption key. HKDF-Expand doesn't need a salt. You just need one salt to make the KDK (whether you use PBKDF2 or HKDF-Extract for that) and no extra salts for deriving the encryption and MAC key.
Re: SecureD Futures (v2.0)
On 05/27/2018 05:11 PM, sarn wrote:
On Sunday, 27 May 2018 at 10:27:45 UTC, Adam Wilson wrote:
struct cryptoHeader {
ubyte hdrVersion; // The version of the header
ubyte encAlg; // The encryption algorithm used
ubyte hashAlg; // The hash algorithm used
uint kdfIters; // The number of PBKDF2 iterations
ubyte authLen; // The length of the authentication value
ubyte saltLen; // The length of the PBKDF2 salt
ubyte ivLen; // The length of the IV
ulong encLen; // The length of the encrypted data
ulong adLen; // The length of the additional data
}
Should there be any kind of key identifier, or do you consider that a
separate issue?
hashAlg is used for everything related to key derivation. Salts and IVs
are random.
- MacKey = PBKDF2 over EncKey once using same inputs as EncKey
How about this?
Use PBKDF2 to generate a key-derivation-key (KDK), then use HKDF-Expand
(https://tools.ietf.org/html/rfc5869) to derive the encryption key and
MAC key separately.
I think that's more standard. I don't know how much it matters in
practice, but a lot of cryptographers don't like generating
MAC/encryption keys from each other directly.
I like it. But it does require more space. We need three salts and three
lengths in the header. One for the PBKDF2 KDK, one for the MAC key, and
one for the encryption key.
Something like this:
struct cryptoHeader {
ubyte hdrVersion;// The version of the header
ubyte encAlg;// The encryption algorithm used
ubyte hashAlg; // The hash algorithm used
uint kdfIters; // The number of PBKDF2 iterations
ubyte kdkSLen; // The length of the KDK Salt
ubyte macSLen; // The length of the MAC Salt
ubyte keySLen; // The length of the KEY Salt
ubyte ivLen; // The length of the IV
ulong encLen;// The length of the encrypted data
ulong adLen; // The length of the additional data
ubyte authLen; // The length of the authentication value
}
Also, it's probably safer to use HKDF-Extract instead of using raw keys
directly when not using PBKDF2.
It is. And we could dot that if you stick to the default encrypt/decrypt
routines. I want to expand the symmetric capabilities of SecureD so I am
going to rebuild the encrypt/decrypt routines as a composition of
smaller methods. That will allow users to either use the default
encryption (simple) or if they have to interoperate with other systems,
use the component methods to perform their operations. I am not looking
to cover all scenarios though, just the common ones. If you have
something truly unusual, you'll still need to drop down the base crypto
libraries.
--
Adam Wilson
IRC: LightBender
import quiet.dlang.dev;
Re: SecureD Futures (v2.0)
On Sunday, 27 May 2018 at 10:27:45 UTC, Adam Wilson wrote:
struct cryptoHeader {
ubyte hdrVersion; // The version of the header
ubyte encAlg; // The encryption algorithm used
ubyte hashAlg; // The hash algorithm used
uint kdfIters; // The number of PBKDF2 iterations
ubyte authLen; // The length of the authentication value
ubyte saltLen; // The length of the PBKDF2 salt
ubyte ivLen;// The length of the IV
ulong encLen; // The length of the encrypted data
ulong adLen;// The length of the additional data
}
Should there be any kind of key identifier, or do you consider
that a separate issue?
- MacKey = PBKDF2 over EncKey once using same inputs as EncKey
How about this?
Use PBKDF2 to generate a key-derivation-key (KDK), then use
HKDF-Expand (https://tools.ietf.org/html/rfc5869) to derive the
encryption key and MAC key separately.
I think that's more standard. I don't know how much it matters
in practice, but a lot of cryptographers don't like generating
MAC/encryption keys from each other directly.
Also, it's probably safer to use HKDF-Extract instead of using
raw keys directly when not using PBKDF2.
Re: SecureD Futures (v2.0)
On 05/27/2018 09:54 AM, Neia Neutuladh wrote: On Sunday, 27 May 2018 at 10:27:45 UTC, Adam Wilson wrote: Now that SecureD v1 is in the books This would have been a great place to insert a brief description of what SecureD is or a link to the project. Good point. SecureD is a cryptography library that is designed to make non transport-layer cryptography simple to use. I created it after working with OpenSSL and observing that the API design of OpenSSL actively encourages broken implementations. Think of it as a cryptography library for the rest of us. You can read more about it here: http://code.dlang.org/packages/secured -- Adam Wilson IRC: LightBender import quiet.dlang.dev;
Re: SecureD Futures (v2.0)
On Sunday, 27 May 2018 at 10:27:45 UTC, Adam Wilson wrote: Now that SecureD v1 is in the books This would have been a great place to insert a brief description of what SecureD is or a link to the project.
Re: SecureD Futures (v2.0)
On Sunday, 27 May 2018 at 10:27:45 UTC, Adam Wilson wrote: Now that SecureD v1 is in the books I thought it would be worthwhile to explore what a second version could like. I specifically want to focus on expanding compatibility with other systems. [...] No, it’s not. Look at IOpipe and no further it provides the exact abstraction that works for any form buffered I/O doing stream processing at top speeds. I’m doing IOpipe regex, it looks to be just what the doctor ordered. But that means pulling in Vibe.D for a simple cryptography library. At this point that doesn't seem like the right idea. If someone is willing to step-up and do the work I'd be willing to look at it, but for now I want to wait on this, preferably for a standard/generic streams interface to be made available. [...]
SecureD Futures (v2.0)
Now that SecureD v1 is in the books I thought it would be worthwhile to
explore what a second version could like. I specifically want to focus
on expanding compatibility with other systems.
For example: AWS uses SHA2-256 for signing requests. As implemented
today SecureD does not support this. However, this is one of the
scenarios that falls under SecureD's mission of ease-to-use cryptography
for non-transport layer security.
I am curious what people would like to see in SecureD moving forward and
I would love to get your feedback on what works and what doesn't work in
v1 of SecureD.
With this in mind here are my ideas for expanding SecureD's
capabilities. I'll start with the idea that I think will be the least
controversial.
1. Full support for SHA-2 and SHA-3 for Hashes and HMAC's.
This means the following modes would be supported:
SHA2: 224, 256, 384, 512, 512/224, 512/256
SHA3: 224, 256, 384, 512
The PBKDF2 implementation would also be updated to support the above
hash functions.
Hash methods supporting salts will be added.
To maintain backwards compatibility with v1, methods that match the v1
signatures and forward to the correct implementation would be provided
as the v1 defaults are still secure.
This requires OpenSSL 1.1.1 at a minimum. Note that SHA3 support could
be delayed until SecureD 2.1 if OpenSSL 1.1.1 is not available on a
majority of systems prior to shipping 2.0. In such case OpenSSL 1.1.0
would become the minimum supported version.
2. More Flexibility for Symmetric Encryption.
Currently SecureD only supports AES256-CTR-HMAC384 and the encrypted
data is laid out as: HASH+IV+DATA. This is only useful in scenarios
where the developer controls both the encryption and decryption of the
data. Additionally, this does not support AEAD modes, only AE modes.
The first thing I would like to do is to support all AES, SHA2, and SHA3
modes. I also want to add support for the CBC cipher mode with PCKS#7
padding as that is the most common cipher mode in use.
One of the primary use cases for SecureD is long-term storage of data.
To facilitate this requirement I would like to add a binary encoded
header to each encrypted blob. This header would contain the necessary
information to decode the blob (minus the key of course).
The header would cost 26 bytes per encrypted blob and would have the
following layout:
struct cryptoHeader {
ubyte hdrVersion; // The version of the header
ubyte encAlg; // The encryption algorithm used
ubyte hashAlg; // The hash algorithm used
uint kdfIters; // The number of PBKDF2 iterations
ubyte authLen; // The length of the authentication value
ubyte saltLen; // The length of the PBKDF2 salt
ubyte ivLen;// The length of the IV
ulong encLen; // The length of the encrypted data
ulong adLen;// The length of the additional data
}
The data layout would be: HEADER+AUTH+SALT+IV+ENC+AD
Any excluded elements would be marked as 0 in the header and completely
excluded from the data layout.
The MAC would be computed as: HMAC(HEADER+SALT+IV+ENC+AD)
The full encryption process would be as follows:
- Create a PBKDF2 salt via RNG
- Create IV via RNG
- Create header from user inputs
- EncKey = PBKDF2 over RawKey unless hashAlg is None or kdfIters == 0
- Use RawKey as-is if hashAlg is None or kdfIters == 0
- MacKey = PBKDF2 over EncKey once using same inputs as EncKey
- Encrypt data
- Compute HMAC(HEADER+SALT+IV+ENC+AD)
- Return RESULT(HEADER+AUTH+SALT+IV+ENC+AD)
Methods would be added to work with the following:
- Simple encrypted blobs (no header/auth/salt/iv/ad included)
- Creating and Verifying AE/AD MACs without headers
- Methods to read SecureD v1 encrypted data
Note that this plan is NOT compatible with the SecureD v1 encryption
data layout and data would need to be re-encrypted.
3. Re-implement the OpenSSL Bindings
The Deimos/DUB bindings are ancient and do not support OpenSSL 1.1.x. We
can re-implement the bindings taking care to only include the symbols we
need.
4. Streams
I get asked about this, and, as much as I want to do this, it is not a
simple topic. std.streams was removed and right now the only viable
replacement is Vibe.D streams. But that means pulling in Vibe.D for a
simple cryptography library. At this point that doesn't seem like the
right idea. If someone is willing to step-up and do the work I'd be
willing to look at it, but for now I want to wait on this, preferably
for a standard/generic streams interface to be made available.
Please let know what you think! I am very interested to hear about what
would make your life easier when working with SecureD an cryptography in
general.
--
Adam Wilson
IRC: LightBender
import quiet.dlang.dev;
