Re: UTF8 decoding, unneeded byte masking
On 8/27/13 1:33 AM, Dave Thompson wrote:
From: owner-openssl-...@openssl.org On Behalf Of Yuan Kang
Sent: Tuesday, 27 August, 2013 00:54
I don't think that it is true that "(signed char)(*p) >= 0"
is always true,
Mr Weimer didn't say it IS always true, he said a compiler
IS ALLOWED TO ASSUME it is. As I adjusted, the compiler does
have to document that it does this.
I don't know who would issue such rules, or who would enforce them.
Whether a given compiler
does it depends on the writers of that compiler, and quite
likely on the target architecture, and quite possibly on
the optimizations implemented by the compiler and used.
As I added, gcc (at least the versions I have) documents
that it doesn't do so.
I really doubt that.
Here is (from memory) a code snippet I tested on gcc a year or so ago
void
testint(int a, int b)
{
int c;
if (a < 0) return;
if (b < 0) return;
c = a+b;
if (c < 0) {
printf("a+b = %d, which is negative\n", c);
} else {
printf(a+b = %d, which is positive\n", c);
}
}
when compiled with optimization -O2 or higher and with and a and b both
20, it prints
"a+b = -294967296 which is positive".
The fact that the value of an operation with signed arithmetic that
overflows (the + operator) is undefined, and the result of and
operation where at least one operand is undefined (the < operator), is
undefined. That allows the compiler to reason using ordinary
mathematical operations rather than computer operations. If there is no
overflow, computer arithmetic and mathematical arithmetic are the same,
and it is okay. If overflow occurs, the result is undefined, so
anything they do is okay. So they reason that a is non-negative and b
is non-negative, and hence a+b must be non-negative, and hence, c is
non-negative, and c<0 if false, and optimize away the test and the true
branch.
The -fwrapv flag tells the optimizer to make the result of signed
arithmetic defined and take into account the wrapping behavior of
computer arithmetic. When compiled with -fwrapv, the output is as expected.
--David
What you need to check (or test) is
all versions of all other compilers used on all platforms
OpenSSL supports, with all optimization settings. Preferably
also all versions or replacements that will be released in
the next several years; people often are seen staying on
old OpenSSL versions 5 years or more, so that seems a good
minimum. It's a lot easier to support code that just works.
I think the optimization that PMHager was thinking about
(based on what I see in the x86_64 assembly from gcc) is
that the compiler could (but apparently not every compiler
does this) use the instruction for checking the sign, like this:
9:84 c9test %cl,%cl
b:78 1bjs 28
This is most likely faster than having to do a logical and instruction
first.
It may be faster but that's far from certain, and I'm very
confident it's not usefully faster; nearly all of the time
will be in the memory references (which don't change)
and the branch if mispredicted (which doesn't change).
And not all CPUs work the same as x64, or x86. Where
speed really matters, like in some of the cipher and
hash cores, OpenSSL has assembler -- separate for each
CPU (more or less), and reliably producing the desired
instructions no matter what the C code generator does.
That's better (though more work) than nonportable C,
which soon needs #ifdef hell to keep working.
__
OpenSSL Project http://www.openssl.org
Development Mailing List openssl-dev@openssl.org
Automated List Manager majord...@openssl.org
__
OpenSSL Project http://www.openssl.org
Development Mailing List openssl-dev@openssl.org
Automated List Manager majord...@openssl.org
make elapsed in speed.c always on, plus a couple questions/comments
Looks like some old code didn't get cleaned up, and the documentation for elapsed isn't always displayed though it now is always compiled in: diff --git a/apps/speed.c b/apps/speed.c index 9232418..f70fd3e 100644 --- a/apps/speed.c +++ b/apps/speed.c @@ -1126,9 +1126,7 @@ int MAIN(int argc, char **argv) BIO_printf(bio_err,"\n"); BIO_printf(bio_err,"Available options:\n"); -#if defined(TIMES) || defined(USE_TOD) BIO_printf(bio_err,"-elapsedmeasure time in real time instead of CPU user time.\n"); -#endif #ifndef OPENSSL_NO_ENGINE BIO_printf(bio_err,"-engine e use engine e, possibly a hardware device.\n"); #endif You should look at making -evp the default now... Or at least warn people that if you have an AES-NI capable CPU that doing: openssl speed aes Is not going to give them the results you think it will... Also, is there a way to specify which engines get used for which ciphers? So maybe you can still use the native AES-NI implementation even when you have cryptodev? Thanks. -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not." __ OpenSSL Project http://www.openssl.org Development Mailing List openssl-dev@openssl.org Automated List Manager majord...@openssl.org
Re: UTF8 decoding, unneeded byte masking
I believe the masking part is there because of the UTF-8 standard:
https://tools.ietf.org/html/rfc3629#section-3
The first byte starts with, say n - 1, consecutive bits with value 1, and
then a bit with value 0 to indicate the number of bytes to read. The
remaining 8 - n bits in the first byte are then read into value. Although
this doesn't really do anything in the case of n = 1 (which is when we mask
with 0x7F), it is needed for all of the other cases, as you can see in the
"if-else" branches, and consistency across the different cases also helps
readability. I think the reader's question would be more easily answered if
the reader knows what 0x7F means. 0x7F is equal to ~0x80, and likewise, in
the other cases, 0x1f = ~0xe0, 0xf = ~0xf0, and so on. Seeing how there is
a lot of repeated or predictable code, we could instead use macros,
something like this:
#define LEADING_BITS_SET(n) (((1 << n) - 1) << (8 - n))
#define BITS_AFTER(n) (~LEADING_BITS_SET(n))
...
if ((*p & LEADING_BITS_SET(n)) == LEADING_BITS_SET(n - 1)) {
...
value = (*p++ & BITS_AFTER(n)) ...
...
}
Alternatively, if we really want to optimize out the unnecessary masking
for n = 1, we can have:
#define BITS_AFTER(n) (~LEADING_BITS_SET(n - 1))
So BITS_AFTER(1) = 0xff, which I believe will get optimized out from the
and instruction.
In any case the first thing the reader will think is "something that's
consistent with UTF-8," and the thought of "technically not necessary for
the first case" may not even occur depending on how we define BITS_AFTER
On Tue, Aug 27, 2013 at 2:13 AM, Michel wrote:
> Thanks for your comment,
> but no, I didn't talk about performance.
> I understand this is not very costly, especially compared with other
> crypto operations.
> My concern was mostly about keeping the source code easy to understand and
> 'logically consistent'.
>
> I am trying to save the reader from asking himself "what is the use of the
> '& 0x7F' masking operation ?".
>
> Le 25/08/2013 12:23, PMHager a écrit :
>
>> If your intention is performance optimization you could even replace
>>
>>if((*p & 0x80) == 0)
>> with
>>if((signed char)(*p) >= 0)
>>
>> as you cannot assume that all compilers will do it correctly themselves.
>>
>> -Original Message-
>> From: owner-openssl-...@openssl.org [mailto:owner-openssl-dev@**
>> openssl.org ] On
>> Behalf Of Michel
>> Sent: Thursday, August 22, 2013 11:44 AM
>> To: openssl-dev@openssl.org
>> Subject: UTF8 decoding, unneeded byte masking
>>
>> In a_utf8.c, lines 85 and 86 (1.0.1e) :
>>
>> ...
>>if((*p & 0x80) == 0) { // as this byte looks like : 0xxx
>>value = *p++ & 0x7f; // this line could as well be written :
>> value = *p++;
>> ...
>>
>> If I don't miss something, it would seems clearer to me.
>>
>>
>
> __**__**__
> OpenSSL Project http://www.openssl.org
> Development Mailing List openssl-dev@openssl.org
> Automated List Manager majord...@openssl.org
>
Missing error handling in SRP implementation
Hi, am I somehow missing something obvious or are there numerous error handling checks missing in the SRP implementation? For example in SRP_create_verifier() in crypto/srp/srp_vfy.c, I would argue that if the first two calls to BN_bin2bn() succeed, but the one for the salt fails, that would produce NULL pointer dereferences somewhere in the bignum calls. In the inverse case, instead, the salt would be leaking, I think? Figuring out what exactly the consequences of each of those unchecked error returns are is tedious, and often, some check deeper into the call tree does indeed catch resulting NULL pointers, but it seems to me like that is more of an accident than intentional design?! Florian __ OpenSSL Project http://www.openssl.org Development Mailing List openssl-dev@openssl.org Automated List Manager majord...@openssl.org
Re: UTF8 decoding, unneeded byte masking
>On Tue, Aug 27, 2013 at 1:33 AM, Dave Thompson wrote: > Mr Weimer didn't say it IS always true, he said a compiler > IS ALLOWED TO ASSUME it is. Sorry about the misinterpretation. I thought that Mr. Weimer meant that the assumption is desired behaviour here, with which I disagreed, but now it makes sense to me. > It may be faster but that's far from certain, and I'm very > confident it's not usefully faster; > Good point. I let tunnel vision get the best of me again.
RE: AES-XTS problem in non-FIPS mode
OK, found the error. I simply did not give a "double size" key to the cipher as required. That would explain why it sometimes worked. Sorry for the trouble Thanks for your time and support LJB > -Original Message- > From: owner-openssl-...@openssl.org [mailto:owner-openssl-...@openssl.org] > On Behalf Of Leon Brits > Sent: 27 August 2013 02:48 PM > To: openssl-dev@openssl.org > Subject: RE: AES-XTS problem in non-FIPS mode > > OK, sorry this stupid error has been resolved. There was some openssl init > code which got disabled when I disabled lines of source for FIPS mode. > > The problem however still persists for me even with this OpenSSL which has > been compiled without fips. I will continue looking at my code. > > Thanks > LJB > > > -Original Message- > > From: owner-openssl-...@openssl.org > > [mailto:owner-openssl-...@openssl.org] > > On Behalf Of Leon Brits > > Sent: 27 August 2013 12:08 PM > > To: openssl-dev@openssl.org > > Subject: RE: AES-XTS problem in non-FIPS mode > > > > Ok, some weirdness happening here... > > > > I've selected to test with option 2 and recompiled my openssl 1.0.1e > > withOUT "fips" in "./config fips". > > > > $ openssl version > > OpenSSL 1.0.1e 11 Feb 2013 > > > > I've verified that the AES-XTS cipher is present with: > > $ openssl list-cipher-algorithms > > > > In my app I resolve the NID_aes_256_xts to a name with OBJ_nid2sn() > > and get the same name as in the list above. > > > > However when I call EVP_get_cipherbyname() with this name I get a NULL. > > I've never had an error with this in the FIPS compiled module. This > > just does not seem possible as an error, so any ideas on what may be > > wrong with my system? I've working in VirtualBox VM with a default > > installed Ubuntu 12.04.02. > > > > Thanks > > LJB > > > > > > > -Original Message- > > > From: owner-openssl-...@openssl.org > > > [mailto:owner-openssl-...@openssl.org] > > > On Behalf Of Dr. Stephen Henson > > > Sent: 26 August 2013 03:23 PM > > > To: openssl-dev@openssl.org > > > Subject: Re: AES-XTS problem in non-FIPS mode > > > > > > On Mon, Aug 26, 2013, Leon Brits wrote: > > > > > > > I am using a FIPS compiled OpenSSL and I switch between FIPS and > > > > non- > > > FIPS mode with the FIPS_mode_set() API call. The selection is made > > > by the application linked to my library based on its configuration. > > > > > > > > > > That's weird. It should be using exactly the same algorithm > > > implementation then. > > > > > > Assuming there's no problem with your code the only thing I can > > > think of is some inconsistency between FIPS and non-FIPS > > > initialisation of EVP. To test that have a look in > > > crypto/evp/evp_enc.c in OpenSSL 1.0.1 for the lines that check > > > FIPS_mode(). Change them so they're always caled and not just if > > > FIPS_mode() is non-zero. > > > > > > If possible also try OpenSSL 1.0.1 without the "fips" compilation > > option: > > > it will then use its internal implementation and not the one in the > > > FIPS module. > > > > > > Steve. > > > -- > > > Dr Stephen N. Henson. OpenSSL project core developer. > > > Commercial tech support now available see: http://www.openssl.org > > > __ > > > OpenSSL Project http://www.openssl.org > > > Development Mailing List openssl-dev@openssl.org > > > Automated List Manager majord...@openssl.org > > > > > > __ > > OpenSSL Project http://www.openssl.org > > Development Mailing List openssl-dev@openssl.org > > Automated List Manager majord...@openssl.org > > > __ > OpenSSL Project http://www.openssl.org > Development Mailing List openssl-dev@openssl.org > Automated List Manager majord...@openssl.org __ OpenSSL Project http://www.openssl.org Development Mailing List openssl-dev@openssl.org Automated List Manager majord...@openssl.org
RE: AES-XTS problem in non-FIPS mode
OK, sorry this stupid error has been resolved. There was some openssl init code which got disabled when I disabled lines of source for FIPS mode. The problem however still persists for me even with this OpenSSL which has been compiled without fips. I will continue looking at my code. Thanks LJB > -Original Message- > From: owner-openssl-...@openssl.org [mailto:owner-openssl-...@openssl.org] > On Behalf Of Leon Brits > Sent: 27 August 2013 12:08 PM > To: openssl-dev@openssl.org > Subject: RE: AES-XTS problem in non-FIPS mode > > Ok, some weirdness happening here... > > I've selected to test with option 2 and recompiled my openssl 1.0.1e > withOUT "fips" in "./config fips". > > $ openssl version > OpenSSL 1.0.1e 11 Feb 2013 > > I've verified that the AES-XTS cipher is present with: > $ openssl list-cipher-algorithms > > In my app I resolve the NID_aes_256_xts to a name with OBJ_nid2sn() and > get the same name as in the list above. > > However when I call EVP_get_cipherbyname() with this name I get a NULL. > I've never had an error with this in the FIPS compiled module. This just > does not seem possible as an error, so any ideas on what may be wrong with > my system? I've working in VirtualBox VM with a default installed Ubuntu > 12.04.02. > > Thanks > LJB > > > > -Original Message- > > From: owner-openssl-...@openssl.org > > [mailto:owner-openssl-...@openssl.org] > > On Behalf Of Dr. Stephen Henson > > Sent: 26 August 2013 03:23 PM > > To: openssl-dev@openssl.org > > Subject: Re: AES-XTS problem in non-FIPS mode > > > > On Mon, Aug 26, 2013, Leon Brits wrote: > > > > > I am using a FIPS compiled OpenSSL and I switch between FIPS and > > > non- > > FIPS mode with the FIPS_mode_set() API call. The selection is made by > > the application linked to my library based on its configuration. > > > > > > > That's weird. It should be using exactly the same algorithm > > implementation then. > > > > Assuming there's no problem with your code the only thing I can think > > of is some inconsistency between FIPS and non-FIPS initialisation of > > EVP. To test that have a look in crypto/evp/evp_enc.c in OpenSSL 1.0.1 > > for the lines that check FIPS_mode(). Change them so they're always > > caled and not just if > > FIPS_mode() is non-zero. > > > > If possible also try OpenSSL 1.0.1 without the "fips" compilation > option: > > it will then use its internal implementation and not the one in the > > FIPS module. > > > > Steve. > > -- > > Dr Stephen N. Henson. OpenSSL project core developer. > > Commercial tech support now available see: http://www.openssl.org > > __ > > OpenSSL Project http://www.openssl.org > > Development Mailing List openssl-dev@openssl.org > > Automated List Manager majord...@openssl.org > > > __ > OpenSSL Project http://www.openssl.org > Development Mailing List openssl-dev@openssl.org > Automated List Manager majord...@openssl.org __ OpenSSL Project http://www.openssl.org Development Mailing List openssl-dev@openssl.org Automated List Manager majord...@openssl.org
Re: UTF8 decoding, unneeded byte masking
I forgot to mention that, even though performance is not my concern here, I do appreciate your comments on that matter. Le 27/08/2013 11:13, Michel a écrit : Thanks for your comment, but no, I didn't talk about performance. I understand this is not very costly, especially compared with other crypto operations. My concern was mostly about keeping the source code easy to understand and 'logically consistent'. I am trying to save the reader from asking himself "what is the use of the '& 0x7F' masking operation ?". __ OpenSSL Project http://www.openssl.org Development Mailing List openssl-dev@openssl.org Automated List Manager majord...@openssl.org
RE: AES-XTS problem in non-FIPS mode
Ok, some weirdness happening here... I've selected to test with option 2 and recompiled my openssl 1.0.1e withOUT "fips" in "./config fips". $ openssl version OpenSSL 1.0.1e 11 Feb 2013 I've verified that the AES-XTS cipher is present with: $ openssl list-cipher-algorithms In my app I resolve the NID_aes_256_xts to a name with OBJ_nid2sn() and get the same name as in the list above. However when I call EVP_get_cipherbyname() with this name I get a NULL. I've never had an error with this in the FIPS compiled module. This just does not seem possible as an error, so any ideas on what may be wrong with my system? I've working in VirtualBox VM with a default installed Ubuntu 12.04.02. Thanks LJB > -Original Message- > From: owner-openssl-...@openssl.org [mailto:owner-openssl-...@openssl.org] > On Behalf Of Dr. Stephen Henson > Sent: 26 August 2013 03:23 PM > To: openssl-dev@openssl.org > Subject: Re: AES-XTS problem in non-FIPS mode > > On Mon, Aug 26, 2013, Leon Brits wrote: > > > I am using a FIPS compiled OpenSSL and I switch between FIPS and non- > FIPS mode with the FIPS_mode_set() API call. The selection is made by the > application linked to my library based on its configuration. > > > > That's weird. It should be using exactly the same algorithm implementation > then. > > Assuming there's no problem with your code the only thing I can think of > is some inconsistency between FIPS and non-FIPS initialisation of EVP. To > test that have a look in crypto/evp/evp_enc.c in OpenSSL 1.0.1 for the > lines that check FIPS_mode(). Change them so they're always caled and not > just if > FIPS_mode() is non-zero. > > If possible also try OpenSSL 1.0.1 without the "fips" compilation option: > it will then use its internal implementation and not the one in the FIPS > module. > > Steve. > -- > Dr Stephen N. Henson. OpenSSL project core developer. > Commercial tech support now available see: http://www.openssl.org > __ > OpenSSL Project http://www.openssl.org > Development Mailing List openssl-dev@openssl.org > Automated List Manager majord...@openssl.org __ OpenSSL Project http://www.openssl.org Development Mailing List openssl-dev@openssl.org Automated List Manager majord...@openssl.org
Re: UTF8 decoding, unneeded byte masking
Thanks for your comment,
but no, I didn't talk about performance.
I understand this is not very costly, especially compared with other
crypto operations.
My concern was mostly about keeping the source code easy to understand
and 'logically consistent'.
I am trying to save the reader from asking himself "what is the use of
the '& 0x7F' masking operation ?".
Le 25/08/2013 12:23, PMHager a écrit :
If your intention is performance optimization you could even replace
if((*p & 0x80) == 0)
with
if((signed char)(*p) >= 0)
as you cannot assume that all compilers will do it correctly themselves.
-Original Message-
From: owner-openssl-...@openssl.org [mailto:owner-openssl-...@openssl.org] On
Behalf Of Michel
Sent: Thursday, August 22, 2013 11:44 AM
To: openssl-dev@openssl.org
Subject: UTF8 decoding, unneeded byte masking
In a_utf8.c, lines 85 and 86 (1.0.1e) :
...
if((*p & 0x80) == 0) { // as this byte looks like : 0xxx
value = *p++ & 0x7f; // this line could as well be written : value =
*p++;
...
If I don't miss something, it would seems clearer to me.
__
OpenSSL Project http://www.openssl.org
Development Mailing List openssl-dev@openssl.org
Automated List Manager majord...@openssl.org
Re: UTF8 decoding, unneeded byte masking
>From: owner-openssl-...@openssl.org On Behalf Of Yuan Kang >Sent: Tuesday, 27 August, 2013 00:54 >I don't think that it is true that "(signed char)(*p) >= 0" >is always true, Mr Weimer didn't say it IS always true, he said a compiler IS ALLOWED TO ASSUME it is. As I adjusted, the compiler does have to document that it does this. Whether a given compiler does it depends on the writers of that compiler, and quite likely on the target architecture, and quite possibly on the optimizations implemented by the compiler and used. As I added, gcc (at least the versions I have) documents that it doesn't do so. What you need to check (or test) is all versions of all other compilers used on all platforms OpenSSL supports, with all optimization settings. Preferably also all versions or replacements that will be released in the next several years; people often are seen staying on old OpenSSL versions 5 years or more, so that seems a good minimum. It's a lot easier to support code that just works. > I think the optimization that PMHager was thinking about > (based on what I see in the x86_64 assembly from gcc) is > that the compiler could (but apparently not every compiler > does this) use the instruction for checking the sign, like this: > 9:84 c9test %cl,%cl > b:78 1bjs 28 >This is most likely faster than having to do a logical and instruction first. It may be faster but that's far from certain, and I'm very confident it's not usefully faster; nearly all of the time will be in the memory references (which don't change) and the branch if mispredicted (which doesn't change). And not all CPUs work the same as x64, or x86. Where speed really matters, like in some of the cipher and hash cores, OpenSSL has assembler -- separate for each CPU (more or less), and reliably producing the desired instructions no matter what the C code generator does. That's better (though more work) than nonportable C, which soon needs #ifdef hell to keep working. __ OpenSSL Project http://www.openssl.org Development Mailing List openssl-dev@openssl.org Automated List Manager majord...@openssl.org
RE: UTF8 decoding, unneeded byte masking
> From: owner-openssl-...@openssl.org On Behalf Of Florian Weimer > Sent: Monday, 26 August, 2013 10:58 > To: openssl-dev@openssl.org > Cc: PMHager > Subject: Re: UTF8 decoding, unneeded byte masking > > On 08/25/2013 12:23 PM, PMHager wrote: > > If your intention is performance optimization you could even replace > > > >if((*p & 0x80) == 0) > > > > with > > > >if((signed char)(*p) >= 0) > > > > as you cannot assume that all compilers will do it > correctly themselves. > > Actually, this proposed change relies on a GCC extension. If *p is > unsigned, compilers are allowed to assume that (signed > char)(*p) >= 0 is > always true (because signed overflow is undefined). > Not quite. Overflow in signed integer computation is Undefined Behavior, which can be even worse than using the wrong value; in particular sometimes it causes a trap/interrupt/abort/etc. But this case is out-of-range conversion to signed integer -- and that is *implementation-defined*: the compiled code must give either a documented value or a documented signal, but this need not be the same as other systems, and need not be the value you want (and if a nonresumable signal, not any value at all). The (oldish) versions of gcc I have conveniently to hand do document unsigned-to-signed as the common and intuitive just-use-the-bits, which means the "fix" works. The gcc maintainers could conceivably decide to change, but IME over the years they MOSTLY avoid changing basic things like this that would upset a lot of people. But that's not a real guarantee, and gcc is not the only compiler. Nit: but only if the char types are 8 bits. The Standard allows more (not less) and in that case the conversion from u-char 0x80:0xFF to s-char is guaranteed never negative, so the "fix" always fails. But I don't think OpenSSL supports any platforms with CHAR_BIT > 8; the known ones mentioned on the C newsgroup don't do IP, making libssl pretty useless, though maybe parts of libcrypto could still be workable. __ OpenSSL Project http://www.openssl.org Development Mailing List openssl-dev@openssl.org Automated List Manager majord...@openssl.org
