Re: [openssl-dev] Running against BoringSSL's SSL test suite

[Emilia Käsper]

FYI for easier use, this patch now lives at
https://github.com/google/openssl-tests, rebased against BoringSSL latest
(thanks David!) and OpenSSL-1.1.0-pre4 (Beta 1). I've also checked in a log
from Beta 1.

Cheers,
Emilia

On Thu, Mar 10, 2016 at 4:33 PM David Benjamin  wrote:

> On Thu, Mar 10, 2016 at 1:30 AM Kanaka Kotamarthy 
> wrote:
>
>> And also Openssl fails with Resume-Client-NoResume cases. Do you have
>> any report on which test cases do fail and reasons for the failure?
>>
>>
>> RT tickets 4387 through 4395 were the failures I've triaged. I'm sure
>> there's more things in there to look through.
>>
>> I don't believe Resume-Client-NoResume fails for me. Perhaps something
>> was fixed between master and 1.1.0-pre2.
>>
>>
>> Openssl doesn't gives any error. For Resume-Client-NoResume-SSL3-TLS11
>> test case, we expect the new session's handshake to be done with TLS11. But
>> with Openssl handshake is done using SSL3. As in ssl3_clear, we set back
>> s->version to s->method->version.
>>
>
> Oh, sorry, I keep forgetting our runner doesn't make it clear when a -test
> option fails to match anything. (I should fix that...) I looked
> for Resume-Client-NoResume without noticing it had suffixes. :-)
>
> I would expect most things addResumptionVersionTests to fail. See
> https://github.com/openssl/openssl/pull/603
>
> David
>
>
>> Thank you
>> Durga.
>>
>> On Wed, Mar 9, 2016 at 10:38 PM, David Benjamin 
>> wrote:
>>
>>> On Wed, Mar 9, 2016 at 5:07 AM Kanaka Kotamarthy 
>>> wrote:
>>>
 Hi

 I am even testing OpenSSL with BoringSSL's test cases using
 Openssl-1.1.0-pre2. Trying to find out reasons of OpenSSL's failures
 for particular cases.

 DTLS 1.0 session resumption has some thing wrong. If s_server started
 with -dtls and s_client -dtls1 -reconnect , session resumption is not
 being done. The reason for this may be, version negotiation for DTLS
 is done after loading previous session and check for s->version and
 s->session->version fails in tls_process_client_hello.

>>>
>>> See RT #4392.
>>> https://rt.openssl.org/Ticket/Display.html?id=4392
>>>
>>>
 And also Openssl fails with Resume-Client-NoResume cases. Do you have
 any report on which test cases do fail and reasons for the failure?

>>>
>>> RT tickets 4387 through 4395 were the failures I've triaged. I'm sure
>>> there's more things in there to look through.
>>>
>>> I don't believe Resume-Client-NoResume fails for me. Perhaps something
>>> was fixed between master and 1.1.0-pre2.
>>>
>>> David
>>>
>>>
 Thank you
 Durga.

 On Tue, Mar 8, 2016 at 3:19 AM, David Benjamin 
 wrote:
 > Hi folks,
 >
 > So, we've by now built up a decent-sized SSL test suite in BoringSSL.
 I was
 > bored and ran it against OpenSSL master. It revealed a number of
 bugs. One
 > is https://github.com/openssl/openssl/pull/603. I'll be filing
 tickets
 > shortly for the remaining ones I've triaged, but I thought I'd send
 this
 > separately rather than duplicate it everywhere.
 >
 > Emilia also suggested there may be room to collaborate on testing. If
 > nothing else, just borrowing ideas or porting tests to/from your
 TLSProxy
 > setup. (Like, say, the ones that caught the bugs I'll be reporting.
 :-) )
 > So, here's an introduction on how it all works:
 >
 > To run the tests on OpenSSL, clone BoringSSL:
 > https://boringssl.googlesource.com/boringssl/
 > Then patch in this change. (Click the "Download" in the upper-right
 for
 > options.)
 > https://boringssl-review.googlesource.com/#/c/7332/
 > Then follow the instructions in the commit message.
 >
 > The tests themselves and the runner logic live in
 ssl/test/runner/runner.go:
 >
 https://boringssl.googlesource.com/boringssl/+/22ce9b2d08a52e399bf2ab86851952d727be034d/ssl/test/runner/runner.go#922
 >
 > They work by running an unmodified TLS stack in a shim binary against
 a copy
 > of Go's. We patch our copy with options for weird behavior to test
 against:
 >
 https://boringssl.googlesource.com/boringssl/+/22ce9b2d08a52e399bf2ab86851952d727be034d/ssl/test/runner/common.go#414
 >
 > Go and shim communicate entirely with sockets and (tons of)
 command-line
 > flags, though it is slightly overfit to BoringSSL's behavior and
 checks
 > error strings a lot. The shim also has options like -async mode which
 we use
 > on a subset of tests to stress state machine resumption. (This has
 saved me
 > from state machine bugs so many times.)
 >
 https://boringssl.googlesource.com/boringssl/+/22ce9b2d08a52e399bf2ab86851952d727be034d/ssl/test/runner/runner.go#2770
 >
 

Re: [openssl-dev] Running against BoringSSL's SSL test suite

[Kanaka Kotamarthy]

And also Openssl fails with Resume-Client-NoResume cases. Do you have
any report on which test cases do fail and reasons for the failure?


RT tickets 4387 through 4395 were the failures I've triaged. I'm sure
there's more things in there to look through.

I don't believe Resume-Client-NoResume fails for me. Perhaps something was
fixed between master and 1.1.0-pre2.


Openssl doesn't gives any error. For Resume-Client-NoResume-SSL3-TLS11 test
case, we expect the new session's handshake to be done with TLS11. But with
Openssl handshake is done using SSL3. As in ssl3_clear, we set back
s->version to s->method->version.

Thank you
Durga.

On Wed, Mar 9, 2016 at 10:38 PM, David Benjamin  wrote:

> On Wed, Mar 9, 2016 at 5:07 AM Kanaka Kotamarthy 
> wrote:
>
>> Hi
>>
>> I am even testing OpenSSL with BoringSSL's test cases using
>> Openssl-1.1.0-pre2. Trying to find out reasons of OpenSSL's failures
>> for particular cases.
>>
>> DTLS 1.0 session resumption has some thing wrong. If s_server started
>> with -dtls and s_client -dtls1 -reconnect , session resumption is not
>> being done. The reason for this may be, version negotiation for DTLS
>> is done after loading previous session and check for s->version and
>> s->session->version fails in tls_process_client_hello.
>>
>
> See RT #4392.
> https://rt.openssl.org/Ticket/Display.html?id=4392
>
>
>> And also Openssl fails with Resume-Client-NoResume cases. Do you have
>> any report on which test cases do fail and reasons for the failure?
>>
>
> RT tickets 4387 through 4395 were the failures I've triaged. I'm sure
> there's more things in there to look through.
>
> I don't believe Resume-Client-NoResume fails for me. Perhaps something was
> fixed between master and 1.1.0-pre2.
>
> David
>
>
>> Thank you
>> Durga.
>>
>> On Tue, Mar 8, 2016 at 3:19 AM, David Benjamin 
>> wrote:
>> > Hi folks,
>> >
>> > So, we've by now built up a decent-sized SSL test suite in BoringSSL. I
>> was
>> > bored and ran it against OpenSSL master. It revealed a number of bugs.
>> One
>> > is https://github.com/openssl/openssl/pull/603. I'll be filing tickets
>> > shortly for the remaining ones I've triaged, but I thought I'd send this
>> > separately rather than duplicate it everywhere.
>> >
>> > Emilia also suggested there may be room to collaborate on testing. If
>> > nothing else, just borrowing ideas or porting tests to/from your
>> TLSProxy
>> > setup. (Like, say, the ones that caught the bugs I'll be reporting. :-)
>> )
>> > So, here's an introduction on how it all works:
>> >
>> > To run the tests on OpenSSL, clone BoringSSL:
>> > https://boringssl.googlesource.com/boringssl/
>> > Then patch in this change. (Click the "Download" in the upper-right for
>> > options.)
>> > https://boringssl-review.googlesource.com/#/c/7332/
>> > Then follow the instructions in the commit message.
>> >
>> > The tests themselves and the runner logic live in
>> ssl/test/runner/runner.go:
>> >
>> https://boringssl.googlesource.com/boringssl/+/22ce9b2d08a52e399bf2ab86851952d727be034d/ssl/test/runner/runner.go#922
>> >
>> > They work by running an unmodified TLS stack in a shim binary against a
>> copy
>> > of Go's. We patch our copy with options for weird behavior to test
>> against:
>> >
>> https://boringssl.googlesource.com/boringssl/+/22ce9b2d08a52e399bf2ab86851952d727be034d/ssl/test/runner/common.go#414
>> >
>> > Go and shim communicate entirely with sockets and (tons of) command-line
>> > flags, though it is slightly overfit to BoringSSL's behavior and checks
>> > error strings a lot. The shim also has options like -async mode which
>> we use
>> > on a subset of tests to stress state machine resumption. (This has
>> saved me
>> > from state machine bugs so many times.)
>> >
>> https://boringssl.googlesource.com/boringssl/+/22ce9b2d08a52e399bf2ab86851952d727be034d/ssl/test/runner/runner.go#2770
>> >
>> https://boringssl.googlesource.com/boringssl/+/22ce9b2d08a52e399bf2ab86851952d727be034d/ssl/test/bssl_shim.cc#826
>> >
>> > I hope this is useful! Bugs and patches will follow this mail, as I
>> write
>> > them up.
>> >
>> > David
>> >
>> > --
>> > openssl-dev mailing list
>> > To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev
>> >
>> --
>> openssl-dev mailing list
>> To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev
>>
>
> --
> openssl-dev mailing list
> To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev
>
>
-- 
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev

Re: [openssl-dev] Running against BoringSSL's SSL test suite

[David Benjamin]

On Wed, Mar 9, 2016 at 5:07 AM Kanaka Kotamarthy 
wrote:

> Hi
>
> I am even testing OpenSSL with BoringSSL's test cases using
> Openssl-1.1.0-pre2. Trying to find out reasons of OpenSSL's failures
> for particular cases.
>
> DTLS 1.0 session resumption has some thing wrong. If s_server started
> with -dtls and s_client -dtls1 -reconnect , session resumption is not
> being done. The reason for this may be, version negotiation for DTLS
> is done after loading previous session and check for s->version and
> s->session->version fails in tls_process_client_hello.
>

See RT #4392.
https://rt.openssl.org/Ticket/Display.html?id=4392


> And also Openssl fails with Resume-Client-NoResume cases. Do you have
> any report on which test cases do fail and reasons for the failure?
>

RT tickets 4387 through 4395 were the failures I've triaged. I'm sure
there's more things in there to look through.

I don't believe Resume-Client-NoResume fails for me. Perhaps something was
fixed between master and 1.1.0-pre2.

David


> Thank you
> Durga.
>
> On Tue, Mar 8, 2016 at 3:19 AM, David Benjamin 
> wrote:
> > Hi folks,
> >
> > So, we've by now built up a decent-sized SSL test suite in BoringSSL. I
> was
> > bored and ran it against OpenSSL master. It revealed a number of bugs.
> One
> > is https://github.com/openssl/openssl/pull/603. I'll be filing tickets
> > shortly for the remaining ones I've triaged, but I thought I'd send this
> > separately rather than duplicate it everywhere.
> >
> > Emilia also suggested there may be room to collaborate on testing. If
> > nothing else, just borrowing ideas or porting tests to/from your TLSProxy
> > setup. (Like, say, the ones that caught the bugs I'll be reporting. :-) )
> > So, here's an introduction on how it all works:
> >
> > To run the tests on OpenSSL, clone BoringSSL:
> > https://boringssl.googlesource.com/boringssl/
> > Then patch in this change. (Click the "Download" in the upper-right for
> > options.)
> > https://boringssl-review.googlesource.com/#/c/7332/
> > Then follow the instructions in the commit message.
> >
> > The tests themselves and the runner logic live in
> ssl/test/runner/runner.go:
> >
> https://boringssl.googlesource.com/boringssl/+/22ce9b2d08a52e399bf2ab86851952d727be034d/ssl/test/runner/runner.go#922
> >
> > They work by running an unmodified TLS stack in a shim binary against a
> copy
> > of Go's. We patch our copy with options for weird behavior to test
> against:
> >
> https://boringssl.googlesource.com/boringssl/+/22ce9b2d08a52e399bf2ab86851952d727be034d/ssl/test/runner/common.go#414
> >
> > Go and shim communicate entirely with sockets and (tons of) command-line
> > flags, though it is slightly overfit to BoringSSL's behavior and checks
> > error strings a lot. The shim also has options like -async mode which we
> use
> > on a subset of tests to stress state machine resumption. (This has saved
> me
> > from state machine bugs so many times.)
> >
> https://boringssl.googlesource.com/boringssl/+/22ce9b2d08a52e399bf2ab86851952d727be034d/ssl/test/runner/runner.go#2770
> >
> https://boringssl.googlesource.com/boringssl/+/22ce9b2d08a52e399bf2ab86851952d727be034d/ssl/test/bssl_shim.cc#826
> >
> > I hope this is useful! Bugs and patches will follow this mail, as I write
> > them up.
> >
> > David
> >
> > --
> > openssl-dev mailing list
> > To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev
> >
> --
> openssl-dev mailing list
> To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev
>
-- 
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev

Re: [openssl-dev] Running against BoringSSL's SSL test suite

[Kanaka Kotamarthy]

Hi

I am even testing OpenSSL with BoringSSL's test cases using
Openssl-1.1.0-pre2. Trying to find out reasons of OpenSSL's failures
for particular cases.

DTLS 1.0 session resumption has some thing wrong. If s_server started
with -dtls and s_client -dtls1 -reconnect , session resumption is not
being done. The reason for this may be, version negotiation for DTLS
is done after loading previous session and check for s->version and
s->session->version fails in tls_process_client_hello.

And also Openssl fails with Resume-Client-NoResume cases. Do you have
any report on which test cases do fail and reasons for the failure?

Thank you
Durga.

On Tue, Mar 8, 2016 at 3:19 AM, David Benjamin  wrote:
> Hi folks,
>
> So, we've by now built up a decent-sized SSL test suite in BoringSSL. I was
> bored and ran it against OpenSSL master. It revealed a number of bugs. One
> is https://github.com/openssl/openssl/pull/603. I'll be filing tickets
> shortly for the remaining ones I've triaged, but I thought I'd send this
> separately rather than duplicate it everywhere.
>
> Emilia also suggested there may be room to collaborate on testing. If
> nothing else, just borrowing ideas or porting tests to/from your TLSProxy
> setup. (Like, say, the ones that caught the bugs I'll be reporting. :-) )
> So, here's an introduction on how it all works:
>
> To run the tests on OpenSSL, clone BoringSSL:
> https://boringssl.googlesource.com/boringssl/
> Then patch in this change. (Click the "Download" in the upper-right for
> options.)
> https://boringssl-review.googlesource.com/#/c/7332/
> Then follow the instructions in the commit message.
>
> The tests themselves and the runner logic live in ssl/test/runner/runner.go:
> https://boringssl.googlesource.com/boringssl/+/22ce9b2d08a52e399bf2ab86851952d727be034d/ssl/test/runner/runner.go#922
>
> They work by running an unmodified TLS stack in a shim binary against a copy
> of Go's. We patch our copy with options for weird behavior to test against:
> https://boringssl.googlesource.com/boringssl/+/22ce9b2d08a52e399bf2ab86851952d727be034d/ssl/test/runner/common.go#414
>
> Go and shim communicate entirely with sockets and (tons of) command-line
> flags, though it is slightly overfit to BoringSSL's behavior and checks
> error strings a lot. The shim also has options like -async mode which we use
> on a subset of tests to stress state machine resumption. (This has saved me
> from state machine bugs so many times.)
> https://boringssl.googlesource.com/boringssl/+/22ce9b2d08a52e399bf2ab86851952d727be034d/ssl/test/runner/runner.go#2770
> https://boringssl.googlesource.com/boringssl/+/22ce9b2d08a52e399bf2ab86851952d727be034d/ssl/test/bssl_shim.cc#826
>
> I hope this is useful! Bugs and patches will follow this mail, as I write
> them up.
>
> David
>
> --
> openssl-dev mailing list
> To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev
>
-- 
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev

Re: [openssl-dev] Running against BoringSSL's SSL test suite

[Matt Caswell]

On 07/03/16 21:49, David Benjamin wrote:
> Hi folks,
> 
> So, we've by now built up a decent-sized SSL test suite in BoringSSL. I
> was bored and ran it against OpenSSL master. It revealed a number of
> bugs. One is https://github.com/openssl/openssl/pull/603. I'll be filing
> tickets shortly for the remaining ones I've triaged, but I thought I'd
> send this separately rather than duplicate it everywhere.

Wow! That's awesome! Thanks David.

> 
> Emilia also suggested there may be room to collaborate on testing. If
> nothing else, just borrowing ideas or porting tests to/from your
> TLSProxy setup. (Like, say, the ones that caught the bugs I'll be
> reporting. :-) ) So, here's an introduction on how it all works:
> 
> To run the tests on OpenSSL, clone BoringSSL:
> https://boringssl.googlesource.com/boringssl/
> Then patch in this change. (Click the "Download" in the upper-right for
> options.)
> https://boringssl-review.googlesource.com/#/c/7332/
> Then follow the instructions in the commit message.
> 
> The tests themselves and the runner logic live in ssl/test/runner/runner.go:
> https://boringssl.googlesource.com/boringssl/+/22ce9b2d08a52e399bf2ab86851952d727be034d/ssl/test/runner/runner.go#922
> 
> They work by running an unmodified TLS stack in a shim binary against a
> copy of Go's. We patch our copy with options for weird behavior to test
> against:
> https://boringssl.googlesource.com/boringssl/+/22ce9b2d08a52e399bf2ab86851952d727be034d/ssl/test/runner/common.go#414
> 
> Go and shim communicate entirely with sockets and (tons of) command-line
> flags, though it is slightly overfit to BoringSSL's behavior and checks
> error strings a lot. The shim also has options like -async mode which we
> use on a subset of tests to stress state machine resumption. (This has
> saved me from state machine bugs so many times.)
> https://boringssl.googlesource.com/boringssl/+/22ce9b2d08a52e399bf2ab86851952d727be034d/ssl/test/runner/runner.go#2770
> https://boringssl.googlesource.com/boringssl/+/22ce9b2d08a52e399bf2ab86851952d727be034d/ssl/test/bssl_shim.cc#826
> 
> I hope this is useful! Bugs and patches will follow this mail, as I
> write them up.

Great. We're in the final few days prior to the 1.1.0 feature freeze and
the team are working flat out at the moment. I'll try and start looking
at them once we're past that milestone later this week.

Matt

-- 
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev