Re: [dmarc-ietf] indeterminisim of ARC-Seal b= value

[Seth Blank]

On Thu, Mar 30, 2017 at 7:35 PM, Dave Crocker  wrote:
>
> So to the extent that you are sure things really are that fragile, the
> answer is not going to be a test suite or excessively demanding algorithms,
> but a re-thinking of the details, to make the implementation and deployment
> issues simpler.


Exactly; we are in agreement. Where some discussion after the last interop
panned out was that deterministic behavior in the new headers would remove
this fragility. That's why the conversation was raised to this list instead
of happening off to the side.

If the consensus here is that the matter is not worth pursuing further,
that is fine - I just want to make sure we're all talking about the same
thing.

Seth


On Thu, Mar 30, 2017 at 7:35 PM, Dave Crocker  wrote:

> On 3/30/2017 7:10 PM, Seth Blank wrote:
>
>> Dave, If we were only talking about ARC Signing messages, I'd generally
>> agree with the comments on this list.
>>
>> However, ARC is fundamentally different. It is about a chain of messages
>>
>
>
> Either you are correct, in which case ARC has been made far too fragile to
> be able to work with any serious degree of reliability at scale,
>
> or you are wrong, in which case the fact of there being a sequence of
> DKIM-ish signatures has the same requirements as for individual signatures.
>
> What you have been getting told by a range of folk with quite a lengthy
> history of DKIM and email deployment experience is in line with the latter.
>
> So to the extent that you are sure things really are that fragile, the
> answer is not going to be a test suite or excessively demanding algorithms,
> but a re-thinking of the details, to make the implementation and deployment
> issues simpler.
>
>
>
> d/
>
> --
> Dave Crocker
> Brandenburg InternetWorking
> bbiw.net
>
> ___
> dmarc mailing list
> dmarc@ietf.org
> https://www.ietf.org/mailman/listinfo/dmarc
>



-- 

[image: logo for sig file.png]

Bringing Trust to Email

Seth Blank | Head of Product for Open Source and Protocols
s...@valimail.com
+1-415-894-2724 <415-894-2724>
___
dmarc mailing list
dmarc@ietf.org
https://www.ietf.org/mailman/listinfo/dmarc

Re: [dmarc-ietf] indeterminisim of ARC-Seal b= value

[John Levine]

>My reading of this too-long message thread is that there is essentially 
>no support for making ARC's header-related issues any different from 
>DKIM's, so I encourage the chair to shut this thread down.

What he said.  Can we please limit subsequent discussion of testing
about how to test ARC as defined and implemented, not some not-ARC
that is not defined and certainly will not be implemented?

R's,
John

___
dmarc mailing list
dmarc@ietf.org
https://www.ietf.org/mailman/listinfo/dmarc

Re: [dmarc-ietf] indeterminisim of ARC-Seal b= value

[Dave Crocker]

On 3/30/2017 7:10 PM, Seth Blank wrote:

Dave, If we were only talking about ARC Signing messages, I'd generally
agree with the comments on this list.

However, ARC is fundamentally different. It is about a chain of messages



Either you are correct, in which case ARC has been made far too fragile 
to be able to work with any serious degree of reliability at scale,


or you are wrong, in which case the fact of there being a sequence of 
DKIM-ish signatures has the same requirements as for individual 
signatures.


What you have been getting told by a range of folk with quite a lengthy 
history of DKIM and email deployment experience is in line with the latter.


So to the extent that you are sure things really are that fragile, the 
answer is not going to be a test suite or excessively demanding 
algorithms, but a re-thinking of the details, to make the implementation 
and deployment issues simpler.



d/

--
Dave Crocker
Brandenburg InternetWorking
bbiw.net

___
dmarc mailing list
dmarc@ietf.org
https://www.ietf.org/mailman/listinfo/dmarc

Re: [dmarc-ietf] indeterminisim of ARC-Seal b= value

[Seth Blank]

Yes, +1000 to many test suites.

But no test suite can function without a deterministic order of fields in
the header unless it also includes an ARC implementation for
signing/validating, which is where this conversation started and why we
care so much about discussing this ordering.

On Thu, Mar 30, 2017 at 6:11 PM, HANSEN, TONY L  wrote:

> One of the reasons DKIM was successful in becoming interoperable was not
> that we had a test suite to force conformance, but instead that we had
> >multiple< test suites that were able to successfully validate DKIM
> signatures from multiple implementations. My test suite caught corner cases
> that Ned’s and Yahoo’s and Google’s did not, and vice versa for each one of
> them. (My company actually had two totally independent implementations of
> DKIM so that we could do internal testing against a different platform. It
> was just prudent to do it that way.)
>
>
>
> When there was an issue in someone’s test suite, we were then able to
> discuss whether it was an issue in the test suite that needed to be fixed,
> or it was an issue in the implementation. A single test suite is never
> sufficient – that’s one reason we prefer multiple interoperating
> implementations in order for something to be considered an internet
> standard.
>
>
>
> By the way, none of us that implemented DKIM test suites had an issue with
> the header order being determined by the sender. It makes the validation
> infinitesimally trickier than having a predetermined header order.
>
>
>
> -1000 to the suggestion of having a single conformance test suite that
> everyone lives and dies by. +1000 to multiple test suites that multiple
> people write and can be used to test against.
>
>
>
> I’m all for you, Seth and Peter, to write a test suite. But nothing you’ve
> said so far has convinced me that a predetermined header order or a single
> conformance test suite is worth pursuing. One of many, sure.
>
>
>
> Tony Hansen
>
>
>
> *From: *dmarc  on behalf of Seth Blank <
> s...@valimail.com>
> *Date: *Thursday, March 30, 2017 at 8:10 PM
> *To: *"dcroc...@bbiw.net" 
> *Cc: *"Murray S. Kucherawy" , "dmarc@ietf.org" <
> dmarc@ietf.org>, Scott Kitterman , Peter Goldstein <
> pe...@valimail.com>, "ned+dm...@mrochek.com" 
> *Subject: *Re: [dmarc-ietf] indeterminisim of ARC-Seal b= value
>
>
>
> Dave, If we were only talking about ARC Signing messages, I'd generally
> agree with the comments on this list.
>
>
>
> However, ARC is fundamentally different. It is about a chain of messages
> that validate each other. And ARC's complexity is not in the signing (where
> most of this conversation has been focused), but in this chain validation
> and the cascading issues that can be created by any member of the chain
> doing something slightly wrong. And since *everyone* in the chain except
> the initial signer is also a receiver, the final receiver simply can't make
> up for mistakes caused by a less savvy intermediary - the chain will
> already be broken and unrecoverable.
>
>
>
> ARC also has a lot of subtlety that is missable at interops because of
> this. For instance, if I'm modifying a message, I need to validate the
> incoming chain, then make my modifications, and then sign the outgoing
> message. An intermediary doing this wrong (say, validating and then signing
> after message modification) isn't a bug a final receiver can fix. And
> without a seriously complex interop with a ton of intermediaries and
> sending relationships mapped out for testing, it's not likely an error in
> any specific implementation here will be discovered until it's too late.
>
>
>
> To my understanding, working on the test suite shook these issues out
> because they were what caused problems running the test suite at the
> interop in the first place. So this is actually a good and healthy part of
> the process, and is the natural consequence of the interops. What worked
> here for DKIM is insufficient with the nuances of ARC.
>
>
>
> What we've been discussing doesn't just guarantee a test suite can be made
> that works for everyone (which is huge in itself), but constrains the
> problem space for everyone involved in an ARC chain. This is clean and
> desirable.
>
>
>
> Since the spec is not finalized, this is the perfect time to set a
> deterministic order for headers that no one has even seen before.
>
>
>
> Seth
>
>
>
>
>
> On Thu, Mar 30, 2017 at 4:15 PM, Dave Crocker  wrote:
>
> On 3/30/2017 12:13 PM, ned+dm...@mrochek.com wrote:
>
> And given the relationship with DKIM people are going to expect ARC
> to work in the same general way, maing such a change a least astonishment
> principle violation.
>
>
>
> +10.
>
> This thread has been active for 6 days.  That is 5.5 days longer than it
> should have lasted, and I'm being generous.
>
> Internet technologies succeed through interoperability testing, not
> conformance tests, or the like.  Conformance testing, and the like, are
> useful for early-stage bug-finding, but 

Re: [dmarc-ietf] indeterminisim of ARC-Seal b= value

[HANSEN, TONY L]

One of the reasons DKIM was successful in becoming interoperable was not that 
we had a test suite to force conformance, but instead that we had >multiple< 
test suites that were able to successfully validate DKIM signatures from 
multiple implementations. My test suite caught corner cases that Ned’s and 
Yahoo’s and Google’s did not, and vice versa for each one of them. (My company 
actually had two totally independent implementations of DKIM so that we could 
do internal testing against a different platform. It was just prudent to do it 
that way.)

When there was an issue in someone’s test suite, we were then able to discuss 
whether it was an issue in the test suite that needed to be fixed, or it was an 
issue in the implementation. A single test suite is never sufficient – that’s 
one reason we prefer multiple interoperating implementations in order for 
something to be considered an internet standard.

By the way, none of us that implemented DKIM test suites had an issue with the 
header order being determined by the sender. It makes the validation 
infinitesimally trickier than having a predetermined header order.

-1000 to the suggestion of having a single conformance test suite that everyone 
lives and dies by. +1000 to multiple test suites that multiple people write and 
can be used to test against.

I’m all for you, Seth and Peter, to write a test suite. But nothing you’ve said 
so far has convinced me that a predetermined header order or a single 
conformance test suite is worth pursuing. One of many, sure.

Tony Hansen

From: dmarc  on behalf of Seth Blank 
Date: Thursday, March 30, 2017 at 8:10 PM
To: "dcroc...@bbiw.net" 
Cc: "Murray S. Kucherawy" , "dmarc@ietf.org" 
, Scott Kitterman , Peter Goldstein 
, "ned+dm...@mrochek.com" 
Subject: Re: [dmarc-ietf] indeterminisim of ARC-Seal b= value

Dave, If we were only talking about ARC Signing messages, I'd generally agree 
with the comments on this list.

However, ARC is fundamentally different. It is about a chain of messages that 
validate each other. And ARC's complexity is not in the signing (where most of 
this conversation has been focused), but in this chain validation and the 
cascading issues that can be created by any member of the chain doing something 
slightly wrong. And since *everyone* in the chain except the initial signer is 
also a receiver, the final receiver simply can't make up for mistakes caused by 
a less savvy intermediary - the chain will already be broken and unrecoverable.

ARC also has a lot of subtlety that is missable at interops because of this. 
For instance, if I'm modifying a message, I need to validate the incoming 
chain, then make my modifications, and then sign the outgoing message. An 
intermediary doing this wrong (say, validating and then signing after message 
modification) isn't a bug a final receiver can fix. And without a seriously 
complex interop with a ton of intermediaries and sending relationships mapped 
out for testing, it's not likely an error in any specific implementation here 
will be discovered until it's too late.

To my understanding, working on the test suite shook these issues out because 
they were what caused problems running the test suite at the interop in the 
first place. So this is actually a good and healthy part of the process, and is 
the natural consequence of the interops. What worked here for DKIM is 
insufficient with the nuances of ARC.

What we've been discussing doesn't just guarantee a test suite can be made that 
works for everyone (which is huge in itself), but constrains the problem space 
for everyone involved in an ARC chain. This is clean and desirable.

Since the spec is not finalized, this is the perfect time to set a 
deterministic order for headers that no one has even seen before.

Seth


On Thu, Mar 30, 2017 at 4:15 PM, Dave Crocker 
mailto:d...@dcrocker.net>> wrote:
On 3/30/2017 12:13 PM, ned+dm...@mrochek.com 
wrote:
And given the relationship with DKIM people are going to expect ARC
to work in the same general way, maing such a change a least astonishment
principle violation.


+10.

This thread has been active for 6 days.  That is 5.5 days longer than it should 
have lasted, and I'm being generous.

Internet technologies succeed through interoperability testing, not conformance 
tests, or the like.  Conformance testing, and the like, are useful for 
early-stage bug-finding, but not for 'certification'.  This point has been a 
distinguishing feature of Internet technical work since before the Internet.

ARC is a small re-casting of DKIM.  DKIM has worked for quite a few years.  
Some details of ARC need to be different from DKIM, but the basic paradigm -- 
including the model of achieving field operation through interoperability 
testing -- should be the same.

My reading of this too-long message thread is that there is essentially no 
support for making ARC's header-related issues any different from DKIM'

Re: [dmarc-ietf] indeterminisim of ARC-Seal b= value

[Seth Blank]

Dave, If we were only talking about ARC Signing messages, I'd generally
agree with the comments on this list.

However, ARC is fundamentally different. It is about a chain of messages
that validate each other. And ARC's complexity is not in the signing (where
most of this conversation has been focused), but in this chain validation
and the cascading issues that can be created by any member of the chain
doing something slightly wrong. And since *everyone* in the chain except
the initial signer is also a receiver, the final receiver simply can't make
up for mistakes caused by a less savvy intermediary - the chain will
already be broken and unrecoverable.

ARC also has a lot of subtlety that is missable at interops because of
this. For instance, if I'm modifying a message, I need to validate the
incoming chain, then make my modifications, and then sign the outgoing
message. An intermediary doing this wrong (say, validating and then signing
after message modification) isn't a bug a final receiver can fix. And
without a seriously complex interop with a ton of intermediaries and
sending relationships mapped out for testing, it's not likely an error in
any specific implementation here will be discovered until it's too late.

To my understanding, working on the test suite shook these issues out
because they were what caused problems running the test suite at the
interop in the first place. So this is actually a good and healthy part of
the process, and is the natural consequence of the interops. What worked
here for DKIM is insufficient with the nuances of ARC.

What we've been discussing doesn't just guarantee a test suite can be made
that works for everyone (which is huge in itself), but constrains the
problem space for everyone involved in an ARC chain. This is clean and
desirable.

Since the spec is not finalized, this is the perfect time to set a
deterministic order for headers that no one has even seen before.

Seth


On Thu, Mar 30, 2017 at 4:15 PM, Dave Crocker  wrote:

> On 3/30/2017 12:13 PM, ned+dm...@mrochek.com wrote:
>
>> And given the relationship with DKIM people are going to expect ARC
>> to work in the same general way, maing such a change a least astonishment
>> principle violation.
>>
>
>
> +10.
>
> This thread has been active for 6 days.  That is 5.5 days longer than it
> should have lasted, and I'm being generous.
>
> Internet technologies succeed through interoperability testing, not
> conformance tests, or the like.  Conformance testing, and the like, are
> useful for early-stage bug-finding, but not for 'certification'.  This
> point has been a distinguishing feature of Internet technical work since
> before the Internet.
>
> ARC is a small re-casting of DKIM.  DKIM has worked for quite a few
> years.  Some details of ARC need to be different from DKIM, but the basic
> paradigm -- including the model of achieving field operation through
> interoperability testing -- should be the same.
>
> My reading of this too-long message thread is that there is essentially no
> support for making ARC's header-related issues any different from DKIM's,
> so I encourage the chair to shut this thread down.
>
> d/
>
> --
> Dave Crocker
> Brandenburg InternetWorking
> bbiw.net
>
>
> ___
> dmarc mailing list
> dmarc@ietf.org
> https://www.ietf.org/mailman/listinfo/dmarc
>



-- 

[image: logo for sig file.png]

Bringing Trust to Email

Seth Blank | Head of Product for Open Source and Protocols
s...@valimail.com
+1-415-894-2724 <415-894-2724>
___
dmarc mailing list
dmarc@ietf.org
https://www.ietf.org/mailman/listinfo/dmarc

Re: [dmarc-ietf] indeterminisim of ARC-Seal b= value

[Dave Crocker]

On 3/30/2017 12:13 PM, ned+dm...@mrochek.com wrote:

And given the relationship with DKIM people are going to expect ARC
to work in the same general way, maing such a change a least astonishment
principle violation.



+10.

This thread has been active for 6 days.  That is 5.5 days longer than it 
should have lasted, and I'm being generous.


Internet technologies succeed through interoperability testing, not 
conformance tests, or the like.  Conformance testing, and the like, are 
useful for early-stage bug-finding, but not for 'certification'.  This 
point has been a distinguishing feature of Internet technical work since 
before the Internet.


ARC is a small re-casting of DKIM.  DKIM has worked for quite a few 
years.  Some details of ARC need to be different from DKIM, but the 
basic paradigm -- including the model of achieving field operation 
through interoperability testing -- should be the same.


My reading of this too-long message thread is that there is essentially 
no support for making ARC's header-related issues any different from 
DKIM's, so I encourage the chair to shut this thread down.


d/

--
Dave Crocker
Brandenburg InternetWorking
bbiw.net

___
dmarc mailing list
dmarc@ietf.org
https://www.ietf.org/mailman/listinfo/dmarc

Re: [dmarc-ietf] indeterminisim of ARC-Seal b= value

[Seth Blank]

On Thu, Mar 30, 2017 at 7:01 AM,  wrote:

> > On Mon, Mar 27, 2017 at 11:49 PM, Scott Kitterman 
> > That sort of organic validation of implementations seemed to work for
> > DKIM.  The unknown here is whether ARC will be as successful with that
> > model.
>
> Exactly. And given the success of the approach with DKIM, the burden is
> on those proposing to use a different model for ARC to explain why it will
> work better.
>
> And note that "this kind of test becomes possible" is not sufficient. All
> kinds of tests become possible when you add constraints - the question is
> what sort of real world problem will those additional tests find.
>


ARC is fundamentally different than DKIM in several key ways. If we were
only talking about ARC Signing messages, I'd generally agree with the
comments on this list.

However, ARC is fundamentally different. It is about a chain of messages
that validate each other. And ARC's complexity is not in the signing (where
most of this conversation has been focused), but in this chain validation
and the cascading issues that can be created by any member of the chain
doing something slightly wrong. And since *everyone* in the chain except
the initial signer is also a receiver, the final receiver simply can't make
up for mistakes caused by a less savvy intermediary - the chain will
already be broken and unrecoverable.

ARC also has a lot of subtlety that is missable at interops because of
this. For instance, if I'm modifying a message, I need to validate the
incoming chain, then make my modifications, and then sign the outgoing
message. An intermediary doing this wrong (say, validating and then signing
after message modification) isn't a bug a final receiver can fix. And
without a seriously complex interop with a ton of intermediaries and
sending relationships mapped out for testing, it's not likely an error in
any specific implementation here will be discovered until it's too late.

To my understanding, working on the test suite shook these issues out
because they were what caused problems running the test suite at the
interop in the first place. So this is actually a good and healthy part of
the process, and is the natural consequence of the interops. What worked
here for DKIM is insufficient with the nuances of ARC.

What we've been discussing doesn't just guarantee a test suite can be made
that works for everyone (which is huge in itself), but constrains the
problem space for everyone involved in an ARC chain. This is clean and
desirable.

Since the spec isn't finalized, this is the perfect time to impose some
order that isn't necessary for DKIM but is critical when we're talking
about multiple operators working together over multiple hops as part of a
chain of trust.

Seth

-- 

[image: logo for sig file.png]

Bringing Trust to Email

Seth Blank | Head of Product for Open Source and Protocols
s...@valimail.com
+1-415-894-2724 <415-894-2724>
___
dmarc mailing list
dmarc@ietf.org
https://www.ietf.org/mailman/listinfo/dmarc

Re: [dmarc-ietf] indeterminisim of ARC-Seal b= value

[ned+dmarc]

> >
> > > I think Peter's proposal (and he can correct me if I'm wrong) is only to
> > > constrain signers.  Verifiers are still expected to handle everything
> > weird
> > > that the infrastructure might do.
> > This proposal creates a situation where verifiers are required to support
> > something that no compliant signer will generate. The result will be that a
> > test suite with reasonable coverage cannot be generated using compliant
> > software.
> > And since most people are not going to bother to create an incompliant
> > signer
> > specifically for purposes of testing, if anything this will create, rather
> > than
> > elimnate, interoperability problems.
> >

> Just to be clear, this is not what I was proposing.  Specifically, I don't
> think the Robustness Principle
>  applies in this case.
> Verification should check the RFC defined spacing, ordering, etc. of the
> tag/value pairs and reject non-compliant values

> As this is a new protocol, and these headers are all new and used
> exclusively for ARC, and for the reasons mentioned by Ned, there's no
> reason to be lenient in what is accepted.

On the contrary, there are very good reasons. People aren't going to magic
up implementations of ARC out of nothing; they will undoubtedly use
existing DKIM code as a starting point. Implementing an ordering
requirement on top of such code could get interesting depending on how
it works.

And given the relationship with DKIM people are going to expect ARC
to work in the same general way, maing such a change a least astonishment
principle violation.

> > That sort of organic validation of implementations seemed to work for
> > > DKIM.  The unknown here is whether ARC will be as successful with that
> > > model.

> > Exactly. And given the success of the approach with DKIM, the burden is
> > on those proposing to use a different model for ARC to explain why it will
> > work better.


> A few quick comments on this.

> As per my earlier email, I have a lot less confidence in how well this
> 'seemed to work' than some of the other participants in this thread.  One
> of the benefits of DMARC re:DKIM is it has exposed some issues with DKIM
> implementations that were not obvious prior to DMARC evaluation.  Without
> DMARC reporting some of the problems with certain DKIM implementations were
> simply not very visible.  So I don't actually think the DKIM model worked
> as well as is being asserted.

Which of these problems had anything to do with field ordering? Which of
these problems would have been exposed by requiring a specific
order?

> Second, the existence of such a test suite would have had material benefits
> just within the last ~18 months over which ARC implementations have been
> developed.  We know this because of the immediate benefits the current test
> suite (which does impose a canonical ordering, spacing, etc.) yielded
> almost immediately.

> The suite exposed a few minor but significant issues with the Python
> implementation that was submitted to dkimpy.  Similarly, it exposed that
> the current OpenARC signing functionality is in a much less mature state
> than was generally understood.  In terms of RFC definition, it exposed the
> fact that the current RFC as written required support for insecure 512 bit
> keys - and triggered a discussion on this very list.  In our internal
> development it's forced us to look hard at some of the corner cases and
> raise them on this list or at interops.  Note that most of these issues had
> escaped detection through multiple interops.

> None of the above is meant to insult the associated developers or spec
> authors.  Their contributions have been essential, and the skill and
> dedication they've brought to their tasks is greatly appreciated.  It is
> simply meant to demonstrate the material benefit that this kind of testing
> has already brought.  This is what automated tests do - bring problems to
> light quickly.

> So yes, I think this approach works better.  Much better.

Simply put, I don't. I'm strongly opposed to this, to the point where I'm
likely to significantly deprioritize implementation of this proposal because of
it.

I have better things to do that waste my time watching the sorts of bugs this
is going to cause get shaken out.

Ned

___
dmarc mailing list
dmarc@ietf.org
https://www.ietf.org/mailman/listinfo/dmarc

Re: [dmarc-ietf] indeterminisim of ARC-Seal b= value

[Peter Goldstein]

>
> > I think Peter's proposal (and he can correct me if I'm wrong) is only to
> > constrain signers.  Verifiers are still expected to handle everything
> weird
> > that the infrastructure might do.
> This proposal creates a situation where verifiers are required to support
> something that no compliant signer will generate. The result will be that a
> test suite with reasonable coverage cannot be generated using compliant
> software.
> And since most people are not going to bother to create an incompliant
> signer
> specifically for purposes of testing, if anything this will create, rather
> than
> elimnate, interoperability problems.
>

Just to be clear, this is not what I was proposing.  Specifically, I don't
think the Robustness Principle
 applies in this case.
Verification should check the RFC defined spacing, ordering, etc. of the
tag/value pairs and reject non-compliant values

As this is a new protocol, and these headers are all new and used
exclusively for ARC, and for the reasons mentioned by Ned, there's no
reason to be lenient in what is accepted.

> That sort of organic validation of implementations seemed to work for
> > DKIM.  The unknown here is whether ARC will be as successful with that
> > model.
> Exactly. And given the success of the approach with DKIM, the burden is
> on those proposing to use a different model for ARC to explain why it will
> work better.


A few quick comments on this.

As per my earlier email, I have a lot less confidence in how well this
'seemed to work' than some of the other participants in this thread.  One
of the benefits of DMARC re:DKIM is it has exposed some issues with DKIM
implementations that were not obvious prior to DMARC evaluation.  Without
DMARC reporting some of the problems with certain DKIM implementations were
simply not very visible.  So I don't actually think the DKIM model worked
as well as is being asserted.

Second, the existence of such a test suite would have had material benefits
just within the last ~18 months over which ARC implementations have been
developed.  We know this because of the immediate benefits the current test
suite (which does impose a canonical ordering, spacing, etc.) yielded
almost immediately.

The suite exposed a few minor but significant issues with the Python
implementation that was submitted to dkimpy.  Similarly, it exposed that
the current OpenARC signing functionality is in a much less mature state
than was generally understood.  In terms of RFC definition, it exposed the
fact that the current RFC as written required support for insecure 512 bit
keys - and triggered a discussion on this very list.  In our internal
development it's forced us to look hard at some of the corner cases and
raise them on this list or at interops.  Note that most of these issues had
escaped detection through multiple interops.

None of the above is meant to insult the associated developers or spec
authors.  Their contributions have been essential, and the skill and
dedication they've brought to their tasks is greatly appreciated.  It is
simply meant to demonstrate the material benefit that this kind of testing
has already brought.  This is what automated tests do - bring problems to
light quickly.

So yes, I think this approach works better.  Much better.

Best,

Peter


>

-- 


[image: logo for sig file.png]

Bringing Trust to Email

Peter Goldstein | CTO & Co-Founder

pe...@valimail.com
+1.415.793.5783
___
dmarc mailing list
dmarc@ietf.org
https://www.ietf.org/mailman/listinfo/dmarc

Re: [dmarc-ietf] indeterminisim of ARC-Seal b= value

[Peter Goldstein]

>
> I agree that it's impossible to design a signer test suite that confirms
> correct output, without doing crytpo checks of its own with a known-good
> verifier, unless we nail down the syntax the output will use.


Great.  :)  I wasn't sure there was agreement on that in the wider thread.


> For this to work, we'd have to mandate a lot of things, including at least
> these:
> - the order of the tags as presented to the hash algorithm
> - which tags will be present (note that many are not required, including
> "t=")
> - the specific values they will all contain
> - for ARC-Message-Signature, which canonicalization will be used
> - the spacing between the tags; since "relaxed" header canonicalization
> compresses spaces but does not add them, "a=foo;b=foo" is not the same as
> "a=foo; b=foo", but "a=foo;\r\n\tb=foo" is
> - similarly, how signature fields will be wrapped (if at all)
> - what signing key will be used
> - the body content to be signed
> - the header content to be signed
> - the set of header fields that will be signed (which becomes "h=")
> It certainly removes many variables to nail down a test in this way, and
> it's faster to run tests that don't require crypto functions or a
> known-good verifier to confirm.  To be honest, I can't remember if we ever
> considered this sort of approach during the development of DKIM.  I don't
> think we did, and after an admittedly brief search I couldn't find anything
> in the archives.  I think when canonicalization was developed, it was plain
> that tag order wouldn't matter to the verifier, and that was sort of the
> end of it.


So while it is true for a given test case you would need to constrain all
of these items, this is a much larger list than would need to be specified
as part of the proposal under consideration.

Many of these items are already constrained by the controlling RFC (ARC or
DKIM) and the other values specified in the signature.  For example, the
order in which different header fields are combined and hashed is defined
as a function of the tag/value pairs in the signature (i.e in DKIM
signature the 'b' value is impacted by the choice of values for the 'a',
'c','h', etc. tag values).  That ordering is therefore an input to the test
case.  The input message content (headers + body), the signing key(s), etc.
would all also be input to a particular test case.

A test suite should handle this by having test cases that exercise the
library with different inputs that give good coverage across allowed
variations.  That has the side benefit of allowing the test suite
implementer(s) to easily develop targeted test cases that cover those
specific variations in isolation, making it easier to find bugs.

That said, some of these items in the list (intertag spacing, required
tags, canonical values for some of the tags) would need to be constrained
as part of this proposal.  This is clearly a change, but it's a change to a
draft standard that has a small # of implementers and should be relatively
easy to lock down.  And I think it's far less of a change than some others
on this list do.

The obvious counter-argument here is that DKIM has been successful without
> ever being strict about what a signature looks like.  ...  We reached a
> point where we had organically developed a bunch of "good" implementations
> based on the fact that they all appeared to agree with each other.


I think the bigger point being argued is that DKIM has been successful
without ever having a global test suite, because that's the impact of not
making this choice (or committing to a global reference implementation).

Which is mostly true, but frankly not entirely true.  I'm aware of at least
two major vendors who have production DKIM signing implementations with
edge case bugs that would've been caught by such a suite.  Those
implementations made it through development, passed whatever implementation
specific test cases the developers built, and made it to production.
That's what a good test suite helps you avoid.

Since the canonicalization algorithms in ARC are the same as the ones in
> DKIM, and the tag=value and key syntaxes are also the same, we've got a lot
> of concepts and code being recycled here.  I'm not sure how much new
> fragility we should really be concerned about.  I'll know more as I
> complete my implementation.


Paul's point that video and security protocols are traditionally much more
> rigid is of course quite true, and I presume he's thinking of things like
> the headers of DNS, IP, TCP, etc.  But I note that those are based on a
> different model where fields have fixed sizes and predictable positions.
> That's antithetical to email, however, where the fields come in arbitrary
> order, and the only way you know what a given header means is that its name
> precedes its value.  There can even be duplication.  And from one MTA to
> another, header fields can be added, removed, or rearranged.  That's
> impossible in a rigid header definition, so it

Re: [dmarc-ietf] indeterminisim of ARC-Seal b= value

[ned+dmarc]

> On Mon, Mar 27, 2017 at 11:49 PM, Scott Kitterman 
> wrote:

> > What's more difficult to is identify all the ways that things get
> > reordered,
> > mangled, etc as they transit the various elements of the mail architecture.
> > If you over specify the allowed order, aren't you risking increasing the
> > brittleness of the solution.
> >

> I think Peter's proposal (and he can correct me if I'm wrong) is only to
> constrain signers.  Verifiers are still expected to handle everything weird
> that the infrastructure might do.

This proposal creates a situation where verifiers are required to support
something that no compliant signer will generate. The result will be that a
test suite with reasonable coverage cannot be generated using compliant
software.

And since most people are not going to bother to create an incompliant signer
specifically for purposes of testing, if anything this will create, rather than
elimnate, interoperability problems.

> > If test cases are automatically generated, then they are cheap and we
> > shouldn't worry about unduly constraining things to keep the number
> > small.  As
> > long as, at some point, the test cases are validated against multiple
> > implementations, I think it's fine.  If you've got a handful of
> > implementations, then the risk of them all having the same bug is pretty
> > low.
> >

> That sort of organic validation of implementations seemed to work for
> DKIM.  The unknown here is whether ARC will be as successful with that
> model.

Exactly. And given the success of the approach with DKIM, the burden is
on those proposing to use a different model for ARC to explain why it will
work better.

And note that "this kind of test becomes possible" is not sufficient. All
kinds of tests become possible when you add constraints - the question is
what sort of real world problem will those additional tests find.

Ned

___
dmarc mailing list
dmarc@ietf.org
https://www.ietf.org/mailman/listinfo/dmarc

Re: [dmarc-ietf] indeterminisim of ARC-Seal b= value

[Murray S. Kucherawy]

On Mon, Mar 27, 2017 at 11:49 PM, Scott Kitterman 
wrote:

> What's more difficult to is identify all the ways that things get
> reordered,
> mangled, etc as they transit the various elements of the mail architecture.
> If you over specify the allowed order, aren't you risking increasing the
> brittleness of the solution.
>

I think Peter's proposal (and he can correct me if I'm wrong) is only to
constrain signers.  Verifiers are still expected to handle everything weird
that the infrastructure might do.


> If test cases are automatically generated, then they are cheap and we
> shouldn't worry about unduly constraining things to keep the number
> small.  As
> long as, at some point, the test cases are validated against multiple
> implementations, I think it's fine.  If you've got a handful of
> implementations, then the risk of them all having the same bug is pretty
> low.
>

That sort of organic validation of implementations seemed to work for
DKIM.  The unknown here is whether ARC will be as successful with that
model.

-MSK
___
dmarc mailing list
dmarc@ietf.org
https://www.ietf.org/mailman/listinfo/dmarc

Re: [dmarc-ietf] indeterminisim of ARC-Seal b= value

[Murray S. Kucherawy]

On Mon, Mar 27, 2017 at 1:06 PM, Brandon Long  wrote:

> What does validating the exact signature generated benefit us over just
> verifying that the signature verifies?
>

The most obvious benefit I can think of is that the output of signing is
entirely deterministic.  You could test it with strcmp() on a disconnected
computer, rather than invoking a complete ARC verifier that might need DNS
and would have to invoke crypto functions.

-MSK
___
dmarc mailing list
dmarc@ietf.org
https://www.ietf.org/mailman/listinfo/dmarc

Re: [dmarc-ietf] indeterminisim of ARC-Seal b= value

[Murray S. Kucherawy]

On Sun, Mar 26, 2017 at 11:25 PM, Peter Goldstein 
wrote:

> Second, the premise that "it's not hard to write test code..." is simply
> not true.  What would be required to actually write such code would be
> either a) pick a preferred ordering/formatting for these tags, and only
> have the tests pass for that ordering or b) include a complete ARC
> verifying system in the test code.  The first is what OpenDKIM did for its
> specs (see below).  The latter is possible, but seems ill-advised for a
> test suite.
>

OpenDKIM does both.  At some point when it was still "dkim-milter", when it
came time to write tests (because at one point there were none), I mainly
wanted to assure that commits made didn't cause any regressions;
correctness could be affirmed in other ways like at interop events or
through other implementers' auto-responders.  So I picked a tag ordering I
liked, generated a bunch of signatures with it, and then wrote tests that
did two things with them:

1) Confirmed that the signing code produces those specific signatures given
fixed inputs (timestamp, key, selector, domain, canonicalization, etc.).
This was inherently a tautology at the time I generated them of course, but
my goal was just to ensure no regressions were introduced.  I didn't know
for sure that the output was right, but I knew it would be consistent.

2) Since OpenDKIM conveniently includes a verifying capability, there are
some tests that instantiate the library to produce a signature, then
re-instantiate the library to verify it.  Obviously this doesn't prove that
its signatures are correct, but they do prove that its code is consistent
with itself, and again that changes made later don't break verifying.

None of this was done with an eye toward making my test suite applicable to
other implementations.  It's only for my own.  What we're talking about
here is different: coming up with tests that could be applied to any
implementation, not to a specific one, so long as they agree on fixed
syntax and inputs.

To see that this is the case, it's useful to look at two earlier email
> standards and their corresponding test suites.  SPF has a language-agnostic
> test suite (http://www.openspf.org/Test_Suite) that makes it easy to test
> implementations in any language.  I've done it myself, and it requires
> minimal effort - https://github.com/ValiMail/
> coppertone/tree/master/spec/open_spf.  I can tell you from personal
> experience that the test suite was really helpful in ensuring the
> implementation was correct in all cases.
>

An inherent difference between SPF and DKIM is that the order of the tokens
in SPF is significant.  In DKIM, reordering tags changes nothing about the
meaning of the header field, though obviously it alters the resulting
header hash.  In contrast, the order of tags in SPF is necessarily
specified.

-MSK
___
dmarc mailing list
dmarc@ietf.org
https://www.ietf.org/mailman/listinfo/dmarc

Re: [dmarc-ietf] indeterminisim of ARC-Seal b= value

[Murray S. Kucherawy]

I have to catch up on the rest of this thread still, but I wanted to chime
in here to get started:

On Sun, Mar 26, 2017 at 5:23 PM, Gene Shuman  wrote:

> Ah that had slipped my mind & is a good point.  However, I think the issue
> here is generally that ARC is more complex protocol than DKIM and therefore
> it's more important to reduce ambiguity & increase standardization while we
> have the chance.  I think this is generally a good idea from a security
> perspective, however this is mostly relevant with respect to testing &
> validation, as ensuring cross-compatibility is a much bigger challenge.
> It's even more important than it was with DKIM to have a test suite that
> can verify signing behavior.  If we don't agree on any sort of standard, a
> test suite will need to select a preferred format for the ARC headers &
> will fail all implementations that don't meet this form.  We've discussed
> this with Murray, and he agrees with this conclusion.
>

I agree that it's impossible to design a signer test suite that confirms
correct output, without doing crytpo checks of its own with a known-good
verifier, unless we nail down the syntax the output will use.  For this to
work, we'd have to mandate a lot of things, including at least these:

- the order of the tags as presented to the hash algorithm
- which tags will be present (note that many are not required, including
"t=")
- the specific values they will all contain
- for ARC-Message-Signature, which canonicalization will be used
- the spacing between the tags; since "relaxed" header canonicalization
compresses spaces but does not add them, "a=foo;b=foo" is not the same as
"a=foo; b=foo", but "a=foo;\r\n\tb=foo" is
- similarly, how signature fields will be wrapped (if at all)
- what signing key will be used
- the body content to be signed
- the header content to be signed
- the set of header fields that will be signed (which becomes "h=")

It certainly removes many variables to nail down a test in this way, and
it's faster to run tests that don't require crypto functions or a
known-good verifier to confirm.  To be honest, I can't remember if we ever
considered this sort of approach during the development of DKIM.  I don't
think we did, and after an admittedly brief search I couldn't find anything
in the archives.  I think when canonicalization was developed, it was plain
that tag order wouldn't matter to the verifier, and that was sort of the
end of it.

The obvious counter-argument here is that DKIM has been successful without
ever being strict about what a signature looks like.  Our interop event in
Dallas way-back-when consisted solely of implementer "A" sending mail at
implementer "B" and seeing what passed and what failed, and digging into
the failure cases to see if they're bugs in code or bugs in the
specification.  Independent of interop events, various implementers also
set up auto-responders that would verify an arriving message, then re-sign
it and send it back so the originator can test its verification.  Any
failure compelled us to implement debugging tools and exchange information
so we could work out the kinks, either in the spec or in the
implementation(s), until all participants were consistent.  We reached a
point where we had organically developed a bunch of "good" implementations
based on the fact that they all appeared to agree with each other.

Since the canonicalization algorithms in ARC are the same as the ones in
DKIM, and the tag=value and key syntaxes are also the same, we've got a lot
of concepts and code being recycled here.  I'm not sure how much new
fragility we should really be concerned about.  I'll know more as I
complete my implementation.

Paul's point that video and security protocols are traditionally much more
rigid is of course quite true, and I presume he's thinking of things like
the headers of DNS, IP, TCP, etc.  But I note that those are based on a
different model where fields have fixed sizes and predictable positions.
That's antithetical to email, however, where the fields come in arbitrary
order, and the only way you know what a given header means is that its name
precedes its value.  There can even be duplication.  And from one MTA to
another, header fields can be added, removed, or rearranged.  That's
impossible in a rigid header definition, so it's perhaps no surprise that
this community isn't exactly warm to the idea.

The question, then, is whether this is a desirable path to follow: Is the
value of a quick evaluation of a deterministic signer high enough to
surrender the flexibility of DKIM style of tag spacing and ordering?  And,
perhaps more importantly, what's the cost of being wrong later?

-MSK
___
dmarc mailing list
dmarc@ietf.org
https://www.ietf.org/mailman/listinfo/dmarc