Re: [commons-crypto] OpenSSL 3.x, FIPS, digests, and HMAC

[Alex Remily]

Gary,

Can you have a look at this?

https://github.com/apache/commons-crypto/pull/242

Alex

On Sun, Aug 27, 2023 at 8:38 AM Gary Gregory  wrote:

> On Fri, Aug 25, 2023 at 9:48 PM Alex Remily  wrote:
> >
> >  Breakup
> > the current module into different maven modules? Not support both?>
> >
> > Agreed.  Just to provide some history, when I was working on the 1.1.x
> > upgrade I was guided by commons committer Marcelo Vanzin.  Marcelo
> required
> > a design that supported runtime discovery of the underlying openssl
> API.  I
> > don't recall all of the rationale for the requirement, but he insisted
> that
> > any commons-crypto upgrade must support legacy and current versions of
> > openssl transparently to the calling program.  The end result is what we
> > have now.  I don't recall where in the code commons-crypto makes the
> > underlying version checks, at library initialization time or when a
> > specific function is called, but the end result is that users need only
> > download the latest commons-crypto release regardless of their underlying
> > openssl API--as long as they are running supported openssl versions.
> >
> > In my view there are a few open questions regarding the current approach
> as
> > compared to an API-specific one.  One, what is the performance penalty
> > associated with the dynamic version checks?  Two, how much complexity
> does
> > it introduce into the codebase?  Finally, what was the use case that
> drove
> > the runtime checking requirement?  Marcelo could answer the last
> question.
> > I don't know if he is still involved in the community (I haven't seen him
> > around for awhile.  IIRC, he was primarily a Spark committer).
> >
> > Another consideration is the FIPS certification.  I work in a heavily
> > regulated industry and FIPS is a real constraint.  I haven't personally
> > encountered a requirement to deploy a FIPS compliant openssl in our
> > application code, but it's probably just a matter of time.  In terms of
> > expanding our user base, it may make sense to provide that capability.
> It
> > doesn't seem to be generally available from an existing provider.
> >
> > Regarding message digests/HMAC, I question whether the performance gain
> > from native code would significantly outperform some of the modern JCA
> > providers.  As Matt Sicker pointed out, there are other implementations
> > supported by major vendors, like AWS, that may be as fast as a JNI
> wrapper
> > on OpenSSL, or at least close enough not to bother with the added
> > complexity of the native stuff.  The only way I know to answer that
> > question is to write the code and run a load test comparison.
> >
> >
> https://aws.amazon.com/blogs/opensource/introducing-amazon-corretto-crypto-provider-accp/
> >
> > In what forum should we discuss these issues?
>
> This developer's mailing list is where all decisions MUST be recorded,
> so it makes sense that discussions occur here. We can also use Jira
> tickets for specific tasks. Different discussions can use these
> different forums as we best see fit. I would not recommend using Slack
> as that data does or will eventually disappear. I do use Slack as a
> convenience at times for reminding people/channels to vote or pay
> attention to something happening in Jira or on a mailing list.
>
> > Are we limited to this
> > distro or do we have other options?
>
> I'm not sure what this means.
>
> > How do we form teams?
> I'm not sure what that means in the context of Apache. Apache has no
> notion of teams that I am aware of. We have Projects like Apache
> Commons, and Commons has Components like Commons Crypto.
> We are all on this mailing list, we can work on whatever we want, and
> we can use Jira to track tasks. If someone wants to work on a task,
> they can announce it here or in a Jira ticket, to avoid clashes or
> just be informative.
>
> > What is our
> > governance model?
> The same as any Apache project with the usual leeway allowed to each PMC.
>
>  > How do we make decisions?
> The same as any Apache project with the usual leeway allowed to each PMC.
> See https://httpd.apache.org/dev/guidelines.html
> A project can come up with it's own processes, for example see
> https://www.apache.org/foundation/voting.html
>
> Gary
>
> >
> > FYI:  There's already issues on the backlog for OpenSSL 3 and HMAC:
> > https://issues.apache.org/jira/browse/CRYPTO-165
> > https://issues.apache.org/jira/browse/CRYPTO-164
> >
> > Alex
> >
> >
> > On Wed, Aug 23, 2023 at 10:21 PM Gary Gregory 
> > wrote:
> >
> > > That would be great. I think this is worth the effort. A big item to
> > > consider is if and how 1.1 vs 3.0 should be handled. Breakup the
> current
> > > module into different maven modules? Not support both?
> > >
> > > Gary
> > >
> > > On Wed, Aug 23, 2023, 8:37 PM Alex Remily 
> wrote:
> > >
> > > >  learn
> > > > how to implement message digests and HMAC on top of OpenSSL 3.0.8.>
> > > >
> > > > Implementing the OpenSSL 3 API and exposing OpenSSL 

Re: [commons-crypto] OpenSSL 3.x, FIPS, digests, and HMAC

[Alex Remily]

Given Marcelo's response, I think it makes sense to retain support for
1.1.x, add support for 3.0.x using dynamic version discovery, and drop
support for anything older than 1.1.  This would align us with the openssl
LTS versions.  Looks like 3.1.x isn't FIPS validated.

https://www.openssl.org/source/

Thoughts?

Alex

On Sun, Aug 27, 2023 at 8:38 AM Gary Gregory  wrote:

> On Fri, Aug 25, 2023 at 9:48 PM Alex Remily  wrote:
> >
> >  Breakup
> > the current module into different maven modules? Not support both?>
> >
> > Agreed.  Just to provide some history, when I was working on the 1.1.x
> > upgrade I was guided by commons committer Marcelo Vanzin.  Marcelo
> required
> > a design that supported runtime discovery of the underlying openssl
> API.  I
> > don't recall all of the rationale for the requirement, but he insisted
> that
> > any commons-crypto upgrade must support legacy and current versions of
> > openssl transparently to the calling program.  The end result is what we
> > have now.  I don't recall where in the code commons-crypto makes the
> > underlying version checks, at library initialization time or when a
> > specific function is called, but the end result is that users need only
> > download the latest commons-crypto release regardless of their underlying
> > openssl API--as long as they are running supported openssl versions.
> >
> > In my view there are a few open questions regarding the current approach
> as
> > compared to an API-specific one.  One, what is the performance penalty
> > associated with the dynamic version checks?  Two, how much complexity
> does
> > it introduce into the codebase?  Finally, what was the use case that
> drove
> > the runtime checking requirement?  Marcelo could answer the last
> question.
> > I don't know if he is still involved in the community (I haven't seen him
> > around for awhile.  IIRC, he was primarily a Spark committer).
> >
> > Another consideration is the FIPS certification.  I work in a heavily
> > regulated industry and FIPS is a real constraint.  I haven't personally
> > encountered a requirement to deploy a FIPS compliant openssl in our
> > application code, but it's probably just a matter of time.  In terms of
> > expanding our user base, it may make sense to provide that capability.
> It
> > doesn't seem to be generally available from an existing provider.
> >
> > Regarding message digests/HMAC, I question whether the performance gain
> > from native code would significantly outperform some of the modern JCA
> > providers.  As Matt Sicker pointed out, there are other implementations
> > supported by major vendors, like AWS, that may be as fast as a JNI
> wrapper
> > on OpenSSL, or at least close enough not to bother with the added
> > complexity of the native stuff.  The only way I know to answer that
> > question is to write the code and run a load test comparison.
> >
> >
> https://aws.amazon.com/blogs/opensource/introducing-amazon-corretto-crypto-provider-accp/
> >
> > In what forum should we discuss these issues?
>
> This developer's mailing list is where all decisions MUST be recorded,
> so it makes sense that discussions occur here. We can also use Jira
> tickets for specific tasks. Different discussions can use these
> different forums as we best see fit. I would not recommend using Slack
> as that data does or will eventually disappear. I do use Slack as a
> convenience at times for reminding people/channels to vote or pay
> attention to something happening in Jira or on a mailing list.
>
> > Are we limited to this
> > distro or do we have other options?
>
> I'm not sure what this means.
>
> > How do we form teams?
> I'm not sure what that means in the context of Apache. Apache has no
> notion of teams that I am aware of. We have Projects like Apache
> Commons, and Commons has Components like Commons Crypto.
> We are all on this mailing list, we can work on whatever we want, and
> we can use Jira to track tasks. If someone wants to work on a task,
> they can announce it here or in a Jira ticket, to avoid clashes or
> just be informative.
>
> > What is our
> > governance model?
> The same as any Apache project with the usual leeway allowed to each PMC.
>
>  > How do we make decisions?
> The same as any Apache project with the usual leeway allowed to each PMC.
> See https://httpd.apache.org/dev/guidelines.html
> A project can come up with it's own processes, for example see
> https://www.apache.org/foundation/voting.html
>
> Gary
>
> >
> > FYI:  There's already issues on the backlog for OpenSSL 3 and HMAC:
> > https://issues.apache.org/jira/browse/CRYPTO-165
> > https://issues.apache.org/jira/browse/CRYPTO-164
> >
> > Alex
> >
> >
> > On Wed, Aug 23, 2023 at 10:21 PM Gary Gregory 
> > wrote:
> >
> > > That would be great. I think this is worth the effort. A big item to
> > > consider is if and how 1.1 vs 3.0 should be handled. Breakup the
> current
> > > module into different maven modules? Not support both?
> > >
> > > Gary
> > 

Re: [commons-crypto] OpenSSL 3.x, FIPS, digests, and HMAC

[Gary Gregory]

On Fri, Aug 25, 2023 at 9:48 PM Alex Remily  wrote:
>
>  the current module into different maven modules? Not support both?>
>
> Agreed.  Just to provide some history, when I was working on the 1.1.x
> upgrade I was guided by commons committer Marcelo Vanzin.  Marcelo required
> a design that supported runtime discovery of the underlying openssl API.  I
> don't recall all of the rationale for the requirement, but he insisted that
> any commons-crypto upgrade must support legacy and current versions of
> openssl transparently to the calling program.  The end result is what we
> have now.  I don't recall where in the code commons-crypto makes the
> underlying version checks, at library initialization time or when a
> specific function is called, but the end result is that users need only
> download the latest commons-crypto release regardless of their underlying
> openssl API--as long as they are running supported openssl versions.
>
> In my view there are a few open questions regarding the current approach as
> compared to an API-specific one.  One, what is the performance penalty
> associated with the dynamic version checks?  Two, how much complexity does
> it introduce into the codebase?  Finally, what was the use case that drove
> the runtime checking requirement?  Marcelo could answer the last question.
> I don't know if he is still involved in the community (I haven't seen him
> around for awhile.  IIRC, he was primarily a Spark committer).
>
> Another consideration is the FIPS certification.  I work in a heavily
> regulated industry and FIPS is a real constraint.  I haven't personally
> encountered a requirement to deploy a FIPS compliant openssl in our
> application code, but it's probably just a matter of time.  In terms of
> expanding our user base, it may make sense to provide that capability.  It
> doesn't seem to be generally available from an existing provider.
>
> Regarding message digests/HMAC, I question whether the performance gain
> from native code would significantly outperform some of the modern JCA
> providers.  As Matt Sicker pointed out, there are other implementations
> supported by major vendors, like AWS, that may be as fast as a JNI wrapper
> on OpenSSL, or at least close enough not to bother with the added
> complexity of the native stuff.  The only way I know to answer that
> question is to write the code and run a load test comparison.
>
> https://aws.amazon.com/blogs/opensource/introducing-amazon-corretto-crypto-provider-accp/
>
> In what forum should we discuss these issues?

This developer's mailing list is where all decisions MUST be recorded,
so it makes sense that discussions occur here. We can also use Jira
tickets for specific tasks. Different discussions can use these
different forums as we best see fit. I would not recommend using Slack
as that data does or will eventually disappear. I do use Slack as a
convenience at times for reminding people/channels to vote or pay
attention to something happening in Jira or on a mailing list.

> Are we limited to this
> distro or do we have other options?

I'm not sure what this means.

> How do we form teams?
I'm not sure what that means in the context of Apache. Apache has no
notion of teams that I am aware of. We have Projects like Apache
Commons, and Commons has Components like Commons Crypto.
We are all on this mailing list, we can work on whatever we want, and
we can use Jira to track tasks. If someone wants to work on a task,
they can announce it here or in a Jira ticket, to avoid clashes or
just be informative.

> What is our
> governance model?
The same as any Apache project with the usual leeway allowed to each PMC.

 > How do we make decisions?
The same as any Apache project with the usual leeway allowed to each PMC.
See https://httpd.apache.org/dev/guidelines.html
A project can come up with it's own processes, for example see
https://www.apache.org/foundation/voting.html

Gary

>
> FYI:  There's already issues on the backlog for OpenSSL 3 and HMAC:
> https://issues.apache.org/jira/browse/CRYPTO-165
> https://issues.apache.org/jira/browse/CRYPTO-164
>
> Alex
>
>
> On Wed, Aug 23, 2023 at 10:21 PM Gary Gregory 
> wrote:
>
> > That would be great. I think this is worth the effort. A big item to
> > consider is if and how 1.1 vs 3.0 should be handled. Breakup the current
> > module into different maven modules? Not support both?
> >
> > Gary
> >
> > On Wed, Aug 23, 2023, 8:37 PM Alex Remily  wrote:
> >
> > >  > > how to implement message digests and HMAC on top of OpenSSL 3.0.8.>
> > >
> > > Implementing the OpenSSL 3 API and exposing OpenSSL HMAC functionality in
> > > commons-crypto are things I've wanted to engage on for a while now.  I
> > was
> > > involved in the commons-crypto OpenSSL 1.1.x upgrade so I have some
> > > familiarity with the code base, albeit dated.  The reason that neither
> > the
> > > OpenSSL 3 API nor the HMAC functionality have been added to
> > commons-crypto
> > > yet, IMHO, is because both 

Re: [commons-crypto] OpenSSL 3.x, FIPS, digests, and HMAC

[Marcelo Vanzin]

Oh, wow, I think I heard my name.

First, big disclaimer that I haven't really worked with any of this stuff
for years, so take what I say with a big pinch of salt. I have no idea of
what's the current state of things in OpenSSL-land.

As far as I can remember, the desire to do runtime discovery was for easy
of packaging software using the library. For my company at the time, the
main use was for Apache Spark and other software that people don't
generally embed into their apps, but instead run directly from a package
provided by Apache (or some distribution). Having to build multiple
versions of an already large package and have to teach people to figure out
what's the version that they need was just too much of a hassle.

Maybe there are better ways to do that runtime discovery without the C code
having to introspect into the shared libraries. Maybe that isn't even
needed these days because the libssl API is more stable and the world has
moved on from 1.0.

Anyway, I won't make suggestions because I don't want to talk about things
that I'm completely out of touch with these days, but please don't hold
back on making changes because of something that, at the time, might have
made sense but these days may not.


On Fri, Aug 25, 2023 at 6:48 PM Alex Remily  wrote:

>  the current module into different maven modules? Not support both?>
>
> Agreed.  Just to provide some history, when I was working on the 1.1.x
> upgrade I was guided by commons committer Marcelo Vanzin.  Marcelo required
> a design that supported runtime discovery of the underlying openssl API.  I
> don't recall all of the rationale for the requirement, but he insisted that
> any commons-crypto upgrade must support legacy and current versions of
> openssl transparently to the calling program.  The end result is what we
> have now.  I don't recall where in the code commons-crypto makes the
> underlying version checks, at library initialization time or when a
> specific function is called, but the end result is that users need only
> download the latest commons-crypto release regardless of their underlying
> openssl API--as long as they are running supported openssl versions.
>
> In my view there are a few open questions regarding the current approach as
> compared to an API-specific one.  One, what is the performance penalty
> associated with the dynamic version checks?  Two, how much complexity does
> it introduce into the codebase?  Finally, what was the use case that drove
> the runtime checking requirement?  Marcelo could answer the last question.
> I don't know if he is still involved in the community (I haven't seen him
> around for awhile.  IIRC, he was primarily a Spark committer).
>
> Another consideration is the FIPS certification.  I work in a heavily
> regulated industry and FIPS is a real constraint.  I haven't personally
> encountered a requirement to deploy a FIPS compliant openssl in our
> application code, but it's probably just a matter of time.  In terms of
> expanding our user base, it may make sense to provide that capability.  It
> doesn't seem to be generally available from an existing provider.
>
> Regarding message digests/HMAC, I question whether the performance gain
> from native code would significantly outperform some of the modern JCA
> providers.  As Matt Sicker pointed out, there are other implementations
> supported by major vendors, like AWS, that may be as fast as a JNI wrapper
> on OpenSSL, or at least close enough not to bother with the added
> complexity of the native stuff.  The only way I know to answer that
> question is to write the code and run a load test comparison.
>
>
> https://aws.amazon.com/blogs/opensource/introducing-amazon-corretto-crypto-provider-accp/
>
> In what forum should we discuss these issues?  Are we limited to this
> distro or do we have other options?  How do we form teams?  What is our
> governance model?  How do we make decisions?
>
> FYI:  There's already issues on the backlog for OpenSSL 3 and HMAC:
> https://issues.apache.org/jira/browse/CRYPTO-165
> https://issues.apache.org/jira/browse/CRYPTO-164
>
> Alex
>
>
> On Wed, Aug 23, 2023 at 10:21 PM Gary Gregory 
> wrote:
>
> > That would be great. I think this is worth the effort. A big item to
> > consider is if and how 1.1 vs 3.0 should be handled. Breakup the current
> > module into different maven modules? Not support both?
> >
> > Gary
> >
> > On Wed, Aug 23, 2023, 8:37 PM Alex Remily  wrote:
> >
> > >  learn
> > > how to implement message digests and HMAC on top of OpenSSL 3.0.8.>
> > >
> > > Implementing the OpenSSL 3 API and exposing OpenSSL HMAC functionality
> in
> > > commons-crypto are things I've wanted to engage on for a while now.  I
> > was
> > > involved in the commons-crypto OpenSSL 1.1.x upgrade so I have some
> > > familiarity with the code base, albeit dated.  The reason that neither
> > the
> > > OpenSSL 3 API nor the HMAC functionality have been added to
> > commons-crypto
> > > yet, IMHO, is because both 

Re: [commons-crypto] OpenSSL 3.x, FIPS, digests, and HMAC

[Alex Remily]

Agreed.  Just to provide some history, when I was working on the 1.1.x
upgrade I was guided by commons committer Marcelo Vanzin.  Marcelo required
a design that supported runtime discovery of the underlying openssl API.  I
don't recall all of the rationale for the requirement, but he insisted that
any commons-crypto upgrade must support legacy and current versions of
openssl transparently to the calling program.  The end result is what we
have now.  I don't recall where in the code commons-crypto makes the
underlying version checks, at library initialization time or when a
specific function is called, but the end result is that users need only
download the latest commons-crypto release regardless of their underlying
openssl API--as long as they are running supported openssl versions.

In my view there are a few open questions regarding the current approach as
compared to an API-specific one.  One, what is the performance penalty
associated with the dynamic version checks?  Two, how much complexity does
it introduce into the codebase?  Finally, what was the use case that drove
the runtime checking requirement?  Marcelo could answer the last question.
I don't know if he is still involved in the community (I haven't seen him
around for awhile.  IIRC, he was primarily a Spark committer).

Another consideration is the FIPS certification.  I work in a heavily
regulated industry and FIPS is a real constraint.  I haven't personally
encountered a requirement to deploy a FIPS compliant openssl in our
application code, but it's probably just a matter of time.  In terms of
expanding our user base, it may make sense to provide that capability.  It
doesn't seem to be generally available from an existing provider.

Regarding message digests/HMAC, I question whether the performance gain
from native code would significantly outperform some of the modern JCA
providers.  As Matt Sicker pointed out, there are other implementations
supported by major vendors, like AWS, that may be as fast as a JNI wrapper
on OpenSSL, or at least close enough not to bother with the added
complexity of the native stuff.  The only way I know to answer that
question is to write the code and run a load test comparison.

https://aws.amazon.com/blogs/opensource/introducing-amazon-corretto-crypto-provider-accp/

In what forum should we discuss these issues?  Are we limited to this
distro or do we have other options?  How do we form teams?  What is our
governance model?  How do we make decisions?

FYI:  There's already issues on the backlog for OpenSSL 3 and HMAC:
https://issues.apache.org/jira/browse/CRYPTO-165
https://issues.apache.org/jira/browse/CRYPTO-164

Alex


On Wed, Aug 23, 2023 at 10:21 PM Gary Gregory 
wrote:

> That would be great. I think this is worth the effort. A big item to
> consider is if and how 1.1 vs 3.0 should be handled. Breakup the current
> module into different maven modules? Not support both?
>
> Gary
>
> On Wed, Aug 23, 2023, 8:37 PM Alex Remily  wrote:
>
> >  > how to implement message digests and HMAC on top of OpenSSL 3.0.8.>
> >
> > Implementing the OpenSSL 3 API and exposing OpenSSL HMAC functionality in
> > commons-crypto are things I've wanted to engage on for a while now.  I
> was
> > involved in the commons-crypto OpenSSL 1.1.x upgrade so I have some
> > familiarity with the code base, albeit dated.  The reason that neither
> the
> > OpenSSL 3 API nor the HMAC functionality have been added to
> commons-crypto
> > yet, IMHO, is because both are non-trivial efforts and we all have day
> > jobs. Until now, there hasn't been much of an ask for either feature, but
> > considering the recent chatter maybe there's enough interest to generate
> > some momentum.  I would be happy to collaborate on this effort if it
> gains
> > traction.
> >
> > Alex
> >
> > On Wed, Aug 23, 2023 at 5:21 PM Matt Sicker  wrote:
> >
> >> You may find this project (also Apache licensed, so fairly safe for
> reuse
> >> with commons-crypto updates) to be helpful for that aspect of message
> >> digests, Macs, etc.
> >>
> >> [image: amazon-corretto-crypto-provider.png]
> >>
> >> corretto/amazon-corretto-crypto-provider: The Amazon Corretto Crypto
> >> Provider is a collection of high-performance cryptographic
> implementations
> >> exposed via standard JCA/JCE interfaces.
> >> 
> >> github.com  >
> >> 
> >>
> >>
> >> On Aug 1, 2023, at 9:17 AM, Jim Showalter 
> >> wrote:
> >>
> >> I'm still trying to come up to speed on your PR/fork. A lot to learn!
> >>
> >> Ack about licensing. The idea wouldn't be to copy the code, but to learn
> >> how to implement message digests and HMAC on top of OpenSSL 3.0.8.
> >>
> >> On Tue, Aug 1, 2023, 5:03 AM Gary Gregory 
> wrote:
> >>
> >> In the short, the best way to help is to provide PRs.
> >> In more detail, we should probably 

Re: [commons-crypto] OpenSSL 3.x, FIPS, digests, and HMAC

[Gary Gregory]

That would be great. I think this is worth the effort. A big item to
consider is if and how 1.1 vs 3.0 should be handled. Breakup the current
module into different maven modules? Not support both?

Gary

On Wed, Aug 23, 2023, 8:37 PM Alex Remily  wrote:

>  how to implement message digests and HMAC on top of OpenSSL 3.0.8.>
>
> Implementing the OpenSSL 3 API and exposing OpenSSL HMAC functionality in
> commons-crypto are things I've wanted to engage on for a while now.  I was
> involved in the commons-crypto OpenSSL 1.1.x upgrade so I have some
> familiarity with the code base, albeit dated.  The reason that neither the
> OpenSSL 3 API nor the HMAC functionality have been added to commons-crypto
> yet, IMHO, is because both are non-trivial efforts and we all have day
> jobs. Until now, there hasn't been much of an ask for either feature, but
> considering the recent chatter maybe there's enough interest to generate
> some momentum.  I would be happy to collaborate on this effort if it gains
> traction.
>
> Alex
>
> On Wed, Aug 23, 2023 at 5:21 PM Matt Sicker  wrote:
>
>> You may find this project (also Apache licensed, so fairly safe for reuse
>> with commons-crypto updates) to be helpful for that aspect of message
>> digests, Macs, etc.
>>
>> [image: amazon-corretto-crypto-provider.png]
>>
>> corretto/amazon-corretto-crypto-provider: The Amazon Corretto Crypto
>> Provider is a collection of high-performance cryptographic implementations
>> exposed via standard JCA/JCE interfaces.
>> 
>> github.com 
>> 
>>
>>
>> On Aug 1, 2023, at 9:17 AM, Jim Showalter 
>> wrote:
>>
>> I'm still trying to come up to speed on your PR/fork. A lot to learn!
>>
>> Ack about licensing. The idea wouldn't be to copy the code, but to learn
>> how to implement message digests and HMAC on top of OpenSSL 3.0.8.
>>
>> On Tue, Aug 1, 2023, 5:03 AM Gary Gregory  wrote:
>>
>> In the short, the best way to help is to provide PRs.
>> In more detail, we should probably come up with some kind of a plan so
>> that
>> everyone uses their time wisely.
>>
>> I'll review my branch this week or next and see where that stands, but
>> feel
>> free to look at it, use it, PR it, as I might not actually be able to take
>> the time this week.
>>
>> WRT openssl4j, you CANNOT bring in anything licensed under the LGPL.
>> IANAL,
>> but our documentation seems clear to me, please see
>> https://www.apache.org/legal/resolved.html#category-x
>>
>> Hope this helps and we can make it work!
>> Gary
>>
>>
>> On Mon, Jul 31, 2023, 8:02 PM Jim Showalter 
>> wrote:
>>
>> A split seems reasonable. We were amazed at how few changes you had to
>>
>> make
>>
>> to support OpenSSL 3.x. The EVPs are very different. But it sounds like
>> there's more to do.
>>
>> The problem with commons-codec is that it doesn't use OpenSSL or any
>>
>> other
>>
>> FIPS-certified cryptographic module. For example, HmacUtils uses Mac,
>>
>> which
>>
>> is supplied by the JRE, which isn't FIPS-certified.
>>
>> In order to qualify for FedRAMP High, which is table stakes for a lot of
>> corporate and government contracts, FIPS has to be used. It's mandated by
>> statute. No wiggle room.
>>
>> The promise of bc-fips is that it is FIPS-certified, is a JSP, and
>> implements the full JCE. The drawback is that the bc-fips org is funded
>> through donations and consulting, and is always very far behind Java
>> releases (it's still on Java 11, for example).
>>
>> What we need is a full-featured JSP that is based on a FIPS-certified
>> cryptographic module that is implemented using native code and JNI.
>>
>> We can get FIPS-certified digests and HMAC from openssl4j. What we need
>> from commons-crypto are the ciphers, and they need to be on OpenSSL
>>
>> 3.0.8.
>>
>>
>> If there's anything we can do to help make that happen, please let us
>>
>> know.
>>
>>
>> On Mon, Jul 31, 2023 at 2:58 PM Gary Gregory 
>> wrote:
>>
>> Hi Jim,
>>
>> My branch has not been merged because it does not fully work. It's
>> challenging to update the code such that one can use either OpenSSL
>>
>> 1.1.1
>>
>> or 3.0.0 or both. We might need the component split into more than one
>> Maven module.
>>
>> The name commons-crypto might have been poorly chosen because it's
>>
>> current
>>
>> remit is an OpelSSL wrapper. That said there is room for more features,
>> which may mean splitting things up into more than one Maven module.
>>
>> Commons Code provides more convenience wrappers for JRE message digests
>> including HMAC:
>> https://commons.apache.org/proper/commons-codec/apidocs/index.html
>>
>> Are you looking to wrap or implement HMAC and message digests
>>
>> differently?
>>
>>
>> Gary
>>
>>
>> On Mon, Jul 31, 2023, 5:04 PM Jim Showalter <
>>
>> jamesleeshowal...@gmail.com
>>
>>
>> wrote:
>>
>> We are trying to replace bc-fips (
>>
>> 

Re: [commons-crypto] OpenSSL 3.x, FIPS, digests, and HMAC

[Alex Remily]

Implementing the OpenSSL 3 API and exposing OpenSSL HMAC functionality in
commons-crypto are things I've wanted to engage on for a while now.  I was
involved in the commons-crypto OpenSSL 1.1.x upgrade so I have some
familiarity with the code base, albeit dated.  The reason that neither the
OpenSSL 3 API nor the HMAC functionality have been added to commons-crypto
yet, IMHO, is because both are non-trivial efforts and we all have day
jobs. Until now, there hasn't been much of an ask for either feature, but
considering the recent chatter maybe there's enough interest to generate
some momentum.  I would be happy to collaborate on this effort if it gains
traction.

Alex

On Wed, Aug 23, 2023 at 5:21 PM Matt Sicker  wrote:

> You may find this project (also Apache licensed, so fairly safe for reuse
> with commons-crypto updates) to be helpful for that aspect of message
> digests, Macs, etc.
>
> [image: amazon-corretto-crypto-provider.png]
>
> corretto/amazon-corretto-crypto-provider: The Amazon Corretto Crypto
> Provider is a collection of high-performance cryptographic implementations
> exposed via standard JCA/JCE interfaces.
> 
> github.com 
> 
>
>
> On Aug 1, 2023, at 9:17 AM, Jim Showalter 
> wrote:
>
> I'm still trying to come up to speed on your PR/fork. A lot to learn!
>
> Ack about licensing. The idea wouldn't be to copy the code, but to learn
> how to implement message digests and HMAC on top of OpenSSL 3.0.8.
>
> On Tue, Aug 1, 2023, 5:03 AM Gary Gregory  wrote:
>
> In the short, the best way to help is to provide PRs.
> In more detail, we should probably come up with some kind of a plan so that
> everyone uses their time wisely.
>
> I'll review my branch this week or next and see where that stands, but feel
> free to look at it, use it, PR it, as I might not actually be able to take
> the time this week.
>
> WRT openssl4j, you CANNOT bring in anything licensed under the LGPL. IANAL,
> but our documentation seems clear to me, please see
> https://www.apache.org/legal/resolved.html#category-x
>
> Hope this helps and we can make it work!
> Gary
>
>
> On Mon, Jul 31, 2023, 8:02 PM Jim Showalter 
> wrote:
>
> A split seems reasonable. We were amazed at how few changes you had to
>
> make
>
> to support OpenSSL 3.x. The EVPs are very different. But it sounds like
> there's more to do.
>
> The problem with commons-codec is that it doesn't use OpenSSL or any
>
> other
>
> FIPS-certified cryptographic module. For example, HmacUtils uses Mac,
>
> which
>
> is supplied by the JRE, which isn't FIPS-certified.
>
> In order to qualify for FedRAMP High, which is table stakes for a lot of
> corporate and government contracts, FIPS has to be used. It's mandated by
> statute. No wiggle room.
>
> The promise of bc-fips is that it is FIPS-certified, is a JSP, and
> implements the full JCE. The drawback is that the bc-fips org is funded
> through donations and consulting, and is always very far behind Java
> releases (it's still on Java 11, for example).
>
> What we need is a full-featured JSP that is based on a FIPS-certified
> cryptographic module that is implemented using native code and JNI.
>
> We can get FIPS-certified digests and HMAC from openssl4j. What we need
> from commons-crypto are the ciphers, and they need to be on OpenSSL
>
> 3.0.8.
>
>
> If there's anything we can do to help make that happen, please let us
>
> know.
>
>
> On Mon, Jul 31, 2023 at 2:58 PM Gary Gregory 
> wrote:
>
> Hi Jim,
>
> My branch has not been merged because it does not fully work. It's
> challenging to update the code such that one can use either OpenSSL
>
> 1.1.1
>
> or 3.0.0 or both. We might need the component split into more than one
> Maven module.
>
> The name commons-crypto might have been poorly chosen because it's
>
> current
>
> remit is an OpelSSL wrapper. That said there is room for more features,
> which may mean splitting things up into more than one Maven module.
>
> Commons Code provides more convenience wrappers for JRE message digests
> including HMAC:
> https://commons.apache.org/proper/commons-codec/apidocs/index.html
>
> Are you looking to wrap or implement HMAC and message digests
>
> differently?
>
>
> Gary
>
>
> On Mon, Jul 31, 2023, 5:04 PM Jim Showalter <
>
> jamesleeshowal...@gmail.com
>
>
> wrote:
>
> We are trying to replace bc-fips (
>
> https://www.bouncycastle.org/fips-java/)
>
> with a JSP that is based on a cryptographic module that is 1) a
>
> native
>
> library and 2) is certified for FIPS 140-2 (
> https://csrc.nist.gov/pubs/fips/140-2/upd2/final).
>
> A native library is faster, plus it doesn't entangle the Java
>
> classpath
>
> with restrictions on Java versions or load order the way bc-fips
>
> does.
>
>
> The two available native libraries we're aware of are BoringSSL and
> OpenSSL.
>
> For various 

Re: [commons-crypto] OpenSSL 3.x, FIPS, digests, and HMAC

[Jim Showalter]

I'm still trying to come up to speed on your PR/fork. A lot to learn!

Ack about licensing. The idea wouldn't be to copy the code, but to learn
how to implement message digests and HMAC on top of OpenSSL 3.0.8.

On Tue, Aug 1, 2023, 5:03 AM Gary Gregory  wrote:

> In the short, the best way to help is to provide PRs.
> In more detail, we should probably come up with some kind of a plan so that
> everyone uses their time wisely.
>
> I'll review my branch this week or next and see where that stands, but feel
> free to look at it, use it, PR it, as I might not actually be able to take
> the time this week.
>
> WRT openssl4j, you CANNOT bring in anything licensed under the LGPL. IANAL,
> but our documentation seems clear to me, please see
> https://www.apache.org/legal/resolved.html#category-x
>
> Hope this helps and we can make it work!
> Gary
>
>
> On Mon, Jul 31, 2023, 8:02 PM Jim Showalter 
> wrote:
>
> > A split seems reasonable. We were amazed at how few changes you had to
> make
> > to support OpenSSL 3.x. The EVPs are very different. But it sounds like
> > there's more to do.
> >
> > The problem with commons-codec is that it doesn't use OpenSSL or any
> other
> > FIPS-certified cryptographic module. For example, HmacUtils uses Mac,
> which
> > is supplied by the JRE, which isn't FIPS-certified.
> >
> > In order to qualify for FedRAMP High, which is table stakes for a lot of
> > corporate and government contracts, FIPS has to be used. It's mandated by
> > statute. No wiggle room.
> >
> > The promise of bc-fips is that it is FIPS-certified, is a JSP, and
> > implements the full JCE. The drawback is that the bc-fips org is funded
> > through donations and consulting, and is always very far behind Java
> > releases (it's still on Java 11, for example).
> >
> > What we need is a full-featured JSP that is based on a FIPS-certified
> > cryptographic module that is implemented using native code and JNI.
> >
> > We can get FIPS-certified digests and HMAC from openssl4j. What we need
> > from commons-crypto are the ciphers, and they need to be on OpenSSL
> 3.0.8.
> >
> > If there's anything we can do to help make that happen, please let us
> know.
> >
> > On Mon, Jul 31, 2023 at 2:58 PM Gary Gregory 
> > wrote:
> >
> > > Hi Jim,
> > >
> > > My branch has not been merged because it does not fully work. It's
> > > challenging to update the code such that one can use either OpenSSL
> 1.1.1
> > > or 3.0.0 or both. We might need the component split into more than one
> > > Maven module.
> > >
> > > The name commons-crypto might have been poorly chosen because it's
> > current
> > > remit is an OpelSSL wrapper. That said there is room for more features,
> > > which may mean splitting things up into more than one Maven module.
> > >
> > > Commons Code provides more convenience wrappers for JRE message digests
> > > including HMAC:
> > > https://commons.apache.org/proper/commons-codec/apidocs/index.html
> > >
> > > Are you looking to wrap or implement HMAC and message digests
> > differently?
> > >
> > > Gary
> > >
> > >
> > > On Mon, Jul 31, 2023, 5:04 PM Jim Showalter <
> jamesleeshowal...@gmail.com
> > >
> > > wrote:
> > >
> > > > We are trying to replace bc-fips (
> > > https://www.bouncycastle.org/fips-java/)
> > > > with a JSP that is based on a cryptographic module that is 1) a
> native
> > > > library and 2) is certified for FIPS 140-2 (
> > > > https://csrc.nist.gov/pubs/fips/140-2/upd2/final).
> > > >
> > > > A native library is faster, plus it doesn't entangle the Java
> classpath
> > > > with restrictions on Java versions or load order the way bc-fips
> does.
> > > >
> > > > The two available native libraries we're aware of are BoringSSL and
> > > > OpenSSL.
> > > >
> > > > For various reasons, we want to use OpenSSL.
> > > >
> > > > OpenSSL 1.1.1 was only FIPS-certified on RedHat (and they had to
> modify
> > > it
> > > > to add FIPS support), and the certification expires soon.
> > > >
> > > > OpenSSL 1.1.1 is the version commons-crypto is currently based on.
> > > >
> > > > OpenSSL 3.0.8 is FIPS-certified on a variety of platforms, supports
> > FIPS
> > > > mode natively, and its successor (3.1.x) will be certified for FIPS
> > > 140-3.
> > > >
> > > > We're very interested in
> > > > https://github.com/garydgregory/commons-crypto/tree/openssl3, which
> > adds
> > > > support for OpenSSL 3.0.8 to commons-crypto, per
> > > > https://issues.apache.org/jira/browse/CRYPTO-164.
> > > >
> > > > But that PR was never merged, hasn't been touched since December 20,
> > > 2022,
> > > > and is currently 92 commits behind the main branch.
> > > >
> > > > What would it take to update that PR with all of the commits since
> > then,
> > > > and get it merged?
> > > >
> > > > Once that's done, we'd be happy to submit a PR to add FIPS mode, per
> > > > https://issues.apache.org/jira/browse/CRYPTO-136.
> > > >
> > > > Also, commons-crypto doesn't support message digests or HMAC. We're
> in
> > > the
> > > > process of 

Re: [commons-crypto] OpenSSL 3.x, FIPS, digests, and HMAC

[Gary Gregory]

In the short, the best way to help is to provide PRs.
In more detail, we should probably come up with some kind of a plan so that
everyone uses their time wisely.

I'll review my branch this week or next and see where that stands, but feel
free to look at it, use it, PR it, as I might not actually be able to take
the time this week.

WRT openssl4j, you CANNOT bring in anything licensed under the LGPL. IANAL,
but our documentation seems clear to me, please see
https://www.apache.org/legal/resolved.html#category-x

Hope this helps and we can make it work!
Gary


On Mon, Jul 31, 2023, 8:02 PM Jim Showalter 
wrote:

> A split seems reasonable. We were amazed at how few changes you had to make
> to support OpenSSL 3.x. The EVPs are very different. But it sounds like
> there's more to do.
>
> The problem with commons-codec is that it doesn't use OpenSSL or any other
> FIPS-certified cryptographic module. For example, HmacUtils uses Mac, which
> is supplied by the JRE, which isn't FIPS-certified.
>
> In order to qualify for FedRAMP High, which is table stakes for a lot of
> corporate and government contracts, FIPS has to be used. It's mandated by
> statute. No wiggle room.
>
> The promise of bc-fips is that it is FIPS-certified, is a JSP, and
> implements the full JCE. The drawback is that the bc-fips org is funded
> through donations and consulting, and is always very far behind Java
> releases (it's still on Java 11, for example).
>
> What we need is a full-featured JSP that is based on a FIPS-certified
> cryptographic module that is implemented using native code and JNI.
>
> We can get FIPS-certified digests and HMAC from openssl4j. What we need
> from commons-crypto are the ciphers, and they need to be on OpenSSL 3.0.8.
>
> If there's anything we can do to help make that happen, please let us know.
>
> On Mon, Jul 31, 2023 at 2:58 PM Gary Gregory 
> wrote:
>
> > Hi Jim,
> >
> > My branch has not been merged because it does not fully work. It's
> > challenging to update the code such that one can use either OpenSSL 1.1.1
> > or 3.0.0 or both. We might need the component split into more than one
> > Maven module.
> >
> > The name commons-crypto might have been poorly chosen because it's
> current
> > remit is an OpelSSL wrapper. That said there is room for more features,
> > which may mean splitting things up into more than one Maven module.
> >
> > Commons Code provides more convenience wrappers for JRE message digests
> > including HMAC:
> > https://commons.apache.org/proper/commons-codec/apidocs/index.html
> >
> > Are you looking to wrap or implement HMAC and message digests
> differently?
> >
> > Gary
> >
> >
> > On Mon, Jul 31, 2023, 5:04 PM Jim Showalter  >
> > wrote:
> >
> > > We are trying to replace bc-fips (
> > https://www.bouncycastle.org/fips-java/)
> > > with a JSP that is based on a cryptographic module that is 1) a native
> > > library and 2) is certified for FIPS 140-2 (
> > > https://csrc.nist.gov/pubs/fips/140-2/upd2/final).
> > >
> > > A native library is faster, plus it doesn't entangle the Java classpath
> > > with restrictions on Java versions or load order the way bc-fips does.
> > >
> > > The two available native libraries we're aware of are BoringSSL and
> > > OpenSSL.
> > >
> > > For various reasons, we want to use OpenSSL.
> > >
> > > OpenSSL 1.1.1 was only FIPS-certified on RedHat (and they had to modify
> > it
> > > to add FIPS support), and the certification expires soon.
> > >
> > > OpenSSL 1.1.1 is the version commons-crypto is currently based on.
> > >
> > > OpenSSL 3.0.8 is FIPS-certified on a variety of platforms, supports
> FIPS
> > > mode natively, and its successor (3.1.x) will be certified for FIPS
> > 140-3.
> > >
> > > We're very interested in
> > > https://github.com/garydgregory/commons-crypto/tree/openssl3, which
> adds
> > > support for OpenSSL 3.0.8 to commons-crypto, per
> > > https://issues.apache.org/jira/browse/CRYPTO-164.
> > >
> > > But that PR was never merged, hasn't been touched since December 20,
> > 2022,
> > > and is currently 92 commits behind the main branch.
> > >
> > > What would it take to update that PR with all of the commits since
> then,
> > > and get it merged?
> > >
> > > Once that's done, we'd be happy to submit a PR to add FIPS mode, per
> > > https://issues.apache.org/jira/browse/CRYPTO-136.
> > >
> > > Also, commons-crypto doesn't support message digests or HMAC. We're in
> > the
> > > process of adding HMAC and FIPS mode to
> > > https://github.com/sfuhrm/openssl4j,
> > > which has message digests, and targets OpenSSL 3.0.8.
> > >
> > > It seems like the message digests and HMAC from openssl4j could be
> merged
> > > into commons-crypto, to bring it closer to being a full JCE
> > implementation.
> > > Is there any interest in seeing that happen?
> > >
> >
>

Re: [commons-crypto] OpenSSL 3.x, FIPS, digests, and HMAC

[Jim Showalter]

A split seems reasonable. We were amazed at how few changes you had to make
to support OpenSSL 3.x. The EVPs are very different. But it sounds like
there's more to do.

The problem with commons-codec is that it doesn't use OpenSSL or any other
FIPS-certified cryptographic module. For example, HmacUtils uses Mac, which
is supplied by the JRE, which isn't FIPS-certified.

In order to qualify for FedRAMP High, which is table stakes for a lot of
corporate and government contracts, FIPS has to be used. It's mandated by
statute. No wiggle room.

The promise of bc-fips is that it is FIPS-certified, is a JSP, and
implements the full JCE. The drawback is that the bc-fips org is funded
through donations and consulting, and is always very far behind Java
releases (it's still on Java 11, for example).

What we need is a full-featured JSP that is based on a FIPS-certified
cryptographic module that is implemented using native code and JNI.

We can get FIPS-certified digests and HMAC from openssl4j. What we need
from commons-crypto are the ciphers, and they need to be on OpenSSL 3.0.8.

If there's anything we can do to help make that happen, please let us know.

On Mon, Jul 31, 2023 at 2:58 PM Gary Gregory  wrote:

> Hi Jim,
>
> My branch has not been merged because it does not fully work. It's
> challenging to update the code such that one can use either OpenSSL 1.1.1
> or 3.0.0 or both. We might need the component split into more than one
> Maven module.
>
> The name commons-crypto might have been poorly chosen because it's current
> remit is an OpelSSL wrapper. That said there is room for more features,
> which may mean splitting things up into more than one Maven module.
>
> Commons Code provides more convenience wrappers for JRE message digests
> including HMAC:
> https://commons.apache.org/proper/commons-codec/apidocs/index.html
>
> Are you looking to wrap or implement HMAC and message digests differently?
>
> Gary
>
>
> On Mon, Jul 31, 2023, 5:04 PM Jim Showalter 
> wrote:
>
> > We are trying to replace bc-fips (
> https://www.bouncycastle.org/fips-java/)
> > with a JSP that is based on a cryptographic module that is 1) a native
> > library and 2) is certified for FIPS 140-2 (
> > https://csrc.nist.gov/pubs/fips/140-2/upd2/final).
> >
> > A native library is faster, plus it doesn't entangle the Java classpath
> > with restrictions on Java versions or load order the way bc-fips does.
> >
> > The two available native libraries we're aware of are BoringSSL and
> > OpenSSL.
> >
> > For various reasons, we want to use OpenSSL.
> >
> > OpenSSL 1.1.1 was only FIPS-certified on RedHat (and they had to modify
> it
> > to add FIPS support), and the certification expires soon.
> >
> > OpenSSL 1.1.1 is the version commons-crypto is currently based on.
> >
> > OpenSSL 3.0.8 is FIPS-certified on a variety of platforms, supports FIPS
> > mode natively, and its successor (3.1.x) will be certified for FIPS
> 140-3.
> >
> > We're very interested in
> > https://github.com/garydgregory/commons-crypto/tree/openssl3, which adds
> > support for OpenSSL 3.0.8 to commons-crypto, per
> > https://issues.apache.org/jira/browse/CRYPTO-164.
> >
> > But that PR was never merged, hasn't been touched since December 20,
> 2022,
> > and is currently 92 commits behind the main branch.
> >
> > What would it take to update that PR with all of the commits since then,
> > and get it merged?
> >
> > Once that's done, we'd be happy to submit a PR to add FIPS mode, per
> > https://issues.apache.org/jira/browse/CRYPTO-136.
> >
> > Also, commons-crypto doesn't support message digests or HMAC. We're in
> the
> > process of adding HMAC and FIPS mode to
> > https://github.com/sfuhrm/openssl4j,
> > which has message digests, and targets OpenSSL 3.0.8.
> >
> > It seems like the message digests and HMAC from openssl4j could be merged
> > into commons-crypto, to bring it closer to being a full JCE
> implementation.
> > Is there any interest in seeing that happen?
> >
>

Re: [commons-crypto] OpenSSL 3.x, FIPS, digests, and HMAC

[Gary Gregory]

Hi Jim,

My branch has not been merged because it does not fully work. It's
challenging to update the code such that one can use either OpenSSL 1.1.1
or 3.0.0 or both. We might need the component split into more than one
Maven module.

The name commons-crypto might have been poorly chosen because it's current
remit is an OpelSSL wrapper. That said there is room for more features,
which may mean splitting things up into more than one Maven module.

Commons Code provides more convenience wrappers for JRE message digests
including HMAC:
https://commons.apache.org/proper/commons-codec/apidocs/index.html

Are you looking to wrap or implement HMAC and message digests differently?

Gary


On Mon, Jul 31, 2023, 5:04 PM Jim Showalter 
wrote:

> We are trying to replace bc-fips (https://www.bouncycastle.org/fips-java/)
> with a JSP that is based on a cryptographic module that is 1) a native
> library and 2) is certified for FIPS 140-2 (
> https://csrc.nist.gov/pubs/fips/140-2/upd2/final).
>
> A native library is faster, plus it doesn't entangle the Java classpath
> with restrictions on Java versions or load order the way bc-fips does.
>
> The two available native libraries we're aware of are BoringSSL and
> OpenSSL.
>
> For various reasons, we want to use OpenSSL.
>
> OpenSSL 1.1.1 was only FIPS-certified on RedHat (and they had to modify it
> to add FIPS support), and the certification expires soon.
>
> OpenSSL 1.1.1 is the version commons-crypto is currently based on.
>
> OpenSSL 3.0.8 is FIPS-certified on a variety of platforms, supports FIPS
> mode natively, and its successor (3.1.x) will be certified for FIPS 140-3.
>
> We're very interested in
> https://github.com/garydgregory/commons-crypto/tree/openssl3, which adds
> support for OpenSSL 3.0.8 to commons-crypto, per
> https://issues.apache.org/jira/browse/CRYPTO-164.
>
> But that PR was never merged, hasn't been touched since December 20, 2022,
> and is currently 92 commits behind the main branch.
>
> What would it take to update that PR with all of the commits since then,
> and get it merged?
>
> Once that's done, we'd be happy to submit a PR to add FIPS mode, per
> https://issues.apache.org/jira/browse/CRYPTO-136.
>
> Also, commons-crypto doesn't support message digests or HMAC. We're in the
> process of adding HMAC and FIPS mode to
> https://github.com/sfuhrm/openssl4j,
> which has message digests, and targets OpenSSL 3.0.8.
>
> It seems like the message digests and HMAC from openssl4j could be merged
> into commons-crypto, to bring it closer to being a full JCE implementation.
> Is there any interest in seeing that happen?
>