Re: CFR: AES-GCM and OpenCrypto work review

[Vsevolod Stakhov]

On 08/11/14 20:45, John-Mark Gurney wrote:

Vsevolod Stakhov wrote this message on Sat, Nov 08, 2014 at 18:55 +:

On 08/11/14 04:23, John-Mark Gurney wrote:

Hello,

Over the last few months, I've been working on a project to add support
for AES-GCM and AES-CTR modes to our OpenCrypto framework.  The work is
sponsored by The FreeBSD Foundation and Netgate.

I plan on committing these patches early next week.  If you need more
time for review, please email me privately and I will make delay.

The code has already been reviewed by Watson Ladd (the software crypto
implementations) and Trevor Perrin (the aesni module part) and I have
integrated these changes into the patch.

There are two patches, one is the changes for OpenCrypto and the test
framework.  The other is the data files used by the test framework.
The data is from NIST's CAVP program, and is about 20MB worth of test
vectors.  (I just realized, should we look at compressing these on
disk?)

Main patch (192KB):
https://www.funkthat.com/~jmg/patches/aes.ipsec.5.patch

Data files (~20MB):
https://www.funkthat.com/~jmg/patches/aes.ipsec.5.testing.patch

A list of notable changes in the patch:
- Replacing crypto(4) w/ NetBSD's version + updates
- Lots of man page updates, including CIOCFINDDEV and crypto(7) which
   adds specifics about restrictions on the modes.
- Allow sane useage of both _HARDWARE and _SOFTWARE flags.
- Add a timing safe bcmp for MAC comparision.
- Add a software implementation of GCM that uses a four bit lookup
   table with parallelization.  This algorithm is possibly vulnerable to
   timing attacks, but best known mitigation methods are used.  Using
   a timing safe version is many times slower.
- Added a CRYPTDEB macro that defaults to off.
- Bring in some of OpenBSD's improvements to the OpenCrypto framework.
- If an mbuf passed to the aesni module is only one segment, don't do
   a copy.  This needs to be improved to support segmented buffers.
- Remove the CRYPTO_F_REL flag.  It was meaningless.  It was used but
   did not change any behavior.
- Add function crypto_mbuftoiov to convert an mbuf to an iov.  This
   also converts the software crypto to only use iov's even for a simple
   linear buffer, and so simplifies the processing.
- Add a dtrace probe for errors from the ioctl.
- Add the CIOCCRYPTAEAD ioctl that allows userland processing (testing)
   of AES-GCM and future AEAD modes.

Future improvements:
- Support IV's longer than 12 bytes for GCM.
- Make AES-NI support segmented buffers (iov or mbuf) so multisegmented
   inputs don't have to be copied.


I have the question regarding to the algorithm of GF field calculations
used in the proposed implementation: why not use the recent researches
in GCM calculations, e.g. described in [1], for further speed optimizations?

[1] - https://eprint.iacr.org/2013/157.pdf


The paper you linked to does not describe a new way of calculating
GHASH, but evalutation of a bug in their implementation using the
PCLMULQDQ instruction...

If you mean, why don't I use OpenSSL's code?  The reason is that their
code is a perl script that generates assmebly...  We don't have
perl in base.. and I didn't want more assembly in our tree (see below)..

Instead, I decided to use code from Intel's whitepaper:
Intel® Carry-Less Multiplication Instruction and its Usage for
Computing the GCM Mode

I didn't use their assembly version because I wanted to have
maintainable code, and also the same code can be used on both i386
and amd64 arches..  This turns out to also be a good thing, as when
I add segmented buffer support, it'll be much easier to add to the C
version, and I only have to do the work once for two arches...

Also, the software GF library that I wrote is using state of the art
algorithms...  An OpenBSD developer has tested my code and has seen
a significant performance improvement over their old code, and are
evaluating if they want to/can include it in their tree...

Hope this answers your question.  If not, please be more specific so
I can answer it.

Thanks.



I'm sorry, I thought that is the paper that is a transcript of the 
following presentation:


http://2013.diac.cr.yp.to/slides/gueron.pdf

made by the same authors. The transcript is not available so far it seems.

And regarding assembler/C maintainability I would argue that the current 
intrinsics based implementation is more readable than the pure assembler 
solution (and it is still machine dependent). Of course, I'm not the 
expert in such optimizations, so that is just my own feeling.


By the way, do you have some concrete numbers about the performance of 
your aes-gcm? (I recently could do aes-128-gcm at about 32 gigabits/sec 
that is not a limit of the modern hardware for sure).

___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: How exactly does the base toolchain determine WHICH language to build with?

[Chris H]

On Sat, 8 Nov 2014 14:46:21 +0100 Dimitry Andric  wrote

> On 08 Nov 2014, at 01:23, Chris H  wrote:
> > Sorry for the long title. I've been [needlessly] struggling
> > with getting ports within the ports tree to build, on a
> > fresh 11-CURRENT install from 2014-11-05. With custom
> > KERNEL and WORLD built, and installed.
> > Here's my situation, which has worked well since ~8.2;
> > make.conf(5)
> > WITHOUT_CLANG=true
> > FAVORITE_COMPILER=gcc
> > src.conf(5)
> > WITHOUT_CLANG=true
> 
> If you don't want to use and build clang at all, I think you also need
> these two settings (at least on recent -CURRENT):
> 
> WITHOUT_CLANG_BOOTSTRAP=foo
> WITHOUT_CLANG_IS_CC=foo
> 
> The latter variable determines whether /usr/bin/cc, /usr/bin/c++ and so
> on are linked to clang or gcc.  I suspect the root of all your problems
> is that you don't have this setting.
> 
> 
> > I'll neither argue, nor defend rational for w/o clang. To
> > boring and out of scope for this thread. That said; I
> > realize that lang/clang(33/34/35) is the default toolchain
> > for 10+, and that's just fine by me. So I shouldn't be
> > terribly surprised when install kernel/world, followed by
> > make delete-old removes the clang built, or provided by
> > the base install from the (initial) install procedure. But
> > what _does_ surprise me, is that the install of lang/gcc-48
> > does _not_ become the compiler of choice with the above
> > $ENV, after [seemingly] deleting clang.
> 
> It does not work that way.  If you configure the base system to use gcc
> only, individual ports can decide that they need newer C++ support, or
> other features not available in the version of gcc in base (which is
> very old by now).  In that case, they will use one of the gcc ports,
> or possibly even one of the clang ports.
Ahh. Thank you. It's now perfectly clear, and makes complete sense, now.
For the record; I have nothing against clang/llvm, per se.
I am just looking to obtain a bit more control over my environment.
As this installation is intended for development purposes. In
fact, I'll probably install all 3 versions of lang/clang. :)
Oh, and GNC C, as well.

Thank you again, for the reply. Now that I have also been informed
about freebsd-toolchain@. I should probably take this opportunity
to apologize for the noise, and thank all for their thoughtful
replies. :)

--Chris
> 
> -Dimitry


___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: CFR: AES-GCM and OpenCrypto work review

[John-Mark Gurney]

Vsevolod Stakhov wrote this message on Sat, Nov 08, 2014 at 18:55 +:
> On 08/11/14 04:23, John-Mark Gurney wrote:
> > Hello,
> > 
> > Over the last few months, I've been working on a project to add support
> > for AES-GCM and AES-CTR modes to our OpenCrypto framework.  The work is
> > sponsored by The FreeBSD Foundation and Netgate.
> > 
> > I plan on committing these patches early next week.  If you need more
> > time for review, please email me privately and I will make delay.
> > 
> > The code has already been reviewed by Watson Ladd (the software crypto
> > implementations) and Trevor Perrin (the aesni module part) and I have
> > integrated these changes into the patch.
> > 
> > There are two patches, one is the changes for OpenCrypto and the test
> > framework.  The other is the data files used by the test framework.
> > The data is from NIST's CAVP program, and is about 20MB worth of test
> > vectors.  (I just realized, should we look at compressing these on
> > disk?)
> > 
> > Main patch (192KB):
> > https://www.funkthat.com/~jmg/patches/aes.ipsec.5.patch
> > 
> > Data files (~20MB):
> > https://www.funkthat.com/~jmg/patches/aes.ipsec.5.testing.patch
> > 
> > A list of notable changes in the patch:
> > - Replacing crypto(4) w/ NetBSD's version + updates
> > - Lots of man page updates, including CIOCFINDDEV and crypto(7) which
> >   adds specifics about restrictions on the modes.
> > - Allow sane useage of both _HARDWARE and _SOFTWARE flags.
> > - Add a timing safe bcmp for MAC comparision.
> > - Add a software implementation of GCM that uses a four bit lookup
> >   table with parallelization.  This algorithm is possibly vulnerable to
> >   timing attacks, but best known mitigation methods are used.  Using
> >   a timing safe version is many times slower.
> > - Added a CRYPTDEB macro that defaults to off.
> > - Bring in some of OpenBSD's improvements to the OpenCrypto framework.
> > - If an mbuf passed to the aesni module is only one segment, don't do
> >   a copy.  This needs to be improved to support segmented buffers.
> > - Remove the CRYPTO_F_REL flag.  It was meaningless.  It was used but
> >   did not change any behavior.
> > - Add function crypto_mbuftoiov to convert an mbuf to an iov.  This
> >   also converts the software crypto to only use iov's even for a simple
> >   linear buffer, and so simplifies the processing.
> > - Add a dtrace probe for errors from the ioctl.
> > - Add the CIOCCRYPTAEAD ioctl that allows userland processing (testing)
> >   of AES-GCM and future AEAD modes.
> > 
> > Future improvements:
> > - Support IV's longer than 12 bytes for GCM.
> > - Make AES-NI support segmented buffers (iov or mbuf) so multisegmented
> >   inputs don't have to be copied.
> 
> I have the question regarding to the algorithm of GF field calculations
> used in the proposed implementation: why not use the recent researches
> in GCM calculations, e.g. described in [1], for further speed optimizations?
> 
> [1] - https://eprint.iacr.org/2013/157.pdf

The paper you linked to does not describe a new way of calculating
GHASH, but evalutation of a bug in their implementation using the
PCLMULQDQ instruction...

If you mean, why don't I use OpenSSL's code?  The reason is that their
code is a perl script that generates assmebly...  We don't have
perl in base.. and I didn't want more assembly in our tree (see below)..

Instead, I decided to use code from Intel's whitepaper:
Intel® Carry-Less Multiplication Instruction and its Usage for
Computing the GCM Mode

I didn't use their assembly version because I wanted to have
maintainable code, and also the same code can be used on both i386
and amd64 arches..  This turns out to also be a good thing, as when
I add segmented buffer support, it'll be much easier to add to the C
version, and I only have to do the work once for two arches...

Also, the software GF library that I wrote is using state of the art
algorithms...  An OpenBSD developer has tested my code and has seen
a significant performance improvement over their old code, and are
evaluating if they want to/can include it in their tree...

Hope this answers your question.  If not, please be more specific so
I can answer it.

Thanks.

-- 
  John-Mark Gurney  Voice: +1 415 225 5579

 "All that I will do, has been done, All that I have, has not."
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: CFR: AES-GCM and OpenCrypto work review

[Vsevolod Stakhov]

On 08/11/14 04:23, John-Mark Gurney wrote:
> Hello,
> 
> Over the last few months, I've been working on a project to add support
> for AES-GCM and AES-CTR modes to our OpenCrypto framework.  The work is
> sponsored by The FreeBSD Foundation and Netgate.
> 
> I plan on committing these patches early next week.  If you need more
> time for review, please email me privately and I will make delay.
> 
> The code has already been reviewed by Watson Ladd (the software crypto
> implementations) and Trevor Perrin (the aesni module part) and I have
> integrated these changes into the patch.
> 
> There are two patches, one is the changes for OpenCrypto and the test
> framework.  The other is the data files used by the test framework.
> The data is from NIST's CAVP program, and is about 20MB worth of test
> vectors.  (I just realized, should we look at compressing these on
> disk?)
> 
> Main patch (192KB):
> https://www.funkthat.com/~jmg/patches/aes.ipsec.5.patch
> 
> Data files (~20MB):
> https://www.funkthat.com/~jmg/patches/aes.ipsec.5.testing.patch
> 
> A list of notable changes in the patch:
> - Replacing crypto(4) w/ NetBSD's version + updates
> - Lots of man page updates, including CIOCFINDDEV and crypto(7) which
>   adds specifics about restrictions on the modes.
> - Allow sane useage of both _HARDWARE and _SOFTWARE flags.
> - Add a timing safe bcmp for MAC comparision.
> - Add a software implementation of GCM that uses a four bit lookup
>   table with parallelization.  This algorithm is possibly vulnerable to
>   timing attacks, but best known mitigation methods are used.  Using
>   a timing safe version is many times slower.
> - Added a CRYPTDEB macro that defaults to off.
> - Bring in some of OpenBSD's improvements to the OpenCrypto framework.
> - If an mbuf passed to the aesni module is only one segment, don't do
>   a copy.  This needs to be improved to support segmented buffers.
> - Remove the CRYPTO_F_REL flag.  It was meaningless.  It was used but
>   did not change any behavior.
> - Add function crypto_mbuftoiov to convert an mbuf to an iov.  This
>   also converts the software crypto to only use iov's even for a simple
>   linear buffer, and so simplifies the processing.
> - Add a dtrace probe for errors from the ioctl.
> - Add the CIOCCRYPTAEAD ioctl that allows userland processing (testing)
>   of AES-GCM and future AEAD modes.
> 
> Future improvements:
> - Support IV's longer than 12 bytes for GCM.
> - Make AES-NI support segmented buffers (iov or mbuf) so multisegmented
>   inputs don't have to be copied.

I have the question regarding to the algorithm of GF field calculations
used in the proposed implementation: why not use the recent researches
in GCM calculations, e.g. described in [1], for further speed optimizations?

[1] - https://eprint.iacr.org/2013/157.pdf

-- 
Vsevolod Stakhov
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: CFR: AES-GCM and OpenCrypto work review

[Dag-Erling Smørgrav]

John-Mark Gurney  writes:
> Over the last few months, I've been working on a project to add support
> for AES-GCM and AES-CTR modes to our OpenCrypto framework.  The work is
> sponsored by The FreeBSD Foundation and Netgate.
>
> I plan on committing these patches early next week.  If you need more
> time for review, please email me privately and I will make delay.

Please remember to bump __FreeBSD_version.

DES
-- 
Dag-Erling Smørgrav - d...@des.no
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: How exactly does the base toolchain determine WHICH language to build with?

[Dimitry Andric]

On 08 Nov 2014, at 01:23, Chris H  wrote:
> Sorry for the long title. I've been [needlessly] struggling
> with getting ports within the ports tree to build, on a
> fresh 11-CURRENT install from 2014-11-05. With custom
> KERNEL and WORLD built, and installed.
> Here's my situation, which has worked well since ~8.2;
> make.conf(5)
> WITHOUT_CLANG=true
> FAVORITE_COMPILER=gcc
> src.conf(5)
> WITHOUT_CLANG=true

If you don't want to use and build clang at all, I think you also need
these two settings (at least on recent -CURRENT):

WITHOUT_CLANG_BOOTSTRAP=foo
WITHOUT_CLANG_IS_CC=foo

The latter variable determines whether /usr/bin/cc, /usr/bin/c++ and so
on are linked to clang or gcc.  I suspect the root of all your problems
is that you don't have this setting.


> I'll neither argue, nor defend rational for w/o clang. To
> boring and out of scope for this thread. That said; I
> realize that lang/clang(33/34/35) is the default toolchain
> for 10+, and that's just fine by me. So I shouldn't be
> terribly surprised when install kernel/world, followed by
> make delete-old removes the clang built, or provided by
> the base install from the (initial) install procedure. But
> what _does_ surprise me, is that the install of lang/gcc-48
> does _not_ become the compiler of choice with the above
> $ENV, after [seemingly] deleting clang.

It does not work that way.  If you configure the base system to use gcc
only, individual ports can decide that they need newer C++ support, or
other features not available in the version of gcc in base (which is
very old by now).  In that case, they will use one of the gcc ports,
or possibly even one of the clang ports.

-Dimitry

___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: pkg 1.4 freeze please test test test!

[Marc UBM]

On Sat, 1 Nov 2014 23:45:49 +0100
Baptiste Daroussin  wrote:

> On Sat, Nov 01, 2014 at 04:13:32PM +0100, Marc UBM wrote:

[snip]

> > 
> > The update is failing for me with:
> > 
> > .../usr/ports/ports-mgmt/pkg-devel# make all install clean
> > ===>  Installing for pkg-1.4.0.a3
> > ===>  Checking if pkg already installed
> > pkg-static: sqlite error while executing DROP INDEX
> > packages_unique;CREATE UNIQUE INDEX packages_unique ON packages(name);
> > in file pkgdb.c:2246: UNIQUE constraint failed: packages.name *** Error
> > code 74
> > 
> > Stop.
> > make[1]: stopped in /usr/ports/ports-mgmt/pkg-devel
> > *** Error code 1
> > 
> > Stop.
> > make: stopped in /usr/ports/ports-mgmt/pkg-devel
> > 
> > 
> > 
> > portmaster fails with:
> > root@ubm:/usr/ports/ports-mgmt/pkg-devel# portmaster -d pkg
> > ===>>> No ORIGIN in /var/db/pkg/pkgconf-0.9.7/+CONTENTS
> > 
> > 
> > ===>>> Cannot continue
> > ===>>> Aborting update
> > 
> > ===>>> Killing background jobs
> > Terminated
> > ===>>> Exiting
> > 
> > make.conf related options:
> > 
> > #enable pkgng (might be superfluous)
> > WITH_PKGNG=yes
> > #enable PKGNG devel
> > WITH_PKGNG=devel
> > 
> > Am I doing something wrong?
> 
> You are doing nothing wrong but that probably means you have ancient packages
> that never got upgraded (in the old time it was allowed to have 2 packages
> installed with the same name) we have fixed that over the time and that is why
> we had unicity set to origin as a hack for a while, we are now moving to 
> unique
> name so you have to make sure that all your installed packages are up to date
> before moving to new pkg.
> 
> At least make sure you do not have 2 packages with the same name.
> 
> Concerning portmaster I have no idea why it now thinks you are not using
> pkg :(

I checked for packages with the same name and found none (via pkg
version). Meanwhile, the upgrade still fails for me. Has anybody got
any new ideas?

Thanks in advance!

Regards,
Marc

___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Jenkins build is back to normal : FreeBSD_HEAD #1796

[jenkins-admin]

See 

___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"