Re: [concurrency-interest] ThreadLocalRandom clinit troubles

[Martin Buchholz]

I support Peter at al's plan to add an API that ThreadLocalRandom et al can
use to get some system entropy without unbounded class dependency loading.

It should not surprise anyone that at Google, we are most interested in
running on Linux, so while we're waiting for a proper fix to happen we are
locally applying a simpler patch that tries explicitly /dev/urandom,
falling back to SecureRandom if necessary:

http://cr.openjdk.java.net/~martin/webrevs/openjdk9/ThreadLocalRandom-system-entropy/


On Mon, Jul 14, 2014 at 1:54 PM, Bernd  wrote:

> SecureRandom is unfortunatelly pretty  complex. It is interpreting the
> seed url in some way (the configuration you mentioned behave very special
> since Java 6) , it is mixing seed and continues data and it reorders the
> implementations used.
>
> JEP 123 intended to clear things, but getInstanceStrong() (which nobody
> uses?!) did not improve things IMHO.
>
> Bernd
>
> PS: I think the webrev changed since then, but the mail from Brad
> describes the problem well:
> http://mail.openjdk.java.net/pipermail/security-dev/2013-January/006288.html
> Am 14.07.2014 21:05 schrieb "Oleksandr Otenko" <
> oleksandr.ote...@oracle.com>:
>
>  Can someone summarize what happened?
>>
>> SecureRandom used to get entropy from /dev/random, which is configurable
>> through a policy file to /dev/urandom. Has this changed?
>>
>> Alex
>>
>> On 12/07/2014 00:33, Martin Buchholz wrote:
>>
>>  Thanks to Peter for digging into the secure seed generator classes and
>> coming up with a patch.  Openjdk security folks, please review.  I confess
>> to getting lost whenever I try to orient myself in the twisty maze of seed
>> generator implementation files.
>>
>>  Anyways, it seems important to have prngs like ThreadLocalRandom be
>> able to get a few bits of seed entropy without loading hundreds of classes
>> and without occupying any file descriptors permanently.  Perhaps at Google
>> we will go back to writing some simple non-portable startup code to read
>> /dev/urandom until openjdk security team comes up with a more principled
>> solution (but one that doesn't drag in too much machinery).
>>
>>
>> ___
>> Concurrency-interest mailing 
>> listConcurrency-interest@cs.oswego.eduhttp://cs.oswego.edu/mailman/listinfo/concurrency-interest
>>
>>
>>

Re: [concurrency-interest] ThreadLocalRandom clinit troubles

[Bernd]

SecureRandom is unfortunatelly pretty  complex. It is interpreting the seed
url in some way (the configuration you mentioned behave very special since
Java 6) , it is mixing seed and continues data and it reorders the
implementations used.

JEP 123 intended to clear things, but getInstanceStrong() (which nobody
uses?!) did not improve things IMHO.

Bernd

PS: I think the webrev changed since then, but the mail from Brad describes
the problem well:
http://mail.openjdk.java.net/pipermail/security-dev/2013-January/006288.html
Am 14.07.2014 21:05 schrieb "Oleksandr Otenko" :

>  Can someone summarize what happened?
>
> SecureRandom used to get entropy from /dev/random, which is configurable
> through a policy file to /dev/urandom. Has this changed?
>
> Alex
>
> On 12/07/2014 00:33, Martin Buchholz wrote:
>
>  Thanks to Peter for digging into the secure seed generator classes and
> coming up with a patch.  Openjdk security folks, please review.  I confess
> to getting lost whenever I try to orient myself in the twisty maze of seed
> generator implementation files.
>
>  Anyways, it seems important to have prngs like ThreadLocalRandom be able
> to get a few bits of seed entropy without loading hundreds of classes and
> without occupying any file descriptors permanently.  Perhaps at Google we
> will go back to writing some simple non-portable startup code to read
> /dev/urandom until openjdk security team comes up with a more principled
> solution (but one that doesn't drag in too much machinery).
>
>
> ___
> Concurrency-interest mailing 
> listConcurrency-interest@cs.oswego.eduhttp://cs.oswego.edu/mailman/listinfo/concurrency-interest
>
>
>

Re: [concurrency-interest] ThreadLocalRandom clinit troubles

[Oleksandr Otenko]

Can someone summarize what happened?

SecureRandom used to get entropy from /dev/random, which is configurable 
through a policy file to /dev/urandom. Has this changed?


Alex

On 12/07/2014 00:33, Martin Buchholz wrote:
Thanks to Peter for digging into the secure seed generator classes and 
coming up with a patch.  Openjdk security folks, please review.  I 
confess to getting lost whenever I try to orient myself in the twisty 
maze of seed generator implementation files.


Anyways, it seems important to have prngs like ThreadLocalRandom be 
able to get a few bits of seed entropy without loading hundreds of 
classes and without occupying any file descriptors permanently. 
 Perhaps at Google we will go back to writing some simple non-portable 
startup code to read /dev/urandom until openjdk security team comes up 
with a more principled solution (but one that doesn't drag in too much 
machinery).



___
Concurrency-interest mailing list
concurrency-inter...@cs.oswego.edu
http://cs.oswego.edu/mailman/listinfo/concurrency-interest

Re: ThreadLocalRandom clinit troubles

[Peter Levart]

Hi Sean, Alex

Here's a sum-up post:

http://mail.openjdk.java.net/pipermail/security-dev/2014-June/010700.html

Regards, Peter


On 07/14/2014 04:44 PM, Sean Mullan wrote:
I don't see a pointer to the webrev/patch -- did you forget to include 
it?


--Sean

On 07/11/2014 07:33 PM, Martin Buchholz wrote:

Thanks to Peter for digging into the secure seed generator classes and
coming up with a patch.  Openjdk security folks, please review. I 
confess
to getting lost whenever I try to orient myself in the twisty maze of 
seed

generator implementation files.

Anyways, it seems important to have prngs like ThreadLocalRandom be 
able to

get a few bits of seed entropy without loading hundreds of classes and
without occupying any file descriptors permanently.  Perhaps at 
Google we

will go back to writing some simple non-portable startup code to read
/dev/urandom until openjdk security team comes up with a more principled
solution (but one that doesn't drag in too much machinery).

Re: ThreadLocalRandom clinit troubles

[Sean Mullan]

I don't see a pointer to the webrev/patch -- did you forget to include it?

--Sean

On 07/11/2014 07:33 PM, Martin Buchholz wrote:

Thanks to Peter for digging into the secure seed generator classes and
coming up with a patch.  Openjdk security folks, please review.  I confess
to getting lost whenever I try to orient myself in the twisty maze of seed
generator implementation files.

Anyways, it seems important to have prngs like ThreadLocalRandom be able to
get a few bits of seed entropy without loading hundreds of classes and
without occupying any file descriptors permanently.  Perhaps at Google we
will go back to writing some simple non-portable startup code to read
/dev/urandom until openjdk security team comes up with a more principled
solution (but one that doesn't drag in too much machinery).

Re: ThreadLocalRandom clinit troubles

[Martin Buchholz]

Thanks to Peter for digging into the secure seed generator classes and
coming up with a patch.  Openjdk security folks, please review.  I confess
to getting lost whenever I try to orient myself in the twisty maze of seed
generator implementation files.

Anyways, it seems important to have prngs like ThreadLocalRandom be able to
get a few bits of seed entropy without loading hundreds of classes and
without occupying any file descriptors permanently.  Perhaps at Google we
will go back to writing some simple non-portable startup code to read
/dev/urandom until openjdk security team comes up with a more principled
solution (but one that doesn't drag in too much machinery).

Re: ThreadLocalRandom clinit troubles

[Bradford Wetmore]

On 6/26/2014 4:10 PM, Doug Lea wrote:

This seems to be the best idea yet, assuming that people are OK
with the changes to sun.security.provider.SeedGenerator and
NativeSeedGenerator.java


I've been meaning to review this thread, but have been chasing several 
urgent escalations.


Brad

Re: ThreadLocalRandom clinit troubles

[Doug Lea]

Peter: Thanks very much for attacking the shocking impact/complexity
of getting a few seed bits.

On 06/25/2014 01:41 PM, Peter Levart wrote:


Peeking around in the sun.security.provider package, I found there already is a
minimal internal infrastructure for obtaining random seed. It's encapsulated in
package-private abstract class sun.security.provider.SeedGenerator with 4
implementations. It turns out that, besides Java-only fall-back implementation
called ThreadedSeedGenerator and generic URLSeedGenerator, there are also two
implementations of NativeSeedGenerator (one for UNIX-es which is just an
extension of URLSeedGenerator and the other for Windows which uses MS
CryptoAPI). I made a few tweaks that allow this sub-infrastructure to be used
directly in ThreadLocalRandom and SplittableRandom:

http://cr.openjdk.java.net/~plevart/jdk9-dev/TLR_SR_SeedGenerator/webrev.01/



This seems to be the best idea yet, assuming that people are OK
with the changes to sun.security.provider.SeedGenerator and
NativeSeedGenerator.java

-Doug

Re: ThreadLocalRandom clinit troubles

[Peter Levart]

To sum-up: We have a problem with TLR initialization since by default it 
uses networking code to compute initial "seeder" value which can execute 
user code in at least two situations:


- when "sun.net.spi.nameservice.provider" system property is defined to 
use custom NameService provider

- when custom SecurityManager is in effect while TLR is being initialized
- also, when "java.util.secureRandomSeed" is defined, at least on 
Windows this means that default SecureRandom algorithm will be using 
networking code too to gather system entropy


We could work-around these problems by delaying initialization of 
NameService provider(s), by-passing SecurityManager when obtaining MAC 
address from NetworkInterface and extending the semantics of 
"java.util.secureRandomSeed" property to specify explicit SecureRandom 
algorithm and provider to use when obtaining SecureRandom instance, like 
in the following patch:


http://cr.openjdk.java.net/~plevart/jdk9-dev/TLR.initialSeed/webrev.01/

But on the other hand this seems too many knobs to worry about. Ideally 
one would like to always use OS provided native seed source, but 
SecureRandom (with all the security providers infrastructure) seems too 
heavy-weight to be used in classes like ThreadLocalRandom or 
SplittableRandom by default since they can be initialized very early in 
the start-up sequence. I made an experiment with class-loading. Recent 
JDK9 build loads 381 classes when running the following empty program on 
Linux:


public class test0 {
public static void main(String[] args) {
}
}

...ThreadLocalRandom is not among them. But in special configurations 
(like when using java agents) or in the future, it could be. The 
following program:


public class test {
public static void main(String[] args) {
java.util.concurrent.ThreadLocalRandom.current();
}
}

...loads 403 classes. That's 22 more than an empty program. The 
following classes are loaded in addition:


+ java.util.Random
+ java.util.concurrent.ThreadLocalRandom
+ java.net.NetworkInterface
+ java.net.NetworkInterface$1
+ java.net.InterfaceAddress
+ java.net.InetAddress
+ sun.security.action.GetBooleanAction
+ java.net.InetAddress$1
+ java.net.InetAddress$InetAddressHolder
+ java.net.InetAddress$Cache
+ java.net.InetAddress$Cache$Type
+ java.net.InetAddressImplFactory
+ java.net.InetAddressImpl
+ java.net.Inet6AddressImpl
+ sun.net.spi.nameservice.NameService
+ java.net.InetAddress$2
+ java.net.Inet4Address
+ java.net.Inet6Address
+ java.net.Inet6Address$Inet6AddressHolder
+ java.net.DefaultInterface
+ java.net.NetworkInterface$2
+ sun.nio.ch.Interruptible

If I run the same program but set the "java.util.secureRandomSeed=true", 
it loads 435 classes. Besides 381 classes loaded by an empty program, 
the following 54 classes are loaded in addition:


+ java.util.Random
+ java.util.concurrent.ThreadLocalRandom
+ java.security.SecureRandom
+ sun.security.jca.Providers
+ java.lang.InheritableThreadLocal
+ sun.security.jca.ProviderList
+ sun.security.jca.ProviderConfig
+ java.security.Provider
+ sun.security.jca.ProviderList$3
+ sun.security.jca.ProviderList$1
+ java.security.Provider$ServiceKey
+ java.security.Provider$EngineDescription
+ sun.misc.FloatingDecimal
+ sun.misc.FloatingDecimal$BinaryToASCIIConverter
+ sun.misc.FloatingDecimal$ExceptionalBinaryToASCIIBuffer
+ sun.misc.FloatingDecimal$BinaryToASCIIBuffer
+ sun.misc.FloatingDecimal$1
+ sun.misc.FloatingDecimal$ASCIIToBinaryConverter
+ sun.misc.FloatingDecimal$PreparedASCIIToBinaryBuffer
+ sun.misc.FDBigInteger
+ sun.security.jca.ProviderList$2
+ java.security.Security
+ java.security.Security$1
+ java.util.Properties$LineReader
+ java.util.AbstractList$Itr
+ sun.security.jca.ProviderConfig$2
+ sun.security.provider.Sun
+ sun.security.provider.SunEntries
+ sun.security.provider.SunEntries$1
+ java.security.SecureRandomSpi
+ sun.security.provider.NativePRNG
+ sun.security.provider.NativePRNG$Variant
+ sun.security.provider.NativePRNG$1
+ sun.security.provider.NativePRNG$2
+ java.net.URI
+ java.net.URI$Parser
+ sun.security.provider.NativePRNG$RandomIO
+ sun.security.provider.NativePRNG$Blocking
+ sun.security.provider.NativePRNG$NonBlocking
+ java.util.LinkedHashMap$LinkedEntrySet
+ java.util.LinkedHashMap$LinkedHashIterator
+ java.util.LinkedHashMap$LinkedEntryIterator
+ java.security.Provider$Service
+ java.security.Provider$UString
+ java.util.LinkedHashSet
+ java.util.LinkedHashMap$LinkedValues
+ java.util.LinkedHashMap$LinkedValueIterator
+ java.util.Collections$UnmodifiableSet
+ java.util.Collections$UnmodifiableCollection$1
+ java.util.LinkedHashMap$LinkedKeySet
+ java.util.LinkedHashMap$LinkedKeyIterator
+ sun.security.jca.GetInstance
+ sun.security.jca.GetInstance$Instance
+ sun.nio.ch.Interruptible


This seems too heavy-weight even if the initialization issue on Windows 
where default SecureRandom algorithm is using networking code to gather 
system entropy is worked-around by reque

Re: ThreadLocalRandom clinit troubles

[Peter Levart]

On 06/24/2014 06:01 PM, Martin Buchholz wrote:




On Tue, Jun 24, 2014 at 7:03 AM, Peter Levart > wrote:



I would rather use SecureRandom.generateSeed() instance method
instead of SecureRandom.nextBytes(). Why? Because every
SecureRandom instance has to initialize it's seed 1st before
getBytes() can provide the next random bytes from the PRNG. Since
we only need the 1st 8 bytes from the SecureRandom instance to
initialize TLR's seeder, we might as well directly call the
SecureRandom.generateSeed() method.


If I strace this program on Linux using strace -ff -q java SecureRandoms:

public class SecureRandoms {
public static void main(String[] args) throws Throwable {
byte[] bytes = new byte[8];
new java.security.SecureRandom().nextBytes(bytes);
}
}

I see a read from /dev/urandom, but not from /dev/random, so I 
conclude your intuitive understanding of how the seeding works must be 
wrong.  It makes sense that NativePRNG doesn't need to do any special 
seeding of its own, since it reuses the operating system's.


You're right. I checked again. The  NativePRNG is actually using 
/dev/urandom (by default unless java.security.egd or securerandom.source 
is defined). It's mixing the /dev/urandom stream with the stream 
obtained from SHA1 generator which is seeded by 20 bytes from 
/dev/urandom too. So by default yes, plain NativePRNG (the default on 
UNIX-es) is using /dev/urandom for nextBytes(), but this can be changed 
by defining java.security.egd or securerandom.source system property. I 
still think that for configuration-independent PRNG seed on UNIX-es it's 
better to invoke generateSeed() on NativePRNG$NonBlocking, which 
hard-codes /dev/urandom and doesn't mix it with SHA1 stream.


On Windows, there's a different story, since the default SecureRandom 
algorithm is SHA1, seeded by SeedGenerator.getSystemEntropy() and 
SeedGenerator.generateSeed(). The former call includes invoking 
networking code and resolving local host name. Which we would like to 
avoid. So I think we need a nicer story on windows then just: new 
SecureRandom().nextBytes(). I propose requesting explicit algorithm / 
provider on each particular platform that we know does best what we 
want, rather than using default which can still be used as a fall-back.


Regards, Peter

Re: ThreadLocalRandom clinit troubles

[Peter Levart]

Hi Martin,

On 06/22/2014 07:12 PM, Martin Buchholz wrote:

We know that loading the networking machinery is problematic.  On Linux we
would be content to hard-code a read from /dev/urandom, which is safer and
strictly more random than the existing network hardware determination, but
y'all will reject that as too system-dependent (insufficient machinery!).
H  maybe not  as long as we code up a good fallback ...

I learned that SecureRandom by default on Unix uses /dev/random for "seed
bytes" and /dev/urandom for nextBytes.

Here's my proposal, that in the default case on Unix doesn't load any
machinery, and as a fallback loads the SecureRandom machinery instead of
the network machinery, while maintaining the ultra-secure behavior of the
java.util.secureRandomSeed system property:


 private static long initialSeed() {
 byte[] seedBytes = initialSeedBytes();
 long s = (long)(seedBytes[0]) & 0xffL;
 for (int i = 1; i < seedBytes.length; ++i)
 s = (s << 8) | ((long)(seedBytes[i]) & 0xffL);
 return s ^ mix64(System.currentTimeMillis()) ^
mix64(System.nanoTime());
 }

 private static byte[] initialSeedBytes() {
 String pp = java.security.AccessController.doPrivileged(
 new sun.security.action.GetPropertyAction(
 "java.util.secureRandomSeed"));
 boolean secureRandomSeed = (pp != null &&
pp.equalsIgnoreCase("true"));
 if (secureRandomSeed)
 return java.security.SecureRandom.getSeed(8);
 final byte[] seedBytes = new byte[8];
 File seedSource = new File("/dev/urandom");
 if (seedSource.exists()) {
 try (FileInputStream stream = new FileInputStream(seedSource)) {
 if (stream.read(seedBytes) == 8)
 return seedBytes;
 } catch (IOException ignore) { }
 }
 new java.security.SecureRandom().nextBytes(seedBytes);
 return seedBytes;
 }


So on platforms lacking "/dev/urandom" special file (Windows only?), the 
fall-back would be to use SecureRandom.nextBytes() whatever default 
SecureRandom PRNG is configured to be on such platform. This might be a 
user-supplied SecureRandom PRNG. The user might want it's own default 
SecureRandom but not use it for TLR's seed computation (unless requested 
by "java.util.secureRandomSeed" system property).


I would rather use SecureRandom.generateSeed() instance method instead 
of SecureRandom.nextBytes(). Why? Because every SecureRandom instance 
has to initialize it's seed 1st before getBytes() can provide the next 
random bytes from the PRNG. Since we only need the 1st 8 bytes from the 
SecureRandom instance to initialize TLR's seeder, we might as well 
directly call the SecureRandom.generateSeed() method. That's one reason. 
The other is the peculiar initialization of default SecureRandom 
algorithm on Windows (see below)...


Even if user does not provide it's own default SecureRandom PRNG, there 
are basically two algorithms that are used by default on OpenJDK:


On Solaris/Linux/Mac/AIX:

the "NativePRNG" algorithm from "SUN" provider (implemented by 
sun.security.provider.NativePRNG) which uses /dev/random for getBytes() 
and /dev/urandom (or whatever is configured with "java.security.egd" or 
"securerandom.source" system properties) for generateSeed(), and


On Windows:

the "SHA1PRNG" algorithm from "SUN" provider (implemented by 
sun.security.provider.SecureRandom) which uses SHA1 message digest for 
generating random numbers with seed computed by gathering system entropy...



The most problematic one is the default on Windows platform (the 
platform that does not have the "/dev/urandom" special file and would be 
used as a fall-back by your proposal) - 
sun.security.provider.SecureRandom. This one seeds itself by 
constructing an instance of itself with the result returned from 
SeedGenerator.getSystemEntropy() method. This method, among other 
things, uses networking code to gather system entropy:


...
md.update
(InetAddress.getLocalHost().toString().getBytes());
...

This is problematic since it not only initializes NameService providers 
but also uses them to resolve local host name. This can block for 
several seconds on unusual configurations and was the main motivation to 
replace similar code in TLR with code that uses 
NetworkInterface.getHardwareAddress() instead. Using 
SecureRandom.generateSeed() instead of SecureRandom.getBytes() does not 
invoke SeedGenerator.getSystemEntropy(), but uses just 
SeedGenerator.generateSeed(), which by default on Windows uses 
ThreadedSeedGenerator.getSeedBytes()...


I showed how we could suppress NameService providers initialization 
while still using NetworkInterface.getHardwareAddress() for TLR's seeder 
initialization. If that's not enough and we would like to get-away 
without using networking code at al

Re: ThreadLocalRandom clinit troubles

[Bradford Wetmore]

Martin,

Thanks for filing.  I was positive there was already a bug for this, but 
for the life of me I can't find it now.  There's some other more minor 
cleanup that needs to take place, but seems like I've been in 
escalation/firefighting mode for more than a year now and it hasn't 
bubbled to the top.


Brad


On 6/21/2014 9:05 PM, Martin Buchholz wrote:

While looking at NativePRNG, I filed

https://bugs.openjdk.java.net/browse/JDK-8047769

SecureRandom should be more frugal with file descriptors

If I run this java program on Linux

public class SecureRandoms {
 public static void main(String[] args) throws Throwable {
 new java.security.SecureRandom();
 }
}

it creates 6 file descriptors for /dev/random and /dev/urandom, as shown
by:

strace -q -ff -e open java SecureRandoms |& grep /dev/
[pid 20769] open("/dev/random", O_RDONLY) = 5
[pid 20769] open("/dev/urandom", O_RDONLY) = 6
[pid 20769] open("/dev/random", O_RDONLY) = 7
[pid 20769] open("/dev/random", O_RDONLY) = 8
[pid 20769] open("/dev/urandom", O_RDONLY) = 9
[pid 20769] open("/dev/urandom", O_RDONLY) = 10

Looking at jdk/src/solaris/classes/sun/security/provider/NativePRNG.java
it looks like 2 file descriptors are created for every variant of
NativePRNG, whether or not they are ever used. Which is wasteful. In
fact, you only ever need at most two file descriptors, one for
/dev/random and one for /dev/urandom.

Further, it would be nice if the file descriptors were closed when idle
and lazily re-created. Especially /dev/random should typically be used
at startup and never thereafter.


On Fri, Jun 20, 2014 at 7:59 AM, Alan Bateman mailto:alan.bate...@oracle.com>> wrote:

On 20/06/2014 15:02, Peter Levart wrote:


And, as Martin pointed out, it seems to be used for tests that
exercise particular responses from NameService API to test the
behaviour of JDK classes. It would be a shame for those tests to
go away.

We've been talking about removing it for many years because it has
been so troublesome. If we really need to having something for
testing then I don't think it needs to be general purpose, we can
get right of the lookup at least.

-Alan.