[jira] [Commented] (RNG-168) LXM family of random number generators
[
https://issues.apache.org/jira/browse/RNG-168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17510923#comment-17510923
]
Alex Herbert commented on RNG-168:
--
I have created a set of LXM generators using the examples provided in Steele &
Vigna (2021). These generators all use the same basic steps:
{noformat}
s = state of LCG
t = state of XBG
generate():
z <- mix(combine(w upper bits of s, w upper bits of t))
s <- LCG state update
t <- XBG state update
return z{noformat}
All the generators use w as 64-bits with the combine operation being an
addition. The mix function is a variant on the final mixing function used in
MurmurHash3. This mix function, with different constants, is used in the
SplitMix64 generator and is provided in Steele and Vigna's paper. One exception
is a faster mix function that matches that used in Vigna's StarStar generators
such as XoRoShiRo128StarStar.
I have added the following classes which implement the same LXM generators as
JDK 17:
||JDK 17||Class||RandomSource||XBG||
|L64X128StarStarRandom|L64X128StarStar|L64_X128_SS|XoRoShiRo128|
|L64X128MixRandom|L64X128Mix|L64_X128_MIX|XoRoShiRo128|
|L64X256MixRandom|L64X256Mix|L64_X256_MIX|XoShiRo256|
|L64X1024MixRandom|L64X1024Mix|L64_X1024_MIX|XoRoShiRo1024|
|L128X128MixRandom|L128X128Mix|L128_X128_MIX|XoRoShiRo128|
|L128X256MixRandom|L128X256Mix|L128_X256_MIX|XoShiRo256|
|L128X1024MixRandom|L128X1024Mix|L128_X1024_MIX|XoRoShiRo1024|
I have used the equivalent class in the JDK 17 these to create reference data
for the new generators. In doing so I found two bugs in the seeding and
construction routines of the JDK generators. These have been raised as issues
([JDK-8283083|https://bugs.java.com/bugdatabase/view_bug.do?bug_id=JDK-8283083]
and
[JDK-8282144|https://bugs.java.com/bugdatabase/view_bug.do?bug_id=JDK-8282144]).
Thus the seeds used for the reference generators could not be a full random
byte[] of the correct length. Six generators can be seeded using a random
byte[] that uses only positively signed bytes in any position that is not
modulus 8 == 0. One generator has to be seeded using a single long value. I
extracted the routine used in JDK 17 to generate the full seed and recorded it
for the unit tests. This routine initialises the state of the 128-bit LCG to 1
so the seeding for the reference data could be improved when the bugs are fixed.
One item to note is that the byte[] seed is converted in the JDK generators
using big-endian format. The bytes create in order: the LCG additive parameter;
the LCG state; and the XBG state. Seed conversion in the RNG simple module uses
little-endian format. So it would be impossible to create a generator using the
same seed the outputs the same sequence without a modification of the byte[]
seed conversion in RNG simple, i.e.
{code:java}
byte[] seed = SecureRandom.generateSeed(Long.Bytes * 6);
RandomGenerator g = RandomGeneratorFactory.of("L128X128MixRandom").create(seed);
UniformRandomProvider rng = RandomSource.L128_X128_MIX.create(seed);
g.nextLong() != rng.nextLong();
{code}
In addition, since the generators in commons RNG have a base XBG implementation
already in the library, the seed must be passed up to the super-class during
construction. It is simplest to leave the initial part of the long[] the same
and add the extra seed bytes to the end. Thus the order of the seed in Commons
RNG is: the XBG state; the LCG state; and the LCG additive parameter. This is a
swap of each part of the seed. Leaving the additive parameter at the end is an
arbitrary choice but note that it is a requirement for this to be odd. The
constructor sets the input value to odd. If this is the last component of the
seed then it can be documented that the seed is used entirely except for the
final trailing bit.
h2. Quality
Here are results from the various test suites:
{noformat}
RNG Dieharder ∩ TestU01 (BigCrush) ∩
PractRand ∩
L64_X128_SS 0,1,0,0,0 0,0,0,0,0 -,-,-
L64_X128_MIX0,0,0,0,0 0,0,0,1,2 -,-,-
L64_X256_MIX0,0,0,0,0 0,0,0,0,0 -,-,-
L64_X1024_MIX 0,0,0,0,0 0,0,1,0,1 -,-,-
L128_X128_MIX 1,0,0,0,0 0,1,2,0,0 -,-,-
L128_X256_MIX 1,0,0,0,0 0,0,0,2,0 -,-,-
L128_X1024_MIX 0,0,0,0,0 0,0,0,2,1 -,-,-
{noformat}
No surprises here. These are high quality generators that pass 4TB of output
through PractRand and have a few spurious fails in Dieharder and BigCrush.
h2. Performance
Here is a JMH benchmark result for nextLong run on JDK 11:
||Random Source||Method||Score||Baselined||Relative to parent XBG||
|
[GitHub] [commons-rng] aherbert opened a new pull request #104: RNG-168: LXM family of random number generators
aherbert opened a new pull request #104: URL: https://github.com/apache/commons-rng/pull/104 Add support for generators included in JDK 17: L64X128StarStar L64X128Mix L64X256Mix L64X1024Mix L128X128Mix L128X256Mix L128X1024Mix Added benchmark for support routines for computing the unsigned long multiplications in the 128-bit LCG (linear congruential generator). Added a benchmark for the jump function to allow comparison with the equivalent base XBG (xor-based generator). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@commons.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (RNG-168) LXM family of random number generators
[
https://issues.apache.org/jira/browse/RNG-168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alex Herbert updated RNG-168:
-
Description:
JDK 17 updates the random number support and adds a LXM family of random number
generators. The generators are described in the following paper:
{noformat}
Steele and Vigna (2021)
LXM: better splittable pseudorandom number generators (and almost as fast).
Proceedings of the ACM on Programming Languages, Volume 5, Article 148, pp 1–31
{noformat}
[https://doi.org/10.1145/3485525]
The generators mix the output of a linear conguential generator (LCG; L) and a
Xor-based generator (XBG; X). The paper provides code examples for
L64X128MixRandom and L128X256MixRandom from JDK 17.
Details of the combinations of the linear conguential generator (LCG; L) and
Xor-based generators (XBG; X) used in JDK 17 are provided in the package
javadoc:
[JDK 17
java.util.random|https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/util/random/package-summary.html]
The current commons RNG library implements all the XBG generators. Adding
support for these generators should add a LCG to combine with the output of the
XBG via the specified mix function.
All constants are supplied in the JDK 17 package javadoc to allow
implementation with the exception of the mixing functions lea32 and lea64. The
lea64 function is provided in the paper above but the reference origin is
personal communication with Doug Lea (2013). Thus the lea32 mix function for
32-bit generators is not published. Obtaining from the JDK source code is
possible but would be subject to the Java licence.
The generators support jump functionality and are suitable for streaming in
large numbers for parallel applications (the paper tests up to 2^24 generators)
by using a different additive constant for the LCG for each generator to ensure
no sequence overlap. Creating a new additive constant is significantly faster
than performing a jump operation.
was:
JDK 17 updates the random number support and adds a LXM family of random number
generators. The generators are described in the following paper:
{noformat}
Blackman and Vigna (2021)
Scrambled Linear Psuedorandom Number Generators.
ACM Transactions on Mathematical Software, vol 47, pp 1–32
{noformat}
[https://dl.acm.org/doi/10.1145/3460772]
The generators mix the output of a linear conguential generator (LCG; L) and a
Xor-based generator (XBG; X). The paper provides code examples for
L64X128MixRandom and L128X256MixRandom from JDK 17.
Details of the combinations of the linear conguential generator (LCG; L) and
Xor-based generators (XBG; X) used in JDK 17 are provided in the package
javadoc:
[JDK 17
java.util.random|https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/util/random/package-summary.html]
The current commons RNG library implements all the XBG generators. Adding
support for these generators should add a LCG to combine with the output of the
XBG via the specified mix function.
All constants are supplied in the JDK 17 package javadoc to allow
implementation with the exception of the mixing functions lea32 and lea64. The
lea64 function is provided in the paper above but the reference origin is
personal communication with Doug Lea (2013). Thus the lea32 mix function for
32-bit generators is not published. Obtaining from the JDK source code is
possible but would be subject to the Java licence.
The generators support jump functionality and are suitable for streaming in
large numbers for parallel applications (the paper tests up to 2^24 generators)
by using a different additive constant for the LCG for each generator to ensure
no sequence overlap. Creating a new additive constant is significantly faster
than performing a jump operation.
> LXM family of random number generators
> --
>
> Key: RNG-168
> URL: https://issues.apache.org/jira/browse/RNG-168
> Project: Commons RNG
> Issue Type: New Feature
> Components: core, simple
>Affects Versions: 1.4
>Reporter: Alex Herbert
>Priority: Major
>
> JDK 17 updates the random number support and adds a LXM family of random
> number generators. The generators are described in the following paper:
> {noformat}
> Steele and Vigna (2021)
> LXM: better splittable pseudorandom number generators (and almost as fast).
> Proceedings of the ACM on Programming Languages, Volume 5, Article 148, pp
> 1–31
> {noformat}
> [https://doi.org/10.1145/3485525]
> The generators mix the output of a linear conguential generator (LCG; L) and
> a Xor-based generator (XBG; X). The paper provides code examples for
> L64X128MixRandom and L128X256MixRandom from JDK 17.
> Details of the combinations of the linear conguential generator (LCG; L) and
> Xor-based generators (XBG; X) used in JDK 17 are provided in
[jira] [Commented] (DAEMON-383) Provide option for service stop using Windows PRESHUTDOWN event
[
https://issues.apache.org/jira/browse/DAEMON-383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17510799#comment-17510799
]
Michael Litwak commented on DAEMON-383:
---
For reference see {{SERVICE_ACCEPT_PRESHUTDOWN}} at
[https://docs.microsoft.com/en-us/windows/win32/services/service-control-handler-function]
> Provide option for service stop using Windows PRESHUTDOWN event
> ---
>
> Key: DAEMON-383
> URL: https://issues.apache.org/jira/browse/DAEMON-383
> Project: Commons Daemon
> Issue Type: Improvement
> Components: Procrun
>Affects Versions: 1.1.0
>Reporter: George McCone
>Priority: Major
>
> Starting with Windows Vista, Services can be configured to be shutdown in
> either one of two groups. The normal group or the preshutdown group. Since
> the normal group is all shutdown concurrently, this can cause services that
> are dependent on other service to fail during shutdown. If the service has
> been registered as a preshutdown service, it is notified to stop prior to the
> normal group.
> Being able to configure preshutdown can be very important if one of your
> services is an infrastructure service (such as a Database or JMS Server),
> being able to have the services start as a dependency and shutdown as a
> preshutdown makes sure these resources are available during the entire
> lifecycle for a system.
> A description can be found here.
> https://msdn.microsoft.com/en-us/library/windows/desktop/ms685149(v=vs.85).aspx
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (VFS-757) [SFTP] Configure whether exec detection is enabled #80
[ https://issues.apache.org/jira/browse/VFS-757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17510749#comment-17510749 ] Gary D. Gregory commented on VFS-757: - >From the PR: One of our customer has an interesting sftp server. The server does not allow any further channels to be established after the execution of the exec command (getUid()). So it is not possible to do any filesystem operations, because at first the exec command is executed to test if exec is disabled. Maybe the server things we try to break the security or something else. So my idea to solve this problem is to make execDisabled configurable. This is done here. > [SFTP] Configure whether exec detection is enabled #80 > -- > > Key: VFS-757 > URL: https://issues.apache.org/jira/browse/VFS-757 > Project: Commons VFS > Issue Type: New Feature >Reporter: Gary D. Gregory >Assignee: Gary D. Gregory >Priority: Major > Fix For: 2.7.0 > > > [SFTP] Configure whether exec detection is enabled #80. > [https://github.com/apache/commons-vfs/pull/80] -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (VFS-816) Apache commons VFS2 code is taking 5 min to resolve the file in SFTP
[ https://issues.apache.org/jira/browse/VFS-816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17510748#comment-17510748 ] Gary D. Gregory commented on VFS-816: - I see that VFS-757 seems related to this. > Apache commons VFS2 code is taking 5 min to resolve the file in SFTP > > > Key: VFS-816 > URL: https://issues.apache.org/jira/browse/VFS-816 > Project: Commons VFS > Issue Type: Bug >Affects Versions: 2.7.0 >Reporter: Veeraswamy Naidu Pulaparti >Priority: Major > Attachments: SFTPTest.rtf > > > Hi Team, when i used the commons-vfs2-2.7.0.jar file in my application it is > taking around 5 min to resolve the file for 1kb as well. > when the same code is used by changing the packages from vf2 -> vfs and using > the jar commons-vfs-2.0.jar it is working normally and the same step is > coming out within milliseconds. > Could you please check the same and let me know if i am missing anything > here. For reference i am attaching reference java code. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (VFS-816) Apache commons VFS2 code is taking 5 min to resolve the file in SFTP
[ https://issues.apache.org/jira/browse/VFS-816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17510734#comment-17510734 ] Veeraswamy Naidu Pulaparti commented on VFS-816: [~ggregory] After deep dive of the jstack and vfs code base , it is blocking at detectExecDisabled() org.apache.commons.vfs2.provider.sftp.SftpFileSystem.executeCommand(SftpFileSystem.java:322). If i set the filesystem options to SftpFileSystemConfigBuilder.getInstance().setDisableDetectExecChannel(opts, true); It is working fine . May i know the functionality of this check implemented in 2.7.0 and what will happen if i always go with setting this option as true like above? Thanks in advance. > Apache commons VFS2 code is taking 5 min to resolve the file in SFTP > > > Key: VFS-816 > URL: https://issues.apache.org/jira/browse/VFS-816 > Project: Commons VFS > Issue Type: Bug >Affects Versions: 2.7.0 >Reporter: Veeraswamy Naidu Pulaparti >Priority: Major > Attachments: SFTPTest.rtf > > > Hi Team, when i used the commons-vfs2-2.7.0.jar file in my application it is > taking around 5 min to resolve the file for 1kb as well. > when the same code is used by changing the packages from vf2 -> vfs and using > the jar commons-vfs-2.0.jar it is working normally and the same step is > coming out within milliseconds. > Could you please check the same and let me know if i am missing anything > here. For reference i am attaching reference java code. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (LANG-1662) Create methods on ReflectionToStringBuilder to reflect only select fields
[ https://issues.apache.org/jira/browse/LANG-1662?focusedWorklogId=745671=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-745671 ] ASF GitHub Bot logged work on LANG-1662: Author: ASF GitHub Bot Created on: 22/Mar/22 09:22 Start Date: 22/Mar/22 09:22 Worklog Time Spent: 10m Work Description: garydgregory commented on pull request #849: URL: https://github.com/apache/commons-lang/pull/849#issuecomment-1074923276 Hi @GutoVeronezi You did not answer my question. See my code example; it should be one of your unit tests. I do not know (yet) what would be reasonable as the behavior. @kinow , any thoughts? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@commons.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 745671) Remaining Estimate: 4h 10m (was: 4h 20m) Time Spent: 1h 50m (was: 1h 40m) > Create methods on ReflectionToStringBuilder to reflect only select fields > - > > Key: LANG-1662 > URL: https://issues.apache.org/jira/browse/LANG-1662 > Project: Commons Lang > Issue Type: Improvement > Components: lang.builder.* >Reporter: Daniel Augusto Veronezi Salvador >Priority: Minor > Original Estimate: 6h > Time Spent: 1h 50m > Remaining Estimate: 4h 10m > > *ReflectionToStringBuilder* has methods to exclude fields from *toString*; If > we have an object with several fields and want to reflect only a fews, we > have to list all the fields that we don't want to reflect and pass to > *excludeFieldNames*. > Would be valid implement a way to pass the fields that we want and reflect > only the selected fields? > > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [commons-lang] garydgregory commented on pull request #849: [LANG-1662] Create methods on ReflectionToStringBuilder to reflect only select fields
garydgregory commented on pull request #849: URL: https://github.com/apache/commons-lang/pull/849#issuecomment-1074923276 Hi @GutoVeronezi You did not answer my question. See my code example; it should be one of your unit tests. I do not know (yet) what would be reasonable as the behavior. @kinow , any thoughts? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@commons.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
