[jira] [Commented] (CRYPTO-169) Unable to load native library on Apple Silicon (M1 Mac)
[ https://issues.apache.org/jira/browse/CRYPTO-169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17788185#comment-17788185 ] Sebb commented on CRYPTO-169: - OpenSslCryptoRandom static block issue reported as CRYPTO-178; now fixed > Unable to load native library on Apple Silicon (M1 Mac) > --- > > Key: CRYPTO-169 > URL: https://issues.apache.org/jira/browse/CRYPTO-169 > Project: Commons Crypto > Issue Type: Bug >Reporter: L. C. Hsieh >Priority: Major > Attachments: image-2023-10-21-17-51-41-260.png, > image-2023-10-21-17-52-02-237.png, image-2023-10-21-17-53-16-898.png, > image-2023-10-21-17-59-13-866.png, image-2023-10-22-23-58-57-254.png > > > Recently as trying to upgrade commons-crypto from 1.1.0 to 1.2.0 in Spark > (SPARK-45617), got the following error in one test: > ``` > [info] Cause: java.security.GeneralSecurityException: Native library is not > loaded > [info] at > org.apache.commons.crypto.random.OpenSslCryptoRandom.checkNative(OpenSslCryptoRandom.java:79) > [info] at > org.apache.commons.crypto.random.OpenSslCryptoRandom.(OpenSslCryptoRandom.java:65) > > [info] at java.base/java.lang.Class.forName0(Native Method) > > [info] at java.base/java.lang.Class.forName(Class.java:467) > [info] at > org.apache.commons.crypto.utils.ReflectionUtils.getClassByNameOrNull(ReflectionUtils.java:93) > [info] at > org.apache.commons.crypto.utils.ReflectionUtils.getClassByName(ReflectionUtils.java:64) > [info] at > org.apache.commons.crypto.random.CryptoRandomFactory.getCryptoRandom(CryptoRandomFactory.java:189) > [info] at > org.apache.spark.security.CryptoStreamUtils$.createInitializationVector(CryptoStreamUtils.scala:138) > [info] at > org.apache.spark.security.CryptoStreamUtils$.createCryptoOutputStream(CryptoStreamUtils.scala:56) > [info] at > org.apache.spark.serializer.SerializerManager.$anonfun$wrapForEncryption$3(SerializerManager.scala:151) > > ``` > I did a quick check. commons-crypto-1.2.0.jar only contain x86_64 Mac native > library, it doesn't provide pre-built native library for Apple Silicon. > But I also checked commons-crypto-1.1.0.jar which Spark uses currently. It > also doesn't provide native library for Apple Silicon. > Since they both don't contain native library for Apple Silicon, why > commons-crypto-1.2.0 doesn't work and what makes the difference between > commons-crypto-1.1.0 and commons-crypto-1.2.0? -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (CRYPTO-169) Unable to load native library on Apple Silicon (M1 Mac)
[ https://issues.apache.org/jira/browse/CRYPTO-169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17787684#comment-17787684 ] Sebb commented on CRYPTO-169: - I think there is a problem with the recent changes to OpenSslCryptoRandom. It seems wrong for the static block to raise an Exception. This means that the method OpenSslCryptoRandom.isNativeCodeEnabled() cannot be used if the class does not load. Rather than checking for errors in the static block, they should be checked by the methods of the class as necessary. > Unable to load native library on Apple Silicon (M1 Mac) > --- > > Key: CRYPTO-169 > URL: https://issues.apache.org/jira/browse/CRYPTO-169 > Project: Commons Crypto > Issue Type: Bug >Reporter: L. C. Hsieh >Priority: Major > Attachments: image-2023-10-21-17-51-41-260.png, > image-2023-10-21-17-52-02-237.png, image-2023-10-21-17-53-16-898.png, > image-2023-10-21-17-59-13-866.png, image-2023-10-22-23-58-57-254.png > > > Recently as trying to upgrade commons-crypto from 1.1.0 to 1.2.0 in Spark > (SPARK-45617), got the following error in one test: > ``` > [info] Cause: java.security.GeneralSecurityException: Native library is not > loaded > [info] at > org.apache.commons.crypto.random.OpenSslCryptoRandom.checkNative(OpenSslCryptoRandom.java:79) > [info] at > org.apache.commons.crypto.random.OpenSslCryptoRandom.(OpenSslCryptoRandom.java:65) > > [info] at java.base/java.lang.Class.forName0(Native Method) > > [info] at java.base/java.lang.Class.forName(Class.java:467) > [info] at > org.apache.commons.crypto.utils.ReflectionUtils.getClassByNameOrNull(ReflectionUtils.java:93) > [info] at > org.apache.commons.crypto.utils.ReflectionUtils.getClassByName(ReflectionUtils.java:64) > [info] at > org.apache.commons.crypto.random.CryptoRandomFactory.getCryptoRandom(CryptoRandomFactory.java:189) > [info] at > org.apache.spark.security.CryptoStreamUtils$.createInitializationVector(CryptoStreamUtils.scala:138) > [info] at > org.apache.spark.security.CryptoStreamUtils$.createCryptoOutputStream(CryptoStreamUtils.scala:56) > [info] at > org.apache.spark.serializer.SerializerManager.$anonfun$wrapForEncryption$3(SerializerManager.scala:151) > > ``` > I did a quick check. commons-crypto-1.2.0.jar only contain x86_64 Mac native > library, it doesn't provide pre-built native library for Apple Silicon. > But I also checked commons-crypto-1.1.0.jar which Spark uses currently. It > also doesn't provide native library for Apple Silicon. > Since they both don't contain native library for Apple Silicon, why > commons-crypto-1.2.0 doesn't work and what makes the difference between > commons-crypto-1.1.0 and commons-crypto-1.2.0? -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (CRYPTO-169) Unable to load native library on Apple Silicon (M1 Mac)
[ https://issues.apache.org/jira/browse/CRYPTO-169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17778428#comment-17778428 ] Yang Jie commented on CRYPTO-169: - OK > Unable to load native library on Apple Silicon (M1 Mac) > --- > > Key: CRYPTO-169 > URL: https://issues.apache.org/jira/browse/CRYPTO-169 > Project: Commons Crypto > Issue Type: Bug >Reporter: L. C. Hsieh >Priority: Major > Attachments: image-2023-10-21-17-51-41-260.png, > image-2023-10-21-17-52-02-237.png, image-2023-10-21-17-53-16-898.png, > image-2023-10-21-17-59-13-866.png, image-2023-10-22-23-58-57-254.png > > > Recently as trying to upgrade commons-crypto from 1.1.0 to 1.2.0 in Spark > (SPARK-45617), got the following error in one test: > ``` > [info] Cause: java.security.GeneralSecurityException: Native library is not > loaded > [info] at > org.apache.commons.crypto.random.OpenSslCryptoRandom.checkNative(OpenSslCryptoRandom.java:79) > [info] at > org.apache.commons.crypto.random.OpenSslCryptoRandom.(OpenSslCryptoRandom.java:65) > > [info] at java.base/java.lang.Class.forName0(Native Method) > > [info] at java.base/java.lang.Class.forName(Class.java:467) > [info] at > org.apache.commons.crypto.utils.ReflectionUtils.getClassByNameOrNull(ReflectionUtils.java:93) > [info] at > org.apache.commons.crypto.utils.ReflectionUtils.getClassByName(ReflectionUtils.java:64) > [info] at > org.apache.commons.crypto.random.CryptoRandomFactory.getCryptoRandom(CryptoRandomFactory.java:189) > [info] at > org.apache.spark.security.CryptoStreamUtils$.createInitializationVector(CryptoStreamUtils.scala:138) > [info] at > org.apache.spark.security.CryptoStreamUtils$.createCryptoOutputStream(CryptoStreamUtils.scala:56) > [info] at > org.apache.spark.serializer.SerializerManager.$anonfun$wrapForEncryption$3(SerializerManager.scala:151) > > ``` > I did a quick check. commons-crypto-1.2.0.jar only contain x86_64 Mac native > library, it doesn't provide pre-built native library for Apple Silicon. > But I also checked commons-crypto-1.1.0.jar which Spark uses currently. It > also doesn't provide native library for Apple Silicon. > Since they both don't contain native library for Apple Silicon, why > commons-crypto-1.2.0 doesn't work and what makes the difference between > commons-crypto-1.1.0 and commons-crypto-1.2.0? -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (CRYPTO-169) Unable to load native library on Apple Silicon (M1 Mac)
[ https://issues.apache.org/jira/browse/CRYPTO-169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17778427#comment-17778427 ] Gary D. Gregory commented on CRYPTO-169: If this is really a different issue, then yes, a new ticket issue makes sense. > Unable to load native library on Apple Silicon (M1 Mac) > --- > > Key: CRYPTO-169 > URL: https://issues.apache.org/jira/browse/CRYPTO-169 > Project: Commons Crypto > Issue Type: Bug >Reporter: L. C. Hsieh >Priority: Major > Attachments: image-2023-10-21-17-51-41-260.png, > image-2023-10-21-17-52-02-237.png, image-2023-10-21-17-53-16-898.png, > image-2023-10-21-17-59-13-866.png, image-2023-10-22-23-58-57-254.png > > > Recently as trying to upgrade commons-crypto from 1.1.0 to 1.2.0 in Spark > (SPARK-45617), got the following error in one test: > ``` > [info] Cause: java.security.GeneralSecurityException: Native library is not > loaded > [info] at > org.apache.commons.crypto.random.OpenSslCryptoRandom.checkNative(OpenSslCryptoRandom.java:79) > [info] at > org.apache.commons.crypto.random.OpenSslCryptoRandom.(OpenSslCryptoRandom.java:65) > > [info] at java.base/java.lang.Class.forName0(Native Method) > > [info] at java.base/java.lang.Class.forName(Class.java:467) > [info] at > org.apache.commons.crypto.utils.ReflectionUtils.getClassByNameOrNull(ReflectionUtils.java:93) > [info] at > org.apache.commons.crypto.utils.ReflectionUtils.getClassByName(ReflectionUtils.java:64) > [info] at > org.apache.commons.crypto.random.CryptoRandomFactory.getCryptoRandom(CryptoRandomFactory.java:189) > [info] at > org.apache.spark.security.CryptoStreamUtils$.createInitializationVector(CryptoStreamUtils.scala:138) > [info] at > org.apache.spark.security.CryptoStreamUtils$.createCryptoOutputStream(CryptoStreamUtils.scala:56) > [info] at > org.apache.spark.serializer.SerializerManager.$anonfun$wrapForEncryption$3(SerializerManager.scala:151) > > ``` > I did a quick check. commons-crypto-1.2.0.jar only contain x86_64 Mac native > library, it doesn't provide pre-built native library for Apple Silicon. > But I also checked commons-crypto-1.1.0.jar which Spark uses currently. It > also doesn't provide native library for Apple Silicon. > Since they both don't contain native library for Apple Silicon, why > commons-crypto-1.2.0 doesn't work and what makes the difference between > commons-crypto-1.1.0 and commons-crypto-1.2.0? -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (CRYPTO-169) Unable to load native library on Apple Silicon (M1 Mac)
[ https://issues.apache.org/jira/browse/CRYPTO-169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17778420#comment-17778420 ] Yang Jie commented on CRYPTO-169: - Is it better to add a follow-up or create a new ticket? > Unable to load native library on Apple Silicon (M1 Mac) > --- > > Key: CRYPTO-169 > URL: https://issues.apache.org/jira/browse/CRYPTO-169 > Project: Commons Crypto > Issue Type: Bug >Reporter: L. C. Hsieh >Priority: Major > Attachments: image-2023-10-21-17-51-41-260.png, > image-2023-10-21-17-52-02-237.png, image-2023-10-21-17-53-16-898.png, > image-2023-10-21-17-59-13-866.png, image-2023-10-22-23-58-57-254.png > > > Recently as trying to upgrade commons-crypto from 1.1.0 to 1.2.0 in Spark > (SPARK-45617), got the following error in one test: > ``` > [info] Cause: java.security.GeneralSecurityException: Native library is not > loaded > [info] at > org.apache.commons.crypto.random.OpenSslCryptoRandom.checkNative(OpenSslCryptoRandom.java:79) > [info] at > org.apache.commons.crypto.random.OpenSslCryptoRandom.(OpenSslCryptoRandom.java:65) > > [info] at java.base/java.lang.Class.forName0(Native Method) > > [info] at java.base/java.lang.Class.forName(Class.java:467) > [info] at > org.apache.commons.crypto.utils.ReflectionUtils.getClassByNameOrNull(ReflectionUtils.java:93) > [info] at > org.apache.commons.crypto.utils.ReflectionUtils.getClassByName(ReflectionUtils.java:64) > [info] at > org.apache.commons.crypto.random.CryptoRandomFactory.getCryptoRandom(CryptoRandomFactory.java:189) > [info] at > org.apache.spark.security.CryptoStreamUtils$.createInitializationVector(CryptoStreamUtils.scala:138) > [info] at > org.apache.spark.security.CryptoStreamUtils$.createCryptoOutputStream(CryptoStreamUtils.scala:56) > [info] at > org.apache.spark.serializer.SerializerManager.$anonfun$wrapForEncryption$3(SerializerManager.scala:151) > > ``` > I did a quick check. commons-crypto-1.2.0.jar only contain x86_64 Mac native > library, it doesn't provide pre-built native library for Apple Silicon. > But I also checked commons-crypto-1.1.0.jar which Spark uses currently. It > also doesn't provide native library for Apple Silicon. > Since they both don't contain native library for Apple Silicon, why > commons-crypto-1.2.0 doesn't work and what makes the difference between > commons-crypto-1.1.0 and commons-crypto-1.2.0? -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (CRYPTO-169) Unable to load native library on Apple Silicon (M1 Mac)
[
https://issues.apache.org/jira/browse/CRYPTO-169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17778419#comment-17778419
]
Yang Jie commented on CRYPTO-169:
-
[~ggregory] It seems that there are some issues with the reentrancy of
`CryptoRandomFactory.getCryptoRandom(properties)`.
For the following test case：
{code:java}
@Test
public void testReentrancyOfExceptionInInitializerErrorRandom() throws
GeneralSecurityException, IOException {
final Properties properties = new Properties();
String classes =
ExceptionInInitializerErrorRandom.class.getName().concat(",")
.concat(CryptoRandomFactory.RandomProvider.JAVA.getClassName());
properties.setProperty(CryptoRandomFactory.CLASSES_KEY, classes);
try (final CryptoRandom random =
CryptoRandomFactory.getCryptoRandom(properties)) {
assertEquals(JavaCryptoRandom.class.getName(),
random.getClass().getName());
}
try (final CryptoRandom random =
CryptoRandomFactory.getCryptoRandom(properties)) {
assertEquals(JavaCryptoRandom.class.getName(),
random.getClass().getName());
}
try (final CryptoRandom random =
CryptoRandomFactory.getCryptoRandom(properties)) {
assertEquals(JavaCryptoRandom.class.getName(),
random.getClass().getName());
}
} {code}
The second call to `CryptoRandomFactory.getCryptoRandom(properties)` will
result in the following error:
!image-2023-10-22-23-58-57-254.png|width=1197,height=314!
It seems that we need to add another layer of protection to the
ExceptionInInitializerError that wraps the NoClassDefFoundError ...
> Unable to load native library on Apple Silicon (M1 Mac)
> ---
>
> Key: CRYPTO-169
> URL: https://issues.apache.org/jira/browse/CRYPTO-169
> Project: Commons Crypto
> Issue Type: Bug
>Reporter: L. C. Hsieh
>Priority: Major
> Attachments: image-2023-10-21-17-51-41-260.png,
> image-2023-10-21-17-52-02-237.png, image-2023-10-21-17-53-16-898.png,
> image-2023-10-21-17-59-13-866.png, image-2023-10-22-23-58-57-254.png
>
>
> Recently as trying to upgrade commons-crypto from 1.1.0 to 1.2.0 in Spark
> (SPARK-45617), got the following error in one test:
> ```
> [info] Cause: java.security.GeneralSecurityException: Native library is not
> loaded
> [info] at
> org.apache.commons.crypto.random.OpenSslCryptoRandom.checkNative(OpenSslCryptoRandom.java:79)
> [info] at
> org.apache.commons.crypto.random.OpenSslCryptoRandom.(OpenSslCryptoRandom.java:65)
>
> [info] at java.base/java.lang.Class.forName0(Native Method)
>
> [info] at java.base/java.lang.Class.forName(Class.java:467)
> [info] at
> org.apache.commons.crypto.utils.ReflectionUtils.getClassByNameOrNull(ReflectionUtils.java:93)
> [info] at
> org.apache.commons.crypto.utils.ReflectionUtils.getClassByName(ReflectionUtils.java:64)
> [info] at
> org.apache.commons.crypto.random.CryptoRandomFactory.getCryptoRandom(CryptoRandomFactory.java:189)
> [info] at
> org.apache.spark.security.CryptoStreamUtils$.createInitializationVector(CryptoStreamUtils.scala:138)
> [info] at
> org.apache.spark.security.CryptoStreamUtils$.createCryptoOutputStream(CryptoStreamUtils.scala:56)
> [info] at
> org.apache.spark.serializer.SerializerManager.$anonfun$wrapForEncryption$3(SerializerManager.scala:151)
>
> ```
> I did a quick check. commons-crypto-1.2.0.jar only contain x86_64 Mac native
> library, it doesn't provide pre-built native library for Apple Silicon.
> But I also checked commons-crypto-1.1.0.jar which Spark uses currently. It
> also doesn't provide native library for Apple Silicon.
> Since they both don't contain native library for Apple Silicon, why
> commons-crypto-1.2.0 doesn't work and what makes the difference between
> commons-crypto-1.1.0 and commons-crypto-1.2.0?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (CRYPTO-169) Unable to load native library on Apple Silicon (M1 Mac)
[ https://issues.apache.org/jira/browse/CRYPTO-169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17778369#comment-17778369 ] Gary D. Gregory commented on CRYPTO-169: Merged to git master. Please test git master or a snapshot build from https://repository.apache.org/content/repositories/snapshots/org/apache/commons/commons-crypto > Unable to load native library on Apple Silicon (M1 Mac) > --- > > Key: CRYPTO-169 > URL: https://issues.apache.org/jira/browse/CRYPTO-169 > Project: Commons Crypto > Issue Type: Bug >Reporter: L. C. Hsieh >Priority: Major > Attachments: image-2023-10-21-17-51-41-260.png, > image-2023-10-21-17-52-02-237.png, image-2023-10-21-17-53-16-898.png, > image-2023-10-21-17-59-13-866.png > > > Recently as trying to upgrade commons-crypto from 1.1.0 to 1.2.0 in Spark > (SPARK-45617), got the following error in one test: > ``` > [info] Cause: java.security.GeneralSecurityException: Native library is not > loaded > [info] at > org.apache.commons.crypto.random.OpenSslCryptoRandom.checkNative(OpenSslCryptoRandom.java:79) > [info] at > org.apache.commons.crypto.random.OpenSslCryptoRandom.(OpenSslCryptoRandom.java:65) > > [info] at java.base/java.lang.Class.forName0(Native Method) > > [info] at java.base/java.lang.Class.forName(Class.java:467) > [info] at > org.apache.commons.crypto.utils.ReflectionUtils.getClassByNameOrNull(ReflectionUtils.java:93) > [info] at > org.apache.commons.crypto.utils.ReflectionUtils.getClassByName(ReflectionUtils.java:64) > [info] at > org.apache.commons.crypto.random.CryptoRandomFactory.getCryptoRandom(CryptoRandomFactory.java:189) > [info] at > org.apache.spark.security.CryptoStreamUtils$.createInitializationVector(CryptoStreamUtils.scala:138) > [info] at > org.apache.spark.security.CryptoStreamUtils$.createCryptoOutputStream(CryptoStreamUtils.scala:56) > [info] at > org.apache.spark.serializer.SerializerManager.$anonfun$wrapForEncryption$3(SerializerManager.scala:151) > > ``` > I did a quick check. commons-crypto-1.2.0.jar only contain x86_64 Mac native > library, it doesn't provide pre-built native library for Apple Silicon. > But I also checked commons-crypto-1.1.0.jar which Spark uses currently. It > also doesn't provide native library for Apple Silicon. > Since they both don't contain native library for Apple Silicon, why > commons-crypto-1.2.0 doesn't work and what makes the difference between > commons-crypto-1.1.0 and commons-crypto-1.2.0? -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (CRYPTO-169) Unable to load native library on Apple Silicon (M1 Mac)
[ https://issues.apache.org/jira/browse/CRYPTO-169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17778338#comment-17778338 ] Yang Jie commented on CRYPTO-169: - Submit https://github.com/apache/commons-crypto/pull/258 > Unable to load native library on Apple Silicon (M1 Mac) > --- > > Key: CRYPTO-169 > URL: https://issues.apache.org/jira/browse/CRYPTO-169 > Project: Commons Crypto > Issue Type: Bug >Reporter: L. C. Hsieh >Priority: Major > Attachments: image-2023-10-21-17-51-41-260.png, > image-2023-10-21-17-52-02-237.png, image-2023-10-21-17-53-16-898.png, > image-2023-10-21-17-59-13-866.png > > > Recently as trying to upgrade commons-crypto from 1.1.0 to 1.2.0 in Spark > (SPARK-45617), got the following error in one test: > ``` > [info] Cause: java.security.GeneralSecurityException: Native library is not > loaded > [info] at > org.apache.commons.crypto.random.OpenSslCryptoRandom.checkNative(OpenSslCryptoRandom.java:79) > [info] at > org.apache.commons.crypto.random.OpenSslCryptoRandom.(OpenSslCryptoRandom.java:65) > > [info] at java.base/java.lang.Class.forName0(Native Method) > > [info] at java.base/java.lang.Class.forName(Class.java:467) > [info] at > org.apache.commons.crypto.utils.ReflectionUtils.getClassByNameOrNull(ReflectionUtils.java:93) > [info] at > org.apache.commons.crypto.utils.ReflectionUtils.getClassByName(ReflectionUtils.java:64) > [info] at > org.apache.commons.crypto.random.CryptoRandomFactory.getCryptoRandom(CryptoRandomFactory.java:189) > [info] at > org.apache.spark.security.CryptoStreamUtils$.createInitializationVector(CryptoStreamUtils.scala:138) > [info] at > org.apache.spark.security.CryptoStreamUtils$.createCryptoOutputStream(CryptoStreamUtils.scala:56) > [info] at > org.apache.spark.serializer.SerializerManager.$anonfun$wrapForEncryption$3(SerializerManager.scala:151) > > ``` > I did a quick check. commons-crypto-1.2.0.jar only contain x86_64 Mac native > library, it doesn't provide pre-built native library for Apple Silicon. > But I also checked commons-crypto-1.1.0.jar which Spark uses currently. It > also doesn't provide native library for Apple Silicon. > Since they both don't contain native library for Apple Silicon, why > commons-crypto-1.2.0 doesn't work and what makes the difference between > commons-crypto-1.1.0 and commons-crypto-1.2.0? -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (CRYPTO-169) Unable to load native library on Apple Silicon (M1 Mac)
[ https://issues.apache.org/jira/browse/CRYPTO-169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17778323#comment-17778323 ] Gary D. Gregory commented on CRYPTO-169: Hi [~LuciferYang] and [~viirya] Feel free to create a PR on GitHub so we can see if there are side effects on various platforms. > Unable to load native library on Apple Silicon (M1 Mac) > --- > > Key: CRYPTO-169 > URL: https://issues.apache.org/jira/browse/CRYPTO-169 > Project: Commons Crypto > Issue Type: Bug >Reporter: L. C. Hsieh >Priority: Major > Attachments: image-2023-10-21-17-51-41-260.png, > image-2023-10-21-17-52-02-237.png, image-2023-10-21-17-53-16-898.png, > image-2023-10-21-17-59-13-866.png > > > Recently as trying to upgrade commons-crypto from 1.1.0 to 1.2.0 in Spark > (SPARK-45617), got the following error in one test: > ``` > [info] Cause: java.security.GeneralSecurityException: Native library is not > loaded > [info] at > org.apache.commons.crypto.random.OpenSslCryptoRandom.checkNative(OpenSslCryptoRandom.java:79) > [info] at > org.apache.commons.crypto.random.OpenSslCryptoRandom.(OpenSslCryptoRandom.java:65) > > [info] at java.base/java.lang.Class.forName0(Native Method) > > [info] at java.base/java.lang.Class.forName(Class.java:467) > [info] at > org.apache.commons.crypto.utils.ReflectionUtils.getClassByNameOrNull(ReflectionUtils.java:93) > [info] at > org.apache.commons.crypto.utils.ReflectionUtils.getClassByName(ReflectionUtils.java:64) > [info] at > org.apache.commons.crypto.random.CryptoRandomFactory.getCryptoRandom(CryptoRandomFactory.java:189) > [info] at > org.apache.spark.security.CryptoStreamUtils$.createInitializationVector(CryptoStreamUtils.scala:138) > [info] at > org.apache.spark.security.CryptoStreamUtils$.createCryptoOutputStream(CryptoStreamUtils.scala:56) > [info] at > org.apache.spark.serializer.SerializerManager.$anonfun$wrapForEncryption$3(SerializerManager.scala:151) > > ``` > I did a quick check. commons-crypto-1.2.0.jar only contain x86_64 Mac native > library, it doesn't provide pre-built native library for Apple Silicon. > But I also checked commons-crypto-1.1.0.jar which Spark uses currently. It > also doesn't provide native library for Apple Silicon. > Since they both don't contain native library for Apple Silicon, why > commons-crypto-1.2.0 doesn't work and what makes the difference between > commons-crypto-1.1.0 and commons-crypto-1.2.0? -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (CRYPTO-169) Unable to load native library on Apple Silicon (M1 Mac)
[ https://issues.apache.org/jira/browse/CRYPTO-169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17778316#comment-17778316 ] Sebb commented on CRYPTO-169: - That should be OK. > Unable to load native library on Apple Silicon (M1 Mac) > --- > > Key: CRYPTO-169 > URL: https://issues.apache.org/jira/browse/CRYPTO-169 > Project: Commons Crypto > Issue Type: Bug >Reporter: L. C. Hsieh >Priority: Major > Attachments: image-2023-10-21-17-51-41-260.png, > image-2023-10-21-17-52-02-237.png, image-2023-10-21-17-53-16-898.png, > image-2023-10-21-17-59-13-866.png > > > Recently as trying to upgrade commons-crypto from 1.1.0 to 1.2.0 in Spark > (SPARK-45617), got the following error in one test: > ``` > [info] Cause: java.security.GeneralSecurityException: Native library is not > loaded > [info] at > org.apache.commons.crypto.random.OpenSslCryptoRandom.checkNative(OpenSslCryptoRandom.java:79) > [info] at > org.apache.commons.crypto.random.OpenSslCryptoRandom.(OpenSslCryptoRandom.java:65) > > [info] at java.base/java.lang.Class.forName0(Native Method) > > [info] at java.base/java.lang.Class.forName(Class.java:467) > [info] at > org.apache.commons.crypto.utils.ReflectionUtils.getClassByNameOrNull(ReflectionUtils.java:93) > [info] at > org.apache.commons.crypto.utils.ReflectionUtils.getClassByName(ReflectionUtils.java:64) > [info] at > org.apache.commons.crypto.random.CryptoRandomFactory.getCryptoRandom(CryptoRandomFactory.java:189) > [info] at > org.apache.spark.security.CryptoStreamUtils$.createInitializationVector(CryptoStreamUtils.scala:138) > [info] at > org.apache.spark.security.CryptoStreamUtils$.createCryptoOutputStream(CryptoStreamUtils.scala:56) > [info] at > org.apache.spark.serializer.SerializerManager.$anonfun$wrapForEncryption$3(SerializerManager.scala:151) > > ``` > I did a quick check. commons-crypto-1.2.0.jar only contain x86_64 Mac native > library, it doesn't provide pre-built native library for Apple Silicon. > But I also checked commons-crypto-1.1.0.jar which Spark uses currently. It > also doesn't provide native library for Apple Silicon. > Since they both don't contain native library for Apple Silicon, why > commons-crypto-1.2.0 doesn't work and what makes the difference between > commons-crypto-1.1.0 and commons-crypto-1.2.0? -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (CRYPTO-169) Unable to load native library on Apple Silicon (M1 Mac)
[ https://issues.apache.org/jira/browse/CRYPTO-169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17778304#comment-17778304 ] L. C. Hsieh commented on CRYPTO-169: How about only catch `ExceptionInInitializerError`? > Unable to load native library on Apple Silicon (M1 Mac) > --- > > Key: CRYPTO-169 > URL: https://issues.apache.org/jira/browse/CRYPTO-169 > Project: Commons Crypto > Issue Type: Bug >Reporter: L. C. Hsieh >Priority: Major > Attachments: image-2023-10-21-17-51-41-260.png, > image-2023-10-21-17-52-02-237.png, image-2023-10-21-17-53-16-898.png, > image-2023-10-21-17-59-13-866.png > > > Recently as trying to upgrade commons-crypto from 1.1.0 to 1.2.0 in Spark > (SPARK-45617), got the following error in one test: > ``` > [info] Cause: java.security.GeneralSecurityException: Native library is not > loaded > [info] at > org.apache.commons.crypto.random.OpenSslCryptoRandom.checkNative(OpenSslCryptoRandom.java:79) > [info] at > org.apache.commons.crypto.random.OpenSslCryptoRandom.(OpenSslCryptoRandom.java:65) > > [info] at java.base/java.lang.Class.forName0(Native Method) > > [info] at java.base/java.lang.Class.forName(Class.java:467) > [info] at > org.apache.commons.crypto.utils.ReflectionUtils.getClassByNameOrNull(ReflectionUtils.java:93) > [info] at > org.apache.commons.crypto.utils.ReflectionUtils.getClassByName(ReflectionUtils.java:64) > [info] at > org.apache.commons.crypto.random.CryptoRandomFactory.getCryptoRandom(CryptoRandomFactory.java:189) > [info] at > org.apache.spark.security.CryptoStreamUtils$.createInitializationVector(CryptoStreamUtils.scala:138) > [info] at > org.apache.spark.security.CryptoStreamUtils$.createCryptoOutputStream(CryptoStreamUtils.scala:56) > [info] at > org.apache.spark.serializer.SerializerManager.$anonfun$wrapForEncryption$3(SerializerManager.scala:151) > > ``` > I did a quick check. commons-crypto-1.2.0.jar only contain x86_64 Mac native > library, it doesn't provide pre-built native library for Apple Silicon. > But I also checked commons-crypto-1.1.0.jar which Spark uses currently. It > also doesn't provide native library for Apple Silicon. > Since they both don't contain native library for Apple Silicon, why > commons-crypto-1.2.0 doesn't work and what makes the difference between > commons-crypto-1.1.0 and commons-crypto-1.2.0? -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (CRYPTO-169) Unable to load native library on Apple Silicon (M1 Mac)
[ https://issues.apache.org/jira/browse/CRYPTO-169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17778297#comment-17778297 ] Sebb commented on CRYPTO-169: - No, it is never acceptable to catch Throwable. Some exceptions should never be caught. > Unable to load native library on Apple Silicon (M1 Mac) > --- > > Key: CRYPTO-169 > URL: https://issues.apache.org/jira/browse/CRYPTO-169 > Project: Commons Crypto > Issue Type: Bug >Reporter: L. C. Hsieh >Priority: Major > Attachments: image-2023-10-21-17-51-41-260.png, > image-2023-10-21-17-52-02-237.png, image-2023-10-21-17-53-16-898.png, > image-2023-10-21-17-59-13-866.png > > > Recently as trying to upgrade commons-crypto from 1.1.0 to 1.2.0 in Spark > (SPARK-45617), got the following error in one test: > ``` > [info] Cause: java.security.GeneralSecurityException: Native library is not > loaded > [info] at > org.apache.commons.crypto.random.OpenSslCryptoRandom.checkNative(OpenSslCryptoRandom.java:79) > [info] at > org.apache.commons.crypto.random.OpenSslCryptoRandom.(OpenSslCryptoRandom.java:65) > > [info] at java.base/java.lang.Class.forName0(Native Method) > > [info] at java.base/java.lang.Class.forName(Class.java:467) > [info] at > org.apache.commons.crypto.utils.ReflectionUtils.getClassByNameOrNull(ReflectionUtils.java:93) > [info] at > org.apache.commons.crypto.utils.ReflectionUtils.getClassByName(ReflectionUtils.java:64) > [info] at > org.apache.commons.crypto.random.CryptoRandomFactory.getCryptoRandom(CryptoRandomFactory.java:189) > [info] at > org.apache.spark.security.CryptoStreamUtils$.createInitializationVector(CryptoStreamUtils.scala:138) > [info] at > org.apache.spark.security.CryptoStreamUtils$.createCryptoOutputStream(CryptoStreamUtils.scala:56) > [info] at > org.apache.spark.serializer.SerializerManager.$anonfun$wrapForEncryption$3(SerializerManager.scala:151) > > ``` > I did a quick check. commons-crypto-1.2.0.jar only contain x86_64 Mac native > library, it doesn't provide pre-built native library for Apple Silicon. > But I also checked commons-crypto-1.1.0.jar which Spark uses currently. It > also doesn't provide native library for Apple Silicon. > Since they both don't contain native library for Apple Silicon, why > commons-crypto-1.2.0 doesn't work and what makes the difference between > commons-crypto-1.1.0 and commons-crypto-1.2.0? -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (CRYPTO-169) Unable to load native library on Apple Silicon (M1 Mac)
[ https://issues.apache.org/jira/browse/CRYPTO-169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17778268#comment-17778268 ] L. C. Hsieh commented on CRYPTO-169: It looks like reasonable to catch Throwable instead of Exception so we can get `JavaCryptoRandom` as 1.1.0. > Unable to load native library on Apple Silicon (M1 Mac) > --- > > Key: CRYPTO-169 > URL: https://issues.apache.org/jira/browse/CRYPTO-169 > Project: Commons Crypto > Issue Type: Bug >Reporter: L. C. Hsieh >Priority: Major > Attachments: image-2023-10-21-17-51-41-260.png, > image-2023-10-21-17-52-02-237.png, image-2023-10-21-17-53-16-898.png, > image-2023-10-21-17-59-13-866.png > > > Recently as trying to upgrade commons-crypto from 1.1.0 to 1.2.0 in Spark > (SPARK-45617), got the following error in one test: > ``` > [info] Cause: java.security.GeneralSecurityException: Native library is not > loaded > [info] at > org.apache.commons.crypto.random.OpenSslCryptoRandom.checkNative(OpenSslCryptoRandom.java:79) > [info] at > org.apache.commons.crypto.random.OpenSslCryptoRandom.(OpenSslCryptoRandom.java:65) > > [info] at java.base/java.lang.Class.forName0(Native Method) > > [info] at java.base/java.lang.Class.forName(Class.java:467) > [info] at > org.apache.commons.crypto.utils.ReflectionUtils.getClassByNameOrNull(ReflectionUtils.java:93) > [info] at > org.apache.commons.crypto.utils.ReflectionUtils.getClassByName(ReflectionUtils.java:64) > [info] at > org.apache.commons.crypto.random.CryptoRandomFactory.getCryptoRandom(CryptoRandomFactory.java:189) > [info] at > org.apache.spark.security.CryptoStreamUtils$.createInitializationVector(CryptoStreamUtils.scala:138) > [info] at > org.apache.spark.security.CryptoStreamUtils$.createCryptoOutputStream(CryptoStreamUtils.scala:56) > [info] at > org.apache.spark.serializer.SerializerManager.$anonfun$wrapForEncryption$3(SerializerManager.scala:151) > > ``` > I did a quick check. commons-crypto-1.2.0.jar only contain x86_64 Mac native > library, it doesn't provide pre-built native library for Apple Silicon. > But I also checked commons-crypto-1.1.0.jar which Spark uses currently. It > also doesn't provide native library for Apple Silicon. > Since they both don't contain native library for Apple Silicon, why > commons-crypto-1.2.0 doesn't work and what makes the difference between > commons-crypto-1.1.0 and commons-crypto-1.2.0? -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (CRYPTO-169) Unable to load native library on Apple Silicon (M1 Mac)
[
https://issues.apache.org/jira/browse/CRYPTO-169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17778262#comment-17778262
]
Yang Jie commented on CRYPTO-169:
-
Yes, crypto has not supported Apple Silicon.
But the current critical issue is that the behavior of the line of code
`org.apache.commons.crypto.random.CryptoRandomFactory.getCryptoRandom(new
Properties())` has changed between version 1.1.0 and 1.2.0.
In 1.1.0, the above statement returns an instance of type
`org.apache.commons.crypto.random.JavaCryptoRandom`.
In 1.2.0, the above statement directly throws an `ExceptionInInitializerError`
type of error due to the execution failure of
`org.apache.commons.crypto.utils.ReflectionUtils.getClassByName("org.apache.commons.crypto.random.OpenSslCryptoRandom")`.
Note, here it throws an Error, not an Exception.
Looking at the logic of `CryptoRandomFactory.getCryptoRandom`, it does not
force `OpenSslCryptoRandom` to return successfully. As long as either
`OpenSslCryptoRandom` or `JavaCryptoRandom` initializes successfully, it is
acceptable. However, in 1.2.0, there is a scenario that
`CryptoRandomFactory.getCryptoRandom` cannot handle, because it handles all
`Exception` scenarios but does not handle any `Error` scenarios，so, when an
`Error` occurs, it does not attempt to initialize `JavaCryptoRandom`, but
directly throws an `ExceptionInInitializerError` instead.
> Unable to load native library on Apple Silicon (M1 Mac)
> ---
>
> Key: CRYPTO-169
> URL: https://issues.apache.org/jira/browse/CRYPTO-169
> Project: Commons Crypto
> Issue Type: Bug
>Reporter: L. C. Hsieh
>Priority: Major
> Attachments: image-2023-10-21-17-51-41-260.png,
> image-2023-10-21-17-52-02-237.png, image-2023-10-21-17-53-16-898.png,
> image-2023-10-21-17-59-13-866.png
>
>
> Recently as trying to upgrade commons-crypto from 1.1.0 to 1.2.0 in Spark
> (SPARK-45617), got the following error in one test:
> ```
> [info] Cause: java.security.GeneralSecurityException: Native library is not
> loaded
> [info] at
> org.apache.commons.crypto.random.OpenSslCryptoRandom.checkNative(OpenSslCryptoRandom.java:79)
> [info] at
> org.apache.commons.crypto.random.OpenSslCryptoRandom.(OpenSslCryptoRandom.java:65)
>
> [info] at java.base/java.lang.Class.forName0(Native Method)
>
> [info] at java.base/java.lang.Class.forName(Class.java:467)
> [info] at
> org.apache.commons.crypto.utils.ReflectionUtils.getClassByNameOrNull(ReflectionUtils.java:93)
> [info] at
> org.apache.commons.crypto.utils.ReflectionUtils.getClassByName(ReflectionUtils.java:64)
> [info] at
> org.apache.commons.crypto.random.CryptoRandomFactory.getCryptoRandom(CryptoRandomFactory.java:189)
> [info] at
> org.apache.spark.security.CryptoStreamUtils$.createInitializationVector(CryptoStreamUtils.scala:138)
> [info] at
> org.apache.spark.security.CryptoStreamUtils$.createCryptoOutputStream(CryptoStreamUtils.scala:56)
> [info] at
> org.apache.spark.serializer.SerializerManager.$anonfun$wrapForEncryption$3(SerializerManager.scala:151)
>
> ```
> I did a quick check. commons-crypto-1.2.0.jar only contain x86_64 Mac native
> library, it doesn't provide pre-built native library for Apple Silicon.
> But I also checked commons-crypto-1.1.0.jar which Spark uses currently. It
> also doesn't provide native library for Apple Silicon.
> Since they both don't contain native library for Apple Silicon, why
> commons-crypto-1.2.0 doesn't work and what makes the difference between
> commons-crypto-1.1.0 and commons-crypto-1.2.0?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (CRYPTO-169) Unable to load native library on Apple Silicon (M1 Mac)
[ https://issues.apache.org/jira/browse/CRYPTO-169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17778219#comment-17778219 ] L. C. Hsieh commented on CRYPTO-169: This is the exception we got: [info] Cause: java.security.GeneralSecurityException: Native library is not loaded [info] at org.apache.commons.crypto.random.OpenSslCryptoRandom.checkNative(OpenSslCryptoRandom.java:79) [info] at org.apache.commons.crypto.random.OpenSslCryptoRandom.(OpenSslCryptoRandom.java:65) There is no additional info indicating what file cannot be loaded and why it cannot be loaded. > Unable to load native library on Apple Silicon (M1 Mac) > --- > > Key: CRYPTO-169 > URL: https://issues.apache.org/jira/browse/CRYPTO-169 > Project: Commons Crypto > Issue Type: Bug >Reporter: L. C. Hsieh >Priority: Major > Attachments: image-2023-10-21-17-51-41-260.png, > image-2023-10-21-17-52-02-237.png, image-2023-10-21-17-53-16-898.png, > image-2023-10-21-17-59-13-866.png > > > Recently as trying to upgrade commons-crypto from 1.1.0 to 1.2.0 in Spark > (SPARK-45617), got the following error in one test: > ``` > [info] Cause: java.security.GeneralSecurityException: Native library is not > loaded > [info] at > org.apache.commons.crypto.random.OpenSslCryptoRandom.checkNative(OpenSslCryptoRandom.java:79) > [info] at > org.apache.commons.crypto.random.OpenSslCryptoRandom.(OpenSslCryptoRandom.java:65) > > [info] at java.base/java.lang.Class.forName0(Native Method) > > [info] at java.base/java.lang.Class.forName(Class.java:467) > [info] at > org.apache.commons.crypto.utils.ReflectionUtils.getClassByNameOrNull(ReflectionUtils.java:93) > [info] at > org.apache.commons.crypto.utils.ReflectionUtils.getClassByName(ReflectionUtils.java:64) > [info] at > org.apache.commons.crypto.random.CryptoRandomFactory.getCryptoRandom(CryptoRandomFactory.java:189) > [info] at > org.apache.spark.security.CryptoStreamUtils$.createInitializationVector(CryptoStreamUtils.scala:138) > [info] at > org.apache.spark.security.CryptoStreamUtils$.createCryptoOutputStream(CryptoStreamUtils.scala:56) > [info] at > org.apache.spark.serializer.SerializerManager.$anonfun$wrapForEncryption$3(SerializerManager.scala:151) > > ``` > I did a quick check. commons-crypto-1.2.0.jar only contain x86_64 Mac native > library, it doesn't provide pre-built native library for Apple Silicon. > But I also checked commons-crypto-1.1.0.jar which Spark uses currently. It > also doesn't provide native library for Apple Silicon. > Since they both don't contain native library for Apple Silicon, why > commons-crypto-1.2.0 doesn't work and what makes the difference between > commons-crypto-1.1.0 and commons-crypto-1.2.0? -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (CRYPTO-169) Unable to load native library on Apple Silicon (M1 Mac)
[ https://issues.apache.org/jira/browse/CRYPTO-169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17778202#comment-17778202 ] Gary D. Gregory commented on CRYPTO-169: Hm, what file is it that it cannot load? Is it there, missing, corrupted, or an incompatible format? > Unable to load native library on Apple Silicon (M1 Mac) > --- > > Key: CRYPTO-169 > URL: https://issues.apache.org/jira/browse/CRYPTO-169 > Project: Commons Crypto > Issue Type: Bug >Reporter: L. C. Hsieh >Priority: Major > Attachments: image-2023-10-21-17-51-41-260.png, > image-2023-10-21-17-52-02-237.png, image-2023-10-21-17-53-16-898.png, > image-2023-10-21-17-59-13-866.png > > > Recently as trying to upgrade commons-crypto from 1.1.0 to 1.2.0 in Spark > (SPARK-45617), got the following error in one test: > ``` > [info] Cause: java.security.GeneralSecurityException: Native library is not > loaded > [info] at > org.apache.commons.crypto.random.OpenSslCryptoRandom.checkNative(OpenSslCryptoRandom.java:79) > [info] at > org.apache.commons.crypto.random.OpenSslCryptoRandom.(OpenSslCryptoRandom.java:65) > > [info] at java.base/java.lang.Class.forName0(Native Method) > > [info] at java.base/java.lang.Class.forName(Class.java:467) > [info] at > org.apache.commons.crypto.utils.ReflectionUtils.getClassByNameOrNull(ReflectionUtils.java:93) > [info] at > org.apache.commons.crypto.utils.ReflectionUtils.getClassByName(ReflectionUtils.java:64) > [info] at > org.apache.commons.crypto.random.CryptoRandomFactory.getCryptoRandom(CryptoRandomFactory.java:189) > [info] at > org.apache.spark.security.CryptoStreamUtils$.createInitializationVector(CryptoStreamUtils.scala:138) > [info] at > org.apache.spark.security.CryptoStreamUtils$.createCryptoOutputStream(CryptoStreamUtils.scala:56) > [info] at > org.apache.spark.serializer.SerializerManager.$anonfun$wrapForEncryption$3(SerializerManager.scala:151) > > ``` > I did a quick check. commons-crypto-1.2.0.jar only contain x86_64 Mac native > library, it doesn't provide pre-built native library for Apple Silicon. > But I also checked commons-crypto-1.1.0.jar which Spark uses currently. It > also doesn't provide native library for Apple Silicon. > Since they both don't contain native library for Apple Silicon, why > commons-crypto-1.2.0 doesn't work and what makes the difference between > commons-crypto-1.1.0 and commons-crypto-1.2.0? -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (CRYPTO-169) Unable to load native library on Apple Silicon (M1 Mac)
[ https://issues.apache.org/jira/browse/CRYPTO-169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17778198#comment-17778198 ] L. C. Hsieh commented on CRYPTO-169: > It's possible that we are looking for a specific version of OpenSSL that's > not on the native path. Hmm, but I saw commons-crypto-1.1.0 and commons-crypto-1.2.0 both use OpenSSL 1.1.1. > Unable to load native library on Apple Silicon (M1 Mac) > --- > > Key: CRYPTO-169 > URL: https://issues.apache.org/jira/browse/CRYPTO-169 > Project: Commons Crypto > Issue Type: Bug >Reporter: L. C. Hsieh >Priority: Major > Attachments: image-2023-10-21-17-51-41-260.png, > image-2023-10-21-17-52-02-237.png, image-2023-10-21-17-53-16-898.png, > image-2023-10-21-17-59-13-866.png > > > Recently as trying to upgrade commons-crypto from 1.1.0 to 1.2.0 in Spark > (SPARK-45617), got the following error in one test: > ``` > [info] Cause: java.security.GeneralSecurityException: Native library is not > loaded > [info] at > org.apache.commons.crypto.random.OpenSslCryptoRandom.checkNative(OpenSslCryptoRandom.java:79) > [info] at > org.apache.commons.crypto.random.OpenSslCryptoRandom.(OpenSslCryptoRandom.java:65) > > [info] at java.base/java.lang.Class.forName0(Native Method) > > [info] at java.base/java.lang.Class.forName(Class.java:467) > [info] at > org.apache.commons.crypto.utils.ReflectionUtils.getClassByNameOrNull(ReflectionUtils.java:93) > [info] at > org.apache.commons.crypto.utils.ReflectionUtils.getClassByName(ReflectionUtils.java:64) > [info] at > org.apache.commons.crypto.random.CryptoRandomFactory.getCryptoRandom(CryptoRandomFactory.java:189) > [info] at > org.apache.spark.security.CryptoStreamUtils$.createInitializationVector(CryptoStreamUtils.scala:138) > [info] at > org.apache.spark.security.CryptoStreamUtils$.createCryptoOutputStream(CryptoStreamUtils.scala:56) > [info] at > org.apache.spark.serializer.SerializerManager.$anonfun$wrapForEncryption$3(SerializerManager.scala:151) > > ``` > I did a quick check. commons-crypto-1.2.0.jar only contain x86_64 Mac native > library, it doesn't provide pre-built native library for Apple Silicon. > But I also checked commons-crypto-1.1.0.jar which Spark uses currently. It > also doesn't provide native library for Apple Silicon. > Since they both don't contain native library for Apple Silicon, why > commons-crypto-1.2.0 doesn't work and what makes the difference between > commons-crypto-1.1.0 and commons-crypto-1.2.0? -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (CRYPTO-169) Unable to load native library on Apple Silicon (M1 Mac)
[ https://issues.apache.org/jira/browse/CRYPTO-169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17778195#comment-17778195 ] Gary D. Gregory commented on CRYPTO-169: Commons Crypto has never supported Apple silicon. > Unable to load native library on Apple Silicon (M1 Mac) > --- > > Key: CRYPTO-169 > URL: https://issues.apache.org/jira/browse/CRYPTO-169 > Project: Commons Crypto > Issue Type: Bug >Reporter: L. C. Hsieh >Priority: Major > Attachments: image-2023-10-21-17-51-41-260.png, > image-2023-10-21-17-52-02-237.png, image-2023-10-21-17-53-16-898.png, > image-2023-10-21-17-59-13-866.png > > > Recently as trying to upgrade commons-crypto from 1.1.0 to 1.2.0 in Spark > (SPARK-45617), got the following error in one test: > ``` > [info] Cause: java.security.GeneralSecurityException: Native library is not > loaded > [info] at > org.apache.commons.crypto.random.OpenSslCryptoRandom.checkNative(OpenSslCryptoRandom.java:79) > [info] at > org.apache.commons.crypto.random.OpenSslCryptoRandom.(OpenSslCryptoRandom.java:65) > > [info] at java.base/java.lang.Class.forName0(Native Method) > > [info] at java.base/java.lang.Class.forName(Class.java:467) > [info] at > org.apache.commons.crypto.utils.ReflectionUtils.getClassByNameOrNull(ReflectionUtils.java:93) > [info] at > org.apache.commons.crypto.utils.ReflectionUtils.getClassByName(ReflectionUtils.java:64) > [info] at > org.apache.commons.crypto.random.CryptoRandomFactory.getCryptoRandom(CryptoRandomFactory.java:189) > [info] at > org.apache.spark.security.CryptoStreamUtils$.createInitializationVector(CryptoStreamUtils.scala:138) > [info] at > org.apache.spark.security.CryptoStreamUtils$.createCryptoOutputStream(CryptoStreamUtils.scala:56) > [info] at > org.apache.spark.serializer.SerializerManager.$anonfun$wrapForEncryption$3(SerializerManager.scala:151) > > ``` > I did a quick check. commons-crypto-1.2.0.jar only contain x86_64 Mac native > library, it doesn't provide pre-built native library for Apple Silicon. > But I also checked commons-crypto-1.1.0.jar which Spark uses currently. It > also doesn't provide native library for Apple Silicon. > Since they both don't contain native library for Apple Silicon, why > commons-crypto-1.2.0 doesn't work and what makes the difference between > commons-crypto-1.1.0 and commons-crypto-1.2.0? -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (CRYPTO-169) Unable to load native library on Apple Silicon (M1 Mac)
[ https://issues.apache.org/jira/browse/CRYPTO-169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17778192#comment-17778192 ] L. C. Hsieh commented on CRYPTO-169: Thanks [~ggregory] for your reply. As I mentioned in the description, Spark currently uses commons-crypto-1.1.0 and the test can pass without the native loading issue. Does it mean commons-crypto-1.1.0 support Apple silicone? If so, is there any change between 1.1.0 and 1.2.0 making such difference (e.g., GCC related change)? > Unable to load native library on Apple Silicon (M1 Mac) > --- > > Key: CRYPTO-169 > URL: https://issues.apache.org/jira/browse/CRYPTO-169 > Project: Commons Crypto > Issue Type: Bug >Reporter: L. C. Hsieh >Priority: Major > Attachments: image-2023-10-21-17-51-41-260.png, > image-2023-10-21-17-52-02-237.png, image-2023-10-21-17-53-16-898.png, > image-2023-10-21-17-59-13-866.png > > > Recently as trying to upgrade commons-crypto from 1.1.0 to 1.2.0 in Spark > (SPARK-45617), got the following error in one test: > ``` > [info] Cause: java.security.GeneralSecurityException: Native library is not > loaded > [info] at > org.apache.commons.crypto.random.OpenSslCryptoRandom.checkNative(OpenSslCryptoRandom.java:79) > [info] at > org.apache.commons.crypto.random.OpenSslCryptoRandom.(OpenSslCryptoRandom.java:65) > > [info] at java.base/java.lang.Class.forName0(Native Method) > > [info] at java.base/java.lang.Class.forName(Class.java:467) > [info] at > org.apache.commons.crypto.utils.ReflectionUtils.getClassByNameOrNull(ReflectionUtils.java:93) > [info] at > org.apache.commons.crypto.utils.ReflectionUtils.getClassByName(ReflectionUtils.java:64) > [info] at > org.apache.commons.crypto.random.CryptoRandomFactory.getCryptoRandom(CryptoRandomFactory.java:189) > [info] at > org.apache.spark.security.CryptoStreamUtils$.createInitializationVector(CryptoStreamUtils.scala:138) > [info] at > org.apache.spark.security.CryptoStreamUtils$.createCryptoOutputStream(CryptoStreamUtils.scala:56) > [info] at > org.apache.spark.serializer.SerializerManager.$anonfun$wrapForEncryption$3(SerializerManager.scala:151) > > ``` > I did a quick check. commons-crypto-1.2.0.jar only contain x86_64 Mac native > library, it doesn't provide pre-built native library for Apple Silicon. > But I also checked commons-crypto-1.1.0.jar which Spark uses currently. It > also doesn't provide native library for Apple Silicon. > Since they both don't contain native library for Apple Silicon, why > commons-crypto-1.2.0 doesn't work and what makes the difference between > commons-crypto-1.1.0 and commons-crypto-1.2.0? -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (CRYPTO-169) Unable to load native library on Apple Silicon (M1 Mac)
[ https://issues.apache.org/jira/browse/CRYPTO-169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17778190#comment-17778190 ] Gary D. Gregory commented on CRYPTO-169: We do not support Apple silicone yet. The issue is that GCC does not support it yet either: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=96168 > Unable to load native library on Apple Silicon (M1 Mac) > --- > > Key: CRYPTO-169 > URL: https://issues.apache.org/jira/browse/CRYPTO-169 > Project: Commons Crypto > Issue Type: Bug >Reporter: L. C. Hsieh >Priority: Major > Attachments: image-2023-10-21-17-51-41-260.png, > image-2023-10-21-17-52-02-237.png, image-2023-10-21-17-53-16-898.png, > image-2023-10-21-17-59-13-866.png > > > Recently as trying to upgrade commons-crypto from 1.1.0 to 1.2.0 in Spark > (SPARK-45617), got the following error in one test: > ``` > [info] Cause: java.security.GeneralSecurityException: Native library is not > loaded > [info] at > org.apache.commons.crypto.random.OpenSslCryptoRandom.checkNative(OpenSslCryptoRandom.java:79) > [info] at > org.apache.commons.crypto.random.OpenSslCryptoRandom.(OpenSslCryptoRandom.java:65) > > [info] at java.base/java.lang.Class.forName0(Native Method) > > [info] at java.base/java.lang.Class.forName(Class.java:467) > [info] at > org.apache.commons.crypto.utils.ReflectionUtils.getClassByNameOrNull(ReflectionUtils.java:93) > [info] at > org.apache.commons.crypto.utils.ReflectionUtils.getClassByName(ReflectionUtils.java:64) > [info] at > org.apache.commons.crypto.random.CryptoRandomFactory.getCryptoRandom(CryptoRandomFactory.java:189) > [info] at > org.apache.spark.security.CryptoStreamUtils$.createInitializationVector(CryptoStreamUtils.scala:138) > [info] at > org.apache.spark.security.CryptoStreamUtils$.createCryptoOutputStream(CryptoStreamUtils.scala:56) > [info] at > org.apache.spark.serializer.SerializerManager.$anonfun$wrapForEncryption$3(SerializerManager.scala:151) > > ``` > I did a quick check. commons-crypto-1.2.0.jar only contain x86_64 Mac native > library, it doesn't provide pre-built native library for Apple Silicon. > But I also checked commons-crypto-1.1.0.jar which Spark uses currently. It > also doesn't provide native library for Apple Silicon. > Since they both don't contain native library for Apple Silicon, why > commons-crypto-1.2.0 doesn't work and what makes the difference between > commons-crypto-1.1.0 and commons-crypto-1.2.0? -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (CRYPTO-169) Unable to load native library on Apple Silicon (M1 Mac)
[ https://issues.apache.org/jira/browse/CRYPTO-169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17778047#comment-17778047 ] Yang Jie commented on CRYPTO-169: - !image-2023-10-21-17-51-41-260.png|width=720,height=184! !image-2023-10-21-17-52-02-237.png|width=707,height=152! [https://github.com/apache/commons-crypto/blob/c14df32ea386eadf1d459e05ca526ab1b6bc9874/src/main/java/org/apache/commons/crypto/random/OpenSslCryptoRandom.java#L64-L68] !image-2023-10-21-17-53-16-898.png|width=340,height=103! It appears to be related to the newly added `checkNative()` process in OpenSslCryptoRandom, the GeneralSecurityException wrapped by IllegalStateException is ultimately wrapped as `java.lang.ExceptionInInitializerError`. !image-2023-10-21-17-59-13-866.png|width=716,height=467! [https://github.com/apache/commons-crypto/blob/c14df32ea386eadf1d459e05ca526ab1b6bc9874/src/main/java/org/apache/commons/crypto/random/CryptoRandomFactory.java#L198-L201] So, should we add handling for `ExceptionInInitializerError` in the `CryptoRandomFactory#getCryptoRandom(java.util.Properties)` method, or change it to catch `Throwable`? > Unable to load native library on Apple Silicon (M1 Mac) > --- > > Key: CRYPTO-169 > URL: https://issues.apache.org/jira/browse/CRYPTO-169 > Project: Commons Crypto > Issue Type: Bug >Reporter: L. C. Hsieh >Priority: Major > Attachments: image-2023-10-21-17-51-41-260.png, > image-2023-10-21-17-52-02-237.png, image-2023-10-21-17-53-16-898.png, > image-2023-10-21-17-59-13-866.png > > > Recently as trying to upgrade commons-crypto from 1.1.0 to 1.2.0 in Spark > (SPARK-45617), got the following error in one test: > ``` > [info] Cause: java.security.GeneralSecurityException: Native library is not > loaded > [info] at > org.apache.commons.crypto.random.OpenSslCryptoRandom.checkNative(OpenSslCryptoRandom.java:79) > [info] at > org.apache.commons.crypto.random.OpenSslCryptoRandom.(OpenSslCryptoRandom.java:65) > > [info] at java.base/java.lang.Class.forName0(Native Method) > > [info] at java.base/java.lang.Class.forName(Class.java:467) > [info] at > org.apache.commons.crypto.utils.ReflectionUtils.getClassByNameOrNull(ReflectionUtils.java:93) > [info] at > org.apache.commons.crypto.utils.ReflectionUtils.getClassByName(ReflectionUtils.java:64) > [info] at > org.apache.commons.crypto.random.CryptoRandomFactory.getCryptoRandom(CryptoRandomFactory.java:189) > [info] at > org.apache.spark.security.CryptoStreamUtils$.createInitializationVector(CryptoStreamUtils.scala:138) > [info] at > org.apache.spark.security.CryptoStreamUtils$.createCryptoOutputStream(CryptoStreamUtils.scala:56) > [info] at > org.apache.spark.serializer.SerializerManager.$anonfun$wrapForEncryption$3(SerializerManager.scala:151) > > ``` > I did a quick check. commons-crypto-1.2.0.jar only contain x86_64 Mac native > library, it doesn't provide pre-built native library for Apple Silicon. > But I also checked commons-crypto-1.1.0.jar which Spark uses currently. It > also doesn't provide native library for Apple Silicon. > Since they both don't contain native library for Apple Silicon, why > commons-crypto-1.2.0 doesn't work and what makes the difference between > commons-crypto-1.1.0 and commons-crypto-1.2.0? -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (CRYPTO-169) Unable to load native library on Apple Silicon (M1 Mac)
[
https://issues.apache.org/jira/browse/CRYPTO-169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17778037#comment-17778037
]
Yang Jie commented on CRYPTO-169:
-
{code:java}
[error] Uncaught exception when running
org.apache.spark.sql.execution.python.RowQueueSuite:
java.lang.ExceptionInInitializerError
[error] sbt.ForkMain$ForkError: java.lang.ExceptionInInitializerError: null
[error] at java.base/java.lang.Class.forName0(Native Method)
[error] at java.base/java.lang.Class.forName(Class.java:467)
[error] at
org.apache.commons.crypto.utils.ReflectionUtils.getClassByNameOrNull(ReflectionUtils.java:93)
[error] at
org.apache.commons.crypto.utils.ReflectionUtils.getClassByName(ReflectionUtils.java:64)
[error] at
org.apache.commons.crypto.random.CryptoRandomFactory.getCryptoRandom(CryptoRandomFactory.java:189)
[error] at
org.apache.spark.security.CryptoStreamUtils$.createInitializationVector(CryptoStreamUtils.scala:138)
[error] at
org.apache.spark.security.CryptoStreamUtils$.createCryptoOutputStream(CryptoStreamUtils.scala:56)
[error] at
org.apache.spark.serializer.SerializerManager.$anonfun$wrapForEncryption$3(SerializerManager.scala:151)
[error] at scala.Option.map(Option.scala:242)
[error] at
org.apache.spark.serializer.SerializerManager.wrapForEncryption(SerializerManager.scala:151)
[error] at
org.apache.spark.sql.execution.python.DiskRowQueue.(RowQueue.scala:119)
[error] at
org.apache.spark.sql.execution.python.RowQueueSuite.$anonfun$new$2(RowQueueSuite.scala:72)
[error] at
org.apache.spark.sql.execution.python.RowQueueSuite.$anonfun$new$2$adapted(RowQueueSuite.scala:68)
[error] at
org.apache.spark.security.EncryptionFunSuite.$anonfun$encryptionTest$2(EncryptionFunSuite.scala:32)
[error] at
scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
[error] at
org.scalatest.enablers.Timed$$anon$1.timeoutAfter(Timed.scala:127)
[error] at
org.scalatest.concurrent.TimeLimits$.failAfterImpl(TimeLimits.scala:282)
[error] at
org.scalatest.concurrent.TimeLimits.failAfter(TimeLimits.scala:231)
[error] at
org.scalatest.concurrent.TimeLimits.failAfter$(TimeLimits.scala:230)
[error] at
org.apache.spark.SparkFunSuite.failAfter(SparkFunSuite.scala:69)
[error] at
org.apache.spark.SparkFunSuite.$anonfun$test$2(SparkFunSuite.scala:155)
[error] at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
[error] at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
[error] at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
[error] at org.scalatest.Transformer.apply(Transformer.scala:22)
[error] at org.scalatest.Transformer.apply(Transformer.scala:20)
[error] at
org.scalatest.funsuite.AnyFunSuiteLike$$anon$1.apply(AnyFunSuiteLike.scala:226)
[error] at
org.apache.spark.SparkFunSuite.withFixture(SparkFunSuite.scala:227)
[error] at
org.scalatest.funsuite.AnyFunSuiteLike.invokeWithFixture$1(AnyFunSuiteLike.scala:224)
[error] at
org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTest$1(AnyFunSuiteLike.scala:236)
[error] at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)
[error] at
org.scalatest.funsuite.AnyFunSuiteLike.runTest(AnyFunSuiteLike.scala:236)
[error] at
org.scalatest.funsuite.AnyFunSuiteLike.runTest$(AnyFunSuiteLike.scala:218)
[error] at
org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterEach$$super$runTest(SparkFunSuite.scala:69)
[error] at
org.scalatest.BeforeAndAfterEach.runTest(BeforeAndAfterEach.scala:234)
[error] at
org.scalatest.BeforeAndAfterEach.runTest$(BeforeAndAfterEach.scala:227)
[error] at
org.apache.spark.SparkFunSuite.runTest(SparkFunSuite.scala:69)
[error] at
org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTests$1(AnyFunSuiteLike.scala:269)
[error] at
org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:413)
[error] at scala.collection.immutable.List.foreach(List.scala:333)
[error] at
org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)
[error] at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:396)
[error] at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:475)
[error] at
org.scalatest.funsuite.AnyFunSuiteLike.runTests(AnyFunSuiteLike.scala:269)
[error] at
org.scalatest.funsuite.AnyFunSuiteLike.runTests$(AnyFunSuiteLike.scala:268)
[error] at
org.scalatest.funsuite.AnyFunSuite.runTests(AnyFunSuite.scala:1564)
[error] at org.scalatest.Suite.run(Suite.scala:1114)
[error] at org.scalatest.Suite.run$(Suite.scala:1096)
[error] at
org.scalatest.funsuite.AnyFunSuite.org$scalatest$funsuite$AnyFunSuiteLike$$super$run(AnyFunSuite.scala:1564)
[error]
[jira] [Commented] (CRYPTO-169) Unable to load native library on Apple Silicon (M1 Mac)
[ https://issues.apache.org/jira/browse/CRYPTO-169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17778033#comment-17778033 ] Gary D. Gregory commented on CRYPTO-169: https://commons.apache.org/proper/commons-crypto/changes-report.html#a1.2.0 > Unable to load native library on Apple Silicon (M1 Mac) > --- > > Key: CRYPTO-169 > URL: https://issues.apache.org/jira/browse/CRYPTO-169 > Project: Commons Crypto > Issue Type: Bug >Reporter: L. C. Hsieh >Priority: Major > > Recently as trying to upgrade commons-crypto from 1.1.0 to 1.2.0 in Spark > (SPARK-45617), got the following error in one test: > ``` > [info] Cause: java.security.GeneralSecurityException: Native library is not > loaded > [info] at > org.apache.commons.crypto.random.OpenSslCryptoRandom.checkNative(OpenSslCryptoRandom.java:79) > [info] at > org.apache.commons.crypto.random.OpenSslCryptoRandom.(OpenSslCryptoRandom.java:65) > > [info] at java.base/java.lang.Class.forName0(Native Method) > > [info] at java.base/java.lang.Class.forName(Class.java:467) > [info] at > org.apache.commons.crypto.utils.ReflectionUtils.getClassByNameOrNull(ReflectionUtils.java:93) > [info] at > org.apache.commons.crypto.utils.ReflectionUtils.getClassByName(ReflectionUtils.java:64) > [info] at > org.apache.commons.crypto.random.CryptoRandomFactory.getCryptoRandom(CryptoRandomFactory.java:189) > [info] at > org.apache.spark.security.CryptoStreamUtils$.createInitializationVector(CryptoStreamUtils.scala:138) > [info] at > org.apache.spark.security.CryptoStreamUtils$.createCryptoOutputStream(CryptoStreamUtils.scala:56) > [info] at > org.apache.spark.serializer.SerializerManager.$anonfun$wrapForEncryption$3(SerializerManager.scala:151) > > ``` > I did a quick check. commons-crypto-1.2.0.jar only contain x86_64 Mac native > library, it doesn't provide pre-built native library for Apple Silicon. > But I also checked commons-crypto-1.1.0.jar which Spark uses currently. It > also doesn't provide native library for Apple Silicon. > Since they both don't contain native library for Apple Silicon, why > commons-crypto-1.2.0 doesn't work and what makes the difference between > commons-crypto-1.1.0 and commons-crypto-1.2.0? -- This message was sent by Atlassian Jira (v8.20.10#820010)
