[ https://issues.apache.org/jira/browse/SPARK-28102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Josh Rosen updated SPARK-28102: ------------------------------- Summary: Failed LZ4 JNI initialization is repeatedly re-attempted, causing lock contention issues (was: Add configuration for selecting LZ4 implementation (safe, unsafe, JNI)) > Failed LZ4 JNI initialization is repeatedly re-attempted, causing lock > contention issues > ---------------------------------------------------------------------------------------- > > Key: SPARK-28102 > URL: https://issues.apache.org/jira/browse/SPARK-28102 > Project: Spark > Issue Type: Improvement > Components: Spark Core > Affects Versions: 3.0.0 > Reporter: Josh Rosen > Assignee: Josh Rosen > Priority: Major > > Spark's use of {{lz4-java}} ends up calling {{Lz4Factory.fastestInstance}}, > which attempts to load JNI libraries and falls back on Java implementations > in case the JNI library cannot be loaded or initialized. > I run Spark in a configuration where the JNI libraries don't work, so I'd > like to configure LZ4 to not even attempt to use JNI code: if the JNI library > loads but cannot be initialized then the fallback code path involves catching > an exception and this is slow because the exception is thrown under a static > initializer lock (leading to significant lock contention because the filling > of stacktraces is done while holding this lock). > I propose to introduce a {{spark.io.compression.lz4.factory}} configuration > for selecting the LZ4 implementation, allowing users to disable the use of > the JNI library without having to recompile Spark. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org