[ https://issues.apache.org/jira/browse/HADOOP-17292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chao Sun resolved HADOOP-17292. ------------------------------- Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > Using lz4-java in Lz4Codec > -------------------------- > > Key: HADOOP-17292 > URL: https://issues.apache.org/jira/browse/HADOOP-17292 > Project: Hadoop Common > Issue Type: New Feature > Components: common > Affects Versions: 3.3.0 > Reporter: L. C. Hsieh > Assignee: L. C. Hsieh > Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 10h 50m > Remaining Estimate: 0h > > In Hadoop, we use native libs for lz4 codec which has several disadvantages: > It requires native libhadoop to be installed in system LD_LIBRARY_PATH, and > they have to be installed separately on each node of the clusters, container > images, or local test environments which adds huge complexities from > deployment point of view. In some environments, it requires compiling the > natives from sources which is non-trivial. Also, this approach is platform > dependent; the binary may not work in different platform, so it requires > recompilation. > It requires extra configuration of java.library.path to load the natives, and > it results higher application deployment and maintenance cost for users. > Projects such as Spark use [lz4-java|https://github.com/lz4/lz4-java] which > is JNI-based implementation. It contains native binaries in jar file, and it > can automatically load the native binaries into JVM from jar without any > setup. If a native implementation can not be found for a platform, it can > fallback to pure-java implementation of lz4. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org