But maybe there is a way to tell Hadoop to build with awareness without installing the ISA-L?
You need the library of isa-l on the host building Hadoop. Hadoop uses CMake to find isa-l. If required libary (libisal.so) is found on library search path, it will be enabled and linked. - https://github.com/apache/hadoop/blob/rel/release-3.2.2/hadoop-common-project/hadoop-common/src/CMakeLists.txt#L124-L161 You can specify header/library searchpath by system properties on running mvn. - https://github.com/apache/hadoop/blob/rel/release-3.2.2/hadoop-common-project/hadoop-common/pom.xml#L679-L681 We are using bigtop_toolchain to set up building environment. You can add installation of isa-l in there. - https://github.com/apache/bigtop/tree/master/bigtop_toolchain Of cource, you need the libisal.so on every node in the Hadoop cluster. Instead of installing it, you can bundle the libisa.so with Hadoop by specifying properties like `-Dbundle.isal -Disal.lib=/usr/lib64` on `-Pdist` build. - https://github.com/apache/hadoop/blob/rel/release-3.2.2/hadoop-project-dist/pom.xml#L47-L48 You need to modify files in bigtop-packages such as install_hadoop.sh, hadoop.spec, ... too for making the .so bundled with .rpm/.deb packages. - https://github.com/apache/bigtop/tree/master/bigtop-packages/src/common/hadoop - https://github.com/apache/bigtop/tree/master/bigtop-packages/src/rpm/hadoop - https://github.com/apache/bigtop/tree/master/bigtop-packages/src/deb/hadoop On 2021/07/21 0:01, Chad William Seys wrote:
I'm glad there was a bare metal version of the build! We're going to test HDFS packaged to be ISA-L aware installed on a cluster without the ISA-L libraries installed. If it handles this case gracefully I think it would it make sense to package ISA-L aware Hadoop. It would give Bigtop Hadoop the flexibility to use or not use ISA-L "out of the box". Not sure how to trigger ISA-L awareness other than by installing ISA-L before packaging. As mentioned in this thread there is a github repo which automates the creation of RPMs. But maybe there is a way to tell Hadoop to build with awareness without installing the ISA-L? Thanks again, Chad.
