This is an automated email from the ASF dual-hosted git repository. srowen pushed a commit to branch branch-3.3 in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.3 by this push: new bea58e4b863 [SPARK-41089][YARN][SHUFFLE] Relocate Netty native arm64 libs bea58e4b863 is described below commit bea58e4b8634fbe3497431cb50351aa186107fb3 Author: Cheng Pan <cheng...@apache.org> AuthorDate: Thu Nov 10 08:57:56 2022 -0600 [SPARK-41089][YARN][SHUFFLE] Relocate Netty native arm64 libs ### What changes were proposed in this pull request? SPARK-27610 relocated the netty x86 native libs, and the recent version netty ships arm64 native libs as well, we should do same thing to make it works on arm64 platform. ### Why are the changes needed? Align arm64 behavior w/ x86 ### Does this PR introduce _any_ user-facing change? Yes, bug fix for ARM64 platform. ### How was this patch tested? Before patch ``` ➜ apache-spark git:(SPARK-41089) ll common/network-yarn/target/exploded/META-INF/native total 752 -rw-r--r-- 1 chengpan staff 101K Oct 11 23:24 libnetty_transport_native_epoll_aarch_64.so -rw-r--r-- 1 chengpan staff 94K Oct 11 17:57 libnetty_transport_native_kqueue_aarch_64.jnilib -rw-r--r-- 1 chengpan staff 93K Oct 11 23:27 liborg_sparkproject_netty_transport_native_epoll_x86_64.so -rw-r--r-- 1 chengpan staff 77K Oct 11 17:51 liborg_sparkproject_netty_transport_native_kqueue_x86_64.jnilib drwxr-xr-x 3 chengpan staff 96B Nov 9 13:46 linux32 drwxr-xr-x 3 chengpan staff 96B Nov 9 13:46 linux64 drwxr-xr-x 3 chengpan staff 96B Nov 9 13:46 osx drwxr-xr-x 3 chengpan staff 96B Nov 9 13:46 windows32 drwxr-xr-x 3 chengpan staff 96B Nov 9 13:46 windows64 ``` After patch ``` ➜ apache-spark git:(SPARK-41089) ll common/network-yarn/target/exploded/META-INF/native total 752 -rw-r--r-- 1 chengpan staff 101K Oct 11 23:24 liborg_sparkproject_netty_transport_native_epoll_aarch_64.so -rw-r--r-- 1 chengpan staff 93K Oct 11 23:27 liborg_sparkproject_netty_transport_native_epoll_x86_64.so -rw-r--r-- 1 chengpan staff 94K Oct 11 17:57 liborg_sparkproject_netty_transport_native_kqueue_aarch_64.jnilib -rw-r--r-- 1 chengpan staff 77K Oct 11 17:51 liborg_sparkproject_netty_transport_native_kqueue_x86_64.jnilib drwxr-xr-x 3 chengpan staff 96B Nov 10 12:07 linux32 drwxr-xr-x 3 chengpan staff 96B Nov 10 12:07 linux64 drwxr-xr-x 3 chengpan staff 96B Nov 10 12:07 osx drwxr-xr-x 3 chengpan staff 96B Nov 10 12:07 windows32 drwxr-xr-x 3 chengpan staff 96B Nov 10 12:07 windows64 ``` Closes #38593 from pan3793/SPARK-41089. Authored-by: Cheng Pan <cheng...@apache.org> Signed-off-by: Sean Owen <sro...@gmail.com> (cherry picked from commit c72d39990182ad2207c8cdd523af06ee4dc02fc5) Signed-off-by: Sean Owen <sro...@gmail.com> --- common/network-yarn/pom.xml | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/common/network-yarn/pom.xml b/common/network-yarn/pom.xml index 14d41802a8b..81146a36c98 100644 --- a/common/network-yarn/pom.xml +++ b/common/network-yarn/pom.xml @@ -174,6 +174,10 @@ tofile="${project.build.directory}/exploded/META-INF/native/lib${spark.shade.native.packageName}_netty_transport_native_epoll_x86_64.so" /> <move file="${project.build.directory}/exploded/META-INF/native/libnetty_transport_native_kqueue_x86_64.jnilib" tofile="${project.build.directory}/exploded/META-INF/native/lib${spark.shade.native.packageName}_netty_transport_native_kqueue_x86_64.jnilib" /> + <move file="${project.build.directory}/exploded/META-INF/native/libnetty_transport_native_epoll_aarch_64.so" + tofile="${project.build.directory}/exploded/META-INF/native/lib${spark.shade.native.packageName}_netty_transport_native_epoll_aarch_64.so" /> + <move file="${project.build.directory}/exploded/META-INF/native/libnetty_transport_native_kqueue_aarch_64.jnilib" + tofile="${project.build.directory}/exploded/META-INF/native/lib${spark.shade.native.packageName}_netty_transport_native_kqueue_aarch_64.jnilib" /> <jar destfile="${shuffle.jar}" basedir="${project.build.directory}/exploded" /> </target> </configuration> --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org