This is an automated email from the ASF dual-hosted git repository.
srowen pushed a commit to branch branch-3.3
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.3 by this push:
new bea58e4b863 [SPARK-41089][YARN][SHUFFLE] Relocate Netty native arm64
libs
bea58e4b863 is described below
commit bea58e4b8634fbe3497431cb50351aa186107fb3
Author: Cheng Pan <cheng...@apache.org>
AuthorDate: Thu Nov 10 08:57:56 2022 -0600
[SPARK-41089][YARN][SHUFFLE] Relocate Netty native arm64 libs
### What changes were proposed in this pull request?
SPARK-27610 relocated the netty x86 native libs, and the recent version
netty ships arm64 native libs as well, we should do same thing to make it works
on arm64 platform.
### Why are the changes needed?
Align arm64 behavior w/ x86
### Does this PR introduce _any_ user-facing change?
Yes, bug fix for ARM64 platform.
### How was this patch tested?
Before patch
```
➜ apache-spark git:(SPARK-41089) ll
common/network-yarn/target/exploded/META-INF/native
total 752
-rw-r--r-- 1 chengpan staff 101K Oct 11 23:24
libnetty_transport_native_epoll_aarch_64.so
-rw-r--r-- 1 chengpan staff 94K Oct 11 17:57
libnetty_transport_native_kqueue_aarch_64.jnilib
-rw-r--r-- 1 chengpan staff 93K Oct 11 23:27
liborg_sparkproject_netty_transport_native_epoll_x86_64.so
-rw-r--r-- 1 chengpan staff 77K Oct 11 17:51
liborg_sparkproject_netty_transport_native_kqueue_x86_64.jnilib
drwxr-xr-x 3 chengpan staff 96B Nov 9 13:46 linux32
drwxr-xr-x 3 chengpan staff 96B Nov 9 13:46 linux64
drwxr-xr-x 3 chengpan staff 96B Nov 9 13:46 osx
drwxr-xr-x 3 chengpan staff 96B Nov 9 13:46 windows32
drwxr-xr-x 3 chengpan staff 96B Nov 9 13:46 windows64
```
After patch
```
➜ apache-spark git:(SPARK-41089) ll
common/network-yarn/target/exploded/META-INF/native
total 752
-rw-r--r-- 1 chengpan staff 101K Oct 11 23:24
liborg_sparkproject_netty_transport_native_epoll_aarch_64.so
-rw-r--r-- 1 chengpan staff 93K Oct 11 23:27
liborg_sparkproject_netty_transport_native_epoll_x86_64.so
-rw-r--r-- 1 chengpan staff 94K Oct 11 17:57
liborg_sparkproject_netty_transport_native_kqueue_aarch_64.jnilib
-rw-r--r-- 1 chengpan staff 77K Oct 11 17:51
liborg_sparkproject_netty_transport_native_kqueue_x86_64.jnilib
drwxr-xr-x 3 chengpan staff 96B Nov 10 12:07 linux32
drwxr-xr-x 3 chengpan staff 96B Nov 10 12:07 linux64
drwxr-xr-x 3 chengpan staff 96B Nov 10 12:07 osx
drwxr-xr-x 3 chengpan staff 96B Nov 10 12:07 windows32
drwxr-xr-x 3 chengpan staff 96B Nov 10 12:07 windows64
```
Closes #38593 from pan3793/SPARK-41089.
Authored-by: Cheng Pan <cheng...@apache.org>
Signed-off-by: Sean Owen <sro...@gmail.com>
(cherry picked from commit c72d39990182ad2207c8cdd523af06ee4dc02fc5)
Signed-off-by: Sean Owen <sro...@gmail.com>
---
common/network-yarn/pom.xml | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/common/network-yarn/pom.xml b/common/network-yarn/pom.xml
index 14d41802a8b..81146a36c98 100644
--- a/common/network-yarn/pom.xml
+++ b/common/network-yarn/pom.xml
@@ -174,6 +174,10 @@
tofile="${project.build.directory}/exploded/META-INF/native/lib${spark.shade.native.packageName}_netty_transport_native_epoll_x86_64.so"
/>
<move
file="${project.build.directory}/exploded/META-INF/native/libnetty_transport_native_kqueue_x86_64.jnilib"
tofile="${project.build.directory}/exploded/META-INF/native/lib${spark.shade.native.packageName}_netty_transport_native_kqueue_x86_64.jnilib"
/>
+ <move
file="${project.build.directory}/exploded/META-INF/native/libnetty_transport_native_epoll_aarch_64.so"
+
tofile="${project.build.directory}/exploded/META-INF/native/lib${spark.shade.native.packageName}_netty_transport_native_epoll_aarch_64.so"
/>
+ <move
file="${project.build.directory}/exploded/META-INF/native/libnetty_transport_native_kqueue_aarch_64.jnilib"
+
tofile="${project.build.directory}/exploded/META-INF/native/lib${spark.shade.native.packageName}_netty_transport_native_kqueue_aarch_64.jnilib"
/>
<jar destfile="${shuffle.jar}"
basedir="${project.build.directory}/exploded" />
</target>
</configuration>
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
