Status of Spark testing on ARM64

[Tianhua huang]

Hi all,
I will give you some informations about ARM CI of Spark:

Our team and community are working on build/test Spark master on ARM64
server, after find and fix some issues[1], we have integrated two ARM
testing jobs[2] to community CI(AMPLAB Jenkins),
they run as daily job and have been stablely running for few weeks, and the
two ARM testing jobs are success generally.
Thanks Sean Owen, Shane Knapp, Dongjoon Hyun and community to help us :)

If you are interested, please have a try:)  Before
https://github.com/apache/spark/pull/26636 merged, you have to download and
maven install org.openlabtesting.leveldbjni:leveldbjni-all.1.8 using
commands:
wget
https://repo1.maven.org/maven2/org/openlabtesting/leveldbjni/leveldbjni-all/1.8/leveldbjni-all-1.8.jar

mvn install:install-file -DgroupId=org.fusesource.leveldbjni
-DartifactId=leveldbjni-all -Dversion=1.8 -Dpackaging=jar
-Dfile=leveldbjni-all-1.8.jar
Then, you can build and test Spark on ARM64 server.

If you have any questions, please don't hesitate to contact me, thanks all!

[1]:
https://issues.apache.org/jira/browse/SPARK-28770 （
https://github.com/apache/spark/pull/25673）
https://issues.apache.org/jira/browse/SPARK-28519 （
https://github.com/apache/spark/pull/25279）
https://issues.apache.org/jira/browse/SPARK-28433 （
https://github.com/apache/spark/pull/25186）
https://issues.apache.org/jira/browse/SPARK-28467 （
https://github.com/apache/spark/pull/25864）
https://issues.apache.org/jira/browse/SPARK-29286 （
https://github.com/apache/spark/pull/26021）
https://issues.apache.org/jira/browse/SPARK-29286
   (
https://github.com/apache/spark/pull/26636) --- this one is in progress

[2]:
   https://amplab.cs.berkeley.edu/jenkins/job/spark-master-test-maven-arm/
 The job spark-master-test-maven-arm same with community x86 job
https://amplab.cs.berkeley.edu/jenkins/job/spark-master-test-maven-hadoop-2.7/
,
 It runs all java/scala tests, the test number is about 21,112.
   https://amplab.cs.berkeley.edu/jenkins/job/spark-master-test-python-arm/
 The job spark-master-test-python-arm runs pyspark tests with python3.6.

Re: Ask for ARM CI for spark

[Tianhua huang]

Thanks for your reply.

As I said before, I met some problem of build or test for spark on aarch64
server, so it will be better to have the ARM CI to make sure the spark
is compatible
for AArch64 platforms.

I’m from OpenLab team(https://openlabtesting.org/ ,a community to do open
source project testing. And we can support some Arm virtual machines to
AMPLab Jenkins, and also we have a developer team that willing to work on
this, we willing to maintain build CI jobs and address the CI issues.
What do you think?


Thanks for your attention.

On Wed, Jun 19, 2019 at 6:39 AM shane knapp  wrote:

> yeah, we don't have any aarch64 systems for testing...  this has been
> asked before but is currently pretty low on our priority list as we don't
> have the hardware.
>
> sorry,
>
> shane
>
> On Mon, Jun 10, 2019 at 7:08 PM Tianhua huang 
> wrote:
>
>> Hi, sorry to disturb you.
>> The CI testing for apache spark is supported by AMPLab Jenkins, and I
>> find there are some computers(most of them are Linux (amd64) arch) for
>> the CI development, but seems there is no Aarch64 computer for spark CI
>> testing. Recently, I build and run test for spark(master and branch-2.4) on
>> my arm server, and unfortunately there are some problems, for example, ut
>> test is failed due to a LEVELDBJNI native package, the details for java
>> test see http://paste.openstack.org/show/752063/ and python test see
>> http://paste.openstack.org/show/752709/
>> So I have a question about the ARM CI testing for spark, is there any
>> plan to support it? Thank you very much and I will wait for your reply!
>>
>
>
> --
> Shane Knapp
> UC Berkeley EECS Research / RISELab Staff Technical Lead
> https://rise.cs.berkeley.edu
>

ARM CI for spark

[Tianhua huang]

Hi all,
The CI testing for apache spark is supported by AMPLab Jenkins, and I find
there are some computers(most of them are Linux (amd64) arch) for the CI
development, but seems there is no Aarch64 computer for spark CI testing.
Recently, I build and run test for spark(master and branch-2.4) on my arm
server, and unfortunately there are some problems, for example, ut test is
failed due to a LEVELDBJNI native package, the details for java test see
http://paste.openstack.org/show/752063/ and also  python test see
http://paste.openstack.org/show/752709/
and I have send a mail for ask for help
http://mail-archives.apache.org/mod_mbox/spark-user/201906.mbox/%3ccakzhfyxi8wpyytgxnz5o9rhrpjve88g0chxww8x31ubkfr8...@mail.gmail.com%3E
So I have a question about the ARM CI testing for spark, is there any plan
to support it? Thank you very much and I will wait for your reply!

spark ./build/mvn test failed on aarch64

[Tianhua huang]

Hi all,
Recently I run './build/mvn test' of spark on aarch64, and master and
branch-2.4 are all failled, the log pieces as below:

..

[INFO] T E S T S
[INFO] ---
[INFO] Running org.apache.spark.util.kvstore.LevelDBTypeInfoSuite
[INFO] Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
0.081 s - in org.apache.spark.util.kvstore.LevelDBTypeInfoSuite
[INFO] Running org.apache.spark.util.kvstore.InMemoryStoreSuite
[INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
0.001 s - in org.apache.spark.util.kvstore.InMemoryStoreSuite
[INFO] Running org.apache.spark.util.kvstore.InMemoryIteratorSuite
[INFO] Tests run: 38, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
0.219 s - in org.apache.spark.util.kvstore.InMemoryIteratorSuite
[INFO] Running org.apache.spark.util.kvstore.LevelDBIteratorSuite
[ERROR] Tests run: 38, Failures: 0, Errors: 38, Skipped: 0, Time elapsed:
0.23 s <<< FAILURE! - in org.apache.spark.util.kvstore.LevelDBIteratorSuite
[ERROR] 
copyIndexDescendingWithStart(org.apache.spark.util.kvstore.LevelDBIteratorSuite)
Time elapsed: 0.2 s <<< ERROR!
java.lang.UnsatisfiedLinkError: Could not load library. Reasons: [no
leveldbjni64-1.8 in java.library.path, no leveldbjni-1.8 in
java.library.path, no leveldbjni in java.library.path,
/usr/local/src/spark/common/kvstore/target/tmp/libleveldbjni-64-1-610267671268036503.8:
/usr/local/src/spark/common/kvstore/target/tmp/libleveldbjni-64-1-610267671268036503.8:
cannot open shared object file: No such file or directory (Possible cause:
can't load AMD 64-bit .so on a AARCH64-bit platform)]
at
org.apache.spark.util.kvstore.LevelDBIteratorSuite.createStore(LevelDBIteratorSuite.java:44)

..

There is a dependency of  leveldbjni-all  , but there is no the native
package for aarch64 i in leveldbjni-1.8(all) .jar, I found aarch64 is
supported after pr https://github.com/fusesource/leveldbjni/pull/82, but it
was not in the 1.8 release, and unfortunately the repo didn't updated
almost for

two years.

So I have a question: does spark support aarch64, and if it is yes, then
how to fix this problem, if it is not, what's

the plan for it? Thank you all!

spark ./build/mvn test failed on aarch64

[Tianhua huang]

Hi all,
Recently I run './build/mvn test' of spark on aarch64, and master and
branch-2.4 are all failled, the log pieces as below:

..

[INFO] T E S T S
[INFO] ---
[INFO] Running org.apache.spark.util.kvstore.LevelDBTypeInfoSuite
[INFO] Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
0.081 s - in org.apache.spark.util.kvstore.LevelDBTypeInfoSuite
[INFO] Running org.apache.spark.util.kvstore.InMemoryStoreSuite
[INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
0.001 s - in org.apache.spark.util.kvstore.InMemoryStoreSuite
[INFO] Running org.apache.spark.util.kvstore.InMemoryIteratorSuite
[INFO] Tests run: 38, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
0.219 s - in org.apache.spark.util.kvstore.InMemoryIteratorSuite
[INFO] Running org.apache.spark.util.kvstore.LevelDBIteratorSuite
[ERROR] Tests run: 38, Failures: 0, Errors: 38, Skipped: 0, Time elapsed:
0.23 s <<< FAILURE! - in org.apache.spark.util.kvstore.LevelDBIteratorSuite
[ERROR] 
copyIndexDescendingWithStart(org.apache.spark.util.kvstore.LevelDBIteratorSuite)
Time elapsed: 0.2 s <<< ERROR!
java.lang.UnsatisfiedLinkError: Could not load library. Reasons: [no
leveldbjni64-1.8 in java.library.path, no leveldbjni-1.8 in
java.library.path, no leveldbjni in java.library.path,
/usr/local/src/spark/common/kvstore/target/tmp/libleveldbjni-64-1-610267671268036503.8:
/usr/local/src/spark/common/kvstore/target/tmp/libleveldbjni-64-1-610267671268036503.8:
cannot open shared object file: No such file or directory (Possible cause:
can't load AMD 64-bit .so on a AARCH64-bit platform)]
at
org.apache.spark.util.kvstore.LevelDBIteratorSuite.createStore(LevelDBIteratorSuite.java:44)

..

There is a dependency of LEVELDBJNI:

org.fusesource.leveldbjni
leveldbjni-all
1.8