Hi all, I am glad to tell you there is a new progress of build/test spark on aarch64 server, the tests are running, see the build/test detail log https://logs.openlabtesting.org/logs/1/1/419fcb11764048d5a3cda186ea76dd43249e1f97/check/spark-build-arm64/75cc6f5/job-output.txt.gz and the aarch64 instance info see https://logs.openlabtesting.org/logs/1/1/419fcb11764048d5a3cda186ea76dd43249e1f97/check/spark-build-arm64/75cc6f5/zuul-info/zuul-info.ubuntu-xenial-arm64.txt In order to enable the test, I made some modification, the major one is to build leveldbjni local package, I forked fusesource/leveldbjni and chirino/leveldb repos, and made some modification to make sure to build the local package, see https://github.com/huangtianhua/leveldbjni/pull/1 and https://github.com/huangtianhua/leveldbjni/pull/2 , then to use it in spark, the detail you can find in https://github.com/theopenlab/spark/pull/1
Now the tests are not all successful, I will try to fix it and any suggestion is welcome, thank you all. On Mon, Jul 1, 2019 at 5:25 PM Tianhua huang <huangtianhua...@gmail.com> wrote: > We are focus on the arm instance of cloud, and now I use the arm instance > of vexxhost cloud to run the build job which mentioned above, the > specification of the arm instance is 8VCPU and 8GB of RAM, > and we can use bigger flavor to create the arm instance to run the job, if > need be. > > On Fri, Jun 28, 2019 at 6:55 PM Steve Loughran <ste...@cloudera.com.invalid> > wrote: > >> >> Be interesting to see how well a Pi4 works; with only 4GB of RAM you >> wouldn't compile with it, but you could try installing the spark jar bundle >> and then run against some NFS mounted disks: >> https://www.raspberrypi.org/magpi/raspberry-pi-4-specs-benchmarks/ ; >> unlikely to be fast, but it'd be an efficient kind of slow >> >> On Fri, Jun 28, 2019 at 3:08 AM Rui Chen <chenrui.m...@gmail.com> wrote: >> >>> > I think any AA64 work is going to have to define very clearly what >>> "works" is defined as >>> >>> +1 >>> It's very valuable to build a clear scope of these projects >>> functionality for ARM platform in upstream community, it bring confidence >>> to end user and customers when they plan to deploy these projects on ARM. >>> >>> This is absolute long term work, let's to make it step by step, CI, >>> testing, issue and resolving. >>> >>> Steve Loughran <ste...@cloudera.com.invalid> 于2019年6月27日周四 下午9:22写道： >>> >>>> level db and native codecs are invariably a problem here, as is >>>> anything else doing misaligned IO. Protobuf has also had "issues" in the >>>> past >>>> >>>> see https://issues.apache.org/jira/browse/HADOOP-16100 >>>> >>>> I think any AA64 work is going to have to define very clearly what >>>> "works" is defined as; spark standalone with a specific set of codecs is >>>> probably the first thing to aim for -no Snappy or lz4. >>>> >>>> Anything which goes near: protobuf, checksums, native code, etc is in >>>> trouble. Don't try and deploy with HDFS as the cluster FS, would be my >>>> recommendation. >>>> >>>> If you want a cluster use NFS or one of google GCS, Azure WASB for the >>>> cluster FS. And before trying either of those cloud store, run the >>>> filesystem connector test suites (hadoop-azure; google gcs github) to see >>>> that they work. If the foundational FS test suites fail, nothing else will >>>> work >>>> >>>> >>>> >>>> On Thu, Jun 27, 2019 at 3:09 AM Tianhua huang < >>>> huangtianhua...@gmail.com> wrote: >>>> >>>>> I took the ut tests on my arm instance before and reported an issue in >>>>> https://issues.apache.org/jira/browse/SPARK-27721, and seems there >>>>> was no leveldbjni native package for aarch64 in leveldbjni-all.jar(or 1.8) >>>>> https://mvnrepository.com/artifact/org.fusesource.leveldbjni/leveldbjni-all/1.8 >>>>> , we can find https://github.com/fusesource/leveldbjni/pull/82 this >>>>> pr added the aarch64 support and merged on 2 Nov 2017, but the latest >>>>> release of the repo is on 17 Oct 2013, unfortunately it didn't >>>>> include the aarch64 supporting. >>>>> >>>>> I will running the test on the job mentioned above, and will try to >>>>> fix the issue above, or if anyone have any idea of it, welcome reply me, >>>>> thank you. >>>>> >>>>> >>>>> On Wed, Jun 26, 2019 at 8:11 PM Sean Owen <sro...@gmail.com> wrote: >>>>> >>>>>> Can you begin by testing yourself? I think the first step is to make >>>>>> sure the build and tests work on ARM. If you find problems you can >>>>>> isolate them and try to fix them, or at least report them. It's only >>>>>> worth getting CI in place when we think builds will work. >>>>>> >>>>>> On Tue, Jun 25, 2019 at 9:26 PM Tianhua huang < >>>>>> huangtianhua...@gmail.com> wrote: >>>>>> > >>>>>> > Thanks Shane :) >>>>>> > >>>>>> > This sounds good, and yes I agree that it's best to keep the >>>>>> test/build infrastructure in one place. If you can't find the ARM >>>>>> resource >>>>>> we are willing to support the ARM instance :) Our goal is to make more >>>>>> open source software to be more compatible for aarch64 platform, so let's >>>>>> to do it. I will be happy if I can give some help for the goal. >>>>>> > >>>>>> > Waiting for you good news :) >>>>>> > >>>>>> > On Wed, Jun 26, 2019 at 9:47 AM shane knapp <skn...@berkeley.edu> >>>>>> wrote: >>>>>> >> >>>>>> >> ...or via VM as you mentioned earlier. :) >>>>>> >> >>>>>> >> shane (who will file a JIRA tomorrow) >>>>>> >> >>>>>> >> On Tue, Jun 25, 2019 at 6:44 PM shane knapp <skn...@berkeley.edu> >>>>>> wrote: >>>>>> >>> >>>>>> >>> i'd much prefer that we keep the test/build infrastructure in one >>>>>> place. >>>>>> >>> >>>>>> >>> we don't have ARM hardware, but there's a slim possibility i can >>>>>> scare something up in our older research stock... >>>>>> >>> >>>>>> >>> another option would be to run the build in a arm-based docker >>>>>> container, which (according to the intarwebs) is possible. >>>>>> >>> >>>>>> >>> shane >>>>>> >>> >>>>>> >>> On Tue, Jun 25, 2019 at 6:35 PM Tianhua huang < >>>>>> huangtianhua...@gmail.com> wrote: >>>>>> >>>> >>>>>> >>>> I forked apache/spark project and propose a job( >>>>>> https://github.com/theopenlab/spark/pull/1) for spark building in >>>>>> OpenLab ARM instance, this is the first step to build spark on ARM, I >>>>>> can >>>>>> enable a periodic job for arm building for apache/spark master if you >>>>>> guys >>>>>> like. Later I will run tests for spark. I also willing to be the >>>>>> maintainer of the arm ci of spark. >>>>>> >>>> >>>>>> >>>> Thanks for you attention. >>>>>> >>>> >>>>>> >>>> On Thu, Jun 20, 2019 at 10:17 AM Tianhua huang < >>>>>> huangtianhua...@gmail.com> wrote: >>>>>> >>>>> >>>>>> >>>>> Thanks Sean. >>>>>> >>>>> >>>>>> >>>>> I am very happy to hear that the community will put effort to >>>>>> fix the ARM-related issues. I'd be happy to help if you like. And could >>>>>> you >>>>>> give the trace link of this issue, then I can check it is fixed or not, >>>>>> thank you. >>>>>> >>>>> As far as I know the old versions of spark support ARM, and now >>>>>> the new versions don't, this just shows that we need a CI to check >>>>>> whether >>>>>> the spark support ARM and whether some modification break it. >>>>>> >>>>> I will add a demo job in OpenLab to build spark on ARM and do a >>>>>> simple UT test. Later I will give the job link. >>>>>> >>>>> >>>>>> >>>>> Let me know what you think. >>>>>> >>>>> >>>>>> >>>>> Thank you all! >>>>>> >>>>> >>>>>> >>>>> >>>>>> >>>>> On Wed, Jun 19, 2019 at 8:47 PM Sean Owen <sro...@gmail.com> >>>>>> wrote: >>>>>> >>>>>> >>>>>> >>>>>> I'd begin by reporting and fixing ARM-related issues in the >>>>>> build. If >>>>>> >>>>>> they're small, of course we should do them. If it requires >>>>>> significant >>>>>> >>>>>> modifications, we can discuss how much Spark can support ARM. >>>>>> I don't >>>>>> >>>>>> think it's yet necessary for the Spark project to run these CI >>>>>> builds >>>>>> >>>>>> until that point, but it's always welcome if people are >>>>>> testing that >>>>>> >>>>>> separately. >>>>>> >>>>>> >>>>>> >>>>>> On Wed, Jun 19, 2019 at 7:41 AM Holden Karau < >>>>>> hol...@pigscanfly.ca> wrote: >>>>>> >>>>>> > >>>>>> >>>>>> > Moving to dev@ for increased visibility among the >>>>>> developers. >>>>>> >>>>>> > >>>>>> >>>>>> > On Wed, Jun 19, 2019 at 1:24 AM Tianhua huang < >>>>>> huangtianhua...@gmail.com> wrote: >>>>>> >>>>>> >> >>>>>> >>>>>> >> Thanks for your reply. >>>>>> >>>>>> >> >>>>>> >>>>>> >> As I said before, I met some problem of build or test for >>>>>> spark on aarch64 server, so it will be better to have the ARM CI to make >>>>>> sure the spark is compatible for AArch64 platforms. >>>>>> >>>>>> >> >>>>>> >>>>>> >> I’m from OpenLab team(https://openlabtesting.org/ ,a >>>>>> community to do open source project testing. And we can support some Arm >>>>>> virtual machines to AMPLab Jenkins, and also we have a developer team >>>>>> that >>>>>> willing to work on this, we willing to maintain build CI jobs and address >>>>>> the CI issues. What do you think? >>>>>> >>>>>> >> >>>>>> >>>>>> >> >>>>>> >>>>>> >> Thanks for your attention. >>>>>> >>>>>> >> >>>>>> >>>>>> >> >>>>>> >>>>>> >> On Wed, Jun 19, 2019 at 6:39 AM shane knapp < >>>>>> skn...@berkeley.edu> wrote: >>>>>> >>>>>> >>> >>>>>> >>>>>> >>> yeah, we don't have any aarch64 systems for testing... >>>>>> this has been asked before but is currently pretty low on our priority >>>>>> list >>>>>> as we don't have the hardware. >>>>>> >>>>>> >>> >>>>>> >>>>>> >>> sorry, >>>>>> >>>>>> >>> >>>>>> >>>>>> >>> shane >>>>>> >>>>>> >>> >>>>>> >>>>>> >>> On Mon, Jun 10, 2019 at 7:08 PM Tianhua huang < >>>>>> huangtianhua...@gmail.com> wrote: >>>>>> >>>>>> >>>> >>>>>> >>>>>> >>>> Hi, sorry to disturb you. >>>>>> >>>>>> >>>> The CI testing for apache spark is supported by AMPLab >>>>>> Jenkins, and I find there are some computers(most of them are Linux >>>>>> (amd64) >>>>>> arch) for the CI development, but seems there is no Aarch64 computer for >>>>>> spark CI testing. Recently, I build and run test for spark(master and >>>>>> branch-2.4) on my arm server, and unfortunately there are some problems, >>>>>> for example, ut test is failed due to a LEVELDBJNI native package, the >>>>>> details for java test see http://paste.openstack.org/show/752063/ >>>>>> and python test see http://paste.openstack.org/show/752709/ >>>>>> >>>>>> >>>> So I have a question about the ARM CI testing for spark, >>>>>> is there any plan to support it? Thank you very much and I will wait for >>>>>> your reply! >>>>>> >>>>>> >>> >>>>>> >>>>>> >>> >>>>>> >>>>>> >>> >>>>>> >>>>>> >>> -- >>>>>> >>>>>> >>> Shane Knapp >>>>>> >>>>>> >>> UC Berkeley EECS Research / RISELab Staff Technical Lead >>>>>> >>>>>> >>> https://rise.cs.berkeley.edu >>>>>> >>>>>> > >>>>>> >>>>>> > >>>>>> >>>>>> > >>>>>> >>>>>> > -- >>>>>> >>>>>> > Twitter: https://twitter.com/holdenkarau >>>>>> >>>>>> > Books (Learning Spark, High Performance Spark, etc.): >>>>>> https://amzn.to/2MaRAG9 >>>>>> >>>>>> > YouTube Live Streams: >>>>>> https://www.youtube.com/user/holdenkarau >>>>>> >>> >>>>>> >>> >>>>>> >>> >>>>>> >>> -- >>>>>> >>> Shane Knapp >>>>>> >>> UC Berkeley EECS Research / RISELab Staff Technical Lead >>>>>> >>> https://rise.cs.berkeley.edu >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> >> -- >>>>>> >> Shane Knapp >>>>>> >> UC Berkeley EECS Research / RISELab Staff Technical Lead >>>>>> >> https://rise.cs.berkeley.edu >>>>>> >>>>>
