level db and native codecs are invariably a problem here, as is anything else doing misaligned IO. Protobuf has also had "issues" in the past
see https://issues.apache.org/jira/browse/HADOOP-16100 I think any AA64 work is going to have to define very clearly what "works" is defined as; spark standalone with a specific set of codecs is probably the first thing to aim for -no Snappy or lz4. Anything which goes near: protobuf, checksums, native code, etc is in trouble. Don't try and deploy with HDFS as the cluster FS, would be my recommendation. If you want a cluster use NFS or one of google GCS, Azure WASB for the cluster FS. And before trying either of those cloud store, run the filesystem connector test suites (hadoop-azure; google gcs github) to see that they work. If the foundational FS test suites fail, nothing else will work On Thu, Jun 27, 2019 at 3:09 AM Tianhua huang <huangtianhua...@gmail.com> wrote: > I took the ut tests on my arm instance before and reported an issue in > https://issues.apache.org/jira/browse/SPARK-27721, and seems there was > no leveldbjni native package for aarch64 in leveldbjni-all.jar(or 1.8) > https://mvnrepository.com/artifact/org.fusesource.leveldbjni/leveldbjni-all/1.8 > , we can find https://github.com/fusesource/leveldbjni/pull/82 this pr > added the aarch64 support and merged on 2 Nov 2017, but the latest release > of the repo is on 17 Oct 2013, unfortunately it didn't include the > aarch64 supporting. > > I will running the test on the job mentioned above, and will try to fix > the issue above, or if anyone have any idea of it, welcome reply me, thank > you. > > > On Wed, Jun 26, 2019 at 8:11 PM Sean Owen <sro...@gmail.com> wrote: > >> Can you begin by testing yourself? I think the first step is to make >> sure the build and tests work on ARM. If you find problems you can >> isolate them and try to fix them, or at least report them. It's only >> worth getting CI in place when we think builds will work. >> >> On Tue, Jun 25, 2019 at 9:26 PM Tianhua huang <huangtianhua...@gmail.com> >> wrote: >> > >> > Thanks Shane :) >> > >> > This sounds good, and yes I agree that it's best to keep the test/build >> infrastructure in one place. If you can't find the ARM resource we are >> willing to support the ARM instance :) Our goal is to make more open >> source software to be more compatible for aarch64 platform, so let's to do >> it. I will be happy if I can give some help for the goal. >> > >> > Waiting for you good news :) >> > >> > On Wed, Jun 26, 2019 at 9:47 AM shane knapp <skn...@berkeley.edu> >> wrote: >> >> >> >> ...or via VM as you mentioned earlier. :) >> >> >> >> shane (who will file a JIRA tomorrow) >> >> >> >> On Tue, Jun 25, 2019 at 6:44 PM shane knapp <skn...@berkeley.edu> >> wrote: >> >>> >> >>> i'd much prefer that we keep the test/build infrastructure in one >> place. >> >>> >> >>> we don't have ARM hardware, but there's a slim possibility i can >> scare something up in our older research stock... >> >>> >> >>> another option would be to run the build in a arm-based docker >> container, which (according to the intarwebs) is possible. >> >>> >> >>> shane >> >>> >> >>> On Tue, Jun 25, 2019 at 6:35 PM Tianhua huang < >> huangtianhua...@gmail.com> wrote: >> >>>> >> >>>> I forked apache/spark project and propose a job( >> https://github.com/theopenlab/spark/pull/1) for spark building in >> OpenLab ARM instance, this is the first step to build spark on ARM, I can >> enable a periodic job for arm building for apache/spark master if you guys >> like. Later I will run tests for spark. I also willing to be the >> maintainer of the arm ci of spark. >> >>>> >> >>>> Thanks for you attention. >> >>>> >> >>>> On Thu, Jun 20, 2019 at 10:17 AM Tianhua huang < >> huangtianhua...@gmail.com> wrote: >> >>>>> >> >>>>> Thanks Sean. >> >>>>> >> >>>>> I am very happy to hear that the community will put effort to fix >> the ARM-related issues. I'd be happy to help if you like. And could you >> give the trace link of this issue, then I can check it is fixed or not, >> thank you. >> >>>>> As far as I know the old versions of spark support ARM, and now the >> new versions don't, this just shows that we need a CI to check whether the >> spark support ARM and whether some modification break it. >> >>>>> I will add a demo job in OpenLab to build spark on ARM and do a >> simple UT test. Later I will give the job link. >> >>>>> >> >>>>> Let me know what you think. >> >>>>> >> >>>>> Thank you all! >> >>>>> >> >>>>> >> >>>>> On Wed, Jun 19, 2019 at 8:47 PM Sean Owen <sro...@gmail.com> wrote: >> >>>>>> >> >>>>>> I'd begin by reporting and fixing ARM-related issues in the build. >> If >> >>>>>> they're small, of course we should do them. If it requires >> significant >> >>>>>> modifications, we can discuss how much Spark can support ARM. I >> don't >> >>>>>> think it's yet necessary for the Spark project to run these CI >> builds >> >>>>>> until that point, but it's always welcome if people are testing >> that >> >>>>>> separately. >> >>>>>> >> >>>>>> On Wed, Jun 19, 2019 at 7:41 AM Holden Karau <hol...@pigscanfly.ca> >> wrote: >> >>>>>> > >> >>>>>> > Moving to dev@ for increased visibility among the developers. >> >>>>>> > >> >>>>>> > On Wed, Jun 19, 2019 at 1:24 AM Tianhua huang < >> huangtianhua...@gmail.com> wrote: >> >>>>>> >> >> >>>>>> >> Thanks for your reply. >> >>>>>> >> >> >>>>>> >> As I said before, I met some problem of build or test for spark >> on aarch64 server, so it will be better to have the ARM CI to make sure the >> spark is compatible for AArch64 platforms. >> >>>>>> >> >> >>>>>> >> I’m from OpenLab team(https://openlabtesting.org/ ,a community >> to do open source project testing. And we can support some Arm virtual >> machines to AMPLab Jenkins, and also we have a developer team that willing >> to work on this, we willing to maintain build CI jobs and address the CI >> issues. What do you think? >> >>>>>> >> >> >>>>>> >> >> >>>>>> >> Thanks for your attention. >> >>>>>> >> >> >>>>>> >> >> >>>>>> >> On Wed, Jun 19, 2019 at 6:39 AM shane knapp < >> skn...@berkeley.edu> wrote: >> >>>>>> >>> >> >>>>>> >>> yeah, we don't have any aarch64 systems for testing... this >> has been asked before but is currently pretty low on our priority list as >> we don't have the hardware. >> >>>>>> >>> >> >>>>>> >>> sorry, >> >>>>>> >>> >> >>>>>> >>> shane >> >>>>>> >>> >> >>>>>> >>> On Mon, Jun 10, 2019 at 7:08 PM Tianhua huang < >> huangtianhua...@gmail.com> wrote: >> >>>>>> >>>> >> >>>>>> >>>> Hi, sorry to disturb you. >> >>>>>> >>>> The CI testing for apache spark is supported by AMPLab >> Jenkins, and I find there are some computers(most of them are Linux (amd64) >> arch) for the CI development, but seems there is no Aarch64 computer for >> spark CI testing. Recently, I build and run test for spark(master and >> branch-2.4) on my arm server, and unfortunately there are some problems, >> for example, ut test is failed due to a LEVELDBJNI native package, the >> details for java test see http://paste.openstack.org/show/752063/ and >> python test see http://paste.openstack.org/show/752709/ >> >>>>>> >>>> So I have a question about the ARM CI testing for spark, is >> there any plan to support it? Thank you very much and I will wait for your >> reply! >> >>>>>> >>> >> >>>>>> >>> >> >>>>>> >>> >> >>>>>> >>> -- >> >>>>>> >>> Shane Knapp >> >>>>>> >>> UC Berkeley EECS Research / RISELab Staff Technical Lead >> >>>>>> >>> https://rise.cs.berkeley.edu >> >>>>>> > >> >>>>>> > >> >>>>>> > >> >>>>>> > -- >> >>>>>> > Twitter: https://twitter.com/holdenkarau >> >>>>>> > Books (Learning Spark, High Performance Spark, etc.): >> https://amzn.to/2MaRAG9 >> >>>>>> > YouTube Live Streams: https://www.youtube.com/user/holdenkarau >> >>> >> >>> >> >>> >> >>> -- >> >>> Shane Knapp >> >>> UC Berkeley EECS Research / RISELab Staff Technical Lead >> >>> https://rise.cs.berkeley.edu >> >> >> >> >> >> >> >> -- >> >> Shane Knapp >> >> UC Berkeley EECS Research / RISELab Staff Technical Lead >> >> https://rise.cs.berkeley.edu >> >
