i'd much prefer that we keep the test/build infrastructure in one place. we don't have ARM hardware, but there's a slim possibility i can scare something up in our older research stock...
another option would be to run the build in a arm-based docker container, which (according to the intarwebs) is possible. shane On Tue, Jun 25, 2019 at 6:35 PM Tianhua huang <huangtianhua...@gmail.com> wrote: > I forked apache/spark project and propose a job( > https://github.com/theopenlab/spark/pull/1) for spark building in OpenLab > ARM instance, this is the first step to build spark on ARM, I can enable a > periodic > job for arm building for apache/spark master if you guys like. Later I > will run tests for spark. I also willing to be the maintainer of the arm ci > of spark. > > Thanks for you attention. > > On Thu, Jun 20, 2019 at 10:17 AM Tianhua huang <huangtianhua...@gmail.com> > wrote: > >> Thanks Sean. >> >> I am very happy to hear that the community will put effort to fix the >> ARM-related issues. I'd be happy to help if you like. And could you give >> the trace link of this issue, then I can check it is fixed or not, thank >> you. >> As far as I know the old versions of spark support ARM, and now the new >> versions don't, this just shows that we need a CI to check whether the >> spark support ARM and whether some modification break it. >> I will add a demo job in OpenLab to build spark on ARM and do a simple UT >> test. Later I will give the job link. >> >> Let me know what you think. >> >> Thank you all! >> >> >> On Wed, Jun 19, 2019 at 8:47 PM Sean Owen <sro...@gmail.com> wrote: >> >>> I'd begin by reporting and fixing ARM-related issues in the build. If >>> they're small, of course we should do them. If it requires significant >>> modifications, we can discuss how much Spark can support ARM. I don't >>> think it's yet necessary for the Spark project to run these CI builds >>> until that point, but it's always welcome if people are testing that >>> separately. >>> >>> On Wed, Jun 19, 2019 at 7:41 AM Holden Karau <hol...@pigscanfly.ca> >>> wrote: >>> > >>> > Moving to dev@ for increased visibility among the developers. >>> > >>> > On Wed, Jun 19, 2019 at 1:24 AM Tianhua huang < >>> huangtianhua...@gmail.com> wrote: >>> >> >>> >> Thanks for your reply. >>> >> >>> >> As I said before, I met some problem of build or test for spark on >>> aarch64 server, so it will be better to have the ARM CI to make sure the >>> spark is compatible for AArch64 platforms. >>> >> >>> >> I’m from OpenLab team(https://openlabtesting.org/ ,a community to do >>> open source project testing. And we can support some Arm virtual machines >>> to AMPLab Jenkins, and also we have a developer team that willing to work >>> on this, we willing to maintain build CI jobs and address the CI issues. >>> What do you think? >>> >> >>> >> >>> >> Thanks for your attention. >>> >> >>> >> >>> >> On Wed, Jun 19, 2019 at 6:39 AM shane knapp <skn...@berkeley.edu> >>> wrote: >>> >>> >>> >>> yeah, we don't have any aarch64 systems for testing... this has >>> been asked before but is currently pretty low on our priority list as we >>> don't have the hardware. >>> >>> >>> >>> sorry, >>> >>> >>> >>> shane >>> >>> >>> >>> On Mon, Jun 10, 2019 at 7:08 PM Tianhua huang < >>> huangtianhua...@gmail.com> wrote: >>> >>>> >>> >>>> Hi, sorry to disturb you. >>> >>>> The CI testing for apache spark is supported by AMPLab Jenkins, and >>> I find there are some computers(most of them are Linux (amd64) arch) for >>> the CI development, but seems there is no Aarch64 computer for spark CI >>> testing. Recently, I build and run test for spark(master and branch-2.4) on >>> my arm server, and unfortunately there are some problems, for example, ut >>> test is failed due to a LEVELDBJNI native package, the details for java >>> test see http://paste.openstack.org/show/752063/ and python test see >>> http://paste.openstack.org/show/752709/ >>> >>>> So I have a question about the ARM CI testing for spark, is there >>> any plan to support it? Thank you very much and I will wait for your reply! >>> >>> >>> >>> >>> >>> >>> >>> -- >>> >>> Shane Knapp >>> >>> UC Berkeley EECS Research / RISELab Staff Technical Lead >>> >>> https://rise.cs.berkeley.edu >>> > >>> > >>> > >>> > -- >>> > Twitter: https://twitter.com/holdenkarau >>> > Books (Learning Spark, High Performance Spark, etc.): >>> https://amzn.to/2MaRAG9 >>> > YouTube Live Streams: https://www.youtube.com/user/holdenkarau >>> >> -- Shane Knapp UC Berkeley EECS Research / RISELab Staff Technical Lead https://rise.cs.berkeley.edu
