Re: Ask for ARM CI for spark

Hi @Sean Owen , Thanks for your reply and patient. First, we are so apologized for the bad words in the previous emails. We just want to make the users can see the current support status in some place of spark community. I'm really appreciated that you and spark community make spark better on

Re: Ask for ARM CI for spark

We can talk about this later, but I have to update some things:) - It (largely) worked previously --- But no one sure about this before the arm testing, and it can't be found anywhere, specify officially will make it more clear - I think you're also saying you don't have 100% tests passing

Re: Ask for ARM CI for spark

Same response as before: - It is in the list of resolved JIRAs, of course - It (largely) worked previously - I think you're also saying you don't have 100% tests passing anyway, though probably just small issues - It does not seem to merit a special announcement from the PMC among the 2000+

Re: Ask for ARM CI for spark

@Sean Owen ， I'm afraid I don't agree with you this time, I still remember no one can tell me whether Spark supports ARM or how much Spark can support ARM when I asked this first time on Dev@, you're very kind and told me to build and test on ARM locally and so sorry I think you were not sure

Re: Ask for ARM CI for spark

The ASF PR team would like something like that "Spark now supports ARM" in press releases. And don't forget: they do you like to be involved in the launch of the final release. On Fri, Nov 15, 2019 at 9:46 AM bo zhaobo wrote: > Hi @Sean Owen , > > Thanks for your idea. > > We may use the bad

Re: Ask for ARM CI for spark

Hi @Sean Owen , Thanks for your idea. We may use the bad words to describe our request. That's true that we cannot just say "Spark support ARM from release 3.0.0", and we also cannot say the past releases cannot run on ARM. But the reality is the past releases didn't get a fully test on ARM

Re: Ask for ARM CI for spark

@Sean Owen, Thanks for attention this. I agree with you, it's probably not very appropriate to say 'support arm from 3.0 release'. How about change to the word "Spark community supports fully tests on arm from 3.0 release"? Let's try to think about it from the user's point of view than

Re: Ask for ARM CI for spark

I'm not against it, but the JIRAs will already show that the small ARM-related difference like floating-point in log() were resolved. Those aren't major enough to highlight as key changes in the 2000+ resolved. it didn't really not-work before either, as I understand; Spark isn't specific to an

Re: Ask for ARM CI for spark

Hi, And I found Spark-3.0.0-preview had released, but there is no releasenotes in [1]. So how about to add support ARM notes in the next releasenotes(maybe the releasenotes of Spark-3.0.0-preview). So sorry to raise this, I'm not familiar with this, if any bad from me please feel free to correct.

Re: Ask for ARM CI for spark

Hi @Sean Owen , Thanks for reply. We know that Spark community has own release date and plan. We are happy to follow Spark community. But we think it's great if community could add a sentence into the next releasenotes and claim "Spark can support Arm from this release." after we finish the test

Re: Ask for ARM CI for spark

@Sean, Yes, you are right, we don't have to create a separate release of Spark for ARM, it's enough to add a releasenote to say that Spark supports arm architecture. About the test failure, one or two tests will timeout on our poor performance arm instance sometimes, now we donate a high

Re: Ask for ARM CI for spark

I don't quite understand. You are saying tests don't pass yet, so why would anyone yet run these tests regularly? If it's because the instances aren't fast enough, use bigger instances? I don't think anyone would create a separate release of Spark for ARM, no. But why would that be necessary? On

Re: Ask for ARM CI for spark

Hi Spark team, Any ideas about the above email? Thank you. BR ZhaoBo [image: Mailtrack] Sender notified by Mailtrack

Re: Ask for ARM CI for spark

Hi all, Spark arm jobs have built for some time, and now there are two jobs[1] spark-master-test-maven-arm and spark-master-test-python-arm , we can

Re: Ask for ARM CI for spark

Hi Guys, Recently, we are trying to test pyspark on ARM, we found some issue but have no idea about them. Could you please have a look if you are free? Thanks. There are two issues: 1. The first one looks like a arm performance issue, the test job in a pyspark test doesn't fully finish when exec

Re: Ask for ARM CI for spark

Hi, Tianhua. Could you summarize the detail on the JIRA once more? It will be very helpful for the community. Also, I've been waiting on that JIRA. :) Bests, Dongjoon. On Mon, Sep 16, 2019 at 11:48 PM Tianhua huang wrote: > @shane knapp thank you very much, I opened an issue > for this

Re: Ask for ARM CI for spark

@shane knapp thank you very much, I opened an issue for this https://issues.apache.org/jira/browse/SPARK-29106, we can tall the details in it :) And we will prepare an arm instance today and will send the info to your email later. On Tue, Sep 17, 2019 at 4:40 AM Shane Knapp wrote: > @Tianhua

Re: Ask for ARM CI for spark

@Tianhua huang sure, i think we can get something sorted for the short-term. all we need is ssh access (i can provide an ssh key), and i can then have our jenkins master launch a remote worker on that instance. instance setup, etc, will be up to you. my support for the time being will be to

Re: Ask for ARM CI for spark

@Sean Owen , so sorry to reply late, we had a Mid-Autumn holiday:) If you hope to integrate ARM CI to amplab jenkins, we can offer the arm instance, and then the ARM job will run together with other x86 jobs, so maybe there is a guideline to do this? @shane knapp would you help us? On Thu, Sep

Re: Ask for ARM CI for spark

I don't know what's involved in actually accepting or operating those machines, so can't comment there, but in the meantime it's good that you are running these tests and can help report changes needed to keep it working with ARM. I would continue with that for now. On Wed, Sep 11, 2019 at 10:06

Re: Ask for ARM CI for spark

Hi Sean, Thanks for reply. And very apologize for making you confused. I know the dependencies will be downloaded from SBT or Maven. But the Spark QA job also exec "mvn clean package", why the log didn't print "downloading some jar from Maven central [1] and build very fast. Is the reason that

Re: Ask for ARM CI for spark

I'm not sure what you mean. The dependencies are downloaded by SBT and Maven like in any other project, and nothing about it is specific to Spark. The worker machines cache artifacts that are downloaded from these, but this is a function of Maven and SBT, not Spark. You may find that the initial

Re: Ask for ARM CI for spark

Hi Sean, Thanks very much for pointing out the roadmap. ;-). Then I think we will continue to focus on our test environment. For the networking problems, I mean that we can access Maven Central, and jobs cloud download the required jar package with a high network speed. What we want to know is

Re: Ask for ARM CI for spark

@Sean Owen , thanks for your reply. I agree with you basically, two points I have to say :) First, maybe I didn't express clear enough, now we download from Maven Central in our test system, seems the community jenkins ci tests never download the jar packages from maven centry repo, our question

Re: Ask for ARM CI for spark

I think the right goal is to fix the remaining issues first. If we set up CI/CD it will only tell us there are still some test failures. If it's stable, and not hard to add to the existing CI/CD, yes it could be done automatically later. You can continue to test on ARM independently for now. It

Re: Ask for ARM CI for spark

Hi all, I want to discuss spark ARM CI again, we took some tests on arm instance based on master and the job includes https://github.com/theopenlab/spark/pull/13 and k8s integration https://github.com/theopenlab/spark/pull/17/ , there are several things I want to talk about: First, about the

Re: Ask for ARM CI for spark

Hi all, About the arm test of spark, recently we found two tests failed after the commit https://github.com/apache/spark/pull/23767: ReplayListenerSuite: - ... - End-to-end replay *** FAILED *** "[driver]" did not equal "[1]" (JsonProtocolSuite.scala:622) -

Re: Ask for ARM CI for spark

Hi shane, Thanks for your reply. I will wait for you back. ;-) Thanks, Best regards ZhaoBo [image: Mailtrack] Sender notified by Mailtrack

Re: Ask for ARM CI for spark

i'm out of town, but will answer some of your questions next week. On Fri, Aug 2, 2019 at 2:39 AM bo zhaobo wrote: > > Hi Team， > > Any updates about the CI details? ;-) > > Also, I will also need your kind help about Spark QA test, could any one > can tell us how to trigger that tests? When?

Re: Ask for ARM CI for spark

Hi Team， Any updates about the CI details? ;-) Also, I will also need your kind help about Spark QA test, could any one can tell us how to trigger that tests? When? How? So far, I haven't notices how it works. Thanks Best Regards, ZhaoBo [image: Mailtrack]

Re: Ask for ARM CI for spark

Hi, team. I want to make the same test on ARM like existing CI does(x86). As building and testing the whole spark projects will cost too long time, so I plan to split them to multiple jobs to run for lower time cost. But I cannot see what the existing CI[1] have done(so many private scripts

Re: Ask for ARM CI for spark

@Sean Owen Thank you very much. And I saw your reply comment in https://issues.apache.org/jira/browse/SPARK-28519, I will test with modification and to see whether there are other similar tests fail, and will address them together in one pull request. On Sat, Jul 27, 2019 at 9:04 PM Sean Owen

Re: Ask for ARM CI for spark

Great thanks - we can take this to JIRAs now. I think it's worth changing the implementation of atanh if the test value just reflects what Spark does, and there's evidence is a little bit inaccurate. There's an equivalent formula which seems to have better accuracy. On Fri, Jul 26, 2019 at 10:02

Re: Ask for ARM CI for spark

Hi, all, FYI: >> @Yuming Wang the results in float8.sql are from PostgreSQL directly? >> Interesting if it also returns the same less accurate result, which >> might suggest it's more to do with underlying OS math libraries. You >> noted that these tests sometimes gave platform-dependent

Re: Ask for ARM CI for spark

Hi all, Thanks for your concern. Yeah, that's worth to also test in backend database. But need to note here, this issue is hit in Spark SQL, as we only test it with spark itself, not integrate other databases. Best Regards, ZhaoBo [image: Mailtrack]

Re: Ask for ARM CI for spark

Interesting. I don't think log(3) is special, it's just that some differences in how it's implemented and floating-point values on aarch64 vs x86, or in the JVM, manifest at some values like this. It's still a little surprising! BTW Wolfram Alpha suggests that the correct value is more like

Re: Ask for ARM CI for spark

Hi, all Sorry to disturb again, there are several sql tests failed on arm64 instance: - pgSQL/float8.sql *** FAILED *** Expected "0.549306144334054[9]", but got "0.549306144334054[8]" Result did not match for query #56 SELECT atanh(double('0.5')) (SQLQueryTestSuite.scala:362) -

Re: Ask for ARM CI for spark

Thanks for your reply. About the first problem we didn't find any other reason in log, just found timeout to wait the executor up, and after increase the timeout from 1 ms to 3(even 2)ms,

Re: Ask for ARM CI for spark

On Wed, Jul 17, 2019 at 6:28 AM Tianhua huang wrote: > Two failed and the reason is 'Can't find 1 executors before 1 > milliseconds elapsed', see below, then we try increase timeout the tests > passed, so wonder if we can increase the timeout? and here I have another > question about >

Re: Ask for ARM CI for spark

Hi all, We run all unit tests for spark on arm64 platform, after effort there are four tests FAILED, see https://logs.openlabtesting.org/logs/4/4/ae5ebaddd6ba6eba5a525b2bf757043ebbe78432/check/spark-build-arm64/9ecccad/job-output.txt.gz Two failed and the reason is 'Can't find 1 executors before

Re: Ask for ARM CI for spark

Hi all, I am glad to tell you there is a new progress of build/test spark on aarch64 server, the tests are running, see the build/test detail log https://logs.openlabtesting.org/logs/1/1/419fcb11764048d5a3cda186ea76dd43249e1f97/check/spark-build-arm64/75cc6f5/job-output.txt.gz and the aarch64

Re: Ask for ARM CI for spark

We are focus on the arm instance of cloud, and now I use the arm instance of vexxhost cloud to run the build job which mentioned above, the specification of the arm instance is 8VCPU and 8GB of RAM, and we can use bigger flavor to create the arm instance to run the job, if need be. On Fri, Jun

Re: Ask for ARM CI for spark

Be interesting to see how well a Pi4 works; with only 4GB of RAM you wouldn't compile with it, but you could try installing the spark jar bundle and then run against some NFS mounted disks: https://www.raspberrypi.org/magpi/raspberry-pi-4-specs-benchmarks/ ; unlikely to be fast, but it'd be an

Re: Ask for ARM CI for spark

> I think any AA64 work is going to have to define very clearly what "works" is defined as +1 It's very valuable to build a clear scope of these projects functionality for ARM platform in upstream community, it bring confidence to end user and customers when they plan to deploy these projects on

Re: Ask for ARM CI for spark

level db and native codecs are invariably a problem here, as is anything else doing misaligned IO. Protobuf has also had "issues" in the past see https://issues.apache.org/jira/browse/HADOOP-16100 I think any AA64 work is going to have to define very clearly what "works" is defined as; spark

Re: Ask for ARM CI for spark

I took the ut tests on my arm instance before and reported an issue in https://issues.apache.org/jira/browse/SPARK-27721, and seems there was no leveldbjni native package for aarch64 in leveldbjni-all.jar(or 1.8) https://mvnrepository.com/artifact/org.fusesource.leveldbjni/leveldbjni-all/1.8 , we

Re: Ask for ARM CI for spark

Can you begin by testing yourself? I think the first step is to make sure the build and tests work on ARM. If you find problems you can isolate them and try to fix them, or at least report them. It's only worth getting CI in place when we think builds will work. On Tue, Jun 25, 2019 at 9:26 PM

Re: Ask for ARM CI for spark

Thanks Shane :) This sounds good, and yes I agree that it's best to keep the test/build infrastructure in one place. If you can't find the ARM resource we are willing to support the ARM instance :) Our goal is to make more open source software to be more compatible for aarch64 platform, so let's

Re: Ask for ARM CI for spark

...or via VM as you mentioned earlier. :) shane (who will file a JIRA tomorrow) On Tue, Jun 25, 2019 at 6:44 PM shane knapp wrote: > i'd much prefer that we keep the test/build infrastructure in one place. > > we don't have ARM hardware, but there's a slim possibility i can scare > something

Re: Ask for ARM CI for spark

i'd much prefer that we keep the test/build infrastructure in one place. we don't have ARM hardware, but there's a slim possibility i can scare something up in our older research stock... another option would be to run the build in a arm-based docker container, which (according to the intarwebs)

Re: Ask for ARM CI for spark

I forked apache/spark project and propose a job( https://github.com/theopenlab/spark/pull/1) for spark building in OpenLab ARM instance, this is the first step to build spark on ARM, I can enable a periodic job for arm building for apache/spark master if you guys like. Later I will run tests for

Re: Ask for ARM CI for spark

Thanks Sean. I am very happy to hear that the community will put effort to fix the ARM-related issues. I'd be happy to help if you like. And could you give the trace link of this issue, then I can check it is fixed or not, thank you. As far as I know the old versions of spark support ARM, and now

Re: Ask for ARM CI for spark

I'd begin by reporting and fixing ARM-related issues in the build. If they're small, of course we should do them. If it requires significant modifications, we can discuss how much Spark can support ARM. I don't think it's yet necessary for the Spark project to run these CI builds until that point,

Re: Ask for ARM CI for spark

Moving to dev@ for increased visibility among the developers. On Wed, Jun 19, 2019 at 1:24 AM Tianhua huang wrote: > Thanks for your reply. > > As I said before, I met some problem of build or test for spark on aarch64 > server, so it will be better to have the ARM CI to make sure the spark is