Hi Илья, On Fri, Jan 17, 2020 at 5:43 PM Илья Шипицин <[email protected]> wrote:
> > > пт, 17 янв. 2020 г. в 19:33, Martin Grigorov <[email protected]>: > >> >> >> On Fri, Jan 17, 2020 at 4:13 PM Martin Grigorov < >> [email protected]> wrote: >> >>> Hi all, >>> >>> Today's build consistently fails on my ARM64 VM: >>> >>> ########################## Starting vtest ########################## >>> Testing with haproxy version: 2.2-dev0-70c5b0-123 >>> # top TEST reg-tests/mcli/mcli_start_progs.vtc FAILED (3.004) exit=2 >>> 1 tests failed, 0 tests skipped, 34 tests passed >>> ########################## Gathering results ########################## >>> ###### Test case: reg-tests/mcli/mcli_start_progs.vtc ###### >>> ## test results in: >>> "/tmp/haregtests-2020-01-17_14-01-45.SGkYcJ/vtc.12807.6adfff44" >>> ---- h1 haproxy h1 PID file check failed: >>> ---- h1 Bad exit status: 0x0100 exit 0x1 signal 0 core 0 >>> Makefile:964: recipe for target 'reg-tests' failed >>> make: *** [reg-tests] Error 1 >>> >> >> git bisect blaims this commit: >> >> 25b569302167e71b32e569a2366027e8e320e80a is the first bad commit >> commit 25b569302167e71b32e569a2366027e8e320e80a >> Author: William Lallemand <[email protected]> >> Date: Tue Jan 14 15:38:43 2020 +0100 >> >> REGTEST: mcli/mcli_start_progs: start 2 programs >> >> This regtest tests the issue #446 by starting 2 programs and checking >> if >> they exist in the "show proc" of the master CLI. >> >> Should be backported as far as 2.0. >> >> >> https://travis-ci.com/haproxy/haproxy is green >> https://cirrus-ci.com/github/haproxy/haproxy is green >> and what is even more interesting is that >> https://travis-ci.org/martin-g/haproxy/builds (my fork with enabled >> ARM64 testing on TravisCI) also just passed (after few failures due to >> timing issues (I guess) >> > > > timing out might happen because of > https://github.com/haproxy/haproxy/commit/ac8147446c7a3d1aa607042bc782095b03bc8dc4 > your fork is 16 commits behind current master. try to rebase to master > Thanks! I've updated HAProxy on my ARM64 VM but didn't update my fork at GitHub so it was behind. I've just did it and the build again passed successfully: https://travis-ci.org/martin-g/haproxy/builds/638568217 > > >> >> >>> Regards, >>> Martin >>> >>> On Fri, Jan 17, 2020 at 9:22 AM Илья Шипицин <[email protected]> >>> wrote: >>> >>>> привет! >>>> >>>> пт, 17 янв. 2020 г. в 11:42, Martin Grigorov <[email protected] >>>> >: >>>> >>>>> Привет Илья, >>>>> >>>>> On Thu, Jan 16, 2020 at 10:37 AM Илья Шипицин <[email protected]> >>>>> wrote: >>>>> >>>>>> >>>>>> >>>>>> чт, 16 янв. 2020 г. в 13:26, Martin Grigorov < >>>>>> [email protected]>: >>>>>> >>>>>>> Hi Илья, >>>>>>> >>>>>>> On Thu, Jan 16, 2020 at 10:19 AM Илья Шипицин <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> >>>>>>>> Hello, Martin! >>>>>>>> >>>>>>>> btw, just curious, how is Apache Foundation (or you personally) >>>>>>>> interested in all that ? >>>>>>>> please do not blame me, I really like to know. >>>>>>>> >>>>>>> >>>>>> ok, so you work in some company that is interested in haproxy on >>>>>> ARM64. >>>>>> you do not want to tell it's name, at least is it legal ? is it >>>>>> related to some government ? >>>>>> if "no" and "no", I guess most people won't ask any more questions :) >>>>>> >>>>> >>>>> It is legal and I do not work for a government of any country! >>>>> >>>>> >>>>>> >>>>>> personally, I do not work at Haproxy Inc, I use haproxy, sometimes I >>>>>> contribute to it. >>>>>> Please do not consider me as an "official representative". >>>>>> >>>>>> >>>>>> I'm interested in testing haproxy on ARM64, I planned to do so. I can >>>>>> priorierize it according to your interest to it. >>>>>> and yes, I can accept hardware donation (even considering I'm not >>>>>> part of Haproxy Inc). >>>>>> >>>>>> also, from my point of view, what would be really useful in your case >>>>>> is testing not just official reg-tests, but also >>>>>> your configs. reg-tests cover only partially. If you enable clang >>>>>> asan in your own workload there's a chance to catch >>>>>> something interesting (or, to make sure your own workload is ok). I >>>>>> can help with that as well. >>>>>> >>>>> >>>>> Thanks for the offer! >>>>> I've discussed it with my managers. >>>>> Our offer is to donate a VM that could be used as an official CI agent >>>>> for the HAProxy project long term. >>>>> >>>> >>>> I'd split this into short term and long term approach. >>>> >>>> if you need to start any time soon, I'd focus on your own workload >>>> testing first. I would build stand which emulates >>>> your production workload, compile haproxy using clang address sanitizer >>>> and give it a try (functional testing, load testing, ...) >>>> >>>> I can help with that. >>>> >>>> As for long term solution, currently haproxy simply cannot attach any >>>> dedicated build agent to their CI. travis-ci does not allow >>>> attaching dedicated agents. And haproxy team is very conservative in >>>> adding new CI servers. >>>> >>>> I think, I will add arm64 (most probably openssl-1.1.1 only for now) >>>> soon. Also, I'm going to investigate your libressl failures. >>>> >>>> so, dedicated vm definetly will help in troubleshooting issues, for >>>> manual builds. It would save bunch of time. I do not mind if you will >>>> add my ssh to that vm. >>>> >>>> >>>> also, I requested access to Linaro. >>>> >>>> >>>>> >>>>> You can use Linaro for short term testing though. >>>>> https://www.linaro.cloud/ >>>>> Here you can request free VM for short periods: >>>>> >>>>> https://servicedesk.linaro.org/servicedesk/customer/portal/19/create/265 >>>>> P.S. Linaro is not my employer! >>>>> >>>>> >>>>> Regards, >>>>> Martin >>>>> >>>>> >>>>>> >>>>>>> >>>>>>>> >>>>>>> @apache.org is just one of my several emails. And it is the default >>>>>>> one in my email client. >>>>>>> ASF is not related anyhow to my participation here. >>>>>>> If I used my work email then it might look like some kind of >>>>>>> advertisement. I'd like to avoid that! >>>>>>> Next time I will use my @gmail.com one, as more neutral. Actually >>>>>>> I've used the GMail one when registering to this mailing list, so >>>>>>> probably >>>>>>> the post from @apache has been moderated. I'll be more careful next >>>>>>> time! >>>>>>> >>>>>>> Thanks, Илья! >>>>>>> >>>>>>> Regards, >>>>>>> Martin >>>>>>> >>>>>>> >>>>>>>> >>>>>>>> чт, 16 янв. 2020 г. в 12:32, Martin Grigorov <[email protected] >>>>>>>> >: >>>>>>>> >>>>>>>>> Hello HAProxy developers, >>>>>>>>> >>>>>>>>> At work we are going to use more and more ARM64 based servers and >>>>>>>>> HAProxy is one of the main products we rely on. >>>>>>>>> So I went checking whether HAProxy project has a CI environment >>>>>>>>> for testing on ARM architecture. >>>>>>>>> >>>>>>>> >>>>>>>> we are looking towards >>>>>>>> https://docs.travis-ci.com/user/multi-cpu-architectures >>>>>>>> >>>>>>>> >>>>>>>>> I've found this recent discussion: >>>>>>>>> https://www.mail-archive.com/[email protected]/msg35302.html >>>>>>>>> (I didn't find a way how to continue on the same mail thread, so I'm >>>>>>>>> starting a new one. Apologies for that!). >>>>>>>>> >>>>>>>> >>>>>>>> I played with arm64 for a while, the issue I encountered is >>>>>>>> travis-ci cache key, i.e. we cache openssl builds between our builds. >>>>>>>> so travis used the same cache key for both amd64 and arm64 builds >>>>>>>> (this might have changed recently, I did not check yet) >>>>>>>> >>>>>>>> arm64 is in my queue (as well as recent s390x arch from travis), >>>>>>>> hope to get back to it within month or so. >>>>>>>> >>>>>>>> >>>>>>>>> From this discussion and from >>>>>>>>> https://github.com/haproxy/haproxy/blob/master/.travis.yml I >>>>>>>>> understand that there is no public CI in use (i.e. TravisCI or >>>>>>>>> CirrusCI) >>>>>>>>> but some of the developers run some tests locally regularly. >>>>>>>>> >>>>>>>> >>>>>>>> it is not completely true. >>>>>>>> there's public CI. we do not use github PR machinery, so sometimes >>>>>>>> tests fail after push to master branch. it is considered as ok, >>>>>>>> failures >>>>>>>> are fixed pretty fast. >>>>>>>> for example, see >>>>>>>> https://www.mail-archive.com/[email protected]/msg35910.html >>>>>>>> it was just perfect, failure detected using CI and fixed within few >>>>>>>> days. no customers affected. >>>>>>>> >>>>>>>> >>>>>>>>> I've forked the project and tested on TravisCI ( >>>>>>>>> https://travis-ci.org/martin-g/haproxy/builds) but unfortunately >>>>>>>>> the builds were not very stable: >>>>>>>>> 1) some tests fail sometimes. I guess it is because of some timing >>>>>>>>> issues >>>>>>>>> For example: >>>>>>>>> - https://travis-ci.org/martin-g/haproxy/jobs/636745241 >>>>>>>>> - https://travis-ci.org/martin-g/haproxy/jobs/636750676 >>>>>>>>> - https://travis-ci.org/martin-g/haproxy/jobs/636763346 >>>>>>>>> >>>>>>>> >>>>>>>> that's very interesting. I'll have a look. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> 2) There was some weird issue on testing with LibreSSL >>>>>>>>> The output redirect at >>>>>>>>> https://github.com/haproxy/haproxy/blob/bb9da0b8e23c46caab34fe6005b66fa8065fe3ac/.travis.yml#L96 >>>>>>>>> for >>>>>>>>> some reason got stuck the build. I've removed temporarily the output >>>>>>>>> redirects and then it passed. So, it looks like some issue with >>>>>>>>> TravisCI >>>>>>>>> environment. >>>>>>>>> >>>>>>>> >>>>>>>> arm64 is slower, I guess we should add "*travis*_*wait* *30" *to >>>>>>>> build-ssl.sh script >>>>>>>> thank for the hint >>>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>>> In addition I've run the build and tests on one of our machines >>>>>>>>> and all was OK! >>>>>>>>> >>>>>>>>> My question to you is: Are you happy with your current way of >>>>>>>>> testing ARM architectures or you want to add more ? >>>>>>>>> Here are some options: >>>>>>>>> 1) enable TravisCI >>>>>>>>> >>>>>>>> >>>>>>>> already done >>>>>>>> >>>>>>>> https://travis-ci.com/haproxy/haproxy >>>>>>>> >>>>>>>> >>>>>>>>> 2) my company is willing to donate an ARM64 based VM, if you are >>>>>>>>> interested. >>>>>>>>> >>>>>>>> >>>>>>>> I do not work at Haproxy Inc :) >>>>>>>> Willy ? >>>>>>>> >>>>>>>> >>>>>>>>> You will have a SSH access and a user with sudo permissions to >>>>>>>>> install anything that is needed. >>>>>>>>> The spec is aarch64 8 cores CPU 2GHz (Kunpeng), 16GB RAM, 500G >>>>>>>>> disk space and 5M network bandwidth. The OS could be any of CentOS >>>>>>>>> 7.4/7.5/7.6, EulerOS 2.8, Fedora 29, OpenSuse 15.0 and Ubuntu >>>>>>>>> 16.04/18.04. >>>>>>>>> >>>>>>>>> In both cases it will be ARM64. From the earlier mail discussion I >>>>>>>>> understand you would prefer ARM32. >>>>>>>>> >>>>>>>> >>>>>>>> as for myself, I prefer both arm64 and arm32. >>>>>>>> however, both AMD64 and ARM64 are the same 64 bits. both of them >>>>>>>> are little-endian. but you mentioned at least 3 builds with ARM64 >>>>>>>> failing >>>>>>>> (we have those tests passing on AMD64)! >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>>> Kind regards, >>>>>>>>> Martin >>>>>>>>> >>>>>>>>
