Buildbot is written in python and the configuration is very flexible, you can do whatever you want, even make a coffee after a git event, the server keep the configurations and can both poll or react to events from the git server.
In my configuration there is an AWS buildbot server with a build configuration for each board I have, a laptop the "worker" that receive what to do and the boards under test connected via n USB hubthen, at the end of the build, Buildbot send a command to a "quality controller" board to begin the hardware tests on the fresh flashed boards, tests are about: -if realtime timings are consistent with the previous build -hardware behavior is still nominal -power consumption is still nominal in case of any anomalies, buildbot server get informed, the build fails and it is marked red. Then you can configure what to do next, re-attempt, drop the build, notifications, burn a rocket.. > On 20 Dec 2019, at 13:28, David Sidrane <david.sidr...@nscdg.com> wrote: > > I am not familiar with buildbot ore this sort of setup so please forgive > for some simple minded questions. > > Is this SW CI or HW CI or both? > > How does the RPi/BBB/Laptop fit into the picture. > > Any Pictures? > > David > > -----Original Message----- > From: Fabio Balzano [mailto:fa...@elfarolab.com] > Sent: Friday, December 20, 2019 5:22 AM > To: dev@nuttx.apache.org > Subject: Re: [DISCUSS - NuttX Workflow] > > 2 hours is a configured parameter, it is to allow burst of commits, it can > reduced to 0 if you need real time building, then the buildbot server can > also provision remote testing of the builds. > >> On 20 Dec 2019, at 13:09, David Sidrane <david.sidr...@nscdg.com> wrote: >> >> Hi Fabio, >> >> What are the capabilities? >> >> It this 1 RPi/BBB per board nuttx board? >> >> David >> >> -----Original Message----- >> From: Fabio Balzano [mailto:fa...@elfarolab.com] >> Sent: Friday, December 20, 2019 5:06 AM >> To: dev@nuttx.apache.org >> Subject: Re: [DISCUSS - NuttX Workflow] >> >> Hello, >> >> yes the buildbot server is down, later today I will bring it up. Yes you >> can >> do remote builds using a RPI/BBB or similars or local builds performed by >> the server itself. I can setup and maintain the server for the Nuttx >> project >> in case you think it is useful. >> >> Thank you so much >> Fabio Balzano >> >>> On 20 Dec 2019, at 13:00, Alan Carvalho de Assis <acas...@gmail.com> >>> wrote: >>> >>> Hi David, >>> >>> Sorry for scolding you in public as well, but I think we don't need to >>> find guilt. >>> >>> So, I got the impression you were doing it to promote PX4 test >>> workflow as the best solution for all the NuttX issues. >>> >>> And although 300K drones are a lot, there are many commercial products >>> using NuttX. Many Sony audio recorders, Moto Z Snaps, Thermal >>> printers, etc. Probably we have products that overcome that number. >>> >>> I think recently Fabio changed the buildbot link. BTW I just remember >>> other alternative that Sebastien and I did about 3 years ago: >>> >>> https://bitbucket.org/acassis/raspi-nuttx-farm/src/master/ >>> >>> The idea was to use low cost Raspberry PIs as a distributed build test >>> for NuttX. It worked fine! You just define a board file with the >>> configuration you want to test and it is done. >>> >>> BR, >>> >>> Alan >>> >>>> On 12/20/19, David Sidrane <davi...@apache.org> wrote: >>>> Hi Alan, >>>> >>>> Sorry if my intent was misunderstood. I am merely stating facts on were >>>> we >>>> are and how got there.I am not assigning blame. I am not forcing >>>> anything >>>> I >>>> am giving some examples of how we can make it the project complete and >>>> better. We can use all of it, some of it none of it. The is a group >>>> decision. >>>> >>>> Also Pleases do fill us in on where we can see the SW CI & HW CI you >>>> mentioned. Do you have links maybe be we can use it now? >>>> >>>> Again Sorry! >>>> >>>> David >>>> >>>>> On 2019/12/20 11:44:23, Alan Carvalho de Assis <acas...@gmail.com> >>>>> wrote: >>>>> Hi David, >>>>> >>>>>> On 12/20/19, David Sidrane <davi...@apache.org> wrote: >>>>>> Hi Nathan, >>>>>> >>>>>> On 2019/12/20 02:51:56, Nathan Hartman <hartman.nat...@gmail.com> >>>>>> wrote: >>>>>>> On Thu, Dec 19, 2019 at 6:24 PM Gregory Nutt <spudan...@gmail.com> >>>>>>> wrote: >>>>>>>>>> ] A bad build system change can cause serious problems for a lot >>>>>>>>>> of >>>>>>> people around the world. A bad change in the core OS can destroy the >>>>>>> good >>>>>>> reputation of the OS. >>>>>>>>> Why is this the case? Users should not be using unreleased code or >>>>>>>>> be >>>>>>> encouraged to use it.. If they are one solution is to make more >>>>>>> frequent >>>>>>> releases. >>>>>>>> I don't think that the number of releases is the factor. It is time >>>>>>>> in >>>>>>>> people's hand. Subtle corruption of OS real time behavior is not >>>>>>>> easily >>>>>>>> testing. You normally have to specially instrument the software >>>>>>>> and >>>>>>>> setup a special test environment perhaps with a logic analyzer to >>>>>>>> detect >>>>>>>> these errors. Errors in the core OS can persists for months and in >>>>>>>> at >>>>>>>> least one case I am aware of, years, until some sets up the correct >>>>>>>> instrumented test. >>>>>>> >>>>>>> And: >>>>>>> >>>>>>> On Thu, Dec 19, 2019 at 4:20 PM Justin Mclean >>>>>>> <jus...@classsoftware.com> >>>>>>> wrote: >>>>>>>>> ] A bad build system change can cause serious problems for a lot >>>>>>>>> of >>>>>>> people around the world. A bad change in the core OS can destroy the >>>>>>> good >>>>>>> reputation of the OS. >>>>>>>> >>>>>>>> Why is this the case? Users should not be using unreleased code or >>>>>>>> be >>>>>>> encouraged to use it.. If they are one solution is to make more >>>>>>> frequent >>>>>>> releases. >>>>>>> >>>>>>> Many users are only using released code. However, whatever is in >>>>>>> "master" >>>>>>> eventually gets released. So if problems creep in unnoticed, >>>>>>> downstream >>>>>>> users will be affected. It is only delayed. >>>>>>> >>>>>>> I can personally attest that those kinds of errors are extremely >>>>>>> difficult >>>>>>> to detect and trace. It does require a special setup with logic >>>>>>> analyzer >>>>>>> or >>>>>>> oscilloscope, and sometimes other tools, not to mention a whole setup >>>>>>> to >>>>>>> produce the right stimuli, several pieces of software that may have >>>>>>> to >>>>>>> be >>>>>>> written specifically for the test.... >>>>>>> >>>>>>> I have been wracking my brain on and off thinking about how we could >>>>>>> set >>>>>>> up >>>>>>> an automated test system to find errors related to timing etc. >>>>>>> Unfortunately unlike ordinary software for which you can write an >>>>>>> automated >>>>>>> test suite, this sort of embedded RTOS will need specialized hardware >>>>>>> to >>>>>>> conduct the tests. That's a subject for another thread and i don't >>>>>>> know >>>>>>> if >>>>>>> now is the time, but I will post my thoughts eventually. >>>>>>> >>>>>>> Nathan >>>>>> >>>>>> From the proposal >>>>>> >>>>>> "Community >>>>>> >>>>>> NuttX has a large, active community. Communication is via a Google >>>>>> group at >>>>>> https://groups.google.com/forum/#!forum/nuttx where there are 395 >>>>>> members as >>>>>> of this writing. Code is currently maintained at Bitbucket.org at >>>>>> https://bitbucket.org/nuttx/. Other communications are through >>>>>> Bitbucket >>>>>> issues and also via Slack for focused, interactive discussions." >>>>>> >>>>>> >>>>>>> Many users are only using released code. >>>>>> >>>>>> Can we ask the 395 members? >>>>>> >>>>>> I can only share my experience with NuttX since I began working on the >>>>>> project in 2012 for multiple companies. >>>>>> >>>>>> Historically (based on my time on the project) releases - were build >>>>>> tested >>>>>> - by this I mean that the configurations were updated and the thus >>>>>> created a >>>>>> set of "Build Test vectors" BTV. Given the number of permutations >>>>>> solely >>>>>> based on the load time of >>>>>> (http://nuttx.org/doku.php?id=documentation:configvars) with 95,338 >>>>>> CONFIG_* >>>>>> hits. Yes there are duplicates on the page and dependencies. This is >>>>>> just >>>>>> meant to give a number of bits.... >>>>>> >>>>>> The total space is very large >>>>>> >>>>>> The BTV space was very sparse coverage. >>>>>> >>>>>> IIRC Greg gave the build testing task a day of time. It was repeated >>>>>> after >>>>>> errors were found. I am not aware of any other testing. Are you? >>>>>> >>>>>> There were no Release Candidate (rc) nor alpha nor beta test that ran >>>>>> this >>>>>> code one real systems and very little, if any Run Test Vectors (RTV) - >>>>>> I >>>>>> have never seen a test report - has anyone? >>>>>> >>>>>> One way to look at this is Sporadic Integration. (SI) with limited BTV >>>>>> and >>>>>> minimal RTV. Total Test Vector Coverage TTVC = BTV + RTV; The ROI of >>>>>> way >>>>>> of working, from a reliability perspective was and is very small. >>>>>> >>>>>> A herculean effort Greg's part with little return: We released code >>>>>> with >>>>>> many significant and critical errors in it. See the ReleaseNotes and >>>>>> the >>>>>> commit log. >>>>>> >>>>>> Over the years Greg referred to TRUNK (yes it was on SVN) and master >>>>>> as >>>>>> his >>>>>> "own sandbox" stating is should not be considered stable or >>>>>> build-able. >>>>>> This >>>>>> is evident in the commit log. >>>>> >>>>> Please stop focusing on the people (Greg) and let talk about how the >>>>> workflow. >>>>> We are here to discuss how we can improve the process, we are not >>>>> talking about throw away NuttX Build System and move to PX4. >>>>> >>>>> You are picturing something that is not true. >>>>> >>>>> We have issues, as FreeRTOS, MBEB and Zephyr also have. But it is not >>>>> Greg or the Build System guilt. >>>>> >>>>> Please, stop! It is disgusting! >>>>> >>>>>> I have personally never used a release from a tarball. Given the above >>>>>> why >>>>>> would I? It is less stable then master at TC = N >>>>>> (https://www.electronics-tutorials.ws/rc/rc_1.html) where N Is some >>>>>> number >>>>>> of days after a release. - unfortunately based on the current >>>>>> practices >>>>>> (a >>>>>> very unprofessional workflow) N is also dictated by when apps and >>>>>> nuttx >>>>>> actually building for a given target's set of BTV. >>>>> >>>>> It is not "unprofessional" it was what we could do based or our >>>>> hardware limitations. >>>>> >>>>>> With the tools and resources that exist in our work today, Quite >>>>>> frankly: >>>>>> This unacceptable and is an embarrassment. >>>>> >>>>> Oh my Gosh! Please don't do it. >>>>> >>>>> >>>>>> I suspect this is why there is a Tizen. The modern era - gets it. >>>>>> (Disclaimer I am an old dog - I am learning to get it) >>>>> >>>>> Tizen exists because companies want to have control. >>>>> This is the same logic why Redhat and others maintain their own Linux >>>>> kernel by themselves. >>>>> >>>>>> --- Disclaimer --- >>>>>> >>>>>> In the following, I'm am not bragging about PX4 or selling tools, I am >>>>>> merely trying to share our experiences for the betterment of NuttX. >>>>>> >>>>>> From what I understand PX4 has the most instances of NuttX running on >>>>>> real >>>>>> HW in the world. Over 300K. (I welcome other users to share their >>>>>> numbers) >>>>>> >>>>>> PX4's Total TTVC is still limited, but much, much greater than NuttX. >>>>>> >>>>>> We use Continuous integration (CI) on Nuttx on PX4 on every commit on >>>>>> PRs. >>>>>> >>>>>> C/C++ CI / build (push) Successful in 3m >>>>>> Compile MacOS Pending — This commit is being built >>>>>> Compile All Boards — This commit looks good >>>>>> Hardware Test — This commit looks good >>>>>> SITL Tests — This commit looks good >>>>>> SITL Tests (code coverage) — This commit looks good >>>>>> ci/circleci — Your tests passed on CircleCI! >>>>>> continuous-integration/appveyor/pr — AppVeyor build succeeded >>>>>> continuous-integration/jenkins/pr-head — This commit looks good >>>>>> >>>>>> >>>>>> We run tests on HW. >>>>>> >>>>>> http://ci.px4.io:8080/blue/organizations/jenkins/PX4_misc%2FFirmware-hardware/detail/pr-mag-str-preflt/1/pipeline >>>>>> >>>>>> I say limited because of the set of arch we use and the way we >>>>>> configure >>>>>> the >>>>>> OS. >>>>>> >>>>>> I believe this to be true of all users. >>>>>> >>>>>> The benefit of a community is that the sum of all TTVC that finds the >>>>>> problems and fix them. >>>>>> >>>>>> Why not maximize TTVC - if it will have a huge ROI and it is free: >>>>>> >>>>>> PX4 will contribute all that we have. We just need to build temporally >>>>>> consistent build. Yeah he is on the submodule thing AGAIN :) >>>>> >>>>> Just to make the history short: we already have solutions for SW and HW >>>>> CI. >>>>> >>>>> Besides the buildbot (https://buildbot.net) that was implemented and >>>>> tested by Fabio Balzano, Xiaomi also has a build test for NuttX. >>>>> >>>>> At end of the day, it is not only Greg testing the system, we all are >>>>> testing it as well. >>>>> >>>>> Don't try to push PX4 down your throat, it will not work this way. >>>>> Let's keep the Apache way, it is a democracy! >>>>> >>>>> BR, >>>>> >>>>> Alan >>>>
