Hello IoTivity Team! It seems that perhaps running the unit_test job in serial was too aggressive. build.iotivity.org is unresponsive, and I'm going to see if I can get a serial console on it after I compose this email. My experience with situations like this tells me that this machine will need a hard reset, and executing jobs will not be completed.
I've proposed a patch <https://gerrit.iotivity.org/gerrit/#/c/17673/> which allows up to four jobs to run concurrently. Perhaps this will allow them, for the time being, to operate concurrently without simultaneous need for an exclusive lock on the shared resource. https://gerrit.iotivity.org/gerrit/#/c/17673/ As a long-term solution, I recommend that the build script itself handle lock management with its peers so that the number of jobs executing concurrently can be set to a level which allows something like the performance we were receiving before while avoiding the inconsistent state problems concurrent execution was causing. Do you think this is a wise decision? Should I revert the previous configuration change entirely instead? Should we delegate management of the shared resource to the participating processes? I do not have enough time this evening, nor do I have the expertise yet in the build execution to make an informed patch to the build scripts. Can we meet on Monday morning (UTC -0800) on IRC and see if we can solve the problem? Thanks in advance, C.J. On Thu, Mar 2, 2017 at 5:59 PM, ??? (Uze Choi) <uzchoi at samsung.com> wrote: > Hi C.J, it?s Great! > > I also hope not to see this kind of bug and not to spend too much time for > this job execution. > > BR, Uze Choi > > *From:* C.J. Collier [mailto:cjcollier at linuxfoundation.org] > *Sent:* Friday, March 03, 2017 6:20 AM > *To:* Philippe Coval > *Cc:* iotivity-dev at lists.iotivity.org; ???(Uze Choi) > *Subject:* Re: [dev] Unit Test execution issue in Jenkins > > > > I received Phil's review and have +2'd and merged. Hopefully this will > keep the build failures from recurring. > > > > > > On Wed, Mar 1, 2017 at 1:37 PM, C.J. Collier < > cjcollier at linuxfoundation.org> wrote: > > Okay folks! Looks like I've got the configuration in place. Please > review and let me know if I can +2. > > > > https://gerrit.iotivity.org/gerrit/#/c/17589 > > > > Cheers, > > > > C.J. > > > > > > On Wed, Mar 1, 2017 at 11:01 AM, C.J. Collier < > cjcollier at linuxfoundation.org> wrote: > > Hi folks! Sorry, this list seems to be filtered. I'll add a rule to > notify me when messages come in from this list. But please be aware that I > check on the RT queue much more frequently, and that the best way to get > attention is to mail iotivity-helpdesk at rt.linuxfoundation.org. > > > > It looks like the correct solution to this problem is to throttle the > builds of this job to just one. The documentation on the JJB config option > is here: > > > > https://docs.openstack.org/infra/jenkins-job-builder/ > properties.html#properties.throttle > > > > And the source for the JJB config lives here: > > > > https://git.iotivity.org/ci-management/tree/jjb/iotivity/ > iotivity-jobs.yaml#n164 > > > > I will create a change request for your review. Phil & Uze, I will > include you as reviewers. > > > > C.J. > > > > > > On Tue, Feb 28, 2017 at 1:17 AM, Philippe Coval < > philippe.coval at osg.samsung.com> wrote: > > On 27/02/17 09:05, ??? (Uze Choi) wrote: > > Hi Phil, LF help or anyone else, > > Hi everyone, I didn't manage to reach CJ yesterday to progress on that > issue. > > > > Currently there are frequent unit test build job failure from Jenkins > verification. > > During Unit Test execution, parallel unit test executing cause racing > issue. > > yes I think it was already reported in the past (was there a bug ticket?) > > For example, A test case expect A resource server but at the same time B > test case find A resource first and A resource server close its session. > > yea, a lock is missing somewhere to prevent task reentrance > > Then A test case cannot find A resource. https://build.iotivity.org/ci/ > job/iotivity-verify-unit_tests/10126/ > > Test case should have considered this cases but every test case does not > consider it well. > > As a fallback you can retrigger build but this is not solving anything. > > > > As a solution, can we selectively disable this parallel execution for > specific job(unit test job) > > yes this is a sane workaround, > or maybe we could be better isolation using containers or vm ? > > or check what is the other issue over there? > > Is it Jenkins system operation responsibility or build script manager > responsibility or others. > > I don't have control on jenkins, but CJ has, > I will try to reach him online again and keep you updated. > > Regards > > > -- > > mailto:philippe.coval at osg.samsung.com <philippe.coval at osg.samsung.com> > gpg:0x467094BC > > https://blogs.s-osg.org/author/pcoval/ > > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.iotivity.org/pipermail/iotivity-dev/attachments/20170303/56f708f9/attachment.html>
