To all, At the risk of being a 'me too', I have to agree with Jon here and reiterate that extending the timeout may be ignoring a real problem. Even if a change set is not the primary reason that the stack is taking a long time to do something, it may be the reason that a review is needed to keep the stack in top performance.
Jon, Do you know if the timeouts can be made finer grain? Can we set timeout on individual tests? If so, we may want to move that way. As more unit tests are added the time to run would grow naturally by the number of test that are run, still prevent them from running forever and ideally tell us exactly what is taking longer than it should. Pat > -----Original Message----- > From: Jon A. Cruz [mailto:jonc at osg.samsung.com] > Sent: Wednesday, October 07, 2015 2:25 PM > To: myeong.jeong at samsung.com; Lankswert, Patrick; ???; Rahul, Rahul; > Kourt, Tim A; Morrow, Joseph L; ???; ???; iotivity-dev at lists.iotivity.org; > Kesavan, Vijay S; ????; ???; ???; ?? > Subject: Re: [dev] [Pat, Uze, Vijay] IoTivity 1.0.0 release schedule (RC3) > > The timeout was added explicitly to avoid things hanging. The proper fix would > be to get stackinit functionality back to 0.2 seconds instead of the 4.5 or > higher. > > Change 3579 also seems odd that it would cause this problem. Seems to have > just exposed some unwanted delays elsewhere. > > Also we should keep in mind that the unit tests failing in this situation is > exactly > the desired behavior. The long term solution is not to suppress nor modify the > tests, but to correct the problem the tests are flagging. > > Do keep in mind, though, that the short term work-around can be very > different. > We just don't want to lose sight of the main goal of these tests. > > > On 10/07/2015 12:31 AM, MyeongGi Jeong wrote: > > Pat, > > > > We found that some stack unit test cases failed by timed-out, > > especially StackInit test cases. > > > > Currently, each StackInit test case should be passed by 5 seconds( > > SHORT_TEST_TIMEOUT ). > > > > Before this change ( https://gerrit.iotivity.org/gerrit/#/c/3579/ ), > > each StackInit test case took about 4.5 seconds. > > > > After applying above change, StackInit test case need more time than 5 > seconds. > > > > So, StackInit test case failed and we think this failure might affect > > entire test cases running time... > > > > So we tested these test cases with 10 seconds on local linux machine, > > all test cases are passed. > > > > But the jenkins build failed with 30 seconds time-out. > > > > ( It might be related jenkins slave performance and loads ) > > > > So, we suggest followings and you can decide ( or please suggest more > > better option.. ) > > > > 1. applying above patchset even the unit test aborted by time-out. > > > > Later, the each StackInit test case timeout value and total build > > timeout criteria should be changed appropriately. > > > > 2. Altenatevely, changing SConscript to avoid unit test time out, > > > > By separating TCP option from the default build script, unit-test can be > > passed. > > > > The CoAP/TCP feature will be enabled by option setting ( like Remote > > Access ) > > > > We finished the operating test with changed SConscript and all > > functions worked correctly, > > > > and pushed it. Please review following change-set. > > > > https://gerrit.iotivity.org/gerrit/#/c/3669/ ( on 1.0.0-dev branch ) > > > > * the same patchset is also pushed to master branch > > > > ( https://gerrit.iotivity.org/gerrit/#/c/3677/ ) > > > > Which one is better ??? > > > > Thanks. > > > > Best Regards, > > > > --- > > > > MyeongGi Jeong > > > > Senior Engineer, Software Architect > > > > Software R&D Center, Samsung Electronics Co., Ltd. > > > > +82-10-3328-1130 > > > > ------- *Original Message* ------- > > > > *Sender* : MyeongGi Jeong<myeong.jeong at samsung.com> S5/Senior > > Engineer/IoT Solution Lab./Samsung Electronics > > > > *Date* : 2015-10-07 09:17 (GMT+09:00) > > > > *Title* : Re: RE: RE: [Pat, Uze, Vijay] [dev] IoTivity 1.0.0 release > > schedule (RC3) > > > > Pat. > > > > Build was successed. But Unit-test was aborted by time-out.. > > > > We looked the log from Jenkins, but we could not find out why the > > unit-test takes time so long.. > > > > And we could see the strange part from log. > > > > The sum of time duration of each test-case is not same with total time > duration... > > > > ( Time stamp shows the total time duration is correct... ) > > > > Do you know why ?? If you know any clue of this problem, please help us.. > > > > 16:44:13 [----------] 8 tests from DiscoveryRTandIF > > 16:44:13 [ RUN ] DiscoveryRTandIF.SingleItemNormal > > 16:44:13 [ OK ] DiscoveryRTandIF.SingleItemNormal (30 ms) > > 16:44:13 [ RUN ] DiscoveryRTandIF.SingleItemFrontTrim > > 16:44:14 [ OK ] DiscoveryRTandIF.SingleItemFrontTrim (7 ms) > > 16:44:14 [ RUN ] DiscoveryRTandIF.SingleItemBackTrim > > 16:44:14 [ OK ] DiscoveryRTandIF.SingleItemBackTrim (6 ms) > > 16:44:14 [ RUN ] DiscoveryRTandIF.SingleItemBothTrim > > 16:44:14 [ OK ] DiscoveryRTandIF.SingleItemBothTrim (6 ms) > > 16:44:14 [ RUN ] DiscoveryRTandIF.MultiItemsNormal > > 16:44:14 [ OK ] DiscoveryRTandIF.MultiItemsNormal (9 ms) > > 16:44:14 [ RUN ] DiscoveryRTandIF.MultiItemExtraLeadSpaces > > 16:44:16 [ OK ] DiscoveryRTandIF.MultiItemExtraLeadSpaces (7 ms) > > 16:44:25 [ RUN ] DiscoveryRTandIF.MultiItemExtraTrailSpaces > > 16:44:26 [ OK ] DiscoveryRTandIF.MultiItemExtraTrailSpaces (8 ms) > > 16:44:27 [ RUN ] DiscoveryRTandIF.MultiItemBothSpaces > > 16:44:31 [ OK ] DiscoveryRTandIF.MultiItemBothSpaces (7 ms) > > 16:44:33 [----------] 8 tests from DiscoveryRTandIF (19716 ms total) > > 16:44:33 > > 16:44:34 [----------] 8 tests from RepresentationEncodingRTandIF > > 16:44:36 [ RUN ] RepresentationEncodingRTandIF.SingleItemNormal > > 16:44:43 [ OK ] RepresentationEncodingRTandIF.SingleItemNormal (14 ms) > > 16:44:43 [ RUN ] RepresentationEncodingRTandIF.SingleItemFrontTrim > > 16:44:43 [ OK ] RepresentationEncodingRTandIF.SingleItemFrontTrim (5 > ms) > > 16:44:43 [ RUN ] RepresentationEncodingRTandIF.SingleItemBackTrim > > 16:44:48 [ OK ] RepresentationEncodingRTandIF.SingleItemBackTrim (6 > > ms) > > 16:44:55 [ RUN ] RepresentationEncodingRTandIF.SingleItemBothTrim > > 16:44:57 [ OK ] RepresentationEncodingRTandIF.SingleItemBothTrim (6 > > ms) > > 16:44:58 [ RUN ] RepresentationEncodingRTandIF.MultiItemsNormal > > 16:44:58 [ OK ] RepresentationEncodingRTandIF.MultiItemsNormal (7 ms) > > 16:44:58 [ RUN ] > RepresentationEncodingRTandIF.MultiItemExtraLeadSpaces > > 16:44:58 [ OK ] RepresentationEncodingRTandIF.MultiItemExtraLeadSpaces > (6 ms) > > 16:44:58 [ RUN ] > RepresentationEncodingRTandIF.MultiItemExtraTrailSpaces > > 16:44:59 [ OK ] > > RepresentationEncodingRTandIF.MultiItemExtraTrailSpaces > (6 ms) > > 16:44:59 [ RUN ] > RepresentationEncodingRTandIF.MultiItemExtraMiddleSpaces > > 16:45:00 [ OK ] > RepresentationEncodingRTandIF.MultiItemExtraMiddleSpaces > > (6 ms) > > 16:45:00 [----------] 8 tests from RepresentationEncodingRTandIF > > (23900 ms total) > > > > Thanks. > > > > Best Regards, > > > > --- > > > > MyeongGi Jeong > > > > Senior Engineer, Software Architect > > > > Software R&D Center, Samsung Electronics Co., Ltd. > > > > +82-10-3328-1130 > > > > ------- *Original Message* ------- > > > > *Sender* : Lankswert, Patrick<patrick.lankswert at intel.com> > > > > *Date* : 2015-10-07 00:49 (GMT+09:00) > > > > *Title* : RE: RE: [Pat, Uze, Vijay] [dev] IoTivity 1.0.0 release > > schedule (RC3) > > > > MJ, > > > > I reviewed it and it looks simple enough. I am just waiting for build > > verification. > > > > Pat > > > > *From:*MyeongGi Jeong [mailto:myeong.jeong at samsung.com] > > *Sent:* Tuesday, October 06, 2015 5:54 AM > > *To:* Lankswert, Patrick; ???; Rahul, Rahul; Kourt, Tim A; Morrow, > > Joseph L; ???; ???; iotivity-dev at lists.iotivity.org; Kesavan, Vijay S; > > ???? > > *Subject:* Re: RE: [Pat, Uze, Vijay] [dev] IoTivity 1.0.0 release > > schedule (RC3) > > > > *_Dear Pat, June._* > > > > I'd like to tell you one thing about CoAP/TCP, one more gerrit review > > should be applied into 1.0.0-dev branch. > > > > https://gerrit.iotivity.org/gerrit/#/c/3579/ > > > > One line change by adding enumeration constant. > > > > Please review and apply it after build-test. > > > > I'm sorry for late reply.... > > > > Thanks in advance. > > > > Best Regards, > > > > --- > > > > MyeongGi Jeong > > > > Senior Engineer, Software Architect > > > > Software R&D Center, Samsung Electronics Co., Ltd. > > > > +82-10-3328-1130 > > > > ------- *Original Message* ------- > > > > *Sender* : Lankswert, Patrick<patrick.lankswert at intel.com > > <mailto:patrick.lankswert at intel.com>> > > > > *Date* : 2015-10-06 11:53 (GMT+09:00) > > > > *Title* : RE: [Pat, Uze, Vijay] [dev] IoTivity 1.0.0 release schedule > > (RC3) > > > > June, > > > > I continue to receive bug fixes for master to be cherry picked onto > > 1.0.0-dev. > > The biggest delay right now is the build verification. We are backed > > up by over > > 150 jobs. > > > > Pat > > > > *From:*???(June Yong Young) [mailto:juney at samsung.com] > > *Sent:* Monday, October 05, 2015 8:48 PM > > *To:* Rahul, Rahul; Kourt, Tim A; Morrow, Joseph L; Lankswert, > > Patrick; sungkyu.ko at samsung.com <mailto:sungkyu.ko at samsung.com>; > > '???'; iotivity-dev at lists.iotivity.org > > <mailto:iotivity-dev at lists.iotivity.org>; > > Kesavan, Vijay S; ???_??; ?????_ConnectivityTG; ???? > > *Subject:* [Pat, Uze, Vijay] [dev] IoTivity 1.0.0 release schedule > > (RC3) > > > > Pat, Uze, Vijay, > > > > Could you report the issues that you have on RC3 that is supposed to > > be released by today. > > > > According to the following schedule, if we cannot release RC3 by > > today, Website open date can be delayed and we need to discuss when it will > be. > > > > > > 1.0.0 : ?OIC Spec 1.0 Compliant release? > > > > ?Type for Release : Major release, Feature Driven, PlugFest #4 > > > > ?Schedule (6 weeks ahead of QA completion) (cf. Original QA request is > > 6 weeks ahead of QA start) > > > > ?08/28,APIfreeze & Programmer?s guide freeze > > > > ?09/21, Feature complete & Review Complete and make ?1.0.0-RC1? > > release to QA > > > > ?09/25, QA 1st round complete > > > > ?09/29, Make ?1.0.0-RC2? release to QA > > > > ?10/02, QA 2nd round complete > > > > ?10/05, Make ?1.0.0-RC3? release to QA > > > > ?10/09, QA 3rd round complete > > > > ?10/12, Website Open > > > > ?10/27~29, Plug Fest #4 > > > > __ > > > > *June Yong Young* > > OIC Open Sourece WG Project Planning & Requirement TG Chair > > > > IoTivity Release Function Lead > > > > > > Samsung Electronics Co.,Ltd. > > > > Software R&D Center, IoT Solution Lab. | Web & Convergence Team > > > > Principal Engineer > > > > T: +82-31-301-6107, M: +82-10-9530-6107 > > > > E-mail :juney at samsung.com > > > > ---?? ???--- > > ??? : Lankswert, Patrick/patrick.lankswert at intel.com > > <mailto:Patrick/patrick.lankswert at intel.com> > > ???? : 2015/09/29 14:55 (GMT+09:00) > > ?? : RE: [dev] IoTivity 1.0.0 release schedule (RC2) > > > > June, > > > > There have been issues in the ZigBee, RD and other reviews. I can > > create the > > RC2 tag but it will not have the code that is currently under review. > > > > Pat > > > > *From:*???(June Yong Young) [mailto:juney at samsung.com] > > *Sent:* Friday, September 25, 2015 4:27 AM > > *To:* Lankswert, Patrick; iotivity-dev at lists.iotivity.org > > <mailto:iotivity-dev at lists.iotivity.org> > > *Subject:* RE: [dev] IoTivity 1.0.0 release schedule (RC2) > > > > Dear IoTivity members, > > > > The next 1.0.0-RC2 will be released on 9/28. > > > > Please complete RC1 QA testing by 9/25, so that we can release RC2 on time. > > > > Vijay, > > > > Could you provide the list of the features will be added on RC2 regarding > ZigBee? > > > > Pat, > > > > Korea has Thanks-giving big holiday during 9/26~9/29, so please tag > > RC2 on 9/28 after Intel feature fixes are completed. > > > > I?ll inform you just in case Samsung?s problems will still remain until > > 9/28. > > > > > > 1.0.0 : ?OIC Spec 1.0 Compliant release? > > > > ?08/28,APIfreeze & Programmer?s guide freeze > > > > ?09/21, Feature complete & Review Complete and make ?1.0.0-RC1? > > release to QA > > > > ?09/25, QA 1st round complete > > > > ?*09/28, Make ?1.0.0-RC2? release to QA* > > > > ?10/02, QA 2nd round complete > > > > ?10/05, Make ?1.0.0-RC3? release to QA > > > > ?10/09, QA 3rd round complete > > > > ?10/12, Website Open > > > > ?10/27~29, Plug Fest #4 > > > > *June Yong Young* > > OIC Open Sourece WG Project Planning & Requirement TG Chair > > > > IoTivity Release Function Lead > > > > > > Samsung Electronics Co.,Ltd. > > > > Software R&D Center, IoT Solution Lab. | Web & Convergence Team > > > > Principal Engineer > > > > T: +82-31-301-6107, M: +82-10-9530-6107 > > > > E-mail :juney at samsung.com > > > > *From:*iotivity-dev-bounces at lists.iotivity.org > > <mailto:iotivity-dev-bounces at lists.iotivity.org>[mailto:iotivity-dev-b > > ounces at lists.iotivity.org] > > *On Behalf Of *???(June Yong Young) > > *Sent:* Wednesday, September 16, 2015 2:17 AM > > *To:* iotivity-dev at lists.iotivity.org > > <mailto:iotivity-dev at lists.iotivity.org> > > *Subject:* Re: [dev] IoTivity 1.0.0 release schedule > > > > Dear IoTivity members > > > > The below is IoTivitiy 1.0.0 release schedule that was shared before. > > > > Please keep in mind that we have to complete feature implementation > > and release > > RC1 by 9/25 for the 1^st activitiy. > > > > *1.0.0 : ?OIC Spec 1.0 Compliant release?* > > > > ?Type for Release : Major release, Feature Driven, PlugFest #4 > > > > ?Schedule (6 weeks ahead of QA completion) (cf. Original QA request is > > 6 weeks ahead of QA start) > > > > ?08/28, API freeze & Programmer?s guide freeze > > > > ?09/18, Feature complete & Review Complete and make ?1.0.0-RC1? > > release to QA > > > > ?09/25, QA 1st round complete > > > > ?09/28, Make ?1.0.0-RC2? release to QA > > > > ?10/02, QA 2nd round complete > > > > ?10/05, Make ?1.0.0-RC3? release to QA > > > > ?10/09, QA 3rd round complete > > > > ?10/12, Website Open > > > > ?10/27~29, Plug Fest #4 > > > > And this is the final conclusion below at the last OIC OSWG F2F > > meeting regarding the mandatory features that have to be in 1.0.0. > > > > ?High priority features to be pushed and released ? > > > > ?Blockwise transfer in CoAP > > > > ?CoAP serialization over TCP > > > > ?TLS support for RA (*Conditional ? Depends on availability by 9/18*) > > > > ?Zigbee plugin > > > > *June Yong Young* > > OIC Open Sourece WG Project Planning & Requirement TG Chair > > > > IoTivity Release Function Lead > > > > > > Samsung Electronics Co.,Ltd. > > > > Software R&D Center, IoT Solution Lab. | Web & Convergence Team > > > > Principal Engineer > > > > T: +82-31-301-6107, M: +82-10-9530-6107 > > > > E-mail :juney at samsung.com > > > > > > > > _______________________________________________ > > iotivity-dev mailing list > > iotivity-dev at lists.iotivity.org > > https://lists.iotivity.org/mailman/listinfo/iotivity-dev > > > > -- > Jon A. Cruz - Senior Open Source Developer Samsung Open Source Group > jonc at osg.samsung.com
