Hi Mat,
Thank you for your questioning even you already are aware of them. CTT GAP: Simple Server Application + IoTivity Framework ?? CTT This will only cover the part of core specification (discovery, messaging, common resource operation, security logic) from the OCF specification perspective. Vertical resource and some of complex operation in the core specification such as collection are out of scope. However, Mitch may update more clear definition if needed. BR, Uze Choi From: Mats Wichmann [mailto:[email protected]] Sent: Tuesday, November 22, 2016 10:13 AM To: Gregg Reynolds; Mitch Kettrick Cc: Jacek Hryszkiewicz; cert_wg at openconnectivity.org; Heldt-Sheller, Nathan; ??? (Uze Choi); ???; iotivity-dev at lists.iotivity.org; Richard Bardini Subject: Re: [cert_wg] Re: [dev] CTT Gap status sharing request You just have to ask the easy questions, do you? The CTT is actually two different things, the one referred to here is the OCF's Compliance Test Tool. There's also a branch in iotivity called ctt which has a test tool (possibly a bit out of date now) with kind of similar aims. One of the things IoTivity wants to be able to do is be compliant with OCF specs, so that someone implementing a device using IoTivity software has a decent chance of achieving OCF certification. Of course IoTivity itself can't be certified since certification is against a specific device implementation, not a source code project. But still, if the IoTivity project does something that is not going to pass when an implementation uses it, that's a "gap". On 11/21/2016 05:11 PM, Gregg Reynolds wrote: since this is after all the iotivity list, and not the OCF list, if someone could take the time to explain what all this stuff about CTT, Gap, etc. is about I would be grateful. ggg On Nov 21, 2016 6:05 PM, "Mitch Kettrick" <cpm at openconnectivity.org> wrote: Hi Uze, I?ll let Nathan and/or Dongik provide specific details about any Security-related issues if needed. Here is my understanding of where we are right now: Items to be fixed on CTT v1.4 ? CT1.7.7.1 updated to append ACE with random UUID rather than adding a ?*? ACE for /oic/p which already exists ? Update CT1.7.8.1 for Clients - CTT sends a GET to /oic/d which has an ACE installed ? Update CT1.7.8.2 to allow any 4.xx error code (CTT now allows 4.01 Unauthorized only) ? Update to align with Security CR46 which allows only doxm and pstat to be accessed over CoAP rather than all SVRs. There is already an IoTivity patch for this (#14137) Items still to be fixed on IoTivity v1.2-rel ? IPv4/IPv6 changes as discussed by Thiago ? NON block-wise transfer (current analysis indicates this is an IoTivity issue ? email attached) ? Add Policy ?p? to any Collection as discussed by Joey from Intel (email attached) Items to be added to the IoTivity Reference Device or still to be tested: ? Collections ? Run all sever-role test cases against a device that is primarily in the Client role Core schema file changes (Richard) ? Update changes to OIC Link schema file as agreed at the F2F ? Update OIC Link schema file to address issues found by Comarch (pull request 30) Thanks, Mitch From: ??? (Uze Choi) [mailto:[email protected]] Sent: Sunday, November 20, 2016 11:32 PM To: ???; Heldt-Sheller, Nathan; Mitch Kettrick Cc: iotivity-dev at lists.iotivity.org; cert_wg at openconnectivity.org Subject: CTT Gap status sharing request Hi Nathan/Dongik/Mitch As a main IoTivity/CTT developer, I?d like to ask you to share IoTivity 1.2.1 and CTT1.4 Gap. Items To be fixed on CTT 1.4 - aaa - bbb Items To be fixed on IoTivity 1.2-rel - ccc - ddd Interoperability Test status - the lastest testing event: IOTIVITY 1.2-rel (changeID: xxx), CTT 1.3.kk - fail 1: aaa/ccc - fail 2: bbb/ddd Currently there are several pieces of mail, but too fragmented and detail. Following format or equivalent simple format will be helpful I think. BR, Uze Choi ---------- Forwarded message ---------- From: Mitch Kettrick <[email protected]> To: <sungkyu.ko at samsung.com>, "'???'" <i.mushfiq at samsung.com> Cc: "'Heldt-Sheller, Nathan'" <nathan.heldt-sheller at intel.com>, "'???'" <js126.lee at samsung.com>, "'Jacek Hryszkiewicz'" <jacek.hryszkiewicz at comarch.com>, "'Agis, Ed'" <ed.agis at intel.com>, "'???'" <uzchoi at samsung.com>, "'???'" <myeong.jeong at samsung.com>, "'???'" <dongik.lee at samsung.com>, "'????'" <dwarka.dayama at samsung.com>, "'???'" <moonki1.hong at samsung.com>, "'???'" <moonyoung.choi at samsung.com>, "'???'" <jminl.choi at samsung.com>, "'???'" <soonhwang.choi at samsung.com>, "'???'" <k1358.kim at samsung.com>, "'Michael Koster'" <michael.koster at smartthings.com>, "'Brian Scriber'" <b.scriber at cablelabs.com>, "'Marek Szkowron'" <marek.szkowron at comarch.com>, "'Jason Smith'" <lab_mgr at openconnectivity.org>, "'Richard Bardini'" <richard.a.bardini at intel.com> Date: Thu, 17 Nov 2016 08:33:58 -0800 Subject: Potential IoTivity /CTT bug: BWT for NON request messages Hi Sungkyu, I can open the .zip file you sent. Thanks! >From the email, there were two issues on CT1.2.2: CT1.2.2 Request with if=oic.if.baseline returned resource Need new logs from Mushfiqul CT1.2.2 Piggyback ACK and CON CTT issue. The Piggyback failure was a CTT issue and that was fixed. This is related to the oic.if.baseline issue and we never did receive new logs as far as I can remember. Regarding the CT1.2.2 oic.if.baseline issue, here is the response for a NONconfirmable request message from the CTT v1.3.16 logs: 66.388s INFO: ======= Testing resource /Vendor/AirConditioner/TimerClock with query parameter if=oic.if.baseline - NON confirmable ======= 66.388s INFO: 66.388s INFO: Sending unicast NON RETRIEVE request with: /Vendor/AirConditioner/TimerClock?if=oic.if.baseline 66.389s VERBOSE: -> 107.109.211.84:49163->107.109.211.223:56175 NON-GET ID=5612, Token=33AC1867, Options=[URI-Port=56175, URI-Path=Vendor, AirConditioner, TimerClock, URI-Query=if=oic.if.baseline, Accept=application/cbor], Secured=true 66.427s INFO: Prompting "Waiting for CoAP response... " 76.331s VERBOSE: <- 107.109.211.223:56175->107.109.211.84:49163 NON-2.05 Content ID=5612, Token=33AC1867, Options=[URI-Path=Vendor, AirConditioner, TimerClock, Content-Type=application/cbor], Secured=true 76.331s VERBOSE: <- {"x.com.vendor.timer.hour":10,"x.com.vendor.timer.minute":30,"x.com.vendor.timer.reset":false,"x.com.vendor.timer.second":30} Looking at the same Resource when using a CONfirmable request messags from the CTT v1.3.16 logs: 22.560s INFO: ======= Testing resource /Vendor/AirConditioner/TimerClock with query parameter if=oic.if.baseline - CONfirmable ======= 22.560s INFO: 22.560s INFO: Sending unicast CON RETRIEVE request with /Vendor/AirConditioner/TimerClock?if=oic.if.baseline 22.560s VERBOSE: -> 107.109.211.84:49163->107.109.211.223:56175 CON-GET ID=5572, Token=33AC1841, Options=[URI-Port=56175, URI-Path=Vendor, AirConditioner, TimerClock, URI-Query=if=oic.if.baseline, Accept=application/cbor], Secured=true 22.570s VERBOSE: <- 107.109.211.223:56175->107.109.211.84:49163 ACK-Empty Message ID=5572, Token=, Options=[], Secured=true 22.600s INFO: Prompting "Waiting for CoAP response... " 32.570s VERBOSE: <- 107.109.211.223:56175->107.109.211.84:49163 CON-2.05 Content ID=5572, Token=33AC1841, Options=[URI-Path=Vendor, AirConditioner, TimerClock, Content-Type=application/cbor, Block2=0+ (1024B/block [6])], Secured=true 32.570s VERBOSE: -> 107.109.211.84:49163->107.109.211.223:56175 ACK-Empty Message ID=5572, Token=, Options=[], Secured=true 32.570s VERBOSE: -> 107.109.211.84:49163->107.109.211.223:56175 CON-GET ID=5573, Token=33AC1841, Options=[URI-Port=56175, URI-Path=Vendor, AirConditioner, TimerClock, URI-Query=if=oic.if.baseline, Accept=application/cbor, Block2=1 (1024B/block [6])], Secured=true 32.570s VERBOSE: <- 107.109.211.223:56175->107.109.211.84:49163 ACK-2.05 Content ID=5573, Token=33AC1841, Options=[URI-Path=Vendor, AirConditioner, TimerClock, Content-Type=application/cbor, Block2=1 (1024B/block [6])], Secured=true 32.570s VERBOSE: <- {"rt":["x.com.vendor.timer"],"if":["oic.if.a","oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline", "oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline", "oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline", "oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline", "oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline", "oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline", "oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline"],"x.com.vendor.timer.hour":10,"x.com.vendor.timer.minute":30,"x.com.vendor.timer.reset":false,"x.com.vendor.timer.second":30} 32.570s DEBUG: AgentBase multiple action complete Why are the responses different between NON and CON? They should be the exact same. I went back to the results from CTT v1.3.14 and this is the response for a NONconfirmable request message: 47.388s INFO: ======= Testing resource /Vendor/AirConditioner/TimerClock - NON confirmable ======= 47.388s INFO: 47.388s INFO: Sending unicast CON RETRIEVE request with /Vendor/AirConditioner/TimerClock?if=oic.if.baseline 47.389s VERBOSE: -> 107.109.211.84:49163->107.109.211.223:38033 CON-GET ID=3623, Token=1FD2D505, Options=[URI-Port=38033, URI-Path=Vendor, AirConditioner, TimerClock, URI-Query=if=oic.if.baseline, Accept=application/cbor], Secured=true 47.399s VERBOSE: <- 107.109.211.223:38033->107.109.211.84:49163 ACK-Empty Message ID=3623, Token=, Options=[], Secured=true 47.468s INFO: Prompting "Waiting for CoAP response... " 57.402s VERBOSE: <- 107.109.211.223:38033->107.109.211.84:49163 CON-2.05 Content ID=3623, Token=1FD2D505, Options=[URI-Path=Vendor, AirConditioner, TimerClock, Content-Type=application/cbor, Block2=0+ (1024B/block [6])], Secured=true 57.402s VERBOSE: -> 107.109.211.84:49163->107.109.211.223:38033 ACK-Empty Message ID=3623, Token=, Options=[], Secured=true 57.402s VERBOSE: -> 107.109.211.84:49163->107.109.211.223:38033 CON-GET ID=3624, Token=1FD2D505, Options=[URI-Port=38033, URI-Path=Vendor, AirConditioner, TimerClock, URI-Query=if=oic.if.baseline, Accept=application/cbor, Block2=1 (1024B/block [6])], Secured=true 57.405s VERBOSE: <- 107.109.211.223:38033->107.109.211.84:49163 ACK-2.05 Content ID=3624, Token=1FD2D505, Options=[URI-Path=Vendor, AirConditioner, TimerClock, Content-Type=application/cbor, Block2=1 (1024B/block [6])], Secured=true 57.405s VERBOSE: <- {"rt":["x.com.vendor.timer"],"if":["oic.if.a","oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline", "oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline", "oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline", "oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline", "oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline", "oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline"], "x.com.vendor.timer.hour":10,"x.com.vendor.timer.minute":30,"x.com.vendor.timer.reset":false,"x.com.vendor.timer.second":30} 57.417s DEBUG: AgentBase multiple action complete Notice that this time, the device responded the same as the CON request message. Also from the email you attached: TC Fail Log Reproducibility Comment CT1.2.2 83.780s ERROR: Request with if=oic.if.baseline returned resource {"x.com.vendor.timer.hour":10,"x.com.vendor.timer.minute":30,"x.com.vendor.timer.reset":false,"x.com.vendor.timer.second":30} which doesn't contain rt... 83.780s ERROR: Request with if=oic.if.baseline returned resource {"x.com.vendor.timer.hour":10,"x.com.vendor.timer.minute":30,"x.com.vendor.timer.reset":false,"x.com.vendor.timer.second":30} which doesn't contain if... 83.780s ERROR: CT1.2.2_Check_8: Resource has not all mandatory properties or interfaces or does not comply to schema 50% CTT Issue It looks like the issue only happens about 50% of the time. The comment says ?CTT Issue? but I don?t think we ever did a thorough log analysis to back up that comment. >From what I can see, this seems like an intermittent BWT issue with the >reference device / IoTivity. The NON response does not always include any >indication that BWT will be used. Can you guys look into this on the IoTivity side? Thanks, Mitch From: ??? [mailto:[email protected]] Sent: Wednesday, November 16, 2016 6:58 PM To: Mitch Kettrick; ??? Cc: 'Heldt-Sheller, Nathan'; ???; 'Jacek Hryszkiewicz'; 'Agis, Ed'; ???; ???; ???; ????; ???; ???; ???; ???; ???; 'Michael Koster'; 'Brian Scriber'; 'Marek Szkowron'; 'Jason Smith'; Richard Bardini Subject: RE: RE: RE: RE: RE: TC result of IoTivity 1.2.0 vs CTT 1.3.16 Execution Summary: Dear Mitch, I can see the logs. I zipped it again and attached. Regarding CT1.2.2 errors, we saw the same errors previous CTT. It was occured when CTT couldn't handle BWT + Slow Response correctly. I attached previous mail about it. @Antu, Cloud you explain further about CT1.2.2 erros? Also please test Iotivity against CTT 1.3.17. (Same iotivity ver. as yesterday) Thanks & BR, Sungkyu Ko. --------- Original Message --------- Sender : Mitch Kettrick <cpm at openconnectivity.org> Date : 2016-11-17 00:51 (GMT+9) Title : RE: RE: RE: RE: TC result of IoTivity 1.2.0 vs CTT 1.3.16 Execution Summary: Hi Mushfiqul, Thanks again for running the tests. For some reason, I can?t open any of the files within the .zip file. Can anyone else open them or is it just me? Regarding the issues you?ve reported below: CT1.2.2 ? As mentioned last time, this looks to be an issue with the IoTivity reference device. Sungkyu, can you look into this one? CT1.7.9.1 ? This is an issue with the OIC Link schema file. We?re waiting for Richard to upload Michael?s schema file change so that it can be pulled into a new CTT release and this error should go away. CT1.7.7.1/CT1.7.8.1/CT1.7.8.2 ? The test should run and those warnings should go away with CTT v1.3.17. Since I can?t open the logs, I can?t do any detailed analysis so I?ll have to wait until either someone can resend these results or I get the next batch. Speaking of the next batch, Comarch just released CTT v1.3.17: OCF CTT Release Version 1.3.17 (2016-11-16) Notes: 1. CTT 1.3.17 has been released to OCF members as an interim release version of the tool and can be used ONLY for testing purposes. 2. Implemented features - Server tests added to client?s profile - CR - CT1.7.8.x test cases apply to Clients This should clear up some issues but it may introduce more when we run the Server role test cases against a Client role (primary) device. Thanks, Mitch From: Muhammad Mushfiqul Islam [mailto:[email protected]] Sent: Wednesday, November 16, 2016 3:36 AM To: Sung-Kyu Ko; Mitch Kettrick Cc: Heldt-Sheller, Nathan; Jongsung Lee; 'Jacek Hryszkiewicz'; Agis, Ed; Uze Choi; MyeongGi Jeong; Lee, Dongik; DWARKAPRASAD DAYAMA; Peter Moonki Hong; MoonYoung Choi; jongmin choi; SOON HWANG CHOI; Gihong Kim; 'Michael Koster'; Brian Scriber; 'Marek Szkowron'; Jason Smith Subject: RE: RE: RE: RE: TC result of IoTivity 1.2.0 vs CTT 1.3.16 Execution Summary: Hello Below is the execution summary of CTT v1.3.16 against IoTivity 1.2.0-RC4(with 6 patches) : Fail: 1.2.2, 1.7.9.1 (client and server) NA: 2.2.4 (usual), and 1.7.7.1, 1.7.8.1, 1.7.8.2 (client and server, although whole procedure is completed, instead of showing pass, N/A is showing) CT1.7.8.3 is now passing for both server & client (without patch https://gerrit.iotivity.org/gerrit/#/c/14137/ ). Here is Details: TC Error Log Status Comment CT1.2.2 76.358s 12:40:34 ERROR:?Request with if=oic.if.baseline returned resource {"x.com.vendor.timer.hour":10,"x.com.vendor.timer.minute":30,"x.com.vendor.timer.reset":false,"x.com.vendor.timer.second":30} which doesn't contain rt... 76.358s 12:40:34 ERROR:?Request with if=oic.if.baseline returned resource {"x.com.vendor.timer.hour":10,"x.com.vendor.timer.minute":30,"x.com.vendor.timer.reset":false,"x.com.vendor.timer.second":30} which doesn't contain if... 76.359s 12:40:34 ERROR:?CT1.2.2_Check_8: Resource has not all mandatory properties or interfaces or does not comply to schema 129.151s 12:41:26 ERROR:?Test case result: FAILED Old CTT Issue Returned This issue was found on CTT version 1.3 & before. After that, it was resolved. CT1.7.9.1 1.939s ERROR: '' is not of type 'array' Failed validating 'type' in schema['allOf'][1]['properties']['aclist']['properties']['aces']['items']['allOf'][0]['properties']['resources']['items']['properties']['rel']: ... 1.944s ERROR: '' is not of type 'array' Failed validating 'type' in schema['allOf'][1]['properties']['aclist']['properties']['aces']['items']['allOf'][0]['properties']['resources']['items']['properties']['rel']: ... 1.949s ERROR: '' is not of type 'array' Failed validating 'type' in schema['allOf'][1]['properties']['aclist']['properties']['aces']['items']['allOf'][0]['properties']['resources']['items']['properties']['rel']: ... 1.949s ERROR: CT1.7.9.1_Check_2: IUT response messages do not follow JSON schema 'oic.r.acl.json' Old Issue Schema issue (Probably) CT1.7.7.1, CT1.7.8.1, CT1.7.8.2 1.764s INFO: ======= End of onboarding ======= 1.764s INFO: 1.764s INFO: Trying to open secure connection... 1.764s DEBUG: DTLS PSK Client: Opening connection for client uuid: 11111111-2222-3333-4444-555555555555 1.764s DEBUG: DTLS PSK Client: Initializing session. Advertised cipher suites: C037, C0A8 1.769s DEBUG: DTLS PSK Client: Server selected cipher suite C037 1.779s DEBUG: DTLS PSK Client: Client Random d8175c32de5449c9e324de22878bbe11b2933768c172a56ec868c308138cbc48 1.779s DEBUG: DTLS PSK Client: Pre-Master Secret 0020da1fb83994299168ea901ce92d66a9de7b6c5782978ab7e692a06ce4daf9516e0010c8c1cfbb5330ae13525296e925930b77 1.779s DEBUG: DTLS PSK Client: Master Secret 322bc6436404f14e40b68a369b24908c7ace8d551c445e79d8a21c2aed8e95c3938965f6c925bb92fc7d0b363b5fa69d 1.779s DEBUG: DTLS PSK Client: Key Block 89fec874d4bc85880ea9eea97199479edd9a19a40fe2bb2be02502b22bcea31210316e762d01db93f792d3f4206a2119a0fb9d8a10d621c3964e3494e8e013d7fd60bff07eb4d55ffa4daa9947e0dc9456141d6a7e6705463edc88caf62d58f9 1.784s DEBUG: DTLS PSK Client: Handshake completed 1.784s INFO: Sending unicast CON RETRIEVE request with /oic/res 1.784s VERBOSE: -> 107.109.211.84:49239->107.109.211.223:36300 CON-GET ID=10943, Token=461B5124, Options=[URI-Port=36300, URI-Path=oic, res, Accept=application/cbor], Secured=true 1.789s VERBOSE: <- 107.109.211.223:36300->107.109.211.84:49239 ACK-2.05 Content ID=10943, Token=461B5124, Options=[URI-Path=oic, res, Content-Type=application/cbor], Secured=true 1.789s VERBOSE: <- [{"di":"6a757374-776f-726b-4465-765575696430","links":[{"href":"/oic/sec/doxm","rt":["oic.r.doxm"],"if":["oic.if.baseline"],"p":{"bm":1,"sec":true,"port":36300}},{"href":"/oic/sec/pstat","rt":["oic.r.pstat"],"if":["oic.if.baseline"],"p":{"bm":1,"sec":true,"port":36300}},{"href":"/oic/d","rt":["oic.wk.d","oic.d.airconditioner"],"if":["oic.if.baseline","oic.if.r"],"p":{"bm":1,"sec":false}},{"href":"/oic/p","rt":["oic.wk.p"],"if":["oic.if.baseline","oic.if.r"],"p":{"bm":1,"sec":false}},{"href":"/BinarySwitchResURI","rt":["oic.r.switch.binary"],"if":["oic.if.baseline","oic.if.a"],"p":{"bm":3,"sec":false}},{"href":"/TemperatureResURI","rt":["oic.r.temperature"],"if":["oic.if.baseline","oic.if.a"],"p":{"bm":3,"sec":false}},{"href":"/AirFlowResURI","rt":["oic.r.airflow"],"if":["oic.if.baseline","oic.if.a"],"p":{"bm":3,"sec":false}},{"href":"/Vendor/AirConditioner/TimerClock","rt":["x.com.vendor.timer"],"if":["oic.if.baseline","oic.if.a"],"p":{"bm":1,"sec":false}},{"href":"/Vendor/AirConditioner/Swinger","rt":["x.com.vendor.swing"],"if":["oic.if.baseline","oic.if.a"],"p":{"bm":1,"sec":false}}]}] 1.794s INFO: Prompting "Waiting for CoAP response... " 1.819s DEBUG: AgentBase multiple action complete 1.819s DEBUG: Starting verification with ID:"CT_Precondition"... 1.819s INFO: CT_Precondition: IUT respond to /oic/res (CON-GET ID:10943) 1.819s DEBUG: Verification with ID:"CT_Precondition" ended with result: PASSED 1.819s WARNING: Did not found any secured resource which href does not start with '/oic' 1.819s DEBUG: Action "PYTHON_SCRIPT" result: NA 1.819s DEBUG: Step "python" result: NA 1.819s DEBUG: DTLS PSK Client: Alert close_notify(0) was raised 2.784s WARNING: Test case result: NA Old Issue After onboarding, TC result is shown as N/A - Thanks & Regards, Mushfiqul Islam Antu --------- Original Message --------- Sender : Jongsung Lee <js126.lee at samsung.com> S5/Senior Engineer/Security Lab./Samsung Electronics Date : 2016-11-16 15:36 (GMT+6) Title : RE: RE: RE: TC result of IoTivity 1.2.0 vs CTT 1.3.15Execution Summary: To : Sung-Kyu Ko<sungkyu.ko at samsung.com>, null<nathan.heldt-sheller at intel.com>, null<cpm at openconnectivity.org>, Muhammad Mushfiqul Islam<i.mushfiq at samsung.com>, Jongsung Lee<js126.lee at samsung.com> CC : null<jacek.hryszkiewicz at comarch.com>, null<ed.agis at intel.com>, Uze Choi<uzchoi at samsung.com>, MyeongGi Jeong<myeong.jeong at samsung.com>, Lee, Dongik<dongik.lee at samsung.com>, DWARKAPRASAD DAYAMA<dwarka.dayama at samsung.com>, Peter Moonki Hong<moonki1.hong at samsung.com>, MoonYoung Choi<moonyoung.choi at samsung.com>, jongmin choi<jminl.choi at samsung.com>, SOON HWANG CHOI<soonhwang.choi at samsung.com>, Gihong Kim<k1358.kim at samsung.com>, null<michael.koster at smartthings.com>, null<b.scriber at cablelabs.com>, null<marek.szkowron at comarch.com>, null<lab_mgr at openconnectivity.org> Dear Mitch and Nathan, Regarding CT1.7.8.3, I think that "CoAP testing for SVRs" should be removed according to CR44 (SVR CoAP Access Allowed). Recently, https://gerrit.iotivity.org/gerrit/#/c/14137/ has been applied, so Iotivity's device is operationg as below (due to security vulnuerability, acl and cred need to be accessible only via coaps). ? doxm pstat acl cred CoAP access access deny deny <CR44> However, unlike other Resources, the SVRs may still be accessed via CoAP, provided there is a wildcard ?*? ACE naming the SVR. So, CT1.7.8.3 is failed, if https://gerrit.iotivity.org/gerrit/#/c/14137/ is included. I attached CTT log for CT1.7.8.1. 56.199s 17:24:01 ERROR:?CT1.7.8.3_Precondition: For message: ID=60353 Token=73E7BB1F expected response code 2.05 Content (69) - received 4.03 Forbidden (131). 56.199s 17:24:01 DEBUG:?Verification with ID:"CT1.7.8.3_Precondition" ended with result: FAILED 56.199s 17:24:01 ERROR:?Response for GET '/oic/sec/cred?if=oic.if.baseline' is not valid response. Could not process procedure for this resource 56.486s 17:24:01 ERROR:?CT1.7.8.3_Precondition: For message: ID=60354 Token=73E7BB20 expected response code 2.05 Content (69) - received 4.03 Forbidden (131). 56.486s 17:24:01 DEBUG:?Verification with ID:"CT1.7.8.3_Precondition" ended with result: FAILED 56.486s 17:24:01 ERROR:?Response for GET '/oic/sec/acl?if=oic.if.baseline' is not valid response. Could not process procedure for this resource Thank you, Jongsung Lee. --------- Original Message --------- Sender : ??? <sungkyu.ko at samsung.com> S5(??)/??/SQE Lab(S/W??)/???? Date : 2016-11-16 13:36 (GMT+9) Title : RE: RE: TC result of IoTivity 1.2.0 vs CTT 1.3.15Execution Summary: Dear Nathan, We are using - 1.2-rel branch 1.2.0-RC4 tag and 6 patches. - https://gerrit.iotivity.org/gerrit/#/c/13625/ - https://gerrit.iotivity.org/gerrit/#/c/13777/ - https://gerrit.iotivity.org/gerrit/#/c/13993/ - https://gerrit.iotivity.org/gerrit/#/c/14053/ - https://gerrit.iotivity.org/gerrit/#/c/14069/ - https://gerrit.iotivity.org/gerrit/#/c/14211 Thanks & BR, Sungkyu Ko. --------- Original Message --------- Sender : Heldt-Sheller, Nathan <nathan.heldt-sheller at intel.com> Date : 2016-11-16 12:12 (GMT+9) Title : RE: TC result of IoTivity 1.2.0 vs CTT 1.3.15Execution Summary: Just to be clear: what version (tag + patches, or just a git version number) are you using to test IUT Simulator against CTT 1.3.16? I see some merges happening against 1.2-rel and want to make sure we are all using the same set of patches to test! Thanks, Nathan From: ??? [mailto:[email protected]] Sent: Tuesday, November 15, 2016 6:52 PM To: Mitch Kettrick <cpm at openconnectivity.org>; ??? <i.mushfiq at samsung.com>; ??? <js126.lee at samsung.com> Cc: 'Jacek Hryszkiewicz' <jacek.hryszkiewicz at comarch.com>; Agis, Ed <Ed.Agis at intel.com>; Heldt-Sheller, Nathan <nathan.heldt-sheller at intel.com>; ??? <uzchoi at samsung.com>; ??? <myeong.jeong at samsung.com>; ??? <dongik.lee at samsung.com>; ???? <dwarka.dayama at samsung.com>; ??? <moonki1.hong at samsung.com>; ??? <moonyoung.choi at samsung.com>; ??? <jminl.choi at samsung.com>; ??? <soonhwang.choi at samsung.com>; ??? <k1358.kim at samsung.com>; 'Michael Koster' <michael.koster at smartthings.com>; Brian Scriber <b.scriber at cablelabs.com>; 'Marek Szkowron' <marek.szkowron at comarch.com>; Jason Smith <lab_mgr at openconnectivity.org> Subject: RE: TC result of IoTivity 1.2.0 vs CTT 1.3.15Execution Summary: Dear Mitch, As usual, Jongmin will chekck security TC and Antu will run whole test caes. :) @Antu, Please test IoTivity 1.2.0 RC4 + patches against CTT 1.3.16 and share the result. I shared new CTT to you. Thanks & BR, Sungkyu Ko. --------- Original Message --------- Sender : Mitch Kettrick <cpm at openconnectivity.org> Date : 2016-11-16 02:42 (GMT+9) Title : TC result of IoTivity 1.2.0 vs CTT 1.3.15Execution Summary: Hi, CTT v1.3.16 has been uploaded to Google Drive. Can someone please run this and share the results? I?m hopeful that this version solves all of the issues other than the failure for CT1.7.9.1 (the oic-link schema file has been updated but not released yet). OCF CTT Release Version 1.3.16 (2016-11-15) Notes: 1. CTT 1.3.16 has been released to OCF members as an interim release version of the tool and can be used ONLY for testing purposes. 2. Implemented features - create CT1.7.8.3 check_3e and prepare workflow - Schema updates - Client rediscovering mechanism - Security test case v8 3. Fixed issues: - #897 (Client Testing: CTT not able to simulate oic.d.tv). Thanks, Mitch From: Mitch Kettrick [mailto:[email protected]] Sent: Monday, November 14, 2016 4:46 PM To: 'i.mushfiq at samsung.com'; 'Jongsung Lee' Cc: 'Jacek Hryszkiewicz'; 'Sung-Kyu Ko'; 'Ed Agis'; 'Heldt-Sheller, Nathan'; 'Uze Choi'; 'MyeongGi Jeong'; 'Lee, Dongik'; 'DWARKAPRASAD DAYAMA'; 'Peter Moonki Hong'; 'MoonYoung Choi'; 'jongmin choi'; 'SOON HWANG CHOI'; 'Gihong Kim'; 'Michael Koster'; Brian Scriber; 'Marek Szkowron'; Jason Smith (lab_mgr at openconnectivity.org) Subject: RE: RE: RE: RE: RE: TC result of IoTivity 1.2.0 vs CTT 1.3.15Execution Summary: Hi Mushfiqul, Thank you so much for running these tests!! Sorry it took me so long to review the results. Last week was busy and chose not to work this week end. ;) It looks like you stopped CT1.7.8.3 before it finished. I really need to see the results all the way until the end to make sure that everything is working as specified. Maybe start the test and go to lunch! J I?m working with Comarch to shorten it a bit but my top priority is to get everything to pass so I need to see the test case run all the way until the end. Once things are passing I?ll put more focus on shortening the test case where possible. I won?t do my full analysis until I have the complete test run because it takes a whole day to do it and I don?t want to spend that much time only to end up with 75% of the answers we need. Can you run all of the test cases in one test campaign? That is what is required for certification so I?d like to simulate that as much as possible. We?ve found that sometimes new failures are introduced when running test cases all together. One way to keep the file size down is not to send the pcap files. Now that the CTT provides full logging information, Wireshark isn?t really needed any more. Comarch will be releasing a new version of the CTT that will resolve most of the errors seen. In my fairly quick analysis: ? All of the issues we see in CT1.7.8.3 should go away. ? The issue in CT1.2.2 is an issue with the reference device. For some reason, the /Vendor/AirConditioner/TimerClock has no ?rt? or ?if? Properties. ? The issue in CT1.7.9.1 was caused by a change in the schema file for the OIC Link related to the ?rel? Property having been changed from a sting to an array of strings. We spoke about it at the ATG meeting last week and Michael will be fixing it. ? I need to review CT1.7.7.1, CT1.7.8.1 and CT1.7.8.2. Since Client?s don?t have any vertical Resources, these test cases may not apply but I need to look at the details. Let?s wait until we have a new release from Comarch. Should be about 16 hours from the time of this message. Then we can see if we cleaned up most of the errors. Thanks again!! Mitch From: Muhammad Mushfiqul Islam [mailto:[email protected]] Sent: Friday, November 11, 2016 2:20 AM To: 'Mitch Kettrick'; Jongsung Lee Cc: Jacek Hryszkiewicz; Sung-Kyu Ko; 'Ed Agis'; 'Heldt-Sheller, Nathan'; Uze Choi; MyeongGi Jeong; Lee, Dongik; DWARKAPRASAD DAYAMA; Peter Moonki Hong; MoonYoung Choi; jongmin choi; SOON HWANG CHOI; Gihong Kim Subject: RE: RE: RE: RE: RE: TC result of IoTivity 1.2.0 vs CTT 1.3.15Execution Summary: Hello, Below is the execution summary of CTT v1.3.15 against IoTIvity : Fail: 1.2.2, 1.7.9.1 (client and server) NA: 2.2.4 (usual), and 1.7.7.1, 1.7.8.1, 1.7.8.2 (client and server, although whole procedure is completed, instead of showing pass, N/A is showing) No End: 1.7.8.3 (client and server), CT1.1.1, CT1.2.3, CT1.26 are passing after applying the patch from Mr. Lee ( <https://gerrit.iotivity.org/gerrit/#/c/14211> https://gerrit.iotivity.org/gerrit/#/c/14211). Here is Details: TC Error Log Status Comment CT1.2.2 76.633s ERROR: Request with if=oic.if.baseline returned resource {"x.com.vendor.timer.hour":10,"x.com.vendor.timer.minute":30,"x.com.vendor.timer.reset":false,"x.com.vendor.timer.second":30} which doesn't contain rt... 76.633s ERROR: Request with if=oic.if.baseline returned resource {"x.com.vendor.timer.hour":10,"x.com.vendor.timer.minute":30,"x.com.vendor.timer.reset":false,"x.com.vendor.timer.second":30} which doesn't contain if... 76.633s ERROR: CT1.2.2_Check_8: Resource has not all mandatory properties or interfaces or does not comply to schema Old CTT Issue Returned This issue was found on CTT version 1.3 & before. After that, it was resolved. CT1.7.9.1 1.939s ERROR: '' is not of type 'array' Failed validating 'type' in schema['allOf'][1]['properties']['aclist']['properties']['aces']['items']['allOf'][0]['properties']['resources']['items']['properties']['rel']: ... 1.944s ERROR: '' is not of type 'array' Failed validating 'type' in schema['allOf'][1]['properties']['aclist']['properties']['aces']['items']['allOf'][0]['properties']['resources']['items']['properties']['rel']: ... 1.949s ERROR: '' is not of type 'array' Failed validating 'type' in schema['allOf'][1]['properties']['aclist']['properties']['aces']['items']['allOf'][0]['properties']['resources']['items']['properties']['rel']: ... 1.949s ERROR: CT1.7.9.1_Check_2: IUT response messages do not follow JSON schema 'oic.r.acl.json' New Issue Schema issue (Probably) CT1.7.8.3 44.597s ERROR: CT1.7.8.3_Precondition: For message: ID=13440 Token=20BA2B5A expected response code 2.05 Content (69) - received 4.03 Forbidden (131). 44.597s ERROR: Response for GET '/Vendor/AirConditioner/TimerClock?if=oic.if.baseline' is not valid response. Could not process procedure for this resource 44.747s ERROR: CT1.7.8.3_Precondition: For message: ID=13441 Token=20BA2B5B expected response code 2.05 Content (69) - received 4.03 Forbidden (131). 44.747s ERROR: Response for GET '/Vendor/AirConditioner/TimerClock?if=oic.if.baseline' is not valid response. Could not process procedure for this resource 47.601s ERROR: CT1.7.8.3_Precondition: For message: ID=13464 Token=20BA2B72 expected response code 2.05 Content (69) - received 4.03 Forbidden (131). 47.601s ERROR: Response for GET '/BinarySwitchResURI?if=oic.if.baseline' is not valid response. Could not process procedure for this resource 47.821s ERROR: CT1.7.8.3_Precondition: For message: ID=13465 Token=20BA2B73 expected response code 2.05 Content (69) - received 4.03 Forbidden (131). 47.821s ERROR: Response for GET '/BinarySwitchResURI?if=oic.if.baseline' is not valid response. Could not process procedure for this resource 48.099s ERROR: CT1.7.8.3_Precondition: For message: ID=13466 Token=20BA2B74 expected response code 2.05 Content (69) - received 4.03 Forbidden (131). 48.099s ERROR: Response for GET '/TemperatureResURI?if=oic.if.baseline' is not valid response. Could not process procedure for this resource 48.346s ERROR: CT1.7.8.3_Precondition: For message: ID=13467 Token=20BA2B75 expected response code 2.05 Content (69) - received 4.03 Forbidden (131). 48.346s ERROR: Response for GET '/TemperatureResURI?if=oic.if.baseline' is not valid response. Could not process procedure for this resource 48.641s ERROR: CT1.7.8.3_Precondition: For message: ID=13468 Token=20BA2B76 expected response code 2.05 Content (69) - received 4.03 Forbidden (131). 48.641s ERROR: Response for GET '/AirFlowResURI?if=oic.if.baseline' is not valid response. Could not process procedure for this resource 48.831s ERROR: CT1.7.8.3_Precondition: For message: ID=13469 Token=20BA2B77 expected response code 2.05 Content (69) - received 4.03 Forbidden (131). 48.831s ERROR: Response for GET '/AirFlowResURI?if=oic.if.baseline' is not valid response. Could not process procedure for this resource 119.549s ERROR: CT1.7.8.3_Precondition: For message: ID=13803 Token=20BA2BED expected response code 2.05 Content (69) - received 4.03 Forbidden (131). 119.549s ERROR: Response for GET '/Vendor/AirConditioner/TimerClock?if=oic.if.baseline' is not valid response. Could not process procedure for this resource 119.879s ERROR: CT1.7.8.3_Precondition: For message: ID=13804 Token=20BA2BEE expected response code 2.05 Content (69) - received 4.03 Forbidden (131). 119.879s ERROR: Response for GET '/Vendor/AirConditioner/TimerClock?if=oic.if.baseline' is not valid response. Could not process procedure for this resource 125.086s ERROR: CT1.7.8.3_Precondition: For message: ID=13827 Token=20BA2C05 expected response code 2.05 Content (69) - received 4.03 Forbidden (131). 125.086s ERROR: Response for GET '/BinarySwitchResURI?if=oic.if.baseline' is not valid response. Could not process procedure for this resource 125.448s ERROR: CT1.7.8.3_Precondition: For message: ID=13828 Token=20BA2C06 expected response code 2.05 Content (69) - received 4.03 Forbidden (131). 125.448s ERROR: Response for GET '/BinarySwitchResURI?if=oic.if.baseline' is not valid response. Could not process procedure for this resource 125.903s ERROR: CT1.7.8.3_Precondition: For message: ID=13829 Token=20BA2C07 expected response code 2.05 Content (69) - received 4.03 Forbidden (131). 125.903s ERROR: Response for GET '/TemperatureResURI?if=oic.if.baseline' is not valid response. Could not process procedure for this resource 126.258s ERROR: CT1.7.8.3_Precondition: For message: ID=13830 Token=20BA2C08 expected response code 2.05 Content (69) - received 4.03 Forbidden (131). 126.258s ERROR: Response for GET '/TemperatureResURI?if=oic.if.baseline' is not valid response. Could not process procedure for this resource 126.688s ERROR: CT1.7.8.3_Precondition: For message: ID=13831 Token=20BA2C09 expected response code 2.05 Content (69) - received 4.03 Forbidden (131). 126.688s ERROR: Response for GET '/AirFlowResURI?if=oic.if.baseline' is not valid response. Could not process procedure for this resource 127.043s ERROR: CT1.7.8.3_Precondition: For message: ID=13832 Token=20BA2C0A expected response code 2.05 Content (69) - received 4.03 Forbidden (131). 127.043s ERROR: Response for GET '/AirFlowResURI?if=oic.if.baseline' is not valid response. Could not process procedure for this resource 227.969s ERROR: CT1.7.8.3_Precondition: For message: ID=14157 Token=20BA2C8C expected response code 2.05 Content (69) - received 4.03 Forbidden (131). 227.969s ERROR: Response for GET '/Vendor/AirConditioner/TimerClock?if=oic.if.baseline' is not valid response. Could not process procedure for this resource 228.464s ERROR: CT1.7.8.3_Precondition: For message: ID=14158 Token=20BA2C8D expected response code 2.05 Content (69) - received 4.03 Forbidden (131). 228.464s ERROR: Response for GET '/Vendor/AirConditioner/TimerClock?if=oic.if.baseline' is not valid response. Could not process procedure for this resource 236.031s ERROR: CT1.7.8.3_Precondit ... -------------- next part -------------- HTML ?????? ??????????????... URL: <http://lists.iotivity.org/pipermail/iotivity-dev/attachments/20161122/23ec6b86/attachment.html>
