since this is after all the iotivity list, and not the OCF list, if someone could take the time to explain what all this stuff about CTT, Gap, etc. is about I would be grateful.
ggg On Nov 21, 2016 6:05 PM, "Mitch Kettrick" <cpm at openconnectivity.org> wrote: > Hi Uze, > > > > I?ll let Nathan and/or Dongik provide specific details about any > Security-related issues if needed. Here is my understanding of where we > are right now: > > > > Items to be fixed on CTT v1.4 > > ? CT1.7.7.1 updated to append ACE with random UUID rather than > adding a ?*? ACE for /oic/p which already exists > > ? Update CT1.7.8.1 for Clients - CTT sends a GET to /oic/d which > has an ACE installed > > ? Update CT1.7.8.2 to allow any 4.xx error code (CTT now allows > 4.01 Unauthorized only) > > ? Update to align with Security CR46 which allows only doxm and > pstat to be accessed over CoAP rather than all SVRs. There is already an > IoTivity patch for this (#14137) > > > > Items still to be fixed on IoTivity v1.2-rel > > ? IPv4/IPv6 changes as discussed by Thiago > > ? NON block-wise transfer (current analysis indicates this is an > IoTivity issue ? email attached) > > ? Add Policy ?p? to any Collection as discussed by Joey from > Intel (email attached) > > > > Items to be added to the IoTivity Reference Device or still to be tested: > > ? Collections > > ? Run all sever-role test cases against a device that is > primarily in the Client role > > > > Core schema file changes (Richard) > > ? Update changes to OIC Link schema file as agreed at the F2F > > ? Update OIC Link schema file to address issues found by Comarch > (pull request 30) > > > > Thanks, > > Mitch > > > > *From:* ??? (Uze Choi) [mailto:uzchoi at samsung.com <uzchoi at samsung.com>] > *Sent:* Sunday, November 20, 2016 11:32 PM > *To:* ???; Heldt-Sheller, Nathan; Mitch Kettrick > *Cc:* iotivity-dev at lists.iotivity.org; cert_wg at openconnectivity.org > *Subject:* CTT Gap status sharing request > > > > Hi Nathan/Dongik/Mitch > > > > As a main IoTivity/CTT developer, I?d like to ask you to share IoTivity > 1.2.1 and CTT1.4 Gap. > > > > Items To be fixed on CTT 1.4 > > - aaa > > - bbb > > Items To be fixed on IoTivity 1.2-rel > > - ccc > > - ddd > > Interoperability Test status > > - the lastest testing event: IOTIVITY 1.2-rel (changeID: xxx), CTT > 1.3.kk > > - fail 1: aaa/ccc > > - fail 2: bbb/ddd > > > > Currently there are several pieces of mail, but too fragmented and detail. > > Following format or equivalent simple format will be helpful I think. > > > > BR, Uze Choi > > > ---------- Forwarded message ---------- > From: Mitch Kettrick <cpm at openconnectivity.org> > To: <sungkyu.ko at samsung.com>, "'???'" <i.mushfiq at samsung.com> > Cc: "'Heldt-Sheller, Nathan'" <nathan.heldt-sheller at intel.com>, "'???'" < > js126.lee at samsung.com>, "'Jacek Hryszkiewicz'" < > jacek.hryszkiewicz at comarch.com>, "'Agis, Ed'" <ed.agis at intel.com>, > "'???'" <uzchoi at samsung.com>, "'???'" <myeong.jeong at samsung.com>, > "'???'" > <dongik.lee at samsung.com>, "'????'" <dwarka.dayama at samsung.com>, "'???'" > < > moonki1.hong at samsung.com>, "'???'" <moonyoung.choi at samsung.com>, > "'???'" < > jminl.choi at samsung.com>, "'???'" <soonhwang.choi at samsung.com>, "'???'" < > k1358.kim at samsung.com>, "'Michael Koster'" <michael.koster at > smartthings.com>, > "'Brian Scriber'" <b.scriber at cablelabs.com>, "'Marek Szkowron'" < > marek.szkowron at comarch.com>, "'Jason Smith'" <lab_mgr at > openconnectivity.org>, > "'Richard Bardini'" <richard.a.bardini at intel.com> > Date: Thu, 17 Nov 2016 08:33:58 -0800 > Subject: Potential IoTivity /CTT bug: BWT for NON request messages > > Hi Sungkyu, > > > > I can open the .zip file you sent. Thanks! > > > > From the email, there were two issues on CT1.2.2: > > > > *CT1.2.2 Request with if=oic.if.baseline returned resource* > > Need new logs from Mushfiqul > > > > *CT1.2.2 Piggyback ACK and CON* > > CTT issue. > > > > The Piggyback failure was a CTT issue and that was fixed. This is related > to the oic.if.baseline issue and we never did receive new logs as far as I > can remember. > > > > Regarding the CT1.2.2 oic.if.baseline issue, here is the response for a > NONconfirmable request message from the CTT v1.3.16 logs: > > > > 66.388s INFO: ======= Testing resource /Vendor/AirConditioner/TimerClock > with query parameter if=oic.if.baseline - NON confirmable ======= > > 66.388s INFO: > > 66.388s INFO: Sending unicast NON RETRIEVE request with: > /Vendor/AirConditioner/TimerClock?if=oic.if.baseline > > 66.389s VERBOSE: -> 107.109.211.84:49163->107.109.211.223:56175 NON-GET > ID=5612, Token=33AC1867, Options=[URI-Port=56175, URI-Path=Vendor, > AirConditioner, TimerClock, URI-Query=if=oic.if.baseline, > Accept=application/cbor], Secured=true > > 66.427s INFO: Prompting "Waiting for CoAP response... " > > 76.331s VERBOSE: <- 107.109.211.223:56175->107.109.211.84:49163 > NON-2.05 Content ID=5612, Token=33AC1867, Options=[URI-Path=Vendor, > AirConditioner, TimerClock, Content-Type=application/cbor], Secured=true > > 76.331s VERBOSE: <- {"x.com.vendor.timer.hour":10, > "x.com.vendor.timer.minute":30,"x.com.vendor.timer.reset": > false,"x.com.vendor.timer.second":30} > > > > Looking at the same Resource when using a CONfirmable request messags from > the CTT v1.3.16 logs: > > > > 22.560s INFO: ======= Testing resource /Vendor/AirConditioner/TimerClock > with query parameter if=oic.if.baseline - CONfirmable ======= > > 22.560s INFO: > > 22.560s INFO: Sending unicast CON RETRIEVE request with > /Vendor/AirConditioner/TimerClock?if=oic.if.baseline > > 22.560s VERBOSE: -> 107.109.211.84:49163->107.109.211.223:56175 CON-GET > ID=5572, Token=33AC1841, Options=[URI-Port=56175, URI-Path=Vendor, > AirConditioner, TimerClock, URI-Query=if=oic.if.baseline, > Accept=application/cbor], Secured=true > > 22.570s VERBOSE: <- 107.109.211.223:56175->107.109.211.84:49163 > ACK-Empty Message ID=5572, Token=, Options=[], Secured=true > > 22.600s INFO: Prompting "Waiting for CoAP response... " > > 32.570s VERBOSE: <- 107.109.211.223:56175->107.109.211.84:49163 > CON-2.05 Content ID=5572, Token=33AC1841, Options=[URI-Path=Vendor, > AirConditioner, TimerClock, Content-Type=application/cbor, Block2=0+ > (1024B/block [6])], Secured=true > > 32.570s VERBOSE: -> 107.109.211.84:49163->107.109.211.223:56175 > ACK-Empty Message ID=5572, Token=, Options=[], Secured=true > > 32.570s VERBOSE: -> 107.109.211.84:49163->107.109.211.223:56175 CON-GET > ID=5573, Token=33AC1841, Options=[URI-Port=56175, URI-Path=Vendor, > AirConditioner, TimerClock, URI-Query=if=oic.if.baseline, > Accept=application/cbor, Block2=1 (1024B/block [6])], Secured=true > > 32.570s VERBOSE: <- 107.109.211.223:56175->107.109.211.84:49163 > ACK-2.05 Content ID=5573, Token=33AC1841, Options=[URI-Path=Vendor, > AirConditioner, TimerClock, Content-Type=application/cbor, Block2=1 > (1024B/block [6])], Secured=true > > 32.570s VERBOSE: <- {"rt":["x.com.vendor.timer"]," > if":["oic.if.a","oic.if.baseline","oic.if.baseline"," > oic.if.baseline","oic.if.baseline","oic.if.baseline"," > oic.if.baseline","oic.if.baseline","oic.if.baseline"," > oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline", > "oic.if.baseline","oic.if.baseline","oic.if.baseline"," > oic.if.baseline","oic.if.baseline","oic.if.baseline"," > oic.if.baseline","oic.if.baseline","oic.if.baseline"," > oic.if.baseline","oic.if.baseline","oic.if.baseline"," > oic.if.baseline","oic.if.baseline", "oic.if.baseline","oic.if. > baseline","oic.if.baseline","oic.if.baseline","oic.if. > baseline","oic.if.baseline","oic.if.baseline","oic.if. > baseline","oic.if.baseline","oic.if.baseline","oic.if. > baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline", > "oic.if.baseline","oic.if.baseline","oic.if.baseline"," > oic.if.baseline","oic.if.baseline","oic.if.baseline"," > oic.if.baseline","oic.if.baseline","oic.if.baseline"," > oic.if.baseline","oic.if.baseline","oic.if.baseline"," > oic.if.baseline","oic.if.baseline", "oic.if.baseline","oic.if. > baseline","oic.if.baseline","oic.if.baseline","oic.if. > baseline","oic.if.baseline","oic.if.baseline","oic.if. > baseline","oic.if.baseline","oic.if.baseline","oic.if. > baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline", > "oic.if.baseline","oic.if.baseline","oic.if.baseline"," > oic.if.baseline","oic.if.baseline","oic.if.baseline"," > oic.if.baseline","oic.if.baseline","oic.if.baseline"," > oic.if.baseline","oic.if.baseline","oic.if.baseline"," > oic.if.baseline","oic.if.baseline", "oic.if.baseline","oic.if. > baseline","oic.if.baseline","oic.if.baseline","oic.if. > baseline","oic.if.baseline"],"x.com.vendor.timer.hour":10," > x.com.vendor.timer.minute":30,"x.com.vendor.timer.reset": > false,"x.com.vendor.timer.second":30} > > 32.570s DEBUG: AgentBase multiple action complete > > > > Why are the responses different between NON and CON? They should be the > exact same. > > > > I went back to the results from CTT v1.3.14 and this is the response for a > NONconfirmable request message: > > > > 47.388s INFO: ======= Testing resource /Vendor/AirConditioner/TimerClock > - NON confirmable ======= > > 47.388s INFO: > > 47.388s INFO: Sending unicast CON RETRIEVE request with > /Vendor/AirConditioner/TimerClock?if=oic.if.baseline > > 47.389s VERBOSE: -> 107.109.211.84:49163->107.109.211.223:38033 CON-GET > ID=3623, Token=1FD2D505, Options=[URI-Port=38033, URI-Path=Vendor, > AirConditioner, TimerClock, URI-Query=if=oic.if.baseline, > Accept=application/cbor], Secured=true > > 47.399s VERBOSE: <- 107.109.211.223:38033->107.109.211.84:49163 > ACK-Empty Message ID=3623, Token=, Options=[], Secured=true > > 47.468s INFO: Prompting "Waiting for CoAP response... " > > 57.402s VERBOSE: <- 107.109.211.223:38033->107.109.211.84:49163 > CON-2.05 Content ID=3623, Token=1FD2D505, Options=[URI-Path=Vendor, > AirConditioner, TimerClock, Content-Type=application/cbor, Block2=0+ > (1024B/block [6])], Secured=true > > 57.402s VERBOSE: -> 107.109.211.84:49163->107.109.211.223:38033 > ACK-Empty Message ID=3623, Token=, Options=[], Secured=true > > 57.402s VERBOSE: -> 107.109.211.84:49163->107.109.211.223:38033 CON-GET > ID=3624, Token=1FD2D505, Options=[URI-Port=38033, URI-Path=Vendor, > AirConditioner, TimerClock, URI-Query=if=oic.if.baseline, > Accept=application/cbor, Block2=1 (1024B/block [6])], Secured=true > > 57.405s VERBOSE: <- 107.109.211.223:38033->107.109.211.84:49163 > ACK-2.05 Content ID=3624, Token=1FD2D505, Options=[URI-Path=Vendor, > AirConditioner, TimerClock, Content-Type=application/cbor, Block2=1 > (1024B/block [6])], Secured=true > > 57.405s VERBOSE: <- {"rt":["x.com.vendor.timer"]," > if":["oic.if.a","oic.if.baseline","oic.if.baseline"," > oic.if.baseline","oic.if.baseline","oic.if.baseline"," > oic.if.baseline","oic.if.baseline","oic.if.baseline"," > oic.if.baseline","oic.if.baseline","oic.if.baseline"," > oic.if.baseline","oic.if.baseline", "oic.if.baseline","oic.if. > baseline","oic.if.baseline","oic.if.baseline","oic.if. > baseline","oic.if.baseline","oic.if.baseline","oic.if. > baseline","oic.if.baseline","oic.if.baseline","oic.if. > baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline","oic.if.baseline", > "oic.if.baseline","oic.if.baseline","oic.if.baseline"," > oic.if.baseline","oic.if.baseline","oic.if.baseline"," > oic.if.baseline","oic.if.baseline","oic.if.baseline"," > oic.if.baseline","oic.if.baseline","oic.if.baseline"," > oic.if.baseline","oic.if.baseline","oic.if.baseline", > "oic.if.baseline","oic.if.baseline","oic.if.baseline"," > oic.if.baseline","oic.if.baseline","oic.if.baseline"," > oic.if.baseline","oic.if.baseline","oic.if.baseline"," > oic.if.baseline","oic.if.baseline","oic.if.baseline"," > oic.if.baseline","oic.if.baseline","oic.if.baseline", > "oic.if.baseline","oic.if.baseline","oic.if.baseline"," > oic.if.baseline","oic.if.baseline","oic.if.baseline"," > oic.if.baseline","oic.if.baseline","oic.if.baseline"," > oic.if.baseline","oic.if.baseline","oic.if.baseline"," > oic.if.baseline","oic.if.baseline","oic.if.baseline", > "oic.if.baseline","oic.if.baseline","oic.if.baseline"," > oic.if.baseline","oic.if.baseline","oic.if.baseline"," > oic.if.baseline","oic.if.baseline","oic.if.baseline"," > oic.if.baseline","oic.if.baseline","oic.if.baseline"," > oic.if.baseline","oic.if.baseline","oic.if.baseline"], > "x.com.vendor.timer.hour":10,"x.com.vendor.timer.minute":30, > "x.com.vendor.timer.reset":false,"x.com.vendor.timer.second":30} > > 57.417s DEBUG: AgentBase multiple action complete > > > > Notice that this time, the device responded the same as the CON request > message. > > > > Also from the email you attached: > > > > TC > > *Fail Log* > > *Reproducibility * > > * Comment* > > CT1.2.2 > > 83.780s ERROR: Request with if=oic.if.baseline returned resource > {"x.com.vendor.timer.hour":10,"x.com.vendor.timer.minute": > 30,"x.com.vendor.timer.reset":false,"x.com.vendor.timer.second":30} which > doesn't contain rt... > 83.780s ERROR: Request with if=oic.if.baseline returned resource > {"x.com.vendor.timer.hour":10,"x.com.vendor.timer.minute": > 30,"x.com.vendor.timer.reset":false,"x.com.vendor.timer.second":30} which > doesn't contain if... > 83.780s ERROR: CT1.2.2_Check_8: Resource has not all mandatory > properties or interfaces or does not comply to schema > > 50% > > CTT Issue > > > > It looks like the issue only happens about 50% of the time. The comment > says ?CTT Issue? but I don?t think we ever did a thorough log analysis to > back up that comment. > > > > From what I can see, this seems like an intermittent BWT issue with the > reference device / IoTivity. The NON response does not always include any > indication that BWT will be used. > > > > Can you guys look into this on the IoTivity side? > > > > Thanks, > > Mitch > > > > *From:* ??? [mailto:sungkyu.ko at samsung.com] > *Sent:* Wednesday, November 16, 2016 6:58 PM > *To:* Mitch Kettrick; ??? > *Cc:* 'Heldt-Sheller, Nathan'; ???; 'Jacek Hryszkiewicz'; 'Agis, Ed'; ???; > ???; ???; ????; ???; ???; ???; ???; ???; 'Michael Koster'; 'Brian > Scriber'; 'Marek Szkowron'; 'Jason Smith'; Richard Bardini > *Subject:* RE: RE: RE: RE: RE: TC result of IoTivity 1.2.0 vs CTT 1.3.16 > Execution Summary: > > > > Dear Mitch, > > > > I can see the logs. I zipped it again and attached. > > Regarding CT1.2.2 errors, we saw the same errors previous CTT. It was > occured when CTT couldn't handle BWT + Slow Response correctly. > > I attached previous mail about it. > > > > @Antu, > > Cloud you explain further about CT1.2.2 erros? > > Also please test Iotivity against CTT 1.3.17. (Same iotivity ver. as > yesterday) > > > > Thanks & BR, > > Sungkyu Ko. > > > > --------- *Original Message* --------- > > *Sender* : Mitch Kettrick <cpm at openconnectivity.org> > > *Date* : 2016-11-17 00:51 (GMT+9) > > *Title* : RE: RE: RE: RE: TC result of IoTivity 1.2.0 vs CTT 1.3.16 > Execution Summary: > > > > Hi Mushfiqul, > > > > Thanks again for running the tests. For some reason, I can?t open any of > the files within the .zip file. Can anyone else open them or is it just me? > > > > Regarding the issues you?ve reported below: > > > > CT1.2.2 ? As mentioned last time, this looks to be an issue with the > IoTivity reference device. Sungkyu, can you look into this one? > > > > CT1.7.9.1 ? This is an issue with the OIC Link schema file. We?re waiting > for Richard to upload Michael?s schema file change so that it can be > pulled into a new CTT release and this error should go away. > > > > CT1.7.7.1/CT1.7.8.1/CT1.7.8.2 ? The test should run and those warnings > should go away with CTT v1.3.17. > > > > Since I can?t open the logs, I can?t do any detailed analysis so I?ll have > to wait until either someone can resend these results or I get the next > batch. > > > Speaking of the next batch, Comarch just released CTT v1.3.17: > > > > *OCF CTT Release Version 1.3.17 (2016-11-16)* > > Notes: > > 1. CTT 1.3.17 has been released to OCF members as an interim release > version of the tool and can be used ONLY for testing purposes. > > 2. Implemented features > > - Server tests added to client?s profile > > - CR - CT1.7.8.x test cases apply to Clients > > > > This should clear up some issues but it may introduce more when we run the > Server role test cases against a Client role (primary) device. > > > > Thanks, > > Mitch > > > > *From:* Muhammad Mushfiqul Islam [mailto:i.mushfiq at samsung.com > <i.mushfiq at samsung.com>] > *Sent:* Wednesday, November 16, 2016 3:36 AM > *To:* Sung-Kyu Ko; Mitch Kettrick > *Cc:* Heldt-Sheller, Nathan; Jongsung Lee; 'Jacek Hryszkiewicz'; Agis, > Ed; Uze Choi; MyeongGi Jeong; Lee, Dongik; DWARKAPRASAD DAYAMA; Peter > Moonki Hong; MoonYoung Choi; jongmin choi; SOON HWANG CHOI; Gihong Kim; > 'Michael Koster'; Brian Scriber; 'Marek Szkowron'; Jason Smith > *Subject:* RE: RE: RE: RE: TC result of IoTivity 1.2.0 vs CTT 1.3.16 > Execution Summary: > > > > Hello > > Below is the execution summary of CTT v1.3.16 against IoTivity > 1.2.0-RC4(with 6 patches) : > > Fail: 1.2.2, 1.7.9.1 (client and server) > > NA: 2.2.4 (usual), and 1.7.7.1, 1.7.8.1, 1.7.8.2 (client and server, > although whole procedure is completed, instead of showing pass, N/A is > showing) > > CT1.7.8.3 is now passing for both server & client (without patch > https://gerrit.iotivity.org/gerrit/#/c/14137/ ). > > Here is Details: > > * TC* > > *Error Log * > > * Status* > > *Comment * > > CT1.2.2 > > 76.358s 12:40:34 ERROR: Request with if=oic.if.baseline returned resource > {"x.com.vendor.timer.hour":10,"x.com.vendor.timer.minute": > 30,"x.com.vendor.timer.reset":false,"x.com.vendor.timer.second":30} which > doesn't contain rt... > 76.358s 12:40:34 ERROR: Request with if=oic.if.baseline returned resource > {"x.com.vendor.timer.hour":10,"x.com.vendor.timer.minute": > 30,"x.com.vendor.timer.reset":false,"x.com.vendor.timer.second":30} which > doesn't contain if... > 76.359s 12:40:34 ERROR: CT1.2.2_Check_8: Resource has not all mandatory > properties or interfaces or does not comply to schema > 129.151s 12:41:26 ERROR: Test case result: FAILED > > Old CTT Issue Returned > > This issue was found on CTT version 1.3 & before. After that, it was > resolved. > > CT1.7.9.1 > > 1.939s ERROR: '' is not of type 'array' > Failed validating 'type' in schema['allOf'][1]['properties']['aclist'][' > properties']['aces']['items']['allOf'][0]['properties'][' > resources']['items']['properties']['rel']: > ... > 1.944s ERROR: '' is not of type 'array' > Failed validating 'type' in schema['allOf'][1]['properties']['aclist'][' > properties']['aces']['items']['allOf'][0]['properties'][' > resources']['items']['properties']['rel']: > ... > 1.949s ERROR: '' is not of type 'array' > Failed validating 'type' in schema['allOf'][1]['properties']['aclist'][' > properties']['aces']['items']['allOf'][0]['properties'][' > resources']['items']['properties']['rel']: > ... > 1.949s ERROR: CT1.7.9.1_Check_2: IUT response messages do not follow > JSON schema 'oic.r.acl.json' > > Old Issue > > Schema issue (Probably) > > CT1.7.7.1, CT1.7.8.1, CT1.7.8.2 > > > > 1.764s INFO: ======= End of onboarding ======= > > 1.764s INFO: > > 1.764s INFO: Trying to open secure connection... > > 1.764s DEBUG: DTLS PSK Client: Opening connection for client uuid: > 11111111-2222-3333-4444-555555555555 > > 1.764s DEBUG: DTLS PSK Client: Initializing session. Advertised cipher > suites: C037, C0A8 > > 1.769s DEBUG: DTLS PSK Client: Server selected cipher suite C037 > > 1.779s DEBUG: DTLS PSK Client: Client Random d8175c32de5449c9e324de22878bbe > 11b2933768c172a56ec868c308138cbc48 > > 1.779s DEBUG: DTLS PSK Client: Pre-Master Secret > 0020da1fb83994299168ea901ce92d66a9de7b6c5782978ab7e692a06ce4 > daf9516e0010c8c1cfbb5330ae13525296e925930b77 > > 1.779s DEBUG: DTLS PSK Client: Master Secret 322bc6436404f14e40b68a369b2490 > 8c7ace8d551c445e79d8a21c2aed8e95c3938965f6c925bb92fc7d0b363b5fa69d > > 1.779s DEBUG: DTLS PSK Client: Key Block 89fec874d4bc85880ea9eea9719947 > 9edd9a19a40fe2bb2be02502b22bcea31210316e762d01db93f792d3f420 > 6a2119a0fb9d8a10d621c3964e3494e8e013d7fd60bff07eb4d55ffa4daa > 9947e0dc9456141d6a7e6705463edc88caf62d58f9 > > 1.784s DEBUG: DTLS PSK Client: Handshake completed > > 1.784s INFO: Sending unicast CON RETRIEVE request with /oic/res > > 1.784s VERBOSE: -> 107.109.211.84:49239->107.109.211.223:36300 CON-GET > ID=10943, Token=461B5124, Options=[URI-Port=36300, URI-Path=oic, res, > Accept=application/cbor], Secured=true > > 1.789s VERBOSE: <- 107.109.211.223:36300->107.109.211.84:49239 ACK-2.05 > Content ID=10943, Token=461B5124, Options=[URI-Path=oic, res, > Content-Type=application/cbor], Secured=true > > 1.789s VERBOSE: <- [{"di":"6a757374-776f-726b- > 4465-765575696430","links":[{"href":"/oic/sec/doxm","rt":[" > oic.r.doxm"],"if":["oic.if.baseline"],"p":{"bm":1,"sec": > true,"port":36300}},{"href":"/oic/sec/pstat","rt":["oic.r. > pstat"],"if":["oic.if.baseline"],"p":{"bm":1,"sec": > true,"port":36300}},{"href":"/oic/d","rt":["oic.wk.d","oic. > d.airconditioner"],"if":["oic.if.baseline","oic.if.r"],"p":{ > "bm":1,"sec":false}},{"href":"/oic/p","rt":["oic.wk.p"],"if" > :["oic.if.baseline","oic.if.r"],"p":{"bm":1,"sec":false}},{" > href":"/BinarySwitchResURI","rt":["oic.r.switch.binary"]," > if":["oic.if.baseline","oic.if.a"],"p":{"bm":3,"sec":false}},{"href":"/ > TemperatureResURI","rt":["oic.r.temperature"],"if":["oic.if. > baseline","oic.if.a"],"p":{"bm":3,"sec":false}},{"href":"/ > AirFlowResURI","rt":["oic.r.airflow"],"if":["oic.if. > baseline","oic.if.a"],"p":{"bm":3,"sec":false}},{"href":"/ > Vendor/AirConditioner/TimerClock","rt":["x.com. > vendor.timer"],"if":["oic.if.baseline","oic.if.a"],"p":{" > bm":1,"sec":false}},{"href":"/Vendor/AirConditioner/Swinger" > ,"rt":["x.com.vendor.swing"],"if":["oic.if.baseline","oic. > if.a"],"p":{"bm":1,"sec":false}}]}] > > 1.794s INFO: Prompting "Waiting for CoAP response... " > > 1.819s DEBUG: AgentBase multiple action complete > > 1.819s DEBUG: Starting verification with ID:"CT_Precondition"... > > 1.819s INFO: CT_Precondition: IUT respond to /oic/res (CON-GET ID:10943) > > 1.819s DEBUG: Verification with ID:"CT_Precondition" ended with result: > PASSED > > 1.819s WARNING: Did not found any secured resource which href does not > start with '/oic' > > 1.819s DEBUG: Action "PYTHON_SCRIPT" result: NA > > 1.819s DEBUG: Step "python" result: NA > > 1.819s DEBUG: DTLS PSK Client: Alert close_notify(0) was raised > > 2.784s WARNING: Test case result: NA > > Old Issue > > After onboarding, TC result is shown as N/A > > > > > > - Thanks & Regards, > > Mushfiqul Islam Antu > > > > > > --------- *Original Message* --------- > > *Sender* : Jongsung Lee <js126.lee at samsung.com> S5/Senior > Engineer/Security Lab./Samsung Electronics > > *Date* : 2016-11-16 15:36 (GMT+6) > > *Title* : RE: RE: RE: TC result of IoTivity 1.2.0 vs CTT 1.3.15Execution > Summary: > > *To : *Sung-Kyu Ko<sungkyu.ko at samsung.com>, null<nathan.heldt-sheller@ > intel.com>, null<cpm at openconnectivity.org>, Muhammad Mushfiqul Islam< > i.mushfiq at samsung.com>, Jongsung Lee<js126.lee at samsung.com> > > *CC : *null<jacek.hryszkiewicz at comarch.com>, null<ed.agis at intel.com>, > Uze > Choi<uzchoi at samsung.com>, MyeongGi Jeong<myeong.jeong at samsung.com>, Lee, > Dongik<dongik.lee at samsung.com>, DWARKAPRASAD DAYAMA<dwarka.dayama at > samsung. > com>, Peter Moonki Hong<moonki1.hong at samsung.com>, MoonYoung Choi< > moonyoung.choi at samsung.com>, jongmin choi<jminl.choi at samsung.com>, SOON > HWANG CHOI<soonhwang.choi at samsung.com>, Gihong Kim<k1358.kim at > samsung.com>, > null<michael.koster at smartthings.com>, null<b.scriber at cablelabs.com>, > null< > marek.szkowron at comarch.com>, null<lab_mgr at openconnectivity.org> > > > > Dear Mitch and Nathan, > > > > Regarding CT1.7.8.3, I think that "CoAP testing for SVRs" should be > removed according to CR44 (SVR CoAP Access Allowed). > > Recently, https://gerrit.iotivity.org/gerrit/#/c/14137/ has been applied, > so Iotivity's device is operationg as below (due to security > vulnuerability, acl and cred need to be accessible only via coaps). > > doxm > > pstat > > acl > > cred > > CoAP > > access > > access > > deny > > deny > > > > <CR44> > > However, unlike other Resources, the SVRs *may *still be accessed via > CoAP, provided there is a wildcard ?*? ACE naming the SVR. > > > > So, CT1.7.8.3 is failed, if https://gerrit.iotivity.org/gerrit/#/c/14137/ is > included. > > I attached CTT log for CT1.7.8.1. > > 56.199s 17:24:01 ERROR: CT1.7.8.3_Precondition: For message: ID=60353 > Token=73E7BB1F expected response code 2.05 Content (69) - received 4.03 > Forbidden (131). > > 56.199s 17:24:01 DEBUG: Verification with ID:"CT1.7.8.3_Precondition" > ended with result: FAILED > > 56.199s 17:24:01 ERROR: Response for GET '/oic/sec/cred?if=oic.if.baseline' > is not valid response. Could not process procedure for this resource > > > > 56.486s 17:24:01 ERROR: CT1.7.8.3_Precondition: For message: ID=60354 > Token=73E7BB20 expected response code 2.05 Content (69) - received 4.03 > Forbidden (131). > > 56.486s 17:24:01 DEBUG: Verification with ID:"CT1.7.8.3_Precondition" > ended with result: FAILED > > 56.486s 17:24:01 ERROR: Response for GET '/oic/sec/acl?if=oic.if.baseline' > is not valid response. Could not process procedure for this resource > > > > Thank you, > > Jongsung Lee. > > > > --------- *Original Message* --------- > > *Sender* : ??? <sungkyu.ko at samsung.com> S5(??)/??/SQE Lab(S/W??)/???? > > *Date* : 2016-11-16 13:36 (GMT+9) > > *Title* : RE: RE: TC result of IoTivity 1.2.0 vs CTT 1.3.15Execution > Summary: > > > > Dear Nathan, > > > > We are using > > - 1.2-rel branch 1.2.0-RC4 tag and 6 patches. > > - https://gerrit.iotivity.org/gerrit/#/c/13625/ > > - https://gerrit.iotivity.org/gerrit/#/c/13777/ > > - https://gerrit.iotivity.org/gerrit/#/c/13993/ > > - https://gerrit.iotivity.org/gerrit/#/c/14053/ > > - https://gerrit.iotivity.org/gerrit/#/c/14069/ > > - https://gerrit.iotivity.org/gerrit/#/c/14211 > > > > Thanks & BR, > > Sungkyu Ko. > > > > --------- *Original Message* --------- > > *Sender* : Heldt-Sheller, Nathan <nathan.heldt-sheller at intel.com> > > *Date* : 2016-11-16 12:12 (GMT+9) > > *Title* : RE: TC result of IoTivity 1.2.0 vs CTT 1.3.15Execution Summary: > > > > Just to be clear: what version (tag + patches, or just a git version > number) are you using to test IUT Simulator against CTT 1.3.16? I see some > merges happening against 1.2-rel and want to make sure we are all using the > same set of patches to test! > > Thanks, > Nathan > > > > *From:* ??? [mailto:sungkyu.ko at samsung.com <sungkyu.ko at samsung.com>] > *Sent:* Tuesday, November 15, 2016 6:52 PM > *To:* Mitch Kettrick <cpm at openconnectivity.org>; ??? < > i.mushfiq at samsung.com>; ??? <js126.lee at samsung.com> > *Cc:* 'Jacek Hryszkiewicz' <jacek.hryszkiewicz at comarch.com>; Agis, Ed < > Ed.Agis at intel.com>; Heldt-Sheller, Nathan <nathan.heldt-sheller at > intel.com>; > ??? <uzchoi at samsung.com>; ??? <myeong.jeong at samsung.com>; ??? < > dongik.lee at samsung.com>; ???? <dwarka.dayama at samsung.com>; ??? < > moonki1.hong at samsung.com>; ??? <moonyoung.choi at samsung.com>; ??? < > jminl.choi at samsung.com>; ??? <soonhwang.choi at samsung.com>; ??? < > k1358.kim at samsung.com>; 'Michael Koster' <michael.koster at > smartthings.com>; > Brian Scriber <b.scriber at cablelabs.com>; 'Marek Szkowron' < > marek.szkowron at comarch.com>; Jason Smith <lab_mgr at openconnectivity.org> > *Subject:* RE: TC result of IoTivity 1.2.0 vs CTT 1.3.15Execution Summary: > > > > Dear Mitch, > > > > As usual, Jongmin will chekck security TC and Antu will run whole test > caes. :) > > > > @Antu, > > Please test IoTivity 1.2.0 RC4 + patches against CTT 1.3.16 and share the > result. > > I shared new CTT to you. > > > > Thanks & BR, > > Sungkyu Ko. > > > > --------- *Original Message* --------- > > *Sender* : Mitch Kettrick <cpm at openconnectivity.org> > > *Date* : 2016-11-16 02:42 (GMT+9) > > *Title* : TC result of IoTivity 1.2.0 vs CTT 1.3.15Execution Summary: > > > > Hi, > > > > CTT v1.3.16 has been uploaded to Google Drive. Can someone please run this > and share the results? I?m hopeful that this version solves all of the > issues other than the failure for CT1.7.9.1 (the oic-link schema file has > been updated but not released yet). > > > > *OCF CTT Release Version 1.3.16 (2016-11-15)* > > Notes: > > 1. CTT 1.3.16 has been released to OCF members as an interim release > version of the tool and can be used ONLY for testing purposes. > 2. Implemented features > > - create CT1.7.8.3 check_3e and prepare workflow > > - Schema updates > > - Client rediscovering mechanism > > - Security test case v8 > > 3. Fixed issues: > > - #897 (Client Testing: CTT not able to simulate oic.d.tv). > > > > Thanks, > > Mitch > > > > > > *From:* Mitch Kettrick [mailto:cpm at openconnectivity.org > <cpm at openconnectivity.org>] > *Sent:* Monday, November 14, 2016 4:46 PM > *To:* 'i.mushfiq at samsung.com'; 'Jongsung Lee' > *Cc:* 'Jacek Hryszkiewicz'; 'Sung-Kyu Ko'; 'Ed Agis'; 'Heldt-Sheller, > Nathan'; 'Uze Choi'; 'MyeongGi Jeong'; 'Lee, Dongik'; 'DWARKAPRASAD > DAYAMA'; 'Peter Moonki Hong'; 'MoonYoung Choi'; 'jongmin choi'; 'SOON HWANG > CHOI'; 'Gihong Kim'; 'Michael Koster'; Brian Scriber; 'Marek Szkowron'; > Jason Smith (lab_mgr at openconnectivity.org) > *Subject:* RE: RE: RE: RE: RE: TC result of IoTivity 1.2.0 vs CTT > 1.3.15Execution Summary: > > > > Hi Mushfiqul, > > > > Thank you so much for running these tests!! Sorry it took me so long to > review the results. Last week was busy and chose not to work this week > end. ;) > > > > It looks like you stopped CT1.7.8.3 before it finished. I really need to > see the results all the way until the end to make sure that everything is > working as specified. Maybe start the test and go to lunch! J I?m > working with Comarch to shorten it a bit but my top priority is to get > everything to pass so I need to see the test case run all the way until the > end. Once things are passing I?ll put more focus on shortening the test > case where possible. I won?t do my full analysis until I have the complete > test run because it takes a whole day to do it and I don?t want to spend > that much time only to end up with 75% of the answers we need. > > > > Can you run all of the test cases in one test campaign? That is what is > required for certification so I?d like to simulate that as much as > possible. We?ve found that sometimes new failures are introduced when > running test cases all together. > > > > One way to keep the file size down is not to send the pcap files. Now > that the CTT provides full logging information, Wireshark isn?t really > needed any more. > > > > Comarch will be releasing a new version of the CTT that will resolve most > of the errors seen. > > > > In my fairly quick analysis: > > > > ? All of the issues we see in CT1.7.8.3 should go away. > > ? The issue in CT1.2.2 is an issue with the reference device. > For some reason, the /Vendor/AirConditioner/TimerClock has no ?rt? or > ?if? Properties. > > ? The issue in CT1.7.9.1 was caused by a change in the schema > file for the OIC Link related to the ?rel? Property having been changed > from a sting to an array of strings. We spoke about it at the ATG meeting > last week and Michael will be fixing it. > > ? I need to review CT1.7.7.1, CT1.7.8.1 and CT1.7.8.2. Since > Client?s don?t have any vertical Resources, these test cases may not apply > but I need to look at the details. > > > > Let?s wait until we have a new release from Comarch. Should be about 16 > hours from the time of this message. Then we can see if we cleaned up most > of the errors. > > > > Thanks again!! > > > > Mitch > > > > *From:* Muhammad Mushfiqul Islam [mailto:i.mushfiq at samsung.com > <i.mushfiq at samsung.com>] > *Sent:* Friday, November 11, 2016 2:20 AM > *To:* 'Mitch Kettrick'; Jongsung Lee > *Cc:* Jacek Hryszkiewicz; Sung-Kyu Ko; 'Ed Agis'; 'Heldt-Sheller, > Nathan'; Uze Choi; MyeongGi Jeong; Lee, Dongik; DWARKAPRASAD DAYAMA; Peter > Moonki Hong; MoonYoung Choi; jongmin choi; SOON HWANG CHOI; Gihong Kim > *Subject:* RE: RE: RE: RE: RE: TC result of IoTivity 1.2.0 vs CTT > 1.3.15Execution Summary: > > > > Hello, > > Below is the execution summary of CTT v1.3.15 against IoTIvity : > > Fail: 1.2.2, 1.7.9.1 (client and server) > > NA: 2.2.4 (usual), and 1.7.7.1, 1.7.8.1, 1.7.8.2 (client and server, > although whole procedure is completed, instead of showing pass, N/A is > showing) > > No End: 1.7.8.3 (client and server), > > CT1.1.1, CT1.2.3, CT1.26 are passing after applying the patch from Mr. > Lee (https://gerrit.iotivity.org/gerrit/#/c/14211). > > Here is Details: > > * TC* > > *Error Log * > > * Status* > > *Comment * > > CT1.2.2 > > 76.633s ERROR: Request with if=oic.if.baseline returned resource > {"x.com.vendor.timer.hour":10,"x.com.vendor.timer.minute": > 30,"x.com.vendor.timer.reset":false,"x.com.vendor.timer.second":30} which > doesn't contain rt... > 76.633s ERROR: Request with if=oic.if.baseline returned resource > {"x.com.vendor.timer.hour":10,"x.com.vendor.timer.minute": > 30,"x.com.vendor.timer.reset":false,"x.com.vendor.timer.second":30} which > doesn't contain if... > 76.633s ERROR: CT1.2.2_Check_8: Resource has not all mandatory > properties or interfaces or does not comply to schema > > Old CTT Issue Returned > > This issue was found on CTT version 1.3 & before. After that, it was > resolved. > > CT1.7.9.1 > > 1.939s ERROR: '' is not of type 'array' > Failed validating 'type' in schema['allOf'][1]['properties']['aclist'][' > properties']['aces']['items']['allOf'][0]['properties'][' > resources']['items']['properties']['rel']: > ... > 1.944s ERROR: '' is not of type 'array' > Failed validating 'type' in schema['allOf'][1]['properties']['aclist'][' > properties']['aces']['items']['allOf'][0]['properties'][' > resources']['items']['properties']['rel']: > ... > 1.949s ERROR: '' is not of type 'array' > Failed validating 'type' in schema['allOf'][1]['properties']['aclist'][' > properties']['aces']['items']['allOf'][0]['properties'][' > resources']['items']['properties']['rel']: > ... > 1.949s ERROR: CT1.7.9.1_Check_2: IUT response messages do not follow > JSON schema 'oic.r.acl.json' > > New Issue > > Schema issue (Probably) > > CT1.7.8.3 > > 44.597s ERROR: CT1.7.8.3_Precondition: For message: ID=13440 > Token=20BA2B5A expected response code 2.05 Content (69) - received 4.03 > Forbidden (131). > 44.597s ERROR: Response for GET > '/Vendor/AirConditioner/TimerClock?if=oic.if.baseline' > is not valid response. Could not process procedure for this resource > 44.747s ERROR: CT1.7.8.3_Precondition: For message: ID=13441 > Token=20BA2B5B expected response code 2.05 Content (69) - received 4.03 > Forbidden (131). > 44.747s ERROR: Response for GET > '/Vendor/AirConditioner/TimerClock?if=oic.if.baseline' > is not valid response. Could not process procedure for this resource > 47.601s ERROR: CT1.7.8.3_Precondition: For message: ID=13464 > Token=20BA2B72 expected response code 2.05 Content (69) - received 4.03 > Forbidden (131). > 47.601s ERROR: Response for GET '/BinarySwitchResURI?if=oic.if.baseline' > is not valid response. Could not process procedure for this resource > 47.821s ERROR: CT1.7.8.3_Precondition: For message: ID=13465 > Token=20BA2B73 expected response code 2.05 Content (69) - received 4.03 > Forbidden (131). > 47.821s ERROR: Response for GET '/BinarySwitchResURI?if=oic.if.baseline' > is not valid response. Could not process procedure for this resource > 48.099s ERROR: CT1.7.8.3_Precondition: For message: ID=13466 > Token=20BA2B74 expected response code 2.05 Content (69) - received 4.03 > Forbidden (131). > 48.099s ERROR: Response for GET '/TemperatureResURI?if=oic.if.baseline' > is not valid response. Could not process procedure for this resource > 48.346s ERROR: CT1.7.8.3_Precondition: For message: ID=13467 > Token=20BA2B75 expected response code 2.05 Content (69) - received 4.03 > Forbidden (131). > 48.346s ERROR: Response for GET '/TemperatureResURI?if=oic.if.baseline' > is not valid response. Could not process procedure for this resource > 48.641s ERROR: CT1.7.8.3_Precondition: For message: ID=13468 > Token=20BA2B76 expected response code 2.05 Content (69) - received 4.03 > Forbidden (131). > 48.641s ERROR: Response for GET '/AirFlowResURI?if=oic.if.baseline' is > not valid response. Could not process procedure for this resource > 48.831s ERROR: CT1.7.8.3_Precondition: For message: ID=13469 > Token=20BA2B77 expected response code 2.05 Content (69) - received 4.03 > Forbidden (131). > 48.831s ERROR: Response for GET '/AirFlowResURI?if=oic.if.baseline' is > not valid response. Could not process procedure for this resource > 119.549s ERROR: CT1.7.8.3_Precondition: For message: ID=13803 > Token=20BA2BED expected response code 2.05 Content (69) - received 4.03 > Forbidden (131). > 119.549s ERROR: Response for GET > '/Vendor/AirConditioner/TimerClock?if=oic.if.baseline' > is not valid response. Could not process procedure for this resource > 119.879s ERROR: CT1.7.8.3_Precondition: For message: ID=13804 > Token=20BA2BEE expected response code 2.05 Content (69) - received 4.03 > Forbidden (131). > 119.879s ERROR: Response for GET > '/Vendor/AirConditioner/TimerClock?if=oic.if.baseline' > is not valid response. Could not process procedure for this resource > 125.086s ERROR: CT1.7.8.3_Precondition: For message: ID=13827 > Token=20BA2C05 expected response code 2.05 Content (69) - received 4.03 > Forbidden (131). > 125.086s ERROR: Response for GET '/BinarySwitchResURI?if=oic.if.baseline' > is not valid response. Could not process procedure for this resource > 125.448s ERROR: CT1.7.8.3_Precondition: For message: ID=13828 > Token=20BA2C06 expected response code 2.05 Content (69) - received 4.03 > Forbidden (131). > 125.448s ERROR: Response for GET '/BinarySwitchResURI?if=oic.if.baseline' > is not valid response. Could not process procedure for this resource > 125.903s ERROR: CT1.7.8.3_Precondition: For message: ID=13829 > Token=20BA2C07 expected response code 2.05 Content (69) - received 4.03 > Forbidden (131). > 125.903s ERROR: Response for GET '/TemperatureResURI?if=oic.if.baseline' > is not valid response. Could not process procedure for this resource > 126.258s ERROR: CT1.7.8.3_Precondition: For message: ID=13830 > Token=20BA2C08 expected response code 2.05 Content (69) - received 4.03 > Forbidden (131). > 126.258s ERROR: Response for GET '/TemperatureResURI?if=oic.if.baseline' > is not valid response. Could not process procedure for this resource > 126.688s ERROR: CT1.7.8.3_Precondition: For message: ID=13831 > Token=20BA2C09 expected response code 2.05 Content (69) - received 4.03 > Forbidden (131). > 126.688s ERROR: Response for GET '/AirFlowResURI?if=oic.if.baseline' is > not valid response. Could not process procedure for this resource > 127.043s ERROR: CT1.7.8.3_Precondition: For message: ID=13832 > Token=20BA2C0A expected response code 2.05 Content (69) - received 4.03 > Forbidden (131). > 127.043s ERROR: Response for GET '/AirFlowResURI?if=oic.if.baseline' is > not valid response. Could not process procedure for this resource > 227.969s ERROR: CT1.7.8.3_Precondition: For message: ID=14157 > Token=20BA2C8C expected response code 2.05 Content (69) - received 4.03 > Forbidden (131). > 227.969s ERROR: Response for GET > '/Vendor/AirConditioner/TimerClock?if=oic.if.baseline' > is not valid response. Could not process procedure for this resource > 228.464s ERROR: CT1.7.8.3_Precondition: For message: ID=14158 > Token=20BA2C8D expected response code 2.05 Content (69) - received 4.03 > Forbidden (131). > 228.464s ERROR: Response for GET > '/Vendor/AirConditioner/TimerClock?if=oic.if.baseline' > is not valid response. Could not process procedure for this resource > 236.031s ERROR: CT1.7.8.3_Precondit > > ... -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.iotivity.org/pipermail/iotivity-dev/attachments/20161121/e83ec655/attachment.html>
