Comments inline. ----- Original Message ----- > From: "Sebastian Hengst" <archaeopte...@coole-files.de> > To: "Jason Smith" <jsm...@mozilla.com>, "dev-b2g" > <dev-b2g@lists.mozilla.org>, "dev-gaia" > <dev-g...@lists.mozilla.org> > Sent: Sunday, August 24, 2014 10:36:38 AM > Subject: Re: Review of QA's Smoketests > > Betreff: Review of QA's Smoketests > Von: Jason Smith <jsm...@mozilla.com> > Datum: 2014-08-20 23:20 > > Hi Everyone, > > > > I'd like to get feedback on our current smoketest suite to see if > there's agreement or disagreement with the test cases currently included > in the test suite. Here is a link to the smoketest suite below: > > > > https://moztrap.mozilla.org/runtests/run/5060/env/347/?pagenumber=1&pagesize=100 > > > > Some questions I'd like feedback on include: > > > > 1. Which test cases do you think make sense to be included in a smoketest > > suite? > > 2. Which test cases do you think don't make sense to be included in a > > smoketest suite? > > 3. Are there test cases you don't see in the test suite that should be > > included? > > Ability to join a hidden Wi-Fi (broken in First-run > https://bugzilla.mozilla.org/show_bug.cgi?id=935487 , was recently > broken in Settings https://bugzilla.mozilla.org/show_bug.cgi?id=1048373 )
This feels like a test case I was use a full functional test run, but not in a smoketest suite, because I don't think this is bare bones test case of the FTU app. > > Unlocking with SIM PIN enabled, e.g. > https://bugzilla.mozilla.org/show_bug.cgi?id=1035739 > https://bugzilla.mozilla.org/show_bug.cgi?id=970022 This is a good idea to explore. > > Battery charge correctly recognized and drain and load reflected by > charge shown. This is a good idea to explore. > > Screenshot can be taken and transfered to a computer connected with USB. I don't think this falls under the scope of bare bones functionality of the phone. I could still use the phone even if I couldn't take screenshots. > > Music and Video can play the expected codecs and file formats. In the > same vein like https://moztrap.mozilla.org/manage/case/2478/ I think there's already a test case for this in smoketests for one particular formats. I do think additional formats are good to test, but not necessarily something I would block a smoketest on. > > Importing contacts from a .vcf file works. I think importing contacts from a data source is a smoketest (which is already covered). Other formats though I think fall into the criteria for full functional testing. > > A graphics stressing test like big canvases or web pages (no crashes or > freeze if it happened before). This feels like a good stress test case, but not a bare bones test case. > > Enabling and disabling the screenreader should work. I don't think I'd block a smoketest if the screenreader broke, as it's not a critical feature to basic phone usage. > > > 4. Do you think the steps in certain test cases make sense or do you think > > there's some refinement needed on those test cases? > OTA update over Wi-Fi: https://moztrap.mozilla.org/manage/case/2313/ > There should be a given time limit for the installation to complete. (My > experience with Keon and Unagi is/was that this can take hours. > https://bugzilla.mozilla.org/show_bug.cgi?id=923753 ) That's a good idea. > > > > Sincerely, > > Jason Smith > > > > Sebastian / Archaeopteryx > _______________________________________________ dev-b2g mailing list dev-b2g@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-b2g
