On Thu, 2011-01-20 at 14:03 -0500, Samuel Greenfeld wrote: > A few more inline comments on a subset of the questions, and two more > thoughts: > > On Thu, Jan 20, 2011 at 11:54 AM, James Laska <jla...@redhat.com> wrote: > > On Thu, 2011-01-20 at 05:54 -0500, Samuel Greenfeld wrote: > > > 1. What is the history of Nitrate and the Fedora Project? What > > > does the Fedora project expect to gain from using it? > > > > This goes back to an eval we did using testopia in Fedora many releases > > agove. Unfortunately, the effort was canceled due to license > > incompatibility between Fedora and testopia. At that point, we invested > > in leveraging the wiki to best of our ability. > > > > https://fedoraproject.org/wiki/QA/Testopia+AF8-Evaluation > > https://bugzilla.redhat.com/show_bug.cgi?id=450013 > > >From the bug I'm guessing this is because Testopia used Ext-JS, and > Ext-JS kept changing licenses. Hopefully there will only be licensing > restrictions for Nitrate on the software itself, and not the created > work of what one does while actively using it.
That was a question I had opened with Hurry. I have no idea how we'd license the content in the system. If anything, I'd hope/expect that to match that of our current wiki. > > > 1. How does Nitrate compare to other open & closed sourced TCMS > > > solutions? Why was it written as opposed to using an existing > > > solution, and what are its strengths & weaknesses? > > > > See history comments above. Also, maybe the nitrate developers [1] can > > offer more insight on how it compares to other open-source solutions? I > > *think* that comparison work has been done in the past, I'm just not > > sure where to find the results. > > > > [1] https://fedorahosted.org/mailman/listinfo/nitrate-devel > > Are the developers actively monitoring/using this list? I looked at > it before, and all I saw were three test posts from July in the > archives. I believe they do, but don't yet have a strong upstream presence since there hasn't been a lot of code/progress to share until recently. This will be something I'm sure they'll want to improve as community interest increases. > > > 1. Can multiple projects share test cases, and even reference > > > older versions of test cases if they are lagging behind the > > > current rawhide/Fedora release? Will Spins be able to make > > > their own (simultaneously running) test plans? > > > > This is the hope. It's not really useful if we can only use it for > > release validation. I don't think we've fully explored how best to > > model other spins/projects, but I don't foresee big problems there. > > That will be fun to explore on the sandbox/staging instance. > > > > With regards to referencing older versions of a test case, I believe > > that support is there, although I'm not certain it's right for our > > needs. Keeping test documentation (plans and cases) updated is a pretty > > sizable maintenance challenge. I've seen many instances where support > > for versioned test cases allows test plans to suffer over time as they > > were linked against old and inaccurate test cases. > > > > Much like how the wiki is used now, we have support for linking against > > older versions of tests (wiki history), but we rarely ever use that > > feature. I expect that trend would continue in the short-term. > > The reason I bring this up is because OLPC's Spin releases tend to lag > behind the official Fedora release. For instance, we just released > our hopefully final Fedora 11-based release this past month, and are > in the early stages of the Fedora 14-based one. At least some Fedora > ARM development work may still be going on with Fedora 13 as well. > > While decently written test cases will survive somewhat over time, > major changes in GUI look & feel or other areas can break their > backwards compatibility. > > Ideally Nitrate will default to using the current version of a > testcase when making a new plan, preferably following the updates of > said testcase until a result is committed which forces the test case > version to be needed. I hope so, yes! :) I'm fairly certain support exists for linking against versioned cases, that is, I know it was present in testopia. Hopefully Hurry, or the nitrate folks can help here. Sounds like we need to add this to our list of requirements. > > > 1. How long will historical test case results be made available? > > > > I suspect the limiting factor here is database size. I'm not aware of > > any rules or process that would require removal/archival of old results. > > However, at some point that could certainly be an issue we'd need to > > plan for. > > Again; this would be to help lagging Spins and similar. Right now it > looks like Bodhi at least publicly hides information for Fedora > versions which have reached end-of-life. (Either that, or I don't > know how to find it.) That or garbage collected. Sounds like we'll need to do some further research on this front. I personally see no reason to purge results for EOL'd releases. I've often referenced old test results to understand when the last time something was tested. Test results aren't disk-hogs like builds (*cough* oo.org *cough*) can be, so I don't expect this to be as much of an issue. Nonetheless, you're point is valid. > I presume that Nitrate has already been determined to be scalable; > otherwise it is a big risk to incorporate it into Fedora. I'll let the nitrate folks speak to scalability issues. That's definitely something we'll want to explore during a pilot. Remember too, I don't expect we'll throw away the wiki, and switch to nitrate without a good bake-in period to identify bugs/gaps etc... > Also: It might be useful to add an "unclear" testcase result, similar > to how Mozilla's Litmus system does it (https://litmus.mozilla.org). I do like litmus! It's a nice evolution from testopia for upstream mozilla. We don't currently have an 'unclear' test result. I'm not opposed to it, but would need better understand how that field is used, and the process around it, in litmus. Thanks, James
signature.asc
Description: This is a digitally signed message part
-- test mailing list test@lists.fedoraproject.org To unsubscribe: https://admin.fedoraproject.org/mailman/listinfo/test
