Re: [Trac] New plugins: a true Test Case Manager

[Roberto Longobardi]

>
> Hi Olemis,
>


> >Cool ! It's very nice to realize that you're really enthusiastic about
> >developing the plugin . Hopefully I will be able to adapt it to my own
> >needs sooner or later (I already bought the hammer ;o)
>
> XD


> >However I'd really like to have an RPC interface to perform these
> >actions (because in my deployment environment there will be external
> >tools e.g. Hudson, ... and a BPMS that will be in «touch» with the
> >test case manager). I can dedicate some time to get this done, and
> >maintain the RPC handlers. I'd really like to join | contribute to the

>project . The major difficulty to do it (from my perspective ;o) is
> >that the VCS is SVN (and I cannot interact with SVN repos since long
> >time ago :-( ... ) .
> >
> >If the switch to a DVCS like bzr, hg, git is not an option , as a
> >workaround , I'll try to create (somehow) an HgSVN repos and publish
> >it as well as a patch queue @ Bitbucket . I'll develop patches and
> >send them back to you . Is it ok ?
>
> I chose SVN just because it is the one used by both sourceforge and
track-hacks...
but I too can't access it from the job, have to do it once back home.
If you manage to do the same, that may be an option. Otherwise, just send
me
the patches and I'll do the merge.
If you give me your account, I may allow you to checkin code into
sourcefprge.


> >PS: Could you please register the plugin in PyPI ?
>

It's already there from the start :-)  Search for "TestManager"

>
> >> The supported statuses are currently:
> >>  - TO_BE_TESTED
> >>  - SUCCESSFUL
> >>  - FAILED
> >>
> >
> >Support for custom test results may be an important addition for
> >future versions (there are lots of threads about this at TiP ML ;o) .
> >An approach to get this done would be to consider test statuses as
> >dumb plain text identifiers, and give them some meaning inside Trac
> >using workflow actions (in a similar way to tickets actions).
>
> Yes, I agree with this. It may be the next stuff into the pipeline :-)


> >>> - There are traceability relationships between TCs and
> >>>tickets (more technically speaking, requirements
> >>>tracked by tickets ;o) so that it's possible to let's say
> >>>know what TCs should be run on changing a ticket
> >>>state manually | by workflow or whatever .
> >>
> >> The traceability is currently only from a Ticket top the corresponding
> Test
> >> Case, but not the other way around.
> >
> >Ok, but ideally it would be nice to have some kind of traceability
> >matrices (probably implemented as sparse matrices ;o) and custom
> >traceability relations. The traceability model in use (in this

>concrete scenario) is based on Requisite Pro traceability matrices,
> >but it would be nice to have something like that inside Trac ;o). Of
> >course , this could be handled at a higher level of abstraction ;o)
> >
>
>> You can open a Ticket and have a traceback to the (e.g. failed) Test Case
> as
> >> follows:
> >> Open a Ticket on a Test Case:
> >> Whether you deploy TracTicketTemplatePlugin or not, you can get the
> >> following URL, where testCaseNumber is the Test Case complete path,
> planid
> >> is the Test Plan ID and planName is its name
> >>
> >[...]
> >
> >Do you use custom ticket fields ?
> >
>

I have an "Open a ticket for this test case" button on the test case (or
test case in plan).
Clickin on it, I open a new ticket for editing with a summary recalling the
test case title and
a wiki reference link in the ticket description, allowing the user to
navigate back to the
corresponding test case.

See below how to allow for programmatic reference.


> >>> - Tickets , test cases , build jobs and everything participate
> >>>in a workflow , even if there are multiple tools
> >>>involved.
> >>
> >> I get the idea, but maybe orchestrating different tools
> >> in a workflow should
> >> be the task of a different tool (a process server?).
> >>
> >
> >Well as I mentioned before in this particular scenario there are two
> >concurrent workflows installed . Trac built-in workflow orchestrates
> >*EVERYTHING* related to software dev process and part of QA. Another
> >BPEL-WS enabled tool considers Trac (and it's workflow ;o) as a block
> >of a wider enterprise process. It encloses other areas (e.g. CRM, HRM,
> >...) and interacts with Trac via RPC .
>
> Cool.


> >>>  - How extensible are TC management components ?
> >> I haven't currently defined any extension points... do you guys have any
> >> idea what could be useful here (besides custom test case execution
> >> verdicts)?
> >>
> >
> >I'll take a look and suggest concrete extension points. I just need to
> >prepare the DVCS stuff  ;o)
>
> I already have received some other ideas for extensions, like custom
properties for test plans (platform, ...).


> >>>  - Wouldn't it be nice to have workflow actions
> >>> related to TC management ?
> >>
> >> Being test cases (and catalogs) actually Wiki pages gives us for free
> the
> >> versioning and histo

Re: [Trac] New plugins: a true Test Case Manager

[Olemis Lang]

On Wed, Aug 18, 2010 at 10:05 AM, Roberto Longobardi  wrote:
> Hi Olemis,

:o)

> I have introduced the "Test Plans" feature in the plugin.

Cool ! It's very nice to realize that you're really enthusiastic about
developing the plugin . Hopefully I will be able to adapt it to my own
needs sooner or later (I already bought the hammer ;o)

> The scenario you describe below would now look like as follows.
>> - TCs are *discovered* (generated ?) from
>>    source code (e.g. ) committed to a repos
>>    and they are using static code analysis,
>>    abstract models , XML specs ...
>> - Test execution , etc , is performed by an external
>>    CI tool (currently Hudson + Bitten ...)
>>
>> In this scenario , I'd like to be able to
>>
>>  - Retrieve test results from CI tool using an API
>>  - Trigger build jobs on changing TC states or when
>>     aforementioned test campaigns (which will be
>>     mapped to in Hudson builds) are started ,
>>     explicitly executed .
>>  - ... well some other , but have no time to say it all
>>
>
> You can create test catalogs and cases programmatically by means of the
> following http requests:
[...]
> Create a root test catalog:
>
[...]
> Create a sub-catalog:
>
[...]
> Create a Test Case:
>
[...]
> Create a Test Plan from a specific catalog:
[...]
>
> Set a Test Case execution verdict, in the context of a Test Plan:
>
[...]

This is really good news !!!

However I'd really like to have an RPC interface to perform these
actions (because in my deployment environment there will be external
tools e.g. Hudson, ... and a BPMS that will be in «touch» with the
test case manager). I can dedicate some time to get this done, and
maintain the RPC handlers. I'd really like to join | contribute to the
project . The major difficulty to do it (from my perspective ;o) is
that the VCS is SVN (and I cannot interact with SVN repos since long
time ago :-( ... ) .

If the switch to a DVCS like bzr, hg, git is not an option , as a
workaround , I'll try to create (somehow) an HgSVN repos and publish
it as well as a patch queue @ Bitbucket . I'll develop patches and
send them back to you . Is it ok ?

PS: Could you please register the plugin in PyPI ?

> The supported statuses are currently:
>  - TO_BE_TESTED
>  - SUCCESSFUL
>  - FAILED
>

Support for custom test results may be an important addition for
future versions (there are lots of threads about this at TiP ML ;o) .
An approach to get this done would be to consider test statuses as
dumb plain text identifiers, and give them some meaning inside Trac
using workflow actions (in a similar way to tickets actions).

>> - There are traceability relationships between TCs and
>>    tickets (more technically speaking, requirements
>>    tracked by tickets ;o) so that it's possible to let's say
>>    know what TCs should be run on changing a ticket
>>    state manually | by workflow or whatever .
>
> The traceability is currently only from a Ticket top the corresponding Test
> Case, but not the other way around.

Ok, but ideally it would be nice to have some kind of traceability
matrices (probably implemented as sparse matrices ;o) and custom
traceability relations. The traceability model in use (in this
concrete scenario) is based on Requisite Pro traceability matrices,
but it would be nice to have something like that inside Trac ;o). Of
course , this could be handled at a higher level of abstraction ;o)

> You can open a Ticket and have a traceback to the (e.g. failed) Test Case as
> follows:
> Open a Ticket on a Test Case:
> Whether you deploy TracTicketTemplatePlugin or not, you can get the
> following URL, where testCaseNumber is the Test Case complete path, planid
> is the Test Plan ID and planName is its name
>
[...]

Do you use custom ticket fields ?

>> - Tickets , test cases , build jobs and everything participate
>>    in a workflow , even if there are multiple tools
>>    involved.
>
> I get the idea, but maybe orchestrating different tools
> in a workflow should
> be the task of a different tool (a process server?).
>

Well as I mentioned before in this particular scenario there are two
concurrent workflows installed . Trac built-in workflow orchestrates
*EVERYTHING* related to software dev process and part of QA. Another
BPEL-WS enabled tool considers Trac (and it's workflow ;o) as a block
of a wider enterprise process. It encloses other areas (e.g. CRM, HRM,
...) and interacts with Trac via RPC .

>>  - How extensible are TC management components ?
> I haven't currently defined any extension points... do you guys have any
> idea what could be useful here (besides custom test case execution
> verdicts)?
>

I'll take a look and suggest concrete extension points. I just need to
prepare the DVCS stuff  ;o)

>>  - Wouldn't it be nice to have workflow actions
>>     related to TC management ?
>
> Being test cases (and catalogs) actually Wiki pages gives us for free the
> versioning and history capabilities.
> I have also added with this release cus

Re: [Trac] New plugins: a true Test Case Manager

[Roberto Longobardi]

Hi Christian,
thanks for the kudos :-)

Mmmh, currently there's no way to add comments when changing the status of a
test case, other than opening a ticket for it... but this is undesirable in
case of success, of course.

This is something that may come into the base code, I think.

About your second point, instead, adding sevral environment information to a
test plan, the first thing that comes to my mind is to provide extension
points to the plugin, instead of including all sorts of information with the
base version.
This way, anyone could add the information more pertinent to his/her working
environment and needs.

I'll think about which extension points may be added, and I'm open to
suggestions of course.
Probably, supporting a list of custom properties associated to each record
in the "testplans" table may be one way...

Also, please, feel free to open enhancements requests, and to provide
patches if you wish to :-)

Thanks, ciao,
Roberto


2010/8/25 Christian Dähn 

>  Hi,
>
> the TestCaseManager seems to really beat any other solutions - it's great!
>
> But I'm looking just one feature - which maybe already exists inside the
> plugin:
> Each test case gets a status (works great) and a comment (didn't found an
> input field -
> do I have to open a ticket for comments?) from the tester.
>
> The goal for a software test is to get a report, where all tests are listed
> with their
> result and the comments of the tester. Based on this the developer /
> reviewer
> creates tickets for failed tests, if needed.
>
> Further, a test plan / report must contain informations about the tested
> subject
> and environment (app version, build date, test date, name of tester, used
> operation
> system...) - this is very important to compare test reports by app version
> and
> to trac changes to an app (e.g. to find regressions).
>
> If my request is not too far away from the plugin's concept and goal,
> I would post this as a feature request.
>
> Thanks for this great plugin! I'm looking forward to use it for my daily
> work.
> Currently I try to port the plugin to Python 2.4 (our production server is
> very old - hope to get a newer one within this year).
>
> ciao,
> Chris
>
>  --
> You received this message because you are subscribed to the Google Groups
> "Trac Users" group.
> To post to this group, send email to trac-us...@googlegroups.com.
> To unsubscribe from this group, send email to
> trac-users+unsubscr...@googlegroups.com
> .
> For more options, visit this group at
> http://groups.google.com/group/trac-users?hl=en.
>

-- 
You received this message because you are subscribed to the Google Groups "Trac 
Users" group.
To post to this group, send email to trac-us...@googlegroups.com.
To unsubscribe from this group, send email to 
trac-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/trac-users?hl=en.

Re: [Trac] New plugins: a true Test Case Manager

[Christian Dähn]

Hi,

the TestCaseManager seems to really beat any other solutions - it's great!

But I'm looking just one feature - which maybe already exists inside the plugin:
Each test case gets a status (works great) and a comment (didn't found an input 
field -
do I have to open a ticket for comments?) from the tester.

The goal for a software test is to get a report, where all tests are listed 
with their
result and the comments of the tester. Based on this the developer / reviewer
creates tickets for failed tests, if needed.

Further, a test plan / report must contain informations about the tested subject
and environment (app version, build date, test date, name of tester, used 
operation
system...) - this is very important to compare test reports by app version and
to trac changes to an app (e.g. to find regressions).

If my request is not too far away from the plugin's concept and goal,
I would post this as a feature request.

Thanks for this great plugin! I'm looking forward to use it for my daily work.
Currently I try to port the plugin to Python 2.4 (our production server is
very old - hope to get a newer one within this year).

ciao,
Chris

-- 
You received this message because you are subscribed to the Google Groups "Trac 
Users" group.
To post to this group, send email to trac-us...@googlegroups.com.
To unsubscribe from this group, send email to 
trac-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/trac-users?hl=en.

Re: [Trac] New plugins: a true Test Case Manager

[Roberto Longobardi]

quot;build/bdist.linux-i686/egg/testmanager/model.py", line 60, in
>> upgrade_environment
>>self._upgrade_db(db)
>>  File "build/bdist.linux-i686/egg/testmanager/model.py", line 110, in
>> _upgrade_db
>>' ', '', 0))
>>  File "build/bdist.linux-i686/egg/trac/db/util.py", line 65, in execute
>>return self.cursor.execute(sql_escape_percent(sql), args)
>>  File "build/bdist.linux-i686/egg/trac/db/sqlite_backend.py", line 78, in
>> execute
>>result = PyFormatCursor.execute(self, *args)
>>  File "build/bdist.linux-i686/egg/trac/db/sqlite_backend.py", line 56, in
>> execute
>>args or [])
>>  File "build/bdist.linux-i686/egg/trac/db/sqlite_backend.py", line 48, in
>> _rollback_on_error
>>return function(self, *args, **kwargs)
>> IntegrityError: columns name, version are not unique
>>
>>
>> - Original Message -
>> > From: "Roberto Longobardi" 
>> > To: "Gary Oberbrunner" 
>> > Cc: trac-users@googlegroups.com, "Stephen Bash" 
>> > Sent: Thursday, August 19, 2010 12:32:48 AM
>> > Subject: Re: [Trac] New plugins: a true Test Case Manager
>> > Hi Gary,
>> > I would like to make it compatible with 0.12, but didn't have a chance
>> > to work on it yet.
>> >
>> >
>> > I'll take a look today at what's wrong and see if it's something
>> > quick.
>> >
>> >
>> > Ciao,
>> > Roberto
>> >
>> >
>> >
>> > 2010/8/18 Gary Oberbrunner < ga...@genarts.com >
>> >
>> >
>> >
>> > On 8/18/2010 10:05 AM, Roberto Longobardi wrote:
>> > > Hi Olemis,
>> > > I have introduced the "Test Plans" feature in the plugin
>> > > < http://trac-hacks.org/wiki/TestManagerForTracPlugin >.
>> >
>> > We tried to install this here but it appears to be incompatible with
>> > Trac 0.12. Are you planning to update it to 0.12?
>> >
>> > -- Gary
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups "Trac 
Users" group.
To post to this group, send email to trac-us...@googlegroups.com.
To unsubscribe from this group, send email to 
trac-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/trac-users?hl=en.

Re: [Trac] New plugins: a true Test Case Manager

[Roberto Longobardi]

Hi Stephen,
the problem you're seeing is caused by the upgrade db code trying to create
the wiki page "/TC" programmatically, by inserting directly into the wiki
table. I now changed this code to use the Wiki class instead of writing into
the table, but haven't released this code yet.
BTW, this wiki page is used as the root of all test catalogs.

The error is complaining about you having already a wiki page with that
name... this is surprising, unless you had already installed the plugin
(maybe a lower version) some time before. Uninstalling the plugin does not
remove this page, so upon uninstall you should delete the "TC" wiki page
manually, if you plan to reinstall the plugin eventually.

In a fresh environment this should never happen, so you can try and deploy
the plugin into a new environment if you can.

Please, let me know if this does not fix the problem and I'll try to provide
you with a script that drops the new tables so to be able to rerun the
update db once again.

Ciao,
Roberto


2010/8/19 Stephen Bash 

> Roberto-
>
> Everything compiled fine.  We ran into an issue during the database
> upgrade.  I've included the trac log output below.  Now trac-admin reports
> the database is up to date, so I can't reproduce the issue anymore.
>
> Thanks,
> Stephen
>
> Trac DEBUG log:
> 2010-08-18 15:29:58,752 Trac[env] INFO:
> testmanager.model.TestManagerModelProvider upgrading...
> 2010-08-18 15:29:58,752 Trac[model] DEBUG: CREATE TABLE testconfig (
>propname text PRIMARY KEY,
>value text
> );
> 2010-08-18 15:29:59,476 Trac[model] DEBUG: CREATE  INDEX
> testconfig_propname_idx ON testconfig (propname);
> 2010-08-18 15:29:59,484 Trac[model] DEBUG: CREATE TABLE testcases (
>id text,
>planid text,
>status text,
>UNIQUE (id,planid)
> );
> 2010-08-18 15:29:59,496 Trac[model] DEBUG: CREATE  INDEX testcases_id_idx
> ON testcases (id);
> 2010-08-18 15:29:59,506 Trac[model] DEBUG: CREATE  INDEX
> testcases_planid_idx ON testcases (planid);
> 2010-08-18 15:29:59,519 Trac[model] DEBUG: CREATE TABLE testcasehistory (
>id text,
>planid text,
>time integer,
>author text,
>status text,
>UNIQUE (id,planid,time)
> );
> 2010-08-18 15:29:59,530 Trac[model] DEBUG: CREATE  INDEX
> testcasehistory_id_planid_time_idx ON testcasehistory (id,planid,time);
> 2010-08-18 15:29:59,543 Trac[model] DEBUG: CREATE TABLE testplans (
>planid text PRIMARY KEY,
>catid text,
>catpath text,
>name text,
>author text,
>time integer
> );
> 2010-08-18 15:29:59,556 Trac[model] DEBUG: CREATE  INDEX
> testplans_planid_idx ON testplans (planid);
> 2010-08-18 15:29:59,574 Trac[model] DEBUG: CREATE  INDEX
> testplans_catid_idx ON testplans (catid);
> 2010-08-18 15:29:59,610 Trac[console] ERROR: Exception in trac-admin
> command:
> Traceback (most recent call last):
>  File "build/bdist.linux-i686/egg/trac/admin/console.py", line 107, in
> onecmd
>rv = cmd.Cmd.onecmd(self, line) or 0
>  File "/usr/lib/python2.5/cmd.py", line 218, in onecmd
>return self.default(line)
>  File "build/bdist.linux-i686/egg/trac/admin/console.py", line 257, in
> default
>return cmd_mgr.execute_command(*args)
>  File "build/bdist.linux-i686/egg/trac/admin/api.py", line 123, in
> execute_command
>return f(*fargs)
>  File "build/bdist.linux-i686/egg/trac/env.py", line 780, in _do_upgrade
>self.env.upgrade(backup=no_backup is None)
>  File "build/bdist.linux-i686/egg/trac/env.py", line 523, in upgrade
>with_transaction(self)(participant.upgrade_environment)
>  File "build/bdist.linux-i686/egg/trac/db/api.py", line 77, in
> transaction_wrapper
>fn(ldb)
>  File "build/bdist.linux-i686/egg/testmanager/model.py", line 60, in
> upgrade_environment
>self._upgrade_db(db)
>  File "build/bdist.linux-i686/egg/testmanager/model.py", line 110, in
> _upgrade_db
>' ', '', 0))
>  File "build/bdist.linux-i686/egg/trac/db/util.py", line 65, in execute
>return self.cursor.execute(sql_escape_percent(sql), args)
>  File "build/bdist.linux-i686/egg/trac/db/sqlite_backend.py", line 78, in
> execute
>result = PyFormatCursor.execute(self, *args)
>  File "build/bdist.linux-i686/egg/trac/db/sqlite_backend.py", line 56, in
> execute
>    args or [])
>  File "build/bdist.linux-i686/egg/trac/db/sqlite_backend.py", line 48, in
> _rollback_on_error
>return function(self, *args, **kwargs)
> IntegrityError: columns name, version are not unique
>
>
> - Original Message -
> > From: "Roberto Longobardi&qu

Re: [Trac] New plugins: a true Test Case Manager

[Roberto Longobardi]

Hi Gary,
I would like to make it compatible with 0.12, but didn't have a chance to
work on it yet.

I'll take a look today at what's wrong and see if it's something quick.

Ciao,
Roberto


2010/8/18 Gary Oberbrunner 

> On 8/18/2010 10:05 AM, Roberto Longobardi wrote:
> > Hi Olemis,
> > I have introduced the "Test Plans" feature in the plugin
> > .
>
> We tried to install this here but it appears to be incompatible with
> Trac 0.12.  Are you planning to update it to 0.12?
>
> -- Gary
>

-- 
You received this message because you are subscribed to the Google Groups "Trac 
Users" group.
To post to this group, send email to trac-us...@googlegroups.com.
To unsubscribe from this group, send email to 
trac-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/trac-users?hl=en.

Re: [Trac] New plugins: a true Test Case Manager

[Gary Oberbrunner]

On 8/18/2010 10:05 AM, Roberto Longobardi wrote:
> Hi Olemis,
> I have introduced the "Test Plans" feature in the plugin
> . 

We tried to install this here but it appears to be incompatible with
Trac 0.12.  Are you planning to update it to 0.12?

-- Gary

-- 
You received this message because you are subscribed to the Google Groups "Trac 
Users" group.
To post to this group, send email to trac-us...@googlegroups.com.
To unsubscribe from this group, send email to 
trac-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/trac-users?hl=en.

Re: [Trac] New plugins: a true Test Case Manager

[Roberto Longobardi]

Hi Olemis,
I have introduced the "Test Plans" feature in the
plugin
.

The scenario you describe below would now look like as follows.

> - TCs are *discovered* (generated ?) from
>source code (e.g. ) committed to a repos
>and they are using static code analysis,
>abstract models , XML specs ...
> - Test execution , etc , is performed by an external
>CI tool (currently Hudson + Bitten ...)
>
> In this scenario , I'd like to be able to
>
>  - Retrieve test results from CI tool using an API
>  - Trigger build jobs on changing TC states or when
> aforementioned test campaigns (which will be
> mapped to in Hudson builds) are started ,
> explicitly executed .
>  - ... well some other , but have no time to say it all
>
> Q:
> - Considering scenarios like that one , is it possible
> to use interfaces so as , let's say , retrieve, synch
>TC history from other sources e.g. Hudson build history


You can create test catalogs and cases programmatically by means of the
following http requests:

*Create a root test catalog:*

Get the following URL:
http://yourserver/yourproject/testcreate?type=catalog&path=TC&title=My%20new%20catalog

this will assign a unique ID (the number 0 in the URL below) to the new
catalog and redirect to a Wiki edit page:
http://yourserver/yourproject/wiki/TC_TT0?action=edit&text=%3D%3D+My+new+catalog+%3D%3D

that you can simply post the form to submit the page and thus create the
catalog.


*Create a sub-catalog:*
Get the wollowing URL, where "path" is the name of the parent catalog.
http://yourserver/yourproject/testcreate?type=catalog&path=TC_TT0&title=My%20sub%20catalog

this will assign a unique ID (the number 1 in the URL below) to the new
catalog and redirect to a Wiki edit page:
http://yourserver/yourproject/wiki/TC_TT0_TT1?action=edit&text=%3D%3D+My+sub+catalog+%3D%3D

that you can simply post the form to submit the page and thus create the
sub-catalog.


*Create a Test Case:*
Get the wollowing URL, where "path" is the name of the parent (sub-)catalog.
http://yourserver/yourproject/testcreate?type=testcase&path=TC_TT0_TT1&title=My%20new%20Test%20Case

this will assign a unique ID (the number 0 in the URL below) to the new test
case and redirect to a Wiki edit page:
http://yourserver/yourproject/wiki/TC_TT0_TT1_TC0?action=edit&text=%3D%3D+My+new+Test+Case+%3D%3D

that you can simply post the form to submit the page and thus create the
test case.


You can create a new Test Plan (e.g. for each nightly build)
programmatically as follows.

*Create a Test Plan from a specific catalog:*
Get the wollowing URL, where "path" is the name of the (sub-)catalog to
create the test plan against.
http://yourserver/yourproject/testcreate?type=testplan&path=TC_TT0&title=Test%20Plan%20for%2020100818

this will assign a unique ID (the number 1 in the URL below) to the new test
plan and redirect to displaying the Test Plan:
http://yourserver/yourproject/wiki/TC_TT0?planid=1

The Test Plan will contain all of the test cases in the specified catalog,
with a verdict of "Untested".


*Set a Test Case execution verdict, in the context of a Test Plan:*

Then, you can set the verdict for any test case in the plan, by means of the
following.

Get the following URL, where "id" is the Test Case ID and planid is the Test
Plan ID:
http://yourserver/yourproject/teststatusupdate?id=5&planid=1&status=SUCCESSFUL

The supported statuses are currently:
 - TO_BE_TESTED
 - SUCCESSFUL
 - FAILED


> - There are traceability relationships between TCs and
>tickets (more technically speaking, requirements
>tracked by tickets ;o) so that it's possible to let's say
>know what TCs should be run on changing a ticket
>state manually | by workflow or whatever .


The traceability is currently only from a Ticket top the corresponding Test
Case, but not the other way around.

You can open a Ticket and have a traceback to the (e.g. failed) Test Case as
follows:

*Open a Ticket on a Test Case:*
Whether you deploy TracTicketTemplatePlugin or not, you can get the
following URL, where testCaseNumber is the Test Case complete path, planid
is the Test Plan ID and planName is its name:

http://yourserver/yourproject/newticket?testCaseNumber=TC_TT0_TC0&planId=1&planName=Test%20Plan%20for%2020100818&description=Test%20Case:%20[wiki:TC_TT0_TC0],%20Test%20Plan:%20Test%20Plan%20for%2020100818%20(1)

this will open a Ticket edit page, with the Test Case-in-Test Plan hyperlink
already added to the ticket description (as Wiki page reference).
You can simply post the form to create the Ticket.

Probably you can directly POST a pre-built form to do everything with a
single call, but I didn't test this...


> - Tickets , test cases , build jobs and everything participate
>in a workflow , even if there are multiple tools
>involved.


I get the idea, but maybe orchestrating different tools in a workflow should
be the task of a different tool (a pro

Re: [Trac] New plugins: a true Test Case Manager

[Olemis Lang]

On Thu, Aug 12, 2010 at 9:05 AM, Roberto Longobardi  wrote:
> 2010/8/12 W. Martin Borgert 
>> Quoting "Roberto Longobardi" :
>>>
[...]

BTW, what's wrong  with the TC -> ticket mapping ? I've seen similar
mappings in other products ...

>>  - In the long run, it would be useful (but not trivial to
>>   implement) to differentiate between the definition of the
>>   tests and running a test campaign. An example: I have a
>>   software product, a plugin for Trac. Before doing a release
>>   "1.0" I have to perform some interactive tests. Therefore
>>   I start a new test campaign with svn revision 1234, where
>>   all test verdicts "Untested". Let's say, one or two tests
>>   failed. I conclude the test campaign unsuccessful. Now I
>>   have to fix the bugs and have to start a new test campaign,
>>   again with all test verdicts set to "Untested".
[...]
>
> About your last point, sure it is something I was thinking about, but as you
> say not easy to implement...
> Anyway, since the data model for the test results is completely separated
> from the one for the test cases themselves (i.e. the wiki table), it may not
> be SO hard to do.
> Help in this direction is welcome of course :-)

Here I go . I'm looking for some kind of tool like this one but
allowing to integrate TC management with CI tools . My environment
will look like this (i.e. part of this is done but the whole will work
in a near future, this is kind of what I want to get ;o).

  - TCs are *discovered* (generated ?) from
 source code (e.g. ) committed to a repos
 and they are using static code analysis,
 abstract models , XML specs ...
  - Test execution , etc , is performed by an external
 CI tool (currently Hudson + Bitten ...)
  - There are traceability relationships between TCs and
 tickets (more technically speaking, requirements
 tracked by tickets ;o) so that it's possible to let's say
 know what TCs should be run on changing a ticket
 state manually | by workflow or whatever .
  - Tickets , test cases , build jobs and everything participate
 in a workflow , even if there are multiple tools
 involved.

In this scenario , I'd like to be able to

  - Retrieve test results from CI tool using an API
  - Trigger build jobs on changing TC states or when
 aforementioned test campaigns (which will be
 mapped to in Hudson builds) are started ,
 explicitly executed .
  - ... well some other , but have no time to say it all

Q:
  - Considering scenarios like that one , is it possible
 to use interfaces so as , let's say , retrieve, synch
 TC history from other sources e.g. Hudson build history
  - How extensible are TC management components ?
  - Wouldn't it be nice to have workflow actions
 related to TC management ?
  - Wouldn't it be nice to provide RPC handlers to
 perform remote actions on TC specs (e.g. from
 inside VCS hooks ;o)

>> Keep on the good work, thanks!

Did I already say this ? ;o)

-- 
Regards,

Olemis.

Blog ES: http://simelo-es.blogspot.com/
Blog EN: http://simelo-en.blogspot.com/

Featured article:

-- 
You received this message because you are subscribed to the Google Groups "Trac 
Users" group.
To post to this group, send email to trac-us...@googlegroups.com.
To unsubscribe from this group, send email to 
trac-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/trac-users?hl=en.

Re: [Trac] New plugins: a true Test Case Manager

[Roberto Longobardi]

Hi Martin,
thanks for your review and for the kudos.

I have opened some enhancement requests on trac-hacks to keep track of your
comments.

Just one thing about openening a ticket on the current test case. I had
tried your way in the first place, but since we also deploy the
TracTicketTemplatePlugin,
it didn't work (this plugin replaces the whole ticket body with the template
upon page load).
So what I did is to patch the abov plugin to accept parameters on the URL
and substitute them in the template. You may have noticed that a couple of
such are passed to the newticket page. This way I actually end up with a
ticket that has a hyperlink (wiki) reference to the test case.

I'll try to make this patch go into the plugin main stream, and try to
support the environment where such plugin isn't found.

About your last point, sure it is something I was thinking about, but as you
say not easy to implement...
Anyway, since the data model for the test results is completely separated
from the one for the test cases themselves (i.e. the wiki table), it may not
be SO hard to do.

Help in this direction is welcome of course :-)

Thanks, ciao
Roberto


2010/8/12 W. Martin Borgert 

> Quoting "Roberto Longobardi" :
>
>> The first is a true test case manager plugin, since I couldn't find
>> anything suitable before, and my users wanted to have test case
>> management and bug tracking integrated in the same place:
>> http://trac-hacks.org/wiki/TestManagerForTracPlugin
>>
>
> OK, here my first observations after a short test:
>
>  - This testing framework is much more to my personal needs
>   than other, similar plugins I tried in the past. Good!
>
>  - It would be nice, if a test case result is not immediately
>   recorded on mouse click, but if there were a "save result"
>   button. It is too easy to click accidently on the coloured
>   bullet.
>
>  - Maybe the "Status change history" should be reversed,
>   showing the latest (= most important) result first.
>
>  - This is only a matter of taste, but I'm used to a
>   different terminology: A "test catalog" for one project
>   would be a "test suite" (TS), a sub catalog (or sub sub
>   catalog) would be a "test group" (TG), and a single test
>   would be a "test case" (TC).
>
>  - When one creates a ticket from a (failed) test, it would
>   be nice, if the new ticket already defaults to a useful
>   subject ("Failed test: Basic Sleep - Sleeping Monster").
>   This can be easily achieved by using the link:
>   .../newticket?summary=Failed%20test:%20Basic%20Sleep...
>
>  - It would be cool, if one could define a set of possible
>   verdicts. You have "Successful", "Untested", and "Failed",
>   which is OK for many purposes. But if you do tests
>   according to one of the many standards, you might have
>   different needs. E.g. ISO-9646 has five verdicts:
>
>   None = no result yet, untested
>   Pass = good, successful
>   Inconc = inconclusive, unclear
>   Fail = bad, failed
>   Error = there was an error in performing the test
>
>   If you test according to POSIX 1003.3 you have the verdicts:
>
>   PASS = good, successful
>   FAIL = bad, failed
>   UNRESOLVED = inconclusive, unclear
>   UNTESTED = no result yet, untested
>
>   Some testing frameworks add XFAIL (= expected fail), UPASS
>   (= unexpected pass) and UNSUPPORTED (= the implementation
>   under test doesn't support a feature) to the POSIX verdicts.
>
>  - In the long run, it would be useful (but not trivial to
>   implement) to differentiate between the definition of the
>   tests and running a test campaign. An example: I have a
>   software product, a plugin for Trac. Before doing a release
>   "1.0" I have to perform some interactive tests. Therefore
>   I start a new test campaign with svn revision 1234, where
>   all test verdicts "Untested". Let's say, one or two tests
>   failed. I conclude the test campaign unsuccessful. Now I
>   have to fix the bugs and have to start a new test campaign,
>   again with all test verdicts set to "Untested".
>
> Keep on the good work, thanks!
>
>

-- 
You received this message because you are subscribed to the Google Groups "Trac 
Users" group.
To post to this group, send email to trac-us...@googlegroups.com.
To unsubscribe from this group, send email to 
trac-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/trac-users?hl=en.

Re: [Trac] New plugins: a true Test Case Manager

[W. Martin Borgert]

Quoting "Roberto Longobardi" :

The first is a true test case manager plugin, since I couldn't find
anything suitable before, and my users wanted to have test case
management and bug tracking integrated in the same place:
http://trac-hacks.org/wiki/TestManagerForTracPlugin


OK, here my first observations after a short test:

 - This testing framework is much more to my personal needs
   than other, similar plugins I tried in the past. Good!

 - It would be nice, if a test case result is not immediately
   recorded on mouse click, but if there were a "save result"
   button. It is too easy to click accidently on the coloured
   bullet.

 - Maybe the "Status change history" should be reversed,
   showing the latest (= most important) result first.

 - This is only a matter of taste, but I'm used to a
   different terminology: A "test catalog" for one project
   would be a "test suite" (TS), a sub catalog (or sub sub
   catalog) would be a "test group" (TG), and a single test
   would be a "test case" (TC).

 - When one creates a ticket from a (failed) test, it would
   be nice, if the new ticket already defaults to a useful
   subject ("Failed test: Basic Sleep - Sleeping Monster").
   This can be easily achieved by using the link:
   .../newticket?summary=Failed%20test:%20Basic%20Sleep...

 - It would be cool, if one could define a set of possible
   verdicts. You have "Successful", "Untested", and "Failed",
   which is OK for many purposes. But if you do tests
   according to one of the many standards, you might have
   different needs. E.g. ISO-9646 has five verdicts:

   None = no result yet, untested
   Pass = good, successful
   Inconc = inconclusive, unclear
   Fail = bad, failed
   Error = there was an error in performing the test

   If you test according to POSIX 1003.3 you have the verdicts:

   PASS = good, successful
   FAIL = bad, failed
   UNRESOLVED = inconclusive, unclear
   UNTESTED = no result yet, untested

   Some testing frameworks add XFAIL (= expected fail), UPASS
   (= unexpected pass) and UNSUPPORTED (= the implementation
   under test doesn't support a feature) to the POSIX verdicts.

 - In the long run, it would be useful (but not trivial to
   implement) to differentiate between the definition of the
   tests and running a test campaign. An example: I have a
   software product, a plugin for Trac. Before doing a release
   "1.0" I have to perform some interactive tests. Therefore
   I start a new test campaign with svn revision 1234, where
   all test verdicts "Untested". Let's say, one or two tests
   failed. I conclude the test campaign unsuccessful. Now I
   have to fix the bugs and have to start a new test campaign,
   again with all test verdicts set to "Untested".

Keep on the good work, thanks!

--
You received this message because you are subscribed to the Google Groups "Trac 
Users" group.
To post to this group, send email to trac-us...@googlegroups.com.
To unsubscribe from this group, send email to 
trac-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/trac-users?hl=en.

Re: [Trac] New plugins: a true Test Case Manager

[W. Martin Borgert]

Hi Roberto,

Quoting "Roberto Longobardi" :

The first is a true test case manager plugin, since I couldn't find
anything suitable before, and my users wanted to have test case
management and bug tracking integrated in the same place:
http://trac-hacks.org/wiki/TestManagerForTracPlugin


The screenshots look very promising. I found the existing plugins
for test management not really up to my needs, so your approach
is very welcome.

Just two hints for the page
http://trac-hacks.org/wiki/TestManagerForTracPlugin:

 1. For some reason I cannot download the .ppt attachments.
I get an error message:
"The requested URL /attachment/wiki/TestManagerForTracPlugin/Test
Manager plugin for Trac - User Guide part 1.ppt was not found on
this server."
I will download the files from sourceforge for now.
But wouldn't it be better to have the complete description
directly in the wiki page? This way people could improve it
and you would have version control.

 2. To directly display the screenshots, you may use a macro:
[[Image(screen2.JPG)]]

Cheers

--
You received this message because you are subscribed to the Google Groups "Trac 
Users" group.
To post to this group, send email to trac-us...@googlegroups.com.
To unsubscribe from this group, send email to 
trac-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/trac-users?hl=en.

[Trac] New plugins: a true Test Case Manager and a Resource planning and reservation

[Roberto Longobardi]

Hi,
just to let you know I have uploaded a couple of new plugins on the
trac-hacks site that I developed for my trac users.

The first is a true test case manager plugin, since I couldn't find
anything suitable before, and my users wanted to have test case
management and bug tracking integrated in the same place:
http://trac-hacks.org/wiki/TestManagerForTracPlugin

The second one is a simple plugin to allow for planning and reserving
the future use of resources on a time schedule. We are using it to
reserve consumable test data (employees in our test database that once
"tested" cannot be used anymore since the test changes their data),
but it could be used as well to plan and coordinate the use of test
machines, for example.
Here is the page on trac-hacks:
http://trac-hacks.org/wiki/ResourceReservationPlugin

Both are currently only tested on Trac 0.11 and python 2.5.

Feedback and bugs/requests for features are both really appreciated,
thanks!
Roberto

-- 
You received this message because you are subscribed to the Google Groups "Trac 
Users" group.
To post to this group, send email to trac-us...@googlegroups.com.
To unsubscribe from this group, send email to 
trac-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/trac-users?hl=en.