Re: Common DBI Driver Test Suite - Requirements
Am 25.09.2013 um 19:27 schrieb Tim Bunce : > On Wed, Sep 25, 2013 at 07:02:07PM +0200, H.Merijn Brand wrote: >> On Wed, 25 Sep 2013 17:28:04 +0100, Tim Bunce >> wrote: >> >>>G9. end users can find the level of DBI API compliance of a driver >>>i.e., by looking at the test configuration for the driver >>>to see what areas of functionality are configured to be skipped. >> >> I have a hard time parsing that. > > Drivers have various limitations, things they don't, or can't, support. > It would be unreasonable for them to fail DBIT for those reasons. > So there needs to be a way for those drivers to tell DBIT that > certain tests should be skipped. Riba put that into a R4: It should be possible to configure which test(case) should be run and which not. This should be possible in a manner which avoids asking cpantesters 100 y/n questions ^^ Maybe this belongs to that point (Riba should be able to explain better). For me, a G10 is interesting (handling SQL::Statement as a DBD) G10: Provide tools/infrastructure to combines several test cases in particular order into tests to allow workflows being tested and stress tests by looping can be performed. > And not just drivers. Things like proxies will need to influence > what's expected to work. This is a key area. > > What G9 is trying to express is that that information (which tests to > skip) is useful for end users, and probably applications as well. > So we should aim to design the system in a way that make that > information available in a useful way. > > Some of that might end up being handled by $dbh->get_info() but there's > bound to be some that isn't. > > >>>S4. DBIT is not meant to test the underlying database. >>>If a driver implements a database itself then it'll >>>need it's own tests to provide good coverage of that. >> >> DBIT will offer the interface to test the underlying database. What is >> important here, as DBIT will offer the possibility to test under >> different circumstances (like with SQL::Nano or under DBI::Proxy) >> without having to rewrite all tests over and over again. >> These test will reside in the test suite of the DBD, not in the test >> suite of DBIT > > Yes. Or, for drivers that want to share tests, those tests could live in > modules installed by a distro that the driver declares it's dependant on. I would prefer the "distributed separately" way. It doesn't have to be dozen of independent distributions, there could be a few thematically grouped ones. Like some for ANSI SQL, some for proxy specific ones (like spreading some requests to see how the scaling is?), …) Cheers -- Jens Rehsack pkgsrc, Perl5 rehs...@cpan.org
Re: Common DBI Driver Test Suite - Requirements
Am 25.09.2013 um 19:02 schrieb H.Merijn Brand : > On Wed, 25 Sep 2013 17:28:04 +0100, Tim Bunce > wrote: > >> Hi all. >> [...] > >>G8. Provide tools that drivers can use/extend for themselves. >>I'm thinking specifically here of the creation of test files >>with combinations of env vars and other settings. >>E.g., test DBD::CSV with Text::CSV and Text::CSV_XS > > As a goal, this is fine. For DBD::CSV, this falls under G7: It will use > Text::CSV_PP (from Text::CSV) under DBI::PurePerl and Text::CSV_XS > under DBI (XS) Don't discuss implementation details here. Keep this argument in mind and come back on that later. Then I tell you why I think you have a 3rd case and so need a G8 :D Cheers -- Jens Rehsack pkgsrc, Perl5 rehs...@cpan.org
Re: Common DBI Driver Test Suite - Requirements
On Wed, Sep 25, 2013 at 07:02:07PM +0200, H.Merijn Brand wrote: > On Wed, 25 Sep 2013 17:28:04 +0100, Tim Bunce > wrote: > > > G9. end users can find the level of DBI API compliance of a driver > > i.e., by looking at the test configuration for the driver > > to see what areas of functionality are configured to be skipped. > > I have a hard time parsing that. Drivers have various limitations, things they don't, or can't, support. It would be unreasonable for them to fail DBIT for those reasons. So there needs to be a way for those drivers to tell DBIT that certain tests should be skipped. And not just drivers. Things like proxies will need to influence what's expected to work. This is a key area. What G9 is trying to express is that that information (which tests to skip) is useful for end users, and probably applications as well. So we should aim to design the system in a way that make that information available in a useful way. Some of that might end up being handled by $dbh->get_info() but there's bound to be some that isn't. > > S4. DBIT is not meant to test the underlying database. > > If a driver implements a database itself then it'll > > need it's own tests to provide good coverage of that. > > DBIT will offer the interface to test the underlying database. What is > important here, as DBIT will offer the possibility to test under > different circumstances (like with SQL::Nano or under DBI::Proxy) > without having to rewrite all tests over and over again. > These test will reside in the test suite of the DBD, not in the test > suite of DBIT Yes. Or, for drivers that want to share tests, those tests could live in modules installed by a distro that the driver declares it's dependant on. Tim.
Re: Common DBI Driver Test Suite - Requirements
On Wed, 25 Sep 2013 17:28:04 +0100, Tim Bunce
wrote:
> Hi all.
>
> This email aims to start a discussion around clarifying the goals, scope
> and requirements of a Common DBI Driver Test Suite, including learning
> from the great work already done on DBI::Test.
>
> Once we've kicked these around to the point were there's reasonable
> agreement, then we'll start a new thread to discuss design/implementation.
> For this thread though, goals, scope, and requirements are the focus.
>
> First, let's spell out some basics...
>
> Stakeholders: (who is affected by this work)
>
> - driver developers
> - cpantesters
> - end users
>
> Terminology:
>
> * We'll use DBIT as the generic term for this thing, whatever it
> turns out to look like or be called.
>
> Goals: ("how will we know if this project is or is not a success")
>
> G1. driver developers improve their compliance with the DBI API
> and so improve consistency and portability of code.
> This is what it's all about!
Yes, my primary goal
> G2. driver developers adopt DBIT as a free test suite with good
> coverage of the DBI API. This is the pay-back _for_ developers.
Yes, also my secondary goal. Going the right way here :) :)
> G3. driver developers write and share reusable generic tests
> (they'll still need to write their own driver-specific tests).
> This is the pay-back _from_ developers.
Most likely. To be honest I never thought about it that way
> G4. end users won't be affected by DBIT evolution causing install
> failures i.e., DBIT runs as 'author testing', *at least for now*.
> This is our promise to end users not to ruin their day.
100% agree. for now it is optional
> G5. be usable for the widest range of DBI drivers, including non-SQL.
> This is a promise to be inclusive.
+1
> G6. enable testing with proxies like DBD::Gofer.
> This is a promise to be flexible.
+1
> G7. enable testing with DBI::PurePerl for pure-perl drivers.
> This is a promise to support fatpackers and perl6 ;)
+2
> G8. Provide tools that drivers can use/extend for themselves.
> I'm thinking specifically here of the creation of test files
> with combinations of env vars and other settings.
> E.g., test DBD::CSV with Text::CSV and Text::CSV_XS
As a goal, this is fine. For DBD::CSV, this falls under G7: It will use
Text::CSV_PP (from Text::CSV) under DBI::PurePerl and Text::CSV_XS
under DBI (XS)
> G9. end users can find the level of DBI API compliance of a driver
> i.e., by looking at the test configuration for the driver
> to see what areas of functionality are configured to be skipped.
I have a hard time parsing that.
> [I thought of adding an extra goal: "other code can determine the
> limitations/quirks of a driver i.e., by looking at the test
> configuration for the driver to see how it's configured." but to be
> generic I think that would require a connected $dbh to determine server
> version/config. It's worth keeping in mind though in terms of having an
> object that represents the server that can be queried for info needed by
> tests. Perhaps that'll end up as extensions to $dbh->get_info().]
>
>
> Scope: (the boundaries and deliverables of the project)
>
> S1. the DBIT will be a separate distribution.
+1. It already is, and there has never been the intention to bundle it
in any other distribution
> S3. the DBI won't have a mandatory dependency on DBIT,
> for now at least, driver developers are the priority.
+100
> S4. DBIT is not meant to test the underlying database.
> If a driver implements a database itself then it'll
> need it's own tests to provide good coverage of that.
DBIT will offer the interface to test the underlying database. What is
important here, as DBIT will offer the possibility to test under
different circumstances (like with SQL::Nano or under DBI::Proxy)
without having to rewrite all tests over and over again.
These test will reside in the test suite of the DBD, not in the test
suite of DBIT
> [Regarding S4: We could enable individual distros to install modules
> that add DBIT tests that'll be included in DBIT runs in certain cases,
> e.g., for certain drivers. I want to avoid running lots of SQL tests
> for drivers where that doesn't make sense. I'm sure we can find a way to
> do that but still allow sharing of SQL tests for SqlEngine based DBDs.]
>
>
> Requirements:
>
> R1. be easy for driver developers to adopt,
> so existing drivers migrate to using it.
+1
> R2. be easy for driver developers to extend/contribute to,
> so driver developers contribute to it.
+1
> R3. work well with cpantesters,
> so we get good coverage (perhaps extend Test::Database)
+1
> So, any thoughts on these goals, scope, and requirements?
>
> (Please avoid design and implementation d
Common DBI Driver Test Suite - Requirements
Hi all.
This email aims to start a discussion around clarifying the goals, scope
and requirements of a Common DBI Driver Test Suite, including learning
from the great work already done on DBI::Test.
Once we've kicked these around to the point were there's reasonable
agreement, then we'll start a new thread to discuss design/implementation.
For this thread though, goals, scope, and requirements are the focus.
First, let's spell out some basics...
Stakeholders: (who is affected by this work)
- driver developers
- cpantesters
- end users
Terminology:
* We'll use DBIT as the generic term for this thing, whatever it
turns out to look like or be called.
Goals: ("how will we know if this project is or is not a success")
G1. driver developers improve their compliance with the DBI API
and so improve consistency and portability of code.
This is what it's all about!
G2. driver developers adopt DBIT as a free test suite with good
coverage of the DBI API. This is the pay-back _for_ developers.
G3. driver developers write and share reusable generic tests
(they'll still need to write their own driver-specific tests).
This is the pay-back _from_ developers.
G4. end users won't be affected by DBIT evolution causing install failures
i.e., DBIT runs as 'author testing', *at least for now*.
This is our promise to end users not to ruin their day.
G5. be usable for the widest range of DBI drivers, including non-SQL.
This is a promise to be inclusive.
G6. enable testing with proxies like DBD::Gofer.
This is a promise to be flexible.
G7. enable testing with DBI::PurePerl for pure-perl drivers.
This is a promise to support fatpackers and perl6 ;)
G8. Provide tools that drivers can use/extend for themselves.
I'm thinking specifically here of the creation of test files
with combinations of env vars and other settings.
E.g., test DBD::CSV with Text::CSV and Text::CSV_XS
G9. end users can find the level of DBI API compliance of a driver
i.e., by looking at the test configuration for the driver
to see what areas of functionality are configured to be skipped.
[I thought of adding an extra goal: "other code can determine the
limitations/quirks of a driver i.e., by looking at the test
configuration for the driver to see how it's configured." but to be
generic I think that would require a connected $dbh to determine server
version/config. It's worth keeping in mind though in terms of having an
object that represents the server that can be queried for info needed by
tests. Perhaps that'll end up as extensions to $dbh->get_info().]
Scope: (the boundaries and deliverables of the project)
S1. the DBIT will be a separate distribution.
S3. the DBI won't have a mandatory dependency on DBIT,
for now at least, driver developers are the priority.
S4. DBIT is not meant to test the underlying database.
If a driver implements a database itself then it'll
need it's own tests to provide good coverage of that.
[Regarding S4: We could enable individual distros to install modules
that add DBIT tests that'll be included in DBIT runs in certain cases,
e.g., for certain drivers. I want to avoid running lots of SQL tests
for drivers where that doesn't make sense. I'm sure we can find a way to
do that but still allow sharing of SQL tests for SqlEngine based DBDs.]
Requirements:
R1. be easy for driver developers to adopt,
so existing drivers migrate to using it.
R2. be easy for driver developers to extend/contribute to,
so driver developers contribute to it.
R3. work well with cpantesters,
so we get good coverage (perhaps extend Test::Database)
So, any thoughts on these goals, scope, and requirements?
(Please avoid design and implementation details for now. If you have any
concerns that relate to design and implementation that you feel need to
be considered now, try to express them in terms of the goals, scope, or
requirements that would lead to that design.)
Tim.
