Re: Common DBI Driver Test Suite - Requirements
On 26/09/13 23:29, Tim Bunce wrote:> On Thu, Sep 26, 2013 at 05:55:56PM +0100, Martin J. Evans wrote: On 25/09/13 17:28, Tim Bunce wrote: G2. driver developers adopt DBIT as a free test suite with good coverage of the DBI API. This is the pay-back _for_ developers. I think one thing many DBD test suites could benefit from is wrappers around many of the DBI methods that wrap that method in tests e.g., when execute is called, did it return undef, a true value (0E0 or a number > 0) or -1 depending on whether it is a select statement or not. If test suites were converted to use those I'm sure we'd find quite a few issues in DBDs but still using existing test code. I don't follow you here Martin. DBIT will implement detailed testing for the return value of execute. All I meant was for DBIT to provide method wrappers which incorporate the tests so when DBD writers write their DBD-specific tests they get the method tests for free. G9. end users can find the level of DBI API compliance of a driver i.e., by looking at the test configuration for the driver to see what areas of functionality are configured to be skipped. This in particular is something I'd like to see and expand on. [...] a capability system beyond what get_info provides. get_info is pretty much ODBC's SQLGetInfo and few drivers beyond DBD::ODBC really support it that well. I'm expecting that one of the side-effects of DBIT will be a great improvement in support for get_info by drivers. That'll be a win for all. It definitely is a win. I can't even begin to tell you the pain I had writing DBD neutral code just for ODBC, Oracle, DB2, mysql and postgres. We persevered for some time but ended up with so much of the class methods overridden we ended up with effectively 5 different applications - although it did not look like it from the front. Once you include the SQL as well - it really is not fun and although I don't use DBIx::Class myself, it must be quite a piece of code. It will be so much nicer to do: if ($h->get_info(parameters_need_colons)) { do this } else { do this } than if ($h->{driver} eq 'Oracle' || $h->{driver} eq 'postres') blah blah It's also worth noting that we can define our own meanings for certain values passed to get_info. With surprising foresight, sometime around 1998 I think, I arranged for the get_info values 9000 thru to be reserved for the Perl DBI in the ISO standard registry: Registry of Values for the SQL Standard (ANSI X3.135 and ISO/IEC 9075) With especial attention to values related to SQL/CLI ([ANSI/]ISO/IEC 9075-3) I actually reserved value ranges for just about all of the SQL CLI functions. http://www.nntp.perl.org/group/perl.dbi.users/2007/04/msg31285.html Nice one. [...] If I put my mind to it (and looked at my code from years ago when I was involved in writing to multiple DBDs from the same application) I could proably come up with a much longer list - Peter probably could too. When the time comes we'll probably need one :) I know this is not DBIT as such and you might see it as a distraction (if you do, ignore) but I think it would be worth while. Very. I think it's important. Excellent. I will be happy to help with this as I can. R3. work well with cpantesters, so we get good coverage (perhaps extend Test::Database) As a side note, I as going to add support to Test::Database for DBD::ODBC because I thought it might get me more smokers actually running the tests. Then I discovered it needed create database support, and that was not going to happen with ODBC as each database has totally different syntax for that and some databases need very high level permissions to create a database. As I mentioned elsewhere, I suspect we'll want to extend or fork Test::Database to bend it to our needs. These seem like worthwhile goals. Whether I can be of any help, I don't know, but I'll at least try and keep up with the discussion and provide any useful feedback. Thanks Martin. Tim. Martin
Re: Best way to retire old code for unsupported database versions
On 27/09/13 03:52, Greg Sabino Mullane wrote: -BEGIN PGP SIGNED MESSAGE- Hash: RIPEMD160 what to do about having something around that you can refer people who want to run 4.x (???!!!). Is a simple information tidbit "if you want to use xxx, you use version x.y.z version of DBD::mysql" ? What have others done about this issue? Yes, it is basically as simple as that. Some thoughts: * Make your abandonment of old versions in large chunks, don't simply phase out older versions over time. In other words, consolidate the pain into as few releases as possible. Ideally, one every 10 years. :) * Have a version or two with lots of warnings in the docs before making the change. * Make the change at a major version bump. Then you can tell people they need version x, rather than x.y.z * This is also a good time to break other things - such as requiring a newer version of DBI. Consolidate that pain! * Provide instructions in the docs about how to reach the old versions. Have the frozen version available somewhere other than CPAN too. * Expect people who have been living off of those #ifdefs for years to suddenly crawl out of the woodwork after you flip the switch and complain. Anticipate their complaints in your docs so you can simply point them there. (MySQL 4.x? I know places still running 3.x!) Everything Greg said ++, plus: * when you make the break try and ensure your RT queue is clear of at least bugs or you'll get people saying they can't upgrade to get a bug fix because you withdrew support for their mysql version. * try and catch someone trying to build against old client libs and output a meaningful message rather than just have the compile fail - preferably do this in your Makefile.PL or Build script and with the correct exit status you'll get a NA on a smoker instead of a Fail. * as well as DBI, think about what Perl version you require - especially wrt unicode support although I see the latest DBI is 5.8.1 already. Martin
Re: Common DBI Driver Test Suite - Requirements
Am 27.09.2013 um 00:03 schrieb Tim Bunce : > On Wed, Sep 25, 2013 at 08:36:22PM +0200, Jens Rehsack wrote: > >> For me, a G10 is interesting (handling SQL::Statement as a DBD) >> >> G10: Provide tools/infrastructure to combines several test cases in >> particular order into tests to allow workflows being tested >> and stress tests by looping can be performed. > > Also here. I'm not quite sure what problem this is trying to address. This is more or less an idea I had during test development. For a lot of tests, we write some sequences of statements to finally prove one or a few things working fine (at least for DBD::CSV, DBD::DBM and a lot of SQL::Statement tests - but IIRC, this is true for SQLite, too). So for testing whether UPDATE/DELETE works, we create some tables, do some inserts and after that, modify them. For SELECT testing, same. For ChopBlank testing (via select, of course), similar. For JOIN testing, more tables - but finally the same. This goes on … (different edge cases to test, but finally all the same). For SQL::Statement I added a performance check )for being run through the profilers we have) which is assembled from one or more of the above tests. But by hand. If there is a design of combining test cases to a sequence of tests (eg. create tables -> fill data -> select -> update -> delete -> drop), it could help simplifying test cases and allow put loops around sequences to redo some test sequence for 100 times for profiling. The design finally choosen for this goal might be a composite for test cases - but the test case design shall allow doing that. Cheers -- Jens Rehsack rehs...@gmail.com
RE: Best way to retire old code for unsupported database versions
Yep one of the more unplesnt parts of being a DBDer, Looking back I had to drop a few #ifdefs when I was the more active with DBD::Oracle. I just made up a chart on the POD http://search.cpan.org/~pythian/DBD-Oracle-1.66/lib/DBD/Oracle.pm#WHICH_VERSION_OF_DBD::ORACLE_IS_FOR_ME? I my case I sort of just followed what the Oracle policy was. Support two versions back only. However nobody has been updating the table:) Really have to find a week or two to get back an lay with DBD::Oracle Cheers John > Date: Fri, 27 Sep 2013 09:28:51 +0100 > From: boh...@ntlworld.com > To: dbi-dev@perl.org > Subject: Re: Best way to retire old code for unsupported database versions > > On 27/09/13 03:52, Greg Sabino Mullane wrote: > > -BEGIN PGP SIGNED MESSAGE- > > Hash: RIPEMD160 > > > > > >> what to do about having something around that you can refer people who > >> want to run 4.x (???!!!). Is a simple information tidbit "if you > >> want to use xxx, you use version x.y.z version of DBD::mysql" ? > >> What have others done about this issue? > > > > Yes, it is basically as simple as that. Some thoughts: > > > > * Make your abandonment of old versions in large chunks, don't simply > > phase out older versions over time. In other words, consolidate the > > pain into as few releases as possible. Ideally, one every 10 years. :) > > > > * Have a version or two with lots of warnings in the docs before > > making the change. > > > > * Make the change at a major version bump. Then you can tell people > > they need version x, rather than x.y.z > > > > * This is also a good time to break other things - such as requiring > > a newer version of DBI. Consolidate that pain! > > > > * Provide instructions in the docs about how to reach the old versions. > > Have the frozen version available somewhere other than CPAN too. > > > > * Expect people who have been living off of those #ifdefs for years to > > suddenly crawl out of the woodwork after you flip the switch and complain. > > Anticipate their complaints in your docs so you can simply point them there. > > > > (MySQL 4.x? I know places still running 3.x!) > > Everything Greg said ++, plus: > > * when you make the break try and ensure your RT queue is clear of at least > bugs or you'll get people saying they can't upgrade to get a bug fix because > you withdrew support for their mysql version. > > * try and catch someone trying to build against old client libs and output a > meaningful message rather than just have the compile fail - preferably do > this in your Makefile.PL or Build script and with the correct exit status > you'll get a NA on a smoker instead of a Fail. > > * as well as DBI, think about what Perl version you require - especially wrt > unicode support although I see the latest DBI is 5.8.1 already. > > Martin
Making trace show more than 2 elements
Is there an undocumented magic variable that will make DBI trace mode show more than the first two arguments to a function like selectrow_array() or similar? Cheers, Yves -- perl -Mre=debug -e "/just|another|perl|hacker/"
Re: Making trace show more than 2 elements
On 27/09/13 15:37, demerphq wrote: Is there an undocumented magic variable that will make DBI trace mode show more than the first two arguments to a function like selectrow_array() or similar? Cheers, Yves Works for me: DBI_TRACE=15=x.log perl -Iblib/lib/ -Iblib/arch/ -MDBI -le 'my $h = DBI->connect("dbi:ODBC:xxx","xx","xx"); my $x = $h->selectrow_arrayref(q/select * from mje where a= ?/, undef, "a");' trace shows 3 arguments: -> selectrow_arrayref for DBD::ODBC::db (DBI::db=HASH(0x9385508)~0x938542c 'select * from mje where a= ?' undef 'a') thr#9215008 There is also DBIx::Log4perl and DBIx::LogAny too if you want to trace just method calls and arguments. Martin
Re: Making trace show more than 2 elements
On 27 September 2013 17:08, Martin J. Evans wrote: > On 27/09/13 15:37, demerphq wrote: >> >> Is there an undocumented magic variable that will make DBI trace mode >> show more than the first two arguments to a function like >> selectrow_array() or similar? >> >> Cheers, >> Yves >> > > Works for me: > > DBI_TRACE=15=x.log perl -Iblib/lib/ -Iblib/arch/ -MDBI -le 'my $h = > DBI->connect("dbi:ODBC:xxx","xx","xx"); my $x = > $h->selectrow_arrayref(q/select * from mje where a= ?/, undef, "a");' > > trace shows 3 arguments: > > -> selectrow_arrayref for DBD::ODBC::db > (DBI::db=HASH(0x9385508)~0x938542c 'select * from mje where a= ?' undef 'a') > thr#9215008 > > There is also DBIx::Log4perl and DBIx::LogAny too if you want to trace just > method calls and arguments. Gah, that outputs too much junk at the same time I want to control it at DBI->trace(1) level. IOW, I dont want low level debug on the DBI calls, I want the output from trace(1) but with all the arguments. I can change how much of a given string I see with $DBI::neat_maxlen but there doesnt seem to a be a $DBI::neat_list_maxitems or equivalent cheers, Yves -- perl -Mre=debug -e "/just|another|perl|hacker/"
Re: Making trace show more than 2 elements
On 27 September 2013 17:29, demerphq wrote: > On 27 September 2013 17:08, Martin J. Evans wrote: >> On 27/09/13 15:37, demerphq wrote: >>> >>> Is there an undocumented magic variable that will make DBI trace mode >>> show more than the first two arguments to a function like >>> selectrow_array() or similar? >>> >>> Cheers, >>> Yves >>> >> >> Works for me: >> >> DBI_TRACE=15=x.log perl -Iblib/lib/ -Iblib/arch/ -MDBI -le 'my $h = >> DBI->connect("dbi:ODBC:xxx","xx","xx"); my $x = >> $h->selectrow_arrayref(q/select * from mje where a= ?/, undef, "a");' >> >> trace shows 3 arguments: >> >> -> selectrow_arrayref for DBD::ODBC::db >> (DBI::db=HASH(0x9385508)~0x938542c 'select * from mje where a= ?' undef 'a') >> thr#9215008 >> >> There is also DBIx::Log4perl and DBIx::LogAny too if you want to trace just >> method calls and arguments. > > Gah, that outputs too much junk at the same time > > I want to control it at DBI->trace(1) level. IOW, I dont want low > level debug on the DBI calls, I want the output from trace(1) but with > all the arguments. > > I can change how much of a given string I see with $DBI::neat_maxlen > but there doesnt seem to a be a $DBI::neat_list_maxitems or > equivalent Do forgive me for being rude, I didnt mean to "gah" at you. Thanks for replying... :-) Yves -- perl -Mre=debug -e "/just|another|perl|hacker/"
Re: Making trace show more than 2 elements
On 27/09/13 16:29, demerphq wrote:> On 27 September 2013 17:08, Martin J. Evans wrote: On 27/09/13 15:37, demerphq wrote: Is there an undocumented magic variable that will make DBI trace mode show more than the first two arguments to a function like selectrow_array() or similar? Cheers, Yves Works for me: DBI_TRACE=15=x.log perl -Iblib/lib/ -Iblib/arch/ -MDBI -le 'my $h = DBI->connect("dbi:ODBC:xxx","xx","xx"); my $x = $h->selectrow_arrayref(q/select * from mje where a= ?/, undef, "a");' trace shows 3 arguments: -> selectrow_arrayref for DBD::ODBC::db (DBI::db=HASH(0x9385508)~0x938542c 'select * from mje where a= ?' undef 'a') thr#9215008 There is also DBIx::Log4perl and DBIx::LogAny too if you want to trace just method calls and arguments. Gah, that outputs too much junk at the same time Which outputs too much junk at the same time? level 15 tracing or DBIx::Log4perl? I want to control it at DBI->trace(1) level. IOW, I dont want low level debug on the DBI calls, I want the output from trace(1) but with all the arguments. You didn't mention trace level 1 before. Now I see what you mean: DBI_TRACE=1=x.log perl -Iblib/lib/ -Iblib/arch/ -MDBI -le 'my $h = DBI->connect("dbi:ODBC:xx","xx","xx"); my $x = $h->selectrow_arrayref(q/select * from mje where a= ?/, undef, "a");' <- selectrow_arrayref('select * from mje where a= ?', undef, ...)= ( [ 'a' 'b' ] ) [1 items] at -e line 1 I can change how much of a given string I see with $DBI::neat_maxlen but there doesnt seem to a be a $DBI::neat_list_maxitems or equivalent cheers, Yves No, there isn't. I wouldn't imagine it would be too hard to add though (not that I looked). Martin
Re: Making trace show more than 2 elements
On 27 September 2013 17:41, Martin J. Evans wrote: > On 27/09/13 16:29, demerphq wrote:> On 27 September 2013 17:08, Martin J. > Evans wrote: >>> >>> On 27/09/13 15:37, demerphq wrote: Is there an undocumented magic variable that will make DBI trace mode show more than the first two arguments to a function like selectrow_array() or similar? Cheers, Yves >>> >>> Works for me: >>> >>> DBI_TRACE=15=x.log perl -Iblib/lib/ -Iblib/arch/ -MDBI -le 'my $h = >>> DBI->connect("dbi:ODBC:xxx","xx","xx"); my $x = >>> $h->selectrow_arrayref(q/select * from mje where a= ?/, undef, "a");' >>> >>> trace shows 3 arguments: >>> >>> -> selectrow_arrayref for DBD::ODBC::db >>> (DBI::db=HASH(0x9385508)~0x938542c 'select * from mje where a= ?' undef >>> 'a') >>> thr#9215008 >>> >>> There is also DBIx::Log4perl and DBIx::LogAny too if you want to trace >>> just >>> method calls and arguments. >> >> >> Gah, that outputs too much junk at the same time > > > Which outputs too much junk at the same time? level 15 tracing or > DBIx::Log4perl? Level 15 tracing. I do my best to avoid log4perl (but that is a religious discussion best left to other forums :-) > >> I want to control it at DBI->trace(1) level. IOW, I dont want low >> level debug on the DBI calls, I want the output from trace(1) but with >> all the arguments. > > > You didn't mention trace level 1 before. Now I see what you mean: > > DBI_TRACE=1=x.log perl -Iblib/lib/ -Iblib/arch/ -MDBI -le 'my $h = > DBI->connect("dbi:ODBC:xx","xx","xx"); my $x = > $h->selectrow_arrayref(q/select * from mje where a= ?/, undef, "a");' > > <- selectrow_arrayref('select * from mje where a= ?', undef, ...)= ( [ > 'a' 'b' ] ) [1 items] at -e line 1 > > >> I can change how much of a given string I see with $DBI::neat_maxlen >> but there doesnt seem to a be a $DBI::neat_list_maxitems or >> equivalent >> >> cheers, >> Yves >> >> > > No, there isn't. I wouldn't imagine it would be too hard to add though (not > that I looked). Ok, thanks. Maybe ill work out a patch. Yves -- perl -Mre=debug -e "/just|another|perl|hacker/"
Re: Making trace show more than 2 elements
On 27/09/13 17:12, demerphq wrote: On 27 September 2013 17:41, Martin J. Evans wrote: On 27/09/13 16:29, demerphq wrote:> On 27 September 2013 17:08, Martin J. Evans wrote: On 27/09/13 15:37, demerphq wrote: Is there an undocumented magic variable that will make DBI trace mode show more than the first two arguments to a function like selectrow_array() or similar? Cheers, Yves Works for me: DBI_TRACE=15=x.log perl -Iblib/lib/ -Iblib/arch/ -MDBI -le 'my $h = DBI->connect("dbi:ODBC:xxx","xx","xx"); my $x = $h->selectrow_arrayref(q/select * from mje where a= ?/, undef, "a");' trace shows 3 arguments: -> selectrow_arrayref for DBD::ODBC::db (DBI::db=HASH(0x9385508)~0x938542c 'select * from mje where a= ?' undef 'a') thr#9215008 There is also DBIx::Log4perl and DBIx::LogAny too if you want to trace just method calls and arguments. Gah, that outputs too much junk at the same time Which outputs too much junk at the same time? level 15 tracing or DBIx::Log4perl? Level 15 tracing. I do my best to avoid log4perl (but that is a religious discussion best left to other forums :-) There is DBIx::LogAny too. Martin
Problem with ReadOnly attribute
The DBI pod says for ReadOnly: http://search.cpan.org/~timb/DBI-1.628/DBI.pm#ReadOnly " An application can set the ReadOnly attribute of a handle to a true value to indicate that it will not be attempting to make any changes using that handle or any children of it. Note that the exact definition of 'read only' is rather fuzzy. For more details see the documentation for the driver you're using. " I have small issue with DBI's ReadOnly attribute. In ODBC it is mapped to SQLSetConnectAttr(SQL_ACCESS_MODE) and it just /suggests/ to the driver you'll be readonly and not making changes in case this allows the driver to optimise things. However, a driver may not support SQL_ACCESS_MODE in which case it returns a SQL_SUCCESS_WITH_INFO and 01S02 - Option value changed, which is described as "the driver did not support the value specified in ValuePtr and substituted a similar value." I don't see a way in DBI's pod to report this back i.e., there is no return value mentioned. Should I just issue a warning if I get "option value changed - 01S02"? The reason this has come up is that I have the following test: # ReadOnly { $dbh->{ReadOnly} = 1; is($dbh->{ReadOnly}, 1, 'ReadOnly set'); $dbh->{ReadOnly} = 0; is($dbh->{ReadOnly}, 0, 'ReadOnly cleared'); } and the SQLite ODBC driver is failing it because any setting of SQL_ACCESS_MODE returns SQL_SUCCESS_WITH_INFO, option value changed, 01S02 and when you go to retrieve it back it is not what you set. Martin -- Martin J. Evans Wetherby, UK
Re: Problem with ReadOnly attribute
On Fri, Sep 27, 2013 at 08:09:16PM +0100, Martin J. Evans wrote: > > However, a driver may not support SQL_ACCESS_MODE in which case it > returns a SQL_SUCCESS_WITH_INFO and 01S02 - Option value changed, > which is described as "the driver did not support the value > specified in ValuePtr and substituted a similar value." > > I don't see a way in DBI's pod to report this back i.e., there is no > return value mentioned. > > Should I just issue a warning if I get "option value changed - 01S02"? > The reason this has come up is that I have the following test: > > $dbh->{ReadOnly} = 1; > is($dbh->{ReadOnly}, 1, 'ReadOnly set'); > $dbh->{ReadOnly} = 0; > is($dbh->{ReadOnly}, 0, 'ReadOnly cleared'); > > and the SQLite ODBC driver is failing it because any setting of > SQL_ACCESS_MODE returns SQL_SUCCESS_WITH_INFO, option value changed, > 01S02 and when you go to retrieve it back it is not what you set. By "issue a warning" do you mean set err to "0", errstr to "option value changed..." and state to "01S02"? If so, yes, that seems like the right thing to do. The test can then be updated to check for that. I'd be happy for you to patch the DBI docs along those lines. Whether $dbh->{ReadOnly} should remain false after an attempt to set it true has 'failed' seems more tricky. If it's false then other code can't tell that the application declared itself to not want to make changes. I'm inclined to let it stay true. Tim.
Re: Problem with ReadOnly attribute
On 27/09/2013 21:01, Tim Bunce wrote: On Fri, Sep 27, 2013 at 08:09:16PM +0100, Martin J. Evans wrote: However, a driver may not support SQL_ACCESS_MODE in which case it returns a SQL_SUCCESS_WITH_INFO and 01S02 - Option value changed, which is described as "the driver did not support the value specified in ValuePtr and substituted a similar value." I don't see a way in DBI's pod to report this back i.e., there is no return value mentioned. Should I just issue a warning if I get "option value changed - 01S02"? The reason this has come up is that I have the following test: $dbh->{ReadOnly} = 1; is($dbh->{ReadOnly}, 1, 'ReadOnly set'); $dbh->{ReadOnly} = 0; is($dbh->{ReadOnly}, 0, 'ReadOnly cleared'); and the SQLite ODBC driver is failing it because any setting of SQL_ACCESS_MODE returns SQL_SUCCESS_WITH_INFO, option value changed, 01S02 and when you go to retrieve it back it is not what you set. By "issue a warning" do you mean set err to "0", errstr to "option value changed..." and state to "01S02"? If so, yes, that seems like the right thing to do. Yes, that is what I was proposing. The test can then be updated to check for that. Exactly. I'd be happy for you to patch the DBI docs along those lines. So the change would say that not all DBDs can necessarily set ReadOnly and if they can't, they will issue a warning? As for changing the docs, I can issue a pull request as I opted out of write access to DBI on github on the basis if I didn't trust myself with git (which I didn't), neither should you - so Merijn removed me. Whether $dbh->{ReadOnly} should remain false after an attempt to set it true has 'failed' seems more tricky. If it's false then other code can't tell that the application declared itself to not want to make changes. I'm inclined to let it stay true. Yes, good point but in this case, setting ReadOnly true results in reading ReadOnly as false (as the driver could not set SQL_ACCESS_MODE to true). So, any subsequent code reading ReadOnly does not know the application attempted to set ReadOnly true. So, if I've understood correctly then setting ReadOnly to true should return true even if the driver could not do that (Option value changed - 01S02). I can do this in DBD::ODBC but it requires maintaining state that ReadOnly was set to true but not acted on in the underlying driver - whereas what I do now is set SQL_ACCESS_MODE, ignore whether it works or not and if someone asks what SQL_ACCESS_MODE (ReadOnly) is I simply call SQLGetConnectAttr which in this case returns false. In summary: 1. setting ReadOnly should warn if setting ReadOnly cannot be achieved. 2. If ReadOnly has been set (even if unsuccessfully in the driver) true should be returned from $dbh->{ReadOnly} I can work with that. Please let me know if I've misunderstood and thanks for clarification. I'll change DBD::ODBC and DBI docs and issue a pull request. And if Christian Werner is by chance reading this, then this is a problem in DBD::ODBC and not the SQLite ODBC Driver which I will correct. Martin -- Martin J. Evans Wetherby, UK