DBD::Gofer middleware thoughts (was Idea for a Gofer transport for translating SQL - primarily intended for testing)
If we have a DBD::Gofer middleware feature then it would be helpful to
add a new passthrough (or "passthru") transport. This would be based on
the null transport, but without the serialization/deserialization, i.e.
something as simple as:
package DBD::Gofer::Transport::passthrough;
# $Id: null.pm 10087 2007-10-16 12:42:37Z timbo
$
#
# Copyright (c) 2007, Tim Bunce,
Ireland
#
# You may distribute under the terms of either the GNU General
Public
# License or the Artistic License, as specified in the Perl README
file.
use strict;
use warnings;
use base qw(DBD::Gofer::Transport::Base);
use DBI::Gofer::Execute;
our $VERSION = sprintf("0.%06d", q$Revision: 10087 $ =~ /(\d+)/o);
my $executor = DBI::Gofer::Execute->new();
sub transmit_request_by_transport {
my ($self, $request) = @_;
return $executor->execute_request( $request );
}
1;
This would be suitable for middleware modules to wrap.
Andrew
Re: Fwd: Idea for a Gofer transport for translating SQL - primarily intended for testing
Thinking about how middleware could be implemented in DBD::Gofer - see
below ...
Tim Bunce wrote:
On Thu, Sep 08, 2011 at 01:19:17PM +0100, Andrew Ford wrote:
and I would quite like to get this module onto CPAN and would like some
feedback as to
whether "the expert" thinks it is a bad idea. I think it would be a useful
addition to the testing
armoury, but then I am obviously biased.
The general idea is certainly useful. I'd like to generalise it though.
There's a conceptual similarity with PSGI/Plack middlewares that I'd
like to build on. Specifically, rather than creating a new transport,
I'd like to see DBD/Gofer/Transport/Base.pm extended to call one or more
'Gofer client middlewares' just before it calls transmit_request_by_transport().
The calling code could be something like:
+ my $mw = $self->middleware;
my $transmit_sub = sub {
my $response = eval {
...
- $self->transmit_request_by_transport($request)
+ $mw->($self, $request);
}
...
}
The middleware attribute would default to:
sub { shift->transmit_request_by_transport(@_) }
new middleware layer would be added by doing:
my $mw = $self->middleware;
$self->middleware( sub {
my ($self, $request) = @_;
...do something with $request...
my $response = $mw->($self, $request);
...do something with $response...
return $response;
} );
[This is all off-the-top-of-my-head, and I'm not very familar with Plack
internals so I may well be missing important issues.]
Then we just need a way to add middlewares via the environment.
I'd be delighted if you would work up a patch to add that to the DBI.
Currently DBD::Gofer::dr:connect() contains code to set up the transport:
my $transport_class = delete $go_attr{go_transport}
or return $drh->set_err($DBI::stderr, "No transport=
argument in '$orig_dsn'");
$transport_class = "DBD::Gofer::Transport::$transport_class"
unless $transport_class =~ /::/;
_load_class($transport_class)
or return $drh->set_err($DBI::stderr, "Can't load
$transport_class: $@");
my $go_transport = eval { $transport_class->new(\%go_attr) }
or return $drh->set_err($DBI::stderr, "Can't instanciate
$transport_class: $@");
I would imagine that we could have a "middleware" keyword that was a
comma-separated list of names that were assumed to be in the
DBDx::GoferMiddleware namespace - these would be installed in sequence
with a similar block of code that follows on from the snippet above,
looking something like:
my $middleware_classes = delete $go_attr{go_middleware};
foreach my $middleware_class (split /,/,
$middleware_classes) {
$middleware_class =
"DBDx::GoferMiddleware::$middleware_class"
unless $middleware_class =~ /::/;
_load_class($middleware_class)
or return $drh->set_err($DBI::stderr, "Can't load
$middleware_class: $@");
$go_transport->middleware( sub { return
$middleware_class->HANDLER(shift, $go_transport->middleware) } );
}
Each middleware handler would take a request object and a reference to
the next handler in the chain.
We would need to have a default method name for the middleware handler
(I've used the placeholder HANDLER in the pseudo-code above).
The DSNs would then look like:
dbi:Gofer:transport=null;middleware=adaptor;transform=MyMapper;dsn=dbi:XXX...
There are a few holes in this that I can think of:
* We might want to instantiate a middleware object for each
component and use that in installing the middleware handler
* I am not sure how we could specify a handler name other than the
default for individual middleware components
* Also not sure how attributes for each middleware component could
be specified such that the component for which the attribute is
intended could be specified - the example DSN above assumes that
the attribute (transform in the example DSN) is just dumped into
the general "go_" attribute hash)
* The go_middleware attribute could be set in the DBI connect
attribute hash to a list of subroutine references (code would need
to differentiate between a comma-separated list value and a list
reference value)
* Should the middleware components be listed innermost first (as
implemented in the pseudo-code above) or outermost first?
* We might want to add some convenience methods to the request class
to build response objects (as suggested in my
DBD::Gofer::Transport::adaptor code)
I'll have a go at getting this coded over the next few days, but it
looks as if the changes required to implement middleware are relatively
trivial.
Your module could then use this mechanism. If you wanted to release it
to CPAN then
Try to Contact SQL::Library Author Doug Gorley
Hello, Is Doug Gorley in this group? I try to contact SQL::Library Author Doug Gorley but failed by the email listed in the module. Could any one help? Thanks, Ge Peng
Re: Fwd: Idea for a Gofer transport for translating SQL - primarily intended for testing
Tim Bunce wrote: On Thu, Sep 08, 2011 at 01:19:17PM +0100, Andrew Ford wrote: and I would quite like to get this module onto CPAN and would like some feedback as to whether "the expert" thinks it is a bad idea. I think it would be a useful addition to the testing armoury, but then I am obviously biased. The general idea is certainly useful. I'd like to generalise it though. OK - I have got Gofer::DBD::Transport::adaptor written and tested (attached to this mail). I will hang off on uploading to CPAN and will look at Plack middleware and how the functionality could be refactored to give patches to DBD::Gofer for installing middleware, plus the rump of my adaptor module as a DBDx::MiddleWare::XXX module. Andrew DBD-Gofer-Transport-adaptor-0.001.tar.gz Description: application/gzip
Re: Fwd: Idea for a Gofer transport for translating SQL - primarily intended for testing
On Thu, Sep 08, 2011 at 01:19:17PM +0100, Andrew Ford wrote:
>Hi Tim
>
>Sorry for mailing you directly, but I don't know whether my emails
>are getting through to the dbi-users mailing list,
[CC'd to dbi-users]
>and I would quite like to get this module onto CPAN and would like some
> feedback as to
>whether "the expert" thinks it is a bad idea. I think it would be a
> useful addition to the testing
>armoury, but then I am obviously biased.
The general idea is certainly useful. I'd like to generalise it though.
There's a conceptual similarity with PSGI/Plack middlewares that I'd
like to build on. Specifically, rather than creating a new transport,
I'd like to see DBD/Gofer/Transport/Base.pm extended to call one or more
'Gofer client middlewares' just before it calls transmit_request_by_transport().
The calling code could be something like:
+ my $mw = $self->middleware;
my $transmit_sub = sub {
my $response = eval {
...
- $self->transmit_request_by_transport($request)
+ $mw->($self, $request);
}
...
}
The middleware attribute would default to:
sub { shift->transmit_request_by_transport(@_) }
new middleware layer would be added by doing:
my $mw = $self->middleware;
$self->middleware( sub {
my ($self, $request) = @_;
...do something with $request...
my $response = $mw->($self, $request);
...do something with $response...
return $response;
} );
[This is all off-the-top-of-my-head, and I'm not very familar with Plack
internals so I may well be missing important issues.]
Then we just need a way to add middlewares via the environment.
I'd be delighted if you would work up a patch to add that to the DBI.
Your module could then use this mechanism. If you wanted to release it
to CPAN then a name like DBDx::GoferMiddleware::FOO would be good.
(Note the DBDx not DBIx since this is client-side. We may well end up
with server-side gofer middlewares as well.)
Tim.
>I have attached the current version of the module file with some
> documentation from my intial post
>included below (I've changed my mind about the attribute naming and
> currently am going for
>transform=modulename and method=methodname for the module/method that
> implements the transformation). I
>will of course add unit tests and everything that goes with a proper CPAN
> release, but I would be
>grateful for some quick feedback.
>
>Regards
>Andrew
>
> Original Message
>
>Subject: Idea for a Gofer transport for translating SQL - primarily
> intended for testing
> Date: Thu, 08 Sep 2011 11:16:41 +0100
> From: Andrew Ford [1]
> To: [2]dbi-users@perl.org
>
>I have a software system that I have to test that uses a very large
> database accessed exclusively with
>stored procedures. It is a pain setting up a test database for simple
> unit tests of the Perl code (I am
>not trying to test the stored procedures).
>
>What I want is to be able to do is set up a simple database (SQLite)
> quickly to use instead, but of
>course there is the issue of the stored procedures being database-specific
> - but then for testing
>purposes most of the stored procedures could be represented by simple SQL
> statements.
>
>What I have done as a proof of concept is to take the
> DBD::Gofer::Transport::null and create a new
>module that maps the stored procedures into simple SQL statements and then
> executes them in-process
>against the test database
>
>The code is pretty simple and included below. Currently it has a map of
> stored procedure
>transformations hard-coded in the code and assumes ODBC style stored
> procedure call syntax, but I was
>thinking of allowing a mapping module to be specified as an attribute in
> the DSN so that the DSN would
>be specified like:
>
>
> dbi:Gofer:transport=adaptor;mapper=My::Mapping::Module;dsn=dbi:SQLite:test.db
>
>Overriding the system's DSN then causes the system to use my test
> database, transparently doing the
>appropriate mappings.
>
>I would appreciate any comments, suggestions, and thoughts as to whether
> this would be a useful addition
>to CPAN.
>
>The immediate issue that I see is: what would the interface to the mapping
> module be (probably the
>module should provide a single function that takes a request object and
> either modifies it or returns a
>new request object).
>
>Andrew
>
>current proof-of-concept code:
>
> package DBD::Gofer::Transport::adaptor;
>
> use strict;
> use warnings;
>
> use DBI::Gofer::Execute;
>
> use base qw(DBD::Gofer::Transport::Base);
>
> __PACKAGE__->mk_accessors( qw( pending_response transmit_count ) );
>
> # Lookup table for stored procedure transformations
>
> my %stored_proc_transform
>
problems install DBD::DB2
Hi, i'm tryng to install DBD::DB2 on my linux box but it fails: install DBD::DB2 Going to read '/root/.cpan/Metadata' Database was generated on Thu, 08 Sep 2011 09:28:48 GMT Running install for module 'DBD::DB2' Running make for I/IB/IBMTORDB2/DBD-DB2-1.81.tar.gz Checksum for /root/.cpan/sources/authors/id/I/IB/IBMTORDB2/DBD-DB2-1.81.tar.gz ok Scanning cache /root/.cpan/build for sizes ..--DONE DEL(1/2): /root/.cpan/build/SOAP-Lite-0.712-CNzvW5 DEL(2/2): /root/.cpan/build/SOAP-Lite-0.712-CNzvW5.yml CPAN.pm: Going to build I/IB/IBMTORDB2/DBD-DB2-1.81.tar.gz Legacy library flush.pl will be removed from the Perl core distribution in the next major release. Please install it from the CPAN distribution Perl4::CoreLibs. It is being used at Makefile.PL, line 11. Configuring DBD::DB2... Remember to actually read the README and CAVEATS files! Using DB2 in "/opt/ibm/db2/V9.7" System: perl5.014001 DBI1.616 linux monitor.regione.toscana.it 2.6.32-131.6.1.el6.x86_64 #1 smp mon jun 20 14:15:38 edt 2011 x86_64 x86_64 x86_64 gnulinux x86_64-linux dl_dlopen.xs Compiler: cc -O2 -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 Includes: -I"/opt/ibm/db2/V9.7/include" -I"/opt/perl5/lib/site_perl/5.14.1/x86_64-linux/auto/DBI" -I"/opt/perl5/lib/5.14.1/x86_64-linux/auto/DBI" -I"/opt/perl5/lib/site_perl/5.14.1/x86_64-linux/auto/DBI" Libraries: -L/opt/ibm/db2/V9.7/lib64 -ldb2 Checking if your kit is complete... Looks good Checking if your kit is complete... Looks good Writing Makefile for DBD::DB2::Constants Writing MYMETA.yml Writing Makefile for DBD::DB2 Writing MYMETA.yml cp DB2.pm blib/lib/DBD/DB2.pm cp lib/Bundle/DBD/DB2.pm blib/lib/Bundle/DBD/DB2.pm cp DB2.pod blib/lib/DBD/DB2.pod make[1]: Entering directory `/root/.cpan/build/DBD-DB2-1.81-xaPjeE/Constants' cp Constants.pm ../blib/lib/DBD/DB2/Constants.pm AutoSplitting ../blib/lib/DBD/DB2/Constants.pm (../blib/lib/auto/DBD/DB2/Constants) /opt/perl5/bin/perl /opt/perl5/lib/5.14.1/ExtUtils/xsubpp -typemap /opt/perl5/lib/5.14.1/ExtUtils/typemap Constants.xs > Constants.xsc && mv Constants.xsc Constants.c cc -c -I"/opt/ibm/db2/V9.7/include" -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -O2 -DVERSION=\"1.81\" -DXS_VERSION=\"1.81\" -fPIC "-I/opt/perl5/lib/5.14.1/x86_64-linux/CORE" Constants.c Running Mkbootstrap for DBD::DB2::Constants () chmod 644 Constants.bs rm -f ../blib/arch/auto/DBD/DB2/Constants/Constants.so cc -shared -O2 -L/usr/local/lib -fstack-protector Constants.o -o ../blib/arch/auto/DBD/DB2/Constants/Constants.so \ \ chmod 755 ../blib/arch/auto/DBD/DB2/Constants/Constants.so cp Constants.bs ../blib/arch/auto/DBD/DB2/Constants/Constants.bs chmod 644 ../blib/arch/auto/DBD/DB2/Constants/Constants.bs make[1]: Leaving directory `/root/.cpan/build/DBD-DB2-1.81-xaPjeE/Constants' /opt/perl5/bin/perl /opt/perl5/lib/5.14.1/ExtUtils/xsubpp -typemap /opt/perl5/lib/5.14.1/ExtUtils/typemap DB2.xs > DB2.xsc && mv DB2.xsc DB2.c cc -c -I"/opt/ibm/db2/V9.7/include" -I"/opt/perl5/lib/site_perl/5.14.1/x86_64-linux/auto/DBI" -I"/opt/perl5/lib/5.14.1/x86_64-linux/auto/DBI" -I"/opt/perl5/lib/site_perl/5.14.1/x86_64-linux/auto/DBI" -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -O2 -DVERSION=\"1.81\" -DXS_VERSION=\"1.81\" -fPIC "-I/opt/perl5/lib/5.14.1/x86_64-linux/CORE" -DDB2_CACHE_FIX DB2.c DB2.xs: In function âXS_DBD__DB2__dr_disconnect_allâ: DB2.xs:49: error: âdirtyâ undeclared (first use in this function) DB2.xs:49: error: (Each undeclared identifier is reported only once DB2.xs:49: error: for each function it appears in.) DB2.xs: In function âXS_DBD__DB2__db__loginâ: DB2.xs:92: error: âsv_yesâ undeclared (first use in this function) DB2.xs:92: error: âsv_noâ undeclared (first use in this function) DB2.xs: In function âXS_DBD__DB2__db_commitâ: DB2.xs:103: error: âsv_yesâ undeclared (first use in this function) DB2.xs:103: error: âsv_noâ undeclared (first use in this function) DB2.xs: In function âXS_DBD__DB2__db_rollbackâ: DB2.xs:112: error: âsv_yesâ undeclared (first use in this function) DB2.xs:112: error: âsv_noâ undeclared (first use in this function) DB2.xs: In function âXS_DBD__DB2__db_disconnectâ: DB2.xs:134: error: âdirtyâ undeclared (first use in this function) DB2.xs:141: error: âsv_yesâ undeclared (first use in this function) DB2.xs:141: error: âsv_noâ undeclared (first use in this function) DB2.xs: In function âXS_DBD__DB2__db_STOREâ: DB2.xs:156: error: âsv_yesâ undeclared (first use in this function) DB2.xs:159: error: âsv_noâ undeclared (first use in this function) DB2.xs: In function âXS_DBD__DB2__db_DESTROYâ: DB2.xs:178: error: âsv_yesâ undeclared (first use in this function) DB2.xs:181: error: âdirtyâ undeclared
