Re: interface ideas for non-blocking mode

[David Nicol]

Okay

I have enough discussion to create a DBD::Async wrapper which
will be used sort of like DBD::Proxy, in terms of how to declare it
in the connect string.

Anyone else who wants to be included tighter in the development
loop of DBD::Async, for instance to help code, or to test, or to
benchmark, please reply to me at [EMAIL PROTECTED]
and we can together proceed.  No more to dbi-dev until a working
release announcement.

We're talking about a convention for async drivers to follow, not
an extention to DBI.

On Fri, 10 Sep 2004 16:44:51 +0100, Tim Bunce <[EMAIL PROTECTED]> wrote:
> On Fri, Sep 10, 2004 at 03:15:38AM -0500, David Nicol wrote:

> > Should I set up an asynchronous DBI mailing list for discussion of
> > asynchronous DBI
> > or should discussion continue here on dbi-dev?  (now I'm talking crazy)
> 
> Here. But...
> 
> > I am also proposing to write at least one wrapper driver that
> > will provide the functionality against a synchronous DBD.
> 
> best to start talking again once you've got something working.

understood
 
> Drivers are, as always, free to add support for anything they like
> using the various driver-specific mechanisms provided by the DBI.
> 
> For now I'd like to shelve this thread. Thanks to all who contributed.
> 
> I'll return to it once I get to address the issue for DBI v2, or
> a driver author brings it up because they're actually working on it
> (or have a patch contributed by someone who has).
> 
> Tim.

Re: interface ideas for non-blocking mode

[Tim Bunce]

On Fri, Sep 10, 2004 at 03:15:38AM -0500, David Nicol wrote:
> I was not aware of the risk of select hanging due to a signal coming in before
> select can set its timer.  Has anyone on this list ever been bitten by such a
> thing?  Is Perl vulnerable to this situation or is it taken care of
> internally?  The
> situation described would turn a nonblocking select call into an
> indefinately blocking
> select, but it sounds unlikely to occur in practice, and would only
> matter when none of
> the pipes are readable anyway.  If one uses wait or waitpid rather
> than redefining
> $SIG{CHLD} is one vulnerable?

There are better mailing lists for those kinds of issues.

> Should I set up an asynchronous DBI mailing list for discussion of
> asynchronous DBI
> or should discussion continue here on dbi-dev?  (now I'm talking crazy)

Here. But...

> I am also proposing to write at least one wrapper driver that
> will provide the functionality against a synchronous DBD.

best to start talking again once you've got something working.

Drivers are, as always, free to add support for anything they like
using the various driver-specific mechanisms provided by the DBI.

For now I'd like to shelve this thread. Thanks to all who contributed.

I'll return to it once I get to address the issue for DBI v2, or
a driver author brings it up because they're actually working on it
(or have a patch contributed by someone who has).

Tim.

> On Thu, 9 Sep 2004 17:52:44 +0100 (BST), Matt Sergeant
> <[EMAIL PROTECTED]> wrote:
> > On Thu, 9 Sep 2004, Dominic Mitchell wrote:
> > 
> > > Even if your DB library doesn't provide access to the file descriptor,
> > > you can still use the self pipe trick to work around that.
> > >
> > >http://cr.yp.to/docs/selfpipe.html
> 
> -- 
> David L Nicol
> IT Consulting since 1986

Re: interface ideas for non-blocking mode

[David Nicol]

I was not aware of the risk of select hanging due to a signal coming in before
select can set its timer.  Has anyone on this list ever been bitten by such a
thing?  Is Perl vulnerable to this situation or is it taken care of
internally?  The
situation described would turn a nonblocking select call into an
indefinately blocking
select, but it sounds unlikely to occur in practice, and would only
matter when none of
the pipes are readable anyway.  If one uses wait or waitpid rather
than redefining
$SIG{CHLD} is one vulnerable?


Should I set up an asynchronous DBI mailing list for discussion of
asynchronous DBI
or should discussion continue here on dbi-dev?  (now I'm talking crazy)


On Thu, 9 Sep 2004 17:52:44 +0100 (BST), Matt Sergeant
<[EMAIL PROTECTED]> wrote:
> On Thu, 9 Sep 2004, Dominic Mitchell wrote:
> 
> > Even if your DB library doesn't provide access to the file descriptor,
> > you can still use the self pipe trick to work around that.
> >
> >http://cr.yp.to/docs/selfpipe.html

-- 
David L Nicol
IT Consulting since 1986

Re: interface ideas for non-blocking mode ( migration path proposal )

[David Nicol]

I am proposing a convention for extension, not an extension to core
compliance requirements.

I am also proposing to write at least one wrapper driver that will provide the
functionality against a synchronous DBD.

I imagine that asynchronous DBDs will, at least initially, exist
separately from their
synchronous counterparts, possibly with an extended DBD name conforming to a
standard suffix, like ora_a or ix_a  -- a for asynchronous.  In much
the same way that
reentrant libraries are suffixed with _r.



On Thu, 9 Sep 2004 00:07:12 -0700, Jonathan Leffler
<[EMAIL PROTECTED]> wrote:
> On Wed, 8 Sep 2004 14:39:45 -0500, David Nicol <[EMAIL PROTECTED]> wrote:
> > Right.  And when your driver is providing a DBI interface to one of
> > them, to provide
> > a fileno() function that works, you may need to open a pipe and write
> > to one end of it when you are expecting some data.  [...]
> 
> We're talking past each other and not connecting.  Probably too much
> asynchronous communication between incompatible database systems :-(

Recognizing the problem is the first step :)
 
> Or we have different plans for what DBI is and/or should be -- I've
> had my disagreements with Tim over this in the past, and can see that
> it may happen again in the future.
> 
> I've gone back to what I believe is the first of your postings (27th
> August - though the email header has 28th August just before 01:00 in
> US/Eastern or maybe US/Central time zone; -05:00).  It's a long
> posting.  In it, you say:
> 
>  ==> In the application I am currently working on, I want prepare and
>  ==> execute to return as soon as they can, even if error reports will get
>  ==> deferred, and I would like to know if data is ready before attempting
>  ==> to fetch a row, so I don't have to wait for the server to send it.
> 
> These perceived requirements obviously colour your opinion on what is
> required - that's OK; Open Source software gets written when it
> scratches someone's itch.
> 
>  ==> Yes this class of issues can be trivially solved by demanding threading,
>  ==> but that does not help when a(n unrealistic?) design constraint limits
>  ==> you to one thread.
> 
> This worries me.  I really think that asynchronous support via
> threading is more sensible.  It is also more likely to be widely
> achievable - IMNSHO.  However, it is your itch, so you get to lay out
> your desires.  However, be aware that not everyone suffers from the
> same sources of irritation as you do.

Supporting asynchronous operation via threading is unnecessary.
With threading, the application author can have one thread
work the database synchronously.  End of story.

>  ==> If pointed to the simplest TCP-based driver, I will cheerfully
>  ==> add this stuff to demonstrate that its practical.
> 
> Simple TCP drivers aren't the problem.  It is the non-simple drivers
> with closed source where there is *NO* access to the file handles that
> are used internally, or where no file handles are used at all, and
> whether or not they are used is irrelevant since there is no access to
> the underlying transport layers regardless of what is used.
> Demonstrate how it will be done in DBD::Oracle -- I'd prefer it to be
> DBD::Informix, but that's probably a tad too partisan to be fair.
> 
> What I don't see in your discussion is how you would scope
> asynchronicity.  Is it at the $dbh level - so that the DBI->connect
> method takes a new attribute that says { Asynchronous => 1 }.  Or is
> it something that is at the statement level, so $dbh->prepare method
> has to take the flag.   Somewhere along the line, you have to indicate
> your desire for asynchronicity -- you cannot add it to everything
> without breaking a lot of happily working applications.

The attribute in my proposal is "Timeout" and its appearance indicates
that the applicaiton author wishes to use an asynchronous mode.  I believe
that DBDs crash when presented with atteributes they don't understand, which
if it is true would mean the following would work in general:

 eval {
 $dbh = DBI->connect($dsn,$username,$password, 
 {RaiseError => 1, Timeout => 0});
 };
  $@ and 
 $dbh = DBI->connect("Async:$dsn",$username,$password, 
 {RaiseError => 1, Timeout => 0});

when Async is the name of an asynchronous wrapper DBD.  I am open to
the idea that having an attribute which the existence of it is what matters
rather than having it set to a true value is too weird for DBI and the attribute
should be renamed to Asynchronous and Timeout would be an additional
attribute that defaults to zero.  I find that cumbersome.

>  ==> Working out the implementation is not our problem yet (well it is our
>  ==> problem if we're maintaining a driver, but it's not our problem if we're
>  ==> wearing "Lords Of The DBI Specification" hats)
> 
> The Lords of the DBI Specification may NOT mandate features that
> cannot be readily be supporte

Re: interface ideas for non-blocking mode

[Matt Sergeant]

On Thu, 9 Sep 2004, Dominic Mitchell wrote:

> Even if your DB library doesn't provide access to the file descriptor,
> you can still use the self pipe trick to work around that.
>
>http://cr.yp.to/docs/selfpipe.html

I believe that's exactly what I'm talking about. Only if your DB doesn't
notify you, and you have to poll it, then you need to do that in a
separate thread for this to work.

I'm going to stop talking about this now, as it's clear I can make the
filehandle thing work for me if I need it to, regardless of what scheme
the DBI implements. I was just trying to think about efficiency for my
apps.

Matt.

RE: interface ideas for non-blocking mode

[Jeff Urlwin]

> 
> In terms of why I'm banging on about file descriptors, let me explain 
> my itch. I work on very highly scalable network applications that 
> process thousands of concurrent connections. In these 
> programs NOTHING 
> can block my event loop, because everything falls apart if I 
> do. And I 
> have to be very careful about slow system calls. If I can have a file 
> descriptor it can fit in cleanly with my current event loop. If I 
> can't, and have to rely on something like $sth->ready, which 
> might call 
> its own select() function, then I'm screwed, because select() 
> is a slow 
> system call.

Just to be clear and not that I think DBI is going down the path of
implementing select(), but since it's mentioned and since I've recently
(today) been screwed by it, I'll bring it up.  You have to be careful with
select() and portability.  Not all fds are the same on all platforms.  For
example, Win32 select() doesn't work with file handles, but it does with
sockets.  (At least as of 5.8.4).

Regards,

Jeff

Re: interface ideas for non-blocking mode

[Dominic Mitchell]

Matt Sergeant wrote:
On 9 Sep 2004, at 08:14, Jonathan Leffler wrote:
 ==> Yes this class of issues can be trivially solved by demanding 
threading,
 ==> but that does not help when a(n unrealistic?) design constraint 
limits
 ==> you to one thread.

This worries me.  I really think that asynchronous support via
threading is more sensible.  It is also more likely to be widely
achievable - IMNSHO.  However, it is your itch, so you get to lay out
your desires.  However, be aware that not everyone suffers from the
same sources of irritation as you do.
So one way to do this to keep everyone happy might be to use threads to 
do things in the background, but use a file descriptor to notify of 
readiness. The DBI could do this transparently, or you could implement 
it natively if you preferred (or had access to your own descriptor that 
you could use).

In terms of why I'm banging on about file descriptors, let me explain my 
itch. I work on very highly scalable network applications that process 
thousands of concurrent connections. In these programs NOTHING can block 
my event loop, because everything falls apart if I do. And I have to be 
very careful about slow system calls. If I can have a file descriptor it 
can fit in cleanly with my current event loop. If I can't, and have to 
rely on something like $sth->ready, which might call its own select() 
function, then I'm screwed, because select() is a slow system call.

I'm absolutely willing to admit that my application is way out there 
compared to most people's.
Even if your DB library doesn't provide access to the file descriptor, 
you can still use the self pipe trick to work around that.

  http://cr.yp.to/docs/selfpipe.html
DJB's talking about a slightly different scenario here (SIGCHLD 
handler), but a similiar principle applies.  If your database supports 
async, it probably has a callback routine.  In that callback, write a 
byte to the pipe.  You'll get woken up in your event loop as expected.

-Dom (an anything but threads person)
--
| Semantico: creators of major online resources  |
|   URL: http://www.semantico.com/   |
|   Tel: +44 (1273) 72 / Fax: +44 (1273) 723232  |
|   Address: 33 Bond St., Brighton, Sussex, BN1 1RD, UK. |

Re: interface ideas for non-blocking mode

[Matt Sergeant]

On 9 Sep 2004, at 08:14, Jonathan Leffler wrote:
 ==> Yes this class of issues can be trivially solved by demanding 
threading,
 ==> but that does not help when a(n unrealistic?) design constraint 
limits
 ==> you to one thread.

This worries me.  I really think that asynchronous support via
threading is more sensible.  It is also more likely to be widely
achievable - IMNSHO.  However, it is your itch, so you get to lay out
your desires.  However, be aware that not everyone suffers from the
same sources of irritation as you do.
So one way to do this to keep everyone happy might be to use threads to 
do things in the background, but use a file descriptor to notify of 
readiness. The DBI could do this transparently, or you could implement 
it natively if you preferred (or had access to your own descriptor that 
you could use).

In terms of why I'm banging on about file descriptors, let me explain 
my itch. I work on very highly scalable network applications that 
process thousands of concurrent connections. In these programs NOTHING 
can block my event loop, because everything falls apart if I do. And I 
have to be very careful about slow system calls. If I can have a file 
descriptor it can fit in cleanly with my current event loop. If I 
can't, and have to rely on something like $sth->ready, which might call 
its own select() function, then I'm screwed, because select() is a slow 
system call.

I'm absolutely willing to admit that my application is way out there 
compared to most people's.

Matt.

Re: interface ideas for non-blocking mode

[Jonathan Leffler]

On Fri, 3 Sep 2004 21:14:23 +0100, Matt Sergeant <[EMAIL PROTECTED]> wrote:
> On 2 Sep 2004, at 23:31, Tim Bunce wrote:
> 
> > I haven't had a chance to read through all the comments in this thread
> > yet,
> > but I'll point out two flies in the ointment:
> >
> > 1. Many database API's don't offer access to the filehandle.
> >
> > 2. Even for those that do, just because the filehandle becomes
> > 'readable'
> >doesn't mean that the appropriate API function call won't block.
> 
> So it doesn't have to be the filehandle that is connected across the
> network. It could be a pipe that you write to when stuff is ready. This
> would be compatible with either local or remote systems.


No - as I said previously, file descriptors (whether pipe, socket,
file or anything else) are not the only mechanism used to achieve IPC
- interprocess communication.


-- 
Jonathan Leffler <[EMAIL PROTECTED]>  #include 
Guardian of DBD::Informix - v2003.04 - http://dbi.perl.org
"I don't suffer from insanity - I enjoy every minute of it."

Re: interface ideas for non-blocking mode

[Matt Sergeant]

On 2 Sep 2004, at 23:31, Tim Bunce wrote:
I haven't had a chance to read through all the comments in this thread 
yet,
but I'll point out two flies in the ointment:

1. Many database API's don't offer access to the filehandle.
2. Even for those that do, just because the filehandle becomes 
'readable'
   doesn't mean that the appropriate API function call won't block.
So it doesn't have to be the filehandle that is connected across the 
network. It could be a pipe that you write to when stuff is ready. This 
would be compatible with either local or remote systems.

Matt.

Re: interface ideas for non-blocking mode

[david nicol]

Changes: fetchavail_arrayref is retracted from the proposal.


On Thu, 2004-09-02 at 18:24, Jonathan Leffler wrote:
> On Thu, 2 Sep 2004 23:31:22 +0100, Tim Bunce <[EMAIL PROTECTED]> wrote:
> > On Thu, Sep 02, 2004 at 01:52:59PM +0100, Matt Sergeant wrote:
> > > On 1 Sep 2004, at 03:13, david nicol wrote:

> > > > summary: define an optional, standard non-blocking interface
> > > > by adding ready(), fileno(), and fetchavail() methods.
> > > > *fileno* would return:
> > > >An array ref rather than one number, so handles that have multiple
> > > >connections open can report them all.

> > > Yes, this would work. Having each DBD implement their own internal
> > > select() wouldn't work as well, because two event loops are worse than
> > > one.

calling select() does not imply you have your own event loop.  Select
can be called with a timeout of zero, in which case it does not block,
and can be used within another event loop. We can strongly reccommend
that the timeout attribute should be set to 0 to have non-blocking
polling.

> > I haven't had a chance to read through all the comments in this thread yet,
> > but I'll point out two flies in the ointment:
> > 
> > 1. Many database API's don't offer access to the filehandle.

> And some API's don't even use files for communicating with the database server.

returning the file descriptor number of the session filehandle of course
only makes sense when it makes sense.  In a threaded environment, or
even in a fork-friendly environment, setting up a C just to
provide a fd that the fully asynchronous driver can write a trigger byte
to when something becomes ready is certainly possible. 

> Informix can use shared memory connections -- and uses semaphores for
> synchronization.

I worry that
I'm getting repetitive here and might do better to sit back and watch
for a while rather than keeping up with this thread.  I will try to
have this one be the last repetition of the proposal for a while
except for altering it in response to constructive criticisms.

The implementation is the driver implementor's problem. An optional
extension to the DBI interface standard needs to

  * not break anything  ("first, do no harm")

  * be general enough to apply everywhere

  * be trivial to map current NB implementations to

  * play nice with external event loops using select() 

  * ease creating a DBD::NonBlockingWrapper for using the nonblocking
interface against a back-end that does not support it

Currently, DBI applications can block for all steps of the DBI
interaction.  In a nonblocking DBD, this behavior can be imitated
by not defining a Timeout attribute, or defining a Timeout of C.
This mode is called "Synchronous" and is the only mode available with
current DBI.

The two asynchronous modes are called "deferred asynchronous" and 
"fully asynchronous."  The Sybase documentation referred to earlier in
this thread defines them well.  The interface I am proposing works the
same, to the DBI coder, in both modes.

The current DBI fetch methods will continue to block, unless this
behavior is overruled by an extension to this extension, which would
redefine the current fetch methods to return an error.  I think 
redefining fetch to fail EWOULDBLOCK is a bad idea because programming
around the lack of non-blocking sockets on platforms that do not support
them is complex.  So, what is needed is a way to know if a fetch() will
block or not, before we call fetch.  My proposal for a method which
answers that question is $sth->ready().

In a synchronous environment, calling ready() is an error, providing
instant legacy support: legacy (synchronous) drivers will throw a "Can't
locate object method "ready" via package ..." error which can be trapped
with block eval.

In a deferred asynchronous environment, calling ready() gives the driver
a chance to non-blockingly poll for data and see if there is enough data
that the next expected call (fetch, or perhaps execute) will not block.

In a fully asynchronous environment, the driver thread (or process) 
sets a flag somewhere the ready() method can check.

To support an external event loop, the fileno() method is introduced.
An FD returned by fileno() becoming readable DOES NOT imply that the
next expected call will not block. It merely indicates that the driver
would appreciate a time-slice, which can be given to it by calling
ready().  

Having a file descriptor actually involved in the back-end
implementation is not needed:  Here is a working fileno() method that
can be switched on and off:

   {
  my ($RH, $WH); # make these per-object for per-object granularity
 INIT { 
pipe $RH, $WH;
my $oldfh = select($WH); $| = 1; select($oldfh)
  }
  sub fileno{ [fileno $RH]}; 
  sub _fileno_inc(){ print $WH, 'a' };
  sub _fileno_dec(){ my $dummy; sysread $RH, $dummy,1 };
   }

For per-object granularity, the INIT code would go in the constructor
and the fileno method would do something like [fileno shif

Re: interface ideas for non-blocking mode

[Jonathan Leffler]

On Thu, 2 Sep 2004 23:31:22 +0100, Tim Bunce <[EMAIL PROTECTED]> wrote:
> On Thu, Sep 02, 2004 at 01:52:59PM +0100, Matt Sergeant wrote:
> > On 1 Sep 2004, at 03:13, david nicol wrote:
> > >Based on this comment I would like to extend the proposed extension
> > >to include an optional $h->fileno() method[...]
> > > ...
> > > buildselectvectors(@Listeners,@Clients,@Callbacks);
> > > select ...
> > >
> > >An array ref rather than one number, so handles that have multiple
> > >connections open can report them all.
> >
> > Yes, this would work. Having each DBD implement their own internal
> > select() wouldn't work as well, because two event loops are worse than
> > one.
> 
> I haven't had a chance to read through all the comments in this thread yet,
> but I'll point out two flies in the ointment:
> 
> 1. Many database API's don't offer access to the filehandle.

And some API's don't even use files for communicating with the database server.

Informix can use shared memory connections -- and uses semaphores for
synchronization.

-- 
Jonathan Leffler <[EMAIL PROTECTED]>  #include 
Guardian of DBD::Informix - v2003.04 - http://dbi.perl.org
"I don't suffer from insanity - I enjoy every minute of it."

Re: interface ideas for non-blocking mode

[Tim Bunce]

On Thu, Sep 02, 2004 at 01:52:59PM +0100, Matt Sergeant wrote:
> On 1 Sep 2004, at 03:13, david nicol wrote:
> 
> >Based on this comment I would like to extend the proposed extension
> >to include an optional $h->fileno() method, which would return
> >an arrayref containing file descriptor numbers for use in the main
> >event loop, so a busy event-driven system will know when it can be sure
> >that calling $h->ready() will not waste cycles.
> >
> >   {
> > my $htmp = $h;
> > my $coderef = sub { $htmp->ready and push @Readies, $htmp };
> > $Callback[$_] = $coderef for @{$h->fileno};
> >   }
> >
> > ...
> > buildselectvectors(@Listeners,@Clients,@Callbacks);
> > select ...
> >
> >An array ref rather than one number, so handles that have multiple
> >connections open can report them all.
> 
> Yes, this would work. Having each DBD implement their own internal 
> select() wouldn't work as well, because two event loops are worse than 
> one.

I haven't had a chance to read through all the comments in this thread yet,
but I'll point out two flies in the ointment:

1. Many database API's don't offer access to the filehandle.

2. Even for those that do, just because the filehandle becomes 'readable'
   doesn't mean that the appropriate API function call won't block.

Tim.

Re: interface ideas for non-blocking mode

[Matt Sergeant]

On 1 Sep 2004, at 03:13, david nicol wrote:
Based on this comment I would like to extend the proposed extension
to include an optional $h->fileno() method, which would return
an arrayref containing file descriptor numbers for use in the main
event loop, so a busy event-driven system will know when it can be sure
that calling $h->ready() will not waste cycles.
   {
 my $htmp = $h;
 my $coderef = sub { $htmp->ready and push @Readies, $htmp };
 $Callback[$_] = $coderef for @{$h->fileno};
   }
 ...
 buildselectvectors(@Listeners,@Clients,@Callbacks);
 select ...
An array ref rather than one number, so handles that have multiple
connections open can report them all.
Yes, this would work. Having each DBD implement their own internal 
select() wouldn't work as well, because two event loops are worse than 
one.

Matt.

Re: interface ideas for non-blocking mode

[david nicol]

On Sat, 2004-08-28 at 06:01, Matt Sergeant wrote:
> As an addition to this thread, I just wanted to add that having each 
> DBD implement their own event loop (via select() or otherwise) is not 
> acceptable. Anyone who has done event driven programming will 
> understand the mantra: There can be only one. Only one event loop that 
> is. If you implement an event loop in your DBD then you BLOCK my event 
> loop, which I cannot allow to happen. I don't believe that perl threads 
> are stable/fast enough to fix this problem in a very large scale 
> program.
> 
> The API *has* to be something that can integrate with other event 
> loops. My suggestion for the only thing scalable enough for this is a 
> file descriptor which can be notified of readiness.
> 
> Matt.

Supporting deferred-asynchronous mode with a ready()
method that polls and returns either immediately or after handling
availble data would not block your event loop.

An event loop inside one's DBD would be appropriate for a fully
asynchronous mode, however you choose build that.

Based on this comment I would like to extend the proposed extension
to include an optional $h->fileno() method, which would return 
an arrayref containing file descriptor numbers for use in the main
event loop, so a busy event-driven system will know when it can be sure
that calling $h->ready() will not waste cycles.

   {
 my $htmp = $h;
 my $coderef = sub { $htmp->ready and push @Readies, $htmp };
 $Callback[$_] = $coderef for @{$h->fileno};
   }  

 ...
 buildselectvectors(@Listeners,@Clients,@Callbacks);
 select ...

An array ref rather than one number, so handles that have multiple
connections open can report them all.

Re: interface ideas for non-blocking mode

[Matt Sergeant]

As an addition to this thread, I just wanted to add that having each 
DBD implement their own event loop (via select() or otherwise) is not 
acceptable. Anyone who has done event driven programming will 
understand the mantra: There can be only one. Only one event loop that 
is. If you implement an event loop in your DBD then you BLOCK my event 
loop, which I cannot allow to happen. I don't believe that perl threads 
are stable/fast enough to fix this problem in a very large scale 
program.

The API *has* to be something that can integrate with other event 
loops. My suggestion for the only thing scalable enough for this is a 
file descriptor which can be notified of readiness.

Matt.

Re: interface ideas for non-blocking mode

[Matt Sergeant]

On 26 Aug 2004, at 16:48, Dean Arnold wrote:
While this handles async execution, esp. in support of multiconnection
operations relevant to Teradata's parallel nature,
it doesn't really handle async completion notification, and
I'm not certain there's a clean, DBMS-independent way to support
that without using timers, signals, or threads, all of which
may be problematic.
I would favour some kind of filehandle for completion notification - 
this way it will integrate much nicer with other event loop systems. It 
could be a simple pipe that you write something to when there's data 
ready, and your event loop can select()/poll() on the other end of the 
pipe.

Matt.

Re: interface ideas for non-blocking mode

[david nicol]

On Thu, 2004-08-26 at 03:04, Tim Bunce wrote:

> We would all like many things but have to settle for what's practical.
> ...
>
> Tim.

umm okay

All I knew, before my unsuccessful attempt to locate instances of
connect(2) in DBD::ODBC's xs and dbdimp.c files, was that I have in 
the past written code, in C and C++, that talked with SQL servers by
opening regular old TCP streams to servers and conversing on them.

Informix and Oracle.  And both times I was extending working code
so I didn't have to know or support all the possibilities.

The only time a TCP stream blocks is when one is waiting to read from
it, and this can be worked around, even on systems that do not have
nonblocking sockets (unless you're multithreading, in which case you
need to set up a mutex to avoid the race condition, so even if you're
multithreading, provided you can set up a nonblocking mutex), by
only trying to read from the socket when select(2) has indicated that
there is something there. It can also block when you're writing a lot
to it, but this can be worked around by being sure to only send
small chunks, when  you can't get non-blocking sockets.

In the application I am currently working on, I want prepare and
execute to return as soon as they can, even if error reports will get
deferred, and I would like to know if data is ready before attempting
to fetch a row, so I don't have to wait for the server to send it.

Expecting the *all* access functions to return partial sets is 
not a core requirement for nonblocking support, mandating that *all*
block would work just as well and would not slow down the blocking
case with needless checks.

Yes this class of issues can be trivially solved by demanding threading,
but that does not help when a(n unrealistic?) design constraint limits
you to one thread.

I realized after posting the feature request for ready(), more(), done()
that $h->timeout() also would make sense.  Select(2) defines the timeout
as a time value or null for indefinate blocking, for those tuning in
late.

Without a mandated interface to non-blocking, drivers will all 
implement it differently. I think it is not only practical but a good
idea to define a standard interface before every driver that can support
a nonblocking mode does it differently.

@{$sth}{qw/more done/} can be defined in terms of $sth->{Active}:

sub more{  $_[0]->{Active}}
sub done{ !$_[0]->{Active}}

supporting $h->ready implies that a driver supports a non-blocking
mode, in which partial state is maintained in the handle and we are
waiting for something from the other end.  In a blocking driver,
$h->ready would always be true because we are never in an incomplete
state.  Blocking DBI either returns or throws, then that phase of the
operation is complete.

In a nonblocking situation, a driver might return immediately, but
register a callback with itself so that when ready() is called, the
callback will run, and the pending operation will either get completed
and ready() will be true, or the pending operation will still be
pending and ready() will still be false.  The ready() would throw
the deferred error.

To make things easier, (prepare execute fetch) could be mandated to all
internally do 

 readycheck: $me->ready() or goto readycheck;

as they start, or even better,

 { 
  my $oldtimeout = $me->{Timeout};
  undef $me->{Timeout}; 
  return $h->set_err($err, "Defered $errmsg", $state)
 unless $me->ready(); # mandated to finish or throw
  $me->{Timeout} = $oldtimeout
 }

Or even better still, a driver could stack requests up, and each would
become ready or not-ready on its own.  No flexibility is taken from the
driver implementors.

So, my proposal is, to declare three read-only access functions
and one attribute.  These are more, done, ready and {Timeout}.

If pointed to the simplest TCP-based driver, I will cheerfully 
add this stuff to demonstrate that its practical.

Referring to M. Peppler's link:
http://sybooks.sybase.com/onlinebooks/group-sd/sdg1251e/ctref/@Generic__BookTextView/1039;pt=799/*#X

the ready() function would be always true in Synchronous mode, would
invoke the ct_poll in Deferred async mode, and would check to see if
the callback has occurred automatically in fully synchronous mode:

   @AsyncStHandles = map {$dbh->prepare($_)} @Statements;
   do{
 $pending = 0;
 for(@AsyncStHandles){
   unless($_->ready()){
  $pending++;
  next;
   };
   ...
   $_->more and $pending++;
 }
   } while $pending

How cool is that?

Full support of the optional NonBlocking attribute (settable by
attempting to define the Timeout attribute?) in general would
do something like Deferred Asyncronous mode.

The exact reason the pending operation is not ready -- CS_BUSY,
EWOULDBLOCK, etc. could be described in $DBI_ERRSTR.  Ready shouldn't
throw just because it isn't ready, so set_err would not be the right
way to set the message.


J Leffler wrote:
> One

Re: interface ideas for non-blocking mode

[Tim Bunce]

On Thu, Aug 26, 2004 at 08:48:42AM -0700, Dean Arnold wrote:
> 
> While this handles async execution, esp. in support of multiconnection
> operations relevant to Teradata's parallel nature,
> it doesn't really handle async completion notification, and
> I'm not certain there's a clean, DBMS-independent way to support
> that without using timers, signals, or threads, all of which
> may be problematic.
> 
> Also, in many (most?) instances, driver support for Perl threads
> may obviate the need for an async API; in fact, I'd prefer to see
> driver developers focus first on thread support, since that doesn't
> really require any API definitions, and provides much the same
> capability.

I agree.

> Perhaps DBI layer enhancements to support sharing
> handles between threads would be useful as well.

See $h->take_imp_data()

and that reminds me to expand the Roadmap in relation to take_imp_data
and driver support for building handles around supplied 'imp data'.

Tim.

Re: interface ideas for non-blocking mode

[Dean Arnold]

H.Merijn Brand wrote:
On Thu 26 Aug 2004 10:04, Tim Bunce <[EMAIL PROTECTED]> wrote:
On Thu, Aug 26, 2004 at 12:02:19AM -0500, david nicol wrote:
In the recent roadmap announcement, Tim Bunce wrote:

=head2 Other Enhancements
* Support non-blocking mode for drivers that can enable it in their
client API.
I have just startedworking with DBI but I'm doing something that would
benefit from a non-blocking mode.
I would like a $sth->ready() function, which would start false but
become true (or throw errors when RaiseError is set) when there is
at least one whole row available to fetch, and a $sth->done() function
to indicate that there is nothing left to fetch.
We would all like many things but have to settle for what's practical.
The APIs of all the main databases which offer a non-blocking mode
(which includes at least ODBC and Oracle OCI) need to be studied first
to ensure that any DBI API design is workable for them.
We might as well start that now...
Can driver authors, and anyone else interested, please reply with a
short summary of the way non-blocking mode works for 'your' database.
Including URLs to relevant docs would be a *big* help.
After that there are deep and non-trivial issues about how non-blocking
mode would interact with the DBI dispatcher and DBI.pm library code.
But those issues can wait.

DBD::Teradata has async support via driver-specific
methods/attributes:
my @dbhs = ();
my @sths = ();
...connect N sessions, storing handles in @dbhs...
foreach (@dbhs) {
push @sths, $_->prepare('insert into table values(?,?,?)',
{ tdat_nowait => 1 });
}
foreach (@sths) {
$_->execute([shift @paramtuples]);
}
while (params to load) {
@avails = $drh->tdat_FirstAvailList([EMAIL PROTECTED], $timeout);
foreach (@avails) {
$rc = $sths[$_]->tdat_Realize();
$sths[$_]->execute([shift @paramtuples]);
}
}
The API also supports including filehandles in the list passed to
tdat_FirstAvailList, in order to handle other async I/O events.
While this handles async execution, esp. in support of multiconnection
operations relevant to Teradata's parallel nature,
it doesn't really handle async completion notification, and
I'm not certain there's a clean, DBMS-independent way to support
that without using timers, signals, or threads, all of which
may be problematic.
Also, in many (most?) instances, driver support for Perl threads
may obviate the need for an async API; in fact, I'd prefer to see
driver developers focus first on thread support, since that doesn't
really require any API definitions, and provides much the same
capability. Perhaps DBI layer enhancements to support sharing
handles between threads would be useful as well.
Dean Arnold
Presicient Corp.

Re: interface ideas for non-blocking mode

[Jonathan Leffler]

On Thu, 26 Aug 2004 09:04:07 +0100, Tim Bunce <[EMAIL PROTECTED]> wrote:
> We would all like many things but have to settle for what's practical.
> 
> The APIs of all the main databases which offer a non-blocking mode
> (which includes at least ODBC and Oracle OCI) need to be studied first
> to ensure that any DBI API design is workable for them.
> 
> We might as well start that now...
> 
> Can driver authors, and anyone else interested, please reply with a
> short summary of the way non-blocking mode works for 'your' database.
> Including URLs to relevant docs would be a *big* help.

The DBD::Informix interface, Informix ESQL/C, does not support a
non-blocking mode.

Whether the Informix ODBC drivers support a non-blocking mode is
debatable - I've not checked, and have no particular plans to do so. 
Manuals are available if you're curious:

http://www.ibm.com/software/data/informix/library

One of the issues I think that the specification will have to address,
probably restrictively, is whether you can have both an asynchronous
(non-blocking) statement and other synchronous (or, indeed, other
asynchronous) statements active on a single $dbh -- I suspect that the
portable answer will be "No; only one statement, whether synchronous
or asynchronous, can be active on a $dbh at any given time".

Also, I think we should review whether Perl threading might be an
alternative (better?) way to achieve asynchronous operations. 
DBD::Informix and ESQL/C could support multi-threading with some work
-- basically, to ensure that any 'globals' are allocated per thread
when necessary.  You'd then be able to have multiple $dbh -
connections - to  a database, and different threads would be able to
use different $dbh synchronously (within the thread) but concurrently
(across threads).

-- 
Jonathan Leffler <[EMAIL PROTECTED]>  #include 
Guardian of DBD::Informix - v2003.04 - http://dbi.perl.org
"I don't suffer from insanity - I enjoy every minute of it."

Re: interface ideas for non-blocking mode

[H.Merijn Brand]

On Thu 26 Aug 2004 10:04, Tim Bunce <[EMAIL PROTECTED]> wrote:
> On Thu, Aug 26, 2004 at 12:02:19AM -0500, david nicol wrote:
> > 
> > In the recent roadmap announcement, Tim Bunce wrote:
> > 
> > > =head2 Other Enhancements
> > > 
> > > * Support non-blocking mode for drivers that can enable it in their
> > >  client API.
> > 
> > I have just startedworking with DBI but I'm doing something that would
> > benefit from a non-blocking mode.
> > 
> > I would like a $sth->ready() function, which would start false but
> > become true (or throw errors when RaiseError is set) when there is
> > at least one whole row available to fetch, and a $sth->done() function
> > to indicate that there is nothing left to fetch.
> 
> We would all like many things but have to settle for what's practical.
> 
> The APIs of all the main databases which offer a non-blocking mode
> (which includes at least ODBC and Oracle OCI) need to be studied first
> to ensure that any DBI API design is workable for them.
> 
> We might as well start that now...
> 
> Can driver authors, and anyone else interested, please reply with a
> short summary of the way non-blocking mode works for 'your' database.
> Including URLs to relevant docs would be a *big* help.
> 
> After that there are deep and non-trivial issues about how non-blocking
> mode would interact with the DBI dispatcher and DBI.pm library code.
> But those issues can wait.

I have people from Unify visiting here this afternoon. I'll ask

-- 
H.Merijn BrandAmsterdam Perl Mongers (http://amsterdam.pm.org/)
using perl-5.6.1, 5.8.0 & 633 on HP-UX 10.20 & 11.00, AIX 4.2, AIX 4.3,
 WinNT 4, Win2K pro & WinCE 2.11 often with Tk800.024 &/| DBD-Unify
ftp://ftp.funet.fi/pub/languages/perl/CPAN/authors/id/H/HM/HMBRAND/

Re: interface ideas for non-blocking mode

[Steffen Goeldner]

Tim Bunce wrote:

[...]
Can driver authors, and anyone else interested, please reply with a
short summary of the way non-blocking mode works for 'your' database.
Including URLs to relevant docs would be a *big* help.
Don't know much about this, but it looks like ADO uses events:
  
Steffen

Re: interface ideas for non-blocking mode

[Michael Peppler]

On Thu, 2004-08-26 at 10:04, Tim Bunce wrote:
> On Thu, Aug 26, 2004 at 12:02:19AM -0500, david nicol wrote:
> > 
> > In the recent roadmap announcement, Tim Bunce wrote:
> > 
> > > =head2 Other Enhancements
> > > 
> > > * Support non-blocking mode for drivers that can enable it in their
> > >  client API.
> > 
> > I have just startedworking with DBI but I'm doing something that would
> > benefit from a non-blocking mode.
> > 
> > I would like a $sth->ready() function, which would start false but
> > become true (or throw errors when RaiseError is set) when there is
> > at least one whole row available to fetch, and a $sth->done() function
> > to indicate that there is nothing left to fetch.
> 
> We would all like many things but have to settle for what's practical.
> 
> The APIs of all the main databases which offer a non-blocking mode
> (which includes at least ODBC and Oracle OCI) need to be studied first
> to ensure that any DBI API design is workable for them.
> 
> We might as well start that now...
> 
> Can driver authors, and anyone else interested, please reply with a
> short summary of the way non-blocking mode works for 'your' database.
> Including URLs to relevant docs would be a *big* help.

Sybase's Client-Library API has a couple of options, but in general you
can say that the client sends the query to the server in
async/non-blocking mode and receives notification of pending results in
a "completion" callback function.

I haven't used it much, though I did implement the basic functionality
in the Sybase::CTlib module.

See:

http://sybooks.sybase.com/onlinebooks/group-sd/sdg1251e/ctref/@Generic__BookTextView/1039;pt=799/*#X

Michael
-- 
Michael Peppler  Data Migrations, Inc.
[EMAIL PROTECTED]   http://www.peppler.org/
Sybase T-SQL/OpenClient/OpenServer/C/Perl developer available for short
or long term contract positions - http://www.peppler.org/resume.html

Re: interface ideas for non-blocking mode

[Tim Bunce]

On Thu, Aug 26, 2004 at 12:02:19AM -0500, david nicol wrote:
> 
> In the recent roadmap announcement, Tim Bunce wrote:
> 
> > =head2 Other Enhancements
> > 
> > * Support non-blocking mode for drivers that can enable it in their
> >  client API.
> 
> I have just startedworking with DBI but I'm doing something that would
> benefit from a non-blocking mode.
> 
> I would like a $sth->ready() function, which would start false but
> become true (or throw errors when RaiseError is set) when there is
> at least one whole row available to fetch, and a $sth->done() function
> to indicate that there is nothing left to fetch.

We would all like many things but have to settle for what's practical.

The APIs of all the main databases which offer a non-blocking mode
(which includes at least ODBC and Oracle OCI) need to be studied first
to ensure that any DBI API design is workable for them.

We might as well start that now...

Can driver authors, and anyone else interested, please reply with a
short summary of the way non-blocking mode works for 'your' database.
Including URLs to relevant docs would be a *big* help.

After that there are deep and non-trivial issues about how non-blocking
mode would interact with the DBI dispatcher and DBI.pm library code.
But those issues can wait.

Tim.