Re: [PERFORM] Suspending SELECTs

2006-01-23 Thread Alessandro Baretta
August Zajonc wrote: Alessandro Baretta wrote: Alessandro, I've very much enjoyed reading your thoughts and the problem your facing and everyone's responses. Thank you for your interest, Agust. Since you control the middle layer, could you not use a cookie to keep a cursor open on the middl

Re: [PERFORM] Suspending SELECTs

2006-01-22 Thread August Zajonc
Alessandro Baretta wrote: > > What I could do relatively easily is instantiate a thread to iteratively > scan a traditional cursor N rows at a time, retrieving only record keys, > and finally send them to the query-cache-manager. The application thread > would then scan through the cursor results

Re: [PERFORM] Suspending SELECTs

2006-01-18 Thread mark
On Wed, Jan 18, 2006 at 03:41:57PM +, Harry Jackson wrote: > There are various reason why google might want to limit the search > result returned ie to encourage people to narrow their search. Prevent > screen scrapers from hitting them really hard blah blah. Perhaps less > than 0.0001% of

Re: [PERFORM] Suspending SELECTs

2006-01-18 Thread Harry Jackson
Your experiment made far too many assumptions and the data does not stand up to scrutiny. On 1/18/06, Alessandro Baretta <[EMAIL PROTECTED]> wrote: > Results: I'll omit the numerical data, which everyone can easily obtain in > only > a few minutes, repeating the experiment. I used several query s

Re: [PERFORM] Suspending SELECTs

2006-01-18 Thread Alessandro Baretta
[EMAIL PROTECTED] wrote: On Wed, Jan 18, 2006 at 09:57:50AM +0100, Alessandro Baretta wrote: I there is to be a change to PostgreSQL to optimize for this case, I suggest it involve the caching of query plans, executor plans, query results (materialized views?), LIMIT and OFFSET. If we had all of

Re: [PERFORM] Suspending SELECTs

2006-01-18 Thread mark
On Wed, Jan 18, 2006 at 09:57:50AM +0100, Alessandro Baretta wrote: > [EMAIL PROTECTED] wrote: > >On Tue, Jan 17, 2006 at 08:56:00PM +0100, Alessandro Baretta wrote: > >>I understand most of these issues, and expected this kind of reply. > >>Please, allow me to insist that we reason on this proble

Re: [PERFORM] Suspending SELECTs

2006-01-18 Thread Alessandro Baretta
Josh Berkus wrote: People: To follow up further, what Alessandro is talking about is known as a "keyset cursor". Sybase and SQL Server used to support them; I beleive that they were strictly read-only and had weird issues with record visibility. I would like to thank everyone for sharing

Re: [PERFORM] Suspending SELECTs

2006-01-18 Thread Tino Wildenhain
Alessandro Baretta schrieb: [EMAIL PROTECTED] wrote: ... It looks like this is the only possible solution at present--and in the future, too--but it has a tremendouse performance impact on queries returning thousands of rows. Well actually one of the better solutions would be persistent

Re: [PERFORM] Suspending SELECTs

2006-01-18 Thread Alessandro Baretta
[EMAIL PROTECTED] wrote: On Tue, Jan 17, 2006 at 08:56:00PM +0100, Alessandro Baretta wrote: I understand most of these issues, and expected this kind of reply. Please, allow me to insist that we reason on this problem and try to find a solution. My reason for doing so is that the future softw

Re: [PERFORM] Suspending SELECTs

2006-01-17 Thread Mark Kirkwood
Tom Lane wrote: Mark Kirkwood <[EMAIL PROTECTED]> writes: SELECT ... FROM table WHERE ... ORDER BY id LIMIT 20; Suppose this displays records for id 1 -> 10020. When the user hits next, and page saves id=10020 in the session state and executes: SELECT ... FROM table WHERE ... AND i

Re: [PERFORM] Suspending SELECTs

2006-01-17 Thread Tom Lane
Mark Kirkwood <[EMAIL PROTECTED]> writes: > SELECT ... FROM table WHERE ... ORDER BY id LIMIT 20; > Suppose this displays records for id 1 -> 10020. > When the user hits next, and page saves id=10020 in the session state > and executes: > SELECT ... FROM table WHERE ... AND id > 10020 ORDER

Re: [PERFORM] Suspending SELECTs

2006-01-17 Thread Craig A. James
Alessandro Baretta wrote: I think you're trying to do something at the wrong layer of your architecture. This task normally goes in your middleware layer, not your database layer. I am developing my applications in Objective Caml, and I have written the middleware layer myself. I could easil

Re: [PERFORM] Suspending SELECTs

2006-01-17 Thread Mark Kirkwood
[EMAIL PROTECTED] wrote: On Tue, Jan 17, 2006 at 08:56:00PM +0100, Alessandro Baretta wrote: What is wrong with LIMIT and OFFSET? I assume your results are ordered in some manner. Especially with web users, who become bored if the page doesn't flicker in a way that appeals to them, how could o

Re: [PERFORM] Suspending SELECTs

2006-01-17 Thread Josh Berkus
People: To follow up further, what Alessandro is talking about is known as a "keyset cursor". Sybase and SQL Server used to support them; I beleive that they were strictly read-only and had weird issues with record visibility. -- --Josh Josh Berkus Aglio Database Solutions San Francisco -

Re: [PERFORM] Suspending SELECTs

2006-01-17 Thread Josh Berkus
Alessandro, > I understand most of these issues, and expected this kind of reply. > Please, allow me to insist that we reason on this problem and try to > find a solution. My reason for doing so is that the future software > industry is likely to see more and more web applications retrieving data

Re: [PERFORM] Suspending SELECTs

2006-01-17 Thread Frank Wiles
On Tue, 17 Jan 2006 16:12:59 -0500 [EMAIL PROTECTED] wrote: > In the mean time, I successfully use LIMIT and OFFSET without such an > optimization, and things have been fine for me. Same here. - Frank Wiles <[EMAIL PROTECTED]> http://www.wiles.org -

Re: [PERFORM] Suspending SELECTs

2006-01-17 Thread Mark Lewis
> I am developing my applications in Objective Caml, and I have written the > middleware layer myself. I could easily implement a cursor-pooling strategy, > but > there is no perfect solution to the problem of guaranteeing that cursors be > closed. Remember that web applications require the use

Re: [PERFORM] Suspending SELECTs

2006-01-17 Thread mark
On Tue, Jan 17, 2006 at 08:56:00PM +0100, Alessandro Baretta wrote: > I understand most of these issues, and expected this kind of reply. Please, > allow me to insist that we reason on this problem and try to find a > solution. My reason for doing so is that the future software industry is > lik

Re: [PERFORM] Suspending SELECTs

2006-01-17 Thread Jim C. Nasby
On Tue, Jan 17, 2006 at 09:06:53PM +0100, Alessandro Baretta wrote: > Craig A. James wrote: > > > >Alessandro Baretta <[EMAIL PROTECTED]> writes: > > > >I think you're trying to do something at the wrong layer of your > >architecture. This task normally goes in your middleware layer, not > >your

Re: [PERFORM] Suspending SELECTs

2006-01-17 Thread Tom Lane
Alessandro Baretta <[EMAIL PROTECTED]> writes: > * When the cursor state is pushed back to the backend, no new > transaction is instantiated, but the XID of the original transaction > is reused. In the MVCC system, this allows us to achieve a perfectly > consistent view of the database at the insta

Re: [PERFORM] Suspending SELECTs

2006-01-17 Thread Alessandro Baretta
Craig A. James wrote: Alessandro Baretta <[EMAIL PROTECTED]> writes: I think you're trying to do something at the wrong layer of your architecture. This task normally goes in your middleware layer, not your database layer. I am developing my applications in Objective Caml, and I have writt

Re: [PERFORM] Suspending SELECTs

2006-01-17 Thread Michael Stone
On Tue, Jan 17, 2006 at 08:56:00PM +0100, Alessandro Baretta wrote: I understand most of these issues, and expected this kind of reply. Please, allow me to insist that we reason on this problem and try to find a solution. My reason for doing so is that the future software industry is likely to

Re: [PERFORM] Suspending SELECTs

2006-01-17 Thread Alessandro Baretta
Tom Lane wrote: Alessandro Baretta <[EMAIL PROTECTED]> writes: I am aware that what I am dreaming of is already available through cursors, but in a web application, cursors are bad boys, and should be avoided. What I would like to be able to do is to plan a query and run the plan to retreive a

Re: [PERFORM] Suspending SELECTs

2006-01-16 Thread Craig A. James
Alessandro Baretta <[EMAIL PROTECTED]> writes: I am aware that what I am dreaming of is already available through cursors, but in a web application, cursors are bad boys, and should be avoided. What I would like to be able to do is to plan a query and run the plan to retreive a limited number of

Re: [PERFORM] Suspending SELECTs

2006-01-16 Thread Mark Lewis
On Mon, 2006-01-16 at 11:13 +0100, Alessandro Baretta wrote: > I am aware that what I am dreaming of is already available through cursors, > but > in a web application, cursors are bad boys, and should be avoided. What I > would > like to be able to do is to plan a query and run the plan to ret

Re: [PERFORM] Suspending SELECTs

2006-01-16 Thread Tom Lane
Alvaro Herrera <[EMAIL PROTECTED]> writes: > I wonder if we could have a way to "suspend" a transaction and restart > it later in another backend. I think we could do something like this > using the 2PC machinery. > Not that I'm up for coding it; just an idea that crossed my mind. It's not imposs

Re: [PERFORM] Suspending SELECTs

2006-01-16 Thread Alvaro Herrera
Tom Lane wrote: > Alessandro Baretta <[EMAIL PROTECTED]> writes: > > I am aware that what I am dreaming of is already available through > > cursors, but in a web application, cursors are bad boys, and should be > > avoided. What I would like to be able to do is to plan a query and run > > the plan

Re: [PERFORM] Suspending SELECTs

2006-01-16 Thread Tom Lane
Alessandro Baretta <[EMAIL PROTECTED]> writes: > I am aware that what I am dreaming of is already available through > cursors, but in a web application, cursors are bad boys, and should be > avoided. What I would like to be able to do is to plan a query and run > the plan to retreive a limited numb

[PERFORM] Suspending SELECTs

2006-01-16 Thread Alessandro Baretta
I am aware that what I am dreaming of is already available through cursors, but in a web application, cursors are bad boys, and should be avoided. What I would like to be able to do is to plan a query and run the plan to retreive a limited number of rows as well as the executor's state. This way