On Fri, Aug 3, 2012 at 12:01 AM, Tom Lane wrote:
> Marko Kreen writes:
>> On Thu, Aug 2, 2012 at 8:19 AM, Tom Lane wrote:
>>> In principle I suppose we could hack PQconsumeInput enough that it
>>> didn't damage the row buffer while still meeting its API contract of
>>> clearing the read-ready co
Marko Kreen writes:
> On Thu, Aug 2, 2012 at 8:19 AM, Tom Lane wrote:
>> In principle I suppose we could hack PQconsumeInput enough that it
>> didn't damage the row buffer while still meeting its API contract of
>> clearing the read-ready condition on the socket; but it wouldn't be
>> simple or c
On Thu, Aug 2, 2012 at 8:19 AM, Tom Lane wrote:
> Marko Kreen writes:
>> On Wed, Aug 1, 2012 at 6:18 PM, Tom Lane wrote:
>>> So I'm working from the first set of patches in your message
>>> <20120721194907.ga28...@gmail.com>.
>
>> Great!
>
> Here's an updated draft patch. I've not looked at the
Marko Kreen writes:
> On Wed, Aug 1, 2012 at 6:18 PM, Tom Lane wrote:
>> So I'm working from the first set of patches in your message
>> <20120721194907.ga28...@gmail.com>.
> Great!
Here's an updated draft patch. I've not looked at the docs yet so
this doesn't include that, but I'm reasonably
On Wed, Aug 1, 2012 at 6:18 PM, Tom Lane wrote:
> Marko Kreen writes:
>> * Is there better API than PQsetSingleRowMode()? New PQsend*
>> functions is my alternative.
>
> After thinking it over, I'm really unexcited about adding new versions
> of all the PQsend functions. If we had the prospec
[ getting back to this now after assorted distractions ]
Marko Kreen writes:
> Just to show agreement: both PQgetRowData() and optimized PGresult
> do not belong to 9.2.
OK, we're all on board with leaving those for later.
> Only open questions are:
> * Is there better API than PQsetSingleRowM
On Mon, Jul 30, 2012 at 10:26 PM, Jan Wieck wrote:
> On 7/30/2012 10:31 PM, Leon Smith wrote:
>>
>> This is not necessarily true, on multiple levels. I mean, some of
>> the programs I write are highly concurrent, and this form of batching
>> would have almost no risk of stalling the network b
On 7/30/2012 10:31 PM, Leon Smith wrote:
This is not necessarily true, on multiple levels. I mean, some of
the programs I write are highly concurrent, and this form of batching
would have almost no risk of stalling the network buffer.And
the possible use case would be when you are dealin
On Mon, Jul 30, 2012 at 9:59 PM, Jan Wieck wrote:
> On 7/30/2012 8:11 PM, Leon Smith wrote:
>
>> One other possibility, Tom Lane fretted ever so slightly about the use
>> of malloc/free per row... what about instead of PQsetSingleRowMode, you
>> have PQsetChunkedRowMode that takes a chunkSize p
On 7/30/2012 8:11 PM, Leon Smith wrote:
One other possibility, Tom Lane fretted ever so slightly about the use
of malloc/free per row... what about instead of PQsetSingleRowMode, you
have PQsetChunkedRowMode that takes a chunkSize parameter. A chunkSize
<= 0 would be equivalent to what we hav
Hey, this thread was pointed out to me just a few days ago, but I'll start
by saying that I think this thread is on exactly the right track. I don't
like the callback API, and think that PQsetSingleRowMode should be offered
in place of it. But I do have one
On Sat, Jun 16, 2012 at 10:22 AM, M
On Tuesday, July 24, 2012, Marko Kreen wrote:
> On Wed, Jul 25, 2012 at 1:29 AM, Merlin Moncure
wrote:
>> On Tue, Jul 24, 2012 at 4:59 PM, Marko Kreen wrote:
>>> So if we give only PQgetResult() in 9.2, I do not see that we
>>> are locked out from any interesting optimizations.
>>
>> Well, you a
On Wed, Jul 25, 2012 at 1:29 AM, Merlin Moncure wrote:
> On Tue, Jul 24, 2012 at 4:59 PM, Marko Kreen wrote:
>> So if we give only PQgetResult() in 9.2, I do not see that we
>> are locked out from any interesting optimizations.
>
> Well, you are locked out of having PQgetResult reuse the conn's r
On Tue, Jul 24, 2012 at 5:29 PM, Merlin Moncure wrote:
> so #2 seems like the lowest common
> denominator (it would permanently preclude #3 and would require #4 to
> introduce two new functions instead of just one). #1 of course would
> bolt on to #2.
oops, got #1 and #2 backwards there.
merlin
On Tue, Jul 24, 2012 at 4:59 PM, Marko Kreen wrote:
> So if we give only PQgetResult() in 9.2, I do not see that we
> are locked out from any interesting optimizations.
Well, you are locked out of having PQgetResult reuse the conn's result
since that would then introduce potentially breaking chan
On Wed, Jul 25, 2012 at 12:35 AM, Merlin Moncure wrote:
> On Tue, Jul 24, 2012 at 3:52 PM, Marko Kreen wrote:
>> * Is there better API than PQsetSingleRowMode()? New PQsend*
>> functions is my alternative.
>
> I would prefer to have PQsetSingleRowMode() over new flavor of PQsend.
>
> To consol
On Tue, Jul 24, 2012 at 3:52 PM, Marko Kreen wrote:
> * Is there better API than PQsetSingleRowMode()? New PQsend*
> functions is my alternative.
I would prefer to have PQsetSingleRowMode() over new flavor of PQsend.
To consolidate my argument above: since you're throwing PQgetResult in
the m
On Tue, Jul 24, 2012 at 10:37 PM, Tom Lane wrote:
> Merlin Moncure writes:
>> Either way, it looks like there's plausible paths to optimizing
>> repeated result fetch without having expose an alternate data-fetching
>> API and that the proposed implementation doesn't appear to be a wall
>> in ter
On Tue, Jul 24, 2012 at 11:34 PM, Merlin Moncure wrote:
> On Tue, Jul 24, 2012 at 2:37 PM, Tom Lane wrote:
>> In particular I agree that PQsetSingleRowMode is a bit ugly, so I'm
>> wondering about your thoughts on providing PQgetSingleRowResult instead.
>> I don't see how to make that work in asy
On Tue, Jul 24, 2012 at 2:37 PM, Tom Lane wrote:
> Merlin Moncure writes:
>> Either way, it looks like there's plausible paths to optimizing
>> repeated result fetch without having expose an alternate data-fetching
>> API and that the proposed implementation doesn't appear to be a wall
>> in term
> Hm, I think it's possible to rig the test to do dummy
> copy of pgresult, thus it's possible to see what kind
> of speed is possible.. Will try.
I added a new method (-x) to rowdump where it asks for row
with PQgetRowData() and then tries to emulate super-efficient
PGresult copy, then loads dat
Merlin Moncure writes:
> Either way, it looks like there's plausible paths to optimizing
> repeated result fetch without having expose an alternate data-fetching
> API and that the proposed implementation doesn't appear to be a wall
> in terms of getting there. So I'm firmly in the non-exposed-row
On Tue, Jul 24, 2012 at 1:49 PM, Merlin Moncure wrote:
> On Tue, Jul 24, 2012 at 1:33 PM, Merlin Moncure wrote:
>> The 'source' result (or source data that would be copied into the
>> destination result) would be stored in the PGconn, right? So, the idea
>> is that when you set up single row mode
On Tue, Jul 24, 2012 at 1:33 PM, Merlin Moncure wrote:
> The 'source' result (or source data that would be copied into the
> destination result) would be stored in the PGconn, right? So, the idea
> is that when you set up single row mode the connection generates a
> template PGconn which is then c
On Tue, Jul 24, 2012 at 11:57 AM, Marko Kreen wrote:
> On Tue, Jul 24, 2012 at 7:52 PM, Merlin Moncure wrote:
>> But, the faster rowbuf method is a generally incompatible way of
>> dealing with data vs current libpq -- this is bad. If it's truly
>> impossible to get those benefits without bypass
Merlin Moncure writes:
> I think the dummy copy of PGresult is plausible (if by that you mean
> optimizing PQgetResult when in single row mode). That would be even
> better: you'd remove the need for the rowbuf mode.
I haven't spent any time looking at this, but my gut tells me that a big
chunk
On Tue, Jul 24, 2012 at 7:52 PM, Merlin Moncure wrote:
> But, the faster rowbuf method is a generally incompatible way of
> dealing with data vs current libpq -- this is bad. If it's truly
> impossible to get those benefits without bypassing result API that
> then I remove my objection on the gro
On Tue, Jul 24, 2012 at 11:39 AM, Marko Kreen wrote:
> On Tue, Jul 24, 2012 at 7:25 PM, Merlin Moncure wrote:
>> On Tue, Jul 24, 2012 at 11:08 AM, Marko Kreen wrote:
I'm arguing that *all* data getting must continue to do so through the
result object, and bypassing the result to get at
On Tue, Jul 24, 2012 at 7:25 PM, Merlin Moncure wrote:
> On Tue, Jul 24, 2012 at 11:08 AM, Marko Kreen wrote:
>>> I'm arguing that *all* data getting must continue to do so through the
>>> result object, and bypassing the result to get at data is breaking the
>>> result abstraction in the libpq a
On Tue, Jul 24, 2012 at 7:08 PM, Marko Kreen wrote:
> The PQgetRowData() is unimportant as it just exposes
> the rowBuf to user and thats all.
So to get back to the more interesting PQgetRowData() discussion...
Did you notice that it's up to app to decide whether it calls
PQgetResult() or PQgetR
On Tue, Jul 24, 2012 at 11:08 AM, Marko Kreen wrote:
>> I'm arguing that *all* data getting must continue to do so through the
>> result object, and bypassing the result to get at data is breaking the
>> result abstraction in the libpq api. I bet you can still maintain
>> data access through resu
On Tue, Jul 24, 2012 at 6:18 PM, Merlin Moncure wrote:
> On Mon, Jul 23, 2012 at 5:07 PM, Marko Kreen wrote:
>>> Does PQgetRowData() break the ability to call PQgetvalue() against the
>>> result as well as other functions like PQgetisnull()? If so, I
>>> object.
>>
>> I don't get what are you ob
On Tue, Jul 24, 2012 at 10:49 AM, Tom Lane wrote:
> Merlin Moncure writes:
>> I'm arguing that *all* data getting must continue to do so through the
>> result object, and bypassing the result to get at data is breaking the
>> result abstraction in the libpq api.
>
> That's a fair point, but the s
Merlin Moncure writes:
> I'm arguing that *all* data getting must continue to do so through the
> result object, and bypassing the result to get at data is breaking the
> result abstraction in the libpq api.
That's a fair point, but the single-row mode without PQgetRowData still
fits that model,
On Mon, Jul 23, 2012 at 5:07 PM, Marko Kreen wrote:
>> Does PQgetRowData() break the ability to call PQgetvalue() against the
>> result as well as other functions like PQgetisnull()? If so, I
>> object.
>
> I don't get what are you objecting to. The PQgetRowData()
> is called instead of PQgetRes
On Tue, Jul 24, 2012 at 12:27 AM, Merlin Moncure wrote:
> On Mon, Jul 23, 2012 at 2:05 PM, Marko Kreen wrote:
> It seems odd (but maybe ok) that you have to set the single row mode
> on the connection only to have the server reset it whenever you call a
> send function: maybe rename to PQsetResul
On Mon, Jul 23, 2012 at 2:05 PM, Marko Kreen wrote:
>
> Here is a simple test program that takes a SELECT
> query, reads it and outputs a COPY-formatted stream
> to standard output, to simulate some activity.
>
> It operates on 3 modes, specified by commant-line switches:
>
> -f Load full result
Here is a simple test program that takes a SELECT
query, reads it and outputs a COPY-formatted stream
to standard output, to simulate some activity.
It operates on 3 modes, specified by commant-line switches:
-f Load full resultset at once - old way.
-s Single-Row mode using PQgetResult().
-
Marko Kreen writes:
> Here is 2 approaches how to get to state where only PQsetSingleRowMode()
> is available. Both on top of REL9_2_STABLE branch.
> a) Remove callback hooks, keep rowBuf, implement single-row-mode on
>top of that. This was posted before, I just additionally removed
>t
Here is 2 approaches how to get to state where only PQsetSingleRowMode()
is available. Both on top of REL9_2_STABLE branch.
a) Remove callback hooks, keep rowBuf, implement single-row-mode on
top of that. This was posted before, I just additionally removed
the PQgetRowData() function.
On Mon, Jul 16, 2012 at 6:47 PM, Tom Lane wrote:
> Marko Kreen writes:
>> On Mon, Jul 16, 2012 at 4:33 PM, Tom Lane wrote:
>>> Mm. I still think we should drop it, because it's still a dangerous API
>>> that's not necessary for the principal benefit of this feature.
>
>> Yes, it is a secondary
Marko Kreen writes:
> On Mon, Jul 16, 2012 at 4:33 PM, Tom Lane wrote:
>> Mm. I still think we should drop it, because it's still a dangerous API
>> that's not necessary for the principal benefit of this feature.
> Yes, it is a secondary feature, but it fits the needs of the actual target
> aud
On Mon, Jul 16, 2012 at 4:33 PM, Tom Lane wrote:
> Marko Kreen writes:
>> On Mon, Jul 16, 2012 at 4:11 AM, Tom Lane wrote:
>>> I'm starting to look at this patch now. I think we could drop the
>>> PQgetRowData() API: it complicates matters for little gain that I can
>>> see. The argument for i
Marko Kreen writes:
> On Mon, Jul 16, 2012 at 4:11 AM, Tom Lane wrote:
>> I'm starting to look at this patch now. I think we could drop the
>> PQgetRowData() API: it complicates matters for little gain that I can
>> see. The argument for it was to avoid the cost of creating a PGresult
>> per ro
On Mon, Jul 16, 2012 at 4:11 AM, Tom Lane wrote:
> Marko Kreen writes:
>> Now, looking at the problem with some perspective, the solution
>> is obvious: when in single-row mode, the PQgetResult() must return
>> proper PGresult for that single row. And everything else follows that.
>
>> Such API
Marko Kreen writes:
> Now, looking at the problem with some perspective, the solution
> is obvious: when in single-row mode, the PQgetResult() must return
> proper PGresult for that single row. And everything else follows that.
> Such API is implemented in attached patch:
I'm starting to look a
On 17 June 2012 19:37, Marko Kreen wrote:
> On Sun, Jun 17, 2012 at 2:07 PM, Simon Riggs wrote:
>> I prefer the description of Marko's API than the one we have now.
>>
>> Adopting one API in 9.2 and another in 9.3 would be fairly bad.
>> Perhaps we can have both?
>
> I see no reason the keep the
On Sun, Jun 17, 2012 at 2:07 PM, Simon Riggs wrote:
> I prefer the description of Marko's API than the one we have now.
>
> Adopting one API in 9.2 and another in 9.3 would be fairly bad.
> Perhaps we can have both?
I see no reason the keep the (public) callback API around,
except if we don't bot
On Sat, Jun 16, 2012 at 7:58 PM, Marko Kreen wrote:
> So my preference would be to simply remove the callback API
> but keep the processing and provide PQgetRowData() instead.
This is implemented in attached patch. It also
converts dblink to use single-row API.
The patch should be applied on to
On 16 June 2012 23:09, Tom Lane wrote:
> Marko Kreen writes:
>>> Now, looking at the problem with some perspective, the solution
>>> is obvious: when in single-row mode, the PQgetResult() must return
>>> proper PGresult for that single row. And everything else follows that.
>>>
>>> * PQgetRowDat
On Sat, Jun 16, 2012 at 6:09 PM, Tom Lane wrote:
> I guess this raises the question of whether we ought to revert the
> row-callback patch entirely and support only this approach. IMO
> it is (barely) not too late to do that for 9.2, if we want to.
> If we don't want to, then this is just another
Marko Kreen writes:
>> Now, looking at the problem with some perspective, the solution
>> is obvious: when in single-row mode, the PQgetResult() must return
>> proper PGresult for that single row. And everything else follows that.
>>
>> * PQgetRowData(): can be called instead PQgetResult() to ge
Demos:
https://github.com/markokr/libpq-rowproc-demos/blob/master/demo-onerow-sync.c
https://github.com/markokr/libpq-rowproc-demos/blob/master/demo-onerow-async.c
Few clarifications below.
On Fri, Jun 15, 2012 at 9:21 PM, Marko Kreen wrote:
> Now, looking at the problem with some perspective,
On Fri, Jun 15, 2012 at 1:21 PM, Marko Kreen wrote:
> The row-processor API is now in 9.2, but it solves only the
> "different-row-storage" problem, but not the "one-row-at-a-time"
> problem, as libpq is still in control until all rows are received.
>
> This means libpq cannet still be used to imp
The row-processor API is now in 9.2, but it solves only the
"different-row-storage" problem, but not the "one-row-at-a-time"
problem, as libpq is still in control until all rows are received.
This means libpq cannet still be used to implement iterative
result processing that almost all high-level
55 matches
Mail list logo
