Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-07-13 Thread Peter Geoghegan
On Thu, Jul 2, 2020 at 8:47 AM James Coleman wrote: > It seems like the consensus over at another discussion on this topic > [1] is that we ought to go ahead and print the zeros [for machine > readable output formats], even though that creates some interesting > scenarios like the fact that disk s

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-07-13 Thread Peter Geoghegan
On Thu, Jul 9, 2020 at 12:06 PM Jonathan S. Katz wrote: > On 7/2/20 11:47 AM, James Coleman wrote: > > It seems like the consensus over at another discussion on this topic > > [1] is that we ought to go ahead and print the zeros [for machine > > readable output formats], even though that creates s

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-07-09 Thread Jonathan S. Katz
On 7/2/20 11:47 AM, James Coleman wrote: > It seems like the consensus over at another discussion on this topic > [1] is that we ought to go ahead and print the zeros [for machine > readable output formats], even though that creates some interesting > scenarios like the fact that disk sorts will pr

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-07-02 Thread James Coleman
It seems like the consensus over at another discussion on this topic [1] is that we ought to go ahead and print the zeros [for machine readable output formats], even though that creates some interesting scenarios like the fact that disk sorts will print 0 for memory even though that's not true. Th

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-06-24 Thread James Coleman
On Fri, Jun 19, 2020 at 12:04 AM Justin Pryzby wrote: > > On Tue, Apr 07, 2020 at 08:40:30AM -0400, James Coleman wrote: > > On Tue, Apr 7, 2020 at 12:25 AM Justin Pryzby wrote: > > > On Mon, Apr 06, 2020 at 09:57:22PM +0200, Tomas Vondra wrote: > > > > I've pushed the fist part of this patch ser

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-06-18 Thread Justin Pryzby
On Tue, Apr 07, 2020 at 08:40:30AM -0400, James Coleman wrote: > On Tue, Apr 7, 2020 at 12:25 AM Justin Pryzby wrote: > > On Mon, Apr 06, 2020 at 09:57:22PM +0200, Tomas Vondra wrote: > > > I've pushed the fist part of this patch series - I've reorganized it a > > > > I scanned through this again

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-05-12 Thread Peter Geoghegan
On Tue, May 12, 2020 at 11:18 AM Tomas Vondra wrote: > I've pushed both patches, fixing typos and explain format. Thanks, Tomas. -- Peter Geoghegan

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-05-12 Thread Tomas Vondra
On Sun, May 10, 2020 at 02:25:23PM +0200, Tomas Vondra wrote: On Sat, May 09, 2020 at 06:48:02PM -0700, Peter Geoghegan wrote: On Sat, May 9, 2020 at 3:19 PM Tomas Vondra wrote: I'm generally OK with most of this - I'd probably keep the single-line format, but I don't feel very strongly about

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-05-10 Thread Tomas Vondra
On Sat, May 09, 2020 at 06:48:02PM -0700, Peter Geoghegan wrote: On Sat, May 9, 2020 at 3:19 PM Tomas Vondra wrote: I'm generally OK with most of this - I'd probably keep the single-line format, but I don't feel very strongly about that and if others think using two lines is better ... Barring

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-05-09 Thread Peter Geoghegan
On Sat, May 9, 2020 at 3:19 PM Tomas Vondra wrote: > I'm generally OK with most of this - I'd probably keep the single-line > format, but I don't feel very strongly about that and if others think > using two lines is better ... > > Barring objections I'll get this polished and pushed soon-ish (say

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-05-09 Thread Tomas Vondra
On Sat, May 09, 2020 at 10:50:12PM +0200, Tomas Vondra wrote: On Sat, May 09, 2020 at 03:18:36PM -0500, Justin Pryzby wrote: Checking if it's possible to address this Opened Item before 13b1. https://wiki.postgresql.org/wiki/PostgreSQL_13_Open_Items consistency of explain output: two spaces, eq

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-05-09 Thread Tomas Vondra
On Sat, May 09, 2020 at 03:18:36PM -0500, Justin Pryzby wrote: Checking if it's possible to address this Opened Item before 13b1. https://wiki.postgresql.org/wiki/PostgreSQL_13_Open_Items consistency of explain output: two spaces, equals vs colons, semicolons (incremental sort) Yes. Now that

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-05-09 Thread Justin Pryzby
Checking if it's possible to address this Opened Item before 13b1. https://wiki.postgresql.org/wiki/PostgreSQL_13_Open_Items consistency of explain output: two spaces, equals vs colons, semicolons (incremental sort) On Sun, Apr 19, 2020 at 09:46:55AM -0400, James Coleman wrote: > On Sat, Apr 18

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-04-19 Thread James Coleman
On Sat, Apr 18, 2020 at 10:36 PM Justin Pryzby wrote: > > On Tue, Apr 07, 2020 at 10:53:05AM -0500, Justin Pryzby wrote: > > On Tue, Apr 07, 2020 at 08:40:30AM -0400, James Coleman wrote: > > > > And, should it use two spaces before "Sort Method", "Memory" and > > > > "Pre-sorted > > ... > > > I

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-04-18 Thread Justin Pryzby
On Tue, Apr 07, 2020 at 10:53:05AM -0500, Justin Pryzby wrote: > On Tue, Apr 07, 2020 at 08:40:30AM -0400, James Coleman wrote: > > > And, should it use two spaces before "Sort Method", "Memory" and > > > "Pre-sorted > ... > > I read through that subthread, and the ending seemed to be Peter > > wa

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-04-16 Thread James Coleman
On Thu, Apr 16, 2020 at 1:10 PM Tom Lane wrote: > > James Coleman writes: > > On Fri, Apr 10, 2020 at 10:12 AM James Coleman wrote: > >> One thing I just noticed and had a question about: in > >> preparePresortedCols (which sets up a function call context), do we > >> need to call pg_proc_aclche

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-04-16 Thread Tom Lane
James Coleman writes: > On Fri, Apr 10, 2020 at 10:12 AM James Coleman wrote: >> One thing I just noticed and had a question about: in >> preparePresortedCols (which sets up a function call context), do we >> need to call pg_proc_aclcheck? > Background: this came up because I noticed that pg_pro

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-04-16 Thread James Coleman
On Fri, Apr 10, 2020 at 10:12 AM James Coleman wrote: > > One thing I just noticed and had a question about: in > preparePresortedCols (which sets up a function call context), do we > need to call pg_proc_aclcheck? Background: this came up because I noticed that pg_proc_aclcheck is called in the

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-04-10 Thread James Coleman
One thing I just noticed and had a question about: in preparePresortedCols (which sets up a function call context), do we need to call pg_proc_aclcheck? James

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-04-08 Thread Tomas Vondra
On Wed, Apr 08, 2020 at 09:54:42AM -0400, James Coleman wrote: On Wed, Apr 8, 2020 at 9:43 AM Tomas Vondra wrote: On Wed, Apr 08, 2020 at 12:51:05PM +0200, Tomas Vondra wrote: >On Tue, Apr 07, 2020 at 11:54:23PM -0400, Tom Lane wrote: >>hyrax is not too happy with this test: >> >>https://build

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-04-08 Thread Tomas Vondra
On Wed, Apr 08, 2020 at 11:13:26AM -0400, James Coleman wrote: On Wed, Apr 8, 2020 at 11:02 AM Tomas Vondra wrote: On Wed, Apr 08, 2020 at 04:08:39PM +0200, Tomas Vondra wrote: >On Wed, Apr 08, 2020 at 09:54:42AM -0400, James Coleman wrote: >>On Wed, Apr 8, 2020 at 9:43 AM Tomas Vondra >> wrot

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-04-08 Thread Tom Lane
James Coleman writes: > Should we change `analyze` to `analyze t` to avoid unnecessarily > re-analyzing all other tables in the regression db? Yes, a global analyze here is a remarkably horrid idea. regards, tom lane

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-04-08 Thread James Coleman
On Wed, Apr 8, 2020 at 11:02 AM Tomas Vondra wrote: > > On Wed, Apr 08, 2020 at 04:08:39PM +0200, Tomas Vondra wrote: > >On Wed, Apr 08, 2020 at 09:54:42AM -0400, James Coleman wrote: > >>On Wed, Apr 8, 2020 at 9:43 AM Tomas Vondra > >> wrote: > >>> > >>>On Wed, Apr 08, 2020 at 12:51:05PM +0200, T

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-04-08 Thread Tomas Vondra
On Wed, Apr 08, 2020 at 04:08:39PM +0200, Tomas Vondra wrote: On Wed, Apr 08, 2020 at 09:54:42AM -0400, James Coleman wrote: On Wed, Apr 8, 2020 at 9:43 AM Tomas Vondra wrote: On Wed, Apr 08, 2020 at 12:51:05PM +0200, Tomas Vondra wrote: On Tue, Apr 07, 2020 at 11:54:23PM -0400, Tom Lane wro

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-04-08 Thread James Coleman
On Wed, Apr 8, 2020 at 9:43 AM Tomas Vondra wrote: > > On Wed, Apr 08, 2020 at 12:51:05PM +0200, Tomas Vondra wrote: > >On Tue, Apr 07, 2020 at 11:54:23PM -0400, Tom Lane wrote: > >>hyrax is not too happy with this test: > >> > >>https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=hyrax&dt=202

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-04-08 Thread Tomas Vondra
On Wed, Apr 08, 2020 at 12:51:05PM +0200, Tomas Vondra wrote: On Tue, Apr 07, 2020 at 11:54:23PM -0400, Tom Lane wrote: hyrax is not too happy with this test: https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=hyrax&dt=2020-04-07%2004%3A55%3A15 It's not too clear to me why CLOBBER_CACHE_A

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-04-08 Thread Tomas Vondra
On Tue, Apr 07, 2020 at 11:54:23PM -0400, Tom Lane wrote: hyrax is not too happy with this test: https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=hyrax&dt=2020-04-07%2004%3A55%3A15 It's not too clear to me why CLOBBER_CACHE_ALWAYS would be breaking EXPLAIN output, but it evidently is.

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-04-07 Thread Tom Lane
hyrax is not too happy with this test: https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=hyrax&dt=2020-04-07%2004%3A55%3A15 It's not too clear to me why CLOBBER_CACHE_ALWAYS would be breaking EXPLAIN output, but it evidently is. regards, tom lane

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-04-07 Thread Tomas Vondra
Hi, I've pushed the second part of this patch series, adding incremental sort to additional places in the planner. As explained in the commit message (and somewhere in this thread) we've decided to only update some of the places that require sorted input (and do create_sort). This might be overly

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-04-07 Thread James Coleman
On Tue, Apr 7, 2020 at 7:58 PM Tomas Vondra wrote: > > On Tue, Apr 07, 2020 at 07:50:26PM -0400, James Coleman wrote: > >On Tue, Apr 7, 2020 at 7:02 PM Tomas Vondra > > wrote: > >> > >> On Mon, Apr 06, 2020 at 11:25:21PM -0500, Justin Pryzby wrote: > >> >On Mon, Apr 06, 2020 at 09:57:22PM +0200, T

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-04-07 Thread Tomas Vondra
On Tue, Apr 07, 2020 at 07:50:26PM -0400, James Coleman wrote: On Tue, Apr 7, 2020 at 7:02 PM Tomas Vondra wrote: On Mon, Apr 06, 2020 at 11:25:21PM -0500, Justin Pryzby wrote: >On Mon, Apr 06, 2020 at 09:57:22PM +0200, Tomas Vondra wrote: >> I've pushed the fist part of this patch series - I'

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-04-07 Thread James Coleman
On Tue, Apr 7, 2020 at 7:02 PM Tomas Vondra wrote: > > On Mon, Apr 06, 2020 at 11:25:21PM -0500, Justin Pryzby wrote: > >On Mon, Apr 06, 2020 at 09:57:22PM +0200, Tomas Vondra wrote: > >> I've pushed the fist part of this patch series - I've reorganized it a > > > >I scanned through this again pos

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-04-07 Thread Tomas Vondra
On Mon, Apr 06, 2020 at 11:25:21PM -0500, Justin Pryzby wrote: On Mon, Apr 06, 2020 at 09:57:22PM +0200, Tomas Vondra wrote: I've pushed the fist part of this patch series - I've reorganized it a I scanned through this again post-commit. Find attached some suggestions. Thanks. The typo fix

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-04-07 Thread Justin Pryzby
On Tue, Apr 07, 2020 at 08:40:30AM -0400, James Coleman wrote: > On Tue, Apr 7, 2020 at 12:25 AM Justin Pryzby wrote: > > On Mon, Apr 06, 2020 at 09:57:22PM +0200, Tomas Vondra wrote: > > > I've pushed the fist part of this patch series - I've reorganized it a > > > > I scanned through this again

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-04-07 Thread Tomas Vondra
On Tue, Apr 07, 2020 at 08:40:30AM -0400, James Coleman wrote: On Tue, Apr 7, 2020 at 12:25 AM Justin Pryzby wrote: On Mon, Apr 06, 2020 at 09:57:22PM +0200, Tomas Vondra wrote: > I've pushed the fist part of this patch series - I've reorganized it a I scanned through this again post-commit.

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-04-07 Thread James Coleman
On Tue, Apr 7, 2020 at 12:25 AM Justin Pryzby wrote: > > On Mon, Apr 06, 2020 at 09:57:22PM +0200, Tomas Vondra wrote: > > I've pushed the fist part of this patch series - I've reorganized it a > > I scanned through this again post-commit. Find attached some suggestions. > > Shouldn't non-text ex

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-04-06 Thread Justin Pryzby
On Mon, Apr 06, 2020 at 09:57:22PM +0200, Tomas Vondra wrote: > I've pushed the fist part of this patch series - I've reorganized it a I scanned through this again post-commit. Find attached some suggestions. Shouldn't non-text explain output always show both disk *and* mem, including zeros ? S

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-04-06 Thread Tomas Vondra
On Mon, Apr 06, 2020 at 11:00:37PM -0400, Tom Lane wrote: Tomas Vondra writes: I came to the same conclusion (that the change in TuplesortMethod definiton is the culprit) a while ago and was about to push a fix that initialized it correctly in ExecSortInitializeDSM. But I agree reverting it bac

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-04-06 Thread Tom Lane
Tomas Vondra writes: > I came to the same conclusion (that the change in TuplesortMethod > definiton is the culprit) a while ago and was about to push a fix that > initialized it correctly in ExecSortInitializeDSM. But I agree reverting > it back to the old definition is probably better. Yeah, fo

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-04-06 Thread Tomas Vondra
On Mon, Apr 06, 2020 at 10:19:41PM -0400, Tom Lane wrote: James Coleman writes: Fair enough. Unsure if Tomas is still online to comment and/or push, but reverting SORT_TYPE_STILL_IN_PROGRESS back to 0 works for me as an initial fix. I'm guessing he went to bed, so I'll push a fix in a moment.

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-04-06 Thread Tom Lane
James Coleman writes: > Fair enough. Unsure if Tomas is still online to comment and/or push, > but reverting SORT_TYPE_STILL_IN_PROGRESS back to 0 works for me as an > initial fix. I'm guessing he went to bed, so I'll push a fix in a moment. The patch has survived enough test cycles here now to m

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-04-06 Thread James Coleman
On Mon, Apr 6, 2020 at 10:09 PM Tom Lane wrote: > > James Coleman writes: > > On Mon, Apr 6, 2020 at 9:46 PM Tom Lane wrote: > >> I think the correct fix is to change the enum declaration. > > > Hmm. I don't actually really like that, because it means the value > > here isn't actually semantical

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-04-06 Thread Tom Lane
James Coleman writes: > On Mon, Apr 6, 2020 at 9:46 PM Tom Lane wrote: >> I think the correct fix is to change the enum declaration. > Hmm. I don't actually really like that, because it means the value > here isn't actually semantically correct. That is, the sort type is > not "in progress"; it'

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-04-06 Thread James Coleman
On Mon, Apr 6, 2020 at 9:46 PM Tom Lane wrote: > > Tomas Vondra writes: > > I don't know, I've tried running the tests on a number of machines, > > similar to those failing. Rapsberry Pi, Fedora 31, ... and it worked > > everywhere while the failures seem consistent. > > On my machine, it reprodu

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-04-06 Thread Tom Lane
Tomas Vondra writes: > I don't know, I've tried running the tests on a number of machines, > similar to those failing. Rapsberry Pi, Fedora 31, ... and it worked > everywhere while the failures seem consistent. On my machine, it reproduces about one time in six with force_parallel_mode = regress.

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-04-06 Thread Tomas Vondra
On Mon, Apr 06, 2020 at 08:42:13PM -0400, Tom Lane wrote: Tomas Vondra writes: It doesn't seem to be particularly platform-specific, but I've been unable to reproduce it so far. It seems on older gcc versions, though. It's looking kind of like an uninitialized-memory problem. Note the latest

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-04-06 Thread Tomas Vondra
On Mon, Apr 06, 2020 at 07:27:19PM -0400, James Coleman wrote: On Mon, Apr 6, 2020 at 7:09 PM James Coleman wrote: On Mon, Apr 6, 2020 at 6:13 PM Tomas Vondra wrote: > > On Mon, Apr 06, 2020 at 05:47:48PM -0400, James Coleman wrote: > >On Mon, Apr 6, 2020 at 5:40 PM Tomas Vondra > > wrote: >

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-04-06 Thread James Coleman
On Mon, Apr 6, 2020 at 7:31 PM Tomas Vondra wrote: > > On Mon, Apr 06, 2020 at 07:09:11PM -0400, James Coleman wrote: > >On Mon, Apr 6, 2020 at 6:13 PM Tomas Vondra > > wrote: > >> > >> On Mon, Apr 06, 2020 at 05:47:48PM -0400, James Coleman wrote: > >> >On Mon, Apr 6, 2020 at 5:40 PM Tomas Vondra

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-04-06 Thread Tom Lane
Tomas Vondra writes: > It doesn't seem to be particularly platform-specific, but I've been > unable to reproduce it so far. It seems on older gcc versions, though. It's looking kind of like an uninitialized-memory problem. Note the latest from spurfowl, https://buildfarm.postgresql.org/cgi-bin/

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-04-06 Thread Tomas Vondra
On Mon, Apr 06, 2020 at 07:09:11PM -0400, James Coleman wrote: On Mon, Apr 6, 2020 at 6:13 PM Tomas Vondra wrote: On Mon, Apr 06, 2020 at 05:47:48PM -0400, James Coleman wrote: >On Mon, Apr 6, 2020 at 5:40 PM Tomas Vondra > wrote: >> >> On Mon, Apr 06, 2020 at 11:12:32PM +0200, Tomas Vondra wr

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-04-06 Thread James Coleman
On Mon, Apr 6, 2020 at 7:09 PM James Coleman wrote: > > On Mon, Apr 6, 2020 at 6:13 PM Tomas Vondra > wrote: > > > > On Mon, Apr 06, 2020 at 05:47:48PM -0400, James Coleman wrote: > > >On Mon, Apr 6, 2020 at 5:40 PM Tomas Vondra > > > wrote: > > >> > > >> On Mon, Apr 06, 2020 at 11:12:32PM +0200,

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-04-06 Thread James Coleman
On Mon, Apr 6, 2020 at 6:13 PM Tomas Vondra wrote: > > On Mon, Apr 06, 2020 at 05:47:48PM -0400, James Coleman wrote: > >On Mon, Apr 6, 2020 at 5:40 PM Tomas Vondra > > wrote: > >> > >> On Mon, Apr 06, 2020 at 11:12:32PM +0200, Tomas Vondra wrote: > >> >On Mon, Apr 06, 2020 at 04:54:38PM -0400, Al

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-04-06 Thread Tomas Vondra
On Mon, Apr 06, 2020 at 06:34:04PM -0400, Tom Lane wrote: Tomas Vondra writes: On Mon, Apr 06, 2020 at 05:51:43PM -0400, Tom Lane wrote: Well, it's *less* unhappy. thorntail is showing that the number of workers field is not stable; that will need to be masked. Yeah, I've already pushed a

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-04-06 Thread Tom Lane
Tomas Vondra writes: > On Mon, Apr 06, 2020 at 05:51:43PM -0400, Tom Lane wrote: >> Well, it's *less* unhappy. thorntail is showing that the number of >> workers field is not stable; that will need to be masked. > Yeah, I've already pushed a fix for that. But there seems to be another > failure

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-04-06 Thread Tomas Vondra
On Mon, Apr 06, 2020 at 05:51:43PM -0400, Tom Lane wrote: Tomas Vondra writes: OK, I've pushed a fix - this should make the buildfarm happy again. Well, it's *less* unhappy. thorntail is showing that the number of workers field is not stable; that will need to be masked. Yeah, I've alread

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-04-06 Thread Tomas Vondra
On Mon, Apr 06, 2020 at 05:47:48PM -0400, James Coleman wrote: On Mon, Apr 6, 2020 at 5:40 PM Tomas Vondra wrote: On Mon, Apr 06, 2020 at 11:12:32PM +0200, Tomas Vondra wrote: >On Mon, Apr 06, 2020 at 04:54:38PM -0400, Alvaro Herrera wrote: >>On 2020-Apr-06, Tom Lane wrote: >> >>>Locally, thin

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-04-06 Thread Tom Lane
Tomas Vondra writes: > OK, I've pushed a fix - this should make the buildfarm happy again. Well, it's *less* unhappy. thorntail is showing that the number of workers field is not stable; that will need to be masked. regards, tom lane

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-04-06 Thread James Coleman
On Mon, Apr 6, 2020 at 5:40 PM Tomas Vondra wrote: > > On Mon, Apr 06, 2020 at 11:12:32PM +0200, Tomas Vondra wrote: > >On Mon, Apr 06, 2020 at 04:54:38PM -0400, Alvaro Herrera wrote: > >>On 2020-Apr-06, Tom Lane wrote: > >> > >>>Locally, things pass without force_parallel_mode, but turning it on

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-04-06 Thread Tomas Vondra
On Mon, Apr 06, 2020 at 11:12:32PM +0200, Tomas Vondra wrote: On Mon, Apr 06, 2020 at 04:54:38PM -0400, Alvaro Herrera wrote: On 2020-Apr-06, Tom Lane wrote: Locally, things pass without force_parallel_mode, but turning it on produces failures that look similar to rhinoceros's (didn't examine

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-04-06 Thread James Coleman
On Mon, Apr 6, 2020 at 5:22 PM James Coleman wrote: > > On Mon, Apr 6, 2020 at 5:20 PM James Coleman wrote: > > > > On Mon, Apr 6, 2020 at 5:12 PM Tomas Vondra > > wrote: > > > > > > On Mon, Apr 06, 2020 at 04:54:38PM -0400, Alvaro Herrera wrote: > > > >On 2020-Apr-06, Tom Lane wrote: > > > > >

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-04-06 Thread James Coleman
On Mon, Apr 6, 2020 at 5:20 PM James Coleman wrote: > > On Mon, Apr 6, 2020 at 5:12 PM Tomas Vondra > wrote: > > > > On Mon, Apr 06, 2020 at 04:54:38PM -0400, Alvaro Herrera wrote: > > >On 2020-Apr-06, Tom Lane wrote: > > > > > >> Locally, things pass without force_parallel_mode, but turning it o

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-04-06 Thread James Coleman
On Mon, Apr 6, 2020 at 5:12 PM Tomas Vondra wrote: > > On Mon, Apr 06, 2020 at 04:54:38PM -0400, Alvaro Herrera wrote: > >On 2020-Apr-06, Tom Lane wrote: > > > >> Locally, things pass without force_parallel_mode, but turning it on > >> produces failures that look similar to rhinoceros's (didn't ex

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-04-06 Thread Tomas Vondra
On Mon, Apr 06, 2020 at 04:54:38PM -0400, Alvaro Herrera wrote: On 2020-Apr-06, Tom Lane wrote: Locally, things pass without force_parallel_mode, but turning it on produces failures that look similar to rhinoceros's (didn't examine other BF members). FWIW I looked at the eight failures there

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-04-06 Thread Alvaro Herrera
On 2020-Apr-06, Tom Lane wrote: > Locally, things pass without force_parallel_mode, but turning it on > produces failures that look similar to rhinoceros's (didn't examine > other BF members). FWIW I looked at the eight failures there were about fifteen minutes ago and they were all identical. I

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-04-06 Thread Tom Lane
Tomas Vondra writes: > On Mon, Apr 06, 2020 at 04:14:38PM -0400, Tom Lane wrote: >> Did you ever use force_parallel_mode = regress? > Ah, not sure - probably not in this round of tests and there were some > changes in the explain code. Thanks for the hint. Locally, things pass without force_para

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-04-06 Thread Tomas Vondra
On Mon, Apr 06, 2020 at 04:14:38PM -0400, Tom Lane wrote: Tomas Vondra writes: Hmmm, I see the buildfarm is not happy about it - a couple of animals failed, but some succeeded. The failure seems like a simple difference in explain output, but it's not clear why would it happen (and I've ran the

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-04-06 Thread Tom Lane
Tomas Vondra writes: > Hmmm, I see the buildfarm is not happy about it - a couple of animals > failed, but some succeeded. The failure seems like a simple difference > in explain output, but it's not clear why would it happen (and I've > ran the tests many times but never seen this failure). Did

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-04-06 Thread Tomas Vondra
On Mon, Apr 06, 2020 at 09:57:22PM +0200, Tomas Vondra wrote: Hi, I've pushed the fist part of this patch series - I've reorganized it a bit by moving the add_partial_path changes to the end. That way I've been able to add regression test demonstrating impact of the change on plans involving inc

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-04-05 Thread Tomas Vondra
On Sun, Apr 05, 2020 at 03:01:10PM +0200, Tomas Vondra wrote: On Thu, Apr 02, 2020 at 09:40:45PM -0400, James Coleman wrote: On Thu, Apr 2, 2020 at 8:46 PM James Coleman wrote: On Thu, Apr 2, 2020 at 8:20 PM Tomas Vondra wrote: ... 5) Overall, I think the costing is OK. I'm sure we'll find

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-04-02 Thread James Coleman
On Thu, Apr 2, 2020 at 8:46 PM James Coleman wrote: > > On Thu, Apr 2, 2020 at 8:20 PM Tomas Vondra > wrote: > > > > Hi, > > > > Thanks, the v52 looks almost ready. I've been looking at the two or > > three things I mentioned, and I have a couple of comments. > > > > > > 1) /* XXX comparison_cost

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-04-02 Thread James Coleman
On Thu, Apr 2, 2020 at 8:20 PM Tomas Vondra wrote: > > Hi, > > Thanks, the v52 looks almost ready. I've been looking at the two or > three things I mentioned, and I have a couple of comments. > > > 1) /* XXX comparison_cost shouldn't be 0? */ > > I'm not worried about this, because this is not rea

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-04-02 Thread Tomas Vondra
Hi, Thanks, the v52 looks almost ready. I've been looking at the two or three things I mentioned, and I have a couple of comments. 1) /* XXX comparison_cost shouldn't be 0? */ I'm not worried about this, because this is not really intriduced by this patch - create_sort_path has the same commen

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-04-01 Thread Tomas Vondra
On Wed, Apr 01, 2020 at 10:09:20PM -0400, James Coleman wrote: On Wed, Apr 1, 2020 at 5:42 PM Tomas Vondra wrote: ... I've realized the way get_useful_pathkeys_for_relation() is coded kinda works against the fastpath we added when comparing pathkeys. That depends on comparing pointers to the li

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-04-01 Thread Tomas Vondra
On Wed, Apr 01, 2020 at 09:05:27AM -0400, James Coleman wrote: On Tue, Mar 31, 2020 at 11:07 PM James Coleman wrote: On Tue, Mar 31, 2020 at 10:44 PM Tomas Vondra wrote: > > On Tue, Mar 31, 2020 at 10:12:29PM -0400, James Coleman wrote: > >On Tue, Mar 31, 2020 at 9:59 PM Tomas Vondra > > wrot

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-03-31 Thread Tomas Vondra
On Tue, Mar 31, 2020 at 10:12:29PM -0400, James Coleman wrote: On Tue, Mar 31, 2020 at 9:59 PM Tomas Vondra wrote: On Tue, Mar 31, 2020 at 08:42:47PM -0400, James Coleman wrote: >On Tue, Mar 31, 2020 at 8:38 PM Tomas Vondra > wrote: >> >> On Tue, Mar 31, 2020 at 08:11:15PM -0400, James Coleman

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-03-31 Thread James Coleman
On Tue, Mar 31, 2020 at 9:59 PM Tomas Vondra wrote: > > On Tue, Mar 31, 2020 at 08:42:47PM -0400, James Coleman wrote: > >On Tue, Mar 31, 2020 at 8:38 PM Tomas Vondra > > wrote: > >> > >> On Tue, Mar 31, 2020 at 08:11:15PM -0400, James Coleman wrote: > >> >On Tue, Mar 31, 2020 at 7:56 PM Tomas Von

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-03-31 Thread Tomas Vondra
On Tue, Mar 31, 2020 at 08:42:47PM -0400, James Coleman wrote: On Tue, Mar 31, 2020 at 8:38 PM Tomas Vondra wrote: On Tue, Mar 31, 2020 at 08:11:15PM -0400, James Coleman wrote: >On Tue, Mar 31, 2020 at 7:56 PM Tomas Vondra > wrote: >> >> On Tue, Mar 31, 2020 at 07:09:04PM -0400, James Coleman

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-03-31 Thread James Coleman
On Tue, Mar 31, 2020 at 8:38 PM Tomas Vondra wrote: > > On Tue, Mar 31, 2020 at 08:11:15PM -0400, James Coleman wrote: > >On Tue, Mar 31, 2020 at 7:56 PM Tomas Vondra > > wrote: > >> > >> On Tue, Mar 31, 2020 at 07:09:04PM -0400, James Coleman wrote: > >> >On Tue, Mar 31, 2020 at 6:54 PM Tomas Von

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-03-31 Thread Tomas Vondra
On Tue, Mar 31, 2020 at 08:11:15PM -0400, James Coleman wrote: On Tue, Mar 31, 2020 at 7:56 PM Tomas Vondra wrote: On Tue, Mar 31, 2020 at 07:09:04PM -0400, James Coleman wrote: >On Tue, Mar 31, 2020 at 6:54 PM Tomas Vondra > wrote: >> >> On Tue, Mar 31, 2020 at 06:35:32PM -0400, Tom Lane wrot

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-03-31 Thread Alvaro Herrera
On 2020-Mar-31, Tom Lane wrote: > James Coleman writes: > > On Tue, Mar 31, 2020 at 1:04 PM Tom Lane wrote: > >> Perhaps the semantics are such that that's actually sensible, but it's > >> far from a straightforward remapping of the old enum. > > > Right, I didn't see the explicit "= 0" in othe

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-03-31 Thread James Coleman
On Tue, Mar 31, 2020 at 7:56 PM Tomas Vondra wrote: > > On Tue, Mar 31, 2020 at 07:09:04PM -0400, James Coleman wrote: > >On Tue, Mar 31, 2020 at 6:54 PM Tomas Vondra > > wrote: > >> > >> On Tue, Mar 31, 2020 at 06:35:32PM -0400, Tom Lane wrote: > >> >Tomas Vondra writes: > >> >> In general, I th

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-03-31 Thread Tomas Vondra
On Tue, Mar 31, 2020 at 07:09:04PM -0400, James Coleman wrote: On Tue, Mar 31, 2020 at 6:54 PM Tomas Vondra wrote: On Tue, Mar 31, 2020 at 06:35:32PM -0400, Tom Lane wrote: >Tomas Vondra writes: >> In general, I think it'd be naive that we can make planner smarter with >> no extra overhead sp

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-03-31 Thread James Coleman
On Tue, Mar 31, 2020 at 6:54 PM Tomas Vondra wrote: > > On Tue, Mar 31, 2020 at 06:35:32PM -0400, Tom Lane wrote: > >Tomas Vondra writes: > >> In general, I think it'd be naive that we can make planner smarter with > >> no extra overhead spent on planning, and we can never accept patches > >> add

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-03-31 Thread Tomas Vondra
On Tue, Mar 31, 2020 at 06:35:32PM -0400, Tom Lane wrote: Tomas Vondra writes: In general, I think it'd be naive that we can make planner smarter with no extra overhead spent on planning, and we can never accept patches adding even tiny overhead. With that approach we'd probably end up with a t

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-03-31 Thread Tomas Vondra
On Tue, Mar 31, 2020 at 06:00:00PM -0400, James Coleman wrote: On Tue, Mar 31, 2020 at 5:53 PM Tomas Vondra wrote: On Tue, Mar 31, 2020 at 02:23:15PM -0400, James Coleman wrote: >On Mon, Mar 30, 2020 at 9:14 PM Tomas Vondra > wrote: >> >> The main thing I've been working on today is benchmarki

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-03-31 Thread Tom Lane
Tomas Vondra writes: > In general, I think it'd be naive that we can make planner smarter with > no extra overhead spent on planning, and we can never accept patches > adding even tiny overhead. With that approach we'd probably end up with > a trivial planner that generates just a single query pla

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-03-31 Thread James Coleman
On Tue, Mar 31, 2020 at 5:53 PM Tomas Vondra wrote: > > On Tue, Mar 31, 2020 at 02:23:15PM -0400, James Coleman wrote: > >On Mon, Mar 30, 2020 at 9:14 PM Tomas Vondra > > wrote: > >> > >> The main thing I've been working on today is benchmarking how this > >> affects planning. And I'm seeing a reg

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-03-31 Thread Tomas Vondra
On Tue, Mar 31, 2020 at 02:23:15PM -0400, James Coleman wrote: On Mon, Mar 30, 2020 at 9:14 PM Tomas Vondra wrote: The main thing I've been working on today is benchmarking how this affects planning. And I'm seeing a regression that worries me a bit, unfortunately. The test I'm doing is prett

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-03-31 Thread Tom Lane
James Coleman writes: > On Tue, Mar 31, 2020 at 1:04 PM Tom Lane wrote: >> Perhaps the semantics are such that that's actually sensible, but it's >> far from a straightforward remapping of the old enum. > Right, I didn't see the explicit "= 0" in other enums there, so it > made me wonder if it w

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-03-31 Thread James Coleman
On Tue, Mar 31, 2020 at 1:04 PM Tom Lane wrote: > > James Coleman writes: > > + * TuplesortMethod is used in a bitmask in Increment Sort's shared memory > > + * instrumentation so needs to have each value be a separate bit. > > >> I don't quite understand why you skipped "1". (Also, is the use o

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-03-31 Thread Tom Lane
James Coleman writes: > + * TuplesortMethod is used in a bitmask in Increment Sort's shared memory > + * instrumentation so needs to have each value be a separate bit. >> I don't quite understand why you skipped "1". (Also, is the use of zero >> a wise choice?) > The assignment of 0 was already

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-03-31 Thread James Coleman
On Tue, Mar 31, 2020 at 12:31 PM Alvaro Herrera wrote: > > On 2020-Mar-30, James Coleman wrote: > > > +/* > > + *Instruementation information for IncrementalSort > > + * > > + */ > > +typedef struct IncrementalSortGroupInfo > > +{ > > + int64 gro

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-03-31 Thread Alvaro Herrera
On 2020-Mar-30, James Coleman wrote: > +/* > + *Instruementation information for IncrementalSort > + * > + */ > +typedef struct IncrementalSortGroupInfo > +{ > + int64 groupCount; > + longmaxDiskSpaceUsed; > + long

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-03-30 Thread Tomas Vondra
On Mon, Mar 30, 2020 at 06:53:47PM -0400, James Coleman wrote: On Mon, Mar 30, 2020 at 8:24 AM Tomas Vondra wrote: On Sun, Mar 29, 2020 at 10:16:53PM -0400, James Coleman wrote: >On Sun, Mar 29, 2020 at 9:44 PM Tomas Vondra > wrote: >> >> Hi, >> >> Attached is a slightly reorganized patch seri

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-03-30 Thread Tomas Vondra
On Sun, Mar 29, 2020 at 10:16:53PM -0400, James Coleman wrote: On Sun, Mar 29, 2020 at 9:44 PM Tomas Vondra wrote: Hi, Attached is a slightly reorganized patch series. I've merged the fixes into the appropriate matches, and I've also combined the two patches adding incremental sort paths to a

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-03-29 Thread James Coleman
On Sun, Mar 29, 2020 at 9:44 PM Tomas Vondra wrote: > > Hi, > > Attached is a slightly reorganized patch series. I've merged the fixes > into the appropriate matches, and I've also combined the two patches > adding incremental sort paths to additional places in planner. > > A couple more comments:

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-03-28 Thread James Coleman
On Sat, Mar 28, 2020 at 11:14 PM Tomas Vondra wrote: > > On Sat, Mar 28, 2020 at 10:47:49PM -0400, James Coleman wrote: > >On Sat, Mar 28, 2020 at 6:59 PM Tomas Vondra > > wrote: > >> > >> Hi, > >> > >> Attached is my take on simplification of the useful pathkeyes thing. It > >> keeps the function

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-03-28 Thread Tomas Vondra
On Sat, Mar 28, 2020 at 10:47:49PM -0400, James Coleman wrote: On Sat, Mar 28, 2020 at 6:59 PM Tomas Vondra wrote: Hi, Attached is my take on simplification of the useful pathkeyes thing. It keeps the function, but it truncates query_pathkeys to only members with EC members in the relation. I

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-03-28 Thread Tomas Vondra
Hi, Attached is my take on simplification of the useful pathkeyes thing. It keeps the function, but it truncates query_pathkeys to only members with EC members in the relation. I think that's essentially the optimization you've proposed. I've also noticed an issue in explain output. EXPLAIN ANAL

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2020-03-28 Thread James Coleman
On Sat, Mar 28, 2020 at 5:30 PM Tomas Vondra wrote: > > On Sat, Mar 28, 2020 at 10:19:04AM -0400, James Coleman wrote: > >On Fri, Mar 27, 2020 at 10:58 PM Tomas Vondra > > wrote: > >> > >> ... > >> > >> The more I look at pathkeys_useful_for_ordering() the more I think the > >> get_useful_pathkeys

  1   2   3   >