Re: Ideas about a better API for postgres_fdw remote estimates

2020-09-08 Thread Ashutosh Bapat
On Wed, 9 Sep 2020 at 02:35, Tomas Vondra wrote > > I think that was the topic of *this* thread as started by Tom, but I now > realize Andrey steered it in the direction to allow re-using remote > stats. Which seems useful too, but it confused me a bit. > I didn't realize that the nearby thread

Re: Ideas about a better API for postgres_fdw remote estimates

2020-09-08 Thread Tomas Vondra
On Tue, Sep 08, 2020 at 05:55:09PM +0530, Ashutosh Bapat wrote: On Fri, 4 Sep 2020 at 20:27, Tomas Vondra wrote 4) I wonder if we actually want/need to simply output pg_statistic data verbatim like this. Is postgres_fdw actually going to benefit from it? I kinda doubt that, and my assumption

Re: Ideas about a better API for postgres_fdw remote estimates

2020-09-08 Thread Ashutosh Bapat
On Fri, 4 Sep 2020 at 20:27, Tomas Vondra wrote > > > 4) I wonder if we actually want/need to simply output pg_statistic data > verbatim like this. Is postgres_fdw actually going to benefit from it? I > kinda doubt that, and my assumption was that we'd return only a small > subset of the data, ne

Re: Ideas about a better API for postgres_fdw remote estimates

2020-09-07 Thread Andrey V. Lepikhov
On 9/4/20 6:23 PM, Ashutosh Bapat wrote: On Thu, 3 Sep 2020 at 10:44, Andrey V. Lepikhov mailto:a.lepik...@postgrespro.ru>> wrote: On 8/31/20 6:19 PM, Ashutosh Bapat wrote: > On Mon, Aug 31, 2020 at 3:36 PM Andrey V. Lepikhov > mailto:a.lepik...@postgrespro.ru>> wrote: >>

Re: Ideas about a better API for postgres_fdw remote estimates

2020-09-04 Thread Tomas Vondra
On Thu, Sep 03, 2020 at 10:14:41AM +0500, Andrey V. Lepikhov wrote: On 8/31/20 6:19 PM, Ashutosh Bapat wrote: On Mon, Aug 31, 2020 at 3:36 PM Andrey V. Lepikhov wrote: Thanks for this helpful feedback. I think the patch has some other problems like it works only for regular tables on foreign

Re: Ideas about a better API for postgres_fdw remote estimates

2020-09-04 Thread Ashutosh Bapat
On Thu, 3 Sep 2020 at 10:44, Andrey V. Lepikhov wrote: > On 8/31/20 6:19 PM, Ashutosh Bapat wrote: > > On Mon, Aug 31, 2020 at 3:36 PM Andrey V. Lepikhov > > wrote: > >> > >> Thanks for this helpful feedback. > > I think the patch has some other problems like it works only for > > regular tables

Re: Ideas about a better API for postgres_fdw remote estimates

2020-09-02 Thread Andrey V. Lepikhov
On 8/31/20 6:19 PM, Ashutosh Bapat wrote: On Mon, Aug 31, 2020 at 3:36 PM Andrey V. Lepikhov wrote: Thanks for this helpful feedback. I think the patch has some other problems like it works only for regular tables on foreign server but a foreign table can be pointing to any relation like a ma

Re: Ideas about a better API for postgres_fdw remote estimates

2020-08-31 Thread Andrey Lepikhov
On 8/31/20 6:19 PM, Ashutosh Bapat wrote: On Mon, Aug 31, 2020 at 3:36 PM Andrey V. Lepikhov wrote: I agreed that this task needs to implement an API for serialization/deserialization of statistics: pg_load_relation_statistics(json_string text); pg_get_relation_statistics(relname text); We can

Re: Ideas about a better API for postgres_fdw remote estimates

2020-08-31 Thread Bruce Momjian
On Mon, Aug 31, 2020 at 05:46:22PM -0400, Bruce Momjian wrote: > On Mon, Aug 31, 2020 at 01:53:01PM -0400, Tom Lane wrote: > > Stephen Frost writes: > > > Feature work either requires changes to pg_dump, or not. I agree that > > > features which don't require pg_dump changes are definitionally le

Re: Ideas about a better API for postgres_fdw remote estimates

2020-08-31 Thread Bruce Momjian
On Mon, Aug 31, 2020 at 01:53:01PM -0400, Tom Lane wrote: > Stephen Frost writes: > > Feature work either requires changes to pg_dump, or not. I agree that > > features which don't require pg_dump changes are definitionally less > > work than features which do (presuming the rest of the feature i

Re: Ideas about a better API for postgres_fdw remote estimates

2020-08-31 Thread Bruce Momjian
On Mon, Aug 31, 2020 at 01:26:59PM -0400, Stephen Frost wrote: > * Bruce Momjian (br...@momjian.us) wrote: > > I actually don't know which statement above is correct, because of the > > "forever" maintenance. > > I can understand not being sure which is correct, and we can all have > different poi

Re: Ideas about a better API for postgres_fdw remote estimates

2020-08-31 Thread Tom Lane
Stephen Frost writes: > Feature work either requires changes to pg_dump, or not. I agree that > features which don't require pg_dump changes are definitionally less > work than features which do (presuming the rest of the feature is the > same in both cases) but that isn't a justification to not

Re: Ideas about a better API for postgres_fdw remote estimates

2020-08-31 Thread Stephen Frost
Greetings, * Bruce Momjian (br...@momjian.us) wrote: > On Mon, Aug 31, 2020 at 12:56:21PM -0400, Stephen Frost wrote: > > The point I was making was that it has value and people did realize it > > but there's only so many resources to go around when it comes to hacking > > on PG and therefore it s

Re: Ideas about a better API for postgres_fdw remote estimates

2020-08-31 Thread Bruce Momjian
On Mon, Aug 31, 2020 at 12:56:21PM -0400, Stephen Frost wrote: > Greetings, > * Bruce Momjian (br...@momjian.us) wrote: > The point I was making was that it has value and people did realize it > but there's only so many resources to go around when it comes to hacking > on PG and therefore it simply

Re: Ideas about a better API for postgres_fdw remote estimates

2020-08-31 Thread Stephen Frost
Greetings, * Bruce Momjian (br...@momjian.us) wrote: > On Mon, Aug 31, 2020 at 12:19:52PM -0400, Stephen Frost wrote: > > * Bruce Momjian (br...@momjian.us) wrote: > > > I don't think there was enough value to do statistics migration just for > > > pg_upgrade, but doing it for pg_upgrade and FDWs

Re: Ideas about a better API for postgres_fdw remote estimates

2020-08-31 Thread Bruce Momjian
On Mon, Aug 31, 2020 at 12:19:52PM -0400, Stephen Frost wrote: > * Bruce Momjian (br...@momjian.us) wrote: > > I don't think there was enough value to do statistics migration just for > > pg_upgrade, but doing it for pg_upgrade and FDWs seems like it might > > have enough demand to justify the requ

Re: Ideas about a better API for postgres_fdw remote estimates

2020-08-31 Thread Stephen Frost
Greetings, * Bruce Momjian (br...@momjian.us) wrote: > On Sat, Aug 29, 2020 at 12:50:59PM -0400, Tom Lane wrote: > > Stephen Frost writes: > > > Isn't this going to create a version dependency that we'll need to deal > > > with..? What if a newer major version has some kind of improved ANALYZE >

Re: Ideas about a better API for postgres_fdw remote estimates

2020-08-31 Thread Bruce Momjian
On Sat, Aug 29, 2020 at 12:50:59PM -0400, Tom Lane wrote: > Stephen Frost writes: > > Isn't this going to create a version dependency that we'll need to deal > > with..? What if a newer major version has some kind of improved ANALYZE > > command, in terms of what it looks at or stores, and it's t

Re: Ideas about a better API for postgres_fdw remote estimates

2020-08-31 Thread Ashutosh Bapat
On Mon, Aug 31, 2020 at 3:36 PM Andrey V. Lepikhov wrote: > > Thanks for this helpful feedback. > > I found several threads related to the problem [1-3]. > I agreed that this task needs to implement an API for > serialization/deserialization of statistics: > pg_load_relation_statistics(json_string

Re: Ideas about a better API for postgres_fdw remote estimates

2020-08-31 Thread Andrey V. Lepikhov
On 8/29/20 9:50 PM, Tom Lane wrote: Years ago (when I was still at Salesforce, IIRC, so ~5 years) we had some discussions about making it possible for pg_dump and/or pg_upgrade to propagate stats data forward to the new database. There is at least one POC patch in the archives for doing that by

Re: Ideas about a better API for postgres_fdw remote estimates

2020-08-29 Thread Andrey Lepikhov
On 8/29/20 9:22 PM, Stephen Frost wrote: I implemented some FDW + pg core machinery to reduce weight of the problem. The ANALYZE command on foreign table executes query on foreign server that extracts statistics tuple, serializes it into json-formatted string and returns to the caller. The c

Re: Ideas about a better API for postgres_fdw remote estimates

2020-08-29 Thread Tom Lane
Stephen Frost writes: > Isn't this going to create a version dependency that we'll need to deal > with..? What if a newer major version has some kind of improved ANALYZE > command, in terms of what it looks at or stores, and it's talking to an > older server? Yeah, this proposal is a nonstarter

Re: Ideas about a better API for postgres_fdw remote estimates

2020-08-29 Thread Stephen Frost
Greetings, * Andrey Lepikhov (a.lepik...@postgrespro.ru) wrote: > During the implementation of sharding related improvements i noticed that if > we use a lot of foreign partitions, we have bad plans because of vacuum > don't update statistics of foreign tables.This is done by the ANALYZE > command

Re: Ideas about a better API for postgres_fdw remote estimates

2020-08-28 Thread Andrey Lepikhov
On 7/14/20 6:32 AM, Bruce Momjian wrote: On Mon, Jul 6, 2020 at 11:28:28AM -0400, Stephen Frost wrote: Yeah, thinking about it as a function that inspects partial planner results, it might be useful for other purposes besides postgres_fdw. As I said before, I don't think this necessarily has

Re: Ideas about a better API for postgres_fdw remote estimates

2020-07-13 Thread Bruce Momjian
On Mon, Jul 6, 2020 at 11:28:28AM -0400, Stephen Frost wrote: > > Yeah, thinking about it as a function that inspects partial planner > > results, it might be useful for other purposes besides postgres_fdw. > > As I said before, I don't think this necessarily has to be bundled as > > part of postg

Re: Ideas about a better API for postgres_fdw remote estimates

2020-07-06 Thread Stephen Frost
Greetings, * Tom Lane (t...@sss.pgh.pa.us) wrote: > Stephen Frost writes: > > * Tom Lane (t...@sss.pgh.pa.us) wrote: > >> Per postgres_fdw's get_remote_estimate(), the only data we use right now > >> is the startup_cost, total_cost, rows and width estimates from the > >> top-level Plan node. Tha

Re: Ideas about a better API for postgres_fdw remote estimates

2020-07-06 Thread Tom Lane
Stephen Frost writes: > * Tom Lane (t...@sss.pgh.pa.us) wrote: >> Per postgres_fdw's get_remote_estimate(), the only data we use right now >> is the startup_cost, total_cost, rows and width estimates from the >> top-level Plan node. That's available immediately from the Plan tree, >> meaning that

Re: Ideas about a better API for postgres_fdw remote estimates

2020-07-06 Thread Stephen Frost
Greetings, * Tom Lane (t...@sss.pgh.pa.us) wrote: > Stephen Frost writes: > > * Tom Lane (t...@sss.pgh.pa.us) wrote: > >> 2. Wedging this into EXPLAIN would be quite ugly, because (at least > >> as I envision it) the output would have just about nothing to do with > >> any existing EXPLAIN output

Re: Ideas about a better API for postgres_fdw remote estimates

2020-07-05 Thread Tom Lane
Stephen Frost writes: > * Tom Lane (t...@sss.pgh.pa.us) wrote: >> 2. Wedging this into EXPLAIN would be quite ugly, because (at least >> as I envision it) the output would have just about nothing to do with >> any existing EXPLAIN output. > This is a better argument for not making it part of EXPL

Re: Ideas about a better API for postgres_fdw remote estimates

2020-07-05 Thread Stephen Frost
Greetings, * Tom Lane (t...@sss.pgh.pa.us) wrote: > Stephen Frost writes: > > * Tom Lane (t...@sss.pgh.pa.us) wrote: > >> * Rather than adding a core-server feature, the remote-end part of the new > >> API should be a function installed by an extension (either postgres_fdw > >> itself, or a new e

Re: Ideas about a better API for postgres_fdw remote estimates

2020-07-05 Thread Tom Lane
Stephen Frost writes: > * Tom Lane (t...@sss.pgh.pa.us) wrote: >> * Rather than adding a core-server feature, the remote-end part of the new >> API should be a function installed by an extension (either postgres_fdw >> itself, or a new extension "postgres_fdw_remote" or the like). > I'm trying to

Re: Ideas about a better API for postgres_fdw remote estimates

2020-07-05 Thread Stephen Frost
Greetings, * Tom Lane (t...@sss.pgh.pa.us) wrote: > In <1116564.1593813...@sss.pgh.pa.us> I wrote: > > I wonder whether someday we ought to invent a new API that's more > > suited to postgres_fdw's needs than EXPLAIN is. It's not like the > > remote planner doesn't know the number we want; it jus

Ideas about a better API for postgres_fdw remote estimates

2020-07-03 Thread Tom Lane
In <1116564.1593813...@sss.pgh.pa.us> I wrote: > I wonder whether someday we ought to invent a new API that's more > suited to postgres_fdw's needs than EXPLAIN is. It's not like the > remote planner doesn't know the number we want; it just fails to > include it in EXPLAIN. I've been thinking abo