Re: Check each of base restriction clauses for constant-FALSE-or-NULL

2023-10-10 Thread David Rowley
On Sat, 7 Oct 2023 at 22:44, Richard Guo wrote: > > In relation_excluded_by_constraints() when we're trying to figure out > whether the relation need not be scanned, one of the checks we do is to > detect constant-FALSE-or-NULL restriction clauses. Currently we perform > this check only when

Re: Use virtual tuple slot for Unique node

2023-10-10 Thread David Rowley
On Wed, 27 Sept 2023 at 20:01, David Rowley wrote: > > On Sat, 23 Sept 2023 at 03:15, Heikki Linnakangas wrote: > > So not a win in this case. Could you peek at the outer slot type, and > > use the same kind of slot for the Unique's result? Or some more > > complicated l

Re: Crash in add_paths_to_append_rel

2023-10-09 Thread David Rowley
On Mon, 9 Oct 2023 at 22:49, Richard Guo wrote: > I came across a crash in add_paths_to_append_rel() with the query below. > It was introduced by commit a8a968a8 where we referenced > cheapest_startup_path->param_info without first checking that we have a > valid cheapest_startup_path. I pushed

Re: Making aggregate deserialization (and WAL receive) functions slightly faster

2023-10-09 Thread David Rowley
On Tue, 10 Oct 2023 at 06:38, Tom Lane wrote: > Hm. I'd be inclined to use maxlen == 0 as the indicator of a read-only > buffer, just because that would not create a problem if we ever want > to change it to an unsigned type. Other than that, I agree with the > idea of using a special maxlen

Re: Crash in add_paths_to_append_rel

2023-10-09 Thread David Rowley
On Mon, 9 Oct 2023 at 22:49, Richard Guo wrote: > I came across a crash in add_paths_to_append_rel() with the query below. > It was introduced by commit a8a968a8 where we referenced > cheapest_startup_path->param_info without first checking that we have a > valid cheapest_startup_path. Thank

Re: Draft LIMIT pushdown to Append and MergeAppend patch

2023-10-09 Thread David Rowley
On Mon, 9 Oct 2023 at 23:35, Ashutosh Bapat wrote: > > On Mon, Oct 9, 2023 at 6:25 AM David Rowley wrote: > > > > However, it may also be worth you reading over [3] and the ultimate > > reason I changed my mind on that being a good idea. Pushing LIMITs > > below an

Re: Making aggregate deserialization (and WAL receive) functions slightly faster

2023-10-09 Thread David Rowley
On Mon, 9 Oct 2023 at 21:17, David Rowley wrote: > Here are some more thoughts on how we could improve this: > > 1. Adjust the definition of StringInfoData.maxlen to define that -1 > means the StringInfoData's buffer is externally managed. > 2. Adjust enlargeStringInfo() to add a c

Re: Making aggregate deserialization (and WAL receive) functions slightly faster

2023-10-09 Thread David Rowley
On Mon, 9 Oct 2023 at 17:37, Tom Lane wrote: > Sorry for not having paid more attention to this thread ... but > I'm pretty desperately unhappy with the patch as-pushed. I agree > with the criticism that this is a very repetitive coding pattern > that could have used a macro. But my real

Re: Making aggregate deserialization (and WAL receive) functions slightly faster

2023-10-08 Thread David Rowley
On Thu, 5 Oct 2023 at 21:24, David Rowley wrote: > > On Thu, 5 Oct 2023 at 18:23, Michael Paquier wrote: > > Ahem, well. Based on this argument my own argument does not hold > > much. Perhaps I'd still use a macro at the top of array_userfuncs.c > > and numeric.c, to

Re: pg16: XX000: could not find pathkey item to sort

2023-10-08 Thread David Rowley
On Mon, 9 Oct 2023 at 12:42, David Rowley wrote: > Maybe it's worth checking the total planning time spent in a run of > the regression tests with and without the patch to see how much > overhead it adds to the "average case". I've now pushed the patch that trims off the Pat

Re: Does anyone ever use OPTIMIZER_DEBUG?

2023-10-08 Thread David Rowley
On Mon, 9 Oct 2023 at 10:28, Tom Lane wrote: > > David Rowley writes: > > It looks like nobody is objecting to this. I understand that not > > everyone who might object will have read this email thread, so what I > > propose to do here is move along and just comm

Re: Draft LIMIT pushdown to Append and MergeAppend patch

2023-10-08 Thread David Rowley
On Sun, 8 Oct 2023 at 18:32, Michał Kłeczek wrote: > On 8 Oct 2023, at 03:33, Andy Fan wrote: >> For the patches for performance improvement, it is better to provide >> an example to show how much benefits we can get. As for this case, >> I'm doubtful it can work as an improvement. > Could

Re: pg16: XX000: could not find pathkey item to sort

2023-10-08 Thread David Rowley
On Sun, 8 Oct 2023 at 23:52, Richard Guo wrote: > On Thu, Oct 5, 2023 at 2:26 PM David Rowley wrote: >> >> So in short, I propose the attached fix without any regression tests >> because I feel that any regression test would just mark that there was >> a big in creat

Re: Problem, partition pruning for prepared statement with IS NULL clause.

2023-10-08 Thread David Rowley
On Sat, 7 Oct 2023 at 03:11, Sergei Glukhov wrote: > I noticed that combination of prepared statement with generic plan and > 'IS NULL' clause could lead partition pruning to crash. > Test case: > -- > set plan_cache_mode to force_generic_plan; > prepare stmt AS select * from hp where a is

Re: Does anyone ever use OPTIMIZER_DEBUG?

2023-10-08 Thread David Rowley
On Tue, 3 Oct 2023 at 12:29, David Rowley wrote: > > On Fri, 29 Sept 2023 at 10:59, Tom Lane wrote: > > We could also discuss keeping the "tracing" aspect of it, but > > replacing debug_print_rel with pprint(rel), which'd still allow > > removal of all the &

Re: Making aggregate deserialization (and WAL receive) functions slightly faster

2023-10-05 Thread David Rowley
On Thu, 5 Oct 2023 at 18:23, Michael Paquier wrote: > > On Wed, Oct 04, 2023 at 07:47:11PM +1300, David Rowley wrote: > > The original patch had a new function in stringinfo.c which allowed a > > StringInfoData to be initialised from an existing string with some > > g

Re: make add_paths_to_append_rel aware of startup cost

2023-10-05 Thread David Rowley
On Thu, 5 Oct 2023 at 14:11, Andy Fan wrote: > Patch LGTM. Thanks. Pushed. David

Re: [PATCH] Fix memory leak in memoize for numeric key

2023-10-05 Thread David Rowley
On Wed, 4 Oct 2023 at 21:08, Orlov Aleksej wrote: > I've finished testing the patch. > I confirm that the patch solves the problem and works just as fast. Thanks for checking that. I've pushed the patch now. David

Re: pg16: XX000: could not find pathkey item to sort

2023-10-05 Thread David Rowley
On Tue, 3 Oct 2023 at 20:16, David Rowley wrote: > I wonder if the attached patch is too much of a special case fix. I > guess from the lack of complaints previously that there are no other > cases where we could possibly have pathkeys that belong to columns > that are aggregated. I

Re: Making aggregate deserialization (and WAL receive) functions slightly faster

2023-10-04 Thread David Rowley
Thanks for taking a look at this. On Wed, 4 Oct 2023 at 16:57, Michael Paquier wrote: > + buf.len = VARSIZE_ANY_EXHDR(sstate); > + buf.maxlen = 0; > + buf.cursor = 0; > > Perhaps it would be worth hiding that in a macro defined in > stringinfo.h? The original patch had a new

Re: make add_paths_to_append_rel aware of startup cost

2023-10-03 Thread David Rowley
On Sun, 1 Oct 2023 at 21:26, Andy Fan wrote: >> But overall, I'm more inclined to just go with the more simple "add a >> cheap unordered startup append path if considering cheap startup >> plans" version. I see your latest patch does both. So, I'd suggest two >> patches as I do see the merit in

Re: [PATCH] Fix memory leak in memoize for numeric key

2023-10-03 Thread David Rowley
On Tue, 3 Oct 2023 at 19:38, Orlov Aleksej wrote: > I found a query which consumes a lot of memory and triggers OOM killer. > Memory leak occurs in memoize node for numeric key. Thanks for the analysis and the patch. > I've attached memoize_memory_leak_numeric_key.patch to address this. Yeah,

Re: pg16: XX000: could not find pathkey item to sort

2023-10-03 Thread David Rowley
On Tue, 3 Oct 2023 at 09:11, David Rowley wrote: > I'm concerned that this patch will be too much overhead when creating > paths when a PathKey's EquivalenceClass has a large number of members > from partitioned tables. I just tried out the patch to see how much it affects the pe

Re: Making aggregate deserialization (and WAL receive) functions slightly faster

2023-10-02 Thread David Rowley
On Sun, 12 Feb 2023 at 23:43, David Rowley wrote: > > On Sun, 12 Feb 2023 at 19:39, Tom Lane wrote: > > It could maybe be okay if we added the capability for StringInfoData > > to understand (and enforce) that its "data" buffer is read-only. > > However, that'd

Re: Fixup some more appendStringInfo misusages

2023-10-02 Thread David Rowley
On Tue, 3 Oct 2023 at 11:24, David Rowley wrote: > This is along the same lines as 8b26769bc, f736e188c, 110d81728 and 8abc13a88. I've pushed the patch to fix the misusages of the functions. David

Re: Does anyone ever use OPTIMIZER_DEBUG?

2023-10-02 Thread David Rowley
On Fri, 29 Sept 2023 at 10:59, Tom Lane wrote: > We could also discuss keeping the "tracing" aspect of it, but > replacing debug_print_rel with pprint(rel), which'd still allow > removal of all the "DEBUG SUPPORT" stuff at the bottom of allpaths.c. > That's pretty much all of the

Fixup some more appendStringInfo misusages

2023-10-02 Thread David Rowley
The attached v1-0001 patch adjusts some code in stringinfo.h to find misusages of the appendStringInfo functions. I don't intend to commit that, but I do intend to commit the patch for the new misusages that it found, which is also attached. This is along the same lines as 8b26769bc, f736e188c,

Re: pg16: XX000: could not find pathkey item to sort

2023-10-02 Thread David Rowley
On Tue, 19 Sept 2023 at 23:45, Richard Guo wrote: > My first thought about the fix is that we artificially add resjunk > target entries to parse->targetList for the ordered aggregates' > arguments that are ORDER BY expressions, as attached. While this can > fix the given query, it would cause

Re: Avoid a possible out-of-bounds access (src/backend/optimizer/util/relnode.c)

2023-09-28 Thread David Rowley
On Fri, 29 Sept 2023 at 01:00, Ranier Vilela wrote: > Perhaps, and using your own words, the leaders on this project seem > to be against reviewers armed with blunderbuss, too. I don't have any ideas on what you're talking about here, but if this is a valid concern that you think is unfair then

Does anyone ever use OPTIMIZER_DEBUG?

2023-09-28 Thread David Rowley
In c4a1933b4 I pushed a fix for a 4 year old bug in print_path() where I'd forgotten to add handling for TidRangePaths while working on bb437f995. 4 years is quite a long time for such a bug. Maybe that's because nobody uses OPTIMIZER_DEBUG. I certainly don't, and Tom mentions [1] he doesn't

Re: Is it worth adding Assert(false) for unknown paths in print_path()?

2023-09-28 Thread David Rowley
On Fri, 29 Sept 2023 at 03:23, Tom Lane wrote: > FWIW, I'd argue for dropping print_path rather than continuing to > maintain it. I never use it, finding pprint() to serve the need > better and more reliably. Then perhaps we just need to open a thread with an appropriate subject to check if

Is it worth adding Assert(false) for unknown paths in print_path()?

2023-09-28 Thread David Rowley
In [1] Andrey highlighted that I'd forgotten to add print_path() handling for TidRangePaths in bb437f995. I know the OPTIMIZER_DEBUG code isn't exactly well used. I never personally use it and I work quite a bit in the planner, however, if we're keeping it, I thought maybe we might get the memo

Re: Avoid a possible out-of-bounds access (src/backend/optimizer/util/relnode.c)

2023-09-27 Thread David Rowley
On Thu, 28 Sept 2023 at 02:37, Ranier Vilela wrote: >> Please check [1] for the mention of: >> >> "The fastest way to get your patch rejected is to make unrelated >> changes. Reformatting lines that haven't changed, changing unrelated >> comments you felt were poorly worded, touching code not

Re: Set enable_seqscan doesn't take effect?

2023-09-27 Thread David Rowley
On Thu, 28 Sept 2023 at 13:47, jacktby jacktby wrote: > > postgres=# SET enable_seqscan = off; > SET > postgres=# explain select * from t; >QUERY PLAN > - > Seq Scan on t

Re: Should consider materializing the cheapest inner path in consider_parallel_nestloop()

2023-09-27 Thread David Rowley
On Fri, 8 Sept 2023 at 19:14, Richard Guo wrote: > explain select * from partsupp join lineitem on l_partkey > ps_partkey; > QUERY PLAN > -- > Gather (cost=0.00..1807085.44

Re: Should consider materializing the cheapest inner path in consider_parallel_nestloop()

2023-09-27 Thread David Rowley
On Fri, 8 Sept 2023 at 09:41, Robert Haas wrote: > > On Tue, Sep 5, 2023 at 8:07 AM Richard Guo wrote: > > Yeah, this seems an omission in commit 45be99f8. > > It's been a while, but I think I omitted this deliberately because I > didn't really understand the value of it and wanted to keep the >

Re: make add_paths_to_append_rel aware of startup cost

2023-09-27 Thread David Rowley
On Mon, 18 Sept 2023 at 22:55, Andy Fan wrote: > Here is an updated version to show what's in my mind. My current thoughts on this are that the fractional cost part adds quite a bit more complexity than the minimalistic approach of just also considering the cheapest startup path. There's also

Re: Avoid a possible out-of-bounds access (src/backend/optimizer/util/relnode.c)

2023-09-27 Thread David Rowley
On Wed, 27 Sept 2023 at 06:10, Ranier Vilela wrote: > As suggested, casting is the best option that does not add any overhead and > improves the robustness of the find_base_rel function. > I propose patch v1, with the additional addition of fixing the > find_base_rel_ignore_join function, >

Re: Use virtual tuple slot for Unique node

2023-09-27 Thread David Rowley
On Sat, 23 Sept 2023 at 03:15, Heikki Linnakangas wrote: > So not a win in this case. Could you peek at the outer slot type, and > use the same kind of slot for the Unique's result? Or some more > complicated logic, like use a virtual slot if all the values are > pass-by-val? I'd also like to

Re: Correct the documentation for work_mem

2023-09-26 Thread David Rowley
On Tue, 12 Sept 2023 at 03:03, Bruce Momjian wrote: > > On Mon, Sep 11, 2023 at 10:02:55PM +1200, David Rowley wrote: > > It's certainly not a show-stopper. I do believe the patch makes some > > improvements. The reason I'd prefer to see either "and" or "and/or

Re: Avoid a possible out-of-bounds access (src/backend/optimizer/util/relnode.c)

2023-09-26 Thread David Rowley
On Wed, 27 Sept 2023 at 01:31, Ranier Vilela wrote: > It seems to me that it adds a LEA instruction. > https://godbolt.org/z/b4jK3PErE There's a fairly significant difference in the optimisability of a comparison with a compile-time constant vs a variable. For example, would you expect the

Re: Avoid a possible out-of-bounds access (src/backend/optimizer/util/relnode.c)

2023-09-26 Thread David Rowley
On Tue, 26 Sept 2023 at 21:45, Ashutosh Bapat wrote: > However, I agree that changing find_base_rel() the way you have done > in your patch is fine and mildly future-proof. +1 to that idea > irrespective of what bitmapset functions do. I'm not a fan of adding additional run-time overhead for

Re: Bug fix in vacuumdb --buffer-usage-limit xxx -Z

2023-09-21 Thread David Rowley
On Thu, 21 Sept 2023 at 17:59, Michael Paquier wrote: > That was fast. If I may ask, why don't you have some regression tests > for the two code paths of vacuumdb that append this option to the > commands generated for VACUUM and ANALYZE? I think we have differing standards for what constitutes

Re: Bug fix in vacuumdb --buffer-usage-limit xxx -Z

2023-09-20 Thread David Rowley
On Thu, 21 Sept 2023 at 16:18, David Rowley wrote: > Thanks for the report and the patch. I agree this has been overlooked. > > I also wonder if we should be escaping the buffer-usage-limit string > sent in the comment. It seems quite an unlikely attack vector, as the > user wou

Re: Bug fix in vacuumdb --buffer-usage-limit xxx -Z

2023-09-20 Thread David Rowley
On Thu, 21 Sept 2023 at 13:45, Ryoga Yoshida wrote: > When --buffer-usage-limit option is specified, vacuumdb issues VACUUM or > VACUUM ANALYZE command with BUFFER_USAGE_LIMIT option. Also if > --buffer-usage-limit and -Z options are specified, vacuumdb should issue > ANALYZE command with

Re: Comment about set_join_pathlist_hook()

2023-09-20 Thread David Rowley
On Wed, 20 Sept 2023 at 22:06, Etsuro Fujita wrote: > So I would like to propose to extend the comment to explain what they > can do, as in the comment about set_rel_pathlist_hook() in allpaths.c. > Attached is a patch for that. Looks good to me. I see you've copy/edited the comment just above

Re: disfavoring unparameterized nested loops

2023-09-20 Thread David Rowley
On Wed, 20 Sept 2023 at 19:56, Andrey Lepikhov wrote: > What could you say about a different way: hybrid join? In MS SQL Server, > they have such a feature [1], and, according to their description, it > requires low overhead. They start from HashJoin and switch to NestLoop > if the inner input

Re: Speed up transaction completion faster after many relations are accessed in a transaction

2023-09-17 Thread David Rowley
On Fri, 15 Sept 2023 at 22:37, Heikki Linnakangas wrote: > > I've added a call to LockAssertNoneHeld(false) in there. > > I don't see it in the patch? hmm. I must've git format-patch before committing that part. I'll try that again... see attached. David

Re: make add_paths_to_append_rel aware of startup cost

2023-09-17 Thread David Rowley
On Mon, 18 Sept 2023 at 01:42, Andy Fan wrote: > On Fri, Sep 15, 2023 at 3:15 PM David Rowley wrote: >> Instead of doing that, why don't you just create a completely new >> AppendPath containing all the cheapest_startup_paths and add that to >> the append rel. You can opti

Re: make add_paths_to_append_rel aware of startup cost

2023-09-15 Thread David Rowley
On Thu, 7 Sept 2023 at 04:37, Andy Fan wrote: > Currently add_paths_to_append_rel overlooked the startup cost for creating > append path, so it may have lost some optimization chances. After a glance, > the following 4 identifiers can be impacted. > - We shouldn't do the optimization if there

Re: What's the eviction algorithm of newest pg version?

2023-09-14 Thread David Rowley
On Fri, 15 Sept 2023 at 00:53, jacktby jacktby wrote: > A normal LRU cache or implement it reference to a research paper? src/backend/storage/buffer/README David

Re: Surely this code in setrefs.c is wrong?

2023-09-13 Thread David Rowley
On Sun, 10 Sept 2023 at 21:07, David Rowley wrote: > > On Sun, 10 Sept 2023 at 11:22, Tom Lane wrote: > > if (!OidIsValid(saop->hashfuncid)) > > record_plan_function_dependency(root, saop->hashfuncid); > > > >

Re: Redundant Unique plan node for table with a unique index

2023-09-13 Thread David Rowley
On Thu, 14 Sept 2023 at 02:28, Damir Belyalov wrote: > create table a (n int); > insert into a (n) select x from generate_series(1, 14) as g(x); > create unique index on a (n); > explain select distinct n from a; > QUERY PLAN >

Re: Speed up transaction completion faster after many relations are accessed in a transaction

2023-09-11 Thread David Rowley
Thank you for having a look at this. Apologies for not getting back to you sooner. On Wed, 5 Jul 2023 at 21:44, Heikki Linnakangas wrote: > > On 10/02/2023 04:51, David Rowley wrote: > > I've attached another set of patches. I do need to spend longer > > looking at this. I'

Re: Correct the documentation for work_mem

2023-09-11 Thread David Rowley
On Sat, 9 Sept 2023 at 14:25, Imseih (AWS), Sami wrote: > > > This looks mostly fine to me modulo "sort or hash". I do see many > > instances of "and/or" in the docs. Maybe that would work better. > > "sort or hash operations at the same time" is clear explanation IMO. Just for anyone else

Re: Surely this code in setrefs.c is wrong?

2023-09-10 Thread David Rowley
On Sun, 10 Sept 2023 at 11:22, Tom Lane wrote: > if (!OidIsValid(saop->hashfuncid)) > record_plan_function_dependency(root, saop->hashfuncid); > > if (!OidIsValid(saop->negfuncid)) > record_plan_function_dependency(root, saop->negfuncid); > > Surely those

Re: Correct the documentation for work_mem

2023-09-07 Thread David Rowley
On Fri, 8 Sept 2023 at 15:24, Bruce Momjian wrote: > Adjusted patch attached. This looks mostly fine to me modulo "sort or hash". I do see many instances of "and/or" in the docs. Maybe that would work better. David

Re: Use virtual tuple slot for Unique node

2023-08-30 Thread David Rowley
On Thu, 31 Aug 2023 at 05:37, Денис Смирнов wrote: > I have inspected the performance of the GROUP BY and DISTINCT queries for the > sorted data streams and found out, that Group node (produced by GROUP BY) > works faster then the Unique node (produced by DISTINCT). The flame graph > should

Re: Sync scan & regression tests

2023-08-30 Thread David Rowley
On Tue, 29 Aug 2023 at 22:35, Heikki Linnakangas wrote: > Looking the new heapgettup_advance_block() function and the code that it > replaced, it's now skipping this ss_report_location() on the last call, > when it has reached the end of the scan: > > > > > /* > >* Report our new

Re: Debian 12 gcc warning

2023-08-28 Thread David Rowley
On Tue, 29 Aug 2023 at 07:37, Bruce Momjian wrote: > nargs = 0; > foreach(lc, args) > { > actual_arg_types[nargs++] = exprType((Node *) lfirst(lc)); > } Does it still produce the warning if you form the above more like? nargs =

Re: meson uses stale pg_config_paths.h left over from make

2023-08-24 Thread David Rowley
On Thu, 24 Aug 2023 at 18:25, Andres Freund wrote: > > On 2023-08-24 08:18:14 +0200, Peter Eisentraut wrote: > > Surely meson should not be required to detect that? > > I think we should try to detect included files, due to the nasty and hard to > debug issues that creates. I've spent quite a bit

Re: Support run-time partition pruning for hash join

2023-08-24 Thread David Rowley
On Thu, 24 Aug 2023 at 21:27, Richard Guo wrote: > I performed some performance comparisons of the worst case with two > tables as below: > > 1. The partitioned table has 1000 children, and 100,000 tuples in total. > > 2. The other table is designed that > a) its tuples occupy every partition

Re: Avoid a possible overflow (src/backend/utils/sort/logtape.c)

2023-08-24 Thread David Rowley
On Fri, 25 Aug 2023 at 13:19, Michael Paquier wrote: > Still that looks entirely different to me. Here we have a problem > where the number of free blocks stored may cause an overflow in the > internal routine retrieving a free block, but your other thread > is about long being not enough on

Re: meson uses stale pg_config_paths.h left over from make

2023-08-23 Thread David Rowley
On Thu, 24 Aug 2023 at 00:52, David Rowley wrote: > Are there any objections to the attached being applied? Pushed. David

Re: PostgreSQL 16 release announcement draft

2023-08-23 Thread David Rowley
On Thu, 24 Aug 2023 at 05:55, Jonathan S. Katz wrote: > We could add something about 1349d2790 -- do you have suggested wording? I think it's worth a mention. See the text added in square brackets below: PostgreSQL 16 improves the performance of existing PostgreSQL functionality through new

meson uses stale pg_config_paths.h left over from make

2023-08-23 Thread David Rowley
I've been having some problems running the regression tests using meson on Windows. This seems to be down to initdb being compiled with a version of pg_config_paths.h left over from the msvc build which had been used on that source tree previously. Generally when there are files left over the

Re: PostgreSQL 16 release announcement draft

2023-08-23 Thread David Rowley
On Wed, 23 Aug 2023 at 22:21, jian he wrote: > > >>> > PostgreSQL 16 improves the performance of existing PostgreSQL functionality > through new query planner optimizations. In this latest release, the query > planner can parallelize `FULL` and `RIGHT` joins, utilize incremental sorts > for >

Re: Support run-time partition pruning for hash join

2023-08-22 Thread David Rowley
On Tue, 22 Aug 2023 at 00:34, Andy Fan wrote: > > On Mon, Aug 21, 2023 at 11:48 AM Richard Guo wrote: >> 1. All the join partition prunning decisions are made in createplan.c >>where the best path tree has been decided. This is not great. Maybe >>it's better to make it happen when we

Re: PG 16 draft release notes ready

2023-08-21 Thread David Rowley
s://www.postgresql.org/docs/16/release-16.html#RELEASE-16-PERFORMANCE > > > > Same as here: > > https://momjian.us/pgsql_docs/release-16.html#RELEASE-16-PERFORMANCE > > > > Allow window functions to use ROWS mode internally when RANGE mode is > > specified but un

Re: Report planning memory in EXPLAIN ANALYZE

2023-08-13 Thread David Rowley
On Thu, 10 Aug 2023 at 20:33, Ashutosh Bapat wrote: > My point is what's relevant here is how much net memory planner asked > for. But that's not what your patch is reporting. All you're reporting is the difference in memory that's *currently* palloc'd from before and after the planner ran. If

Re: Report planning memory in EXPLAIN ANALYZE

2023-08-09 Thread David Rowley
On Thu, 10 Aug 2023 at 03:12, Ashutosh Bapat wrote: > Thinking more about it, I think memory used is the only right metrics. > It's an optimization in MemoryContext implementation that malloc'ed > memory is not freed when it is returned using free(). I guess it depends on the problem you're

Re: Separate memory contexts for relcache and catcache

2023-08-09 Thread David Rowley
On Thu, 10 Aug 2023 at 01:23, Alvaro Herrera wrote: > > On 2023-Aug-09, Melih Mutlu wrote: > > > --Patch > > name | used_bytes | free_bytes | total_bytes > > ---+++- > > RelCacheMemoryContext |4706464 |3682144 |

Re: [PoC] Reducing planning time when tables have many partitions

2023-08-09 Thread David Rowley
On Wed, 9 Aug 2023 at 20:15, Yuya Watari wrote: > I agree with your opinion that my patch lacks some explanations, so I > will consider adding more comments. However, I received the following > message from David in March. > > On Thu, Mar 9, 2023 at 6:23 AM David Rowley wrote: &

Re: [PoC] Reducing planning time when tables have many partitions

2023-08-09 Thread David Rowley
On Wed, 9 Aug 2023 at 22:28, David Rowley wrote: > i.e: > > + Bitmapset *matching_ems = NULL; > + memcpy(_iter, iter, sizeof(EquivalenceMemberIterator)); > + memcpy(_iter, iter, sizeof(EquivalenceMemberIterator)); > + > + idx_iter.use_index = true; > + noidx_iter.use_inde

Re: [PoC] Reducing planning time when tables have many partitions

2023-08-09 Thread David Rowley
On Wed, 5 Jul 2023 at 21:58, Yuya Watari wrote: > > Hello, > > On Fri, Mar 10, 2023 at 5:38 PM Yuya Watari wrote: > > Thank you for pointing it out. I have attached the rebased version to > > this email. > > Recent commits, such as a8c09daa8b [1], have caused conflicts and > compilation errors

Re: Avoid stack frame setup in performance critical routines using tail calls

2023-08-09 Thread David Rowley
On Fri, 21 Jul 2023 at 14:03, David Rowley wrote: > I'll reply back with a more detailed review next week. Here's a review of v2-0001: 1. /* * XXX: Should this also be moved into alloc()? We could possibly avoid * zeroing in some cases (e.g. if we used mmap() ourselves. */ MemSetAligned(ret

Re: Reducing memory consumed by RestrictInfo list translations in partitionwise join planning

2023-08-08 Thread David Rowley
On Fri, 28 Jul 2023 at 02:06, Ashutosh Bapat wrote: > 0001 - to measure memory consumption during planning. This is the same > one as attached to [1]. I see you're recording the difference in the CurrentMemoryContext of palloc'd memory before and after planning. That won't really alert us to

Re: [PATCH] [zh_CN.po] fix a typo in simplified Chinese translation file

2023-08-07 Thread David Rowley
On Wed, 2 Aug 2023 at 15:45, jian he wrote: > I think it's pretty obviously. anyway. I created an commitfest entry. > https://commitfest.postgresql.org/44/4470/ I saw that there were two CF entries for this patch. I marked one as committed and the other as withdrawn. For the future, I believe

Re: 2023-08-10 release announcement draft

2023-08-07 Thread David Rowley
On Tue, 8 Aug 2023 at 13:49, Jonathan S. Katz wrote: > > On 8/7/23 9:45 PM, David Rowley wrote: > > >> * Fix a performance regression when running concurrent > >> [`COPY`](https://www.postgresql.org/docs/16/sql-copy.html) statements on a > >> single

Re: 2023-08-10 release announcement draft

2023-08-07 Thread David Rowley
On Tue, 8 Aug 2023 at 13:15, Jonathan S. Katz wrote: > Attached is the release announcement draft for the 2023-08-10 update > release, which also includes the release of PostgreSQL 16 Beta 3. Thanks for drafting this. > * Fix a performance regression when running concurrent >

Re: Check volatile functions in ppi_clauses for memoize node

2023-08-07 Thread David Rowley
On Fri, 4 Aug 2023 at 22:26, Richard Guo wrote: > explain (costs off) > select * from t t1 left join lateral > (select t1.a as t1a, t2.a as t2a from t t2) s > on t1.a = s.t2a + random(); > QUERY PLAN > --- > Nested Loop Left Join

Re: Fix a comment in paraminfo_get_equal_hashops

2023-08-07 Thread David Rowley
On Fri, 4 Aug 2023 at 18:48, Richard Guo wrote: > As stated in [1], there is a typo in the comment in > paraminfo_get_equal_hashops. > [1] > https://www.postgresql.org/message-id/cambws49dehrpe8pom_k39r2uosaozcg+y0b5a8tf7vw3uvr...@mail.gmail.com Thanks. Pushed. David

Re: Performance degradation on concurrent COPY into a single relation in PG16.

2023-08-07 Thread David Rowley
On Wed, 2 Aug 2023 at 13:35, David Rowley wrote: > So, it looks like this item can be closed off. I'll hold off from > doing that for a few days just in case anyone else wants to give > feedback or test themselves. Alright, closed. David

Re: Improve join_search_one_level readibilty (one line change)

2023-08-06 Thread David Rowley
On Fri, 4 Aug 2023 at 16:05, Richard Guo wrote: > > > On Fri, Aug 4, 2023 at 10:36 AM David Rowley wrote: >> >> The whole lnext() stuff all feels a bit old now that Lists are arrays. >> I think we'd be better adjusting the code to pass the List index where &

Re: Improve join_search_one_level readibilty (one line change)

2023-08-03 Thread David Rowley
On Tue, 1 Aug 2023 at 01:48, Julien Rouhaud wrote: > Apart from that +1 from me for the patch, I think it helps focusing the > attention on what actually matters here. I think it's worth doing something to improve this code. However, I think we should go a bit further than what the proposed

Re: Fix incorrect start up costs for WindowAgg paths (bug #17862)

2023-08-03 Thread David Rowley
On Fri, 4 Aug 2023 at 11:54, Nathan Bossart wrote: > I'm seeing some reliable test failures for 32-bit builds on cfbot [0]. At > a glance, it looks like the relations are swapped in the plan. Thank you for the report. I've just pushed a patch which I'm hoping will fix it. David

Re: Fix incorrect start up costs for WindowAgg paths (bug #17862)

2023-08-03 Thread David Rowley
On Fri, 4 Aug 2023 at 02:02, Andy Fan wrote: > I have checked the updated patch and LGTM. Thank you for reviewing. I've pushed the patch to master only. David

Re: Fix incorrect start up costs for WindowAgg paths (bug #17862)

2023-08-03 Thread David Rowley
Thanks for having a look at this. On Thu, 3 Aug 2023 at 18:49, Andy Fan wrote: > 1. ORDER BY or PARTITION BY > > select *, count(two) over (order by unique1) from tenk1 limit 1; > DEBUG: startup_tuples = 1 > DEBUG: startup_tuples = 1 > > select *, count(two) over (partition by unique1) from

Re: Fix incorrect start up costs for WindowAgg paths (bug #17862)

2023-08-02 Thread David Rowley
On Wed, 31 May 2023 at 12:59, David Rowley wrote: > > On Wed, 12 Apr 2023 at 21:03, David Rowley wrote: > > I'll add this to the "Older bugs affecting stable branches" section of > > the PG 16 open items list > > When I wrote the above, it was very soon afte

Re: Performance degradation on concurrent COPY into a single relation in PG16.

2023-08-01 Thread David Rowley
On Wed, 2 Aug 2023 at 12:25, David Rowley wrote: > master @ 3845577cb > latency average = 1575.879 ms > >6.79% postgres [.] pg_strtoint32_safe > > master~1 > latency average = 1968.004 ms > > 14.28% postgres [.] pg_strtoint32_safe > >

Re: Performance degradation on concurrent COPY into a single relation in PG16.

2023-08-01 Thread David Rowley
On Wed, 2 Aug 2023 at 07:38, Dean Rasheed wrote: > Running the new test on slightly older Intel hardware (i9-9900K, gcc > 11.3), I get the following: Thanks for running those tests. I've now pushed the fastpath4.patch after making a few adjustments to the header comments to mention the new stuff

Re: Performance degradation on concurrent COPY into a single relation in PG16.

2023-08-01 Thread David Rowley
On Wed, 2 Aug 2023 at 01:26, Dean Rasheed wrote: > > On Tue, 1 Aug 2023 at 13:55, David Rowley wrote: > > > > I tried adding the "at least 1 digit check" by adding an else { goto > > slow; } in the above code, but it seems to generate slower code than >

Re: Performance degradation on concurrent COPY into a single relation in PG16.

2023-08-01 Thread David Rowley
On Tue, 1 Aug 2023 at 13:25, Andres Freund wrote: > There's a lot of larger numbers in the file, which likely reduces the impact > some. And there's the overhead of actually inserting the rows into the table, > making the difference appear smaller than it is. It might be worth special casing the

Re: Performance degradation on concurrent COPY into a single relation in PG16.

2023-08-01 Thread David Rowley
On Mon, 31 Jul 2023 at 21:39, John Naylor wrote: > master + pg_strtoint_fastpath1.patch > latency average = 938.146 ms > latency stddev = 9.354 ms > > master + pg_strtoint_fastpath2.patch > latency average = 902.808 ms > latency stddev = 3.957 ms Thanks for checking those two on your machine.

Re: Performance degradation on concurrent COPY into a single relation in PG16.

2023-07-27 Thread David Rowley
On Thu, 27 Jul 2023 at 14:51, David Rowley wrote: > Just to keep this moving and to make it easier for people to test the > pg_strtoint patches, I've pushed the fix_COPY_DEFAULT.patch patch. > The only thing I changed was to move the line that was allocating the > array to a location

Re: Performance degradation on concurrent COPY into a single relation in PG16.

2023-07-26 Thread David Rowley
On Wed, 26 Jul 2023 at 03:50, Andres Freund wrote: > On 2023-07-25 23:37:08 +1200, David Rowley wrote: > > On Tue, 25 Jul 2023 at 17:34, Andres Freund wrote: > > I've not really studied the fix_COPY_DEFAULT.patch patch. Is there a > > reason to delay committing th

Re: Performance degradation on concurrent COPY into a single relation in PG16.

2023-07-26 Thread David Rowley
> On 2023-07-25 23:37:08 +1200, David Rowley wrote: > > On Tue, 25 Jul 2023 at 17:34, Andres Freund wrote: > > > HEAD: 812.690 > > > > > > your patch: 821.354 > > > > > > strtoint from 8692f6644

Re: Partition pruning not working on updates

2023-07-25 Thread David Rowley
On Tue, 25 Jul 2023 at 20:45, Mr.Bim wrote: > Partition pruning is not working on the updates query, am I missing something? In PG13, partition pruning for UPDATE and DELETE only works during query planning. Because you're using CURRENT_TIMESTAMP, that's not an immutable expression which can be

Re: Performance degradation on concurrent COPY into a single relation in PG16.

2023-07-25 Thread David Rowley
On Tue, 25 Jul 2023 at 17:34, Andres Freund wrote: > prep: > COPY (SELECT generate_series(1, 200) a, (random() * 10 - 5)::int > b, 3243423 c) TO '/tmp/lotsaints.copy'; > DROP TABLE lotsaints; CREATE UNLOGGED TABLE lotsaints(a int, b int, c int); > > benchmark: > psql -qX -c 'truncate

Re: Performance degradation on concurrent COPY into a single relation in PG16.

2023-07-20 Thread David Rowley
On Thu, 20 Jul 2023 at 20:37, Dean Rasheed wrote: > > On Thu, 20 Jul 2023 at 00:56, David Rowley wrote: > I agree with the principal though. In the attached updated patch, I > replaced that test with a simpler one: > > +/* > + * Process the number's digits. We optim

<    1   2   3   4   5   6   7   8   9   10   >