Re: POC, WIP: OR-clause support for indexes

2024-03-14 Thread Andrei Lepikhov
On 14/3/2024 16:31, Alexander Korotkov wrote: On Wed, Mar 13, 2024 at 2:16 PM Andrei Lepikhov mailto:a.lepik...@postgrespro.ru>> wrote: > On 13/3/2024 18:05, Alexander Korotkov wrote: > > On Wed, Mar 13, 2024 at 7:52 AM Andrei Lepikhov > > Given all of the

Re: POC, WIP: OR-clause support for indexes

2024-03-13 Thread Andrei Lepikhov
On 13/3/2024 18:05, Alexander Korotkov wrote: On Wed, Mar 13, 2024 at 7:52 AM Andrei Lepikhov Given all of the above, I think moving transformation to the canonicalize_qual() would be the right way to go. Ok, I will try to move the code. I have no idea about the timings so far. I recall the

Re: POC, WIP: OR-clause support for indexes

2024-03-12 Thread Andrei Lepikhov
On 12/3/2024 22:20, Alexander Korotkov wrote: On Mon, Mar 11, 2024 at 2:43 PM Andrei Lepikhov I think you are right. It is probably a better place than any other to remove duplicates in an array. I just think we should sort and remove duplicates from entry->consts in one pass. Thus, t

Re: a wrong index choose when statistics is out of date

2024-03-12 Thread Andrei Lepikhov
tion that makes the index-picking technique less dependent on the ordering of index lists [1]. [1] https://www.postgresql.org/message-id/9b3dbf6d-776a-4953-a5a4-609929393...@postgrespro.ru -- regards, Andrei Lepikhov Postgres Professional diff --git a/src/backend/utils/adt/like_support.c

Re: POC, WIP: OR-clause support for indexes

2024-03-11 Thread Andrei Lepikhov
s caused by this approach. Anyway, it should be done as quickly as possible to increase the effect of the optimization. -- regards, Andrei Lepikhov Postgres Professional

Re: POC, WIP: OR-clause support for indexes

2024-03-10 Thread Andrei Lepikhov
On 7/3/2024 21:51, Alexander Korotkov wrote: Hi! On Tue, Mar 5, 2024 at 9:59 AM Andrei Lepikhov mailto:a.lepik...@postgrespro.ru>> wrote: > On 5/3/2024 12:30, Andrei Lepikhov wrote: > > On 4/3/2024 09:26, jian he wrote: > ... and the new version of the patchset is atta

Re: a wrong index choose when statistics is out of date

2024-03-08 Thread Andrei Lepikhov
On 7/3/2024 17:32, David Rowley wrote: On Thu, 7 Mar 2024 at 21:17, Andrei Lepikhov wrote: I would like to ask David why the var_eq_const estimator doesn't have an option for estimation with a histogram. Having that would relieve a problem with skewed data. Detecting the situation

Re: a wrong index choose when statistics is out of date

2024-03-07 Thread Andrei Lepikhov
esult: planned rows=8604, actual rows=3501 EXPLAIN ANALYZE SELECT * FROM norm_test WHERE val = 700; -- result: planned rows=8604, actual rows=1705 EXPLAIN ANALYZE SELECT * FROM norm_test WHERE val = 1000; -- result: planned rows=8604, actual rows=91 -- regards, Andrei Lepikhov Postgres Professional

Re: "type with xxxx does not exist" when doing ExecMemoize()

2024-03-05 Thread Andrei Lepikhov
On 6/3/2024 10:10, Tender Wang wrote: Andrei Lepikhov <mailto:a.lepik...@postgrespro.ru>> 于2024年3月5日周二 17:36写道: On 1/3/2024 14:18, Tender Wang wrote: > During debug, I learned that numeric_add doesn't have type check like > rangetype, so aboved query w

Re: Hooking into ExplainOneQuery() complicated by missing standard_ExplainOneQuery

2024-03-05 Thread Andrei Lepikhov
3f677%40garret.ru -- regards, Andrei Lepikhov Postgres Professional

Re: "type with xxxx does not exist" when doing ExecMemoize()

2024-03-05 Thread Andrei Lepikhov
On 1/3/2024 14:18, Tender Wang wrote: During debug, I learned that numeric_add doesn't have type check like rangetype, so aboved query will not report "type with xxx does not exist". And I realize that  the test case added by Andrei Lepikhov  in v3 is right. So in v6 pat

Re: POC, WIP: OR-clause support for indexes

2024-03-04 Thread Andrei Lepikhov
On 5/3/2024 12:30, Andrei Lepikhov wrote: On 4/3/2024 09:26, jian he wrote: ... and the new version of the patchset is attached. -- regards, Andrei Lepikhov Postgres Professional From 1c3ac3e006cd66ff40f1ddaaa09e3fc0f3a75ca5 Mon Sep 17 00:00:00 2001 From: Alena Rybakina Date: Fri, 2 Feb 2024

Re: POC, WIP: OR-clause support for indexes

2024-03-04 Thread Andrei Lepikhov
On 4/3/2024 09:26, jian he wrote: On Thu, Feb 29, 2024 at 4:59 PM Andrei Lepikhov Feel free to add, change or totally rewrite these changes. On replacement of static ScalarArrayOpExpr dest with dynamic allocated one: After discussion [1] I agree with that replacement. Some style (and language

Re: a wrong index choose when statistics is out of date

2024-03-03 Thread Andrei Lepikhov
of the boundaries when an index shows us that we have min/max outside known statistics? Because it would be used for the values out of the histogram, it should only add an overhead with a reason. -- regards, Andrei Lepikhov Postgres Professional

Re: a wrong index choose when statistics is out of date

2024-03-03 Thread Andrei Lepikhov
rk, but it needs a new parameter in pg_type and a lot of additional code for such a rare case. I'm looking forward to the demo patch. -- regards, Andrei Lepikhov Postgres Professional

Re: POC, WIP: OR-clause support for indexes

2024-03-03 Thread Andrei Lepikhov
he logic above. What about arrays? As I see, arrays don't have typarray and we can avoid to spend more cycles after detection of TYPCATEGORY_ARRAY. I haven't done it yet because have a second thought: what if to combine arrays into the larger one? I'm unsure on that, so we can forbid it too. -- regards, Andrei Lepikhov Postgres Professional

Re: POC, WIP: OR-clause support for indexes

2024-02-29 Thread Andrei Lepikhov
e smoothly. All modifications are integrated into the two new patches. Feel free to add, change or totally rewrite these changes. -- regards, Andrei Lepikhov Postgres Professional From 015a564cc784139c806a7004f25bf5f7a4b4a29d Mon Sep 17 00:00:00 2001 From: Alena Rybakina Date: Fri, 2 Feb 2024 22:01:0

Re: POC, WIP: OR-clause support for indexes

2024-02-28 Thread Andrei Lepikhov
transformBoolExprOr and generate_saop_pathlist (including cross-referencing each other). These are starting points to understand the transformation and, therefore, a good place for a detailed explanation. -- regards, Andrei Lepikhov Postgres Professional

Re: "type with xxxx does not exist" when doing ExecMemoize()

2024-02-27 Thread Andrei Lepikhov
ch looks better. Also, You don't need to initialize tts_values[i] at all if tts_isnull[i] set to true. -- regards, Andrei Lepikhov Postgres Professional

Re: POC, WIP: OR-clause support for indexes

2024-02-27 Thread Andrei Lepikhov
ant to be on the right side. 2. We should describe the second part of the feature, where the optimiser can split an array to fit the optimal BitmapOr scan path. -- regards, Andrei Lepikhov Postgres Professional

Re: The const expression evaluation routine should always return a copy

2024-02-27 Thread Andrei Lepikhov
On 28/2/2024 04:19, Tom Lane wrote: Andrei Lepikhov writes: IMO, the routine eval_const_expressions_mutator contains some stale code: I'd like to push back against the idea that eval_const_expressions is expected to return a freshly-copied tree. Its API specification contains no such

The const expression evaluation routine should always return a copy

2024-02-26 Thread Andrei Lepikhov
as a result, allow to return a Const instead of SubPlan? But at least we can return a flat copy of the SubPplan node just for the convention — the same thing for the AlternativeSubPlan. See the patch in the attachment. -- regards, Andrei Lepikhov Postgres Professiona

Re: "type with xxxx does not exist" when doing ExecMemoize()

2024-02-26 Thread Andrei Lepikhov
On 26/2/2024 18:34, Richard Guo wrote: On Mon, Feb 26, 2024 at 3:54 PM Andrei Lepikhov mailto:a.lepik...@postgrespro.ru>> wrote: On 26/2/2024 12:44, Tender Wang wrote: > Make sense. I found MemoizeState already has a MemoryContext, so I used it. > I upda

Re: "type with xxxx does not exist" when doing ExecMemoize()

2024-02-25 Thread Andrei Lepikhov
On 26/2/2024 12:44, Tender Wang wrote: Andrei Lepikhov <mailto:a.lepik...@postgrespro.ru>> 于2024年2月26日周一 10:57写道: On 25/2/2024 20:32, Tender Wang wrote: > I think in prepare_probe_slot(), should called datumCopy as the attached > patch does. >

Re: "type with xxxx does not exist" when doing ExecMemoize()

2024-02-25 Thread Andrei Lepikhov
On 26/2/2024 09:52, Andrei Lepikhov wrote: On 25/2/2024 20:32, Tender Wang wrote: I think in prepare_probe_slot(), should called datumCopy as the attached patch does. Any thoughts? Thanks. Thanks for the report. I think it is better to invent a Runtime Memory Context; likewise, it is

Re: "type with xxxx does not exist" when doing ExecMemoize()

2024-02-25 Thread Andrei Lepikhov
. Here, you just allocate the value in some upper memory context. Also, I'm curious why such a trivial error hasn't been found for a long time -- regards, Andrei Lepikhov Postgres Professional

Re: Removing unneeded self joins

2024-02-22 Thread Andrei Lepikhov
se I follow another commenting style, but technically, it's still OK. -- regards, Andrei Lepikhov Postgres Professional

Re: POC: GROUP BY optimization

2024-02-21 Thread Andrei Lepikhov
tension of the relation blocks. If Maxim will answer that it's enough to resolve his issue, why not? -- regards, Andrei Lepikhov Postgres Professional

Re: Unlinking Parallel Hash Join inner batch files sooner

2024-02-21 Thread Andrei Lepikhov
On 22/2/2024 06:42, Thomas Munro wrote: On Wed, Feb 21, 2024 at 7:34 PM Andrei Lepikhov wrote: I see in [1] that the reporter mentioned a delay between the error message in parallel HashJoin and the return control back from PSQL. Your patch might reduce this delay. Also, I have the same

Re: POC: GROUP BY optimization

2024-02-21 Thread Andrei Lepikhov
around 10 cost points. [1] https://www.postgresql.org/message-id/CACG=ezaYM1tr6Lmp8PRH1aeZq=rbkxeotwgzmclad5mphfw...@mail.gmail.com -- regards, Andrei Lepikhov Postgres Professional

Re: Removing unneeded self joins

2024-02-21 Thread Andrei Lepikhov
need two different subtrees for the same query. I will look into your fix. -- regards, Andrei Lepikhov Postgres Professional

Re: Unlinking Parallel Hash Join inner batch files sooner

2024-02-20 Thread Andrei Lepikhov
stuck into the unlink of tons of temporary files. So, are you going to do something with this code? [1] https://www.postgresql.org/message-id/18349-83d33dd3d0c855c3%40postgresql.org -- regards, Andrei Lepikhov Postgres Professional

Re: [POC] Allow flattening of subquery with a link to upper query

2024-02-20 Thread Andrei Lepikhov
On 20/2/2024 17:43, David Rowley wrote: On Tue, 20 Feb 2024 at 22:57, Andrei Lepikhov wrote: I agree that it would be nice to teach the planner how to do this, but I think it just has to be a cost-based decision. Imagine how the transformed query would perform of pg_am had a billion rows and

Re: [POC] Allow flattening of subquery with a link to upper query

2024-02-20 Thread Andrei Lepikhov
ted to fix links from a non-parent query block. So, in my opinion, the reason for this patch still exists, and we can continue this work further, maybe elaborating on flattening LATERAL references - this needs some research. [1] https://www.postgresql.org/message-id/35c8a3e8-d080-dfa8-2be3-cf5fe702010

Re: Optimize planner memory consumption for huge arrays

2024-02-19 Thread Andrei Lepikhov
think we should design small memory contexts - one per scalable direction of memory utilization, like selectivity or partitioning (appending ?). My coding experience shows that short-lived GEQO memory context forces people to learn on Postgres internals more precisely :). -- regards, Andrei Le

Re: POC, WIP: OR-clause support for indexes

2024-02-19 Thread Andrei Lepikhov
o, by reducing the clause list, we eliminate many calls of the equal() routine, too. `leftop operator rightop` the operator can also be volatile. Do we need to check (op_volatile(opno) == PROVOLATILE_VOLATILE) within transformBoolExprOr? As usual, could you provide a test case to discuss it mo

Re: Optimize planner memory consumption for huge arrays

2024-02-19 Thread Andrei Lepikhov
same memory piece for the next array element. It finds this piece more quickly than before that optimization. -- regards, Andrei Lepikhov Postgres Professional From 2e89dc8b743953068174c777d7a014e1ea71f659 Mon Sep 17 00:00:00 2001 From: "Andrey V. Lepikhov" Date: Tue, 20 Feb 2024 11:0

Re: POC, WIP: OR-clause support for indexes

2024-02-19 Thread Andrei Lepikhov
in this operation */ + continue; + arrayconst = lsecond_node(Const, saop->args); + dest = makeNode(ScalarArrayOpExpr); Thanks for the review! I'm not sure I understand you clearly. Does the patch in attachment fix the issue you raised? -- regards, Andrei Lepikhov Postgres Professional

Re: Memory consumed by paths during partitionwise join planning

2024-02-19 Thread Andrei Lepikhov
On 19/2/2024 19:25, Ashutosh Bapat wrote: On Fri, Feb 16, 2024 at 8:42 AM Andrei Lepikhov wrote: Live example: right now, I am working on the code like MSSQL has - a combination of NestLoop and HashJoin paths and switching between them in real-time. It requires both paths in the path list at

Re: POC, WIP: OR-clause support for indexes

2024-02-19 Thread Andrei Lepikhov
or both patches. As I see it, the only general explanation of the idea is not addressed. I'm not sure how deeply we should explain it. -- regards, Andrei Lepikhov Postgres Professional From 3a3b6aa36320a06b64f2f608e3526255e53ed655 Mon Sep 17 00:00:00 2001 From: Alena Rybakina Date: Fri, 2 Feb

Re: Removing unneeded self joins

2024-02-18 Thread Andrei Lepikhov
d you, please, recheck? I reviewed this patch. Why do you check only the target list? I guess these links can be everywhere. See the patch in the attachment with the elaborated test and slightly changed code. -- regards, Andrei Lepikhov Postgres Professional From 7f94a3c96fd410522b87e570240cdb96b30

Re: Reducing memory consumed by RestrictInfo list translations in partitionwise join planning

2024-02-18 Thread Andrei Lepikhov
ability on many partitions, we should introduce per-partition memory context and reset it in between. GEQO already has a short-lived memory context, making designing extensions a bit more challenging but nothing too painful. -- regards, Andrei Lepikhov Postgres Professional

Re: POC, WIP: OR-clause support for indexes

2024-02-15 Thread Andrei Lepikhov
On 16/2/2024 07:00, jian he wrote: On Wed, Feb 14, 2024 at 11:21 AM Andrei Lepikhov wrote: My OS: Ubuntu 22.04.3 LTS I already set the max_parallel_workers_per_gather to 10. So for all cases, it should use parallelism first? a better question would be: how to make the number of OR less than 29

Re: Memory consumed by paths during partitionwise join planning

2024-02-15 Thread Andrei Lepikhov
On 15/2/2024 19:06, Ashutosh Bapat wrote: On Thu, Feb 15, 2024 at 9:41 AM Andrei Lepikhov But I'm not sure about freeing unreferenced paths. I would have to see alternatives in the pathlist. I didn't understand this. Can you please elaborate? A path in any pathlist is refe

Re: planner chooses incremental but not the best one

2024-02-15 Thread Andrei Lepikhov
On 15/2/2024 18:10, Tomas Vondra wrote: On 2/15/24 07:50, Andrei Lepikhov wrote: On 18/12/2023 19:53, Tomas Vondra wrote: On 12/18/23 11:40, Richard Guo wrote: The challenge is where to get usable information about correlation between columns. I only have a couple very rought ideas of what

Re: planner chooses incremental but not the best one

2024-02-14 Thread Andrei Lepikhov
add a test directly checking appendrel->tuples correction. -- regards, Andrei Lepikhov Postgres Professional

Re: planner chooses incremental but not the best one

2024-02-14 Thread Andrei Lepikhov
f grouping? -- regards, Andrei Lepikhov Postgres Professional

Re: Memory consumed by paths during partitionwise join planning

2024-02-14 Thread Andrei Lepikhov
message-id/flat/CAOP8fzaVL_2SCJayLL9kj5pCA46PJOXXjuei6-3aFUV45j4LJQ%40mail.gmail.com [2] https://www.postgresql.org/message-id/flat/CAMbWs496%2BN%3DUAjOc%3DrcD3P7B6oJe4rZw08e_TZRUsWbPxZW3Tw%40mail.gmail.com -- regards, Andrei Lepikhov Postgres Professional

Re: Memory consumed by child SpecialJoinInfo in partitionwise join planning

2024-02-13 Thread Andrei Lepikhov
On 14/2/2024 13:32, Ashutosh Bapat wrote: On Wed, Feb 14, 2024 at 9:50 AM Andrei Lepikhov wrote: On 30/1/2024 12:44, Ashutosh Bapat wrote: Thanks Vignesh. PFA patches rebased on the latest HEAD. The patch addressing Amit's comments is still a separate patch for him to review. Thank

Re: Propagate pathkeys from CTEs up to the outer query

2024-02-13 Thread Andrei Lepikhov
nt backburner - pull parameterisation through the GROUP-BY and the query block fence up to the JOIN searching code of the parent query. -- regards, Andrei Lepikhov Postgres Professional

Re: Memory consumed by child SpecialJoinInfo in partitionwise join planning

2024-02-13 Thread Andrei Lepikhov
some extensions, such as pg_hint_plan, call build_child_join_sjinfo. It is OK to break the interface with a major version. But what if they need child_sjinfo a bit longer and collect links to this structure? I don't think it is a real stopper, but it is worth additional analysis. -- rega

Re: POC, WIP: OR-clause support for indexes

2024-02-13 Thread Andrei Lepikhov
On 13/2/2024 17:03, Andrei Lepikhov wrote: On 13/2/2024 07:00, jian he wrote: The time is the last result of the 10 iterations. I'm not sure about the origins of such behavior, but it seems to be an issue of parallel workers, not this specific optimization. Having written that, I&#x

Re: POC, WIP: OR-clause support for indexes

2024-02-13 Thread Andrei Lepikhov
On 12/2/2024 17:51, Alena Rybakina wrote: On 12.02.2024 12:01, Andrei Lepikhov wrote: On 12/2/2024 15:55, Alena Rybakina wrote: As I understand it, match_clauses_to_index is necessary if you have a RestrictInfo (rinfo1) variable, so maybe we should run it after the make_restrictonfo procedure

Re: POC, WIP: OR-clause support for indexes

2024-02-13 Thread Andrei Lepikhov
igins of such behavior, but it seems to be an issue of parallel workers, not this specific optimization. -- regards, Andrei Lepikhov Postgres Professional

Re: POC, WIP: OR-clause support for indexes

2024-02-12 Thread Andrei Lepikhov
1, haven't we? -- regards, Andrei Lepikhov Postgres Professional

Re: POC, WIP: OR-clause support for indexes

2024-02-11 Thread Andrei Lepikhov
Thanks for the review! It was the first version for discussion. Of course, refactoring and polishing cycles will be needed, even so we can discuss the general idea earlier. On 10/2/2024 12:00, jian he wrote: On Thu, Feb 8, 2024 at 1:34 PM Andrei Lepikhov 1235 | PredicatesData

Re: pg_stat_advisor extension

2024-02-07 Thread Andrei Lepikhov
. -- regards, Andrei Lepikhov Postgres Professional

Re: POC, WIP: OR-clause support for indexes

2024-02-07 Thread Andrei Lepikhov
il.gmail.com [2] https://www.postgresql.org/message-id/CAPpHfduJtO0s9E%3DSHUTzrCD88BH0eik0UNog1_q3XBF2wLmH6g%40mail.gmail.com -- regards, Andrei Lepikhov Postgres Professional From 0ac511ab94959e87d1525d5ea8c4855643a6ccbe Mon Sep 17 00:00:00 2001 From: Alena Rybakina Date: Fri, 2 Feb 2024 22:01:0

Re: POC: GROUP BY optimization

2024-02-01 Thread Andrei Lepikhov
l be already sorted before the MergeJoin. Why not use Incremental Sort on (x,z,w) instead of full sort? 2. For memo, IMO, this test shows us the future near perspective of this feature: It is cheaper to use grouping order (w,z) instead of (z,w). -- regards, Andrei Lepikhov Postgres Professional

Re: POC: GROUP BY optimization

2024-02-01 Thread Andrei Lepikhov
ession run. Nor do I see any particular reason for pg_class to be especially suited to the test. Yeah, It is my fault. Please, see in the attachment the patch fixing that. -- regards, Andrei Lepikhov Postgres Professional From 11a049d95ee48e38ad569aab7663d8de91f946ad Mon Sep 17 00:00:00 2001 F

Re: POC: GROUP BY optimization

2024-01-18 Thread Andrei Lepikhov
Just forgotten second attachment. -- regards, Andrei Lepikhov Postgres Professional diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c index 1095b73dac..b612420547 100644 --- a/src/backend/optimizer/path/pathkeys.c +++ b/src/backend/optimizer/path

Re: POC: GROUP BY optimization

2024-01-18 Thread Andrei Lepikhov
pk_eclass->ec_sortref' is valid before calling get_sortgroupref_clause_noerr with it. Also, how can we guarantee that the returned GROUP BY item is sortable? Should we check that with OidIsValid(sgc->sortop)? Added. Reviewed it, looks good. -- regards, Andrei Lepikhov Postgre

Re: POC: GROUP BY optimization

2024-01-15 Thread Andrei Lepikhov
set looks good to me.  I'm going to push it if there are no objections. Seems I'm late for the party.  Can we hold for several more days?  I'd like to have a review on this patch. Get on board! It looks like this feature needs as much review as possible (likewise SJE). --

Re: POC: GROUP BY optimization

2024-01-14 Thread Andrei Lepikhov
On 13/1/2024 22:00, Alexander Korotkov wrote: On Sat, Jan 13, 2024 at 11:09 AM Andrei Lepikhov wrote: On 11/1/2024 18:30, Alexander Korotkov wrote: On Tue, Jan 9, 2024 at 1:14 PM Pavel Borisov wrote: Hmm, I don't see this old code in these patches. Resend 0002-* because of trailing s

Re: POC: GROUP BY optimization

2024-01-13 Thread Andrei Lepikhov
additional comments and tests to make the code more understandable (see attachment). I intended to look into this part of the code more, but the tests show a difference between PostgreSQL v.15 and v.16, which causes changes in the code of this feature. -- regards, Andrei Lepikhov Postgres Pr

Re: introduce dynamic shared memory registry

2024-01-10 Thread Andrei Lepikhov
t; I kept this the same, as I didn't see any need to tie the key size to > NAMEDATALEN. IMO, we should avoid magic numbers whenever possible. Current logic according to which first users of this feature will be extensions naturally bonds this size to the size of the 'name' type

Re: Custom explain options

2024-01-10 Thread Andrei Lepikhov
On 10/1/2024 20:27, Konstantin Knizhnik wrote: On 10/01/2024 8:46 am, Michael Paquier wrote: On Wed, Jan 10, 2024 at 01:29:30PM +0700, Andrei Lepikhov wrote: What do you think about this really useful feature? Do you wish to develop it further? I am biased here.  This seems like a lot of

Re: Multidimensional Histograms

2024-01-10 Thread Andrei Lepikhov
er for one combination, building the dependency statistics is still massive work. So, in the multicolumn case, having something like a histogram may be more effective. -- regards, Andrei Lepikhov Postgres Professional

Re: Custom explain options

2024-01-09 Thread Andrei Lepikhov
code in my PR to send actual amount of custom instrumentation data. But it can not help with the cases above. What do you think about this really useful feature? Do you wish to develop it further? -- regards, Andrei Lepikhov Postgres Professional

Re: POC: GROUP BY optimization

2024-01-09 Thread Andrei Lepikhov
On 9/1/2024 16:45, vignesh C wrote: On Tue, 9 Jan 2024 at 14:31, Andrei Lepikhov wrote: Here is a new version of GROUP-BY optimization without sort model. On 21/12/2023 17:53, Alexander Korotkov wrote: I'd like to make some notes. 1) As already mentioned, there is clearly a repet

Re: POC: GROUP BY optimization

2024-01-09 Thread Andrei Lepikhov
= off. Then in get_useful_group_keys_orderings() we should only deal with input_path fully matching the group-by clause, and try only full match of group-by output to the required order. Hm, is it really make sense in current implementation? -- regards, Andrei Lepikhov Postgres Professional From ea068e221a754a463142575f6

Re: Multidimensional Histograms

2024-01-07 Thread Andrei Lepikhov
On 8/1/2024 01:36, Tomas Vondra wrote: On 1/7/24 18:26, Andrei Lepikhov wrote: On 7/1/2024 17:51, Tomas Vondra wrote: On 1/7/24 11:22, Andrei Lepikhov wrote: On 7/1/2024 06:54, Tomas Vondra wrote: It's an interesting are for experiments, no doubt about it. And if you choose to explo

Re: Multidimensional Histograms

2024-01-07 Thread Andrei Lepikhov
On 7/1/2024 17:51, Tomas Vondra wrote: On 1/7/24 11:22, Andrei Lepikhov wrote: On 7/1/2024 06:54, Tomas Vondra wrote: It's an interesting are for experiments, no doubt about it. And if you choose to explore it, that's fine. But it's better to be aware it may not end with a

Re: Multidimensional Histograms

2024-01-07 Thread Andrei Lepikhov
imation over a set of ANDed quals on different columns. In your opinion, is it possible to add a hook into the extended statistics to allow for an extension to propose alternative estimation? [1] https://github.com/danolivo/pg_index_stats -- regards, Andrei Lepikhov Postgres Professional

Re: Removing unneeded self joins

2023-12-29 Thread Andrei Lepikhov
lace the relid here. Could you check your issue with the patch in the attachment? Does it resolve this case? -- regards, Andrei Lepikhov Postgres Professional diff --git a/src/backend/optimizer/plan/analyzejoins.c b/src/backend/optimizer/plan/analyzejoins.c index 6c02fe8908..f79c673afd 1006

Re: POC: GROUP BY optimization

2023-12-28 Thread Andrei Lepikhov
On 28/12/2023 18:29, Alexander Korotkov wrote: On Thu, Dec 28, 2023 at 10:22 AM Andrei Lepikhov wrote: But arrangement with an ORDER BY clause doesn't work: DROP INDEX abc; explain SELECT x,w,z FROM t GROUP BY (w,x,z) ORDER BY (x,z,w); I think the reason is that the sort_pathkey

Re: POC: GROUP BY optimization

2023-12-28 Thread Andrei Lepikhov
On 27/12/2023 12:07, Tom Lane wrote: Andrei Lepikhov writes: To be clear. In [1], I mentioned we can perform micro-benchmarks and structure costs of operators. At least for fixed-length operators, it is relatively easy. I repeat what I said: this is a fool's errand. You will no

Re: POC: GROUP BY optimization

2023-12-26 Thread Andrei Lepikhov
rection of turning on choosing between different sort column orderings if we have extended statistics on the columns? [1] https://www.postgresql.org/message-id/e3602ccb-e643-2e79-ed2c-1175a8053...@postgrespro.ru -- regards, Andrei Lepikhov Postgres Professional

Re: POC: GROUP BY optimization

2023-12-26 Thread Andrei Lepikhov
On 21/12/2023 17:53, Alexander Korotkov wrote: On Sun, Oct 1, 2023 at 11:45 AM Andrei Lepikhov wrote: New version of the patch. Fixed minor inconsistencies and rebased onto current master. Thank you (and other authors) for working on this subject. Indeed to GROUP BY clauses are order

Specify description of the SpecialJoinInfo structure

2023-12-26 Thread Andrei Lepikhov
Hi, Working on Asymmetric Join, I found slight inconsistency in the description of SpecialJoinInfo: join type JOIN_ANTI can be accompanied by a zero value of the ojrelid if this join was created by the transformation of the NOT EXISTS subquery. -- regards, Andrei Lepikhov Postgres

Re: Optimization outcome depends on the index order

2023-12-25 Thread Andrei Lepikhov
On 25/12/2023 18:36, Alexander Korotkov wrote: On Fri, Dec 22, 2023 at 7:24 PM Andrei Lepikhov wrote: On 22/12/2023 11:48, Alexander Korotkov wrote: > Because we must trust all predictions made by the planner, we just > choose the most trustworthy path. According to the planner log

Re: Optimization outcome depends on the index order

2023-12-22 Thread Andrei Lepikhov
se of COSTS_EQUAL. -- regards, Andrei Lepikhov Postgres Professional From 45bda9784d28dc9cec90c5b33285023a49850800 Mon Sep 17 00:00:00 2001 From: "Andrey V. Lepikhov" Date: Mon, 27 Nov 2023 11:23:48 +0700 Subject: [PATCH] Choose an index path with the best selectivity estimation. In the c

Optimization outcome depends on the index order

2023-12-21 Thread Andrei Lepikhov
;re willing to put more > work into this. Done > 9. https://www.postgresql.org/message-id/154f786a-06a0-4fb1- > b8a4-16c66149731b%40postgrespro.ru -- regards, Andrei Lepikhov Postgres ProfessionalFrom 7b044de1449a5fdc450cb629caafb4e15ded7a93 Mon Sep 17 00:00:00 2001 From: "Andre

Re: Postgres picks suboptimal index after building of an extended statistics

2023-12-21 Thread Andrei Lepikhov
y be used in some way to improve estimates even when there is no full match. I have tried to use the knowledge about unique indexes in the selectivity estimation routine. But it looks invasive and adds a lot of overhead. -- regards, Andrei Lepikhov Postgres Professional

Re: introduce dynamic shared memory registry

2023-12-20 Thread Andrei Lepikhov
On 20/12/2023 17:33, Nathan Bossart wrote: On Wed, Dec 20, 2023 at 11:02:58AM +0200, Andrei Lepikhov wrote: In that case, maybe change the test case to make it closer to real-life usage - with locks and concurrent access (See attachment)? I'm not following why we should make this test

Re: introduce dynamic shared memory registry

2023-12-20 Thread Andrei Lepikhov
On 20/12/2023 07:04, Michael Paquier wrote: On Tue, Dec 19, 2023 at 10:14:44AM -0600, Nathan Bossart wrote: On Tue, Dec 19, 2023 at 10:49:23AM -0500, Robert Haas wrote: On Mon, Dec 18, 2023 at 3:32 AM Andrei Lepikhov wrote: 2. I think a separate file for this feature looks too expensive

Re: introduce dynamic shared memory registry

2023-12-18 Thread Andrei Lepikhov
On 18/12/2023 13:39, Andrei Lepikhov wrote: On 5/12/2023 10:46, Nathan Bossart wrote: I don't presently have any concrete plans to use this for anything, but I thought it might be useful for extensions for caching, etc. and wanted to see whether there was any interest in the feature.

Re: introduce dynamic shared memory registry

2023-12-17 Thread Andrei Lepikhov
hread. Designing extensions, every time I feel pain introducing one shared value or some global stat, the extension must be required to be loadable on startup only. It reduces the flexibility of even very lightweight extensions, which look harmful to use in a cloud. -- regards, Andrei Lepikhov Pos

Re: Statistics Import and Export

2023-12-15 Thread Andrei Lepikhov
ostgresql.org/message-id/7a40707d-1758-85a2-7bb1-6e5775518e64%40postgrespro.ru -- regards, Andrei Lepikhov Postgres Professional

Re: Oversight in reparameterize_path_by_child leading to executor crash

2023-12-12 Thread Andrei Lepikhov
the first reparameterization up to the plan creation, we have some inconsistency in expressions (partitions refer to expressions with the parent relid). What if an extension in the middle changes the parent expression? By and large, this patch is in a good state and may be committed. -- reg

Re: Assert failure on 'list_member_ptr(rel->joininfo, restrictinfo)'

2023-12-10 Thread Andrei Lepikhov
On 11/12/2023 09:31, Richard Guo wrote: On Fri, Dec 8, 2023 at 3:13 PM Alexander Pyhalov mailto:a.pyha...@postgrespro.ru>> wrote: Andrei Lepikhov писал(а) 2023-12-08 07:37: > I'd already clashed with Tom on copying the required_relids field and > voluntarily

Re: Assert failure on 'list_member_ptr(rel->joininfo, restrictinfo)'

2023-12-07 Thread Andrei Lepikhov
current hotfix? 0003 - I think it is really what we need right now: SJE is quite a rare optimization and executes before the entries expansion procedure. So it looks less risky. [1] Asymmetric partition-wise JOIN https://www.postgresql.org/message-id/flat/CAOP8fzaVL_2SCJayLL9kj5pCA46PJOXXju

Re: POC, WIP: OR-clause support for indexes

2023-12-05 Thread Andrei Lepikhov
al indexes) -- regards, Andrei Lepikhov Postgres Professional From 71746caae3eb0c771792b563fd53244fc1ac575b Mon Sep 17 00:00:00 2001 From: "Andrey V. Lepikhov" Date: Thu, 23 Nov 2023 16:00:13 +0700 Subject: [PATCH] Transform OR clause to ANY expressions. Replace (expr op C1) OR (expr op C2) ...

Re: POC, WIP: OR-clause support for indexes

2023-12-03 Thread Andrei Lepikhov
gex expression, which should be corrected further. -- regards, Andrei Lepikhov Postgres Professional From 73031b7acae68494ddd0f9b1faf4c94aae3bd6b0 Mon Sep 17 00:00:00 2001 From: "Andrey V. Lepikhov" Date: Thu, 23 Nov 2023 16:00:13 +0700 Subject: [PATCH] Transform OR clause to ANY expressions.

Re: Custom explain options

2023-11-30 Thread Andrei Lepikhov
On 30/11/2023 22:40, Konstantin Knizhnik wrote: On 30/11/2023 5:59 am, Andrei Lepikhov wrote: On 21/10/2023 19:16, Konstantin Knizhnik wrote: EXPLAIN statement has a list of options (i.e. ANALYZE, BUFFERS, COST,...) which help to provide useful details of query execution. In Neon we have

Re: Report planning memory in EXPLAIN ANALYZE

2023-11-30 Thread Andrei Lepikhov
s on this point. I'm only one vote here. I agree; it should be disabled by default. The fact that memory consumption outputs with byte precision (very uncomfortable for perception) is a sign that the primary audience is developers and for debugging purposes. -- regards, Andrei Lepikhov Postgres Professional

Re: POC, WIP: OR-clause support for indexes

2023-11-30 Thread Andrei Lepikhov
ions, where uniform ORs are grouped into arrays, the optimizer will do such work with less overhead. -- regards, Andrei Lepikhov Postgres Professional

Re: Custom explain options

2023-11-29 Thread Andrei Lepikhov
utine. -- regards, Andrei Lepikhov Postgres Professional

Re: POC, WIP: OR-clause support for indexes

2023-11-28 Thread Andrei Lepikhov
On 28/11/2023 04:03, Robert Haas wrote: On Mon, Nov 27, 2023 at 3:02 AM Andrei Lepikhov wrote: On 25/11/2023 08:23, Alexander Korotkov wrote: I think patch certainly gets better in this aspect. One thing I can't understand is why do we use home-grown code for resolving hash-collisions.

Re: POC, WIP: OR-clause support for indexes

2023-11-27 Thread Andrei Lepikhov
-- regards, Andrei Lepikhov Postgres Professional From 8a43f5b50be6cb431046ab352fbcb9bdd3e4376c Mon Sep 17 00:00:00 2001 From: "Andrey V. Lepikhov" Date: Thu, 23 Nov 2023 16:00:13 +0700 Subject: [PATCH] Transform OR clause to ANY expressions. Replace (X=N1) OR (X=N2) ... with X = ANY(N1, N2

<    1   2   3   >