Re: Table AM Interface Enhancements

2024-05-24 Thread Mats Kindahl
hat slot implementation should be used, but that is a separate discussion.) Best wishes, Mats Kindahl

Re: Hooking into ExplainOneQuery() complicated by missing standard_ExplainOneQuery

2024-03-11 Thread Mats Kindahl
to have this standard explain wrapper at the beginning of next week. > > If others have any comments, feel free. > > Well, done as of a04ddd077e61. > Thanks Michael! -- Best wishes, Mats Kindahl, Timescale

Re: Hooking into ExplainOneQuery() complicated by missing standard_ExplainOneQuery

2024-03-05 Thread Mats Kindahl
dd something to the node explain and the > summary, if only because they can significantly influence the planner > and executor's behaviour. > > [1] > > https://www.postgresql.org/message-id/flat/6cd5caa7-06e1-4460-bf35-00a59da3f677%40garret.ru This is an excellent example of where such a hook would be useful. -- Best wishes, Mats Kindahl, Timescale

Re: Hooking into ExplainOneQuery() complicated by missing standard_ExplainOneQuery

2024-03-04 Thread Mats Kindahl
also allow this information to propagate upwards. We happen to have buffer information right now, but allowing something similar to be added dynamically by extending ExplainNode and passing down a callback to standard_ExplainOneQuery. Best wishes, Mats Kindahl

Hooking into ExplainOneQuery() complicated by missing standard_ExplainOneQuery

2024-03-04 Thread Mats Kindahl
and summarize information from custom scan nodes, but I had no immediate use for that so did not add it here. I would still be interested in hearing if you think this is something that would be useful to the community. Best wishes, Mats Kindahl, Timescale From eaab4d7c088ff3ee9b0e6ec3de96677bafd184c0 Mon

Re: glibc qsort() vulnerability

2024-02-17 Thread Mats Kindahl
On Fri, Feb 16, 2024 at 9:09 PM Nathan Bossart wrote: > On Fri, Feb 16, 2024 at 01:45:52PM +0100, Mats Kindahl wrote: > > Looks good to me. > > Committed. > Thanks Nathan! > > -- > Nathan Bossart > Amazon Web Services: https://aws.amazon.com >

Re: glibc qsort() vulnerability

2024-02-16 Thread Mats Kindahl
On Fri, Feb 16, 2024 at 12:32 AM Nathan Bossart wrote: > Here is what I have staged for commit. > Looks good to me. Checked that all of the comparisons are in the expected order, except inside compDESC, cmp_lsn, and resource_priority_cmp, where the order is reversed. Best wishes, Mats K

Re: glibc qsort() vulnerability

2024-02-13 Thread Mats Kindahl
a < b ? true : a > b ? false : false a < b ? true : false a < b When it comes to (a < b) - (a > b) there are no such rules added since it is not a very common case. Clang fares better for this case and at least shows the two alternatives as equal. Maybe we should change to use the original version equivalent to the inline function above since that works better with surrounding code? Best wishes, Mats Kindahl > > Greetings, > > Andres >

Re: glibc qsort() vulnerability

2024-02-12 Thread Mats Kindahl
On Mon, Feb 12, 2024 at 4:57 PM Nathan Bossart wrote: > On Sun, Feb 11, 2024 at 03:44:42PM +0100, Mats Kindahl wrote: > > On Sat, Feb 10, 2024 at 9:53 PM Nathan Bossart > > > wrote: > >> and I think we should expand on some of the commentary in int.h. > >>

Re: glibc qsort() vulnerability

2024-02-11 Thread Mats Kindahl
On Sat, Feb 10, 2024 at 9:53 PM Nathan Bossart wrote: > On Sat, Feb 10, 2024 at 08:59:06AM +0100, Mats Kindahl wrote: > > Split the code into two patches: one that just adds the functions > > (including the new pg_cmp_size()) to common/int.h and one that starts > using

Re: glibc qsort() vulnerability

2024-02-09 Thread Mats Kindahl
On Fri, Feb 9, 2024 at 9:08 PM Nathan Bossart wrote: > On Fri, Feb 09, 2024 at 08:43:21PM +0100, Mats Kindahl wrote: > > QQ: right now it looks like this: > > > > static inline int > > pg_cmp_u16(uint16 a, uint16 b) > > { &

Re: glibc qsort() vulnerability

2024-02-09 Thread Mats Kindahl
On Fri, Feb 9, 2024 at 8:26 PM Mats Kindahl wrote: > On Fri, Feb 9, 2024 at 5:24 PM Nathan Bossart > wrote: > >> On Fri, Feb 09, 2024 at 08:52:26AM +0100, Mats Kindahl wrote: >> > Here is a new version introducing pg_cmp_s32 and friends and use them >> > instea

Re: glibc qsort() vulnerability

2024-02-09 Thread Mats Kindahl
On Fri, Feb 9, 2024 at 5:27 PM Tom Lane wrote: > Nathan Bossart writes: > > On Fri, Feb 09, 2024 at 08:52:26AM +0100, Mats Kindahl wrote: > >> The types "int" and "size_t" are treated as s32 and u32 respectively > since > >> that seems to

Re: glibc qsort() vulnerability

2024-02-09 Thread Mats Kindahl
On Fri, Feb 9, 2024 at 5:27 PM Tom Lane wrote: > Nathan Bossart writes: > > On Fri, Feb 09, 2024 at 08:52:26AM +0100, Mats Kindahl wrote: > >> The types "int" and "size_t" are treated as s32 and u32 respectively > since > >> that seems to

Re: glibc qsort() vulnerability

2024-02-09 Thread Mats Kindahl
On Fri, Feb 9, 2024 at 5:24 PM Nathan Bossart wrote: > On Fri, Feb 09, 2024 at 08:52:26AM +0100, Mats Kindahl wrote: > > Here is a new version introducing pg_cmp_s32 and friends and use them > > instead of the INT_CMP macro introduced before. It also moves the > > defini

Re: glibc qsort() vulnerability

2024-02-08 Thread Mats Kindahl
r most of the code, even if strictly not correct (size_t can be an unsigned long int for some architecture). I also noted that in many situations size_t values are treated as "int" so there is an overflow risk here, even if small. For example, the result of "list_length" is

Re: glibc qsort() vulnerability

2024-02-08 Thread Mats Kindahl
th following int.h's pattern of including > [s]igned/[u]nsigned > > in the name, an efficient implementation for signed might not be the > same as > > for unsigned. And if we use static inlines, we need to do so for correct > > semantics anyway. > > Seems reasonable to me. > Agree. Best wishes, Mats Kindahl > > -- > Nathan Bossart > Amazon Web Services: https://aws.amazon.com >

Re: glibc qsort() vulnerability

2024-02-08 Thread Mats Kindahl
On Thu, Feb 8, 2024 at 7:44 PM Tom Lane wrote: > Nathan Bossart writes: > > On Thu, Feb 08, 2024 at 02:16:11PM +0100, Mats Kindahl wrote: > >> +/* > >> + * Compare two integers and return -1, 0, or 1 without risking > overflow. > >> + * > >> + * T

Re: glibc qsort() vulnerability

2024-02-08 Thread Mats Kindahl
integers and other common types. Also noted it is quite common to have this pattern in various places to do lexicographic sort of multiple values and continue the comparison if they are equal. Not sure if that is something we should look at. Best wishes, Mats Kindahl > > -- > Nathan Bos

Re: glibc qsort() vulnerability

2024-02-08 Thread Mats Kindahl
On Thu, Feb 8, 2024 at 12:01 PM Mats Kindahl wrote: > On Wed, Feb 7, 2024 at 9:56 PM Nathan Bossart > wrote: > >> On Wed, Feb 07, 2024 at 08:46:56PM +0200, Heikki Linnakangas wrote: >> > Doesn't hurt to fix the comparison functions, and +1 on using the same >>

Re: glibc qsort() vulnerability

2024-02-08 Thread Mats Kindahl
On Thu, Feb 8, 2024 at 4:08 AM Nathan Bossart wrote: > Mats, I apologize for steamrolling a bit here. I'll take a step back into > a reviewer role. > No worries. :) > > -- > Nathan Bossart > Amazon Web Services: https://aws.amazon.com >

Re: glibc qsort() vulnerability

2024-02-08 Thread Mats Kindahl
On Wed, Feb 7, 2024 at 9:56 PM Nathan Bossart wrote: > On Wed, Feb 07, 2024 at 08:46:56PM +0200, Heikki Linnakangas wrote: > > Doesn't hurt to fix the comparison functions, and +1 on using the same > > pattern everywhere. > > I attached a new version of the patch with some small adjustments. I

Re: glibc qsort() vulnerability

2024-02-07 Thread Mats Kindahl
verflow. This method is used for some int16 as well but since standard conversions in C will perform the arithmetics in "int" precision, this cannot overflow, so added a comment there. It might still be a good idea to follow the same pattern for the int16 routines, but since there is no b

Re: glibc qsort() vulnerability

2024-02-07 Thread Mats Kindahl
On Tue, Feb 6, 2024 at 4:11 PM Tom Lane wrote: > Mats Kindahl writes: > > There is a bug in glibc's qsort() algorithm that runs the risk of > creating > > an out-of-bounds error if the comparison function is not transitive, for > > example, if subtraction is us

glibc qsort() vulnerability

2024-02-06 Thread Mats Kindahl
cases and add a comment to one case that was not clear that it could not overflow. Best wishes, Mats Kindahl, Timescale From 83e2f14ab259f568034b07c2f99e34c6dff2c7b5 Mon Sep 17 00:00:00 2001 From: Mats Kindahl Date: Tue, 6 Feb 2024 14:53:53 +0100 Subject: Ensure comparison functions are transitive

Re: Add test module for Table Access Method

2024-01-16 Thread Mats Kindahl
wanted to implement is a table access method that you can use to test the interface, something like a "mock TAM" where you can programmatically decide on the responses to unit-test the API. I was thinking that you could implement a framework that allows you to implement the TAM in

Re: Add test module for Table Access Method

2024-01-16 Thread Mats Kindahl
a proxy TAM for a heap TAM > which does nothing but logging used TAM methods, its arguments and > return values. This would be a good example and also potentially can > be used as a debugging tool. > We wrote a table access method for experimenting with and to be able to

Re: Table AM and DROP TABLE [ Was: Table AM and DDLs]

2022-11-16 Thread Mats Kindahl
e the full power of this concept. On Tue, Aug 2, 2022 at 1:44 AM Andres Freund wrote: > Hi, > > On 2021-04-05 21:57:12 +0200, Mats Kindahl wrote: > >2. In the storage layer, the function RelationDropStorage is called, > >which will record the table to be dropped in

Re: Table AM and DROP TABLE [ Was: Table AM and DDLs]

2022-07-29 Thread Mats Kindahl
by placing restrictions on how DDLs can be used in transactions, but that would create unnecessary restrictions for the end-user. It might also be possible to find implementation workarounds by placing code at strategic points in the implementation, but this is error prone and the risk of making an error is higher. Best wishes, Mats Kindahl > > - Heikki >

Re: RFC: Table access methods and scans

2021-06-04 Thread Mats Kindahl
Hi Jeff, On Fri, Jun 4, 2021 at 2:52 AM Jeff Davis wrote: > Hi, > > On Wed, 2021-03-31 at 22:10 +0200, Mats Kindahl wrote: > > As an example of how this is useful, I noticed the work by Heikki and > > Ashwin [1], where they return a `TableScanDesc` that contains >

Table AM and DROP TABLE [ Was: Table AM and DDLs]

2021-04-05 Thread Mats Kindahl
On Mon, Mar 22, 2021 at 12:16 AM Andres Freund wrote: > Hi, > > On 2021-03-03 22:15:18 +0100, Mats Kindahl wrote: > > On Tue, Feb 23, 2021 at 2:11 AM Andres Freund > wrote: > > Thanks for the answer and sorry about the late reply. > > Mine is even later ;) >

RFC: Table access methods and scans

2021-03-31 Thread Mats Kindahl
gly enough, `ScanKey` is generated for `IndexScan` and I think that the same approach could be used for sequential scans: pick out the quals that can be used for filtering and offer them to the table access method through the `scan_begin` callback. Thoughts around this? Best wishes, Mats Kindahl [1] ht

Re: Table AM and DDLs

2021-03-03 Thread Mats Kindahl
On Tue, Feb 23, 2021 at 2:11 AM Andres Freund wrote: > Hi, > Hi Andres, Thanks for the answer and sorry about the late reply. > On 2021-02-22 08:33:21 +0100, Mats Kindahl wrote: > > I started to experiment with the table access method interface to see if > it > > can

Table AM and DDLs

2021-02-21 Thread Mats Kindahl
it might want to optimize the internal storage when the schema changes. I have not been able to find any discussions around this, but it seems like a natural thing to do with a table. Have I misunderstood how this works? Best wishes, Mats Kindahl Timescale From