On Thu, 18 May 2023 at 12:52, Aleksander Alekseev <aleksan...@timescale.com> wrote: > > Hi Nikita, > > > this part of the PostgreSQL screams to be revised and improved > > I completely agree. The problem with TOAST pointers is that they are > not extendable at the moment which prevents adding new compression > algorithms (e.g. ZSTD), new features like compression dictionaries > [1], etc. I suggest we add extensibility in order to solve this > problem for the foreseeable future for everyone. > > > where Custom TOAST Pointer is distinguished from Regular one by va_flag > > field > > which is a part of varlena header > > I don't think that varlena header is the best place to distinguish a > classical TOAST pointer from an extended one. On top of that I don't > see any free bits that would allow adding such a flag to the on-disk > varlena representation [2]. > > The current on-disk TOAST pointer representation is following: > > ``` > typedef struct varatt_external > { > int32 va_rawsize; /* Original data size (includes header) */ > uint32 va_extinfo; /* External saved size (without header) and > * compression method */ > Oid va_valueid; /* Unique ID of value within TOAST table */ > Oid va_toastrelid; /* RelID of TOAST table containing it */ > } varatt_external; > ```
No, that's inaccurate. The complete on-disk representation of a varatt is { uint8 va_header; /* Always 0x80 or 0x01 */ uint8 va_tag; /* Type of datum */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Type-dependent data, for toasted values that's currently only a varatt_external */ } varattrib_1b_e; With va_tag being filled with one of the vartag_external values: typedef enum vartag_external { VARTAG_INDIRECT = 1, VARTAG_EXPANDED_RO = 2, VARTAG_EXPANDED_RW = 3, VARTAG_ONDISK = 18 } vartag_external; This enum still has many options to go before it exceeds the maximum of the uint8 va_tag field. Therefore, I don't think we have no disk representations left, nor do I think we'll need to add another option to the ToastCompressionId enum. As an example, we can add another VARTAG option for dictionary-enabled external toast; like what the pluggable toast patch worked on. I think we can salvage some ideas from that patch, even if the main idea got stuck. Kind regards, Matthias van de Meent Neon Inc.