On Fri, Nov 5, 2021 at 4:53 AM Jeff Davis <pg...@j-davis.com> wrote: > > On Thu, 2021-11-04 at 14:33 +0530, Amit Kapila wrote: > > Can't different tableAM's have different representations > > for toast or may be some different concept like speculative > > insertions? > > The decoding plugin should just use the common interfaces to toast, and > if the table AM supports toast, everything should work fine. >
I think that is true only if table AM uses same format as heap for toast. It also seems to be relying heap tuple format. > The only > special thing it needs to do is check VARATT_IS_EXTERNAL_ONDISK(), > because on-disk toast data can't be decoded (which is true for heap, > too). > > I haven't looked as much into speculative insertions, but I don't think > those are a problem either. Shouldn't they be handled before they make > it into the change stream that the plugin sees? > They will be handled before the plugin sees them but I was talking about what if the table AM has some other WAL similar to WAL of speculative insertions that needs special handling. > > Similarly, I remember that for zheap we didn't had > > TransactionIds for subtransactions so we need to make some changes in > > logical decoding to compensate for that. I guess similar stuff could > > be required for another table AMs. Then a different table AM can have > > a different tuple format which won't work for current change records > > unless we convert it to heap tuple format before writing WAL for it. > > Columnar certainly has a different format. That makes me wonder whether > ReorderBufferChange and/or ReorderBufferTupleBuf are too low-level, and > we should have a higher-level representation of a change that is based > on slots. > > Can you tell me more about the changes you made for zheap? I still > don't understand why the decoding plugin would have to know what table > AM the change came from. > I am not talking about decoding plugin but rather decoding itself, basically, the work we do in reorderbuffer.c, decode.c, etc. The two things I remember were tuple format and transaction ids as mentioned in my previous email. I think we should try to find a solution for tuple format as the current decoding code relies on it if we want decoding to deal with another table AMs transparently. -- With Regards, Amit Kapila.