Hi, I just discovered this very interesting thread on batch execution. It's very relevant to some of the work we've done on columnar engines and table access methods in TimescaleDB. I am happy to share some of our learnings. I have not digested the full history of the thread yet, so I apologize in advance if I repeat things you've already discussed. I intend to study the current patch set in detail later. I'll just describe at a high level what we did and the approaches we took. I am not sure yet how much this overlaps with the ongoing batch executor work, but hopefully it still provides valuable insights---if nothing else, perhaps only to validate your current approach.
Obviously, for TimescaleDB, we approached batch processing from the point-of-view of an extension where we could not change the existing interfaces. That led us to initially build a columnar engine using custom executor nodes to implement columnar processing on top of the heap engine. This approach wasn't very principled, making our code less compatible with both existing and future PostgreSQL APIs, which is not ideal. One example is the table access method API (TAM), which didn't exist when we built the first version of our columnar engine. However, as TAM became available, along with other new APIs, we attempted to be more principled by building a TAM over our existing columnar engine. Perhaps that work can inform and help improve this work on batch execution? I think the initial instinct when looking at this problem is to extend the TAM API with new batch functions. However, since that wasn't an option, we decided to build it mostly as a custom TupleTableSlot implementation using its extensible interface. On reflection, I think that's actually the more principled approach because it keeps the TAM and underlying storage very transparent to "upper" executor nodes and APIs. So, for this purpose we built a custom TupleTableSlot for batching that we called ArrowTupleTableSlot because we use the Arrow format internally for processing columnar data. The implementation of the slot can be found here: https://github.com/timescale/timescaledb/blob/2.21.x/tsl/src/hypercore/arrow_tts.h So, the way this functions is as follows: If the TAM supports columnar format or batching (which is transparent to nodes from the TAM), it produces an ArrowTupleTableSlot which internally holds a batch of data and an index into that batch. Initially, when the TAM's `getnextslot()` produces the slot, the batch is not even materialized; it's only a reference to the relevant data in storage (which could be compressed). When a processing node (let's say SeqScan for simplicity) eventually calls getsomeattrs() on the slot, the batch is materialized (e.g., decompressed) and the slot's index points to the first entry in the batch array. This "late materialization" is critical for good performance because it delays decompression until after filtering, or it decompresses only the columns needed for the filtering. Internally, the slot's batch might hold relevant metadata, like min/max values so an executor node can filter the entire batch slot before materialization. Even without min/max values, a vectorized filter can quickly scan a column in the batch and determine that no "rows" pass the filter. Skipping the entire batch is done by setting the slot's batch index to "end". When TAM's getnextslot() is called for the second time, one of two things happens: 1. The TAM increments the index in the existing slot's batch and sees that it is not yet done, just returning the existing slot again without reading new data. 2. The TAM notices that the slot's batch index is at "end", and then reads the next batch into the slot, setting the index to 0. I am skipping many other details for now, but hopefully this is enough to get the gist of it. So, what's the advantage of this approach? 1. It is entirely transparent to existing processing nodes and the TAM interface. In fact, SeqScan and any other similar nodes can work on top of the TAM and the slot without knowing it produces batches internally. However, a SeqScan doesn't take full advantage of all the benefits either. 2. Custom "batch-aware" nodes (e.g., a "BatchSeqScan") can detect that they read an ArrowTupleTableSlot from the TAM based on the slot type, and therefore initiate special batch processing. E.g., filtering entire batches or running vectorized expressions over them. 3. The TAM and slot interfaces remain largely the same. However, to really take advantage of columnar processing, some API extensions are needed. 4. The batch slot can carry the batch data up to a vector aggregation node, which can efficiently compute functions like avg() over the entire batch in one go. Here's a non-exhaustive discussion of some API changes that might be necessary with the above approach: TupleTableSlot: - getsomeattrs(): This API will materialize all columns up to the given index. This does not work for columnar data because each column needs separate decompression. Instead, it should be possible to materialize only the columns used by filters or projections. In fact, if your query projects columns 2,1 (in that order) you might initially decompress only column 4 for vectorized filtering, end then _after_ filtering, columns 2,1 for projection on the batches that survive. We worked around this by keeping filter and projection column masks in the ArrowTupleTableSlot so that we only materialized the necessary columns whenever getsomeattrs was called, using the separate masks for filtering and projection. - For batch processing, the TupleTableSlot API probably needs some additional API functions for batch-aware processing nodes, e.g., getbatchsize(), increment_batch_index(), etc. - One issue we had was with projections: When passing the slot up to an Agg node, PostgreSQL typically wants to do projection in order to convert the slot to a VirtualTupleTableSlot with only the projected columns and data and maybe change the column order. However, that removes the batch data and prevents vectorized aggregation. Instead, there must be a way to project columns while preserving the batch and/or columnar format. One way to do this is to introduce indirection: keep the batch data in its original column order and, instead, apply projection to an array of column indexes in the slot to reorder them, rather than actually reordering and copying the data into a VirtualTupleTableSlot. TAM: For basic batch processing, the TAM API didn't actually need any changes with the approach we took. However, you probably want a way to push down certain expressions, for example, skipping entire RowGroups (in Parquet terminology). Someone already proposed doing this in beginscan(), which might work. We used ScanKeys to push down some basic min/max filters. But maybe the API can be improved? One thing that is worth keeping in mind is that the semantics of columnar filtering might differ from existing filtering in PostgreSQL. Normally, when providing a ScanKey the contract is that only data passing the filter is passed upward and no additioanl quals filtering is needed. However, with batches and "sparse" metadata, like min/max you can only exclude batches that cannot possibly contain any relevant values. For example, if min/max "temp" of a batch is 5 and 10, and the filter is "temp" > 7, the entire batch must be passed "upward" even though some values within the batch should be excluded. Therefore, you must still run the same qual filters on the batch even though you passed down ScanKeys. Otherwise the entire filtering and expression engines must move down into the TAM, which I don't think is the right approach. That said, some TAMs might provide a full expression and filtering engine internally, and should probably be allowed to do that. Other columnar/batch TAMs might want to offload at least some filtering to PostgreSQL's filtering engine instead. For example, they could perform basic min/max batch filtering internally but offload advanced expression filtering to higher-level processing nodes in PostgreSQL. FWIW, SeqScan currently doesn't provide ScanKeys to TAMs even if the API exists, so we implemented a custom SeqScan node to push down the relevant keys. Still, executor nodes can perform similar filtering using metadata within the TupleTableSlot, as I described. If no materialization has occurred, there is no performance loss. For columnar data, the TID format was a challenge. I know there have been proposals to change it already, so won't add much to that discussion for now, except mentioning that we hacked around it by re-encoding the TID to hold <block, offset, batch-index>. This was an imperfect approach, but it worked with some limitations. This approach allowed the standard index AM API to work on top of our columnar TAM, enabling support for cursors and other features. We also dealt with many other issues, such as cost models for planning when producing columnar data, multi-stage filtering (batch skipping -> vectorized filters -> regular quals over each value), batch deletions and inserts, index support via IAM (e.g., should min/max filters be an IAM or internal to TAM, or both). I will stop here for now, as this is already a lengthy addition to this thread. Happy to continue the discussion and answer questions if you have any. Best regards, Erik On Tue, Jul 21, 2026 at 8:03 AM Amit Langote <[email protected]> wrote: > Hi, > > I still haven't gotten a chance to review Denis's proposal and the > patches closely, but I wanted to second what Antonin wrote below. > > On Fri, Jul 17, 2026 at 9:12 PM Antonin Houska <[email protected]> wrote: > > Denis Smirnov <[email protected]> wrote: > > > > > I am not sure adding a new table AM callback for this is the right > > > direction, at least for this patch. > > > > > > My concern is that scan_getnextbatch still looks like a row-oriented > > > interface. For a Parquet-like AM, with columnar storage and block-level > > > filters such as bloom/fuse filters, the useful API would need to pass > > > down things like the required columns, pushed-down predicates, and > maybe > > > a limit. Just asking the AM for the next batch of rows does not give > the > > > storage layer enough information to avoid unnecessary work. > > > > Is there a reason not to include this information in the scan descriptor > of > > particular AM? > > Yeah, I think that information is better passed to beginscan() than to > getnextslot() or getnextbatch(). The scan descriptor is scan-lifetime > state, so telling the AM which columns and predicates it can use > belongs at scan setup; what the getnext* functions return per call is > a separate question. That work could be undertaken independently of > allowing them to return batches. > > I'll try to reply properly on the rest of Denis's points later this week. > > -- > Thanks, Amit Langote > > >
