> The even better implementation, that would keep the advantages of the > partitioning scheme while avoiding increased memory consumption and > rereading all bitmaps by all participants, is to keep the partitions in > shared memory from the staert and having each participant directly > insert the TIDs into the right partition, e.g. > > BlockNumber blockno = ItemPointerGetBlockNumber(cur_tid); > uint32 p = murmurhash32(blockno / 256) % nparticipants; > tbm_add_tuple(tidbitmaps[p], cur_tid); > > If we use enough partitions, say 8 per participant, then we might get > away with locking the entire tbm_add_tuple() operation, hoping that the > TIDs distribute more or less uniformly across all partitions. Now when > thinking about it, that might be the better approach, also for the > TIDBitmap-based variant.
I just remembered why I didn't go for this variant: TIDBitmap uses simplehash.h which cannot easily live in shared memory. The hash map array itself can by using a custom allocator (TIDBitmap does that already) but the control struct that contains the size, etc. currently cannot. > If we do it this way, integrating the TID radix tree would actually be > straightforward. If the TID radix tree can live in shared memory we could first go with my variant that repartitions and then later swap that part out with one TID radix tree per partition stored in shared memory and updated by all participants as described in my previous mail. -- David Geier
