On Wed, 13 Jul 2022 at 17:20, David Rowley <[email protected]> wrote:
> I did consider that in all cases where the allocations are above
> allocChunkLimit that the chunk is put on a dedicated block and in
> fact, the blockoffset is always the same for those. I wondered if we
> could use the full 60 bits for the chunksize for those cases. The
> reason I didn't pursue that is because:
As it turns out, we don't really need to explicitly store the chunk
size for chunks which are on dedicated blocks. We can just calculate
this by subtracting the pointer to the memory from the block's endptr.
The block offset is always fixed too, like I mentioned above.
I've now revised the patch to completely get rid of the concept of
"large" chunks and instead memory chunks are always 8 bytes in size.
I've created a struct to this effect and named it MemoryChunk. All
memory contexts make use of this new struct. The header bit which I
was previously using to denote a "large" chunk now marks if the chunk
is "external", meaning that the MemoryChunk does not have knowledge of
the chunk size and/or block offset. The MemoryContext itself is
expected to manage this when the chunk is external. I've coded up
aset.c and generation.c to always use these external chunks when size
> set->allocChunkLimit. There is now a coding pattern like the
following (taken from AllocSetRealloc:
if (MemoryChunkIsExternal(chunk))
{
block = ExternalChunkGetBlock(chunk);
oldsize = block->endptr - (char *) pointer;
}
else
{
block = MemoryChunkGetBlock(chunk);
oldsize = MemoryChunkGetSize(chunk);
}
Here the ExternalChunkGetBlock() macro is just subtracting the
ALLOC_BLOCKHDRSZ from the chunk pointer to obtain the block pointer,
whereas MemoryChunkGetBlock() is subtracting the blockoffset as is
stored in the 30-bits of the chunk header.
Andres and I had a conversation off-list about the storage of freelist
pointers. Currently I have these stored in the actual allocated
memory. The minimum allocation size is 8-bytes, which is big enough
for storing sizeof(void *). Andres suggested that it might be safer to
store this link at the end of the chunk rather than at the start.
I've not quite done that in the attached, but doing this should just
be a matter of adjusting the GetFreeListLink() macro to add the
chunksize - sizeof(AllocFreelistLink).
I did a little bit of benchmarking of the attached with a scale 1 pgbench.
master = 0b039e3a8
master
drowley@amd3990x:~$ pgbench -c 156 -j 156 -T 60 -S postgres
tps = 1638436.367793 (without initial connection time)
tps = 1652688.009579 (without initial connection time)
tps = 1671281.281856 (without initial connection time)
master + v2 patch
drowley@amd3990x:~$ pgbench -c 156 -j 156 -T 60 -S postgres
tps = 1825346.734486 (without initial connection time)
tps = 1824539.294142 (without initial connection time)
tps = 1807359.758822 (without initial connection time)
~10% faster.
There are a couple of things to note that might require discussion:
1. I've added a new restriction that block sizes cannot be above 1GB.
This is because the 30-bits in the MemoryChunk used for storing the
offset between the chunk and the block wouldn't be able to store the
offset if the chunk was offset more than 1GB from the block. I used
debian code search to see if I could find any user code that used
blocks this big. I found nothing.
2. slab.c has a new restriction that the chunk size cannot be >= 1GB.
I'm not at all concerned about this. I think if someone needed chunks
that big there'd be no benefits from slab context over an aset
context.
3. As mentioned above, aset.c now stores freelist pointers in the
allocated chunk's memory. This allows us to get the header down to 8
bytes instead of today's 16 bytes. There's an increased danger that
buggy code that writes to memory after a pfree could stomp on this.
I think the patch is now starting to take shape. I've added it to the
September commitfest [1].
David
[1] https://commitfest.postgresql.org/39/3810/
From 13034cd62f47eb26aca717e0c65a21f8c976477e Mon Sep 17 00:00:00 2001
From: David Rowley <[email protected]>
Date: Thu, 9 Jun 2022 20:09:22 +1200
Subject: [PATCH v2] Improve performance of and reduce overheads of memory
management
Whenever we palloc a chunk of memory, traditionally, we prefix the
returned pointer with a pointer to the memory context to which the chunk
belongs. This is required so that we're able to easily determine the
owning context when performing operations such as pfree() and repalloc().
For the AllocSet context, prior to this commit we additionally prefixed
the pointer to the owning context with the size of the chunk. This made
the header 16 bytes in size. This 16-byte overhead was required for all
AllocSet allocations regardless of the allocation size.
For the generation context, the problem was worse; in addition to the
pointer to the owning context and chunk size, we also stored a pointer to
the owning block so that we could track the number of freed chunks on a
block.
The slab allocator had a 16-byte chunk header.
The changes being made here reduce the chunk header size down to just 8
bytes for all 3 of our memory context types. For small to medium sized
allocations, this significantly increases the number of chunks that we can
fit on a given block which results in much more efficient use of memory.
Additionally, this commit completely changes the rule that pointers must
be directly prefixed by the owning memory context and instead, we now
insist that they're directly prefixed by an 8-byte value where the least
significant 3-bits are set to a value to indicate which type of memory
context the pointer belongs to. Using those 3 bits as an index to a new
array which stores the methods for each memory context type, we're now
able to pass the pointer given to functions such as pfree and repalloc to
the function specific to that context implementation to allow them to
devise their own methods of finding the memory context which owns the
given allocated chunk of memory.
The reason we're able to reduce the chunk header down to just 8 bytes is
because of the way we make use of the remaining 61 bits of the required
8-byte chunk header. Here we also implement a general-purpose MemoryChunk
struct which makes use of those 61 remaining bits to store the size of the
chunk and the number of bytes which must be subtracted from the chunk to
get a reference to the block that the chunk is stored on. We reserve 30
bits each for these fields and the 1 additional remaining bit is used to
mark chunks with a chunk size larger than the value we can store in 30
bits (1GB). All 3 of our memory context types now use MemoryChunk as a
chunk header.
With this MemoryChunk, each of our 3 memory context types can quickly
obtain a reference to the block any given chunk is located on. AllocSet
is able to find the context to which the chunk is owned by first obtaining
a reference to the block and then referencing the block's 'aset' field.
Generation contexts did not have such a field in the GenerationBlock, so
one is added in this commit.
For allocations of 1GB or more, both AllocSet and the generation memory
context types always allocate these on dedicated blocks. Since we cannot
store the chunk size for these in the given 30 bits, we use the remaining
1 bit in the chunk header to mark such chunks as "external". Here
external means the memory context implementation itself must be able to
determine the size of the chunk and the block to which the chunk belongs.
For AllocSet and generation contexts this is easy information to obtain as
these larger chunks are always on dedicated blocks, finding the block is
just a matter of subtracting the size of the block header from the chunk
pointer. We obtain the size of the chunk just by subtracting the size of
the block header from the size of the block which we already know the size
of by way of the block's endptr.
For the slab allocator, this commit adds a new restriction that slab
chunks cannot be >= 1GB in size. If there happened to be any users of
slab.c which used chunk sizes this large, they really should be using
AllocSet instead.
We also add a restriction that block sizes for all 3 of the memory
allocators cannot be 1GB or larger. We would be unable to store the
number of bytes that the block is offset from the chunk stored beyond this
1GB boundary on any block that was larger than 1GB.
---
src/backend/utils/mmgr/README | 60 ++-
src/backend/utils/mmgr/aset.c | 451 ++++++++++++-----------
src/backend/utils/mmgr/generation.c | 326 ++++++++--------
src/backend/utils/mmgr/mcxt.c | 90 ++++-
src/backend/utils/mmgr/slab.c | 190 ++++------
src/include/nodes/memnodes.h | 7 +-
src/include/utils/memutils.h | 48 +--
src/include/utils/memutils_internal.h | 117 ++++++
src/include/utils/memutils_memorychunk.h | 185 ++++++++++
9 files changed, 913 insertions(+), 561 deletions(-)
create mode 100644 src/include/utils/memutils_internal.h
create mode 100644 src/include/utils/memutils_memorychunk.h
diff --git a/src/backend/utils/mmgr/README b/src/backend/utils/mmgr/README
index 221b4bd343..2c98e92b69 100644
--- a/src/backend/utils/mmgr/README
+++ b/src/backend/utils/mmgr/README
@@ -378,35 +378,55 @@ MemoryContext like the parent and child contexts, and the
name of the
context.
This is essentially an abstract superclass, and the behavior is
-determined by the "methods" pointer is its virtual function table
-(struct MemoryContextMethods). Specific memory context types will use
-derived structs having these fields as their first fields. All the
+determined by the "methods" pointer which references which set of
+MemoryContextMethods are to be used. Specific memory context types will
+use derived structs having these fields as their first fields. All the
contexts of a specific type will have methods pointers that point to
-the same static table of function pointers.
+the corresponding element in the mcxt_methods[] array as defined in mcxt.c.
While operations like allocating from and resetting a context take the
relevant MemoryContext as a parameter, operations like free and
realloc are trickier. To make those work, we require all memory
context types to produce allocated chunks that are immediately,
-without any padding, preceded by a pointer to the corresponding
-MemoryContext.
+without any padding, preceded by a uint64 value of which the least
+significant 3 bits are set to the owning context's MemoryContextMethodID.
+This allows the code to determine the correct MemoryContextMethods to
+use by looking up the mcxt_methods[] array using the 3 bits as an index
+into that array.
If a type of allocator needs additional information about its chunks,
like e.g. the size of the allocation, that information can in turn
-precede the MemoryContext. This means the only overhead implied by
-the memory context mechanism is a pointer to its context, so we're not
-constraining context-type designers very much.
-
-Given this, routines like pfree determine their corresponding context
-with an operation like (although that is usually encapsulated in
-GetMemoryChunkContext())
-
- MemoryContext context = *(MemoryContext*) (((char *) pointer) -
sizeof(void *));
-
-and then invoke the corresponding method for the context
-
- context->methods->free_p(pointer);
-
+either be encoded into the remaining 61 bits of the preceding uint64 value
+or if more space is required, additional values may be stored directly prior
+to the uint64 value. It is up to the context implementation to manage this.
+
+Given this, routines like pfree can determine which set of
+MemoryContextMethods to call the free_p function for by calling
+GetMemoryChunkMethodID() and finding the corresponding MemoryContextMethods
+in the mcxt_methods array[]. For convenience, the MCXT_METHOD() macro is
+provided, making the code as simple as:
+
+void
+pfree(void *pointer)
+{
+ MCXT_METHOD(pointer, free_p)(pointer);
+}
+
+All of the current memory contexts make use of the MemoryChunk header type
+which is defined in memutils_memorychunk.h. This suits all of the existing
+context types well as it makes use of the remaining 61-bits of the uint64
+header to efficiently encode the size of the chunk of memory and the number of
+bytes which must be subtracted from the chunk in order to obtain a reference
+to the block that the chunk belongs to. 30 bits are used for each of these.
+If more than 30 bits are required (i.e. > 1GB) then the memory context must
+manage that itself. This can be done by calling the
+MemoryChunkSetHdrMaskExternal() function on the given chunk. Currently, each
+memory context type stores large allocations on dedicated blocks (which always
+contain only a single chunk). For these, finding the block is simple as we
+know that the chunk must be the first on the given block, so the block is
+always at a fixed offset to the chunk. For these, finding the size of the
+chunk is also simple as the block always stores an endptr which we can use to
+calculate the size of the chunk.
More Control Over aset.c Behavior
---------------------------------
diff --git a/src/backend/utils/mmgr/aset.c b/src/backend/utils/mmgr/aset.c
index ec3e264a73..f052deee8a 100644
--- a/src/backend/utils/mmgr/aset.c
+++ b/src/backend/utils/mmgr/aset.c
@@ -49,6 +49,8 @@
#include "port/pg_bitutils.h"
#include "utils/memdebug.h"
#include "utils/memutils.h"
+#include "utils/memutils_memorychunk.h"
+#include "utils/memutils_internal.h"
/*--------------------
* Chunk freelist k holds chunks of size 1 << (k + ALLOC_MINBITS),
@@ -66,7 +68,9 @@
* CAUTION: ALLOC_MINBITS must be large enough so that
* 1<<ALLOC_MINBITS is at least MAXALIGN,
* or we may fail to align the smallest chunks adequately.
- * 8-byte alignment is enough on all currently known machines.
+ * 8-byte alignment is enough on all currently known machines. This 8-byte
+ * minimum also allows us to store a pointer to the next freelist item within
+ * the chunk of memory itself.
*
* With the current parameters, request sizes up to 8K are treated as chunks,
* larger requests go into dedicated blocks. Change ALLOCSET_NUM_FREELISTS
@@ -98,10 +102,9 @@
*/
#define ALLOC_BLOCKHDRSZ MAXALIGN(sizeof(AllocBlockData))
-#define ALLOC_CHUNKHDRSZ sizeof(struct AllocChunkData)
+#define ALLOC_CHUNKHDRSZ sizeof(MemoryChunk)
typedef struct AllocBlockData *AllocBlock; /* forward reference */
-typedef struct AllocChunkData *AllocChunk;
/*
* AllocPointer
@@ -109,6 +112,25 @@ typedef struct AllocChunkData *AllocChunk;
*/
typedef void *AllocPointer;
+/*
+ * AllocFreelistLink
+ * When pfreeing memory, if we maintain a freelist for the given
chunk's
+ * size then we use a AllocFreelistLink to point to the current
item in
+ * the AllocSetContext's freelist and then set the given freelist
element
+ * to point to the chunk being freed.
+ */
+typedef struct AllocFreelistLink
+{
+ MemoryChunk *next;
+} AllocFreelistLink;
+
+/*
+ * Obtain a AllocFreelistLink for the given chunk. Allocation sizes are
+ * always at least sizeof(AllocFreelistLink), so we reuse the pointer's memory
+ * itself to store the freelist link
+ */
+#define GetFreeListLink(c, sz) ((AllocFreelistLink *) ((char *) (c) +
ALLOC_CHUNKHDRSZ))
+
/*
* AllocSetContext is our standard implementation of MemoryContext.
*
@@ -123,7 +145,7 @@ typedef struct AllocSetContext
MemoryContextData header; /* Standard memory-context fields */
/* Info about storage allocated in this context: */
AllocBlock blocks; /* head of list of blocks in
this set */
- AllocChunk freelist[ALLOCSET_NUM_FREELISTS]; /* free chunk
lists */
+ MemoryChunk *freelist[ALLOCSET_NUM_FREELISTS]; /* free chunk lists */
/* Allocation parameters for this context: */
Size initBlockSize; /* initial block size */
Size maxBlockSize; /* maximum block size */
@@ -139,8 +161,8 @@ typedef AllocSetContext *AllocSet;
/*
* AllocBlock
* An AllocBlock is the unit of memory that is obtained by aset.c
- * from malloc(). It contains one or more AllocChunks, which are
- * the units requested by palloc() and freed by pfree().
AllocChunks
+ * from malloc(). It contains one or more MemoryChunks, which are
+ * the units requested by palloc() and freed by pfree().
MemoryChunks
* cannot be returned to malloc() individually, instead they are
put
* on freelists by pfree() and re-used by the next palloc() that
has
* a matching request size.
@@ -158,48 +180,12 @@ typedef struct AllocBlockData
} AllocBlockData;
/*
- * AllocChunk
- * The prefix of each piece of memory in an AllocBlock
- *
- * Note: to meet the memory context APIs, the payload area of the chunk must
- * be maxaligned, and the "aset" link must be immediately adjacent to the
- * payload area (cf. GetMemoryChunkContext). We simplify matters for this
- * module by requiring sizeof(AllocChunkData) to be maxaligned, and then
- * we can ensure things work by adding any required alignment padding before
- * the "aset" field. There is a static assertion below that the alignment
- * is done correctly.
- */
-typedef struct AllocChunkData
-{
- /* size is always the size of the usable space in the chunk */
- Size size;
-#ifdef MEMORY_CONTEXT_CHECKING
- /* when debugging memory usage, also store actual requested size */
- /* this is zero in a free chunk */
- Size requested_size;
-
-#define ALLOCCHUNK_RAWSIZE (SIZEOF_SIZE_T * 2 + SIZEOF_VOID_P)
-#else
-#define ALLOCCHUNK_RAWSIZE (SIZEOF_SIZE_T + SIZEOF_VOID_P)
-#endif /*
MEMORY_CONTEXT_CHECKING */
-
- /* ensure proper alignment by adding padding if needed */
-#if (ALLOCCHUNK_RAWSIZE % MAXIMUM_ALIGNOF) != 0
- char padding[MAXIMUM_ALIGNOF - ALLOCCHUNK_RAWSIZE %
MAXIMUM_ALIGNOF];
-#endif
-
- /* aset is the owning aset if allocated, or the freelist link if free */
- void *aset;
- /* there must not be any padding to reach a MAXALIGN boundary here! */
-} AllocChunkData;
-
-/*
- * Only the "aset" field should be accessed outside this module.
+ * Only the "hdrmask" field should be accessed outside this module.
* We keep the rest of an allocated chunk's header marked NOACCESS when using
* valgrind. But note that chunk headers that are in a freelist are kept
* accessible, for simplicity.
*/
-#define ALLOCCHUNK_PRIVATE_LEN offsetof(AllocChunkData, aset)
+#define ALLOCCHUNK_PRIVATE_LEN offsetof(MemoryChunk, hdrmask)
/*
* AllocPointerIsValid
@@ -213,10 +199,12 @@ typedef struct AllocChunkData
*/
#define AllocSetIsValid(set) PointerIsValid(set)
-#define AllocPointerGetChunk(ptr) \
- ((AllocChunk)(((char *)(ptr)) -
ALLOC_CHUNKHDRSZ))
-#define AllocChunkGetPointer(chk) \
- ((AllocPointer)(((char *)(chk)) +
ALLOC_CHUNKHDRSZ))
+/*
+ * We always store external chunks on a dedicated block. This makes fetching
+ * the block from an external chunk easy since it's always the first and only
+ * chunk on the block.
+ */
+#define ExternalChunkGetBlock(chunk) (AllocBlock) ((char *) chunk -
ALLOC_BLOCKHDRSZ)
/*
* Rather than repeatedly creating and deleting memory contexts, we keep some
@@ -260,42 +248,6 @@ static AllocSetFreeList context_freelists[2] =
}
};
-/*
- * These functions implement the MemoryContext API for AllocSet contexts.
- */
-static void *AllocSetAlloc(MemoryContext context, Size size);
-static void AllocSetFree(MemoryContext context, void *pointer);
-static void *AllocSetRealloc(MemoryContext context, void *pointer, Size size);
-static void AllocSetReset(MemoryContext context);
-static void AllocSetDelete(MemoryContext context);
-static Size AllocSetGetChunkSpace(MemoryContext context, void *pointer);
-static bool AllocSetIsEmpty(MemoryContext context);
-static void AllocSetStats(MemoryContext context,
- MemoryStatsPrintFunc
printfunc, void *passthru,
- MemoryContextCounters *totals,
- bool print_to_stderr);
-
-#ifdef MEMORY_CONTEXT_CHECKING
-static void AllocSetCheck(MemoryContext context);
-#endif
-
-/*
- * This is the virtual function table for AllocSet contexts.
- */
-static const MemoryContextMethods AllocSetMethods = {
- AllocSetAlloc,
- AllocSetFree,
- AllocSetRealloc,
- AllocSetReset,
- AllocSetDelete,
- AllocSetGetChunkSpace,
- AllocSetIsEmpty,
- AllocSetStats
-#ifdef MEMORY_CONTEXT_CHECKING
- ,AllocSetCheck
-#endif
-};
-
/* ----------
* AllocSetFreeIndex -
@@ -387,18 +339,20 @@ AllocSetContextCreateInternal(MemoryContext parent,
AllocSet set;
AllocBlock block;
- /* Assert we padded AllocChunkData properly */
+ /* Assert we padded MemoryChunk properly */
StaticAssertStmt(ALLOC_CHUNKHDRSZ == MAXALIGN(ALLOC_CHUNKHDRSZ),
- "sizeof(AllocChunkData) is not
maxaligned");
- StaticAssertStmt(offsetof(AllocChunkData, aset) + sizeof(MemoryContext)
==
- ALLOC_CHUNKHDRSZ,
- "padding calculation in AllocChunkData
is wrong");
+ "sizeof(MemoryChunk) is not
maxaligned");
/*
* First, validate allocation parameters. Once these were regular
runtime
* test and elog's, but in practice Asserts seem sufficient because
nobody
* varies their parameters at runtime. We somewhat arbitrarily enforce
a
- * minimum 1K block size.
+ * minimum 1K block size. We restrict the maximum block size to
+ * MEMORYCHUNK_MAX_SIZE as MemoryChunks are limited to this in regards
to
+ * addressing the offset between the chunk and the block that the chunk
is
+ * stored on. We would be unable to store the offset between the chunk
+ * and block for any chunks that were beyond MEMORYCHUNK_MAX_SIZE bytes
+ * into the block if the block was to be larger than this.
*/
Assert(initBlockSize == MAXALIGN(initBlockSize) &&
initBlockSize >= 1024);
@@ -409,6 +363,7 @@ AllocSetContextCreateInternal(MemoryContext parent,
(minContextSize == MAXALIGN(minContextSize) &&
minContextSize >= 1024 &&
minContextSize <= maxBlockSize));
+ Assert(maxBlockSize <= MEMORYCHUNK_MAX_SIZE);
/*
* Check whether the parameters match either available freelist. We do
@@ -443,7 +398,7 @@ AllocSetContextCreateInternal(MemoryContext parent,
/* Reinitialize its header, installing correct name and
parent */
MemoryContextCreate((MemoryContext) set,
T_AllocSetContext,
-
&AllocSetMethods,
+ MCTX_ASET_ID,
parent,
name);
@@ -519,7 +474,8 @@ AllocSetContextCreateInternal(MemoryContext parent,
*
* We have to have allocChunkLimit a power of two, because the requested
* and actually-allocated sizes of any chunk must be on the same side of
- * the limit, else we get confused about whether the chunk is "big".
+ * the limit, else we get confused about whether the chunk is on a
+ * dedicated block.
*
* Also, allocChunkLimit must not exceed ALLOCSET_SEPARATE_THRESHOLD.
*/
@@ -534,7 +490,7 @@ AllocSetContextCreateInternal(MemoryContext parent,
/* Finally, do the type-independent part of context creation */
MemoryContextCreate((MemoryContext) set,
T_AllocSetContext,
- &AllocSetMethods,
+ MCTX_ASET_ID,
parent,
name);
@@ -555,7 +511,7 @@ AllocSetContextCreateInternal(MemoryContext parent,
* thrash malloc() when a context is repeatedly reset after small allocations,
* which is typical behavior for per-tuple contexts.
*/
-static void
+void
AllocSetReset(MemoryContext context)
{
AllocSet set = (AllocSet) context;
@@ -623,7 +579,7 @@ AllocSetReset(MemoryContext context)
*
* Unlike AllocSetReset, this *must* free all resources of the set.
*/
-static void
+void
AllocSetDelete(MemoryContext context)
{
AllocSet set = (AllocSet) context;
@@ -717,12 +673,12 @@ AllocSetDelete(MemoryContext context)
* is marked, as mcxt.c will set it to UNDEFINED. In some paths we will
* return space that is marked NOACCESS - AllocSetRealloc has to beware!
*/
-static void *
+void *
AllocSetAlloc(MemoryContext context, Size size)
{
AllocSet set = (AllocSet) context;
AllocBlock block;
- AllocChunk chunk;
+ MemoryChunk *chunk;
int fidx;
Size chunk_size;
Size blksize;
@@ -746,18 +702,20 @@ AllocSetAlloc(MemoryContext context, Size size)
block->aset = set;
block->freeptr = block->endptr = ((char *) block) + blksize;
- chunk = (AllocChunk) (((char *) block) + ALLOC_BLOCKHDRSZ);
- chunk->aset = set;
- chunk->size = chunk_size;
+ chunk = (MemoryChunk *) (((char *) block) + ALLOC_BLOCKHDRSZ);
+
+ /* mark the MemoryChunk as externally managed */
+ MemoryChunkSetHdrMaskExternal(chunk, MCTX_ASET_ID);
+
#ifdef MEMORY_CONTEXT_CHECKING
chunk->requested_size = size;
/* set mark to catch clobber of "unused" space */
if (size < chunk_size)
- set_sentinel(AllocChunkGetPointer(chunk), size);
+ set_sentinel(MemoryChunkGetPointer(chunk), size);
#endif
#ifdef RANDOMIZE_ALLOCATED_MEMORY
/* fill the allocated space with junk */
- randomize_mem((char *) AllocChunkGetPointer(chunk), size);
+ randomize_mem((char *) MemoryChunkGetPointer(chunk), size);
#endif
/*
@@ -780,13 +738,13 @@ AllocSetAlloc(MemoryContext context, Size size)
}
/* Ensure any padding bytes are marked NOACCESS. */
- VALGRIND_MAKE_MEM_NOACCESS((char *) AllocChunkGetPointer(chunk)
+ size,
+ VALGRIND_MAKE_MEM_NOACCESS((char *)
MemoryChunkGetPointer(chunk) + size,
chunk_size -
size);
/* Disallow external access to private part of chunk header. */
VALGRIND_MAKE_MEM_NOACCESS(chunk, ALLOCCHUNK_PRIVATE_LEN);
- return AllocChunkGetPointer(chunk);
+ return MemoryChunkGetPointer(chunk);
}
/*
@@ -799,31 +757,37 @@ AllocSetAlloc(MemoryContext context, Size size)
chunk = set->freelist[fidx];
if (chunk != NULL)
{
- Assert(chunk->size >= size);
+ AllocFreelistLink *link;
- set->freelist[fidx] = (AllocChunk) chunk->aset;
+ Assert(MemoryChunkGetSize(chunk) >= size);
- chunk->aset = (void *) set;
+ chunk_size = (1 << ALLOC_MINBITS) << fidx;
+ link = GetFreeListLink(chunk, chunk_size);
+
+ /* pop this chunk off the freelist */
+ VALGRIND_MAKE_MEM_DEFINED(link, sizeof(AllocFreelistLink));
+ set->freelist[fidx] = link->next;
+ VALGRIND_MAKE_MEM_NOACCESS(link, sizeof(AllocFreelistLink));
#ifdef MEMORY_CONTEXT_CHECKING
chunk->requested_size = size;
/* set mark to catch clobber of "unused" space */
- if (size < chunk->size)
- set_sentinel(AllocChunkGetPointer(chunk), size);
+ if (size < MemoryChunkGetSize(chunk))
+ set_sentinel(MemoryChunkGetPointer(chunk), size);
#endif
#ifdef RANDOMIZE_ALLOCATED_MEMORY
/* fill the allocated space with junk */
- randomize_mem((char *) AllocChunkGetPointer(chunk), size);
+ randomize_mem((char *) MemoryChunkGetPointer(chunk), size);
#endif
/* Ensure any padding bytes are marked NOACCESS. */
- VALGRIND_MAKE_MEM_NOACCESS((char *) AllocChunkGetPointer(chunk)
+ size,
- chunk->size
- size);
+ VALGRIND_MAKE_MEM_NOACCESS((char *)
MemoryChunkGetPointer(chunk) + size,
+
MemoryChunkGetSize(chunk) - size);
/* Disallow external access to private part of chunk header. */
VALGRIND_MAKE_MEM_NOACCESS(chunk, ALLOCCHUNK_PRIVATE_LEN);
- return AllocChunkGetPointer(chunk);
+ return MemoryChunkGetPointer(chunk);
}
/*
@@ -856,6 +820,7 @@ AllocSetAlloc(MemoryContext context, Size size)
*/
while (availspace >= ((1 << ALLOC_MINBITS) +
ALLOC_CHUNKHDRSZ))
{
+ AllocFreelistLink *link;
Size availchunk = availspace -
ALLOC_CHUNKHDRSZ;
int a_fidx =
AllocSetFreeIndex(availchunk);
@@ -871,22 +836,26 @@ AllocSetAlloc(MemoryContext context, Size size)
availchunk = ((Size) 1 << (a_fidx +
ALLOC_MINBITS));
}
- chunk = (AllocChunk) (block->freeptr);
+ chunk = (MemoryChunk *) (block->freeptr);
/* Prepare to initialize the chunk header. */
VALGRIND_MAKE_MEM_UNDEFINED(chunk,
ALLOC_CHUNKHDRSZ);
-
block->freeptr += (availchunk +
ALLOC_CHUNKHDRSZ);
availspace -= (availchunk + ALLOC_CHUNKHDRSZ);
- chunk->size = availchunk;
+ MemoryChunkSetHdrMask(chunk, block, availchunk,
MCTX_ASET_ID);
#ifdef MEMORY_CONTEXT_CHECKING
chunk->requested_size = 0; /* mark it free
*/
#endif
- chunk->aset = (void *) set->freelist[a_fidx];
+ /* push this chunk onto the freelist */
+ link = GetFreeListLink(chunk, availspace);
+
+ VALGRIND_MAKE_MEM_DEFINED(link,
sizeof(AllocFreelistLink));
+ link->next = set->freelist[a_fidx];
+ VALGRIND_MAKE_MEM_NOACCESS(link,
sizeof(AllocFreelistLink));
+
set->freelist[a_fidx] = chunk;
}
-
/* Mark that we need to create a new block */
block = NULL;
}
@@ -954,7 +923,7 @@ AllocSetAlloc(MemoryContext context, Size size)
/*
* OK, do the allocation
*/
- chunk = (AllocChunk) (block->freeptr);
+ chunk = (MemoryChunk *) (block->freeptr);
/* Prepare to initialize the chunk header. */
VALGRIND_MAKE_MEM_UNDEFINED(chunk, ALLOC_CHUNKHDRSZ);
@@ -962,67 +931,66 @@ AllocSetAlloc(MemoryContext context, Size size)
block->freeptr += (chunk_size + ALLOC_CHUNKHDRSZ);
Assert(block->freeptr <= block->endptr);
- chunk->aset = (void *) set;
- chunk->size = chunk_size;
+ MemoryChunkSetHdrMask(chunk, block, chunk_size, MCTX_ASET_ID);
#ifdef MEMORY_CONTEXT_CHECKING
chunk->requested_size = size;
/* set mark to catch clobber of "unused" space */
- if (size < chunk->size)
- set_sentinel(AllocChunkGetPointer(chunk), size);
+ if (size < chunk_size)
+ set_sentinel(MemoryChunkGetPointer(chunk), size);
#endif
#ifdef RANDOMIZE_ALLOCATED_MEMORY
/* fill the allocated space with junk */
- randomize_mem((char *) AllocChunkGetPointer(chunk), size);
+ randomize_mem((char *) MemoryChunkGetPointer(chunk), size);
#endif
/* Ensure any padding bytes are marked NOACCESS. */
- VALGRIND_MAKE_MEM_NOACCESS((char *) AllocChunkGetPointer(chunk) + size,
+ VALGRIND_MAKE_MEM_NOACCESS((char *) MemoryChunkGetPointer(chunk) + size,
chunk_size - size);
/* Disallow external access to private part of chunk header. */
VALGRIND_MAKE_MEM_NOACCESS(chunk, ALLOCCHUNK_PRIVATE_LEN);
- return AllocChunkGetPointer(chunk);
+ return MemoryChunkGetPointer(chunk);
}
/*
* AllocSetFree
* Frees allocated memory; memory is removed from the set.
*/
-static void
-AllocSetFree(MemoryContext context, void *pointer)
+void
+AllocSetFree(void *pointer)
{
- AllocSet set = (AllocSet) context;
- AllocChunk chunk = AllocPointerGetChunk(pointer);
+ AllocSet set;
+ MemoryChunk *chunk = PointerGetMemoryChunk(pointer);
/* Allow access to private part of chunk header. */
VALGRIND_MAKE_MEM_DEFINED(chunk, ALLOCCHUNK_PRIVATE_LEN);
+ if (MemoryChunkIsExternal(chunk))
+ {
+
+ AllocBlock block = ExternalChunkGetBlock(chunk);
+
+ set = block->aset;
+
#ifdef MEMORY_CONTEXT_CHECKING
- /* Test for someone scribbling on unused space in chunk */
- if (chunk->requested_size < chunk->size)
- if (!sentinel_ok(pointer, chunk->requested_size))
- elog(WARNING, "detected write past chunk end in %s %p",
- set->header.name, chunk);
+ {
+ Size chunk_size = block->endptr - (char *)
pointer;
+
+ /* Test for someone scribbling on unused space in chunk
*/
+ if (chunk->requested_size < chunk_size)
+ if (!sentinel_ok(pointer,
chunk->requested_size))
+ elog(WARNING, "detected write past
chunk end in %s %p",
+ set->header.name, chunk);
+ }
#endif
- if (chunk->size > set->allocChunkLimit)
- {
- /*
- * Big chunks are certain to have been allocated as single-chunk
- * blocks. Just unlink that block and return it to malloc().
- */
- AllocBlock block = (AllocBlock) (((char *) chunk) -
ALLOC_BLOCKHDRSZ);
/*
- * Try to verify that we have a sane block pointer: it should
- * reference the correct aset, and freeptr and endptr should
point
- * just past the chunk.
+ * Try to verify that we have a sane block pointer, the freeptr
should
+ * match the endptr.
*/
- if (block->aset != set ||
- block->freeptr != block->endptr ||
- block->freeptr != ((char *) block) +
- (chunk->size + ALLOC_BLOCKHDRSZ + ALLOC_CHUNKHDRSZ))
+ if (block->freeptr != block->endptr)
elog(ERROR, "could not find block containing chunk %p",
chunk);
/* OK, remove block from aset's list and free it */
@@ -1033,7 +1001,7 @@ AllocSetFree(MemoryContext context, void *pointer)
if (block->next)
block->next->prev = block->prev;
- context->mem_allocated -= block->endptr - ((char *) block);
+ set->header.mem_allocated -= block->endptr - ((char *) block);
#ifdef CLOBBER_FREED_MEMORY
wipe_mem(block, block->freeptr - ((char *) block));
@@ -1042,20 +1010,37 @@ AllocSetFree(MemoryContext context, void *pointer)
}
else
{
- /* Normal case, put the chunk into appropriate freelist */
- int fidx = AllocSetFreeIndex(chunk->size);
+ AllocBlock block = MemoryChunkGetBlock(chunk);
+ Size chunk_size = MemoryChunkGetSize(chunk);
+ AllocFreelistLink *link = GetFreeListLink(chunk, chunksize);
+ int fidx;
- chunk->aset = (void *) set->freelist[fidx];
+ set = block->aset;
+
+#ifdef MEMORY_CONTEXT_CHECKING
+ /* Test for someone scribbling on unused space in chunk */
+ if (chunk->requested_size < chunk_size)
+ if (!sentinel_ok(pointer, chunk->requested_size))
+ elog(WARNING, "detected write past chunk end in
%s %p",
+ set->header.name, chunk);
+#endif
#ifdef CLOBBER_FREED_MEMORY
- wipe_mem(pointer, chunk->size);
+ wipe_mem(pointer, chunk_size);
#endif
+ /* Normal case, put the chunk into appropriate freelist */
+ fidx = AllocSetFreeIndex(chunk_size);
+
+ /* push this chunk onto the top of the freelist for this chunk
size */
+ VALGRIND_MAKE_MEM_DEFINED(link, sizeof(AllocFreelistLink));
+ link->next = set->freelist[fidx];
+ VALGRIND_MAKE_MEM_NOACCESS(link, sizeof(AllocFreelistLink));
+ set->freelist[fidx] = chunk;
#ifdef MEMORY_CONTEXT_CHECKING
/* Reset requested_size to 0 in chunks that are on freelist */
chunk->requested_size = 0;
#endif
- set->freelist[fidx] = chunk;
}
}
@@ -1071,17 +1056,28 @@ AllocSetFree(MemoryContext context, void *pointer)
* (In principle, we could use VALGRIND_GET_VBITS() to rediscover the old
* request size.)
*/
-static void *
-AllocSetRealloc(MemoryContext context, void *pointer, Size size)
+void *
+AllocSetRealloc(void *pointer, Size size)
{
- AllocSet set = (AllocSet) context;
- AllocChunk chunk = AllocPointerGetChunk(pointer);
+ AllocBlock block;
+ AllocSet set;
+ MemoryChunk *chunk = PointerGetMemoryChunk(pointer);
Size oldsize;
/* Allow access to private part of chunk header. */
VALGRIND_MAKE_MEM_DEFINED(chunk, ALLOCCHUNK_PRIVATE_LEN);
- oldsize = chunk->size;
+ if (MemoryChunkIsExternal(chunk))
+ {
+ block = ExternalChunkGetBlock(chunk);
+ oldsize = block->endptr - (char *) pointer;
+ }
+ else
+ {
+ block = MemoryChunkGetBlock(chunk);
+ oldsize = MemoryChunkGetSize(chunk);
+ }
+ set = block->aset;
#ifdef MEMORY_CONTEXT_CHECKING
/* Test for someone scribbling on unused space in chunk */
@@ -1091,37 +1087,25 @@ AllocSetRealloc(MemoryContext context, void *pointer,
Size size)
set->header.name, chunk);
#endif
- if (oldsize > set->allocChunkLimit)
+ if (MemoryChunkIsExternal(chunk))
{
/*
* The chunk must have been allocated as a single-chunk block.
Use
* realloc() to make the containing block bigger, or smaller,
with
* minimum space wastage.
*/
- AllocBlock block = (AllocBlock) (((char *) chunk) -
ALLOC_BLOCKHDRSZ);
Size chksize;
Size blksize;
Size oldblksize;
/*
- * Try to verify that we have a sane block pointer: it should
- * reference the correct aset, and freeptr and endptr should
point
- * just past the chunk.
+ * Try to verify that we have a sane block pointer, the freeptr
should
+ * match the endptr.
*/
- if (block->aset != set ||
- block->freeptr != block->endptr ||
- block->freeptr != ((char *) block) +
- (oldsize + ALLOC_BLOCKHDRSZ + ALLOC_CHUNKHDRSZ))
+ if (block->freeptr != block->endptr)
elog(ERROR, "could not find block containing chunk %p",
chunk);
- /*
- * Even if the new request is less than set->allocChunkLimit,
we stick
- * with the single-chunk block approach. Therefore we need
- * chunk->size to be bigger than set->allocChunkLimit, so we
don't get
- * confused about the chunk's status in future calls.
- */
- chksize = Max(size, set->allocChunkLimit + 1);
- chksize = MAXALIGN(chksize);
+ chksize = MAXALIGN(size);
/* Do the realloc */
blksize = chksize + ALLOC_BLOCKHDRSZ + ALLOC_CHUNKHDRSZ;
@@ -1136,21 +1120,20 @@ AllocSetRealloc(MemoryContext context, void *pointer,
Size size)
}
/* updated separately, not to underflow when (oldblksize >
blksize) */
- context->mem_allocated -= oldblksize;
- context->mem_allocated += blksize;
+ set->header.mem_allocated -= oldblksize;
+ set->header.mem_allocated += blksize;
block->freeptr = block->endptr = ((char *) block) + blksize;
/* Update pointers since block has likely been moved */
- chunk = (AllocChunk) (((char *) block) + ALLOC_BLOCKHDRSZ);
- pointer = AllocChunkGetPointer(chunk);
+ chunk = (MemoryChunk *) (((char *) block) + ALLOC_BLOCKHDRSZ);
+ pointer = MemoryChunkGetPointer(chunk);
if (block->prev)
block->prev->next = block;
else
set->blocks = block;
if (block->next)
block->next->prev = block;
- chunk->size = chksize;
#ifdef MEMORY_CONTEXT_CHECKING
#ifdef RANDOMIZE_ALLOCATED_MEMORY
@@ -1172,9 +1155,8 @@ AllocSetRealloc(MemoryContext context, void *pointer,
Size size)
#endif
chunk->requested_size = size;
-
/* set mark to catch clobber of "unused" space */
- if (size < chunk->size)
+ if (size < chksize)
set_sentinel(pointer, size);
#else /*
!MEMORY_CONTEXT_CHECKING */
@@ -1289,34 +1271,58 @@ AllocSetRealloc(MemoryContext context, void *pointer,
Size size)
memcpy(newPointer, pointer, oldsize);
/* free old chunk */
- AllocSetFree((MemoryContext) set, pointer);
+ AllocSetFree(pointer);
return newPointer;
}
}
+/*
+ * AllocSetGetChunkContext
+ * Return the MemoryContext that 'pointer' belongs to.
+ */
+MemoryContext
+AllocSetGetChunkContext(void *pointer)
+{
+ MemoryChunk *chunk = PointerGetMemoryChunk(pointer);
+ AllocBlock block;
+ AllocSet set;
+
+ if (MemoryChunkIsExternal(chunk))
+ block = ExternalChunkGetBlock(chunk);
+ else
+ block = (AllocBlock) MemoryChunkGetBlock(chunk);
+
+ set = block->aset;
+
+ return &set->header;
+}
+
/*
* AllocSetGetChunkSpace
* Given a currently-allocated chunk, determine the total space
* it occupies (including all memory-allocation overhead).
*/
-static Size
-AllocSetGetChunkSpace(MemoryContext context, void *pointer)
+Size
+AllocSetGetChunkSpace(void *pointer)
{
- AllocChunk chunk = AllocPointerGetChunk(pointer);
- Size result;
+ MemoryChunk *chunk = PointerGetMemoryChunk(pointer);
- VALGRIND_MAKE_MEM_DEFINED(chunk, ALLOCCHUNK_PRIVATE_LEN);
- result = chunk->size + ALLOC_CHUNKHDRSZ;
- VALGRIND_MAKE_MEM_NOACCESS(chunk, ALLOCCHUNK_PRIVATE_LEN);
- return result;
+ if (MemoryChunkIsExternal(chunk))
+ {
+ AllocBlock block = ExternalChunkGetBlock(chunk);
+
+ return block->endptr - (char *) chunk;
+ }
+
+ return MemoryChunkGetSize(chunk) + ALLOC_CHUNKHDRSZ;
}
/*
* AllocSetIsEmpty
* Is an allocset empty of any allocated space?
*/
-static bool
+bool
AllocSetIsEmpty(MemoryContext context)
{
/*
@@ -1339,7 +1345,7 @@ AllocSetIsEmpty(MemoryContext context)
* totals: if not NULL, add stats about this context into *totals.
* print_to_stderr: print stats to stderr if true, elog otherwise.
*/
-static void
+void
AllocSetStats(MemoryContext context,
MemoryStatsPrintFunc printfunc, void *passthru,
MemoryContextCounters *totals, bool print_to_stderr)
@@ -1363,13 +1369,21 @@ AllocSetStats(MemoryContext context,
}
for (fidx = 0; fidx < ALLOCSET_NUM_FREELISTS; fidx++)
{
- AllocChunk chunk;
+ MemoryChunk *chunk = set->freelist[fidx];
- for (chunk = set->freelist[fidx]; chunk != NULL;
- chunk = (AllocChunk) chunk->aset)
+ while (chunk != NULL)
{
+ Size chksz = MemoryChunkGetSize(chunk);
+ AllocFreelistLink *link = GetFreeListLink(chunk, chksz);
+
+ Assert(((1 << ALLOC_MINBITS) << fidx) == chksz);
+
freechunks++;
- freespace += chunk->size + ALLOC_CHUNKHDRSZ;
+ freespace += chksz + ALLOC_CHUNKHDRSZ;
+
+ VALGRIND_MAKE_MEM_DEFINED(link,
sizeof(AllocFreelistLink));
+ chunk = link->next;
+ VALGRIND_MAKE_MEM_NOACCESS(link,
sizeof(AllocFreelistLink));
}
}
@@ -1404,7 +1418,7 @@ AllocSetStats(MemoryContext context,
* find yourself in an infinite loop when trouble occurs, because this
* routine will be entered again when elog cleanup tries to release memory!
*/
-static void
+void
AllocSetCheck(MemoryContext context)
{
AllocSet set = (AllocSet) context;
@@ -1421,6 +1435,7 @@ AllocSetCheck(MemoryContext context)
long blk_used = block->freeptr - bpoz;
long blk_data = 0;
long nchunks = 0;
+ bool has_external_chunk = false;
if (set->keeper == block)
total_allocated += block->endptr - ((char *) set);
@@ -1452,14 +1467,25 @@ AllocSetCheck(MemoryContext context)
*/
while (bpoz < block->freeptr)
{
- AllocChunk chunk = (AllocChunk) bpoz;
+ MemoryChunk *chunk = (MemoryChunk *) bpoz;
+ AllocBlock chunkblock;
Size chsize,
dsize;
/* Allow access to private part of chunk header. */
VALGRIND_MAKE_MEM_DEFINED(chunk,
ALLOCCHUNK_PRIVATE_LEN);
- chsize = chunk->size; /* aligned chunk size */
+ if (MemoryChunkIsExternal(chunk))
+ {
+ chsize = block->endptr - (char *)
MemoryChunkGetPointer(chunk); /* aligned chunk size */
+ chunkblock = ExternalChunkGetBlock(chunk);
+ has_external_chunk = true;
+ }
+ else
+ {
+ chsize = MemoryChunkGetSize(chunk); /* aligned
chunk size */
+ chunkblock = (AllocBlock)
MemoryChunkGetBlock(chunk);
+ }
dsize = chunk->requested_size; /* real data */
/*
@@ -1478,20 +1504,15 @@ AllocSetCheck(MemoryContext context)
elog(WARNING, "problem in alloc set %s: bad
single-chunk %p in block %p",
name, chunk, block);
- /*
- * If chunk is allocated, check for correct aset
pointer. (If it's
- * free, the aset is the freelist pointer, which we
can't check as
- * easily...) Note this is an incomplete test, since
palloc(0)
- * produces an allocated chunk with requested_size == 0.
- */
- if (dsize > 0 && chunk->aset != (void *) set)
- elog(WARNING, "problem in alloc set %s: bogus
aset link in block %p, chunk %p",
- name, block, chunk);
+ /* Check the block offset correctly matches this block
*/
+ if (block != chunkblock)
+ elog(WARNING, "problem in alloc set %s: bad
block offset for chunk %p in block %p",
+ name, chunk, block);
/*
* Check for overwrite of padding space in an allocated
chunk.
*/
- if (chunk->aset == (void *) set && dsize < chsize &&
+ if (dsize > 0 && dsize < chsize &&
!sentinel_ok(chunk, ALLOC_CHUNKHDRSZ + dsize))
elog(WARNING, "problem in alloc set %s:
detected write past chunk end in block %p, chunk %p",
name, block, chunk);
@@ -1500,7 +1521,7 @@ AllocSetCheck(MemoryContext context)
* If chunk is allocated, disallow external access to
private part
* of chunk header.
*/
- if (chunk->aset == (void *) set)
+ if (dsize > 0)
VALGRIND_MAKE_MEM_NOACCESS(chunk,
ALLOCCHUNK_PRIVATE_LEN);
blk_data += chsize;
@@ -1512,6 +1533,10 @@ AllocSetCheck(MemoryContext context)
if ((blk_data + (nchunks * ALLOC_CHUNKHDRSZ)) != blk_used)
elog(WARNING, "problem in alloc set %s: found
inconsistent memory block %p",
name, block);
+
+ if (has_external_chunk && nchunks > 1)
+ elog(WARNING, "problem in alloc set %s: external chunk
on non-dedicated block %p",
+ name, block);
}
Assert(total_allocated == context->mem_allocated);
diff --git a/src/backend/utils/mmgr/generation.c
b/src/backend/utils/mmgr/generation.c
index e530e272e0..02709c4bf6 100644
--- a/src/backend/utils/mmgr/generation.c
+++ b/src/backend/utils/mmgr/generation.c
@@ -39,15 +39,16 @@
#include "port/pg_bitutils.h"
#include "utils/memdebug.h"
#include "utils/memutils.h"
+#include "utils/memutils_memorychunk.h"
+#include "utils/memutils_internal.h"
#define Generation_BLOCKHDRSZ MAXALIGN(sizeof(GenerationBlock))
-#define Generation_CHUNKHDRSZ sizeof(GenerationChunk)
+#define Generation_CHUNKHDRSZ sizeof(MemoryChunk)
#define Generation_CHUNK_FRACTION 8
typedef struct GenerationBlock GenerationBlock; /* forward reference */
-typedef struct GenerationChunk GenerationChunk;
typedef void *GenerationPointer;
@@ -89,6 +90,7 @@ typedef struct GenerationContext
struct GenerationBlock
{
dlist_node node; /* doubly-linked list of blocks
*/
+ GenerationContext *context; /* pointer back to the owning context */
Size blksize; /* allocated size of this block
*/
int nchunks; /* number of chunks in
the block */
int nfree; /* number of free
chunks */
@@ -97,104 +99,36 @@ struct GenerationBlock
};
/*
- * GenerationChunk
- * The prefix of each piece of memory in a GenerationBlock
- *
- * Note: to meet the memory context APIs, the payload area of the chunk must
- * be maxaligned, and the "context" link must be immediately adjacent to the
- * payload area (cf. GetMemoryChunkContext). We simplify matters for this
- * module by requiring sizeof(GenerationChunk) to be maxaligned, and then
- * we can ensure things work by adding any required alignment padding before
- * the pointer fields. There is a static assertion below that the alignment
- * is done correctly.
- */
-struct GenerationChunk
-{
- /* size is always the size of the usable space in the chunk */
- Size size;
-#ifdef MEMORY_CONTEXT_CHECKING
- /* when debugging memory usage, also store actual requested size */
- /* this is zero in a free chunk */
- Size requested_size;
-
-#define GENERATIONCHUNK_RAWSIZE (SIZEOF_SIZE_T * 2 + SIZEOF_VOID_P * 2)
-#else
-#define GENERATIONCHUNK_RAWSIZE (SIZEOF_SIZE_T + SIZEOF_VOID_P * 2)
-#endif /*
MEMORY_CONTEXT_CHECKING */
-
- /* ensure proper alignment by adding padding if needed */
-#if (GENERATIONCHUNK_RAWSIZE % MAXIMUM_ALIGNOF) != 0
- char padding[MAXIMUM_ALIGNOF - GENERATIONCHUNK_RAWSIZE %
MAXIMUM_ALIGNOF];
-#endif
-
- GenerationBlock *block; /* block owning this chunk */
- GenerationContext *context; /* owning context, or NULL if freed chunk */
- /* there must not be any padding to reach a MAXALIGN boundary here! */
-};
-
-/*
- * Only the "context" field should be accessed outside this module.
+ * Only the "hdrmask" field should be accessed outside this module.
* We keep the rest of an allocated chunk's header marked NOACCESS when using
* valgrind. But note that freed chunk headers are kept accessible, for
* simplicity.
*/
-#define GENERATIONCHUNK_PRIVATE_LEN offsetof(GenerationChunk, context)
-
+#define GENERATIONCHUNK_PRIVATE_LEN offsetof(MemoryChunk, hdrmask)
/*
* GenerationIsValid
* True iff set is valid allocation set.
*/
#define GenerationIsValid(set) PointerIsValid(set)
-#define GenerationPointerGetChunk(ptr) \
- ((GenerationChunk *)(((char *)(ptr)) - Generation_CHUNKHDRSZ))
-#define GenerationChunkGetPointer(chk) \
- ((GenerationPointer *)(((char *)(chk)) + Generation_CHUNKHDRSZ))
+/*
+ * We always store external chunks on a dedicated block. This makes fetching
+ * the block from an external chunk easy since it's always the first and only
+ * chunk on the block.
+ */
+#define ExternalChunkGetBlock(chunk) \
+ (GenerationBlock *) ((char *) chunk - Generation_BLOCKHDRSZ)
/* Inlined helper functions */
-static inline void GenerationBlockInit(GenerationBlock *block, Size blksize);
+static inline void GenerationBlockInit(GenerationContext *context,
+
GenerationBlock *block,
+ Size
blksize);
static inline bool GenerationBlockIsEmpty(GenerationBlock *block);
static inline void GenerationBlockMarkEmpty(GenerationBlock *block);
static inline Size GenerationBlockFreeBytes(GenerationBlock *block);
static inline void GenerationBlockFree(GenerationContext *set,
GenerationBlock *block);
-/*
- * These functions implement the MemoryContext API for Generation contexts.
- */
-static void *GenerationAlloc(MemoryContext context, Size size);
-static void GenerationFree(MemoryContext context, void *pointer);
-static void *GenerationRealloc(MemoryContext context, void *pointer, Size
size);
-static void GenerationReset(MemoryContext context);
-static void GenerationDelete(MemoryContext context);
-static Size GenerationGetChunkSpace(MemoryContext context, void *pointer);
-static bool GenerationIsEmpty(MemoryContext context);
-static void GenerationStats(MemoryContext context,
- MemoryStatsPrintFunc
printfunc, void *passthru,
- MemoryContextCounters
*totals,
- bool print_to_stderr);
-
-#ifdef MEMORY_CONTEXT_CHECKING
-static void GenerationCheck(MemoryContext context);
-#endif
-
-/*
- * This is the virtual function table for Generation contexts.
- */
-static const MemoryContextMethods GenerationMethods = {
- GenerationAlloc,
- GenerationFree,
- GenerationRealloc,
- GenerationReset,
- GenerationDelete,
- GenerationGetChunkSpace,
- GenerationIsEmpty,
- GenerationStats
-#ifdef MEMORY_CONTEXT_CHECKING
- ,GenerationCheck
-#endif
-};
-
/*
* Public routines
@@ -225,15 +159,17 @@ GenerationContextCreate(MemoryContext parent,
/* Assert we padded GenerationChunk properly */
StaticAssertStmt(Generation_CHUNKHDRSZ ==
MAXALIGN(Generation_CHUNKHDRSZ),
- "sizeof(GenerationChunk) is not
maxaligned");
- StaticAssertStmt(offsetof(GenerationChunk, context) +
sizeof(MemoryContext) ==
- Generation_CHUNKHDRSZ,
- "padding calculation in
GenerationChunk is wrong");
+ "sizeof(MemoryChunk) is not
maxaligned");
/*
* First, validate allocation parameters. Asserts seem sufficient
because
* nobody varies their parameters at runtime. We somewhat arbitrarily
- * enforce a minimum 1K block size.
+ * enforce a minimum 1K block size. We restrict the maximum block size
to
+ * MEMORYCHUNK_MAX_SIZE as MemoryChunks are limited to this in regards
to
+ * addressing the offset between the chunk and the block that the chunk
is
+ * stored on. We would be unable to store the offset between the chunk
+ * and block for any chunks that were beyond MEMORYCHUNK_MAX_SIZE bytes
+ * into the block if the block was to be larger than this.
*/
Assert(initBlockSize == MAXALIGN(initBlockSize) &&
initBlockSize >= 1024);
@@ -244,6 +180,7 @@ GenerationContextCreate(MemoryContext parent,
(minContextSize == MAXALIGN(minContextSize) &&
minContextSize >= 1024 &&
minContextSize <= maxBlockSize));
+ Assert(maxBlockSize <= MEMORYCHUNK_MAX_SIZE);
/* Determine size of initial block */
allocSize = MAXALIGN(sizeof(GenerationContext)) +
@@ -278,7 +215,7 @@ GenerationContextCreate(MemoryContext parent,
block = (GenerationBlock *) (((char *) set) +
MAXALIGN(sizeof(GenerationContext)));
/* determine the block size and initialize it */
firstBlockSize = allocSize - MAXALIGN(sizeof(GenerationContext));
- GenerationBlockInit(block, firstBlockSize);
+ GenerationBlockInit(set, block, firstBlockSize);
/* add it to the doubly-linked list of blocks */
dlist_push_head(&set->blocks, &block->node);
@@ -310,7 +247,7 @@ GenerationContextCreate(MemoryContext parent,
/* Finally, do the type-independent part of context creation */
MemoryContextCreate((MemoryContext) set,
T_GenerationContext,
- &GenerationMethods,
+ MCTX_GENERATION_ID,
parent,
name);
@@ -326,7 +263,7 @@ GenerationContextCreate(MemoryContext parent,
* The code simply frees all the blocks in the context - we don't keep any
* keeper blocks or anything like that.
*/
-static void
+void
GenerationReset(MemoryContext context)
{
GenerationContext *set = (GenerationContext *) context;
@@ -371,7 +308,7 @@ GenerationReset(MemoryContext context)
* GenerationDelete
* Free all memory which is allocated in the given context.
*/
-static void
+void
GenerationDelete(MemoryContext context)
{
/* Reset to release all releasable GenerationBlocks */
@@ -393,12 +330,12 @@ GenerationDelete(MemoryContext context)
* is marked, as mcxt.c will set it to UNDEFINED. In some paths we will
* return space that is marked NOACCESS - GenerationRealloc has to beware!
*/
-static void *
+void *
GenerationAlloc(MemoryContext context, Size size)
{
GenerationContext *set = (GenerationContext *) context;
GenerationBlock *block;
- GenerationChunk *chunk;
+ MemoryChunk *chunk;
Size chunk_size = MAXALIGN(size);
Size required_size = chunk_size + Generation_CHUNKHDRSZ;
@@ -414,6 +351,7 @@ GenerationAlloc(MemoryContext context, Size size)
context->mem_allocated += blksize;
/* block with a single (used) chunk */
+ block->context = set;
block->blksize = blksize;
block->nchunks = 1;
block->nfree = 0;
@@ -421,33 +359,33 @@ GenerationAlloc(MemoryContext context, Size size)
/* the block is completely full */
block->freeptr = block->endptr = ((char *) block) + blksize;
- chunk = (GenerationChunk *) (((char *) block) +
Generation_BLOCKHDRSZ);
- chunk->block = block;
- chunk->context = set;
- chunk->size = chunk_size;
+ chunk = (MemoryChunk *) (((char *) block) +
Generation_BLOCKHDRSZ);
+
+ /* mark the MemoryChunk as externally managed */
+ MemoryChunkSetHdrMaskExternal(chunk, MCTX_GENERATION_ID);
#ifdef MEMORY_CONTEXT_CHECKING
chunk->requested_size = size;
/* set mark to catch clobber of "unused" space */
if (size < chunk_size)
- set_sentinel(GenerationChunkGetPointer(chunk), size);
+ set_sentinel(MemoryChunkGetPointer(chunk), size);
#endif
#ifdef RANDOMIZE_ALLOCATED_MEMORY
/* fill the allocated space with junk */
- randomize_mem((char *) GenerationChunkGetPointer(chunk), size);
+ randomize_mem((char *) MemoryChunkGetPointer(chunk), size);
#endif
/* add the block to the list of allocated blocks */
dlist_push_head(&set->blocks, &block->node);
/* Ensure any padding bytes are marked NOACCESS. */
- VALGRIND_MAKE_MEM_NOACCESS((char *)
GenerationChunkGetPointer(chunk) + size,
+ VALGRIND_MAKE_MEM_NOACCESS((char *)
MemoryChunkGetPointer(chunk) + size,
chunk_size -
size);
/* Disallow external access to private part of chunk header. */
VALGRIND_MAKE_MEM_NOACCESS(chunk, GENERATIONCHUNK_PRIVATE_LEN);
- return GenerationChunkGetPointer(chunk);
+ return MemoryChunkGetPointer(chunk);
}
/*
@@ -516,7 +454,7 @@ GenerationAlloc(MemoryContext context, Size size)
context->mem_allocated += blksize;
/* initialize the new block */
- GenerationBlockInit(block, blksize);
+ GenerationBlockInit(set, block, blksize);
/* add it to the doubly-linked list of blocks */
dlist_push_head(&set->blocks, &block->node);
@@ -533,7 +471,7 @@ GenerationAlloc(MemoryContext context, Size size)
Assert(block != NULL);
Assert((block->endptr - block->freeptr) >= Generation_CHUNKHDRSZ +
chunk_size);
- chunk = (GenerationChunk *) block->freeptr;
+ chunk = (MemoryChunk *) block->freeptr;
/* Prepare to initialize the chunk header. */
VALGRIND_MAKE_MEM_UNDEFINED(chunk, Generation_CHUNKHDRSZ);
@@ -543,29 +481,26 @@ GenerationAlloc(MemoryContext context, Size size)
Assert(block->freeptr <= block->endptr);
- chunk->block = block;
- chunk->context = set;
- chunk->size = chunk_size;
-
+ MemoryChunkSetHdrMask(chunk, block, chunk_size, MCTX_GENERATION_ID);
#ifdef MEMORY_CONTEXT_CHECKING
chunk->requested_size = size;
/* set mark to catch clobber of "unused" space */
- if (size < chunk->size)
- set_sentinel(GenerationChunkGetPointer(chunk), size);
+ if (size < chunk_size)
+ set_sentinel(MemoryChunkGetPointer(chunk), size);
#endif
#ifdef RANDOMIZE_ALLOCATED_MEMORY
/* fill the allocated space with junk */
- randomize_mem((char *) GenerationChunkGetPointer(chunk), size);
+ randomize_mem((char *) MemoryChunkGetPointer(chunk), size);
#endif
/* Ensure any padding bytes are marked NOACCESS. */
- VALGRIND_MAKE_MEM_NOACCESS((char *) GenerationChunkGetPointer(chunk) +
size,
+ VALGRIND_MAKE_MEM_NOACCESS((char *) MemoryChunkGetPointer(chunk) + size,
chunk_size - size);
/* Disallow external access to private part of chunk header. */
VALGRIND_MAKE_MEM_NOACCESS(chunk, GENERATIONCHUNK_PRIVATE_LEN);
- return GenerationChunkGetPointer(chunk);
+ return MemoryChunkGetPointer(chunk);
}
/*
@@ -574,8 +509,10 @@ GenerationAlloc(MemoryContext context, Size size)
* mem_allocated field.
*/
static inline void
-GenerationBlockInit(GenerationBlock *block, Size blksize)
+GenerationBlockInit(GenerationContext *context, GenerationBlock *block,
+ Size blksize)
{
+ block->context = context;
block->blksize = blksize;
block->nchunks = 0;
block->nfree = 0;
@@ -661,33 +598,48 @@ GenerationBlockFree(GenerationContext *set,
GenerationBlock *block)
* Update number of chunks in the block, and if all chunks in the
block
* are now free then discard the block.
*/
-static void
-GenerationFree(MemoryContext context, void *pointer)
+void
+GenerationFree(void *pointer)
{
- GenerationContext *set = (GenerationContext *) context;
- GenerationChunk *chunk = GenerationPointerGetChunk(pointer);
+ MemoryChunk *chunk = PointerGetMemoryChunk(pointer);
GenerationBlock *block;
+ GenerationContext *set;
+#if defined(MEMORY_CONTEXT_CHECKING) || defined(CLOBBER_FREED_MEMORY)
+ Size chunksize;
+#endif
+
+ if (MemoryChunkIsExternal(chunk))
+ {
+ block = ExternalChunkGetBlock(chunk);
+#if defined(MEMORY_CONTEXT_CHECKING) || defined(CLOBBER_FREED_MEMORY)
+ chunksize = block->endptr - (char *) pointer;
+#endif
+ }
+ else
+ {
+ block = MemoryChunkGetBlock(chunk);
+#if defined(MEMORY_CONTEXT_CHECKING) || defined(CLOBBER_FREED_MEMORY)
+ chunksize = MemoryChunkGetSize(chunk);
+#endif
+ }
/* Allow access to private part of chunk header. */
VALGRIND_MAKE_MEM_DEFINED(chunk, GENERATIONCHUNK_PRIVATE_LEN);
- block = chunk->block;
+ set = block->context;
#ifdef MEMORY_CONTEXT_CHECKING
/* Test for someone scribbling on unused space in chunk */
- if (chunk->requested_size < chunk->size)
+ if (chunk->requested_size < chunksize)
if (!sentinel_ok(pointer, chunk->requested_size))
elog(WARNING, "detected write past chunk end in %s %p",
((MemoryContext) set)->name, chunk);
#endif
#ifdef CLOBBER_FREED_MEMORY
- wipe_mem(pointer, chunk->size);
+ wipe_mem(pointer, chunksize);
#endif
- /* Reset context to NULL in freed chunks */
- chunk->context = NULL;
-
#ifdef MEMORY_CONTEXT_CHECKING
/* Reset requested_size to 0 in freed chunks */
chunk->requested_size = 0;
@@ -732,7 +684,7 @@ GenerationFree(MemoryContext context, void *pointer)
*/
dlist_delete(&block->node);
- context->mem_allocated -= block->blksize;
+ set->header.mem_allocated -= block->blksize;
free(block);
}
@@ -742,18 +694,30 @@ GenerationFree(MemoryContext context, void *pointer)
* and discard the old one. The only exception is when the new
size fits
* into the old chunk - in that case we just update chunk header.
*/
-static void *
-GenerationRealloc(MemoryContext context, void *pointer, Size size)
+void *
+GenerationRealloc(void *pointer, Size size)
{
- GenerationContext *set = (GenerationContext *) context;
- GenerationChunk *chunk = GenerationPointerGetChunk(pointer);
+ MemoryChunk *chunk = PointerGetMemoryChunk(pointer);
+ GenerationContext *set;
+ GenerationBlock *block;
GenerationPointer newPointer;
Size oldsize;
/* Allow access to private part of chunk header. */
VALGRIND_MAKE_MEM_DEFINED(chunk, GENERATIONCHUNK_PRIVATE_LEN);
- oldsize = chunk->size;
+ if (MemoryChunkIsExternal(chunk))
+ {
+ block = ExternalChunkGetBlock(chunk);
+ oldsize = block->endptr - (char *) pointer;
+ }
+ else
+ {
+ block = MemoryChunkGetBlock(chunk);
+ oldsize = MemoryChunkGetSize(chunk);
+ }
+
+ set = block->context;
#ifdef MEMORY_CONTEXT_CHECKING
/* Test for someone scribbling on unused space in chunk */
@@ -848,7 +812,7 @@ GenerationRealloc(MemoryContext context, void *pointer,
Size size)
memcpy(newPointer, pointer, oldsize);
/* free old chunk */
- GenerationFree((MemoryContext) set, pointer);
+ GenerationFree(pointer);
return newPointer;
}
@@ -858,23 +822,51 @@ GenerationRealloc(MemoryContext context, void *pointer,
Size size)
* Given a currently-allocated chunk, determine the total space
* it occupies (including all memory-allocation overhead).
*/
-static Size
-GenerationGetChunkSpace(MemoryContext context, void *pointer)
+MemoryContext
+GenerationGetChunkContext(void *pointer)
{
- GenerationChunk *chunk = GenerationPointerGetChunk(pointer);
- Size result;
+ MemoryChunk *chunk = PointerGetMemoryChunk(pointer);
+ GenerationBlock *block;
- VALGRIND_MAKE_MEM_DEFINED(chunk, GENERATIONCHUNK_PRIVATE_LEN);
- result = chunk->size + Generation_CHUNKHDRSZ;
- VALGRIND_MAKE_MEM_NOACCESS(chunk, GENERATIONCHUNK_PRIVATE_LEN);
- return result;
+ if (MemoryChunkIsExternal(chunk))
+ block = ExternalChunkGetBlock(chunk);
+ else
+ block = (GenerationBlock *) MemoryChunkGetBlock(chunk);
+
+ return &block->context->header;
+}
+
+/*
+ * GenerationGetChunkSpace
+ * Given a currently-allocated chunk, determine the total space
+ * it occupies (including all memory-allocation overhead).
+ */
+Size
+GenerationGetChunkSpace(void *pointer)
+{
+ MemoryChunk *chunk = PointerGetMemoryChunk(pointer);
+ GenerationBlock *block;
+ Size chunksize;
+
+ if (MemoryChunkIsExternal(chunk))
+ {
+ block = ExternalChunkGetBlock(chunk);
+ chunksize = block->endptr - (char *) pointer;
+ }
+ else
+ {
+ block = MemoryChunkGetBlock(chunk);
+ chunksize = MemoryChunkGetSize(chunk);
+ }
+
+ return Generation_CHUNKHDRSZ + chunksize;
}
/*
* GenerationIsEmpty
* Is a GenerationContext empty of any allocated space?
*/
-static bool
+bool
GenerationIsEmpty(MemoryContext context)
{
GenerationContext *set = (GenerationContext *) context;
@@ -903,7 +895,7 @@ GenerationIsEmpty(MemoryContext context)
* XXX freespace only accounts for empty space at the end of the block, not
* space of freed chunks (which is unknown).
*/
-static void
+void
GenerationStats(MemoryContext context,
MemoryStatsPrintFunc printfunc, void *passthru,
MemoryContextCounters *totals, bool
print_to_stderr)
@@ -961,7 +953,7 @@ GenerationStats(MemoryContext context,
* find yourself in an infinite loop when trouble occurs, because this
* routine will be entered again when elog cleanup tries to release memory!
*/
-static void
+void
GenerationCheck(MemoryContext context)
{
GenerationContext *gen = (GenerationContext *) context;
@@ -976,6 +968,7 @@ GenerationCheck(MemoryContext context)
int nfree,
nchunks;
char *ptr;
+ bool has_external_chunk = false;
total_allocated += block->blksize;
@@ -987,6 +980,11 @@ GenerationCheck(MemoryContext context)
elog(WARNING, "problem in Generation %s: number of free
chunks %d in block %p exceeds %d allocated",
name, block->nfree, block, block->nchunks);
+ /* check block belongs to the correct context */
+ if (block->context != gen)
+ elog(WARNING, "problem in Generation %s: bogus context
link in block %p",
+ name, block);
+
/* Now walk through the chunks and count them. */
nfree = 0;
nchunks = 0;
@@ -994,42 +992,47 @@ GenerationCheck(MemoryContext context)
while (ptr < block->freeptr)
{
- GenerationChunk *chunk = (GenerationChunk *) ptr;
+ MemoryChunk *chunk = (MemoryChunk *) ptr;
+ GenerationBlock *chunkblock;
+ Size chunksize;
/* Allow access to private part of chunk header. */
VALGRIND_MAKE_MEM_DEFINED(chunk,
GENERATIONCHUNK_PRIVATE_LEN);
+ if (MemoryChunkIsExternal(chunk))
+ {
+ chunkblock = ExternalChunkGetBlock(chunk);
+ chunksize = block->endptr - (char *)
MemoryChunkGetPointer(chunk);
+ has_external_chunk = true;
+ }
+ else
+ {
+ chunkblock = MemoryChunkGetBlock(chunk);
+ chunksize = MemoryChunkGetSize(chunk);
+ }
+
/* move to the next chunk */
- ptr += (chunk->size + Generation_CHUNKHDRSZ);
+ ptr += (chunksize + Generation_CHUNKHDRSZ);
nchunks += 1;
/* chunks have both block and context pointers, so
check both */
- if (chunk->block != block)
+ if (chunkblock != block)
elog(WARNING, "problem in Generation %s: bogus
block link in block %p, chunk %p",
name, block, chunk);
- /*
- * Check for valid context pointer. Note this is an
incomplete
- * test, since palloc(0) produces an allocated chunk
with
- * requested_size == 0.
- */
- if ((chunk->requested_size > 0 && chunk->context !=
gen) ||
- (chunk->context != gen && chunk->context !=
NULL))
- elog(WARNING, "problem in Generation %s: bogus
context link in block %p, chunk %p",
- name, block, chunk);
/* now make sure the chunk size is correct */
- if (chunk->size < chunk->requested_size ||
- chunk->size != MAXALIGN(chunk->size))
+ if (chunksize < chunk->requested_size ||
+ chunksize != MAXALIGN(chunksize))
elog(WARNING, "problem in Generation %s: bogus
chunk size in block %p, chunk %p",
name, block, chunk);
/* is chunk allocated? */
- if (chunk->context != NULL)
+ if (chunk->requested_size > 0)
{
- /* check sentinel, but only in allocated blocks
*/
- if (chunk->requested_size < chunk->size &&
+ /* check sentinel */
+ if (chunk->requested_size < chunksize &&
!sentinel_ok(chunk,
Generation_CHUNKHDRSZ + chunk->requested_size))
elog(WARNING, "problem in Generation
%s: detected write past chunk end in block %p, chunk %p",
name, block, chunk);
@@ -1041,7 +1044,7 @@ GenerationCheck(MemoryContext context)
* If chunk is allocated, disallow external access to
private part
* of chunk header.
*/
- if (chunk->context != NULL)
+ if (chunk->requested_size > 0)
VALGRIND_MAKE_MEM_NOACCESS(chunk,
GENERATIONCHUNK_PRIVATE_LEN);
}
@@ -1056,6 +1059,11 @@ GenerationCheck(MemoryContext context)
if (nfree != block->nfree)
elog(WARNING, "problem in Generation %s: number of free
chunks %d in block %p does not match header %d",
name, nfree, block, block->nfree);
+
+ if (has_external_chunk && nchunks > 1)
+ elog(WARNING, "problem in Generation %s: external chunk
on non-dedicated block %p",
+ name, block);
+
}
Assert(total_allocated == context->mem_allocated);
diff --git a/src/backend/utils/mmgr/mcxt.c b/src/backend/utils/mmgr/mcxt.c
index e12be1b9bd..7f9514c450 100644
--- a/src/backend/utils/mmgr/mcxt.c
+++ b/src/backend/utils/mmgr/mcxt.c
@@ -29,12 +29,60 @@
#include "utils/fmgrprotos.h"
#include "utils/memdebug.h"
#include "utils/memutils.h"
+#include "utils/memutils_internal.h"
/*****************************************************************************
* GLOBAL MEMORY
*
*****************************************************************************/
+static const MemoryContextMethods mcxt_methods[] = {
+ [MCTX_ASET_ID] = {
+ AllocSetAlloc,
+ AllocSetFree,
+ AllocSetRealloc,
+ AllocSetReset,
+ AllocSetDelete,
+ AllocSetGetChunkContext,
+ AllocSetGetChunkSpace,
+ AllocSetIsEmpty,
+ AllocSetStats
+#ifdef MEMORY_CONTEXT_CHECKING
+ ,AllocSetCheck
+#endif
+ },
+
+ [MCTX_GENERATION_ID] = {
+ GenerationAlloc,
+ GenerationFree,
+ GenerationRealloc,
+ GenerationReset,
+ GenerationDelete,
+ GenerationGetChunkContext,
+ GenerationGetChunkSpace,
+ GenerationIsEmpty,
+ GenerationStats
+#ifdef MEMORY_CONTEXT_CHECKING
+ ,GenerationCheck
+#endif
+ },
+
+ [MCTX_SLAB_ID] = {
+ SlabAlloc,
+ SlabFree,
+ SlabRealloc,
+ SlabReset,
+ SlabDelete,
+ SlabGetChunkContext,
+ SlabGetChunkSpace,
+ SlabIsEmpty,
+ SlabStats
+#ifdef MEMORY_CONTEXT_CHECKING
+ ,SlabCheck
+#endif
+ },
+};
+
/*
* CurrentMemoryContext
* Default memory context for allocations.
@@ -73,6 +121,13 @@ static void MemoryContextStatsPrint(MemoryContext context,
void *passthru,
#define AssertNotInCriticalSection(context) \
Assert(CritSectionCount == 0 || (context)->allowInCritSection)
+/*
+ * Call the given function in the MemoryContextMethods for the memory context
+ * type that 'pointer' belongs to.
+ */
+#define MCXT_METHOD(pointer, method) \
+ mcxt_methods[GetMemoryChunkMethodID(pointer)].method
+
/*****************************************************************************
* EXPORTED ROUTINES
*
@@ -422,6 +477,17 @@ MemoryContextAllowInCriticalSection(MemoryContext context,
bool allow)
context->allowInCritSection = allow;
}
+/*
+ * GetMemoryChunkContext
+ * Given a currently-allocated chunk, determine the MemoryContext
that
+ * the chunk belongs to.
+ */
+MemoryContext
+GetMemoryChunkContext(void *pointer)
+{
+ return MCXT_METHOD(pointer, get_chunk_context) (pointer);
+}
+
/*
* GetMemoryChunkSpace
* Given a currently-allocated chunk, determine the total space
@@ -433,9 +499,7 @@ MemoryContextAllowInCriticalSection(MemoryContext context,
bool allow)
Size
GetMemoryChunkSpace(void *pointer)
{
- MemoryContext context = GetMemoryChunkContext(pointer);
-
- return context->methods->get_chunk_space(context, pointer);
+ return MCXT_METHOD(pointer, get_chunk_space) (pointer);
}
/*
@@ -814,7 +878,7 @@ MemoryContextContains(MemoryContext context, void *pointer)
void
MemoryContextCreate(MemoryContext node,
NodeTag tag,
- const MemoryContextMethods *methods,
+ MemoryContextMethodID method_id,
MemoryContext parent,
const char *name)
{
@@ -824,7 +888,7 @@ MemoryContextCreate(MemoryContext node,
/* Initialize all standard fields of memory context header */
node->type = tag;
node->isReset = true;
- node->methods = methods;
+ node->methods = &mcxt_methods[method_id];
node->parent = parent;
node->firstchild = NULL;
node->mem_allocated = 0;
@@ -1174,9 +1238,11 @@ palloc_extended(Size size, int flags)
void
pfree(void *pointer)
{
+#ifdef USE_VALGRIND
MemoryContext context = GetMemoryChunkContext(pointer);
+#endif
- context->methods->free_p(context, pointer);
+ MCXT_METHOD(pointer, free_p) (pointer);
VALGRIND_MEMPOOL_FREE(context, pointer);
}
@@ -1187,7 +1253,9 @@ pfree(void *pointer)
void *
repalloc(void *pointer, Size size)
{
+#if defined(USE_ASSERT_CHECKING) || defined(USE_VALGRIND)
MemoryContext context = GetMemoryChunkContext(pointer);
+#endif
void *ret;
if (!AllocSizeIsValid(size))
@@ -1198,9 +1266,11 @@ repalloc(void *pointer, Size size)
/* isReset must be false already */
Assert(!context->isReset);
- ret = context->methods->realloc(context, pointer, size);
+ ret = MCXT_METHOD(pointer, realloc) (pointer, size);
if (unlikely(ret == NULL))
{
+ MemoryContext context = GetMemoryChunkContext(pointer);
+
MemoryContextStats(TopMemoryContext);
ereport(ERROR,
(errcode(ERRCODE_OUT_OF_MEMORY),
@@ -1257,7 +1327,9 @@ MemoryContextAllocHuge(MemoryContext context, Size size)
void *
repalloc_huge(void *pointer, Size size)
{
+#if defined(USE_ASSERT_CHECKING) || defined(USE_VALGRIND)
MemoryContext context = GetMemoryChunkContext(pointer);
+#endif
void *ret;
if (!AllocHugeSizeIsValid(size))
@@ -1268,9 +1340,11 @@ repalloc_huge(void *pointer, Size size)
/* isReset must be false already */
Assert(!context->isReset);
- ret = context->methods->realloc(context, pointer, size);
+ ret = MCXT_METHOD(pointer, realloc) (pointer, size);
if (unlikely(ret == NULL))
{
+ MemoryContext context = GetMemoryChunkContext(pointer);
+
MemoryContextStats(TopMemoryContext);
ereport(ERROR,
(errcode(ERRCODE_OUT_OF_MEMORY),
diff --git a/src/backend/utils/mmgr/slab.c b/src/backend/utils/mmgr/slab.c
index 67d97b22e5..77f9e12049 100644
--- a/src/backend/utils/mmgr/slab.c
+++ b/src/backend/utils/mmgr/slab.c
@@ -55,6 +55,8 @@
#include "lib/ilist.h"
#include "utils/memdebug.h"
#include "utils/memutils.h"
+#include "utils/memutils_memorychunk.h"
+#include "utils/memutils_internal.h"
/*
* SlabContext is a specialized implementation of MemoryContext.
@@ -90,75 +92,23 @@ typedef struct SlabBlock
dlist_node node; /* doubly-linked list */
int nfree; /* number of free
chunks */
int firstFreeChunk; /* index of the first free
chunk in the block */
-} SlabBlock;
-
-/*
- * SlabChunk
- * The prefix of each piece of memory in a SlabBlock
- *
- * Note: to meet the memory context APIs, the payload area of the chunk must
- * be maxaligned, and the "slab" link must be immediately adjacent to the
- * payload area (cf. GetMemoryChunkContext). Since we support no machines on
- * which MAXALIGN is more than twice sizeof(void *), this happens without any
- * special hacking in this struct declaration. But there is a static
- * assertion below that the alignment is done correctly.
- */
-typedef struct SlabChunk
-{
- SlabBlock *block; /* block owning this chunk */
SlabContext *slab; /* owning context */
- /* there must not be any padding to reach a MAXALIGN boundary here! */
-} SlabChunk;
+} SlabBlock;
+#define Slab_CHUNKHDRSZ sizeof(MemoryChunk)
#define SlabPointerGetChunk(ptr) \
- ((SlabChunk *)(((char *)(ptr)) - sizeof(SlabChunk)))
+ ((MemoryChunk *)(((char *)(ptr)) - sizeof(MemoryChunk)))
#define SlabChunkGetPointer(chk) \
- ((void *)(((char *)(chk)) + sizeof(SlabChunk)))
+ ((void *)(((char *)(chk)) + sizeof(MemoryChunk)))
#define SlabBlockGetChunk(slab, block, idx) \
- ((SlabChunk *) ((char *) (block) + sizeof(SlabBlock) \
+ ((MemoryChunk *) ((char *) (block) + sizeof(SlabBlock) \
+ (idx * slab->fullChunkSize)))
#define SlabBlockStart(block) \
((char *) block + sizeof(SlabBlock))
#define SlabChunkIndex(slab, block, chunk) \
(((char *) chunk - SlabBlockStart(block)) / slab->fullChunkSize)
-/*
- * These functions implement the MemoryContext API for Slab contexts.
- */
-static void *SlabAlloc(MemoryContext context, Size size);
-static void SlabFree(MemoryContext context, void *pointer);
-static void *SlabRealloc(MemoryContext context, void *pointer, Size size);
-static void SlabReset(MemoryContext context);
-static void SlabDelete(MemoryContext context);
-static Size SlabGetChunkSpace(MemoryContext context, void *pointer);
-static bool SlabIsEmpty(MemoryContext context);
-static void SlabStats(MemoryContext context,
- MemoryStatsPrintFunc printfunc, void
*passthru,
- MemoryContextCounters *totals,
- bool print_to_stderr);
-#ifdef MEMORY_CONTEXT_CHECKING
-static void SlabCheck(MemoryContext context);
-#endif
-
-/*
- * This is the virtual function table for Slab contexts.
- */
-static const MemoryContextMethods SlabMethods = {
- SlabAlloc,
- SlabFree,
- SlabRealloc,
- SlabReset,
- SlabDelete,
- SlabGetChunkSpace,
- SlabIsEmpty,
- SlabStats
-#ifdef MEMORY_CONTEXT_CHECKING
- ,SlabCheck
-#endif
-};
-
-
/*
* SlabContextCreate
* Create a new Slab context.
@@ -168,8 +118,7 @@ static const MemoryContextMethods SlabMethods = {
* blockSize: allocation block size
* chunkSize: allocation chunk size
*
- * The chunkSize may not exceed:
- * MAXALIGN_DOWN(SIZE_MAX) - MAXALIGN(sizeof(SlabBlock)) -
sizeof(SlabChunk)
+ * The chunkSize may not exceed MEMORYCHUNK_MAX_SIZE
*/
MemoryContext
SlabContextCreate(MemoryContext parent,
@@ -184,19 +133,14 @@ SlabContextCreate(MemoryContext parent,
SlabContext *slab;
int i;
- /* Assert we padded SlabChunk properly */
- StaticAssertStmt(sizeof(SlabChunk) == MAXALIGN(sizeof(SlabChunk)),
- "sizeof(SlabChunk) is not maxaligned");
- StaticAssertStmt(offsetof(SlabChunk, slab) + sizeof(MemoryContext) ==
- sizeof(SlabChunk),
- "padding calculation in SlabChunk is
wrong");
+ Assert(chunkSize <= MEMORYCHUNK_MAX_SIZE);
/* Make sure the linked list node fits inside a freed chunk */
if (chunkSize < sizeof(int))
chunkSize = sizeof(int);
/* chunk, including SLAB header (both addresses nicely aligned) */
- fullChunkSize = sizeof(SlabChunk) + MAXALIGN(chunkSize);
+ fullChunkSize = Slab_CHUNKHDRSZ + MAXALIGN(chunkSize);
/* Make sure the block can store at least one chunk. */
if (blockSize < fullChunkSize + sizeof(SlabBlock))
@@ -265,7 +209,7 @@ SlabContextCreate(MemoryContext parent,
/* Finally, do the type-independent part of context creation */
MemoryContextCreate((MemoryContext) slab,
T_SlabContext,
- &SlabMethods,
+ MCTX_SLAB_ID,
parent,
name);
@@ -279,7 +223,7 @@ SlabContextCreate(MemoryContext parent,
* The code simply frees all the blocks in the context - we don't keep any
* keeper blocks or anything like that.
*/
-static void
+void
SlabReset(MemoryContext context)
{
int i;
@@ -322,7 +266,7 @@ SlabReset(MemoryContext context)
* SlabDelete
* Free all memory which is allocated in the given context.
*/
-static void
+void
SlabDelete(MemoryContext context)
{
/* Reset to release all the SlabBlocks */
@@ -336,12 +280,12 @@ SlabDelete(MemoryContext context)
* Returns pointer to allocated memory of given size or NULL if
* request could not be completed; memory is added to the slab.
*/
-static void *
+void *
SlabAlloc(MemoryContext context, Size size)
{
SlabContext *slab = castNode(SlabContext, context);
SlabBlock *block;
- SlabChunk *chunk;
+ MemoryChunk *chunk;
int idx;
Assert(slab);
@@ -370,6 +314,7 @@ SlabAlloc(MemoryContext context, Size size)
block->nfree = slab->chunksPerBlock;
block->firstFreeChunk = 0;
+ block->slab = slab;
/*
* Put all the chunks on a freelist. Walk the chunks and point
each
@@ -378,7 +323,7 @@ SlabAlloc(MemoryContext context, Size size)
for (idx = 0; idx < slab->chunksPerBlock; idx++)
{
chunk = SlabBlockGetChunk(slab, block, idx);
- *(int32 *) SlabChunkGetPointer(chunk) = (idx + 1);
+ *(int32 *) MemoryChunkGetPointer(chunk) = (idx + 1);
}
/*
@@ -426,8 +371,8 @@ SlabAlloc(MemoryContext context, Size size)
* Remove the chunk from the freelist head. The index of the next free
* chunk is stored in the chunk itself.
*/
- VALGRIND_MAKE_MEM_DEFINED(SlabChunkGetPointer(chunk), sizeof(int32));
- block->firstFreeChunk = *(int32 *) SlabChunkGetPointer(chunk);
+ VALGRIND_MAKE_MEM_DEFINED(MemoryChunkGetPointer(chunk), sizeof(int32));
+ block->firstFreeChunk = *(int32 *) MemoryChunkGetPointer(chunk);
Assert(block->firstFreeChunk >= 0);
Assert(block->firstFreeChunk <= slab->chunksPerBlock);
@@ -464,47 +409,47 @@ SlabAlloc(MemoryContext context, Size size)
slab->minFreeChunks = 0;
/* Prepare to initialize the chunk header. */
- VALGRIND_MAKE_MEM_UNDEFINED(chunk, sizeof(SlabChunk));
-
- chunk->block = block;
- chunk->slab = slab;
+ VALGRIND_MAKE_MEM_UNDEFINED(chunk, Slab_CHUNKHDRSZ);
+ MemoryChunkSetHdrMask(chunk, block, MAXALIGN(slab->chunkSize),
+ MCTX_SLAB_ID);
#ifdef MEMORY_CONTEXT_CHECKING
/* slab mark to catch clobber of "unused" space */
- if (slab->chunkSize < (slab->fullChunkSize - sizeof(SlabChunk)))
+ if (slab->chunkSize < (slab->fullChunkSize - Slab_CHUNKHDRSZ))
{
- set_sentinel(SlabChunkGetPointer(chunk), size);
+ set_sentinel(MemoryChunkGetPointer(chunk), size);
VALGRIND_MAKE_MEM_NOACCESS(((char *) chunk) +
-
sizeof(SlabChunk) + slab->chunkSize,
+
Slab_CHUNKHDRSZ + slab->chunkSize,
slab->fullChunkSize -
-
(slab->chunkSize + sizeof(SlabChunk)));
+
(slab->chunkSize + Slab_CHUNKHDRSZ));
}
#endif
+
#ifdef RANDOMIZE_ALLOCATED_MEMORY
/* fill the allocated space with junk */
- randomize_mem((char *) SlabChunkGetPointer(chunk), size);
+ randomize_mem((char *) MemoryChunkGetPointer(chunk), size);
#endif
Assert(slab->nblocks * slab->blockSize == context->mem_allocated);
- return SlabChunkGetPointer(chunk);
+ return MemoryChunkGetPointer(chunk);
}
/*
* SlabFree
* Frees allocated memory; memory is removed from the slab.
*/
-static void
-SlabFree(MemoryContext context, void *pointer)
+void
+SlabFree(void *pointer)
{
int idx;
- SlabContext *slab = castNode(SlabContext, context);
- SlabChunk *chunk = SlabPointerGetChunk(pointer);
- SlabBlock *block = chunk->block;
+ MemoryChunk *chunk = PointerGetMemoryChunk(pointer);
+ SlabBlock *block = MemoryChunkGetBlock(chunk);
+ SlabContext *slab = block->slab;
#ifdef MEMORY_CONTEXT_CHECKING
/* Test for someone scribbling on unused space in chunk */
- if (slab->chunkSize < (slab->fullChunkSize - sizeof(SlabChunk)))
+ if (slab->chunkSize < (slab->fullChunkSize - Slab_CHUNKHDRSZ))
if (!sentinel_ok(pointer, slab->chunkSize))
elog(WARNING, "detected write past chunk end in %s %p",
slab->header.name, chunk);
@@ -560,13 +505,13 @@ SlabFree(MemoryContext context, void *pointer)
{
free(block);
slab->nblocks--;
- context->mem_allocated -= slab->blockSize;
+ slab->header.mem_allocated -= slab->blockSize;
}
else
dlist_push_head(&slab->freelist[block->nfree], &block->node);
Assert(slab->nblocks >= 0);
- Assert(slab->nblocks * slab->blockSize == context->mem_allocated);
+ Assert(slab->nblocks * slab->blockSize == slab->header.mem_allocated);
}
/*
@@ -582,13 +527,14 @@ SlabFree(MemoryContext context, void *pointer)
* rather pointless - Slab is meant for chunks of constant size, and moreover
* realloc is usually used to enlarge the chunk.
*/
-static void *
-SlabRealloc(MemoryContext context, void *pointer, Size size)
+void *
+SlabRealloc(void *pointer, Size size)
{
- SlabContext *slab = castNode(SlabContext, context);
+ MemoryChunk *chunk = PointerGetMemoryChunk(pointer);
+ SlabBlock *block = MemoryChunkGetBlock(chunk);
+ SlabContext *slab = block->slab;
Assert(slab);
-
/* can't do actual realloc with slab, but let's try to be gentle */
if (size == slab->chunkSize)
return pointer;
@@ -597,15 +543,33 @@ SlabRealloc(MemoryContext context, void *pointer, Size
size)
return NULL; /* keep compiler quiet */
}
+/*
+ * SlabGetChunkContext
+ * Return the MemoryContext that 'pointer' belongs to.
+ */
+MemoryContext
+SlabGetChunkContext(void *pointer)
+{
+ MemoryChunk *chunk = PointerGetMemoryChunk(pointer);
+ SlabBlock *block = MemoryChunkGetBlock(chunk);
+ SlabContext *slab = block->slab;
+
+ Assert(slab != NULL);
+
+ return &slab->header;
+}
+
/*
* SlabGetChunkSpace
* Given a currently-allocated chunk, determine the total space
* it occupies (including all memory-allocation overhead).
*/
-static Size
-SlabGetChunkSpace(MemoryContext context, void *pointer)
+Size
+SlabGetChunkSpace(void *pointer)
{
- SlabContext *slab = castNode(SlabContext, context);
+ MemoryChunk *chunk = PointerGetMemoryChunk(pointer);
+ SlabBlock *block = MemoryChunkGetBlock(chunk);
+ SlabContext *slab = block->slab;
Assert(slab);
@@ -616,7 +580,7 @@ SlabGetChunkSpace(MemoryContext context, void *pointer)
* SlabIsEmpty
* Is an Slab empty of any allocated space?
*/
-static bool
+bool
SlabIsEmpty(MemoryContext context)
{
SlabContext *slab = castNode(SlabContext, context);
@@ -635,7 +599,7 @@ SlabIsEmpty(MemoryContext context)
* totals: if not NULL, add stats about this context into *totals.
* print_to_stderr: print stats to stderr if true, elog otherwise.
*/
-static void
+void
SlabStats(MemoryContext context,
MemoryStatsPrintFunc printfunc, void *passthru,
MemoryContextCounters *totals,
@@ -697,7 +661,7 @@ SlabStats(MemoryContext context,
* find yourself in an infinite loop when trouble occurs, because this
* routine will be entered again when elog cleanup tries to release memory!
*/
-static void
+void
SlabCheck(MemoryContext context)
{
int i;
@@ -742,7 +706,7 @@ SlabCheck(MemoryContext context)
nfree = 0;
while (idx < slab->chunksPerBlock)
{
- SlabChunk *chunk;
+ MemoryChunk *chunk;
/* count the chunk as free, add it to the
bitmap */
nfree++;
@@ -750,8 +714,8 @@ SlabCheck(MemoryContext context)
/* read index of the next free chunk */
chunk = SlabBlockGetChunk(slab, block, idx);
-
VALGRIND_MAKE_MEM_DEFINED(SlabChunkGetPointer(chunk), sizeof(int32));
- idx = *(int32 *) SlabChunkGetPointer(chunk);
+
VALGRIND_MAKE_MEM_DEFINED(MemoryChunkGetPointer(chunk), sizeof(int32));
+ idx = *(int32 *) MemoryChunkGetPointer(chunk);
}
for (j = 0; j < slab->chunksPerBlock; j++)
@@ -759,19 +723,23 @@ SlabCheck(MemoryContext context)
/* non-zero bit in the bitmap means chunk the
chunk is used */
if (!slab->freechunks[j])
{
- SlabChunk *chunk =
SlabBlockGetChunk(slab, block, j);
-
- /* chunks have both block and slab
pointers, so check both */
- if (chunk->block != block)
+ MemoryChunk *chunk =
SlabBlockGetChunk(slab, block, j);
+ SlabBlock *chunkblock = (SlabBlock *)
MemoryChunkGetBlock(chunk);
+
+ /*
+ * check the chunk's blockoffset
correctly points back to
+ * the block
+ */
+ if (chunkblock != block)
elog(WARNING, "problem in slab
%s: bogus block link in block %p, chunk %p",
name, block, chunk);
- if (chunk->slab != slab)
+ if (block->slab != slab)
elog(WARNING, "problem in slab
%s: bogus slab link in block %p, chunk %p",
name, block, chunk);
/* there might be sentinel (thanks to
alignment) */
- if (slab->chunkSize <
(slab->fullChunkSize - sizeof(SlabChunk)))
+ if (slab->chunkSize <
(slab->fullChunkSize - Slab_CHUNKHDRSZ))
if (!sentinel_ok(chunk,
slab->chunkSize))
elog(WARNING, "problem
in slab %s: detected write past chunk end in block %p, chunk %p",
name, block,
chunk);
diff --git a/src/include/nodes/memnodes.h b/src/include/nodes/memnodes.h
index 16cd56da6d..63d07358cd 100644
--- a/src/include/nodes/memnodes.h
+++ b/src/include/nodes/memnodes.h
@@ -59,11 +59,12 @@ typedef struct MemoryContextMethods
{
void *(*alloc) (MemoryContext context, Size size);
/* call this free_p in case someone #define's free() */
- void (*free_p) (MemoryContext context, void *pointer);
- void *(*realloc) (MemoryContext context, void *pointer, Size
size);
+ void (*free_p) (void *pointer);
+ void *(*realloc) (void *pointer, Size size);
void (*reset) (MemoryContext context);
void (*delete_context) (MemoryContext context);
- Size (*get_chunk_space) (MemoryContext context, void
*pointer);
+ MemoryContext (*get_chunk_context) (void *pointer);
+ Size (*get_chunk_space) (void *pointer);
bool (*is_empty) (MemoryContext context);
void (*stats) (MemoryContext context,
MemoryStatsPrintFunc
printfunc, void *passthru,
diff --git a/src/include/utils/memutils.h b/src/include/utils/memutils.h
index 495d1af201..244f7bf4ff 100644
--- a/src/include/utils/memutils.h
+++ b/src/include/utils/memutils.h
@@ -79,6 +79,7 @@ extern void MemoryContextDeleteChildren(MemoryContext
context);
extern void MemoryContextSetIdentifier(MemoryContext context, const char *id);
extern void MemoryContextSetParent(MemoryContext context,
MemoryContext new_parent);
+extern MemoryContext GetMemoryChunkContext(void *pointer);
extern Size GetMemoryChunkSpace(void *pointer);
extern MemoryContext MemoryContextGetParent(MemoryContext context);
extern bool MemoryContextIsEmpty(MemoryContext context);
@@ -98,53 +99,6 @@ extern bool MemoryContextContains(MemoryContext context,
void *pointer);
#define MemoryContextCopyAndSetIdentifier(cxt, id) \
MemoryContextSetIdentifier(cxt, MemoryContextStrdup(cxt, id))
-/*
- * GetMemoryChunkContext
- * Given a currently-allocated chunk, determine the context
- * it belongs to.
- *
- * All chunks allocated by any memory context manager are required to be
- * preceded by the corresponding MemoryContext stored, without padding, in the
- * preceding sizeof(void*) bytes. A currently-allocated chunk must contain a
- * backpointer to its owning context. The backpointer is used by pfree() and
- * repalloc() to find the context to call.
- */
-#ifndef FRONTEND
-static inline MemoryContext
-GetMemoryChunkContext(void *pointer)
-{
- MemoryContext context;
-
- /*
- * Try to detect bogus pointers handed to us, poorly though we can.
- * Presumably, a pointer that isn't MAXALIGNED isn't pointing at an
- * allocated chunk.
- */
- Assert(pointer != NULL);
- Assert(pointer == (void *) MAXALIGN(pointer));
-
- /*
- * OK, it's probably safe to look at the context.
- */
- context = *(MemoryContext *) (((char *) pointer) - sizeof(void *));
-
- AssertArg(MemoryContextIsValid(context));
-
- return context;
-}
-#endif
-
-/*
- * This routine handles the context-type-independent part of memory
- * context creation. It's intended to be called from context-type-
- * specific creation routines, and noplace else.
- */
-extern void MemoryContextCreate(MemoryContext node,
- NodeTag tag,
- const
MemoryContextMethods *methods,
- MemoryContext
parent,
- const char
*name);
-
extern void HandleLogMemoryContextInterrupt(void);
extern void ProcessLogMemoryContextInterrupt(void);
diff --git a/src/include/utils/memutils_internal.h
b/src/include/utils/memutils_internal.h
new file mode 100644
index 0000000000..2dcfdd7ec3
--- /dev/null
+++ b/src/include/utils/memutils_internal.h
@@ -0,0 +1,117 @@
+/*-------------------------------------------------------------------------
+ *
+ * memutils_internal.h
+ * This file contains declarations for memory allocation utility
+ * functions for internal use.
+ *
+ *
+ * Portions Copyright (c) 2022, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/utils/memutils_internal.h
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#ifndef MEMUTILS_INTERNAL_H
+#define MEMUTILS_INTERNAL_H
+
+#include "utils/memutils.h"
+
+extern void *AllocSetAlloc(MemoryContext context, Size size);
+extern void AllocSetFree(void *pointer);
+extern void *AllocSetRealloc(void *pointer, Size size);
+extern void AllocSetReset(MemoryContext context);
+extern void AllocSetDelete(MemoryContext context);
+extern MemoryContext AllocSetGetChunkContext(void *pointer);
+extern Size AllocSetGetChunkSpace(void *pointer);
+extern bool AllocSetIsEmpty(MemoryContext context);
+extern void AllocSetStats(MemoryContext context,
+ MemoryStatsPrintFunc
printfunc, void *passthru,
+ MemoryContextCounters *totals,
+ bool print_to_stderr);
+
+#ifdef MEMORY_CONTEXT_CHECKING
+extern void AllocSetCheck(MemoryContext context);
+#endif
+
+
+extern void *GenerationAlloc(MemoryContext context, Size size);
+extern void GenerationFree(void *pointer);
+extern void *GenerationRealloc(void *pointer, Size size);
+extern void GenerationReset(MemoryContext context);
+extern void GenerationDelete(MemoryContext context);
+extern MemoryContext GenerationGetChunkContext(void *pointer);
+extern Size GenerationGetChunkSpace(void *pointer);
+extern bool GenerationIsEmpty(MemoryContext context);
+extern void GenerationStats(MemoryContext context,
+ MemoryStatsPrintFunc
printfunc, void *passthru,
+ MemoryContextCounters
*totals,
+ bool print_to_stderr);
+
+#ifdef MEMORY_CONTEXT_CHECKING
+extern void GenerationCheck(MemoryContext context);
+#endif
+
+
+/*
+ * These functions implement the MemoryContext API for Slab contexts.
+ */
+extern void *SlabAlloc(MemoryContext context, Size size);
+extern void SlabFree(void *pointer);
+extern void *SlabRealloc(void *pointer, Size size);
+extern void SlabReset(MemoryContext context);
+extern void SlabDelete(MemoryContext context);
+extern MemoryContext SlabGetChunkContext(void *pointer);
+extern Size SlabGetChunkSpace(void *pointer);
+extern bool SlabIsEmpty(MemoryContext context);
+extern void SlabStats(MemoryContext context,
+ MemoryStatsPrintFunc printfunc, void
*passthru,
+ MemoryContextCounters *totals,
+ bool print_to_stderr);
+#ifdef MEMORY_CONTEXT_CHECKING
+extern void SlabCheck(MemoryContext context);
+#endif
+
+/*
+ * MemoryContextMethodID
+ * A unique identifier for each MemoryContext implementation which
+ * indicates the index into the mcxt_methods[] array. See mcxt.c.
+ */
+typedef enum MemoryContextMethodID
+{
+ MCTX_ASET_ID = 0,
+ MCTX_GENERATION_ID,
+ MCTX_SLAB_ID,
+} MemoryContextMethodID;
+
+/*
+ * This routine handles the context-type-independent part of memory
+ * context creation. It's intended to be called from context-type-
+ * specific creation routines, and noplace else.
+ */
+extern void MemoryContextCreate(MemoryContext node,
+ NodeTag tag,
+
MemoryContextMethodID method_id,
+ MemoryContext
parent,
+ const char
*name);
+
+static inline MemoryContextMethodID
+GetMemoryChunkMethodID(void *pointer)
+{
+ uint64 header;
+
+ /*
+ * Try to detect bogus pointers handed to us, poorly though we can.
+ * Presumably, a pointer that isn't MAXALIGNED isn't pointing at an
+ * allocated chunk.
+ */
+ Assert(pointer != NULL);
+ Assert(pointer == (void *) MAXALIGN(pointer));
+
+ header = *((uint64 *) ((char *) pointer - sizeof(uint64)));
+
+ return header & 7;
+}
+
+#endif /* MEMUTILS_INTERNAL_H
*/
diff --git a/src/include/utils/memutils_memorychunk.h
b/src/include/utils/memutils_memorychunk.h
new file mode 100644
index 0000000000..6239cf9008
--- /dev/null
+++ b/src/include/utils/memutils_memorychunk.h
@@ -0,0 +1,185 @@
+/*-------------------------------------------------------------------------
+ *
+ * memutils_memorychunk.h
+ * Here we define a struct named MemoryChunk which implementations of
+ * MemoryContexts may use as a header for chunks of memory they allocate.
+ *
+ * A MemoryChunk provides a lightweight header which a MemoryContext can use
+ * to store the size of an allocation and a reference back to the block which
+ * the given chunk is allocated on.
+ *
+ * Although MemoryChunks are used by each of our MemoryContexts, other
+ * implementations may choose to implement their own method for storing chunk
+ * headers. The only requirement is that the header end with an 8-byte value
+ * which the least significant 3-bits of are set to the MemoryContextMethodID
+ * of the given context.
+ *
+ * By default, a MemoryChunk is 8 bytes in size, however when
+ * MEMORY_CONTEXT_CHECKING is defined the header becomes 16 bytes in size due
+ * to the additional requested_size field. The MemoryContext may use this
+ * field for whatever they wish, but it is intended to be used for additional
+ * checks which are only done in MEMORY_CONTEXT_CHECKING builds.
+ *
+ * The MemoryChunk contains a uint64 field named 'hdrmask'. This field is
+ * used to encode 4 separate pieces of information. Starting with the least
+ * significant bits of 'hdrmask', the bits of this field as used as follows:
+ *
+ * 1. 3-bits to indicate the MemoryContextMethodID
+ * 2. 1-bit to indicate if the chunk is externally managed (see below)
+ * 3. 30-bits for the amount of memory which was reserved for the chunk
+ * 4. 30-bits for the number of bytes that must be subtracted from the chunk
+ * to obtain the address of the block that the chunk is stored on.
+ *
+ * Because we're limited to a block offset and chunk size of 1GB (30-bits),
+ * any allocation which exceeds this amount must call MemoryChunkSetExternal()
+ * and the MemoryContext must devise its own method for storing the offset for
+ * the block and size of the chunk.
+ *
+ * Interface:
+ *
+ * MemoryChunkSetHdrMask:
+ * Used to set up a non-external MemoryChunk.
+ *
+ * MemoryChunkSetHdrMaskExternal:
+ * Used to set up an externally managed MemoryChunk.
+ *
+ * MemoryChunkIsExternal:
+ * Determine if the given MemoryChunk is externally managed, i.e.
+ * MemoryChunkSetHdrMaskExternal() was called on the chunk.
+ *
+ * MemoryChunkGetSize:
+ * For non-external chunks, return the size of the chunk as it was
set
+ * in the call to MemoryChunkSetHdrMask.
+ *
+ * MemoryChunkGetBlock:
+ * For non-external chunks, return a pointer to the block as it
was set
+ * in the call to MemoryChunkSetHdrMask.
+ *
+ * Also exports:
+ * MEMORYCHUNK_MAX_SIZE
+ * PointerGetMemoryChunk
+ * MemoryChunkGetPointer
+ *
+ * Portions Copyright (c) 2022, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/utils/memutils_memorychunk.h
+ *
+ *-------------------------------------------------------------------------
+ */
+
+#ifndef MEMUTILS_MEMORYCHUNK_H
+#define MEMUTILS_MEMORYCHUNK_H
+
+#include "utils/memutils_internal.h"
+
+ /*
+ * The maximum size for a memory chunk before it must be externally managed.
+ */
+#define MEMORYCHUNK_MAX_SIZE 0x3FFFFFFF
+
+ /*
+ * The value to AND onto the hdrmask to determine if it's an externally
+ * managed memory chunk.
+ */
+#define MEMORYCHUNK_EXTERNAL_BIT (1 << 3)
+
+typedef struct MemoryChunk
+{
+#ifdef MEMORY_CONTEXT_CHECKING
+ Size requested_size;
+#endif
+
+ /* bitfield for storing details about the chunk */
+ uint64 hdrmask; /* must be last */
+} MemoryChunk;
+
+/* Get the MemoryChunk from the pointer */
+#define PointerGetMemoryChunk(p) ((MemoryChunk *) ((char *) (p) -
sizeof(MemoryChunk)))
+/* Get the pointer from the MemoryChunk */
+#define MemoryChunkGetPointer(c) ((void *) ((char *) (c) +
sizeof(MemoryChunk)))
+
+/* private macros for making the inline functions below more simple */
+#define HdrMaskIsExternal(hdrmask) ((hdrmask) &
MEMORYCHUNK_EXTERNAL_BIT)
+#define HdrMaskChunkSize(hdrmask) (((hdrmask) >> 4) &
MEMORYCHUNK_MAX_SIZE)
+#define HdrMaskBlockOffset(hdrmask) ((hdrmask) >> 34)
+
+/*
+ * MemoryChunkSetHdrMask
+ * Store the given 'block', 'chunk_size' and 'methodid' in the
given
+ * MemoryChunk.
+ *
+ * The number of bytes between 'block' and 'chunk' must be <=
+ * MEMORYCHUNK_MAX_SIZE
+ */
+static inline void
+MemoryChunkSetHdrMask(MemoryChunk *chunk, void *block,
+ Size chunk_size,
MemoryContextMethodID methodid)
+{
+ Size blockoffset = (char *) chunk - (char *) block;
+
+ Assert((char *) chunk > (char *) block);
+ Assert(blockoffset <= MEMORYCHUNK_MAX_SIZE);
+ Assert(chunk_size <= MEMORYCHUNK_MAX_SIZE);
+ Assert(methodid <= 7);
+
+ chunk->hdrmask = (blockoffset << 34) | (chunk_size << 4) | methodid;
+}
+
+/*
+ * MemoryChunkSetHdrMaskExternal
+ * Set 'chunk' as an externally managed chunk. Here we only
record the
+ * MemoryContextMethodID and set the external chunk bit.
+ */
+static inline void
+MemoryChunkSetHdrMaskExternal(MemoryChunk *chunk,
+ MemoryContextMethodID
methodid)
+{
+ Assert(methodid <= 7);
+
+ chunk->hdrmask = MEMORYCHUNK_EXTERNAL_BIT | methodid;
+}
+
+/*
+ * MemoryChunkIsExternal
+ * Return true if 'chunk' is marked as external.
+ */
+static inline bool
+MemoryChunkIsExternal(MemoryChunk *chunk)
+{
+ return HdrMaskIsExternal(chunk->hdrmask);
+}
+
+/*
+ * MemoryChunkGetChunkSize
+ * For non-external chunks, returns the size of the chunk as was
set
+ * in MemoryChunkSetHdrMask.
+ */
+static inline Size
+MemoryChunkGetSize(MemoryChunk *chunk)
+{
+ Assert(!HdrMaskIsExternal(chunk->hdrmask));
+
+ return HdrMaskChunkSize(chunk->hdrmask);
+}
+
+/*
+ * MemoryChunkGetBlock
+ * For non-external chunks, returns the pointer to the block as
was set
+ * in MemoryChunkSetHdrMask.
+ */
+static inline void *
+MemoryChunkGetBlock(MemoryChunk *chunk)
+{
+ Assert(!HdrMaskIsExternal(chunk->hdrmask));
+
+ return (void *) ((char *) chunk - HdrMaskBlockOffset(chunk->hdrmask));
+}
+
+/* cleanup all internal definitions */
+#undef MEMORYCHUNK_EXTERNAL_BIT
+#undef HdrMaskIsExternal
+#undef HdrMaskChunkSize
+#undef HdrMaskBlockOffset
+
+#endif /*
MEMUTILS_MEMORYCHUNK_H */
--
2.34.1
