Hey Wyatt, Looking at the code, it's not `NewRecord` that calls mem.Allocate, it's the call to `Append` which does the allocating.
The reason why `builder.cap` is zeroed out on reset for array Builders is because there's no way to know that the array has been released and that it is safe to re-use the same buffers when Append is called on the builder. Your example is simply not a commonly reported workflow, in most cases after building a record it is handed off to something else (put into a channel, added to a slice, etc.) rather than an operation being done immediately on it and then releasing it. Whenever `NewRecord` is called, which ultimately calls `NewArray` on the underlying columns, it is necessary to ensure that the builder does not retain the references to the buffers that were built since we can't be sure that they will be released before or after the next call to Append on the builder. Unfortunately, there's currently only two ways you could avoid allocating for each record in your example: 1. You would need to build up the buffers themselves and use `NewData` / `MakeFromData` manually, then just re-use the buffers for each record you build. 2. You could create a custom Allocator object which utilizes a pool of memory under the hood so that when the buffers are freed, the next call to Allocate could return those same buffers back --Matt On Mon, Jun 24, 2024 at 11:37 PM Wyatt Alt <[email protected]> wrote: > Hello, > I'm just getting started with the go library and I may have > misunderstandings. I'm wondering if it is possible to rewrite the following > test in a way that will not allocate new buffers on successive calls to > NewRecord. > > Thanks for any advice - code below. > Wyatt > > func TestBuilder(t *testing.T) { > pool := memory.NewGoAllocator() > schema := arrow.NewSchema([]arrow.Field{ > {Name: "data", Type: arrow.PrimitiveTypes.Float64}, > > }, nil) > > > builder := array.NewRecordBuilder(pool, schema) > > defer builder.Release() > > > fb := builder.Field(0).(*array.Float64Builder) > > > > // Max size the field attains > > fb.Reserve(2) > > > // Write 100 record batches > for i := 0; i < 100; i++ { > > > > // Two float observations per record > for j := 0; j < 2; j++ { > fb.Append(float64(j)) > > } > > > // Why does this call mem.Allocate() on every call? > // Why is builder.cap zeroed on calls to reset instead of > retained? > record := builder.NewRecord() > // do something with the record... > record.Release() > > } > } >
