On 8/20/21 4:01 PM, Tony Whyman wrote:
......
With my test data, the buffer overflow occurs after several thousand
calls to IBatch::add. The number of message buffers lost is of the
order of hundreds. I don't want to have to keep a copy of every
message buffer as that could result in other out of memory issues.
My problem is that the number of message buffers that may be lost on
buffer overflow appears to be indeterminate. It would be more useful
if when IBatch is created, I could specify a message buffer limit and
the internal buffers were allocated in advance.
You can specify such limit - use IBatch::TAG_BUFFER_BYTES_SIZE parameter
in batch parameters block when creating a batch, default is 16Mb, hard
limit - 256Mb. Ywo such buffers will be available - one for messages
data, other - for blobs.
Well, in theory OOM can take place later, but that is abnormal case.
Alternatively, I could also work with an IBatch method that told me
the minimum number of guaranteed calls to IBatch::add (for a given
message buffer size). Other suggestions are welcome.
Divide what you've requested in TAG_BUFFER_BYTES_SIZE by
getAlignedLength() of your message. But be aware that this will not work
when/if we move to variable message size and string field with
unrestricted length.
I believe that a solution is necessary because you cannot use IBatch
for any large scale use without knowing what you need to do to recover
from a buffer overflow error without losing user data.
With current fixed format of messages solution is trivial. I believe
that when adding var-length forma we will add required support to easily
get it's actual length as stored in a buffer.
Blobs are more serious issue. To make sure blob data also fits into
buffer one need to calculate it's actual size for which I suppose is
worth adding a function to IBatch iface which will help with that task
(there is getBlobAlignment but it's not enough).
Firebird-Devel mailing list, web interface at
https://lists.sourceforge.net/lists/listinfo/firebird-devel
