curcur commented on pull request #13501: URL: https://github.com/apache/flink/pull/13501#issuecomment-702802919
1. What is the partial-record information: - Whether the FIRST record BEGINS as a complete and new record or not. - If YES, the recorded length = MIN(the entire record size, buffer size); If NOT, the recorded length = MIN(the remaining record size, buffer size) the recorded length is used to decide how much data to skip in the NOT case; the recorded length is not used in the YES case. Here, the **FIRST** record means: starting with memory segment position 0 for both writing (builder) and reading (consumer) Partial-record information is set up like this because a record can miss its header part only when a bufferBuilder ends with a full buffer but not a full record; when flushing buffer to network, we either flush a complete buffer or a complete record; So we never have a chance to start reading a partial record if reading position of bufferConsumer > 0 when polling buffers to send through the network. 2. Hence, partial-record information is used by the buffer reader to decide 1). whether needs to skip data when reading a buffer; and 2). how much data to skip; I am lean-to to consider `partial-record information` as a buffer-level concept, instead of record-level; Or at least, as a record writer, it should not care about how data layout in the underlying memory segment as well. and if I want to decide how much data to skip (which is the whole purpose for partial record clean-up), it seems that I have to have some way to manipulate buffer write/read anyway. 3. Currently, BufferConsumer is created and added into PrioritizedDeque<BufferConsumer> when requesting a new bufferBuilder in BufferWritingResultPartition#getNewSubpartitionBufferBuilder; (In broadcasting mode, it is similar, but adding multiple copies) After that, write and read is completely separated/independent and only coordinated through the `write positionMarker`. If I want to encode the `partial-record information` into PrioritizedDeque<BufferConsumerWithOffset>, I have to either encode offset information before writing (that's when BufferConsumerWithOffset is created, but offset information is available only when writing the FIRST RECORD); or updating BufferConsumerWithOffset's offset when writing the FIRST RECORD, which is strange as well. 4. Since `partial-record information` is shared between BufferBuilder and BufferConsumer both, so I was wondering whether I can put this information (encoded in integer) as a field of MemorySegment? ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
