vishnuprakaz commented on PR #17315:
URL: https://github.com/apache/iceberg/pull/17315#issuecomment-5252207388
I was curious so i tried tracing where `DynamicWriteResult` bytes could
actually end up in a snapshot. From what I can see, the writer is stateless,
the aggregator only holds results in memory and flushes them in
`prepareSnapshotPreBarrier`, and what the committer persists is
`DynamicCommittable`, which has its own versioned serializer. So the only path
I could find is unaligned checkpoint in flight data.
Also for context, I was checking the implementation #14810 and the
conclusion there was that this serializer is only used at runtime between the
writer and the aggregator.
I was trying to understand the path the old bytes would take to reach the
new deserializer, so maybe some more context about the setup where it got hit
could help @wangyum
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
