[
https://issues.apache.org/jira/browse/FLINK-41352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18137965#comment-18137965
]
Sonal Maheshwari commented on FLINK-41352:
------------------------------------------
Hi, I'd like to work on this. I have an implementation with tests ready and
will open a PR against master shortly. Could someone please assign it to me?
> [mongodb] Support change events larger than 16MB with
> $changeStreamSplitLargeEvent
> ----------------------------------------------------------------------------------
>
> Key: FLINK-41352
> URL: https://issues.apache.org/jira/browse/FLINK-41352
> Project: Flink
> Issue Type: Improvement
> Components: Flink CDC
> Affects Versions: cdc-3.7.0
> Reporter: Sonal Maheshwari
> Priority: Major
>
> When a change event is larger than 16MB, e.g. an update whose looked-up full
> document ({{fullDocument: updateLookup}}) plus update description exceed the
> limit, MongoDB fails the getMore with BSONObjectTooLarge (10334).
> {{MongoDBStreamFetchTask}} treats 10334 as an invalid change stream and
> reopens the cursor from the same resume token, which hits the same event
> again. The stream never moves past it and the lag keeps growing.
> MongoDB 6.0.9+ provides the {{$changeStreamSplitLargeEvent}} stage, which
> splits such events into fragments of at most 16MB. Proposal:
> - Add an opt-in option {{scan.split-large-event.enabled}} (default false)
> that appends the stage to the change stream pipeline.
> - Reassemble the fragments in the stream fetch task into one change event
> that carries the resume token of the last fragment, so checkpoints never
> store a mid-event token.
> - Handle the edge cases: cursor reopen mid-event, heartbeats while
> reassembling, resuming from a fragment token whose event is no longer split,
> unsupported MongoDB versions.
> Related: FLINK-38782 reports the same 10334 error.
> I have a working implementation with unit tests and Testcontainers tests on
> MongoDB 8.0 (Flink 1.20 and 2.2) and would like to contribute it.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
