SEZ9 commented on issue #13330:
URL: https://github.com/apache/seatunnel/issues/13330#issuecomment-6994106490

   Thanks for the thorough report. The four cases, the config, and the measured 
read counts on MongoDB 8.0.4 make the problem clear, and the silent loss or 
re-read of documents with no error or warning is the most concerning part.
   
   Since you've indicated you're willing to submit a PR, that would be very 
welcome. A few suggestions:
   
   - The four cases look independent (mixed BSON types in the split key, 
missing/null/NaN bounds, `match.query` replaced when it names the split key, 
and a dotted split key such as `a.b`). Separate fixes, or at least separate 
commits, may be easier to review.
   - Please turn the rows of your table into regression tests for 
`SamplingSplitStrategy` / `MongoSplitUtils.createMongoSplit`, asserting that 
the splits together read every document exactly once and that `match.query` is 
preserved.
   - Before you start, could you briefly outline the approach you have in mind 
for each case, particularly how you plan to handle bounds of different BSON 
types and null/missing split-key values? That would help us agree on the 
direction early.
   
   <!-- streview-comment:2441 -->


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to