SEPURI-SAI-KRISHNA opened a new issue, #13330: URL: https://github.com/apache/seatunnel/issues/13330
### Search before asking - [X] I had searched in the [issues](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22Bug%22) and found no similar issues. ### What happened The MongoDB source splits a collection with `SamplingSplitStrategy`: it samples documents sorted by `partition.split-key` (default `_id`) and turns every few sampled values into the bound of a range split, `{key: {$lt: b1}}`, `{key: {$gte: b1, $lt: b2}}`, ..., `{key: {$gte: bn}}` (`MongoSplitUtils.createMongoSplit`). Once a collection is larger than one split (64 MB by default), the job can lose documents or read them more than once, and it still succeeds: 1. **Documents are lost when the split key has more than one BSON type.** A MongoDB range filter only matches values of the type bracket of its bound (numbers, strings, ObjectIds, dates and so on). When two neighbouring bounds have different types, for example `{_id: {$gte: 760, $lt: "s0155"}}`, the split matches nothing, so every number above 760 and every string below `"s0155"` is never read. This happens with the default `_id` when a collection mixes ObjectId, string and numeric ids, which is common after imports. 2. **Documents are read again when the split key is missing or null in some documents.** `samples.get(i).get(splitKey)` returns Java `null` for a sampled document without the key, and `createMongoSplit` treats a null bound as "no bound": a split between two such bounds is `{}` and reads the whole collection. A null or NaN value can also become a bound, `{key: {$lt: null}}` or `{key: {$lt: NaN}}`, of a range that matches nothing. 3. **`match.query` is dropped on the first and last split when it names the split key.** `createMongoSplit` copies the match query into the split query and then appends the bound with the same field name, which replaces the condition of the match query. 4. **A split key with a dot, such as `a.b`, makes every split `{}`.** `BsonDocument.get("a.b")` does not follow the path, so every bound is null and each split reads the whole collection. Measured with the connector's own `SamplingSplitStrategy` on MongoDB 8.0.4 (64 KB splits, every document of every split read back and compared by `_id`): | Collection | Split key | Documents | Read | |---|---|---|---| | `_id` is 1000 ObjectId, 1000 string, 1000 long | `_id` (default) | 3000 | 2520, 2340 and 2428 in three runs: 480, 660 and 572 lost | | `code` is a number, missing (500), null (500) or NaN (5) | `code`, indexed | 3000 | 2927 and 2849: 73 and 151 lost (NaN and numbers); 8995 in a third run: 5995 read again | | `code` is a number, missing (500) or null (500) | `code` | 3000 | 6000: one split `{}` read every document again | | int `_id`, `match.query = "{_id: {$gte: 1500}}"` | `_id` | 1500 | 3000: 1500 outside the match query | | `a.b` is a number | `a.b` | 3000 | 36000: all 12 splits `{}` | | `_id` is ObjectId only | `_id` | 20000 | 20000 | The samples are random, so the exact counts change from run to run. No error or warning is logged. ### SeaTunnel Version dev (`f1ab36f042`). The split code is unchanged since #9895. ### SeaTunnel Config ```conf env { parallelism = 2 job.mode = "BATCH" } source { MongoDB { uri = "mongodb://127.0.0.1:27017/probe" database = "probe" collection = "mixed_id" schema = { fields { pad = string } } } } sink { Console {} } ``` with a collection larger than `partition.split-size` whose `_id` values are ObjectIds, strings and numbers. ### Running Command ```shell SamplingSplitStrategy.split() against MongoDB 8.0.4, then find(split.getQuery()) for every split ``` ### Error Exception ```log None. The job succeeds and reads fewer or more documents than the collection holds. ``` ### Zeta or Flink or Spark Version All engines: the splits are created by the source's enumerator. ### Java or Scala Version JDK 17 locally. ### Screenshots _No response_ ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
