SEPURI-SAI-KRISHNA opened a new pull request, #13332:
URL: https://github.com/apache/seatunnel/pull/13332
Yes, glad to contribute to Apache SeaTunnel.
### Purpose of this pull request
Closes #13330.
The MongoDB source splits a collection into ranges of `partition.split-key`
(default `_id`), bounded by sampled values. Once a collection is larger than
one split, the job could lose documents or read them more than once, and still
succeed:
- A MongoDB range only matches values of the type bracket of its bound, so a
split between bounds of two types, such as `{_id: {$gte: 760, $lt: "s0155"}}`,
read nothing. With ObjectId, string and numeric `_id` values, 480, 660 and 572
of 3000 documents were lost in three runs.
- A sampled document without the split key gave a Java `null` bound, so the
split without that bound was `{}` and read the whole collection again (6000
reads of 3000 documents). A null or NaN bound gave a range that matches nothing.
- A `match.query` on the split key was replaced by the bound of the first
and the last split (3000 reads where the match query selects 1500).
- A dotted split key such as `a.b` gave null bounds, and all 12 splits were
`{}` (36000 reads of 3000 documents).
Changes, all in the source split code:
- `SamplingSplitStrategy` samples only documents with a split key value,
reads the value along a dotted path, and skips values that cannot bound a
range: null, NaN, arrays, and the types it builds no range for, such as regex.
- The sorted bounds are grouped by type bracket (numbers, strings,
documents, binary, ObjectId, boolean, date, timestamp), and each bracket gets
its own ranges, from below its first bound to above its last one.
- One more split reads every other document: `{$or: [{key: null}, {key:
NaN}, {key: NumberDecimal(NaN)}, {key: {$type: [types without a range]}}]}`.
`{key: null}` matches missing and null; `undefined` is listed among the types
because MongoDB 8.0 no longer matches it with null. Each part can use an index
on the split key: on `_id` the split examined 1 or 2 index keys.
- When no sampled value can bound a range, one split reads the whole
collection, as for a collection of one split.
- `MongoSplitUtils` combines the match query and the bound with `$and`
instead of merging their fields.
- Docs: a tip on the MongoDB source page (EN and ZH) on split keys with
several types, missing values, dotted paths and arrays.
The split state (`MongoSplit`) is unchanged, so checkpoints restore as
before.
Not covered here: a split key that holds arrays. A range matches an array by
each element, so such a document can be read by more than one split; the tip
says the split key must not hold arrays. The restore behaviour described on the
page (pending splits restored and the strategy run again) is unchanged.
I searched open and closed issues and PRs for MongoDB split problems and
checked the open PRs that touch `connector-mongodb`: #13200 (sink option
defaults) and #12626 (class loader references) do not change the source split
code. #13200 also adds a test to `MongodbIT`, in another place of the file.
### Does this PR introduce _any_ user-facing change?
Yes. A MongoDB source job now reads each document of the collection, or of
`match.query`, exactly once, also when the split key has values of several
types or is missing, null or NaN in some documents, and a `match.query` on the
split key applies to every split. Jobs on collections that fit one split, or
whose split key has one type and a value in every document, read the same
documents as before; their splits also include one more split, which reads
nothing for them. No option, default or API changes.
### How was this patch tested?
- `SamplingSplitStrategyTest`: six new cases for bounds of three types,
missing, null and NaN values, no usable bound, the match query on the split key
and dotted paths. With the match query merged as before,
`testEverySplitKeepsTheMatchQuery` fails; with all types in one bracket,
`testSplitsCoverEachTypeOfTheSampledBoundsWithItsOwnRanges` fails. All 69
`connector-mongodb` unit tests pass.
- Locally against MongoDB 8.0.4 with the strategy of dev and of this PR (64
KB splits, every split read back and compared by `_id`):
| Collection | dev | this PR |
|---|---|---|
| `_id` ObjectId, string and long (3000) | 480, 660 and 572 lost | 3000, no
duplicate |
| `code` number, missing, null, NaN (3000), indexed | 73 and 151 lost; 8995
read in a third run | 3000, no duplicate |
| int `_id`, `match.query` `{_id: {$gte: 1500}}` | 3000 read, 1500 outside
the match | 1500 |
| `a.b` number (3000) | 36000 read | 3000 |
| ObjectId `_id` only (20000) | 20000 | 20000 |
- E2E: `MongodbIT#testMongodbSourceSplitKeyOfSeveralTypes` reads a
collection whose `_id` is an ObjectId, a string or a long and whose `c_key` is
an int, a string, missing or null, split by `c_key`, by `_id`, and by `c_key`
with `match.query = "{c_key: {$gte: 150}}"`, and checks that each job writes
exactly the expected `c_id` values. It passes on Zeta, on Flink 1.13.6, 1.15.3,
1.18.0 and 1.20.1, and on Spark 2.4.6 and 3.3.0. With the split classes of dev
it fails on Zeta at the first job: the sink holds 523 rows for the 300
documents, 17 documents are missing (13 with an int `c_key`, 4 with a string
one) and 120 were written more than once.
- `spotless:check`, `MarkdownTest`, `git diff --check`.
### Check list
* [x] If any new Jar binary package adding in your PR, please add License
Notice according
[New License
Guide](https://github.com/apache/seatunnel/blob/dev/docs/en/developer/new-license.md)
(no new jar)
* [x] If necessary, please update the documentation to describe the new
feature. https://github.com/apache/seatunnel/tree/dev/docs
* [x] If necessary, please update `incompatible-changes.md` to describe the
incompatibility caused by this PR. (not needed: no incompatible change)
* [ ] If you are contributing the connector code, please check that the
following files are updated: (not a new connector)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]