ogiogidayo opened a new issue, #39900:
URL: https://github.com/apache/beam/issues/39900
### What happened?
### What happened?
Reading a MongoDB collection whose `_id` values are not ObjectIds (e.g.
application-defined string IDs, which MongoDB fully supports) fails as soon as
read splitting is enabled via `withBucketAuto(true)` (or `withNumSplits(n)`).
Observed with Beam 2.75.0 on Dataflow, reading a MongoDB Atlas collection
with uniform string `_id` values:
```
org.apache.beam.sdk.util.UserCodeException:
java.lang.IllegalArgumentException: state should be: hexString has 24 characters
at org.bson.types.ObjectId.parseHexString(ObjectId.java:384)
at org.bson.types.ObjectId.<init>(ObjectId.java:193)
at org.bson.json.JsonReader.visitObjectIdConstructor(JsonReader.java:733)
...
at org.bson.Document.parse(Document.java:129)
at
org.apache.beam.sdk.io.mongodb.MongoDbIO$BoundedMongoDbSource.split(MongoDbIO.java:546)
at
org.apache.beam.sdk.io.Read$BoundedSourceAsSDFWrapperFn.splitRestriction(Read.java:304)
```
### Root cause
`BoundedMongoDbSource.splitKeysToFilters` unconditionally formats split
boundaries as `ObjectId("%s")`, regardless of the boundary value's actual BSON
type:
https://github.com/apache/beam/blob/v2.75.0/sdks/java/io/mongodb/src/main/java/org/apache/beam/sdk/io/mongodb/MongoDbIO.java#L603-L643
The `$bucketAuto` boundary computation itself works fine for string `_id`
values; only the conversion of boundaries into range filters is broken. When
the generated filter string is parsed back with `Document.parse` inside
`split()`, the `ObjectId("<non-hex-string>")` constructor throws.
The AggregationQuery path has the same assumption (`splitKeysToMatch` calls
`splitKeys.get(i).getObjectId("_id")`):
https://github.com/apache/beam/blob/v2.75.0/sdks/java/io/mongodb/src/main/java/org/apache/beam/sdk/io/mongodb/MongoDbIO.java#L675
So there is no way to read a non-ObjectId-keyed collection with parallel
splitting.
### The existing unit test asserts unparseable output
`MongoDbIOTest#testSplitIntoFilters` feeds an Integer `_id` (56) and asserts
the output contains `ObjectId("56")` — a filter that can never be parsed back
(ObjectId requires a 24-character hex string). The test passes only because it
compares strings without parsing them, which indicates the current behavior is
accidental rather than intended:
https://github.com/apache/beam/blob/v2.75.0/sdks/java/io/mongodb/src/test/java/org/apache/beam/sdk/io/mongodb/MongoDbIOTest.java#L100-L107
### Steps to reproduce
1. Create a collection where `_id` values are uniform strings (e.g.
`"user-abc123"`).
2. Read it with:
```java
MongoDbIO.read()
.withUri(uri)
.withDatabase("db")
.withCollection("coll")
.withBucketAuto(true)
```
3. The job fails in `split()` with the exception above. (`withNumSplits(n)`
without bucketAuto fails the same way once `splitVector` returns string
boundaries.)
### Relationship to existing issues
- #18600 (imported from Jira
[BEAM-3165](https://issues.apache.org/jira/browse/BEAM-3165), originally
reported in 2017) describes the same root cause. Back then splitting ran
unconditionally, so the `ObjectId(...)`-formatted filter blew up later, when
parsed in `BoundedMongoDbReader.start()`. #17084 — attached to the same Jira —
made unsplit reads the default (`numSplits <= 0` returns a single source),
which resolved the default-configuration symptom but left `splitKeysToFilters`
unchanged, so enabling `bucketAuto`/`numSplits` still hits it today (as also
noted in the 2023 discussion on #18600). Filing this issue with a current-code
reproduction; happy to consolidate into #18600 if preferred.
- The Python SDK already supports splitting on uniform integer/string `_id`
values (#14460, BEAM-12119/BEAM-12122), so fixing this brings the Java SDK to
parity.
[`mongodbio.py`](https://github.com/apache/beam/blob/v2.75.0/sdks/python/apache_beam/io/mongodbio.py#L376-L383)
explicitly dispatches on the `_id` type when creating range trackers:
```python
if isinstance(start_position, ObjectId):
return _ObjectIdRangeTracker(start_position, stop_position)
if isinstance(start_position, int):
return OffsetRangeTracker(start_position, stop_position)
if isinstance(start_position, str):
return LexicographicKeyRangeTracker(start_position, stop_position)
```
- Out of scope: collections with *mixed* `_id` types (see #30472, closed as
not planned). This issue is only about collections with a uniform non-ObjectId
`_id` type.
### Proposed fix (backward compatible)
Serialize split boundaries with their actual BSON types using extended JSON
(`Document#toJson`) instead of unconditional `ObjectId("...")` string
formatting. For ObjectId-keyed collections this produces semantically identical
range filters, so existing users are unaffected; the public API does not
change. `testSplitIntoFilters` expectations need updating (they currently
assert the unparseable output described above).
I'd like to contribute a fix (we already run this change in production) with
unit tests for string, ObjectId, and single-split-key cases.
### Issue Priority
Priority: 2 (default / most bugs should be filed as P2)
### Issue Components
- [ ] Component: Python SDK
- [x] Component: Java SDK
- [ ] Component: Go SDK
- [ ] Component: Typescript SDK
- [x] Component: IO connector
- [ ] Component: Beam YAML
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Infrastructure
- [ ] Component: Spark Runner
- [ ] Component: Flink Runner
- [ ] Component: Prism Runner
- [ ] Component: Twister2 Runner
- [ ] Component: Hazelcast Jet Runner
- [ ] Component: Google Cloud Dataflow Runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
