zhang-arvin opened a new pull request, #20060:
URL: https://github.com/apache/druid/pull/20060
### Description
Adds more robust testing to streaming ingestion integration tests as
described in #10971.
### Changes
1. **New `FaultyStreamEventStreamGenerator`** — A `SyntheticStreamGenerator`
subclass that can inject faulty data at a configurable ratio:
- `INVALID_JSON` — Malformed JSON bytes that cannot be parsed (e.g.,
`{"broken": }`)
- `NULL_FIELDS` — Valid JSON objects where all field values are `null`
- `EMPTY_JSON` — Empty JSON objects (`{}`)
- `MULTI_ROW` — JSON arrays containing multiple objects
- `EMPTY_STRING` — Completely empty strings
- `ALL_VALID` — No faults injected (baseline)
2. **New `KafkaStreamIngestionRobustnessTest`** — Six integration tests that
verify the streaming ingestion pipeline handles faulty data gracefully:
- `test_supervisorHandlesInvalidJsonGracefully` — Supervisor stays
healthy with 20% invalid JSON
- `test_supervisorHandlesNullFieldsGracefully` — Supervisor stays healthy
with 20% null-field records
- `test_supervisorHandlesEmptyJsonGracefully` — Supervisor stays healthy
with 20% empty JSON objects
- `test_supervisorHandlesMultiRowDataGracefully` — Supervisor stays
healthy with 20% multi-row JSON arrays
- `test_supervisorHandlesEmptyStringGracefully` — Supervisor stays
healthy with 20% empty strings
- `test_supervisorHandlesAllValidDataCorrectly` — Baseline test with all
valid data
Each test verifies that:
- The supervisor remains healthy (`RUNNING` state)
- Valid records are ingested successfully
- The row count matches the expected valid records
Closes #10971.
<hr>
This PR has:
- [x] been self-reviewed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
