corgy-w opened a new pull request, #13252:
URL: https://github.com/apache/seatunnel/pull/13252
### What
The Druid sink no longer re-submits the rows of previous batches: the row
buffer is cleared after each successful submission, and an empty buffer is not
submitted at all.
### Why
`DruidWriter` keeps every row it has written in a single `StringBuffer`
field, and `flush()` builds the indexing task from that whole buffer:
```java
// DruidWriter.java
private final transient StringBuffer data; // created once in the
constructor
...
data.append(joiner); // write() appends the
row
if (currentBatchSize >= batchSize) {
flush();
currentBatchSize = 0; // the counter is reset,
the buffer is not
}
...
ParallelIndexIOConfig provideDruidIOConfig(final StringBuffer data) {
return new ParallelIndexIOConfig(null, new
InlineInputSource(data.toString()), ...);
}
```
So the second flush submits batch 1 + batch 2, the third submits batches
1-3, and `close()` flushes the entire buffer one more time. The buffer is never
truncated anywhere in the class.
The duplicates are stored, not merged away: the data schema is built with
`new UniformGranularitySpec(HOUR, MINUTE, false, null)`, i.e. rollup disabled,
so Druid ingests every re-sent row as a new row. Nothing fails and no log line
reports it.
Measured with the unit test below (5 rows, `batchSize = 2`), the three
submissions carry 2, 4 and 5 lines instead of 2, 2 and 1:
```
batch 1: rows 1,2
batch 2: rows 1,2,3,4 <- rows 1,2 sent again
batch 3: rows 1,2,3,4,5 <- rows 1-4 sent again (close())
```
The growth is quadratic: for `N` rows and batch size `B` the sink ingests
about `N² / (2B)` rows. 25k rows at the default batch size of 10k ingest 55k
rows (2.2x); 1M rows ingest about 50M (50x). The buffer also grows without
bound, so a large job holds its entire dataset in memory in the writer.
### Fix
- clear the buffer after a successful submission (`data.setLength(0)`), so a
failed submission keeps its rows for the retry;
- return early from `flush()` when the buffer is empty, so `close()` does
not create an empty indexing task when the last batch was just flushed.
### How verified
- New unit test `DruidWriterTest#everyBatchSubmitsOnlyItsOwnRows`: five rows
with `batchSize = 2` through a local stand-in for the Druid Overlord, asserting
the submissions carry `[2, 2, 1]` lines.
- before the fix: `expected: <[2, 2, 1]> but was: <[2, 4, 5]>`
- after the fix: `Tests run: 3, Failures: 0, Errors: 0, Skipped: 0` (`mvn
-o -pl seatunnel-connectors-v2/connector-druid test -Dtest=DruidWriterTest`)
- New IT `DruidIT#testDruidSinkDoesNotResubmitRowsAcrossBatches` with
`fakesource_to_druid_with_batch.conf` (six rows, `batchSize = 2`, per-run
datasource): asserts the datasource holds exactly six rows with ids 1-6 after
the job, which fails with the duplicated rows of the old behaviour.
### Compatibility
No option name, default value, or public API change. Only the number of rows
submitted per indexing task changes, from "everything written so far" to "the
rows of this batch".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]