corgy-w opened a new pull request, #13252:
URL: https://github.com/apache/seatunnel/pull/13252

   ### What
   
   The Druid sink no longer re-submits the rows of previous batches: the row 
buffer is cleared after each successful submission, and an empty buffer is not 
submitted at all.
   
   ### Why
   
   `DruidWriter` keeps every row it has written in a single `StringBuffer` 
field, and `flush()` builds the indexing task from that whole buffer:
   
   ```java
   // DruidWriter.java
   private final transient StringBuffer data;          // created once in the 
constructor
   ...
   data.append(joiner);                                // write() appends the 
row
   if (currentBatchSize >= batchSize) {
       flush();
       currentBatchSize = 0;                           // the counter is reset, 
the buffer is not
   }
   ...
   ParallelIndexIOConfig provideDruidIOConfig(final StringBuffer data) {
       return new ParallelIndexIOConfig(null, new 
InlineInputSource(data.toString()), ...);
   }
   ```
   
   So the second flush submits batch 1 + batch 2, the third submits batches 
1-3, and `close()` flushes the entire buffer one more time. The buffer is never 
truncated anywhere in the class.
   
   The duplicates are stored, not merged away: the data schema is built with 
`new UniformGranularitySpec(HOUR, MINUTE, false, null)`, i.e. rollup disabled, 
so Druid ingests every re-sent row as a new row. Nothing fails and no log line 
reports it.
   
   Measured with the unit test below (5 rows, `batchSize = 2`), the three 
submissions carry 2, 4 and 5 lines instead of 2, 2 and 1:
   
   ```
   batch 1: rows 1,2
   batch 2: rows 1,2,3,4      <- rows 1,2 sent again
   batch 3: rows 1,2,3,4,5    <- rows 1-4 sent again (close())
   ```
   
   The growth is quadratic: for `N` rows and batch size `B` the sink ingests 
about `N² / (2B)` rows. 25k rows at the default batch size of 10k ingest 55k 
rows (2.2x); 1M rows ingest about 50M (50x). The buffer also grows without 
bound, so a large job holds its entire dataset in memory in the writer.
   
   ### Fix
   
   - clear the buffer after a successful submission (`data.setLength(0)`), so a 
failed submission keeps its rows for the retry;
   - return early from `flush()` when the buffer is empty, so `close()` does 
not create an empty indexing task when the last batch was just flushed.
   
   ### How verified
   
   - New unit test `DruidWriterTest#everyBatchSubmitsOnlyItsOwnRows`: five rows 
with `batchSize = 2` through a local stand-in for the Druid Overlord, asserting 
the submissions carry `[2, 2, 1]` lines.
     - before the fix: `expected: <[2, 2, 1]> but was: <[2, 4, 5]>`
     - after the fix: `Tests run: 3, Failures: 0, Errors: 0, Skipped: 0` (`mvn 
-o -pl seatunnel-connectors-v2/connector-druid test -Dtest=DruidWriterTest`)
   - New IT `DruidIT#testDruidSinkDoesNotResubmitRowsAcrossBatches` with 
`fakesource_to_druid_with_batch.conf` (six rows, `batchSize = 2`, per-run 
datasource): asserts the datasource holds exactly six rows with ids 1-6 after 
the job, which fails with the duplicated rows of the old behaviour.
   
   ### Compatibility
   
   No option name, default value, or public API change. Only the number of rows 
submitted per indexing task changes, from "everything written so far" to "the 
rows of this batch".
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to