Doris-Breakwater commented on issue #67475: URL: https://github.com/apache/doris/issues/67475#issuecomment-5521960826
Initial triage (Doris 4.1.3; the issue currently has no labels): ### Verified facts * In 4.1.3, `BDBJEJournal.write(JournalBatch)` commits the BDBJE transaction **before** emitting this warning. The warning is triggered when the committed batch is larger than `batch_edit_log_max_byte_size`, whose default is 640 KiB (655,360 bytes). Therefore, the reported line by itself does not indicate a failed journal write or data loss. The reported record was 714,271 bytes, 58,911 bytes above that advisory threshold. * `batch size 1` is important: this was one oversized journal entity, not too many small entries grouped together. Reducing `batch_edit_log_max_item_num` cannot split that entity. * A 4.1.3 MySQL Streaming Job can persist the complete JDBC offset-provider state inside an `OP_UPDATE_SCHEDULER_JOB` edit log. During snapshot processing that state grows with split progress and is persisted on offset commits. * [PR #66238](https://github.com/apache/doris/pull/66238), backported to branch-4.1 by [PR #66539](https://github.com/apache/doris/pull/66539) after the 4.1.3 tag, specifically reduces this behavior: it rate-limits snapshot offset persistence, increases the default snapshot split size, and removes obsolete snapshot recovery state after entering the binlog phase. ### Assessment The warning is informational/advisory unless it is accompanied by BDBJE write errors, high journal-write latency, or follower replay lag. The most likely explanation is the known 4.1.3 snapshot-offset persistence behavior addressed by #66238/#66539, **but this is not yet confirmed** because the issue does not identify the operation type of journal `1517333`, the job phase, or the warning frequency. If journal `1517333` is `OP_UPDATE_SCHEDULER_JOB` (numeric opcode 451) and the warnings occur while the MySQL job is taking its initial snapshot, the preferred resolution is to upgrade to a 4.1 build/release containing #66539 and verify the result. If the warning continues during the incremental/binlog phase after that change, the remaining serialized payload may be intrinsically large (for example, schema metadata for many/wide source tables) and should be investigated separately. I do **not** recommend increasing `batch_edit_log_max_byte_size` merely to hide the warning. That changes the batching threshold and can allow larger BDBJE transactions; it does not reduce the metadata being persisted. An isolated warning can be left alone if journal latency and FE replay remain healthy. ### Information needed to confirm the cause Please provide: 1. The exact build/commit identifier in addition to `4.1.3`, and whether this is shared-nothing or compute-storage-decoupled deployment. 2. Warning timestamps/frequency and whether they occur during job creation, initial snapshot, snapshot-to-binlog transition, or steady incremental consumption. 3. A credential-redacted `CREATE JOB` statement, the number of synchronized source tables, approximate total column count, and any explicitly configured snapshot split size. 4. The nearby CDC-client `commit offset ... tableSchemasSize=...` log line and any BDBJE write-error/slow-write lines from the same interval. Please redact connection properties and credentials. 5. If safely available from verbose logging or an offline BDBJE metadata copy, only the `OpType` and `Size` for journal ID `1517333`. Do not publish the journal `Data` field because a serialized job may contain sensitive connection properties. Maintainer next step: first confirm opcode 451 and correlate the warning with snapshot commits. If confirmed, treat #66539 as the existing branch-4.1 remediation and determine the first released 4.1.x version that contains it; otherwise inspect the identified operation's serialized fields before classifying this as a Streaming Job bug. Breakwater-GitHub-Analysis-Slot: slot_453c7dc7d079 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
