suxiaogang223 opened a new pull request, #66685: URL: https://github.com/apache/doris/pull/66685
### What - introduce a connector-independent external write distribution plan with safe single-writer fallback and stateless hash routing - route Iceberg writes by evaluated partition transform results so each physical partition is owned by one writer - route supported Paimon fixed-bucket writes by native Paimon-compatible partition and bucket expressions over Doris blocks - keep dynamic, key-dynamic, bucket-unaware, postpone, custom-bucket, and unsupported Paimon cases on the single-writer path - plan external writer parallelism from writer capacity and estimated ownership cardinality, and expose writer/file-size planning metadata in sink profiles - reuse standard Project and HASH/GATHER Exchange nodes instead of connector-specific exchange protocols ### Why External table writers previously did not consistently colocate rows belonging to the same Iceberg partition transform or Paimon fixed bucket. That could prevent effective parallel writes or allow one ownership unit to be split across writers, increasing small files and risking connector writer-state conflicts. This change establishes one common distribution boundary: connector providers produce hidden routing expressions, standard Doris exchanges assign each ownership key to a unique writer, and unsupported or stateful modes fall back safely to one writer. Stateful dynamic-bucket assigners are intentionally outside this change. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
