xylaaaaa commented on issue #69884: URL: https://github.com/apache/doris/issues/69884#issuecomment-6683292937
### Follow-up assessment: prefer statement-local scan-plan reuse first The requested follow-up was read-only: no production/test code changes, no new build or cluster run, and no new performance claims. This refines the recommendation in the [checkpoint diagnosis](https://github.com/apache/doris/issues/69884#issuecomment-6682873512). **Recommended first step: wire Delta into the existing statement-local reuse of equivalent synchronous scan plans.** The current branch already has this in [Iceberg](https://github.com/apache/doris/blob/fb4e695560bafc1862ac398fe22bfe6338c7f02e/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java#L428), Hive, Paimon and Hudi. `ConnectorStatementScope.computeIfAbsentForPlanning` and `enable_external_scan_task_reuse` already exist; Delta currently re-enumerates every scan. This needs no shared FE/SPI/BE change and can avoid both repeated IO and Kernel parsing, rather than only network reads. Offline grouping of the saved plans by table and predicate, removing slot IDs only, gives these **candidate** groups: | Query | Scan nodes | Candidate groups | | --- | ---: | ---: | | Q88 | 32 | 11 | | Q80 | 18 | 12 | | Q4 | 18 | 6 | | Q22 | 3 | 3 | These are not observed cache hits: the full request key still needs tests. Q88's store_sales/store/household_demographics each repeat eight equivalent filters, while its eight time_dim predicates remain distinct. If other request dimensions match, the measured five-GET-per-enumeration pattern suggests 160→55 checkpoint GETs for Q88 and 90→60 for Q80; this is workload arithmetic, not a latency prediction. The Delta key must cover table path/UUID/version, typed filter, required partitions and the pruned-to-empty flag, COUNT mode, split size and timezone. Reuse only complete immutable ranges plus per-node metrics; keep cancellation/validation on hits. The existing range descriptor copy behavior is a suitable base. Preserve a per-statement retention budget and release at planning end. Streaming/unknown-size scans must keep their current path without materialization. The scope loader must only create a private memo: scan planning re-enters the statement scope for counts, so doing IO directly inside that loader would introduce map re-entry. **Checkpoint byte caching remains a second candidate.** The retained 24 checkpoint objects total only 1,002,655 bytes; Q88's four unique objects total 165,159 bytes, versus 3,331,960 transferred per EXPLAIN. A scoped Engine/DefaultParquetHandler/FileIO wrapper is possible through public Kernel 4.4.1 APIs, but the catalog-shared `newInputFile(path, size)` alone lacks user/table identity. Snapshot discovery should stay outside that cache until the table UUID is established. Also, [the pinned Delta protocol](https://github.com/delta-io/delta/blob/v4.4.1/PROTOCOL.md#classic-checkpoint) allows classic checkpoints to be overwritten by a competing writer with equivalent content; byte-layout immutability must not be assumed. Whole-object consistency, refresh fencing, concurrent-load memory and interruptible waits require explicit design. Framework retained-weight limits do not bound byte arrays being downloaded or still held by readers. Acceptance for the first candidate: identity/filter/partition/count/size/timezone and failure/lifecycle tests; filesystem/dedicated HMS/HMS gateway and mixed-format results; same-build paired reuse-on/off probes with actual hit/GET/profile evidence; the retained million-file streaming fixture under 512 MiB; then full TPC-DS. No claim that this fixes Q22, Q1 early termination, or the full +4.69% gap. END-03 remains open. Local tasklist and `optimization-assessment.md` contain the scoped design and review checklist; services remain stopped and retained fixtures untouched. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
