realAaronWu opened a new issue, #435: URL: https://github.com/apache/paimon-cpp/issues/435
Parent integration: sirius-db/sirius#2181. Consumer bridge tracking: polardb/duckdb-paimon#139. ## Need A database embedding paimon-cpp needs to admit planning work before materializing an arbitrarily large native plan, and stop submitting work after cooperative cancellation. This is a feature request, not a claim that existing standalone planning is incorrect. The integration pins `53f9c86d45aabb0a6f1a379271da07d7a9f27a3d`. The following API gap also remains in inspected main `242decffdfdefef01dcd0e52e48c9a1e69792479`: - `ScanContextBuilder::WithMemoryPool` routes pool-backed allocations, but does not bound ordinary STL planning containers or their transitive metadata allocations. - `TableScan::CreatePlan()` returns a complete plan; it exposes no planning cancellation/budget contract. `SetLimit` is a row-result pushdown, not a planning-memory quota: `DataTableBatchScan::CreatePlan` calls `starting_scanner_->Scan` before `ApplyPushDownLimit`. - `FileStoreScan::ReadFileEntries` builds a vector of futures for all supplied manifest metadata, then collects complete per-manifest entry vectors. A custom serial executor can prevent concurrent task execution, but does not itself bound those retained containers/results. Source references at the inspected main: - [Scan context API](https://github.com/apache/paimon-cpp/blob/242decffdfdefef01dcd0e52e48c9a1e69792479/include/paimon/scan_context.h) - [TableScan API](https://github.com/apache/paimon-cpp/blob/242decffdfdefef01dcd0e52e48c9a1e69792479/include/paimon/table/source/table_scan.h) - [Batch scan / limit ordering](https://github.com/apache/paimon-cpp/blob/242decffdfdefef01dcd0e52e48c9a1e69792479/src/paimon/core/table/source/data_table_batch_scan.cpp) - [Manifest task/results materialization](https://github.com/apache/paimon-cpp/blob/242decffdfdefef01dcd0e52e48c9a1e69792479/src/paimon/core/operation/file_store_scan.cpp) ## Proposed Contract To Discuss Provide optional query-scoped planning controls, preserving existing unlimited behavior by default. The API shape should be agreed with maintainers; a post-plan split-count check is not sufficient. - Check conservative manifest/entry/split/retained-metadata quotas before the corresponding allocation or task submission; identify which allocations are covered and what remains outside the guarantee. - Bound outstanding manifest work and retained results, including ordinary containers, rather than only pool-backed buffers. - Check a cooperative cancellation token before I/O/submission and inside substantial decode/merge loops. Do not promise interruption of an arbitrary blocking filesystem call unless the filesystem supports it. - Return a distinguishable resource-refusal/cancellation status, never partial success or EOF. Stop new submissions and drain accepted work before returning, including exceptions and submission failure. - Avoid changing process-global executors, Arrow thread capacity, or allocation resources. Acceptance tests should use many small manifests and injected allocation/I/O/submission failures. Assert the limit is enforced before excess work is accepted, accepted jobs are drained on every terminal path, and cancellation cannot return a partial successful plan. Include serial/nested executor configurations to guard against deadlock. Keep row LIMIT semantics separate. This issue does not request another Arrow batch-lifetime implementation; that work already exists in #259/#260. No hard whole-process-memory bound or prompt cancellation of uninterruptible I/O is claimed by the current integration. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
