yinli-systems commented on issue #24117: URL: https://github.com/apache/datafusion/issues/24117#issuecomment-5672865424
Implementation update: I now have a working, fail-closed end-to-end prototype stacked on #24227: https://github.com/yinli-systems/datafusion/commit/2bdf91d9f Key points: - Adds an opt-in physical optimizer rule for direct `Utf8View` GROUP BY columns. - Carries per-file physical facts from the Parquet footer through listing metadata and the metadata cache. - Selectively asks the Parquet source to emit `Dictionary<Int32, Utf8>` only for profitable group keys; it does not insert a runtime dictionary cast. - Requires exact row count, null count, and NDV, every data page in every input file to be dictionary encoded, average decoded value width >= 16 bytes, and at least 10,000 non-null rows per distinct value. - Missing or ambiguous evidence, filters, limits, file ranges, complex scan projections, grouping expressions, or aggregate references to a retyped key all leave the plan unchanged. - Restores the original aggregate output schema and field metadata above the aggregate, and the rewrite is idempotent. The conservative gates come from the expanded benchmark in #25303, now including `Utf8View`. With 10M rows, dictionaries still win at NDV 1,000 for 16-byte and 64-byte values, but lose clearly by NDV 10,000; short inline `Utf8View` values are also a poor target. I updated the PR description with those results. Validation completed locally: - Real Parquet end-to-end test proves the footer facts, verifies the scan actually changes from `Utf8View` to native `Dictionary<Int32, Utf8>`, compares results byte-for-byte with the disabled baseline, and repeats planning through the metadata-cache path. - `datafusion-physical-optimizer`: 41 passed. - `datafusion-datasource-parquet`: 265 passed. - `datafusion-common`: 610 passed. - `datafusion-datasource`: 176 passed. - `datafusion-execution`: 113 passed. - `datafusion-catalog-listing`: 19 passed. - targeted `information_schema` sqllogictests: 4 files passed. - affected packages pass all-target no-deps clippy with `-D warnings`; Rust formatting, diff checks, and Prettier 2.7.1 checks pass. I have pushed the implementation branch but deliberately have not opened it against `main` yet: #24227 is still unmerged, and opening now would mix that contributor's dependency commits into this PR. Once #24227 lands, I will rebase this single feature commit onto current `main`, regenerate the config docs there, and open the clean implementation PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
