kz930 opened a new pull request, #8512: URL: https://github.com/apache/texera/pull/8512
### What changes were proposed in this PR? A source that reads a Parquet file. Texera read CSV, JSONL, Arrow and plain text off disk but not Parquet, and converting one to CSV first loses what the file knew: the column written as an INTEGER came back as text for the schema to guess at again. The format states its own types in a footer, so this source infers nothing. It reads a row group at a time rather than the whole file, which is the thing a columnar format is chosen to avoid, and a column that is a group, a list or a map is refused by name instead of being dropped in silence. A timestamp is read with UTC arithmetic, matching what `ArrowUtils` means by a Texera TIMESTAMP: the count from the epoch lands on a wall clock, and no zone of the machine's own enters it. Reading it any other way would move the value and part the engine from the script the export writes. No new dependency. `parquet-hadoop` and `parquet-column` are already on the classpath under `iceberg-parquet`, and the reader takes a `LocalInputFile`, so none of Hadoop's own file plumbing is involved. ### Any related issues, documentation, discussions? Not part of #8325: nothing here makes a workflow exportable. It does ship standalone code, so it reads the trait that issue introduced. Closes #8511, the task this change is the whole of. ### How was this PR tested? Thirteen tests in the operator's own spec, over a Parquet file the spec writes: the columns the footer states, the values read back, a row that wrote no field at all, the timestamp that a zone could have moved, a nested column refused by name, a file that is not Parquet, and the Python the export emits with each scan window. The parity harness needs no per-operator code for it. A scan source is fixtured by the format it declares, so an encoder for `"Parquet"` is the whole of what it asks for, and that entry lands with the harness in #8364. With both in place the operator runs on the canonical table both ways and the two answers match. ### Was this PR authored or co-authored using generative AI tooling? Generated-by: Claude Code (Claude Opus 5) 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
