The GitHub Actions job "Benchmarks" on texera.git/feat/parquet-source has 
failed.
Run started by GitHub user kz930 (triggered by kz930).

Head commit for run:
fcd65b6a9453a9fbe212fa8c35aef78229c15baa / kary zheng <[email protected]>
fix(operator): read every column a Parquet footer states, index or not

A file pandas wrote from a frame keyed by one of its columns records that
in the footer, and pandas reads those columns back as the frame's index
rather than as columns. The executor reads the columns the footer states
and has no notion of an index, so a file written from a frame keyed by
`customer_id` kept that column on the one side and dropped it on the
other, where a projection naming it raised.

The footer is handed back to the reader stripped of its metadata, so
pandas reads every column the file holds, under the name the footer gives
it and in the order it states them. Restoring the index afterwards would
not do: an unnamed one comes back as `index` where the file calls it
`__index_level_0__`, and it lands first where the footer puts it last.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

Report URL: https://github.com/apache/texera/actions/runs/35575731525

With regards,
GitHub Actions via GitBox

Reply via email to