aarmens702-hub opened a new pull request, #76699: URL: https://github.com/apache/airflow/pull/76699
## Human Summary Following up on #76389, in the same file. Files saved from Excel as “CSV UTF-8”, or from PowerShell, start with an invisible byte-order mark (BOM). JSON files then fail the task with “Unexpected UTF-8 BOM”. CSVs still load, but the first column name has a hidden \ufeff stuck to the front. The operator has no encoding option, so users can’t work around it. This change decodes with utf-8-sig. That strips the BOM when there is one and otherwise works exactly like utf-8. ## AI Summary <details><summary>Click here</summary> Files saved by Excel's "CSV UTF-8" export or by Windows PowerShell start with a UTF-8 byte order mark. `_decode_text` decoded them as plain `utf-8`, which keeps the mark: - a `.json` file fails the task with `JSONDecodeError: Unexpected UTF-8 BOM (decode using utf-8-sig)`; - a `.csv` file's first column name becomes `id`, which is invisible in logs but not to the model. The operator has no `encoding` option, so users can't work around it. This decodes with `utf-8-sig`, which drops the mark when it's there and is otherwise identical to `utf-8`. Test: `test_utf8_byte_order_mark_is_not_part_of_the_content`, parametrized over `json`, `csv` and `txt`. All three fail on main and pass with the change. The rest of `test_file_analysis.py` passes (61 passed, 3 skipped). </details> --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes (please specify the tool below) Generated-by: Claude Code following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
