its-amann commented on code in PR #76699:
URL: https://github.com/apache/airflow/pull/76699#discussion_r4505576915
##########
providers/common/ai/src/airflow/providers/common/ai/utils/file_analysis.py:
##########
@@ -649,7 +649,9 @@ def _read_limited_bytes(handle: io.BufferedIOBase, *, path:
ObjectStoragePath, m
def _decode_text(data: bytes) -> str:
- return data.decode("utf-8", errors="replace")
+ # utf-8-sig drops the byte order mark Excel and Windows tools write, which
json.loads
+ # refuses and which would otherwise be read as part of the first CSV
column's name.
+ return data.decode("utf-8-sig", errors="replace")
Review Comment:
Windows PowerShell 5.1 `>` and `Out-File` write UTF-16LE with a BOM, so
those files still decode as `[\x00{\x00...` and json.loads fails. A UTF-8 file
can't start with 0xFF or 0xFE, so checking for the UTF-16 BOM first is safe:
```suggestion
if data.startswith((b"\xff\xfe", b"\xfe\xff")):
return data.decode("utf-16", errors="replace")
return data.decode("utf-8-sig", errors="replace")
```
pushed it with a test here if you want to cherry pick:
https://github.com/its-amann/airflow/commit/597af72178660a03418fa4502088f83033666408
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
