PG1204 opened a new issue, #8962:
URL: https://github.com/apache/texera/issues/8962
### Task Summary
`_read_binary_value` turns a cell into bytes with no limit on the cell's
size, while a remote fetch of the same payload is capped at 25 MiB. Apply an
equivalent bound to column values.
### Context
`_read_binary_value` (`HuggingFaceCodegenBase.scala`, lines 980-997)
processes the whole string once per row:
val = str(value).strip()
if not val:
return None
if self._looks_like_html(val):
return self._html_to_image_bytes(val)
if val.startswith("data:"):
_, encoded = val.split(",", 1)
return base64.b64decode(encoded)
...
return base64.b64decode(val, validate=True)
Every branch scales with the cell: `_looks_like_html` and
`_html_to_image_bytes` run regexes and hand parsers over it, a `data:` value is
base64-decoded in full (allocating roughly three quarters of its length again),
and otherwise the value is decoded with `validate=True` and UTF-8 encoded on
failure.
Remote input is already bounded. `_fetch_remote_url` caps a download at
`MAX_REMOTE_FETCH_BYTES` (25 MiB, line 131) and aborts past it with a clear
error. A value arriving through the table has no equivalent limit, so the same
payload is refused over HTTP and accepted from a column. A column holding a
very large string, such as an inlined image or the output of an upstream
operator, drives per-row CPU and memory with no ceiling.
This is resource use rather than correctness: the operator works on small
inputs and degrades on large ones, paid per row.
### Proposed change
Bound the cell before any parsing. Compare the value's length against a cap
and raise the same style of clear error when it is exceeded. Reusing
`MAX_REMOTE_FETCH_BYTES` would keep column and remote input consistent and
avoids introducing a second, unrelated limit.
### Required test
Add a test asserting the generated Python checks the value's size before
`_looks_like_html` and the decode branches, and that an oversized value raises
rather than being processed.
### Task Type
- [ ] Refactor / Cleanup
- [ ] DevOps / Deployment / CI
- [ ] Testing / QA
- [ ] Documentation
- [x] Performance
- [ ] Other
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
