xlsongc opened a new issue, #77259: URL: https://github.com/apache/airflow/issues/77259
### Description Follow-up to #73926, as agreed in review (https://github.com/apache/airflow/pull/73926#discussion_r4607050135). #73926 makes `DocumentLoaderOperator` read DOCX tables and caps each table at 1000 grid columns and about 1,000,000 rendered characters. Two gaps remain, both outside one table: 1. **No document-wide limit.** The caps apply per table, so the output grows with the number of tables. A table of 8-row merges nested 6 deep renders to about 786K characters, which is under the per-table cap. 100 copies of it fit in a 40 KB file and produce 78.6M characters (+162 MB). Each extra copy adds about 44 bytes to the zip. This growth comes with table extraction in #73926; before it, tables were not read. 2. **No decompressed-size check.** python-docx reads each zip part whole into memory before parsing, and nothing checks the part's size first. This was already true before #73926. A 93 KB DOCX whose `document.xml` is 40 MB of empty paragraphs took 58 s and +1.2 GB peak memory to load (python-docx 1.2.0). Proposed change: - A character budget for the whole document, **charged per cell rather than per row**. Today a cell holding many nested tables builds all of them before its row is counted, so the budget has to be charged as each cell's text (nested tables included) is produced, and extraction has to stop as soon as the budget is spent instead of building the output and checking it afterwards. - A check of the zip entries' uncompressed sizes (`ZipInfo.file_size`) before the file is handed to python-docx. CPython's `zipfile` stops reading an entry at its declared `file_size` and raises `BadZipFile` on a CRC mismatch, so a forged size cannot make it return more than declared. This still needs to be confirmed against the supported Python versions in the PR. - When a limit is exceeded, follow the existing skip-with-warning behaviour where possible (stop adding content and log a warning) rather than failing the whole file. The PR can settle the exact behaviour and default limits. Prior art: Apache POI's `ZipSecureFile` checks a minimum inflate ratio (0.01) and a maximum entry size, and caps extracted text at 10M characters by default. ### Use case/motivation `DocumentLoaderOperator` is often pointed at files uploaded by users or collected from outside sources. A small DOCX should not be able to take minutes of CPU and gigabytes of memory on a worker, or produce tens of millions of characters that are then sent for embedding. Under the security model, Dag authors are responsible for what they pass to third-party parsers, so this is hardening rather than a vulnerability. Built-in limits still make the operator safer to use with untrusted files. ### Related issues #73926 ### Are you willing to submit a PR? - [x] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
