xlsongc opened a new issue, #77259:
URL: https://github.com/apache/airflow/issues/77259

   ### Description
   
   Follow-up to #73926, as agreed in review
   (https://github.com/apache/airflow/pull/73926#discussion_r4607050135).
   
   #73926 makes `DocumentLoaderOperator` read DOCX tables and caps each table 
at 1000 grid
   columns and about 1,000,000 rendered characters. Two gaps remain, both 
outside one table:
   
   1. **No document-wide limit.** The caps apply per table, so the output grows 
with the
      number of tables. A table of 8-row merges nested 6 deep renders to about 
786K
      characters, which is under the per-table cap. 100 copies of it fit in a 
40 KB file and
      produce 78.6M characters (+162 MB). Each extra copy adds about 44 bytes 
to the zip.
      This growth comes with table extraction in #73926; before it, tables were 
not read.
   2. **No decompressed-size check.** python-docx reads each zip part whole 
into memory before
      parsing, and nothing checks the part's size first. This was already true 
before #73926.
      A 93 KB DOCX whose `document.xml` is 40 MB of empty paragraphs took 58 s 
and +1.2 GB peak
      memory to load (python-docx 1.2.0).
   
   Proposed change:
   
   - A character budget for the whole document, **charged per cell rather than 
per row**.
     Today a cell holding many nested tables builds all of them before its row 
is counted, so
     the budget has to be charged as each cell's text (nested tables included) 
is produced,
     and extraction has to stop as soon as the budget is spent instead of 
building the output
     and checking it afterwards.
   - A check of the zip entries' uncompressed sizes (`ZipInfo.file_size`) 
before the file is
     handed to python-docx. CPython's `zipfile` stops reading an entry at its 
declared
     `file_size` and raises `BadZipFile` on a CRC mismatch, so a forged size 
cannot make it
     return more than declared. This still needs to be confirmed against the 
supported Python
     versions in the PR.
   - When a limit is exceeded, follow the existing skip-with-warning behaviour 
where possible
     (stop adding content and log a warning) rather than failing the whole 
file. The PR can
     settle the exact behaviour and default limits.
   
   Prior art: Apache POI's `ZipSecureFile` checks a minimum inflate ratio 
(0.01) and a maximum
   entry size, and caps extracted text at 10M characters by default.
   
   ### Use case/motivation
   
   `DocumentLoaderOperator` is often pointed at files uploaded by users or 
collected from
   outside sources. A small DOCX should not be able to take minutes of CPU and 
gigabytes of
   memory on a worker, or produce tens of millions of characters that are then 
sent for
   embedding. Under the security model, Dag authors are responsible for what 
they pass to
   third-party parsers, so this is hardening rather than a vulnerability. 
Built-in limits
   still make the operator safer to use with untrusted files.
   
   ### Related issues
   
   #73926
   
   
   ### Are you willing to submit a PR?
   
   - [x] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [x] I agree to follow this project's [Code of 
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to