xlsongc commented on code in PR #73926:
URL: https://github.com/apache/airflow/pull/73926#discussion_r4291172895
##########
providers/common/ai/src/airflow/providers/common/ai/operators/document_loader.py:
##########
@@ -450,21 +453,77 @@ def _parse_pdf_stream(self, stream: BinaryIO) ->
list[dict[str, Any]]:
documents.append({"text": text, "metadata": {"page_number":
page_num + 1}})
return documents
- def _parse_docx_stream(self, stream: BinaryIO) -> list[dict[str, Any]]:
+ def _parse_docx_stream(self, stream: BinaryIO, *, source_hint: str) ->
list[dict[str, Any]]:
"""
- Parse a DOCX stream into documents.
+ Parse a DOCX stream into a single document.
- Extracts paragraph text only. Tables, headers, footers, and footnotes
- are not included. For richer DOCX parsing, plug in a dedicated
- extraction tool (``Unstructured``, ``docling``) as a custom parser
- backend.
+ Paragraphs and tables in the document body are extracted in document
+ order. Each table row becomes one "| cell | cell |" line, and a nested
+ table is flattened into its cell. Headers, footers, footnotes, content
+ controls, text boxes, and pending tracked insertions are not included.
"""
try:
from docx import Document
+ from docx.oxml.exceptions import InvalidXmlError
+ from docx.table import Table
except ImportError as e:
raise AirflowOptionalProviderFeatureException(e)
doc = Document(stream)
- paragraphs = [p.text for p in doc.paragraphs if p.text.strip()]
- text = "\n\n".join(paragraphs)
+ blocks = []
+ for block in doc.iter_inner_content():
+ if isinstance(block, Table):
+ try:
+ rows = self._get_docx_table_rows(block, Table)
+ except (ValueError, InvalidXmlError, RecursionError) as e:
+ # Malformed table XML raises ValueError (a vMerge with no
cell above) or
+ # InvalidXmlError (a missing required attribute).
python-docx walks vertical
+ # merges up recursively, so one spanning about 1000 rows
raises RecursionError.
+ self.log.warning(
+ "Skipping a table in %s that python-docx could not
read: %r", source_hint, e
+ )
+ continue
+ text = "\n".join(f"| {' | '.join(cells)} |" for cells in rows)
+ else:
+ text = block.text
+ if text.strip():
+ blocks.append(text)
+ text = "\n\n".join(blocks)
return [{"text": text, "metadata": {}}]
+
+ def _get_docx_table_rows(self, table: Table, table_cls: type[Table]) ->
list[list[str]]:
+ # w:gridBefore, w:gridAfter and w:gridSpan come straight from the XML,
so the empty
+ # cells added for them are capped at the table's column count.
+ column_count = len(table.columns)
+
+ rows = []
+ for row in table.rows:
+ cells: list[str] = [""] * min(row.grid_cols_before, column_count)
+ previous_cell = None
+ for cell in row.cells:
Review Comment:
Done, building on your suggestion. A per-cell check can still be passed by
many cells that each fit on their own: a 39 KB file with 5 rows of 1000 cells,
each spanning all 20000 columns, built 100 million entries in about 6.5 s with
it. So the spans of each row are added up and compared with the column count
before `row.cells` is read, and that file is now skipped in 0.02 s. A span
below 1 is rejected too, since python-docx drops that cell together with its
text. `test_docx_unreadable_table_skipped_with_warning` gains
`gridspan-wider-than-table` (with `100_000_000`, asserting the warning),
`gridspan-zero` and `gridspans-together-wider-than-table`, and each fails
without the check. The `len(cells) < column_count` guard is removed, since the
sum check makes it unreachable.
The same check also skips a table whose row has more cells than `w:tblGrid`
declares. Word never writes one: I checked that it adds `w:gridAfter` when a
cell is deleted and widens `w:tblGrid` when one is added.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
