[
https://issues.apache.org/jira/browse/NIFI-16467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jakub Augustyniak updated NIFI-16467:
-------------------------------------
Description:
When a record reader infers the schema of JSON whose nested records differ in
shape at many levels, inference takes exponentially longer with depth. The time
roughly doubles with each nesting level. Rich text documents with deeply nested
lists produce exactly this structure, for example:
{code:java}
{"type":"doc","content":[{"type":"bulletList","content":[{"type":"listItem","content":[
{"type":"paragraph","content":[{"type":"text","text":"a"}]},
{"type":"bulletList","content":[{"type":"listItem","content":[ ... repeated N
times ... ]}]}
]}]}]}{code}
At every level, a {{content}} array holds a {{{}paragraph{}}}, whose children
have a {{text}} field, next to a nested {{{}bulletList{}}}, whose children have
a {{content}} field. Neither record type is wider than the other.
Observed with JsonTreeReader using Infer Schema, on a 1.7 MB array of 100
records, two of which contain a 38-level nested document:
- Inference did not finish within 5 minutes. In a real deployment it ran for
hours without completing.
- Either of the two deep documents alone is enough: it still didn't finish
after 60 s.
- With those documents removed, inference on the same input takes 26 ms.
- The processor thread can't be stopped, because inference doesn't respond to
interruption, and the same FlowFile is picked up again after every restart.
That blocks the queue indefinitely.
was:
When a record reader infers the schema of JSON whose nested records differ in
shape at many levels, inference takes exponentially longer with depth. The time
roughly doubles with each nesting level. Rich text documents with deeply nested
lists produce exactly this structure, for example:
{code:java}
{"type":"doc","content":[{"type":"bulletList","content":[{"type":"listItem","content":[
{"type":"paragraph","content":[{"type":"text","text":"a"}]},
{"type":"bulletList","content":[{"type":"listItem","content":[ ... repeated N
times ... ]}]}
]}]}]}{code}
At every level, a {{content}} array holds a {{{}paragraph{}}}, whose children
have a {{text}} field, next to a nested {{{}bulletList{}}}, whose children have
a {{content}} field. Neither record type is wider than the other.
Observed with JsonTreeReader using Infer Schema, on main:
- Inference on about 28 nesting levels takes about 96 s, and over 120 s at 30
levels.
- A 2 MB input with a 38-level document did not finish within 5 minutes; in a
real deployment it ran for hours.
- The processor thread can't be stopped, because inference doesn't respond to
interruption, and the same FlowFile is picked up again after every restart.
That blocks the queue indefinitely.
> Schema inference hangs on deeply nested JSON records
> ----------------------------------------------------
>
> Key: NIFI-16467
> URL: https://issues.apache.org/jira/browse/NIFI-16467
> Project: Apache NiFi
> Issue Type: Bug
> Reporter: Jakub Augustyniak
> Assignee: Jakub Augustyniak
> Priority: Major
>
> When a record reader infers the schema of JSON whose nested records differ in
> shape at many levels, inference takes exponentially longer with depth. The
> time roughly doubles with each nesting level. Rich text documents with deeply
> nested lists produce exactly this structure, for example:
> {code:java}
> {"type":"doc","content":[{"type":"bulletList","content":[{"type":"listItem","content":[
> {"type":"paragraph","content":[{"type":"text","text":"a"}]},
> {"type":"bulletList","content":[{"type":"listItem","content":[ ... repeated
> N times ... ]}]}
> ]}]}]}{code}
> At every level, a {{content}} array holds a {{{}paragraph{}}}, whose children
> have a {{text}} field, next to a nested {{{}bulletList{}}}, whose children
> have a {{content}} field. Neither record type is wider than the other.
> Observed with JsonTreeReader using Infer Schema, on a 1.7 MB array of 100
> records, two of which contain a 38-level nested document:
> - Inference did not finish within 5 minutes. In a real deployment it ran for
> hours without completing.
> - Either of the two deep documents alone is enough: it still didn't finish
> after 60 s.
> - With those documents removed, inference on the same input takes 26 ms.
> - The processor thread can't be stopped, because inference doesn't respond to
> interruption, and the same FlowFile is picked up again after every restart.
> That blocks the queue indefinitely.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
