[
https://issues.apache.org/jira/browse/NIFI-16467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jakub Augustyniak updated NIFI-16467:
-------------------------------------
Description:
When a record reader infers the schema of JSON whose nested records differ in
shape at many levels, inference takes exponentially longer with depth. The time
roughly doubles with each nesting level. Rich text documents with deeply nested
lists produce exactly this structure, for example:
{code:java}
{"type":"doc","content":[{"type":"bulletList","content":[{"type":"listItem","content":[
{"type":"paragraph","content":[{"type":"text","text":"a"}]},
{"type":"bulletList","content":[{"type":"listItem","content":[ ... repeated N
times ... ]}]}
]}]}]}{code}
At every level, a {{content}} array holds a {{{}paragraph{}}}, whose children
have a {{text}} field, next to a nested {{{}bulletList{}}}, whose children have
a {{content}} field. Neither record type is wider than the other.
Observed with JsonTreeReader using Infer Schema, on main:
- Inference on about 28 nesting levels takes about 96 s, and over 120 s at 30
levels.
- A 2 MB input with a 38-level document did not finish within 5 minutes; in a
real deployment it ran for hours.
- The processor thread can't be stopped, because inference doesn't respond to
interruption, and the same FlowFile is picked up again after every restart.
That blocks the queue indefinitely.
was:
When a record reader infers the schema of JSON whose nested records differ in
shape at many levels, inference takes exponentially longer with depth. The time
roughly doubles with each nesting level. Rich text documents with deeply nested
lists produce exactly this structure, for example:
{code:java}
{"type":"doc","content":[{"type":"bulletList","content":[{"type":"listItem","content":[
{"type":"paragraph","content":[{"type":"text","text":"a"}]},
{"type":"bulletList","content":[{"type":"listItem","content":[ ... repeated N
times ... ]}]}
]}]}]}{code}
At every level, a {{content}} array holds a {{{}paragraph{}}}, whose children
have a {{text}} field, next to a nested {{{}bulletList{}}}, whose children have
a {{content}} field. Neither record type is wider than the other.
Root cause: {{DataTypeUtils.mergeDataTypes}} calls {{getWiderType}} before it
checks whether both types are records (or arrays of records) that should be
merged. For two record types where neither is wider, {{getWiderRecordType}}
calls {{isRecordWider}} in both directions. Each call recurses into the nested
types, and at the next level both directions are checked again. After
{{getWiderType}} finds nothing, the method falls through to the RECORD and
ARRAY<RECORD> merge added in NIFI-15745 anyway, so the expensive check doesn't
change the result.
Observed with JsonTreeReader using Infer Schema, on main with NIFI-15745
included:
- Inference on about 28 nesting levels takes about 96 s, and over 120 s at 30
levels.
- A 2 MB input with a 38-level document did not finish within 5 minutes; in a
real deployment it ran for hours.
- The processor thread can't be stopped, because inference doesn't respond to
interruption, and the same FlowFile is picked up again after every restart.
That blocks the queue indefinitely.
> Schema inference hangs on deeply nested JSON records
> ----------------------------------------------------
>
> Key: NIFI-16467
> URL: https://issues.apache.org/jira/browse/NIFI-16467
> Project: Apache NiFi
> Issue Type: Bug
> Reporter: Jakub Augustyniak
> Assignee: Jakub Augustyniak
> Priority: Major
>
> When a record reader infers the schema of JSON whose nested records differ in
> shape at many levels, inference takes exponentially longer with depth. The
> time roughly doubles with each nesting level. Rich text documents with deeply
> nested lists produce exactly this structure, for example:
> {code:java}
> {"type":"doc","content":[{"type":"bulletList","content":[{"type":"listItem","content":[
> {"type":"paragraph","content":[{"type":"text","text":"a"}]},
> {"type":"bulletList","content":[{"type":"listItem","content":[ ... repeated
> N times ... ]}]}
> ]}]}]}{code}
> At every level, a {{content}} array holds a {{{}paragraph{}}}, whose children
> have a {{text}} field, next to a nested {{{}bulletList{}}}, whose children
> have a {{content}} field. Neither record type is wider than the other.
> Observed with JsonTreeReader using Infer Schema, on main:
> - Inference on about 28 nesting levels takes about 96 s, and over 120 s at
> 30 levels.
> - A 2 MB input with a 38-level document did not finish within 5 minutes; in
> a real deployment it ran for hours.
> - The processor thread can't be stopped, because inference doesn't respond
> to interruption, and the same FlowFile is picked up again after every
> restart. That blocks the queue indefinitely.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
