[
https://issues.apache.org/jira/browse/FLINK-41439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Moritz Manner closed FLINK-41439.
---------------------------------
Resolution: Won't Fix
Closing this for now. It only happens for a variant with a long key in a very
large array, close to the 16 MiB size limit, and we don’t expect that in
practice. Other string functions like REPEAT and JSON_STRING have no output
limit either. We can reopen this if it comes up.
> XML_STRING can run out of memory for a variant within the size limit
> --------------------------------------------------------------------
>
> Key: FLINK-41439
> URL: https://issues.apache.org/jira/browse/FLINK-41439
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Runtime
> Reporter: Moritz Manner
> Priority: Major
> Labels: pull-request-available
>
> The output of {{XML_STRING}} isn’t bounded by the size of the input variant.
> A variant stores each key once, but {{XML_STRING}} writes the key name for
> every array item. For an array of nulls it also adds an {{xsi:nil}} attribute
> to each item. So a small variant can turn into a very large string.
> {{XmlStringFunction.eval}} rethrows an {{{}OutOfMemoryError{}}}. A single row
> can fail the TaskManager instead of the query failing with a
> {{{}TableRuntimeException{}}}.
> The variant and {{JSON_STRING}} write the key once per array, {{XML_STRING}}
> writes it once per item. The table shows the structure with a small example,
> and the measured sizes for a key of 1,000 characters and an array of nulls:
> ||Nulls||Variant||{{JSON_STRING}}||{{XML_STRING}}||
> |2, key {{{{kkk}}}} (example)|{{{"r":\{"kkk":[null,null]}}}}
> |{{{"r":\{"kkk":[null,null]}}}}|{{<r
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";>}}
> {{ <kkk xsi:nil="true"/>}}
> {{ <kkk xsi:nil="true"/>}}
> {{</r>}}|
> |10,000, key of 1,000 chars|31 KB|51,012 chars|10,180,061 chars (328x)|
> |3,000,000, key of 1,000 chars|12 MB|–|{{OutOfMemoryError}}|
> The last variant is below the variant size limit of 16 MiB. Depending on the
> heap, it fails with {{Java heap space}} or because the output exceeds the
> maximum length of a Java string:
>
>
> *How to reproduce:*
>
> {code:java}
> SELECT XML_STRING(PARSE_JSON(CONCAT('{"r":{"', REPEAT('k', 1000), '":[null',
> REPEAT(',null', 2999999), ']}}')));{code}
>
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
