[ https://issues.apache.org/jira/browse/ARROW-15583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Weston Pace resolved ARROW-15583. --------------------------------- Fix Version/s: 9.0.0 Resolution: Fixed Issue resolved by pull request 12852 [https://github.com/apache/arrow/pull/12852] > [C++] The Substrait consumer could potentially use a massive amount of RAM if > the producer uses large anchors > ------------------------------------------------------------------------------------------------------------- > > Key: ARROW-15583 > URL: https://issues.apache.org/jira/browse/ARROW-15583 > Project: Apache Arrow > Issue Type: Improvement > Components: C++ > Reporter: Weston Pace > Assignee: Sanjiban Sengupta > Priority: Major > Labels: pull-request-available, substrait > Fix For: 9.0.0 > > Time Spent: 5.5h > Remaining Estimate: 0h > > In Substrait a function is referred to by a "fully qualified name" which > consists of a URI and a function name. For example, the "add" function is > something like > {{https://github.com/substrait-io/substrait/blob/main/extensions/functions_arithmetic.yaml}}. > To avoid serializing these long names multiple times in the plan the > producer should pick an anchor value (an int32 in protobuf) and use that > everywhere (with a single lookup table at the top level of the plan). > To avoid map lookups the Arrow C++ consumer currently assumes that this > lookup table will be small enough it can be stored in a vector... > {noformat} > { > > "https://github.com/substrait-io/substrait/blob/main/extensions/functions_arithmetic.yaml#add", > > "https://github.com/substrait-io/substrait/blob/main/extensions/functions_arithmetic.yaml#subtract" > } > {noformat} > However, this sort of assumes that a plan is going to use numbers like 0, 1, > 2, ... N to create N anchors. There is nothing that prevents a consumer from > using whatever numbers it wants (e.g. a pointer value). If the producer uses > a really large anchor value then the C++ Substrait consumer will create a > lookup table with a lot of blank values. This could lead to a lot of wasted > memory. > We could try and request the Substrait spec enfoce small anchors or we could > change the extension set handling in the C++ consumer to use an unordered_map. -- This message was sent by Atlassian Jira (v8.20.7#820007)