[
https://issues.apache.org/jira/browse/CALCITE-3784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17034842#comment-17034842
]
Haisheng Yuan commented on CALCITE-3784:
----------------------------------------
A filter condition with 10,000 expressions, would use much more memory than
that. Suppose we have N expression, all of which are OR conditions. If it is
left deep tree, its overall digest will consume {{(N*(N-1)/2 + (N-1)) * 100}}
bytes memory. For RexCall with 10,000 expressions, it is almost 48 MB. With all
kinds of transformations and rulecall match digest, we would have more.
> RexCall toString digest gives OOM while huge expression is evaluated
> --------------------------------------------------------------------
>
> Key: CALCITE-3784
> URL: https://issues.apache.org/jira/browse/CALCITE-3784
> Project: Calcite
> Issue Type: Bug
> Components: core
> Reporter: Ravi Kapoor
> Priority: Critical
>
> I have some complex query which has tens of thousands of rex expressions and
> this expression is used in filter expression in the query.
> On creating a filter below code gets called:
> {code}
> public RelBuilder filter(Iterable<CorrelationId> variablesSet,
> Iterable<? extends RexNode> predicates) {
> final RexNode simplifiedPredicates =
> simplifier.simplifyFilterPredicates(predicates);
> if (simplifiedPredicates == null) {
> return empty();
> }
> {code}
> RexSimplify further adds the rexnode in the Set<RexNode> calling hashcode()
> internally which calls toString()
> Is there any way to avoid this computeDigest Call which creates complex
> string object and blows up the RAM about 14GB?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
