[
https://issues.apache.org/jira/browse/CALCITE-7805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18130668#comment-18130668
]
Frank Chen commented on CALCITE-7805:
-------------------------------------
[~vladimirsitnikov] is asking the following question in the github comment:
[https://github.com/apache/calcite/pull/5284#issuecomment-6342314780]
> Could you also explain why Druid does not pass a large list of IDs to Calcite
> as a single array value? Even an array bound to a ? parameter is expanded by
> Druid's SqlParameterizerShuttle#createArrayLiteral into ARRAY[...] with one
> SqlLiteral per element, so planning still processes a million literals. A
> RexDynamicParam has a constant-size digest and needs no per-element
> validation or type inference.
Maybe we can discuss here.
A little more context: The druid SCALAR_IN_ARRAY(col, ARRAY[x,y,z,...])
operator was introduced by [~gian] via druid PR
[#16388|[https://github.com/apache/druid/pull/16388]]. The motivation behind it
is to address the large in filter performance which was raised CALCITE-3178
long time ago that Calcite turned the IN into a list of OR operators.
For this new operator, I did NOT know why it was not designed to accept a
RexDynamicParam, but I think the problem at that time was not the memory
allocation based on above context. And maybe Gian can can provide any context
or consideration for the design.
Another thing I can think of about the design is that rewriting IN into
SCALAR_IN_ARRAY(col, ARRAY[x,y,z,xxx]) is a more intuitive way for people to
understand and implement instead of using of RexDynamicParam.
When #16388 was done, the benchmark showed a significant improvement, that
might also explained that we stopped there. The memory allocation memory was
raised after CALCITE-5948 in 1.37.0 which was fixed by CALCITE-7782 a month ago.
Back to the proposal, based on current underlying implementation of Calcite, I
think the rewriting IN as SCALAR_IN_ARRAY(x, ARRAY[?]) by using RexDynamicParam
provides a new way for us to address this problem. It turns the problem into a
pure druid side work which does not depends on optimization at Calcite side. We
will try to see if it works.
> Reduce temporary object allocation when validating large string ARRAYs to
> lower GC pressure
> -------------------------------------------------------------------------------------------
>
> Key: CALCITE-7805
> URL: https://issues.apache.org/jira/browse/CALCITE-7805
> Project: Calcite
> Issue Type: Improvement
> Components: core
> Affects Versions: 1.42.0
> Reporter: Frank Chen
> Priority: Major
> Labels: pull-request-available
>
> [CALCITE-7782|https://issues.apache.org/jira/browse/CALCITE-7782] eliminated
> quadratic operand-list copying while validating large ARRAY and MAP
> constructors, allowing a 1,000,000-element string ARRAY to complete. Planning
> remains allocation-heavy: Druid's benchmark allocates approximately 15.4 GB
> and takes approximately 7.8 seconds for one million string literals.
> Allocation is cumulative bytes per operation, not retained or peak heap.
> h2. Reproducer
> {code:sql}
> SELECT string1
> FROM foo
> WHERE string1 IN ('1', '2', ..., '1000000')
> {code}
> Druid rewrites this predicate to:
> {code}
> SCALAR_IN_ARRAY(string1, ARRAY['1', '2', ..., '1000000'])
> {code}
> The benchmark is {{InPlanningBenchmark.queryStringInSqlPlanOnly}}.
> h2. Work items
> The remaining allocation sources are tracked independently:
> * [CALCITE-7806|https://issues.apache.org/jira/browse/CALCITE-7806] - Avoid
> resolving generated CAST calls during ARRAY validation.
> * [CALCITE-7807|https://issues.apache.org/jira/browse/CALCITE-7807] - Reuse
> unchanged SQL types during least-restrictive inference.
> * [CALCITE-7808|https://issues.apache.org/jira/browse/CALCITE-7808] - Cache
> recently created and decorated SQL types.
> * [CALCITE-7809|https://issues.apache.org/jira/browse/CALCITE-7809] - Avoid
> copying unchanged SQL and Rex operands.
> * [CALCITE-7810|https://issues.apache.org/jira/browse/CALCITE-7810] - Write
> Rex string literal digests directly.
> Combined prototypes reduced allocation by approximately 70% in this Druid
> workload. Each sub-task contains its isolated measurement and scope.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
