[
https://issues.apache.org/jira/browse/IGNITE-16315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aleksey Plekhanov updated IGNITE-16315:
---------------------------------------
Description:
For simple queries SQL engine most of the time spend in writing/reading query
start requests, which contains a lot of data. Nested instances of
{{ColocationGroup}} class contain assignments for each partition
({{{}List<List<UUID>>{}}}). Transferred size can be reduced if we compact
assignments somehow. The target colocation group from fragment description
contains redundant synthetic partitions, this also can be optimized.
Messages workflow is not optimal too. First, we send {{QueryStartRequest}} to
the remote nodes, remotes reply with the QueryStartResponse messages. After
that remotes send batches with data to the target nodes and receive acks for
each batch (acks required to limit inbox workload). When query execution is
finished, the node initiator sends {{QueryCloseMessage}} to the remote nodes,
remotes close queries, and sends back {{ErrorMessage}} to the initiator with
the {{ExecutionCancelledException}} error (which is ignored on the initiator
node).
Also, some other optimizations are possible. Proposed changes:
* Implement compaction of assignments of {{ColocationGroup}}
* Reduce target colocation group partitions count
* Fix caching of query plans (store original SQL as key, not parsed SQL, to
avoid redundant parsing)
* Change messages workflow (don't send ack messages for the last batch since
it is redundant, self-close remote queries, and don't send close query messages
to remote nodes, if we know for sure that it's already self-closed)
* Reduce count of {{RexBuilder}} creation on the execution phase (RexBuilder
is stateless and can be used one static instance)
* Reduce count of Calcite types creation on the execution phase
was:For simple queries SQL engine most of the time spend in writing/reading
query start requests, which contains a lot of data. Nested instances of
{{ColocationGroup}} class contain assignments for each partition
(\{{List<List<UUID>>}}). Transferred size can be reduced if we compact
assignments somehow. Also, the target colocation group from fragment
description contains redundant synthetic partitions, this also can be optimized.
> Calcite engine. Query start request contains a lot of data
> ----------------------------------------------------------
>
> Key: IGNITE-16315
> URL: https://issues.apache.org/jira/browse/IGNITE-16315
> Project: Ignite
> Issue Type: Improvement
> Reporter: Aleksey Plekhanov
> Assignee: Aleksey Plekhanov
> Priority: Major
> Labels: calcite2-required, calcite3-required
> Time Spent: 10m
> Remaining Estimate: 0h
>
> For simple queries SQL engine most of the time spend in writing/reading query
> start requests, which contains a lot of data. Nested instances of
> {{ColocationGroup}} class contain assignments for each partition
> ({{{}List<List<UUID>>{}}}). Transferred size can be reduced if we compact
> assignments somehow. The target colocation group from fragment description
> contains redundant synthetic partitions, this also can be optimized.
> Messages workflow is not optimal too. First, we send {{QueryStartRequest}} to
> the remote nodes, remotes reply with the QueryStartResponse messages. After
> that remotes send batches with data to the target nodes and receive acks for
> each batch (acks required to limit inbox workload). When query execution is
> finished, the node initiator sends {{QueryCloseMessage}} to the remote nodes,
> remotes close queries, and sends back {{ErrorMessage}} to the initiator with
> the {{ExecutionCancelledException}} error (which is ignored on the initiator
> node).
> Also, some other optimizations are possible. Proposed changes:
> * Implement compaction of assignments of {{ColocationGroup}}
> * Reduce target colocation group partitions count
> * Fix caching of query plans (store original SQL as key, not parsed SQL, to
> avoid redundant parsing)
> * Change messages workflow (don't send ack messages for the last batch since
> it is redundant, self-close remote queries, and don't send close query
> messages to remote nodes, if we know for sure that it's already self-closed)
> * Reduce count of {{RexBuilder}} creation on the execution phase (RexBuilder
> is stateless and can be used one static instance)
> * Reduce count of Calcite types creation on the execution phase
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
