morningman opened a new pull request #4885:
URL: https://github.com/apache/incubator-doris/pull/4885
## Proposed changes
This CL mainly changes:
1. Send and execute Fragments in batch
In the previous implementation, a SQL execution plan may generate
multiple Fragments.
Even if these Fragments are sent to the same BE node, they are sent
separately in multiple RPCs.
These Fragments will have some common information, such as DescriptorTable,
QueryGlobals, etc.
Sending one by one through RPC will repeatedly DeSer these information,
resulting in unnecessary overhead.
After the modification, the Coordinator merges multiple Fragments of
same BE and sends them in one RPC,
and extracts common information to avoid multiple repeated DeSer operations.
At the same time,
the Fragment execution part on the BE side is modified, multiple Fragments
are executed in one request,
and the common structure is avoided constructing repeatedly.
In the local single-node test, a SQL which generates 300 fragments, the
execution time is reduced from
7 seconds to 2 seconds. The main reduction part is the receiving and
executing Fragment part on the BE side.
The remaining time is mainly spent on query planning on the FE side, which
will be improved in next PR.
2. Add the time-consuming part of FE logic in Profile
Including SQL analysis, planning, Fragment scheduling and sending on the
FE side, and the time to fetch data.
## Types of changes
- [x] Code refactor (Modify the code structure, format the code, etc...)
## Checklist
- [x] I have create an issue on (Fix #4656), and have described the
bug/feature there in detail
## Further comments
The query planner is still time-consuming when processing complex SQL. It
will be optimized later.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
