HonestManXin opened a new pull request, #67228:
URL: https://github.com/apache/doris/pull/67228
In an online point query workload, a wide table with about 80 columns was
queried through nearly 10,000 client connections. The Backend
`LookupConnectionCache` usage on a single BE could grow to more than 10 GB.
The root cause is that short circuit point query contexts used random
UUIDs as Backend lookup cache keys. Even when different connections executed
the same query shape, each context generated a different cache ID. As a result,
the Backend could not reuse the pre-calculated lookup cache entries across
connections, and many duplicate cache entries were created. This problem
becomes much more obvious on wide tables because serialized descriptors and
output expressions are larger.
This PR derives the short circuit cache ID from the serialized query
context, including descriptor table, output expressions, and query options.
Therefore, identical query contexts are mapped to a bounded set of cache IDs
instead of always generating random IDs. To avoid concentrating a hot query on a
single Backend cache shard, a round-robin bucket is also mixed into the
hash, so hot identical queries can still be spread across multiple
`LookupConnectionCache` shards and reduce lock contention.
In the online workload above, the lookup cache usage on a single BE
dropped from more than 10 GB to about 270 MB.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
