bziobrowski opened a new pull request, #15049:
URL: https://github.com/apache/pinot/pull/15049
PR improves performance and reduces temporary object generation in
`JSON_MATCH()` function calls by:
- reducing and/or delaying immutable to mutable conversion as much as
possible
- using immutable objects instead of allocating
- for != and NOT IN expressions - switching from OR-ing all bitmaps except
those for unwanted values to reading bitmap with all values for key and then
removing values present in bitmaps of unwanted values
- comparing dict ids instead of string values
Running the following query on `githubComplexTypeEvents` table with 2.5mil
records (same base data, but multiplied) and JSON index on payload column shows
(in async profiler/JFR) :
- response time reduction from ~ 500ms to about ~250ms
- allocation reduction from 400MiB to about 20MiB .
```sql
SELECT
COUNT(*) AS event_count,
SUM(payload.size) AS event_size,
SUM( CASE WHEN created_at_timestamp >= ago('PT3H') THEN 1 ELSE 0 END ) AS
transaction_count_3h,
SUM( CASE WHEN created_at_timestamp >= ago('PT3H') THEN
payload.distinct_size END ) AS transaction_sum_3h,
SUM( CASE WHEN created_at_timestamp >= ago('PT15M') THEN 1 END ) AS
transaction_count_15min,
SUM( CASE WHEN created_at_timestamp >= ago('PT15M') THEN
payload.distinct_size END ) AS transaction_sum_15min,
type
FROM githubComplexTypeEvents
WHERE created_at_timestamp between ago('P10000D') AND now()
AND payload.size > 0
AND JSON_MATCH(
payload,
'("$.ref" = ''refs/heads/master'' OR ("$.ref" = ''refs/heads/gh-pages''
AND ("$.commits[0].author.name" != ''chao1995'' OR "$.commits[0].author.name"
IS NOT NULL)))'
)
GROUP BY type
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
