[
https://issues.apache.org/jira/browse/KYLIN-5602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17739463#comment-17739463
]
huangsheng commented on KYLIN-5602:
-----------------------------------
h1. Design
Information to add #
Derived dimension query related:
##
Is it a query for derived dimensions? (Judging by hitting the index and
snapshot at the same time)
##
Is the Filter on the snapshot? (Get the filterColumns in the OLAPContext, and
then filter the filterColumns on snapShots)
##
How many parquet files did you finally scan? (already)
#
Segment Prunning Effect
##
How many segments are there in total, and how many segments are pruned out?
###
How many partition prunes are filtered?
###
How many segments pruned?
###
How many derived seg prnue filters?
Both Segment Pruning and Derived Segment Pruning were previously recorded as
{{dimension filters }}, separating the two as
{{Dimension filter }}and {{derived dimension filter}}
{code:java}
Segment Num: Before filter: 45, After time partition filter: 45, After
dimension filter: 45, After derived dimension filter: 11.// code placeholder
{code}
h1. Test Suggestion
#
Make a query for derived dimensions to see if the following three fields are
correct.
##
Derived query
###
Filter on dimension table
###
Filter is not on dimension table
##
Non-derived query
Is Dervied: xxx Snapshot Names: [xxx] Snapshot Filter: [xxx] #
View statistics
Check if the logs are consistent before and after the Segment Filter.
Before optimization
traceId: 97a07ee3-990a-5479-9309-70926d922e2f 2023-04-01T02:39:53,170 INFO
[derived_seg_prune] [Query eeecb641-43c2-2796-b8e7-ab21722a1320-219]
datasource.FilePruner : Segment Num: Before filter: 45, After time partition
filter: 45, After dimension filter: 11.
After optimization
traceId: 97a07ee3-990a-5479-9309-70926d922e2f 2023-04-01T02:39:53,170 INFO
[derived_seg_prune] [Query eeecb641-43c2-2796-b8e7-ab21722a1320-219]
datasource.FilePruner : Segment Num: Before filter: 45, After time partition
filter: 45, After dimension filter: 45, After derived dimension filter: 11.
{{}}
> Added logs related to Segment Pruning for derived dimension queries
> --------------------------------------------------------------------
>
> Key: KYLIN-5602
> URL: https://issues.apache.org/jira/browse/KYLIN-5602
> Project: Kylin
> Issue Type: New Feature
> Components: Query Engine
> Affects Versions: 5.0-alpha
> Reporter: huangsheng
> Assignee: huangsheng
> Priority: Major
> Fix For: 5.0-alpha
>
>
> Supplement Derived segment pruning related logs
> # Query whether Derived Segment Pruning is used, and filter a few
> # Add Metric to quantify the effect of Derived Segment Pruning, such as:
> counting the filtered segment size and counting the time of Bloom Filter
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
