[
https://issues.apache.org/jira/browse/KYLIN-5788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17835409#comment-17835409
]
pengfei.zhan commented on KYLIN-5788:
-------------------------------------
h1. Some Evidence
1) two building task
{panel}
*
segment [07-01,
08-01)：2b9a13e4-c9e1-7263-36a4-ac11dc0cd5a4-59bd4ae5-403d-4f49-8eb4-83b063ef23b8
=> says: job_m7
*
segment [08-01,
09-01)：2811946e-870e-5508-db80-a76cd1904926-59bd4ae5-403d-4f49-8eb4-83b063ef23b8
=> says: job_m8{panel}
2) Look again at when the construction of several versions of the dictionary
was completed (in chronological order)
job_m8: version_1692827967895 (no resize occurred, generated by dictionary
build process, new dictionary data available, bucket count 5859)
{panel}
* 2023-08-24T05:59:27,898 INFO [logger-thread-0] dict.NGlobalDictHDFSStore :
Commit from
hdfs://isilonidc1prod.starbucks.net:8020/user/User_ZP/kylin231/kylin/DV/dict/global_dict/DV_REPORT.ADS_ORDER_CUBE/ORDER_ID/working
to
hdfs://isilonidc1prod.starbucks.net:8020/user/User_ZP/kylin231/kylin/DV/dict/global_dict/DV_REPORT.ADS_ORDER_CUBE/ORDER_ID/version_1692827967895{panel}
job_m7: version_1692828374437 (generated by resize, bucket count expanded from
5859 to 5864)
{panel}
* 2023-08-24T06:06:14,439 INFO [logger-thread-0] dict.NGlobalDictHDFSStore :
Commit from
hdfs://isilonidc1prod.starbucks.net:8020/user/User_ZP/kylin231/kylin/DV/dict/global_dict/DV_REPORT.ADS_ORDER_CUBE/ORDER_ID/working
to
hdfs://isilonidc1prod.starbucks.net:8020/user/User_ZP/kylin231/kylin/DV/dict/global_dict/DV_REPORT.ADS_ORDER_CUBE/ORDER_ID/version_1692828374437
*
* 2023-08-24T06:06:14,444 INFO [logger-thread-0]
builder.DictionaryBuilderHelper : End building a global dictionary column for
ORDER_ID, need resize from 5859 to 5864{panel}
job_m7: version_1692828826204 (generated by dictionary build process, no new
dictionary data, bucket count 5864)
{panel}
* 2023-08-24T06:13:46,206 INFO [logger-thread-0] dict.NGlobalDictHDFSStore :
Commit from
hdfs://isilonidc1prod.starbucks.net:8020/user/User_ZP/kylin231/kylin/DV/dict/global_dict/DV_REPORT.ADS_ORDER_CUBE/ORDER_ID/working
to
hdfs://isilonidc1prod.starbucks.net:8020/user/User_ZP/kylin231/kylin/DV/dict/global_dict/DV_REPORT.ADS_ORDER_CUBE/ORDER_ID/version_1692828826204{panel}
3) Look at the log of the dictionary file fetched by the spark executor for
job_m8
Since the number of partitions in job_m8's flat table encode process is 5859,
its spark task id and dictionary bucket id are in the range of 0 - 5858, so
let's take the first, middle, and last task respectively.
(Dictionary version version_ is highlighted at the beginning of the directory
name.)
Beginning 0:
{panel}
* 2023-08-24T06:06:56,087 INFO [Executor task launch worker for task 0.0 in
stage 69.0 (TID 8625)] dict.NGlobalDictHDFSStore :
[bucketId:0][isForColumnEncoding:true] bucketFiles size: 1
* 2023-08-24T06:06:56,087 INFO [Executor task launch worker for task 0.0 in
stage 69.0 (TID 8625)] dict.NGlobalDictHDFSStore :
[bucketId:0][isForColumnEncoding:true] Listing dict file:
hdfs://isilonidc1prod.starbucks.net:8020/user/User_ZP/kylin231/kylin/DV/dict/global_dict/DV_REPORT.ADS_ORDER_CUBE/ORDER_ID/version_1692828374437/PREV_0,
file size: 7743123{panel}
Middle 3000:
{panel}
* 2023-08-24T06:10:08,930 INFO [Executor task launch worker for task 3000.0 in
stage 69.0 (TID 11625)] dict.NGlobalDictHDFSStore :
[bucketId:3000][isForColumnEncoding:true] bucketFiles size: 1
* 2023-08-24T06:10:08,930 INFO [Executor task launch worker for task 3000.0 in
stage 69.0 (TID 11625)] dict.NGlobalDictHDFSStore :
[bucketId:3000][isForColumnEncoding:true] Listing dict file:
hdfs://isilonidc1prod.starbucks.net:8020/user/User_ZP/kylin231/kylin/DV/dict/global_dict/DV_REPORT.ADS_ORDER_CUBE/ORDER_ID/version_1692828374437/PREV_3000,
file size: 7756093{panel}
Middle 5719:
{panel}
* 2023-08-24T06:13:41,968 INFO [Executor task launch worker for task 5719.0 in
stage 69.0 (TID 14344)] dict.NGlobalDictHDFSStore :
[bucketId:5719][isForColumnEncoding:true] bucketFiles size: 1
* 2023-08-24T06:13:41,968 INFO [Executor task launch worker for task 5719.0 in
stage 69.0 (TID 14344)] dict.NGlobalDictHDFSStore :
[bucketId:5719][isForColumnEncoding:true] Listing dict file:
hdfs://isilonidc1prod.starbucks.net:8020/user/User_ZP/kylin231/kylin/DV/dict/global_dict/DV_REPORT.ADS_ORDER_CUBE/ORDER_ID/version_1692828374437/PREV_5719,
file size: 7759167{panel}
Middle 5720:
{panel}
* 2023-08-24T06:13:48,124 INFO [Executor task launch worker for task 5720.0 in
stage 69.0 (TID 14345)] dict.NGlobalDictHDFSStore :
[bucketId:5720][isForColumnEncoding:true] bucketFiles size: 1
* 2023-08-24T06:13:48,124 INFO [Executor task launch worker for task 5720.0 in
stage 69.0 (TID 14345)] dict.NGlobalDictHDFSStore :
[bucketId:5720][isForColumnEncoding:true] Listing dict file:
hdfs://isilonidc1prod.starbucks.net:8020/user/User_ZP/kylin231/kylin/DV/dict/global_dict/DV_REPORT.ADS_ORDER_CUBE/ORDER_ID/version_1692828826204/PREV_5720,
file size: 7731073{panel}
End 5858:
{panel}
* 2023-08-24T06:14:18,030 INFO [Executor task launch worker for task 5858.0 in
stage 69.0 (TID 14483)] dict.NGlobalDictHDFSStore :
[bucketId:5858][isForColumnEncoding:true] bucketFiles size: 1
* 2023-08-24T06:14:18,030 INFO [Executor task launch worker for task 5858.0 in
stage 69.0 (TID 14483)] dict.NGlobalDictHDFSStore :
[bucketId:5858][isForColumnEncoding:true] Listing dict file:
hdfs://isilonidc1prod.starbucks.net:8020/user/User_ZP/kylin231/kylin/DV/dict/global_dict/DV_REPORT.ADS_ORDER_CUBE/ORDER_ID/version_1692828826204/PREV_5858,
file size: 7755919{panel}
4) Compare the time it took to build several versions of the dictionary with
the time it took to fetch each version of the dictionary file from the executor.
Based on the above logs, we can see that there are two demarcation points
(note: the logs we posted are executor logs for job_m8!). The following is the
cutoff point:
- In task 0, the dictionary version used is version_1692828374437 built by
job_m7, not version_1692827967895 built by job_m8, and there has been an
expansion between these two dictionary versions.
- Between 5719 and 5720, there is another split in the use of dictionary
versions, except that the 2 dictionary versions produced by job_m7 are used,
further validating our RC
> Enhance global dict on flat table encoding stage logging & retry
> ----------------------------------------------------------------
>
> Key: KYLIN-5788
> URL: https://issues.apache.org/jira/browse/KYLIN-5788
> Project: Kylin
> Issue Type: Bug
> Components: Job Engine
> Affects Versions: 5.0-beta
> Reporter: pengfei.zhan
> Assignee: pengfei.zhan
> Priority: Major
> Fix For: 5.0-beta
>
> Attachments: pic1.jpeg, pic2.jpeg, pic3.jpeg
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
