Alexander created KYLIN-5985:
--------------------------------
Summary: Spark build indexes persormance issue. Many
pauses/freezes in spark UI.
Key: KYLIN-5985
URL: https://issues.apache.org/jira/browse/KYLIN-5985
Project: Kylin
Issue Type: Bug
Components: Job Engine
Affects Versions: 5.0.0
Environment: Rocky linux 8.
Hadoop - Bigtop 3.3.0 distribution.
Reporter: Alexander
Attachments: SparkUI.jpeg,
eventLogs-application_1738068922293_0056.zip,
image-2025-01-29-12-40-18-326.png, image-2025-01-29-12-40-58-725.png,
image-2025-01-29-12-46-03-795.png, image-2025-01-29-12-46-50-561.png,
ns3246587.ip-57-128-229.eu_7070_job_2025_01_29_09_27_19_862C85.zip
Spark build job freezes and a lot of time spend.
Build job works 88,94 minutes.
!image-2025-01-29-12-40-18-326.png!
Loads a model with only 71,734 rows and 98 indexes.
!image-2025-01-29-12-40-58-725.png!
Builded segment size 1.46mb
!image-2025-01-29-12-46-03-795.png!
With next parameters
|spark.driver.memory|6G|
|spark.driver.memoryOverhead|2G|
|spark.executor.instances|6|
|spark.executor.memory|6G|
|spark.executor.memoryOverhead|2G|
in spark UI we have a lot of pauses/freezer. It looks like nothing happend at
this pauses. No any tasks on executors.
!image-2025-01-29-12-46-50-561.png!
Full screen of a spark job - SparkUI.jpeg.
Diagnostic information in attachment.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
