[
https://issues.apache.org/jira/browse/KYLIN-5985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexander updated KYLIN-5985:
-----------------------------
Attachment: learn_kylin_model_metadata_build_diagnostic.zip
> Spark build indexes persormance issue. Many pauses/freezes in spark UI.
> -----------------------------------------------------------------------
>
> Key: KYLIN-5985
> URL: https://issues.apache.org/jira/browse/KYLIN-5985
> Project: Kylin
> Issue Type: Bug
> Components: Job Engine
> Affects Versions: 5.0.0
> Environment: Rocky linux 8.
> Hadoop - Bigtop 3.3.0 distribution.
> Reporter: Alexander
> Priority: Blocker
> Attachments: SparkUI.jpeg, evenLog_print.pdf,
> eventLogs-application_1737114540896_0916.zip,
> eventLogs-application_1738068922293_0056.zip,
> image-2025-01-29-12-40-18-326.png, image-2025-01-29-12-40-58-725.png,
> image-2025-01-29-12-46-03-795.png, image-2025-01-29-12-46-50-561.png,
> image-2025-01-29-12-52-20-907.png, image-2025-01-29-12-53-01-478.png,
> image-2025-02-17-12-48-56-969.png, image-2025-02-17-13-28-55-544.png,
> image-2025-02-17-13-30-48-044.png, learn_kylin_model_metadata.zip,
> learn_kylin_model_metadata_build_diagnostic.zip,
> ns3246587.ip-57-128-229.eu_7070_job_2025_01_29_09_27_19_862C85.zip
>
>
> Spark build job freezes and a lot of time spend.
> Build job works 88,94 minutes.
> !image-2025-01-29-12-40-18-326.png!
> Loads a model with only 71,734 rows and 97 indexes.
> !image-2025-01-29-12-52-20-907.png!
>
> Builded segment size 1.46mb
> !image-2025-01-29-12-53-01-478.png!
> With next parameters
> |spark.driver.memory|6G|
> |spark.driver.memoryOverhead|2G|
> |spark.executor.instances|6|
> |spark.executor.memory|6G|
> |spark.executor.memoryOverhead|2G|
>
> in spark UI we have a lot of pauses/freezes. It looks like nothing happend at
> this pauses. No any tasks on executors.
> !image-2025-01-29-12-46-50-561.png!
> Full screen of a spark job - SparkUI.jpeg.
> Diagnostic information in attachment.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
