Srinivasu Majeti created SPARK-47622:
----------------------------------------
Summary: Spark creates lot of tiny blocks for a single driverLog
file of size less than a dfs.blocksize
Key: SPARK-47622
URL: https://issues.apache.org/jira/browse/SPARK-47622
Project: Spark
Issue Type: Bug
Components: Spark Shell, Spark Submit
Affects Versions: 3.3.2
Reporter: Srinivasu Majeti
Upon reviewing the spark code, found that /user/spark/driverLogs are synced to
HDFS with hsync option as shown below.
{code:java}
hdfsStream.hsync(EnumSet.allOf(classOf[HdfsDataOutputStream.SyncFlag]))
Ref:
https://github.com/apache/spark/blob/a3c04ec1145662e4227d57cd953bffce96b8aad7/core/src/main/scala/org/apache/spark/util/logging/DriverLogger.scala{code}
As a result of this we see lot of tiny blocks getting synced every 5 seconds
with a new block. So we see a small HDFS file with 8 blocks as shown in the
below example.
{code:java}
[r...@ccycloud-3.smajeti.root.comops.site subdir0]# hdfs fsck
/user/spark/driverLogs/application_1710495774861_0002_driver.log
Connecting to namenode via
https://ccycloud-3.smajeti.root.comops.site:20102/fsck?ugi=hdfs&path=%2Fuser%2Fspark%2FdriverLogs%2Fapplication_1710495774861_0002_driver.log
FSCK started by hdfs (auth:KERBEROS_SSL) from /10.140.136.139 for path
/user/spark/driverLogs/application_1710495774861_0002_driver.log at Thu Mar 28
06:37:29 UTC 2024
Status: HEALTHY
Number of data-nodes: 4
Number of racks: 1
Total dirs: 0
Total symlinks: 0
Replicated Blocks:
Total size: 157574 B
Total files: 1
Total blocks (validated): 8 (avg. block size 19696 B)
Minimally replicated blocks: 8 (100.0 %) {code}
HdfsDataOutputStream.SyncFlag includes two flags UPDATE_LENGTH and END_BLOCK.
This has been an expected behavior for some time now and these flags will help
visualize the latest size of the HDFS Driver log file and to achieve that,
blocks are being ended/closed every 5-second sync. Every new sync will create a
new block for the same HDFS driver log file. This hysnc behavior was started
after fixing SPARK-29105 (SHS may delete driver log file of in-progress
application) 5 years back.
But this leaves Namenode to manage a lot of meta and becomes an overhead at
times in large clusters.
{code:java}
public static enum SyncFlag {
UPDATE_LENGTH,
END_BLOCK;
private SyncFlag() {
}
}
{code}
I don't see any configurable option to avoid this and avoiding this type of
hsync may have some side effects in spark as we saw SPARK-29105 bug.
We only have two options that needs manual intevention
1. Keep cleaning these Driver logs after some time
2. Keep merging these small block files into files with 128MB
Can we provide some customizable option to merge these blocks while closing the
spark-shell or during closing the driver log file?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
