camper42 commented on PR #17296:
URL: https://github.com/apache/iceberg/pull/17296#issuecomment-5087385147
We deployed Iceberg 1.11.0 with commit 2493bb7 from this PR cherry-picked
into our Spark runtime and observed a substantial improvement in a
production
Structured Streaming workload.
Environment/workload:
- Spark 3.5.8, Scala 2.12, Java 17
- Iceberg format-version 3, merge-on-read
- 102 concurrent streaming queries performing MERGE operations
- 20 executors, 4 cores each
- 16 GiB executor heap + 24 GiB memory overhead (40 GiB pod limit)
- spark.sql.shuffle.partitions=400
- Same application code, tables, and checkpoints before and after the
change
Representative observations:
Before the fix:
- Peak executor DirectPoolMemory: 15.43 GiB
- Peak container RSS: 33.19 GiB
- Peak working set: 33.58 GiB
- The application became unhealthy after about 7 minutes, with 54 failed
jobs,
314 failed tasks, executor losses, and subsequent shuffle fetch
failures.
After cherry-picking this PR:
- Peak executor DirectPoolMemory: 0.017 GiB (~17 MiB)
- Peak container RSS: 18.05 GiB
- Peak working set: 19.31 GiB
- After 17 minutes, 755 jobs had succeeded with no failed jobs or tasks.
- The executors had processed approximately 771 GiB of shuffle reads and
922 GiB of shuffle writes.
- JVM heap pressure remained comparable (15.34 GiB before vs 15.86 GiB
after),
indicating that the workload was not materially lighter.
This is a greater than 900x reduction in peak DirectPoolMemory. Container
memory usage can still approach the cgroup limit because of reclaimable
inactive file cache, but RSS and working set remain stable around 16–18
GiB,
with no observed OOMKills.
AI Disclosure
Model: GPT-5.6 Sol
Platform/Tool: Pi
Human Oversight: reviewed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
