Hi, I would love to get your input on an issue we have stumbled upon, originally reported in https://issues.apache.org/jira/browse/SPARK-44003. The issue appears to be that:
> a combination of a large number of writers created via Spark's DynamicPartitionDataSingleWriter and the global Parquet MemoryManager synchronization bottleneck drastically reduces the performance by starving the writer threads Is this a known issue/"feature"? What would you suggest to workaround this issue? Cheers - Rafal
