[GitHub] [spark] dongjoon-hyun commented on issue #26085: [SPARK-29434][Core] Improve the MapStatuses Serialization Performance

dongjoon-hyun commented on issue #26085: [SPARK-29434][Core] Improve the MapStatuses Serialization Performance URL: https://github.com/apache/spark/pull/26085#issuecomment-545034306 Got it. Thanks, @tgravescs . This is an

[GitHub] [spark] dongjoon-hyun commented on issue #26085: [SPARK-29434][Core] Improve the MapStatuses Serialization Performance

dongjoon-hyun commented on issue #26085: [SPARK-29434][Core] Improve the MapStatuses Serialization Performance URL: https://github.com/apache/spark/pull/26085#issuecomment-544772675 In this PR, ZSTD reduces the size like the following. If we choose `LZ4`, it seems to be a regression over

[GitHub] [spark] dongjoon-hyun commented on issue #26085: [SPARK-29434][Core] Improve the MapStatuses Serialization Performance

dongjoon-hyun commented on issue #26085: [SPARK-29434][Core] Improve the MapStatuses Serialization Performance URL: https://github.com/apache/spark/pull/26085#issuecomment-544590037 No. I don't think it's tried. To make it sure, let's ping @dbtsai . :)

[GitHub] [spark] dongjoon-hyun commented on issue #26085: [SPARK-29434][Core] Improve the MapStatuses Serialization Performance

dongjoon-hyun commented on issue #26085: [SPARK-29434][Core] Improve the MapStatuses Serialization Performance URL: https://github.com/apache/spark/pull/26085#issuecomment-544560483 That sounds like a reasonable idea. Could you make a JIRA and a PR for the configuration, @tgravescs ? I

[GitHub] [spark] dongjoon-hyun commented on issue #26085: [SPARK-29434][Core] Improve the MapStatuses Serialization Performance

dongjoon-hyun commented on issue #26085: [SPARK-29434][Core] Improve the MapStatuses Serialization Performance URL: https://github.com/apache/spark/pull/26085#issuecomment-544291723 Thank you, @dbtsai , @tgravescs , @viirya , @MaxGekk , @advancedxy . Merged to master.

[GitHub] [spark] dongjoon-hyun commented on issue #26085: [SPARK-29434][Core] Improve the MapStatuses Serialization Performance

dongjoon-hyun commented on issue #26085: [SPARK-29434][Core] Improve the MapStatuses Serialization Performance URL: https://github.com/apache/spark/pull/26085#issuecomment-543979549 Hi, @dbtsai . I created a PR to your branch. Could you review and merge the updated benchmark result? -

[GitHub] [spark] dongjoon-hyun commented on issue #26085: [SPARK-29434] [Core] Improve the MapStatuses Serialization Performance

dongjoon-hyun commented on issue #26085: [SPARK-29434] [Core] Improve the MapStatuses Serialization Performance URL: https://github.com/apache/spark/pull/26085#issuecomment-540880356 Thank you for pinging me, @dbtsai . I'll take a look tomorrow.