zhengruifeng commented on PR #58278: URL: https://github.com/apache/spark/pull/58278#issuecomment-5420318267
Thanks for the detailed review. I reproduced the comparison with a CountVectorizer-like workload on Java 17: 262,144 String keys, 2 million lookups, and a 90% hit rate. | Representation | Lookup | Estimated heap | JavaSerializer + LZ4 payload | Compressed deserialization | | --- | ---: | ---: | ---: | ---: | | Scala immutable Map | 69.5 ns/op | 25.6 MiB | 3.37 MB | 127.7 ms | | Spark OpenHashMap | 38.3 ns/op | 17.5 MiB | 3.39 MB | 35.7 ms | | java.util.HashMap | 20.7 ns/op | 22.3 MiB | 2.50 MB | 67.6 ms | The checked-in OpenHashMapBenchmark agrees on lookup performance: java.util.HashMap is about 2.1x faster for random String-key lookup. OpenHashMap remains best for deserialized heap and deserialization speed, but java.util.HashMap gives the best result for this lookup-heavy use case while also producing the smallest default compressed broadcast payload. I will switch this PR to java.util.HashMap. That also removes the zero-sentinel contract and the new OpenHashMap Kryo-registration issue. I will hoist Broadcast.value out of the per-token loop and remove the OpenHashMap-specific helper and test, which addresses the sizing, offset-contract, placement, and helper-test comments. More broadly, these results suggest that we should revisit existing OpenHashMap/OpenHashSet call sites. The right replacement depends on the operation: JDK collections appear preferable for String-key insert/lookup workloads, while specialized primitive storage and changeValue-style aggregation can still favor the Spark collections. That should be evaluated separately per call site, including lookup/insert performance, heap footprint, serialization, and configured Kryo behavior. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
