Wenchen Fan created SPARK-25317: ----------------------------------- Summary: MemoryBlock performance regression Key: SPARK-25317 URL: https://issues.apache.org/jira/browse/SPARK-25317 Project: Spark Issue Type: Bug Components: SQL Affects Versions: 2.4.0 Reporter: Wenchen Fan
There is a performance regression when calculating hash code for UTF8String: {code} test("hashing") { import org.apache.spark.unsafe.hash.Murmur3_x86_32 import org.apache.spark.unsafe.types.UTF8String val hasher = new Murmur3_x86_32(0) val str = UTF8String.fromString("b" * 10001) val numIter = 100000 val start = System.nanoTime for (i <- 0 until numIter) { Murmur3_x86_32.hashUTF8String(str, 0) Murmur3_x86_32.hashUTF8String(str, 0) Murmur3_x86_32.hashUTF8String(str, 0) Murmur3_x86_32.hashUTF8String(str, 0) Murmur3_x86_32.hashUTF8String(str, 0) Murmur3_x86_32.hashUTF8String(str, 0) Murmur3_x86_32.hashUTF8String(str, 0) Murmur3_x86_32.hashUTF8String(str, 0) Murmur3_x86_32.hashUTF8String(str, 0) Murmur3_x86_32.hashUTF8String(str, 0) Murmur3_x86_32.hashUTF8String(str, 0) Murmur3_x86_32.hashUTF8String(str, 0) Murmur3_x86_32.hashUTF8String(str, 0) Murmur3_x86_32.hashUTF8String(str, 0) Murmur3_x86_32.hashUTF8String(str, 0) Murmur3_x86_32.hashUTF8String(str, 0) Murmur3_x86_32.hashUTF8String(str, 0) Murmur3_x86_32.hashUTF8String(str, 0) Murmur3_x86_32.hashUTF8String(str, 0) Murmur3_x86_32.hashUTF8String(str, 0) Murmur3_x86_32.hashUTF8String(str, 0) Murmur3_x86_32.hashUTF8String(str, 0) Murmur3_x86_32.hashUTF8String(str, 0) Murmur3_x86_32.hashUTF8String(str, 0) Murmur3_x86_32.hashUTF8String(str, 0) Murmur3_x86_32.hashUTF8String(str, 0) Murmur3_x86_32.hashUTF8String(str, 0) Murmur3_x86_32.hashUTF8String(str, 0) Murmur3_x86_32.hashUTF8String(str, 0) Murmur3_x86_32.hashUTF8String(str, 0) } val duration = (System.nanoTime() - start) / 1000 / numIter println(s"duration $duration us") } {code} To run this test in 2.3, we need to add {code} public static int hashUTF8String(UTF8String str, int seed) { return hashUnsafeBytes(str.getBaseObject(), str.getBaseOffset(), str.numBytes(), seed); } {code} to `Murmur3_x86_32` In my laptop, the result for master vs 2.3 is: 120 us vs 40 us -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org