Shixiong Zhu created SPARK-41045: ------------------------------------ Summary: Pre-compute to eliminate ScalaReflection calls after deserializer is created Key: SPARK-41045 URL: https://issues.apache.org/jira/browse/SPARK-41045 Project: Spark Issue Type: Improvement Components: SQL Affects Versions: 3.4.0 Reporter: Shixiong Zhu
Currently when `ScalaReflection` returns a deserializer, for a few complex types, such as array, map, udt, etc, it creates functions that may still touch `ScalaReflection` after the deserializer is created. `ScalaReflection` is a performance bottleneck for multiple threads as it holds multiple global locks. We can refactor `ScalaReflection.deserializerFor` to pre-compute everything that needs to touch `ScalaReflection` before creating the deserializer. After this, once the deserializer is created, it can be reused by multiple threads without touching `ScalaReflection.deserializerFor` any more and it will be much faster. -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org