[ https://issues.apache.org/jira/browse/SPARK-41045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Apache Spark reassigned SPARK-41045: ------------------------------------ Assignee: Apache Spark (was: Shixiong Zhu) > Pre-compute to eliminate ScalaReflection calls after deserializer is created > ---------------------------------------------------------------------------- > > Key: SPARK-41045 > URL: https://issues.apache.org/jira/browse/SPARK-41045 > Project: Spark > Issue Type: Improvement > Components: SQL > Affects Versions: 3.4.0 > Reporter: Shixiong Zhu > Assignee: Apache Spark > Priority: Major > > Currently when `ScalaReflection` returns a deserializer, for a few complex > types, such as array, map, udt, etc, it creates functions that may still > touch `ScalaReflection` after the deserializer is created. > `ScalaReflection` is a performance bottleneck for multiple threads as it > holds multiple global locks. We can refactor > `ScalaReflection.deserializerFor` to pre-compute everything that needs to > touch `ScalaReflection` before creating the deserializer. After this, once > the deserializer is created, it can be reused by multiple threads without > touching `ScalaReflection.deserializerFor` any more and it will be much > faster. -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org