[jira] [Assigned] (SPARK-41045) Pre-compute to eliminate ScalaReflection calls after deserializer is created
[ https://issues.apache.org/jira/browse/SPARK-41045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-41045: Assignee: Shixiong Zhu (was: Apache Spark) > Pre-compute to eliminate ScalaReflection calls after deserializer is created > > > Key: SPARK-41045 > URL: https://issues.apache.org/jira/browse/SPARK-41045 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.4.0 >Reporter: Shixiong Zhu >Assignee: Shixiong Zhu >Priority: Major > > Currently when `ScalaReflection` returns a deserializer, for a few complex > types, such as array, map, udt, etc, it creates functions that may still > touch `ScalaReflection` after the deserializer is created. > `ScalaReflection` is a performance bottleneck for multiple threads as it > holds multiple global locks. We can refactor > `ScalaReflection.deserializerFor` to pre-compute everything that needs to > touch `ScalaReflection` before creating the deserializer. After this, once > the deserializer is created, it can be reused by multiple threads without > touching `ScalaReflection.deserializerFor` any more and it will be much > faster. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-41045) Pre-compute to eliminate ScalaReflection calls after deserializer is created
[ https://issues.apache.org/jira/browse/SPARK-41045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-41045: Assignee: Apache Spark (was: Shixiong Zhu) > Pre-compute to eliminate ScalaReflection calls after deserializer is created > > > Key: SPARK-41045 > URL: https://issues.apache.org/jira/browse/SPARK-41045 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.4.0 >Reporter: Shixiong Zhu >Assignee: Apache Spark >Priority: Major > > Currently when `ScalaReflection` returns a deserializer, for a few complex > types, such as array, map, udt, etc, it creates functions that may still > touch `ScalaReflection` after the deserializer is created. > `ScalaReflection` is a performance bottleneck for multiple threads as it > holds multiple global locks. We can refactor > `ScalaReflection.deserializerFor` to pre-compute everything that needs to > touch `ScalaReflection` before creating the deserializer. After this, once > the deserializer is created, it can be reused by multiple threads without > touching `ScalaReflection.deserializerFor` any more and it will be much > faster. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-41045) Pre-compute to eliminate ScalaReflection calls after deserializer is created
[ https://issues.apache.org/jira/browse/SPARK-41045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu reassigned SPARK-41045: Assignee: Shixiong Zhu > Pre-compute to eliminate ScalaReflection calls after deserializer is created > > > Key: SPARK-41045 > URL: https://issues.apache.org/jira/browse/SPARK-41045 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.4.0 >Reporter: Shixiong Zhu >Assignee: Shixiong Zhu >Priority: Major > > Currently when `ScalaReflection` returns a deserializer, for a few complex > types, such as array, map, udt, etc, it creates functions that may still > touch `ScalaReflection` after the deserializer is created. > `ScalaReflection` is a performance bottleneck for multiple threads as it > holds multiple global locks. We can refactor > `ScalaReflection.deserializerFor` to pre-compute everything that needs to > touch `ScalaReflection` before creating the deserializer. After this, once > the deserializer is created, it can be reused by multiple threads without > touching `ScalaReflection.deserializerFor` any more and it will be much > faster. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org