[ 
https://issues.apache.org/jira/browse/SPARK-39146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17553828#comment-17553828
 ] 

Sean R. Owen commented on SPARK-39146:
--------------------------------------

Sounds good, do you want to open a PR?

> The singleton Jackson ObjectMapper should be preferred
> ------------------------------------------------------
>
>                 Key: SPARK-39146
>                 URL: https://issues.apache.org/jira/browse/SPARK-39146
>             Project: Spark
>          Issue Type: Improvement
>          Components: Spark Core, SQL
>    Affects Versions: 3.4.0
>            Reporter: Yang Jie
>            Priority: Minor
>
> I write a mirco-benchmark to test Jackson ObjectWriter read and write:
> [https://github.com/LuciferYang/spark/blob/objectMapper/sql/catalyst/src/test/scala/org/apache/spark/sql/connector/catalog/JacksonBenchmark.scala]
> and run this use GA:
>  
> {code:java}
> OpenJDK 64-Bit Server VM 1.8.0_332-b09 on Linux 5.13.0-1022-azure
> Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
> Test create ObjectMapper:                 Best Time(ms)   Avg Time(ms)   
> Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
> ------------------------------------------------------------------------------------------------------------------------
> Test create ObjectMapper                            648            652        
>    4          0.0       64819.0       1.0XOpenJDK 64-Bit Server VM 
> 1.8.0_332-b09 on Linux 5.13.0-1022-azure
> Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
> Test write map to json:                   Best Time(ms)   Avg Time(ms)   
> Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
> ------------------------------------------------------------------------------------------------------------------------
> Test Multiple                                      2116           2127        
>   15          0.0      211556.5       1.0X
> Test Single                                           4              4        
>    0          2.4         416.1     508.4XOpenJDK 64-Bit Server VM 
> 1.8.0_332-b09 on Linux 5.13.0-1022-azure
> Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
> Test read json to map:                    Best Time(ms)   Avg Time(ms)   
> Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
> ------------------------------------------------------------------------------------------------------------------------
> Test Multiple                                      8848           8867        
>   27          0.0      884776.2       1.0X
> Test Single   {code}
>  
>  
> From the test results, we should use singleton Jackson ObjectMapper, because 
> it seems expensive to new a ObjectMapper instance.
>  
> The following code in Spark not use singleton:
>  
> {code:java}
> common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java
> core/src/main/scala/org/apache/spark/status/api/v1/JacksonMessageWriter.scala
> core/src/main/scala/org/apache/spark/storage/DiskBlockManager.scala
> resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/SparkKubernetesClientFactory.scala
> sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/FileDataSourceV2.scala
> sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Utils.scala
> sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/RebaseDateTime.scala
>  {code}
> can find hotpath and fix them



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

Reply via email to