Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/22179#discussion_r214479500 --- Diff: pom.xml --- @@ -1770,6 +1770,10 @@ <groupId>org.apache.hive</groupId> <artifactId>hive-storage-api</artifactId> </exclusion> + <exclusion> + <groupId> com.esotericsoftware</groupId> + <artifactId>kryo-shaded</artifactId> --- End diff -- @srowen In short, the current Spark always uses the same Kryo version for read/write `SearchArgument` and it's used only on runtime. 1. Old OrcFileFormat always uses `org.spark-project.hive:hive-exec:1.2.1.spark2` which uses the shaded one in `hive-exec`. - `com.esotericsoftware.kryo:kryo:2.21`. 2. New OrcFileFormat uses `org.apache.orc` which uses the one provided by Spark. - `com.esotericsoftware:kryo-shaded:3.0.3` (All Spark/Orc/Hive uses this version for now) 3. New OrcFileFormat (in this PR) uses `org.apache.orc` which uses the one provided by Spark. - `com.esotericsoftware:kryo-shaded:4.0.2` So, (1) is unchanged by this PR. (2) and (3) also doesn't use a mixed version of Kryo. So, it should be fine because Apache Spark doesn't allow a mixed Spark version(master and executor). BTW, during investigation, there was some performance issue in `createFilter`. I'll file a new JIRA for that.
--- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org