LuciferYang commented on PR #44477:
URL: https://github.com/apache/spark/pull/44477#issuecomment-1869925312
@dongjoon-hyun @bjornjorgensen Synchronization:
1. The test failure appears to be due to incompatibility between the
metadata created by Ivy versions 2.5.1 and 2.5.2 in the `ivy.cache.dir`. This
can be resolved by using a separate `ivy.cache.dir` for the older versions of
Spark being tested. The current pull request has already been tested
successfully.
2. However, I think this issue is not just limited to testing. I conducted
additional functional tests on my local machine as follows:
- Prepared a Spark client using Ivy 2.5.2 with version 4.0.0-SNAPSHOT and
another Spark 3.x version, such as Spark 3.4.2, using Ivy 2.5.1.
- After clearing the `~/.ivy2` directory, submitted jobs in the order of
3.4.2, 4.0.0-SNAPSHOT with the config `spark.sql.hive.metastore.jars=maven`.
Both versions were able to run successfully.
- After clearing the `~/.ivy2` directory again, submitted jobs in the order
of 4.0.0-SNAPSHOT, 3.4.2 with the same config
`spark.sql.hive.metastore.jars=maven`. The execution of Spark 3.4.2 failed with
the following error:
```
[info] : java.lang.RuntimeException: problem during retrieve of
org.apache.spark#spark-submit-parent-8bd00540-3ae3-45c0-b8cb-adf54c547a85:
java.lang.RuntimeException: Multiple artifacts of the module log4j#log4j;1.2.17
are retrieved to the same file! Update the retrieve pattern to fix this error.
[info] at
org.apache.ivy.core.retrieve.RetrieveEngine.retrieve(RetrieveEngine.java:238)`
```
Therefore, it seems that upgrading to Ivy 2.5.2 can cause issues for end
users. If a client with Ivy 2.5.2 is used first and then a client with Ivy
2.5.1 is intended to be used again, it is necessary to specify a different
`ivy.cache.dir` or manually clear the `ivy.cache.dir` updated by Ivy 2.5.2 in
order to use it again properly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
