After upgrading from apache spark 2.1.1 to 2.2.0 our integration test fail with
an exception:
java.lang.IllegalAccessError: tried to access method
com.google.common.base.Stopwatch.<init>()V from class
org.apache.hadoop.mapred.FileInputFormat
at
org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:312)
at
org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:194)
Seems there is an incompatibility with hadoop artifacts and guava, more details
are described here:
https://stackoverflow.com/questions/36427291/illegalaccesserror-to-guavas-stopwatch-from-org-apache-hadoop-mapreduce-lib-inp
Advising maven to use more recent versions of affected artifacts solved this
issue.
Questions: Is this a reasonable workaround for deployment into production? Or
should we still stay on 2.1.1 until these dependencies are fixed in spark? Is
there a ticket referencing this issue?
