[ https://issues.apache.org/jira/browse/SPARK-3965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sean Owen resolved SPARK-3965. ------------------------------ Resolution: Duplicate > Spark assembly for hadoop2 contains avro-mapred for hadoop1 > ----------------------------------------------------------- > > Key: SPARK-3965 > URL: https://issues.apache.org/jira/browse/SPARK-3965 > Project: Spark > Issue Type: Bug > Components: Input/Output > Affects Versions: 1.0.2, 1.1.0, 1.2.0 > Environment: hadoop2, HDP2.1 > Reporter: David Jacot > > When building Spark assembly for hadoop2, org.apache.avro:avro-mapred for > hadoop1 is picked and added to the assembly which leads to following > exception at runtime. > {code} > java.lang.IncompatibleClassChangeError: Found interface > org.apache.hadoop.mapreduce.TaskAttemptContext, but class was expected > at > org.apache.avro.mapreduce.AvroKeyInputFormat.createRecordReader(AvroKeyInputFormat.java:47) > at > org.apache.spark.rdd.NewHadoopRDD$$anon$1.<init>(NewHadoopRDD.scala:111) > ... > {code} > The patch for SPARK-3039 works well at compile time but artefact's classifier > is not applied when assembly is built. I'm not a maven expert but I don't > think that classifiers are applied on transitive dependencies. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org