[ https://issues.apache.org/jira/browse/HIVE-8722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14251557#comment-14251557 ]
Rui Li commented on HIVE-8722: ------------------------------ I got this exception which also seems related: {noformat} 2014-12-18 12:25:18,399 INFO [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) - 14/12/18 12:25:18 DEBUG rdd.HadoopRDD: SplitLocationInfo and other new Hadoop classes are unavailable. Using the older Hadoop location info code. 2014-12-18 12:25:18,399 INFO [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) - java.lang.ClassNotFoundException: org.apache.hadoop.mapred.InputSplitWithLocationInfo 2014-12-18 12:25:18,399 INFO [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) - at java.net.URLClassLoader$1.run(URLClassLoader.java:366) 2014-12-18 12:25:18,399 INFO [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) - at java.net.URLClassLoader$1.run(URLClassLoader.java:355) 2014-12-18 12:25:18,399 INFO [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) - at java.security.AccessController.doPrivileged(Native Method) 2014-12-18 12:25:18,399 INFO [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) - at java.net.URLClassLoader.findClass(URLClassLoader.java:354) 2014-12-18 12:25:18,399 INFO [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) - at java.lang.ClassLoader.loadClass(ClassLoader.java:425) 2014-12-18 12:25:18,399 INFO [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) - at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) 2014-12-18 12:25:18,399 INFO [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) - at java.lang.ClassLoader.loadClass(ClassLoader.java:358) 2014-12-18 12:25:18,399 INFO [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) - at java.lang.Class.forName0(Native Method) 2014-12-18 12:25:18,399 INFO [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) - at java.lang.Class.forName(Class.java:190) 2014-12-18 12:25:18,400 INFO [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) - at org.apache.spark.rdd.HadoopRDD$SplitInfoReflections.<init>(HadoopRDD.scala:381) 2014-12-18 12:25:18,400 INFO [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) - at org.apache.spark.rdd.HadoopRDD$.liftedTree1$1(HadoopRDD.scala:391) 2014-12-18 12:25:18,400 INFO [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) - at org.apache.spark.rdd.HadoopRDD$.<init>(HadoopRDD.scala:390) 2014-12-18 12:25:18,400 INFO [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) - at org.apache.spark.rdd.HadoopRDD$.<clinit>(HadoopRDD.scala) 2014-12-18 12:25:18,400 INFO [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) - at org.apache.spark.rdd.HadoopRDD.getInputFormat(HadoopRDD.scala:179) 2014-12-18 12:25:18,400 INFO [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) - at org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:197) 2014-12-18 12:25:18,400 INFO [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) - at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:206) 2014-12-18 12:25:18,400 INFO [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) - at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:204) 2014-12-18 12:25:18,400 INFO [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) - at scala.Option.getOrElse(Option.scala:120) 2014-12-18 12:25:18,400 INFO [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) - at org.apache.spark.rdd.RDD.partitions(RDD.scala:204) 2014-12-18 12:25:18,400 INFO [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) - at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:32) 2014-12-18 12:25:18,400 INFO [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) - at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:206) 2014-12-18 12:25:18,400 INFO [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) - at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:204) 2014-12-18 12:25:18,400 INFO [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) - at scala.Option.getOrElse(Option.scala:120) 2014-12-18 12:25:18,400 INFO [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) - at org.apache.spark.rdd.RDD.partitions(RDD.scala:204) 2014-12-18 12:25:18,400 INFO [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) - at org.apache.spark.ShuffleDependency.<init>(Dependency.scala:79) 2014-12-18 12:25:18,400 INFO [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) - at org.apache.spark.rdd.ShuffledRDD.getDependencies(ShuffledRDD.scala:80) 2014-12-18 12:25:18,400 INFO [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) - at org.apache.spark.rdd.RDD$$anonfun$dependencies$2.apply(RDD.scala:193) 2014-12-18 12:25:18,401 INFO [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) - at org.apache.spark.rdd.RDD$$anonfun$dependencies$2.apply(RDD.scala:191) 2014-12-18 12:25:18,401 INFO [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) - at scala.Option.getOrElse(Option.scala:120) 2014-12-18 12:25:18,401 INFO [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) - at org.apache.spark.rdd.RDD.dependencies(RDD.scala:191) 2014-12-18 12:25:18,401 INFO [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) - at org.apache.spark.scheduler.DAGScheduler.visit$1(DAGScheduler.scala:301) 2014-12-18 12:25:18,401 INFO [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) - at org.apache.spark.scheduler.DAGScheduler.getParentStages(DAGScheduler.scala:313) 2014-12-18 12:25:18,401 INFO [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) - at org.apache.spark.scheduler.DAGScheduler.newStage(DAGScheduler.scala:247) 2014-12-18 12:25:18,401 INFO [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) - at org.apache.spark.scheduler.DAGScheduler.handleJobSubmitted(DAGScheduler.scala:734) 2014-12-18 12:25:18,401 INFO [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) - at org.apache.spark.scheduler.DAGSchedulerEventProcessActor$$anonfun$receive$2.applyOrElse(DAGScheduler.scala:1389) 2014-12-18 12:25:18,401 INFO [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) - at akka.actor.Actor$class.aroundReceive(Actor.scala:465) 2014-12-18 12:25:18,401 INFO [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) - at org.apache.spark.scheduler.DAGSchedulerEventProcessActor.aroundReceive(DAGScheduler.scala:1375) 2014-12-18 12:25:18,401 INFO [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) - at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516) 2014-12-18 12:25:18,401 INFO [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) - at akka.actor.ActorCell.invoke(ActorCell.scala:487) 2014-12-18 12:25:18,401 INFO [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) - at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238) 2014-12-18 12:25:18,401 INFO [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) - at akka.dispatch.Mailbox.run(Mailbox.scala:220) 2014-12-18 12:25:18,401 INFO [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) - at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393) 2014-12-18 12:25:18,401 INFO [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) - at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) 2014-12-18 12:25:18,401 INFO [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) - at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) 2014-12-18 12:25:18,401 INFO [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) - at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) 2014-12-18 12:25:18,402 INFO [stderr-redir-1]: client.SparkClientImpl (SparkClientImpl.java:run(435)) - at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) {noformat} > Enhance InputSplitShims to extend InputSplitWithLocationInfo [Spark Branch] > --------------------------------------------------------------------------- > > Key: HIVE-8722 > URL: https://issues.apache.org/jira/browse/HIVE-8722 > Project: Hive > Issue Type: Improvement > Reporter: Jimmy Xiang > > We got thie following exception in hive.log: > {noformat} > 2014-11-03 11:45:49,865 DEBUG rdd.HadoopRDD > (Logging.scala:logDebug(84)) - Failed to use InputSplitWithLocations. > java.lang.ClassCastException: Cannot cast > org.apache.hadoop.hive.ql.io.CombineHiveInputFormat$CombineHiveInputSplit > to org.apache.hadoop.mapred.InputSplitWithLocationInfo > at java.lang.Class.cast(Class.java:3094) > at > org.apache.spark.rdd.HadoopRDD.getPreferredLocations(HadoopRDD.scala:278) > at > org.apache.spark.rdd.RDD$$anonfun$preferredLocations$2.apply(RDD.scala:216) > at > org.apache.spark.rdd.RDD$$anonfun$preferredLocations$2.apply(RDD.scala:216) > at scala.Option.getOrElse(Option.scala:120) > at org.apache.spark.rdd.RDD.preferredLocations(RDD.scala:215) > at > org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$getPreferredLocsInternal(DAGScheduler.scala:1303) > at > org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$spark$scheduler$DAGScheduler$$getPreferredLocsInternal$2$$anonfun$apply$2.apply$mcVI$sp(DAGScheduler.scala:1313) > at > org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$spark$scheduler$DAGScheduler$$getPreferredLocsInternal$2$$anonfun$apply$2.apply(DAGScheduler.scala:1312) > {noformat} > My understanding is that the split location info helps Spark to execute tasks > more efficiently. This could help other execution engine too. So we should > consider to enhance InputSplitShim to implement InputSplitWithLocationInfo if > possible. -- This message was sent by Atlassian JIRA (v6.3.4#6332)