Grant Henke created KUDU-2796: --------------------------------- Summary: Restore job mangles HDFS paths Key: KUDU-2796 URL: https://issues.apache.org/jira/browse/KUDU-2796 Project: Kudu Issue Type: Bug Affects Versions: 1.9.0 Reporter: Grant Henke Assignee: Grant Henke
It looks like there is an issue with certain HDFS deployments where converting to and from a Java and HDFS path can break the job. Here is a sample stack trace: {noformat} 19/04/23 10:43:24 ERROR yarn.ApplicationMaster: User class threw exception: org.apache.spark.sql.AnalysisException: Path does not exist: hdfs://ns1/ns1/kudu-backup-tests/default.loadgen_auto_0c24803e79b04c4eb010abaff6a290b9/1556039820974; org.apache.spark.sql.AnalysisException: Path does not exist: hdfs://ns1/ns1/kudu-backup-tests/default.loadgen_auto_0c24803e79b04c4eb010abaff6a290b9/1556039820974; at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$org$apache$spark$sql$execution$datasources$DataSource$$checkAndGlobPathIfNecessary$1.apply(DataSource.scala:557) at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$org$apache$spark$sql$execution$datasources$DataSource$$checkAndGlobPathIfNecessary$1.apply(DataSource.scala:544) at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241) at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241) at scala.collection.immutable.List.foreach(List.scala:392) at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:241) at scala.collection.immutable.List.flatMap(List.scala:355) at org.apache.spark.sql.execution.datasources.DataSource.org$apache$spark$sql$execution$datasources$DataSource$$checkAndGlobPathIfNecessary(DataSource.scala:544) at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:358) at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:223) at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:211) at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:178) at org.apache.kudu.backup.KuduRestore$$anonfun$run$1$$anonfun$apply$1.apply(KuduRestore.scala:79) at org.apache.kudu.backup.KuduRestore$$anonfun$run$1$$anonfun$apply$1.apply(KuduRestore.scala:58) at scala.collection.immutable.List.foreach(List.scala:392) at org.apache.kudu.backup.KuduRestore$$anonfun$run$1.apply(KuduRestore.scala:58) at org.apache.kudu.backup.KuduRestore$$anonfun$run$1.apply(KuduRestore.scala:55) at scala.collection.immutable.List.foreach(List.scala:392) at org.apache.kudu.backup.KuduRestore$.run(KuduRestore.scala:55) at org.apache.kudu.backup.KuduRestore$.main(KuduRestore.scala:166) at org.apache.kudu.backup.KuduRestore.main(KuduRestore.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:673){noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005)