RushabhK commented on issue #9801:
URL:
https://github.com/apache/incubator-gluten/issues/9801#issuecomment-2929038203
Hello @JkSelf, I did try running with this fix in my environment. Although
now there are job failures with FileNotFoundException. Please find the error
logs below.
`Caused by: java.io.FileNotFoundException: listStatus(hadoopPath:
gs://<some_path>/.spark-staging-6815c37c-acfc-4167-95ba-cd9fefaf325a/_temporary/a6b6b0b2-bc38-4620-8ca2-18be8938fbc4/00/tasks/attempt_202505301505413719290422332255928_0002_m_000298_1958):
'gs://<some_path>/.spark-staging-6815c37c-acfc-4167-95ba-cd9fefaf325a/_temporary/a6b6b0b2-bc38-4620-8ca2-18be8938fbc4/00/tasks/attempt_202505301505413719290422332255928_0002_m_000298_1958'
does not exist.
at
com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystemBase.listStatus(GoogleHadoopFileSystemBase.java:937)
at org.apache.hadoop.fs.FileSystem.listStatusBatch(FileSystem.java:2066)
at
org.apache.hadoop.fs.FileSystem$DirListingIterator.<init>(FileSystem.java:2317)
at
org.apache.hadoop.fs.FileSystem.listStatusIterator(FileSystem.java:2361)
at
org.apache.hadoop.mapreduce.lib.output.committer.manifest.impl.ManifestStoreOperationsThroughFileSystem.listStatusIterator(ManifestStoreOperationsThroughFileSystem.java:126)
at
org.apache.hadoop.mapreduce.lib.output.committer.manifest.stages.AbstractJobOrTaskStage.lambda$listStatusIterator$5(AbstractJobOrTaskStage.java:488)
at
org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.invokeTrackingDuration(IOStatisticsBinding.java:547)
at
org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.lambda$trackDurationOfOperation$5(IOStatisticsBinding.java:528)
at
org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.trackDuration(IOStatisticsBinding.java:449)
at
org.apache.hadoop.mapreduce.lib.output.committer.manifest.stages.AbstractJobOrTaskStage.listStatusIterator(AbstractJobOrTaskStage.java:487)
at
org.apache.hadoop.mapreduce.lib.output.committer.manifest.stages.TaskAttemptScanDirectoryStage.scanDirectoryTree(TaskAttemptScanDirectoryStage.java:148)
at
org.apache.hadoop.mapreduce.lib.output.committer.manifest.stages.TaskAttemptScanDirectoryStage.executeStage(TaskAttemptScanDirectoryStage.java:77)
at
org.apache.hadoop.mapreduce.lib.output.committer.manifest.stages.TaskAttemptScanDirectoryStage.executeStage(TaskAttemptScanDirectoryStage.java:51)
at
org.apache.hadoop.mapreduce.lib.output.committer.manifest.stages.AbstractJobOrTaskStage.apply(AbstractJobOrTaskStage.java:210)
at
org.apache.hadoop.mapreduce.lib.output.committer.manifest.stages.CommitTaskStage.executeStage(CommitTaskStage.java:66)
at
org.apache.hadoop.mapreduce.lib.output.committer.manifest.stages.CommitTaskStage.executeStage(CommitTaskStage.java:41)
at
org.apache.hadoop.mapreduce.lib.output.committer.manifest.stages.AbstractJobOrTaskStage.apply(AbstractJobOrTaskStage.java:210)
at
org.apache.hadoop.mapreduce.lib.output.committer.manifest.ManifestCommitter.commitTask(ManifestCommitter.java:310)
at
org.apache.spark.mapred.SparkHadoopMapRedUtil$.$anonfun$commitTask$1(SparkHadoopMapRedUtil.scala:51)
at
scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
at org.apache.spark.util.Utils$.timeTakenMs(Utils.scala:552)
at
org.apache.spark.mapred.SparkHadoopMapRedUtil$.performCommit$1(SparkHadoopMapRedUtil.scala:51)
at
org.apache.spark.mapred.SparkHadoopMapRedUtil$.commitTask(SparkHadoopMapRedUtil.scala:78)
at
org.apache.spark.internal.io.HadoopMapReduceCommitProtocol.commitTask(HadoopMapReduceCommitProtocol.scala:401)
at
org.apache.spark.sql.execution.SparkWriteFilesCommitProtocol.$anonfun$commitTask$1(SparkWriteFilesCommitProtocol.scala:91)
at org.apache.spark.util.Utils$.timeTakenMs(Utils.scala:552)
at
org.apache.spark.sql.execution.SparkWriteFilesCommitProtocol.commitTask(SparkWriteFilesCommitProtocol.scala:91)
at
org.apache.spark.sql.execution.VeloxColumnarWriteFilesRDD.$anonfun$compute$2(VeloxColumnarWriteFilesExec.scala:218)
at
scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
at
org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1397)
at
org.apache.spark.sql.execution.VeloxColumnarWriteFilesRDD.compute(VeloxColumnarWriteFilesExec.scala:201)
... 13 more
Caused by: java.io.FileNotFoundException: Item not found:
gs://<some_path>/.spark-staging-6815c37c-acfc-4167-95ba-cd9fefaf325a/_temporary/a6b6b0b2-bc38-4620-8ca2-18be8938fbc4/00/tasks/attempt_202505301505413719290422332255928_0002_m_000298_1958
at
com.google.cloud.hadoop.repackaged.gcs.com.google.cloud.hadoop.gcsio.GoogleCloudStorageFileSystem.listFileInfo(GoogleCloudStorageFileSystem.java:1052)
at
com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystemBase.listStatus(GoogleHadoopFileSystemBase.java:928)
... 43 more
Caused by: org.apache.spark.SparkException: Task failed while writing rows
to staging path:
gs://<some_path>/.spark-staging-6815c37c-acfc-4167-95ba-cd9fefaf325a, output
path: gs://<some_path>/.spark-staging-6815c37c-acfc-4167-95ba-cd9fefaf325a`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
