Jia Yu created SEDONA-494:
-----------------------------
Summary: Raster data source cannot write to HDFS
Key: SEDONA-494
URL: https://issues.apache.org/jira/browse/SEDONA-494
Project: Apache Sedona
Issue Type: Bug
Reporter: Jia Yu
h2. Reproduce
When run the following code
var df = spark.read.format("binaryFile").load("/user/spark/raster/input.tif")
df.write.format("raster").mode(org.apache.spark.sql.SaveMode.Overwrite).save("output")
Just a _SUCCESS file found in the path.
I can find tiff file created in HDFS audit.log , but there's not 'rename' cmd .
I can find "SparkHadoopMapRedUtil: No need to commit output of task because
needsTaskCommit=false: attempt_xxx
BasicWriteTaskStatsTracker: Expected 1 files, but only saw 0. This could be due
to the output format not writing empty files, or files being not immediately
visible in the filesystem." in executor log.
h2. Solution:
in "org.apache.spark.sql.sedona_sql.io.raster.RasterFileFormat.scala"
val out = hfs.create(new Path(Paths.get(savePath, new
Path(rasterFilePath).getName).toString))
=>
val out = hfs.create(new Path(savePath, new Path(rasterFilePath).getName))
will solve the problem
Paths.get should not be used in FileSystem implements
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
