[
https://issues.apache.org/jira/browse/SPARK-28841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16912719#comment-16912719
]
Shixiong Zhu edited comment on SPARK-28841 at 8/21/19 10:35 PM:
----------------------------------------------------------------
Okey, at least the following codes are legit but failing.
{code}
scala> :paste
// Entering paste mode (ctrl-D to finish)
import sys.process._
"touch /tmp/test/foo:bar".!!
spark.read.text("/tmp/test/foo?bar")
// Exiting paste mode, now interpreting.
java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path
in absolute URI: foo:bar
at org.apache.hadoop.fs.Path.initialize(Path.java:205)
at org.apache.hadoop.fs.Path.<init>(Path.java:171)
at org.apache.hadoop.fs.Path.<init>(Path.java:93)
at org.apache.hadoop.fs.Globber.glob(Globber.java:241)
at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:1657)
at org.apache.spark.deploy.SparkHadoopUtil.globPath(SparkHadoopUtil.scala:245)
at
org.apache.spark.deploy.SparkHadoopUtil.globPathIfNecessary(SparkHadoopUtil.scala:255)
at
org.apache.spark.sql.execution.datasources.DataSource$$anonfun$org$apache$spark$sql$execution$datasources$DataSource$$checkAndGlobPathIfNecessary$1.apply(DataSource.scala:549)
at
org.apache.spark.sql.execution.datasources.DataSource$$anonfun$org$apache$spark$sql$execution$datasources$DataSource$$checkAndGlobPathIfNecessary$1.apply(DataSource.scala:545)
at
scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
at
scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
at scala.collection.immutable.List.foreach(List.scala:392)
at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:241)
at scala.collection.immutable.List.flatMap(List.scala:355)
at
org.apache.spark.sql.execution.datasources.DataSource.org$apache$spark$sql$execution$datasources$DataSource$$checkAndGlobPathIfNecessary(DataSource.scala:545)
at
org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:359)
at
org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:223)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:211)
at org.apache.spark.sql.DataFrameReader.text(DataFrameReader.scala:711)
at org.apache.spark.sql.DataFrameReader.text(DataFrameReader.scala:683)
... 57 elided
Caused by: java.net.URISyntaxException: Relative path in absolute URI: foo:bar
at java.net.URI.checkPath(URI.java:1823)
at java.net.URI.<init>(URI.java:745)
at org.apache.hadoop.fs.Path.initialize(Path.java:202)
... 76 more
{code}
was (Author: zsxwing):
Okey, at least the following codes are legit but failing.
{code}
scala> :paste
// Entering paste mode (ctrl-D to finish)
import org.apache.hadoop.fs._
val fs = new Path("/").getFileSystem(spark.sessionState.newHadoopConf())
fs.create(new Path("/tmp/test/foo:bar")).close()
// Exiting paste mode, now interpreting.
java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path
in absolute URI: .foo:bar.crc
at org.apache.hadoop.fs.Path.initialize(Path.java:205)
at org.apache.hadoop.fs.Path.<init>(Path.java:171)
at org.apache.hadoop.fs.Path.<init>(Path.java:93)
at
org.apache.hadoop.fs.ChecksumFileSystem.getChecksumFile(ChecksumFileSystem.java:90)
at
org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.<init>(ChecksumFileSystem.java:402)
at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:461)
at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:440)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:911)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:892)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:789)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:778)
... 59 elided
Caused by: java.net.URISyntaxException: Relative path in absolute URI:
.foo:bar.crc
at java.net.URI.checkPath(URI.java:1823)
at java.net.URI.<init>(URI.java:745)
at org.apache.hadoop.fs.Path.initialize(Path.java:202)
... 69 more
{code}
> Spark cannot read a relative path containing ":"
> ------------------------------------------------
>
> Key: SPARK-28841
> URL: https://issues.apache.org/jira/browse/SPARK-28841
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.4.3
> Reporter: Shixiong Zhu
> Priority: Major
>
> Reproducer:
> {code}
> spark.read.parquet("test:test")
> {code}
> Error:
> {code}
> java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative
> path in absolute URI: test:test
> {code}
> This is actually a Hadoop issue since the error is thrown from "new
> Path("test:test")". I'm creating this ticket to see if we can work around
> this issue in Spark.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
