[
https://issues.apache.org/jira/browse/HADOOP-13618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yulei Li updated HADOOP-13618:
------------------------------
Attachment: HADOOP-13618.patch
I have tested both cases you mentioned using the patch, the error no more
occured. You can test the patch in your envrioment, but you should first apply
the patch in your source code, then build the openstack jar. If you can't do
that, please tell me, I will e-mail the jar package for you.
> IllegalArgumentException when accessing Swift object with name containing
> space character
> -----------------------------------------------------------------------------------------
>
> Key: HADOOP-13618
> URL: https://issues.apache.org/jira/browse/HADOOP-13618
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/swift
> Affects Versions: 2.6.0
> Environment: Linux EL6
> Reporter: Steve Yang
> Assignee: Yulei Li
> Attachments: HADOOP-13618.patch, avro_test.zip
>
>
> We are using Spark and hadoop-openstack-2.6.0.jar
> (compile('org.apache.hadoop:hadoop-openstack:2.6.0')) to access Oracle
> Storage Service which is Swift-based:
> DataFrame df =
> hiveCtx.read().format("com.databricks.spark.csv").option(...).load(objectName);
> When accessing a Swift URL like "swift://Linda.oracleswift/non-matching
> records.csv" where the object name "non-matching records.csv" contains a
> space character, the following exception is thrown:
> 2016-08-23 15:56:03 DEBUG SwiftNativeFileSystem:126 - SwiftFileSystem
> initialized
> java.lang.IllegalArgumentException: Illegal character in path at index 13:
> /non-matching records.csv
> at java.net.URI.create(URI.java:859)
> at
> org.apache.hadoop.fs.swift.util.SwiftObjectPath.<init>(SwiftObjectPath.java:59)
> at
> org.apache.hadoop.fs.swift.util.SwiftObjectPath.fromPath(SwiftObjectPath.java:183)
> at
> org.apache.hadoop.fs.swift.util.SwiftObjectPath.fromPath(SwiftObjectPath.java:145)
> at
> org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystemStore.toObjectPath(SwiftNativeFileSystemStore.java:434)
> at
> org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystemStore.getObjectMetadata(SwiftNativeFileSystemStore.java:211)
> at
> org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystemStore.getObjectMetadata(SwiftNativeFileSystemStore.java:181)
> at
> org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystem.getFileStatus(SwiftNativeFileSystem.java:173)
> at org.apache.hadoop.fs.Globber.getFileStatus(Globber.java:64)
> at org.apache.hadoop.fs.Globber.doGlob(Globber.java:272)
> at org.apache.hadoop.fs.Globber.glob(Globber.java:151)
> at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:1653)
> at
> org.apache.hadoop.mapred.FileInputFormat.singleThreadedListStatus(FileInputFormat.java:259)
> ...
> Apparently it is complaining about the space character. However, checking the
> debug messages earlier before this error is raised we can see:
> 2016-08-23 15:56:03 DEBUG SwiftNativeFileSystem:122 - Initializing
> SwiftNativeFileSystem against URI
> swift://Linda.oracleswift/non-matching%20records.csv and working dir
> swift://Linda.oracleswift/user/syang
> 2016-08-23 15:56:03 DEBUG RestClientBindings:141 - Filesystem
> swift://Linda.oracleswift/non-matching%20records.csv is using configuration
> keys fs.swift.service.oracleswift
> ...
> The space character has already been encoded into "%20" and so it seems the
> Swift URL enters into SwiftNativeFileSystem is properly encoded.
> Because of this error any Swift object with file name contains space
> character (and may be slash '/' character as well?) cannot be accessed.
> As an additional data point, if we first encode the object name("non-matching
> records.csv"=>"non-matching%20records.csv") before giving it to OpenStack
> Swift API, a different error is raised. This time somehow the path separator
> '/' after the container name 'Linda' got encoded by
> SwiftNativeFileSystemStore:
> 2016-08-23 10:56:41 DEBUG SwiftRestClient:1731 - Status code = 400
> 2016-08-23 10:56:41 DEBUG SwiftRestClient:1445 - Method HEAD on
> https://storage.oraclecorp.com/v1/Storage-dfisher/Linda%2Fnon-matching%20records.csv
> failed, status code: 400, status line: HTTP/1.1 400 Bad Request
> BadRequest: Bad request against
> https://storage.oraclecorp.com/v1/Storage-dfisher/Linda%2Fnon-matching%20records.csv
> HEAD
> https://storage.oraclecorp.com/v1/Storage-dfisher/Linda%2Fnon-matching%20records.csv
> => 400
> at
> org.apache.hadoop.fs.swift.http.SwiftRestClient.buildException(SwiftRestClient.java:1456)
> at
> org.apache.hadoop.fs.swift.http.SwiftRestClient.perform(SwiftRestClient.java:1403)
> at
> org.apache.hadoop.fs.swift.http.SwiftRestClient.headRequest(SwiftRestClient.java:1016)
> at
> org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystemStore.stat(SwiftNativeFileSystemStore.java:257)
> at
> org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystemStore.getObjectMetadata(SwiftNativeFileSystemStore.java:212)
> at
> org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystemStore.getObjectMetadata(SwiftNativeFileSystemStore.java:181)
> at
> org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystem.getFileStatus(SwiftNativeFileSystem.java:173)
> So here it always error out whether the Swift object name is URL-encoded or
> not.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
