[jira] (SPARK-19407) defaultFS is used FileSystem.get instead of getting it from uri scheme

2017-01-31 Thread Amit Assudani (JIRA) 
Title: Message Title
 
 
 
 
 
 
 
 
 
 
  
 
 Amit Assudani commented on  SPARK-19407 
 
 
 
 
 
 
 
 
 
 

 
 
 
 
 
 
 
  Re: defaultFS is used FileSystem.get instead of getting it from uri scheme  
 
 
 
 
 
 
 
 
 
 
I'll do it over the weekend, you may assign it to me. I may need some logistics support as it will be my first PR. 
 
 
 
 
 
 
 
 
 
 
 
 

 
 Add Comment 
 
 
 
 
 
 
 
 
 
 

 
 
 
 
 
 
 
 
 
 

 This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d)

[jira] (SPARK-19407) defaultFS is used FileSystem.get instead of getting it from uri scheme

2017-01-30 Thread Shixiong Zhu (JIRA) 
Title: Message Title
 
 
 
 
 
 
 
 
 
 
  
 
 Shixiong Zhu updated an issue 
 
 
 
 
 
 
 
 
 
 

 
 
 
 
 
 
 
 Spark /  SPARK-19407 
 
 
 
  defaultFS is used FileSystem.get instead of getting it from uri scheme  
 
 
 
 
 
 
 
 
 

Change By:
 
 Shixiong Zhu 
 
 
 

Labels:
 
 checkpoint filesystem  starter  streaming 
 
 
 
 
 
 
 
 
 
 
 
 

 
 Add Comment 
 
 
 
 
 
 
 
 
 
 

 
 
 
 
 
 
 
 
 
 

 This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d)

[jira] (SPARK-19407) defaultFS is used FileSystem.get instead of getting it from uri scheme

2017-01-30 Thread Shixiong Zhu (JIRA) 
Title: Message Title
 
 
 
 
 
 
 
 
 
 
  
 
 Shixiong Zhu edited a comment on  SPARK-19407 
 
 
 
 
 
 
 
 
 
 

 
 
 
 
 
 
 
  Re: defaultFS is used FileSystem.get instead of getting it from uri scheme  
 
 
 
 
 
 
 
 
 
 Good catch. Do you want to submit a PR to fix  it ? It's just replacing "FileSystem.get(hadoopConf)" with "metadataFile.getFileSystem(hadoopConf)". 
 
 
 
 
 
 
 
 
 
 
 
 

 
 Add Comment 
 
 
 
 
 
 
 
 
 
 

 
 
 
 
 
 
 
 
 
 

 This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d)

[jira] (SPARK-19407) defaultFS is used FileSystem.get instead of getting it from uri scheme

2017-01-30 Thread Shixiong Zhu (JIRA) 
Title: Message Title
 
 
 
 
 
 
 
 
 
 
  
 
 Shixiong Zhu commented on  SPARK-19407 
 
 
 
 
 
 
 
 
 
 

 
 
 
 
 
 
 
  Re: defaultFS is used FileSystem.get instead of getting it from uri scheme  
 
 
 
 
 
 
 
 
 
 
Good catch. Do you want to submit a PR to fix? It's just replacing "FileSystem.get(hadoopConf)" with "metadataFile.getFileSystem(hadoopConf)". 
 
 
 
 
 
 
 
 
 
 
 
 

 
 Add Comment 
 
 
 
 
 
 
 
 
 
 

 
 
 
 
 
 
 
 
 
 

 This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d)

[jira] (SPARK-19407) defaultFS is used FileSystem.get instead of getting it from uri scheme

2017-01-30 Thread Amit Assudani (JIRA) 
Title: Message Title
 
 
 
 
 
 
 
 
 
 
  
 
 Amit Assudani created an issue 
 
 
 
 
 
 
 
 
 
 

 
 
 
 
 
 
 
 Spark /  SPARK-19407 
 
 
 
  defaultFS is used FileSystem.get instead of getting it from uri scheme  
 
 
 
 
 
 
 
 
 

Issue Type:
 
  Bug 
 
 
 

Affects Versions:
 

 2.1.0 
 
 
 

Assignee:
 

 Unassigned 
 
 
 

Components:
 

 Structured Streaming 
 
 
 

Created:
 

 30/Jan/17 22:24 
 
 
 

Labels:
 

 streaming checkpoint filesystem 
 
 
 

Priority:
 
  Major 
 
 
 

Reporter:
 
 Amit Assudani 
 
 
 
 
 
 
 
 
 
 
Caused by: java.lang.IllegalArgumentException: Wrong FS: s3a://**/checkpoint/7b2231a3-d845-4740-bfa3-681850e5987f/metadata, expected: file:/// at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:649) at org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:82) at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:606) at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:824) at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:601) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:421) at