sqlContext.parquetFile doesn't work with s3n in version 1.3.0
Hi All, I just upgrade the system to use version 1.3.0, but then the sqlContext.parquetFile doesn't work with s3n. I have test the same code with 1.2.1 and it works. A simple test running in spark-shell: val parquetFile = sqlContext.parquetFile("""s3n:///test/2.parq """) java.lang.IllegalArgumentException: Wrong FS: s3n:///test/2.parq, expected: file:/// And same test work with spark-shell under 1.2.1 Regards, Shuai
Re: sqlContext.parquetFile doesn't work with s3n in version 1.3.0
See https://issues.apache.org/jira/browse/SPARK-6351 ~ Jonathan From: Shuai Zheng mailto:szheng.c...@gmail.com>> Date: Monday, March 16, 2015 at 11:46 AM To: "user@spark.apache.org<mailto:user@spark.apache.org>" mailto:user@spark.apache.org>> Subject: sqlContext.parquetFile doesn't work with s3n in version 1.3.0 Hi All, I just upgrade the system to use version 1.3.0, but then the sqlContext.parquetFile doesn’t work with s3n. I have test the same code with 1.2.1 and it works. A simple test running in spark-shell: val parquetFile = sqlContext.parquetFile("""s3n:///test/2.parq """) java.lang.IllegalArgumentException: Wrong FS: s3n:///test/2.parq, expected: file:/// And same test work with spark-shell under 1.2.1 Regards, Shuai
RE: sqlContext.parquetFile doesn't work with s3n in version 1.3.0
I see, but this is really a. big issue. anyway for me to work around? I try to set the fs.default.name = s3n, but looks like it doesn't work. I must upgrade to 1.3.0 because I face the package incompatible issue in 1.2.1, and if I must patch something, I rather go with latest version. Regards, Shuai From: Kelly, Jonathan [mailto:jonat...@amazon.com] Sent: Monday, March 16, 2015 2:54 PM To: Shuai Zheng; user@spark.apache.org Subject: Re: sqlContext.parquetFile doesn't work with s3n in version 1.3.0 See https://issues.apache.org/jira/browse/SPARK-6351 ~ Jonathan From: Shuai Zheng Date: Monday, March 16, 2015 at 11:46 AM To: "user@spark.apache.org" Subject: sqlContext.parquetFile doesn't work with s3n in version 1.3.0 Hi All, I just upgrade the system to use version 1.3.0, but then the sqlContext.parquetFile doesn't work with s3n. I have test the same code with 1.2.1 and it works. A simple test running in spark-shell: val parquetFile = sqlContext.parquetFile("""s3n:///test/2.parq """) java.lang.IllegalArgumentException: Wrong FS: s3n:///test/2.parq, expected: file:/// And same test work with spark-shell under 1.2.1 Regards, Shuai
Re: sqlContext.parquetFile doesn't work with s3n in version 1.3.0
We will be including this fix in Spark 1.3.1 which we hope to make in the next week or so. On Mon, Mar 16, 2015 at 12:01 PM, Shuai Zheng wrote: > I see, but this is really a… big issue. anyway for me to work around? I > try to set the fs.default.name = s3n, but looks like it doesn’t work. > > > > I must upgrade to 1.3.0 because I face the package incompatible issue in > 1.2.1, and if I must patch something, I rather go with latest version. > > > > Regards, > > > > Shuai > > > > *From:* Kelly, Jonathan [mailto:jonat...@amazon.com] > *Sent:* Monday, March 16, 2015 2:54 PM > *To:* Shuai Zheng; user@spark.apache.org > *Subject:* Re: sqlContext.parquetFile doesn't work with s3n in version > 1.3.0 > > > > See https://issues.apache.org/jira/browse/SPARK-6351 > > > > ~ Jonathan > > > > *From: *Shuai Zheng > *Date: *Monday, March 16, 2015 at 11:46 AM > *To: *"user@spark.apache.org" > *Subject: *sqlContext.parquetFile doesn't work with s3n in version 1.3.0 > > > > Hi All, > > > > I just upgrade the system to use version 1.3.0, but then the > sqlContext.parquetFile doesn’t work with s3n. I have test the same code > with 1.2.1 and it works. > > > > A simple test running in spark-shell: > > > > val parquetFile = sqlContext.parquetFile("""s3n:///test/2.parq """) > > java.lang.IllegalArgumentException: Wrong FS: s3n:///test/2.parq, > expected: file:/// > > > > And same test work with spark-shell under 1.2.1 > > > > Regards, > > > > Shuai >