sqlContext.parquetFile doesn't work with s3n in version 1.3.0
Hi All,
I just upgrade the system to use version 1.3.0, but then the
sqlContext.parquetFile doesn't work with s3n. I have test the same code with
1.2.1 and it works.
A simple test running in spark-shell:
val parquetFile = sqlContext.parquetFile("""s3n:///test/2.parq """)
java.lang.IllegalArgumentException: Wrong FS: s3n:///test/2.parq,
expected: file:///
And same test work with spark-shell under 1.2.1
Regards,
Shuai
Re: sqlContext.parquetFile doesn't work with s3n in version 1.3.0
See https://issues.apache.org/jira/browse/SPARK-6351
~ Jonathan
From: Shuai Zheng mailto:szheng.c...@gmail.com>>
Date: Monday, March 16, 2015 at 11:46 AM
To: "user@spark.apache.org<mailto:user@spark.apache.org>"
mailto:user@spark.apache.org>>
Subject: sqlContext.parquetFile doesn't work with s3n in version 1.3.0
Hi All,
I just upgrade the system to use version 1.3.0, but then the
sqlContext.parquetFile doesn’t work with s3n. I have test the same code with
1.2.1 and it works.
A simple test running in spark-shell:
val parquetFile = sqlContext.parquetFile("""s3n:///test/2.parq """)
java.lang.IllegalArgumentException: Wrong FS: s3n:///test/2.parq, expected:
file:///
And same test work with spark-shell under 1.2.1
Regards,
Shuai
RE: sqlContext.parquetFile doesn't work with s3n in version 1.3.0
I see, but this is really a. big issue. anyway for me to work around? I try
to set the fs.default.name = s3n, but looks like it doesn't work.
I must upgrade to 1.3.0 because I face the package incompatible issue in
1.2.1, and if I must patch something, I rather go with latest version.
Regards,
Shuai
From: Kelly, Jonathan [mailto:jonat...@amazon.com]
Sent: Monday, March 16, 2015 2:54 PM
To: Shuai Zheng; user@spark.apache.org
Subject: Re: sqlContext.parquetFile doesn't work with s3n in version 1.3.0
See https://issues.apache.org/jira/browse/SPARK-6351
~ Jonathan
From: Shuai Zheng
Date: Monday, March 16, 2015 at 11:46 AM
To: "user@spark.apache.org"
Subject: sqlContext.parquetFile doesn't work with s3n in version 1.3.0
Hi All,
I just upgrade the system to use version 1.3.0, but then the
sqlContext.parquetFile doesn't work with s3n. I have test the same code with
1.2.1 and it works.
A simple test running in spark-shell:
val parquetFile = sqlContext.parquetFile("""s3n:///test/2.parq """)
java.lang.IllegalArgumentException: Wrong FS: s3n:///test/2.parq,
expected: file:///
And same test work with spark-shell under 1.2.1
Regards,
Shuai
Re: sqlContext.parquetFile doesn't work with s3n in version 1.3.0
We will be including this fix in Spark 1.3.1 which we hope to make in the
next week or so.
On Mon, Mar 16, 2015 at 12:01 PM, Shuai Zheng wrote:
> I see, but this is really a… big issue. anyway for me to work around? I
> try to set the fs.default.name = s3n, but looks like it doesn’t work.
>
>
>
> I must upgrade to 1.3.0 because I face the package incompatible issue in
> 1.2.1, and if I must patch something, I rather go with latest version.
>
>
>
> Regards,
>
>
>
> Shuai
>
>
>
> *From:* Kelly, Jonathan [mailto:jonat...@amazon.com]
> *Sent:* Monday, March 16, 2015 2:54 PM
> *To:* Shuai Zheng; user@spark.apache.org
> *Subject:* Re: sqlContext.parquetFile doesn't work with s3n in version
> 1.3.0
>
>
>
> See https://issues.apache.org/jira/browse/SPARK-6351
>
>
>
> ~ Jonathan
>
>
>
> *From: *Shuai Zheng
> *Date: *Monday, March 16, 2015 at 11:46 AM
> *To: *"user@spark.apache.org"
> *Subject: *sqlContext.parquetFile doesn't work with s3n in version 1.3.0
>
>
>
> Hi All,
>
>
>
> I just upgrade the system to use version 1.3.0, but then the
> sqlContext.parquetFile doesn’t work with s3n. I have test the same code
> with 1.2.1 and it works.
>
>
>
> A simple test running in spark-shell:
>
>
>
> val parquetFile = sqlContext.parquetFile("""s3n:///test/2.parq """)
>
> java.lang.IllegalArgumentException: Wrong FS: s3n:///test/2.parq,
> expected: file:///
>
>
>
> And same test work with spark-shell under 1.2.1
>
>
>
> Regards,
>
>
>
> Shuai
>
