[jira] [Commented] (SPARK-11413) Java 8 build has problem with joda-time and s3 request, should bump joda-time version
[
https://issues.apache.org/jira/browse/SPARK-11413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15302873#comment-15302873
]
Steve Loughran commented on SPARK-11413:
Note that joda-time is only explicitly pulled in with Hive builds; any version
of spark built without {{-Phive}} will omit it. This gives whoever compiled
their private version of Spark the opportunity and duty to put the version of
Joda time compatible with the AWS SDK/JVM on their own classpath.
> Java 8 build has problem with joda-time and s3 request, should bump joda-time
> version
> -
>
> Key: SPARK-11413
> URL: https://issues.apache.org/jira/browse/SPARK-11413
> Project: Spark
> Issue Type: Improvement
> Components: Build
>Reporter: Yongjia Wang
>Assignee: Yongjia Wang
>Priority: Minor
> Fix For: 1.6.0
>
>
> Joda-time has problems with formatting time zones starting with Java 1.8u60,
> and this will cause s3 request to fail. It is said to have been fixed at
> joda-time 2.8.1.
> Spark is still using joda-time 2.5 by fault, if java8 is used to build spark,
> should set -Djoda.version=2.8.1 or above.
> I was hit by this problem, and -Djoda.version=2.9 worked.
> I don't see any reason not to bump up joda-time version in pom.xml
> Should I create a pull request for this? It is trivial.
> https://github.com/aws/aws-sdk-java/issues/484
> https://github.com/aws/aws-sdk-java/issues/444
> http://stackoverflow.com/questions/32058431/aws-java-sdk-aws-authentication-requires-a-valid-date-or-x-amz-date-header
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-11413) Java 8 build has problem with joda-time and s3 request, should bump joda-time version
[ https://issues.apache.org/jira/browse/SPARK-11413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14982909#comment-14982909 ] Yongjia Wang commented on SPARK-11413: -- There is no more transitive dependencies from joda-time. So as long as it passes the tests, it should be good? > Java 8 build has problem with joda-time and s3 request, should bump joda-time > version > - > > Key: SPARK-11413 > URL: https://issues.apache.org/jira/browse/SPARK-11413 > Project: Spark > Issue Type: Improvement > Components: Build >Reporter: Yongjia Wang >Priority: Minor > > Joda-time has problems with formatting time zones starting with Java 1.8u60, > and this will cause s3 request to fail. It is said to have been fixed at > joda-time 2.8.1. > Spark is still using joda-time 2.5 by fault, if java8 is used to build spark, > should set -Djoda.version=2.8.1 or above. > I was hit by this problem, and -Djoda.version=2.9 worked. > I don't see any reason not to bump up joda-time version in pom.xml > Should I create a pull request for this? It is trivial. > https://github.com/aws/aws-sdk-java/issues/484 > https://github.com/aws/aws-sdk-java/issues/444 > http://stackoverflow.com/questions/32058431/aws-java-sdk-aws-authentication-requires-a-valid-date-or-x-amz-date-header -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-11413) Java 8 build has problem with joda-time and s3 request, should bump joda-time version
[ https://issues.apache.org/jira/browse/SPARK-11413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14982906#comment-14982906 ] Apache Spark commented on SPARK-11413: -- User 'yongjiaw' has created a pull request for this issue: https://github.com/apache/spark/pull/9379 > Java 8 build has problem with joda-time and s3 request, should bump joda-time > version > - > > Key: SPARK-11413 > URL: https://issues.apache.org/jira/browse/SPARK-11413 > Project: Spark > Issue Type: Improvement > Components: Build >Reporter: Yongjia Wang >Priority: Minor > > Joda-time has problems with formatting time zones starting with Java 1.8u60, > and this will cause s3 request to fail. It is said to have been fixed at > joda-time 2.8.1. > Spark is still using joda-time 2.5 by fault, if java8 is used to build spark, > should set -Djoda.version=2.8.1 or above. > I was hit by this problem, and -Djoda.version=2.9 worked. > I don't see any reason not to bump up joda-time version in pom.xml > Should I create a pull request for this? It is trivial. > https://github.com/aws/aws-sdk-java/issues/484 > https://github.com/aws/aws-sdk-java/issues/444 > http://stackoverflow.com/questions/32058431/aws-java-sdk-aws-authentication-requires-a-valid-date-or-x-amz-date-header -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-11413) Java 8 build has problem with joda-time and s3 request, should bump joda-time version
[ https://issues.apache.org/jira/browse/SPARK-11413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14982883#comment-14982883 ] Sean Owen commented on SPARK-11413: --- I imagine it will be as well to update to 2.9, but yes that would be good to investigate while you're having a look. I don't expect any problems, it's just good practice. > Java 8 build has problem with joda-time and s3 request, should bump joda-time > version > - > > Key: SPARK-11413 > URL: https://issues.apache.org/jira/browse/SPARK-11413 > Project: Spark > Issue Type: Improvement > Components: Build >Reporter: Yongjia Wang >Priority: Minor > > Joda-time has problems with formatting time zones starting with Java 1.8u60, > and this will cause s3 request to fail. It is said to have been fixed at > joda-time 2.8.1. > Spark is still using joda-time 2.5 by fault, if java8 is used to build spark, > should set -Djoda.version=2.8.1 or above. > I was hit by this problem, and -Djoda.version=2.9 worked. > I don't see any reason not to bump up joda-time version in pom.xml > Should I create a pull request for this? It is trivial. > https://github.com/aws/aws-sdk-java/issues/484 > https://github.com/aws/aws-sdk-java/issues/444 > http://stackoverflow.com/questions/32058431/aws-java-sdk-aws-authentication-requires-a-valid-date-or-x-amz-date-header -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-11413) Java 8 build has problem with joda-time and s3 request, should bump joda-time version
[ https://issues.apache.org/jira/browse/SPARK-11413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14982874#comment-14982874 ] Yongjia Wang commented on SPARK-11413: -- I can follow up with a PR first. The latest joda-time release is 2.9. Do you suggest just using 2.8.1 as the fix with minimum changes? > Java 8 build has problem with joda-time and s3 request, should bump joda-time > version > - > > Key: SPARK-11413 > URL: https://issues.apache.org/jira/browse/SPARK-11413 > Project: Spark > Issue Type: Improvement > Components: Build >Reporter: Yongjia Wang >Priority: Minor > > Joda-time has problems with formatting time zones starting with Java 1.8u60, > and this will cause s3 request to fail. It is said to have been fixed at > joda-time 2.8.1. > Spark is still using joda-time 2.5 by fault, if java8 is used to build spark, > should set -Djoda.version=2.8.1 or above. > I was hit by this problem, and -Djoda.version=2.9 worked. > I don't see any reason not to bump up joda-time version in pom.xml > Should I create a pull request for this? It is trivial. > https://github.com/aws/aws-sdk-java/issues/484 > https://github.com/aws/aws-sdk-java/issues/444 > http://stackoverflow.com/questions/32058431/aws-java-sdk-aws-authentication-requires-a-valid-date-or-x-amz-date-header -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-11413) Java 8 build has problem with joda-time and s3 request, should bump joda-time version
[ https://issues.apache.org/jira/browse/SPARK-11413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14982854#comment-14982854 ] Sean Owen commented on SPARK-11413: --- To assume? no, that's why I'm suggesting you try it by submitting a PR to run the tests, but also investigate the list of changes from 2.5 -> 2.8.1 and to diff the dependency trees. > Java 8 build has problem with joda-time and s3 request, should bump joda-time > version > - > > Key: SPARK-11413 > URL: https://issues.apache.org/jira/browse/SPARK-11413 > Project: Spark > Issue Type: Improvement > Components: Build >Reporter: Yongjia Wang >Priority: Minor > > Joda-time has problems with formatting time zones starting with Java 1.8u60, > and this will cause s3 request to fail. It is said to have been fixed at > joda-time 2.8.1. > Spark is still using joda-time 2.5 by fault, if java8 is used to build spark, > should set -Djoda.version=2.8.1 or above. > I was hit by this problem, and -Djoda.version=2.9 worked. > I don't see any reason not to bump up joda-time version in pom.xml > Should I create a pull request for this? It is trivial. > https://github.com/aws/aws-sdk-java/issues/484 > https://github.com/aws/aws-sdk-java/issues/444 > http://stackoverflow.com/questions/32058431/aws-java-sdk-aws-authentication-requires-a-valid-date-or-x-amz-date-header -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-11413) Java 8 build has problem with joda-time and s3 request, should bump joda-time version
[ https://issues.apache.org/jira/browse/SPARK-11413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14982790#comment-14982790 ] Sean Owen commented on SPARK-11413: --- I understand all that, but my question is about the side-effects of the update. Does it cause any other potential incompatibility, or change transitive dependencies? That's always the question when someone wants to update a dependency. > Java 8 build has problem with joda-time and s3 request, should bump joda-time > version > - > > Key: SPARK-11413 > URL: https://issues.apache.org/jira/browse/SPARK-11413 > Project: Spark > Issue Type: Improvement > Components: Build >Reporter: Yongjia Wang >Priority: Minor > > Joda-time has problems with formatting time zones starting with Java 1.8u60, > and this will cause s3 request to fail. It is said to have been fixed at > joda-time 2.8.1. > Spark is still using joda-time 2.5 by fault, if java8 is used to build spark, > should set -Djoda.version=2.8.1 or above. > I was hit by this problem, and -Djoda.version=2.9 worked. > I don't see any reason not to bump up joda-time version in pom.xml > Should I create a pull request for this? It is trivial. > https://github.com/aws/aws-sdk-java/issues/484 > https://github.com/aws/aws-sdk-java/issues/444 > http://stackoverflow.com/questions/32058431/aws-java-sdk-aws-authentication-requires-a-valid-date-or-x-amz-date-header -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-11413) Java 8 build has problem with joda-time and s3 request, should bump joda-time version
[ https://issues.apache.org/jira/browse/SPARK-11413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14982802#comment-14982802 ] Yongjia Wang commented on SPARK-11413: -- I see. I don't know, is it safe to assume joda-time is backward compatible as far as Spark is concerned? > Java 8 build has problem with joda-time and s3 request, should bump joda-time > version > - > > Key: SPARK-11413 > URL: https://issues.apache.org/jira/browse/SPARK-11413 > Project: Spark > Issue Type: Improvement > Components: Build >Reporter: Yongjia Wang >Priority: Minor > > Joda-time has problems with formatting time zones starting with Java 1.8u60, > and this will cause s3 request to fail. It is said to have been fixed at > joda-time 2.8.1. > Spark is still using joda-time 2.5 by fault, if java8 is used to build spark, > should set -Djoda.version=2.8.1 or above. > I was hit by this problem, and -Djoda.version=2.9 worked. > I don't see any reason not to bump up joda-time version in pom.xml > Should I create a pull request for this? It is trivial. > https://github.com/aws/aws-sdk-java/issues/484 > https://github.com/aws/aws-sdk-java/issues/444 > http://stackoverflow.com/questions/32058431/aws-java-sdk-aws-authentication-requires-a-valid-date-or-x-amz-date-header -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-11413) Java 8 build has problem with joda-time and s3 request, should bump joda-time version
[ https://issues.apache.org/jira/browse/SPARK-11413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14982755#comment-14982755 ] Yongjia Wang commented on SPARK-11413: -- My last statement was wrong. It's a problem when using jre 8u60 or above, and when joda-time 2.8.0 and belong was loaded at run time by s3 client. The following error was reproduced when I use the official spark release with JRE 1.8.0_65 (Oracle Corporation) Note that the aws-java-sdk-s3 library already depends on joda-time-2.8.1. It's due to Spark assembly's inclusion of joda-time-2.5 which was loaded by JVM that caused this problem. com.amazonaws.services.s3.model.AmazonS3Exception: AWS authentication requires a valid Date or x-amz-date header (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: E0D714AF923221DD), S3 Extended Request ID: 8OJlBMYKVzwrEXHANpBTsiLRmaFn2lHQ4sAvkU/HF66wWGVj8VR2/Jh4Wl8QuFUt at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1182) at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:770) at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:489) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:310) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3785) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3738) at com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:653) > Java 8 build has problem with joda-time and s3 request, should bump joda-time > version > - > > Key: SPARK-11413 > URL: https://issues.apache.org/jira/browse/SPARK-11413 > Project: Spark > Issue Type: Improvement > Components: Build >Reporter: Yongjia Wang >Priority: Minor > > Joda-time has problems with formatting time zones starting with Java 1.8u60, > and this will cause s3 request to fail. It is said to have been fixed at > joda-time 2.8.1. > Spark is still using joda-time 2.5 by fault, if java8 is used to build spark, > should set -Djoda.version=2.8.1 or above. > I was hit by this problem, and -Djoda.version=2.9 worked. > I don't see any reason not to bump up joda-time version in pom.xml > Should I create a pull request for this? It is trivial. > https://github.com/aws/aws-sdk-java/issues/484 > https://github.com/aws/aws-sdk-java/issues/444 > http://stackoverflow.com/questions/32058431/aws-java-sdk-aws-authentication-requires-a-valid-date-or-x-amz-date-header -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-11413) Java 8 build has problem with joda-time and s3 request, should bump joda-time version
[ https://issues.apache.org/jira/browse/SPARK-11413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14982662#comment-14982662 ] Yongjia Wang commented on SPARK-11413: -- yea, the fix is to update the joda.version number from 2.5 to 2.8.1 or to the newest 2.9 in Spark's pom.xml. This PR would just be a 1 line change in pom.xml It's only an issue when compiling with java 1.8u60, due to the changes in time format, and s3 request in some way relies on the time format in its header. Those links in my comment should explain better. It's no new dependency, but can be considered a java8 s3 combo bug. So this only affects the combination of java 8u60 + using s3a, which should become quite common and hits more people. If I understand correctly, building with java7 or java8 before u60 should be fine, even running with jre after java 8u60. > Java 8 build has problem with joda-time and s3 request, should bump joda-time > version > - > > Key: SPARK-11413 > URL: https://issues.apache.org/jira/browse/SPARK-11413 > Project: Spark > Issue Type: Improvement > Components: Build >Reporter: Yongjia Wang >Priority: Minor > > Joda-time has problems with formatting time zones starting with Java 1.8u60, > and this will cause s3 request to fail. It is said to have been fixed at > joda-time 2.8.1. > Spark is still using joda-time 2.5 by fault, if java8 is used to build spark, > should set -Djoda.version=2.8.1 or above. > I was hit by this problem, and -Djoda.version=2.9 worked. > I don't see any reason not to bump up joda-time version in pom.xml > Should I create a pull request for this? It is trivial. > https://github.com/aws/aws-sdk-java/issues/484 > https://github.com/aws/aws-sdk-java/issues/444 > http://stackoverflow.com/questions/32058431/aws-java-sdk-aws-authentication-requires-a-valid-date-or-x-amz-date-header -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-11413) Java 8 build has problem with joda-time and s3 request, should bump joda-time version
[ https://issues.apache.org/jira/browse/SPARK-11413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14982044#comment-14982044 ] Sean Owen commented on SPARK-11413: --- How about updating joda time to 2.8.1 everywhere? what are the compatibility issues, new dependencies, etc? It doesn't seem to make sense to only do it for java 8. > Java 8 build has problem with joda-time and s3 request, should bump joda-time > version > - > > Key: SPARK-11413 > URL: https://issues.apache.org/jira/browse/SPARK-11413 > Project: Spark > Issue Type: Improvement > Components: Build >Reporter: Yongjia Wang >Priority: Minor > > Joda-time has problems with formatting time zones starting with Java 1.8u60, > and this will cause s3 request to fail. It is said to have been fixed at > joda-time 2.8.1. > Spark is still using joda-time 2.5 by fault, if java8 is used to build spark, > should set -Djoda.version=2.8.1 or above. > I was hit by this problem, and -Djoda.version=2.9 worked. > I don't see any reason not to bump up joda-time version in pom.xml > Should I create a pull request for this? It is trivial. > https://github.com/aws/aws-sdk-java/issues/484 > https://github.com/aws/aws-sdk-java/issues/444 > http://stackoverflow.com/questions/32058431/aws-java-sdk-aws-authentication-requires-a-valid-date-or-x-amz-date-header -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
