[jira] [Commented] (HADOOP-14305) S3A SSE tests won't run in parallel: Bad request in directory GetFileStatus
[
https://issues.apache.org/jira/browse/HADOOP-14305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15982717#comment-15982717
]
Steve Loughran commented on HADOOP-14305:
-
ooh, I think you went to more effort than was needed for that branch-2 patch;
the {{intercept()}} call works on b2, it just needs to take a
{{Callable}}. I'll just take the trunk patch and do that tweak.
Also, and I'm not going to change it, your intercept lambdas don't need to
raise an exception the clause didn't raise the exception you wanted. Intercept
generates its own exception, the latest versions (in s3guard) including the
response of the clause
Example:
{code}
intercept(IOException.class, "", () -> {{ return File.getStatus(new Path(new
URI("s3a://bucket/no-such-path"); })
{code}
This will raise an assertion saying something like {{"Expected an IOException,
got FileStatus("s3a://bucket/no-such-path", ..."}}. It's aim is to provide as
much useful diagnostics automatically
> S3A SSE tests won't run in parallel: Bad request in directory GetFileStatus
> ---
>
> Key: HADOOP-14305
> URL: https://issues.apache.org/jira/browse/HADOOP-14305
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3, test
>Affects Versions: 2.9.0
>Reporter: Steve Loughran
>Assignee: Steve Moist
>Priority: Minor
> Attachments: HADOOP-14305-001.patch, HADOOP-14305-002.patch,
> HADOOP-14305-branch-2-002.patch
>
>
> The S3a encryption tests all run in parallel (they were interfering with each
> other, apparently). This adds ~1 min to the test runs.
> They should run in serial. That they fail when this is attempted due to Bad
> Auth problems must be considered a serious problem, as is indicates issues
> related to working with SSE encryption from hadoop
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14305) S3A SSE tests won't run in parallel: Bad request in directory GetFileStatus
[ https://issues.apache.org/jira/browse/HADOOP-14305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15981802#comment-15981802 ] Hudson commented on HADOOP-14305: - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11625 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/11625/]) HADOOP-14305 S3A SSE tests won't run in parallel: Bad request in (stevel: rev 5f934f838619ff73fcd80ad5fd6aee4c2134371a) * (edit) hadoop-tools/hadoop-aws/pom.xml * (edit) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AEncryptionSSEC.java * (edit) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/AbstractTestS3AEncryption.java * (edit) hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md > S3A SSE tests won't run in parallel: Bad request in directory GetFileStatus > --- > > Key: HADOOP-14305 > URL: https://issues.apache.org/jira/browse/HADOOP-14305 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3, test >Affects Versions: 2.9.0 >Reporter: Steve Loughran >Assignee: Steve Moist >Priority: Minor > Attachments: HADOOP-14305-001.patch, HADOOP-14305-002.patch > > > The S3a encryption tests all run in parallel (they were interfering with each > other, apparently). This adds ~1 min to the test runs. > They should run in serial. That they fail when this is attempted due to Bad > Auth problems must be considered a serious problem, as is indicates issues > related to working with SSE encryption from hadoop -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14305) S3A SSE tests won't run in parallel: Bad request in directory GetFileStatus
[ https://issues.apache.org/jira/browse/HADOOP-14305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15981749#comment-15981749 ] Steve Loughran commented on HADOOP-14305: - OK, committed to trunk. Steve: if you can do the branch-2 patch and just attach it to this with the --branch-2-002.patch suffix, it can be reviewed by Jenkins for me to pull in tomorrow. > S3A SSE tests won't run in parallel: Bad request in directory GetFileStatus > --- > > Key: HADOOP-14305 > URL: https://issues.apache.org/jira/browse/HADOOP-14305 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3, test >Affects Versions: 2.9.0 >Reporter: Steve Loughran >Assignee: Steve Moist >Priority: Minor > Attachments: HADOOP-14305-001.patch, HADOOP-14305-002.patch > > > The S3a encryption tests all run in parallel (they were interfering with each > other, apparently). This adds ~1 min to the test runs. > They should run in serial. That they fail when this is attempted due to Bad > Auth problems must be considered a serious problem, as is indicates issues > related to working with SSE encryption from hadoop -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14305) S3A SSE tests won't run in parallel: Bad request in directory GetFileStatus
[ https://issues.apache.org/jira/browse/HADOOP-14305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15981700#comment-15981700 ] Steve Loughran commented on HADOOP-14305: - +1 > S3A SSE tests won't run in parallel: Bad request in directory GetFileStatus > --- > > Key: HADOOP-14305 > URL: https://issues.apache.org/jira/browse/HADOOP-14305 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3, test >Affects Versions: 2.9.0 >Reporter: Steve Loughran >Assignee: Steve Moist >Priority: Minor > Attachments: HADOOP-14305-001.patch, HADOOP-14305-002.patch > > > The S3a encryption tests all run in parallel (they were interfering with each > other, apparently). This adds ~1 min to the test runs. > They should run in serial. That they fail when this is attempted due to Bad > Auth problems must be considered a serious problem, as is indicates issues > related to working with SSE encryption from hadoop -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14305) S3A SSE tests won't run in parallel: Bad request in directory GetFileStatus
[ https://issues.apache.org/jira/browse/HADOOP-14305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15981578#comment-15981578 ] Steve Loughran commented on HADOOP-14305: - thanks, LGTM, testing locally & will +1 it after if all is well. Nice to see some troubleshooting docs —always appreciated > S3A SSE tests won't run in parallel: Bad request in directory GetFileStatus > --- > > Key: HADOOP-14305 > URL: https://issues.apache.org/jira/browse/HADOOP-14305 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3, test >Affects Versions: 2.9.0 >Reporter: Steve Loughran >Assignee: Steve Moist >Priority: Minor > Attachments: HADOOP-14305-001.patch, HADOOP-14305-002.patch > > > The S3a encryption tests all run in parallel (they were interfering with each > other, apparently). This adds ~1 min to the test runs. > They should run in serial. That they fail when this is attempted due to Bad > Auth problems must be considered a serious problem, as is indicates issues > related to working with SSE encryption from hadoop -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14305) S3A SSE tests won't run in parallel: Bad request in directory GetFileStatus
[ https://issues.apache.org/jira/browse/HADOOP-14305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15981459#comment-15981459 ] Steve Moist commented on HADOOP-14305: -- us-west-2 (Oregon) > S3A SSE tests won't run in parallel: Bad request in directory GetFileStatus > --- > > Key: HADOOP-14305 > URL: https://issues.apache.org/jira/browse/HADOOP-14305 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3, test >Affects Versions: 2.9.0 >Reporter: Steve Loughran >Assignee: Steve Moist >Priority: Minor > Attachments: HADOOP-14305-001.patch, HADOOP-14305-002.patch > > > The S3a encryption tests all run in parallel (they were interfering with each > other, apparently). This adds ~1 min to the test runs. > They should run in serial. That they fail when this is attempted due to Bad > Auth problems must be considered a serious problem, as is indicates issues > related to working with SSE encryption from hadoop -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14305) S3A SSE tests won't run in parallel: Bad request in directory GetFileStatus
[ https://issues.apache.org/jira/browse/HADOOP-14305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15981340#comment-15981340 ] Steve Loughran commented on HADOOP-14305: - OK. As usual, which s3 endpoint did you run the full S3a integration suite against. Thanks > S3A SSE tests won't run in parallel: Bad request in directory GetFileStatus > --- > > Key: HADOOP-14305 > URL: https://issues.apache.org/jira/browse/HADOOP-14305 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3, test >Affects Versions: 2.9.0 >Reporter: Steve Loughran >Assignee: Steve Moist >Priority: Minor > Attachments: HADOOP-14305-001.patch, HADOOP-14305-002.patch > > > The S3a encryption tests all run in parallel (they were interfering with each > other, apparently). This adds ~1 min to the test runs. > They should run in serial. That they fail when this is attempted due to Bad > Auth problems must be considered a serious problem, as is indicates issues > related to working with SSE encryption from hadoop -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14305) S3A SSE tests won't run in parallel: Bad request in directory GetFileStatus
[ https://issues.apache.org/jira/browse/HADOOP-14305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15980160#comment-15980160 ] Steve Moist commented on HADOOP-14305: -- Uploaded a patch for trunk, if everyone is satisfied with it I will then work on back porting it to branch-2. > S3A SSE tests won't run in parallel: Bad request in directory GetFileStatus > --- > > Key: HADOOP-14305 > URL: https://issues.apache.org/jira/browse/HADOOP-14305 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3, test >Affects Versions: 2.9.0 >Reporter: Steve Loughran >Assignee: Steve Moist >Priority: Minor > Attachments: HADOOP-14305-001.patch, HADOOP-14305-002.patch > > > The S3a encryption tests all run in parallel (they were interfering with each > other, apparently). This adds ~1 min to the test runs. > They should run in serial. That they fail when this is attempted due to Bad > Auth problems must be considered a serious problem, as is indicates issues > related to working with SSE encryption from hadoop -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14305) S3A SSE tests won't run in parallel: Bad request in directory GetFileStatus
[ https://issues.apache.org/jira/browse/HADOOP-14305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15975556#comment-15975556 ] Thomas Demoor commented on HADOOP-14305: [~ste...@apache.org] We currently only support SSE-S3, so I fear we won't be able to provide additional insight on the SSE-C stuff. Will run the SSE-S3 stuff through our testing. > S3A SSE tests won't run in parallel: Bad request in directory GetFileStatus > --- > > Key: HADOOP-14305 > URL: https://issues.apache.org/jira/browse/HADOOP-14305 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3, test >Affects Versions: 2.9.0 >Reporter: Steve Loughran >Assignee: Steve Moist >Priority: Minor > Attachments: HADOOP-14305-001.patch > > > The S3a encryption tests all run in parallel (they were interfering with each > other, apparently). This adds ~1 min to the test runs. > They should run in serial. That they fail when this is attempted due to Bad > Auth problems must be considered a serious problem, as is indicates issues > related to working with SSE encryption from hadoop -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14305) S3A SSE tests won't run in parallel: Bad request in directory GetFileStatus
[ https://issues.apache.org/jira/browse/HADOOP-14305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15974527#comment-15974527 ] Steve Loughran commented on HADOOP-14305: - Created HADOOP-14325 to cover full todo list for SSE-KMS & SSE-C. We can restrict this patch to the tests and round off the other bits elsewhere. I don't see much other than docs and property lockdown; some downstream integration testing will be handy too > S3A SSE tests won't run in parallel: Bad request in directory GetFileStatus > --- > > Key: HADOOP-14305 > URL: https://issues.apache.org/jira/browse/HADOOP-14305 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3, test >Affects Versions: 2.9.0 >Reporter: Steve Loughran >Assignee: Steve Moist >Priority: Minor > Attachments: HADOOP-14305-001.patch > > > The S3a encryption tests all run in parallel (they were interfering with each > other, apparently). This adds ~1 min to the test runs. > They should run in serial. That they fail when this is attempted due to Bad > Auth problems must be considered a serious problem, as is indicates issues > related to working with SSE encryption from hadoop -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14305) S3A SSE tests won't run in parallel: Bad request in directory GetFileStatus
[ https://issues.apache.org/jira/browse/HADOOP-14305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15974467#comment-15974467 ] Steve Loughran commented on HADOOP-14305: - I'm trying to think of what would make some good extra tests. There's already one for rename, but it's for a file. How about * recursive rename of a 2-level directory tree, verify ls of directories find dirs, loading files still works. listFiles(path, recursive=true) should also be used for listing, on account of it being a different list mechanism. * open , seek to end + read to -1, then seek to (0) and read there. Why? Helps verify that filelengths match data, and that the re-opens done by seek() stay valid. Maybe we could also subclass a Huge test and add encryption, again using SSE-C as the worst-case example. That will test multipart operations and large file copies. With that there'd be coverage of the set of S3 verbs: PUT, mulipart POST, COPY, LIST, GET, HEAD. IT might also be fun to do a getFileStatus() and open() on the stock landsat CSV, expecting failures. We'll be able to make an assert about the error type/text. verify anything I do w.r.t diagnostics text in HADOOP-14291 doesn't break the exception class type. (and provides details on encryption type) Note that I've added something to the release notes of HADOOP-13075, made clear this must be considered experimental for now. I'm confident you think it's ready, but everything related to S3 usually takes one iteration to stabilise. The phase II stuff all went through the HDCloud 4-6 week dev cycle before shipping in longer-lived releases, and yes, we found things. I'll expect to see things here too, now that you are shipping it. If they go to you internally, don't be shy of matching them with things related to the ASF code, test and docs. > S3A SSE tests won't run in parallel: Bad request in directory GetFileStatus > --- > > Key: HADOOP-14305 > URL: https://issues.apache.org/jira/browse/HADOOP-14305 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3, test >Affects Versions: 2.9.0 >Reporter: Steve Loughran >Assignee: Steve Moist >Priority: Minor > Attachments: HADOOP-14305-001.patch > > > The S3a encryption tests all run in parallel (they were interfering with each > other, apparently). This adds ~1 min to the test runs. > They should run in serial. That they fail when this is attempted due to Bad > Auth problems must be considered a serious problem, as is indicates issues > related to working with SSE encryption from hadoop -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14305) S3A SSE tests won't run in parallel: Bad request in directory GetFileStatus
[ https://issues.apache.org/jira/browse/HADOOP-14305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15973589#comment-15973589 ] Steve Moist commented on HADOOP-14305: -- I've made the pom changes, but want to spend some time writing some more unit tests that deal with SSE-C. I'll submit a patch in a few days. > S3A SSE tests won't run in parallel: Bad request in directory GetFileStatus > --- > > Key: HADOOP-14305 > URL: https://issues.apache.org/jira/browse/HADOOP-14305 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3, test >Affects Versions: 2.9.0 >Reporter: Steve Loughran >Assignee: Steve Moist >Priority: Minor > Attachments: HADOOP-14305-001.patch > > > The S3a encryption tests all run in parallel (they were interfering with each > other, apparently). This adds ~1 min to the test runs. > They should run in serial. That they fail when this is attempted due to Bad > Auth problems must be considered a serious problem, as is indicates issues > related to working with SSE encryption from hadoop -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14305) S3A SSE tests won't run in parallel: Bad request in directory GetFileStatus
[ https://issues.apache.org/jira/browse/HADOOP-14305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15973319#comment-15973319 ] Steve Moist commented on HADOOP-14305: -- Sounds good. I'll submit a few patches today. > S3A SSE tests won't run in parallel: Bad request in directory GetFileStatus > --- > > Key: HADOOP-14305 > URL: https://issues.apache.org/jira/browse/HADOOP-14305 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3, test >Affects Versions: 2.9.0 >Reporter: Steve Loughran >Priority: Minor > Attachments: HADOOP-14305-001.patch > > > The S3a encryption tests all run in parallel (they were interfering with each > other, apparently). This adds ~1 min to the test runs. > They should run in serial. That they fail when this is attempted due to Bad > Auth problems must be considered a serious problem, as is indicates issues > related to working with SSE encryption from hadoop -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14305) S3A SSE tests won't run in parallel: Bad request in directory GetFileStatus
[ https://issues.apache.org/jira/browse/HADOOP-14305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15972625#comment-15972625 ] Steve Loughran commented on HADOOP-14305: - Thanks for helping track this down. bq. I don't really advise people to use SSE-C. That's the kind of thing we need in the documentation and release notes. You can add this detail to the release notes of the original JIRA. for this patch, how about restricting it to serializing that SSE-C test, and updating the docs about how to safely use the feature. I'll review and we can have a fast turnaround here: no production code changes to worry about > S3A SSE tests won't run in parallel: Bad request in directory GetFileStatus > --- > > Key: HADOOP-14305 > URL: https://issues.apache.org/jira/browse/HADOOP-14305 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3, test >Affects Versions: 2.9.0 >Reporter: Steve Loughran >Priority: Minor > Attachments: HADOOP-14305-001.patch > > > The S3a encryption tests all run in parallel (they were interfering with each > other, apparently). This adds ~1 min to the test runs. > They should run in serial. That they fail when this is attempted due to Bad > Auth problems must be considered a serious problem, as is indicates issues > related to working with SSE encryption from hadoop -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14305) S3A SSE tests won't run in parallel: Bad request in directory GetFileStatus
[ https://issues.apache.org/jira/browse/HADOOP-14305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15971778#comment-15971778 ] Steve Moist commented on HADOOP-14305: -- I'm pretty sure it's SSE-C related. I disabled the SSE-C tests and it ran fine in parallel mode. So I entirely suspect its that its using the wrong key, wrong method or no key supplied when trying to re-access the fork-#/ prefix in the tests (either with SSE-C, and the other tests failing because it's not cleaned up properly) and s3 is failing due to the wrong key being provided. If I do a "aws s3api head-object --bucket my-bucket --key sse-c-encrypted.txt", I get a "An error occurred (400) when calling the HeadObject operation: Bad Request" when trying to get the head of an object with no encryption key, or the wrong encryption key. Which is the same http error in the failed tests. SSE-C was one of those s3 features where the encryption key must always be provided for all operations or you get a 400 error. So I suspect this is the real culprit of the issue. In general, I don't really advise people to use SSE-C. In the web ui, you can't tell if it's encrypted, which encryption key was used, nor can you get the meta-data about the object without the correct encryption key through the command line. If a customer wishes to use this, the best solution is to use 1 SSE-C key across the cluster. If they wish to use multiple, they need to keep track of which key maps to where and should NEVER share buckets. I also recommend the documentation advising only to use this for advanced users. > S3A SSE tests won't run in parallel: Bad request in directory GetFileStatus > --- > > Key: HADOOP-14305 > URL: https://issues.apache.org/jira/browse/HADOOP-14305 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3, test >Affects Versions: 2.9.0 >Reporter: Steve Loughran >Priority: Minor > Attachments: HADOOP-14305-001.patch > > > The S3a encryption tests all run in parallel (they were interfering with each > other, apparently). This adds ~1 min to the test runs. > They should run in serial. That they fail when this is attempted due to Bad > Auth problems must be considered a serious problem, as is indicates issues > related to working with SSE encryption from hadoop -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14305) S3A SSE tests won't run in parallel: Bad request in directory GetFileStatus
[
https://issues.apache.org/jira/browse/HADOOP-14305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15971774#comment-15971774
]
Hadoop QA commented on HADOOP-14305:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 15m
16s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
11s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
17s{color} | {color:green} hadoop-aws in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 34m 13s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:0ac17dc |
| JIRA Issue | HADOOP-14305 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12863336/HADOOP-14305-001.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit xml |
| uname | Linux 162cb43c69e9 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / f1de2c8 |
| Default Java | 1.8.0_121 |
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/12110/testReport/ |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/12110/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> S3A SSE tests won't run in parallel: Bad request in directory GetFileStatus
> ---
>
> Key: HADOOP-14305
> URL: https://issues.apache.org/jira/browse/HADOOP-14305
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3, test
>Affects Versions: 2.9.0
>Reporter: Steve Loughran
>Priority: Minor
> Attachments: HADOOP-14305-001.patch
>
>
> The S3a encryption tests all run in parallel (they were interfering with each
> other, apparently). This adds ~1 min to the test runs.
> They should run in serial. That they fail when this is attempted due to Bad
> Auth problems must be considered a serious problem, as is indicates issues
> related to working with SSE encryption from hadoop
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
[jira] [Commented] (HADOOP-14305) S3A SSE tests won't run in parallel: Bad request in directory GetFileStatus
[ https://issues.apache.org/jira/browse/HADOOP-14305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15971469#comment-15971469 ] Steve Loughran commented on HADOOP-14305: - Not > 1 thread, >1 client process. Our tests run in different VMs. This is making me wonder if, once one client with encryption turned on creates some mock empty directories or something, does that stop clients with other encryption policies *including the no-encryption policy* from working with the directory structure. I'm not worry about whether process A can read data created by process B if they have different keys, but HEAD requests are things I'd expect to work, If that's not the case, then this is a serious issue which is either going to need to be fixed (somehow), or used to constrain the usage situations of the encryption mech, i.e. one policy/key per bucket. We don't see this problem with AWS-managed SSE, presumably because it sorts out keys when needed. > S3A SSE tests won't run in parallel: Bad request in directory GetFileStatus > --- > > Key: HADOOP-14305 > URL: https://issues.apache.org/jira/browse/HADOOP-14305 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3, test >Affects Versions: 2.9.0 >Reporter: Steve Loughran >Priority: Minor > Attachments: HADOOP-14305-001.patch > > > The S3a encryption tests all run in parallel (they were interfering with each > other, apparently). This adds ~1 min to the test runs. > They should run in serial. That they fail when this is attempted due to Bad > Auth problems must be considered a serious problem, as is indicates issues > related to working with SSE encryption from hadoop -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14305) S3A SSE tests won't run in parallel: Bad request in directory GetFileStatus
[
https://issues.apache.org/jira/browse/HADOOP-14305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15969466#comment-15969466
]
Steve Moist commented on HADOOP-14305:
--
{quote}
Steve Moist : have you been using this feature yet? In particular, using it
with writers to the same bucket with different policies? It may be that you
just can't do this —something the release notes will need to cover.
{quote}
I've mainly been using it during testing with a cluster and parts of a demo.
Most of it has been fairly light usage, except when I ran terragen a few times.
I haven't seen this kind of issue. During a demo I did change from SSE-S3 to
SSE-KMS to SSE-C to write to the same bucket, but I used Cloudera Manager to
push those changes to the cluster and restarted the services. So I never had
multiple threads running with different encryption types.
> S3A SSE tests won't run in parallel: Bad request in directory GetFileStatus
> ---
>
> Key: HADOOP-14305
> URL: https://issues.apache.org/jira/browse/HADOOP-14305
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3, test
>Affects Versions: 2.9.0
>Reporter: Steve Loughran
>Priority: Minor
> Attachments: HADOOP-14305-001.patch
>
>
> The S3a encryption tests all run in parallel (they were interfering with each
> other, apparently). This adds ~1 min to the test runs.
> They should run in serial. That they fail when this is attempted due to Bad
> Auth problems must be considered a serious problem, as is indicates issues
> related to working with SSE encryption from hadoop
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14305) S3A SSE tests won't run in parallel: Bad request in directory GetFileStatus
[ https://issues.apache.org/jira/browse/HADOOP-14305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15968086#comment-15968086 ] Steve Loughran commented on HADOOP-14305: - I'm not going to have chance to look at this. Until we understand what's happening, we can't consider KMC/C encryption ready for use > S3A SSE tests won't run in parallel: Bad request in directory GetFileStatus > --- > > Key: HADOOP-14305 > URL: https://issues.apache.org/jira/browse/HADOOP-14305 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3, test >Affects Versions: 2.9.0 >Reporter: Steve Loughran >Priority: Minor > Attachments: HADOOP-14305-001.patch > > > The S3a encryption tests all run in parallel (they were interfering with each > other, apparently). This adds ~1 min to the test runs. > it should be straighforward for them to use unique names. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
