[jira] [Updated] (HADOOP-13113) Enable parallel test execution for hadoop-aws.
[ https://issues.apache.org/jira/browse/HADOOP-13113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth updated HADOOP-13113: --- Issue Type: Sub-task (was: Improvement) Parent: HADOOP-11694 > Enable parallel test execution for hadoop-aws. > -- > > Key: HADOOP-13113 > URL: https://issues.apache.org/jira/browse/HADOOP-13113 > Project: Hadoop Common > Issue Type: Sub-task > Components: test >Reporter: Chris Nauroth >Assignee: Chris Nauroth >Priority: Minor > Fix For: 2.8.0 > > Attachments: HADOOP-13113.001.patch, HADOOP-13113.002.patch, > HADOOP-13113.003.patch, HADOOP-13113.004.patch > > > The full hadoop-aws test suite takes ~30 minutes to execute. The tests spend > most of their time blocked on network I/O with the S3 back-end, but they > don't saturate the bandwidth of the NIC. We can improve overall execution > time by enabling parallel test execution. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13113) Enable parallel test execution for hadoop-aws.
[ https://issues.apache.org/jira/browse/HADOOP-13113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth updated HADOOP-13113: --- Priority: Minor (was: Major) > Enable parallel test execution for hadoop-aws. > -- > > Key: HADOOP-13113 > URL: https://issues.apache.org/jira/browse/HADOOP-13113 > Project: Hadoop Common > Issue Type: Improvement > Components: test >Reporter: Chris Nauroth >Assignee: Chris Nauroth >Priority: Minor > Fix For: 2.8.0 > > Attachments: HADOOP-13113.001.patch, HADOOP-13113.002.patch, > HADOOP-13113.003.patch, HADOOP-13113.004.patch > > > The full hadoop-aws test suite takes ~30 minutes to execute. The tests spend > most of their time blocked on network I/O with the S3 back-end, but they > don't saturate the bandwidth of the NIC. We can improve overall execution > time by enabling parallel test execution. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13113) Enable parallel test execution for hadoop-aws.
[ https://issues.apache.org/jira/browse/HADOOP-13113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-13113: Resolution: Fixed Fix Version/s: 2.8.0 Status: Resolved (was: Patch Available) > Enable parallel test execution for hadoop-aws. > -- > > Key: HADOOP-13113 > URL: https://issues.apache.org/jira/browse/HADOOP-13113 > Project: Hadoop Common > Issue Type: Improvement > Components: test >Reporter: Chris Nauroth >Assignee: Chris Nauroth > Fix For: 2.8.0 > > Attachments: HADOOP-13113.001.patch, HADOOP-13113.002.patch, > HADOOP-13113.003.patch, HADOOP-13113.004.patch > > > The full hadoop-aws test suite takes ~30 minutes to execute. The tests spend > most of their time blocked on network I/O with the S3 back-end, but they > don't saturate the bandwidth of the NIC. We can improve overall execution > time by enabling parallel test execution. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13113) Enable parallel test execution for hadoop-aws.
[ https://issues.apache.org/jira/browse/HADOOP-13113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth updated HADOOP-13113: --- Attachment: HADOOP-13113.004.patch Thank you, Steve. Here is patch v004 with the doc change and a minor rebase after the recent commits. > Enable parallel test execution for hadoop-aws. > -- > > Key: HADOOP-13113 > URL: https://issues.apache.org/jira/browse/HADOOP-13113 > Project: Hadoop Common > Issue Type: Improvement > Components: test >Reporter: Chris Nauroth >Assignee: Chris Nauroth > Attachments: HADOOP-13113.001.patch, HADOOP-13113.002.patch, > HADOOP-13113.003.patch, HADOOP-13113.004.patch > > > The full hadoop-aws test suite takes ~30 minutes to execute. The tests spend > most of their time blocked on network I/O with the S3 back-end, but they > don't saturate the bandwidth of the NIC. We can improve overall execution > time by enabling parallel test execution. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13113) Enable parallel test execution for hadoop-aws.
[
https://issues.apache.org/jira/browse/HADOOP-13113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated HADOOP-13113:
---
Attachment: HADOOP-13113.003.patch
I'm attaching patch v003. This adds the rety loop on
{{TestS3AContractRootDir#testListEmptyRootDirectory}}. I also took the
opportunity to clear some Checkstyle nits in files I'm touching.
bq. Do we need to rely on a new system property, or could we just rely on the
test class name being unique? that is {{this.getClass().getName()}} could
provide the path?
Currently the patch relies on overriding
{{AbstractBondedFSContract#getTestPath}} as a convenient place to parameterize
the test path per unique fork. This layer of the code doesn't have direct
access to the test suite class. We could change the contract classes so that
the test suite class needs to be passed in during construction, but then every
test suite would need to be changed. Overall, I prefer what the current patch
is doing, but I could be convinced otherwise if you feel strongly about it.
> Enable parallel test execution for hadoop-aws.
> --
>
> Key: HADOOP-13113
> URL: https://issues.apache.org/jira/browse/HADOOP-13113
> Project: Hadoop Common
> Issue Type: Improvement
> Components: test
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
> Attachments: HADOOP-13113.001.patch, HADOOP-13113.002.patch,
> HADOOP-13113.003.patch
>
>
> The full hadoop-aws test suite takes ~30 minutes to execute. The tests spend
> most of their time blocked on network I/O with the S3 back-end, but they
> don't saturate the bandwidth of the NIC. We can improve overall execution
> time by enabling parallel test execution.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13113) Enable parallel test execution for hadoop-aws.
[
https://issues.apache.org/jira/browse/HADOOP-13113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated HADOOP-13113:
---
Attachment: HADOOP-13113.002.patch
I'm attaching patch v002, adding {{TestJets3tNativeS3FileSystemContract}} to
the include/exclude lists. This is in anticipation that someday we'll resolve
HADOOP-13116 too. It's another test suite that can't run in parallel with
others.
> Enable parallel test execution for hadoop-aws.
> --
>
> Key: HADOOP-13113
> URL: https://issues.apache.org/jira/browse/HADOOP-13113
> Project: Hadoop Common
> Issue Type: Improvement
> Components: test
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
> Attachments: HADOOP-13113.001.patch, HADOOP-13113.002.patch
>
>
> The full hadoop-aws test suite takes ~30 minutes to execute. The tests spend
> most of their time blocked on network I/O with the S3 back-end, but they
> don't saturate the bandwidth of the NIC. We can improve overall execution
> time by enabling parallel test execution.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13113) Enable parallel test execution for hadoop-aws.
[
https://issues.apache.org/jira/browse/HADOOP-13113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated HADOOP-13113:
---
Attachment: HADOOP-13113.001.patch
I'm attaching patch v001. This uses the same parallel test execution
techniques that have been effective in hadoop-common and hadoop-hdfs. I
updated pom.xml to add the {{parallel-tests}} profile and changed some test
code to generate testing paths based on the Surefire fork number, so that
they're guaranteed to be isolated from each other. HADOOP-12801 is a
pre-requisite for this patch.
There are some limitations to this approach. Tests that cover root directory
access, multi-part upload, or YARN usage would run into conflicts on shared
resources. To handle this, I've moved those tests to a separate Surefire
execution step, which runs sequentially instead of parallelized. A future
improvement could be to refactor tests related to root out of
{{FileSystemContractBaseTest}}, so that the non-root tests in that suite can
run in parallel. {{TestS3AFileSystemContract}} takes ~90 seconds to run, so it
would be nice to parallelize that with other suites. I chose not to step
outside hadoop-aws for the scope of the current patch though.
While I was editing pom.xml, I also removed an unused dependency on Jackson.
This change is backwards-compatible, because clients who had been relying on
receiving Jackson via transitive dependency through hadoop-aws will continue to
receive it through the dependency chain hadoop-aws -> hadoop-common -> Jackson.
In my environment, plain {{mvn test}} runs in ~30 minutes, and {{mvn
-Pparallel-tests test}} runs in ~13 minutes.
[~ste...@apache.org], would you please review when you get a chance?
> Enable parallel test execution for hadoop-aws.
> --
>
> Key: HADOOP-13113
> URL: https://issues.apache.org/jira/browse/HADOOP-13113
> Project: Hadoop Common
> Issue Type: Improvement
> Components: test
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
> Attachments: HADOOP-13113.001.patch
>
>
> The full hadoop-aws test suite takes ~30 minutes to execute. The tests spend
> most of their time blocked on network I/O with the S3 back-end, but they
> don't saturate the bandwidth of the NIC. We can improve overall execution
> time by enabling parallel test execution.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13113) Enable parallel test execution for hadoop-aws.
[ https://issues.apache.org/jira/browse/HADOOP-13113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth updated HADOOP-13113: --- Status: Patch Available (was: Open) > Enable parallel test execution for hadoop-aws. > -- > > Key: HADOOP-13113 > URL: https://issues.apache.org/jira/browse/HADOOP-13113 > Project: Hadoop Common > Issue Type: Improvement > Components: test >Reporter: Chris Nauroth >Assignee: Chris Nauroth > Attachments: HADOOP-13113.001.patch > > > The full hadoop-aws test suite takes ~30 minutes to execute. The tests spend > most of their time blocked on network I/O with the S3 back-end, but they > don't saturate the bandwidth of the NIC. We can improve overall execution > time by enabling parallel test execution. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
