[ https://issues.apache.org/jira/browse/HADOOP-18508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17848361#comment-17848361 ]
ASF GitHub Bot commented on HADOOP-18508: ----------------------------------------- mukund-thakur commented on code in PR #5081: URL: https://github.com/apache/hadoop/pull/5081#discussion_r1608853482 ########## hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/testing.md: ########## @@ -539,12 +539,51 @@ Otherwise, set a large timeout in `fs.s3a.scale.test.timeout` The tests are executed in an order to only clean up created files after the end of all the tests. If the tests are interrupted, the test data will remain. +## <a name="CI"/> Testing through continuous integration + +### Parallel CI builds. +For CI testing of the module, including the integration tests, +it is generally necessary to support testing multiple PRs simultaneously. + +To do this +1. A job ID must be supplied in the `job.id` property, so each job works on an isolated directory + tree. This should be a number or unique string, which will be used within a path element, so + must only contain characters valid in an S3/hadoop path element. +2. Root directory tests need to be disabled by setting `fs.s3a.root.tests.enabled` to + `false`, either in the command line to maven or in the XML configurations. + +``` +mvn verify -T 1C -Dparallel-tests -DtestsThreadCount=14 -Dscale -Dfs.s3a.root.tests.enabled=false -Djob.id=001 +``` + +This parallel execution feature is only for isolated builds sharing a single S3 bucket; it does +not support parallel builds and tests from the same local source tree. Review Comment: ah..already mentioned here so we are good. > support multiple s3a integration test runs on same bucket in parallel > --------------------------------------------------------------------- > > Key: HADOOP-18508 > URL: https://issues.apache.org/jira/browse/HADOOP-18508 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3, test > Affects Versions: 3.3.9 > Reporter: Steve Loughran > Assignee: Steve Loughran > Priority: Minor > Labels: pull-request-available > > to have (internal, sorry) jenkins test runs work in parallel, they need to > share the same bucket so > # must have a prefix for job id which is passed in to the path used for forks > # support disabling root tests so they don't stamp on each other -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org