[
https://issues.apache.org/jira/browse/HADOOP-16207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16934024#comment-16934024
]
Siddharth Seth edited comment on HADOOP-16207 at 9/20/19 3:58 AM:
------------------------------------------------------------------
Also, to run the tests in parallel - the jobs need to start using a different
directory name. Currently, all of them use testMRJob (The method name in the
common class that all tests inherit from).
The issue with the local dir conflict is a MR configuration afaik (Likely the
MR tmp dir config property). YARN clusters should already be able to run in
parallel (different ports, random dir names, etc)
I'd also be careful trying to run too many of these in parallel, given the
amount of memory they consume. Maybe a different parallelism flag for any tests
running on a cluster?
was (Author: sseth):
Also, to run the tests in parallel - the jobs need to start using a different
directory name. Currently, all of them use testMRJob (The method name in the
common class that all tests inherit from).
The issue with the local dir conflict is a MR configuration afaik (Likely the
MR tmp dir config property). YARN clusters should already be able to run in
parallel (different ports, random dir names, etc)
> Fix ITestDirectoryCommitMRJob.testMRJob
> ---------------------------------------
>
> Key: HADOOP-16207
> URL: https://issues.apache.org/jira/browse/HADOOP-16207
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3, test
> Affects Versions: 3.3.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Critical
>
> Reported failure of {{ITestDirectoryCommitMRJob}} in validation runs of
> HADOOP-16186; assertIsDirectory with s3guard enabled and a parallel test run:
> Path "is recorded as deleted by S3Guard"
> {code}
> waitForConsistency();
> assertIsDirectory(outputPath) /* here */
> {code}
> The file is there but there's a tombstone. Possibilities
> * some race condition with another test
> * tombstones aren't timing out
> * committers aren't creating that base dir in a way which cleans up S3Guard's
> tombstones.
> Remember: we do have to delete that dest dir before the committer runs unless
> overwrite==true, so at the start of the run there will be a tombstone. It
> should be overwritten by a success.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
