[ https://issues.apache.org/jira/browse/HADOOP-16644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16947209#comment-16947209 ]
Siddharth Seth commented on HADOOP-16644: ----------------------------------------- Looks like a PUTRequest gives back the modification time, a multipart upload does not. Given a multipart upload is likely a long operation anyway - a HEAD request following a MultiPartComplete call likely doesn't add a large percentage to the operation time (only is S3Guard enabled). For a direct PUT - we have the data anyway. Will definitely make me happy to avoid writing to DDB during a getSTatus operation. Using S3 for resource localization - that's got at least one issue which I'm aware of. Need to test this, and then file a YARN jira. Essentially - I suspect the localizer does not use the JobClient config - so any credentials there will not be available to YARN for localization (e.g. client sets up access_key and secret_key in config). > Intermittent failure of ITestS3ATerasortOnS3A: timestamp differences > -------------------------------------------------------------------- > > Key: HADOOP-16644 > URL: https://issues.apache.org/jira/browse/HADOOP-16644 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3, test > Affects Versions: 3.3.0 > Environment: -Dparallel-tests -DtestsThreadCount=8 > -Dfailsafe.runOrder=balanced -Ds3guard -Ddynamo -Dscale > h2. Hypothesis: > the timestamp of the source file is being picked up from S3Guard, but when > the NM does a getFileStatus call, a HEAD check is made -and this (due to the > overloaded test system) is out of sync with the listing. S3Guard is updated, > the corrected date returned and the localisation fails. > Reporter: Steve Loughran > Priority: Major > > Terasort of directory committer failing in resource localisaton -the > partitions.lst file has a different TS from that expected > Happens under loaded integration tests (threads = 8; not standalone); > non-auth s3guard > {code} > 2019-10-08 11:50:29,774 [IPC Server handler 4 on 55983] WARN > localizer.ResourceLocalizationService > (ResourceLocalizationService.java:processHeartbeat(1150)) - { > s3a://hwdev-steve-ireland-new/terasort-directory/sortout/_partition.lst, > 1570531828143, FILE, null } failed: Resource > s3a://hwdev-steve-ireland-new/terasort-directory/sortout/_partition.lst > changed on src filesystem (expected 1570531828143, was 1570531828000 > java.io.IOException: Resource > s3a://hwdev-steve-ireland-new/terasort-directory/sortout/_partition.lst > changed on src filesystem (expected 1570531828143, was 1570531828000 > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org