[ https://issues.apache.org/jira/browse/HADOOP-16644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16977567#comment-16977567 ]
Steve Loughran commented on HADOOP-16644: ----------------------------------------- AWS SDK copyManager doesn't pass the modtime from a single file copy back to the caller, although it gets it back from S3. We'd have to move off the AWS xfer manager to get this, which is a big undertaking only justifiable for other reasons > Retrive modtime of PUT file from store, via response or HEAD > ------------------------------------------------------------ > > Key: HADOOP-16644 > URL: https://issues.apache.org/jira/browse/HADOOP-16644 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3, test > Affects Versions: 3.3.0 > Environment: -Dparallel-tests -DtestsThreadCount=8 > -Dfailsafe.runOrder=balanced -Ds3guard -Ddynamo -Dscale > h2. Hypothesis: > the timestamp of the source file is being picked up from S3Guard, but when > the NM does a getFileStatus call, a HEAD check is made -and this (due to the > overloaded test system) is out of sync with the listing. S3Guard is updated, > the corrected date returned and the localisation fails. > Reporter: Steve Loughran > Priority: Major > > Terasort of directory committer failing in resource localisaton -the > partitions.lst file has a different TS from that expected > Happens under loaded integration tests (threads = 8; not standalone); > non-auth s3guard > {code} > 2019-10-08 11:50:29,774 [IPC Server handler 4 on 55983] WARN > localizer.ResourceLocalizationService > (ResourceLocalizationService.java:processHeartbeat(1150)) - { > s3a://hwdev-steve-ireland-new/terasort-directory/sortout/_partition.lst, > 1570531828143, FILE, null } failed: Resource > s3a://hwdev-steve-ireland-new/terasort-directory/sortout/_partition.lst > changed on src filesystem (expected 1570531828143, was 1570531828000 > java.io.IOException: Resource > s3a://hwdev-steve-ireland-new/terasort-directory/sortout/_partition.lst > changed on src filesystem (expected 1570531828143, was 1570531828000 > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org