[
https://issues.apache.org/jira/browse/HADOOP-16644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16947525#comment-16947525
]
Steve Loughran commented on HADOOP-16644:
-----------------------------------------
yeah, I'd just seen that too, it comes back in the metadata. I just need to
pass it in through the finishedWrite.
My initial PR always does the HEAD on a non-dir PUT; we can enhance that.
There's a risk for overwrites the HEAD returns the previous version. If we have
the version ID all is good, but if not we can use the etag to verify we have
the right value -we'd have to retry to get the new one. And as we know, those
load balancers can cache for many seconds.
regarding localisation and credentials, see HADOOP-16233 -we have to mark the
status entries as encrypted so the shared cache is not used (it checks for
"world readable and ! encrypted for the shared cache). With that patch in, the
localisation is done as the user, and uses their DT.
I believe that this will then use the jobconf -we would have to check.
> Intermittent failure of ITestS3ATerasortOnS3A: timestamp differences
> --------------------------------------------------------------------
>
> Key: HADOOP-16644
> URL: https://issues.apache.org/jira/browse/HADOOP-16644
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3, test
> Affects Versions: 3.3.0
> Environment: -Dparallel-tests -DtestsThreadCount=8
> -Dfailsafe.runOrder=balanced -Ds3guard -Ddynamo -Dscale
> h2. Hypothesis:
> the timestamp of the source file is being picked up from S3Guard, but when
> the NM does a getFileStatus call, a HEAD check is made -and this (due to the
> overloaded test system) is out of sync with the listing. S3Guard is updated,
> the corrected date returned and the localisation fails.
> Reporter: Steve Loughran
> Priority: Major
>
> Terasort of directory committer failing in resource localisaton -the
> partitions.lst file has a different TS from that expected
> Happens under loaded integration tests (threads = 8; not standalone);
> non-auth s3guard
> {code}
> 2019-10-08 11:50:29,774 [IPC Server handler 4 on 55983] WARN
> localizer.ResourceLocalizationService
> (ResourceLocalizationService.java:processHeartbeat(1150)) - {
> s3a://hwdev-steve-ireland-new/terasort-directory/sortout/_partition.lst,
> 1570531828143, FILE, null } failed: Resource
> s3a://hwdev-steve-ireland-new/terasort-directory/sortout/_partition.lst
> changed on src filesystem (expected 1570531828143, was 1570531828000
> java.io.IOException: Resource
> s3a://hwdev-steve-ireland-new/terasort-directory/sortout/_partition.lst
> changed on src filesystem (expected 1570531828143, was 1570531828000
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
