[ https://issues.apache.org/jira/browse/HADOOP-15625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16772965#comment-16772965 ]
Steve Loughran commented on HADOOP-15625: ----------------------------------------- third party stores are always somewhat trouble: if you look @ the tests we always provide a way to turn off things (encryption, sessions, tokens, &c) we assume aren't there. you tend to get some different kinds of store * Full AWS compatible by people who put in the effort to help test (Kudos to WDC here) * ones which are fairly complete, but for which there are the odd corner case (if modified since etc). A good one: some don't handle a GET of content length 0-0 on a zero byte file, which we naively ask for on a 0 byte file. We should fix that our side for performance alone. * work in progress ones which should be using our client as part of their test suite (example: Ozone's S3 adapter, which is still rounding out stuff like multipart uploads) I think we should actually have a configurable policy here for versioning with a property like fs.s3a.change.detection * {{none}}: no checks. Turn on if either it's 3rd party or some problem with it is surfacing * {{etag}} server side with if-modified since * {{client}} client etag * {{warn}} client etag with a warning over a failure then if versioning support is added, a new option could be added. Yes, this complicates testing. Imagine in a new parameterized test which would be skipped entirely if the base test configuration for the test store was set to "none" looking at {{ITestS3AMiscOperations}} we have a test there (testChecksumLengthPastEOF) which relies on the checksum being non-null, hence the etag. And nobody has complained *yet*. But we do have those checksums disabled by default as it broke distcp (HADOOP-15297) so it may not have yet surfaced in the wild. It's actually that if-modified-since check which I worry about, because even though it's part of the HTTP spec, I can imagine some S3 implementation not doing it. A client-side variant would allow the checks but softly The other reason I like "warn" is it how we could downgrade handling of inconsistencies with s3guard to logging issues, but not failing. > S3A input stream to use etags to detect changed source files > ------------------------------------------------------------ > > Key: HADOOP-15625 > URL: https://issues.apache.org/jira/browse/HADOOP-15625 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 > Affects Versions: 3.2.0 > Reporter: Brahma Reddy Battula > Assignee: Brahma Reddy Battula > Priority: Major > Attachments: HADOOP-15625-001.patch, HADOOP-15625-002.patch, > HADOOP-15625-003.patch > > > S3A input stream doesn't handle changing source files any better than the > other cloud store connectors. Specifically: it doesn't noticed it has > changed, caches the length from startup, and whenever a seek triggers a new > GET, you may get one of: old data, new data, and even perhaps go from new > data to old data due to eventual consistency. > We can't do anything to stop this, but we could detect changes by > # caching the etag of the first HEAD/GET (we don't get that HEAD on open with > S3Guard, BTW) > # on future GET requests, verify the etag of the response > # raise an IOE if the remote file changed during the read. > It's a more dramatic failure, but it stops changes silently corrupting things. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org