[ https://issues.apache.org/jira/browse/HADOOP-15229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16727091#comment-16727091 ]
Sameer Choudhary commented on HADOOP-15229: ------------------------------------------- [~ste...@apache.org] Thanks for providing the inputs on https://issues.apache.org/jira/browse/HADOOP-15364. Will use this Jira for all future feedback and discussion related to S3 Select support for S3A. {quote}SelectInputStream uses Invoker.once() to translate exceptions raised in the read() operation of the wrapped stream. Because now its not just an HTTP connection, it's a chain of paged results from the S3 select call, and it can fail in new ways...even socket exceptions get translated into AWS SDK exceptions. These are now converted back to some form of IOE. {quote} This is partially correct. As I mentioned before in HADOOP-15364, client receives 200 HTTP Status Code at the start of streaming. Not all exceptions will be reported by the SDK as EventStreamingException. For SDK to return EventStreamingException, it must receive a "Request Level Error Message" from the service. The specifics of the protocol are documented at [https://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectSELECTContent.html.] There could be cases where the service doesn't send this Message to the client (in case of an unknown service side bug that results in early termination of some query) or any other network issues. The client in that case will not get any exception and will have partial output which could be treated as OK response, which would be incorrect. Thus assertion on EndEvent is required. Adding the link to sample code showing proper handling again for reference [https://docs.aws.amazon.com/AmazonS3/latest/dev/SelectObjectContentUsingJava.html.] {quote}S3A Invoker is used to wrap IO the initial [select call|https://github.com/steveloughran/hadoop/blob/filesystem/HADOOP-15229-openfile/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java#L3527] is retried with the [normal retry policy|https://github.com/steveloughran/hadoop/blob/filesystem/HADOOP-15229-openfile/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3ARetryPolicy.java]. If there are specific errors which need extra retry/fast fail, that'd be handy. There's no attempt to recover from failures in a read() itself, the SDK exceptions are simply mapped/wrapped to IOEs. [https://github.com/steveloughran/hadoop/blob/802fcc9f4d80f6d11582c175efa57ed580d0b25d/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/select/SelectInputStream.java#L210] looking @ that code, I think we should be collecting stats on failures there {quote} Could we raise IOE from the Filesystem in this case along with collecting the stats? This would require additional code to track the EndEvent in addition to wrapping SDK exceptions. {quote}always. See [https://github.com/steveloughran/hadoop/blob/filesystem/HADOOP-15229-openfile/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/s3_select.md] The ITest* Tests do this, [in this package|https://github.com/steveloughran/hadoop/tree/filesystem/HADOOP-15229-openfile/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/select] Test policy is covered in [S3a test policy|https://hadoop.apache.org/docs/r3.1.1/hadoop-aws/tools/hadoop-aws/testing.html]. {quote} Thanks for sharing the pointers. Will take a look. {quote}Get set up for test runs first and then think about what more can go in. {quote} Sure, will review current test suite and provide feedback. {quote}For s3 select it'd be good to have some other standard source files to test against; landsat.csv.gz only stresses the gzip and csv logic, something with timestamps which can be parsed and in bzip2 would be nice. These should be public & free: our developers don't have the time nor necessarily the funding to create/store these themselves. One thing which isn't tested yet is handling of bad data: what if the CSV has inconsistent numbers of rows, mixes tabs and spaces, datatypes inconsistent, not all datetimes parseable etc. Someone should really write those, if just to have more error messages and stack traces for the docs. {quote} I agree, having authoritative test datasets to assert implementation correctness for projects would be great. Let me discuss this internally with the team and get back to you. Can't provide an ETA on this though at this point in time. > Add FileSystem builder-based openFile() API to match createFile() > ----------------------------------------------------------------- > > Key: HADOOP-15229 > URL: https://issues.apache.org/jira/browse/HADOOP-15229 > Project: Hadoop Common > Issue Type: New Feature > Components: fs, fs/azure, fs/s3 > Affects Versions: 3.0.0 > Reporter: Steve Loughran > Assignee: Steve Loughran > Priority: Major > Attachments: HADOOP-15229-001.patch, HADOOP-15229-002.patch, > HADOOP-15229-003.patch, HADOOP-15229-004.patch, HADOOP-15229-004.patch, > HADOOP-15229-005.patch, HADOOP-15229-006.patch, HADOOP-15229-007.patch, > HADOOP-15229-009.patch, HADOOP-15229-010.patch, HADOOP-15229-011.patch, > HADOOP-15229-012.patch, HADOOP-15229-013.patch, HADOOP-15229-014.patch > > > Replicate HDFS-1170 and HADOOP-14365 with an API to open files. > A key requirement of this is not HDFS, it's to put in the fadvise policy for > working with object stores, where getting the decision to do a full GET and > TCP abort on seek vs smaller GETs is fundamentally different: the wrong > option can cost you minutes. S3A and Azure both have adaptive policies now > (first backward seek), but they still don't do it that well. > Columnar formats (ORC, Parquet) should be able to say "fs.input.fadvise" > "random" as an option when they open files; I can imagine other options too. > The Builder model of [~eddyxu] is the one to mimic, method for method. > Ideally with as much code reuse as possible -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org