Hi Chris,
This looks like a valid fix to me. Would you like to create a JIRA (in
Hadoop Common) for this with a patch?
Thanks,
Tom
On Sat, Sep 1, 2012 at 6:29 PM, Chris Collins <ch...@geekychris.com> wrote:
> Any comment on this?
> On Aug 28, 2012, at 11:43 PM, Chris Collins <ch...@geekychris.com> wrote:
>
>> I was attempting to use the natives3 file system outside of doing any map
>> reduce tasks. A simple task of trying to create a directory:
>>
>>
>> FileSystem fs = FileSystem.get(uri, conf);
>> Path currPath = new Path("/a/b/c");
>> fs.mkdirs(currPath);
>>
>> ( I can provide full code if needed).
>>
>> Anyway the class Jets3tNativeFileSystemStore attempts to detect if each key
>> part of the object path exists expecting a 404 response if it does not:
>>
>> public FileMetadata retrieveMetadata(String key) throws IOException {
>> try {
>> S3Object object = s3Service.getObjectDetails(bucket, key);
>> return new FileMetadata(key, object.getContentLength(),
>> object.getLastModifiedDate().getTime());
>> } catch (S3ServiceException e) {
>> // Following is brittle. Is there a better way?
>> if (e.getMessage().contains("ResponseCode=404")) {
>> return null;
>> }
>> if (e.getCause() instanceof IOException) {
>> throw (IOException) e.getCause();
>> }
>> throw new S3Exception(e);
>> }
>> }
>>
>> All version of jets3 I have looked at that seem to have a compatible class
>> structure (don't blow on AWSCredentials) actually return an exception
>> containing ".....ResponseCode: 404....
>>
>> I took a copy of the code in this directory and fixed the following to read:
>>
>> public FileMetadata retrieveMetadata(String key) throws IOException {
>> try {
>> S3Object object = s3Service.getObjectDetails(bucket, key);
>> return new FileMetadata(key, object.getContentLength(),
>> object.getLastModifiedDate().getTime());
>> } catch (S3ServiceException e) {
>> // Following is brittle. Is there a better way?
>> if (e.getResponseCode() == 404) {
>> return null;
>> }
>> if (e.getCause() instanceof IOException) {
>> throw (IOException) e.getCause();
>> }
>> throw new S3Exception(e);
>> }
>> }
>>
>> which seems to fix the issue. Am I missing something? Also this seems to
>> of been broken for a variety of hadoop versions. Does anyone actually use
>> this code path and if so is there a valid version combination that should of
>> worked for me?
>>
>> Comments welcome.
>>
>> Chris
>
