[jira] [Commented] (HADOOP-16193) add extra S3A MPU test to see what happens if a file is created during the MPU

[ https://issues.apache.org/jira/browse/HADOOP-16193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16918669#comment-16918669 ] Thomas Demoor commented on HADOOP-16193: [~ehiggs] is correct. On AWS, the multipart put, the

[jira] [Commented] (HADOOP-9565) Add a Blobstore interface to add to blobstore FileSystems

[ https://issues.apache.org/jira/browse/HADOOP-9565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997360#comment-15997360 ] Thomas Demoor commented on HADOOP-9565: --- This ticket has been worked on by multiple people. Steve

[jira] [Commented] (HADOOP-14305) S3A SSE tests won't run in parallel: Bad request in directory GetFileStatus

[ https://issues.apache.org/jira/browse/HADOOP-14305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15975556#comment-15975556 ] Thomas Demoor commented on HADOOP-14305: [~ste...@apache.org] We currently only support SSE-S3,

[jira] [Commented] (HADOOP-13786) Add S3Guard committer for zero-rename commits to consistent S3 endpoints

[ https://issues.apache.org/jira/browse/HADOOP-13786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15937149#comment-15937149 ] Thomas Demoor commented on HADOOP-13786: On the staging wrappedCommitter exception on s3a. Could

[jira] [Commented] (HADOOP-13786) Add S3Guard committer for zero-rename commits to consistent S3 endpoints

[ https://issues.apache.org/jira/browse/HADOOP-13786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15905342#comment-15905342 ] Thomas Demoor commented on HADOOP-13786: Hi Ryan, thanks for contributing to this effort. Both

[jira] [Commented] (HADOOP-13946) Document how HDFS updates timestamps in the FS spec; compare with object stores

[ https://issues.apache.org/jira/browse/HADOOP-13946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15899437#comment-15899437 ] Thomas Demoor commented on HADOOP-13946: my previous remarks have been addressed. didn't find

[jira] [Commented] (HADOOP-13282) S3 blob etags to be made visible in status/getFileChecksum() calls

[ https://issues.apache.org/jira/browse/HADOOP-13282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870601#comment-15870601 ] Thomas Demoor commented on HADOOP-13282: ETag: * singlepart= md5sum * multipart = (md5sum of the

[jira] [Commented] (HADOOP-14028) S3A block output streams don't delete temporary files in multipart uploads

[ https://issues.apache.org/jira/browse/HADOOP-14028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15868162#comment-15868162 ] Thomas Demoor commented on HADOOP-14028: Not sure yet, looking at the aws sdk code but I'm still

[jira] [Commented] (HADOOP-14071) S3a: Failed to reset the request input stream

[ https://issues.apache.org/jira/browse/HADOOP-14071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15861404#comment-15861404 ] Thomas Demoor commented on HADOOP-14071: For the ByteArrayInputStream & ByteBufferInputStream I

[jira] [Commented] (HADOOP-14028) S3A block output streams don't delete temporary files in multipart uploads

[ https://issues.apache.org/jira/browse/HADOOP-14028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15854824#comment-15854824 ] Thomas Demoor commented on HADOOP-14028: I'll do some testing tomorrow. > S3A block output

[jira] [Commented] (HADOOP-13974) S3a CLI to support list/purge of pending multipart commits

[ https://issues.apache.org/jira/browse/HADOOP-13974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15848410#comment-15848410 ] Thomas Demoor commented on HADOOP-13974: For purging we use TransferManagers

[jira] [Assigned] (HADOOP-9565) Add a Blobstore interface to add to blobstore FileSystems

[ https://issues.apache.org/jira/browse/HADOOP-9565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Demoor reassigned HADOOP-9565: - Assignee: Thomas Demoor (was: Pieter Reuse) > Add a Blobstore interface to add to

[jira] [Commented] (HADOOP-13946) Document how HDFS updates timestamps in the FS spec; compare with object stores

[ https://issues.apache.org/jira/browse/HADOOP-13946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15809887#comment-15809887 ] Thomas Demoor commented on HADOOP-13946: A copy is an independent immutable object (which

[jira] [Commented] (HADOOP-13946) Document how HDFS updates timestamps in the FS spec; compare with object stores

[ https://issues.apache.org/jira/browse/HADOOP-13946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15801582#comment-15801582 ] Thomas Demoor commented on HADOOP-13946: Yup, same for multipart. The timestamp of the initiate

[jira] [Commented] (HADOOP-13946) Document how HDFS updates timestamps in the FS spec; compare with object stores

[ https://issues.apache.org/jira/browse/HADOOP-13946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15796426#comment-15796426 ] Thomas Demoor commented on HADOOP-13946: Another late review remark: bq. The file only becomes

[jira] [Commented] (HADOOP-13912) S3a Multipart Committer (avoid rename)

[ https://issues.apache.org/jira/browse/HADOOP-13912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15761057#comment-15761057 ] Thomas Demoor commented on HADOOP-13912: Extensive write-up by [~ste...@apache.org] based on the

[jira] [Commented] (HADOOP-13786) add output committer which uses s3guard for consistent commits to S3

[ https://issues.apache.org/jira/browse/HADOOP-13786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15755099#comment-15755099 ] Thomas Demoor commented on HADOOP-13786: Had my write-up ready but you beat me too actually

[jira] [Created] (HADOOP-13912) S3a Multipart Committer (avoid rename)

Thomas Demoor created HADOOP-13912: -- Summary: S3a Multipart Committer (avoid rename) Key: HADOOP-13912 URL: https://issues.apache.org/jira/browse/HADOOP-13912 Project: Hadoop Common Issue

[jira] [Commented] (HADOOP-13826) S3A Deadlock in multipart copy due to thread pool limits.

[ https://issues.apache.org/jira/browse/HADOOP-13826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15743421#comment-15743421 ] Thomas Demoor commented on HADOOP-13826: [~mackrorysd] your last patch seems simple but I think

[jira] [Commented] (HADOOP-13695) S3A to use a thread pool for async path operations

[ https://issues.apache.org/jira/browse/HADOOP-13695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15726823#comment-15726823 ] Thomas Demoor commented on HADOOP-13695: The netflix solution is "dirty but efficient" if you

[jira] [Commented] (HADOOP-13826) S3A Deadlock in multipart copy due to thread pool limits.

[ https://issues.apache.org/jira/browse/HADOOP-13826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15709695#comment-15709695 ] Thomas Demoor commented on HADOOP-13826: I think [~mackrorysd]'s implementation is heading in

[jira] [Commented] (HADOOP-13421) Switch to v2 of the S3 List Objects API in S3A

[ https://issues.apache.org/jira/browse/HADOOP-13421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15705631#comment-15705631 ] Thomas Demoor commented on HADOOP-13421: Thanks for reaching out [~ste...@apache.org]

[jira] [Commented] (HADOOP-13826) S3A Deadlock in multipart copy due to thread pool limits.

[ https://issues.apache.org/jira/browse/HADOOP-13826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15691219#comment-15691219 ] Thomas Demoor commented on HADOOP-13826: Yes, that is set on the AmazonS3Client, which we use

[jira] [Commented] (HADOOP-13826) S3A Deadlock in multipart copy due to thread pool limits.

[ https://issues.apache.org/jira/browse/HADOOP-13826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15689382#comment-15689382 ] Thomas Demoor commented on HADOOP-13826: Mmmh, sorry, I caused this when introducing the

[jira] [Commented] (HADOOP-13695) S3A to use a thread pool for async path operations

[ https://issues.apache.org/jira/browse/HADOOP-13695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15605184#comment-15605184 ] Thomas Demoor commented on HADOOP-13695: We really like the idea of the separate threadpools per

[jira] [Commented] (HADOOP-13262) set multipart delete timeout to 5 * 60s in S3ATestUtils.createTestFileSystem

[ https://issues.apache.org/jira/browse/HADOOP-13262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15506494#comment-15506494 ] Thomas Demoor commented on HADOOP-13262: +1 > set multipart delete timeout to 5 * 60s in

[jira] [Commented] (HADOOP-13568) S3AFastOutputStream to implement flush()

[ https://issues.apache.org/jira/browse/HADOOP-13568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15473830#comment-15473830 ] Thomas Demoor commented on HADOOP-13568: This might be hard as only the last part of a MPU can

[jira] [Commented] (HADOOP-11684) S3a to use thread pool that blocks clients

[ https://issues.apache.org/jira/browse/HADOOP-11684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15473544#comment-15473544 ] Thomas Demoor commented on HADOOP-11684: [~ste...@apache.org] you should not make the queue

[jira] [Comment Edited] (HADOOP-9565) Add a Blobstore interface to add to blobstore FileSystems

[ https://issues.apache.org/jira/browse/HADOOP-9565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15409534#comment-15409534 ] Thomas Demoor edited comment on HADOOP-9565 at 8/5/16 2:41 PM: --- Steve the

[jira] [Commented] (HADOOP-9565) Add a Blobstore interface to add to blobstore FileSystems

[ https://issues.apache.org/jira/browse/HADOOP-9565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15409534#comment-15409534 ] Thomas Demoor commented on HADOOP-9565: --- Steve the "avoid data write" thing you mention is exactly

[jira] [Commented] (HADOOP-12878) Impersonate hosts in s3a for better data locality handling

[ https://issues.apache.org/jira/browse/HADOOP-12878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15367772#comment-15367772 ] Thomas Demoor commented on HADOOP-12878: [~cnauroth] pointed out that the issue I originally ran

[jira] [Commented] (HADOOP-12878) Impersonate hosts in s3a for better data locality handling

[ https://issues.apache.org/jira/browse/HADOOP-12878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15324313#comment-15324313 ] Thomas Demoor commented on HADOOP-12878: [~cnauroth]: I assumed this would be used as follows:

[jira] [Commented] (HADOOP-13131) Add tests to verify that S3A supports SSE-S3 encryption

[ https://issues.apache.org/jira/browse/HADOOP-13131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15292150#comment-15292150 ] Thomas Demoor commented on HADOOP-13131: [~PieterReuse] ran some test on our "non-AWS s3

[jira] [Commented] (HADOOP-11687) Ignore x-emc-* headers when copying an Amazon S3 object

[ https://issues.apache.org/jira/browse/HADOOP-11687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15179718#comment-15179718 ] Thomas Demoor commented on HADOOP-11687: This should also fix another bug. The current clone

[jira] [Created] (HADOOP-12878) Impersonate hosts in s3a for better data locality handling

Thomas Demoor created HADOOP-12878: -- Summary: Impersonate hosts in s3a for better data locality handling Key: HADOOP-12878 URL: https://issues.apache.org/jira/browse/HADOOP-12878 Project: Hadoop

[jira] [Commented] (HADOOP-12444) Consider implementing lazy seek in S3AInputStream

[ https://issues.apache.org/jira/browse/HADOOP-12444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15155079#comment-15155079 ] Thomas Demoor commented on HADOOP-12444: Ran tests: get through readFully fails: HTTP 416-

[jira] [Commented] (HADOOP-12548) read s3 creds from a Credential Provider

[ https://issues.apache.org/jira/browse/HADOOP-12548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15145959#comment-15145959 ] Thomas Demoor commented on HADOOP-12548: Hi [~lmccay], thanks, finally someone cleaned up our

[jira] [Commented] (HADOOP-12548) read s3 creds from a Credential Provider

[ https://issues.apache.org/jira/browse/HADOOP-12548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15145997#comment-15145997 ] Thomas Demoor commented on HADOOP-12548: Right, I forgot we started using [~aw]'s magic. Thanks!

[jira] [Commented] (HADOOP-12353) S3 Native filesystem does not retry all connection failures

[ https://issues.apache.org/jira/browse/HADOOP-12353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15146008#comment-15146008 ] Thomas Demoor commented on HADOOP-12353: [~ste...@apache.org]: s3a does retry as the AWS SDK has

[jira] [Assigned] (HADOOP-9565) Add a Blobstore interface to add to blobstore FileSystems

[ https://issues.apache.org/jira/browse/HADOOP-9565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Demoor reassigned HADOOP-9565: - Assignee: Thomas Demoor (was: Steve Loughran) > Add a Blobstore interface to add to

[jira] [Updated] (HADOOP-9565) Add a Blobstore interface to add to blobstore FileSystems

[ https://issues.apache.org/jira/browse/HADOOP-9565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Demoor updated HADOOP-9565: -- Attachment: HADOOP-9565-005.patch Adjusted distcp: removed tempfile usage for object stores

[jira] [Commented] (HADOOP-9565) Add a Blobstore interface to add to blobstore FileSystems

[ https://issues.apache.org/jira/browse/HADOOP-9565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15146014#comment-15146014 ] Thomas Demoor commented on HADOOP-9565: --- Also fixed the javadoc issues (thanks [~emres]). > Add a

[jira] [Resolved] (HADOOP-12319) S3AFastOutputStream has no ability to apply backpressure

[ https://issues.apache.org/jira/browse/HADOOP-12319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Demoor resolved HADOOP-12319. Resolution: Fixed Fixed by HADOOP-11684 > S3AFastOutputStream has no ability to apply

[jira] [Commented] (HADOOP-12292) Make use of DeleteObjects optional

[ https://issues.apache.org/jira/browse/HADOOP-12292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15135998#comment-15135998 ] Thomas Demoor commented on HADOOP-12292: Thanks [~ste...@apache.org] for adding the test and

[jira] [Resolved] (HADOOP-12763) S3AFileSystem And Hadoop FsShell Operations

[ https://issues.apache.org/jira/browse/HADOOP-12763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Demoor resolved HADOOP-12763. Resolution: Duplicate Hi Stephen, I think this has been fixed in HADOOP-11918. Closing as

[jira] [Commented] (HADOOP-11262) Enable YARN to use S3A

[ https://issues.apache.org/jira/browse/HADOOP-11262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15095907#comment-15095907 ] Thomas Demoor commented on HADOOP-11262: Thank you for supporting this feature and for your

[jira] [Commented] (HADOOP-12671) Inconsistent configuration values and incorrect comments

[ https://issues.apache.org/jira/browse/HADOOP-12671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15069467#comment-15069467 ] Thomas Demoor commented on HADOOP-12671: Good catches, [~tianyin]! >From inspecting the history

[jira] [Commented] (HADOOP-12632) Allow S3A to use Temporary Credentials for Auth

[ https://issues.apache.org/jira/browse/HADOOP-12632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15050268#comment-15050268 ] Thomas Demoor commented on HADOOP-12632: Hi Joe, at first glance this seems to duplicate

[jira] [Resolved] (HADOOP-12632) Allow S3A to use Temporary Credentials for Auth

[ https://issues.apache.org/jira/browse/HADOOP-12632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Demoor resolved HADOOP-12632. Resolution: Duplicate > Allow S3A to use Temporary Credentials for Auth >

[jira] [Commented] (HADOOP-12548) read s3 creds from a Credential Provider

[ https://issues.apache.org/jira/browse/HADOOP-12548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15006034#comment-15006034 ] Thomas Demoor commented on HADOOP-12548: Relating to HADOOP-11694: s3a phase II improvements and

[jira] [Commented] (HADOOP-12548) read s3 creds from a file

[ https://issues.apache.org/jira/browse/HADOOP-12548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14989393#comment-14989393 ] Thomas Demoor commented on HADOOP-12548: I would like to point you to HADOOP-11670, where an

[jira] [Commented] (HADOOP-11262) Enable YARN to use S3A

[ https://issues.apache.org/jira/browse/HADOOP-11262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14989408#comment-14989408 ] Thomas Demoor commented on HADOOP-11262: Thanks for your time reviewing this. [~cnauroth] is

[jira] [Commented] (HADOOP-10400) Incorporate new S3A FileSystem implementation

[ https://issues.apache.org/jira/browse/HADOOP-10400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14970895#comment-14970895 ] Thomas Demoor commented on HADOOP-10400: Alternatively, the last patch uploaded in HADOOP-9565

[jira] [Commented] (HADOOP-12496) AWS SDK version

[ https://issues.apache.org/jira/browse/HADOOP-12496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14966398#comment-14966398 ] Thomas Demoor commented on HADOOP-12496: Hi Yongjia, HADOOP-12269 upgraded aws-sdk-s3 to 1.10.6

[jira] [Commented] (HADOOP-12444) Consider implementing lazy seek in S3AInputStream

[ https://issues.apache.org/jira/browse/HADOOP-12444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14958539#comment-14958539 ] Thomas Demoor commented on HADOOP-12444: Hi [~rajesh.balamohan], that performance improvement

[jira] [Commented] (HADOOP-11684) S3a to use thread pool that blocks clients

[ https://issues.apache.org/jira/browse/HADOOP-11684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14939600#comment-14939600 ] Thomas Demoor commented on HADOOP-11684: S3a has 2 modes for uploading: *

[jira] [Updated] (HADOOP-12444) Consider implementing lazy seek in S3AInputStream

[ https://issues.apache.org/jira/browse/HADOOP-12444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Demoor updated HADOOP-12444: --- Issue Type: Sub-task (was: Improvement) Parent: HADOOP-11694 > Consider

[jira] [Commented] (HADOOP-12444) Consider implementing lazy seek in S3AInputStream

[ https://issues.apache.org/jira/browse/HADOOP-12444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14936637#comment-14936637 ] Thomas Demoor commented on HADOOP-12444: I recall a discussion about this functionality in the

[jira] [Commented] (HADOOP-11684) S3a to use thread pool that blocks clients

[ https://issues.apache.org/jira/browse/HADOOP-11684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14936822#comment-14936822 ] Thomas Demoor commented on HADOOP-11684: One has to take into account that s3a runs within the

[jira] [Moved] (HADOOP-12444) Consider implementing lazy seek in S3AInputStream

[ https://issues.apache.org/jira/browse/HADOOP-12444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Demoor moved HDFS-9150 to HADOOP-12444: -- Component/s: (was: HDFS) Issue Type: Improvement (was: Bug)

[jira] [Updated] (HADOOP-11684) S3a to use thread pool that blocks clients

[ https://issues.apache.org/jira/browse/HADOOP-11684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Demoor updated HADOOP-11684: --- Attachment: HADOOP-11684-003.patch 003.patch throttles using {{CallerRunsPolicy}}. Changed

[jira] [Updated] (HADOOP-12292) Make use of DeleteObjects optional

[ https://issues.apache.org/jira/browse/HADOOP-12292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Demoor updated HADOOP-12292: --- Attachment: HADOOP-12292-003.patch Note to self: rerun test-patch, even if you only changed

[jira] [Updated] (HADOOP-12292) Make use of DeleteObjects optional

[ https://issues.apache.org/jira/browse/HADOOP-12292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Demoor updated HADOOP-12292: --- Attachment: HADOOP-12292-002.patch I agree we'd rather not be blatant :p So, 002.patch

[jira] [Updated] (HADOOP-11918) Listing an empty s3a root directory throws FileNotFound.

[ https://issues.apache.org/jira/browse/HADOOP-11918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Demoor updated HADOOP-11918: --- Attachment: HADOOP-11918-003.patch I uploaded patch-003, which is what we are running. Feel

[jira] [Commented] (HADOOP-12420) While trying to access Amazon S3 through hadoop-aws(Spark basically) I was getting Exception in thread "main" java.lang.NoSuchMethodError: com.amazonaws.services.s3.t

[ https://issues.apache.org/jira/browse/HADOOP-12420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14875693#comment-14875693 ] Thomas Demoor commented on HADOOP-12420: Ok, just see now that it was backported to 2 already,

[jira] [Commented] (HADOOP-11684) S3a to use thread pool that blocks clients

[ https://issues.apache.org/jira/browse/HADOOP-11684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14875712#comment-14875712 ] Thomas Demoor commented on HADOOP-11684: As said previously, I agree with your points above. I

[jira] [Commented] (HADOOP-12420) While trying to access Amazon S3 through hadoop-aws(Spark basically) I was getting Exception in thread "main" java.lang.NoSuchMethodError: com.amazonaws.services.s3.t

[ https://issues.apache.org/jira/browse/HADOOP-12420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14875689#comment-14875689 ] Thomas Demoor commented on HADOOP-12420: I'm not sure I understand the issue completely. Spark

[jira] [Created] (HADOOP-12387) Improve S3AFastOutputStream memory management

Thomas Demoor created HADOOP-12387: -- Summary: Improve S3AFastOutputStream memory management Key: HADOOP-12387 URL: https://issues.apache.org/jira/browse/HADOOP-12387 Project: Hadoop Common

[jira] [Commented] (HADOOP-11684) S3a to use thread pool that blocks clients

[ https://issues.apache.org/jira/browse/HADOOP-11684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14727276#comment-14727276 ] Thomas Demoor commented on HADOOP-11684: Thanks for the review Aaron. I did. I think that with

[jira] [Commented] (HADOOP-12353) S3 Native filesystem does not retry all connection failures

[ https://issues.apache.org/jira/browse/HADOOP-12353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14712842#comment-14712842 ] Thomas Demoor commented on HADOOP-12353: Hi [~szczeles], Just giving you a

[jira] [Commented] (HADOOP-9565) Add a Blobstore interface to add to blobstore FileSystems

[ https://issues.apache.org/jira/browse/HADOOP-9565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14702718#comment-14702718 ] Thomas Demoor commented on HADOOP-9565: --- Hi Nick, [~ste...@apache.org] also

[jira] [Commented] (HADOOP-12319) S3AFastOutputStream has no ability to apply backpressure

[ https://issues.apache.org/jira/browse/HADOOP-12319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14694923#comment-14694923 ] Thomas Demoor commented on HADOOP-12319: Hi Colin. Completely correct

[jira] [Updated] (HADOOP-11684) S3a to use thread pool that blocks clients

[ https://issues.apache.org/jira/browse/HADOOP-11684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Demoor updated HADOOP-11684: --- Target Version/s: 2.8.0, 3.0.0 (was: 3.0.0) Status: Patch Available (was:

[jira] [Commented] (HADOOP-10643) Add NativeS3Fs that delgates calls from FileContext apis to native s3 fs implementation

[ https://issues.apache.org/jira/browse/HADOOP-10643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14681995#comment-14681995 ] Thomas Demoor commented on HADOOP-10643: This request was originally for s3n.

[jira] [Updated] (HADOOP-11684) S3a to use thread pool that blocks clients

[ https://issues.apache.org/jira/browse/HADOOP-11684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Demoor updated HADOOP-11684: --- Attachment: HADOOP-11684-002.patch The aws-sdk version bump was seperated out to

[jira] [Updated] (HADOOP-11918) Listing an empty s3a root directory throws FileNotFound.

[ https://issues.apache.org/jira/browse/HADOOP-11918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Demoor updated HADOOP-11918: --- Issue Type: Sub-task (was: Bug) Parent: HADOOP-11694 Listing an empty s3a root

[jira] [Updated] (HADOOP-12292) Make use of DeleteObjects optional

[ https://issues.apache.org/jira/browse/HADOOP-12292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Demoor updated HADOOP-12292: --- Attachment: HADOOP-12292-001.patch Make use of DeleteObjects optional

[jira] [Updated] (HADOOP-12292) Make use of DeleteObjects optional

[ https://issues.apache.org/jira/browse/HADOOP-12292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Demoor updated HADOOP-12292: --- Status: Patch Available (was: Open) Make use of DeleteObjects optional

[jira] [Commented] (HADOOP-12292) Make use of DeleteObjects optional

[ https://issues.apache.org/jira/browse/HADOOP-12292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14651506#comment-14651506 ] Thomas Demoor commented on HADOOP-12292: Oh, now I get what you

[jira] [Created] (HADOOP-12292) Make use of DeleteObjects optional

Thomas Demoor created HADOOP-12292: -- Summary: Make use of DeleteObjects optional Key: HADOOP-12292 URL: https://issues.apache.org/jira/browse/HADOOP-12292 Project: Hadoop Common Issue Type:

[jira] [Updated] (HADOOP-12269) Update aws-sdk dependency to 1.10.6

[ https://issues.apache.org/jira/browse/HADOOP-12269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Demoor updated HADOOP-12269: --- Status: Patch Available (was: Open) Update aws-sdk dependency to 1.10.6

[jira] [Updated] (HADOOP-12269) Update aws-sdk dependency to 1.10.6

[ https://issues.apache.org/jira/browse/HADOOP-12269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Demoor updated HADOOP-12269: --- Attachment: HADOOP-12269-002.patch Patch 2 sets multipart threshold back to ~ 2GB and

[jira] [Commented] (HADOOP-12269) Update aws-sdk dependency to 1.10.6

[ https://issues.apache.org/jira/browse/HADOOP-12269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14644403#comment-14644403 ] Thomas Demoor commented on HADOOP-12269: Setting defaults is difficult as the

[jira] [Updated] (HADOOP-12269) Update aws-sdk dependency to 1.10.6

[ https://issues.apache.org/jira/browse/HADOOP-12269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Demoor updated HADOOP-12269: --- Target Version/s: 2.8.0 As soon as possible (just like all my other patches :P). For

[jira] [Created] (HADOOP-12269) Update aws-sdk dependency version

Thomas Demoor created HADOOP-12269: -- Summary: Update aws-sdk dependency version Key: HADOOP-12269 URL: https://issues.apache.org/jira/browse/HADOOP-12269 Project: Hadoop Common Issue Type:

[jira] [Commented] (HADOOP-12267) s3a failure due to integer overflow bug in AWS SDK

[ https://issues.apache.org/jira/browse/HADOOP-12267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14640184#comment-14640184 ] Thomas Demoor commented on HADOOP-12267: Hi Aaron, in HADOOP-11684 I have

[jira] [Updated] (HADOOP-12269) Update aws-sdk dependency version

[ https://issues.apache.org/jira/browse/HADOOP-12269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Demoor updated HADOOP-12269: --- Description: This was originally part of HADOOP-11684 Update aws-sdk dependency version

[jira] [Updated] (HADOOP-12269) Update aws-sdk dependency to 1.10.6

[ https://issues.apache.org/jira/browse/HADOOP-12269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Demoor updated HADOOP-12269: --- Summary: Update aws-sdk dependency to 1.10.6 (was: Update aws-sdk dependency version)

[jira] [Updated] (HADOOP-12269) Update aws-sdk dependency to 1.10.6

[ https://issues.apache.org/jira/browse/HADOOP-12269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Demoor updated HADOOP-12269: --- Attachment: HADOOP-12269-001.patch Patch bumps from aws-sdk-1.7.4 to aws-sdk-s3-1.10.6. *

[jira] [Commented] (HADOOP-12267) s3a failure due to integer overflow bug in AWS SDK

[ https://issues.apache.org/jira/browse/HADOOP-12267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14640584#comment-14640584 ] Thomas Demoor commented on HADOOP-12267: I have isolated the aws-sdk bump in

[jira] [Updated] (HADOOP-12269) Update aws-sdk dependency version

[ https://issues.apache.org/jira/browse/HADOOP-12269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Demoor updated HADOOP-12269: --- Description: This was originally part of HADOOP-11684, pulling out to this separate subtask

[jira] [Commented] (HADOOP-9565) Add a Blobstore interface to add to blobstore FileSystems

[ https://issues.apache.org/jira/browse/HADOOP-9565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14622385#comment-14622385 ] Thomas Demoor commented on HADOOP-9565: --- Thanks for your review [~eddyxu]. Indeed,

[jira] [Updated] (HADOOP-9565) Add a Blobstore interface to add to blobstore FileSystems

[ https://issues.apache.org/jira/browse/HADOOP-9565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Demoor updated HADOOP-9565: -- Attachment: HADOOP-9565-004.patch 004 introduces an objectStore-aware FileOutputCommitter: it

[jira] [Updated] (HADOOP-11262) Enable YARN to use S3A

[ https://issues.apache.org/jira/browse/HADOOP-11262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Demoor updated HADOOP-11262: --- Assignee: Pieter Reuse (was: Thomas Demoor) Enable YARN to use S3A

[jira] [Updated] (HADOOP-12169) ListStatus on empty dir in S3A lists itself instead of returning an empty list

[ https://issues.apache.org/jira/browse/HADOOP-12169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Demoor updated HADOOP-12169: --- Issue Type: Sub-task (was: Bug) Parent: HADOOP-11694 ListStatus on empty dir in

[jira] [Commented] (HADOOP-11918) Listing an empty s3a root directory throws FileNotFound.

[ https://issues.apache.org/jira/browse/HADOOP-11918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14591914#comment-14591914 ] Thomas Demoor commented on HADOOP-11918: We preferred the test approach of this

[jira] [Commented] (HADOOP-11262) Enable YARN to use S3A

[ https://issues.apache.org/jira/browse/HADOOP-11262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14578985#comment-14578985 ] Thomas Demoor commented on HADOOP-11262: We (well, [~PieterReuse] actually :p)

[jira] [Commented] (HADOOP-11918) Listing an empty s3a root directory throws FileNotFound.

[ https://issues.apache.org/jira/browse/HADOOP-11918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14547750#comment-14547750 ] Thomas Demoor commented on HADOOP-11918: Sorry, here you go:

[jira] [Commented] (HADOOP-9565) Add a Blobstore interface to add to blobstore FileSystems

[ https://issues.apache.org/jira/browse/HADOOP-9565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14537716#comment-14537716 ] Thomas Demoor commented on HADOOP-9565: --- FYI, currently testing my implementation.

[jira] [Commented] (HADOOP-11918) Listing an empty s3a root directory throws FileNotFound.

[ https://issues.apache.org/jira/browse/HADOOP-11918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14537712#comment-14537712 ] Thomas Demoor commented on HADOOP-11918: Not yet, focusing on HADOOP-9565 first.

  1   2   3   >