Re: [PR] HADOOP-19256 Integrate PutIfNotExist functionality into S3A [hadoop]

2024-09-14 Thread via GitHub
steveloughran commented on PR #7011: URL: https://github.com/apache/hadoop/pull/7011#issuecomment-2351041617 @shameersss1 exactly. which means that fs.create(path, false).close() is equivalent to other filesystems, though ``` st = fs.create(path, false) st.write("I think I have excl

Re: [PR] HADOOP-19256 Integrate PutIfNotExist functionality into S3A [hadoop]

2024-09-11 Thread via GitHub
shameersss1 commented on PR #7011: URL: https://github.com/apache/hadoop/pull/7011#issuecomment-2343342550 > FYI I'm working on a document covering how we can modify the output stream, rename code and more to deliver this more broadly. Will share soon +1, I too think the scope can be

Re: [PR] HADOOP-19256 Integrate PutIfNotExist functionality into S3A [hadoop]

2024-09-11 Thread via GitHub
steveloughran commented on PR #7011: URL: https://github.com/apache/hadoop/pull/7011#issuecomment-2343308507 FYI I'm working on a document covering how we can modify the output stream, rename code and more to deliver this more broadly. Will share soon -- This is an automated message from

Re: [PR] HADOOP-19256 Integrate PutIfNotExist functionality into S3A [hadoop]

2024-08-29 Thread via GitHub
hadoop-yetus commented on PR #7011: URL: https://github.com/apache/hadoop/pull/7011#issuecomment-2319468985 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reexe

Re: [PR] HADOOP-19256 Integrate PutIfNotExist functionality into S3A [hadoop]

2024-08-29 Thread via GitHub
hadoop-yetus commented on PR #7011: URL: https://github.com/apache/hadoop/pull/7011#issuecomment-2318825242 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reexe

Re: [PR] HADOOP-19256 Integrate PutIfNotExist functionality into S3A [hadoop]

2024-08-29 Thread via GitHub
diljotgrewal commented on PR #7011: URL: https://github.com/apache/hadoop/pull/7011#issuecomment-2318317399 Sorry, I didnt intend to push those changes here. I've removed those commits. I did start a separate [PR](https://github.com/apache/hadoop/pull/7015) for the sdk update yesterday und

Re: [PR] HADOOP-19256 Integrate PutIfNotExist functionality into S3A [hadoop]

2024-08-29 Thread via GitHub
steveloughran commented on PR #7011: URL: https://github.com/apache/hadoop/pull/7011#issuecomment-2317807423 @diljotgrewal upgrading an aws SDK MUST be done as a standalone action. we have a whole section of the testing doc [that covers this](https://hadoop.apache.org/docs/stable/hadoop-aws

Re: [PR] HADOOP-19256 Integrate PutIfNotExist functionality into S3A [hadoop]

2024-08-28 Thread via GitHub
steveloughran commented on PR #7011: URL: https://github.com/apache/hadoop/pull/7011#issuecomment-2316111280 +test this on google gcs through the connector and see what failure exception it raises; have it translate to the same ioe -- This is an automated message from the Apache Git Serv

Re: [PR] HADOOP-19256 Integrate PutIfNotExist functionality into S3A [hadoop]

2024-08-28 Thread via GitHub
steveloughran commented on code in PR #7011: URL: https://github.com/apache/hadoop/pull/7011#discussion_r1735181866 ## hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/impl/ITestS3APutIfMatch.java: ## @@ -0,0 +1,113 @@ +package org.apache.hadoop.fs.s3a.impl; + +imp

Re: [PR] HADOOP-19256 Integrate PutIfNotExist functionality into S3A [hadoop]

2024-08-27 Thread via GitHub
virajjasani commented on code in PR #7011: URL: https://github.com/apache/hadoop/pull/7011#discussion_r1733151248 ## hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/RequestFactoryImpl.java: ## @@ -517,12 +518,22 @@ public CreateMultipartUploadRequest.Builder

Re: [PR] HADOOP-19256 Integrate PutIfNotExist functionality into S3A [hadoop]

2024-08-27 Thread via GitHub
virajjasani commented on PR #7011: URL: https://github.com/apache/hadoop/pull/7011#issuecomment-2312950444 As per the discussion on the Jira, we also need the ability to override the configured values of these params, using s3afs implementation so that clients can use same s3afs instance an

Re: [PR] HADOOP-19256 Integrate PutIfNotExist functionality into S3A [hadoop]

2024-08-27 Thread via GitHub
ahmarsuhail commented on code in PR #7011: URL: https://github.com/apache/hadoop/pull/7011#discussion_r1732579400 ## hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3ABlockOutputStream.java: ## @@ -608,6 +611,16 @@ private long putObject() throws IOException {

Re: [PR] HADOOP-19256 Integrate PutIfNotExist functionality into S3A [hadoop]

2024-08-27 Thread via GitHub
ahmarsuhail commented on code in PR #7011: URL: https://github.com/apache/hadoop/pull/7011#discussion_r1732466736 ## hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/RequestFactoryImpl.java: ## @@ -517,12 +518,22 @@ public CreateMultipartUploadRequest.Builder

Re: [PR] HADOOP-19256 Integrate PutIfNotExist functionality into S3A [hadoop]

2024-08-24 Thread via GitHub
virajjasani commented on code in PR #7011: URL: https://github.com/apache/hadoop/pull/7011#discussion_r1730141410 ## hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/RequestFactoryImpl.java: ## @@ -517,12 +518,22 @@ public CreateMultipartUploadRequest.Builder

Re: [PR] HADOOP-19256 Integrate PutIfNotExist functionality into S3A [hadoop]

2024-08-24 Thread via GitHub
raphaelazzolini commented on code in PR #7011: URL: https://github.com/apache/hadoop/pull/7011#discussion_r1730127202 ## hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/RequestFactoryImpl.java: ## @@ -517,12 +518,22 @@ public CreateMultipartUploadRequest.Buil

Re: [PR] HADOOP-19256 Integrate PutIfNotExist functionality into S3A [hadoop]

2024-08-23 Thread via GitHub
hadoop-yetus commented on PR #7011: URL: https://github.com/apache/hadoop/pull/7011#issuecomment-2307980012 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reexe

Re: [PR] HADOOP-19256 Integrate PutIfNotExist functionality into S3A [hadoop]

2024-08-23 Thread via GitHub
virajjasani commented on code in PR #7011: URL: https://github.com/apache/hadoop/pull/7011#discussion_r1729596489 ## hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/RequestFactoryImpl.java: ## @@ -517,12 +518,22 @@ public CreateMultipartUploadRequest.Builder

[PR] HADOOP-19256 Integrate PutIfNotExist functionality into S3A [hadoop]

2024-08-23 Thread via GitHub
diljotgrewal opened a new pull request, #7011: URL: https://github.com/apache/hadoop/pull/7011 ### Description of PR This PR adds support for S3 Conditional Write (Put-if-absent) capability which is now generally available - https://aws.amazon.com/about-aws/whats-new/2024/08/amaz