[GitHub] [hadoop] snvijaya commented on a change in pull request #1790: changes for combined append+flush calls for blockblob(traditionan) an…
snvijaya commented on a change in pull request #1790: changes for combined append+flush calls for blockblob(traditionan) an… URL: https://github.com/apache/hadoop/pull/1790#discussion_r365177009 ## File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java ## @@ -55,7 +55,7 @@ public static final Logger LOG = LoggerFactory.getLogger(AbfsClient.class); private final URL baseUrl; private final SharedKeyCredentials sharedKeyCredentials; - private final String xMsVersion = "2018-11-09"; + private final String xMsVersion = "2019-12-12"; Review comment: Server behaviour is controlled with this API version. So until server is ready, we can not bump up the api version support in driver. This is not recommended to be a config control either, to prevent clients from changing api versions to any older one. All dev work for new features is done with a feature config control that will turn off the feature. When the api version is incremented, the feature config will be turned on too. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #1790: changes for combined append+flush calls for blockblob(traditionan) an…
snvijaya commented on a change in pull request #1790: changes for combined
append+flush calls for blockblob(traditionan) an…
URL: https://github.com/apache/hadoop/pull/1790#discussion_r365196475
##
File path:
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsOutputStream.java
##
@@ -287,13 +297,13 @@ private synchronized void writeCurrentBufferToService()
throws IOException {
@Override
public Void call() throws Exception {
client.append(path, offset, bytes, 0,
Review comment:
In the trunk, client calls have AbfsPerfTracker tracking all the calls. Not
sure why the PR view is failing to show it as a delta from trunk. But looks
like there is some issue here with merge from trunk. Please do a file to file
compare from PR branch to trunk.
example: from trunk:
` public Void call() throws Exception {
AbfsPerfTracker tracker = client.getAbfsPerfTracker();
try (AbfsPerfInfo perfInfo = new AbfsPerfInfo(tracker,
"writeCurrentBufferToService", "append")) {
AbfsRestOperation op = client.append(path, offset, bytes, 0,
bytesLength);
perfInfo.registerResult(op.getResult());
byteBufferPool.putBuffer(ByteBuffer.wrap(bytes));
perfInfo.registerSuccess(true);
return null;
}`
(this is not the only delta)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #1790: changes for combined append+flush calls for blockblob(traditionan) an…
snvijaya commented on a change in pull request #1790: changes for combined
append+flush calls for blockblob(traditionan) an…
URL: https://github.com/apache/hadoop/pull/1790#discussion_r365179445
##
File path:
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AbfsConfiguration.java
##
@@ -427,6 +443,14 @@ public boolean isFlushEnabled() {
return this.enableFlush;
}
+ public boolean isAppendWithFlushEnabled() {
+return this.enableAppendWithFlush;
+ }
+
+ public boolean isOutputStreamFlushDisabled() {
Review comment:
Many changes from trunk merge is showing up as new updates even though not
added as part of this PR.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #1790: changes for combined append+flush calls for blockblob(traditionan) an…
snvijaya commented on a change in pull request #1790: changes for combined
append+flush calls for blockblob(traditionan) an…
URL: https://github.com/apache/hadoop/pull/1790#discussion_r365177842
##
File path:
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsOutputStream.java
##
@@ -257,17 +265,16 @@ public synchronized void close() throws IOException {
private synchronized void flushInternal(boolean isClose) throws IOException {
maybeThrowLastError();
-writeCurrentBufferToService();
-flushWrittenBytesToService(isClose);
+writeAndFlushWrittenBytesToService(isClose);
}
private synchronized void flushInternalAsync() throws IOException {
maybeThrowLastError();
-writeCurrentBufferToService();
+writeCurrentBufferToService(true, false);
flushWrittenBytesToServiceAsync();
}
- private synchronized void writeCurrentBufferToService() throws IOException {
+ private synchronized void writeCurrentBufferToService(final boolean flush,
final boolean isClose) throws IOException {
Review comment:
Param passing was missing earlier iteration. Can see in new update.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #1790: changes for combined append+flush calls for blockblob(traditionan) an…
snvijaya commented on a change in pull request #1790: changes for combined append+flush calls for blockblob(traditionan) an… URL: https://github.com/apache/hadoop/pull/1790#discussion_r365177009 ## File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java ## @@ -55,7 +55,7 @@ public static final Logger LOG = LoggerFactory.getLogger(AbfsClient.class); private final URL baseUrl; private final SharedKeyCredentials sharedKeyCredentials; - private final String xMsVersion = "2018-11-09"; + private final String xMsVersion = "2019-12-12"; Review comment: Server behaviour is controlled with this API version. So until server is ready, we can not bump up the api version support in driver. This is not recommended to be a config control either, to prevent clients from changing api versions to any older one. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #1790: changes for combined append+flush calls for blockblob(traditionan) an…
snvijaya commented on a change in pull request #1790: changes for combined
append+flush calls for blockblob(traditionan) an…
URL: https://github.com/apache/hadoop/pull/1790#discussion_r365177280
##
File path:
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java
##
@@ -278,7 +282,8 @@ public AbfsRestOperation renamePath(final String source,
final String destinatio
}
public AbfsRestOperation append(final String path, final long position,
final byte[] buffer, final int offset,
- final int length) throws
AzureBlobFileSystemException {
+ final int length, boolean flush, boolean
isClose)
Review comment:
New iteration has the caller updated.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #1790: changes for combined append+flush calls for blockblob(traditionan) an…
snvijaya commented on a change in pull request #1790: changes for combined
append+flush calls for blockblob(traditionan) an…
URL: https://github.com/apache/hadoop/pull/1790#discussion_r365174656
##
File path:
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java
##
@@ -344,40 +366,53 @@ public void deleteFilesystem() throws
AzureBlobFileSystemException {
public OutputStream createFile(final Path path, final boolean overwrite,
final FsPermission permission,
final FsPermission umask) throws
AzureBlobFileSystemException {
-boolean isNamespaceEnabled = getIsNamespaceEnabled();
-LOG.debug("createFile filesystem: {} path: {} overwrite: {} permission: {}
umask: {} isNamespaceEnabled: {}",
-client.getFileSystem(),
-path,
-overwrite,
-permission.toString(),
-umask.toString(),
-isNamespaceEnabled);
-
-client.createPath(AbfsHttpConstants.FORWARD_SLASH + getRelativePath(path),
true, overwrite,
-isNamespaceEnabled ? getOctalNotation(permission) : null,
-isNamespaceEnabled ? getOctalNotation(umask) : null);
+try (AbfsPerfInfo perfInfo = startTracking("createFile", "createPath")) {
+ boolean isNamespaceEnabled = getIsNamespaceEnabled();
+ LOG.debug("createFile filesystem: {} path: {} overwrite: {} permission:
{} umask: {} isNamespaceEnabled: {}",
+ client.getFileSystem(),
+ path,
+ overwrite,
+ permission.toString(),
+ umask.toString(),
+ isNamespaceEnabled);
+
+boolean appendBlob = false;
+if (isAppendBlobKey(path.toString())) {
+ appendBlob = true;
+}
-return new AbfsOutputStream(
-client,
-AbfsHttpConstants.FORWARD_SLASH + getRelativePath(path),
-0,
-abfsConfiguration.getWriteBufferSize(),
-abfsConfiguration.isFlushEnabled());
+ client.createPath(AbfsHttpConstants.FORWARD_SLASH +
getRelativePath(path), true, overwrite,
Review comment:
Is support for AppendBlob over queryparam "blobtype" also enabled with new
Dec-2019 version ? If yes, add a config control for AppendBlob as well and have
the config off, which can be turned on when the API version is upgraded in
driver.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #1790: changes for combined append+flush calls for blockblob(traditionan) an…
snvijaya commented on a change in pull request #1790: changes for combined append+flush calls for blockblob(traditionan) an… URL: https://github.com/apache/hadoop/pull/1790#discussion_r365171372 ## File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AbfsConfiguration.java ## @@ -146,6 +150,10 @@ DefaultValue = DEFAULT_ENABLE_FLUSH) private boolean enableFlush; + @BooleanConfigurationValidatorAnnotation(ConfigurationKey = FS_AZURE_ENABLE_APPEND_WITH_FLUSH, + DefaultValue = DEFAULT_ENABLE_APPEND_WITH_FLUSH) + private boolean enableAppendWithFlush; Review comment: Last iteration added has bought in the missing code piece. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #1790: changes for combined append+flush calls for blockblob(traditionan) an…
snvijaya commented on a change in pull request #1790: changes for combined append+flush calls for blockblob(traditionan) an… URL: https://github.com/apache/hadoop/pull/1790#discussion_r365171120 ## File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AbfsConfiguration.java ## @@ -130,6 +130,10 @@ DefaultValue = DEFAULT_FS_AZURE_ATOMIC_RENAME_DIRECTORIES) private String azureAtomicDirs; + @StringConfigurationValidatorAnnotation(ConfigurationKey = FS_AZURE_APPEND_BLOB_KEY, + DefaultValue = DEFAULT_FS_AZURE_APPEND_BLOB_DIRECTORIES) + private String azureAppendBlobDirs; Review comment: Dev introducing new config needs to add the documentation for it. You need to edit the abfs.md file mentioned in above comment. Follow format already present in it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #1790: changes for combined append+flush calls for blockblob(traditionan) an…
snvijaya commented on a change in pull request #1790: changes for combined append+flush calls for blockblob(traditionan) an… URL: https://github.com/apache/hadoop/pull/1790#discussion_r365170305 ## File path: hadoop-tools/hadoop-azure/pom.xml ## @@ -194,7 +203,8 @@ org.wildfly.openssl wildfly-openssl - runtime + 1.0.9.Final Review comment: Wildfly jar version is picked from the parent project. Retain the original config. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #1790: changes for combined append+flush calls for blockblob(traditionan) an…
snvijaya commented on a change in pull request #1790: changes for combined append+flush calls for blockblob(traditionan) an… URL: https://github.com/apache/hadoop/pull/1790#discussion_r365170489 ## File path: hadoop-tools/hadoop-azure/pom.xml ## @@ -291,6 +301,7 @@ +
[GitHub] [hadoop] snvijaya commented on a change in pull request #1790: changes for combined append+flush calls for blockblob(traditionan) an…
snvijaya commented on a change in pull request #1790: changes for combined
append+flush calls for blockblob(traditionan) an…
URL: https://github.com/apache/hadoop/pull/1790#discussion_r363154151
##
File path:
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java
##
@@ -278,7 +282,8 @@ public AbfsRestOperation renamePath(final String source,
final String destinatio
}
public AbfsRestOperation append(final String path, final long position,
final byte[] buffer, final int offset,
- final int length) throws
AzureBlobFileSystemException {
+ final int length, boolean flush, boolean
isClose)
Review comment:
This method is supposed to be called from AbfsOutputStream.java ->
writeCurrentBufferToService(), but there is no change for call to this API
within that method. How did the compilation pass.
And, (probably a repeat question of config query posted above, still) can
you please explain why flush and close needs to be added to Append method.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #1790: changes for combined append+flush calls for blockblob(traditionan) an…
snvijaya commented on a change in pull request #1790: changes for combined
append+flush calls for blockblob(traditionan) an…
URL: https://github.com/apache/hadoop/pull/1790#discussion_r363154151
##
File path:
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java
##
@@ -278,7 +282,8 @@ public AbfsRestOperation renamePath(final String source,
final String destinatio
}
public AbfsRestOperation append(final String path, final long position,
final byte[] buffer, final int offset,
- final int length) throws
AzureBlobFileSystemException {
+ final int length, boolean flush, boolean
isClose)
Review comment:
This method is supposed to be called from AbfsOutputStream.java ->
writeCurrentBufferToService(), but there is no change for call to this API
within that method. How did the compilation pass.
And, (probably a repeat question of config query posted above, still) can
you please why add flush and close needs to be added to Append method.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #1790: changes for combined append+flush calls for blockblob(traditionan) an…
snvijaya commented on a change in pull request #1790: changes for combined
append+flush calls for blockblob(traditionan) an…
URL: https://github.com/apache/hadoop/pull/1790#discussion_r363153206
##
File path:
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsOutputStream.java
##
@@ -309,15 +316,23 @@ private synchronized void
flushWrittenBytesToService(boolean isClose) throws IOE
throw new FileNotFoundException(ex.getMessage());
}
}
-
if (ex.getCause() instanceof AzureBlobFileSystemException) {
ex = (AzureBlobFileSystemException) ex.getCause();
}
lastError = new IOException(ex);
throw lastError;
}
}
-flushWrittenBytesToServiceInternal(position, false, isClose);
+shrinkWriteOperationQueue();
Review comment:
AbfsOutputStream.close() call flow is:
close() -> flushInternal() -> flushWrittenBytesToService() **->
flushWrittenBytesToServiceInternal()**
Within flushWrittenBytesToServiceInternal() is where service "Flush" gets
called.
Above change removes call to flushWrittenBytesToServiceInternal() and
replaces it with shrinkWriteOperationQueue(). This is going to prevent flush
for AbfsOutputStream.close() calls.
This might be ok for AppendBlob, but BlockBlob behaviour needs to be
retained where AbfsClient->Flush() will get triggered.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #1790: changes for combined append+flush calls for blockblob(traditionan) an…
snvijaya commented on a change in pull request #1790: changes for combined
append+flush calls for blockblob(traditionan) an…
URL: https://github.com/apache/hadoop/pull/1790#discussion_r363152120
##
File path:
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsOutputStream.java
##
@@ -257,17 +265,16 @@ public synchronized void close() throws IOException {
private synchronized void flushInternal(boolean isClose) throws IOException {
maybeThrowLastError();
-writeCurrentBufferToService();
-flushWrittenBytesToService(isClose);
+writeAndFlushWrittenBytesToService(isClose);
}
private synchronized void flushInternalAsync() throws IOException {
maybeThrowLastError();
-writeCurrentBufferToService();
+writeCurrentBufferToService(true, false);
flushWrittenBytesToServiceAsync();
}
- private synchronized void writeCurrentBufferToService() throws IOException {
+ private synchronized void writeCurrentBufferToService(final boolean flush,
final boolean isClose) throws IOException {
Review comment:
As mentioned in the comments for AbfsConfiguration.java, dont see flush in
use inside the method. Also applies to the new isClose argument.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #1790: changes for combined append+flush calls for blockblob(traditionan) an…
snvijaya commented on a change in pull request #1790: changes for combined append+flush calls for blockblob(traditionan) an… URL: https://github.com/apache/hadoop/pull/1790#discussion_r363151946 ## File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java ## @@ -55,7 +55,7 @@ public static final Logger LOG = LoggerFactory.getLogger(AbfsClient.class); private final URL baseUrl; private final SharedKeyCredentials sharedKeyCredentials; - private final String xMsVersion = "2018-11-09"; + private final String xMsVersion = "2019-12-12"; Review comment: Is this version present on all the production clusters already ? If not, this change should go in only after the support is available. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #1790: changes for combined append+flush calls for blockblob(traditionan) an…
snvijaya commented on a change in pull request #1790: changes for combined
append+flush calls for blockblob(traditionan) an…
URL: https://github.com/apache/hadoop/pull/1790#discussion_r363151735
##
File path:
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AbfsConfiguration.java
##
@@ -411,6 +419,10 @@ public String getAzureAtomicRenameDirs() {
return this.azureAtomicDirs;
}
+ public String getAppendBlobDirs() {
Review comment:
Why not return a HashSet as all callers are going to need it so.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #1790: changes for combined append+flush calls for blockblob(traditionan) an…
snvijaya commented on a change in pull request #1790: changes for combined append+flush calls for blockblob(traditionan) an… URL: https://github.com/apache/hadoop/pull/1790#discussion_r363151653 ## File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AbfsConfiguration.java ## @@ -130,6 +130,10 @@ DefaultValue = DEFAULT_FS_AZURE_ATOMIC_RENAME_DIRECTORIES) private String azureAtomicDirs; + @StringConfigurationValidatorAnnotation(ConfigurationKey = FS_AZURE_APPEND_BLOB_KEY, + DefaultValue = DEFAULT_FS_AZURE_APPEND_BLOB_DIRECTORIES) + private String azureAppendBlobDirs; Review comment: New configs need documentation. For reference, recent documentation done for flush related config : https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-azure/src/site/markdown/abfs.md#-flush-options This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] snvijaya commented on a change in pull request #1790: changes for combined append+flush calls for blockblob(traditionan) an…
snvijaya commented on a change in pull request #1790: changes for combined append+flush calls for blockblob(traditionan) an… URL: https://github.com/apache/hadoop/pull/1790#discussion_r363151516 ## File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AbfsConfiguration.java ## @@ -146,6 +150,10 @@ DefaultValue = DEFAULT_ENABLE_FLUSH) private boolean enableFlush; + @BooleanConfigurationValidatorAnnotation(ConfigurationKey = FS_AZURE_ENABLE_APPEND_WITH_FLUSH, + DefaultValue = DEFAULT_ENABLE_APPEND_WITH_FLUSH) + private boolean enableAppendWithFlush; Review comment: This config value is passed until AbfsOutputStream.java -> writeCurrentBufferToService() as the first argument. but no specific action is seen inside writeCurrentBufferToService() if the value is true or false. What is intended of this config ? is it required ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
