[jira] [Commented] (HADOOP-15659) ABFS: Code changes for bug fix and new tests
[ https://issues.apache.org/jira/browse/HADOOP-15659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16676370#comment-16676370 ] Sunil Govindan commented on HADOOP-15659: - Updated fixed version to 3.2.0 as HADOOP-15407 branch is merged to trunk > ABFS: Code changes for bug fix and new tests > > > Key: HADOOP-15659 > URL: https://issues.apache.org/jira/browse/HADOOP-15659 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Da Zhou >Assignee: Da Zhou >Priority: Major > Fix For: 3.2.0 > > Attachments: HADOOP-15659-HADOOP-15407-001.patch, > HADOOP-15659-HADOOP-15407-002.patch, HADOOP-15659-HADOOP-15407-003.patch > > > - add bug fixes. > - remove unnecessary dependencies. > - add new tests for code changes. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15659) ABFS: Code changes for bug fix and new tests
[ https://issues.apache.org/jira/browse/HADOOP-15659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16624931#comment-16624931 ] Hudson commented on HADOOP-15659: - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #15037 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/15037/]) HADOOP-15659. Code changes for bug fix and new tests. Contributed by Da (tmarq: rev b54b0c1b676c616aef9574e4e88ea30c314c79dc) * (edit) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java * (edit) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/contracts/services/ListResultSchema.java * (edit) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsInputStream.java * (add) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/KeyProvider.java * (add) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemFinalize.java * (edit) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemFlush.java * (edit) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsOutputStream.java * (edit) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestFileSystemProperties.java * (add) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/contracts/exceptions/KeyProviderException.java * (edit) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsHttpOperation.java * (add) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAbfsClient.java * (edit) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/AbstractAbfsIntegrationTest.java * (edit) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemInitAndCreate.java * (edit) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/constants/FileSystemConfigurations.java * (edit) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java * (edit) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/constants/ConfigurationKeys.java * (edit) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemRename.java * (edit) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/contract/AbfsFileSystemContract.java * (delete) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/services/TestAbfsConfigurationFieldsValidation.java * (add) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/services/TestAbfsClient.java * (edit) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsUriQueryBuilder.java * (add) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/ShellDecryptionKeyProvider.java * (edit) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemE2EScale.java * (add) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAbfsReadWriteAndSeek.java * (delete) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/services/ITestAbfsReadWriteAndSeek.java * (edit) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystem.java * (edit) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsRestOperation.java * (add) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/TestAbfsConfigurationFieldsValidation.java * (delete) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsConfiguration.java * (edit) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/contracts/services/AzureServiceErrorCode.java * (edit) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/contracts/services/ListResultEntrySchema.java * (add) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/SimpleKeyProvider.java * (add) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/services/TestShellDecryptionKeyProvider.java * (add) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AbfsConfiguration.java * (edit) hadoop-tools/hadoop-azure/pom.xml * (edit) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemE2E.java > ABFS: Code changes for bug fix and new tests > > > Key: HADOOP-15659 > URL: https://issues.apache.org/jira/browse/HADOOP-15659 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Da Zhou >Assignee: Da Zhou >Priority: Major > Attachments: HAD
[jira] [Commented] (HADOOP-15659) ABFS: Code changes for bug fix and new tests
[ https://issues.apache.org/jira/browse/HADOOP-15659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16578944#comment-16578944 ] Thomas Marquardt commented on HADOOP-15659: --- One of the analytic platforms has a scenario where they disable the filesystem cache and create many filesystems in a single process. It is also a multi-tenant scenario, where they have multiple users running in the same process. They rely on user's code to close the filesystem, and sometimes users write bad code, and if we don't override finalize then the filesystem doesn't get released and they hit an OOM condition. I agree with you that it is weird, and unusual, but without this change their scenario does not work. > ABFS: Code changes for bug fix and new tests > > > Key: HADOOP-15659 > URL: https://issues.apache.org/jira/browse/HADOOP-15659 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Da Zhou >Assignee: Da Zhou >Priority: Major > Attachments: HADOOP-15659-HADOOP-15407-001.patch, > HADOOP-15659-HADOOP-15407-002.patch, HADOOP-15659-HADOOP-15407-003.patch > > > - add bug fixes. > - remove unnecessary dependencies. > - add new tests for code changes. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15659) ABFS: Code changes for bug fix and new tests
[
https://issues.apache.org/jira/browse/HADOOP-15659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16578905#comment-16578905
]
Steve Loughran commented on HADOOP-15659:
-
out of curiosity, why the {{finalize()}} for AzureBlobFileSystem? Generally
unused as we expect close() to be called, and if not: tough
> ABFS: Code changes for bug fix and new tests
>
>
> Key: HADOOP-15659
> URL: https://issues.apache.org/jira/browse/HADOOP-15659
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Reporter: Da Zhou
>Assignee: Da Zhou
>Priority: Major
> Attachments: HADOOP-15659-HADOOP-15407-001.patch,
> HADOOP-15659-HADOOP-15407-002.patch, HADOOP-15659-HADOOP-15407-003.patch
>
>
> - add bug fixes.
> - remove unnecessary dependencies.
> - add new tests for code changes.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15659) ABFS: Code changes for bug fix and new tests
[ https://issues.apache.org/jira/browse/HADOOP-15659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16576985#comment-16576985 ] Thomas Marquardt commented on HADOOP-15659: --- Committed to HADOOP-15407 branch > ABFS: Code changes for bug fix and new tests > > > Key: HADOOP-15659 > URL: https://issues.apache.org/jira/browse/HADOOP-15659 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Da Zhou >Assignee: Da Zhou >Priority: Major > Attachments: HADOOP-15659-HADOOP-15407-001.patch, > HADOOP-15659-HADOOP-15407-002.patch, HADOOP-15659-HADOOP-15407-003.patch > > > - add bug fixes. > - remove unnecessary dependencies. > - add new tests for code changes. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15659) ABFS: Code changes for bug fix and new tests
[
https://issues.apache.org/jira/browse/HADOOP-15659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16576974#comment-16576974
]
genericqa commented on HADOOP-15659:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} docker {color} | {color:red} 12m
23s{color} | {color:red} Docker failed to build yetus/hadoop:abb62dd. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HADOOP-15659 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12935215/HADOOP-15659-HADOOP-15407-003.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/15017/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> ABFS: Code changes for bug fix and new tests
>
>
> Key: HADOOP-15659
> URL: https://issues.apache.org/jira/browse/HADOOP-15659
> Project: Hadoop Common
> Issue Type: Sub-task
>Reporter: Da Zhou
>Assignee: Da Zhou
>Priority: Major
> Attachments: HADOOP-15659-HADOOP-15407-001.patch,
> HADOOP-15659-HADOOP-15407-002.patch, HADOOP-15659-HADOOP-15407-003.patch
>
>
> - add bug fixes.
> - remove unnecessary dependencies.
> - add new tests for code changes.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15659) ABFS: Code changes for bug fix and new tests
[ https://issues.apache.org/jira/browse/HADOOP-15659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16576965#comment-16576965 ] Thomas Marquardt commented on HADOOP-15659: --- Looks good to me, +1. I had to make a simple fix to ITestAbfsClient.testContinuationTokenHavingEqualSign and that is attached in the 003 patch. [^HADOOP-15659-HADOOP-15407-003.patch] All tests pass with this update: Tests run: 261, Failures: 0, Errors: 0, Skipped: 11 Tests run: 1, Failures: 0, Errors: 0, Skipped: 0 Tests run: 777, Failures: 0, Errors: 0, Skipped: 182 Tests run: 186, Failures: 0, Errors: 0, Skipped: 10 Since Yetus is not working, I'm running test-patch locally and if that goes well I will commit the patch. > ABFS: Code changes for bug fix and new tests > > > Key: HADOOP-15659 > URL: https://issues.apache.org/jira/browse/HADOOP-15659 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Da Zhou >Assignee: Da Zhou >Priority: Major > Attachments: HADOOP-15659-HADOOP-15407-001.patch, > HADOOP-15659-HADOOP-15407-002.patch, HADOOP-15659-HADOOP-15407-003.patch > > > - add bug fixes. > - remove unnecessary dependencies. > - add new tests for code changes. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15659) ABFS: Code changes for bug fix and new tests
[
https://issues.apache.org/jira/browse/HADOOP-15659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16576777#comment-16576777
]
genericqa commented on HADOOP-15659:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} docker {color} | {color:red} 0m
9s{color} | {color:red} Docker failed to build yetus/hadoop:abb62dd. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HADOOP-15659 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12935182/HADOOP-15659-HADOOP-15407-002.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/15014/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> ABFS: Code changes for bug fix and new tests
>
>
> Key: HADOOP-15659
> URL: https://issues.apache.org/jira/browse/HADOOP-15659
> Project: Hadoop Common
> Issue Type: Sub-task
>Reporter: Da Zhou
>Assignee: Da Zhou
>Priority: Major
> Attachments: HADOOP-15659-HADOOP-15407-001.patch,
> HADOOP-15659-HADOOP-15407-002.patch
>
>
> - add bug fixes.
> - remove unnecessary dependencies.
> - add new tests for code changes.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15659) ABFS: Code changes for bug fix and new tests
[ https://issues.apache.org/jira/browse/HADOOP-15659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16576771#comment-16576771 ] Da Zhou commented on HADOOP-15659: -- Attaching HADOOP-15659-HADOOP-15407-002.patch. Thanks [~tmarquardt] for your review. AbfsClient L48, L63: Fixed AbfsRestOperation: L51: removed static. Calls to getLogger will retrieve the cached AbfsClient logger. AbfsFileSystem: L473: Updated. Since HEAD request response won't have message body, so we can not check its exception based on storage error code, so I did a work around based on HTTP status code. Below are the description of the code changes: - Added a configuration key to enable/disable retrieving usergroup info when initializing AzureBlobFileSystem - Added a configuration key to enable/disable flush operation - Added finalize() for AzureBlobFileSystem - Updated default write buffer size and read ahead queue size - Replaced denpendency on com.fasterxml.jackson.core with org.codehaus.jackson - Removed dependency on joda-time - Added ShellDecruptionKeyProvider - Fixed a bug of HTTP status code handling in AbfsRestOperation - Fixed a bug of percent encoding query parameter - Fixed a bug of async flush - Fixed checkstyle issues > ABFS: Code changes for bug fix and new tests > > > Key: HADOOP-15659 > URL: https://issues.apache.org/jira/browse/HADOOP-15659 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Da Zhou >Assignee: Da Zhou >Priority: Major > Attachments: HADOOP-15659-HADOOP-15407-001.patch > > > - add bug fixes. > - remove unnecessary dependencies. > - add new tests for code changes. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15659) ABFS: Code changes for bug fix and new tests
[ https://issues.apache.org/jira/browse/HADOOP-15659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16575792#comment-16575792 ] Thomas Marquardt commented on HADOOP-15659: --- Overall looks good to me. I would like a list with a short description of each change, and then please address feedback below: Use git apply --whitespace=fix to remove whitespace from your patch. AbfsClient.java L48 should have xMsVersion = "2018-06-17" L63 Remove "this" AbfsRestOperation.java L51 Why static? Originally, this was using the public static logger field from AbfsClient, but now calls getLogger which I assume returns a cached logger? I definitely think we should use a single AbfsClient logger. AbfsFileSystem.java L473 Can you update this to call abfsStore.getFilesystemProperties() directly? My test results match yours: Tests run: 261, Failures: 0, Errors: 0, Skipped: 11 Tests run: 1, Failures: 0, Errors: 0, Skipped: 0 Tests run: 777, Failures: 0, Errors: 0, Skipped: 182 Tests run: 186, Failures: 0, Errors: 0, Skipped: 10 > ABFS: Code changes for bug fix and new tests > > > Key: HADOOP-15659 > URL: https://issues.apache.org/jira/browse/HADOOP-15659 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Da Zhou >Assignee: Da Zhou >Priority: Major > Attachments: HADOOP-15659-HADOOP-15407-001.patch > > > - add bug fixes. > - remove unnecessary dependencies. > - add new tests for code changes. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15659) ABFS: Code changes for bug fix and new tests
[ https://issues.apache.org/jira/browse/HADOOP-15659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16575742#comment-16575742 ] Thomas Marquardt commented on HADOOP-15659: --- Can you add a short description of each bug fix? > ABFS: Code changes for bug fix and new tests > > > Key: HADOOP-15659 > URL: https://issues.apache.org/jira/browse/HADOOP-15659 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Da Zhou >Assignee: Da Zhou >Priority: Major > Attachments: HADOOP-15659-HADOOP-15407-001.patch > > > - add bug fixes. > - remove unnecessary dependencies. > - add new tests for code changes. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15659) ABFS: Code changes for bug fix and new tests
[
https://issues.apache.org/jira/browse/HADOOP-15659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16575550#comment-16575550
]
genericqa commented on HADOOP-15659:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} docker {color} | {color:red} 0m
9s{color} | {color:red} Docker failed to build yetus/hadoop:abb62dd. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HADOOP-15659 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12935057/HADOOP-15659-HADOOP-15407-001.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/15010/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> ABFS: Code changes for bug fix and new tests
>
>
> Key: HADOOP-15659
> URL: https://issues.apache.org/jira/browse/HADOOP-15659
> Project: Hadoop Common
> Issue Type: Sub-task
>Reporter: Da Zhou
>Assignee: Da Zhou
>Priority: Major
> Attachments: HADOOP-15659-HADOOP-15407-001.patch
>
>
> - add bug fixes.
> - remove unnecessary dependencies.
> - add new tests for code changes.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
