[
https://issues.apache.org/jira/browse/HADOOP-18501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17628730#comment-17628730
]
ASF GitHub Bot commented on HADOOP-18501:
-----------------------------------------
pranavsaxena-microsoft opened a new pull request, #5109:
URL: https://github.com/apache/hadoop/pull/5109
<!--
Thanks for sending a pull request!
1. If this is your first time, please read our contributor guidelines:
https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute
2. Make sure your PR title starts with JIRA issue id, e.g.,
'HADOOP-17799. Your PR title ...'.
-->
### Description of PR
JIRA: https://issues.apache.org/jira/browse/HADOOP-18501
Error Description:
For partial read (due to account backend throttling), the ABFS driver retry
but doesn't add up in the throttling metrics.
In case of partial read with connection-reset exception, ABFS driver retry
for the full request and doesn't add up in throttling metrics.
Mitigation:
In case of partial read, ABFS Driver should retry for the remaining bytes,
and it should be added in throttling metrics.
### How was this patch tested?
Ran integeration and unit tests on the following accounts:
1. pranavsaxenahns: EAST_US: HNS account
2. pranavsaxenanonhns: EAST_US: Non-HNS account
Test results:
NonHNS-SharedKey
========================
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR]
TestAccountConfiguration.testConfigPropNotFound:386->testMissingConfigKey:399
Expected a
org.apache.hadoop.fs.azurebfs.contracts.exceptions.TokenAccessProviderException
to be thrown, but got the result: :
"org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider"
[INFO]
[ERROR] Tests run: 107, Failures: 1, Errors: 0, Skipped: 2
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR]
ITestAzureBlobFileSystemRandomRead.testValidateSeekBounds:269->Assert.assertTrue:42->Assert.fail:89
There should not be any network I/O (elapsedTimeMs=24).
[ERROR]
ITestPartialRead.testPartialReadWithConnectionReset:189->Assert.assertEquals:633->Assert.assertEquals:647->Assert.failNotEquals:835->Assert.fail:89
expected:<4> but was:<8>
[ERROR]
ITestPartialRead.testRecoverPartialRead:112->Assert.assertEquals:633->Assert.assertEquals:647->Assert.failNotEquals:835->Assert.fail:89
expected:<4> but was:<0>
[ERROR] Errors:
[ERROR]
ITestAzureBlobFileSystemLease.testAcquireRetry:344->lambda$testAcquireRetry$6:345
» TestTimedOut
[INFO]
[ERROR] Tests
run: 568, Failures: 3, Errors: 1, Skipped: 276
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR]
ITestAbfsTerasort.test_110_teragen:244->executeStage:211->Assert.assertEquals:647->Assert.failNotEquals:835->Assert.fail:89
teragen(1000,
abfs://testcontai...@pranavsaxenanonhns.dfs.core.windows.net/ITestAbfsTerasort/sortin)
failed expected:<0> but was:<1>
[ERROR] ITestReadBufferManager.testPurgeBufferManagerForParallelStreams:85
[After closing all streams free list contents should match with [0, 1, 2, 3, 4,
5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]]
Expected size:<16> but was:<9> in:
<[9, 12, 10, 13, 11, 8, 7, 15, 14]>
[ERROR] Errors:
[ERROR] ITestAbfsJobThroughManifestCommitter.test_0420_validateJob »
OutputValidation ...
[ERROR] ITestAbfsManifestCommitProtocol.testCommitLifecycle »
OutputValidation `abfs:/...
[ERROR]
ITestAbfsManifestCommitProtocol.testCommitterWithDuplicatedCommit »
OutputValidation
[ERROR] ITestAbfsManifestCommitProtocol.testConcurrentCommitTaskWithSubDir »
OutputValidation
[ERROR] ITestAbfsManifestCommitProtocol.testMapFileOutputCommitter »
OutputValidation ...
[ERROR] ITestAbfsManifestCommitProtocol.testOutputFormatIntegration »
OutputValidation
[ERROR] ITestAbfsManifestCommitProtocol.testParallelJobsToAdjacentPaths
» OutputValidation
[ERROR] ITestAbfsManifestCommitProtocol.testTwoTaskAttemptsCommit »
OutputValidation `...
[INFO]
[ERROR] Tests run: 333, Failures: 2, Errors: 8, Skipped: 46
### For code changes:
- [ ] Does the title or this PR starts with the corresponding JIRA issue id
(e.g. 'HADOOP-17799. Your PR title ...')?
- [ ] Object storage: have the integration tests been executed and the
endpoint declared according to the connector-specific documentation?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`,
`NOTICE-binary` files?
> [ABFS]: Partial Read should add to throttling metric
> ----------------------------------------------------
>
> Key: HADOOP-18501
> URL: https://issues.apache.org/jira/browse/HADOOP-18501
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/azure
> Affects Versions: 3.3.4
> Reporter: Pranav Saxena
> Assignee: Pranav Saxena
> Priority: Minor
>
> Error Description:
> For partial read (due to account backend throttling), the ABFS driver retry
> but doesn't add up in the throttling metrics.
> In case of partial read with connection-reset exception, ABFS driver retry
> for the full request and doesn't add up in throttling metrics.
> Mitigation:
> In case of partial read, Abfs Driver should retry for the remaining bytes and
> it should be added in throttling metrics.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
