[
https://issues.apache.org/jira/browse/HADOOP-14660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16088647#comment-16088647
]
Steve Loughran commented on HADOOP-14660:
-----------------------------------------
Brief look at this, I go offline for 2 weeks so expect silence from me. as far
as I'm concerned this is good to go in once the style checks are up and you
make the declaration of the specific endpoint you tested all the azure tests
against (sorry, I'm setting up the process here)
* try to get those line lengths under 80, including in comments. I know it
seems out of date, and I'd prefer a higher number, but current consensus is 80
except when it's really hard to do.
* my IDE complains about a few general things; have a look in yours (IntelliJ?)
to see what it says. Example (ClientThrottlingAnalyzer's {{new
AtomicReference()}} should be a {{new AtomicReference<>()}}.
* {{ClientThrottlingAnalyzer}} L150: move {{++consecutiveNoErrorCount *
analysisPeriodMs }} into is own var, to make ++ operation more obvious (ie. not
mix with the condition)
* comments above each new class should be /** javadocs */, so they get included
in the docs
* tests look good
If you want this in branch-2, as well as the trunk patch, include one with the
syntax HADOOP-14660-branch-2-123.patch; I'm not going to pick it in until a
java 7 JVM is happy with the build. Thanks
> wasb: improve throughput by 34% when account limit exceeded
> -----------------------------------------------------------
>
> Key: HADOOP-14660
> URL: https://issues.apache.org/jira/browse/HADOOP-14660
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/azure
> Reporter: Thomas
> Assignee: Thomas
> Attachments: HADOOP-14660-001.patch, HADOOP-14660-002.patch,
> HADOOP-14660-003.patch, HADOOP-14660-004.patch
>
>
> Big data workloads frequently exceed the Azure Storage max ingress and egress
> limits
> (https://docs.microsoft.com/en-us/azure/azure-subscription-service-limits).
> For example, the max ingress limit for a GRS account in the United States is
> currently 10 Gbps. When the limit is exceeded, the Azure Storage service
> fails a percentage of incoming requests, and this causes the client to
> initiate the retry policy. The retry policy delays requests by sleeping, but
> the sleep duration is independent of the client throughput and account limit.
> This results in low throughput, due to the high number of failed requests
> and thrashing causes by the retry policy.
> To fix this, we introduce a client-side throttle which minimizes failed
> requests and maximizes throughput. Tests have shown that this improves
> throughtput by ~34% when the storage account max ingress and/or egress limits
> are exceeded.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
