[ https://issues.apache.org/jira/browse/HADOOP-14660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16087208#comment-16087208 ]
Steve Loughran commented on HADOOP-14660: ----------------------------------------- Interesting: numbers look good, and its the kind of problem which doesn't surface in testing unless you are playing with fault injection. What's going to happen over long haul links, especially if there's a bit of unreliability in them? I'm thinking of (a) overeacting to transient failures, and conflict with the usual TCP congestion control I'm fixing up S3A to handle AWS throttling, which has to be in our client (503 -> throttled exception -> custom retry policy with exponential backoff (maybe I should add jitter too)). I plan to actually log this as part of the instrumentation, so when you collect the results of a job, you can see that a lot of throttling took place. Is there a way to pick this up here? # updating SDK must be a separate, standalone patch, linked as depended on by HADOOP-9991; mark this throttling patch as depending on the SDK update. That helps identifying high-risk version updates, simplifies rollback, etc. # new key should, if settable by users, listed in a public constant. I'd also prefer "enabled" as the element name, rather than "enable", though that's very subjective # log @ debug when throttling is on & what the options are. # {{BlobOperationDescriptor.getContentLengthIfKnown}}: add tests for parsing invalid input strings; they should be be rejected rather than trigger failures (which is probably what happens in this patch) # Use SLF4J for all logging; we've moved off commons logging. This lets you do logging as {{LOG.info("timer is {}", timer)}} & have on-demand string construction only when the logging is needed # if an outer class is tagged as {{InterfaceAudience.Private}}, no need to mark the inner classes/interfaces. * are you confident that {{TimerTaskImpl}} is always stopped? And can you make sure the thread has a meaningful name, like "wasb-timer-container-mycontainer". h3. {{ClientThrottlingAnalyzer}} * L81. you can use {{Preconditions.checkArgument}} here, and {{StringUtils.isNotEmpty()}} as the check for non-empty strings. * L183: to avoid needless string construction, guard the log@ debug with a {{LOG.isDebugEnabled()}} h3. {{TestBlobOperationDescriptor}} * extend {{AbstractWasbTestBase}} * output streams &c must be closed even when asserts are raised; try-with-resources can do this. * {{AbstractWasbTestBase}} & others. Just say {{throws Exception}}; more flexible in future as tests change. h3. {{TestClientThrottlingAnalyzer}} * In HADOOP-14553 I'm going to switch all possible tests to parallel runs, and don't want to add new sequential runs if possible. How will a parallel run impact timing estimates, especially if run over a long-haul link. Some of those assertions look britlle. * lot of repetition in the tests: Can you refactor this out? > wasb: improve throughput by 34% when account limit exceeded > ----------------------------------------------------------- > > Key: HADOOP-14660 > URL: https://issues.apache.org/jira/browse/HADOOP-14660 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/azure > Reporter: Thomas > Assignee: Thomas > Attachments: HADOOP-14660-001.patch, HADOOP-14660-002.patch > > > Big data workloads frequently exceed the Azure Storage max ingress and egress > limits > (https://docs.microsoft.com/en-us/azure/azure-subscription-service-limits). > For example, the max ingress limit for a GRS account in the United States is > currently 10 Gbps. When the limit is exceeded, the Azure Storage service > fails a percentage of incoming requests, and this causes the client to > initiate the retry policy. The retry policy delays requests by sleeping, but > the sleep duration is independent of the client throughput and account limit. > This results in low throughput, due to the high number of failed requests > and thrashing causes by the retry policy. > To fix this, we introduce a client-side throttle which minimizes failed > requests and maximizes throughput. Tests have shown that this improves > throughtput by ~34% when the storage account max ingress and/or egress limits > are exceeded. -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org