[
https://issues.apache.org/jira/browse/HADOOP-15991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16720106#comment-16720106
]
Steve Loughran commented on HADOOP-15991:
-----------------------------------------
"Interesting", as they say. Stack trace implies it was in DynamoDB during test
teardown itself.
# what happens if you don't run with s3guard on?
# what are the s3guard settings for that bucket (e.g IO allocation).
# How far is that AWS region from you?
# and what is your bandwidth, especially uploading
3.2+ has a lot of support for dealing with DDB throttling -from the printed
metrics I don't see any throttling events
you can set the test timeout for scale tests by setting the property
fs.s3a.scale.test.timeout to a value in seconds. Do that on the maven build and
it will also tune the maven timeout too, so the test runner itself doesn't
break.
Looking at the code in the MPU test it should be timing out after 30 minutes,
not 3 which is something to worry about. Maybe we're passing it up wrong (it's
retrofitted the scale params to a contract test, see)
You can try setting a smaller partition size at
{{-Dfs.s3a.scale.test.huge.partitionsize=5M}} ; this uploads half as much data.
Looking at where timeouts can come in from the test runner, if you run the test
as a unit test (-Dtest=ITest....) you'd see more problems, but I'm not sure
that any of the scale tests will kick in then.
Returning to the problem:
# switch to a smaller partition size (5M is the minimum)
# see if a bigger scale test timeout on the build command line helps
# see if you want to spend time debugging how timeouts get picked up; maybe
some log statement
> testMultipartUpload timing out
> ------------------------------
>
> Key: HADOOP-15991
> URL: https://issues.apache.org/jira/browse/HADOOP-15991
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3, test
> Affects Versions: 3.2.0
> Reporter: lqjacklee
> Priority: Minor
>
> timeout of S3 mpu tests
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
