[
https://issues.apache.org/jira/browse/HDFS-13934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17167094#comment-17167094
]
Ayush Saxena commented on HDFS-13934:
-------------------------------------
Hi [~ste...@apache.org], [~fabbri]
This tends to break {{TestHDFSContractMultipartUploader.testConcurrentUploads}}
Should be a test only issue, I guess :
{code:java}
eventually(timeToBecomeConsistentMillis(),
() -> verifyFileLength(file, size2),
new LambdaTestUtils.ProportionalRetryInterval(
CONSISTENCY_INTERVAL, timeToBecomeConsistentMillis() == 0 ?
CONSISTENCY_INTERVAL :
timeToBecomeConsistentMillis())); // This is 0 for HDFS
{code}
The reason being {{timeToBecomeConsistentMillis()}} is 0, and there is
{{PreCondition}} check in {{LambdaTestUtils.ProportionalRetryInterval}} making
sure it shouldn't be 0,
Earlier that {{LambdaTestUtils.FixedRetryInterval}} was being used, which
didn't had this issue
We can change back to it?, or handle this specifically for HDFS?
This test is even extended by {{ITestS3AContractMultipartUploader}} which I can
not verify, and any change here would impact that as well, So, would be good if
either of you, help fix this.
Ref :
[https://builds.apache.org/job/PreCommit-HDFS-Build/29566/testReport/org.apache.hadoop.fs.contract.hdfs/TestHDFSContractMultipartUploader/testConcurrentUploads/]
> Multipart uploaders to be created through API call to FileSystem/FileContext,
> not service loader
> ------------------------------------------------------------------------------------------------
>
> Key: HDFS-13934
> URL: https://issues.apache.org/jira/browse/HDFS-13934
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: fs, fs/s3, hdfs
> Affects Versions: 3.2.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Major
> Fix For: 3.3.1
>
>
> the Multipart Uploaders are created via service loaders. This is troublesome
> # HADOOP-12636, HADOOP-13323, HADOOP-13625 highlight how the load process
> forces the transient loading of dependencies. If a dependent class cannot be
> loaded (e.g aws-sdk is not on the classpath), that service won't load.
> Without error handling round the load process, this stops any uploader from
> loading. Even with that error handling, the performance hit of that load,
> especially with reshaded dependencies, hurts performance (HADOOP-13138).
> # it makes wrapping the the load with any filter impossible, stops transitive
> binding through viewFS, mocking, etc.
> # It complicates security in a kerberized world. If you have an FS instance
> of user A, then you should be able to create an MPU instance with that user's
> permissions. currently, if a service were to try to create one, you'd be
> looking at doAs() games around the service loading, and a more complex bind
> process.
> Proposed
> # remove the service loader mech entirely
> # add to FS & FC as createMultipartUploader(path) call, which will create one
> bound to the current FS, with its permissions, DTs, etc.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
