[
https://issues.apache.org/jira/browse/HDFS-13713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16602927#comment-16602927
]
Ewan Higgs commented on HDFS-13713:
-----------------------------------
{quote}something about how all listed handles must be in
keys(FS.uploads(uploadHandle).parts){quote}
{quote}what if there are handles in the MPU which aren't included in the
map?{quote}
They are not part of the resulting file and should be cleaned up by the MPU
uploader implementation if the underlying backing store does not do this
automatically.
{quote}what if there are duplicate entries in the hand in the parts list
provided?{quote}
This fails in HDFS with {{org.apache.hadoop.HadoopIllegalArgumentException:
concat: at least two of the source files are the same}}
This fails in S3 with {{AmazonS3Exception: The list of parts was not in
ascending order.}}
{quote}what if, at the point of completion, there is now a directory at the
destination?{quote}
AFAICS, this is a race condition that can only be left as implementation
dependent undefined behaviour since some backing stores don't have directories,
and the FileSystem API doesn't have transactions or CaS to only write a file if
nothing it already there.
> Add specification of Multipart Upload API to FS specification, with contract
> tests
> ----------------------------------------------------------------------------------
>
> Key: HDFS-13713
> URL: https://issues.apache.org/jira/browse/HDFS-13713
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: fs, test
> Affects Versions: 3.2.0
> Reporter: Steve Loughran
> Assignee: Ewan Higgs
> Priority: Blocker
> Attachments: HDFS-13713.001.patch, HDFS-13713.002.patch,
> multipartuploader.md
>
>
> There's nothing in the FS spec covering the new API. Add it in a new .md file
> * add FS model with the notion of a function mapping (uploadID -> Upload),
> the operations (list, commit, abort). The [TLA+
> mode|https://issues.apache.org/jira/secure/attachment/12865161/objectstore.pdf]l
> of HADOOP-13786 shows how to do this.
> * Contract tests of not just the successful path, but all the invalid ones.
> * implementations of the contract tests of all FSs which support the new API.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
