geruh commented on code in PR #9292:
URL: https://github.com/apache/iceberg/pull/9292#discussion_r1426221963
##########
open-api/rest-catalog-open-api.yaml:
##########
@@ -2367,6 +2368,21 @@ components:
items:
type: string
+ AppendFilesUpdate:
+ allOf:
+ - $ref: '#/components/schemas/BaseUpdate'
+ required:
+ - appended-manifests
+ properties:
+ action:
+ type: string
+ enum: [ "append-files" ]
+ appended-manifests:
Review Comment:
Hey Jack, you make a good point. The intentions behind the appended-manifest
is to showcase the potential need for compression, and it was a middle ground
between a List of DataFiles and a ManifestList. In the case of a large scale
Append operation this list can become very large therefore, we can trade some
overhead for these operations. However, at the end of the day this is a list of
string locations. How the service appends these files is depending on it's
ability to read
[DataFiles](https://iceberg.apache.org/javadoc/master/org/apache/iceberg/DataFiles.html)
into their service from the Request body.
Some other potential approaches we could take are:
* Pagination/batching
* Compression (GZIP, Brotli)
* File Upload of a different type other than avro.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For additional commands, e-mail: issues-h...@iceberg.apache.org
