amogh-jahagirdar commented on code in PR #17198:
URL: https://github.com/apache/iceberg/pull/17198#discussion_r3786896796
##########
format/spec.md:
##########
@@ -674,11 +674,13 @@ A manifest file must store the partition spec and other
metadata as properties i
| _optional_ | _required_ | `format-version` | Table format version
number of the manifest as a string
|
| | _required_ | `content` | Type of content files
tracked by the manifest: "data" or "deletes"
|
-The schema of a manifest file is defined by the `manifest_entry` struct,
described in the following section.
+#### Content file uniqueness
+
+Within a snapshot, each content file must be referenced by at most one
manifest entry across all manifests; otherwise, the snapshot has undefined
behavior. Writers are not required to validate uniqueness at commit time.
Review Comment:
I think this is probably stemming from my review a while back
https://github.com/apache/iceberg/pull/17198#discussion_r3588578656
We've talked offline about this. I really think the spec needs to take a
clear stance around the ADDED + DELETED case, and the "at most one _live_
entry" kinda steps over that. I think basically everyone agrees that it's not
recommended and can lead to needless ambiguity when it comes to change
detection and file cleanup. The scan planning portion covers readers and that's
great but I feel like if we know there are scenarios that we know generally are
bad ideas, and aren't much of a burden to writers to actually have to handle,
we should probably not allow those in the spec in my opinion. It gives stronger
guarantees about the format for consumers, which I think is worthwhile.
If we feel strongly about only "live" here, I'm OK because we could always
make implementations better but this just feels like a simple thing to keep in
the spec and have better guarantees to work with.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
