anoopj commented on code in PR #2922:
URL: https://github.com/apache/iceberg-rust/pull/2922#discussion_r3686841017
##########
crates/iceberg/src/spec/manifest_list/manifest_file.rs:
##########
@@ -195,8 +195,61 @@ impl ManifestFile {
entry.inherit_data(self);
}
+ self.assign_first_row_ids(&mut entries)?;
Review Comment:
Fixed. Folded \`first_row_id\` into the cache key
(\`CachedObjectKey::Manifest((String, Option<u64>))\`) so the same manifest
path referenced with different offsets across snapshots/branches no longer
shares an entry. Added \`test_get_manifest_keys_on_first_row_id\` in
\`object_cache.rs\` that loads the same path twice with different offsets and
asserts distinct inherited ids.
##########
crates/iceberg/src/spec/manifest_list/manifest_file.rs:
##########
@@ -195,8 +195,61 @@ impl ManifestFile {
entry.inherit_data(self);
}
+ self.assign_first_row_ids(&mut entries)?;
+
Ok(Manifest::new(metadata, entries))
}
+
+ /// Assigns `first_row_id` to data-file entries that do not already have
one,
+ /// starting from the manifest-level `first_row_id` and advancing by each
+ /// entry's record count. See the row-lineage inheritance rules in
+ ///
<https://github.com/apache/iceberg/blob/main/format/spec.md#first-row-id-inheritance>.
+ fn assign_first_row_ids(&self, entries: &mut [ManifestEntry]) ->
Result<()> {
+ let Some(manifest_first_row_id) = self.first_row_id else {
+ return Ok(());
+ };
+
+ // A `first_row_id` is only valid on data manifests; delete files
always
+ // have a null `first_row_id`.
+ if self.content != ManifestContentType::Data {
Review Comment:
Changed to no-op + \`tracing::warn!\`, matching Java's
\`readDeleteManifest\` (which never passes \`firstRowId\` to the reader). Test
updated accordingly (\`test_assign_first_row_ids_ignores_delete_manifest\`).
##########
crates/iceberg/src/spec/manifest_list/manifest_file.rs:
##########
@@ -195,8 +195,61 @@ impl ManifestFile {
entry.inherit_data(self);
}
+ self.assign_first_row_ids(&mut entries)?;
+
Ok(Manifest::new(metadata, entries))
}
+
+ /// Assigns `first_row_id` to data-file entries that do not already have
one,
+ /// starting from the manifest-level `first_row_id` and advancing by each
+ /// entry's record count. See the row-lineage inheritance rules in
+ ///
<https://github.com/apache/iceberg/blob/main/format/spec.md#first-row-id-inheritance>.
+ fn assign_first_row_ids(&self, entries: &mut [ManifestEntry]) ->
Result<()> {
+ let Some(manifest_first_row_id) = self.first_row_id else {
Review Comment:
Good catch. Added the null-clearing path: on a data manifest with no
manifest-level \`first_row_id\`, each entry's \`first_row_id\` is set to
\`None\`. This is Java's committed \`idAssigner\` branch. Note the
\`!isCommitted\` branch (preserve for uncommitted merges) has no equivalent
here — \`load_manifest\` is only ever a committed read path in iceberg-rust;
the uncommitted case is write-side (\`MergingSnapshotProducer\`), which we
don't go through. Covered by
\`test_assign_first_row_ids_clears_without_manifest_first_row_id\`.
##########
crates/iceberg/src/spec/manifest_list/manifest_file.rs:
##########
@@ -195,8 +195,61 @@ impl ManifestFile {
entry.inherit_data(self);
}
+ self.assign_first_row_ids(&mut entries)?;
+
Ok(Manifest::new(metadata, entries))
}
+
+ /// Assigns `first_row_id` to data-file entries that do not already have
one,
+ /// starting from the manifest-level `first_row_id` and advancing by each
+ /// entry's record count. See the row-lineage inheritance rules in
+ ///
<https://github.com/apache/iceberg/blob/main/format/spec.md#first-row-id-inheritance>.
+ fn assign_first_row_ids(&self, entries: &mut [ManifestEntry]) ->
Result<()> {
+ let Some(manifest_first_row_id) = self.first_row_id else {
+ return Ok(());
+ };
+
+ // A `first_row_id` is only valid on data manifests; delete files
always
+ // have a null `first_row_id`.
+ if self.content != ManifestContentType::Data {
+ return Err(Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "first_row_id is not valid for delete manifests (found
{manifest_first_row_id} on {})",
+ self.manifest_path
+ ),
+ ));
+ }
+
+ let mut next_row_id = i64::try_from(manifest_first_row_id).map_err(|_|
{
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!("Invalid first_row_id: {manifest_first_row_id}
(exceeds i64::MAX)"),
+ )
+ })?;
+
+ for entry in entries {
+ if !entry.is_alive() {
+ continue;
+ }
+
+ if entry.data_file.first_row_id.is_none() {
+ entry.data_file.first_row_id = Some(next_row_id);
+ let record_count = entry.data_file.record_count;
+ next_row_id =
next_row_id.checked_add_unsigned(record_count).ok_or_else(|| {
+ Error::new(
+ ErrorKind::DataInvalid,
+ format!(
+ "Row ID overflow assigning first_row_id in {}.
Next Row ID: {next_row_id}, Record Count: {record_count}",
Review Comment:
Reworded to \`"File first_row_id: {file_first_row_id}, record count:
{record_count}"\`, and introduced a \`file_first_row_id\` binding so the
message names the triggering file's own id rather than "next".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
