JeonDaehong commented on code in PR #1188:
URL: https://github.com/apache/paimon-rust/pull/1188#discussion_r4337571972
##########
crates/paimon/src/table/table_write.rs:
##########
@@ -1561,7 +1595,15 @@ impl TableWrite {
let partition_path = self.resolve_partition_path(&partition_bytes)?;
let writer = if self.primary_key_indices.is_empty() {
- self.create_append_writer(partition_path, bucket)?
+ let key = (partition_bytes.clone(), bucket);
+ let config = self
+ .append_write_configs
+ .get(&key)
+ .cloned()
+ .unwrap_or_else(|| self.append_write_config());
+ let writer = self.create_append_writer(partition_path, bucket,
&config)?;
+ self.append_write_configs.insert(key, config);
Review Comment:
Nothing removes an entry from `append_write_configs`, so a partition/bucket
keeps its first settings for the whole life of the `TableWrite`. Java only
keeps them while the writer is alive. `AbstractFileStoreWrite.prepareCommit`
drops a writer that wasn't modified since the last committed identifier, and
the next write to that bucket creates a writer with the current `writeType` /
`BlobConsumer`.
I checked with paimon-bundle 2.1.0, DE append table `(pt, id, a, b)`
partitioned by `pt`:
1. Write `(1, 1, 10, 100)`, then `prepareCommit` + commit.
2. Write `(9, 9, 9, 9)` to another partition, then `prepareCommit` + commit.
This is the round where pt=1 is idle.
3. `withWriteType(pt, id, a)`, write `(1, 2, 20)`, commit.
Java reads back `pt=1 id=2 a=20 b=null`. With no idle round in between, Java
fails in step 3 (`ArrayIndexOutOfBoundsException`), which matches what this PR
keeps.
The same sequence here (`with_write_type([pt, id])`, write pt=1, commit,
write pt=9, commit, `with_write_type([pt, omitted])`, write pt=1) fails with
`field name mismatch at index 1: expected 'id', actual 'omitted'`, after 0, 1
or 2 idle rounds.
Should a config be dropped once its bucket had no writes in a committed
round, like the Java writer? That would also bound this map for a long-running
writer that touches many partitions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
