hameizi commented on a change in pull request #2916:
URL: https://github.com/apache/iceberg/pull/2916#discussion_r687430019
##########
File path: core/src/main/java/org/apache/iceberg/io/BaseTaskWriter.java
##########
@@ -105,7 +112,11 @@ protected BaseEqualityDeltaWriter(StructLike partition,
Schema schema, Schema de
this.dataWriter = new RollingFileWriter(partition);
this.eqDeleteWriter = new RollingEqDeleteWriter(partition);
- this.posDeleteWriter = new SortedPosDeleteWriter<>(appenderFactory,
fileFactory, format, partition);
+ this.posDeleteWriter = new SortedPosDeleteWriter<>(appenderFactory,
+ fileFactory,
+ format,
+ partition,
+ getRecordsNumThreshold(properties));
Review comment:
> > Sometimes certain maps made by Guava etc (ImmutableMap comes to
mind) are not serializable, particularly when using Kryo (which is not the
default but the de facto default with Spark).
>
> I'm sorry i did't test this, i will try test these case. And pass around
table properties conveniency for develop later. If there is some bug to pass
around table properties i can fix it to just the value of this propertie.
@kbendick I found all my change of this PR just RowDataTaskWriterFactory is
serializable, but it will not be serializd by Kryo in flink, so i think there
is no worry in Kyro serialize in this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
