amogh-jahagirdar commented on code in PR #11273:
URL: https://github.com/apache/iceberg/pull/11273#discussion_r1828091714
##########
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkPositionDeltaWrite.java:
##########
@@ -361,6 +389,10 @@ private static class PositionDeltaWriteFactory implements
DeltaWriterFactory {
@Override
public DeltaWriter<InternalRow> createWriter(int partitionId, long taskId)
{
Table table = tableBroadcast.value();
+ Map<String, DeleteFileSet> rewritableDeletes = Maps.newHashMap();
Review Comment:
Good suggestion, done! Note, since it's possible now that rewritable deletes
is null in the writer, I explicitly pass in the previous loader to be a
function path -> null. Previously I was passing in an empty map so the lookup
would return null, but i think your suggestion is better because we can avoid
having to do a lookup in the map entirely
##########
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkPositionDeltaWrite.java:
##########
@@ -169,7 +174,13 @@ public DeltaWriterFactory
createBatchWriterFactory(PhysicalWriteInfo info) {
// broadcast the table metadata as the writer factory will be sent to
executors
Broadcast<Table> tableBroadcast =
sparkContext.broadcast(SerializableTableWithSize.copyOf(table));
- return new PositionDeltaWriteFactory(tableBroadcast, command, context,
writeProperties);
+ Broadcast<Map<String, DeleteFileSet>> rewritableDeletes = null;
+ if (context.deleteGranularity() == DeleteGranularity.FILE && scan !=
null) {
+ rewritableDeletes = sparkContext.broadcast(scan.rewritableDeletes());
Review Comment:
This is fixed now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
