RussellSpitzer commented on code in PR #15150:
URL: https://github.com/apache/iceberg/pull/15150#discussion_r2754918198
##########
spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/SparkWriteRequirements.java:
##########
@@ -26,18 +26,32 @@
/** A set of requirements such as distribution and ordering reported to Spark
during writes. */
public class SparkWriteRequirements {
+ public static final long NO_ADVISORY_PARTITION_SIZE = 0;
public static final SparkWriteRequirements EMPTY =
- new SparkWriteRequirements(Distributions.unspecified(), new
SortOrder[0], 0);
+ new SparkWriteRequirements(
+ Distributions.unspecified(),
+ new SortOrder[0],
+ org.apache.iceberg.SortOrder.unsorted(),
+ NO_ADVISORY_PARTITION_SIZE);
private final Distribution distribution;
private final SortOrder[] ordering;
+ private final org.apache.iceberg.SortOrder icebergOrdering;
private final long advisoryPartitionSize;
SparkWriteRequirements(
Review Comment:
I'm not sure we should be passing the Iceberg ordering all the way down, I
feel like all the writer actually needs to know is the ID it should be using in
the Datafile for the sort order used and we should be doing all the work to
determine what sortorder is and how to pass it as a spark ordering in code
calling this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
