aokolnychyi commented on code in PR #6371:
URL: https://github.com/apache/iceberg/pull/6371#discussion_r1042622671
##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkSQLProperties.java:
##########
@@ -42,4 +42,9 @@ private SparkSQLProperties() {}
// Controls whether to check the order of fields during writes
public static final String CHECK_ORDERING =
"spark.sql.iceberg.check-ordering";
public static final boolean CHECK_ORDERING_DEFAULT = true;
+
+ // Controls whether to preserve the existing grouping of data while planning
splits
+ public static final String PRESERVE_DATA_GROUPING =
Review Comment:
I am afraid this one will be pretty public. Users will have to explicitly
enable this as we don't know if Spark can benefit from the reported
distribution and skip shuffles. Hence, we disable it by default to avoid any
performance regressions caused by less dense packing of splits.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
