amogh-jahagirdar commented on code in PR #6651:
URL: https://github.com/apache/iceberg/pull/6651#discussion_r1084484101
##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/source/SparkWrite.java:
##########
@@ -290,7 +293,12 @@ public String toString() {
private class BatchAppend extends BaseBatchWrite {
@Override
public void commit(WriterCommitMessage[] messages) {
- AppendFiles append = table.newAppend();
+ AppendFiles append;
+ if (branch != null) {
+ append = table.newAppend().toBranch(branch);
+ } else {
+ append = table.newAppend();
+ }
Review Comment:
If we do what I mentioned above we don't need if/else here. A branch will
always be defined.
##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/source/SparkWrite.java:
##########
@@ -312,7 +320,13 @@ public void commit(WriterCommitMessage[] messages) {
return;
}
- ReplacePartitions dynamicOverwrite = table.newReplacePartitions();
+ ReplacePartitions dynamicOverwrite;
+ if (branch != null) {
+ dynamicOverwrite = table.newReplacePartitions().toBranch(branch);
+ } else {
+ dynamicOverwrite = table.newReplacePartitions();
+ }
Review Comment:
Same as above we can remove the if/else after the change for a branch to
always be defined.
##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkWriteConf.java:
##########
@@ -304,4 +304,9 @@ public boolean caseSensitive() {
.defaultValue(SQLConf.CASE_SENSITIVE().defaultValueString())
.parse();
}
+
+ public String branch() {
+ return
confParser.stringConf().option(SparkWriteOptions.BRANCH).parseOptional();
Review Comment:
Can this return main if no branch is passed in?
##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/source/SparkWrite.java:
##########
@@ -411,7 +429,12 @@ private Expression conflictDetectionFilter() {
@Override
public void commit(WriterCommitMessage[] messages) {
- OverwriteFiles overwriteFiles = table.newOverwrite();
+ OverwriteFiles overwriteFiles;
+ if (branch != null) {
+ overwriteFiles = table.newOverwrite().toBranch(branch);
+ } else {
+ overwriteFiles = table.newOverwrite();
+ }
Review Comment:
Same as above
##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/source/SparkWrite.java:
##########
@@ -349,8 +363,12 @@ private OverwriteByFilter(Expression overwriteExpr) {
@Override
public void commit(WriterCommitMessage[] messages) {
- OverwriteFiles overwriteFiles = table.newOverwrite();
- overwriteFiles.overwriteByRowFilter(overwriteExpr);
+ OverwriteFiles overwriteFiles;
+ if (branch != null) {
+ overwriteFiles =
table.newOverwrite().toBranch(branch).overwriteByRowFilter(overwriteExpr);
+ } else {
+ overwriteFiles =
table.newOverwrite().overwriteByRowFilter(overwriteExpr);
Review Comment:
Same as above we can remove the if/else since a branch will always be
defined.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
