This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-1.1-lts
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-1.1-lts by this push:
new d0fc8323bf [fix](spark load)The where condition does not take effect
when spark load loads the file #13804
d0fc8323bf is described below
commit d0fc8323bf08d37d0af7d29a71c8f9532fe072cb
Author: jiafeng.zhang <[email protected]>
AuthorDate: Mon Oct 31 15:01:46 2022 +0800
[fix](spark load)The where condition does not take effect when spark load
loads the file #13804
The where condition does not take effect when spark load loads the file
---
.../src/main/java/org/apache/doris/load/loadv2/dpp/SparkDpp.java | 3 +++
1 file changed, 3 insertions(+)
diff --git
a/fe/spark-dpp/src/main/java/org/apache/doris/load/loadv2/dpp/SparkDpp.java
b/fe/spark-dpp/src/main/java/org/apache/doris/load/loadv2/dpp/SparkDpp.java
index 5e0fbe2086..763a0e9ee4 100644
--- a/fe/spark-dpp/src/main/java/org/apache/doris/load/loadv2/dpp/SparkDpp.java
+++ b/fe/spark-dpp/src/main/java/org/apache/doris/load/loadv2/dpp/SparkDpp.java
@@ -727,6 +727,9 @@ public final class SparkDpp implements java.io.Serializable
{
);
Dataset<Row> dataframe = spark.createDataFrame(rowRDD, srcSchema);
+ if (!Strings.isNullOrEmpty(fileGroup.where)) {
+ dataframe = dataframe.where(fileGroup.where);
+ }
return dataframe;
}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
