Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/20647#discussion_r171143887
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Relation.scala
---
@@ -107,19 +104,36 @@ case class DataSourceV2Relation(
}
/**
- * A specialization of DataSourceV2Relation with the streaming bit set to
true. Otherwise identical
- * to the non-streaming relation.
+ * A specialization of [[DataSourceV2Relation]] with the streaming bit set
to true.
+ *
+ * Note that, this plan has a mutable reader, so Spark won't apply
operator push-down for this plan,
+ * to avoid making the plan mutable. We should consolidate this plan and
[[DataSourceV2Relation]]
+ * after we figure out how to apply operator push-down for streaming data
sources.
*/
case class StreamingDataSourceV2Relation(
output: Seq[AttributeReference],
+ source: DataSourceV2,
+ options: Map[String, String],
reader: DataSourceReader)
- extends LeafNode with DataSourceReaderHolder with
MultiInstanceRelation {
+ extends LeafNode with MultiInstanceRelation with
DataSourceV2StringFormat {
+
override def isStreaming: Boolean = true
- override def canEqual(other: Any): Boolean =
other.isInstanceOf[StreamingDataSourceV2Relation]
+ override def simpleString: String = "Streaming RelationV2 " +
metadataString
override def newInstance(): LogicalPlan = copy(output =
output.map(_.newInstance()))
+ // TODO: unify the equal/hashCode implementation for all data source v2
query plans.
+ override def equals(other: Any): Boolean = other match {
+ case other: StreamingDataSourceV2Relation =>
+ output == other.output && reader.getClass == other.reader.getClass
&& options == other.options
--- End diff --
Now it's exactly same as before. We should clean it up after figure out how
to push down operators to streaming relation.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
