viirya commented on a change in pull request #28996:
URL: https://github.com/apache/spark/pull/28996#discussion_r453000366
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala
##########
@@ -2030,7 +2030,25 @@ class Dataset[T] private[sql](
* @group typedrel
* @since 2.3.0
*/
- def unionByName(other: Dataset[T]): Dataset[T] = withSetOperator {
+ def unionByName(other: Dataset[T]): Dataset[T] = unionByName(other, false)
+
+ /**
+ * Returns a new Dataset containing union of rows in this Dataset and
another Dataset.
+ *
+ * This is different from both `UNION ALL` and `UNION DISTINCT` in SQL. To
do a SQL-style set
+ * union (that does deduplication of elements), use this function followed
by a [[distinct]].
Review comment:
Actually in original `unionByName`, its doc has this section too:
> This is different from both `UNION ALL` and `UNION DISTINCT` in SQL. To do
a SQL-style set
> union (that does deduplication of elements), use this function followed by
a [[distinct]].
Re-read this doc, even with original `unionByName` behavior, it is a bit
confusing to me. Do you think we should remove "To do a SQL-style set union
(that does deduplication of elements), use this function followed by a
[[distinct]]."?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
