Github user TRANTANKHOA commented on a diff in the pull request:
https://github.com/apache/spark/pull/20947#discussion_r178444354
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala ---
@@ -1593,7 +1596,9 @@ class Dataset[T] private[sql](
def groupBy(col1: String, cols: String*): RelationalGroupedDataset = {
val colNames: Seq[String] = col1 +: cols
RelationalGroupedDataset(
- toDF(), colNames.map(colName => resolve(colName)),
RelationalGroupedDataset.GroupByType)
+ toDF(),
+ colNames.distinct.map(colName => resolve(colName)),
--- End diff --
Yes, this ticket is only about making this behavior change. So the question
really is if the team think that it is the expected behavior. I personally find
it helps eliminate bugs in our ETL. I don't think anyone need to have
duplicated columns in their grouped data set by intention.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
