[GitHub] spark pull request #20674: [SPARK-23465][SQL] Introduce new function to rena...
Github user misutoth closed the pull request at: https://github.com/apache/spark/pull/20674 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20674: [SPARK-23465][SQL] Introduce new function to rena...
Github user misutoth commented on a diff in the pull request:
https://github.com/apache/spark/pull/20674#discussion_r171386636
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala ---
@@ -2239,6 +2239,34 @@ class Dataset[T] private[sql](
}
}
+ /**
+* Returns a new Dataset with altered column names.
+* This is a no-op if schema doesn't contain existingName.
+*
+* @param convert conversion function from the old to the new name
+*
+* @group untypedrel
+* @since 2.0.0
+*/
+ def withAllColumnsRenamed(convert: String => String): DataFrame = {
+val output = queryExecution.analyzed.output
+var containsRename = false
--- End diff --
Thats correct. Removed that optimization and the code became much smaller.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20674: [SPARK-23465][SQL] Introduce new function to rena...
Github user attilapiros commented on a diff in the pull request:
https://github.com/apache/spark/pull/20674#discussion_r171341653
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala ---
@@ -2239,6 +2239,34 @@ class Dataset[T] private[sql](
}
}
+ /**
+* Returns a new Dataset with altered column names.
+* This is a no-op if schema doesn't contain existingName.
+*
+* @param convert conversion function from the old to the new name
+*
+* @group untypedrel
+* @since 2.0.0
+*/
+ def withAllColumnsRenamed(convert: String => String): DataFrame = {
+val output = queryExecution.analyzed.output
+var containsRename = false
--- End diff --
This should not be optimised for the case when convert function is the
identity. This way you could get rid of the flag 'containsRename'.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20674: [SPARK-23465][SQL] Introduce new function to rena...
GitHub user misutoth opened a pull request: https://github.com/apache/spark/pull/20674 [SPARK-23465][SQL] Introduce new function to rename columns using an algoritm ## What changes were proposed in this pull request? Add an additional convenient method to rename multiple of columns by specifying a mapping between the old and the new column name. ## How was this patch tested? Wrote additional unit test cases, ran scalastyle. You can merge this pull request into a Git repository by running: $ git pull https://github.com/misutoth/spark column-rename Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/20674.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #20674 commit f03884c3582cdb3bc2eaa034cbc0bcdcfd1c8250 Author: Mihaly Toth Date: 2018-02-24T20:19:19Z [SPARK-23465][SQL] Introduce new function to rename columns using an algorithm --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
