GitHub user HyukjinKwon reopened a pull request: https://github.com/apache/spark/pull/17785
[SPARK-20493][R] De-deuplicate parse logics for DDL-like type strings in R ## What changes were proposed in this pull request? It seems we are using `SQLUtils.getSQLDataType` for type string in structField. It looks we can replace this with `CatalystSqlParser.parseDataType`. They look similar DDL-like type definitions as below: ```scala scala> Seq(Tuple1(Tuple1("a"))).toDF.show() ``` ``` +---+ | _1| +---+ |[a]| +---+ ``` ```scala scala> Seq(Tuple1(Tuple1("a"))).toDF.select($"_1".cast("struct<_1:string>")).show() ``` ``` +---+ | _1| +---+ |[a]| +---+ ``` Such type strings looks identical when Râs one as below: ```R > write.df(sql("SELECT named_struct('_1', 'a') as struct"), "/tmp/aa", "parquet") > collect(read.df("/tmp/aa", "parquet", structType(structField("struct", "struct<_1:string>")))) struct 1 a ``` Râs one is stricter because we are checking the types via regular expressions in R side ahead. Actual logics there look a bit different but as we check it ahead in R side, it looks replacing it would not introduce (I think) no behaviour changes. To make this sure, the tests dedicated for it were added in SPARK-20105. (It looks `structField` is the only place that calls this method). ## How was this patch tested? Existing tests - https://github.com/apache/spark/blob/master/R/pkg/inst/tests/testthat/test_sparkSQL.R#L143-L194 should cover this. You can merge this pull request into a Git repository by running: $ git pull https://github.com/HyukjinKwon/spark SPARK-20493 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/17785.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #17785 ---- commit 8874de1901b54db740fd439905748a4a2e87c795 Author: hyukjinkwon <gurwls...@gmail.com> Date: 2017-04-27T15:18:12Z De-deuplicate parse logics for DDL-like type string in R commit b802e36666afedb78f30def538bb905b647334b6 Author: hyukjinkwon <gurwls...@gmail.com> Date: 2017-04-27T15:32:06Z Minimised fix ---- --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org