GitHub user HyukjinKwon opened a pull request:
https://github.com/apache/spark/pull/17785
[SPARK-20493][R] De-deuplicate parse logics for DDL-like type strings in R
## What changes were proposed in this pull request?
It seems we are using `SQLUtils.getSQLDataType` for type string in
structField. It looks we can replace this with
`CatalystSqlParser.parseDataType`.
They look similar DDL-like type definitions as below:
```scala
scala> Seq(Tuple1(Tuple1("a"))).toDF.show()
```
```
+---+
| _1|
+---+
|[a]|
+---+
```
```scala
scala>
Seq(Tuple1(Tuple1("a"))).toDF.select($"_1".cast("struct<_1:string>")).show()
```
```
+---+
| _1|
+---+
|[a]|
+---+
```
Such type strings looks identical when Râs one as below:
```R
> write.df(sql("SELECT named_struct('_1', 'a') as struct"), "/tmp/aa",
"parquet")
> collect(read.df("/tmp/aa", "parquet", structType(structField("struct",
"struct<_1:string>"))))
struct
1 a
```
Râs one is more stricter because we are checking the types via regular
expressions in R side.
Actual logics there look a bit different but as we check it ahead in R
side, it looks replacing it would not introduce (I think) no behaviour changes.
To make this sure, the tests dedicated for it were added in SPARK-20105. (It
looks `structField` is the only place that calls this method).
## How was this patch tested?
Existing tests -
https://github.com/apache/spark/blob/master/R/pkg/inst/tests/testthat/test_sparkSQL.R#L143-L194
should cover this.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/HyukjinKwon/spark SPARK-20493
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/17785.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #17785
----
commit 8874de1901b54db740fd439905748a4a2e87c795
Author: hyukjinkwon <gurwls...@gmail.com>
Date: 2017-04-27T15:18:12Z
De-deuplicate parse logics for DDL-like type string in R
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
