[GitHub] spark pull request #22954: [SPARK-25981][R] Enables Arrow optimization from ...

2018-11-12 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22954#discussion_r232619364 --- Diff: R/pkg/R/SQLContext.R --- @@ -172,36 +257,72 @@ getDefaultSqlSource <- function() { createDataFrame <- function(data, schema

[GitHub] spark pull request #22954: [SPARK-25981][R] Enables Arrow optimization from ...

2018-11-12 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22954#discussion_r232618853 --- Diff: R/pkg/R/SQLContext.R --- @@ -172,36 +257,72 @@ getDefaultSqlSource <- function() { createDataFrame <- function(data, schema

[GitHub] spark pull request #22954: [SPARK-25981][R] Enables Arrow optimization from ...

2018-11-12 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22954#discussion_r232620582 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/api/r/SQLUtils.scala --- @@ -225,4 +226,25 @@ private[sql] object SQLUtils extends Logging

[GitHub] spark pull request #22944: [SPARK-25942][SQL] Aggregate expressions shouldn'...

2018-11-12 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22944#discussion_r232573269 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/DatasetSuite.scala --- @@ -1556,6 +1556,20 @@ class DatasetSuite extends QueryTest

[GitHub] spark pull request #22944: [SPARK-25942][SQL] Aggregate expressions shouldn'...

2018-11-12 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22944#discussion_r232571836 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/DatasetSuite.scala --- @@ -1556,6 +1556,20 @@ class DatasetSuite extends QueryTest

[GitHub] spark pull request #23012: [SPARK-26014][R] Deprecate R prior to version 3.4...

2018-11-11 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/23012#discussion_r232549062 --- Diff: docs/index.md --- @@ -31,7 +31,8 @@ Spark runs on both Windows and UNIX-like systems (e.g. Linux, Mac OS). It's easy locally on one machine

[GitHub] spark pull request #23012: [SPARK-26014][R] Deprecate R prior to version 3.4...

2018-11-11 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/23012#discussion_r232548211 --- Diff: docs/index.md --- @@ -31,7 +31,8 @@ Spark runs on both Windows and UNIX-like systems (e.g. Linux, Mac OS). It's easy locally on one machine

[GitHub] spark pull request #22989: [SPARK-25986][Build] Banning throw new OutOfMemor...

2018-11-11 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22989#discussion_r232493703 --- Diff: scalastyle-config.xml --- @@ -240,6 +240,18 @@ This file is divided into 3 sections: ]]> + +throw

[GitHub] spark pull request #22986: [SPARK-25959][ML] GBTClassifier picks wrong impur...

2018-11-10 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22986#discussion_r232446160 --- Diff: mllib/src/main/scala/org/apache/spark/ml/tree/treeParams.scala --- @@ -538,7 +538,7 @@ private[ml] object GBTClassifierParams { Array

[GitHub] spark pull request #22986: [SPARK-25959][ML] GBTClassifier picks wrong impur...

2018-11-10 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22986#discussion_r232445980 --- Diff: mllib/src/main/scala/org/apache/spark/ml/tree/treeParams.scala --- @@ -538,7 +538,7 @@ private[ml] object GBTClassifierParams { Array

[GitHub] spark issue #22275: [SPARK-25274][PYTHON][SQL] In toPandas with Arrow send u...

2018-11-09 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/22275 LGTM the current change looks clearer. Thanks @BryanCutler --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org

[GitHub] spark pull request #21732: [SPARK-24762][SQL] Enable Option of Product encod...

2018-11-09 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/21732#discussion_r232302254 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/encoders/ExpressionEncoder.scala --- @@ -207,7 +198,7 @@ case class ExpressionEncoder[T

[GitHub] spark pull request #22989: [SPARK-25986][Build] Banning throw new OutOfMemor...

2018-11-09 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22989#discussion_r232245886 --- Diff: scalastyle-config.xml --- @@ -240,6 +240,18 @@ This file is divided into 3 sections: ]]> + +throw

[GitHub] spark issue #22944: [SPARK-25942][SQL] Aggregate expressions shouldn't be re...

2018-11-09 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/22944 ping @cloud-fan --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #22880: [SPARK-25407][SQL] Ensure we pass a compatible pruned sc...

2018-11-07 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/22880 retest this please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #22944: [SPARK-25942][SQL] Fix Dataset.groupByKey to make it wor...

2018-11-07 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/22944 @cloud-fan I have a simpler fix for this issue. Can you take another look? Thanks. --- - To unsubscribe, e-mail: reviews

[GitHub] spark pull request #22944: [SPARK-25942][SQL] Fix Dataset.groupByKey to make...

2018-11-06 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22944#discussion_r231359624 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/object.scala --- @@ -262,25 +262,39 @@ object AppendColumns { def

[GitHub] spark pull request #22944: [SPARK-25942][SQL] Fix Dataset.groupByKey to make...

2018-11-06 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22944#discussion_r231350156 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/object.scala --- @@ -262,25 +262,39 @@ object AppendColumns { def

[GitHub] spark pull request #22944: [SPARK-25942][SQL] Fix Dataset.groupByKey to make...

2018-11-06 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22944#discussion_r231171593 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/object.scala --- @@ -262,25 +262,39 @@ object AppendColumns { def

[GitHub] spark pull request #22944: [SPARK-25942][SQL] Fix Dataset.groupByKey to make...

2018-11-06 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22944#discussion_r231170974 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/object.scala --- @@ -262,25 +262,39 @@ object AppendColumns { def

[GitHub] spark pull request #22880: [SPARK-25407][SQL] Ensure we pass a compatible pr...

2018-11-06 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22880#discussion_r231049073 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetRowConverter.scala --- @@ -202,11 +204,15 @@ private[parquet

[GitHub] spark pull request #22944: [SPARK-25942][SQL] Fix Dataset.groupByKey to make...

2018-11-06 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22944#discussion_r231038560 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/object.scala --- @@ -262,25 +262,39 @@ object AppendColumns { def

[GitHub] spark pull request #22944: [SPARK-25942][SQL] Fix Dataset.groupByKey to make...

2018-11-05 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22944#discussion_r231002129 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/object.scala --- @@ -262,25 +262,39 @@ object AppendColumns { def

[GitHub] spark pull request #22944: [SPARK-25942][SQL] Fix Dataset.groupByKey to make...

2018-11-05 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22944#discussion_r231001655 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/object.scala --- @@ -262,25 +262,39 @@ object AppendColumns { def

[GitHub] spark pull request #22944: [SPARK-25942][SQL] Fix Dataset.groupByKey to make...

2018-11-05 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22944#discussion_r230995240 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/object.scala --- @@ -262,25 +262,39 @@ object AppendColumns { def

[GitHub] spark pull request #22944: [SPARK-25942][SQL] Fix Dataset.groupByKey to make...

2018-11-05 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22944#discussion_r230989493 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/object.scala --- @@ -262,25 +262,39 @@ object AppendColumns { def

[GitHub] spark pull request #22944: [SPARK-25942][SQL] Fix Dataset.groupByKey to make...

2018-11-05 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22944#discussion_r230986888 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/object.scala --- @@ -262,25 +262,39 @@ object AppendColumns { def

[GitHub] spark pull request #22944: [SPARK-25942][SQL] Fix Dataset.groupByKey to make...

2018-11-05 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22944#discussion_r230983077 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/object.scala --- @@ -262,25 +262,39 @@ object AppendColumns { def

[GitHub] spark pull request #22087: [SPARK-25097][ML] Support prediction on single in...

2018-11-05 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22087#discussion_r230968204 --- Diff: mllib/src/test/scala/org/apache/spark/ml/util/MLTest.scala --- @@ -155,4 +155,16 @@ trait MLTest extends StreamTest with TempDirectory { self

[GitHub] spark pull request #22944: [SPARK-25942][SQL] Fix Dataset.groupByKey to make...

2018-11-05 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22944#discussion_r230781583 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/DatasetSuite.scala --- @@ -1556,6 +1556,14 @@ class DatasetSuite extends QueryTest

[GitHub] spark pull request #22947: [SPARK-24913][SQL] Make AssertNotNull and AssertT...

2018-11-05 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22947#discussion_r230741956 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/misc.scala --- @@ -66,6 +66,8 @@ case class AssertTrue(child: Expression

[GitHub] spark pull request #21732: [SPARK-24762][SQL] Enable Option of Product encod...

2018-11-05 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/21732#discussion_r230739979 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/encoders/ExpressionEncoder.scala --- @@ -207,7 +198,7 @@ case class ExpressionEncoder[T

[GitHub] spark pull request #22919: [SPARK-25906][SHELL] Documents '-I' option (from ...

2018-11-05 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22919#discussion_r230695150 --- Diff: bin/spark-shell --- @@ -32,7 +32,10 @@ if [ -z "${SPARK_HOME}" ]; then source "$(dirname "$0")"/find-spark-

[GitHub] spark pull request #22944: [SPARK-25942][SQL] Fix Dataset.groupByKey to make...

2018-11-05 Thread viirya
GitHub user viirya opened a pull request: https://github.com/apache/spark/pull/22944 [SPARK-25942][SQL] Fix Dataset.groupByKey to make it work on primitive data ## What changes were proposed in this pull request? `Dataset.groupByKey` can't work on primitive data now. When

[GitHub] spark issue #22939: [SPARK-25446][R] Add schema_of_json() and schema_of_csv(...

2018-11-04 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/22939 > In addition, it also proposes to make from_csv and from_json accept structType, DDL-formatted string, DDL-formatted string literal, and schema_of_[csv|json] as schema so that we can utilise b

[GitHub] spark pull request #22939: [SPARK-25446][R] Add schema_of_json() and schema_...

2018-11-04 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22939#discussion_r230589804 --- Diff: R/pkg/R/functions.R --- @@ -202,14 +202,18 @@ NULL #' \itemize{ #' \item \code{from_json}: a structType object to use

[GitHub] spark issue #22919: [SPARK-25906][SHELL] Revives '-i' option in spark-shell

2018-11-01 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/22919 hmm, I think this follows previous behavior, but I'm wondering should we follow scala REPL to replace `-i` with `-I

[GitHub] spark pull request #22905: [SPARK-25894][SQL] Add a ColumnarFileFormat type ...

2018-10-31 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22905#discussion_r229932838 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/ColumnarFileFormat.scala --- @@ -0,0 +1,32 @@ +/* + * Licensed

[GitHub] spark pull request #22905: [SPARK-25894][SQL] Add a ColumnarFileFormat type ...

2018-10-31 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22905#discussion_r229697350 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala --- @@ -306,7 +306,15 @@ case class FileSourceScanExec

[GitHub] spark pull request #22898: [SPARK-25746][SQL][followup] do not add unnecessa...

2018-10-31 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22898#discussion_r229631397 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/encoders/ExpressionEncoder.scala --- @@ -124,14 +124,9 @@ object ExpressionEncoder

[GitHub] spark pull request #22898: [SPARK-25746][SQL][followup] do not add unnecessa...

2018-10-31 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22898#discussion_r229619306 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/encoders/ExpressionEncoder.scala --- @@ -124,14 +124,9 @@ object ExpressionEncoder

[GitHub] spark issue #22898: [SPARK-25746][SQL][followup] do not add unnecessary If e...

2018-10-31 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/22898 LGTM --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark pull request #22880: [SPARK-25407][SQL] Ensure we pass a compatible pr...

2018-10-30 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22880#discussion_r229530694 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetReadSupport.scala --- @@ -49,34 +49,82 @@ import

[GitHub] spark pull request #22880: [SPARK-25407][SQL] Ensure we pass a compatible pr...

2018-10-30 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22880#discussion_r229530287 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetReadSupport.scala --- @@ -93,13 +141,14 @@ private[parquet

[GitHub] spark pull request #22880: [SPARK-25407][SQL] Ensure we pass a compatible pr...

2018-10-30 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22880#discussion_r229530592 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetRowConverter.scala --- @@ -202,11 +204,15 @@ private[parquet

[GitHub] spark pull request #22857: [SPARK-25860][SQL] Replace Literal(null, _) with ...

2018-10-30 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22857#discussion_r229529772 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala --- @@ -736,3 +736,60 @@ object CombineConcats extends Rule

[GitHub] spark pull request #22857: [SPARK-25860][SQL] Replace Literal(null, _) with ...

2018-10-30 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22857#discussion_r229528767 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala --- @@ -736,3 +736,60 @@ object CombineConcats extends Rule

[GitHub] spark pull request #22713: [SPARK-25691][SQL] Use semantic equality in Alias...

2018-10-30 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22713#discussion_r229313621 --- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/AnalysisSuite.scala --- @@ -604,4 +606,28 @@ class AnalysisSuite extends

[GitHub] spark issue #22713: [SPARK-25691][SQL] Use semantic equality in AliasViewChi...

2018-10-30 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/22713 LGTM --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark pull request #22880: [SPARK-25407][SQL] Ensure we pass a compatible pr...

2018-10-30 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22880#discussion_r229212944 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetReadSupport.scala --- @@ -93,13 +141,14 @@ private[parquet

[GitHub] spark pull request #22880: [SPARK-25407][SQL] Ensure we pass a compatible pr...

2018-10-30 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22880#discussion_r229208839 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetReadSupport.scala --- @@ -49,34 +49,82 @@ import

[GitHub] spark pull request #22880: [SPARK-25407][SQL] Ensure we pass a compatible pr...

2018-10-30 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22880#discussion_r229204016 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetRowConverter.scala --- @@ -182,18 +182,20 @@ private[parquet

[GitHub] spark pull request #22880: [SPARK-25407][SQL] Ensure we pass a compatible pr...

2018-10-30 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22880#discussion_r229211086 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetRowConverter.scala --- @@ -202,11 +204,15 @@ private[parquet

[GitHub] spark pull request #22857: [SPARK-25860][SQL] Replace Literal(null, _) with ...

2018-10-29 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22857#discussion_r229165719 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala --- @@ -736,3 +736,60 @@ object CombineConcats extends Rule

[GitHub] spark pull request #22857: [SPARK-25860][SQL] Replace Literal(null, _) with ...

2018-10-29 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22857#discussion_r229165497 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala --- @@ -736,3 +736,60 @@ object CombineConcats extends Rule

[GitHub] spark issue #22873: [SPARK-25866][ML] Update KMeans formatVersion

2018-10-29 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/22873 Are we considering to remove it? If no, as `save` now calls `SaveLoadV2_0`, it seems good to update this variable

[GitHub] spark issue #22870: [SPARK-25862][SQL] Remove rangeBetween APIs introduced i...

2018-10-29 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/22870 retest this please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark pull request #21732: [SPARK-24762][SQL] Enable Option of Product encod...

2018-10-29 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/21732#discussion_r228820434 --- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/ScalaReflectionSuite.scala --- @@ -362,4 +362,38 @@ class ScalaReflectionSuite extends

[GitHub] spark pull request #22514: [SPARK-25271][SQL] Hive ctas commands should use ...

2018-10-29 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22514#discussion_r228817044 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/CreateHiveTableAsSelectCommand.scala --- @@ -45,6 +46,11 @@ case class

[GitHub] spark pull request #22514: [SPARK-25271][SQL] Hive ctas commands should use ...

2018-10-29 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22514#discussion_r228813738 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/CreateHiveTableAsSelectCommand.scala --- @@ -45,6 +46,11 @@ case class

[GitHub] spark pull request #21732: [SPARK-24762][SQL] Enable Option of Product encod...

2018-10-28 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/21732#discussion_r228785348 --- Diff: docs/sql-migration-guide-upgrade.md --- @@ -17,6 +17,8 @@ displayTitle: Spark SQL Upgrading Guide - The `ADD JAR` command previously

[GitHub] spark pull request #21732: [SPARK-24762][SQL] Enable Option of Product encod...

2018-10-28 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/21732#discussion_r228785236 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/TypedAggregateExpression.scala --- @@ -76,7 +76,7 @@ object

[GitHub] spark pull request #22514: [SPARK-25271][SQL] Hive ctas commands should use ...

2018-10-28 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22514#discussion_r228776103 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/DataFrameFunctionsSuite.scala --- @@ -2648,7 +2648,7 @@ class DataFrameFunctionsSuite extends

[GitHub] spark pull request #22857: [SPARK-25860][SQL] Replace Literal(null, _) with ...

2018-10-28 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22857#discussion_r228739894 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala --- @@ -736,3 +736,65 @@ object CombineConcats extends Rule

[GitHub] spark issue #22309: [SPARK-20384][SQL] Support value class in schema of Data...

2018-10-28 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/22309 retest this please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #22309: [SPARK-20384][SQL] Support value class in schema of Data...

2018-10-27 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/22309 According to the second rule, value class in Array needs to be instantiated. In current code, I think it should go to the special handling case of value class. Can you also add a test to verify

[GitHub] spark issue #22309: [SPARK-20384][SQL] Support value class in schema of Data...

2018-10-27 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/22309 How about value class in Array? Do we support it? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org

[GitHub] spark pull request #22309: [SPARK-20384][SQL] Support value class in schema ...

2018-10-27 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22309#discussion_r228734242 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/ScalaReflection.scala --- @@ -635,13 +675,17 @@ object ScalaReflection extends

[GitHub] spark pull request #22309: [SPARK-20384][SQL] Support value class in schema ...

2018-10-27 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22309#discussion_r228733929 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/ScalaReflection.scala --- @@ -376,6 +387,23 @@ object ScalaReflection extends

[GitHub] spark pull request #22309: [SPARK-20384][SQL] Support value class in schema ...

2018-10-27 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22309#discussion_r228733678 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/ScalaReflection.scala --- @@ -376,6 +387,23 @@ object ScalaReflection extends

[GitHub] spark pull request #22309: [SPARK-20384][SQL] Support value class in schema ...

2018-10-27 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22309#discussion_r228733525 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/ScalaReflection.scala --- @@ -376,6 +387,23 @@ object ScalaReflection extends

[GitHub] spark pull request #22309: [SPARK-20384][SQL] Support value class in schema ...

2018-10-27 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22309#discussion_r228733511 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/ScalaReflection.scala --- @@ -376,6 +387,23 @@ object ScalaReflection extends

[GitHub] spark issue #22309: [SPARK-20384][SQL] Support value class in schema of Data...

2018-10-27 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/22309 ``` [error] /home/jenkins/workspace/SparkPullRequestBuilder/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/ScalaReflection.scala:401: value isDefined is not a member

[GitHub] spark pull request #22309: [SPARK-20384][SQL] Support value class in schema ...

2018-10-27 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22309#discussion_r228733287 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/ScalaReflection.scala --- @@ -376,6 +387,23 @@ object ScalaReflection extends

[GitHub] spark pull request #22309: [SPARK-20384][SQL] Support value class in schema ...

2018-10-27 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22309#discussion_r228732860 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/ScalaReflection.scala --- @@ -376,6 +387,23 @@ object ScalaReflection extends

[GitHub] spark pull request #22309: [SPARK-20384][SQL] Support value class in schema ...

2018-10-27 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22309#discussion_r228731507 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/ScalaReflection.scala --- @@ -376,6 +387,23 @@ object ScalaReflection extends

[GitHub] spark issue #21732: [SPARK-24762][SQL] Enable Option of Product encoders

2018-10-27 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/21732 Rebased this after refactoring ExpressionEncoder. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org

[GitHub] spark issue #22843: [SPARK-16693][SPARKR] Remove methods deprecated

2018-10-26 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/22843 Some deprecated functions (e.g., createExternalTable, registerTempTable, dropTempTable) are mentioned in `Migration Guide` of Spark R document, but seems some are not. Should we explicitly declare

[GitHub] spark issue #22830: [SPARK-25838][ML] Remove formatVersion from Saveable

2018-10-26 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/22830 cc @mengxr any idea why we have this `formatVersion`? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org

[GitHub] spark issue #22812: [SPARK-25817][SQL] Dataset encoder should support combin...

2018-10-25 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/22812 LGTM --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark pull request #22812: [SPARK-25817][SQL] Dataset encoder should support...

2018-10-25 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22812#discussion_r228390150 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala --- @@ -2384,14 +2384,23 @@ class Analyzer

[GitHub] spark pull request #22812: [SPARK-25817][SQL] Dataset encoder should support...

2018-10-25 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22812#discussion_r228381728 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala --- @@ -2384,14 +2384,23 @@ class Analyzer

[GitHub] spark pull request #22812: [SPARK-25817][SQL] Dataset encoder should support...

2018-10-25 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22812#discussion_r228380982 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/encoders/ExpressionEncoder.scala --- @@ -216,7 +215,6 @@ case class ExpressionEncoder[T

[GitHub] spark issue #22825: [SPARK-25772][SQL][FOLLOWUP] remove GetArrayFromMap

2018-10-25 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/22825 Nice catch! LGTM --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark pull request #22790: [SPARK-25793][ML]call SaveLoadV2_0.load for class...

2018-10-25 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22790#discussion_r228208961 --- Diff: mllib/src/main/scala/org/apache/spark/mllib/clustering/BisectingKMeansModel.scala --- @@ -109,7 +109,7 @@ class BisectingKMeansModel private

[GitHub] spark issue #22749: [SPARK-25746][SQL] Refactoring ExpressionEncoder to get ...

2018-10-25 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/22749 Thanks @cloud-fan --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark pull request #22749: [SPARK-25746][SQL] Refactoring ExpressionEncoder ...

2018-10-25 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22749#discussion_r228135443 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/encoders/ExpressionEncoder.scala --- @@ -212,21 +181,91 @@ object ExpressionEncoder

[GitHub] spark pull request #22749: [SPARK-25746][SQL] Refactoring ExpressionEncoder ...

2018-10-25 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22749#discussion_r228133724 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/encoders/ExpressionEncoder.scala --- @@ -212,21 +181,91 @@ object ExpressionEncoder

[GitHub] spark pull request #22514: [SPARK-25271][SQL] Hive ctas commands should use ...

2018-10-25 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22514#discussion_r228096896 --- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveParquetSuite.scala --- @@ -92,4 +92,18 @@ class HiveParquetSuite extends QueryTest

[GitHub] spark issue #22795: [SPARK-25798][PYTHON] Internally document type conversio...

2018-10-24 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/22795 Sorry for late. This LGTM. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark pull request #22795: [SPARK-25798][PYTHON] Internally document type co...

2018-10-24 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22795#discussion_r228012874 --- Diff: python/pyspark/sql/functions.py --- @@ -3023,6 +3023,42 @@ def pandas_udf(f=None, returnType=None, functionType=None): conversion

[GitHub] spark issue #22514: [SPARK-25271][SQL] Hive ctas commands should use data so...

2018-10-24 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/22514 retest this please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark pull request #22514: [SPARK-25271][SQL] Hive ctas commands should use ...

2018-10-24 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22514#discussion_r228009289 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/CreateHiveTableAsSelectCommand.scala --- @@ -45,6 +46,11 @@ case class

[GitHub] spark pull request #22749: [SPARK-25746][SQL] Refactoring ExpressionEncoder ...

2018-10-24 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22749#discussion_r227996228 --- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/ScalaReflectionSuite.scala --- @@ -280,59 +281,59 @@ class ScalaReflectionSuite extends

[GitHub] spark pull request #22749: [SPARK-25746][SQL] Refactoring ExpressionEncoder ...

2018-10-24 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22749#discussion_r227797378 --- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/ScalaReflectionSuite.scala --- @@ -280,59 +281,59 @@ class ScalaReflectionSuite extends

[GitHub] spark pull request #22749: [SPARK-25746][SQL] Refactoring ExpressionEncoder ...

2018-10-24 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22749#discussion_r227796616 --- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/ScalaReflectionSuite.scala --- @@ -280,59 +281,59 @@ class ScalaReflectionSuite extends

[GitHub] spark pull request #22749: [SPARK-25746][SQL] Refactoring ExpressionEncoder ...

2018-10-24 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22749#discussion_r227745900 --- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/ScalaReflectionSuite.scala --- @@ -351,11 +347,15 @@ class ScalaReflectionSuite extends

[GitHub] spark pull request #22749: [SPARK-25746][SQL] Refactoring ExpressionEncoder ...

2018-10-24 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22749#discussion_r227695574 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/ScalaReflection.scala --- @@ -434,17 +426,34 @@ object ScalaReflection extends

[GitHub] spark pull request #22749: [SPARK-25746][SQL] Refactoring ExpressionEncoder ...

2018-10-24 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22749#discussion_r227678714 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/encoders/RowEncoder.scala --- @@ -58,12 +58,10 @@ object RowEncoder { def apply

[GitHub] spark pull request #22749: [SPARK-25746][SQL] Refactoring ExpressionEncoder ...

2018-10-24 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22749#discussion_r227677176 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/encoders/RowEncoder.scala --- @@ -58,12 +58,10 @@ object RowEncoder { def apply

<    1   2   3   4   5   6   7   8   9   10   >