[GitHub] spark issue #22569: [SPARK-25542][SQL][Test] Move flaky test in OpenHashMapS...

2018-09-27 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/22569 retest this please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #22569: [SPARK-25542][SQL][Test] Move flaky test in OpenHashMapS...

2018-09-27 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/22569 cc @dongjoon-hyun @cloud-fan --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark pull request #22569: [SPARK-25542][SQL][Test] Move flaky test in OpenH...

2018-09-27 Thread viirya
GitHub user viirya opened a pull request: https://github.com/apache/spark/pull/22569 [SPARK-25542][SQL][Test] Move flaky test in OpenHashMapSuite to OpenHashSetSuite ## What changes were proposed in this pull request? The specified test in OpenHashMapSuite to test large

[GitHub] spark issue #22524: [SPARK-25497][SQL] Limit operation within whole stage co...

2018-09-27 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/22524 retest this please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #22556: [MINOR] Remove useless InSubquery expression

2018-09-27 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/22556 LGTM --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark pull request #22544: [SPARK-25522][SQL] Improve type promotion for inp...

2018-09-26 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22544#discussion_r220765052 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala --- @@ -974,6 +974,25 @@ object TypeCoercion

[GitHub] spark issue #22514: [SPARK-25271][SQL] Hive ctas commands should use data so...

2018-09-26 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/22514 retest this please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #22514: [SPARK-25271][SQL] Hive ctas commands should use data so...

2018-09-26 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/22514 retest this please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark pull request #22544: [SPARK-25522][SQL] Improve type promotion for inp...

2018-09-26 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22544#discussion_r220460094 --- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercionSuite.scala --- @@ -257,12 +257,48 @@ class TypeCoercionSuite

[GitHub] spark pull request #22544: [SPARK-25522][SQL] Improve type promotion for inp...

2018-09-26 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22544#discussion_r220459877 --- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercionSuite.scala --- @@ -257,12 +257,48 @@ class TypeCoercionSuite

[GitHub] spark issue #22514: [SPARK-25271][SQL] Hive ctas commands should use data so...

2018-09-26 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/22514 retest this please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #22544: [SPARK-25522][SQL] Improve type promotion for input argu...

2018-09-26 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/22544 retest this please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #22514: [SPARK-25271][SQL] Hive ctas commands should use data so...

2018-09-25 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/22514 retest this please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #22524: [SPARK-25497][SQL] Limit operation within whole stage co...

2018-09-25 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/22524 Oh, I see. May the code template make some confusion. I'd change it. The downstream code is wrapped inside an if block. I don't clearly show how downstream operators work in codegen. Let me update

[GitHub] spark pull request #22429: [SPARK-25440][SQL] Dumping query execution info t...

2018-09-25 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22429#discussion_r220409552 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/package.scala --- @@ -167,6 +170,56 @@ package object util

[GitHub] spark issue #22524: [SPARK-25497][SQL] Limit operation within whole stage co...

2018-09-25 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/22524 hmm, in above suggested way, isn't `setStopEarly(true)` called when consuming `given_limit` + 1 records? Otherwise I may misunderstand

[GitHub] spark pull request #22544: [SPARK-25522][SQL] Improve type promotion for inp...

2018-09-25 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22544#discussion_r220244473 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala --- @@ -971,9 +971,36 @@ object TypeCoercion

[GitHub] spark pull request #22544: [SPARK-25522][SQL] Improve type promotion for inp...

2018-09-25 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22544#discussion_r220241139 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -2140,21 +2140,34 @@ case class ElementAt

[GitHub] spark issue #22524: [SPARK-25497][SQL] Limit operation within whole stage co...

2018-09-25 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/22524 Doesn't this way consume one more record than given limit number? --- - To unsubscribe, e-mail: reviews-unsubscr

[GitHub] spark pull request #22429: [SPARK-25440][SQL] Dumping query execution info t...

2018-09-25 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22429#discussion_r220195279 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/package.scala --- @@ -167,6 +170,56 @@ package object util

[GitHub] spark pull request #22544: [SPARK-25522][SQL] Improve type promotion for inp...

2018-09-25 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22544#discussion_r220193922 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -2140,21 +2140,34 @@ case class ElementAt

[GitHub] spark pull request #22544: [SPARK-25522][SQL] Improve type promotion for inp...

2018-09-25 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22544#discussion_r220140660 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala --- @@ -971,9 +971,36 @@ object TypeCoercion

[GitHub] spark pull request #22544: [SPARK-25522][SQL] Improve type promotion for inp...

2018-09-25 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22544#discussion_r220140252 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala --- @@ -971,9 +971,36 @@ object TypeCoercion

[GitHub] spark pull request #22544: [SPARK-25522][SQL] Improve type promotion for inp...

2018-09-25 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22544#discussion_r220147139 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala --- @@ -971,9 +971,36 @@ object TypeCoercion

[GitHub] spark issue #22544: [SPARK-25522][SQL] Improve type promotion for input argu...

2018-09-25 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/22544 @dilipbiswal The two examples shown in the PR description are allowed in current and this proposes to disallow them? I'm a bit confused by the description

[GitHub] spark pull request #22524: [SPARK-25497][SQL] Limit operation within whole s...

2018-09-24 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22524#discussion_r220054697 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/limit.scala --- @@ -71,22 +71,14 @@ trait BaseLimitExec extends UnaryExecNode

[GitHub] spark issue #22524: [SPARK-25497][SQL] Limit operation within whole stage co...

2018-09-24 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/22524 > It will be great to explain how limit works in whole stage codegen, in general. This part is a little hard to understand and I believe many operators need to deal with limit as well.

[GitHub] spark pull request #22524: [SPARK-25497][SQL] Limit operation within whole s...

2018-09-24 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22524#discussion_r220048264 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/limit.scala --- @@ -71,22 +71,14 @@ trait BaseLimitExec extends UnaryExecNode

[GitHub] spark pull request #22524: [SPARK-25497][SQL] Limit operation within whole s...

2018-09-24 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22524#discussion_r220046213 --- Diff: sql/core/src/main/java/org/apache/spark/sql/execution/BufferedRowIterator.java --- @@ -73,14 +78,21 @@ public void append(InternalRow row

[GitHub] spark pull request #22524: [SPARK-25497][SQL] Limit operation within whole s...

2018-09-24 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22524#discussion_r220046092 --- Diff: sql/core/src/main/java/org/apache/spark/sql/execution/BufferedRowIterator.java --- @@ -73,14 +78,21 @@ public void append(InternalRow row

[GitHub] spark pull request #22524: [SPARK-25497][SQL] Limit operation within whole s...

2018-09-24 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22524#discussion_r220044740 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/limit.scala --- @@ -71,22 +71,14 @@ trait BaseLimitExec extends UnaryExecNode

[GitHub] spark pull request #22524: [SPARK-25497][SQL] Limit operation within whole s...

2018-09-24 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22524#discussion_r220044584 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/basicPhysicalOperators.scala --- @@ -465,13 +465,18 @@ case class RangeExec(range

[GitHub] spark pull request #22524: [SPARK-25497][SQL] Limit operation within whole s...

2018-09-24 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22524#discussion_r220040370 --- Diff: sql/core/src/main/java/org/apache/spark/sql/execution/BufferedRowIterator.java --- @@ -38,6 +38,11 @@ protected int partitionIndex

[GitHub] spark issue #22535: [SPARK-17636][SQL][WIP] Parquet predicate pushdown in ne...

2018-09-24 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/22535 retest this please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #22524: [SPARK-25497][SQL] Limit operation within whole stage co...

2018-09-24 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/22524 cc @cloud-fan --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #22524: [WIP][SPARK-25497][SQL] Limit operation within whole sta...

2018-09-24 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/22524 retest this please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark pull request #22524: [WIP][SPARK-25497][SQL] Limit operation within wh...

2018-09-23 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22524#discussion_r219731695 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/DataFrameAggregateSuite.scala --- @@ -556,7 +556,7 @@ class DataFrameAggregateSuite extends QueryTest

[GitHub] spark issue #22524: [WIP][SPARK-25497][SQL] Limit operation within whole sta...

2018-09-23 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/22524 @xuanyuanking Thanks. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews

[GitHub] spark issue #22524: [WIP][SPARK-25497][SQL] Limit operation within whole sta...

2018-09-23 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/22524 > Does this means you got a Reason: Error during SSL Handshake with remote server after open the jenkins link?

[GitHub] spark pull request #22227: [SPARK-25202] [SQL] Implements split with limit s...

2018-09-23 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/7#discussion_r219690825 --- Diff: R/pkg/R/functions.R --- @@ -3404,19 +3404,27 @@ setMethod("collect_set", #' Equivalent to \code{split} SQL function. #'

[GitHub] spark pull request #22227: [SPARK-25202] [SQL] Implements split with limit s...

2018-09-23 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/7#discussion_r219691017 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/functions.scala --- @@ -2546,15 +2546,39 @@ object functions { def soundex(e: Column): Column

[GitHub] spark pull request #22227: [SPARK-25202] [SQL] Implements split with limit s...

2018-09-23 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/7#discussion_r219690753 --- Diff: R/pkg/R/functions.R --- @@ -3404,19 +3404,27 @@ setMethod("collect_set", #' Equivalent to \code{split} SQL function. #'

[GitHub] spark issue #22524: [WIP][SPARK-25497][SQL] Limit operation within whole sta...

2018-09-23 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/22524 Is jenkins down now? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews

[GitHub] spark pull request #22524: [WIP][SPARK-25497][SQL] Limit operation within wh...

2018-09-23 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22524#discussion_r219690438 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/limit.scala --- @@ -84,9 +84,10 @@ trait BaseLimitExec extends UnaryExecNode

[GitHub] spark pull request #22524: [WIP][SPARK-25497][SQL] Limit operation within wh...

2018-09-23 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22524#discussion_r219690413 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/basicPhysicalOperators.scala --- @@ -465,13 +465,18 @@ case class RangeExec(range

[GitHub] spark pull request #22524: [SPARK-25497][SQL] Limit operation within whole s...

2018-09-22 Thread viirya
GitHub user viirya opened a pull request: https://github.com/apache/spark/pull/22524 [SPARK-25497][SQL] Limit operation within whole stage codegen should not consume all the inputs ## What changes were proposed in this pull request? This issue was discovered during https

[GitHub] spark issue #22514: [SPARK-25271][SQL] Hive ctas commands should use data so...

2018-09-21 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/22514 cc @cloud-fan @HyukjinKwon --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark pull request #22455: [SPARK-24572][SPARKR] "eager execution" for R she...

2018-09-21 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22455#discussion_r219410274 --- Diff: R/pkg/R/DataFrame.R --- @@ -226,7 +226,8 @@ setMethod("showDF", #' show #' -#' Print class and type information

[GitHub] spark issue #22455: [SPARK-24572][SPARKR] "eager execution" for R shell, IDE

2018-09-21 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/22455 Let's also update the doc of `REPL_EAGER_EVAL_ENABLED` in `SQLConf`. After this patch, eager evaluation is not only supported in PySpark

[GitHub] spark pull request #22455: [SPARK-24572][SPARKR] "eager execution" for R she...

2018-09-21 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22455#discussion_r219404319 --- Diff: R/pkg/R/DataFrame.R --- @@ -244,11 +245,15 @@ setMethod("showDF", #' @note show(SparkDataFrame) since 1.4.0 setMe

[GitHub] spark issue #22514: [SPARK-25271][SQL] Hive ctas commands should use data so...

2018-09-21 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/22514 retest this please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark pull request #22514: [SPARK-25271][SQL] Hive ctas commands should use ...

2018-09-21 Thread viirya
GitHub user viirya opened a pull request: https://github.com/apache/spark/pull/22514 [SPARK-25271][SQL] Hive ctas commands should use data source if it is convertible ## What changes were proposed in this pull request? We have a [regression](https://github.com/apache

[GitHub] spark pull request #22455: [SPARK-24572][SPARKR] "eager execution" for R she...

2018-09-20 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22455#discussion_r219351707 --- Diff: R/pkg/R/DataFrame.R --- @@ -244,11 +245,15 @@ setMethod("showDF", #' @note show(SparkDataFrame) since 1.4.0 setMe

[GitHub] spark pull request #22455: [SPARK-24572][SPARKR] "eager execution" for R she...

2018-09-20 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22455#discussion_r219348795 --- Diff: R/pkg/R/DataFrame.R --- @@ -244,11 +245,15 @@ setMethod("showDF", #' @note show(SparkDataFrame) since 1.4.0 setMe

[GitHub] spark pull request #22455: [SPARK-24572][SPARKR] "eager execution" for R she...

2018-09-20 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22455#discussion_r219347786 --- Diff: R/pkg/R/DataFrame.R --- @@ -244,11 +245,15 @@ setMethod("showDF", #' @note show(SparkDataFrame) since 1.4.0 setMe

[GitHub] spark issue #22479: [MINOR][PYTHON][TEST] Use collect() instead of show() to...

2018-09-20 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/22479 Thanks @HyukjinKwon. LGTM --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark pull request #22479: [MINOR][PYTHON][TEST] Use collect() instead of sh...

2018-09-20 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22479#discussion_r219053623 --- Diff: python/pyspark/sql/tests.py --- @@ -1168,7 +1168,7 @@ def test_simple_udt_in_df(self): df = self.spark.createDataFrame

[GitHub] spark pull request #22464: Revert [SPARK-19355][SPARK-25352]

2018-09-19 Thread viirya
Github user viirya closed the pull request at: https://github.com/apache/spark/pull/22464 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #22481: Revert [SPARK-19355][SPARK-25352]

2018-09-19 Thread viirya
GitHub user viirya opened a pull request: https://github.com/apache/spark/pull/22481 Revert [SPARK-19355][SPARK-25352] ## What changes were proposed in this pull request? This goes to revert sequential PRs based on some discussion and comments at https://github.com/apache

[GitHub] spark issue #22481: Revert [SPARK-19355][SPARK-25352]

2018-09-19 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/22481 cc @cloud-fan --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #22464: Revert [SPARK-19355][SPARK-25352]

2018-09-19 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/22464 @cloud-fan Shall we have 4 PRs to revert these PRs individually? --- - To unsubscribe, e-mail: reviews-unsubscr

[GitHub] spark issue #22464: Revert [SPARK-19355][SPARK-25352]

2018-09-19 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/22464 @cloud-fan Ok. Let me try it. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark pull request #22465: [SPARK-25457][SQL] IntegralDivide returns data ty...

2018-09-19 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22465#discussion_r218998728 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala --- @@ -1561,6 +1561,13 @@ object SQLConf { "are perf

[GitHub] spark issue #22464: Revert [SPARK-19355][SPARK-25352]

2018-09-19 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/22464 Thanks @HyukjinKwon @maropu. I've put it in the PR description. --- - To unsubscribe, e-mail: reviews-unsubscr

[GitHub] spark issue #22464: Revert [SPARK-19355][SPARK-25352]

2018-09-19 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/22464 cc @cloud-fan @rxin @hvanhovell --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e

[GitHub] spark pull request #22464: Revert [SPARK-19355][SPARK-25352]

2018-09-19 Thread viirya
GitHub user viirya opened a pull request: https://github.com/apache/spark/pull/22464 Revert [SPARK-19355][SPARK-25352] ## What changes were proposed in this pull request? This goes to revert [SPARK-19355](https://github.com/apache/spark/pull/16677) and [SPARK-25352](https

[GitHub] spark issue #16677: [SPARK-19355][SQL] Use map output statistics to improve ...

2018-09-19 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/16677 I understood the two major concerns regarding this change. I'm going to submit a pr to revert the change. I will look into this idea further with new design

[GitHub] spark pull request #22456: [SPARK-19355][SQL] Fix variable names numberOfOut...

2018-09-19 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22456#discussion_r218685917 --- Diff: core/src/main/scala/org/apache/spark/scheduler/MapStatus.scala --- @@ -31,7 +31,7 @@ import org.apache.spark.util.Utils

[GitHub] spark pull request #16677: [SPARK-19355][SQL] Use map output statistics to i...

2018-09-18 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/16677#discussion_r218652707 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/limit.scala --- @@ -93,25 +96,93 @@ trait BaseLimitExec extends UnaryExecNode

[GitHub] spark pull request #16677: [SPARK-19355][SQL] Use map output statistics to i...

2018-09-18 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/16677#discussion_r218639550 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/limit.scala --- @@ -93,25 +96,93 @@ trait BaseLimitExec extends UnaryExecNode

[GitHub] spark pull request #16677: [SPARK-19355][SQL] Use map output statistics to i...

2018-09-18 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/16677#discussion_r218639483 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/limit.scala --- @@ -93,25 +96,93 @@ trait BaseLimitExec extends UnaryExecNode

[GitHub] spark issue #16677: [SPARK-19355][SQL] Use map output statistics to improve ...

2018-09-18 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/16677 I'm not sure where it can cause perf regressions. Basically this just changes the way we retrieve records from partitions when performing limit. This doesn't do shuffling them together to single

[GitHub] spark pull request #22344: [SPARK-25352][SQL] Perform ordered global limit w...

2018-09-18 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22344#discussion_r218638143 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/limit.scala --- @@ -98,7 +98,8 @@ case class LocalLimitExec(limit: Int, child: SparkPlan

[GitHub] spark pull request #22344: [SPARK-25352][SQL] Perform ordered global limit w...

2018-09-18 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22344#discussion_r218631953 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/limit.scala --- @@ -98,7 +98,8 @@ case class LocalLimitExec(limit: Int, child: SparkPlan

[GitHub] spark issue #16677: [SPARK-19355][SQL] Use map output statistics to improve ...

2018-09-18 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/16677 @rxin Thanks for the comment. I will improve the document in a pr. --- - To unsubscribe, e-mail: reviews-unsubscr

[GitHub] spark pull request #22344: [SPARK-25352][SQL] Perform ordered global limit w...

2018-09-18 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22344#discussion_r218631052 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/limit.scala --- @@ -98,7 +98,8 @@ case class LocalLimitExec(limit: Int, child: SparkPlan

[GitHub] spark pull request #22344: [SPARK-25352][SQL] Perform ordered global limit w...

2018-09-18 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22344#discussion_r218630241 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/limit.scala --- @@ -98,7 +98,8 @@ case class LocalLimitExec(limit: Int, child: SparkPlan

[GitHub] spark issue #22344: [SPARK-25352][SQL] Perform ordered global limit when lim...

2018-09-18 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/22344 Thanks @rxin. I will submit a pr to improve the readability. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org

[GitHub] spark pull request #22344: [SPARK-25352][SQL] Perform ordered global limit w...

2018-09-18 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22344#discussion_r218627396 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/limit.scala --- @@ -98,7 +98,8 @@ case class LocalLimitExec(limit: Int, child: SparkPlan

[GitHub] spark pull request #22440: [SPARK-24151][SQL] Case insensitive resolution of...

2018-09-17 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22440#discussion_r218021130 --- Diff: docs/sql-programming-guide.md --- @@ -1879,6 +1879,7 @@ working with timestamps in `pandas_udf`s to get the best performance, see

[GitHub] spark issue #21217: [SPARK-24151][SQL] Fix CURRENT_DATE, CURRENT_TIMESTAMP t...

2018-09-17 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/21217 @mgaido91 No problem. Please submit the PR. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21217: [SPARK-24151][SQL] Fix CURRENT_DATE, CURRENT_TIMESTAMP t...

2018-09-17 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/21217 Thanks @jamesthomp for you work. When we pick this up, I think we can still give the credit of the work to you

[GitHub] spark pull request #22237: [SPARK-25243][SQL] Use FailureSafeParser in from_...

2018-09-17 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22237#discussion_r217995920 --- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/JsonExpressionsSuite.scala --- @@ -402,13 +402,13 @@ class

[GitHub] spark pull request #22237: [SPARK-25243][SQL] Use FailureSafeParser in from_...

2018-09-17 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22237#discussion_r218000227 --- Diff: docs/sql-programming-guide.md --- @@ -1877,6 +1877,10 @@ working with timestamps in `pandas_udf`s to get the best performance, see

[GitHub] spark pull request #22237: [SPARK-25243][SQL] Use FailureSafeParser in from_...

2018-09-17 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22237#discussion_r218003046 --- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/JsonExpressionsSuite.scala --- @@ -450,7 +450,7 @@ class

[GitHub] spark pull request #22237: [SPARK-25243][SQL] Use FailureSafeParser in from_...

2018-09-17 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22237#discussion_r218000572 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala --- @@ -595,10 +607,7 @@ case class JsonToStructs

[GitHub] spark pull request #22395: [SPARK-16323][SQL] Add IntegralDivide expression

2018-09-17 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22395#discussion_r217981686 --- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/ArithmeticExpressionSuite.scala --- @@ -143,16 +143,14 @@ class

[GitHub] spark pull request #22429: [SPARK-25440][SQL] Dumping query execution info t...

2018-09-16 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22429#discussion_r217915212 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/QueryExecutionSuite.scala --- @@ -16,11 +16,33 @@ */ package

[GitHub] spark issue #22417: [SPARK-25426][SQL] Remove the duplicate fallback logic i...

2018-09-14 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/22417 LGTM too. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #22395: [SPARK-16323][SQL] Add IntegralDivide expression

2018-09-14 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/22395 LGTM --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark pull request #22395: [SPARK-16323][SQL] Add IntegralDivide expression

2018-09-14 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22395#discussion_r217747363 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/arithmetic.scala --- @@ -314,6 +314,32 @@ case class Divide(left

[GitHub] spark pull request #22395: [SPARK-16323][SQL] Add IntegralDivide expression

2018-09-14 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22395#discussion_r217631634 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/arithmetic.scala --- @@ -314,6 +314,27 @@ case class Divide(left

[GitHub] spark pull request #22417: [SPARK-25426][SQL] Handles subexpression eliminat...

2018-09-14 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22417#discussion_r217619116 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/CodeGeneratorWithInterpretedFallback.scala --- @@ -59,6 +59,6 @@ abstract

[GitHub] spark issue #21217: [SPARK-24151][SQL] Fix CURRENT_DATE, CURRENT_TIMESTAMP t...

2018-09-13 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/21217 @HyukjinKwon thanks for pinging me. I'd wait for others to take over this first, if no one does, I can do it later

[GitHub] spark pull request #22375: [SPARK-25388][Test][SQL] Detect incorrect nullabl...

2018-09-13 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22375#discussion_r217433074 --- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/ExpressionEvalHelper.scala --- @@ -223,8 +223,8 @@ trait

[GitHub] spark issue #22409: [SPARK-25352][SQL][Followup] Add helper method and addre...

2018-09-13 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/22409 retest this please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #22409: [SPARK-25352][SQL][Followup] Add helper method and addre...

2018-09-13 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/22409 retest this please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #22409: [SPARK-25352][SQL][Followup] Add helper method and addre...

2018-09-12 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/22409 cc @hvanhovell @dbtsai @cloud-fan --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e

[GitHub] spark pull request #22409: [SPARK-25352][SQL][Followup] Add helper method an...

2018-09-12 Thread viirya
GitHub user viirya opened a pull request: https://github.com/apache/spark/pull/22409 [SPARK-25352][SQL][Followup] Add helper method and address style issue ## What changes were proposed in this pull request? This patch address [the review comment](https://github.com/apache

[GitHub] spark pull request #20521: [SPARK-22977][SQL] fix web UI SQL tab for CTAS

2018-09-12 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/20521#discussion_r217255531 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/CreateHiveTableAsSelectCommand.scala --- @@ -56,34 +57,36 @@ case class

<    1   2   3   4   5   6   7   8   9   10   >