[GitHub] spark issue #20937: [SPARK-23094][SPARK-23723][SPARK-23724][SQL] Support cus...

2018-04-21 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/20937 I think we are almost there - sorry for late response. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For

[GitHub] spark pull request #20937: [SPARK-23094][SPARK-23723][SPARK-23724][SQL] Supp...

2018-04-21 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/20937#discussion_r183227330 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/json/JsonBenchmarks.scala --- @@ -0,0 +1,179 @@ +/* + * Licensed t

[GitHub] spark pull request #20937: [SPARK-23094][SPARK-23723][SPARK-23724][SQL] Supp...

2018-04-21 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/20937#discussion_r183227312 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/text/TextOptions.scala --- @@ -41,19 +41,25 @@ private[text] class TextOpt

[GitHub] spark pull request #20937: [SPARK-23094][SPARK-23723][SPARK-23724][SQL] Supp...

2018-04-21 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/20937#discussion_r183227276 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/json/JsonDataSource.scala --- @@ -175,11 +187,15 @@ object MultiLineJsonDa

[GitHub] spark pull request #21056: [SPARK-23849][SQL] Tests for samplingRatio of jso...

2018-04-21 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/21056#discussion_r183227100 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/json/JsonSuite.scala --- @@ -2128,38 +2128,77 @@ class JsonSuite extends Q

[GitHub] spark pull request #21056: [SPARK-23849][SQL] Tests for samplingRatio of jso...

2018-04-21 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/21056#discussion_r183227071 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/json/JsonSuite.scala --- @@ -2128,38 +2128,77 @@ class JsonSuite extends Q

[GitHub] spark pull request #21052: [SPARK-23799][SQL] FilterEstimation.evaluateInSet...

2018-04-21 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/21052 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21031: [SPARK-23923][SQL] Add cardinality function

2018-04-21 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21031 @kiszk Could you also update the PR description? LGTM --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For a

[GitHub] spark issue #21052: [SPARK-23799][SQL] FilterEstimation.evaluateInSet produc...

2018-04-21 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21052 Thanks! Merged to master/2.3 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mai

[GitHub] spark pull request #20959: [SPARK-23846][SQL] The samplingRatio option for C...

2018-04-21 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/20959#discussion_r183226807 --- Diff: python/pyspark/sql/readwriter.py --- @@ -882,6 +882,9 @@ def csv(self, path, mode=None, compression=None, sep=None, quote=None, escape=No

[GitHub] spark pull request #21107: [DO-NOT-MERGE][WIP] Explicitly print out skipped ...

2018-04-21 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/21107#discussion_r183226283 --- Diff: python/run-tests.py --- @@ -152,65 +172,17 @@ def parse_opts(): return opts -def _check_dependencies(python_exec, modu

[GitHub] spark issue #21107: [DO-NOT-MERGE][WIP] Explicitly print out skipped tests f...

2018-04-21 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/21107 @BryanCutler, will check and update after testing out. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For

[GitHub] spark issue #20280: [SPARK-22232][PYTHON][SQL] Fixed Row pickling to include...

2018-04-21 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/20280 @BryanCutler, mind if I ask to clarify what happens for end-to-end cases in the PR description (like before & after with explaining the reasons)? the change looks small but possibly a breaking c

[GitHub] spark issue #21120: [SPARK-22448][ML] Added sum function to Summerizer and M...

2018-04-21 Thread dedunumax
Github user dedunumax commented on the issue: https://github.com/apache/spark/pull/21120 cc @rxin @cloud-fan @gatorsmile --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-

[GitHub] spark issue #20280: [SPARK-22232][PYTHON][SQL] Fixed Row pickling to include...

2018-04-21 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/20280 BTW, I believe it's not so easy to pass a configuration from a very quick look because the exception usually would be thrown in a Python worker process. --- ---

[GitHub] spark issue #20280: [SPARK-22232][PYTHON][SQL] Fixed Row pickling to include...

2018-04-21 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/20280 If the renaming scenario works in most of cases as expected, I think it'd be worthwhile to have a configuration; however, the previous behaviour looks actually odd because it's going to work onl

[GitHub] spark issue #20930: [SPARK-23811][Core] FetchFailed comes before Success of ...

2018-04-21 Thread Ngone51
Github user Ngone51 commented on the issue: https://github.com/apache/spark/pull/20930 > because we can get the MapStatus, but get a 'null'. If I'm not mistaken, this also because the ExecutorLost trigger removeOutputsOnExecutor If there's a `null` MapStatus for stage 2, how c

[GitHub] spark pull request #21116: [SPARK-24038][SS] Refactor continuous writing to ...

2018-04-21 Thread xuanyuanking
Github user xuanyuanking commented on a diff in the pull request: https://github.com/apache/spark/pull/21116#discussion_r183224838 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/continuous/WriteToContinuousDataSourceExec.scala --- @@ -0,0 +1,126 @@

[GitHub] spark issue #20280: [SPARK-22232][PYTHON][SQL] Fixed Row pickling to include...

2018-04-21 Thread felixcheung
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/20280 I'm kinda worry the example you give above is actually fairly common - construct with kwargs, and then (re-)name the columns. perhaps worthwhile to consider a config switch? ---

[GitHub] spark issue #21071: [SPARK-21962][CORE] Distributed Tracing in Spark

2018-04-21 Thread felixcheung
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/21071 yap... HTrace is [retired](http://mail-archives.apache.org/mod_mbox/htrace-dev/201804.mbox/%3Cpony-b7497055821405926d63668ab1112e0f108e2346-2561e81afc434e2d237bbeb5b5921941503445e4%40dev.htrace.a

[GitHub] spark issue #20940: [SPARK-23429][CORE] Add executor memory metrics to heart...

2018-04-21 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20940 **[Test build #89685 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89685/testReport)** for PR 20940 at commit [`ae8a388`](https://github.com/apache/spark/commit/ae

[GitHub] spark issue #20930: [SPARK-23811][Core] FetchFailed comes before Success of ...

2018-04-21 Thread xuanyuanking
Github user xuanyuanking commented on the issue: https://github.com/apache/spark/pull/20930 ![image](https://user-images.githubusercontent.com/4833765/39091106-ff11d0a6-461f-11e8-968f-7fcbe6652bb3.png) Stage 0\1\2\3 same with 20\21\22\23 in this screenshot, stage2's shuffleId

[GitHub] spark issue #21082: [SPARK-22239][SQL][Python] Enable grouped aggregate pand...

2018-04-21 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21082 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89682/ Test FAILed. ---

[GitHub] spark issue #21052: [SPARK-23799][SQL] FilterEstimation.evaluateInSet produc...

2018-04-21 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21052 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89684/ Test PASSed. ---

[GitHub] spark issue #21052: [SPARK-23799][SQL] FilterEstimation.evaluateInSet produc...

2018-04-21 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21052 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21082: [SPARK-22239][SQL][Python] Enable grouped aggregate pand...

2018-04-21 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21082 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21082: [SPARK-22239][SQL][Python] Enable grouped aggregate pand...

2018-04-21 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21082 **[Test build #89682 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89682/testReport)** for PR 21082 at commit [`657a6a5`](https://github.com/apache/spark/commit/6

[GitHub] spark issue #21052: [SPARK-23799][SQL] FilterEstimation.evaluateInSet produc...

2018-04-21 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21052 **[Test build #89684 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89684/testReport)** for PR 21052 at commit [`8369cbc`](https://github.com/apache/spark/commit/8

[GitHub] spark issue #21122: [SPARK-24017] [SQL] Refactor ExternalCatalog to be an in...

2018-04-21 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21122 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89681/ Test PASSed. ---

[GitHub] spark issue #21122: [SPARK-24017] [SQL] Refactor ExternalCatalog to be an in...

2018-04-21 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21122 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21122: [SPARK-24017] [SQL] Refactor ExternalCatalog to be an in...

2018-04-21 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21122 **[Test build #89681 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89681/testReport)** for PR 21122 at commit [`c62bba1`](https://github.com/apache/spark/commit/c

[GitHub] spark pull request #20787: [MINOR][DOCS] Documenting months_between directio...

2018-04-21 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/20787#discussion_r183221673 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala --- @@ -1117,11 +1117,21 @@ case class AddMonths(s

[GitHub] spark issue #21121: [SPARK-24042][SQL] Collection function: zip_with_index

2018-04-21 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21121 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21121: [SPARK-24042][SQL] Collection function: zip_with_index

2018-04-21 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21121 **[Test build #89683 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89683/testReport)** for PR 21121 at commit [`a599544`](https://github.com/apache/spark/commit/a

[GitHub] spark issue #21121: [SPARK-24042][SQL] Collection function: zip_with_index

2018-04-21 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21121 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89683/ Test FAILed. ---

[GitHub] spark issue #21052: [SPARK-23799][SQL] FilterEstimation.evaluateInSet produc...

2018-04-21 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21052 **[Test build #89684 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89684/testReport)** for PR 21052 at commit [`8369cbc`](https://github.com/apache/spark/commit/83

[GitHub] spark issue #21121: [SPARK-24042][SQL] Collection function: zip_with_index

2018-04-21 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21121 **[Test build #89683 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89683/testReport)** for PR 21121 at commit [`a599544`](https://github.com/apache/spark/commit/a5

[GitHub] spark pull request #21121: [SPARK-24042][SQL] Collection function: zip_with_...

2018-04-21 Thread mn-mikke
Github user mn-mikke commented on a diff in the pull request: https://github.com/apache/spark/pull/21121#discussion_r183220685 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -883,3 +884,139 @@ case class Concat(chi

[GitHub] spark issue #21052: [SPARK-23799][SQL] FilterEstimation.evaluateInSet produc...

2018-04-21 Thread mshtelma
Github user mshtelma commented on the issue: https://github.com/apache/spark/pull/21052 @gatorsmile I have removed explain() and changed formatting --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org F

[GitHub] spark pull request #21052: [SPARK-23799][SQL] FilterEstimation.evaluateInSet...

2018-04-21 Thread mshtelma
Github user mshtelma commented on a diff in the pull request: https://github.com/apache/spark/pull/21052#discussion_r183220650 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/StatisticsCollectionSuite.scala --- @@ -382,4 +382,32 @@ class StatisticsCollectionSuite extends St

[GitHub] spark pull request #21052: [SPARK-23799][SQL] FilterEstimation.evaluateInSet...

2018-04-21 Thread mshtelma
Github user mshtelma commented on a diff in the pull request: https://github.com/apache/spark/pull/21052#discussion_r183220647 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/StatisticsCollectionSuite.scala --- @@ -382,4 +382,32 @@ class StatisticsCollectionSuite extends St

[GitHub] spark issue #21082: [SPARK-22239][SQL][Python] Enable grouped aggregate pand...

2018-04-21 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21082 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21082: [SPARK-22239][SQL][Python] Enable grouped aggregate pand...

2018-04-21 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21082 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2563/ Tes

[GitHub] spark issue #21082: [SPARK-22239][SQL][Python] Enable grouped aggregate pand...

2018-04-21 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21082 **[Test build #89682 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89682/testReport)** for PR 21082 at commit [`657a6a5`](https://github.com/apache/spark/commit/65

[GitHub] spark pull request #21082: [SPARK-22239][SQL][Python] Enable grouped aggrega...

2018-04-21 Thread icexelloss
Github user icexelloss commented on a diff in the pull request: https://github.com/apache/spark/pull/21082#discussion_r183220435 --- Diff: python/pyspark/sql/tests.py --- @@ -5156,6 +5156,15 @@ def test_retain_group_columns(self): expected1 = df.groupby(df.id).agg(s

[GitHub] spark pull request #21082: [SPARK-22239][SQL][Python] Enable grouped aggrega...

2018-04-21 Thread icexelloss
Github user icexelloss commented on a diff in the pull request: https://github.com/apache/spark/pull/21082#discussion_r183220392 --- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/AnalysisErrorSuite.scala --- @@ -149,7 +149,7 @@ class AnalysisErrorSuite e

[GitHub] spark issue #21052: [SPARK-23799][SQL] FilterEstimation.evaluateInSet produc...

2018-04-21 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21052 LGTM except two minor comments. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-

[GitHub] spark pull request #21052: [SPARK-23799][SQL] FilterEstimation.evaluateInSet...

2018-04-21 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21052#discussion_r183219812 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/StatisticsCollectionSuite.scala --- @@ -382,4 +382,32 @@ class StatisticsCollectionSuite extends

[GitHub] spark pull request #21052: [SPARK-23799][SQL] FilterEstimation.evaluateInSet...

2018-04-21 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21052#discussion_r183219803 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/StatisticsCollectionSuite.scala --- @@ -382,4 +382,32 @@ class StatisticsCollectionSuite extends

[GitHub] spark issue #21122: [SPARK-24017] [SQL] Refactor ExternalCatalog to be an in...

2018-04-21 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21122 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2562/ Tes

[GitHub] spark issue #21122: [SPARK-24017] [SQL] Refactor ExternalCatalog to be an in...

2018-04-21 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21122 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21122: [SPARK-24017] [SQL] Refactor ExternalCatalog to be an in...

2018-04-21 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21122 **[Test build #89681 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89681/testReport)** for PR 21122 at commit [`c62bba1`](https://github.com/apache/spark/commit/c6

[GitHub] spark issue #21122: [SPARK-24017] [SQL] Refactor ExternalCatalog to be an in...

2018-04-21 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21122 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews

[GitHub] spark issue #21122: [SPARK-24017] [SQL] Refactor ExternalCatalog to be an in...

2018-04-21 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21122 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21122: [SPARK-24017] [SQL] Refactor ExternalCatalog to be an in...

2018-04-21 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21122 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89680/ Test FAILed. ---

[GitHub] spark issue #21122: [SPARK-24017] [SQL] Refactor ExternalCatalog to be an in...

2018-04-21 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21122 **[Test build #89680 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89680/testReport)** for PR 21122 at commit [`c62bba1`](https://github.com/apache/spark/commit/c

[GitHub] spark issue #12154: [SPARK-12133][STREAMING] Streaming dynamic allocation

2018-04-21 Thread sugix
Github user sugix commented on the issue: https://github.com/apache/spark/pull/12154 @tdas - Why we cannot see this in the documentation and I am not sure if AWS EMR supports this feature? --- - To unsubscribe, e-m

[GitHub] spark issue #21121: [SPARK-24042][SQL] Collection function: zip_with_index

2018-04-21 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21121 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21121: [SPARK-24042][SQL] Collection function: zip_with_index

2018-04-21 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21121 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89679/ Test FAILed. ---

[GitHub] spark issue #21121: [SPARK-24042][SQL] Collection function: zip_with_index

2018-04-21 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21121 **[Test build #89679 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89679/testReport)** for PR 21121 at commit [`551d04d`](https://github.com/apache/spark/commit/5

[GitHub] spark issue #21121: [SPARK-24042][SQL] Collection function: zip_with_index

2018-04-21 Thread mn-mikke
Github user mn-mikke commented on the issue: https://github.com/apache/spark/pull/21121 @gatorsmile I'm not aware of any. From user experience, I strongly feel that such a function is missing. Escpecially, when [transform](https://issues.apache.org/jira/browse/SPARK-23908) function is

[GitHub] spark issue #21056: [SPARK-23849][SQL] Tests for samplingRatio of json datas...

2018-04-21 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21056 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21056: [SPARK-23849][SQL] Tests for samplingRatio of json datas...

2018-04-21 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21056 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89678/ Test PASSed. ---

[GitHub] spark issue #21056: [SPARK-23849][SQL] Tests for samplingRatio of json datas...

2018-04-21 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21056 **[Test build #89678 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89678/testReport)** for PR 21056 at commit [`fdeac84`](https://github.com/apache/spark/commit/f

[GitHub] spark issue #21122: [SPARK-24017] [SQL] Refactor ExternalCatalog to be an in...

2018-04-21 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21122 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2561/ Tes

[GitHub] spark issue #21122: [SPARK-24017] [SQL] Refactor ExternalCatalog to be an in...

2018-04-21 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21122 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark pull request #21115: [SPARK-24033] [SQL] Fix Mismatched of Window Fram...

2018-04-21 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/21115 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21122: [SPARK-24017] [SQL] Refactor ExternalCatalog to be an in...

2018-04-21 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21122 **[Test build #89680 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89680/testReport)** for PR 21122 at commit [`c62bba1`](https://github.com/apache/spark/commit/c6

[GitHub] spark issue #21115: [SPARK-24033] [SQL] Fix Mismatched of Window Frame speci...

2018-04-21 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21115 Thanks! Merged to master/2.3 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mai

[GitHub] spark pull request #21122: [SPARK-24017] [SQL] Refactor ExternalCatalog to b...

2018-04-21 Thread gatorsmile
GitHub user gatorsmile opened a pull request: https://github.com/apache/spark/pull/21122 [SPARK-24017] [SQL] Refactor ExternalCatalog to be an interface ## What changes were proposed in this pull request? This refactors the external catalog to be an interface. It can be easier f

[GitHub] spark issue #21122: [SPARK-24017] [SQL] Refactor ExternalCatalog to be an in...

2018-04-21 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21122 cc @rxin @cloud-fan --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: revie

[GitHub] spark pull request #21121: [SPARK-24042][SQL] Collection function: zip_with_...

2018-04-21 Thread kiszk
Github user kiszk commented on a diff in the pull request: https://github.com/apache/spark/pull/21121#discussion_r183214860 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -883,3 +884,139 @@ case class Concat(childr

[GitHub] spark issue #21121: [SPARK-24042][SQL] Collection function: zip_with_index

2018-04-21 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21121 Which database has this function? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands,

[GitHub] spark issue #20959: [SPARK-23846][SQL] The samplingRatio option for CSV data...

2018-04-21 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20959 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #20959: [SPARK-23846][SQL] The samplingRatio option for CSV data...

2018-04-21 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20959 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89677/ Test PASSed. ---

[GitHub] spark issue #20959: [SPARK-23846][SQL] The samplingRatio option for CSV data...

2018-04-21 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20959 **[Test build #89677 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89677/testReport)** for PR 20959 at commit [`0737bf7`](https://github.com/apache/spark/commit/0

[GitHub] spark pull request #21121: [SPARK-24042][SQL] Collection function: zip_with_...

2018-04-21 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/21121#discussion_r183214185 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -883,3 +884,139 @@ case class Concat(

[GitHub] spark pull request #21121: [SPARK-24042][SQL] Collection function: zip_with_...

2018-04-21 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/21121#discussion_r183214315 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/functions.scala --- @@ -3340,6 +3340,17 @@ object functions { */ def reverse(e: Col

[GitHub] spark pull request #21121: [SPARK-24042][SQL] Collection function: zip_with_...

2018-04-21 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/21121#discussion_r183214167 --- Diff: python/pyspark/sql/functions.py --- @@ -2191,6 +2191,24 @@ def reverse(col): return Column(sc._jvm.functions.reverse(_to_java_column(c

[GitHub] spark issue #20280: [SPARK-22232][PYTHON][SQL] Fixed Row pickling to include...

2018-04-21 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/20280 Right. Will triple check for sure but I am with you for now. Yup, something in the migration guide makes much more sense to me too. ---

[GitHub] spark issue #21121: [SPARK-24042][SQL] Collection function: zip_with_index

2018-04-21 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21121 **[Test build #89679 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89679/testReport)** for PR 21121 at commit [`551d04d`](https://github.com/apache/spark/commit/55

[GitHub] spark issue #21121: [SPARK-24042][SQL] Collection function: zip_with_index

2018-04-21 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/21121 ok to test --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@s

[GitHub] spark pull request #21110: [SPARK-24029][core] Set SO_REUSEADDR on listen so...

2018-04-21 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/21110 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21052: [SPARK-23799][SQL] FilterEstimation.evaluateInSet produc...

2018-04-21 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21052 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21052: [SPARK-23799][SQL] FilterEstimation.evaluateInSet produc...

2018-04-21 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21052 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89675/ Test PASSed. ---

[GitHub] spark issue #21110: [SPARK-24029][core] Set SO_REUSEADDR on listen sockets.

2018-04-21 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/21110 Merged to master. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews

[GitHub] spark issue #21052: [SPARK-23799][SQL] FilterEstimation.evaluateInSet produc...

2018-04-21 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21052 **[Test build #89675 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89675/testReport)** for PR 21052 at commit [`8d21488`](https://github.com/apache/spark/commit/8

[GitHub] spark issue #21056: [SPARK-23849][SQL] Tests for samplingRatio of json datas...

2018-04-21 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21056 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89674/ Test PASSed. ---

[GitHub] spark issue #21056: [SPARK-23849][SQL] Tests for samplingRatio of json datas...

2018-04-21 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21056 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21056: [SPARK-23849][SQL] Tests for samplingRatio of json datas...

2018-04-21 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21056 **[Test build #89674 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89674/testReport)** for PR 21056 at commit [`f96134c`](https://github.com/apache/spark/commit/f

[GitHub] spark issue #21056: [SPARK-23849][SQL] Tests for samplingRatio of json datas...

2018-04-21 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21056 **[Test build #89678 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89678/testReport)** for PR 21056 at commit [`fdeac84`](https://github.com/apache/spark/commit/fd

[GitHub] spark issue #20959: [SPARK-23846][SQL] The samplingRatio option for CSV data...

2018-04-21 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20959 **[Test build #89677 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89677/testReport)** for PR 20959 at commit [`0737bf7`](https://github.com/apache/spark/commit/07

[GitHub] spark issue #20280: [SPARK-22232][PYTHON][SQL] Fixed Row pickling to include...

2018-04-21 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/20280 oops, I missed this. will take a look shortly. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additio

[GitHub] spark issue #21121: [SPARK-24042][SQL] Collection function: zip_with_index

2018-04-21 Thread mn-mikke
Github user mn-mikke commented on the issue: https://github.com/apache/spark/pull/21121 cc @gatorsmile @ueshin @kiszk --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark issue #20930: [SPARK-23811][Core] FetchFailed comes before Success of ...

2018-04-21 Thread Ngone51
Github user Ngone51 commented on the issue: https://github.com/apache/spark/pull/20930 Hi, @xuanyuanking , thank for your patient explanation, sincerely. With regard to your latest explanation: > stage 2's shuffleID is 1, but stage 3 failed by missing an output for s

[GitHub] spark issue #21121: [SPARK-24042][SQL] Collection function: zip_with_index

2018-04-21 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21121 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21121: [SPARK-24042][SQL] Collection function: zip_with_index

2018-04-21 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21121 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #21121: [SPARK-24042][SQL] Collection function: zip_with_...

2018-04-21 Thread mn-mikke
GitHub user mn-mikke opened a pull request: https://github.com/apache/spark/pull/21121 [SPARK-24042][SQL] Collection function: zip_with_index ## What changes were proposed in this pull request? Implement function zip_with_index(array[, indexFirst]) that transforms the input

[GitHub] spark issue #20350: [SPARK-23179][SQL] Support option to throw exception if ...

2018-04-21 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20350 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #20350: [SPARK-23179][SQL] Support option to throw exception if ...

2018-04-21 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20350 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89673/ Test PASSed. ---

  1   2   >