[GitHub] spark pull request #21333: [SPARK-23778][CORE] Avoid unneeded shuffle when u...

2018-06-19 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/21333#discussion_r196521913 --- Diff: core/src/test/scala/org/apache/spark/rdd/RDDSuite.scala --- @@ -154,6 +154,13 @@ class RDDSuite extends SparkFunSuite with SharedSparkContext {

[GitHub] spark issue #21564: [SPARK-24556][SQL] Always rewrite output partitioning in...

2018-06-19 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/21564 thanks, merging to master! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #21587: [SPARK-24588][SS] streaming join should require HashClus...

2018-06-19 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21587 **[Test build #92094 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92094/testReport)** for PR 21587 at commit [`6fc7913`](https://github.com/apache/spark/commit/6f

[GitHub] spark issue #21587: [SPARK-24588][SS] streaming join should require HashClus...

2018-06-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21587 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/4217/ Tes

[GitHub] spark issue #21587: [SPARK-24588][SS] streaming join should require HashClus...

2018-06-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21587 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21587: [SPARK-24588][SS] streaming join should require HashClus...

2018-06-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21587 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21587: [SPARK-24588][SS] streaming join should require HashClus...

2018-06-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21587 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/323/

[GitHub] spark pull request #21531: [SPARK-24521][SQL][TEST] Fix ineffective test in ...

2018-06-19 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/21531 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21531: [SPARK-24521][SQL][TEST] Fix ineffective test in CachedT...

2018-06-19 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21531 LGTM Thanks! Merged to master/2.3 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional c

[GitHub] spark pull request #21577: [SPARK-24589][core] Correctly identify tasks in o...

2018-06-19 Thread mridulm
Github user mridulm commented on a diff in the pull request: https://github.com/apache/spark/pull/21577#discussion_r196514785 --- Diff: core/src/main/scala/org/apache/spark/scheduler/OutputCommitCoordinator.scala --- @@ -109,20 +116,21 @@ private[spark] class OutputCommitCoordinato

[GitHub] spark pull request #21577: [SPARK-24589][core] Correctly identify tasks in o...

2018-06-19 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/21577#discussion_r196514319 --- Diff: core/src/main/scala/org/apache/spark/scheduler/OutputCommitCoordinator.scala --- @@ -109,20 +116,21 @@ private[spark] class OutputCommitCoordinator

[GitHub] spark pull request #21577: [SPARK-24589][core] Correctly identify tasks in o...

2018-06-19 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/21577#discussion_r196513151 --- Diff: core/src/main/scala/org/apache/spark/scheduler/OutputCommitCoordinator.scala --- @@ -109,20 +116,21 @@ private[spark] class OutputCommitCoordina

[GitHub] spark issue #21585: [SPARK-24583][SQL] Wrong schema type in InsertIntoDataSo...

2018-06-19 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21585 **[Test build #92093 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92093/testReport)** for PR 21585 at commit [`049844e`](https://github.com/apache/spark/commit/04

[GitHub] spark issue #21585: [SPARK-24583][SQL] Wrong schema type in InsertIntoDataSo...

2018-06-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21585 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/4216/ Tes

[GitHub] spark issue #21585: [SPARK-24583][SQL] Wrong schema type in InsertIntoDataSo...

2018-06-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21585 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21585: [SPARK-24583][SQL] Wrong schema type in InsertIntoDataSo...

2018-06-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21585 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/322/

[GitHub] spark issue #21585: [SPARK-24583][SQL] Wrong schema type in InsertIntoDataSo...

2018-06-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21585 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21585: [SPARK-24583][SQL] Wrong schema type in InsertIntoDataSo...

2018-06-19 Thread maryannxue
Github user maryannxue commented on the issue: https://github.com/apache/spark/pull/21585 Done with the changes. Thanks a lot, @cloud-fan ! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For addit

[GitHub] spark issue #21585: [SPARK-24583][SQL] Wrong schema type in InsertIntoDataSo...

2018-06-19 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21585 **[Test build #92092 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92092/testReport)** for PR 21585 at commit [`03c3c90`](https://github.com/apache/spark/commit/03

[GitHub] spark issue #21585: [SPARK-24583][SQL] Wrong schema type in InsertIntoDataSo...

2018-06-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21585 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/4215/ Tes

[GitHub] spark issue #21585: [SPARK-24583][SQL] Wrong schema type in InsertIntoDataSo...

2018-06-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21585 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21585: [SPARK-24583][SQL] Wrong schema type in InsertIntoDataSo...

2018-06-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21585 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/321/

[GitHub] spark issue #21585: [SPARK-24583][SQL] Wrong schema type in InsertIntoDataSo...

2018-06-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21585 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark pull request #21585: [SPARK-24583][SQL] Wrong schema type in InsertInt...

2018-06-19 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/21585#discussion_r196507229 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/sources/InsertSuite.scala --- @@ -520,4 +544,29 @@ class InsertSuite extends DataSourceTest with S

[GitHub] spark issue #21585: [SPARK-24583][SQL] Wrong schema type in InsertIntoDataSo...

2018-06-19 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/21585 thanks, LGTM --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@s

[GitHub] spark pull request #21585: [SPARK-24583][SQL] Wrong schema type in InsertInt...

2018-06-19 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/21585#discussion_r196506912 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/InsertIntoDataSourceCommand.scala --- @@ -38,9 +38,8 @@ case class InsertInt

[GitHub] spark issue #21585: [SPARK-24583][SQL] Wrong schema type in InsertIntoDataSo...

2018-06-19 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21585 **[Test build #92091 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92091/testReport)** for PR 21585 at commit [`bb9fa03`](https://github.com/apache/spark/commit/bb

[GitHub] spark issue #21585: [SPARK-24583][SQL] Wrong schema type in InsertIntoDataSo...

2018-06-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21585 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/320/

[GitHub] spark issue #21585: [SPARK-24583][SQL] Wrong schema type in InsertIntoDataSo...

2018-06-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21585 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21585: [SPARK-24583][SQL] Wrong schema type in InsertIntoDataSo...

2018-06-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21585 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21585: [SPARK-24583][SQL] Wrong schema type in InsertIntoDataSo...

2018-06-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21585 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/4214/ Tes

[GitHub] spark issue #21591: [SQL][WIP] Added column name listing option to INSERT IN...

2018-06-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21591 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21591: [SQL][WIP] Added column name listing option to INSERT IN...

2018-06-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21591 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21591: [SQL][WIP] Added column name listing option to INSERT IN...

2018-06-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21591 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #21591: [SQL][WIP] Added column name listing option to IN...

2018-06-19 Thread misutoth
GitHub user misutoth opened a pull request: https://github.com/apache/spark/pull/21591 [SQL][WIP] Added column name listing option to INSERT INTO ## What changes were proposed in this pull request? Added possibility to specify column list to INSERT INTO. Source column list

[GitHub] spark issue #21577: [SPARK-24589][core] Correctly identify tasks in output c...

2018-06-19 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/21577 I'm fine with separating them but we need a jira or need to update the v2 jira to handle all cases --- - To unsubscribe, e-mai

[GitHub] spark pull request #21577: [SPARK-24589][core] Correctly identify tasks in o...

2018-06-19 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/21577#discussion_r196494533 --- Diff: core/src/main/scala/org/apache/spark/scheduler/OutputCommitCoordinator.scala --- @@ -109,20 +116,21 @@ private[spark] class OutputCommitCoordina

[GitHub] spark issue #21357: [SPARK-24311][SS] Refactor HDFSBackedStateStoreProvider ...

2018-06-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21357 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21357: [SPARK-24311][SS] Refactor HDFSBackedStateStoreProvider ...

2018-06-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21357 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92088/ Test PASSed. ---

[GitHub] spark issue #21357: [SPARK-24311][SS] Refactor HDFSBackedStateStoreProvider ...

2018-06-19 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21357 **[Test build #92088 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92088/testReport)** for PR 21357 at commit [`55256b5`](https://github.com/apache/spark/commit/5

[GitHub] spark pull request #21590: [SPARK-24423][SQL] Add a new option for JDBC sour...

2018-06-19 Thread dilipbiswal
Github user dilipbiswal commented on a diff in the pull request: https://github.com/apache/spark/pull/21590#discussion_r196490214 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCOptions.scala --- @@ -65,13 +65,38 @@ class JDBCOptions( /

[GitHub] spark pull request #21590: [SPARK-24423][SQL] Add a new option for JDBC sour...

2018-06-19 Thread dilipbiswal
Github user dilipbiswal commented on a diff in the pull request: https://github.com/apache/spark/pull/21590#discussion_r196489749 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCOptions.scala --- @@ -109,6 +134,20 @@ class JDBCOptions(

[GitHub] spark issue #20350: [SPARK-23179][SQL] Support option to throw exception if ...

2018-06-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20350 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92087/ Test PASSed. ---

[GitHub] spark issue #20350: [SPARK-23179][SQL] Support option to throw exception if ...

2018-06-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20350 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #20350: [SPARK-23179][SQL] Support option to throw exception if ...

2018-06-19 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20350 **[Test build #92087 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92087/testReport)** for PR 20350 at commit [`069b861`](https://github.com/apache/spark/commit/0

[GitHub] spark pull request #21590: [SPARK-24423][SQL] Add a new option for JDBC sour...

2018-06-19 Thread dilipbiswal
Github user dilipbiswal commented on a diff in the pull request: https://github.com/apache/spark/pull/21590#discussion_r196487627 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCOptions.scala --- @@ -65,13 +65,38 @@ class JDBCOptions( /

[GitHub] spark issue #21403: [SPARK-24341][WIP][SQL] Support IN subqueries with struc...

2018-06-19 Thread mgaido91
Github user mgaido91 commented on the issue: https://github.com/apache/spark/pull/21403 I updated the PR according to the previous discussion. @hvanhovell @juliuszsompolski may you please take a look at it now? Thanks. ---

[GitHub] spark pull request #21577: [SPARK-24589][core] Correctly identify tasks in o...

2018-06-19 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/21577#discussion_r196482353 --- Diff: core/src/main/scala/org/apache/spark/scheduler/OutputCommitCoordinator.scala --- @@ -109,20 +116,21 @@ private[spark] class OutputCommitCoordinator

[GitHub] spark pull request #21577: [SPARK-24589][core] Correctly identify tasks in o...

2018-06-19 Thread jiangxb1987
Github user jiangxb1987 commented on a diff in the pull request: https://github.com/apache/spark/pull/21577#discussion_r196479588 --- Diff: core/src/main/scala/org/apache/spark/scheduler/OutputCommitCoordinator.scala --- @@ -109,20 +116,21 @@ private[spark] class OutputCommitCoordi

[GitHub] spark issue #21577: [SPARK-24589][core] Correctly identify tasks in output c...

2018-06-19 Thread jiangxb1987
Github user jiangxb1987 commented on the issue: https://github.com/apache/spark/pull/21577 This in general looks good, IMO we shall focus on fixing the output commit coordinator issue in this PR, and discuss the data source issue in a separated thread. I'm OOO this week but will

[GitHub] spark issue #21588: [WIP][SPARK-24590][BUILD] Make Jenkins tests passed with...

2018-06-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21588 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21588: [WIP][SPARK-24590][BUILD] Make Jenkins tests passed with...

2018-06-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21588 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92086/ Test FAILed. ---

[GitHub] spark issue #21588: [WIP][SPARK-24590][BUILD] Make Jenkins tests passed with...

2018-06-19 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21588 **[Test build #92086 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92086/testReport)** for PR 21588 at commit [`ef5ab0d`](https://github.com/apache/spark/commit/e

[GitHub] spark issue #21587: [SPARK-24588][SS] streaming join should require HashClus...

2018-06-19 Thread bogdanrdc
Github user bogdanrdc commented on the issue: https://github.com/apache/spark/pull/21587 maybe also fix `SinglePartition.satisfies`. It is only checking for ClusteredDistribution and defaults to true otherwise. Luckily, `SinglePartition.numPartitions` is 1 so `EnsureRequirements` will

[GitHub] spark issue #21556: [SPARK-24549][SQL] 32BitDecimalType and 64BitDecimalType...

2018-06-19 Thread wangyum
Github user wangyum commented on the issue: https://github.com/apache/spark/pull/21556 cc @gatorsmile @rdblue --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: revi

[GitHub] spark issue #21403: [SPARK-24341][WIP][SQL] Support IN subqueries with struc...

2018-06-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21403 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21403: [SPARK-24341][WIP][SQL] Support IN subqueries with struc...

2018-06-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21403 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92084/ Test PASSed. ---

[GitHub] spark issue #21403: [SPARK-24341][WIP][SQL] Support IN subqueries with struc...

2018-06-19 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21403 **[Test build #92084 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92084/testReport)** for PR 21403 at commit [`df7d3ee`](https://github.com/apache/spark/commit/d

[GitHub] spark issue #18424: [SPARK-17091] Add rule to convert IN predicate to equiva...

2018-06-19 Thread wangyum
Github user wangyum commented on the issue: https://github.com/apache/spark/pull/18424 @ptkool Are you still working on? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-m

[GitHub] spark pull request #21427: [SPARK-24324][PYTHON] Pandas Grouped Map UDF shou...

2018-06-19 Thread icexelloss
Github user icexelloss commented on a diff in the pull request: https://github.com/apache/spark/pull/21427#discussion_r196457433 --- Diff: python/pyspark/worker.py --- @@ -110,9 +116,20 @@ def wrapped(key_series, value_series): "Number of columns of the returned

[GitHub] spark issue #21527: [SPARK-24519] MapStatus has 2000 hardcoded

2018-06-19 Thread hthuynh2
Github user hthuynh2 commented on the issue: https://github.com/apache/spark/pull/21527 I updated it. Thanks. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: review

[GitHub] spark issue #21527: [SPARK-24519] MapStatus has 2000 hardcoded

2018-06-19 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/21527 I updated the jira description @hthuynh2 please update the description on the PR to match --- - To unsubscribe, e-mail: revie

[GitHub] spark issue #21403: [SPARK-24341][WIP][SQL] Support IN subqueries with struc...

2018-06-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21403 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92085/ Test FAILed. ---

[GitHub] spark issue #21403: [SPARK-24341][WIP][SQL] Support IN subqueries with struc...

2018-06-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21403 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21403: [SPARK-24341][WIP][SQL] Support IN subqueries with struc...

2018-06-19 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21403 **[Test build #92085 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92085/testReport)** for PR 21403 at commit [`c9a36e0`](https://github.com/apache/spark/commit/c

[GitHub] spark pull request #21427: [SPARK-24324][PYTHON] Pandas Grouped Map UDF shou...

2018-06-19 Thread icexelloss
Github user icexelloss commented on a diff in the pull request: https://github.com/apache/spark/pull/21427#discussion_r196442152 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/arrow/ArrowUtils.scala --- @@ -120,4 +121,19 @@ object ArrowUtils { StructFi

[GitHub] spark issue #21570: [SPARK-24564][TEST] Add test suite for RecordBinaryCompa...

2018-06-19 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21570 **[Test build #92090 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92090/testReport)** for PR 21570 at commit [`5be5a7a`](https://github.com/apache/spark/commit/5b

[GitHub] spark issue #21570: [SPARK-24564][TEST] Add test suite for RecordBinaryCompa...

2018-06-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21570 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/4213/ Tes

[GitHub] spark issue #21570: [SPARK-24564][TEST] Add test suite for RecordBinaryCompa...

2018-06-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21570 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21570: [SPARK-24564][TEST] Add test suite for RecordBinaryCompa...

2018-06-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21570 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21570: [SPARK-24564][TEST] Add test suite for RecordBinaryCompa...

2018-06-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21570 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/319/

[GitHub] spark issue #21427: [SPARK-24324][PYTHON] Pandas Grouped Map UDF should assi...

2018-06-19 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/21427 Seems fine and I am okay with it. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands,

[GitHub] spark pull request #21427: [SPARK-24324][PYTHON] Pandas Grouped Map UDF shou...

2018-06-19 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/21427#discussion_r196438132 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala --- @@ -1161,6 +1161,16 @@ object SQLConf { .booleanConf

[GitHub] spark pull request #21427: [SPARK-24324][PYTHON] Pandas Grouped Map UDF shou...

2018-06-19 Thread icexelloss
Github user icexelloss commented on a diff in the pull request: https://github.com/apache/spark/pull/21427#discussion_r196437909 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/python/WindowInPandasExec.scala --- @@ -97,7 +98,7 @@ case class WindowInPandasExec(

[GitHub] spark pull request #21427: [SPARK-24324][PYTHON] Pandas Grouped Map UDF shou...

2018-06-19 Thread icexelloss
Github user icexelloss commented on a diff in the pull request: https://github.com/apache/spark/pull/21427#discussion_r196437623 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/python/FlatMapGroupsInPandasExec.scala --- @@ -77,7 +78,7 @@ case class FlatMapGroupsIn

[GitHub] spark pull request #21427: [SPARK-24324][PYTHON] Pandas Grouped Map UDF shou...

2018-06-19 Thread icexelloss
Github user icexelloss commented on a diff in the pull request: https://github.com/apache/spark/pull/21427#discussion_r196437348 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/python/ArrowPythonRunner.scala --- @@ -58,18 +58,18 @@ class ArrowPythonRunner(

[GitHub] spark pull request #21570: [SPARK-24564][TEST] Add test suite for RecordBina...

2018-06-19 Thread jiangxb1987
Github user jiangxb1987 commented on a diff in the pull request: https://github.com/apache/spark/pull/21570#discussion_r196437321 --- Diff: sql/core/src/test/java/test/org/apache/spark/sql/execution/sort/RecordBinaryComparatorSuite.java --- @@ -0,0 +1,255 @@ +/* + * Licens

[GitHub] spark pull request #21427: [SPARK-24324][PYTHON] Pandas Grouped Map UDF shou...

2018-06-19 Thread icexelloss
Github user icexelloss commented on a diff in the pull request: https://github.com/apache/spark/pull/21427#discussion_r196436658 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/python/ArrowEvalPythonExec.scala --- @@ -63,7 +64,7 @@ case class ArrowEvalPythonExec(u

[GitHub] spark pull request #21427: [SPARK-24324][PYTHON] Pandas Grouped Map UDF shou...

2018-06-19 Thread icexelloss
Github user icexelloss commented on a diff in the pull request: https://github.com/apache/spark/pull/21427#discussion_r196435526 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/arrow/ArrowUtils.scala --- @@ -120,4 +121,19 @@ object ArrowUtils { StructFi

[GitHub] spark issue #21577: [SPARK-24589][core] Correctly identify tasks in output c...

2018-06-19 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/21577 So I think the commit/delete thing is also an issue for existing v1 and hadoop committers as well. So this doesn't fully solve the problem. spark uses a file format like (HadoopMapReduceWriteCo

[GitHub] spark issue #21558: [SPARK-24552][SQL] Use task ID instead of attempt number...

2018-06-19 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/21558 Correct TaskId is unique. I agree with @squito can we rename to tid. Hmm, good point @rdblue we need to make sure the output directories/files and such would be unique. That problem ma

[GitHub] spark issue #21567: [SPARK-24560][CORE][MESOS] Fix some getTimeAsMs as getTi...

2018-06-19 Thread jiangxb1987
Github user jiangxb1987 commented on the issue: https://github.com/apache/spark/pull/21567 Overall I don't think the current logic shall be modified. However, it shall be useful to document some the configs mentioned in this PR. --- --

[GitHub] spark issue #21109: [SPARK-24020][SQL] Sort-merge join inner range optimizat...

2018-06-19 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21109 **[Test build #92089 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92089/testReport)** for PR 21109 at commit [`9889ba1`](https://github.com/apache/spark/commit/98

[GitHub] spark pull request #21567: [SPARK-24560][CORE][MESOS] Fix some getTimeAsMs a...

2018-06-19 Thread jiangxb1987
Github user jiangxb1987 commented on a diff in the pull request: https://github.com/apache/spark/pull/21567#discussion_r196433023 --- Diff: core/src/main/scala/org/apache/spark/ui/ConsoleProgressBar.scala --- @@ -34,7 +34,7 @@ private[spark] class ConsoleProgressBar(sc: SparkContex

[GitHub] spark pull request #21567: [SPARK-24560][CORE][MESOS] Fix some getTimeAsMs a...

2018-06-19 Thread jiangxb1987
Github user jiangxb1987 commented on a diff in the pull request: https://github.com/apache/spark/pull/21567#discussion_r196432597 --- Diff: core/src/main/scala/org/apache/spark/executor/Executor.scala --- @@ -613,7 +614,7 @@ private[spark] class Executor( private[this] val

[GitHub] spark pull request #21567: [SPARK-24560][CORE][MESOS] Fix some getTimeAsMs a...

2018-06-19 Thread jiangxb1987
Github user jiangxb1987 commented on a diff in the pull request: https://github.com/apache/spark/pull/21567#discussion_r196431492 --- Diff: core/src/main/scala/org/apache/spark/deploy/worker/DriverRunner.scala --- @@ -58,7 +59,7 @@ private[deploy] class DriverRunner( /

[GitHub] spark pull request #21567: [SPARK-24560][CORE][MESOS] Fix some getTimeAsMs a...

2018-06-19 Thread jiangxb1987
Github user jiangxb1987 commented on a diff in the pull request: https://github.com/apache/spark/pull/21567#discussion_r196431134 --- Diff: core/src/main/scala/org/apache/spark/api/python/PythonRunner.scala --- @@ -354,7 +355,8 @@ private[spark] abstract class BasePythonRunner[IN,

[GitHub] spark pull request #21575: [SPARK-24566][CORE] spark.storage.blockManagerSla...

2018-06-19 Thread jiangxb1987
Github user jiangxb1987 commented on a diff in the pull request: https://github.com/apache/spark/pull/21575#discussion_r196429048 --- Diff: core/src/main/scala/org/apache/spark/HeartbeatReceiver.scala --- @@ -75,16 +76,18 @@ private[spark] class HeartbeatReceiver(sc: SparkContext,

[GitHub] spark pull request #21575: [SPARK-24566][CORE] spark.storage.blockManagerSla...

2018-06-19 Thread jiangxb1987
Github user jiangxb1987 commented on a diff in the pull request: https://github.com/apache/spark/pull/21575#discussion_r196428732 --- Diff: core/src/main/scala/org/apache/spark/HeartbeatReceiver.scala --- @@ -75,16 +76,18 @@ private[spark] class HeartbeatReceiver(sc: SparkContext,

[GitHub] spark issue #21589: [SPARK-24591][CORE] Number of cores and executors in the...

2018-06-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21589 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92081/ Test PASSed. ---

[GitHub] spark issue #21589: [SPARK-24591][CORE] Number of cores and executors in the...

2018-06-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21589 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21589: [SPARK-24591][CORE] Number of cores and executors in the...

2018-06-19 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21589 **[Test build #92081 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92081/testReport)** for PR 21589 at commit [`483743d`](https://github.com/apache/spark/commit/4

[GitHub] spark issue #21564: [SPARK-24556][SQL] Always rewrite output partitioning in...

2018-06-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21564 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21564: [SPARK-24556][SQL] Always rewrite output partitioning in...

2018-06-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21564 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92082/ Test PASSed. ---

[GitHub] spark issue #21333: [SPARK-23778][CORE] Avoid unneeded shuffle when union ge...

2018-06-19 Thread mgaido91
Github user mgaido91 commented on the issue: https://github.com/apache/spark/pull/21333 cc @cloud-fan @JoshRosen --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: re

[GitHub] spark issue #21564: [SPARK-24556][SQL] Always rewrite output partitioning in...

2018-06-19 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21564 **[Test build #92082 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92082/testReport)** for PR 21564 at commit [`dcd0ce9`](https://github.com/apache/spark/commit/d

[GitHub] spark pull request #21590: [SPARK-24423][SQL] Add a new option for JDBC sour...

2018-06-19 Thread maropu
Github user maropu commented on a diff in the pull request: https://github.com/apache/spark/pull/21590#discussion_r196407423 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCOptions.scala --- @@ -65,13 +65,38 @@ class JDBCOptions( // Req

[GitHub] spark pull request #21590: [SPARK-24423][SQL] Add a new option for JDBC sour...

2018-06-19 Thread maropu
Github user maropu commented on a diff in the pull request: https://github.com/apache/spark/pull/21590#discussion_r196406329 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCOptions.scala --- @@ -109,6 +134,20 @@ class JDBCOptions( s"W

[GitHub] spark issue #21357: [SPARK-24311][SS] Refactor HDFSBackedStateStoreProvider ...

2018-06-19 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21357 **[Test build #92088 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92088/testReport)** for PR 21357 at commit [`55256b5`](https://github.com/apache/spark/commit/55

[GitHub] spark issue #21357: [SPARK-24311][SS] Refactor HDFSBackedStateStoreProvider ...

2018-06-19 Thread HeartSaVioR
Github user HeartSaVioR commented on the issue: https://github.com/apache/spark/pull/21357 retest this, please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: revie

<    1   2   3   4   5   6   >