[GitHub] spark issue #17190: [SPARK-19478][SS] JDBC Sink

2017-10-24 Thread wuciawe
Github user wuciawe commented on the issue: https://github.com/apache/spark/pull/17190 @GaalDornick hi, I think ``` def quote(colName: String): String = { s$colName } ``` should be ``` def quote(colName: String): String = {

[GitHub] spark issue #19572: [SPARK-22349]In on-heap mode, when allocating memory fro...

2017-10-24 Thread 10110346
Github user 10110346 commented on the issue: https://github.com/apache/spark/pull/19572 cc @sameeragarwal @ericl --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: rev

[GitHub] spark issue #19572: [SPARK-22349]In on-heap mode, when allocating memory fro...

2017-10-24 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19572 **[Test build #83035 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83035/testReport)** for PR 19572 at commit [`95ae9c3`](https://github.com/apache/spark/commit/95

[GitHub] spark pull request #19572: [SPARK-22349]In on-heap mode, when allocating mem...

2017-10-24 Thread 10110346
GitHub user 10110346 opened a pull request: https://github.com/apache/spark/pull/19572 [SPARK-22349]In on-heap mode, when allocating memory from pool,we should fill memory with `MEMORY_DEBUG_FILL_CLEAN_VALUE` ## What changes were proposed in this pull request? In on-heap mode, w

[GitHub] spark issue #19571: [SPARK-15474][SQL] Write and read back non-emtpy schema ...

2017-10-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19571 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #19571: [SPARK-15474][SQL] Write and read back non-emtpy schema ...

2017-10-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19571 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83030/ Test PASSed. ---

[GitHub] spark issue #19571: [SPARK-15474][SQL] Write and read back non-emtpy schema ...

2017-10-24 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19571 **[Test build #83030 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83030/testReport)** for PR 19571 at commit [`be7ba9b`](https://github.com/apache/spark/commit/b

[GitHub] spark issue #19569: [SPARK-22348][SQL] The table cache providing ColumnarBat...

2017-10-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19569 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #19569: [SPARK-22348][SQL] The table cache providing ColumnarBat...

2017-10-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19569 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83032/ Test PASSed. ---

[GitHub] spark issue #19569: [SPARK-22348][SQL] The table cache providing ColumnarBat...

2017-10-24 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19569 **[Test build #83032 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83032/testReport)** for PR 19569 at commit [`568e791`](https://github.com/apache/spark/commit/5

[GitHub] spark issue #19217: [SPARK-21991][LAUNCHER] Fix race condition in LauncherSe...

2017-10-24 Thread nivox
Github user nivox commented on the issue: https://github.com/apache/spark/pull/19217 @vanzin @ash211 I just modified the title of the PR as per your suggestion --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.

[GitHub] spark issue #19569: [SPARK-22348][SQL] The table cache providing ColumnarBat...

2017-10-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19569 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #19569: [SPARK-22348][SQL] The table cache providing ColumnarBat...

2017-10-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19569 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83031/ Test PASSed. ---

[GitHub] spark issue #19569: [SPARK-22348][SQL] The table cache providing ColumnarBat...

2017-10-24 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19569 **[Test build #83031 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83031/testReport)** for PR 19569 at commit [`b9e238c`](https://github.com/apache/spark/commit/b

[GitHub] spark issue #19458: [SPARK-22227][CORE] DiskBlockManager.getAllBlocks now to...

2017-10-24 Thread jerryshao
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19458 There's a UT failure (https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83014/testReport/junit/org.apache.spark.storage/BlockIdSuite/test_bad_deserialization/). @superbobry please

[GitHub] spark issue #19458: [SPARK-22227][CORE] DiskBlockManager.getAllBlocks now to...

2017-10-24 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19458 **[Test build #83034 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83034/testReport)** for PR 19458 at commit [`ff9a6ae`](https://github.com/apache/spark/commit/ff

[GitHub] spark pull request #19451: SPARK-22181 Adds ReplaceExceptWithNotFilter rule

2017-10-24 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/19451#discussion_r146763856 --- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/PropagateEmptyRelationSuite.scala --- @@ -30,6 +30,7 @@ class PropagateEmp

[GitHub] spark pull request #19556: [SPARK-22328][Core] ClosureCleaner should not mis...

2017-10-24 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/19556#discussion_r146763677 --- Diff: core/src/main/scala/org/apache/spark/util/ClosureCleaner.scala --- @@ -91,6 +91,50 @@ private[spark] object ClosureCleaner extends Logging {

[GitHub] spark issue #19458: [SPARK-22227][CORE] DiskBlockManager.getAllBlocks now to...

2017-10-24 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/19458 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-

[GitHub] spark pull request #19451: SPARK-22181 Adds ReplaceExceptWithNotFilter rule

2017-10-24 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/19451#discussion_r146763654 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/ReplaceExceptWithFilter.scala --- @@ -0,0 +1,111 @@ +/* + * Licens

[GitHub] spark pull request #19451: SPARK-22181 Adds ReplaceExceptWithNotFilter rule

2017-10-24 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/19451#discussion_r146763571 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/ReplaceExceptWithFilter.scala --- @@ -0,0 +1,111 @@ +/* + * Licens

[GitHub] spark pull request #17100: [SPARK-13947][SQL] The error message from using a...

2017-10-24 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/17100 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #19451: SPARK-22181 Adds ReplaceExceptWithNotFilter rule

2017-10-24 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/19451#discussion_r146761735 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala --- @@ -922,6 +922,17 @@ object SQLConf { .intConf

[GitHub] spark pull request #19451: SPARK-22181 Adds ReplaceExceptWithNotFilter rule

2017-10-24 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/19451#discussion_r146761684 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala --- @@ -1200,6 +1211,8 @@ class SQLConf extends Serializable with Loggi

[GitHub] spark pull request #18527: [SPARK-21101][SQL] Catch IllegalStateException wh...

2017-10-24 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/18527 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17100: [SPARK-13947][SQL] The error message from using an inval...

2017-10-24 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/17100 Thanks! Merged to master. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #18527: [SPARK-21101][SQL] Catch IllegalStateException when CREA...

2017-10-24 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/18527 LGTM --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.ap

[GitHub] spark issue #18527: [SPARK-21101][SQL] Catch IllegalStateException when CREA...

2017-10-24 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/18527 Thanks! Merged to master. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #19570: [SPARK-22335][SQL] Clarify union behavior on Dataset of ...

2017-10-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19570 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #19570: [SPARK-22335][SQL] Clarify union behavior on Dataset of ...

2017-10-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19570 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83029/ Test PASSed. ---

[GitHub] spark issue #19560: [SPARK-22334][SQL] Check table size from filesystem in c...

2017-10-24 Thread jinxing64
Github user jinxing64 commented on the issue: https://github.com/apache/spark/pull/19560 >My main concern is, we'd better not to put burden on Spark to deal with metastore failures I think this make sense. I was also thinking about this when proposing this pr. I do agree with

[GitHub] spark issue #19570: [SPARK-22335][SQL] Clarify union behavior on Dataset of ...

2017-10-24 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19570 **[Test build #83029 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83029/testReport)** for PR 19570 at commit [`eab627a`](https://github.com/apache/spark/commit/e

[GitHub] spark pull request #19556: [SPARK-22328][Core] ClosureCleaner should not mis...

2017-10-24 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/19556#discussion_r146760101 --- Diff: core/src/main/scala/org/apache/spark/util/ClosureCleaner.scala --- @@ -91,6 +91,50 @@ private[spark] object ClosureCleaner extends Logging {

[GitHub] spark pull request #19557: [SPARK-22281][SPARKR][WIP] Handle R method breaki...

2017-10-24 Thread shivaram
Github user shivaram commented on a diff in the pull request: https://github.com/apache/spark/pull/19557#discussion_r146760035 --- Diff: R/pkg/R/DataFrame.R --- @@ -3249,9 +3249,12 @@ setMethod("as.data.frame", #' @note attach since 1.6.0 setMethod("attach",

[GitHub] spark issue #19390: [SPARK-18935][MESOS] Fix dynamic reservations on mesos

2017-10-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19390 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #19390: [SPARK-18935][MESOS] Fix dynamic reservations on mesos

2017-10-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19390 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83028/ Test PASSed. ---

[GitHub] spark issue #19390: [SPARK-18935][MESOS] Fix dynamic reservations on mesos

2017-10-24 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19390 **[Test build #83028 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83028/testReport)** for PR 19390 at commit [`c90d351`](https://github.com/apache/spark/commit/c

[GitHub] spark issue #19550: [SPARK-22327][SPARKR][TEST][BACKPORT-2.0] check for vers...

2017-10-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19550 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #19550: [SPARK-22327][SPARKR][TEST][BACKPORT-2.0] check for vers...

2017-10-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19550 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83033/ Test PASSed. ---

[GitHub] spark issue #19550: [SPARK-22327][SPARKR][TEST][BACKPORT-2.0] check for vers...

2017-10-24 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19550 **[Test build #83033 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83033/consoleFull)** for PR 19550 at commit [`3a5b6fa`](https://github.com/apache/spark/commit/

[GitHub] spark issue #19560: [SPARK-22334][SQL] Check table size from filesystem in c...

2017-10-24 Thread wzhfy
Github user wzhfy commented on the issue: https://github.com/apache/spark/pull/19560 My main concern is, we'd better not to put burden on Spark to deal with metastore failures, because Spark doesn't have control on metastores. The system using Spark and metastore should be responsible

[GitHub] spark pull request #19556: [SPARK-22328][Core] ClosureCleaner should not mis...

2017-10-24 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/19556#discussion_r146759438 --- Diff: core/src/main/scala/org/apache/spark/util/ClosureCleaner.scala --- @@ -91,6 +91,50 @@ private[spark] object ClosureCleaner extends Logging {

[GitHub] spark pull request #19569: [SPARK-22348][SQL] The table cache providing Colu...

2017-10-24 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/19569 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19569: [SPARK-22348][SQL] The table cache providing ColumnarBat...

2017-10-24 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/19569 good catch! merging to master, thanks! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comman

[GitHub] spark issue #19560: [SPARK-22334][SQL] Check table size from filesystem in c...

2017-10-24 Thread wzhfy
Github user wzhfy commented on the issue: https://github.com/apache/spark/pull/19560 > Users always do not know there's error in stats. Isn't there any exceptions or error messages when updating table/stats fails? I suppose the system is able to know it through logging or prot

[GitHub] spark issue #19433: [SPARK-3162] [MLlib] Add local tree training for decisio...

2017-10-24 Thread smurching
Github user smurching commented on the issue: https://github.com/apache/spark/pull/19433 @WeichenXu123 Thanks for the comments! I'll respond inline: > In your doc, you said "Specifically, we only need to store sufficient stats for each bin of a single feature, as opposed to ea

[GitHub] spark pull request #19557: [SPARK-22281][SPARKR][WIP] Handle R method breaki...

2017-10-24 Thread felixcheung
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/19557#discussion_r146757169 --- Diff: R/pkg/R/DataFrame.R --- @@ -3249,9 +3249,12 @@ setMethod("as.data.frame", #' @note attach since 1.6.0 setMethod("attach",

[GitHub] spark pull request #19557: [SPARK-22281][SPARKR][WIP] Handle R method breaki...

2017-10-24 Thread felixcheung
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/19557#discussion_r146757193 --- Diff: R/pkg/R/DataFrame.R --- @@ -3249,9 +3249,12 @@ setMethod("as.data.frame", #' @note attach since 1.6.0 setMethod("attach",

[GitHub] spark pull request #19557: [SPARK-22281][SPARKR][WIP] Handle R method breaki...

2017-10-24 Thread felixcheung
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/19557#discussion_r146757250 --- Diff: R/run-tests.sh --- @@ -38,6 +38,7 @@ FAILED=$((PIPESTATUS[0]||$FAILED)) NUM_CRAN_WARNING="$(grep -c WARNING$ $CRAN_CHECK_LOG_FILE)" N

[GitHub] spark pull request #19568: SPARK-22345: Fix sort-merge joins with conditions...

2017-10-24 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/19568#discussion_r146757237 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/joins/SortMergeJoinExec.scala --- @@ -585,21 +585,26 @@ case class SortMergeJoinExec(

[GitHub] spark pull request #19568: SPARK-22345: Fix sort-merge joins with conditions...

2017-10-24 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/19568#discussion_r146756914 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/joins/SortMergeJoinExec.scala --- @@ -585,21 +585,26 @@ case class SortMergeJoinExec(

[GitHub] spark issue #19550: [SPARK-22327][SPARKR][TEST][BACKPORT-2.0] check for vers...

2017-10-24 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19550 **[Test build #83033 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83033/consoleFull)** for PR 19550 at commit [`3a5b6fa`](https://github.com/apache/spark/commit/3

[GitHub] spark issue #19550: [SPARK-22327][SPARKR][TEST][BACKPORT-2.0] check for vers...

2017-10-24 Thread felixcheung
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/19550 Jenkins, retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mai

[GitHub] spark pull request #19568: SPARK-22345: Fix sort-merge joins with conditions...

2017-10-24 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/19568#discussion_r146755690 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/joins/SortMergeJoinExec.scala --- @@ -615,6 +620,7 @@ case class SortMergeJoinExec(

[GitHub] spark issue #19569: [SPARK-22348][SQL] The table cache providing ColumnarBat...

2017-10-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19569 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83027/ Test PASSed. ---

[GitHub] spark issue #19569: [SPARK-22348][SQL] The table cache providing ColumnarBat...

2017-10-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19569 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #19569: [SPARK-22348][SQL] The table cache providing ColumnarBat...

2017-10-24 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19569 **[Test build #83027 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83027/testReport)** for PR 19569 at commit [`f2c8266`](https://github.com/apache/spark/commit/f

[GitHub] spark issue #19480: [SPARK-22226][SQL] splitExpression can create too many m...

2017-10-24 Thread kiszk
Github user kiszk commented on the issue: https://github.com/apache/spark/pull/19480 Sounds good to me. Sorry for being late since I was busy last week. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.

[GitHub] spark pull request #19571: [SPARK-15474][SQL] Write and read back non-emtpy ...

2017-10-24 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/19571#discussion_r146752367 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/orc/OrcFileFormat.scala --- @@ -252,6 +253,13 @@ private[orc] class OrcOutputWriter( overr

[GitHub] spark pull request #19571: [SPARK-15474][SQL] Write and read back non-emtpy ...

2017-10-24 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/19571#discussion_r146752170 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/orc/OrcFileFormat.scala --- @@ -252,6 +253,13 @@ private[orc] class OrcOutputWriter( overr

[GitHub] spark issue #19569: [SPARK-22348][SQL] The table cache providing ColumnarBat...

2017-10-24 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/19569 also cc @cloud-fan for review. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark pull request #19571: [SPARK-15474][SQL] Write and read back non-emtpy ...

2017-10-24 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/19571#discussion_r146751242 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcFileFormat.scala --- @@ -39,4 +45,33 @@ private[sql] object OrcFileForma

[GitHub] spark issue #19569: [SPARK-22348][SQL] The table cache providing ColumnarBat...

2017-10-24 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19569 **[Test build #83032 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83032/testReport)** for PR 19569 at commit [`568e791`](https://github.com/apache/spark/commit/56

[GitHub] spark issue #19571: [SPARK-15474][SQL] Write and read back non-emtpy schema ...

2017-10-24 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19571 **[Test build #83030 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83030/testReport)** for PR 19571 at commit [`be7ba9b`](https://github.com/apache/spark/commit/be

[GitHub] spark issue #19569: [SPARK-22348][SQL] The table cache providing ColumnarBat...

2017-10-24 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19569 **[Test build #83031 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83031/testReport)** for PR 19569 at commit [`b9e238c`](https://github.com/apache/spark/commit/b9

[GitHub] spark pull request #19571: [SPARK-15474][SQL] Write and read back non-emtpy ...

2017-10-24 Thread dongjoon-hyun
GitHub user dongjoon-hyun opened a pull request: https://github.com/apache/spark/pull/19571 [SPARK-15474][SQL] Write and read back non-emtpy schema with empty dataframe ## What changes were proposed in this pull request? Previously, ORC file format cannot write a correct sch

[GitHub] spark pull request #19569: [SPARK-22348][SQL] The table cache providing Colu...

2017-10-24 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/19569#discussion_r146749535 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/InMemoryTableScanExec.scala --- @@ -201,35 +193,50 @@ case class InMemoryTableScan

[GitHub] spark pull request #19569: [SPARK-22348][SQL] The table cache providing Colu...

2017-10-24 Thread CodingCat
Github user CodingCat commented on a diff in the pull request: https://github.com/apache/spark/pull/19569#discussion_r146748032 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/InMemoryTableScanExec.scala --- @@ -201,35 +193,50 @@ case class InMemoryTableS

[GitHub] spark pull request #18664: [SPARK-21375][PYSPARK][SQL] Add Date and Timestam...

2017-10-24 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/18664#discussion_r146747982 --- Diff: python/pyspark/serializers.py --- @@ -224,7 +225,13 @@ def _create_batch(series): # If a nullable integer series has been promoted to float

[GitHub] spark issue #19570: [SPARK-22335][SQL] Clarify union behavior on Dataset of ...

2017-10-24 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19570 **[Test build #83029 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83029/testReport)** for PR 19570 at commit [`eab627a`](https://github.com/apache/spark/commit/ea

[GitHub] spark pull request #19570: [SPARK-22335][SQL] Clarify union behavior on Data...

2017-10-24 Thread viirya
GitHub user viirya opened a pull request: https://github.com/apache/spark/pull/19570 [SPARK-22335][SQL] Clarify union behavior on Dataset of typed objects in the document ## What changes were proposed in this pull request? Seems that end users can be confused by the union's

[GitHub] spark issue #19560: [SPARK-22334][SQL] Check table size from filesystem in c...

2017-10-24 Thread jinxing64
Github user jinxing64 commented on the issue: https://github.com/apache/spark/pull/19560 @wzhfy Thanks for comment; I know your point. In my cluster, namenode is under heavy pressure. Errors in stats happen with big chance. Users always do not know there's error in stats. T

[GitHub] spark issue #19568: SPARK-22345: Fix sort-merge joins with conditions and co...

2017-10-24 Thread kiszk
Github user kiszk commented on the issue: https://github.com/apache/spark/pull/19568 Could you please change title from `SPARK-22345` to `[SPARK-22345]`? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache

[GitHub] spark pull request #19568: SPARK-22345: Fix sort-merge joins with conditions...

2017-10-24 Thread kiszk
Github user kiszk commented on a diff in the pull request: https://github.com/apache/spark/pull/19568#discussion_r146743704 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/joins/SortMergeJoinExec.scala --- @@ -615,6 +620,7 @@ case class SortMergeJoinExec(

[GitHub] spark issue #19563: [SPARK-22284][SQL] Fix 64KB JVM bytecode limit problem i...

2017-10-24 Thread kiszk
Github user kiszk commented on the issue: https://github.com/apache/spark/pull/19563 ping @cloud-fan --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@s

[GitHub] spark issue #19569: [SPARK-22348][SQL] The table cache providing ColumnarBat...

2017-10-24 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/19569 @kiszk Thanks. I've roughly checked existing tests. Seems that there are related ones for pruning the table cache. Let me see if I can add one. --- -

[GitHub] spark issue #19390: [SPARK-18935][MESOS] Fix dynamic reservations on mesos

2017-10-24 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19390 **[Test build #83028 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83028/testReport)** for PR 19390 at commit [`c90d351`](https://github.com/apache/spark/commit/c9

[GitHub] spark issue #19569: [SPARK-22348][SQL] The table cache providing ColumnarBat...

2017-10-24 Thread kiszk
Github user kiszk commented on the issue: https://github.com/apache/spark/pull/19569 Good catch, thank you. Would it be possible to add a test case for pruning with table cache? --- - To unsubscribe, e-mail: reviews

[GitHub] spark pull request #19516: [SPARK-22277][ML]fix the bug of ChiSqSelector on ...

2017-10-24 Thread mpjlu
Github user mpjlu commented on a diff in the pull request: https://github.com/apache/spark/pull/19516#discussion_r146741139 --- Diff: mllib/src/main/scala/org/apache/spark/ml/feature/ChiSqSelector.scala --- @@ -291,9 +291,13 @@ final class ChiSqSelectorModel private[ml] (

[GitHub] spark issue #19569: [SPARK-22348][SQL] The table cache providing ColumnarBat...

2017-10-24 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19569 **[Test build #83027 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83027/testReport)** for PR 19569 at commit [`f2c8266`](https://github.com/apache/spark/commit/f2

[GitHub] spark issue #19390: [SPARK-18935][MESOS] Fix dynamic reservations on mesos

2017-10-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19390 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #19390: [SPARK-18935][MESOS] Fix dynamic reservations on mesos

2017-10-24 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19390 **[Test build #83026 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83026/testReport)** for PR 19390 at commit [`9ca6902`](https://github.com/apache/spark/commit/9

[GitHub] spark issue #19390: [SPARK-18935][MESOS] Fix dynamic reservations on mesos

2017-10-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19390 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83026/ Test FAILed. ---

[GitHub] spark pull request #19569: [SPARK-22348][SQL] The table cache providing Colu...

2017-10-24 Thread viirya
GitHub user viirya opened a pull request: https://github.com/apache/spark/pull/19569 [SPARK-22348][SQL] The table cache providing ColumnarBatch should also do partition batch pruning ## What changes were proposed in this pull request? We enable table cache `InMemoryTableSca

[GitHub] spark issue #19390: [SPARK-18935][MESOS] Fix dynamic reservations on mesos

2017-10-24 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19390 **[Test build #83026 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83026/testReport)** for PR 19390 at commit [`9ca6902`](https://github.com/apache/spark/commit/9c

[GitHub] spark issue #19390: [SPARK-18935][MESOS] Fix dynamic reservations on mesos

2017-10-24 Thread skonto
Github user skonto commented on the issue: https://github.com/apache/spark/pull/19390 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h..

[GitHub] spark issue #19567: [SPARK-22291] Postgresql UUID[] to Cassandra: Conversion...

2017-10-24 Thread jmchung
Github user jmchung commented on the issue: https://github.com/apache/spark/pull/19567 Thanks @wangyum and @viirya, I'll add the corresponding tests in `PostgresIntegrationSuite`. To @viirya , I'm not sure if the other data types will work, will consider them into tests, thanks!

[GitHub] spark issue #19433: [SPARK-3162] [MLlib] Add local tree training for decisio...

2017-10-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19433 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #19433: [SPARK-3162] [MLlib] Add local tree training for decisio...

2017-10-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19433 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83025/ Test PASSed. ---

[GitHub] spark issue #19433: [SPARK-3162] [MLlib] Add local tree training for decisio...

2017-10-24 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19433 **[Test build #83025 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83025/testReport)** for PR 19433 at commit [`fd6cdbb`](https://github.com/apache/spark/commit/f

[GitHub] spark issue #19560: [SPARK-22334][SQL] Check table size from filesystem in c...

2017-10-24 Thread wzhfy
Github user wzhfy commented on the issue: https://github.com/apache/spark/pull/19560 I wonder when this config should be used. If user knows there's some error in stats, why not just analyze the table (specify "noscan" if only size is needed)? This can fix the problem instead of verif

[GitHub] spark issue #19567: [SPARK-22291] Postgresql UUID[] to Cassandra: Conversion...

2017-10-24 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/19567 Besides uuid, other PostgreSQL types such as "cidr", "inet", they are treated as StringType too, will they work? --- - To unsubs

[GitHub] spark issue #19567: [SPARK-22291] Postgresql UUID[] to Cassandra: Conversion...

2017-10-24 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/19567 Seems that we don't even have test against uuid column, when you create test for uuid[], can you also create test for uuid? Thanks. --- -

[GitHub] spark issue #19567: [SPARK-22291] Postgresql UUID[] to Cassandra: Conversion...

2017-10-24 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/19567 @jmchung You can add unit test into `PostgresIntegrationSuite`. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org

[GitHub] spark pull request #19519: [SPARK-21840][core] Add trait that allows conf to...

2017-10-24 Thread jerryshao
Github user jerryshao commented on a diff in the pull request: https://github.com/apache/spark/pull/19519#discussion_r146737263 --- Diff: core/src/main/scala/org/apache/spark/deploy/SparkApplication.scala --- @@ -0,0 +1,55 @@ +/* + * Licensed to the Apache Software Foundati

[GitHub] spark issue #19568: SPARK-22345: Fix sort-merge joins with conditions and co...

2017-10-24 Thread rdblue
Github user rdblue commented on the issue: https://github.com/apache/spark/pull/19568 @dongjoon-hyun, yes, I'm currently working on it. I just wanted to get the rest up. --- - To unsubscribe, e-mail: reviews-unsubsc

[GitHub] spark issue #19383: [SPARK-20643][core] Add listener implementation to colle...

2017-10-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19383 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83024/ Test PASSed. ---

[GitHub] spark issue #19383: [SPARK-20643][core] Add listener implementation to colle...

2017-10-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19383 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #19383: [SPARK-20643][core] Add listener implementation to colle...

2017-10-24 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19383 **[Test build #83024 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83024/testReport)** for PR 19383 at commit [`53357a1`](https://github.com/apache/spark/commit/5

[GitHub] spark pull request #19433: [SPARK-3162] [MLlib] Add local tree training for ...

2017-10-24 Thread WeichenXu123
Github user WeichenXu123 commented on a diff in the pull request: https://github.com/apache/spark/pull/19433#discussion_r146735946 --- Diff: mllib/src/main/scala/org/apache/spark/ml/tree/impl/LocalDecisionTree.scala --- @@ -0,0 +1,250 @@ +/* + * Licensed to the Apache Softw

  1   2   3   4   >