[GitHub] spark issue #18655: [SPARK-21440][SQL][PYSPARK] Refactor ArrowConverters and...

2017-07-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18655 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #18655: [SPARK-21440][SQL][PYSPARK] Refactor ArrowConverters and...

2017-07-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18655 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79786/ Test PASSed. ---

[GitHub] spark issue #18655: [SPARK-21440][SQL][PYSPARK] Refactor ArrowConverters and...

2017-07-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18655 **[Test build #79786 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79786/testReport)** for PR 18655 at commit

[GitHub] spark issue #18649: [SPARK-21395][SQL] Spark SQL hive-thriftserver doesn't r...

2017-07-20 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/18649 I'm not familiar with this code and don't know who I should ping, cc @srowen do we have a maintainer for thrift-server? --- If your project is set up for it, you can reply to this email and have

[GitHub] spark issue #18651: [SPARK-21383][Core] Fix the YarnAllocator allocates more...

2017-07-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18651 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79789/ Test PASSed. ---

[GitHub] spark issue #18651: [SPARK-21383][Core] Fix the YarnAllocator allocates more...

2017-07-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18651 **[Test build #79789 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79789/testReport)** for PR 18651 at commit

[GitHub] spark issue #18668: [SPARK-21451][SQL]get `spark.hadoop.*` properties from s...

2017-07-20 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/18668 ok to test --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if

[GitHub] spark pull request #18680: [SPARK-21472][SQL] Introduce ArrowColumnVector as...

2017-07-20 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/18680#discussion_r128449351 --- Diff: sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/ArrowColumnVector.java --- @@ -0,0 +1,545 @@ +/* + * Licensed to the

[GitHub] spark issue #18682: [MINOR][DOCS] Fix some missing notes for Python 2.6 supp...

2017-07-20 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/18682 Merged to master --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or

[GitHub] spark pull request #18682: [MINOR][DOCS] Fix some missing notes for Python 2...

2017-07-20 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/18682 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] spark pull request #18680: [SPARK-21472][SQL] Introduce ArrowColumnVector as...

2017-07-20 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/18680#discussion_r128449421 --- Diff: sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/ArrowColumnVector.java --- @@ -0,0 +1,545 @@ +/* + * Licensed to the

[GitHub] spark issue #18683: [SPARK-21474][CORE] Make number of parallel fetches from...

2017-07-20 Thread jiangxb1987
Github user jiangxb1987 commented on the issue: https://github.com/apache/spark/pull/18683 I'm not convinced we need this configuration, we can always control the size of each fetch request by `maxSizeInFlights`, so why bother importing this new config? --- If your project is set

[GitHub] spark pull request #18468: [SPARK-20783][SQL] Create CachedBatchColumnVector...

2017-07-20 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/18468#discussion_r128457753 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/vectorized/ColumnarBatchSuite.scala --- @@ -326,6 +329,80 @@ class ColumnarBatchSuite

[GitHub] spark pull request #18185: [SPARK-20962][SQL] Support subquery column aliase...

2017-07-20 Thread maropu
Github user maropu commented on a diff in the pull request: https://github.com/apache/spark/pull/18185#discussion_r128439050 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala --- @@ -753,17 +753,26 @@ class AstBuilder(conf: SQLConf)

[GitHub] spark issue #12646: [SPARK-14878][SQL] Trim characters string function suppo...

2017-07-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/12646 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark pull request #18687: [SPARK-21484][SQL] Fix inconsistent query plans o...

2017-07-20 Thread viirya
GitHub user viirya opened a pull request: https://github.com/apache/spark/pull/18687 [SPARK-21484][SQL] Fix inconsistent query plans of Dataset after persist/unpersist ## What changes were proposed in this pull request? After the call of persist/unpersis, the query plans

[GitHub] spark pull request #17848: [SPARK-20586] [SQL] Add deterministic to ScalaUDF...

2017-07-20 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/17848#discussion_r128444612 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/UDFRegistration.scala --- @@ -130,460 +138,507 @@ class UDFRegistration private[sql]

[GitHub] spark pull request #17848: [SPARK-20586] [SQL] Add deterministic to ScalaUDF...

2017-07-20 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/17848#discussion_r128443096 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/UDFRegistration.scala --- @@ -79,8 +79,15 @@ class UDFRegistration private[sql] (functionRegistry:

[GitHub] spark pull request #17848: [SPARK-20586] [SQL] Add deterministic to ScalaUDF...

2017-07-20 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/17848#discussion_r12832 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/UDFRegistration.scala --- @@ -130,460 +138,507 @@ class UDFRegistration private[sql]

[GitHub] spark pull request #17848: [SPARK-20586] [SQL] Add deterministic to ScalaUDF...

2017-07-20 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/17848#discussion_r128444945 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/functions.scala --- @@ -3185,170 +3185,207 @@ object functions { val inputTypes = (1 to

[GitHub] spark pull request #18492: [SPARK-19326] Speculated task attempts do not get...

2017-07-20 Thread jiangxb1987
Github user jiangxb1987 commented on a diff in the pull request: https://github.com/apache/spark/pull/18492#discussion_r128452352 --- Diff: core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala --- @@ -572,20 +572,35 @@ private[spark] class

[GitHub] spark pull request #18281: [SPARK-21027][ML][PYTHON] Added tunable paralleli...

2017-07-20 Thread MLnick
Github user MLnick commented on a diff in the pull request: https://github.com/apache/spark/pull/18281#discussion_r128456263 --- Diff: python/pyspark/ml/tests.py --- @@ -1389,11 +1389,25 @@ def test_output_columns(self): (2.0,

[GitHub] spark pull request #18281: [SPARK-21027][ML][PYTHON] Added tunable paralleli...

2017-07-20 Thread MLnick
Github user MLnick commented on a diff in the pull request: https://github.com/apache/spark/pull/18281#discussion_r128455590 --- Diff: python/pyspark/ml/classification.py --- @@ -1560,8 +1581,9 @@ def trainSingleClass(index):

[GitHub] spark pull request #18281: [SPARK-21027][ML][PYTHON] Added tunable paralleli...

2017-07-20 Thread MLnick
Github user MLnick commented on a diff in the pull request: https://github.com/apache/spark/pull/18281#discussion_r128456402 --- Diff: python/pyspark/ml/tests.py --- @@ -1389,11 +1389,25 @@ def test_output_columns(self): (2.0,

[GitHub] spark pull request #18281: [SPARK-21027][ML][PYTHON] Added tunable paralleli...

2017-07-20 Thread MLnick
Github user MLnick commented on a diff in the pull request: https://github.com/apache/spark/pull/18281#discussion_r128453820 --- Diff: mllib/src/main/scala/org/apache/spark/ml/classification/OneVsRest.scala --- @@ -294,6 +296,18 @@ final class OneVsRest @Since("1.4.0") (

[GitHub] spark pull request #18281: [SPARK-21027][ML][PYTHON] Added tunable paralleli...

2017-07-20 Thread MLnick
Github user MLnick commented on a diff in the pull request: https://github.com/apache/spark/pull/18281#discussion_r128455001 --- Diff: mllib/src/test/scala/org/apache/spark/ml/classification/OneVsRestSuite.scala --- @@ -101,6 +101,45 @@ class OneVsRestSuite extends SparkFunSuite

[GitHub] spark issue #18688: delete the superfluous ‘ symbol in rdd-programming-gui...

2017-07-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18688 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark pull request #18651: [SPARK-21383][Core] Fix the YarnAllocator allocat...

2017-07-20 Thread djvulee
Github user djvulee commented on a diff in the pull request: https://github.com/apache/spark/pull/18651#discussion_r128435807 --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala --- @@ -525,8 +535,9 @@ private[yarn] class

[GitHub] spark pull request #18503: [SPARK-21271][SQL] Ensure Unsafe.sizeInBytes is a...

2017-07-20 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/18503#discussion_r128439416 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreProvider.scala --- @@ -363,7 +363,8 @@

[GitHub] spark pull request #18680: [SPARK-21472][SQL] Introduce ArrowColumnVector as...

2017-07-20 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/18680#discussion_r128448056 --- Diff: sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/ArrowColumnVector.java --- @@ -0,0 +1,545 @@ +/* + * Licensed to the

[GitHub] spark pull request #18684: [SPARK-21475][Core] Use NIO's Files API to replac...

2017-07-20 Thread jiangxb1987
Github user jiangxb1987 commented on a diff in the pull request: https://github.com/apache/spark/pull/18684#discussion_r128449639 --- Diff: core/src/main/java/org/apache/spark/shuffle/sort/BypassMergeSortShuffleWriter.java --- @@ -188,17 +189,20 @@ public void

[GitHub] spark pull request #18684: [SPARK-21475][Core] Use NIO's Files API to replac...

2017-07-20 Thread jiangxb1987
Github user jiangxb1987 commented on a diff in the pull request: https://github.com/apache/spark/pull/18684#discussion_r128447912 --- Diff: common/network-common/src/main/java/org/apache/spark/network/buffer/FileSegmentManagedBuffer.java --- @@ -132,7 +134,8 @@ public Object

[GitHub] spark pull request #18684: [SPARK-21475][Core] Use NIO's Files API to replac...

2017-07-20 Thread jiangxb1987
Github user jiangxb1987 commented on a diff in the pull request: https://github.com/apache/spark/pull/18684#discussion_r128448743 --- Diff: core/src/main/java/org/apache/spark/shuffle/sort/BypassMergeSortShuffleWriter.java --- @@ -188,17 +189,20 @@ public void

[GitHub] spark pull request #18688: delete the superfluous ‘ symbol

2017-07-20 Thread wangyangting
GitHub user wangyangting opened a pull request: https://github.com/apache/spark/pull/18688 delete the superfluous ‘ symbol **'ByKey** --> **ByKey** and `'ByKey` operations --> and `ByKey` operations ## How was this patch tested? before --> ![5x9 z l

[GitHub] spark issue #17419: [SPARK-19634][ML] Multivariate summarizer - dataframes A...

2017-07-20 Thread WeichenXu123
Github user WeichenXu123 commented on the issue: https://github.com/apache/spark/pull/17419 As the dataframe version is much slower than RDD version (currently test against vector of size 1) I also guess there is some performance issue in

[GitHub] spark issue #18649: [SPARK-21395][SQL] Spark SQL hive-thriftserver doesn't r...

2017-07-20 Thread debugger87
Github user debugger87 commented on the issue: https://github.com/apache/spark/pull/18649 @cloud-fan This patch is very simple and clear, is there any problem to review or merge it? --- If your project is set up for it, you can reply to this email and have your reply appear on

[GitHub] spark issue #18651: [SPARK-21383][Core] Fix the YarnAllocator allocates more...

2017-07-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18651 **[Test build #79789 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79789/testReport)** for PR 18651 at commit

[GitHub] spark issue #18680: [SPARK-21472][SQL] Introduce ArrowColumnVector as a read...

2017-07-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18680 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark pull request #18185: [SPARK-20962][SQL] Support subquery column aliase...

2017-07-20 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/18185#discussion_r128443990 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala --- @@ -753,17 +753,26 @@ class AstBuilder(conf: SQLConf)

[GitHub] spark issue #18353: [SPARK-21142][SS] spark-streaming-kafka-0-10 should depe...

2017-07-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18353 **[Test build #3846 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3846/testReport)** for PR 18353 at commit

[GitHub] spark issue #18668: [SPARK-21451][SQL]get `spark.hadoop.*` properties from s...

2017-07-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18668 **[Test build #79792 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79792/testReport)** for PR 18668 at commit

[GitHub] spark pull request #18680: [SPARK-21472][SQL] Introduce ArrowColumnVector as...

2017-07-20 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/18680#discussion_r128451896 --- Diff: sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/ArrowColumnVector.java --- @@ -0,0 +1,545 @@ +/* + * Licensed to the

[GitHub] spark issue #18687: [SPARK-21484][SQL] Fix inconsistent query plans of Datas...

2017-07-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18687 **[Test build #79791 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79791/testReport)** for PR 18687 at commit

[GitHub] spark issue #18680: [SPARK-21472][SQL] Introduce ArrowColumnVector as a read...

2017-07-20 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/18680 @BryanCutler all classes under the `execution` package are meant to be private, in the future we will move them to a new package if we are ready to public them. --- If your project is set up

[GitHub] spark pull request #17419: [SPARK-19634][ML] Multivariate summarizer - dataf...

2017-07-20 Thread WeichenXu123
Github user WeichenXu123 commented on a diff in the pull request: https://github.com/apache/spark/pull/17419#discussion_r128429389 --- Diff: mllib/src/test/scala/org/apache/spark/ml/stat/SummarizerSuite.scala --- @@ -0,0 +1,406 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request #17419: [SPARK-19634][ML] Multivariate summarizer - dataf...

2017-07-20 Thread WeichenXu123
Github user WeichenXu123 commented on a diff in the pull request: https://github.com/apache/spark/pull/17419#discussion_r128428434 --- Diff: mllib/src/main/scala/org/apache/spark/ml/stat/Summarizer.scala --- @@ -0,0 +1,746 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request #17419: [SPARK-19634][ML] Multivariate summarizer - dataf...

2017-07-20 Thread WeichenXu123
Github user WeichenXu123 commented on a diff in the pull request: https://github.com/apache/spark/pull/17419#discussion_r128429254 --- Diff: mllib/src/main/scala/org/apache/spark/ml/stat/Summarizer.scala --- @@ -0,0 +1,799 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request #17419: [SPARK-19634][ML] Multivariate summarizer - dataf...

2017-07-20 Thread WeichenXu123
Github user WeichenXu123 commented on a diff in the pull request: https://github.com/apache/spark/pull/17419#discussion_r128428604 --- Diff: mllib/src/main/scala/org/apache/spark/ml/stat/Summarizer.scala --- @@ -0,0 +1,746 @@ +/* + * Licensed to the Apache Software

[GitHub] spark issue #18680: [SPARK-21472][SQL] Introduce ArrowColumnVector as a read...

2017-07-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18680 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79787/ Test PASSed. ---

[GitHub] spark issue #18680: [SPARK-21472][SQL] Introduce ArrowColumnVector as a read...

2017-07-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18680 **[Test build #79787 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79787/testReport)** for PR 18680 at commit

[GitHub] spark issue #18651: [SPARK-21383][Core] Fix the YarnAllocator allocates more...

2017-07-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18651 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #12646: [SPARK-14878][SQL] Trim characters string function suppo...

2017-07-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/12646 **[Test build #79788 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79788/testReport)** for PR 12646 at commit

[GitHub] spark issue #12646: [SPARK-14878][SQL] Trim characters string function suppo...

2017-07-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/12646 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79788/ Test FAILed. ---

[GitHub] spark pull request #18185: [SPARK-20962][SQL] Support subquery column aliase...

2017-07-20 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/18185#discussion_r128438692 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala --- @@ -753,17 +753,26 @@ class AstBuilder(conf: SQLConf)

[GitHub] spark issue #18687: [SPARK-21484][SQL] Fix inconsistent query plans of Datas...

2017-07-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18687 **[Test build #79790 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79790/testReport)** for PR 18687 at commit

[GitHub] spark pull request #17848: [SPARK-20586] [SQL] Add deterministic to ScalaUDF...

2017-07-20 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/17848#discussion_r128444145 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/UDFRegistration.scala --- @@ -130,460 +138,507 @@ class UDFRegistration private[sql]

[GitHub] spark pull request #18185: [SPARK-20962][SQL] Support subquery column aliase...

2017-07-20 Thread maropu
Github user maropu commented on a diff in the pull request: https://github.com/apache/spark/pull/18185#discussion_r128453437 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala --- @@ -753,17 +753,26 @@ class AstBuilder(conf: SQLConf)

[GitHub] spark pull request #18680: [SPARK-21472][SQL] Introduce ArrowColumnVector as...

2017-07-20 Thread ueshin
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/18680#discussion_r128477508 --- Diff: sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/ArrowColumnVector.java --- @@ -0,0 +1,545 @@ +/* + * Licensed to the

[GitHub] spark pull request #18388: [SPARK-21175] Reject OpenBlocks when memory short...

2017-07-20 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/18388#discussion_r128485815 --- Diff: common/network-common/src/main/java/org/apache/spark/network/util/TransportConf.java --- @@ -257,4 +257,7 @@ public Properties cryptoConf() {

[GitHub] spark issue #17461: [SPARK-20082][ml][WIP] LDA incremental model learning

2017-07-20 Thread mdespriee
Github user mdespriee commented on the issue: https://github.com/apache/spark/pull/17461 Hi @hhbyyh, @jkbradley a gentle ping on this PR, if you could have a look at the code, and give me your opinion regarding my question hereabove (the use of Param API to provide a previous

[GitHub] spark issue #18323: [SPARK-21117][SQL] Built-in SQL Function Support - WIDTH...

2017-07-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18323 **[Test build #79794 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79794/testReport)** for PR 18323 at commit

[GitHub] spark issue #18323: [SPARK-21117][SQL] Built-in SQL Function Support - WIDTH...

2017-07-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18323 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #18323: [SPARK-21117][SQL] Built-in SQL Function Support - WIDTH...

2017-07-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18323 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79794/ Test FAILed. ---

[GitHub] spark issue #18323: [SPARK-21117][SQL] Built-in SQL Function Support - WIDTH...

2017-07-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18323 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #16924: [SPARK-19531] Send UPDATE_LENGTH for Spark History servi...

2017-07-20 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/16924 Is there any concern regarding checking file length? I guess we have no much choices here to detect file changes? There are comments for why using file length to detect file changes:

[GitHub] spark issue #18680: [SPARK-21472][SQL] Introduce ArrowColumnVector as a read...

2017-07-20 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/18680 thanks, merging to master! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and

[GitHub] spark issue #18323: [SPARK-21117][SQL] Built-in SQL Function Support - WIDTH...

2017-07-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18323 **[Test build #79799 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79799/testReport)** for PR 18323 at commit

[GitHub] spark pull request #18388: [SPARK-21175] Reject OpenBlocks when memory short...

2017-07-20 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/18388#discussion_r128526865 --- Diff: common/network-common/src/main/java/org/apache/spark/network/server/TransportRequestHandler.java --- @@ -145,7 +172,12 @@ private void

[GitHub] spark pull request #18388: [SPARK-21175] Reject OpenBlocks when memory short...

2017-07-20 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/18388#discussion_r128502814 --- Diff: common/network-common/src/main/java/org/apache/spark/network/server/TransportRequestHandler.java --- @@ -145,7 +172,12 @@ private void

[GitHub] spark issue #18689: [SPARK-10063] Follow-up: remove dead code related to an ...

2017-07-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18689 **[Test build #79797 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79797/testReport)** for PR 18689 at commit

[GitHub] spark issue #18655: [SPARK-21440][SQL][PYSPARK] Refactor ArrowConverters and...

2017-07-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18655 **[Test build #79798 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79798/testReport)** for PR 18655 at commit

[GitHub] spark pull request #18388: [SPARK-21175] Reject OpenBlocks when memory short...

2017-07-20 Thread jinxing64
Github user jinxing64 commented on a diff in the pull request: https://github.com/apache/spark/pull/18388#discussion_r128519475 --- Diff: common/network-common/src/main/java/org/apache/spark/network/server/TransportRequestHandler.java --- @@ -145,7 +172,12 @@ private void

[GitHub] spark issue #18353: [SPARK-21142][SS] spark-streaming-kafka-0-10 should depe...

2017-07-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18353 **[Test build #3847 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3847/testReport)** for PR 18353 at commit

[GitHub] spark issue #18323: [SPARK-21117][SQL] Built-in SQL Function Support - WIDTH...

2017-07-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18323 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79795/ Test FAILed. ---

[GitHub] spark issue #18323: [SPARK-21117][SQL] Built-in SQL Function Support - WIDTH...

2017-07-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18323 **[Test build #79795 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79795/testReport)** for PR 18323 at commit

[GitHub] spark issue #16924: [SPARK-19531] Send UPDATE_LENGTH for Spark History servi...

2017-07-20 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/16924 ok then this LGTM, but I'm not very confident about this part of code, cc @vanzin --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well.

[GitHub] spark issue #18388: [SPARK-21175] Reject OpenBlocks when memory shortage on ...

2017-07-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18388 **[Test build #79800 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79800/testReport)** for PR 18388 at commit

[GitHub] spark pull request #18651: [SPARK-21383][Core] Fix the YarnAllocator allocat...

2017-07-20 Thread djvulee
Github user djvulee commented on a diff in the pull request: https://github.com/apache/spark/pull/18651#discussion_r128432387 --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala --- @@ -242,7 +244,7 @@ private[yarn] class

[GitHub] spark pull request #18503: [SPARK-21271][SQL] Ensure Unsafe.sizeInBytes is a...

2017-07-20 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/18503#discussion_r128438054 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamSuite.scala --- @@ -479,6 +479,61 @@ class StreamSuite extends StreamTest {

[GitHub] spark pull request #17848: [SPARK-20586] [SQL] Add deterministic to ScalaUDF...

2017-07-20 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/17848#discussion_r128439889 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/expressions/UserDefinedFunction.scala --- @@ -84,22 +90,38 @@ case class UserDefinedFunction

[GitHub] spark pull request #17848: [SPARK-20586] [SQL] Add deterministic to ScalaUDF...

2017-07-20 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/17848#discussion_r128439969 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/UDFRegistration.scala --- @@ -79,8 +79,15 @@ class UDFRegistration private[sql] (functionRegistry:

[GitHub] spark pull request #18680: [SPARK-21472][SQL] Introduce ArrowColumnVector as...

2017-07-20 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/18680#discussion_r128446137 --- Diff: sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/ReadOnlyColumnVector.java --- @@ -0,0 +1,250 @@ +/* + * Licensed to

[GitHub] spark pull request #18680: [SPARK-21472][SQL] Introduce ArrowColumnVector as...

2017-07-20 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/18680#discussion_r128449912 --- Diff: sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/ArrowColumnVector.java --- @@ -0,0 +1,545 @@ +/* + * Licensed to the

[GitHub] spark pull request #18492: [SPARK-19326] Speculated task attempts do not get...

2017-07-20 Thread jiangxb1987
Github user jiangxb1987 commented on a diff in the pull request: https://github.com/apache/spark/pull/18492#discussion_r128452499 --- Diff: core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala --- @@ -572,20 +572,35 @@ private[spark] class

[GitHub] spark pull request #18305: [SPARK-20988][ML] Logistic regression uses aggreg...

2017-07-20 Thread MLnick
Github user MLnick commented on a diff in the pull request: https://github.com/apache/spark/pull/18305#discussion_r128458132 --- Diff: mllib/src/test/scala/org/apache/spark/ml/optim/aggregator/LogisticAggregatorSuite.scala --- @@ -0,0 +1,254 @@ +/* + * Licensed to the

[GitHub] spark pull request #18680: [SPARK-21472][SQL] Introduce ArrowColumnVector as...

2017-07-20 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/18680#discussion_r128503537 --- Diff: sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/ArrowColumnVector.java --- @@ -0,0 +1,545 @@ +/* + * Licensed to the

[GitHub] spark pull request #18680: [SPARK-21472][SQL] Introduce ArrowColumnVector as...

2017-07-20 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/18680 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] spark issue #18680: [SPARK-21472][SQL] Introduce ArrowColumnVector as a read...

2017-07-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18680 **[Test build #79793 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79793/testReport)** for PR 18680 at commit

[GitHub] spark pull request #18388: [SPARK-21175] Reject OpenBlocks when memory short...

2017-07-20 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/18388#discussion_r128503793 --- Diff: common/network-common/src/main/java/org/apache/spark/network/util/TransportConf.java --- @@ -257,4 +257,7 @@ public Properties cryptoConf() {

[GitHub] spark issue #18680: [SPARK-21472][SQL] Introduce ArrowColumnVector as a read...

2017-07-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18680 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79793/ Test PASSed. ---

[GitHub] spark issue #18680: [SPARK-21472][SQL] Introduce ArrowColumnVector as a read...

2017-07-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18680 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark pull request #18468: [SPARK-20783][SQL] Create CachedBatchColumnVector...

2017-07-20 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/18468#discussion_r128507888 --- Diff: sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/CachedBatchColumnVector.java --- @@ -0,0 +1,417 @@ +/* + * Licensed

[GitHub] spark issue #18388: [SPARK-21175] Reject OpenBlocks when memory shortage on ...

2017-07-20 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/18388 a question: can we track the number of sending chunks in `TransportRequestHandler` instead of `StreamManager`? It looks weird to me that only `OneForOneStreamManager` does the tracking while we

[GitHub] spark issue #18652: [WIP] Pull non-deterministic joining keys from Join oper...

2017-07-20 Thread baibaichen
Github user baibaichen commented on the issue: https://github.com/apache/spark/pull/18652 @viirya , @jiangxb1987 @gatorsmile In general, Hive doesn't consider non-deterministic in join condition. Some terms: 1 equi-joins with key, i.e. a.key = b.key, using

[GitHub] spark issue #18323: [SPARK-21117][SQL] Built-in SQL Function Support - WIDTH...

2017-07-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18323 **[Test build #79794 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79794/testReport)** for PR 18323 at commit

[GitHub] spark issue #18680: [SPARK-21472][SQL] Introduce ArrowColumnVector as a read...

2017-07-20 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/18680 LGTM, pending jenkins --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes

[GitHub] spark pull request #18681: [MINOR][ML] Reorg RFormula params.

2017-07-20 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/18681 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] spark pull request #18388: [SPARK-21175] Reject OpenBlocks when memory short...

2017-07-20 Thread jinxing64
Github user jinxing64 commented on a diff in the pull request: https://github.com/apache/spark/pull/18388#discussion_r128498015 --- Diff: common/network-common/src/main/java/org/apache/spark/network/util/TransportConf.java --- @@ -257,4 +257,7 @@ public Properties cryptoConf() {

[GitHub] spark issue #18687: [SPARK-21484][SQL] Fix inconsistent query plans of Datas...

2017-07-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18687 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79790/ Test FAILed. ---

[GitHub] spark issue #18687: [SPARK-21484][SQL] Fix inconsistent query plans of Datas...

2017-07-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18687 **[Test build #79790 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79790/testReport)** for PR 18687 at commit

  1   2   3   4   >