[GitHub] spark pull request: [WIP] remove expensive api from InternalRow

2015-06-17 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/6869#discussion_r32703264 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/InternalRow.scala --- @@ -19,39 +19,193 @@ package org.apache.spark.sql.catalyst i

[GitHub] spark pull request: [SPARK-8209[SQL][WIP]Add function conv

2015-06-17 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6872#issuecomment-113056318 [Test build #35099 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/35099/consoleFull) for PR 6872 at commit [`0e35c54`](https://gith

[GitHub] spark pull request: [WIP] remove expensive api from InternalRow

2015-06-17 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/6869#discussion_r32703247 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/InternalRow.scala --- @@ -19,39 +19,193 @@ package org.apache.spark.sql.catalyst i

[GitHub] spark pull request: [SPARK-8209[SQL][WIP]Add function conv

2015-06-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6872#issuecomment-113056144 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not h

[GitHub] spark pull request: [SPARK-8209[SQL][WIP]Add function conv

2015-06-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6872#issuecomment-113056176 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-8209[SQL][WIP]Add function conv

2015-06-17 Thread zhichao-li
GitHub user zhichao-li opened a pull request: https://github.com/apache/spark/pull/6872 [SPARK-8209[SQL][WIP]Add function conv cc @chenghao-intel @adrian-wang You can merge this pull request into a Git repository by running: $ git pull https://github.com/zhichao-li/spark conv

[GitHub] spark pull request: [SPARK-8238][SPARK-8239][SPARK-8242][SPARK-824...

2015-06-17 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/6843#discussion_r32703083 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringOperations.scala --- @@ -313,3 +313,131 @@ case class StringLength(child:

[GitHub] spark pull request: [SPARK-8363][SQL] Move sqrt to math and extend...

2015-06-17 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/6823#discussion_r32702990 --- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/MathFunctionsSuite.scala --- @@ -191,6 +192,24 @@ class MathFunctionsSuite exten

[GitHub] spark pull request: [SPARK-8391][Core] More efficient usage of mem...

2015-06-17 Thread viirya
Github user viirya closed the pull request at: https://github.com/apache/spark/pull/6859 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enab

[GitHub] spark pull request: [SPARK-8391][Core] More efficient usage of mem...

2015-06-17 Thread viirya
Github user viirya commented on the pull request: https://github.com/apache/spark/pull/6859#issuecomment-113055385 hmm, @srowen I think you are correct. Close this now. Thanks. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as we

[GitHub] spark pull request: [SPARK-8202] [PYSPARK] fix infinite loop durin...

2015-06-17 Thread JoshRosen
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/6714#issuecomment-113055264 After this patch, it looks like `_next_limit` is only called once per `ExternalSorter.sorted` call. Since `_next_limit` is only called from there, I wonder whether we

[GitHub] spark pull request: [SPARK-8202] [PYSPARK] fix infinite loop durin...

2015-06-17 Thread davies
Github user davies commented on a diff in the pull request: https://github.com/apache/spark/pull/6714#discussion_r32702982 --- Diff: python/pyspark/tests.py --- @@ -179,9 +179,12 @@ def test_in_memory_sort(self): list(sorter.sorted(l, key=lambda x: -x,

[GitHub] spark pull request: [SPARK-8234][SQL] misc function: md5

2015-06-17 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/6779#issuecomment-113054853 Looks good - but please add a codegen version of this. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If y

[GitHub] spark pull request: [SPARK-8363][SQL] Move sqrt to math and extend...

2015-06-17 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/6823#discussion_r32702912 --- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/MathFunctionsSuite.scala --- @@ -191,6 +192,24 @@ class MathFunctionsSuite ext

[GitHub] spark pull request: [SPARK-8348][SQL] Add in operator to DataFrame...

2015-06-17 Thread yu-iskw
Github user yu-iskw commented on the pull request: https://github.com/apache/spark/pull/6824#issuecomment-113054779 @rxin Thank you for your feedback. I'll fix the test cases soon. As you said, it is hard to implement `%in%` operator in SparkR. And we don't have any operator

[GitHub] spark pull request: [SPARK-8234][SQL] misc function: md5

2015-06-17 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/6779#discussion_r32702891 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/misc.scala --- @@ -0,0 +1,45 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request: [SPARK-8234][SQL] misc function: md5

2015-06-17 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/6779#discussion_r32702874 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/misc.scala --- @@ -0,0 +1,45 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request: [SPARK-7017][HOTFIX][Project Infra]: Refactor ...

2015-06-17 Thread brennonyork
Github user brennonyork commented on the pull request: https://github.com/apache/spark/pull/6865#issuecomment-113054631 Closing in favor of #6866 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not h

[GitHub] spark pull request: [SPARK-7017][HOTFIX][Project Infra]: Refactor ...

2015-06-17 Thread brennonyork
Github user brennonyork closed the pull request at: https://github.com/apache/spark/pull/6865 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] spark pull request: [SPARK-8391][Core] More efficient usage of mem...

2015-06-17 Thread srowen
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/6859#issuecomment-113054457 Well, the JVM will often use `StringBuilder` anyway for certain string concatenations. I think this may make memory usage _worse_ as you hold on to references to large st

[GitHub] spark pull request: [SPARK-8283][SQL] Resolve udf_struct test fail...

2015-06-17 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/6828#issuecomment-113054219 Thanks. Merging in master. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have th

[GitHub] spark pull request: [SPARK-8240] [SPARK-8241] [SQL] string functio...

2015-06-17 Thread adrian-wang
Github user adrian-wang commented on a diff in the pull request: https://github.com/apache/spark/pull/6775#discussion_r32702637 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringOperations.scala --- @@ -313,3 +312,78 @@ case class StringLength(c

[GitHub] spark pull request: [SPARK-8363][SQL] Move sqrt to math and extend...

2015-06-17 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/6823#discussion_r32702614 --- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/MathFunctionsSuite.scala --- @@ -191,6 +192,24 @@ class MathFunctionsSuite exten

[GitHub] spark pull request: [SPARK-8207][SQL] Add math function bin

2015-06-17 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/6721#discussion_r32702596 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/math.scala --- @@ -51,16 +54,28 @@ abstract class LeafMathExpression(c: Double

[GitHub] spark pull request: [SPARK-8348][SQL] Add in operator to DataFrame...

2015-06-17 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/6824#discussion_r32702554 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/ColumnExpressionSuite.scala --- @@ -296,6 +296,15 @@ class ColumnExpressionSuite extends QueryTest {

[GitHub] spark pull request: [SPARK-8238][SPARK-8239][SPARK-8242][SPARK-824...

2015-06-17 Thread adrian-wang
Github user adrian-wang commented on a diff in the pull request: https://github.com/apache/spark/pull/6843#discussion_r32702589 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringOperations.scala --- @@ -313,3 +313,131 @@ case class StringLength(

[GitHub] spark pull request: [SPARK-5750][SPARK-3441][SPARK-5836][CORE] Add...

2015-06-17 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/5074#discussion_r32702504 --- Diff: docs/programming-guide.md --- @@ -1090,6 +1090,67 @@ for details. +### Shuffle operations + +Certain operations within S

[GitHub] spark pull request: [SPARK-8291] [MLlib] [PySpark] Add parse funct...

2015-06-17 Thread MechCoder
Github user MechCoder commented on the pull request: https://github.com/apache/spark/pull/6746#issuecomment-113053477 MLUtils.LabeledPoints is just a wrapper around the scala code right? Were you trying to infer that this also should be a wrapper around the parse method in Scala? Is t

[GitHub] spark pull request: [SPARK-8348][SQL] Add in operator to DataFrame...

2015-06-17 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/6824#issuecomment-113053359 I think it is ok to add the R stuff later. It is pretty hard for everyone to learn R and add all the expressions. Also considering there is a selectExpr workaround ... -

[GitHub] spark pull request: [SPARK-8207][SQL] Add math function bin

2015-06-17 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6721#issuecomment-113053074 [Test build #35098 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/35098/consoleFull) for PR 6721 at commit [`0677f1a`](https://gith

[GitHub] spark pull request: [WIP] remove expensive api from InternalRow

2015-06-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6869#issuecomment-113052926 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark pull request: [SPARK-8371][SQL] improve unit test for MaxOf ...

2015-06-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6825#issuecomment-113052909 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark pull request: [SPARK-8280][SPARK-8281][SQL]Handle NaN, null ...

2015-06-17 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/6835#issuecomment-113052936 btw I was thinking maybe Log2 and the other log functions can just be child class of the Logarithm class added in #6725? --- If your project is set up for it, you can repl

[GitHub] spark pull request: [SPARK-8218][SQL] Binary log math function upd...

2015-06-17 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6871#issuecomment-113052959 [Test build #35097 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/35097/consoleFull) for PR 6871 at commit [`a7c1522`](https://gith

[GitHub] spark pull request: [WIP] remove expensive api from InternalRow

2015-06-17 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6869#issuecomment-113052914 [Test build #35092 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/35092/console) for PR 6869 at commit [`444382e`](https://github.

[GitHub] spark pull request: [SPARK-8218][SQL] Binary log math function upd...

2015-06-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6871#issuecomment-113052865 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not h

[GitHub] spark pull request: [SPARK-8207][SQL] Add math function bin

2015-06-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6721#issuecomment-113052885 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-8218][SQL] Binary log math function upd...

2015-06-17 Thread rxin
GitHub user rxin opened a pull request: https://github.com/apache/spark/pull/6871 [SPARK-8218][SQL] Binary log math function update. Some minor updates based on after merging #6725. You can merge this pull request into a Git repository by running: $ git pull https://github.com/

[GitHub] spark pull request: [SPARK-8218][SQL] Binary log math function upd...

2015-06-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6871#issuecomment-113052884 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-8207][SQL] Add math function bin

2015-06-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6721#issuecomment-113052869 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not h

[GitHub] spark pull request: [SPARK-8218][SQL] Binary log math function upd...

2015-06-17 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/6871#issuecomment-113052849 cc @davies for review. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this f

[GitHub] spark pull request: [SPARK-8218][SQL] Add binary log math function

2015-06-17 Thread viirya
Github user viirya commented on the pull request: https://github.com/apache/spark/pull/6725#issuecomment-113052722 Thanks. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark pull request: [SPARK-8291] [MLlib] [PySpark] Add parse funct...

2015-06-17 Thread MechCoder
Github user MechCoder commented on the pull request: https://github.com/apache/spark/pull/6746#issuecomment-113052658 I have added tests to verify it. from pyspark.mllib.regression import LabeledPoint lb = LabeledPoint(2, [0.1, 1.2, 3.4]) rdd = sc.paralleliz

[GitHub] spark pull request: [SPARK-8207][SQL] Add math function bin

2015-06-17 Thread davies
Github user davies commented on a diff in the pull request: https://github.com/apache/spark/pull/6721#discussion_r32702212 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/math.scala --- @@ -51,16 +54,28 @@ abstract class LeafMathExpression(c: Double

[GitHub] spark pull request: [SPARK-8202] [PYSPARK] fix infinite loop durin...

2015-06-17 Thread JoshRosen
Github user JoshRosen commented on a diff in the pull request: https://github.com/apache/spark/pull/6714#discussion_r32702243 --- Diff: python/pyspark/tests.py --- @@ -179,9 +179,12 @@ def test_in_memory_sort(self): list(sorter.sorted(l, key=lambda x: -

[GitHub] spark pull request: [SPARK-8218][SQL] Add binary log math function

2015-06-17 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/6725 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enab

[GitHub] spark pull request: [SPARK-8218][SQL] Add binary log math function

2015-06-17 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/6725#issuecomment-113051708 Thanks - I'm going to merge this and fix the since version. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well.

[GitHub] spark pull request: [SPARK-8202] [PYSPARK] fix infinite loop durin...

2015-06-17 Thread JoshRosen
Github user JoshRosen commented on a diff in the pull request: https://github.com/apache/spark/pull/6714#discussion_r32702098 --- Diff: python/pyspark/tests.py --- @@ -179,9 +179,12 @@ def test_in_memory_sort(self): list(sorter.sorted(l, key=lambda x: -

[GitHub] spark pull request: [SPARK-8207][SQL] Add math function bin

2015-06-17 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/6721#discussion_r32702082 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/math.scala --- @@ -51,16 +54,28 @@ abstract class LeafMathExpression(c: Double

[GitHub] spark pull request: [SPARK-8280][SPARK-8281][SQL]Handle NaN, null ...

2015-06-17 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/6835#issuecomment-113050673 @yijieshen it'd make sense to merge this after https://github.com/apache/spark/pull/6725 It might change your code here. --- If your project is set up for it, yo

[GitHub] spark pull request: [SPARK-7943][SQL]support DataFrame created by ...

2015-06-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6868#issuecomment-113050215 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark pull request: [SPARK-7943][SQL]support DataFrame created by ...

2015-06-17 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6868#issuecomment-113050188 [Test build #35089 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/35089/console) for PR 6868 at commit [`8937597`](https://github.

[GitHub] spark pull request: [SPARK-7961][SQL]Refactor SQLConf to display b...

2015-06-17 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/6747 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enab

[GitHub] spark pull request: [SPARK-8429] [EC2] Add ability to set addition...

2015-06-17 Thread armisael
Github user armisael commented on the pull request: https://github.com/apache/spark/pull/6857#issuecomment-113049738 Thank you @andrewor14 and @srowen and sorry if I missed that. Hope it's fine now. --- If your project is set up for it, you can reply to this email and have your reply

[GitHub] spark pull request: [SPARK-7961][SQL]Refactor SQLConf to display b...

2015-06-17 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/6747#issuecomment-113049708 Alright merging this. Thanks. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-8371][SQL] improve unit test for MaxOf ...

2015-06-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6825#issuecomment-113049606 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not h

[GitHub] spark pull request: [SPARK-8371][SQL] improve unit test for MaxOf ...

2015-06-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6825#issuecomment-113049620 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-7884] Move block deserialization from B...

2015-06-17 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6423#issuecomment-113049213 [Test build #35095 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/35095/consoleFull) for PR 6423 at commit [`4ea1712`](https://gith

[GitHub] spark pull request: [SPARK-7180][SPARK-8090][SPARK-8091] Fix a num...

2015-06-17 Thread JoshRosen
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/6625#issuecomment-113049132 Gentle bump again: do we want to merge this as-is and fix the comments later? --- If your project is set up for it, you can reply to this email and have your reply ap

[GitHub] spark pull request: [SPARK-8240] [SPARK-8241] [SQL] string functio...

2015-06-17 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/6775#discussion_r32701661 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringOperations.scala --- @@ -313,3 +312,78 @@ case class StringLength(chi

[GitHub] spark pull request: [SPARK-8280][SPARK-8281][SQL]Handle NaN, null ...

2015-06-17 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/6835#discussion_r32701611 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/math.scala --- @@ -81,9 +80,31 @@ abstract class UnaryMathExpression(f: Double =

[GitHub] spark pull request: [SPARK-8425][Core][WIP] Add blacklist mechanis...

2015-06-17 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6870#issuecomment-113048966 [Test build #35094 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/35094/consoleFull) for PR 6870 at commit [`fb3821f`](https://gith

[GitHub] spark pull request: [SPARK-7884] Move block deserialization from B...

2015-06-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6423#issuecomment-113048892 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-7884] Move block deserialization from B...

2015-06-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6423#issuecomment-113048878 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not h

[GitHub] spark pull request: [SPARK-8425][Core][WIP] Add blacklist mechanis...

2015-06-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6870#issuecomment-113048874 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not h

[GitHub] spark pull request: [SPARK-8425][Core][WIP] Add blacklist mechanis...

2015-06-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6870#issuecomment-113048891 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-7884] Move block deserialization from B...

2015-06-17 Thread JoshRosen
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/6423#issuecomment-113048811 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not h

[GitHub] spark pull request: [SPARK-8391][Core] More efficient usage of mem...

2015-06-17 Thread viirya
Github user viirya commented on the pull request: https://github.com/apache/spark/pull/6859#issuecomment-113048670 If you still think this can't be helpful, I will close this. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as wel

[GitHub] spark pull request: [SPARK-8240] [SPARK-8241] [SQL] string functio...

2015-06-17 Thread davies
Github user davies commented on a diff in the pull request: https://github.com/apache/spark/pull/6775#discussion_r32701341 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringOperations.scala --- @@ -313,3 +312,78 @@ case class StringLength(child:

[GitHub] spark pull request: [SPARK-8238][SPARK-8239][SPARK-8242][SPARK-824...

2015-06-17 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/6843#discussion_r32701319 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringOperations.scala --- @@ -313,3 +313,131 @@ case class StringLength(child:

[GitHub] spark pull request: [SPARK-8238][SPARK-8239][SPARK-8242][SPARK-824...

2015-06-17 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/6843#discussion_r32701290 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringOperations.scala --- @@ -313,3 +313,131 @@ case class StringLength(child:

[GitHub] spark pull request: [SQL][SPARK-7088] Fix analysis for 3rd party l...

2015-06-17 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/6853#issuecomment-11304 @smola can you put WIP on the title until you feel it is ready? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as we

[GitHub] spark pull request: [SPARK-7943][SQL][WIP]support DataFrame create...

2015-06-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6494#issuecomment-113047136 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark pull request: [SPARK-7026] [SQL] fix left semi join with equ...

2015-06-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/5643#issuecomment-113047107 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark pull request: [SPARK-7943][SQL][WIP]support DataFrame create...

2015-06-17 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6494#issuecomment-113047113 [Test build #35088 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/35088/console) for PR 6494 at commit [`8833772`](https://github.

[GitHub] spark pull request: [SPARK-7026] [SQL] fix left semi join with equ...

2015-06-17 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5643#issuecomment-113047081 [Test build #35090 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/35090/console) for PR 5643 at commit [`15f9707`](https://github.

[GitHub] spark pull request: [SPARK-8240] [SPARK-8241] [SQL] string functio...

2015-06-17 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/6775#discussion_r32701067 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringOperations.scala --- @@ -313,3 +312,78 @@ case class StringLength(child: E

[GitHub] spark pull request: [SPARK-8425][Core][WIP] Add blacklist mechanis...

2015-06-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6870#issuecomment-113046430 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark pull request: [SPARK-8425][Core][WIP] Add blacklist mechanis...

2015-06-17 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6870#issuecomment-113046429 [Test build #35093 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/35093/console) for PR 6870 at commit [`b8066f3`](https://github.

[GitHub] spark pull request: [SPARK-8425][Core][WIP] Add blacklist mechanis...

2015-06-17 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6870#issuecomment-113046185 [Test build #35093 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/35093/consoleFull) for PR 6870 at commit [`b8066f3`](https://gith

[GitHub] spark pull request: [SPARK-8425][Core][WIP] Add blacklist mechanis...

2015-06-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6870#issuecomment-113045761 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-8425][Core][WIP] Add blacklist mechanis...

2015-06-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6870#issuecomment-113045719 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not h

[GitHub] spark pull request: [WIP] remove expensive api from InternalRow

2015-06-17 Thread cloud-fan
Github user cloud-fan commented on the pull request: https://github.com/apache/spark/pull/6869#issuecomment-113045584 hi @davies , I met a bug at https://github.com/apache/spark/pull/6825 that we didn't handle `BinaryType(byte[])` correctly at `equals` and `hashCode`. We should use `j

[GitHub] spark pull request: [SPARK-8240] [SPARK-8241] [SQL] string functio...

2015-06-17 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/6775#discussion_r32700846 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionRegistry.scala --- @@ -172,7 +174,7 @@ object FunctionRegistry {

[GitHub] spark pull request: [SPARK-8381][SQL]reuse typeConvert when conver...

2015-06-17 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/6831 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enab

[GitHub] spark pull request: [SPARK-8291] [MLlib] [PySpark] Add parse funct...

2015-06-17 Thread davies
Github user davies commented on the pull request: https://github.com/apache/spark/pull/6746#issuecomment-113045343 Also the format we used here, should match with LabelPoint.__str__(), then it can load the RDD after saveAsTextFile. --- If your project is set up for it, you can reply

[GitHub] spark pull request: [SPARK-8291] [MLlib] [PySpark] Add parse funct...

2015-06-17 Thread davies
Github user davies commented on the pull request: https://github.com/apache/spark/pull/6746#issuecomment-113045261 I see, in order to have similar functionality as Scala, we should do similar things as in #685, having MLUtils.loadLabelPoints() ... --- If your project is set up for it

[GitHub] spark pull request: [SPARK-8425][Core][WIP] Add blacklist mechanis...

2015-06-17 Thread jerryshao
GitHub user jerryshao opened a pull request: https://github.com/apache/spark/pull/6870 [SPARK-8425][Core][WIP] Add blacklist mechanism in task scheduling This is a proposal to add blacklist mechanism in Spark for better scheduling task to avoid running on the bad enough executors, t

[GitHub] spark pull request: [SPARK-8240] [SPARK-8241] [SQL] string functio...

2015-06-17 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/6775#discussion_r32700753 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringOperations.scala --- @@ -313,3 +312,78 @@ case class StringLength(child: E

[GitHub] spark pull request: [SPARK-8381][SQL]reuse typeConvert when conver...

2015-06-17 Thread JoshRosen
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/6831#issuecomment-113045048 LGTM, so I'm going to merge this into master. Thanks @lianhuiwang! --- If your project is set up for it, you can reply to this email and have your reply appear on Git

[GitHub] spark pull request: [SPARK-8240] [SPARK-8241] [SQL] string functio...

2015-06-17 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/6775#discussion_r32700704 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringOperations.scala --- @@ -313,3 +312,78 @@ case class StringLength(child: E

[GitHub] spark pull request: [WIP] remove expensive api from InternalRow

2015-06-17 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6869#issuecomment-113044880 [Test build #35092 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/35092/consoleFull) for PR 6869 at commit [`444382e`](https://gith

[GitHub] spark pull request: [SPARK-8240] [SPARK-8241] [SQL] string functio...

2015-06-17 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/6775#discussion_r32700685 --- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/StringFunctionsSuite.scala --- @@ -226,5 +226,13 @@ class StringFunctionsSuite e

[GitHub] spark pull request: [SPARK-8240] [SPARK-8241] [SQL] string functio...

2015-06-17 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/6775#discussion_r32700683 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/DataFrameFunctionsSuite.scala --- @@ -85,6 +86,52 @@ class DataFrameFunctionsSuite extends QueryTest {

[GitHub] spark pull request: [WIP] remove expensive api from InternalRow

2015-06-17 Thread davies
Github user davies commented on the pull request: https://github.com/apache/spark/pull/6869#issuecomment-113044642 cc @liancheng @rxin @JoshRosen --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not h

[GitHub] spark pull request: [WIP] remove expensive api from InternalRow

2015-06-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6869#issuecomment-113044641 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not h

[GitHub] spark pull request: [WIP] remove expensive api from InternalRow

2015-06-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6869#issuecomment-113044658 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [WIP] remove expensive api from InternalRow

2015-06-17 Thread davies
GitHub user davies opened a pull request: https://github.com/apache/spark/pull/6869 [WIP] remove expensive api from InternalRow This is a follow up PR for #6792, to use narrow down the APIs in InternalRow (remove those expensive ones). You can merge this pull request into a Git rep

[GitHub] spark pull request: [SPARK-8291] [MLlib] [PySpark] Add parse funct...

2015-06-17 Thread MechCoder
Github user MechCoder commented on the pull request: https://github.com/apache/spark/pull/6746#issuecomment-113044527 Hmm. The present supported format is coherent with that done in Scala. --- If your project is set up for it, you can reply to this email and have your reply appear on

[GitHub] spark pull request: [SPARK-8291] [MLlib] [PySpark] Add parse funct...

2015-06-17 Thread davies
Github user davies commented on the pull request: https://github.com/apache/spark/pull/6746#issuecomment-113044436 There could be many different formats in a string, so I'm not sure which format we should support. It's better to be done by user. cc @mengxr --- If your proje

  1   2   3   4   5   6   7   8   >