[GitHub] spark issue #20361: [SPARK-23188][SQL] Make vectorized columar reader batch ...

2018-01-31 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/20361 Hi, All. Can we have this in Spark 2.3, too? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For

[GitHub] spark issue #20455: [SPARK-23284][SQL] Document the behavior of several Colu...

2018-01-31 Thread kiszk
Github user kiszk commented on the issue: https://github.com/apache/spark/pull/20455 LGTM for this behavior and comments. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands,

[GitHub] spark issue #20464: [SPARK-23291][SQL][R] R's substr should not reduce start...

2018-01-31 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/20464 Just in case, I am testing with: ```R df <- createDataFrame(list(list(a="abcdef"))) collect(select(df, substr(df$a, 4, 5))) substr("abcdef", 4, 5) ``` just in

[GitHub] spark issue #20463: [SQL][MINOR] Inline SpecifiedWindowFrame.defaultWindowFr...

2018-01-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20463 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/86902/ Test PASSed. ---

[GitHub] spark issue #20463: [SQL][MINOR] Inline SpecifiedWindowFrame.defaultWindowFr...

2018-01-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20463 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20463: [SQL][MINOR] Inline SpecifiedWindowFrame.defaultWindowFr...

2018-01-31 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/20463 Thanks! Merged to master. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark pull request #20361: [SPARK-23188][SQL] Make vectorized columar reader...

2018-01-31 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/20361 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17886: [SPARK-13983][SQL] Fix HiveThriftServer2 can not get "--...

2018-01-31 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/17886 cc @liufengdb --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #20463: [SQL][MINOR] Inline SpecifiedWindowFrame.defaultWindowFr...

2018-01-31 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20463 **[Test build #86902 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86902/testReport)** for PR 20463 at commit

[GitHub] spark issue #20465: [SPARK-23292][TEST] always run python tests

2018-01-31 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20465 **[Test build #86910 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86910/testReport)** for PR 20465 at commit

[GitHub] spark issue #20464: [SPARK-23291][SQL][R] R's substr should not reduce start...

2018-01-31 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/20464 I was just manually double checking both substr in R and this. It seems correct; however, I think we should add a note in the doc and release note ... One followup question is though,

[GitHub] spark issue #20465: [SPARK-23292][TEST] always run python tests

2018-01-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20465 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/460/

[GitHub] spark issue #20465: [SPARK-23292][TEST] always run python tests

2018-01-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20465 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20361: [SPARK-23188][SQL] Make vectorized columar reader batch ...

2018-01-31 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/20361 thanks, merging to master! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark pull request #19041: [SPARK-21097][CORE] Add option to recover cached ...

2018-01-31 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/19041#discussion_r165259010 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockManagerMasterEndpoint.scala --- @@ -246,6 +251,38 @@ class BlockManagerMasterEndpoint(

[GitHub] spark pull request #20465: [SPARK-23292][TEST] always run python tests

2018-01-31 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/20465#discussion_r165259756 --- Diff: python/pyspark/sql/tests.py --- @@ -2858,16 +2833,6 @@ def test_create_dataframe_from_pandas_with_timestamp(self):

[GitHub] spark pull request #20465: [SPARK-23292][TEST] always run python tests

2018-01-31 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/20465#discussion_r165259740 --- Diff: python/pyspark/sql/tests.py --- @@ -2819,13 +2802,6 @@ def test_to_pandas(self): self.assertEquals(types[4], 'datetime64[ns]')

[GitHub] spark issue #20465: [SPARK-23292][TEST] always run python tests

2018-01-31 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/20465 cc @yhuai @icexelloss @BryanCutler @ueshin @shaneknapp --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For

[GitHub] spark pull request #20433: [SPARK-23264][SQL] Support interval values withou...

2018-01-31 Thread maropu
Github user maropu commented on a diff in the pull request: https://github.com/apache/spark/pull/20433#discussion_r165259575 --- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/ExpressionParserSuite.scala --- @@ -561,8 +561,11 @@ class ExpressionParserSuite

[GitHub] spark pull request #20465: [SPARK-23292][TEST] always run python tests

2018-01-31 Thread cloud-fan
GitHub user cloud-fan opened a pull request: https://github.com/apache/spark/pull/20465 [SPARK-23292][TEST] always run python tests ## What changes were proposed in this pull request? We should not skip any tests, otherwise we can't trust the jenkins report. This

[GitHub] spark pull request #20385: [SPARK-21396][SQL] Fixes MatchError when UDTs are...

2018-01-31 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/20385 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20385: [SPARK-21396][SQL] Fixes MatchError when UDTs are passed...

2018-01-31 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/20385 LGTM @atallahhezbor Could you submit another PR to address the comment from @liufengdb ? This fix is nice to have in Spark 2.3. Let merge this now. Thanks! Merged to

[GitHub] spark pull request #20400: [SPARK-23084][PYTHON]Add unboundedPreceding(), un...

2018-01-31 Thread huaxingao
Github user huaxingao commented on a diff in the pull request: https://github.com/apache/spark/pull/20400#discussion_r165258763 --- Diff: python/pyspark/sql/functions.py --- @@ -809,6 +809,48 @@ def ntile(n): return Column(sc._jvm.functions.ntile(int(n)))

[GitHub] spark issue #20455: [SPARK-23284][SQL] Document the behavior of several Colu...

2018-01-31 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/20455 BTW we should also update `ColumnarBatchSuite` for the "return null" behavior --- - To unsubscribe, e-mail:

[GitHub] spark issue #20464: [SPARK-23291][SQL][R] R's substr should not reduce start...

2018-01-31 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/20464 @shivaram Thanks for pointing out it. I made change to the description. Hopefully it is clearer now. Basically I just want to clarify why R's substr tests are correct previously. ---

[GitHub] spark issue #20361: [SPARK-23188][SQL] Make vectorized columar reader batch ...

2018-01-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20361 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/86901/ Test PASSed. ---

[GitHub] spark issue #20361: [SPARK-23188][SQL] Make vectorized columar reader batch ...

2018-01-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20361 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #20435: [SPARK-23268][SQL]Reorganize packages in data sou...

2018-01-31 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/20435 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20361: [SPARK-23188][SQL] Make vectorized columar reader batch ...

2018-01-31 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20361 **[Test build #86901 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86901/testReport)** for PR 20361 at commit

[GitHub] spark issue #20435: [SPARK-23268][SQL]Reorganize packages in data source V2

2018-01-31 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/20435 Thanks! Merged to master/2.3 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands,

[GitHub] spark issue #20457: [SPARK-23110][MINOR] Make linearRegressionModel construc...

2018-01-31 Thread WeichenXu123
Github user WeichenXu123 commented on the issue: https://github.com/apache/spark/pull/20457 It's covered in this PR #20459 So go there discuss. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For

[GitHub] spark issue #20457: [SPARK-23110][MINOR] Make linearRegressionModel construc...

2018-01-31 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20457 **[Test build #86909 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86909/testReport)** for PR 20457 at commit

[GitHub] spark issue #20457: [SPARK-23110][MINOR] Make linearRegressionModel construc...

2018-01-31 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/20457 Just to be sure -- this was never public in a released version of Spark right? so hiding it isn't changing an API. --- - To

[GitHub] spark pull request #20457: [SPARK-23110][MINOR] Make linearRegressionModel c...

2018-01-31 Thread WeichenXu123
Github user WeichenXu123 closed the pull request at: https://github.com/apache/spark/pull/20457 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark pull request #20457: [SPARK-23110][MINOR] Make linearRegressionModel c...

2018-01-31 Thread WeichenXu123
GitHub user WeichenXu123 reopened a pull request: https://github.com/apache/spark/pull/20457 [SPARK-23110][MINOR] Make linearRegressionModel constructor private ## What changes were proposed in this pull request? make linearRegressionModel constructor private[ml]

[GitHub] spark issue #20464: [SPARK-23291][SQL][R] R's substr should not reduce start...

2018-01-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20464 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/86908/ Test PASSed. ---

[GitHub] spark issue #20464: [SPARK-23291][SQL][R] R's substr should not reduce start...

2018-01-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20464 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20464: [SPARK-23291][SQL][R] R's substr should not reduce start...

2018-01-31 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20464 **[Test build #86908 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86908/testReport)** for PR 20464 at commit

[GitHub] spark issue #17185: [SPARK-19602][SQL] Support column resolution of fully qu...

2018-01-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17185 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/86899/ Test PASSed. ---

[GitHub] spark issue #17185: [SPARK-19602][SQL] Support column resolution of fully qu...

2018-01-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17185 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #17185: [SPARK-19602][SQL] Support column resolution of fully qu...

2018-01-31 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17185 **[Test build #86899 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86899/testReport)** for PR 17185 at commit

[GitHub] spark pull request #20400: [SPARK-23084][PYTHON]Add unboundedPreceding(), un...

2018-01-31 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/20400#discussion_r165256200 --- Diff: python/pyspark/sql/window.py --- @@ -217,11 +242,16 @@ def rangeBetween(self, start, end): :param end: boundary end, inclusive.

[GitHub] spark pull request #20400: [SPARK-23084][PYTHON]Add unboundedPreceding(), un...

2018-01-31 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/20400#discussion_r165256128 --- Diff: python/pyspark/sql/window.py --- @@ -217,11 +242,16 @@ def rangeBetween(self, start, end): :param end: boundary end, inclusive.

[GitHub] spark pull request #20415: [SPARK-23247][SQL]combines Unsafe operations and ...

2018-01-31 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/20415 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20415: [SPARK-23247][SQL]combines Unsafe operations and statist...

2018-01-31 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/20415 thanks, merging to master! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #20415: [SPARK-23247][SQL]combines Unsafe operations and statist...

2018-01-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20415 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20415: [SPARK-23247][SQL]combines Unsafe operations and statist...

2018-01-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20415 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/86898/ Test PASSed. ---

[GitHub] spark issue #20464: [SPARK-23291][SQL][R] R's substr should not reduce start...

2018-01-31 Thread shivaram
Github user shivaram commented on the issue: https://github.com/apache/spark/pull/20464 Thanks for clarifying @viirya. Is the PR description accurate ? I read it as `..SQL's substr also accepts zero-based starting position` while R uses a 1-based starting position. ---

[GitHub] spark issue #20415: [SPARK-23247][SQL]combines Unsafe operations and statist...

2018-01-31 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20415 **[Test build #86898 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86898/testReport)** for PR 20415 at commit

[GitHub] spark pull request #20424: [Spark-23240][python] Better error message when e...

2018-01-31 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/20424#discussion_r165255299 --- Diff: core/src/main/scala/org/apache/spark/api/python/PythonWorkerFactory.scala --- @@ -191,7 +191,20 @@ private[spark] class

[GitHub] spark pull request #20400: [SPARK-23084][PYTHON]Add unboundedPreceding(), un...

2018-01-31 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/20400#discussion_r165254437 --- Diff: python/pyspark/sql/window.py --- @@ -129,11 +131,34 @@ def rangeBetween(start, end): :param end: boundary end, inclusive.

[GitHub] spark pull request #20400: [SPARK-23084][PYTHON]Add unboundedPreceding(), un...

2018-01-31 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/20400#discussion_r165254380 --- Diff: python/pyspark/sql/window.py --- @@ -129,11 +131,34 @@ def rangeBetween(start, end): :param end: boundary end, inclusive.

[GitHub] spark pull request #20400: [SPARK-23084][PYTHON]Add unboundedPreceding(), un...

2018-01-31 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/20400#discussion_r165254262 --- Diff: python/pyspark/sql/functions.py --- @@ -809,6 +809,48 @@ def ntile(n): return Column(sc._jvm.functions.ntile(int(n)))

[GitHub] spark pull request #20400: [SPARK-23084][PYTHON]Add unboundedPreceding(), un...

2018-01-31 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/20400#discussion_r165254925 --- Diff: python/pyspark/sql/window.py --- @@ -217,11 +242,16 @@ def rangeBetween(self, start, end): :param end: boundary end, inclusive.

[GitHub] spark pull request #20400: [SPARK-23084][PYTHON]Add unboundedPreceding(), un...

2018-01-31 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/20400#discussion_r165254861 --- Diff: python/pyspark/sql/window.py --- @@ -129,11 +131,34 @@ def rangeBetween(start, end): :param end: boundary end, inclusive.

[GitHub] spark issue #20464: [SPARK-23291][SQL][R] R's substr should not reduce start...

2018-01-31 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/20464 @shivaram This fix is to make it correctly 1-based. Previously SparkR substr API substracts starting position by 1, so it becomes zero-based. This fix matches R's substr in above link as I

[GitHub] spark pull request #20450: [SPARK-23280][SQL] add map type support to Column...

2018-01-31 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/20450 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20450: [SPARK-23280][SQL] add map type support to ColumnVector

2018-01-31 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/20450 thanks, merging to master/2.3! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands,

[GitHub] spark issue #20464: [SPARK-23291][SQL][R] R's substr should not reduce start...

2018-01-31 Thread shivaram
Github user shivaram commented on the issue: https://github.com/apache/spark/pull/20464 One thing to keep in mind is what the user's perception of the API is. If R users are going to use 1-based indexing then this might not be the right fix ?

[GitHub] spark pull request #20456: [SPARK-22624][PYSPARK] Expose range partitioning ...

2018-01-31 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/20456#discussion_r165254109 --- Diff: python/pyspark/sql/dataframe.py --- @@ -667,6 +667,55 @@ def repartition(self, numPartitions, *cols): else:

[GitHub] spark issue #20464: [SPARK-23291][SQL][R] R's substr should not reduce start...

2018-01-31 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/20464 Also @shivaram --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #20464: [SPARK-23291][SQL][R] R's substr should not reduce start...

2018-01-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20464 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/459/

[GitHub] spark issue #20464: [SPARK-23291][SQL][R] R's substr should not reduce start...

2018-01-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20464 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #20437: [SPARK-23270][Streaming][WEB-UI]FileInputDStream ...

2018-01-31 Thread guoxiaolongzte
Github user guoxiaolongzte commented on a diff in the pull request: https://github.com/apache/spark/pull/20437#discussion_r165253828 --- Diff: streaming/src/main/scala/org/apache/spark/streaming/dstream/FileInputDStream.scala --- @@ -157,7 +157,7 @@ class FileInputDStream[K, V, F

[GitHub] spark pull request #19872: [SPARK-22274][PYTHON][SQL] User-defined aggregati...

2018-01-31 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/19872#discussion_r165253818 --- Diff: python/pyspark/sql/tests.py --- @@ -4353,6 +4347,446 @@ def test_unsupported_types(self): df.groupby('id').apply(f).collect()

[GitHub] spark issue #20464: [SPARK-23291][SQL][R] R's substr should not reduce start...

2018-01-31 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/20464 cc @felixcheung @HyukjinKwon --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #20464: [SPARK-23291][SQL][R] R's substr should not reduce start...

2018-01-31 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/20464 One more thing to notice is that the two parameters (starting and ending positions) of R's substr API is also unaligned with Scala's substr which takes starting position and substring length.

[GitHub] spark issue #20463: [SQL][MINOR] Inline SpecifiedWindowFrame.defaultWindowFr...

2018-01-31 Thread hvanhovell
Github user hvanhovell commented on the issue: https://github.com/apache/spark/pull/20463 LGTM --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark pull request #19872: [SPARK-22274][PYTHON][SQL] User-defined aggregati...

2018-01-31 Thread yhuai
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/19872#discussion_r165253514 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/planning/patterns.scala --- @@ -199,7 +200,7 @@ object ExtractFiltersAndInnerJoins

[GitHub] spark issue #20464: [SPARK-23291][SQL][R] R's substr should not reduce start...

2018-01-31 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20464 **[Test build #86908 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86908/testReport)** for PR 20464 at commit

[GitHub] spark pull request #20464: [SPARK-23291][SQL][R] R's substr should not reduc...

2018-01-31 Thread viirya
GitHub user viirya opened a pull request: https://github.com/apache/spark/pull/20464 [SPARK-23291][SQL][R] R's substr should not reduce starting position by 1 when calling Scala API ## What changes were proposed in this pull request? Seems R's substr API treats Scala

[GitHub] spark pull request #20462: [SPARK-23020][core] Fix another race in the in-pr...

2018-01-31 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/20462#discussion_r165253301 --- Diff: launcher/src/main/java/org/apache/spark/launcher/InProcessAppHandle.java --- @@ -66,7 +66,7 @@ synchronized void start(String appName, Method

[GitHub] spark pull request #20462: [SPARK-23020][core] Fix another race in the in-pr...

2018-01-31 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/20462#discussion_r165253320 --- Diff: launcher/src/main/java/org/apache/spark/launcher/LauncherServer.java --- @@ -363,17 +362,28 @@ public void close() throws IOException {

[GitHub] spark pull request #20462: [SPARK-23020][core] Fix another race in the in-pr...

2018-01-31 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/20462#discussion_r165253103 --- Diff: launcher/src/main/java/org/apache/spark/launcher/InProcessAppHandle.java --- @@ -66,7 +66,7 @@ synchronized void start(String appName, Method

[GitHub] spark issue #20387: [SPARK-23203][SPARK-23204][SQL]: DataSourceV2: Use immut...

2018-01-31 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/20387 > We can add things like limit pushdown later, by adding it properly to the existing code. I tried and can't figure out how to do it with `PhysicalOperation`, that's why I build

[GitHub] spark pull request #20437: [SPARK-23270][Streaming][WEB-UI]FileInputDStream ...

2018-01-31 Thread jerryshao
Github user jerryshao commented on a diff in the pull request: https://github.com/apache/spark/pull/20437#discussion_r165252724 --- Diff: streaming/src/main/scala/org/apache/spark/streaming/dstream/FileInputDStream.scala --- @@ -157,7 +157,7 @@ class FileInputDStream[K, V, F <:

[GitHub] spark issue #20387: [SPARK-23203][SPARK-23204][SQL]: DataSourceV2: Use immut...

2018-01-31 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/20387 Currently `DataSourceOptions` is the major way for Spark and users to pass information to the data source. It's very flexible and only defines one rule: the option key lookup should be

[GitHub] spark issue #20422: [SPARK-23253][Core][Shuffle]Only write shuffle temporary...

2018-01-31 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20422 **[Test build #86907 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86907/testReport)** for PR 20422 at commit

[GitHub] spark issue #20422: [SPARK-23253][Core][Shuffle]Only write shuffle temporary...

2018-01-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20422 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #20456: [SPARK-22624][PYSPARK] Expose range partitioning ...

2018-01-31 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/20456#discussion_r165252395 --- Diff: python/pyspark/sql/dataframe.py --- @@ -667,6 +667,55 @@ def repartition(self, numPartitions, *cols): else:

[GitHub] spark issue #20422: [SPARK-23253][Core][Shuffle]Only write shuffle temporary...

2018-01-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20422 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/458/

[GitHub] spark issue #20450: [SPARK-23280][SQL] add map type support to ColumnVector

2018-01-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20450 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20450: [SPARK-23280][SQL] add map type support to ColumnVector

2018-01-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20450 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/86896/ Test PASSed. ---

[GitHub] spark issue #20450: [SPARK-23280][SQL] add map type support to ColumnVector

2018-01-31 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20450 **[Test build #86896 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86896/testReport)** for PR 20450 at commit

[GitHub] spark pull request #20437: [SPARK-23270][Streaming][WEB-UI]FileInputDStream ...

2018-01-31 Thread guoxiaolongzte
Github user guoxiaolongzte commented on a diff in the pull request: https://github.com/apache/spark/pull/20437#discussion_r165251810 --- Diff: streaming/src/main/scala/org/apache/spark/streaming/dstream/FileInputDStream.scala --- @@ -157,7 +157,7 @@ class FileInputDStream[K, V, F

[GitHub] spark pull request #20422: [SPARK-23253][Core][Shuffle]Only write shuffle te...

2018-01-31 Thread jiangxb1987
Github user jiangxb1987 commented on a diff in the pull request: https://github.com/apache/spark/pull/20422#discussion_r165250968 --- Diff: core/src/test/scala/org/apache/spark/shuffle/sort/IndexShuffleBlockResolverSuite.scala --- @@ -89,26 +96,39 @@ class

[GitHub] spark pull request #20422: [SPARK-23253][Core][Shuffle]Only write shuffle te...

2018-01-31 Thread jiangxb1987
Github user jiangxb1987 commented on a diff in the pull request: https://github.com/apache/spark/pull/20422#discussion_r165250833 --- Diff: core/src/test/scala/org/apache/spark/shuffle/sort/IndexShuffleBlockResolverSuite.scala --- @@ -89,26 +96,39 @@ class

[GitHub] spark pull request #20437: [SPARK-23270][Streaming][WEB-UI]FileInputDStream ...

2018-01-31 Thread jerryshao
Github user jerryshao commented on a diff in the pull request: https://github.com/apache/spark/pull/20437#discussion_r165249764 --- Diff: streaming/src/main/scala/org/apache/spark/streaming/dstream/FileInputDStream.scala --- @@ -157,7 +157,7 @@ class FileInputDStream[K, V, F <:

[GitHub] spark pull request #20422: [SPARK-23253][Core][Shuffle]Only write shuffle te...

2018-01-31 Thread jiangxb1987
Github user jiangxb1987 commented on a diff in the pull request: https://github.com/apache/spark/pull/20422#discussion_r165249708 --- Diff: core/src/test/scala/org/apache/spark/shuffle/sort/IndexShuffleBlockResolverSuite.scala --- @@ -17,7 +17,7 @@ package

[GitHub] spark issue #20450: [SPARK-23280][SQL] add map type support to ColumnVector

2018-01-31 Thread jiangxb1987
Github user jiangxb1987 commented on the issue: https://github.com/apache/spark/pull/20450 LGTM --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #20422: [SPARK-23253][Core][Shuffle]Only write shuffle temporary...

2018-01-31 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20422 **[Test build #86906 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86906/testReport)** for PR 20422 at commit

[GitHub] spark issue #20422: [SPARK-23253][Core][Shuffle]Only write shuffle temporary...

2018-01-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20422 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20422: [SPARK-23253][Core][Shuffle]Only write shuffle temporary...

2018-01-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20422 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/457/

[GitHub] spark issue #20460: [SPARK-23285][K8S] Allow fractional values for spark.exe...

2018-01-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20460 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20460: [SPARK-23285][K8S] Allow fractional values for spark.exe...

2018-01-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20460 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/86895/ Test FAILed. ---

[GitHub] spark issue #20460: [SPARK-23285][K8S] Allow fractional values for spark.exe...

2018-01-31 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20460 **[Test build #86895 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86895/testReport)** for PR 20460 at commit

[GitHub] spark issue #20422: [SPARK-23253][Core][Shuffle]Only write shuffle temporary...

2018-01-31 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20422 **[Test build #86905 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86905/testReport)** for PR 20422 at commit

[GitHub] spark issue #20422: [SPARK-23253][Core][Shuffle]Only write shuffle temporary...

2018-01-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20422 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/456/

[GitHub] spark issue #20435: [SPARK-23268][SQL]Reorganize packages in data source V2

2018-01-31 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/20435 LGTM, also cc @rdblue --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #20422: [SPARK-23253][Core][Shuffle]Only write shuffle temporary...

2018-01-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20422 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

<    1   2   3   4   5   6   7   >