[GitHub] spark issue #19893: [SPARK-16139][TEST] Add logging functionality for leaked...

2018-01-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19893 **[Test build #85802 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85802/testReport)** for PR 19893 at commit [`87c4852`](https://github.com/apache/spark/commit/8

[GitHub] spark issue #19893: [SPARK-16139][TEST] Add logging functionality for leaked...

2018-01-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19893 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #19893: [SPARK-16139][TEST] Add logging functionality for leaked...

2018-01-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19893 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85802/ Test PASSed. ---

[GitHub] spark pull request #20174: [SPARK-22951][SQL] aggregate should not produce e...

2018-01-08 Thread liufengdb
Github user liufengdb commented on a diff in the pull request: https://github.com/apache/spark/pull/20174#discussion_r160217718 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/HashAggregateExec.scala --- @@ -230,6 +236,7 @@ case class HashAggregateExec(

[GitHub] spark issue #20146: [SPARK-11215][ML] Add multiple columns support to String...

2018-01-08 Thread felixcheung
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/20146 ok SGTM --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spar

[GitHub] spark issue #20187: [SPARK-22992][K8S] Remove assumption of the DNS domain

2018-01-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20187 **[Test build #85805 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85805/testReport)** for PR 20187 at commit [`bfaf466`](https://github.com/apache/spark/commit/b

[GitHub] spark issue #20187: [SPARK-22992][K8S] Remove assumption of the DNS domain

2018-01-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20187 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85805/ Test PASSed. ---

[GitHub] spark issue #20187: [SPARK-22992][K8S] Remove assumption of the DNS domain

2018-01-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20187 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #20187: [SPARK-22992][K8S] Remove assumption of the DNS domain

2018-01-08 Thread foxish
Github user foxish commented on the issue: https://github.com/apache/spark/pull/20187 cc/ @zihongz @bowei from K8s networking - can you guys confirm that using `..svc` is strictly better than using the FQDN which made an assumption of the dns zone (cluster.local)? --- -

[GitHub] spark issue #20187: [SPARK-22992][K8S] Remove assumption of the DNS domain

2018-01-08 Thread bowei
Github user bowei commented on the issue: https://github.com/apache/spark/pull/20187 Confirmed -- the FQDN is not going to work on clusters that have a custom cluster suffix. --- - To unsubscribe, e-mail: reviews-un

[GitHub] spark pull request #20013: [SPARK-20657][core] Speed up rendering of the sta...

2018-01-08 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/20013#discussion_r16052 --- Diff: core/src/main/scala/org/apache/spark/status/LiveEntity.scala --- @@ -119,118 +121,115 @@ private class LiveTask( import LiveEntityHelpe

[GitHub] spark issue #20013: [SPARK-20657][core] Speed up rendering of the stages pag...

2018-01-08 Thread vanzin
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/20013 > I suggest running more tests(real workloads) I've tested all of these PRs attached to SPARK-18085 with lots of real and synthetic app logs. --- --

[GitHub] spark issue #20187: [SPARK-22992][K8S] Remove assumption of the DNS domain

2018-01-08 Thread liyinan926
Github user liyinan926 commented on the issue: https://github.com/apache/spark/pull/20187 @foxish have you run the integration tests against this on both minikube and GKE? --- - To unsubscribe, e-mail: reviews-unsub

[GitHub] spark issue #20098: [SPARK-22914][DEPLOY] Register history.ui.port

2018-01-08 Thread gerashegalov
Github user gerashegalov commented on the issue: https://github.com/apache/spark/pull/20098 Thank you for review and commit @vanzin ! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20187: [SPARK-22992][K8S] Remove assumption of the DNS domain

2018-01-08 Thread foxish
Github user foxish commented on the issue: https://github.com/apache/spark/pull/20187 @liyinan926 -- not yet, will be running them shortly. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For addit

[GitHub] spark issue #20013: [SPARK-20657][core] Speed up rendering of the stages pag...

2018-01-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20013 **[Test build #85806 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85806/testReport)** for PR 20013 at commit [`ed59b4d`](https://github.com/apache/spark/commit/ed

[GitHub] spark pull request #20171: [SPARK-22978] [PySpark] Register Vectorized UDFs ...

2018-01-08 Thread icexelloss
Github user icexelloss commented on a diff in the pull request: https://github.com/apache/spark/pull/20171#discussion_r160229142 --- Diff: python/pyspark/sql/tests.py --- @@ -3616,6 +3616,34 @@ def test_vectorized_udf_basic(self): bool_f(col('bool')))

[GitHub] spark pull request #19943: [SPARK-16060][SQL] Support Vectorized ORC Reader

2018-01-08 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/19943#discussion_r160232913 --- Diff: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/orc/OrcColumnarBatchReader.java --- @@ -0,0 +1,509 @@ +/* + * Lic

[GitHub] spark issue #19872: [SPARK-22274][PYTHON][SQL] User-defined aggregation func...

2018-01-08 Thread icexelloss
Github user icexelloss commented on the issue: https://github.com/apache/spark/pull/19872 Ping @ueshin, do you think of the current state of the PR? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org

[GitHub] spark issue #20019: [SPARK-22361][SQL][TEST] Add unit test for Window Frames

2018-01-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20019 **[Test build #85803 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85803/testReport)** for PR 20019 at commit [`d1e2454`](https://github.com/apache/spark/commit/d

[GitHub] spark issue #20019: [SPARK-22361][SQL][TEST] Add unit test for Window Frames

2018-01-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20019 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #20019: [SPARK-22361][SQL][TEST] Add unit test for Window Frames

2018-01-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20019 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85803/ Test PASSed. ---

[GitHub] spark pull request #19943: [SPARK-16060][SQL] Support Vectorized ORC Reader

2018-01-08 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/19943#discussion_r160236021 --- Diff: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/orc/OrcColumnarBatchReader.java --- @@ -0,0 +1,509 @@ +/* + * Lic

[GitHub] spark pull request #19943: [SPARK-16060][SQL] Support Vectorized ORC Reader

2018-01-08 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/19943#discussion_r160236163 --- Diff: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/orc/OrcColumnarBatchReader.java --- @@ -0,0 +1,509 @@ +/* + * Lic

[GitHub] spark pull request #19943: [SPARK-16060][SQL] Support Vectorized ORC Reader

2018-01-08 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/19943#discussion_r160236240 --- Diff: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/orc/OrcColumnarBatchReader.java --- @@ -0,0 +1,509 @@ +/* + * Lic

[GitHub] spark pull request #20178: [Spark-22952][CORE] Deprecate stageAttemptId in f...

2018-01-08 Thread zsxwing
Github user zsxwing commented on a diff in the pull request: https://github.com/apache/spark/pull/20178#discussion_r160236611 --- Diff: core/src/main/scala/org/apache/spark/scheduler/StageInfo.scala --- @@ -56,6 +56,8 @@ class StageInfo( completionTime = Some(System.current

[GitHub] spark issue #19943: [SPARK-16060][SQL] Support Vectorized ORC Reader

2018-01-08 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/19943 @henrify , @cloud-fan . For @henrify 's question, I got the answer. The answer is negative like the official document. Even ORC reader side, the data for a VectorizedRowBatch comes from more

[GitHub] spark issue #20187: [SPARK-22992][K8S] Remove assumption of the DNS domain

2018-01-08 Thread vanzin
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/20187 Let me know how those tests go and then I'll merge this. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For ad

[GitHub] spark issue #19943: [SPARK-16060][SQL] Support Vectorized ORC Reader

2018-01-08 Thread henrify
Github user henrify commented on the issue: https://github.com/apache/spark/pull/19943 @dongjoon-hyun Ok thanks. It is pity that the single buffer cannot be used, would have reduced number of arraycopy() calls by 5 orders of magnitude.. Btw have you tested the inlining behaviour or tr

[GitHub] spark pull request #20188: [SPARK-22993][ML] Clarify HasCheckpointInterval p...

2018-01-08 Thread sethah
GitHub user sethah opened a pull request: https://github.com/apache/spark/pull/20188 [SPARK-22993][ML] Clarify HasCheckpointInterval param doc ## What changes were proposed in this pull request? Add a note to the `HasCheckpointInterval` parameter doc that clarifies that thi

[GitHub] spark issue #16209: [SPARK-10849][SQL] Adds option to the JDBC data source w...

2018-01-08 Thread cbyn
Github user cbyn commented on the issue: https://github.com/apache/spark/pull/16209 Thanks @sureshthalamati. I thought the idea was to specify the destination database type. E.g. writing spark sql strings as VARCHAR works but VARCHAR is not a spark sql type. (I'm using the VARCHAR fea

[GitHub] spark issue #20188: [SPARK-22993][ML] Clarify HasCheckpointInterval param do...

2018-01-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20188 **[Test build #85807 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85807/testReport)** for PR 20188 at commit [`752d0ba`](https://github.com/apache/spark/commit/75

[GitHub] spark issue #20188: [SPARK-22993][ML] Clarify HasCheckpointInterval param do...

2018-01-08 Thread sethah
Github user sethah commented on the issue: https://github.com/apache/spark/pull/20188 cc @srowen @holdenk The MLlib counterparts actually make mention of this, but for some reason the note never got ported over to ML package. The only caveat I can think of is that this doc is

[GitHub] spark issue #19943: [SPARK-16060][SQL] Support Vectorized ORC Reader

2018-01-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19943 **[Test build #85808 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85808/testReport)** for PR 19943 at commit [`ba03d20`](https://github.com/apache/spark/commit/ba

[GitHub] spark pull request #19943: [SPARK-16060][SQL] Support Vectorized ORC Reader

2018-01-08 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/19943#discussion_r160251456 --- Diff: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/orc/OrcColumnarBatchReader.java --- @@ -0,0 +1,509 @@ +/* + * Lic

[GitHub] spark pull request #20097: [SPARK-22912] v2 data source support in MicroBatc...

2018-01-08 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/20097#discussion_r160251566 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/MicroBatchExecution.scala --- @@ -392,6 +443,21 @@ class MicroBatchExecution(

[GitHub] spark issue #20097: [SPARK-22912] v2 data source support in MicroBatchExecut...

2018-01-08 Thread tdas
Github user tdas commented on the issue: https://github.com/apache/spark/pull/20097 LGTM. Merging it to master. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: rev

[GitHub] spark issue #20187: [SPARK-22992][K8S] Remove assumption of the DNS domain

2018-01-08 Thread foxish
Github user foxish commented on the issue: https://github.com/apache/spark/pull/20187 ``` Discovery starting. Discovery completed in 191 milliseconds. Run starting. Expected test count is: 8 KubernetesSuite: - Run SparkPi with no resources - Run SparkPi with a ve

[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning

2018-01-08 Thread Gauravshah
Github user Gauravshah commented on the issue: https://github.com/apache/spark/pull/16578 @mallman do you foresee any issues ? planning to backport it to spark 2.2 on personal fork. will probably make jitpack release --- --

[GitHub] spark issue #20187: [SPARK-22992][K8S] Remove assumption of the DNS domain

2018-01-08 Thread vanzin
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/20187 Merging to master / 2.3. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: revie

[GitHub] spark pull request #20187: [SPARK-22992][K8S] Remove assumption of the DNS d...

2018-01-08 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/20187 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20168: SPARK-22730 Add ImageSchema support for non-integ...

2018-01-08 Thread imatiach-msft
Github user imatiach-msft commented on a diff in the pull request: https://github.com/apache/spark/pull/20168#discussion_r160255634 --- Diff: python/pyspark/ml/image.py --- @@ -201,8 +243,9 @@ def readImages(self, path, recursive=False, numPartitions=-1, .. versionadde

[GitHub] spark pull request #20168: SPARK-22730 Add ImageSchema support for non-integ...

2018-01-08 Thread imatiach-msft
Github user imatiach-msft commented on a diff in the pull request: https://github.com/apache/spark/pull/20168#discussion_r160255996 --- Diff: python/pyspark/ml/tests.py --- @@ -1843,6 +1844,28 @@ def tearDown(self): class ImageReaderTest(SparkSessionTestCase): +

[GitHub] spark issue #20186: [SPARK-22990][Core] Fix method isFairScheduler in JobsTa...

2018-01-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20186 **[Test build #85804 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85804/testReport)** for PR 20186 at commit [`10b5518`](https://github.com/apache/spark/commit/1

[GitHub] spark issue #20186: [SPARK-22990][Core] Fix method isFairScheduler in JobsTa...

2018-01-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20186 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #20186: [SPARK-22990][Core] Fix method isFairScheduler in JobsTa...

2018-01-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20186 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85804/ Test PASSed. ---

[GitHub] spark pull request #20168: SPARK-22730 Add ImageSchema support for non-integ...

2018-01-08 Thread imatiach-msft
Github user imatiach-msft commented on a diff in the pull request: https://github.com/apache/spark/pull/20168#discussion_r160257066 --- Diff: python/pyspark/ml/tests.py --- @@ -1843,6 +1844,28 @@ def tearDown(self): class ImageReaderTest(SparkSessionTestCase): +

[GitHub] spark issue #20013: [SPARK-20657][core] Speed up rendering of the stages pag...

2018-01-08 Thread sameeragarwal
Github user sameeragarwal commented on the issue: https://github.com/apache/spark/pull/20013 @vanzin Thanks for the great work! Given this is a sizable change, I'll like get this in before cutting RC1. --- - To unsu

[GitHub] spark pull request #20168: SPARK-22730 Add ImageSchema support for non-integ...

2018-01-08 Thread imatiach-msft
Github user imatiach-msft commented on a diff in the pull request: https://github.com/apache/spark/pull/20168#discussion_r160258674 --- Diff: mllib/src/main/scala/org/apache/spark/ml/image/ImageSchema.scala --- @@ -37,20 +37,51 @@ import org.apache.spark.sql.types._ @Since("2.

[GitHub] spark pull request #20168: SPARK-22730 Add ImageSchema support for non-integ...

2018-01-08 Thread imatiach-msft
Github user imatiach-msft commented on a diff in the pull request: https://github.com/apache/spark/pull/20168#discussion_r160259006 --- Diff: mllib/src/main/scala/org/apache/spark/ml/image/ImageSchema.scala --- @@ -143,12 +174,12 @@ object ImageSchema { val height

[GitHub] spark pull request #20097: [SPARK-22912] v2 data source support in MicroBatc...

2018-01-08 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/20097 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20168: SPARK-22730 Add ImageSchema support for non-integ...

2018-01-08 Thread imatiach-msft
Github user imatiach-msft commented on a diff in the pull request: https://github.com/apache/spark/pull/20168#discussion_r160260468 --- Diff: python/pyspark/ml/tests.py --- @@ -1843,6 +1844,28 @@ def tearDown(self): class ImageReaderTest(SparkSessionTestCase): +

[GitHub] spark issue #20168: SPARK-22730 Add ImageSchema support for non-integer imag...

2018-01-08 Thread imatiach-msft
Github user imatiach-msft commented on the issue: https://github.com/apache/spark/pull/20168 @tomasatdatabricks nice PR! I've added a few comments. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org

[GitHub] spark issue #20188: [SPARK-22993][ML] Clarify HasCheckpointInterval param do...

2018-01-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20188 **[Test build #85807 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85807/testReport)** for PR 20188 at commit [`752d0ba`](https://github.com/apache/spark/commit/7

[GitHub] spark issue #20188: [SPARK-22993][ML] Clarify HasCheckpointInterval param do...

2018-01-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20188 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85807/ Test PASSed. ---

[GitHub] spark issue #20188: [SPARK-22993][ML] Clarify HasCheckpointInterval param do...

2018-01-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20188 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark pull request #20168: SPARK-22730 Add ImageSchema support for non-integ...

2018-01-08 Thread WeichenXu123
Github user WeichenXu123 commented on a diff in the pull request: https://github.com/apache/spark/pull/20168#discussion_r160264829 --- Diff: python/pyspark/ml/image.py --- @@ -71,9 +88,30 @@ def ocvTypes(self): """ if self._ocvTypes is None: -

[GitHub] spark pull request #20168: SPARK-22730 Add ImageSchema support for non-integ...

2018-01-08 Thread WeichenXu123
Github user WeichenXu123 commented on a diff in the pull request: https://github.com/apache/spark/pull/20168#discussion_r160265175 --- Diff: python/pyspark/ml/image.py --- @@ -55,7 +72,7 @@ def imageSchema(self): """ if self._imageSchema is None:

[GitHub] spark issue #19943: [SPARK-16060][SQL] Support Vectorized ORC Reader

2018-01-08 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/19943 Thanks. I checked the as-is inline behavior. As you told, ORC nextBatch is not inlined so far while Parquet nextBatch does. I'll try to optimize that. --- ---

[GitHub] spark pull request #20168: SPARK-22730 Add ImageSchema support for non-integ...

2018-01-08 Thread WeichenXu123
Github user WeichenXu123 commented on a diff in the pull request: https://github.com/apache/spark/pull/20168#discussion_r160264533 --- Diff: python/pyspark/ml/image.py --- @@ -71,9 +88,30 @@ def ocvTypes(self): """ if self._ocvTypes is None: -

[GitHub] spark issue #20186: [SPARK-22990][Core] Fix method isFairScheduler in JobsTa...

2018-01-08 Thread vanzin
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/20186 Is it easy to add a unit test in `StagePageSuite`? Also this is the kind of stuff the compiler should be complaining about... --- ---

[GitHub] spark pull request #20168: SPARK-22730 Add ImageSchema support for non-integ...

2018-01-08 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/20168#discussion_r160271945 --- Diff: python/pyspark/ml/image.py --- @@ -71,9 +88,30 @@ def ocvTypes(self): """ if self._ocvTypes is None: -

[GitHub] spark issue #19943: [SPARK-16060][SQL] Support Vectorized ORC Reader

2018-01-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19943 **[Test build #85808 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85808/testReport)** for PR 19943 at commit [`ba03d20`](https://github.com/apache/spark/commit/b

[GitHub] spark issue #19943: [SPARK-16060][SQL] Support Vectorized ORC Reader

2018-01-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19943 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #19943: [SPARK-16060][SQL] Support Vectorized ORC Reader

2018-01-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19943 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85808/ Test FAILed. ---

[GitHub] spark issue #20151: [SPARK-22959][PYTHON] Configuration to select the module...

2018-01-08 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/20151 @rxin or @joshrosen could you guys take a quick look and see if it makes sense? --- - To unsubscribe, e-mail: reviews-unsubs

[GitHub] spark issue #20013: [SPARK-20657][core] Speed up rendering of the stages pag...

2018-01-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20013 **[Test build #85809 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85809/testReport)** for PR 20013 at commit [`44c9647`](https://github.com/apache/spark/commit/44

[GitHub] spark pull request #20189: [SPARK-22975] MetricsReporter should not throw ex...

2018-01-08 Thread mgaido91
GitHub user mgaido91 opened a pull request: https://github.com/apache/spark/pull/20189 [SPARK-22975] MetricsReporter should not throw exception when there was no progress reported ## What changes were proposed in this pull request? `MetricsReporter ` assumes that there has

[GitHub] spark issue #20189: [SPARK-22975] MetricsReporter should not throw exception...

2018-01-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20189 **[Test build #85810 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85810/testReport)** for PR 20189 at commit [`1185a51`](https://github.com/apache/spark/commit/11

[GitHub] spark issue #20013: [SPARK-20657][core] Speed up rendering of the stages pag...

2018-01-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20013 **[Test build #85806 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85806/testReport)** for PR 20013 at commit [`ed59b4d`](https://github.com/apache/spark/commit/e

[GitHub] spark issue #20013: [SPARK-20657][core] Speed up rendering of the stages pag...

2018-01-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20013 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85806/ Test PASSed. ---

[GitHub] spark issue #20013: [SPARK-20657][core] Speed up rendering of the stages pag...

2018-01-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20013 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #19943: [SPARK-16060][SQL] Support Vectorized ORC Reader

2018-01-08 Thread henrify
Github user henrify commented on the issue: https://github.com/apache/spark/pull/19943 @dongjoon-hyun Thanks. I don't think it matters if nextBatch() is inlined or not. I think what matters is 1) how the putX() etc methods calls inside the tight loops are inlined and 2) how complex th

[GitHub] spark issue #20174: [SPARK-22951][SQL] aggregate should not produce empty ro...

2018-01-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20174 **[Test build #85811 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85811/testReport)** for PR 20174 at commit [`6e164d5`](https://github.com/apache/spark/commit/6e

[GitHub] spark issue #20096: [SPARK-22908] Add kafka source and sink for continuous p...

2018-01-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20096 **[Test build #85812 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85812/testReport)** for PR 20096 at commit [`d94461a`](https://github.com/apache/spark/commit/d9

[GitHub] spark pull request #13599: [SPARK-13587] [PYSPARK] Support virtualenv in pys...

2018-01-08 Thread zjffdu
Github user zjffdu commented on a diff in the pull request: https://github.com/apache/spark/pull/13599#discussion_r160285377 --- Diff: core/src/main/scala/org/apache/spark/api/python/VirtualEnvFactory.scala --- @@ -0,0 +1,151 @@ +/* + * Licensed to the Apache Software Found

[GitHub] spark issue #20176: [SPARK-22981][SQL] Fix incorrect results of Casting Stru...

2018-01-08 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/20176 @cloud-fan ping --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@s

[GitHub] spark issue #19290: [SPARK-22063][R] Fixes lint check failures in R by lates...

2018-01-08 Thread shaneknapp
Github user shaneknapp commented on the issue: https://github.com/apache/spark/pull/19290 i filed a bug about this on the spark jira: https://issues.apache.org/jira/browse/SPARK-22996 as we're about to move all the spark builds to new ubuntu machines, w/much more up2date p

[GitHub] spark issue #19943: [SPARK-16060][SQL] Support Vectorized ORC Reader

2018-01-08 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/19943 After minimizing `nextBatch`, it becomes smaller than Parquet's `nextBatch`. But, it's inlined only some cases, but mostly not. It's not helpful. For the other technique, I'll try later. `

[GitHub] spark pull request #20190: [SPARK-22976][Core]: Cluster mode driver director...

2018-01-08 Thread RussellSpitzer
GitHub user RussellSpitzer opened a pull request: https://github.com/apache/spark/pull/20190 [SPARK-22976][Core]: Cluster mode driver directories can be removed w… …hile running ## What changes were proposed in this pull request? The clean up logic on

[GitHub] spark issue #20190: [SPARK-22976][Core]: Cluster mode driver directories can...

2018-01-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20190 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #20191: [SPARK-22997] Add additional defenses against use...

2018-01-08 Thread JoshRosen
GitHub user JoshRosen opened a pull request: https://github.com/apache/spark/pull/20191 [SPARK-22997] Add additional defenses against use of freed MemoryBlocks ## What changes were proposed in this pull request? This patch modifies Spark's `MemoryAllocator` implementations s

[GitHub] spark issue #20191: [SPARK-22997] Add additional defenses against use of fre...

2018-01-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20191 **[Test build #85813 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85813/testReport)** for PR 20191 at commit [`a7f8c07`](https://github.com/apache/spark/commit/a7

[GitHub] spark issue #19943: [SPARK-16060][SQL] Support Vectorized ORC Reader

2018-01-08 Thread henrify
Github user henrify commented on the issue: https://github.com/apache/spark/pull/19943 @dongjoon-hyun the nextBatch() is invoked 4096x less often than the main copy loops, so it doesn't matter much.. --- - To unsubs

[GitHub] spark pull request #20192: [SPARK-22994][k8s] Use a single image for all Spa...

2018-01-08 Thread vanzin
GitHub user vanzin opened a pull request: https://github.com/apache/spark/pull/20192 [SPARK-22994][k8s] Use a single image for all Spark containers. This change allows a user to submit a Spark application on kubernetes having to provide a single image, instead of one image for ea

[GitHub] spark issue #20192: [SPARK-22994][k8s] Use a single image for all Spark cont...

2018-01-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20192 **[Test build #85814 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85814/testReport)** for PR 20192 at commit [`bfad831`](https://github.com/apache/spark/commit/bf

[GitHub] spark issue #20096: [SPARK-22908] Add kafka source and sink for continuous p...

2018-01-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20096 **[Test build #85815 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85815/testReport)** for PR 20096 at commit [`94d5f50`](https://github.com/apache/spark/commit/94

[GitHub] spark pull request #20191: [SPARK-22997] Add additional defenses against use...

2018-01-08 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/20191#discussion_r160295529 --- Diff: common/unsafe/src/main/java/org/apache/spark/unsafe/memory/UnsafeMemoryAllocator.java --- @@ -38,9 +38,20 @@ public MemoryBlock allocate(long size

[GitHub] spark issue #13599: [SPARK-13587] [PYSPARK] Support virtualenv in pyspark

2018-01-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/13599 **[Test build #85817 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85817/testReport)** for PR 13599 at commit [`fb59f9a`](https://github.com/apache/spark/commit/fb

[GitHub] spark pull request #20193: [SPARK-22998][K8S] Set missing value for SPARK_MO...

2018-01-08 Thread liyinan926
GitHub user liyinan926 opened a pull request: https://github.com/apache/spark/pull/20193 [SPARK-22998][K8S] Set missing value for SPARK_MOUNTED_CLASSPATH in the executors ## What changes were proposed in this pull request? The environment variable `SPARK_MOUNTED_CLASSPATH`

[GitHub] spark issue #20193: [SPARK-22998][K8S] Set missing value for SPARK_MOUNTED_C...

2018-01-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20193 **[Test build #85816 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85816/testReport)** for PR 20193 at commit [`640e785`](https://github.com/apache/spark/commit/64

[GitHub] spark issue #20189: [SPARK-22975] MetricsReporter should not throw exception...

2018-01-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20189 **[Test build #85810 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85810/testReport)** for PR 20189 at commit [`1185a51`](https://github.com/apache/spark/commit/1

[GitHub] spark issue #20189: [SPARK-22975] MetricsReporter should not throw exception...

2018-01-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20189 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85810/ Test FAILed. ---

[GitHub] spark pull request #20191: [SPARK-22997] Add additional defenses against use...

2018-01-08 Thread JoshRosen
Github user JoshRosen commented on a diff in the pull request: https://github.com/apache/spark/pull/20191#discussion_r160296722 --- Diff: common/unsafe/src/main/java/org/apache/spark/unsafe/memory/UnsafeMemoryAllocator.java --- @@ -38,9 +38,20 @@ public MemoryBlock allocate(long s

[GitHub] spark issue #20189: [SPARK-22975] MetricsReporter should not throw exception...

2018-01-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20189 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark pull request #20191: [SPARK-22997] Add additional defenses against use...

2018-01-08 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/20191#discussion_r160296848 --- Diff: common/unsafe/src/main/java/org/apache/spark/unsafe/memory/UnsafeMemoryAllocator.java --- @@ -38,9 +38,20 @@ public MemoryBlock allocate(long size

[GitHub] spark pull request #20191: [SPARK-22997] Add additional defenses against use...

2018-01-08 Thread JoshRosen
Github user JoshRosen commented on a diff in the pull request: https://github.com/apache/spark/pull/20191#discussion_r160297024 --- Diff: common/unsafe/src/main/java/org/apache/spark/unsafe/memory/UnsafeMemoryAllocator.java --- @@ -38,9 +38,20 @@ public MemoryBlock allocate(long s

[GitHub] spark issue #20193: [SPARK-22998][K8S] Set missing value for SPARK_MOUNTED_C...

2018-01-08 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20193 **[Test build #85816 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85816/testReport)** for PR 20193 at commit [`640e785`](https://github.com/apache/spark/commit/6

[GitHub] spark issue #20193: [SPARK-22998][K8S] Set missing value for SPARK_MOUNTED_C...

2018-01-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20193 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85816/ Test PASSed. ---

[GitHub] spark issue #20193: [SPARK-22998][K8S] Set missing value for SPARK_MOUNTED_C...

2018-01-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20193 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

<    1   2   3   4   5   6   >