[GitHub] spark issue #20638: [SPARK-23417][python] Fix the build instructions supplie...

2018-02-27 Thread bersprockets
Github user bersprockets commented on the issue: https://github.com/apache/spark/pull/20638 Do we want to merge this or does it need more eyes? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For a

[GitHub] spark issue #20638: [SPARK-23417][python] Fix the build instructions supplie...

2018-02-27 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/20638 Merged to master. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews

[GitHub] spark pull request #20678: [SPARK-23380][PYTHON] Adds a conf for Arrow fallb...

2018-02-27 Thread holdenk
Github user holdenk commented on a diff in the pull request: https://github.com/apache/spark/pull/20678#discussion_r17186 --- Diff: python/pyspark/sql/dataframe.py --- @@ -1986,55 +1986,89 @@ def toPandas(self): timezone = None if self.sql_ctx

[GitHub] spark pull request #20678: [SPARK-23380][PYTHON] Adds a conf for Arrow fallb...

2018-02-27 Thread holdenk
Github user holdenk commented on a diff in the pull request: https://github.com/apache/spark/pull/20678#discussion_r17018 --- Diff: docs/sql-programming-guide.md --- @@ -1689,6 +1689,10 @@ using the call `toPandas()` and when creating a Spark DataFrame from a Pandas Da `cr

[GitHub] spark pull request #20638: [SPARK-23417][python] Fix the build instructions ...

2018-02-27 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/20638 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20638: [SPARK-23417][python] Fix the build instructions supplie...

2018-02-27 Thread bersprockets
Github user bersprockets commented on the issue: https://github.com/apache/spark/pull/20638 @HyukjinKwon Oh, thanks. I guess I learned what "approved these changes" means. --- - To unsubscribe, e-mail: reviews-unsub

[GitHub] spark issue #20522: [SPARK-23355][SQL] convertMetastore should not ignore ta...

2018-02-27 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/20522 Sure. I'll try, @gatorsmile . It'll take some time for me. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.o

[GitHub] spark issue #15670: [SPARK-18161] [Python] Allow pickle to serialize >4 GB o...

2018-02-27 Thread holdenk
Github user holdenk commented on the issue: https://github.com/apache/spark/pull/15670 So yeah if anyone has the spoons to take this over it _should_ just be fixing the merge conflicts. Feel free to ping me on the PR. --- -

[GitHub] spark issue #20046: [SPARK-22362][SQL] Add unit test for Window Aggregate Fu...

2018-02-27 Thread attilapiros
Github user attilapiros commented on the issue: https://github.com/apache/spark/pull/20046 cc @gatorsmile @hvanhovell @jiangxb1987 your review comments are applied, is there something else I should work on regarding this PR? ---

[GitHub] spark issue #20676: [SPARK-23516][CORE] It is unnecessary to transfer unroll...

2018-02-27 Thread ConeyLiu
Github user ConeyLiu commented on the issue: https://github.com/apache/spark/pull/20676 This is for compatibility reasons. The memory management also support legacy memory management (`StaticMemoryManager`). In `StaticMemoryManager`, the storage memory and unroll memory is managed sep

[GitHub] spark issue #19876: [ML][SPARK-11171][SPARK-11239] Add PMML export to Spark ...

2018-02-27 Thread holdenk
Github user holdenk commented on the issue: https://github.com/apache/spark/pull/19876 So now that it looks like 2.3 is pretty much wrapped up do folks have any thoughts? @mlnick @jkbradley @sethah ? --- - To unsubs

[GitHub] spark issue #18339: [SPARK-21094][PYTHON] Add popen_kwargs to launch_gateway

2018-02-27 Thread holdenk
Github user holdenk commented on the issue: https://github.com/apache/spark/pull/18339 Lets see what @BryanCutler thinks --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-m

[GitHub] spark pull request #20676: [SPARK-23516][CORE] It is unnecessary to transfer...

2018-02-27 Thread ConeyLiu
Github user ConeyLiu commented on a diff in the pull request: https://github.com/apache/spark/pull/20676#discussion_r171115071 --- Diff: core/src/main/scala/org/apache/spark/storage/memory/MemoryStore.scala --- @@ -246,18 +246,18 @@ private[spark] class MemoryStore( va

[GitHub] spark pull request #20676: [SPARK-23516][CORE] It is unnecessary to transfer...

2018-02-27 Thread 10110346
Github user 10110346 commented on a diff in the pull request: https://github.com/apache/spark/pull/20676#discussion_r171115245 --- Diff: core/src/main/scala/org/apache/spark/storage/memory/MemoryStore.scala --- @@ -246,18 +246,18 @@ private[spark] class MemoryStore( va

[GitHub] spark pull request #20687: [SPARK-25000][SQL] Fix complex type simplificatio...

2018-02-27 Thread henryr
GitHub user henryr opened a pull request: https://github.com/apache/spark/pull/20687 [SPARK-25000][SQL] Fix complex type simplification rules to apply to entire plan ## What changes were proposed in this pull request? Complex type simplification optimizer rules were not app

[GitHub] spark issue #20687: [SPARK-25000][SQL] Fix complex type simplification rules...

2018-02-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20687 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/1121/ Tes

[GitHub] spark issue #20687: [SPARK-25000][SQL] Fix complex type simplification rules...

2018-02-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20687 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #20687: [SPARK-25000][SQL] Fix complex type simplification rules...

2018-02-27 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20687 **[Test build #87739 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87739/testReport)** for PR 20687 at commit [`f446fa2`](https://github.com/apache/spark/commit/f4

[GitHub] spark issue #20676: [SPARK-23516][CORE] It is unnecessary to transfer unroll...

2018-02-27 Thread 10110346
Github user 10110346 commented on the issue: https://github.com/apache/spark/pull/20676 In `StaticMemoryManager`, the storage memory and unroll memory is managed separately, but, unroll memory is also storage memory, so we do not need release unroll memory really,Just need to release

[GitHub] spark pull request #20676: [SPARK-23516][CORE] It is unnecessary to transfer...

2018-02-27 Thread 10110346
Github user 10110346 commented on a diff in the pull request: https://github.com/apache/spark/pull/20676#discussion_r171116974 --- Diff: core/src/main/scala/org/apache/spark/storage/memory/MemoryStore.scala --- @@ -246,18 +246,18 @@ private[spark] class MemoryStore( va

[GitHub] spark pull request #20667: [SPARK-23508][CORE] Fix BlockmanagerId in case bl...

2018-02-27 Thread caneGuy
Github user caneGuy commented on a diff in the pull request: https://github.com/apache/spark/pull/20667#discussion_r171121778 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockManagerId.scala --- @@ -132,10 +133,15 @@ private[spark] object BlockManagerId { getCac

[GitHub] spark issue #20666: [SPARK-23448][SQL] Clarify JSON and CSV parser behavior ...

2018-02-27 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20666 **[Test build #87735 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87735/testReport)** for PR 20666 at commit [`daa326d`](https://github.com/apache/spark/commit/d

[GitHub] spark issue #20666: [SPARK-23448][SQL] Clarify JSON and CSV parser behavior ...

2018-02-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20666 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/87735/ Test PASSed. ---

[GitHub] spark issue #20666: [SPARK-23448][SQL] Clarify JSON and CSV parser behavior ...

2018-02-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20666 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #20295: [SPARK-23011] Support alternative function form with gro...

2018-02-27 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20295 **[Test build #87736 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87736/testReport)** for PR 20295 at commit [`9ed3779`](https://github.com/apache/spark/commit/9

[GitHub] spark issue #13599: [SPARK-13587] [PYSPARK] Support virtualenv in pyspark

2018-02-27 Thread zjffdu
Github user zjffdu commented on the issue: https://github.com/apache/spark/pull/13599 I am afraid I would not be present in Strata SJ, I live in Shanghai China, and may not be able to travel at time. --- - To unsub

[GitHub] spark issue #20295: [SPARK-23011] Support alternative function form with gro...

2018-02-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20295 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #20295: [SPARK-23011] Support alternative function form with gro...

2018-02-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20295 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/87736/ Test PASSed. ---

[GitHub] spark issue #13599: [SPARK-13587] [PYSPARK] Support virtualenv in pyspark

2018-02-27 Thread holdenk
Github user holdenk commented on the issue: https://github.com/apache/spark/pull/13599 Totally reasonable, how about if we did a group hangout or similar and then summarized our discussions back to the PR? It could be after Strata and I can get up early or go to bed late as needed for

[GitHub] spark issue #13599: [SPARK-13587] [PYSPARK] Support virtualenv in pyspark

2018-02-27 Thread zjffdu
Github user zjffdu commented on the issue: https://github.com/apache/spark/pull/13599 That would be awesome. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: review

[GitHub] spark pull request #20676: [SPARK-23516][CORE] It is unnecessary to transfer...

2018-02-27 Thread Ngone51
Github user Ngone51 commented on a diff in the pull request: https://github.com/apache/spark/pull/20676#discussion_r171124468 --- Diff: core/src/main/scala/org/apache/spark/storage/memory/MemoryStore.scala --- @@ -246,18 +246,18 @@ private[spark] class MemoryStore( val

[GitHub] spark issue #20666: [SPARK-23448][SQL] Clarify JSON and CSV parser behavior ...

2018-02-27 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/20666 Merged to master and branch-2.3. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands,

[GitHub] spark issue #19108: [SPARK-21898][ML] Feature parity for KolmogorovSmirnovTe...

2018-02-27 Thread WeichenXu123
Github user WeichenXu123 commented on the issue: https://github.com/apache/spark/pull/19108 @sethah Thanks! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews

[GitHub] spark pull request #20666: [SPARK-23448][SQL] Clarify JSON and CSV parser be...

2018-02-27 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/20666 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20676: [SPARK-23516][CORE] It is unnecessary to transfer unroll...

2018-02-27 Thread ConeyLiu
Github user ConeyLiu commented on the issue: https://github.com/apache/spark/pull/20676 Yeah, I see that. I'm not sure it's OK to change. But I think we should follow the interface design, not the underlying implementation. --- --

[GitHub] spark issue #20676: [SPARK-23516][CORE] It is unnecessary to transfer unroll...

2018-02-27 Thread Ngone51
Github user Ngone51 commented on the issue: https://github.com/apache/spark/pull/20676 Hi, @ConeyLiu. I don't think it's about compatibility. Because both ```StaticMemoryManager``` and ```UnifiedMemoryManager``` call the same function for release unroll memory, which is ```releaseUnro

[GitHub] spark pull request #20688: [SPARK-23096][SS] Migrate rate source to V2

2018-02-27 Thread jerryshao
GitHub user jerryshao opened a pull request: https://github.com/apache/spark/pull/20688 [SPARK-23096][SS] Migrate rate source to V2 ## What changes were proposed in this pull request? This PR migrate micro batch rate source to V2 API and rewrite UTs to suite V2 test.

[GitHub] spark issue #20688: [SPARK-23096][SS] Migrate rate source to V2

2018-02-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20688 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #20688: [SPARK-23096][SS] Migrate rate source to V2

2018-02-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20688 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/1122/ Tes

[GitHub] spark issue #20688: [SPARK-23096][SS] Migrate rate source to V2

2018-02-27 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20688 **[Test build #87740 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87740/testReport)** for PR 20688 at commit [`538223e`](https://github.com/apache/spark/commit/53

[GitHub] spark issue #20666: [SPARK-23448][SQL] Clarify JSON and CSV parser behavior ...

2018-02-27 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/20666 Thanks @HyukjinKwon @cloud-fan! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark pull request #20676: [SPARK-23516][CORE] It is unnecessary to transfer...

2018-02-27 Thread Ngone51
Github user Ngone51 commented on a diff in the pull request: https://github.com/apache/spark/pull/20676#discussion_r171128021 --- Diff: core/src/main/scala/org/apache/spark/storage/memory/MemoryStore.scala --- @@ -246,18 +246,18 @@ private[spark] class MemoryStore( val

[GitHub] spark issue #20685: [SPARK-23524] Big local shuffle blocks should not be che...

2018-02-27 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20685 **[Test build #87741 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87741/testReport)** for PR 20685 at commit [`110c851`](https://github.com/apache/spark/commit/11

[GitHub] spark issue #20639: [SPARK-23288][SS] Fix output metrics with parquet sink

2018-02-27 Thread tdas
Github user tdas commented on the issue: https://github.com/apache/spark/pull/20639 @zsxwing as well. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@s

[GitHub] spark issue #20685: [SPARK-23524] Big local shuffle blocks should not be che...

2018-02-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20685 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/1123/ Tes

[GitHub] spark issue #20685: [SPARK-23524] Big local shuffle blocks should not be che...

2018-02-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20685 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #20639: [SPARK-23288][SS] Fix output metrics with parquet sink

2018-02-27 Thread tdas
Github user tdas commented on the issue: https://github.com/apache/spark/pull/20639 this is ok to test --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...

[GitHub] spark issue #18906: [SPARK-21692][PYSPARK][SQL] Add nullability support to P...

2018-02-27 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18906 **[Test build #87742 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87742/testReport)** for PR 18906 at commit [`e94960c`](https://github.com/apache/spark/commit/e9

[GitHub] spark issue #18906: [SPARK-21692][PYSPARK][SQL] Add nullability support to P...

2018-02-27 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18906 **[Test build #87742 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87742/testReport)** for PR 18906 at commit [`e94960c`](https://github.com/apache/spark/commit/e

[GitHub] spark issue #18906: [SPARK-21692][PYSPARK][SQL] Add nullability support to P...

2018-02-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18906 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/87742/ Test FAILed. ---

[GitHub] spark issue #18906: [SPARK-21692][PYSPARK][SQL] Add nullability support to P...

2018-02-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18906 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #20681: [SPARK-23518][SQL] Avoid metastore access when the users...

2018-02-27 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20681 **[Test build #87737 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87737/testReport)** for PR 20681 at commit [`21c3374`](https://github.com/apache/spark/commit/2

[GitHub] spark issue #20681: [SPARK-23518][SQL] Avoid metastore access when the users...

2018-02-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20681 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #20681: [SPARK-23518][SQL] Avoid metastore access when the users...

2018-02-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20681 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/87737/ Test PASSed. ---

[GitHub] spark issue #20679: [SPARK-23514] Use SessionState.newHadoopConf() to propag...

2018-02-27 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20679 **[Test build #87738 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87738/testReport)** for PR 20679 at commit [`b37f24f`](https://github.com/apache/spark/commit/b

[GitHub] spark issue #20679: [SPARK-23514] Use SessionState.newHadoopConf() to propag...

2018-02-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20679 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #20679: [SPARK-23514] Use SessionState.newHadoopConf() to propag...

2018-02-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20679 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/87738/ Test FAILed. ---

[GitHub] spark issue #18906: [SPARK-21692][PYSPARK][SQL] Add nullability support to P...

2018-02-27 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18906 **[Test build #87743 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87743/testReport)** for PR 18906 at commit [`e6e6dbf`](https://github.com/apache/spark/commit/e6

[GitHub] spark issue #20687: [SPARK-25000][SQL] Fix complex type simplification rules...

2018-02-27 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20687 **[Test build #87739 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87739/testReport)** for PR 20687 at commit [`f446fa2`](https://github.com/apache/spark/commit/f

[GitHub] spark issue #20687: [SPARK-25000][SQL] Fix complex type simplification rules...

2018-02-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20687 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #20687: [SPARK-25000][SQL] Fix complex type simplification rules...

2018-02-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20687 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/87739/ Test FAILed. ---

[GitHub] spark issue #20676: [SPARK-23516][CORE] It is unnecessary to transfer unroll...

2018-02-27 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/20676 I don't think unroll memory is storage memory from the interface. It's only true for unified memory manager. I'm -1 on this change unless you can convince the community to remove the static memory

[GitHub] spark issue #20667: [SPARK-23508][CORE] Fix BlockmanagerId in case blockMana...

2018-02-27 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/20667 LGTM --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apa

[GitHub] spark issue #20667: [SPARK-23508][CORE] Fix BlockmanagerId in case blockMana...

2018-02-27 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/20667 ok to test --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spa

[GitHub] spark issue #13599: [SPARK-13587] [PYSPARK] Support virtualenv in pyspark

2018-02-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/13599 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #20667: [SPARK-23508][CORE] Fix BlockmanagerId in case blockMana...

2018-02-27 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20667 **[Test build #87744 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87744/testReport)** for PR 20667 at commit [`3379899`](https://github.com/apache/spark/commit/33

[GitHub] spark issue #20667: [SPARK-23508][CORE] Fix BlockmanagerId in case blockMana...

2018-02-27 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/20667 add to whitelist --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h.

[GitHub] spark issue #13599: [SPARK-13587] [PYSPARK] Support virtualenv in pyspark

2018-02-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/13599 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/1124/ Tes

[GitHub] spark issue #13599: [SPARK-13587] [PYSPARK] Support virtualenv in pyspark

2018-02-27 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/13599 **[Test build #87745 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87745/testReport)** for PR 13599 at commit [`3da68c7`](https://github.com/apache/spark/commit/3d

[GitHub] spark issue #20667: [SPARK-23508][CORE] Fix BlockmanagerId in case blockMana...

2018-02-27 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20667 **[Test build #87746 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87746/testReport)** for PR 20667 at commit [`3379899`](https://github.com/apache/spark/commit/33

[GitHub] spark pull request #20667: [SPARK-23508][CORE] Fix BlockmanagerId in case bl...

2018-02-27 Thread jiangxb1987
Github user jiangxb1987 commented on a diff in the pull request: https://github.com/apache/spark/pull/20667#discussion_r171136135 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockManagerId.scala --- @@ -132,10 +133,17 @@ private[spark] object BlockManagerId { ge

[GitHub] spark issue #13599: [SPARK-13587] [PYSPARK] Support virtualenv in pyspark

2018-02-27 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/13599 **[Test build #87745 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87745/testReport)** for PR 13599 at commit [`3da68c7`](https://github.com/apache/spark/commit/3

[GitHub] spark issue #13599: [SPARK-13587] [PYSPARK] Support virtualenv in pyspark

2018-02-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/13599 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #13599: [SPARK-13587] [PYSPARK] Support virtualenv in pyspark

2018-02-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/13599 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/87745/ Test FAILed. ---

[GitHub] spark issue #20681: [SPARK-23518][SQL] Avoid metastore access when the users...

2018-02-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20681 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/1125/ Tes

[GitHub] spark issue #20681: [SPARK-23518][SQL] Avoid metastore access when the users...

2018-02-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20681 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #20681: [SPARK-23518][SQL] Avoid metastore access when the users...

2018-02-27 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20681 **[Test build #87747 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87747/testReport)** for PR 20681 at commit [`999f86f`](https://github.com/apache/spark/commit/99

[GitHub] spark issue #20667: [SPARK-23508][CORE] Fix BlockmanagerId in case blockMana...

2018-02-27 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20667 **[Test build #87748 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87748/testReport)** for PR 20667 at commit [`bf79f4d`](https://github.com/apache/spark/commit/bf

[GitHub] spark pull request #20678: [SPARK-23380][PYTHON] Adds a conf for Arrow fallb...

2018-02-27 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/20678#discussion_r171138898 --- Diff: python/pyspark/sql/dataframe.py --- @@ -1986,55 +1986,89 @@ def toPandas(self): timezone = None if self.sql

[GitHub] spark pull request #20678: [SPARK-23380][PYTHON] Adds a conf for Arrow fallb...

2018-02-27 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/20678#discussion_r171139748 --- Diff: docs/sql-programming-guide.md --- @@ -1689,6 +1689,10 @@ using the call `toPandas()` and when creating a Spark DataFrame from a Pandas Da

[GitHub] spark issue #20678: [SPARK-23380][PYTHON] Adds a conf for Arrow fallback in ...

2018-02-27 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/20678 Will try to clean up soon. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark issue #20688: [SPARK-23096][SS] Migrate rate source to V2

2018-02-27 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20688 **[Test build #87740 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87740/testReport)** for PR 20688 at commit [`538223e`](https://github.com/apache/spark/commit/5

[GitHub] spark issue #20688: [SPARK-23096][SS] Migrate rate source to V2

2018-02-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20688 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #20688: [SPARK-23096][SS] Migrate rate source to V2

2018-02-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20688 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/87740/ Test FAILed. ---

[GitHub] spark issue #13599: [SPARK-13587] [PYSPARK] Support virtualenv in pyspark

2018-02-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/13599 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #13599: [SPARK-13587] [PYSPARK] Support virtualenv in pyspark

2018-02-27 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/13599 **[Test build #87749 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87749/testReport)** for PR 13599 at commit [`86484d6`](https://github.com/apache/spark/commit/86

[GitHub] spark issue #13599: [SPARK-13587] [PYSPARK] Support virtualenv in pyspark

2018-02-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/13599 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/1126/ Tes

[GitHub] spark issue #20667: [SPARK-23508][CORE] Fix BlockmanagerId in case blockMana...

2018-02-27 Thread Ngone51
Github user Ngone51 commented on the issue: https://github.com/apache/spark/pull/20667 Hi, @caneGuy , sorry for my previous comment as I mixed up ```BlockId``` with ```BlockManagerId```, and leave some wrong comments. And thanks for your reply. Back to now, I have the same qu

[GitHub] spark issue #20647: [SPARK-23303][SQL] improve the explain result for data s...

2018-02-27 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20647 **[Test build #87750 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87750/testReport)** for PR 20647 at commit [`c5af52e`](https://github.com/apache/spark/commit/c5

[GitHub] spark issue #20647: [SPARK-23303][SQL] improve the explain result for data s...

2018-02-27 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/20647 Hi @rdblue , I've opened https://issues.apache.org/jira/browse/SPARK-23531 to include the type info. I'd like to do it later as it's a general problem in Spark SQL and many plans need to be update

[GitHub] spark issue #20647: [SPARK-23303][SQL] improve the explain result for data s...

2018-02-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20647 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/1127/ Tes

[GitHub] spark issue #20647: [SPARK-23303][SQL] improve the explain result for data s...

2018-02-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20647 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #13599: [SPARK-13587] [PYSPARK] Support virtualenv in pyspark

2018-02-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/13599 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #13599: [SPARK-13587] [PYSPARK] Support virtualenv in pyspark

2018-02-27 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/13599 **[Test build #87749 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87749/testReport)** for PR 13599 at commit [`86484d6`](https://github.com/apache/spark/commit/8

[GitHub] spark issue #13599: [SPARK-13587] [PYSPARK] Support virtualenv in pyspark

2018-02-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/13599 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/87749/ Test FAILed. ---

[GitHub] spark pull request #20647: [SPARK-23303][SQL] improve the explain result for...

2018-02-27 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/20647#discussion_r171143887 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Relation.scala --- @@ -107,19 +104,36 @@ case class DataSourc

[GitHub] spark issue #20667: [SPARK-23508][CORE] Fix BlockmanagerId in case blockMana...

2018-02-27 Thread jiangxb1987
Github user jiangxb1987 commented on the issue: https://github.com/apache/spark/pull/20667 In case the same `BlockManagerId` being created multiple times, this cache will ensure we always use the first one that is created, which make it possible for the rest `BlockManagerId` instances

[GitHub] spark pull request #20647: [SPARK-23303][SQL] improve the explain result for...

2018-02-27 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/20647#discussion_r171143986 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2StringFormat.scala --- @@ -0,0 +1,94 @@ +/* + * Licen

[GitHub] spark issue #20667: [SPARK-23508][CORE] Fix BlockmanagerId in case blockMana...

2018-02-27 Thread Ngone51
Github user Ngone51 commented on the issue: https://github.com/apache/spark/pull/20667 Hi, @jiangxb1987 , thanks for your kindly explanation. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For add

[GitHub] spark issue #20685: [SPARK-23524] Big local shuffle blocks should not be che...

2018-02-27 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20685 **[Test build #87741 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87741/testReport)** for PR 20685 at commit [`110c851`](https://github.com/apache/spark/commit/1

<    1   2   3   4   5   >