Re: Strange behavior with 'not' and filter pushdown

Oh, Thanks for checking! On Tue, Feb 14, 2017 at 12:32 PM, Xiao Li wrote: > https://github.com/apache/spark/pull/16894 > > Already backported to Spark 2.0 > > Thanks! > > Xiao > > 2017-02-13 17:41 GMT-08:00 Takeshi Yamamuro : > >> cc: xiao >> >> IIUC

Re: Strange behavior with 'not' and filter pushdown

https://github.com/apache/spark/pull/16894 Already backported to Spark 2.0 Thanks! Xiao 2017-02-13 17:41 GMT-08:00 Takeshi Yamamuro : > cc: xiao > > IIUC a xiao's commit below fixed this issue in master. >

Re: welcoming Takuya Ueshin as a new Apache Spark committer

Thank you very much everyone! I really look forward to working with you! On Tue, Feb 14, 2017 at 9:47 AM, Yanbo Liang wrote: > Congratulations! > > On Mon, Feb 13, 2017 at 3:29 PM, Kazuaki Ishizaki > wrote: > >> Congrats! >> >> Kazuaki Ishizaki >> >>

Re: Strange behavior with 'not' and filter pushdown

cc: xiao IIUC a xiao's commit below fixed this issue in master. https://github.com/apache/spark/commit/2eb093decb5e87a1ea71bbaa28092876a8c84996 Is this fix worth backporting to the v2.0 branch? I checked I could reproduce there: --- scala> Seq((1, "a"), (2, "b"), (3, null)).toDF("c0",

Re: Request for comments: Java 7 removal

I think the biggest concern is enterprise users/operators who do not have the authority or access to upgrade hadoop/yarn clusters to java8. As a reference point, apparently CDH 5.3

Re: welcoming Takuya Ueshin as a new Apache Spark committer

Congratulations! On Mon, Feb 13, 2017 at 3:29 PM, Kazuaki Ishizaki wrote: > Congrats! > > Kazuaki Ishizaki > > > > From:Reynold Xin > To:"dev@spark.apache.org" > Date:2017/02/14 04:18 > Subject:

Re: welcoming Takuya Ueshin as a new Apache Spark committer

Congrats! Kazuaki Ishizaki From: Reynold Xin To: "dev@spark.apache.org" Date: 2017/02/14 04:18 Subject:welcoming Takuya Ueshin as a new Apache Spark committer Hi all, Takuya-san has recently been elected an Apache Spark

Re: welcoming Takuya Ueshin as a new Apache Spark committer

Congrats! Asher Krim Senior Software Engineer On Mon, Feb 13, 2017 at 6:24 PM, Kousuke Saruta wrote: > Congratulations, Takuya! > > - Kousuke > On 2017/02/14 7:38, Herman van Hövell tot Westerflier wrote: > > Congrats Takuya! > > On Mon, Feb 13, 2017 at 11:27 PM,

Re: welcoming Takuya Ueshin as a new Apache Spark committer

Congratulations, Takuya! - Kousuke On 2017/02/14 7:38, Herman van Hövell tot Westerflier wrote: Congrats Takuya! On Mon, Feb 13, 2017 at 11:27 PM, Neelesh Salian > wrote: Congratulations, Takuya! On Mon, Feb 13, 2017 at

Re: [PYTHON][DISCUSS] Moving to cloudpickle and or Py4J as a dependencies?

It's a good question. Py4J seems to have been updated 5 times in 2016 and is a bit involved (from a review point of view verifying the zip file contents is somewhat tedious). cloudpickle is a bit difficult to tell since we can have changes to cloudpickle which aren't correctly tagged as

Re: [PYTHON][DISCUSS] Moving to cloudpickle and or Py4J as a dependencies?

With any dependency update (or refactoring of existing code), I always ask this question: what's the benefit? In this case it looks like the benefit is to reduce efforts in backports. Do you know how often we needed to do those? On Tue, Feb 14, 2017 at 12:01 AM, Holden Karau

[PYTHON][DISCUSS] Moving to cloudpickle and or Py4J as a dependencies?

Hi PySpark Developers, Cloudpickle is a core part of PySpark, and is originally copied from (and improved from) picloud. Since then other projects have found cloudpickle useful and a fork of cloudpickle was created and is now maintained as its own

Re: welcoming Takuya Ueshin as a new Apache Spark committer

Congrats Takuya! On Mon, Feb 13, 2017 at 11:27 PM, Neelesh Salian wrote: > Congratulations, Takuya! > > On Mon, Feb 13, 2017 at 11:16 AM, Reynold Xin wrote: > >> Hi all, >> >> Takuya-san has recently been elected an Apache Spark committer. He's

Re: welcoming Takuya Ueshin as a new Apache Spark committer

Congratulations, Takuya! On Mon, Feb 13, 2017 at 11:16 AM, Reynold Xin wrote: > Hi all, > > Takuya-san has recently been elected an Apache Spark committer. He's been > active in the SQL area and writes very small, surgical patches that are > high quality. Please join me in

Re: welcoming Takuya Ueshin as a new Apache Spark committer

Congrats Takuya! On Mon, Feb 13, 2017 at 2:17 PM, Dilip Biswal wrote: > Congratulations, Takuya! > > Regards, > Dilip Biswal > Tel: 408-463-4980 <(408)%20463-4980> > dbis...@us.ibm.com > > > > - Original message - > From: Takeshi Yamamuro >

Re: welcoming Takuya Ueshin as a new Apache Spark committer

Congratulations, Takuya!   Regards,Dilip BiswalTel: 408-463-4980dbis...@us.ibm.com     - Original message -From: Takeshi Yamamuro To: dev Cc:Subject: Re: welcoming Takuya Ueshin as a new Apache Spark committerDate: Mon, Feb 13, 2017 2:14 PM 

Re: welcoming Takuya Ueshin as a new Apache Spark committer

congrats! On Tue, Feb 14, 2017 at 6:05 AM, Sam Elamin wrote: > Congrats Takuya-san! Clearly well deserved! Well done :) > > On Mon, Feb 13, 2017 at 9:02 PM, Maciej Szymkiewicz < > mszymkiew...@gmail.com> wrote: > >> Congratulations! >> >> >> On 02/13/2017 08:16 PM,

Re: welcoming Takuya Ueshin as a new Apache Spark committer

Congrats Takuya-san! Clearly well deserved! Well done :) On Mon, Feb 13, 2017 at 9:02 PM, Maciej Szymkiewicz wrote: > Congratulations! > > > On 02/13/2017 08:16 PM, Reynold Xin wrote: > > Hi all, > > > > Takuya-san has recently been elected an Apache Spark committer.

Re: welcoming Takuya Ueshin as a new Apache Spark committer

Congratulations! On 02/13/2017 08:16 PM, Reynold Xin wrote: > Hi all, > > Takuya-san has recently been elected an Apache Spark committer. He's > been active in the SQL area and writes very small, surgical patches > that are high quality. Please join me in congratulating Takuya-san! >

Re: welcoming Takuya Ueshin as a new Apache Spark committer

Congratulations, Takuya!  On Mon, Feb 13, 2017 at 2:34 PM Felix Cheung wrote: > Congratulations! > > > -- > *From:* Xuefu Zhang > *Sent:* Monday, February 13, 2017 11:29:12 AM > *To:* Xiao Li > *Cc:* Holden Karau; Reynold

Re: welcoming Takuya Ueshin as a new Apache Spark committer

Congratulations! From: Xuefu Zhang Sent: Monday, February 13, 2017 11:29:12 AM To: Xiao Li Cc: Holden Karau; Reynold Xin; dev@spark.apache.org Subject: Re: welcoming Takuya Ueshin as a new Apache Spark committer Congratulations, Takuya! --Xuefu

Re: welcoming Takuya Ueshin as a new Apache Spark committer

Congratulations, Takuya! --Xuefu On Mon, Feb 13, 2017 at 11:25 AM, Xiao Li wrote: > Congratulations, Takuya! > > Xiao > > 2017-02-13 11:24 GMT-08:00 Holden Karau : > >> Congratulations Takuya-san :D! >> >> On Mon, Feb 13, 2017 at 11:16 AM, Reynold

welcoming Takuya Ueshin as a new Apache Spark committer

Hi all, Takuya-san has recently been elected an Apache Spark committer. He's been active in the SQL area and writes very small, surgical patches that are high quality. Please join me in congratulating Takuya-san!

Re: Add hive-site.xml at runtime

Shivam, We add hive-site.xml at runtime. We use --driver-class-path to add it to the driver and --jars to add it for the executors. rb On Sun, Feb 12, 2017 at 9:10 AM, Shivam Sharma <28shivamsha...@gmail.com> wrote: > Hi, > > I have multiple hive configurations(hive-site.xml) and because of

Re: Spark Improvement Proposals

Here's a new draft that incorporated most of the feedback: https://docs.google.com/document/d/1-Zdi_W-wtuxS9hTK0P9qb2x-nRanvXmnZ7SUi4qMljg/edit# I added a specific role for SPIP Author and another one for SPIP Shepherd. On Sat, Feb 11, 2017 at 6:13 PM, Xiao Li wrote: >

Re: Executors exceed maximum memory defined with `--executor-memory` in Spark 2.1.0

I've filed a JIRA about this problem. https://issues.apache.org/jira/browse/SPARK-19532 I've tried to set `spark.speculation` to `false`, but the off-heap also exceed about 10G after triggering a FullGC to the Executor