Re: [SPARK-2.0][SQL] UDF containing non-serializable object does not work as expected

Yes, it is. You can define a udf like that. Basically, it's a udf Int => Int which is a closure contains a non serializable object. The latter should cause Task not serializable exception. Hao On Mon, Aug 8, 2016 at 5:08 AM, Muthu Jayakumar wrote: > Hello Hao Ren, > > Doesn't the code... > > va

Spark 2.0 sql module empty columns in result over parquet tables

I run into this very strange issue. After loading parquet tables and trying to run an sql query with the sql module the results are not correct with Spark 2.0 although over the same exactly dataset Spark 1.6 results are correct. With Textfiles however both versions of Spark work as expected. I have

RE: [SPARK-2.0][SQL] UDF containing non-serializable object does not work as expected

But does the “notSer” object have to be serialized? The object is immutable by the definition of A, so the only thing that needs to be serialized is the (immutable) Int value? And Ints are serializable? Just thinking out loud Simon Scott Research Developer @ viavisolutions.com From: Hao Ren [

Re: Kafka Support new topic subscriptions without requiring restart of the streaming context

The Kafka 0.10 support in spark 2.0 allows for pattern based topic subscription On Aug 8, 2016 1:12 AM, "r7raul1...@163.com" wrote: > How to add new topic to kafka without requiring restart of the streaming > context? > > -- > r7raul1...@163.com >

Welcoming Felix Cheung as a committer

Hi all, The PMC recently voted to add Felix Cheung as a committer. Felix has been a major contributor to SparkR and we're excited to have him join officially. Congrats and welcome, Felix! Matei - To unsubscribe e-mail: dev-unsu

Re: Welcoming Felix Cheung as a committer

Congratulations, Felix. On Mon, Aug 8, 2016 at 11:15 AM, Matei Zaharia wrote: > Hi all, > > The PMC recently voted to add Felix Cheung as a committer. Felix has been > a major contributor to SparkR and we're excited to have him join > officially. Congrats and welcome, Felix! > > Matei >

Re: Welcoming Felix Cheung as a committer

Congratulation, Felix! Bests, Dongjoon. On Monday, August 8, 2016, Ted Yu wrote: > Congratulations, Felix. > > On Mon, Aug 8, 2016 at 11:15 AM, Matei Zaharia > wrote: > >> Hi all, >> >> The PMC recently voted to add Felix Cheung as a committer. Felix has been >> a major contributor to SparkR a

Re: Welcoming Felix Cheung as a committer

👏👏👏 Do we now have 2 SparkR-focused committers (Shivaram + Felix)? Or are there more? Nick On Mon, Aug 8, 2016 at 2:17 PM Dongjoon Hyun wrote: > Congratulation, Felix! > > Bests, > Dongjoon. > > > On Monday, August 8, 2016, Ted Yu wrote: > >> Congratulations, Felix. >> >> On Mon, Aug 8, 2016

Scaling partitioned Hive table support

Hello, I'd like to propose a modification in the way Hive table partition metadata are loaded and cached. Currently, when a user reads from a partitioned Hive table whose metadata are not cached (and for which Hive table conversion is enabled and supported), all partition metadata is fetched fr

Re: Welcoming Felix Cheung as a committer

Congratulations , Felix! > On Aug 8, 2016, at 11:15 AM, Ted Yu wrote: > > Congratulations, Felix. > > On Mon, Aug 8, 2016 at 11:15 AM, Matei Zaharia > wrote: > Hi all, > > The PMC recently voted to add Felix Cheung as a committer. Felix has been a > major co

Re: Welcoming Felix Cheung as a committer

Congrats Felix! Tim On Mon, Aug 8, 2016 at 11:15 AM, Matei Zaharia wrote: > Hi all, > > The PMC recently voted to add Felix Cheung as a committer. Felix has been a > major contributor to SparkR and we're excited to have him join officially. > Congrats and welcome, Felix! > > Matei > --

Re: Welcoming Felix Cheung as a committer

Congrats Felix! Tarun On Tue, Aug 9, 2016 at 12:57 AM, Timothy Chen wrote: > Congrats Felix! > > Tim > > On Mon, Aug 8, 2016 at 11:15 AM, Matei Zaharia > wrote: > > Hi all, > > > > The PMC recently voted to add Felix Cheung as a committer. Felix has > been a major contributor to SparkR and we'r

Re: Scaling partitioned Hive table support

I like the former approach -- it seems more generally applicable to other catalogs and IIUC would let you defer pruning until execution time. Pruning is work that should be done by the catalog anyways, as is the case when querying over an (unconverted) hive table. You might also want to look at ht

Re: [SPARK-2.0][SQL] UDF containing non-serializable object does not work as expected

That is unfortunately the way how Scala compiler captures (and defines) closures. Nothing is really final in the JVM. You can always use reflection or unsafe to modify the value of fields. On Mon, Aug 8, 2016 at 8:16 PM, Simon Scott wrote: > But does the “notSer” object have to be serialized? >

Re: Scaling partitioned Hive table support

Hi Eric, Thanks for your feedback. I'm rebasing my code for the first approach on a more recent Spark master and am resolving some conflicts. I'll have a better understanding of the relationship to your PR once my rebase is complete. Cheers, Michael > On Aug 8, 2016, at 12:51 PM, Eric Liang

Re: [SPARK-2.0][SQL] UDF containing non-serializable object does not work as expected

@Reynold Some questions to make things clear: 1. As nothing is really final in the JVM, is the generated code during the execution of `df.show()` different from the one of `df.filter($"key" === 2).show()` in my snippet ? 2. When `df.show()` is being executed, it seems that the 'notSer' object is

Re: Welcoming Felix Cheung as a committer

Congrats Felix! On Mon, Aug 8, 2016 at 2:28 PM, Tarun Kumar wrote: > Congrats Felix! > > Tarun > > On Tue, Aug 9, 2016 at 12:57 AM, Timothy Chen wrote: > >> Congrats Felix! >> >> Tim >> >> On Mon, Aug 8, 2016 at 11:15 AM, Matei Zaharia >> wrote: >> > Hi all, >> > >> > The PMC recently voted to

Re: [SPARK-2.0][SQL] UDF containing non-serializable object does not work as expected

The show thing was the result of an optimization that short-circuited any real Spark computation when the input is a local collection, and the result was simply the first few rows. That's why it completed without serializing anything. It is somewhat inconsistent. One way to eliminate the inconsist

Re: Welcoming Felix Cheung as a committer

Congrats Felix! On Mon, Aug 8, 2016 at 11:57 PM, dhruve ashar wrote: > Congrats Felix! > > On Mon, Aug 8, 2016 at 2:28 PM, Tarun Kumar wrote: > >> Congrats Felix! >> >> Tarun >> >> On Tue, Aug 9, 2016 at 12:57 AM, Timothy Chen wrote: >> >>> Congrats Felix! >>> >>> Tim >>> >>> On Mon, Aug 8, 20

Re: Welcoming Felix Cheung as a committer

Congrats Felix! 2016-08-08 15:04 GMT-07:00 Herman van Hövell tot Westerflier : > Congrats Felix! > > On Mon, Aug 8, 2016 at 11:57 PM, dhruve ashar wrote: >> >> Congrats Felix! >> >> On Mon, Aug 8, 2016 at 2:28 PM, Tarun Kumar wrote: >>> >>> Congrats Felix! >>> >>> Tarun >>> >>> On Tue, Aug 9, 20

SASL Support

I was checking if RPC calls can be encrypted and I saw here that the docs here (*http://spark.apache.org/docs/latest/configuration.html ) *say that SASL encryption is "currently only supported by the block transfer service." However, it seems

Re: SASL Support

Please send a pull request to update the doc. Thanks. On Tue, Aug 9, 2016 at 6:48 AM, Michael Gummelt wrote: > I was checking if RPC calls can be encrypted and I saw here that the docs > here (*http://spark.apache.org/docs/latest/configuration.html >

Re: Welcoming Felix Cheung as a committer

Congratulations! 2016-08-09 7:47 GMT+09:00 Xiao Li : > Congrats Felix! > > 2016-08-08 15:04 GMT-07:00 Herman van Hövell tot Westerflier > : > > Congrats Felix! > > > > On Mon, Aug 8, 2016 at 11:57 PM, dhruve ashar > wrote: > >> > >> Congrats Felix! > >> > >> On Mon, Aug 8, 2016 at 2:28 PM, Tarun

Re: Welcoming Felix Cheung as a committer

Congrats Felix! On Tue, Aug 9, 2016 at 8:49 AM, Hyukjin Kwon wrote: > Congratulations! > > 2016-08-09 7:47 GMT+09:00 Xiao Li : > >> Congrats Felix! >> >> 2016-08-08 15:04 GMT-07:00 Herman van Hövell tot Westerflier >> : >> > Congrats Felix! >> > >> > On Mon, Aug 8, 2016 at 11:57 PM, dhruve ashar

Re: Welcoming Felix Cheung as a committer

Congrats Felix! On Mon, Aug 8, 2016, 18:14 Jeff Zhang wrote: > Congrats Felix! > > On Tue, Aug 9, 2016 at 8:49 AM, Hyukjin Kwon wrote: > >> Congratulations! >> >> 2016-08-09 7:47 GMT+09:00 Xiao Li : >> >>> Congrats Felix! >>> >>> 2016-08-08 15:04 GMT-07:00 Herman van Hövell tot Westerflier >>>

Re: Source API requires unbounded distributed storage?

Created SPARK-16963 to cover this issue. Fred On Thu, Aug 4, 2016 at 4:52 PM, Michael Armbrust wrote: > Yeah, this API is in the private execution package because we are planning > to continue to iterate on it. Today, we will only ever go back one batch, > though that might change in the futur

Re: Welcoming Felix Cheung as a committer

Congrats Felix! 2016-08-08 18:21 GMT-07:00 Kai Jiang : > Congrats Felix! > > On Mon, Aug 8, 2016, 18:14 Jeff Zhang wrote: > >> Congrats Felix! >> >> On Tue, Aug 9, 2016 at 8:49 AM, Hyukjin Kwon wrote: >> >>> Congratulations! >>> >>> 2016-08-09 7:47 GMT+09:00 Xiao Li : >>> Congrats Felix! >

Re: Welcoming Felix Cheung as a committer

Thank you! Looking forward to work with you all! On Mon, Aug 8, 2016 at 7:41 PM -0700, "Yanbo Liang" mailto:yblia...@gmail.com>> wrote: Congrats Felix! 2016-08-08 18:21 GMT-07:00 Kai Jiang mailto:jiang...@gmail.com>>: Congrats Felix! On Mon, Aug 8, 2016, 18:14 Jeff Zhang mailto:zjf...@g

Re: Welcoming Felix Cheung as a committer

Awesome - congrats Felix! On Mon, Aug 8, 2016 at 9:44 PM Felix Cheung wrote: > Thank you! > Looking forward to work with you all! > > > > > > On Mon, Aug 8, 2016 at 7:41 PM -0700, "Yanbo Liang" > wrote: > > Congrats Felix! > > 2016-08-08 18:21 GMT-07:00 Kai Jiang : > >> Congrats Felix! >> >> On