Re: Welcoming Felix Cheung as a committer

Awesome - congrats Felix! On Mon, Aug 8, 2016 at 9:44 PM Felix Cheung wrote: > Thank you! > Looking forward to work with you all! > > > > > > On Mon, Aug 8, 2016 at 7:41 PM -0700, "Yanbo Liang" > wrote: > > Congrats Felix! > > 2016-08-08 18:21

Re: Welcoming Felix Cheung as a committer

Congrats Felix! 2016-08-08 18:21 GMT-07:00 Kai Jiang : > Congrats Felix! > > On Mon, Aug 8, 2016, 18:14 Jeff Zhang wrote: > >> Congrats Felix! >> >> On Tue, Aug 9, 2016 at 8:49 AM, Hyukjin Kwon wrote: >> >>> Congratulations! >>> >>>

Re: Source API requires unbounded distributed storage?

Created SPARK-16963 to cover this issue. Fred On Thu, Aug 4, 2016 at 4:52 PM, Michael Armbrust wrote: > Yeah, this API is in the private execution package because we are planning > to continue to iterate on it. Today, we will only ever go back one batch, > though that

Re: Welcoming Felix Cheung as a committer

Congrats Felix! On Mon, Aug 8, 2016, 18:14 Jeff Zhang wrote: > Congrats Felix! > > On Tue, Aug 9, 2016 at 8:49 AM, Hyukjin Kwon wrote: > >> Congratulations! >> >> 2016-08-09 7:47 GMT+09:00 Xiao Li : >> >>> Congrats Felix! >>> >>>

Re: Welcoming Felix Cheung as a committer

Congratulations! 2016-08-09 7:47 GMT+09:00 Xiao Li : > Congrats Felix! > > 2016-08-08 15:04 GMT-07:00 Herman van Hövell tot Westerflier > : > > Congrats Felix! > > > > On Mon, Aug 8, 2016 at 11:57 PM, dhruve ashar > wrote:

Re: SASL Support

Please send a pull request to update the doc. Thanks. On Tue, Aug 9, 2016 at 6:48 AM, Michael Gummelt wrote: > I was checking if RPC calls can be encrypted and I saw here that the docs > here (*http://spark.apache.org/docs/latest/configuration.html >

SASL Support

I was checking if RPC calls can be encrypted and I saw here that the docs here (*http://spark.apache.org/docs/latest/configuration.html ) *say that SASL encryption is "currently only supported by the block transfer service." However, it

Re: Welcoming Felix Cheung as a committer

Congrats Felix! 2016-08-08 15:04 GMT-07:00 Herman van Hövell tot Westerflier : > Congrats Felix! > > On Mon, Aug 8, 2016 at 11:57 PM, dhruve ashar wrote: >> >> Congrats Felix! >> >> On Mon, Aug 8, 2016 at 2:28 PM, Tarun Kumar

Re: Welcoming Felix Cheung as a committer

Congrats Felix! On Mon, Aug 8, 2016 at 11:57 PM, dhruve ashar wrote: > Congrats Felix! > > On Mon, Aug 8, 2016 at 2:28 PM, Tarun Kumar wrote: > >> Congrats Felix! >> >> Tarun >> >> On Tue, Aug 9, 2016 at 12:57 AM, Timothy Chen

Re: [SPARK-2.0][SQL] UDF containing non-serializable object does not work as expected

The show thing was the result of an optimization that short-circuited any real Spark computation when the input is a local collection, and the result was simply the first few rows. That's why it completed without serializing anything. It is somewhat inconsistent. One way to eliminate the

Re: Welcoming Felix Cheung as a committer

Congrats Felix! On Mon, Aug 8, 2016 at 2:28 PM, Tarun Kumar wrote: > Congrats Felix! > > Tarun > > On Tue, Aug 9, 2016 at 12:57 AM, Timothy Chen wrote: > >> Congrats Felix! >> >> Tim >> >> On Mon, Aug 8, 2016 at 11:15 AM, Matei Zaharia

Re: Scaling partitioned Hive table support

Hi Eric, Thanks for your feedback. I'm rebasing my code for the first approach on a more recent Spark master and am resolving some conflicts. I'll have a better understanding of the relationship to your PR once my rebase is complete. Cheers, Michael > On Aug 8, 2016, at 12:51 PM, Eric Liang

Re: [SPARK-2.0][SQL] UDF containing non-serializable object does not work as expected

That is unfortunately the way how Scala compiler captures (and defines) closures. Nothing is really final in the JVM. You can always use reflection or unsafe to modify the value of fields. On Mon, Aug 8, 2016 at 8:16 PM, Simon Scott wrote: > But does the “notSer”

Re: Scaling partitioned Hive table support

I like the former approach -- it seems more generally applicable to other catalogs and IIUC would let you defer pruning until execution time. Pruning is work that should be done by the catalog anyways, as is the case when querying over an (unconverted) hive table. You might also want to look at

Re: Welcoming Felix Cheung as a committer

Congrats Felix! Tarun On Tue, Aug 9, 2016 at 12:57 AM, Timothy Chen wrote: > Congrats Felix! > > Tim > > On Mon, Aug 8, 2016 at 11:15 AM, Matei Zaharia > wrote: > > Hi all, > > > > The PMC recently voted to add Felix Cheung as a committer. Felix has

Re: Welcoming Felix Cheung as a committer

Congrats Felix! Tim On Mon, Aug 8, 2016 at 11:15 AM, Matei Zaharia wrote: > Hi all, > > The PMC recently voted to add Felix Cheung as a committer. Felix has been a > major contributor to SparkR and we're excited to have him join officially. > Congrats and welcome,

Re: Welcoming Felix Cheung as a committer

Congratulations , Felix! > On Aug 8, 2016, at 11:15 AM, Ted Yu wrote: > > Congratulations, Felix. > > On Mon, Aug 8, 2016 at 11:15 AM, Matei Zaharia > wrote: > Hi all, > > The PMC recently voted to add Felix

Scaling partitioned Hive table support

Hello, I'd like to propose a modification in the way Hive table partition metadata are loaded and cached. Currently, when a user reads from a partitioned Hive table whose metadata are not cached (and for which Hive table conversion is enabled and supported), all partition metadata is fetched

Re: Welcoming Felix Cheung as a committer

 Do we now have 2 SparkR-focused committers (Shivaram + Felix)? Or are there more? Nick On Mon, Aug 8, 2016 at 2:17 PM Dongjoon Hyun wrote: > Congratulation, Felix! > > Bests, > Dongjoon. > > > On Monday, August 8, 2016, Ted Yu wrote: > >>

Re: Welcoming Felix Cheung as a committer

Congratulation, Felix! Bests, Dongjoon. On Monday, August 8, 2016, Ted Yu wrote: > Congratulations, Felix. > > On Mon, Aug 8, 2016 at 11:15 AM, Matei Zaharia > wrote: > >> Hi all, >> >> The

Re: Welcoming Felix Cheung as a committer

Congratulations, Felix. On Mon, Aug 8, 2016 at 11:15 AM, Matei Zaharia wrote: > Hi all, > > The PMC recently voted to add Felix Cheung as a committer. Felix has been > a major contributor to SparkR and we're excited to have him join > officially. Congrats and welcome,

Welcoming Felix Cheung as a committer

Hi all, The PMC recently voted to add Felix Cheung as a committer. Felix has been a major contributor to SparkR and we're excited to have him join officially. Congrats and welcome, Felix! Matei - To unsubscribe e-mail:

Re: Kafka Support new topic subscriptions without requiring restart of the streaming context

The Kafka 0.10 support in spark 2.0 allows for pattern based topic subscription On Aug 8, 2016 1:12 AM, "r7raul1...@163.com" wrote: > How to add new topic to kafka without requiring restart of the streaming > context? > > -- > r7raul1...@163.com >

Spark 2.0 sql module empty columns in result over parquet tables

I run into this very strange issue. After loading parquet tables and trying to run an sql query with the sql module the results are not correct with Spark 2.0 although over the same exactly dataset Spark 1.6 results are correct. With Textfiles however both versions of Spark work as expected. I

Re: [SPARK-2.0][SQL] UDF containing non-serializable object does not work as expected

Yes, it is. You can define a udf like that. Basically, it's a udf Int => Int which is a closure contains a non serializable object. The latter should cause Task not serializable exception. Hao On Mon, Aug 8, 2016 at 5:08 AM, Muthu Jayakumar wrote: > Hello Hao Ren, > >

Kafka Support new topic subscriptions without requiring restart of the streaming context

How to add new topic to kafka without requiring restart of the streaming context? r7raul1...@163.com