Re: Moving forward with the timestamp proposal

I think this is the right direction to go, but I'm wondering how can Spark support these new types if the underlying data sources(like parquet files) do not support them yet. I took a quick look at the new doc for file formats, but not sure what's the proposal. Are we going to implement these new

Re: [VOTE] Release Apache Spark 2.4.1 (RC2)

Could you hold for a bit - I have one more fix to get in From: d_t...@apple.com on behalf of DB Tsai Sent: Wednesday, February 20, 2019 12:25 PM To: Spark dev list Cc: Cesar Delgado Subject: Re: [VOTE] Release Apache Spark 2.4.1 (RC2) Okay. Let's fail rc2, and

Re: Unsubscribe

Please send an email to dev-unsubscr...@spark.apache.org to unsubscribe. You should receive an email with instruction to confirm the unsubscribe. On Wed, Feb 20, 2019 at 3:58 PM Reena Agrawal wrote: > Unsubscribe pls. >

Unsubscribe

Unsubscribe pls.

Re: Thoughts on dataframe cogroup?

Alessandro, Thanks for the reply. I assume by "equi-join", you mean "equality full outer join" . Two issues I see with equity outer join is: (1) equity outer join will give n * m rows for each key (n and m being the corresponding number of rows in df1 and df2 for each key) (2) User needs to do

Re: [VOTE] Release Apache Spark 2.4.1 (RC2)

Okay. Let's fail rc2, and I'll prepare rc3 with SPARK-26859. DB Tsai | Siri Open Source Technologies [not a contribution] |  Apple, Inc > On Feb 20, 2019, at 12:11 PM, Marcelo Vanzin > wrote: > > Just wanted to point out that > https://issues.apache.org/jira/browse/SPARK-26859 is not in

Unsubscribe

Unsubscribe pls

Re: [VOTE] Release Apache Spark 2.4.1 (RC2)

Just wanted to point out that https://issues.apache.org/jira/browse/SPARK-26859 is not in this RC, and is marked as a correctness bug. (The fix is in the 2.4 branch, just not in rc2.) On Wed, Feb 20, 2019 at 12:07 PM DB Tsai wrote: > > Please vote on releasing the following candidate as Apache

[VOTE] Release Apache Spark 2.4.1 (RC2)

Please vote on releasing the following candidate as Apache Spark version 2.4.1. The vote is open until Feb 24 PST and passes if a majority +1 PMC votes are cast, with a minimum of 3 +1 votes. [ ] +1 Release this package as Apache Spark 2.4.1 [ ] -1 Do not release this package because ... To

Moving forward with the timestamp proposal

Hi, Last december we shared a timestamp harmonization proposal with the Hive, Spark and Impala communities. This was followed by an extensive discussion in January that lead to various updates and improvements to the proposal, as well as the creation of a new document for

Re: Thoughts on dataframe cogroup?

Hello, I fail to see how an equi-join on the key columns is different than the cogroup you propose. I think the accepted answer can shed some light: https://stackoverflow.com/questions/43960583/whats-the-difference-between-join-and-cogroup-in-apache-spark Now you apply an udf on each iterable,

Re: Missing SparkR in CRAN

Thanks! On Wed, Feb 20, 2019 at 12:10 PM Felix Cheung wrote: > We are waiting for update from CRAN. Please hold on. > > > -- > *From:* Takeshi Yamamuro > *Sent:* Tuesday, February 19, 2019 2:53 PM > *To:* dev > *Subject:* Re: Missing SparkR in CRAN > > Hi, guys > >

Re: [VOTE] SPIP: Identifiers for multi-catalog Spark

+1 On Wed, Feb 20, 2019 at 4:59 PM JackyLee wrote: > +1 > > > > -- > Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/ > > - > To unsubscribe e-mail: dev-unsubscr...@spark.apache.org > > -- --- Takeshi