Re: [build system] tonight's downtime

happy update: we put the host serving the reverse proxy to jenkins on a UPS/genny circuit we expect to last overnight. this means that there should be no impact on jenkins during the maintenance period. pull requests should continue to build w/o issue overnight, and hopefully well in to the

Re: Welcoming Saisai (Jerry) Shao as a committer

Congratulations, Jerry! On Tue, Aug 29, 2017 at 2:14 PM, Suresh Thalamati < suresh.thalam...@gmail.com> wrote: > Congratulations, Jerry > > > On Aug 28, 2017, at 6:28 PM, Matei Zaharia > wrote: > > > > Hi everyone, > > > > The PMC recently voted to add Saisai (Jerry)

Re: Welcoming Saisai (Jerry) Shao as a committer

Congratulations, Jerry > On Aug 28, 2017, at 6:28 PM, Matei Zaharia wrote: > > Hi everyone, > > The PMC recently voted to add Saisai (Jerry) Shao as a committer. Saisai has > been contributing to many areas of the project for a long time, so it’s great > to see him

Re: Welcoming Saisai (Jerry) Shao as a committer

Congratulations, Jerry! Kazuaki Ishizaki From: Hyukjin Kwon To: dev Date: 2017/08/29 12:24 Subject:Re: Welcoming Saisai (Jerry) Shao as a committer Congratulations! Very well deserved. 2017-08-29 11:41 GMT+09:00 Liwei Lin

Re: Welcoming Saisai (Jerry) Shao as a committer

Congratulations Jerry!! Well deserved! Thanks, Yucai On 29/08/2017, 12:06, "Cheng, Hao" wrote: Congratulations!! Jerry, you really deserve it. Hao -Original Message- From: Mridul Muralidharan [mailto:mri...@gmail.com] Sent: Tuesday,

RE: Welcoming Saisai (Jerry) Shao as a committer

Congratulations!! Jerry, you really deserve it. Hao -Original Message- From: Mridul Muralidharan [mailto:mri...@gmail.com] Sent: Tuesday, August 29, 2017 12:04 PM To: Matei Zaharia Cc: dev ; Saisai Shao Subject:

Re: Welcoming Saisai (Jerry) Shao as a committer

Congratulations Jerry, well deserved ! Regards, Mridul On Mon, Aug 28, 2017 at 6:28 PM, Matei Zaharia wrote: > Hi everyone, > > The PMC recently voted to add Saisai (Jerry) Shao as a committer. Saisai has > been contributing to many areas of the project for a long

Re: Welcoming Saisai (Jerry) Shao as a committer

Congratulations! Very well deserved. 2017-08-29 11:41 GMT+09:00 Liwei Lin : > Congratulations, Jerry! > > Cheers, > Liwei > > On Tue, Aug 29, 2017 at 10:15 AM, 蒋星博 wrote: > >> congs！ >> >> Takeshi Yamamuro 于2017年8月28日 周一下午7:11写道：

Re: [VOTE] [SPIP] SPARK-15689: Data Source API V2

Copying from the code review comments I just submitted on the draft API (https://github.com/cloud-fan/spark/pull/10#pullrequestreview-59088745): Context here is that I've spent some time implementing a Spark datasource and have had some issues with the current API which are made worse in V2.

Re: [VOTE] [SPIP] SPARK-15689: Data Source API V2

Copying from the code review comments I just submitted on the draft API (https://github.com/cloud-fan/spark/pull/10#pullrequestreview-59088745): Context here is that I've spent some time implementing a Spark datasource and have had some issues with the current API which are made worse in V2.

Re: [VOTE] [SPIP] SPARK-15689: Data Source API V2

Copying from the code review comments I just submitted on the draft API (https://github.com/cloud-fan/spark/pull/10#pullrequestreview-59088745): Context here is that I've spent some time implementing a Spark datasource and have had some issues with the current API which are made worse in V2.

Re: Welcoming Saisai (Jerry) Shao as a committer

Congratulations, Jerry! Cheers, Liwei On Tue, Aug 29, 2017 at 10:15 AM, 蒋星博 wrote: > congs！ > > Takeshi Yamamuro 于2017年8月28日 周一下午7:11写道： > >> Congrats! >> >> On Tue, Aug 29, 2017 at 11:04 AM, zhichao wrote: >> >>>

Re: Welcoming Saisai (Jerry) Shao as a committer

Congratulations! Jerry! > On Aug 29, 2017, at 10:04 AM, zhichao wrote: > > Congratulations, Jerry! > >> On Tue, Aug 29, 2017 at 9:57 AM, Weiqing Yang >> wrote: >> Congratulations, Jerry! >> >>> On Mon, Aug 28, 2017 at 6:44 PM, Yanbo Liang

Re: Welcoming Saisai (Jerry) Shao as a committer

congs！ Takeshi Yamamuro 于2017年8月28日 周一下午7:11写道： > Congrats! > > On Tue, Aug 29, 2017 at 11:04 AM, zhichao wrote: > >> Congratulations, Jerry! >> >> On Tue, Aug 29, 2017 at 9:57 AM, Weiqing Yang >> wrote: >> >>>

Re: Welcoming Saisai (Jerry) Shao as a committer

Congrats! On Tue, Aug 29, 2017 at 11:04 AM, zhichao wrote: > Congratulations, Jerry! > > On Tue, Aug 29, 2017 at 9:57 AM, Weiqing Yang > wrote: > >> Congratulations, Jerry! >> >> On Mon, Aug 28, 2017 at 6:44 PM, Yanbo Liang

Re: Welcoming Saisai (Jerry) Shao as a committer

Congratulations, Jerry! On Tue, Aug 29, 2017 at 9:57 AM, Weiqing Yang wrote: > Congratulations, Jerry! > > On Mon, Aug 28, 2017 at 6:44 PM, Yanbo Liang wrote: > >> Congratulations, Jerry. >> >> On Tue, Aug 29, 2017 at 9:42 AM, John Deng

Re: Welcoming Saisai (Jerry) Shao as a committer

Congratulations, Jerry! Weiqing Yang wrote > Congratulations, Jerry! > > On Mon, Aug 28, 2017 at 6:44 PM, Yanbo Liang > ybliang8@ > wrote: > >> Congratulations, Jerry. >> >> On Tue, Aug 29, 2017 at 9:42 AM, John Deng > mailthis@ > wrote: >> >>> >>> Congratulations, Jerry ! >>> >>> On

Re: Welcoming Saisai (Jerry) Shao as a committer

Congratulations, Jerry! On Mon, Aug 28, 2017 at 6:44 PM, Yanbo Liang wrote: > Congratulations, Jerry. > > On Tue, Aug 29, 2017 at 9:42 AM, John Deng wrote: > >> >> Congratulations, Jerry ! >> >> On 8/29/2017 09:28，Matei Zaharia >>

Re: Welcoming Saisai (Jerry) Shao as a committer

Congratulations, Jerry. On Tue, Aug 29, 2017 at 9:42 AM, John Deng wrote: > > Congratulations, Jerry ! > > On 8/29/2017 09:28，Matei Zaharia > wrote： > > Hi everyone, > > The PMC recently voted to add Saisai (Jerry) Shao as a >

Re: Welcoming Saisai (Jerry) Shao as a committer

Congratulations, Jerry ! On 8/29/2017 09:28，Matei Zaharia wrote： Hi everyone, The PMC recently voted to add Saisai (Jerry) Shao as a committer. Saisai has been contributing to many areas of the project for a long time, so it’s great to see him join. Join me in

Re: Welcoming Saisai (Jerry) Shao as a committer

Congratulation, Jerry! -- View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/Welcoming-Saisai-Jerry-Shao-as-a-committer-tp22241p22244.html Sent from the Apache Spark Developers List mailing list archive at Nabble.com.

Re: Welcoming Saisai (Jerry) Shao as a committer

Great! Congratulation, Jerry! :D Bests, Dongjoon. On Mon, Aug 28, 2017 at 6:30 PM, Ted Yu wrote: > Congratulations, Jerry ! > > On Mon, Aug 28, 2017 at 6:28 PM, Matei Zaharia > wrote: > >> Hi everyone, >> >> The PMC recently voted to add Saisai

Re: Welcoming Saisai (Jerry) Shao as a committer

Congratulations, Jerry ! On Mon, Aug 28, 2017 at 6:28 PM, Matei Zaharia wrote: > Hi everyone, > > The PMC recently voted to add Saisai (Jerry) Shao as a committer. Saisai > has been contributing to many areas of the project for a long time, so it’s > great to see him

Welcoming Saisai (Jerry) Shao as a committer

Hi everyone, The PMC recently voted to add Saisai (Jerry) Shao as a committer. Saisai has been contributing to many areas of the project for a long time, so it’s great to see him join. Join me in thanking and congratulating him! Matei

use WithColumn with external function of a Java jar

I have data in a DataFrame with below columns 1)Fileformat is csv 2)All below column datatypes are String employeeid,pexpense,cexpense Now I need to create a new DataFrame which has new column called `expense`, which is calculated based on columns `pexpense`, `cexpense`. The tricky part

Re: [VOTE] [SPIP] SPARK-15689: Data Source API V2

+1 (Non-binding) Xiao Li 于2017年8月28日 周一下午5:38写道： > +1 > > 2017-08-28 12:45 GMT-07:00 Cody Koeninger : > >> Just wanted to point out that because the jira isn't labeled SPIP, it >> won't have shown up linked from >> >>

Re: [VOTE] [SPIP] SPARK-15689: Data Source API V2

+1 2017-08-28 12:45 GMT-07:00 Cody Koeninger : > Just wanted to point out that because the jira isn't labeled SPIP, it > won't have shown up linked from > > http://spark.apache.org/improvement-proposals.html > > On Mon, Aug 28, 2017 at 2:20 PM, Wenchen Fan

[build system] tonight's downtime

we are shutting down our servers (including our lab webserver) tonite ~1030pm PDT, which will effectively take jenkins off-line until the electrical feed repairs to our building are finished. the estimate from the contractors is that they should be done tomorrow morning (8-29) by 7am PDT. more

Re: SPIP: Spark on Kubernetes

In addition to the engineering & software aspects of the native Kubernetes community project, we have also worked at building out the community, with the goal of providing the foundation for sustaining engineering on the Kubernetes scheduler back-end. That said, I agree 100% with your point that

Re: Are there multiple processes out there running JIRA <-> Github maintenance tasks?

This should be fixed now. The problem was that debug code had been pushed while investigating the JIRA linkage failure but was not removed and this problem went unnoticed because linking was failing well before the debug code was hit. Once the JIRA connectivity issues were resolved, the

Re: SPIP: Spark on Kubernetes

> > In my opinion, the fact that there are nearly no changes to spark-core, > and most of our changes are additive should go to prove that this adds > little complexity to the workflow of the committers. Actually (and somewhat perversely), the otherwise praiseworthy isolation of the Kubernetes

Re: [VOTE] [SPIP] SPARK-15689: Data Source API V2

Just wanted to point out that because the jira isn't labeled SPIP, it won't have shown up linked from http://spark.apache.org/improvement-proposals.html On Mon, Aug 28, 2017 at 2:20 PM, Wenchen Fan wrote: > Hi all, > > It has been almost 2 weeks since I proposed the data

Re: [VOTE] [SPIP] SPARK-15689: Data Source API V2

+1 (Non-binding) The clustering approach covers most of my requirements on saving some shuffles. We kind of left the "should the user be allowed to provide a full partitioner" discussion on the table. I understand that would require exposing a lot of internals so this is perhaps a good

[VOTE] [SPIP] SPARK-15689: Data Source API V2

Hi all, It has been almost 2 weeks since I proposed the data source V2 for discussion, and we already got some feedbacks on the JIRA ticket and the prototype PR, so I'd like to call for a vote. The full document of the Data Source API V2 is:

Re: Are there multiple processes out there running JIRA <-> Github maintenance tasks?

It seems a little wonky, though. Feels like it's updating JIRA every time you comment on a PR. Or maybe it's still working through the backlog... On Mon, Aug 28, 2017 at 9:57 AM, Reynold Xin wrote: > The process for doing that was down before, and might've come back up and >

Re: Are there multiple processes out there running JIRA <-> Github maintenance tasks?

The process for doing that was down before, and might've come back up and are going through the huge backlog. On Mon, Aug 28, 2017 at 6:56 PM, Sean Owen wrote: > Like whatever reassigns JIRAs after a PR is closed? > > It seems to be going crazy, or maybe there are many

Are there multiple processes out there running JIRA <-> Github maintenance tasks?

Like whatever reassigns JIRAs after a PR is closed? It seems to be going crazy, or maybe there are many running. Not sure who owns that, but can he/she take a look?

Time window on Processing Time

Hi, As I am playing with structured streaming, I observed that window function always requires a time column in input data.So that means it's event time. Is it possible to old spark streaming style window function based on processing time. I don't see any documentation on the same. -- Regards,