Re: Spark Improvement Proposals

Responding to your request for a vote, I meant that this isn't required per se and the consensus here was not to vote on it. Hence the jokes about meta-voting protocol. In that sense nothing new happened process-wise, nothing against ASF norms, if that's your concern. I think it's just an agreed

Re: Spark Improvement Proposals

Another thing I think you should send out is when exactly does this take affect.  Is it any major new feature without a pull request?   Is it anything major starting with the 2.3 release?   Tom On Monday, March 13, 2017 1:08 PM, Tom Graves wrote: I'm not

Re: Spark Improvement Proposals

I'm not sure how you can say its not a new process.  If that is the case why do we need a page documenting it?   As a developer if I want to put up a major improvement I have to now follow the SPIP whereas before I didn't, that certain seems like a new process.  As a PMC member I now have the

Re: Spark Improvement Proposals

It's not a new process, in that it doesn't entail anything not already in http://apache.org/foundation/voting.html . We're just deciding to call a VOTE for this type of code modification. To your point -- yes, it's been around a long time with no further comment, and I called several times for

Re: Spark Improvement Proposals

It seems like if you are adding responsibilities you should do a vote.  SPIP'S require votes from PMC members so you are now putting more responsibility on them. It feels like we should have an official vote to make sure they (PMC members) agree with that and to make sure everyone pays

Re: Spark Improvement Proposals

This ended up proceeding as a normal doc change, instead of precipitating a meta-vote. However, the text that's on the web site now can certainly be further amended if anyone wants to propose a change from here. On Mon, Mar 13, 2017 at 1:50 PM Tom Graves wrote: > I think a

Re: Spark Improvement Proposals

I think a vote here would be good. I think most of the discussion was done by 4 or 5 people and its a long thread.  If nothing else it summarizes everything and gets people attention to the change. Tom On Thursday, March 9, 2017 10:55 AM, Sean Owen wrote: I think a

Re: Spark Improvement Proposals

We can just start using spip label and link to it. On Fri, Mar 10, 2017 at 9:18 AM, Cody Koeninger wrote: > So to be clear, if I translate that google doc to markup and submit a > PR, you will merge it? > > If we're just using "spip" label, that's probably fine, but we

Re: Spark Improvement Proposals

Can someone with filter share permissions can make a filter for open SPIP and one for closed SPIP and share it? e.g. project = SPARK AND status in (Open, Reopened, "In Progress") AND labels=SPIP ORDER BY createdDate DESC and another with the status closed equivalent I just made an open ticket

Re: Spark Improvement Proposals

So to be clear, if I translate that google doc to markup and submit a PR, you will merge it? If we're just using "spip" label, that's probably fine, but we still need shared filters for open and closed SPIPs so the page can link to them. I do not believe I have jira permissions to share filters,

Re: Spark Improvement Proposals

I think it ought to be its own page, linked from the more / community menu dropdowns. We also need the jira tag, and for the page to clearly link to filters that show proposed / completed SPIPs On Fri, Mar 10, 2017 at 3:39 AM, Sean Owen wrote: > Alrighty, if nobody is

Re: Spark Improvement Proposals

Alrighty, if nobody is objecting, and nobody calls for a VOTE, then, let's say this document is the SPIP 1.0 process. I think the next step is just to translate the text to some suitable location. I suggest adding it to https://github.com/apache/spark-website/blob/asf-site/contributing.md On

Re: Spark Improvement Proposals

gonna end up with a stackoverflow on recursive votes here On Thu, Mar 9, 2017 at 1:17 PM, Mark Hamstra wrote: > -0 on voting on whether we need a vote. > > On Thu, Mar 9, 2017 at 9:00 AM, Reynold Xin wrote: > >> I'm fine without a vote. (are we

Re: Spark Improvement Proposals

-0 on voting on whether we need a vote. On Thu, Mar 9, 2017 at 9:00 AM, Reynold Xin wrote: > I'm fine without a vote. (are we voting on wether we need a vote?) > > > On Thu, Mar 9, 2017 at 8:55 AM, Sean Owen wrote: > >> I think a VOTE is over-thinking

Re: Spark Improvement Proposals

Many of us have issue with "shepherd role " , i think we should go with vote. Regards, Vaquar khan On Thu, Mar 9, 2017 at 11:00 AM, Reynold Xin wrote: > I'm fine without a vote. (are we voting on wether we need a vote?) > > > On Thu, Mar 9, 2017 at 8:55 AM, Sean Owen

Re: Spark Improvement Proposals

I'm fine without a vote. (are we voting on wether we need a vote?) On Thu, Mar 9, 2017 at 8:55 AM, Sean Owen wrote: > I think a VOTE is over-thinking it, and is rarely used, but, can't hurt. > Nah, anyone can call a vote. This really isn't that formal. We just want to >

Re: Spark Improvement Proposals

I think a VOTE is over-thinking it, and is rarely used, but, can't hurt. Nah, anyone can call a vote. This really isn't that formal. We just want to declare and document consensus. I think SPIP is just a remix of existing process anyway, and don't think it will actually do much anyway, which is

Re: Spark Improvement Proposals

I started this idea as a fork with a merge-able change to docs. Reynold moved it to his google doc, and has suggested during this email thread that a vote should occur. If a vote needs to occur, I can't see anything on http://apache.org/foundation/voting.html suggesting that I can call for a vote,

Re: Spark Improvement Proposals

Do we need a VOTE? heck I think anyone can call one, anyway. Pre-flight vote check: anyone have objections to the text as-is? See https://docs.google.com/document/d/1-Zdi_W-wtuxS9hTK0P9qb2x-nRanvXmnZ7SUi4qMljg/edit# If so let's hash out specific suggest changes. If not, then I think the next

Re: Spark Improvement Proposals

Another week, another ping. Anyone on the PMC willing to call a vote on this? On Mon, Feb 27, 2017 at 3:08 PM, Ryan Blue wrote: > I'd like to see more discussion on the issues I raised. I don't think > there was a response for why voting is limited to PMC members. > > Tim

Re: Spark Improvement Proposals

To me, no new process is being invented here, on purpose, and so we should just rely on whatever governs any large JIRA or vote, because SPIPs are really just guidance for making a big JIRA. http://apache.org/foundation/voting.html suggests that PMC members have the binding votes in general, and

Re: Spark Improvement Proposals

I'd like to see more discussion on the issues I raised. I don't think there was a response for why voting is limited to PMC members. Tim was kind enough to reply with his rationale for a shepherd, but I don't think that it justifies failing proposals. I think it boiled down to "shepherds can be

Re: Spark Improvement Proposals

The current draft LGTM. I agree some of the various concerns may need to be addressed in the future, depending on how SPIPs progress in practice. If others agree, let's put it to a vote and revisit the proposal in a few months. Joseph On Fri, Feb 24, 2017 at 5:35 AM, Cody Koeninger

Re: Spark Improvement Proposals

It's been a week since any further discussion. Do PMC members think the current draft is OK to vote on? On Fri, Feb 17, 2017 at 10:41 PM, vaquar khan wrote: > I like document and happy to see SPIP draft version however i feel shepherd > role is again hurdle in process

Re: Spark Improvement Proposals

I like document and happy to see SPIP draft version however i feel shepherd role is again hurdle in process improvement ,It's like everything depends only on shepherd . Also want to add point that SPIP should be time bound with define SLA else will defeats purpose. Regards, Vaquar khan On

Re: Spark Improvement Proposals

> [The shepherd] can advise on technical and procedural considerations for people outside the community The sentiment is good, but this doesn't justify requiring a shepherd for a proposal. There are plenty of people that wouldn't need this, would get feedback during discussion, or would ask a

Re: Spark Improvement Proposals

Hi Folks I thought id chime in as someone new to the process so feel free to disregard it if it doesn't make sense. I definitely agree that we need a new forum to identify or discuss changes as JIRA isnt exactly the best place to do that, its a Bug tracker first and foremost. For example I was

Re: Spark Improvement Proposals

Reynold, thanks, LGTM. Sean, great concerns. I agree that behavior is largely cultural and writing down a process won't necessarily solve any problems one way or the other. But one outwardly visible change I'm hoping for out of this a way for people who have a stake in Spark, but can't follow

Re: Spark Improvement Proposals

The text seems fine to me. Really, this is not describing a fundamentally new process, which is good. We've always had JIRAs, we've always been able to call a VOTE for a big question. This just writes down a sensible set of guidelines for putting those two together when a major change is proposed.

Re: Spark Improvement Proposals

> >>> > are: >>>>>>>>>>> >> >>> > >>>>>>>>>>> >> >>> > 1. Visibility: There are so much happening that it is >>>>>>>>>>> difficult to >&g

Re: Spark Improvement Proposals

gt;>>>>>>>> >> >>> > know what the important initiatives are. Even for people >>>>>>>>>> that do >>>>>>>>>> >> >>> > follow, it >>>>>>>>>> >> >>> > is difficult to know what specific things require their >>>>>>>>&

Re: Spark Improvement Proposals

gt;>>>>>>> >> >>> > input >>>>>>>>> >> >>> > more proactively: At the end of the day the project >>>>>>>>> provides value >>>>>>>>> >> >>> > because >&

Re: Spark Improvement Proposals

gt;>> >> >>> > >>>>>>>> >> >>> > >>>>>>>> >> >>> > https://docs.google.com/docume >>>>>>>> nt/d/1-Zdi_W-wtuxS9hTK0P9qb2x-nRanvXmnZ7SUi4qMljg/edit#headi >>>>>&g

Re: Spark Improvement Proposals

d to voting. The reason being in voting there can >>>>>>> easily be a >>>>>>> >> >>> > "loser' >>>>>>> >> >>> > that gets outvoted. >>>>>>> >> >>> > >>>>>&g

Re: Spark Improvement Proposals

>>> >> >>> > implementations in PRs is not something that has not worked >>>>>> so far". >>>>>> >> >>> > >>>>>> >> >>> > 3. I made some the language tweaks to focus mo

Re: Spark Improvement Proposals

two emails that go >>>>> to >>>>> >> >>> > dev@. >>>>> >> >>> > >>>>> >> >>> > >>>>> >> >>> > While I was editing this, I thought we really needed a >>&

Re: Spark Improvement Proposals

gt;> Most things looked OK to me too, although I do plan to take a >>>> >> >>> >> closer >>>> >> >>> >> look >>>> >> >>> >> after Nov 1st when we cut the release branch for 2.1. >>>&g

Re: Spark Improvement Proposals

s not >>> >> >>> >>> explicitly >>> >> >>> >>> called, that voting would happen by e-mail? A template for the >>> >> >>> >>> proposal document (instead of just a bullet nice) would also >&

Re: Spark Improvement Proposals

;> >>> somewhat matches the proposed format. So if anyone wants to try >> >> >>> >>> out >> >> >>> >>> the process... >> >> >>> >>> >> >> >>> >>> On Mon, Oct 31, 2016 at 1

Re: Spark Improvement Proposals

gt;>> > in > >> >>> >>> > moving forward with this? > >> >>> >>> > > >> >>> >>> > > >> >>> >>> > > >> >>> >>> > > >> >&

Re: Spark Improvement Proposals

>>> >>> >> The > >>> >>> >> idea with benchmarks was to show two things: > >>> >>> >> > >>> >>> >> - why some people are doing bad PR for Spark > >>> >>> >> > >>&g

Re: Spark Improvement Proposals

t;>> >> - why some people are doing bad PR for Spark >>> >>> >> >>> >>> >> - how - in easy way - we can change it and show that Spark is >>> >>> >> still on >>> >>> >> the >>> >>> >> top >>> >>&

Re: Spark Improvement Proposals

n Spark :) On the Spark main page there is >> still >> >>> >> chart >> >>> >> "Spark vs Hadoop". It is important to show that framework is not >> the >> >>> >> same >> >>> >> Spark with other A

Re: Spark Improvement Proposals

gt;> faster than other frameworks. > >>> >> > >>> >> > >>> >> About real-time streaming, I think it would be just good to see it > in > >>> >> Spark. > >>> >> I very like current Spark model, but m

Re: Spark Improvement Proposals

I think narrowly focusing on Flink or benchmarks is missing my point. My point is evolve or die. Spark's governance and organization is hampering its ability to evolve technologically, and it needs to change. On Sun, Oct 16, 2016 at 9:21 PM, Debasish Das wrote: >

Re: Spark Improvement Proposals(Internet mail)

can be. --- Sincerely Andy 原始邮件 发件人: Debasish Das<debasish.da...@gmail.com> 收件人: Tomasz Gawęda<tomasz.gaw...@outlook.com> 抄送: dev@spark.apache.org<dev@spark.apache.org>; Cody Koeninger<c...@koeninger.org> 发送时间: 2016年10月17日(周一) 10:21 主题: Re: Spark Improvement Proposals

Re: Spark Improvement Proposals

Thanks Cody for bringing up a valid point...I picked up Spark in 2014 as soon as I looked into it since compared to writing Java map-reduce and Cascading code, Spark made writing distributed code fun...But now as we went deeper with Spark and real-time streaming use-case gets more prominent, I

Re: Spark Improvement Proposals

Hi everyone, I'm quite late with my answer, but I think my suggestions may help a little bit. :) Many technical and organizational topics were mentioned, but I want to focus on these negative posts about Spark and about "haters" I really like Spark. Easy of use, speed, very good community -

Re: Spark Improvement Proposals

;> >> > problem is >> >>>> >> > that writing a good document takes time. This way we can >> leverage >> >>>> >> > non >> >>>> >> > committers to do some of this work (it is just another way to >>

Re: Spark Improvement Proposals

t;>>> >> > contribute). > >>>> >> > > >>>> >> > > >>>> >> > > >>>> >> > As for strategy, in many cases implementation strategy can affect > >>>> >> >

Re: Spark Improvement Proposals

gt; > strategy, >>>> >> > we group by the time to achieve a sliding window. This is >>>> >> > definitely an >>>> >> > implementation decision and not a goal. However, I can think of >>>> >> > several >>>>

Re: Spark Improvement Proposals

inside their calculation >>> >> > buffer. >>> >> > For example, let’s say we want to return a set of all distinct >>> values. >>> >> > One >>> >> > way to implement this would be to make the set into a map and have >>> the >>> >> >

Re: Spark Improvement Proposals

I'm not a fan of the SEP acronym. Besides it prior established meaning of "Somebody else's problem", the are other inappropriate or offensive connotations such as this Australian slang that often gets shortened to just "sep": http://www.urbandictionary.com/define.php?term=Seppo On Sun, Oct 9,

Re: Spark Improvement Proposals

t;>>> Rejected Strategies: >>>>>>>>>>>>> >>>>>>>>>>>>> Having someone who understands the problem implement it first >>>>>>>>>>>>> works, >>>>>>>>>>>&

Re: Spark Improvement Proposals

gt; > One >> >>> > way to implement this would be to make the set into a map and have >> >>> > the >> >>> > value >> >>> > contain the last time seen. Multiplying it across the groupby would >> >>> > cost a >

Re: Spark Improvement Proposals

reluctant to allow changes >>> >>> >>>>>> to >>> >>> >>>>>> @experimental code that are needed in order for it to really >>> >>> >>>>>> work >>> >>> >>>>>>

Re: Spark Improvement Proposals

hat these cases are rare enough so that the strategy is still good >> > enough >> > but how would we know it without user feedback? >> > >> > I believe this example is exactly what Cody was talking about. Since >> > many >> > times implementation str

Re: Spark Improvement Proposals

; >>> >>>>>>>> more than > >>> >>>>>>>> sending stuff to dev@. One very lightweight idea is to have a > >>> >>>>>>>> new > >>&g

Re: Spark Improvement Proposals

; > Assaf. > > > > From: Cody Koeninger-2 [via Apache Spark Developers List] > [mailto:ml-node+[hidden email]] > Sent: Monday, October 10, 2016 2:25 AM > To: Mendelson, Assaf > Subject: Re: Spark Improvement Proposals > > > > Only committers should formally

Re: Spark Improvement Proposals

Only committers should formally submit SIPs because in an apache project only commiters have explicit political power. If a user can't find a commiter willing to sponsor an SIP idea, they have no way to get the idea passed in any case. If I can't find a committer to sponsor this meta-SIP idea,

Re: Spark Improvement Proposals

On Sun, Oct 9, 2016 at 5:19 PM Cody Koeninger wrote: > Regarding name, if the SIP overlap is a concern, we can pick a different > name. > > My tongue in cheek suggestion would be > > Spark Lightweight Improvement process (SPARKLI) > If others share my minor concern about the

Re: Spark Improvement Proposals

Well, I think there are a few things here that don't make sense. First, why should only committers submit SIPs? Development in the project should be open to all contributors, whether they're committers or not. Second, I think unrealistic goals can be found just by inspecting the goals, and I'm

Re: Spark Improvement Proposals

Yeah, I've looked at KIPs and Scala SIPs. I'm reluctant to use the Kafka structured streaming as an example because of the pre-existing conflict around it. If Michael or another committer wanted to put it forth as an example, I'd participate in good faith though. On Sun, Oct 9, 2016 at 5:07 PM,

Re: Spark Improvement Proposals

Users instead of people, sure. Commiters and contributors are (or at least should be) a subset of users. Non goals, sure. I don't care what the name is, but we need to clearly say e.g. 'no we are not maintaining compatibility with XYZ right now'. API, what I care most about is whether it allows

Re: Spark Improvement Proposals

This is a great discussion! Maybe you could have a look at Kafka's process - it also uses Rejected Alternatives and I personally find it very clear actually (the link also leads to all KIPs): https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals Cody - maybe you could take

Re: Spark Improvement Proposals

Yup, this is the stuff that I found unclear. Thanks for clarifying here, but we should also clarify it in the writeup. In particular: - Goals needs to be about user-facing behavior ("people" is broad) - I'd rename Rejected Goals to Non-Goals. Otherwise someone will dig up one of these and say

Re: Spark Improvement Proposals

Regarding name, if the SIP overlap is a concern, we can pick a different name. My tongue in cheek suggestion would be Spark Lightweight Improvement process (SPARKLI) On Sun, Oct 9, 2016 at 4:14 PM, Cody Koeninger wrote: > So to focus the discussion on the specific strategy

Re: Spark Improvement Proposals

So to focus the discussion on the specific strategy I'm suggesting, documented at https://github.com/koeninger/spark-1/blob/SIP-0/docs/spark-improvement-proposals.md "Goals: What must this allow people to do, that they can't currently?" Is it unclear that this is focusing specifically on

Re: Spark Improvement Proposals

If there's confusion there, the document is specifically what I'm proposing. The email is just by way of introduction. On Sun, Oct 9, 2016 at 3:47 PM, Nicholas Chammas wrote: > Oh, hmm… I guess I’m a little confused on the relation between Cody’s > email and the

Re: Spark Improvement Proposals

Oh, hmm… I guess I’m a little confused on the relation between Cody’s email and the document he linked to, which says: https://github.com/koeninger/spark-1/blob/SIP-0/docs/spark-improvement-proposals.md#when SIPs should be used for significant user-facing or cross-cutting changes, not day-to-day

Re: Spark Improvement Proposals

Yup, but the example you gave is for alternatives about *user-facing behavior*, not implementation. The current SIP doc describes "strategy" more as implementation strategy. I'm just saying there are different possible goals for these types of docs. BTW, PEPs and Scala SIPs focus primarily on

Re: Spark Improvement Proposals

- Rejected strategies: I personally wouldn’t put this, because what’s the point of voting to reject a strategy before you’ve really begun designing and implementing something? What if you discover that the strategy is actually better when you start doing stuff? I would guess the point

Re: Spark Improvement Proposals

Here's my specific proposal (meta-proposal?) Spark Improvement Proposals (SIP) Background: The current problem is that design and implementation of large features are often done in private, before soliciting user feedback. When feedback is solicited, it is often as to detailed design

Re: Spark Improvement Proposals

+1 for SIP lebles,waiting for Reynolds detailed proposal . Regards, Vaquar khan On 8 Oct 2016 16:22, "Matei Zaharia" wrote: > Sounds good. Just to comment on the compatibility part: > > > I meant changing public user interfaces. I think the first design is > >

Re: Spark Improvement Proposals

Sounds good. Just to comment on the compatibility part: > I meant changing public user interfaces. I think the first design is > unlikely to be right, because it's done at a time when you have the > least information. As a user, I find it considerably more frustrating > to be unable to use a

Re: Spark Improvement Proposals

Alright looks like there are quite a bit of support. We should wait to hear from more people too. To push this forward, Cody and I will be working together in the next couple of weeks to come up with a concrete, detailed proposal on what this entails, and then we can discuss this the specific

Re: Spark Improvement Proposals

Yeah, in case it wasn't clear, I was talking about SIPs for major user-facing or cross-cutting changes, not minor feature adds. On Fri, Oct 7, 2016 at 3:58 PM, Stavros Kontopoulos < stavros.kontopou...@lightbend.com> wrote: > +1 to the SIP label as long as it does not slow down things and it

Re: Spark Improvement Proposals

+1 to adding an SIP label and linking it from the website. I think it needs - template that focuses it towards soliciting user goals / non goals - clear resolution as to which strategy was chosen to pursue. I'd recommend a vote. Matei asked me to clarify what I meant by changing interfaces, I

Re: Spark Improvement Proposals

I like the lightweight proposal to add a SIP label. During Spark 2.0 development, Tom (Graves) and I suggested using wiki to track the list of major changes, but that never really materialized due to the overhead. Adding a SIP label on major JIRAs and then link to them prominently on the Spark

Re: Spark Improvement Proposals

I am glad that it was not only what I was thinking. I also do agree with Holden, Sean and Cody. All I wanted to say were all said. 2016-10-08 1:16 GMT+09:00 Holden Karau : > First off, thanks Cody for taking the time to put together these proposals > - I think it has

Re: Spark Improvement Proposals

For the improvement proposals, I think one major point was to make them really visible to users who are not contributors, so we should do more than sending stuff to dev@. One very lightweight idea is to have a new type of JIRA called a SIP and have a link to a filter that shows all such JIRAs

Re: Spark Improvement Proposals

I called Cody last night and talked about some of the topics in his email. It became clear to me Cody genuinely cares about the project. Some of the frustrations come from the success of the project itself becoming very "hot", and it is difficult to get clarity from people who don't dedicate all

Re: Spark Improvement Proposals

There are several important discussions happening simultaneously. Should we perhaps split them up into separate threads? Otherwise it’s really difficult to follow. It seems like the discussion about having a more formal “Spark Improvement Proposal” process should take priority here. Other

Re: Spark Improvement Proposals

I think people misunderstood my comment about trolls a bit -- I'm not saying to just dismiss what people say, but to focus on what improves the project instead of being upset that people criticize stuff. This stuff happens all the time to any project in a "hot" area, as Sean said. I don't think

Re: Spark Improvement Proposals

First off, thanks Cody for taking the time to put together these proposals - I think it has kicked off some wonderful discussion. I think dismissing people's complaints with Spark as largely trolls does us a disservice, it’s important for us to recognize our own shortcomings - otherwise we are

Re: Spark Improvement Proposals

Sean, that was very eloquently put, and I 100% agree. If I ever meet you in person, I'll buy you multiple rounds of beverages of your choice ;) This is probably reiterating some of what you said in a less clear manner, but I'll throw more of my 2 cents in. - Design. Yes, design by committee

Re: Spark Improvement Proposals

Suggestion actions way at the bottom. On Fri, Oct 7, 2016 at 5:14 AM Matei Zaharia wrote: since March. But it's true that other things such as the Kafka source for it didn't have as much design on JIRA. Nonetheless, this component is still early on and there's still a

Re: Spark Improvement Proposals

Let us continue to improve Apache Spark! I volunteer to go through all the SQL-related open JIRAs. Xiao Li 2016-10-06 21:14 GMT-07:00 Matei Zaharia : > Hey Cody, > > Thanks for bringing these things up. You're talking about quite a few > different things here, but let

Re: Spark Improvement Proposals

Hey Cody, Thanks for bringing these things up. You're talking about quite a few different things here, but let me get to them each in turn. 1) About technical / design discussion -- I fully agree that everything big should go through a lot of review, and I like the idea of a more formal way to