Alright looks like there are quite a bit of support. We should wait to hear from more people too.
To push this forward, Cody and I will be working together in the next couple of weeks to come up with a concrete, detailed proposal on what this entails, and then we can discuss this the specific proposal as well. On Fri, Oct 7, 2016 at 2:29 PM, Cody Koeninger <c...@koeninger.org> wrote: > Yeah, in case it wasn't clear, I was talking about SIPs for major > user-facing or cross-cutting changes, not minor feature adds. > > On Fri, Oct 7, 2016 at 3:58 PM, Stavros Kontopoulos <stavros.kontopoulos@ > lightbend.com> wrote: > >> +1 to the SIP label as long as it does not slow down things and it >> targets optimizing efforts, coordination etc. For example really small >> features should not need to go through this process (assuming they dont >> touch public interfaces) or re-factorings and hope it will be kept this >> way. So as a guideline doc should be provided, like in the KIP case. >> >> IMHO so far aside from tagging things and linking them elsewhere simply >> having design docs and prototypes implementations in PRs is not something >> that has not worked so far. What is really a pain in many projects out >> there is discontinuity in progress of PRs, missing features, slow reviews >> which is understandable to some extent... it is not only about Spark but >> things can be improved for sure for this project in particular as already >> stated. >> >> On Fri, Oct 7, 2016 at 11:14 PM, Cody Koeninger <c...@koeninger.org> >> wrote: >> >>> +1 to adding an SIP label and linking it from the website. I think it >>> needs >>> >>> - template that focuses it towards soliciting user goals / non goals >>> - clear resolution as to which strategy was chosen to pursue. I'd >>> recommend a vote. >>> >>> Matei asked me to clarify what I meant by changing interfaces, I think >>> it's directly relevant to the SIP idea so I'll clarify here, and split >>> a thread for the other discussion per Nicholas' request. >>> >>> I meant changing public user interfaces. I think the first design is >>> unlikely to be right, because it's done at a time when you have the >>> least information. As a user, I find it considerably more frustrating >>> to be unable to use a tool to get my job done, than I do having to >>> make minor changes to my code in order to take advantage of features. >>> I've seen committers be seriously reluctant to allow changes to >>> @experimental code that are needed in order for it to really work >>> right. You need to be able to iterate, and if people on both sides of >>> the fence aren't going to respect that some newer apis are subject to >>> change, then why even mark them as such? >>> >>> Ideally a finished SIP should give me a checklist of things that an >>> implementation must do, and things that it doesn't need to do. >>> Contributors/committers should be seriously discouraged from putting >>> out a version 0.1 that doesn't have at least a prototype >>> implementation of all those things, especially if they're then going >>> to argue against interface changes necessary to get the the rest of >>> the things done in the 0.2 version. >>> >>> >>> On Fri, Oct 7, 2016 at 2:18 PM, Reynold Xin <r...@databricks.com> wrote: >>> > I like the lightweight proposal to add a SIP label. >>> > >>> > During Spark 2.0 development, Tom (Graves) and I suggested using wiki >>> to >>> > track the list of major changes, but that never really materialized >>> due to >>> > the overhead. Adding a SIP label on major JIRAs and then link to them >>> > prominently on the Spark website makes a lot of sense. >>> > >>> > >>> > On Fri, Oct 7, 2016 at 10:50 AM, Matei Zaharia < >>> matei.zaha...@gmail.com> >>> > wrote: >>> >> >>> >> For the improvement proposals, I think one major point was to make >>> them >>> >> really visible to users who are not contributors, so we should do >>> more than >>> >> sending stuff to dev@. One very lightweight idea is to have a new >>> type of >>> >> JIRA called a SIP and have a link to a filter that shows all such >>> JIRAs from >>> >> http://spark.apache.org. I also like the idea of SIP and design doc >>> >> templates (in fact many projects have them). >>> >> >>> >> Matei >>> >> >>> >> On Oct 7, 2016, at 10:38 AM, Reynold Xin <r...@databricks.com> wrote: >>> >> >>> >> I called Cody last night and talked about some of the topics in his >>> email. >>> >> It became clear to me Cody genuinely cares about the project. >>> >> >>> >> Some of the frustrations come from the success of the project itself >>> >> becoming very "hot", and it is difficult to get clarity from people >>> who >>> >> don't dedicate all their time to Spark. In fact, it is in some ways >>> similar >>> >> to scaling an engineering team in a successful startup: old processes >>> that >>> >> worked well might not work so well when it gets to a certain size, >>> cultures >>> >> can get diluted, building culture vs building process, etc. >>> >> >>> >> I also really like to have a more visible process for larger changes, >>> >> especially major user facing API changes. Historically we upload >>> design docs >>> >> for major changes, but it is not always consistent and difficult to >>> quality >>> >> of the docs, due to the volunteering nature of the organization. >>> >> >>> >> Some of the more concrete ideas we discussed focus on building a >>> culture >>> >> to improve clarity: >>> >> >>> >> - Process: Large changes should have design docs posted on JIRA. One >>> thing >>> >> Cody and I didn't discuss but an idea that just came to me is we >>> should >>> >> create a design doc template for the project and ask everybody to >>> follow. >>> >> The design doc template should also explicitly list goals and >>> non-goals, to >>> >> make design doc more consistent. >>> >> >>> >> - Process: Email dev@ to solicit feedback. We have some this with >>> some >>> >> changes, but again very inconsistent. Just posting something on JIRA >>> isn't >>> >> sufficient, because there are simply too many JIRAs and the signal >>> get lost >>> >> in the noise. While this is generally impossible to enforce because >>> we can't >>> >> force all volunteers to conform to a process (or they might not even >>> be >>> >> aware of this), those who are more familiar with the project can >>> help by >>> >> emailing the dev@ when they see something that hasn't been. >>> >> >>> >> - Culture: The design doc author(s) should be open to feedback. A >>> design >>> >> doc should serve as the base for discussion and is by no means the >>> final >>> >> design. Of course, this does not mean the author has to accept every >>> >> feedback. They should also be comfortable accepting / rejecting ideas >>> on >>> >> technical grounds. >>> >> >>> >> - Process / Culture: For major ongoing projects, it can be useful to >>> have >>> >> some monthly Google hangouts that are open to the world. I am >>> actually not >>> >> sure how well this will work, because of the volunteering nature and >>> we need >>> >> to adjust for timezones for people across the globe, but it seems >>> worth >>> >> trying. >>> >> >>> >> - Culture: Contributors (including committers) should be more direct >>> in >>> >> setting expectations, including whether they are working on a specific >>> >> issue, whether they will be working on a specific issue, and whether >>> an >>> >> issue or pr or jira should be rejected. Most people I know in this >>> community >>> >> are nice and don't enjoy telling other people no, but it is often more >>> >> annoying to a contributor to not know anything than getting a no. >>> >> >>> >> >>> >> On Fri, Oct 7, 2016 at 10:03 AM, Matei Zaharia < >>> matei.zaha...@gmail.com> >>> >> wrote: >>> >>> >>> >>> >>> >>> Love the idea of a more visible "Spark Improvement Proposal" process >>> that >>> >>> solicits user input on new APIs. For what it's worth, I don't think >>> >>> committers are trying to minimize their own work -- every committer >>> cares >>> >>> about making the software useful for users. However, it is always >>> hard to >>> >>> get user input and so it helps to have this kind of process. I've >>> certainly >>> >>> looked at the *IPs a lot in other software I use just to see the >>> biggest >>> >>> things on the roadmap. >>> >>> >>> >>> When you're talking about "changing interfaces", are you talking >>> about >>> >>> public or internal APIs? I do think many people hate changing public >>> APIs >>> >>> and I actually think that's for the best of the project. That's a >>> technical >>> >>> debate, but basically, the worst thing when you're using a piece of >>> software >>> >>> is that the developers constantly ask you to rewrite your app to >>> update to a >>> >>> new version (and thus benefit from bug fixes, etc). Cue anyone who's >>> used >>> >>> Protobuf, or Guava. The "let's get everyone to change their code this >>> >>> release" model works well within a single large company, but doesn't >>> work >>> >>> well for a community, which is why nearly all *very* widely used >>> programming >>> >>> interfaces (I'm talking things like Java standard library, Windows >>> API, etc) >>> >>> almost *never* break backwards compatibility. All this is done >>> within reason >>> >>> though, e.g. we do change things in major releases (2.x, 3.x, etc). >>> >> >>> >> >>> >> >>> >> >>> > >>> >>> --------------------------------------------------------------------- >>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org >>> >>> >> >> >> -- >> Stavros Kontopoulos >> >> *Senior Software Engineer* >> *Lightbend, Inc.* >> >> *p: +30 6977967274 <%2B1%20650%20678%200020>* >> *e: stavros.kontopou...@lightbend.com* <dave.mar...@lightbend.com> >> >> >> >
