I like the idea of having design docs be kept up to date and tracked in git.
If the Apache repo isn't a good fit, perhaps we can have a separate repo just for design docs? Maybe something like github.com/spark-docs/spark-docs/ ? If there's other stuff we want to track but haven't, perhaps we can generalize the purpose of the repo a bit and rename it accordingly (e.g. spark-misc/spark-misc). Nick On Mon, Apr 27, 2015 at 1:21 PM Sandy Ryza <sandy.r...@cloudera.com> wrote: > My only issue with Google Docs is that they're mutable, so it's difficult > to follow a design's history through its revisions and link up JIRA > comments with the relevant version. > > -Sandy > > On Mon, Apr 27, 2015 at 7:54 AM, Steve Loughran <ste...@hortonworks.com> > wrote: > > > > > One thing to consider is that while docs as PDFs in JIRAs do document the > > original proposal, that's not the place to keep living specifications. > That > > stuff needs to live in SCM, in a format which can be easily maintained, > can > > generate readable documents, and, in an unrealistically ideal world, even > > be used by machines to validate compliance with the design. Test suites > > tend to be the implicit machine-readable part of the specification, > though > > they aren't usually viewed as such. > > > > PDFs of word docs in JIRAs are not the place for ongoing work, even if > the > > early drafts can contain them. Given it's just as easy to point to > markdown > > docs in github by commit ID, that could be an alternative way to publish > > docs, with the document itself being viewed as one of the deliverables. > > When the time comes to update a document, then its there in the source > tree > > to edit. > > > > If there's a flaw here, its that design docs are that: the design. The > > implementation may not match, ongoing work will certainly diverge. If the > > design docs aren't kept in sync, then they can mislead people. > Accordingly, > > once the design docs are incorporated into the source tree, keeping them > in > > sync with changes has be viewed as essential as keeping tests up to date > > > > > On 26 Apr 2015, at 22:34, Patrick Wendell <pwend...@gmail.com> wrote: > > > > > > I actually don't totally see why we can't use Google Docs provided it > > > is clearly discoverable from the JIRA. It was my understanding that > > > many projects do this. Maybe not (?). > > > > > > If it's a matter of maintaining public record on ASF infrastructure, > > > perhaps we can just automate that if an issue is closed we capture the > > > doc content and attach it to the JIRA as a PDF. > > > > > > My sense is that in general the ASF infrastructure policy is becoming > > > more and more lenient with regards to using third party services, > > > provided the are broadly accessible (such as a public google doc) and > > > can be definitively archived on ASF controlled storage. > > > > > > - Patrick > > > > > > On Fri, Apr 24, 2015 at 4:57 PM, Sean Owen <so...@cloudera.com> wrote: > > >> I know I recently used Google Docs from a JIRA, so am guilty as > > >> charged. I don't think there are a lot of design docs in general, but > > >> the ones I've seen have simply pushed docs to a JIRA. (I did the same, > > >> mirroring PDFs of the Google Doc.) I don't think this is hard to > > >> follow. > > >> > > >> I think you can do what you like: make a JIRA and attach files. Make a > > >> WIP PR and attach your notes. Make a Google Doc if you're feeling > > >> transgressive. > > >> > > >> I don't see much of a problem to solve here. In practice there are > > >> plenty of workable options, all of which are mainstream, and so I do > > >> not see an argument that somehow this is solved by letting people make > > >> wikis. > > >> > > >> On Fri, Apr 24, 2015 at 7:42 PM, Punyashloka Biswal > > >> <punya.bis...@gmail.com> wrote: > > >>> Okay, I can understand wanting to keep Git history clean, and avoid > > >>> bottlenecking on committers. Is it reasonable to establish a > > convention of > > >>> having a label, component or (best of all) an issue type for issues > > that are > > >>> associated with design docs? For example, if we used the existing > > >>> "Brainstorming" issue type, and people put their design doc in the > > >>> description of the ticket, it would be relatively easy to figure out > > what > > >>> designs are in progress. > > >>> > > >>> Given the push-back against design docs in Git or on the wiki and the > > strong > > >>> preference for keeping docs on ASF property, I'm a bit surprised that > > all > > >>> the existing design docs are on Google Docs. Perhaps Apache should > > consider > > >>> opening up parts of the wiki to a larger group, to better serve this > > use > > >>> case. > > >>> > > >>> Punya > > >>> > > >>> On Fri, Apr 24, 2015 at 5:01 PM Patrick Wendell <pwend...@gmail.com> > > wrote: > > >>>> > > >>>> Using our ASF git repository as a working area for design docs, it > > >>>> seems potentially concerning to me. It's difficult process wise > > >>>> because all commits need to go through committers and also, we'd > > >>>> pollute our git history a lot with random incremental design > updates. > > >>>> > > >>>> The git history is used a lot by downstream packagers, us during our > > >>>> QA process, etc... we really try to keep it oriented around code > > >>>> patches: > > >>>> > > >>>> https://git-wip-us.apache.org/repos/asf?p=spark.git;a=shortlog > > >>>> > > >>>> Committing a polished design doc along with a feature, maybe that's > > >>>> something we could consider. But I still think JIRA is the best > > >>>> location for these docs, consistent with what most other ASF > projects > > >>>> do that I know. > > >>>> > > >>>> On Fri, Apr 24, 2015 at 1:19 PM, Cody Koeninger <c...@koeninger.org > > > > >>>> wrote: > > >>>>> Why can't pull requests be used for design docs in Git if people > who > > >>>>> aren't > > >>>>> committers want to contribute changes (as opposed to just > comments)? > > >>>>> > > >>>>> On Fri, Apr 24, 2015 at 2:57 PM, Sean Owen <so...@cloudera.com> > > wrote: > > >>>>> > > >>>>>> Only catch there is it requires commit access to the repo. We > need a > > >>>>>> way for people who aren't committers to write and collaborate (for > > >>>>>> point #1) > > >>>>>> > > >>>>>> On Fri, Apr 24, 2015 at 3:56 PM, Punyashloka Biswal > > >>>>>> <punya.bis...@gmail.com> wrote: > > >>>>>>> Sandy, doesn't keeping (in-progress) design docs in Git satisfy > the > > >>>>>> history > > >>>>>>> requirement? Referring back to my Gradle example, it seems that > > >>>>>>> > > >>>>>> > > >>>>>> > > > https://github.com/gradle/gradle/commits/master/design-docs/build-comparison.md > > >>>>>>> is a really good way to see why the design doc evolved the way it > > >>>>>>> did. > > >>>>>> When > > >>>>>>> keeping the doc in Jira (presumably as an attachment) it's not > easy > > >>>>>>> to > > >>>>>> see > > >>>>>>> what changed between successive versions of the doc. > > >>>>>>> > > >>>>>>> Punya > > >>>>>>> > > >>>>>>> On Fri, Apr 24, 2015 at 3:53 PM Sandy Ryza < > > sandy.r...@cloudera.com> > > >>>>>> wrote: > > >>>>>>>> > > >>>>>>>> I think there are maybe two separate things we're talking about? > > >>>>>>>> > > >>>>>>>> 1. Design discussions and in-progress design docs. > > >>>>>>>> > > >>>>>>>> My two cents are that JIRA is the best place for this. It > allows > > >>>>>> tracking > > >>>>>>>> the progression of a design across multiple PRs and > > contributors. A > > >>>>>> piece > > >>>>>>>> of useful feedback that I've gotten in the past is to make > design > > >>>>>>>> docs > > >>>>>>>> immutable. When updating them in response to feedback, post a > new > > >>>>>> version > > >>>>>>>> rather than editing the existing one. This enables tracking the > > >>>>>> history of > > >>>>>>>> a design and makes it possible to read comments about previous > > >>>>>>>> designs > > >>>>>> in > > >>>>>>>> context. Otherwise it's really difficult to understand why > > >>>>>>>> particular > > >>>>>>>> approaches were chosen or abandoned. > > >>>>>>>> > > >>>>>>>> 2. Completed design docs for features that we've implemented. > > >>>>>>>> > > >>>>>>>> Perhaps less essential to project progress, but it would be > really > > >>>>>> lovely > > >>>>>>>> to have a central repository to all the projects design doc. If > > >>>>>>>> anyone > > >>>>>>>> wants to step up to maintain it, it would be cool to have a wiki > > >>>>>>>> page > > >>>>>> with > > >>>>>>>> links to all the final design docs posted on JIRA. > > >>>>>>>> > > >>>>>> > > > > > > --------------------------------------------------------------------- > > > To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org > > > For additional commands, e-mail: dev-h...@spark.apache.org > > > > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org > > For additional commands, e-mail: dev-h...@spark.apache.org > > > > >
