Yup, I'm already started on that process. And it's not that I disagree with any particular change that was merged per se -- I haven't seen anything merged that most users won't want. It's more that I object to the burden that our current development/versioning/release process puts on Spark users responsible for production code. For them, adopting a new patch-level release should be a decision requiring almost no thinking since the new release should be essentially just bug-fixes that maintain full binary compatibility. With our current process, those users have to suck in a bunch of new, less-tested, less-mature code that may comprise new features or functionality that the user doesn't want (at least not right away in production), but that they can't cleanly separate from the bug-fixes that they do want. Our process simply has to change if we place users' desires ahead of Spark developers' desires.
On Sun, Dec 8, 2013 at 2:12 PM, Patrick Wendell <pwend...@gmail.com> wrote: > Hey Mark, > > One constructive action you and other people can take to help us > assess the quality and completeness of this release is to download the > release, run the tests, run the release in your dev environment, read > through the documentation, etc. This is one of the main points of > releasing an RC to the community... even if you disagree with some > patches that were merged in, this is still a way you can help validate > the release. > > - Patrick > > On Sun, Dec 8, 2013 at 1:30 PM, Mark Hamstra <m...@clearstorydata.com> > wrote: > > I'm aware of the changes file, but it really doesn't address the issue > that > > I am raising. The changes file just tells me what has gone into the > > release candidate. In general, it doesn't tell me why those changes went > > in or provide any rationale by which to judge whether that is the > complete > > set of changes that should go in. > > > > I talked some with Matei about related versioning and release issues last > > week, and I've raised them in other contexts previously, but I'm taking > the > > liberty to annoy people again because I really am not happy with our > > current versioning and release process, and I really am of the opinion > that > > we've got to start doing much better before I can vote in favor of a 1.0 > > release. I fully realize that this is not a 1.0 release, and that > because > > we are pre-1.0 we still have a lot of flexibility with releases that > break > > backward or forward compatibility and with version numbers that have > > nothing like the semantic meaning that they will eventually need to have; > > but it is not going to be easy to change our process and culture so that > we > > produce the kind of stability and reliability that Spark users need to be > > able to depend upon and version numbers that clearly communicate what > those > > users expect them to mean. I think that we should start making those > > changes now. Just because we have flexibility pre-1.0, that doesn't mean > > that we shouldn't start training ourselves now to work within the > > constraints of post-1.0 Spark. If I'm to be happy voting for an eventual > > 1.0 release candidate, I'll need to have seen at least one full > development > > cycle that already adheres to the post-1.0 constraints, demonstrating the > > maturity of our development process. > > > > That demonstration cycle is clearly not this one -- and I understand that > > there were some compelling reasons (particularly with regard too getting > a > > "full" release of Spark based on Scala 2.9.3 before we make the jump to > > 2.10. This "patch-level" release breaks binary compatibility and > contains > > a lot of code that isn't anywhere close to meeting the criterion for > > inclusion in a real, post-1.0 patch-level release: essentially "changes > > that every, or nearly every, existing Spark user needs (not just wants), > > and that work with all existing and future binaries built with the prior > > patch-level version of Spark as a dependency." Like I said, we are > clearly > > nowhere close to that with the move from 0.8.0 to 0.8.1; but I also > haven't > > been able to recognize any alternative criterion by which to judge the > > quality and completeness of this release candidate. > > > > Maybe there just isn't one, and I'm just going to have to swallow my > > concerns while watching 0.8.1 go out the door; but if we don't start > doing > > better on this kind of thing in the future, you are going to start > hearing > > more complaining from me. I just hope that it doesn't get to the point > > where I feel compelled to actively oppose an eventual 1.0 release > > candidate. > > > > > > On Sun, Dec 8, 2013 at 12:37 PM, Henry Saputra <henry.sapu...@gmail.com > >wrote: > > > >> Ah, sorry for the confusion Patrick, like you said I was just trying to > let > >> people aware about this file and the purpose of it. > >> > >> On Sunday, December 8, 2013, Patrick Wendell wrote: > >> > >> > Hey Henry, > >> > > >> > Are you suggesting we need to change something about or changes file? > >> > Or are you just pointing people to the file? > >> > > >> > - Patrick > >> > > >> > On Sun, Dec 8, 2013 at 11:37 AM, Henry Saputra < > henry.sapu...@gmail.com> > >> > wrote: > >> > > HI Spark devs, > >> > > > >> > > I have modified the Subject to avoid polluting the VOTE thread since > >> > > it related to more info how and which commits merge back to 0.8.* > >> > > branch. > >> > > Please respond to the previous question to this thread. > >> > > > >> > > Technically the CHANGES.txt [1] file should describe the changes in > a > >> > > particular release and it is the main requirement needed to cut an > ASF > >> > > release. > >> > > > >> > > > >> > > - Henry > >> > > > >> > > [1] > >> > https://github.com/apache/incubator-spark/blob/branch-0.8/CHANGES.txt > >> > > > >> > > On Sun, Dec 8, 2013 at 12:03 AM, Josh Rosen <rosenvi...@gmail.com> > >> > wrote: > >> > >> We can use git log to figure out which changes haven't made it into > >> > >> branch-0.8. Here's a quick attempt, which only lists pull requests > >> that > >> > >> were only merged into one of the branches. For completeness, this > >> > could be > >> > >> extended to find commits that weren't part of a merge and are only > >> > present > >> > >> in one branch. > >> > >> > >> > >> *Script:* > >> > >> > >> > >> MASTER_BRANCH=origin/master > >> > >> RELEASE_BRANCH=origin/branch-0.8 > >> > >> > >> > >> git log --oneline --grep "Merge pull request" $MASTER_BRANCH | > cut -f > >> > 2- > >> > >> -d ' ' | sort > master-prs > >> > >> git log --oneline --grep "Merge pull request" $RELEASE_BRANCH | > cut -f > >> > 2- > >> > >> -d ' ' | sort > release-prs > >> > >> > >> > >> comm -23 master-prs release-prs > master-only > >> > >> comm -23 release-prs master-prs > release-only > >> > >> > >> > >> > >> > >> *Master Branch Only:* > >> > >> Merge pull request #1 from colorant/yarn-client-2.2 > >> > >> Merge pull request #105 from pwendell/doc-fix > >> > >> Merge pull request #110 from pwendell/master > >> > >> Merge pull request #146 from JoshRosen/pyspark-custom-serializers > >> > >> Merge pull request #151 from russellcardullo/add-graphite-sink > >> > >> Merge pull request #154 from soulmachine/ClusterScheduler > >> > >> Merge pull request #156 from haoyuan/master > >> > >> Merge pull request #159 from liancheng/dagscheduler-actor-refine > >> > >> Merge pull request #16 from pwendell/master > >> > >> Merge pull request #185 from mkolod/random-number-generator > >> > >> Merge pull request #187 from aarondav/example-bcast-test > >> > >> Merge pull request #190 from markhamstra/Stages4Jobs > >> > >> Merge pull request #198 from > >> > ankurdave/zipPartitions-preservesPartitioning > >> > >> Merge pull request #2 from colorant/yarn-client-2.2 > >> > >> Merge pull request #203 from witgo/master > >> > >> Merge pull request #204 from rxin/hash > >> > >> Merge pull request #205 from kayousterhout/logging > >> > >> Merge pull request #206 from ash211/patch-2 > >> > >> Merge pull request #207 from henrydavidge/master > >> > >> Merge pull request #209 from pwendell/better-docs > >> > >> Merge pull request #210 from haitaoyao/http-timeout > >> > >> Merge pull request #212 from markhamstra/SPARK-963 > >> > >> Merge pull request #216 from liancheng/fix-spark-966 > >> > >> Merge pull request #217 from aarondav/mesos-urls > >> > >> Merge pull request #22 from GraceH/metrics-naming > >> > >> Merge pull request #220 from rxin/zippart > >> > >> Merge pull request #225 from ash211/patch-3 > >> > >> Merge pull request #226 from ash211/patch-4 > >> > >> Merge pull request #233 from hsaputra/changecontexttobackend > >> > >> Merge pull request #239 from aarondav/nit > >> > >> Merge pull request #242 from pwendell/master > >> > >> Merge pull request #3 from aarondav/pv-test > >> > >> Merge pull request #36 from pwendell/versions > >> > >> Merge pull request #37 from pwendell/merge-0.8 > >> > >> Merge pull request #39 from pwendell/master > >> > >> Merge pull request #45 from pwendell/metrics_units > >> > >> Merge pull request #56 from jerryshao/kafka-0.8-dev > >> > >> Merge pull request #64 from prabeesh/master > >> > >> Merge pull request #66 from shivaram/sbt-assembly-deps > >> > >> Merge pull request #670 from jey/ec2-ssh-improvements > >> > >> Merge pull request #71 from aarondav/scdefaults > >> > >> Merge pull request #78 from mosharaf/master > >> > >> Merge pull request #8 from vchekan/checkpoint-ttl-restore > >> > >> Merge pull request #80 from rxin/build > >> > >> Merge pull request #82 from JoshRosen/map-output-t > >> >
