On 10/24/11 1:12 PM, "Olga Natkovich" <[email protected]> wrote:
>I am also is in favor of #2 and assumed that's the way we go: This is exactly what is needed IMO. Define what is acceptable for a major, minor, and patch change, but leave many of the minor details flexible for the project. "1.0" is just a number, its not a promise that there won't be API changes, that the project is 'production ready', or that there won't be a 2.0 any time soon. > >(1) Major version change: major disruptive or non-backward compatible >changes >(2) Minor version changes: small features + bug fixes; *no breakage of >backward compatibility* >(3) Patch version changes: P1 bug fixes; no new features, no >compatibility breakage. Minor version: "small features" is not well defined. I'd suggest leaving that out. A feature that does not affect backwards compatibility at all may be acceptable regardless of its subjective 'size'. I suggest the minor version definition should be broad: A minor release is a release that is neither a bug fix release nor major version release. It may include big new features, but is not a disruptive upgrade -- Users are not expected to have to modify any Pig scripts or UDFs. This allows major work to progress across many minor releases as long as the old APIs are still available. At a major release, users may have to change their Pig scrips, UDF classes, or APIs may be removed. I also suggest leaving out "no new features" from the Patch version definition -- a 'new feature' is often poorly defined and people will debate its meaning. Is an additional command line parameter that helps out some users in a backwards-compatible fashion a 'new feature' or a 'bug fix'? Its subjective. What is important is that what is in the patch version is very low risk and does not break compatibility in any way -- compatibility is objective, and 'low risk' although subjective, is more reasonable for committers to discuss than whether it is in the 'bug fix' or 'feature' category. -Scott > >Olga > >-----Original Message----- >From: Daniel Dai [mailto:[email protected]] >Sent: Monday, October 24, 2011 12:00 PM >To: [email protected] >Subject: Re: Next Pig release proposal > >Yes, we need a versioning scheme. There are two versioning scheme I can >think of: > >Scheme 1: ><major>.<patch> ><major> will be the feature rich release every 3 month ><patch> will be the bug fix release when necessary > >Nov release will be 1.0, Feb release will be 2.0. There will be 1.1, 2.1 >etc >for bug fixes. > >Scheme 2: ><major>.<minor>.<patch> >Most of our 3 month release will be counted as <minor> release unless >there >are major user facing/disruptive changes. > >Nov release will be 1.0.0, Feb release will be 1.1.0. There will be 1.0.1, >1.1.1 etc for bug fixes. > >I personally prefer scheme 2, increasing major version too frequently >might >be confusing to users. How's other folks feel? > >Daniel > > >On Sat, Oct 22, 2011 at 2:31 AM, Gianmarco De Francisci Morales < >[email protected]> wrote: > >> Hi, >> >> just my 2 cents. >> >> I think the issue here is not 1.0 vs 0.10, but what's the versioning >>scheme >> we want to use for Pig. >> Up to now it has been just an increasing number after a '0.' prefix, >> changed >> when the community felt it was time. I think this works well for a small >> project, but it is somewhat fuzzy. >> >> I like the idea of having <major>.<minor>.<patch> versions like many >>other >> projects. It's a very clear and almost standard way of versioning a >>piece >> of >> software. It has clear rules on when to change each of the numbers, and >> lets >> the user get an idea of backward compatibility at a glance. >> >> So, to conclude, I am in favor of going 1.0 (or 1.0.0) as long as we >>decide >> a clear versioning policy (whichever it is). >> So that the 1.0 milestone would mark the beginning of our new policy. >> >> Cheers, >> -- >> Gianmarco >> >> >> >> On Fri, Oct 21, 2011 at 23:10, <[email protected]> wrote: >> >> > If one were to rewrite input and output formats to use the webhdfs:// >> > APIs, this would not be an issue, right ? >> > >> > - milind >> > >> > >> > On 10/21/11 1:50 PM, "Santhosh Srinivasan" <[email protected]> wrote: >> > >> > >If I was not clear in my earlier email, I apologize for the lack of >> > >clarity. I am no longer in favour of waiting for Hadoop API stability >> > >across Hadoop versions. It's a pipe dream. >> > > >> > >When we had PigInputFormat and PigOutputFormat, your reasoning would >>be >> > >spot on. I am concerned about the following. Our tight integration >>with >> > >Hadoop due to the use of Input and Output format might lead to a >>break >> in >> > >backward compatibility. I am not sure if the comparison with that of >> Java >> > >is valid. Probably a majority of the users don't use JNI. Its very >>hard >> > >to use Pig without writing custom load and store functions. The >>default >> > >load and store don't suffice for a majority of use cases that I have >> > >observed. >> > > >> > >I am trying to get all factors that might influence this decision. >>From >> > >the few emails that have been exchanged since yesterday, we have the >> > >following factors: >> > > >> > >1. Hadoop 0.20.205 (support for Append) >> > >2. Hadoop 0.22 >> > >3. Hadoop 0.23 >> > >4. Maturity of the new parser >> > >5. Stability of the new logical plan >> > >6. Other components in the eco system. >> > > - Avro (1.5.4, 1.4.1, ...) >> > > - Cassandra (1.0.0, 0.8.7, ...) >> > > - Chukwa (0.4.0, 0.3.0, ...) >> > > - Hama (0.3.0, 0.2.0, ...) >> > > - Hbase (0.90.4, 0.90.3, 0.90.2, 0.90.1, ...) >> > > - Hive (Releases - 0.7.1, 0.7.0, 0.6.0, ...) >> > > - Zookeeper (3.3.3, 3.3.2, 3.2.2, 3.1.2, ...) >> > > >> > >Santhosh >> > > >> > > >> > >-----Original Message----- >> > >From: Thejas Nair [mailto:[email protected]] >> > >Sent: Friday, October 21, 2011 11:22 AM >> > >To: [email protected] >> > >Subject: Re: Next Pig release proposal >> > > >> > > >> > >Santosh, >> > >I thought you meant API stability for hadoop across major versions, >>but >> I >> > >guess you are referring to stability within 0.23 versions. But >>argument >> > >applies to that as well, if 0.23.1 is not compatible with 0.23.0, we >> need >> > >to call the release for 0.23.1 as 'pig 1.x for 0.23.1 api' . >> > > >> > >We just need to communicate to the users that the >> > >InputFormat/OutputFormat api's (and any anything else we expose from >> > >hadoop) depends on the hadoop version they are using. >> > > >> > >I think it is just like different JNI libraries that you would write >>for >> > >different OS. But the java version remains the same across OSs. >> > > >> > >-Thejas >> > > >> > > >> > >On 10/21/11 10:59 AM, Santhosh Srinivasan wrote: >> > >> Thejas, >> > >> >> > >> I guess you did not read my email completely. You are referring to >>the >> > >>premise without examining the conclusion. I am repasting my entire >> email >> > >>to avoid confusion (I hate truncated references). If you could >>respond >> > >>again, it will bring us onto the same page. >> > >> >> > >> <email> >> > >> >> > >> Ref: http://tinyurl.com/4ng8upa (last discussion on 1.0) >> > >> >> > >> How far have we progressed from our last discussion in March. There >> was >> > >>no consensus on the 1.0 release. Opinions ranged from having more >> > >>releases to bake in the maturity of the new parser and logical plan >> > >>changes to compatibility with Hadoop API (was compared to Social >> > >>Security - a very hot topic these days). >> > >> >> > >> My concerns were around Hadoop API stability. I have heard that the >> > >>APIs will not be stable for at least 1 year. This is taking me away >> from >> > >>the Hadoop API stability factor (They passed healthcare in that >> > >>duration. Really!) Do we want compatibility with 0.23 as a gating >> factor >> > >>- not sure if this is anywhere close to getting done in the near >> future. >> > >>Will we support append (0.20.205)? >> > >> >> > >> Btw, Hbase has been doing 0.90.1, 0.90.2, etc. So we can take a >>look >> at >> > >>this option too. >> > >> >> > >> Santhosh >> > >> >> > >> >> > >> >> > >> -----Original Message----- >> > >> From: Olga Natkovich [mailto:[email protected]] >> > >> Sent: Thursday, October 20, 2011 4:40 PM >> > >> To: [email protected] >> > >> Subject: Next Pig release proposal >> > >> >> > >> Hi, >> > >> >> > >> Here is what I propose we do for the next Pig release: >> > >> >> > >> >> > >> (1) Branch early next week - we have major features and many >>bug >> > >>fixes in and will be fixing remaining bugs on the branch >> > >> >> > >> (2) Publish the release by 11/15 - that will give us a couple of >> > >>weeks to stabilize the branch and get last minute bug fixes in >> > >> >> > >> (3) Make this release a 1.0 release. Reasons to go for 1.0 and >>not >> > >>0.10 >> > >> >> > >> a. This release has minimal number of features and was >>focused >> on >> > >>code stabilization and bug fixes. We believe it will be a stable >> release >> > >> >> > >> <email/> >> > >> >> > >> Thanks, >> > >> Santhosh >> > >> >> > >> -----Original Message----- >> > >> From: Thejas Nair [mailto:[email protected]] >> > >> Sent: Friday, October 21, 2011 10:45 AM >> > >> To: [email protected] >> > >> Subject: Re: Next Pig release proposal >> > >> >> > >> On 10/20/11 4:58 PM, Santhosh Srinivasan wrote: >> > >>> Ref: http://tinyurl.com/4ng8upa (last discussion on 1.0) >> > >>> >> > >>> How far have we progressed from our last discussion in March. >>There >> > >>>was no consensus on the 1.0 release. Opinions ranged from having >>more >> > >>>releases to bake in the maturity of the new parser and logical plan >> > >>>changes to compatibility with Hadoop API (was compared to Social >> > >>>Security - a very hot topic these days). >> > >>> >> > >>> My concerns were around Hadoop API stability. >> > >> >> > >> Over the next year or so, there are going to be two API versions of >> > >>hadoop to be supported - 0.20.x api's and 0.23 apis, as we will have >> > >>userbase on both. >> > >> >> > >> I think it is just a matter of releasing pig 1.0 for 0.20.x api's >>and >> > >>1.0 for 0.23.x api's. We will have to come up with a numbering >>scheme >> > >>that reflects 'for hadoop version X' in our pig releases, >>regardless of >> > >>it being 0.10 or 1.0. >> > >> >> > >> As there will be support for different api's of hadoop in pig >> releases, >> > >>I don't see a reason why the hadoop api stability should stop pig >>from >> > >>going 1.0 . >> > >> >> > >> -Thejas >> > > >> > > >> > >> > >>
