If I was not clear in my earlier email, I apologize for the lack of clarity. I
am no longer in favour of waiting for Hadoop API stability across Hadoop
versions. It's a pipe dream.
When we had PigInputFormat and PigOutputFormat, your reasoning would be spot
on. I am concerned about the following. Our tight integration with Hadoop due
to the use of Input and Output format might lead to a break in backward
compatibility. I am not sure if the comparison with that of Java is valid.
Probably a majority of the users don't use JNI. Its very hard to use Pig
without writing custom load and store functions. The default load and store
don't suffice for a majority of use cases that I have observed.
I am trying to get all factors that might influence this decision. From the few
emails that have been exchanged since yesterday, we have the following factors:
1. Hadoop 0.20.205 (support for Append)
2. Hadoop 0.22
3. Hadoop 0.23
4. Maturity of the new parser
5. Stability of the new logical plan
6. Other components in the eco system.
- Avro (1.5.4, 1.4.1, ...)
- Cassandra (1.0.0, 0.8.7, ...)
- Chukwa (0.4.0, 0.3.0, ...)
- Hama (0.3.0, 0.2.0, ...)
- Hbase (0.90.4, 0.90.3, 0.90.2, 0.90.1, ...)
- Hive (Releases - 0.7.1, 0.7.0, 0.6.0, ...)
- Zookeeper (3.3.3, 3.3.2, 3.2.2, 3.1.2, ...)
Santhosh
-----Original Message-----
From: Thejas Nair [mailto:the...@hortonworks.com]
Sent: Friday, October 21, 2011 11:22 AM
To: dev@pig.apache.org
Subject: Re: Next Pig release proposal
Santosh,
I thought you meant API stability for hadoop across major versions, but I guess
you are referring to stability within 0.23 versions. But argument applies to
that as well, if 0.23.1 is not compatible with 0.23.0, we need to call the
release for 0.23.1 as 'pig 1.x for 0.23.1 api' .
We just need to communicate to the users that the InputFormat/OutputFormat
api's (and any anything else we expose from
hadoop) depends on the hadoop version they are using.
I think it is just like different JNI libraries that you would write for
different OS. But the java version remains the same across OSs.
-Thejas
On 10/21/11 10:59 AM, Santhosh Srinivasan wrote:
> Thejas,
>
> I guess you did not read my email completely. You are referring to the
> premise without examining the conclusion. I am repasting my entire email to
> avoid confusion (I hate truncated references). If you could respond again, it
> will bring us onto the same page.
>
> <email>
>
> Ref: http://tinyurl.com/4ng8upa (last discussion on 1.0)
>
> How far have we progressed from our last discussion in March. There was no
> consensus on the 1.0 release. Opinions ranged from having more releases to
> bake in the maturity of the new parser and logical plan changes to
> compatibility with Hadoop API (was compared to Social Security - a very hot
> topic these days).
>
> My concerns were around Hadoop API stability. I have heard that the APIs will
> not be stable for at least 1 year. This is taking me away from the Hadoop API
> stability factor (They passed healthcare in that duration. Really!) Do we
> want compatibility with 0.23 as a gating factor - not sure if this is
> anywhere close to getting done in the near future. Will we support append
> (0.20.205)?
>
> Btw, Hbase has been doing 0.90.1, 0.90.2, etc. So we can take a look at this
> option too.
>
> Santhosh
>
>
>
> -----Original Message-----
> From: Olga Natkovich [mailto:ol...@yahoo-inc.com]
> Sent: Thursday, October 20, 2011 4:40 PM
> To: dev@pig.apache.org
> Subject: Next Pig release proposal
>
> Hi,
>
> Here is what I propose we do for the next Pig release:
>
>
> (1) Branch early next week - we have major features and many bug fixes in
> and will be fixing remaining bugs on the branch
>
> (2) Publish the release by 11/15 - that will give us a couple of weeks to
> stabilize the branch and get last minute bug fixes in
>
> (3) Make this release a 1.0 release. Reasons to go for 1.0 and not 0.10
>
> a. This release has minimal number of features and was focused on code
> stabilization and bug fixes. We believe it will be a stable release
>
> <email/>
>
> Thanks,
> Santhosh
>
> -----Original Message-----
> From: Thejas Nair [mailto:the...@hortonworks.com]
> Sent: Friday, October 21, 2011 10:45 AM
> To: dev@pig.apache.org
> Subject: Re: Next Pig release proposal
>
> On 10/20/11 4:58 PM, Santhosh Srinivasan wrote:
>> Ref: http://tinyurl.com/4ng8upa (last discussion on 1.0)
>>
>> How far have we progressed from our last discussion in March. There was no
>> consensus on the 1.0 release. Opinions ranged from having more releases to
>> bake in the maturity of the new parser and logical plan changes to
>> compatibility with Hadoop API (was compared to Social Security - a very hot
>> topic these days).
>>
>> My concerns were around Hadoop API stability.
>
> Over the next year or so, there are going to be two API versions of hadoop to
> be supported - 0.20.x api's and 0.23 apis, as we will have userbase on both.
>
> I think it is just a matter of releasing pig 1.0 for 0.20.x api's and 1.0 for
> 0.23.x api's. We will have to come up with a numbering scheme that reflects
> 'for hadoop version X' in our pig releases, regardless of it being 0.10 or
> 1.0.
>
> As there will be support for different api's of hadoop in pig releases, I
> don't see a reason why the hadoop api stability should stop pig from going
> 1.0 .
>
> -Thejas
