I don't mind trying to put together a patch for what I described above if there's a general consensus on the strategy we should take (or at least no big objections).
I think the multiple jars solution could be troublesome, but maybe after seeing what a patch looks like for a single jar solution it'll seem nicer. On Mon, Sep 30, 2013 at 2:45 PM, Cheolsoo Park <piaozhe...@gmail.com> wrote: > Hi Jeremy, > > What you're saying makes sense, and patch is welcome. ;-) But complexity > comes from that there are many classes that are associated with one > another, and it seems necessary to bring back all of them together in order > to provide full backward compatibility. > > After spending many hours on the weekend, I concluded that adding more > workarounds (classes, methods, packages, etc) to the current code makes it > only less maintainable and readable. So I prefer a simpler approach. > > For eg, we can just publish two jars - pig.jar w/ old API and pig-new.jar > w/ new API - maybe not in 0.12 but in 0.13. Since we already have a > tez-branch, we can use it to manage the new version of classes. Then, users > can switch to pig-new.jar gradually in 0.13 and 0.14. When we finally merge > tez-branch into trunk, we can publish a single jar again. > > Of course, this is not trivial either because we have to maintain two > branches. But I feel that managing two branches independently is easier > than maintaining all sorts of workarounds for backward compatibility in the > source code. In addition, we will have more flexibility in terms of > designing new API because we will be completely free from backward > compatibility. No? > > Thanks, > Cheolsoo > > On Mon, Sep 30, 2013 at 11:12 AM, Jeremy Karn <jk...@mortardata.com> > wrote: > > > What about the option of leaving all of the MR specific logic in the > > original classes but marking those methods as deprecated and telling > people > > to switch to using a MR specific object that extends the original class. > > So for example: > > > > JobStats - Reverted to being as it was before PIG-3419 but with all MR > > specific logic deprecated. > > MRJobStats - Would just extend JobStats. > > > > If we did this, external software could switch their code from using > > JobStats to MRJobStats at their own pace and without breaking against any > > specific version of Pig. After a few versions the MR specific logic > could > > be removed from JobStats and pushed into MRJobStats and it shouldn't > break > > anything for people that had made that change. > > > > I'm not familiar with all of the changes in PIG-3419 so this might not > work > > everywhere. > > > > > > On Mon, Sep 30, 2013 at 1:43 PM, Cheolsoo Park <piaozhe...@gmail.com> > > wrote: > > > > > To be specific, we will need to revert all the following commits in > > order: > > > > > > > > > commit ad1b87d4ba073680ad0a7fc8c76baeb8b611c982 > > > Author: Cheolsoo Park <cheol...@apache.org> > > > Date: Fri Sep 20 22:47:29 2013 +0000 > > > > > > PIG-3471: Add a base abstract class for ExecutionEngine (cheolsoo) > > > > > > git-svn-id: > > > > > > > > > https://svn.apache.org/repos/asf/pig/trunk@152516513f79535-47bb-0310-9956-ffa450edef68 > > > > > > commit 4305a6f4737d07396ae13fd95d7c1da7933b38a1 > > > Author: Jianyong Dai <da...@apache.org> > > > Date: Wed Sep 18 19:09:49 2013 +0000 > > > > > > PIG-3457: Provide backward compatibility for PigStatsUtil and > > JobStats > > > > > > git-svn-id: > > > > > > > > > https://svn.apache.org/repos/asf/pig/trunk@152453213f79535-47bb-0310-9956-ffa450edef68 > > > > > > commit e85cf34c92713aa697a1cda7a9c2b3db139350f7 > > > Author: Cheolsoo Park <cheol...@apache.org> > > > Date: Wed Sep 18 15:37:58 2013 +0000 > > > > > > PIG-3464: Mark ExecType and ExecutionEngine interfaces as evolving > > > (cheolsoo) > > > > > > commit fd8b7cdf9292b305f02386d560c25298ab492a0b > > > Author: Cheolsoo Park <cheol...@apache.org> > > > Date: Fri Aug 30 20:04:29 2013 +0000 > > > > > > PIG-3419: Pluggable Execution Engine (achalsoni81 via cheolsoo) > > > > > > git-svn-id: > > > > > > > > > https://svn.apache.org/repos/asf/pig/trunk@151906213f79535-47bb-0310-9956-ffa450edef68 > > > > > > > > > > > > > > > On Mon, Sep 30, 2013 at 10:33 AM, Daniel Dai <da...@hortonworks.com> > > > wrote: > > > > > > > Thanks Cheolsoo! My opinion is provide backward compatibility for > > > PigStats > > > > is a must, otherwise it could be a havoc. I imagine PigStats is > widely > > > used > > > > by Pig users via PigRunner and PPNL interface. People use PigStats to > > > > collect MR job details of the Pig job. Though PigStats is marked for > > > > Evolving, this is mostly for extending PigStats, not limiting > PigStats > > as > > > > PIG-3419 did. Even if we really need to change, we need to very well > > > > communicate with users over time, Pig 0.12 is not an option. > > > > > > > > PIG-3457 is trying to provide a backward compatibility way for > > PigStats, > > > > but just like Cheolsoo said, it is far from ideal. I now tend to > agree > > > > Rohini's suggestion on PIG-3419, rollback PIG-3419, until we find a > > > better > > > > way. Seems PIG-3419 is a little premature. Besides the above > mentioned > > > > PigStats issue, I've already found 2 additional issues: > > > > 1. "explain" shows unoptimized logical plan instead of optimized one > > > > 2. HangingJobKiller is removed > > > > > > > > How does others think? > > > > > > > > Thanks, > > > > Daniel > > > > > > > > > > > > > > > > > > > > On Mon, Sep 30, 2013 at 9:51 AM, Cheolsoo Park <piaozhe...@gmail.com > > > > > > wrote: > > > > > > > > > Hi devs, > > > > > > > > > > PIG-3419 <https://issues.apache.org/jira/browse/PIG-3419> broke > > > backward > > > > > compatibility for downstream applications such as Oozie, and > > > > > PIG-3457<https://issues.apache.org/jira/browse/PIG-3457> is > > > > > trying to fix it. > > > > > > > > > > In summary, we need to keep the old MR-specific JobStats and > PigStats > > > for > > > > > backward compatibility sadly because they're currently exposed in > > > several > > > > > user-facing API including PigRunner, PigServer, etc. However, this > > > > defeats > > > > > the purpose of PIG-3419 > > > > > <https://issues.apache.org/jira/browse/PIG-3419> because > > > > > we cannot implement non-MR execution engines in the back-end if the > > > > > front-end API is tied to MR. > > > > > > > > > > Daniel and I were having a discussion in the jira, but it seems > more > > > > > complicated than I thought. I am wondering whether anyone has a > good > > > > > suggestion on how to solve it. This is blocking the 0.12 release. > > > > > > > > > > Thanks, > > > > > Cheolsoo > > > > > > > > > > > > > -- > > > > CONFIDENTIALITY NOTICE > > > > NOTICE: This message is intended for the use of the individual or > > entity > > > to > > > > which it is addressed and may contain information that is > confidential, > > > > privileged and exempt from disclosure under applicable law. If the > > reader > > > > of this message is not the intended recipient, you are hereby > notified > > > that > > > > any printing, copying, dissemination, distribution, disclosure or > > > > forwarding of this communication is strictly prohibited. If you have > > > > received this communication in error, please contact the sender > > > immediately > > > > and delete it from your system. Thank You. > > > > > > > > > > > > > > > -- > > > > Jeremy Karn / Lead Developer > > MORTAR DATA / 519 277 4391 / www.mortardata.com > > > -- Jeremy Karn / Lead Developer MORTAR DATA / 519 277 4391 / www.mortardata.com
