Re: Time to address the Guava version problem
I've a patch for HADOOP-11102 which rolls curator back to v 2.4.1, which only pulls in Guava 14...hadoop should now be weakly consistent -at least not "strongly inconsistent"- in its guava versions. allowing hadoop to work on 16.x while still remaining compatible with 11.x is still something to work on -there's some patches there already On 24 September 2014 07:35, Billie Rinaldi wrote: > The use of an unnecessarily old dependency encourages problems like > HDFS-7040. The current Guava dependency is a big problem for downstream > apps and I'd really like to see it addressed. > > On Tue, Sep 23, 2014 at 2:09 PM, Steve Loughran > wrote: > > > I'm using curator elsewhere, it does log a lot (as does the ZK client), > but > > it solves a lot of problem. It's being adopted more downstream too. > > > > I'm wondering if we can move the code to the extent we know it works with > > Guava 16, with the hadoop core being 16-compatible, but not actually > > migrated to 16.x only. Then hadoop ships with 16 for curator & downstream > > apps, but we say "you can probably roll back to 11 provided you don't use > > features x-y-z". > > > > On 23 September 2014 21:55, Robert Kanter wrote: > > > > > At the same time, not being able to use Curator will require a lot of > > extra > > > code, a lot of which we probably already have from the ZKRMStateStore, > > but > > > it's not available to use in hadoop-auth. We'd need to create our own > ZK > > > libraries that Hadoop components can use, but (a) that's going to take > a > > > while, and (b) it seems silly to reinvent the wheel when Curator > already > > > does all this. > > > > > > I agree that upgrading Guava will be a compatibility problem though... > > > > > > On Tue, Sep 23, 2014 at 9:30 AM, Sandy Ryza > > > wrote: > > > > > > > If we've broken compatibility in branch-2, that's a bug that we need > to > > > > fix. HADOOP-10868 has not yet made it into a release; I don't see it > > as a > > > > justification for solidifying the breakage. > > > > > > > > -1 to upgrading Guava in branch-2. > > > > > > > > On Tue, Sep 23, 2014 at 3:06 AM, Steve Loughran < > > ste...@hortonworks.com> > > > > wrote: > > > > > > > > > +1 to upgrading guava. Irrespective of downstream apps, the hadoop > > > source > > > > > tree is now internally inconsistent > > > > > > > > > > On 22 September 2014 17:56, Sangjin Lee wrote: > > > > > > > > > > > I agree that a more robust solution is to have better > classloading > > > > > > isolation. > > > > > > > > > > > > Still, IMHO guava (and possibly protobuf as well) sticks out > like a > > > > sore > > > > > > thumb. There are just too many issues in trying to support both > > guava > > > > 11 > > > > > > and guava 16. Independent of what we may do with the classloading > > > > > > isolation, we should still consider upgrading guava. > > > > > > > > > > > > My 2 cents. > > > > > > > > > > > > On Sun, Sep 21, 2014 at 3:11 PM, Karthik Kambatla < > > > ka...@cloudera.com> > > > > > > wrote: > > > > > > > > > > > > > Upgrading Guava version is tricky. While it helps in many > cases, > > it > > > > can > > > > > > > break existing applications/deployments. I understand we do not > > > have > > > > a > > > > > > > policy for updating dependencies, but still we should be > careful > > > with > > > > > > > Guava. > > > > > > > > > > > > > > I would be more inclined towards a more permanent solution to > > this > > > > > > problem > > > > > > > - how about prioritizing classpath isolation so applications > > aren't > > > > > > > affected by Hadoop dependency updates at all? I understand that > > > will > > > > > also > > > > > > > break user applications, but it might be the driving feature > for > > > > Hadoop > > > > > > > 3.0? > > > > > > > > > > > > > > On Fri, Sep 19, 2014 at 5:13 PM, Sangjin Lee > > > > > wrote: > > > > > > > > > > > > > > > I would also agree on upgrading guava. Yes I am aware of the > > > > > potential > > > > > > > > impact on customers who might rely on hadoop bringing in > guava > > > 11. > > > > > > > However, > > > > > > > > IMHO the balance tipped over to the other side a while ago; > > i.e. > > > I > > > > > > think > > > > > > > > there are far more people using guava 16 in their code and > > > > scrambling > > > > > > to > > > > > > > > make things work than the other way around. > > > > > > > > > > > > > > > > On Thu, Sep 18, 2014 at 2:40 PM, Steve Loughran < > > > > > > ste...@hortonworks.com> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > I know we've been ignoring the Guava version problem, but > > > > > > HADOOP-10868 > > > > > > > > > added a transitive dependency on Guava 16 by way of Curator > > > 2.6. > > > > > > > > > > > > > > > > > > Maven currently forces the build to use Guava 11.0.2, but > > this > > > is > > > > > > > hiding > > > > > > > > at > > > > > > > > > compile timeall code paths from curator which may use > > classes & > > > > > > methods > > > > > > > > > that aren't there. > > > > >
Re: Time to address the Guava version problem
The use of an unnecessarily old dependency encourages problems like HDFS-7040. The current Guava dependency is a big problem for downstream apps and I'd really like to see it addressed. On Tue, Sep 23, 2014 at 2:09 PM, Steve Loughran wrote: > I'm using curator elsewhere, it does log a lot (as does the ZK client), but > it solves a lot of problem. It's being adopted more downstream too. > > I'm wondering if we can move the code to the extent we know it works with > Guava 16, with the hadoop core being 16-compatible, but not actually > migrated to 16.x only. Then hadoop ships with 16 for curator & downstream > apps, but we say "you can probably roll back to 11 provided you don't use > features x-y-z". > > On 23 September 2014 21:55, Robert Kanter wrote: > > > At the same time, not being able to use Curator will require a lot of > extra > > code, a lot of which we probably already have from the ZKRMStateStore, > but > > it's not available to use in hadoop-auth. We'd need to create our own ZK > > libraries that Hadoop components can use, but (a) that's going to take a > > while, and (b) it seems silly to reinvent the wheel when Curator already > > does all this. > > > > I agree that upgrading Guava will be a compatibility problem though... > > > > On Tue, Sep 23, 2014 at 9:30 AM, Sandy Ryza > > wrote: > > > > > If we've broken compatibility in branch-2, that's a bug that we need to > > > fix. HADOOP-10868 has not yet made it into a release; I don't see it > as a > > > justification for solidifying the breakage. > > > > > > -1 to upgrading Guava in branch-2. > > > > > > On Tue, Sep 23, 2014 at 3:06 AM, Steve Loughran < > ste...@hortonworks.com> > > > wrote: > > > > > > > +1 to upgrading guava. Irrespective of downstream apps, the hadoop > > source > > > > tree is now internally inconsistent > > > > > > > > On 22 September 2014 17:56, Sangjin Lee wrote: > > > > > > > > > I agree that a more robust solution is to have better classloading > > > > > isolation. > > > > > > > > > > Still, IMHO guava (and possibly protobuf as well) sticks out like a > > > sore > > > > > thumb. There are just too many issues in trying to support both > guava > > > 11 > > > > > and guava 16. Independent of what we may do with the classloading > > > > > isolation, we should still consider upgrading guava. > > > > > > > > > > My 2 cents. > > > > > > > > > > On Sun, Sep 21, 2014 at 3:11 PM, Karthik Kambatla < > > ka...@cloudera.com> > > > > > wrote: > > > > > > > > > > > Upgrading Guava version is tricky. While it helps in many cases, > it > > > can > > > > > > break existing applications/deployments. I understand we do not > > have > > > a > > > > > > policy for updating dependencies, but still we should be careful > > with > > > > > > Guava. > > > > > > > > > > > > I would be more inclined towards a more permanent solution to > this > > > > > problem > > > > > > - how about prioritizing classpath isolation so applications > aren't > > > > > > affected by Hadoop dependency updates at all? I understand that > > will > > > > also > > > > > > break user applications, but it might be the driving feature for > > > Hadoop > > > > > > 3.0? > > > > > > > > > > > > On Fri, Sep 19, 2014 at 5:13 PM, Sangjin Lee > > > wrote: > > > > > > > > > > > > > I would also agree on upgrading guava. Yes I am aware of the > > > > potential > > > > > > > impact on customers who might rely on hadoop bringing in guava > > 11. > > > > > > However, > > > > > > > IMHO the balance tipped over to the other side a while ago; > i.e. > > I > > > > > think > > > > > > > there are far more people using guava 16 in their code and > > > scrambling > > > > > to > > > > > > > make things work than the other way around. > > > > > > > > > > > > > > On Thu, Sep 18, 2014 at 2:40 PM, Steve Loughran < > > > > > ste...@hortonworks.com> > > > > > > > wrote: > > > > > > > > > > > > > > > I know we've been ignoring the Guava version problem, but > > > > > HADOOP-10868 > > > > > > > > added a transitive dependency on Guava 16 by way of Curator > > 2.6. > > > > > > > > > > > > > > > > Maven currently forces the build to use Guava 11.0.2, but > this > > is > > > > > > hiding > > > > > > > at > > > > > > > > compile timeall code paths from curator which may use > classes & > > > > > methods > > > > > > > > that aren't there. > > > > > > > > > > > > > > > > I need curator for my own work (2.4.1 & Guava 14.0 was what > I'd > > > > been > > > > > > > > using), so don't think we can go back. > > > > > > > > > > > > > > > > HADOOP-11102 covers the problem -but doesn't propose a > specific > > > > > > solution. > > > > > > > > But to me the one that seems most likely to work is: update > > Guava > > > > > > > > > > > > > > > > -steve > > > > > > > > > > > > > > > > -- > > > > > > > > CONFIDENTIALITY NOTICE > > > > > > > > NOTICE: This message is intended for the use of the > individual > > or > > > > > > entity > > > > > > > to > > > > > > > > which it is addressed and may contain informat
Re: Time to address the Guava version problem
I'm using curator elsewhere, it does log a lot (as does the ZK client), but it solves a lot of problem. It's being adopted more downstream too. I'm wondering if we can move the code to the extent we know it works with Guava 16, with the hadoop core being 16-compatible, but not actually migrated to 16.x only. Then hadoop ships with 16 for curator & downstream apps, but we say "you can probably roll back to 11 provided you don't use features x-y-z". On 23 September 2014 21:55, Robert Kanter wrote: > At the same time, not being able to use Curator will require a lot of extra > code, a lot of which we probably already have from the ZKRMStateStore, but > it's not available to use in hadoop-auth. We'd need to create our own ZK > libraries that Hadoop components can use, but (a) that's going to take a > while, and (b) it seems silly to reinvent the wheel when Curator already > does all this. > > I agree that upgrading Guava will be a compatibility problem though... > > On Tue, Sep 23, 2014 at 9:30 AM, Sandy Ryza > wrote: > > > If we've broken compatibility in branch-2, that's a bug that we need to > > fix. HADOOP-10868 has not yet made it into a release; I don't see it as a > > justification for solidifying the breakage. > > > > -1 to upgrading Guava in branch-2. > > > > On Tue, Sep 23, 2014 at 3:06 AM, Steve Loughran > > wrote: > > > > > +1 to upgrading guava. Irrespective of downstream apps, the hadoop > source > > > tree is now internally inconsistent > > > > > > On 22 September 2014 17:56, Sangjin Lee wrote: > > > > > > > I agree that a more robust solution is to have better classloading > > > > isolation. > > > > > > > > Still, IMHO guava (and possibly protobuf as well) sticks out like a > > sore > > > > thumb. There are just too many issues in trying to support both guava > > 11 > > > > and guava 16. Independent of what we may do with the classloading > > > > isolation, we should still consider upgrading guava. > > > > > > > > My 2 cents. > > > > > > > > On Sun, Sep 21, 2014 at 3:11 PM, Karthik Kambatla < > ka...@cloudera.com> > > > > wrote: > > > > > > > > > Upgrading Guava version is tricky. While it helps in many cases, it > > can > > > > > break existing applications/deployments. I understand we do not > have > > a > > > > > policy for updating dependencies, but still we should be careful > with > > > > > Guava. > > > > > > > > > > I would be more inclined towards a more permanent solution to this > > > > problem > > > > > - how about prioritizing classpath isolation so applications aren't > > > > > affected by Hadoop dependency updates at all? I understand that > will > > > also > > > > > break user applications, but it might be the driving feature for > > Hadoop > > > > > 3.0? > > > > > > > > > > On Fri, Sep 19, 2014 at 5:13 PM, Sangjin Lee > > wrote: > > > > > > > > > > > I would also agree on upgrading guava. Yes I am aware of the > > > potential > > > > > > impact on customers who might rely on hadoop bringing in guava > 11. > > > > > However, > > > > > > IMHO the balance tipped over to the other side a while ago; i.e. > I > > > > think > > > > > > there are far more people using guava 16 in their code and > > scrambling > > > > to > > > > > > make things work than the other way around. > > > > > > > > > > > > On Thu, Sep 18, 2014 at 2:40 PM, Steve Loughran < > > > > ste...@hortonworks.com> > > > > > > wrote: > > > > > > > > > > > > > I know we've been ignoring the Guava version problem, but > > > > HADOOP-10868 > > > > > > > added a transitive dependency on Guava 16 by way of Curator > 2.6. > > > > > > > > > > > > > > Maven currently forces the build to use Guava 11.0.2, but this > is > > > > > hiding > > > > > > at > > > > > > > compile timeall code paths from curator which may use classes & > > > > methods > > > > > > > that aren't there. > > > > > > > > > > > > > > I need curator for my own work (2.4.1 & Guava 14.0 was what I'd > > > been > > > > > > > using), so don't think we can go back. > > > > > > > > > > > > > > HADOOP-11102 covers the problem -but doesn't propose a specific > > > > > solution. > > > > > > > But to me the one that seems most likely to work is: update > Guava > > > > > > > > > > > > > > -steve > > > > > > > > > > > > > > -- > > > > > > > CONFIDENTIALITY NOTICE > > > > > > > NOTICE: This message is intended for the use of the individual > or > > > > > entity > > > > > > to > > > > > > > which it is addressed and may contain information that is > > > > confidential, > > > > > > > privileged and exempt from disclosure under applicable law. If > > the > > > > > reader > > > > > > > of this message is not the intended recipient, you are hereby > > > > notified > > > > > > that > > > > > > > any printing, copying, dissemination, distribution, disclosure > or > > > > > > > forwarding of this communication is strictly prohibited. If you > > > have > > > > > > > received this communication in error, please contact the sender > > > > > > immediately > > > > > > > and delete
Re: Time to address the Guava version problem
At the same time, not being able to use Curator will require a lot of extra code, a lot of which we probably already have from the ZKRMStateStore, but it's not available to use in hadoop-auth. We'd need to create our own ZK libraries that Hadoop components can use, but (a) that's going to take a while, and (b) it seems silly to reinvent the wheel when Curator already does all this. I agree that upgrading Guava will be a compatibility problem though... On Tue, Sep 23, 2014 at 9:30 AM, Sandy Ryza wrote: > If we've broken compatibility in branch-2, that's a bug that we need to > fix. HADOOP-10868 has not yet made it into a release; I don't see it as a > justification for solidifying the breakage. > > -1 to upgrading Guava in branch-2. > > On Tue, Sep 23, 2014 at 3:06 AM, Steve Loughran > wrote: > > > +1 to upgrading guava. Irrespective of downstream apps, the hadoop source > > tree is now internally inconsistent > > > > On 22 September 2014 17:56, Sangjin Lee wrote: > > > > > I agree that a more robust solution is to have better classloading > > > isolation. > > > > > > Still, IMHO guava (and possibly protobuf as well) sticks out like a > sore > > > thumb. There are just too many issues in trying to support both guava > 11 > > > and guava 16. Independent of what we may do with the classloading > > > isolation, we should still consider upgrading guava. > > > > > > My 2 cents. > > > > > > On Sun, Sep 21, 2014 at 3:11 PM, Karthik Kambatla > > > wrote: > > > > > > > Upgrading Guava version is tricky. While it helps in many cases, it > can > > > > break existing applications/deployments. I understand we do not have > a > > > > policy for updating dependencies, but still we should be careful with > > > > Guava. > > > > > > > > I would be more inclined towards a more permanent solution to this > > > problem > > > > - how about prioritizing classpath isolation so applications aren't > > > > affected by Hadoop dependency updates at all? I understand that will > > also > > > > break user applications, but it might be the driving feature for > Hadoop > > > > 3.0? > > > > > > > > On Fri, Sep 19, 2014 at 5:13 PM, Sangjin Lee > wrote: > > > > > > > > > I would also agree on upgrading guava. Yes I am aware of the > > potential > > > > > impact on customers who might rely on hadoop bringing in guava 11. > > > > However, > > > > > IMHO the balance tipped over to the other side a while ago; i.e. I > > > think > > > > > there are far more people using guava 16 in their code and > scrambling > > > to > > > > > make things work than the other way around. > > > > > > > > > > On Thu, Sep 18, 2014 at 2:40 PM, Steve Loughran < > > > ste...@hortonworks.com> > > > > > wrote: > > > > > > > > > > > I know we've been ignoring the Guava version problem, but > > > HADOOP-10868 > > > > > > added a transitive dependency on Guava 16 by way of Curator 2.6. > > > > > > > > > > > > Maven currently forces the build to use Guava 11.0.2, but this is > > > > hiding > > > > > at > > > > > > compile timeall code paths from curator which may use classes & > > > methods > > > > > > that aren't there. > > > > > > > > > > > > I need curator for my own work (2.4.1 & Guava 14.0 was what I'd > > been > > > > > > using), so don't think we can go back. > > > > > > > > > > > > HADOOP-11102 covers the problem -but doesn't propose a specific > > > > solution. > > > > > > But to me the one that seems most likely to work is: update Guava > > > > > > > > > > > > -steve > > > > > > > > > > > > -- > > > > > > CONFIDENTIALITY NOTICE > > > > > > NOTICE: This message is intended for the use of the individual or > > > > entity > > > > > to > > > > > > which it is addressed and may contain information that is > > > confidential, > > > > > > privileged and exempt from disclosure under applicable law. If > the > > > > reader > > > > > > of this message is not the intended recipient, you are hereby > > > notified > > > > > that > > > > > > any printing, copying, dissemination, distribution, disclosure or > > > > > > forwarding of this communication is strictly prohibited. If you > > have > > > > > > received this communication in error, please contact the sender > > > > > immediately > > > > > > and delete it from your system. Thank You. > > > > > > > > > > > > > > > > > > > > > > -- > > CONFIDENTIALITY NOTICE > > NOTICE: This message is intended for the use of the individual or entity > to > > which it is addressed and may contain information that is confidential, > > privileged and exempt from disclosure under applicable law. If the reader > > of this message is not the intended recipient, you are hereby notified > that > > any printing, copying, dissemination, distribution, disclosure or > > forwarding of this communication is strictly prohibited. If you have > > received this communication in error, please contact the sender > immediately > > and delete it from your system. Thank You. > > >
Re: Time to address the Guava version problem
If we've broken compatibility in branch-2, that's a bug that we need to fix. HADOOP-10868 has not yet made it into a release; I don't see it as a justification for solidifying the breakage. -1 to upgrading Guava in branch-2. On Tue, Sep 23, 2014 at 3:06 AM, Steve Loughran wrote: > +1 to upgrading guava. Irrespective of downstream apps, the hadoop source > tree is now internally inconsistent > > On 22 September 2014 17:56, Sangjin Lee wrote: > > > I agree that a more robust solution is to have better classloading > > isolation. > > > > Still, IMHO guava (and possibly protobuf as well) sticks out like a sore > > thumb. There are just too many issues in trying to support both guava 11 > > and guava 16. Independent of what we may do with the classloading > > isolation, we should still consider upgrading guava. > > > > My 2 cents. > > > > On Sun, Sep 21, 2014 at 3:11 PM, Karthik Kambatla > > wrote: > > > > > Upgrading Guava version is tricky. While it helps in many cases, it can > > > break existing applications/deployments. I understand we do not have a > > > policy for updating dependencies, but still we should be careful with > > > Guava. > > > > > > I would be more inclined towards a more permanent solution to this > > problem > > > - how about prioritizing classpath isolation so applications aren't > > > affected by Hadoop dependency updates at all? I understand that will > also > > > break user applications, but it might be the driving feature for Hadoop > > > 3.0? > > > > > > On Fri, Sep 19, 2014 at 5:13 PM, Sangjin Lee wrote: > > > > > > > I would also agree on upgrading guava. Yes I am aware of the > potential > > > > impact on customers who might rely on hadoop bringing in guava 11. > > > However, > > > > IMHO the balance tipped over to the other side a while ago; i.e. I > > think > > > > there are far more people using guava 16 in their code and scrambling > > to > > > > make things work than the other way around. > > > > > > > > On Thu, Sep 18, 2014 at 2:40 PM, Steve Loughran < > > ste...@hortonworks.com> > > > > wrote: > > > > > > > > > I know we've been ignoring the Guava version problem, but > > HADOOP-10868 > > > > > added a transitive dependency on Guava 16 by way of Curator 2.6. > > > > > > > > > > Maven currently forces the build to use Guava 11.0.2, but this is > > > hiding > > > > at > > > > > compile timeall code paths from curator which may use classes & > > methods > > > > > that aren't there. > > > > > > > > > > I need curator for my own work (2.4.1 & Guava 14.0 was what I'd > been > > > > > using), so don't think we can go back. > > > > > > > > > > HADOOP-11102 covers the problem -but doesn't propose a specific > > > solution. > > > > > But to me the one that seems most likely to work is: update Guava > > > > > > > > > > -steve > > > > > > > > > > -- > > > > > CONFIDENTIALITY NOTICE > > > > > NOTICE: This message is intended for the use of the individual or > > > entity > > > > to > > > > > which it is addressed and may contain information that is > > confidential, > > > > > privileged and exempt from disclosure under applicable law. If the > > > reader > > > > > of this message is not the intended recipient, you are hereby > > notified > > > > that > > > > > any printing, copying, dissemination, distribution, disclosure or > > > > > forwarding of this communication is strictly prohibited. If you > have > > > > > received this communication in error, please contact the sender > > > > immediately > > > > > and delete it from your system. Thank You. > > > > > > > > > > > > > > > > -- > CONFIDENTIALITY NOTICE > NOTICE: This message is intended for the use of the individual or entity to > which it is addressed and may contain information that is confidential, > privileged and exempt from disclosure under applicable law. If the reader > of this message is not the intended recipient, you are hereby notified that > any printing, copying, dissemination, distribution, disclosure or > forwarding of this communication is strictly prohibited. If you have > received this communication in error, please contact the sender immediately > and delete it from your system. Thank You. >
Re: Time to address the Guava version problem
+1 to upgrading guava. Irrespective of downstream apps, the hadoop source tree is now internally inconsistent On 22 September 2014 17:56, Sangjin Lee wrote: > I agree that a more robust solution is to have better classloading > isolation. > > Still, IMHO guava (and possibly protobuf as well) sticks out like a sore > thumb. There are just too many issues in trying to support both guava 11 > and guava 16. Independent of what we may do with the classloading > isolation, we should still consider upgrading guava. > > My 2 cents. > > On Sun, Sep 21, 2014 at 3:11 PM, Karthik Kambatla > wrote: > > > Upgrading Guava version is tricky. While it helps in many cases, it can > > break existing applications/deployments. I understand we do not have a > > policy for updating dependencies, but still we should be careful with > > Guava. > > > > I would be more inclined towards a more permanent solution to this > problem > > - how about prioritizing classpath isolation so applications aren't > > affected by Hadoop dependency updates at all? I understand that will also > > break user applications, but it might be the driving feature for Hadoop > > 3.0? > > > > On Fri, Sep 19, 2014 at 5:13 PM, Sangjin Lee wrote: > > > > > I would also agree on upgrading guava. Yes I am aware of the potential > > > impact on customers who might rely on hadoop bringing in guava 11. > > However, > > > IMHO the balance tipped over to the other side a while ago; i.e. I > think > > > there are far more people using guava 16 in their code and scrambling > to > > > make things work than the other way around. > > > > > > On Thu, Sep 18, 2014 at 2:40 PM, Steve Loughran < > ste...@hortonworks.com> > > > wrote: > > > > > > > I know we've been ignoring the Guava version problem, but > HADOOP-10868 > > > > added a transitive dependency on Guava 16 by way of Curator 2.6. > > > > > > > > Maven currently forces the build to use Guava 11.0.2, but this is > > hiding > > > at > > > > compile timeall code paths from curator which may use classes & > methods > > > > that aren't there. > > > > > > > > I need curator for my own work (2.4.1 & Guava 14.0 was what I'd been > > > > using), so don't think we can go back. > > > > > > > > HADOOP-11102 covers the problem -but doesn't propose a specific > > solution. > > > > But to me the one that seems most likely to work is: update Guava > > > > > > > > -steve > > > > > > > > -- > > > > CONFIDENTIALITY NOTICE > > > > NOTICE: This message is intended for the use of the individual or > > entity > > > to > > > > which it is addressed and may contain information that is > confidential, > > > > privileged and exempt from disclosure under applicable law. If the > > reader > > > > of this message is not the intended recipient, you are hereby > notified > > > that > > > > any printing, copying, dissemination, distribution, disclosure or > > > > forwarding of this communication is strictly prohibited. If you have > > > > received this communication in error, please contact the sender > > > immediately > > > > and delete it from your system. Thank You. > > > > > > > > > > -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: Time to address the Guava version problem
I agree that a more robust solution is to have better classloading isolation. Still, IMHO guava (and possibly protobuf as well) sticks out like a sore thumb. There are just too many issues in trying to support both guava 11 and guava 16. Independent of what we may do with the classloading isolation, we should still consider upgrading guava. My 2 cents. On Sun, Sep 21, 2014 at 3:11 PM, Karthik Kambatla wrote: > Upgrading Guava version is tricky. While it helps in many cases, it can > break existing applications/deployments. I understand we do not have a > policy for updating dependencies, but still we should be careful with > Guava. > > I would be more inclined towards a more permanent solution to this problem > - how about prioritizing classpath isolation so applications aren't > affected by Hadoop dependency updates at all? I understand that will also > break user applications, but it might be the driving feature for Hadoop > 3.0? > > On Fri, Sep 19, 2014 at 5:13 PM, Sangjin Lee wrote: > > > I would also agree on upgrading guava. Yes I am aware of the potential > > impact on customers who might rely on hadoop bringing in guava 11. > However, > > IMHO the balance tipped over to the other side a while ago; i.e. I think > > there are far more people using guava 16 in their code and scrambling to > > make things work than the other way around. > > > > On Thu, Sep 18, 2014 at 2:40 PM, Steve Loughran > > wrote: > > > > > I know we've been ignoring the Guava version problem, but HADOOP-10868 > > > added a transitive dependency on Guava 16 by way of Curator 2.6. > > > > > > Maven currently forces the build to use Guava 11.0.2, but this is > hiding > > at > > > compile timeall code paths from curator which may use classes & methods > > > that aren't there. > > > > > > I need curator for my own work (2.4.1 & Guava 14.0 was what I'd been > > > using), so don't think we can go back. > > > > > > HADOOP-11102 covers the problem -but doesn't propose a specific > solution. > > > But to me the one that seems most likely to work is: update Guava > > > > > > -steve > > > > > > -- > > > CONFIDENTIALITY NOTICE > > > NOTICE: This message is intended for the use of the individual or > entity > > to > > > which it is addressed and may contain information that is confidential, > > > privileged and exempt from disclosure under applicable law. If the > reader > > > of this message is not the intended recipient, you are hereby notified > > that > > > any printing, copying, dissemination, distribution, disclosure or > > > forwarding of this communication is strictly prohibited. If you have > > > received this communication in error, please contact the sender > > immediately > > > and delete it from your system. Thank You. > > > > > >
Re: Time to address the Guava version problem
On 21 September 2014 23:11, Karthik Kambatla wrote: > Upgrading Guava version is tricky. While it helps in many cases, it can > break existing applications/deployments. I understand we do not have a > policy for updating dependencies, but still we should be careful with > Guava. > I agree, but the classpath is currently in an inconsistent state: it incudes Guava 11 and a library built against Guava 16. > > I would be more inclined towards a more permanent solution to this problem > - how about prioritizing classpath isolation so applications aren't > affected by Hadoop dependency updates at all? I understand that will also > break user applications, but it might be the driving feature for Hadoop > 3.0? > I think this would be good; if you look at where we're going with YARN-deployed apps, there is a trend towards pushing up all the JARs, using the distributed cache to reduce the cost of that upload. All you should really need at the far end are the .xml files. Except this has caused a problem with branch-2 to surface, the native libs aren't binary-signature-compatible with 2.5 or earlier JARs -look at HADOOP-11064 This something to be fixed —but it highlights the problem where even the native .lib, .so. .dll files are implicitly part of the in-hadoop compatibility layer. so: the "upload all JARs" strategy has weaknesses too; some OSGi solution could address that, though we need time playing with that before being able to claim it solves all problems ... I worry that it may help address JAR dependencies at the price of performance. Returning to Guava, which can't be put off as we are already in a mess > > On Fri, Sep 19, 2014 at 5:13 PM, Sangjin Lee wrote: > > > I would also agree on upgrading guava. Yes I am aware of the potential > > impact on customers who might rely on hadoop bringing in guava 11. > However, > > IMHO the balance tipped over to the other side a while ago; i.e. I think > > there are far more people using guava 16 in their code and scrambling to > > make things work than the other way around. > > > I concur ... to many things you build downstream need a 15+ guava. Which includes Hadoop now ... we just haven't fully admitted it yet. Steve > > On Thu, Sep 18, 2014 at 2:40 PM, Steve Loughran > > wrote: > > > > > I know we've been ignoring the Guava version problem, but HADOOP-10868 > > > added a transitive dependency on Guava 16 by way of Curator 2.6. > > > > > > Maven currently forces the build to use Guava 11.0.2, but this is > hiding > > at > > > compile timeall code paths from curator which may use classes & methods > > > that aren't there. > > > > > > I need curator for my own work (2.4.1 & Guava 14.0 was what I'd been > > > using), so don't think we can go back. > > > > > > HADOOP-11102 covers the problem -but doesn't propose a specific > solution. > > > But to me the one that seems most likely to work is: update Guava > > > > > > -steve > > > > > > -- > > > CONFIDENTIALITY NOTICE > > > NOTICE: This message is intended for the use of the individual or > entity > > to > > > which it is addressed and may contain information that is confidential, > > > privileged and exempt from disclosure under applicable law. If the > reader > > > of this message is not the intended recipient, you are hereby notified > > that > > > any printing, copying, dissemination, distribution, disclosure or > > > forwarding of this communication is strictly prohibited. If you have > > > received this communication in error, please contact the sender > > immediately > > > and delete it from your system. Thank You. > > > > > > -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: Time to address the Guava version problem
Upgrading Guava version is tricky. While it helps in many cases, it can break existing applications/deployments. I understand we do not have a policy for updating dependencies, but still we should be careful with Guava. I would be more inclined towards a more permanent solution to this problem - how about prioritizing classpath isolation so applications aren't affected by Hadoop dependency updates at all? I understand that will also break user applications, but it might be the driving feature for Hadoop 3.0? On Fri, Sep 19, 2014 at 5:13 PM, Sangjin Lee wrote: > I would also agree on upgrading guava. Yes I am aware of the potential > impact on customers who might rely on hadoop bringing in guava 11. However, > IMHO the balance tipped over to the other side a while ago; i.e. I think > there are far more people using guava 16 in their code and scrambling to > make things work than the other way around. > > On Thu, Sep 18, 2014 at 2:40 PM, Steve Loughran > wrote: > > > I know we've been ignoring the Guava version problem, but HADOOP-10868 > > added a transitive dependency on Guava 16 by way of Curator 2.6. > > > > Maven currently forces the build to use Guava 11.0.2, but this is hiding > at > > compile timeall code paths from curator which may use classes & methods > > that aren't there. > > > > I need curator for my own work (2.4.1 & Guava 14.0 was what I'd been > > using), so don't think we can go back. > > > > HADOOP-11102 covers the problem -but doesn't propose a specific solution. > > But to me the one that seems most likely to work is: update Guava > > > > -steve > > > > -- > > CONFIDENTIALITY NOTICE > > NOTICE: This message is intended for the use of the individual or entity > to > > which it is addressed and may contain information that is confidential, > > privileged and exempt from disclosure under applicable law. If the reader > > of this message is not the intended recipient, you are hereby notified > that > > any printing, copying, dissemination, distribution, disclosure or > > forwarding of this communication is strictly prohibited. If you have > > received this communication in error, please contact the sender > immediately > > and delete it from your system. Thank You. > > >
Re: Time to address the Guava version problem
I would also agree on upgrading guava. Yes I am aware of the potential impact on customers who might rely on hadoop bringing in guava 11. However, IMHO the balance tipped over to the other side a while ago; i.e. I think there are far more people using guava 16 in their code and scrambling to make things work than the other way around. On Thu, Sep 18, 2014 at 2:40 PM, Steve Loughran wrote: > I know we've been ignoring the Guava version problem, but HADOOP-10868 > added a transitive dependency on Guava 16 by way of Curator 2.6. > > Maven currently forces the build to use Guava 11.0.2, but this is hiding at > compile timeall code paths from curator which may use classes & methods > that aren't there. > > I need curator for my own work (2.4.1 & Guava 14.0 was what I'd been > using), so don't think we can go back. > > HADOOP-11102 covers the problem -but doesn't propose a specific solution. > But to me the one that seems most likely to work is: update Guava > > -steve > > -- > CONFIDENTIALITY NOTICE > NOTICE: This message is intended for the use of the individual or entity to > which it is addressed and may contain information that is confidential, > privileged and exempt from disclosure under applicable law. If the reader > of this message is not the intended recipient, you are hereby notified that > any printing, copying, dissemination, distribution, disclosure or > forwarding of this communication is strictly prohibited. If you have > received this communication in error, please contact the sender immediately > and delete it from your system. Thank You. >
Time to address the Guava version problem
I know we've been ignoring the Guava version problem, but HADOOP-10868 added a transitive dependency on Guava 16 by way of Curator 2.6. Maven currently forces the build to use Guava 11.0.2, but this is hiding at compile timeall code paths from curator which may use classes & methods that aren't there. I need curator for my own work (2.4.1 & Guava 14.0 was what I'd been using), so don't think we can go back. HADOOP-11102 covers the problem -but doesn't propose a specific solution. But to me the one that seems most likely to work is: update Guava -steve -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
