Re: Difference between netty and netty-all
Nicholas, FYI, there's some patch for Hadoop 2.8? 2.9? to move up to Netty https://issues.apache.org/jira/browse/HADOOP-13866 https://issues.apache.org/jira/browse/HADOOP-12854 On 5 Dec 2016, at 19:46, Nicholas Chammas <nicholas.cham...@gmail.com<mailto:nicholas.cham...@gmail.com>> wrote: So if I'm running Spark 2.0.2 built against Hadoop 2.6, I should be running [Netty 4.0.29.Final](https://github.com/apache/spark/blob/v2.0.2/dev/deps/spark-deps-hadoop-2.6#L141<https://github.com/apache/spark/blob/553aac56bd5284e84391c05e2ef54d8bd7ad3a12/dev/deps/spark-deps-hadoop-2.6#L141>), right? And since [the Netty PR I'm interested in](https://github.com/netty/netty/pull/5345) is tagged 4.0.37.Final, then I guess Spark 2.0.2 isn't using a version of Netty that includes that PR. This correlates with what I'm seeing in my environment (warnings related to low entropy followed by executor failures). OK cool! Thanks for the pointers. Nick On Mon, Dec 5, 2016 at 2:18 PM Sean Owen <so...@cloudera.com<mailto:so...@cloudera.com>> wrote: netty should be Netty 3.x. It is all but unused but I couldn't manage to get rid of it: https://issues.apache.org/jira/browse/SPARK-17875 I don't know why there's also a dependency on netty 3.x there, except to note that HADOOP-12928 covers keeping it in sync with ZK. Cutting it entirely would achieve that. Patches there welcome; anything that cuts a dependency is less traumatic than those which increment them, which is why I'm happy Hadoop had just got rid of Jackson 1.9.x entirely. netty-all should be 4.x, actually used. On Tue, Dec 6, 2016 at 12:54 AM Nicholas Chammas <nicholas.cham...@gmail.com<mailto:nicholas.cham...@gmail.com>> wrote: I’m looking at the list of dependencies here: https://github.com/apache/spark/search?l=Groff=netty=Code=%E2%9C%93 What’s the difference between netty and netty-all? The reason I ask is because I’m looking at a Netty PR<https://github.com/netty/netty/pull/5345> and trying to figure out if Spark 2.0.2 is using a version of Netty that includes that PR or not. Nick
Re: Difference between netty and netty-all
No. I meant only updating master. It's not worth to update a maintenance branch unless there are critical issues. On Mon, Dec 5, 2016 at 5:39 PM, Nicholas Chammas <nicholas.cham...@gmail.com > wrote: > You mean just for branch-2.0, right? > > > On Mon, Dec 5, 2016 at 8:35 PM Shixiong(Ryan) Zhu <shixi...@databricks.com> > wrote: > >> Hey Nick, >> >> It should be safe to upgrade Netty to the latest 4.0.x version. Could you >> submit a PR, please? >> >> On Mon, Dec 5, 2016 at 11:47 AM, Nicholas Chammas < >> nicholas.cham...@gmail.com> wrote: >> >> That file is in Netty 4.0.29, but I believe the PR I referenced is not. >> It's only in Netty 4.0.37 and up. >> >> >> On Mon, Dec 5, 2016 at 1:57 PM Ted Yu <yuzhih...@gmail.com> wrote: >> >> This should be in netty-all : >> >> $ jar tvf >> /home/x/.m2/repository/io/netty/netty-all/4.0.29.Final/netty-all-4.0.29.Final.jar >> | grep ThreadLocalRandom >>967 Tue Jun 23 11:10:30 UTC 2015 io/netty/util/internal/ >> ThreadLocalRandom$1.class >> 1079 Tue Jun 23 11:10:30 UTC 2015 io/netty/util/internal/ >> ThreadLocalRandom$2.class >> 5973 Tue Jun 23 11:10:30 UTC 2015 io/netty/util/internal/ >> ThreadLocalRandom.class >> >> On Mon, Dec 5, 2016 at 8:53 AM, Nicholas Chammas < >> nicholas.cham...@gmail.com> wrote: >> >> I’m looking at the list of dependencies here: >> >> https://github.com/apache/spark/search?l=Groff=netty; >> type=Code=%E2%9C%93 >> >> What’s the difference between netty and netty-all? >> >> The reason I ask is because I’m looking at a Netty PR >> <https://github.com/netty/netty/pull/5345> and trying to figure out if >> Spark 2.0.2 is using a version of Netty that includes that PR or not. >> >> Nick >> >> >> >> >>
Re: Difference between netty and netty-all
You mean just for branch-2.0, right? On Mon, Dec 5, 2016 at 8:35 PM Shixiong(Ryan) Zhu <shixi...@databricks.com> wrote: > Hey Nick, > > It should be safe to upgrade Netty to the latest 4.0.x version. Could you > submit a PR, please? > > On Mon, Dec 5, 2016 at 11:47 AM, Nicholas Chammas < > nicholas.cham...@gmail.com> wrote: > > That file is in Netty 4.0.29, but I believe the PR I referenced is not. > It's only in Netty 4.0.37 and up. > > > On Mon, Dec 5, 2016 at 1:57 PM Ted Yu <yuzhih...@gmail.com> wrote: > > This should be in netty-all : > > $ jar tvf > /home/x/.m2/repository/io/netty/netty-all/4.0.29.Final/netty-all-4.0.29.Final.jar > | grep ThreadLocalRandom >967 Tue Jun 23 11:10:30 UTC 2015 > io/netty/util/internal/ThreadLocalRandom$1.class > 1079 Tue Jun 23 11:10:30 UTC 2015 > io/netty/util/internal/ThreadLocalRandom$2.class > 5973 Tue Jun 23 11:10:30 UTC 2015 > io/netty/util/internal/ThreadLocalRandom.class > > On Mon, Dec 5, 2016 at 8:53 AM, Nicholas Chammas < > nicholas.cham...@gmail.com> wrote: > > I’m looking at the list of dependencies here: > > > https://github.com/apache/spark/search?l=Groff=netty=Code=%E2%9C%93 > > What’s the difference between netty and netty-all? > > The reason I ask is because I’m looking at a Netty PR > <https://github.com/netty/netty/pull/5345> and trying to figure out if > Spark 2.0.2 is using a version of Netty that includes that PR or not. > > Nick > > > > >
Re: Difference between netty and netty-all
Hey Nick, It should be safe to upgrade Netty to the latest 4.0.x version. Could you submit a PR, please? On Mon, Dec 5, 2016 at 11:47 AM, Nicholas Chammas < nicholas.cham...@gmail.com> wrote: > That file is in Netty 4.0.29, but I believe the PR I referenced is not. > It's only in Netty 4.0.37 and up. > > > On Mon, Dec 5, 2016 at 1:57 PM Ted Yu <yuzhih...@gmail.com> wrote: > >> This should be in netty-all : >> >> $ jar tvf >> /home/x/.m2/repository/io/netty/netty-all/4.0.29.Final/netty-all-4.0.29.Final.jar >> | grep ThreadLocalRandom >>967 Tue Jun 23 11:10:30 UTC 2015 io/netty/util/internal/ >> ThreadLocalRandom$1.class >> 1079 Tue Jun 23 11:10:30 UTC 2015 io/netty/util/internal/ >> ThreadLocalRandom$2.class >> 5973 Tue Jun 23 11:10:30 UTC 2015 io/netty/util/internal/ >> ThreadLocalRandom.class >> >> On Mon, Dec 5, 2016 at 8:53 AM, Nicholas Chammas < >> nicholas.cham...@gmail.com> wrote: >> >> I’m looking at the list of dependencies here: >> >> https://github.com/apache/spark/search?l=Groff=netty; >> type=Code=%E2%9C%93 >> >> What’s the difference between netty and netty-all? >> >> The reason I ask is because I’m looking at a Netty PR >> <https://github.com/netty/netty/pull/5345> and trying to figure out if >> Spark 2.0.2 is using a version of Netty that includes that PR or not. >> >> Nick >> >> >> >>
Re: Difference between netty and netty-all
That file is in Netty 4.0.29, but I believe the PR I referenced is not. It's only in Netty 4.0.37 and up. On Mon, Dec 5, 2016 at 1:57 PM Ted Yu <yuzhih...@gmail.com> wrote: > This should be in netty-all : > > $ jar tvf > /home/x/.m2/repository/io/netty/netty-all/4.0.29.Final/netty-all-4.0.29.Final.jar > | grep ThreadLocalRandom >967 Tue Jun 23 11:10:30 UTC 2015 > io/netty/util/internal/ThreadLocalRandom$1.class > 1079 Tue Jun 23 11:10:30 UTC 2015 > io/netty/util/internal/ThreadLocalRandom$2.class > 5973 Tue Jun 23 11:10:30 UTC 2015 > io/netty/util/internal/ThreadLocalRandom.class > > On Mon, Dec 5, 2016 at 8:53 AM, Nicholas Chammas < > nicholas.cham...@gmail.com> wrote: > > I’m looking at the list of dependencies here: > > > https://github.com/apache/spark/search?l=Groff=netty=Code=%E2%9C%93 > > What’s the difference between netty and netty-all? > > The reason I ask is because I’m looking at a Netty PR > <https://github.com/netty/netty/pull/5345> and trying to figure out if > Spark 2.0.2 is using a version of Netty that includes that PR or not. > > Nick > > > >
Re: Difference between netty and netty-all
netty should be Netty 3.x. It is all but unused but I couldn't manage to get rid of it: https://issues.apache.org/jira/browse/SPARK-17875 netty-all should be 4.x, actually used. On Tue, Dec 6, 2016 at 12:54 AM Nicholas Chammas <nicholas.cham...@gmail.com> wrote: > I’m looking at the list of dependencies here: > > > https://github.com/apache/spark/search?l=Groff=netty=Code=%E2%9C%93 > > What’s the difference between netty and netty-all? > > The reason I ask is because I’m looking at a Netty PR > <https://github.com/netty/netty/pull/5345> and trying to figure out if > Spark 2.0.2 is using a version of Netty that includes that PR or not. > > Nick > >
Re: Difference between netty and netty-all
This should be in netty-all : $ jar tvf /home/x/.m2/repository/io/netty/netty-all/4.0.29.Final/netty-all-4.0.29.Final.jar | grep ThreadLocalRandom 967 Tue Jun 23 11:10:30 UTC 2015 io/netty/util/internal/ThreadLocalRandom$1.class 1079 Tue Jun 23 11:10:30 UTC 2015 io/netty/util/internal/ThreadLocalRandom$2.class 5973 Tue Jun 23 11:10:30 UTC 2015 io/netty/util/internal/ThreadLocalRandom.class On Mon, Dec 5, 2016 at 8:53 AM, Nicholas Chammas <nicholas.cham...@gmail.com > wrote: > I’m looking at the list of dependencies here: > > https://github.com/apache/spark/search?l=Groff=netty; > type=Code=%E2%9C%93 > > What’s the difference between netty and netty-all? > > The reason I ask is because I’m looking at a Netty PR > <https://github.com/netty/netty/pull/5345> and trying to figure out if > Spark 2.0.2 is using a version of Netty that includes that PR or not. > > Nick > >
Difference between netty and netty-all
I’m looking at the list of dependencies here: https://github.com/apache/spark/search?l=Groff=netty=Code=%E2%9C%93 What’s the difference between netty and netty-all? The reason I ask is because I’m looking at a Netty PR <https://github.com/netty/netty/pull/5345> and trying to figure out if Spark 2.0.2 is using a version of Netty that includes that PR or not. Nick
