Re: Good way to test when topology in local cluster is "fully active"
This did work. Thanks! On Tue, Aug 5, 2014 at 2:23 PM, P. Taylor Goetz wrote: > My guess is that the slowdown you are seeing is a result of the new > version of ZooKeeper and how it handles IPv4/6. > > Try adding the following JVM parameter when running your tests: > > -Djava.net.preferIPv4Stack=true > > -Taylor > > On Aug 4, 2014, at 8:49 PM, Corey Nolet wrote: > > > I'm testing some sliding window algorithms with tuples emitted from a > mock spout based on a timer but the amount of time it takes the topology to > fully start up and activate seems to vary from computer to computer. > Specifically, I just updated from 0.8.2 to 0.9.2-incubating and all of my > tests are breaking because the time to activate the topology is taking > longer (because of Netty possibly?). I'd like to make my tests more > resilient to things like this. > > > > Is there something I can look at in LocalCluster where I could do > "while(!notActive) { Thread.sleep(50) }" ? > > > > This is what my test looks like currently: > > > > StormTopology topology = buildTopology(...); > > Config conf = new Config(); > > conf.setNumWorkers(1); > > > > LocalCluster cluster = new LocalCluster(); > > cluster.submitTopology(getTopologyName(), conf, topology); > > > > try { > > Thread.sleep(4000); > > } catch (InterruptedException e) { > > e.printStackTrace(); > > } > > > > cluster.shutdown(); > > > > assertEquals(4, MockSinkBolt.getEvents().size()); > > > > > > > > Thanks! > > > > > >
Re: Good way to test when topology in local cluster is "fully active"
Sorry- the ipv4 fix worked. On Tue, Aug 5, 2014 at 9:13 PM, Corey Nolet wrote: > This did work. Thanks! > > > On Tue, Aug 5, 2014 at 2:23 PM, P. Taylor Goetz wrote: > >> My guess is that the slowdown you are seeing is a result of the new >> version of ZooKeeper and how it handles IPv4/6. >> >> Try adding the following JVM parameter when running your tests: >> >> -Djava.net.preferIPv4Stack=true >> >> -Taylor >> >> On Aug 4, 2014, at 8:49 PM, Corey Nolet wrote: >> >> > I'm testing some sliding window algorithms with tuples emitted from a >> mock spout based on a timer but the amount of time it takes the topology to >> fully start up and activate seems to vary from computer to computer. >> Specifically, I just updated from 0.8.2 to 0.9.2-incubating and all of my >> tests are breaking because the time to activate the topology is taking >> longer (because of Netty possibly?). I'd like to make my tests more >> resilient to things like this. >> > >> > Is there something I can look at in LocalCluster where I could do >> "while(!notActive) { Thread.sleep(50) }" ? >> > >> > This is what my test looks like currently: >> > >> > StormTopology topology = buildTopology(...); >> > Config conf = new Config(); >> > conf.setNumWorkers(1); >> > >> > LocalCluster cluster = new LocalCluster(); >> > cluster.submitTopology(getTopologyName(), conf, topology); >> > >> > try { >> > Thread.sleep(4000); >> > } catch (InterruptedException e) { >> > e.printStackTrace(); >> > } >> > >> > cluster.shutdown(); >> > >> > assertEquals(4, MockSinkBolt.getEvents().size()); >> > >> > >> > >> > Thanks! >> > >> > >> >> >
Re: Good way to test when topology in local cluster is "fully active"
Vincent & P.Taylor, I played with the testing framework for a little bit last night and don't see any easy way to provide pauses in between the emissions of the mock tuples. For instance, my sliding window semantics are heavily orchestrated by time evictions and triggers which mean that I need to be able to time the tuples being fed into the tests (i.e. emit a tuple every 500ms and run the test for 25 secons). On Tue, Aug 5, 2014 at 2:23 PM, P. Taylor Goetz wrote: > My guess is that the slowdown you are seeing is a result of the new > version of ZooKeeper and how it handles IPv4/6. > > Try adding the following JVM parameter when running your tests: > > -Djava.net.preferIPv4Stack=true > > -Taylor > > On Aug 4, 2014, at 8:49 PM, Corey Nolet wrote: > > > I'm testing some sliding window algorithms with tuples emitted from a > mock spout based on a timer but the amount of time it takes the topology to > fully start up and activate seems to vary from computer to computer. > Specifically, I just updated from 0.8.2 to 0.9.2-incubating and all of my > tests are breaking because the time to activate the topology is taking > longer (because of Netty possibly?). I'd like to make my tests more > resilient to things like this. > > > > Is there something I can look at in LocalCluster where I could do > "while(!notActive) { Thread.sleep(50) }" ? > > > > This is what my test looks like currently: > > > > StormTopology topology = buildTopology(...); > > Config conf = new Config(); > > conf.setNumWorkers(1); > > > > LocalCluster cluster = new LocalCluster(); > > cluster.submitTopology(getTopologyName(), conf, topology); > > > > try { > > Thread.sleep(4000); > > } catch (InterruptedException e) { > > e.printStackTrace(); > > } > > > > cluster.shutdown(); > > > > assertEquals(4, MockSinkBolt.getEvents().size()); > > > > > > > > Thanks! > > > > > >
Re: Good way to test when topology in local cluster is "fully active"
My guess is that the slowdown you are seeing is a result of the new version of ZooKeeper and how it handles IPv4/6. Try adding the following JVM parameter when running your tests: -Djava.net.preferIPv4Stack=true -Taylor On Aug 4, 2014, at 8:49 PM, Corey Nolet wrote: > I'm testing some sliding window algorithms with tuples emitted from a mock > spout based on a timer but the amount of time it takes the topology to fully > start up and activate seems to vary from computer to computer. Specifically, > I just updated from 0.8.2 to 0.9.2-incubating and all of my tests are > breaking because the time to activate the topology is taking longer (because > of Netty possibly?). I'd like to make my tests more resilient to things like > this. > > Is there something I can look at in LocalCluster where I could do > "while(!notActive) { Thread.sleep(50) }" ? > > This is what my test looks like currently: > > StormTopology topology = buildTopology(...); > Config conf = new Config(); > conf.setNumWorkers(1); > > LocalCluster cluster = new LocalCluster(); > cluster.submitTopology(getTopologyName(), conf, topology); > > try { > Thread.sleep(4000); > } catch (InterruptedException e) { > e.printStackTrace(); > } > > cluster.shutdown(); > > assertEquals(4, MockSinkBolt.getEvents().size()); > > > > Thanks! > > signature.asc Description: Message signed with OpenPGP using GPGMail
Re: Good way to test when topology in local cluster is "fully active"
No worries. I haven't found the testing framework the easiest thing to use. If you see errors that say something like "cannot convert object to iterable" I use the code below to disable storm from being able to call System.exit() private static class ExitTrappedException extends SecurityException { } private static void forbidSystemExitCall() { final SecurityManager securityManager = new SecurityManager() { public void checkPermission(Permission permission) { if (permission.getName().startsWith("exitVM")) { throw new ExitTrappedException(); } } }; System.setSecurityManager(securityManager); } private static void enableSystemExitCall() { System.setSecurityManager(null); } On Mon, Aug 4, 2014 at 11:02 PM, Corey Nolet wrote: > Nevermind, I wrote that before looking. This has been around since 0.8.1. > Thanks again Vincent! > > > On Mon, Aug 4, 2014 at 11:01 PM, Corey Nolet wrote: > >> Oh Nice. Is this new in 0.9.*? I just updated so I haven't looked much >> into what's changed yet, other than Netty. >> >> >> On Mon, Aug 4, 2014 at 10:40 PM, Vincent Russell < >> vincent.russ...@gmail.com> wrote: >> >>> Corey, >>> >>> Have you tried using the integration testing framework that comes with >>> storm? >>> >>> >>> Testing.withSimulatedTimeLocalCluster(mkClusterParam, >>> new TestJob() { >>> @Override >>> public void run(ILocalCluster cluster) throws Exception { >>> >>> CompleteTopologyParam completeTopologyParam = new >>> CompleteTopologyParam(); >>> completeTopologyParam >>> .setMockedSources(mockedSources); >>> completeTopologyParam.setStormConf(daemonConf); >>> >>> completeTopologyParam.setTopologyName(getTopologyName()); >>> Map result = Testing.completeTopology(cluster, >>> topology, completeTopologyParam); >>> >>> }); >>> >>> -Vincent >>> >>> On Mon, Aug 4, 2014 at 8:49 PM, Corey Nolet wrote: >>> I'm testing some sliding window algorithms with tuples emitted from a mock spout based on a timer but the amount of time it takes the topology to fully start up and activate seems to vary from computer to computer. Specifically, I just updated from 0.8.2 to 0.9.2-incubating and all of my tests are breaking because the time to activate the topology is taking longer (because of Netty possibly?). I'd like to make my tests more resilient to things like this. Is there something I can look at in LocalCluster where I could do "while(!notActive) { Thread.sleep(50) }" ? This is what my test looks like currently: StormTopology topology = buildTopology(...); Config conf = new Config(); conf.setNumWorkers(1); LocalCluster cluster = new LocalCluster(); cluster.submitTopology(getTopologyName(), conf, topology); try { Thread.sleep(4000); } catch (InterruptedException e) { e.printStackTrace(); } cluster.shutdown(); assertEquals(4, MockSinkBolt.getEvents().size()); Thanks! >>> >> >
Re: Good way to test when topology in local cluster is "fully active"
Nevermind, I wrote that before looking. This has been around since 0.8.1. Thanks again Vincent! On Mon, Aug 4, 2014 at 11:01 PM, Corey Nolet wrote: > Oh Nice. Is this new in 0.9.*? I just updated so I haven't looked much > into what's changed yet, other than Netty. > > > On Mon, Aug 4, 2014 at 10:40 PM, Vincent Russell < > vincent.russ...@gmail.com> wrote: > >> Corey, >> >> Have you tried using the integration testing framework that comes with >> storm? >> >> >> Testing.withSimulatedTimeLocalCluster(mkClusterParam, >> new TestJob() { >> @Override >> public void run(ILocalCluster cluster) throws Exception { >> >> CompleteTopologyParam completeTopologyParam = new CompleteTopologyParam(); >> completeTopologyParam >> .setMockedSources(mockedSources); >> completeTopologyParam.setStormConf(daemonConf); >> >> completeTopologyParam.setTopologyName(getTopologyName()); >> Map result = Testing.completeTopology(cluster, >> topology, completeTopologyParam); >> >> }); >> >> -Vincent >> >> On Mon, Aug 4, 2014 at 8:49 PM, Corey Nolet wrote: >> >>> I'm testing some sliding window algorithms with tuples emitted from a >>> mock spout based on a timer but the amount of time it takes the topology to >>> fully start up and activate seems to vary from computer to computer. >>> Specifically, I just updated from 0.8.2 to 0.9.2-incubating and all of my >>> tests are breaking because the time to activate the topology is taking >>> longer (because of Netty possibly?). I'd like to make my tests more >>> resilient to things like this. >>> >>> Is there something I can look at in LocalCluster where I could do >>> "while(!notActive) { Thread.sleep(50) }" ? >>> >>> This is what my test looks like currently: >>> >>> StormTopology topology = buildTopology(...); >>> Config conf = new Config(); >>> conf.setNumWorkers(1); >>> >>> LocalCluster cluster = new LocalCluster(); >>> cluster.submitTopology(getTopologyName(), conf, topology); >>> >>> try { >>> Thread.sleep(4000); >>> } catch (InterruptedException e) { >>> e.printStackTrace(); >>> } >>> >>> cluster.shutdown(); >>> >>> assertEquals(4, MockSinkBolt.getEvents().size()); >>> >>> >>> >>> Thanks! >>> >>> >>> >> >
Re: Good way to test when topology in local cluster is "fully active"
Oh Nice. Is this new in 0.9.*? I just updated so I haven't looked much into what's changed yet, other than Netty. On Mon, Aug 4, 2014 at 10:40 PM, Vincent Russell wrote: > Corey, > > Have you tried using the integration testing framework that comes with > storm? > > > Testing.withSimulatedTimeLocalCluster(mkClusterParam, > new TestJob() { > @Override > public void run(ILocalCluster cluster) throws Exception { > > CompleteTopologyParam completeTopologyParam = new CompleteTopologyParam(); > completeTopologyParam > .setMockedSources(mockedSources); > completeTopologyParam.setStormConf(daemonConf); > > completeTopologyParam.setTopologyName(getTopologyName()); > Map result = Testing.completeTopology(cluster, > topology, completeTopologyParam); > > }); > > -Vincent > > On Mon, Aug 4, 2014 at 8:49 PM, Corey Nolet wrote: > >> I'm testing some sliding window algorithms with tuples emitted from a >> mock spout based on a timer but the amount of time it takes the topology to >> fully start up and activate seems to vary from computer to computer. >> Specifically, I just updated from 0.8.2 to 0.9.2-incubating and all of my >> tests are breaking because the time to activate the topology is taking >> longer (because of Netty possibly?). I'd like to make my tests more >> resilient to things like this. >> >> Is there something I can look at in LocalCluster where I could do >> "while(!notActive) { Thread.sleep(50) }" ? >> >> This is what my test looks like currently: >> >> StormTopology topology = buildTopology(...); >> Config conf = new Config(); >> conf.setNumWorkers(1); >> >> LocalCluster cluster = new LocalCluster(); >> cluster.submitTopology(getTopologyName(), conf, topology); >> >> try { >> Thread.sleep(4000); >> } catch (InterruptedException e) { >> e.printStackTrace(); >> } >> >> cluster.shutdown(); >> >> assertEquals(4, MockSinkBolt.getEvents().size()); >> >> >> >> Thanks! >> >> >> >
Re: Good way to test when topology in local cluster is "fully active"
Corey, Have you tried using the integration testing framework that comes with storm? Testing.withSimulatedTimeLocalCluster(mkClusterParam, new TestJob() { @Override public void run(ILocalCluster cluster) throws Exception { CompleteTopologyParam completeTopologyParam = new CompleteTopologyParam(); completeTopologyParam .setMockedSources(mockedSources); completeTopologyParam.setStormConf(daemonConf); completeTopologyParam.setTopologyName(getTopologyName()); Map result = Testing.completeTopology(cluster, topology, completeTopologyParam); }); -Vincent On Mon, Aug 4, 2014 at 8:49 PM, Corey Nolet wrote: > I'm testing some sliding window algorithms with tuples emitted from a mock > spout based on a timer but the amount of time it takes the topology to > fully start up and activate seems to vary from computer to computer. > Specifically, I just updated from 0.8.2 to 0.9.2-incubating and all of my > tests are breaking because the time to activate the topology is taking > longer (because of Netty possibly?). I'd like to make my tests more > resilient to things like this. > > Is there something I can look at in LocalCluster where I could do > "while(!notActive) { Thread.sleep(50) }" ? > > This is what my test looks like currently: > > StormTopology topology = buildTopology(...); > Config conf = new Config(); > conf.setNumWorkers(1); > > LocalCluster cluster = new LocalCluster(); > cluster.submitTopology(getTopologyName(), conf, topology); > > try { > Thread.sleep(4000); > } catch (InterruptedException e) { > e.printStackTrace(); > } > > cluster.shutdown(); > > assertEquals(4, MockSinkBolt.getEvents().size()); > > > > Thanks! > > >
Good way to test when topology in local cluster is "fully active"
I'm testing some sliding window algorithms with tuples emitted from a mock spout based on a timer but the amount of time it takes the topology to fully start up and activate seems to vary from computer to computer. Specifically, I just updated from 0.8.2 to 0.9.2-incubating and all of my tests are breaking because the time to activate the topology is taking longer (because of Netty possibly?). I'd like to make my tests more resilient to things like this. Is there something I can look at in LocalCluster where I could do "while(!notActive) { Thread.sleep(50) }" ? This is what my test looks like currently: StormTopology topology = buildTopology(...); Config conf = new Config(); conf.setNumWorkers(1); LocalCluster cluster = new LocalCluster(); cluster.submitTopology(getTopologyName(), conf, topology); try { Thread.sleep(4000); } catch (InterruptedException e) { e.printStackTrace(); } cluster.shutdown(); assertEquals(4, MockSinkBolt.getEvents().size()); Thanks!