Re: Good way to test when topology in local cluster is fully active
My guess is that the slowdown you are seeing is a result of the new version of
ZooKeeper and how it handles IPv4/6.
Try adding the following JVM parameter when running your tests:
-Djava.net.preferIPv4Stack=true
-Taylor
On Aug 4, 2014, at 8:49 PM, Corey Nolet cjno...@gmail.com wrote:
I'm testing some sliding window algorithms with tuples emitted from a mock
spout based on a timer but the amount of time it takes the topology to fully
start up and activate seems to vary from computer to computer. Specifically,
I just updated from 0.8.2 to 0.9.2-incubating and all of my tests are
breaking because the time to activate the topology is taking longer (because
of Netty possibly?). I'd like to make my tests more resilient to things like
this.
Is there something I can look at in LocalCluster where I could do
while(!notActive) { Thread.sleep(50) } ?
This is what my test looks like currently:
StormTopology topology = buildTopology(...);
Config conf = new Config();
conf.setNumWorkers(1);
LocalCluster cluster = new LocalCluster();
cluster.submitTopology(getTopologyName(), conf, topology);
try {
Thread.sleep(4000);
} catch (InterruptedException e) {
e.printStackTrace();
}
cluster.shutdown();
assertEquals(4, MockSinkBolt.getEvents().size());
Thanks!
signature.asc
Description: Message signed with OpenPGP using GPGMail
Re: Good way to test when topology in local cluster is fully active
Vincent P.Taylor,
I played with the testing framework for a little bit last night and don't
see any easy way to provide pauses in between the emissions of the mock
tuples. For instance, my sliding window semantics are heavily orchestrated
by time evictions and triggers which mean that I need to be able to time
the tuples being fed into the tests (i.e. emit a tuple every 500ms and run
the test for 25 secons).
On Tue, Aug 5, 2014 at 2:23 PM, P. Taylor Goetz ptgo...@gmail.com wrote:
My guess is that the slowdown you are seeing is a result of the new
version of ZooKeeper and how it handles IPv4/6.
Try adding the following JVM parameter when running your tests:
-Djava.net.preferIPv4Stack=true
-Taylor
On Aug 4, 2014, at 8:49 PM, Corey Nolet cjno...@gmail.com wrote:
I'm testing some sliding window algorithms with tuples emitted from a
mock spout based on a timer but the amount of time it takes the topology to
fully start up and activate seems to vary from computer to computer.
Specifically, I just updated from 0.8.2 to 0.9.2-incubating and all of my
tests are breaking because the time to activate the topology is taking
longer (because of Netty possibly?). I'd like to make my tests more
resilient to things like this.
Is there something I can look at in LocalCluster where I could do
while(!notActive) { Thread.sleep(50) } ?
This is what my test looks like currently:
StormTopology topology = buildTopology(...);
Config conf = new Config();
conf.setNumWorkers(1);
LocalCluster cluster = new LocalCluster();
cluster.submitTopology(getTopologyName(), conf, topology);
try {
Thread.sleep(4000);
} catch (InterruptedException e) {
e.printStackTrace();
}
cluster.shutdown();
assertEquals(4, MockSinkBolt.getEvents().size());
Thanks!
Re: Good way to test when topology in local cluster is fully active
Sorry- the ipv4 fix worked.
On Tue, Aug 5, 2014 at 9:13 PM, Corey Nolet cjno...@gmail.com wrote:
This did work. Thanks!
On Tue, Aug 5, 2014 at 2:23 PM, P. Taylor Goetz ptgo...@gmail.com wrote:
My guess is that the slowdown you are seeing is a result of the new
version of ZooKeeper and how it handles IPv4/6.
Try adding the following JVM parameter when running your tests:
-Djava.net.preferIPv4Stack=true
-Taylor
On Aug 4, 2014, at 8:49 PM, Corey Nolet cjno...@gmail.com wrote:
I'm testing some sliding window algorithms with tuples emitted from a
mock spout based on a timer but the amount of time it takes the topology to
fully start up and activate seems to vary from computer to computer.
Specifically, I just updated from 0.8.2 to 0.9.2-incubating and all of my
tests are breaking because the time to activate the topology is taking
longer (because of Netty possibly?). I'd like to make my tests more
resilient to things like this.
Is there something I can look at in LocalCluster where I could do
while(!notActive) { Thread.sleep(50) } ?
This is what my test looks like currently:
StormTopology topology = buildTopology(...);
Config conf = new Config();
conf.setNumWorkers(1);
LocalCluster cluster = new LocalCluster();
cluster.submitTopology(getTopologyName(), conf, topology);
try {
Thread.sleep(4000);
} catch (InterruptedException e) {
e.printStackTrace();
}
cluster.shutdown();
assertEquals(4, MockSinkBolt.getEvents().size());
Thanks!
Re: Good way to test when topology in local cluster is fully active
This did work. Thanks!
On Tue, Aug 5, 2014 at 2:23 PM, P. Taylor Goetz ptgo...@gmail.com wrote:
My guess is that the slowdown you are seeing is a result of the new
version of ZooKeeper and how it handles IPv4/6.
Try adding the following JVM parameter when running your tests:
-Djava.net.preferIPv4Stack=true
-Taylor
On Aug 4, 2014, at 8:49 PM, Corey Nolet cjno...@gmail.com wrote:
I'm testing some sliding window algorithms with tuples emitted from a
mock spout based on a timer but the amount of time it takes the topology to
fully start up and activate seems to vary from computer to computer.
Specifically, I just updated from 0.8.2 to 0.9.2-incubating and all of my
tests are breaking because the time to activate the topology is taking
longer (because of Netty possibly?). I'd like to make my tests more
resilient to things like this.
Is there something I can look at in LocalCluster where I could do
while(!notActive) { Thread.sleep(50) } ?
This is what my test looks like currently:
StormTopology topology = buildTopology(...);
Config conf = new Config();
conf.setNumWorkers(1);
LocalCluster cluster = new LocalCluster();
cluster.submitTopology(getTopologyName(), conf, topology);
try {
Thread.sleep(4000);
} catch (InterruptedException e) {
e.printStackTrace();
}
cluster.shutdown();
assertEquals(4, MockSinkBolt.getEvents().size());
Thanks!
Re: Good way to test when topology in local cluster is fully active
Corey,
Have you tried using the integration testing framework that comes with
storm?
Testing.withSimulatedTimeLocalCluster(mkClusterParam,
new TestJob() {
@Override
public void run(ILocalCluster cluster) throws Exception {
CompleteTopologyParam completeTopologyParam = new CompleteTopologyParam();
completeTopologyParam
.setMockedSources(mockedSources);
completeTopologyParam.setStormConf(daemonConf);
completeTopologyParam.setTopologyName(getTopologyName());
Map result = Testing.completeTopology(cluster,
topology, completeTopologyParam);
});
-Vincent
On Mon, Aug 4, 2014 at 8:49 PM, Corey Nolet cjno...@gmail.com wrote:
I'm testing some sliding window algorithms with tuples emitted from a mock
spout based on a timer but the amount of time it takes the topology to
fully start up and activate seems to vary from computer to computer.
Specifically, I just updated from 0.8.2 to 0.9.2-incubating and all of my
tests are breaking because the time to activate the topology is taking
longer (because of Netty possibly?). I'd like to make my tests more
resilient to things like this.
Is there something I can look at in LocalCluster where I could do
while(!notActive) { Thread.sleep(50) } ?
This is what my test looks like currently:
StormTopology topology = buildTopology(...);
Config conf = new Config();
conf.setNumWorkers(1);
LocalCluster cluster = new LocalCluster();
cluster.submitTopology(getTopologyName(), conf, topology);
try {
Thread.sleep(4000);
} catch (InterruptedException e) {
e.printStackTrace();
}
cluster.shutdown();
assertEquals(4, MockSinkBolt.getEvents().size());
Thanks!
Re: Good way to test when topology in local cluster is fully active
Oh Nice. Is this new in 0.9.*? I just updated so I haven't looked much into
what's changed yet, other than Netty.
On Mon, Aug 4, 2014 at 10:40 PM, Vincent Russell vincent.russ...@gmail.com
wrote:
Corey,
Have you tried using the integration testing framework that comes with
storm?
Testing.withSimulatedTimeLocalCluster(mkClusterParam,
new TestJob() {
@Override
public void run(ILocalCluster cluster) throws Exception {
CompleteTopologyParam completeTopologyParam = new CompleteTopologyParam();
completeTopologyParam
.setMockedSources(mockedSources);
completeTopologyParam.setStormConf(daemonConf);
completeTopologyParam.setTopologyName(getTopologyName());
Map result = Testing.completeTopology(cluster,
topology, completeTopologyParam);
});
-Vincent
On Mon, Aug 4, 2014 at 8:49 PM, Corey Nolet cjno...@gmail.com wrote:
I'm testing some sliding window algorithms with tuples emitted from a
mock spout based on a timer but the amount of time it takes the topology to
fully start up and activate seems to vary from computer to computer.
Specifically, I just updated from 0.8.2 to 0.9.2-incubating and all of my
tests are breaking because the time to activate the topology is taking
longer (because of Netty possibly?). I'd like to make my tests more
resilient to things like this.
Is there something I can look at in LocalCluster where I could do
while(!notActive) { Thread.sleep(50) } ?
This is what my test looks like currently:
StormTopology topology = buildTopology(...);
Config conf = new Config();
conf.setNumWorkers(1);
LocalCluster cluster = new LocalCluster();
cluster.submitTopology(getTopologyName(), conf, topology);
try {
Thread.sleep(4000);
} catch (InterruptedException e) {
e.printStackTrace();
}
cluster.shutdown();
assertEquals(4, MockSinkBolt.getEvents().size());
Thanks!
Re: Good way to test when topology in local cluster is fully active
Nevermind, I wrote that before looking. This has been around since 0.8.1.
Thanks again Vincent!
On Mon, Aug 4, 2014 at 11:01 PM, Corey Nolet cjno...@gmail.com wrote:
Oh Nice. Is this new in 0.9.*? I just updated so I haven't looked much
into what's changed yet, other than Netty.
On Mon, Aug 4, 2014 at 10:40 PM, Vincent Russell
vincent.russ...@gmail.com wrote:
Corey,
Have you tried using the integration testing framework that comes with
storm?
Testing.withSimulatedTimeLocalCluster(mkClusterParam,
new TestJob() {
@Override
public void run(ILocalCluster cluster) throws Exception {
CompleteTopologyParam completeTopologyParam = new CompleteTopologyParam();
completeTopologyParam
.setMockedSources(mockedSources);
completeTopologyParam.setStormConf(daemonConf);
completeTopologyParam.setTopologyName(getTopologyName());
Map result = Testing.completeTopology(cluster,
topology, completeTopologyParam);
});
-Vincent
On Mon, Aug 4, 2014 at 8:49 PM, Corey Nolet cjno...@gmail.com wrote:
I'm testing some sliding window algorithms with tuples emitted from a
mock spout based on a timer but the amount of time it takes the topology to
fully start up and activate seems to vary from computer to computer.
Specifically, I just updated from 0.8.2 to 0.9.2-incubating and all of my
tests are breaking because the time to activate the topology is taking
longer (because of Netty possibly?). I'd like to make my tests more
resilient to things like this.
Is there something I can look at in LocalCluster where I could do
while(!notActive) { Thread.sleep(50) } ?
This is what my test looks like currently:
StormTopology topology = buildTopology(...);
Config conf = new Config();
conf.setNumWorkers(1);
LocalCluster cluster = new LocalCluster();
cluster.submitTopology(getTopologyName(), conf, topology);
try {
Thread.sleep(4000);
} catch (InterruptedException e) {
e.printStackTrace();
}
cluster.shutdown();
assertEquals(4, MockSinkBolt.getEvents().size());
Thanks!
Re: Good way to test when topology in local cluster is fully active
No worries. I haven't found the testing framework the easiest thing to
use. If you see errors that say something like cannot convert object to
iterable I use the code below to disable storm from being able to call
System.exit()
private static class ExitTrappedException extends SecurityException {
}
private static void forbidSystemExitCall() {
final SecurityManager securityManager = new SecurityManager() {
public void checkPermission(Permission permission) {
if (permission.getName().startsWith(exitVM)) {
throw new ExitTrappedException();
}
}
};
System.setSecurityManager(securityManager);
}
private static void enableSystemExitCall() {
System.setSecurityManager(null);
}
On Mon, Aug 4, 2014 at 11:02 PM, Corey Nolet cjno...@gmail.com wrote:
Nevermind, I wrote that before looking. This has been around since 0.8.1.
Thanks again Vincent!
On Mon, Aug 4, 2014 at 11:01 PM, Corey Nolet cjno...@gmail.com wrote:
Oh Nice. Is this new in 0.9.*? I just updated so I haven't looked much
into what's changed yet, other than Netty.
On Mon, Aug 4, 2014 at 10:40 PM, Vincent Russell
vincent.russ...@gmail.com wrote:
Corey,
Have you tried using the integration testing framework that comes with
storm?
Testing.withSimulatedTimeLocalCluster(mkClusterParam,
new TestJob() {
@Override
public void run(ILocalCluster cluster) throws Exception {
CompleteTopologyParam completeTopologyParam = new
CompleteTopologyParam();
completeTopologyParam
.setMockedSources(mockedSources);
completeTopologyParam.setStormConf(daemonConf);
completeTopologyParam.setTopologyName(getTopologyName());
Map result = Testing.completeTopology(cluster,
topology, completeTopologyParam);
});
-Vincent
On Mon, Aug 4, 2014 at 8:49 PM, Corey Nolet cjno...@gmail.com wrote:
I'm testing some sliding window algorithms with tuples emitted from a
mock spout based on a timer but the amount of time it takes the topology to
fully start up and activate seems to vary from computer to computer.
Specifically, I just updated from 0.8.2 to 0.9.2-incubating and all of my
tests are breaking because the time to activate the topology is taking
longer (because of Netty possibly?). I'd like to make my tests more
resilient to things like this.
Is there something I can look at in LocalCluster where I could do
while(!notActive) { Thread.sleep(50) } ?
This is what my test looks like currently:
StormTopology topology = buildTopology(...);
Config conf = new Config();
conf.setNumWorkers(1);
LocalCluster cluster = new LocalCluster();
cluster.submitTopology(getTopologyName(), conf, topology);
try {
Thread.sleep(4000);
} catch (InterruptedException e) {
e.printStackTrace();
}
cluster.shutdown();
assertEquals(4, MockSinkBolt.getEvents().size());
Thanks!
