Hi Carsten,
Sorry, I think didn’t make myself clear.
I am only replacing ONE listener with a BlockingListener, and then I am
blocking ONE event delivery. So I do expect to loose ONE event, not 70k…
Or am I missing something?
Erwin
@Test
public void testEventing() throws Exception {
// this.addListener(PREFIX + "/0", null);
this.addBlockingListener(PREFIX + "/0", null);
this.addListener(PREFIX + "/1", null);
this.addListener(PREFIX + "/2", null);
...
On Nov 29, 2017, at 09:21, Carsten Ziegeler
<[email protected]<mailto:[email protected]>> wrote:
Thanks
I'm not sure if event admin could/should do anything in this case. Each
blocking listener takes away one thread from the thread pool. As soon as
you have as many blocking listeners as the pool has threads, event admin
will not deliver any event anymore
Regards
Carsten
Erwin Hogeweg wrote
Hi Carsten,
After analyzing a bunch of log files we found some evidence that suggested that
an EventAdminThread might be blocked.
I was able to duplicate it with a JUnit test. I created a BlockingListener with
extends Listener and override the handleEvent() method. As you can see below we
are loosing almost 70000 events because of this one block.
I know, eventHandlers are not supposed to block, so we definitely have a design
issue here, but I figured I share the results with you because I am not sure
this is what’s expected.
Kind Regards,
Erwin
@Override
public void handleEvent(final Event event) {
this.test.handleEvent(event, this.payload);
count++;
if (count == 10) {
logger.info<http://logger.info>("{} is blocking forever",
Thread.currentThread().getName());
while (true) {
try {
Thread.sleep(60000);
} catch (InterruptedException e) {
//
}
}
}
}
[org.apache.felix.eventadmin.ittests.StressTestIT] : Starting eventing test
StressTestIT
[org.apache.felix.eventadmin.ittests.StressTestIT] : Preparing test with 15
threads and 10000 events per thread.
[org.apache.felix.eventadmin.ittests.StressTestIT] : Expecting 1050000 events.
[org.apache.felix.eventadmin.ittests.StressTestIT] : Started test with 15
threads and 10000 events.
[org.apache.felix.eventadmin.ittests.StressTestIT] : Received 0 events so far.
[org.apache.felix.eventadmin.ittests.StressTestIT] : Received 0 events so far.
[org.apache.felix.eventadmin.ittests.StressTestIT] :
[org.apache.felix.eventadmin.ittests.StressTestIT] : Started thread
[org.apache.felix.eventadmin.ittests.StressTestIT] : Started thread
Started thread
[org.apache.felix.eventadmin.ittests.StressTestIT] :
[org.apache.felix.eventadmin.ittests.StressTestIT] :
[org.apache.felix.eventadmin.ittests.StressTestIT] : Started thread
[org.apache.felix.eventadmin.ittests.StressTestIT] : Started thread
[org.apache.felix.eventadmin.ittests.StressTestIT] : Started thread
[org.apache.felix.eventadmin.ittests.StressTestIT] : Started thread
[org.apache.felix.eventadmin.ittests.StressTestIT] : Started thread
[org.apache.felix.eventadmin.ittests.StressTestITStarted thread
Started thread
] : Started thread
[org.apache.felix.eventadmin.ittests.StressTestIT] : Started thread
[org.apache.felix.eventadmin.ittests.StressTestIT] : Started thread
[org.apache.felix.eventadmin.ittests.StressTestIT] : Started thread
[org.apache.felix.eventadmin.ittests.StressTestIT] : Started thread
[org.apache.felix.eventadmin.ittests.StressTestIT] : Received 12 events so far.
[org.apache.felix.eventadmin.ittests.BlockingListener] : EventAdminThread #6 is
blocking forever
[org.apache.felix.eventadmin.ittests.StressTestIT] : Send 10000 events.
[org.apache.felix.eventadmin.ittests.StressTestIT] : Send 10000 events.
[org.apache.felix.eventadmin.ittests.StressTestIT] : Send 10000 events.
[org.apache.felix.eventadmin.ittests.StressTestIT] : Send 10000 events.
[org.apache.felix.eventadmin.ittests.StressTestIT] : Send 10000 events.
[org.apache.felix.eventadmin.ittests.StressTestIT] : Send 10000 events.
[org.apache.felix.eventadmin.ittests.StressTestIT] : Send 10000 events.
[org.apache.felix.eventadmin.ittests.StressTestIT] : Send 10000 events.
[org.apache.felix.eventadmin.ittests.StressTestIT] : Send 10000 events.
[org.apache.felix.eventadmin.ittests.StressTestIT] : Send 10000 events.
[org.apache.felix.eventadmin.ittests.StressTestIT] : Send 10000 events.
[org.apache.felix.eventadmin.ittests.StressTestIT] : Send 10000 events.
[org.apache.felix.eventadmin.ittests.StressTestIT] : Send 10000 events.
[org.apache.felix.eventadmin.ittests.StressTestIT] : Send 10000 events.
[org.apache.felix.eventadmin.ittests.StressTestIT] : Send 10000 events.
[org.apache.felix.eventadmin.ittests.StressTestIT] : Received 980007 events so
far.
[org.apache.felix.eventadmin.ittests.StressTestIT] : Received 980007 events so
far.
[org.apache.felix.eventadmin.ittests.StressTestIT] : Received 980007 events so
far.
[org.apache.felix.eventadmin.ittests.StressTestIT] : Received 980007 events so
far.
[org.apache.felix.eventadmin.ittests.StressTestIT] : Received 980007 events so
far.
[org.apache.felix.eventadmin.ittests.StressTestIT] : Received 980007 events so
far.
[org.apache.felix.eventadmin.ittests.StressTestIT] : Received 980007 events so
far.
…
On Nov 15, 2017, at 21:04, Carsten Ziegeler
<[email protected]<mailto:[email protected]><mailto:[email protected]>>
wrote:
Hi,
we haven't experienced anything like this so far and I wouldn't know if
any better out of the box way to trouble shoot. I guess the best would
be to add additional logging to event admin, so you can follow what's
happening inside event admin.
If it is somehow reproducible, then switching from postEvent to
sendEvent could also help in identifying whether there is a problem in
the postEvent handling (which is more complicated than sendEvent).
Regards
Carsten
Erwin Hogeweg wrote
Hi,
I have a really bizarre problem. It looks like the eventAdmin is intermittent
but frequently loosing events. I find this very hard to believe but the
evidence seems to support my suspicion.
I spit out a log msgs just before the event is posted and also when the event
is handled. There is only one eventHandler for this topic, and sure enough
every now and again I see and event be posted but it never comes out at the
other end.
I have the event admin log level set to 4, but there aren’t any msgs to
indicate that something wrong.
I am really at a loss here. Any suggestions for further trouble shooting would
be greatly appreciated.
BTW… this seems to happen (and gradually increase) after the system has been up
for several weeks which points in the direction of a resource leak, but I don’t
see that either.
Regards,
Erwin
Java8
EventAdmin-1.4.6
Equinox: 3.10.2.v20150203-1939
Producer side:
...
if (eventAdmin != null) {
if (LOG.isDebugEnabled()) {
LOG.debug("Posting: " + newEvent);
}
eventAdmin.postEvent(newEvent);
} else {
LOG.error("eventAdmin is null.");
}
Consumer side:
@Override
public void handleEvent( Event event ) {
LOG.info<http://LOG.info><http://LOG.info><http://LOG.info>("handleEvent: " +
event);
...
Erwin Hogeweg
CTO
3690 Airport Road
Boca Raton, FL 33431
P. +1 (954) 556-6565
M. +1 (561) 306-7395
F. +1 (561) 948-2730
[Seecago]<http://www.seecago.com>
--
Carsten Ziegeler
Adobe Research Switzerland
[email protected]<mailto:[email protected]><mailto:[email protected]>
Erwin Hogeweg
CTO
3690 Airport Road
Boca Raton, FL 33431
P. +1 (954) 556-6565
M. +1 (561) 306-7395
F. +1 (561) 948-2730
[Seecago]<http://www.seecago.com>
--
Carsten Ziegeler
Adobe Research Switzerland
[email protected]<mailto:[email protected]>
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Erwin Hogeweg
CTO
3690 Airport Road
Boca Raton, FL 33431
P. +1 (954) 556-6565
M. +1 (561) 306-7395
F. +1 (561) 948-2730
[Seecago]<http://www.seecago.com>
