JH, Thank you so much for the detailed analysis. Let me take a look into this issue when I get back home.
2008-02-07 (목), 23:59 -0800, jhcha 쓰시길:
> Thank Mike for replying my question.
>
> I read your answers but...
>
> I think MINA already knew the session was being closed in my questioned
> situation.
> because my server session always close after sending the echo data and
> I debugged MINA source
> org.apache.mina.transport.socket.nio.SocketIoProcessor.Worker.run method
> always call doRemove() at (SocketIoProcessor.java:489 line).
> then doRemove method always call
> session.getServiceListeners().fireSessionDestroyed(session);
> at (SocketIoProcessor.java:188 line).
>
> try {
> key.cancel();
> ch.close();
> } catch (IOException e) {
> session.getFilterChain().fireExceptionCaught(session, e);
> } finally {
> releaseWriteBuffers(session);
> session.getServiceListeners().fireSessionDestroyed(session); <---
> SocketIoProcessor.java:188 line
> }
>
>
> and the above session parameter of the fireSessionDestroyed method was
> always not null.
>
> the orgin of the problem is the sessions having serviceAddress is not in
> managedSessions hashmap
> in fireSessionDestroyed method at (IoServiceListenerSupport.java:192 line)
>
> // Get the session set.
> Set<IoSession> sessions = managedSessions.get(serviceAddress); <---
> IoServiceListenerSupport.java:192 line
> // Ignore if unknown.
> if (sessions == null) {
> return;
> }
>
> The fireSessionDestroyed method call propagation stopped only sometimes
> Because sessions is not gotten from managedSessions at
> (IoServiceListenerSupport.java:192 line).
>
> I used setIdleTimeout with read mode, but
> if sessionClosed method is not called, sessionIdle or sessionClosed method
> will never be called after that.
>
> My question is that the filter chain fireSessionClosed method and listener'
> sessionDestroyed method
> Should not know if the destroying session be not in the managedSessions
> hashmap ?
>
> // Fire session events.
> session.getFilterChain().fireSessionClosed(session); <---
> IoServiceListenerSupport.java:208 line
>
> // Fire listener events.
> try {
> for (IoServiceListener listener : listeners) {
> listener.sessionDestroyed(session); <---
> }
> }
>
>
> If fireSessionDestroyed method were called with the parameter the closing
> session
> but not in managedSessions hashmap, Should filter chain fireSessionClosed
> method and
> listener' sessionDestroyed method not called ?
>
>
> in the following MINA source below the IoServiceListenerSupport.java:147
> line.
>
>
> public void fireSessionCreated(IoSession session) {
> SocketAddress serviceAddress = session.getServiceAddress();
>
> // Get the session set.
> Set<IoSession> s = new IdentityHashSet<IoSession>();
> Set<IoSession> sessions =
> managedSessions.putIfAbsent(serviceAddress,
> Collections.synchronizedSet(s));
> boolean firstSession;
>
> if (null == sessions) {
> sessions = s;
> firstSession = true;
> } else {
> firstSession = false;
> }
>
> // If already registered, ignore.
> if (!sessions.add(session)) { <---
> return;
> }
>
> // If the first connector session, fire a virtual service activation
> event.
> if (session.getService() instanceof IoConnector && firstSession) {
> fireServiceActivated(session.getService(), session
> .getServiceAddress(), session.getHandler(), session
> .getServiceConfig());
> }
>
> // Fire session events.
> session.getFilterChain().fireSessionCreated(session);
> session.getFilterChain().fireSessionOpened(session);
>
> // Fire listener events.
> for (IoServiceListener listener : listeners) {
> listener.sessionCreated(session);
> }
> }
>
>
> I think the code "if (!sessions.add(session))" is always true,
> So filter chain fireSessionCreated method and listener' sessionCreated
> method is
> always called in spite of the heavy unstable tcp stress condition.
>
> I think
> Creaing callback method and destroying callback method should have the same
> calling count.
>
> But creating callback logic and managedSessions hashmap are not related
> directly
> in the above fireSessionCreated method.
>
> "not related directly" means that
> whether sessions put managedSessions hashmap or not, fireSessionCreated and
> sessionCreated are called.
>
> Then destroying callback logic and managedSessions hashmap should be also
> not related directly.
> but MINA has the direct relation with managedSessions hashmap
>
> "direct relation" means that
> as the below code shows, if sessions is not found in managedSessions hashmap
> then
> fireSessionClosed and sessionDestroyed method are not called.
>
>
> in the code "public void fireSessionDestroyed(IoSession session)" method
> of IoServiceListenerSupport.java
>
> ...
>
> // Get the session set.
> Set<IoSession> sessions = managedSessions.get(serviceAddress); <---
> IoServiceListenerSupport.java:192 line
> // Ignore if unknown.
> if (sessions == null) { <---
> return;
> }
>
> sessions.remove(session);
>
> boolean lastSession = false;
>
> // Try to remove the remaining empty session set after removal.
> if (sessions.isEmpty()) {
> lastSession = managedSessions.remove(serviceAddress, sessions);
> <====
> }
>
> // Fire session events.
> session.getFilterChain().fireSessionClosed(session);
>
> // Fire listener events.
> try {
> for (IoServiceListener listener : listeners) {
> listener.sessionDestroyed(session);
> }
> }
>
> ...
>
> My inference is that
> When heavy thread competition (and other session parameters but with the
> same serviceAddress),
> the one thread can remove the other using sessions from managedSessions
> hashmap.
>
> 1) the one thread : sessions.isEmpty() succeeded in fireSessionDestroyed
> 2) the other thread : called fireSessionCreated
> 3) the other thread : sessions.add(session) suceeded in fireSessionCreated
> 4) the one thread : managedSessions.remove(serviceAddress, sessions);
> (above "<===" line)
> 5) the other thread : called fireSessionDestroyed
> 6) the other thread : managedSessions.get(serviceAddress); fail
>
> (above "<---" line)
>
> other thread fails to get sessions from managedSessions hashmap with the
> serviceAddress .
> then fireSessionClosed propagation stops. (but should it continue!)
>
>
> It is just my inference.
> and it's possible for me to misunderstand the MINA logic.
>
>
> Thank you again.
>
>
> J. H. Cha
>
>
>
> Mike Heath-4 wrote:
> >
> > It's possible that the server never received the TCP FIN packet and
> > therefor did not know that it should close the IoSession. There are
> > situations where a host has no way of knowing that the TCP socket has
> > closed unless it tries to write data to the socket and that write fails.
> > This is not a limitation of MINA. This is just the way socket I/O works.
> >
> > If this is the cause of your problem, you have a couple of options. You
> > could use some type of watch dog mechanisms that sends data at a regular
> > interval. If the connection is no longer valid, the write will fail and
> > the socket will get closed thereby triggering a sessionClosed event. Or
> > you could use the MINA read idle notification. This way after a certain
> > amount of time has passed and your server hasn't received any data for a
> > particular IoSession, you can manually close the IoSession.
> >
> > If your problem still persists after implementing one of these
> > solutions, please let us know.
> >
> > -Mike
> >
> > jhcha wrote:
> >> Dear Mina developers.
> >> I'm trying to use mina 1.1.5 to develop the very big stress server.
> >> I test my server with jmeter like some echo test now.
> >>
> >> test environment
> >>
> >> * jmeter :
> >> echo message size : 1024 bytes
> >> jmeter threads : 100
> >> loop count : 1000
> >> sampler : TCP Sampler
> >> reuse connection : false
> >>
> >> * server : HP-UX 2 cores
> >> java HotSpot(TM) Server VM (build 1.5.0.07 jinteg:03.20.07-11:05
> >> IA64, mixed mode)
> >> my server(mina)
> >>
> >> * client : pc (dual core).
> >> jmeter
> >>
> >> when I tested the stress, I found sessionCreated called counts are a
> >> little
> >> bigger than sessionClosed called counts.
> >> so, I tried to find out why that mismatch occured during several days.
> >> and I found the followings :
> >>
> >>
> >> org.apache.mina.common.support.IoServiceListenerSupport
> >> {
> >>
> >> public void fireSessionCreated(IoSession session) {
> >> SocketAddress serviceAddress = session.getServiceAddress();
> >>
> >> // Get the session set.
> >> Set<IoSession> s = new IdentityHashSet<IoSession>();
> >> Set<IoSession> sessions =
> >> managedSessions.putIfAbsent(serviceAddress,
> >> Collections
> >> .synchronizedSet(s));
> >> boolean firstSession;
> >>
> >> if (null == sessions) {
> >> sessions = s;
> >> firstSession = true;
> >> } else {
> >> firstSession = false;
> >> }
> >>
> >> // If already registered, ignore.
> >> if (!sessions.add(session)) {
> >> return;
> >> }
> >>
> >> // If the first connector session, fire a virtual service
> >> activation
> >> // event.
> >> if (session.getService() instanceof IoConnector &&
> >> firstSession) {
> >> fireServiceActivated(session.getService(),
> >> session.getServiceAddress(),
> >> session
> >> .getHandler(),
> >> session.getServiceConfig());
> >> }
> >>
> >> // Fire session events.
> >> session.getFilterChain().fireSessionCreated(session); <--
> >> called
> >> correctly
> >> session.getFilterChain().fireSessionOpened(session); <-- called
> >> correctly
> >>
> >> // Fire listener events.
> >> for (IoServiceListener listener : listeners) {
> >> listener.sessionCreated(session); <-- called correctly
> >> }
> >> }
> >>
> >>
> >> public void fireSessionDestroyed(IoSession session) {
> >> boolean lastSession = false;
> >> SocketAddress serviceAddress = session.getServiceAddress();
> >>
> >> // Get the session set.
> >> Set<IoSession> sessions = managedSessions.get(serviceAddress);
> >> // Ignore if unknown.
> >> if (sessions == null) {
> >> // logger.error("fireSessionDestroyed() : sessions ==
> >> null < " +
> >> session);
> >> return; <-- occured sometimes !!!!!!!!!!!!!!!!!!
> >> }
> >>
> >> sessions.remove(session);
> >>
> >> // Try to remove the remaining empty session set after removal.
> >> if (sessions.isEmpty()) {
> >> lastSession = managedSessions.remove(serviceAddress,
> >> sessions);
> >> }
> >>
> >> // Fire session events.
> >> session.getFilterChain().fireSessionClosed(session); <-- skipped
> >> sometimes
> >>
> >> // Fire listener events.
> >> try {
> >> for (IoServiceListener listener : listeners) {
> >> listener.sessionDestroyed(session); <-- skipped
> >> sometimes
> >> }
> >> } finally {
> >> // Fire a virtual service deactivation event for the
> >> last session of
> >> // the connector.
> >> // TODO double-check that this is *STILL* the last
> >> session. May not
> >> // be the case
> >> if (session.getService() instanceof IoConnector &&
> >> lastSession) {
> >> fireServiceDeactivated(session.getService(),
> >> session.getServiceAddress(), session
> >> .getHandler(),
> >> session.getServiceConfig()); <-- skipped sometimes
> >> }
> >> }
> >> }
> >>
> >> the above fireSessionDestroyed method called as the same number as
> >> fireSessionCreated method.
> >>
> >> but the mysterious event is that the sessions is not gotten from
> >> managedSessions in fireSessionDestroyed,
> >> i.e. SocketAddress key is not found in managedSessions.
> >> then the remaining logic was skipped.
> >> so fireSessionClosed, sessionDestroyed ... are not called sometimes.
> >>
> >> [IoServiceListenerSupport] - fireSessionDestroyed() : sessions == null <
> >> (SOCKET, R: /172.26.73.107:15059, L: /10.88.50.205:30000, S:
> >> 0.0.0.0/0.0.0.0:30000)
> >> [IoServiceListenerSupport] - fireSessionDestroyed() : sessions == null <
> >> (SOCKET, R: /172.26.73.107:15961, L: /10.88.50.205:30000, S:
> >> 0.0.0.0/0.0.0.0:30000)
> >> [IoServiceListenerSupport] - fireSessionDestroyed() : sessions == null <
> >> (SOCKET, R: /172.26.73.107:23883, L: /10.88.50.205:30000, S:
> >> 0.0.0.0/0.0.0.0:30000)
> >> [IoServiceListenerSupport] - fireSessionDestroyed() : sessions == null <
> >> (SOCKET, R: /172.26.73.107:24036, L: /10.88.50.205:30000, S:
> >> 0.0.0.0/0.0.0.0:30000)
> >> [IoServiceListenerSupport] - fireSessionDestroyed() : sessions == null <
> >> (SOCKET, R: /172.26.73.107:31341, L: /10.88.50.205:30000, S:
> >> 0.0.0.0/0.0.0.0:30000)
> >> [IoServiceListenerSupport] - fireSessionDestroyed() : sessions == null <
> >> (SOCKET, R: /172.26.73.107:31343, L: /10.88.50.205:30000, S:
> >> 0.0.0.0/0.0.0.0:30000)
> >> [IoServiceListenerSupport] - fireSessionDestroyed() : sessions == null <
> >> (SOCKET, R: /172.26.73.107:31345, L: /10.88.50.205:30000, S:
> >> 0.0.0.0/0.0.0.0:30000)
> >> [IoServiceListenerSupport] - fireSessionDestroyed() : sessions == null <
> >> (SOCKET, R: /172.26.73.107:45839, L: /10.88.50.205:30000, S:
> >> 0.0.0.0/0.0.0.0:30000)
> >> [IoServiceListenerSupport] - fireSessionDestroyed() : sessions == null <
> >> (SOCKET, R: /172.26.73.107:5317, L: /10.88.50.205:30000, S:
> >> 0.0.0.0/0.0.0.0:30000)
> >> [IoServiceListenerSupport] - fireSessionDestroyed() : sessions == null <
> >> (SOCKET, R: /172.26.73.107:8471, L: /10.88.50.205:30000, S:
> >> 0.0.0.0/0.0.0.0:30000)
> >>
> >> I think the reason that the 100 client threads tries to proceed "conect,
> >> send, receive, close",
> >> different sessions has the same client SocketAddress at times in server,
> >> and the origin of that problem is client's stress environment.
> >>
> >> It is possible the following senario :
> >>
> >> I/O processor thread I/O processor thread client thread client
> >> thread
> >> one two one two
> >> --------------------------------------------------------------------------------
> >> 1) connect(the same
> >> client SocketAddress)
> >> 2)fireSessionCreated
> >> 3)putIfAbsent
> >> 4)disconnect
> >>
> >> 5)connect(the same client SocketAddress)
> >> 6)fireSessionCreated
> >> 7)putIfAbsent
> >> 8)fireSessionDestroyed
> >> 9)remove
> >>
> >> 10)disconnect
> >> 11)fireSessionDestroyed
> >> 12)get fail
> >>
> >>
> >>
> >> I changed the mina source IoServiceListenerSupport.java
> >>
> >> org.apache.mina.common.support.IoServiceListenerSupport
> >> ...
> >>
> >> public void fireSessionDestroyed(IoSession session) {
> >> boolean lastSession = false;
> >> SocketAddress serviceAddress = session.getServiceAddress();
> >>
> >> // Get the session set.
> >> Set<IoSession> sessions = managedSessions.get(serviceAddress);
> >> // Ignore if unknown.
> >> if (sessions == null) {
> >> // return; <-- delete
> >> } else { // <-- add
> >>
> >> sessions.remove(session);
> >>
> >> // Try to remove the remaining empty session set after
> >> removal.
> >> if (sessions.isEmpty()) {
> >> lastSession =
> >> managedSessions.remove(serviceAddress, sessions);
> >> }
> >> } // <-- add
> >> ....
> >>
> >> then the Create, Destroy call mismatch was cleared.
> >> but I don't know this is correct solution.
> >>
> >> Would you tell me this is correct or not?
> >>
> >> I think this situation is very rare but...
> >> I hope the mina is the best network framework
> >> and this situation shoud be also considered.
> >>
> >>
> >> Thank you
> >>
> >> J. H. Cha
> >
> >
> >
>
--
what we call human nature is actually human habit
--
http://gleamynode.net/
signature.asc
Description: This is a digitally signed message part
