[jira] Created: (DIRMINA-767) Move encoder/decoder out of the session Attributes
Move encoder/decoder out of the session Attributes
--
Key: DIRMINA-767
URL: https://issues.apache.org/jira/browse/DIRMINA-767
Project: MINA
Issue Type: Improvement
Affects Versions: 2.0.0-RC1
Reporter: Emmanuel Lecharny
Priority: Minor
Fix For: 3.0.0-M1
When we initialize a new session with a ProrocolCodecFilter in the chain, the
filter is storing the encoder and decoder into the session's Attribute :
private void initCodec(IoSession session) throws Exception {
// Creates the decoder and stores it into the newly created session
ProtocolDecoder decoder = factory.getDecoder(session);
session.setAttribute(DECODER, decoder);
// Creates the encoder and stores it into the newly created session
ProtocolEncoder encoder = factory.getEncoder(session);
session.setAttribute(ENCODER, encoder);
}
Assuming that the codec is stateless, there is no reason to do that. If we have
thousands of sessions, that would store thousands of references to the encoder
and the decoder in the session hashmap, forever.
The factory used to get those elements can efficiently provide references to
those elements to the codec when needed.
One could think that two sessions could use two different codec for an
application, but this is a non-sense, as the codec is stored in the attributes
at the beginning, and the encoder/decoder selection does not use any context,
so they will always be the same.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
Re: Debugging session on MINA ... ouch !
expressions are a mix of :-) nd :-( Next time just let me know when you folks plan a debugging marathon :-) May be we can use some screen-sharing tool and I can also participate means listen :-) It would have been a good learning. thanks ashish On Fri, Feb 19, 2010 at 8:04 PM, Emmanuel Lecharny elecha...@gmail.com wrote: Hi, here is a memo describing the debugging session we did yesterday with Julien. I have added TODO each time I think we can do better. MINA analyze when processing one single message. The server receives hello and should return HELLO. - We start in the main IoProcessor loop, after a select() returning 1 on a channel set with OP_READ, as we have just received the message. - the process() method is called. It does two things : 1) if the session is readable and not suspended, read the channel 2) is the session is writable and not suspended, add the session to the queue of session ready for flush, if not already there. TODO : at this point, we have no idea if the session has some pending writes. Check if the OP_WRTE flag has been set on the selectionKey - in the read() method, we allocate a buffer with the size configured for the session. Usually, it's way to big if you just have hundreds of bytes to read, and if you set the config to 65000… TODO : use smaller buffers - Then we call the chain's messageReceived() method - In the ProtocolCodecFilter, we grab the decoder instance, which is stored in the session's Attributes TODO : There is no good reason to store the instance in the attributes. It should be part of the session parameters. - then we create a new instance of a decoderOut. TODO : The only reason we have such an object created here is that we need to create a queue to store the result of the decoding. - We now decode the incoming data, until the buffer is empty. We may have more than one message in the buffer, so each one of them is decoded and enqueued in the encoderOut queue. - The doDecode() method is called. This is the user provided code. The decoded message is enqueued, and we do that until the doDecode() method returns false (meaning we don't have anymore message to decode) - We call the encodedOut.flush() method to go up in the chain TODO : it should be done directly here - For each decoded message in the queue, call the next filter messageReceived() event TODO : we went through the tail filter, which is updating some stats used by the idle session handler, but this should be done in another place. Everyone does not want to deal with idle sessions … My be an IdleFilter could help ? - In the handler, we process the message, and write the response calling session.write( response ) - The session.write() method create a writeFuture and a WriteRequest object containing the response, the recipient's address and the future TODO : this WriteFuture is never used. - We go back to the chain in reverse order, firing the FilterWrite method TODO : This should be a separate chain - We get the encoded and we create an encoderOut instance, containing a queue TODO : here, unless the encoder creates more than one buffer to be sent, there is no need to create a queue. We can also call the encoder, get back the result and flush it, doing so receptively until all the encoded pieces has been generated (but I feel like we better get back a set of buffers instead as a response. No need of a queue) - The user's provided encoder method is called, the encoded buffer is enqueued. It can still be a plain Message, if we want to go though another encoder - At this point, we call the flushWithoutFuture() method which loop on the encoderOut queue to send all the encoded buffers - for each encoded message in the queue, we create an EncodedWriteRequest encapsulating the encoded buffer, and we go down in the chain - The WriteRequest is enqueued in the session WriteRequestQueue, and if the session is not suspended for write, we call the processor.flus() method TODO : there is no reason to call this method now. Also the session.getProcessor() method ask the processor pool for the session's processor, which is useless, as each session is attached to a single IoProtocol. We must store not the pool, but the processor - If the session is not already in the queue of session waiting to be flushed, we add it into this queue, then we wake up the selector TODO : Waking up the selector at this point is questionable. It's probably totally useless. - Ok, we go back, and call the filterWrite() method once again in the protocolCodecFilter, with a new MessageWriteRequest instance, encapsulating the response. TODO : WTF ??? Why do we process the same message twice ??? - In the HeadFilter, we grab the message, which is empty. In fact, this empty message is used as a marker for the 'end of message'. Nevertheless, we add this empty message to the WriteRequestQueue - We ask the processor to flush again the session,
Fwd: Re: Debugging session on MINA ... ouch !
Ooops, mailed to the wrong list... Forwarded to mina On 2/20/10 12:37 PM, Ashish wrote: expressions are a mix of :-) nd :-( Next time just let me know when you folks plan a debugging marathon :-) May be we can use some screen-sharing tool and I can also participate means listen :-) It would have been a good learning. The reason I wrote this long mail was to be able to summup what we saw so that everyone on the ML can benefit from this session. I don't know about tools that allows many people to share a common env (I mean, tools that work), but if we have one, we could organize such debugging session. In this very case, we didn't planned to do such a debugging session, Julien just called me the day before that he will be in Paris for the evening. Usually, we go to a restaurant, have a couple of beers and wine plus good food, and we brag about people on the project, about politicians, about girls, about other coders and share our opinion hat we are the only good coders on earth ;) We decided that may be it would be better to have a pizza and discuss about MINA 2 and 3 face to face, instead of having good time outside, and it was really a good session. That's what ApacheCon are good for : F2F meetings are good for sharing feelings about the code. But it's only twice a year ... In any case, ML remains the place where discussions about the project should be done, and when we discuss either F2F or on IRC/IM, these discussions should be reflected on the ML. It's all about community ! -- Regards, Cordialement, Emmanuel Lécharny www.nextury.com
[jira] Created: (DIRMINA-768) Don't use a queue to store the encoded/decoded messages
Don't use a queue to store the encoded/decoded messages --- Key: DIRMINA-768 URL: https://issues.apache.org/jira/browse/DIRMINA-768 Project: MINA Issue Type: Improvement Affects Versions: 2.0.0-RC1 Reporter: Emmanuel Lecharny Fix For: 3.0.0-M1 Encoding/decoding messages is done using an encoder/decoder and a dedicated data structure storing the encoded/decoded result. This data structure contains a queue to store the results (we may have more than one). The problems with this approach are that : - it's more complex than necessary, - we have to allocate those data structure for each session - we have to allocate a queue for each new session - we will store in memory all the results until the encoding/decoding is fully done, before processing them The last point is really worrying, as if we have a huge piece of data coming from a client, and if we decode it to produce tens of messages, it's perfectly possible that when processing the very first message, the handler might decide that the connection should be killed (for instance if the fisrt message is not expected). We then have decoded all the remaining messages for nothing, wasting CPU and memory. I suggest that the decode method returns the result one by one, until we can't anymore decode anything. From the client POV, that does not change a lot of things, in fact, it should be totally transparent. From the performance POV, we might have a small gain here, as we won't create anymore those intermediate structures, so we will gain on the spared instance creation, and also on the GC. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
Build failed in Hudson: MINA-trunk-j dk1.6-ubuntu » Apache MINA Legal #226
See
http://hudson.zones.apache.org/hudson/job/MINA-trunk-jdk1.6-ubuntu/org.apache.mina$mina-legal/226/
--
[INFO]
[INFO] Building Apache MINA Legal
[INFO]task-segment: [clean, install]
[INFO]
[INFO] [clean:clean {execution: default-clean}]
[INFO] Deleting directory
http://hudson.zones.apache.org/hudson/job/MINA-trunk-jdk1.6-ubuntu/org.apache.mina$mina-legal/ws/target
[INFO] [resources:resources {execution: default-resources}]
[WARNING] Using platform encoding (UTF-8 actually) to copy filtered resources,
i.e. build is platform dependent!
[INFO] Copying 1 resource
[INFO] [compiler:compile {execution: default-compile}]
[INFO] No sources to compile
[INFO] [resources:testResources {execution: default-testResources}]
[WARNING] Using platform encoding (UTF-8 actually) to copy filtered resources,
i.e. build is platform dependent!
[INFO] skip non existing resourceDirectory
http://hudson.zones.apache.org/hudson/job/MINA-trunk-jdk1.6-ubuntu/org.apache.mina$mina-legal/ws/src/test/resources
[INFO] [compiler:testCompile {execution: default-testCompile}]
[INFO] Not compiling test sources
[INFO] [surefire:test {execution: default-test}]
[INFO] Tests are skipped.
[INFO] [jar:jar {execution: default-jar}]
[INFO] Building jar:
http://hudson.zones.apache.org/hudson/job/MINA-trunk-jdk1.6-ubuntu/org.apache.mina$mina-legal/ws/target/mina-legal-2.0.0-RC2-SNAPSHOT.jar
[INFO] Preparing source:jar
[WARNING] Removing: jar from forked lifecycle, to prevent recursive invocation.
[INFO] No goals needed for project - skipping
[INFO] [source:jar {execution: attach-source}]
[INFO] Building jar:
http://hudson.zones.apache.org/hudson/job/MINA-trunk-jdk1.6-ubuntu/org.apache.mina$mina-legal/ws/target/mina-legal-2.0.0-RC2-SNAPSHOT-sources.jar
[INFO] [install:install {execution: default-install}]
[INFO] Installing
http://hudson.zones.apache.org/hudson/job/MINA-trunk-jdk1.6-ubuntu/org.apache.mina$mina-legal/ws/target/mina-legal-2.0.0-RC2-SNAPSHOT.jar
to
/home/hudson/.m2/repository/org/apache/mina/mina-legal/2.0.0-RC2-SNAPSHOT/mina-legal-2.0.0-RC2-SNAPSHOT.jar
[HUDSON] Archiving
http://hudson.zones.apache.org/hudson/job/MINA-trunk-jdk1.6-ubuntu/org.apache.mina$mina-legal/ws/pom.xml
to
/export/home/hudson/hudson/jobs/MINA-trunk-jdk1.6-ubuntu/modules/org.apache.mina$mina-legal/builds/2010-02-20_18-57-33/archive/org.apache.mina/mina-legal/2.0.0-RC2-SNAPSHOT/pom.xml
[HUDSON] Archiving
http://hudson.zones.apache.org/hudson/job/MINA-trunk-jdk1.6-ubuntu/org.apache.mina$mina-legal/ws/target/mina-legal-2.0.0-RC2-SNAPSHOT.jar
to
/export/home/hudson/hudson/jobs/MINA-trunk-jdk1.6-ubuntu/modules/org.apache.mina$mina-legal/builds/2010-02-20_18-57-33/archive/org.apache.mina/mina-legal/2.0.0-RC2-SNAPSHOT/mina-legal-2.0.0-RC2-SNAPSHOT.jar
[HUDSON] Archiving
http://hudson.zones.apache.org/hudson/job/MINA-trunk-jdk1.6-ubuntu/org.apache.mina$mina-legal/ws/target/mina-legal-2.0.0-RC2-SNAPSHOT-sources.jar
to
/export/home/hudson/hudson/jobs/MINA-trunk-jdk1.6-ubuntu/modules/org.apache.mina$mina-legal/builds/2010-02-20_18-57-33/archive/org.apache.mina/mina-legal/2.0.0-RC2-SNAPSHOT/mina-legal-2.0.0-RC2-SNAPSHOT-sources.jar
[INFO]
[ERROR] BUILD ERROR
[INFO]
[INFO] Error installing artifact: Failed to copy full contents from
http://hudson.zones.apache.org/hudson/job/MINA-trunk-jdk1.6-ubuntu/org.apache.mina$mina-legal/ws/target/mina-legal-2.0.0-RC2-SNAPSHOT.jar
to
/home/hudson/.m2/repository/org/apache/mina/mina-legal/2.0.0-RC2-SNAPSHOT/mina-legal-2.0.0-RC2-SNAPSHOT.jar
[INFO]
[INFO] For more information, run Maven with the -e switch
[INFO]
[INFO] Total time: 14 seconds
[INFO] Finished at: Sat Feb 20 18:57:59 UTC 2010
[INFO] Final Memory: 30M/216M
[INFO]
Waiting for Hudson to finish collecting data
Hudson build is still unstable: MINA-tru nk-jdk1.6-windows » Apache MINA Core #13
See http://hudson.zones.apache.org/hudson/job/MINA-trunk-jdk1.6-windows/org.apache.mina$mina-core/changes
Hudson build became unstable: MINA-trunk-jdk1 .6-windows » Apache MINA XBean Integration #13
See http://hudson.zones.apache.org/hudson/job/MINA-trunk-jdk1.6-windows/org.apache.mina$mina-integration-xbean/13/
Hudson build is still unstable: MINA-trunk-jdk1.6-windows #13
See http://hudson.zones.apache.org/hudson/job/MINA-trunk-jdk1.6-windows/changes
Hudson build is back to normal : ftpserver-trunk-jd k1.5-solaris » FtpServer OSGi Spring-DM example #336
See http://hudson.zones.apache.org/hudson/job/ftpserver-trunk-jdk1.5-solaris/org.apache.ftpserver.examples$ftpserver-osgi-spring-service/336/
Hudson build is back to normal : ftpserver-trunk-jdk1. 5-solaris » FtpServer OSGi Ftplet service example #336
See http://hudson.zones.apache.org/hudson/job/ftpserver-trunk-jdk1.5-solaris/org.apache.ftpserver.examples$ftpserver-osgi-ftplet-service/336/
Hudson build is back to normal : ftpserver-trunk -jdk1.5-solaris » Apache FtpServer Examples #336
See http://hudson.zones.apache.org/hudson/job/ftpserver-trunk-jdk1.5-solaris/org.apache.ftpserver$ftpserver-examples/336/
Hudson build is back to normal : ftpserver-tru nk-jdk1.5-solaris » Apache FtpServer Core #336
See http://hudson.zones.apache.org/hudson/job/ftpserver-trunk-jdk1.5-solaris/org.apache.ftpserver$ftpserver-core/336/
Hudson build is back to normal : ftpserver-trun k-jdk1.5-solaris » Apache FtpServer Parent #336
See http://hudson.zones.apache.org/hudson/job/ftpserver-trunk-jdk1.5-solaris/org.apache.ftpserver$ftpserver-parent/336/
Hudson build is back to normal : ftpserver-t runk-jdk1.5-solaris » Apache Ftplet API #336
See http://hudson.zones.apache.org/hudson/job/ftpserver-trunk-jdk1.5-solaris/org.apache.ftpserver$ftplet-api/336/
Hudson strat to p*ss me off ...
Hi guys, is there any reason why this stupid Hudson *always* fail when we commit some new code, and one hour later come back with again 6 useless mails telling us that Hey, sorry, I was totally f*cked up last time I sent you 6 mails ? If Hudson can't inform us when there are *real* errors, at some point, we will simply ignore the alerts and lose all interest of having Hudson as a safety belt ! Sounds to me like Peter and the wolf... -- Regards, Cordialement, Emmanuel Lécharny www.nextury.com
Re: Hudson strat to p*ss me off ...
It may be worth looking into whether we can use Bamboo which is much better at CI. Its an atlassian project and I think it falls under the usual free open source license. Jeff On Feb 20, 2010, at 2:59 PM, Emmanuel Lecharny wrote: Hi guys, is there any reason why this stupid Hudson *always* fail when we commit some new code, and one hour later come back with again 6 useless mails telling us that Hey, sorry, I was totally f*cked up last time I sent you 6 mails ? If Hudson can't inform us when there are *real* errors, at some point, we will simply ignore the alerts and lose all interest of having Hudson as a safety belt ! Sounds to me like Peter and the wolf... -- Regards, Cordialement, Emmanuel Lécharny www.nextury.com
Hudson build is still unstable: MINA-tru nk-jdk1.6-windows » Apache MINA Core #14
See http://hudson.zones.apache.org/hudson/job/MINA-trunk-jdk1.6-windows/org.apache.mina$mina-core/changes
Hudson build is back to stable : MINA-trunk-jdk 1.6-windows » Apache MINA XBean Integration #14
See http://hudson.zones.apache.org/hudson/job/MINA-trunk-jdk1.6-windows/org.apache.mina$mina-integration-xbean/14/
Hudson build is still unstable: MINA-trunk-jdk1.6-windows #14
See http://hudson.zones.apache.org/hudson/job/MINA-trunk-jdk1.6-windows/changes
Hudson build is back to normal : MINA-tru nk-jdk1.6-ubuntu » Apache MINA Legal #227
See http://hudson.zones.apache.org/hudson/job/MINA-trunk-jdk1.6-ubuntu/org.apache.mina$mina-legal/227/
Re: sshd 0.3.0 performance regression
On Tue, Dec 15, 2009 at 05:05, Shawn Pearce s...@google.com wrote: Shawn Pearce s...@google.com wrote: We're seeing a performance regression in SSHD 0.3.0, the throughput is about 1/2 of what it was in SSHD 0.2.0. Bisecting the problem in git led me to revision 882491, which was the bug fix for https://issues.apache.org/jira/browse/SSHD-49. I committed a few changes that actually improve this situation considerably, we're now back to where we were with 0.2.0, and even slightly faster as our write buffer allocation is now better sized to meet packet writes. Thx! Guillaume, I really didn't follow how the delta in 882491 fixed SSHD-49's problem. You changed a bunch of stuff there all at once. But it seems to me the root cause was using putBytes() and not putRawBytes(), so the split packets were corrupted with an extra 4 bytes of data due to the unnecessary length being inserted. IIRC, there were two problems with the test case for SSHD-49. The first one was the use of putBytes instead of putRawBytes, but there was also a problem with the buffering strategy, where in some cases (a very big message from server to client), the server was waiting forever for some space that would never come. The problem has been fixed in the following commit http://svn.apache.org/viewvc?view=revisionrevision=835419 where I've added a new method to the Window object to make sure the server was not trying to send too much data. The real problem is that the loop / send / wait logic was done inside the write() method but was not taking into account all the parameters (the current buffer size for example). So the above commit actually move the logic into the flush method. The next commit (http://svn.apache.org/viewvc/mina/sshd/trunk/sshd-core/src/main/java/org/apache/sshd/common/channel/ChannelOutputStream.java?r1=882491r2=882490pathrev=882491) fixes another problem I found during testing. I thought, given the loop was actually moved to the flush() method that we did not really need it in the write() method. What if we remove the loop in the write() method and simply call flush() after having filled the buffer ? The other changes related to buffering everything on the stream until the application calls flush() seem to me to be completely unrelated to solving SSHD-49, and actually caused problems for Commands that write a lot of data and don't ever call flush(). I'm still a bit unhappy with throughput. I guess I need to dig into SSHD-54 here. The throughput still isn't as fast as we'd like it to be, especially over LAN connections. -- Cheers, Guillaume Nodet Blog: http://gnodet.blogspot.com/ Open Source SOA http://fusesource.com
Re: Hudson strat to p*ss me off ...
I got Hudson running @work without a trouble for a year. Look like ASF one is overload or something like that.. If infra got bamboo or continuum or anything else installed in a stable way I'm +1 for moving. Le Sat, 20 Feb 2010 15:04:30 -0700, Jeff Genender jgenen...@apache.org a écrit : It may be worth looking into whether we can use Bamboo which is much better at CI. Its an atlassian project and I think it falls under the usual free open source license. Jeff On Feb 20, 2010, at 2:59 PM, Emmanuel Lecharny wrote: Hi guys, is there any reason why this stupid Hudson *always* fail when we commit some new code, and one hour later come back with again 6 useless mails telling us that Hey, sorry, I was totally f*cked up last time I sent you 6 mails ? If Hudson can't inform us when there are *real* errors, at some point, we will simply ignore the alerts and lose all interest of having Hudson as a safety belt ! Sounds to me like Peter and the wolf... -- Regards, Cordialement, Emmanuel Lécharny www.nextury.com -- Julien Vermillard Archean Technologies http://www.archean.fr signature.asc Description: PGP signature
