Re: [MINA 3.0] filter chains
Hi, Because a filterchain can be shared across different sessions and threads, so you need to pass the local chain index because you can store it locally in the filter chain. Perhaps there is something smarter to do, because it's the dark point of this API. Julien On Sat, Aug 20, 2011 at 12:41 AM, Alan D. Cabrera l...@toolazydogs.com wrote: Why do we pass the current position? We also seem to pass it twice in the method and the controller. Regards, Alan On Aug 19, 2011, at 1:10 PM, Julien Vermillard wrote: Ok I committed the modification, for passing a chain controller to the filter for delegating the call to next filter. First I did it only for read write chaining because the other events (created,open,closed,idle) are fine like they are. They don't need to block an event or send it multiple time to the following filter. Here the modified IoFilter, some controller interface are passed to read write events : http://svn.apache.org/repos/asf/mina/branches/3.0/core/src/main/java/org/apache/mina/api/IoFilter.java Here sample implementation of LoggingFilter : http://svn.apache.org/repos/asf/mina/branches/3.0/core/src/main/java/org/apache/mina/filter/logging/LoggingFilter.java Here the FilterChain implementation, still simple : http://svn.apache.org/repos/asf/mina/branches/3.0/core/src/main/java/org/apache/mina/filterchain/DefaultIoFilterChain.java Now I need to figure how to remove the current position argument of the filter call. Julien On Fri, Aug 19, 2011 at 2:46 PM, Julien Vermillard jvermill...@gmail.com wrote: I half implemented the controller idea, it's looking like working, I'll finish that during the weekend or next week and commit it for review. Julien On Fri, Aug 19, 2011 at 2:39 PM, Steve Ulrich steve.ulr...@proemion.com wrote: Hi! Besides the points you mentioned, there are some other flaws: 1) How do you handle post-message-forwarding logic? 2) How do you handle filters that transparently push messages to the underlying filters (e.g. keep alive)? So the filters should decide about what to do, the chain about the where. regards Steve There could be specific (empty, single-result, multi-result) implementations that can be extended as needed. Julien Vermillard [mailto:jvermill...@gmail.com] wrote: Hi, I implemented some really simple chain for MINA 3 based on a list of chain processed by a filter chain. The implementation is very simple, sounded like a good idea : http://svn.apache.org/repos/asf/mina/branches/3.0/core/src/main/java/or g/apache/mina/filterchain/DefaultIoFilterChain.java But when i started implementing an HTTP codec I started to see the real design issues. The problem is when you need to produce more than one object during a filter processing, like when you find more than one PDU in a ByteBuffer. We could change IoFilter method : Object messageReceived(IoSession session, Object message); To : ListObject messageReceived(IoSession session, Object message); But starting to use collection here sound like an overkill and not really a nice idea if we want to keep the memory usage low enough. So I'm scraping the current implementation, I'm going to try something else suggested by Emmanuel : When a filter want to call the next filter, he ask it to the filter controller (aka the FilterChain). Let's see if it's going somewhere this time ;) Julien
Re: [MINA 3.0] filter chains
Here another idea (not really sure it's a good one) http://nopaste.info/0c43874851.html It would be regrouping session created/open/close in one IoFilter method, with a SessionEvent are a parameter. I doesn't change much the inner logic, but perhaps it can make impelemention of IoFilter less verbose. WDYT ? Julien On Sat, Aug 20, 2011 at 8:49 AM, Julien Vermillard jvermill...@gmail.com wrote: Hi, Because a filterchain can be shared across different sessions and threads, so you need to pass the local chain index because you can store it locally in the filter chain. Perhaps there is something smarter to do, because it's the dark point of this API. Julien On Sat, Aug 20, 2011 at 12:41 AM, Alan D. Cabrera l...@toolazydogs.com wrote: Why do we pass the current position? We also seem to pass it twice in the method and the controller. Regards, Alan On Aug 19, 2011, at 1:10 PM, Julien Vermillard wrote: Ok I committed the modification, for passing a chain controller to the filter for delegating the call to next filter. First I did it only for read write chaining because the other events (created,open,closed,idle) are fine like they are. They don't need to block an event or send it multiple time to the following filter. Here the modified IoFilter, some controller interface are passed to read write events : http://svn.apache.org/repos/asf/mina/branches/3.0/core/src/main/java/org/apache/mina/api/IoFilter.java Here sample implementation of LoggingFilter : http://svn.apache.org/repos/asf/mina/branches/3.0/core/src/main/java/org/apache/mina/filter/logging/LoggingFilter.java Here the FilterChain implementation, still simple : http://svn.apache.org/repos/asf/mina/branches/3.0/core/src/main/java/org/apache/mina/filterchain/DefaultIoFilterChain.java Now I need to figure how to remove the current position argument of the filter call. Julien On Fri, Aug 19, 2011 at 2:46 PM, Julien Vermillard jvermill...@gmail.com wrote: I half implemented the controller idea, it's looking like working, I'll finish that during the weekend or next week and commit it for review. Julien On Fri, Aug 19, 2011 at 2:39 PM, Steve Ulrich steve.ulr...@proemion.com wrote: Hi! Besides the points you mentioned, there are some other flaws: 1) How do you handle post-message-forwarding logic? 2) How do you handle filters that transparently push messages to the underlying filters (e.g. keep alive)? So the filters should decide about what to do, the chain about the where. regards Steve There could be specific (empty, single-result, multi-result) implementations that can be extended as needed. Julien Vermillard [mailto:jvermill...@gmail.com] wrote: Hi, I implemented some really simple chain for MINA 3 based on a list of chain processed by a filter chain. The implementation is very simple, sounded like a good idea : http://svn.apache.org/repos/asf/mina/branches/3.0/core/src/main/java/or g/apache/mina/filterchain/DefaultIoFilterChain.java But when i started implementing an HTTP codec I started to see the real design issues. The problem is when you need to produce more than one object during a filter processing, like when you find more than one PDU in a ByteBuffer. We could change IoFilter method : Object messageReceived(IoSession session, Object message); To : ListObject messageReceived(IoSession session, Object message); But starting to use collection here sound like an overkill and not really a nice idea if we want to keep the memory usage low enough. So I'm scraping the current implementation, I'm going to try something else suggested by Emmanuel : When a filter want to call the next filter, he ask it to the filter controller (aka the FilterChain). Let's see if it's going somewhere this time ;) Julien
Re: [MINA 3.0] filter chains
On 8/20/11 8:49 AM, Julien Vermillard wrote: Hi, Because a filterchain can be shared across different sessions and threads, so you need to pass the local chain index because you can store it locally in the filter chain. Perhaps there is something smarter to do, because it's the dark point of this API. The filter chain is copied into the session (at least, it was what was done for MINA 2). Assuming that two different sessions might use two different chains, and assumng that the chain might be dynamically changed, it makes sense to do this copy. Now, if we can split the session (using an executor for instance), then the chain must be copied. It would make sense to store the chain into a ThreadLocal variable instead of storing it into a session object. Julien On Sat, Aug 20, 2011 at 12:41 AM, Alan D. Cabreral...@toolazydogs.com wrote: Why do we pass the current position? We also seem to pass it twice in the method and the controller. Regards, Alan On Aug 19, 2011, at 1:10 PM, Julien Vermillard wrote: Ok I committed the modification, for passing a chain controller to the filter for delegating the call to next filter. First I did it only for read write chaining because the other events (created,open,closed,idle) are fine like they are. They don't need to block an event or send it multiple time to the following filter. Here the modified IoFilter, some controller interface are passed to read write events : http://svn.apache.org/repos/asf/mina/branches/3.0/core/src/main/java/org/apache/mina/api/IoFilter.java Here sample implementation of LoggingFilter : http://svn.apache.org/repos/asf/mina/branches/3.0/core/src/main/java/org/apache/mina/filter/logging/LoggingFilter.java Here the FilterChain implementation, still simple : http://svn.apache.org/repos/asf/mina/branches/3.0/core/src/main/java/org/apache/mina/filterchain/DefaultIoFilterChain.java Now I need to figure how to remove the current position argument of the filter call. Julien On Fri, Aug 19, 2011 at 2:46 PM, Julien Vermillard jvermill...@gmail.com wrote: I half implemented the controller idea, it's looking like working, I'll finish that during the weekend or next week and commit it for review. Julien On Fri, Aug 19, 2011 at 2:39 PM, Steve Ulrichsteve.ulr...@proemion.com wrote: Hi! Besides the points you mentioned, there are some other flaws: 1) How do you handle post-message-forwarding logic? 2) How do you handle filters that transparently push messages to the underlying filters (e.g. keep alive)? So the filters should decide about what to do, the chain about the where. regards Steve There could be specific (empty, single-result, multi-result) implementations that can be extended as needed. Julien Vermillard [mailto:jvermill...@gmail.com] wrote: Hi, I implemented some really simple chain for MINA 3 based on a list of chain processed by a filter chain. The implementation is very simple, sounded like a good idea : http://svn.apache.org/repos/asf/mina/branches/3.0/core/src/main/java/or g/apache/mina/filterchain/DefaultIoFilterChain.java But when i started implementing an HTTP codec I started to see the real design issues. The problem is when you need to produce more than one object during a filter processing, like when you find more than one PDU in a ByteBuffer. We could change IoFilter method : Object messageReceived(IoSession session, Object message); To : ListObject messageReceived(IoSession session, Object message); But starting to use collection here sound like an overkill and not really a nice idea if we want to keep the memory usage low enough. So I'm scraping the current implementation, I'm going to try something else suggested by Emmanuel : When a filter want to call the next filter, he ask it to the filter controller (aka the FilterChain). Let's see if it's going somewhere this time ;) Julien -- Regards, Cordialement, Emmanuel Lécharny www.iktek.com
Re: [MINA 3.0] filter chains
Thanks,got it, I Will try to implement that tomorrow. -- Julien Vermillard Le 20 août 2011 à 09:30, Emmanuel Lecharny elecha...@gmail.com a écrit : On 8/20/11 8:49 AM, Julien Vermillard wrote: Hi, Because a filterchain can be shared across different sessions and threads, so you need to pass the local chain index because you can store it locally in the filter chain. Perhaps there is something smarter to do, because it's the dark point of this API. The filter chain is copied into the session (at least, it was what was done for MINA 2). Assuming that two different sessions might use two different chains, and assumng that the chain might be dynamically changed, it makes sense to do this copy. Now, if we can split the session (using an executor for instance), then the chain must be copied. It would make sense to store the chain into a ThreadLocal variable instead of storing it into a session object. Julien On Sat, Aug 20, 2011 at 12:41 AM, Alan D. Cabreral...@toolazydogs.com wrote: Why do we pass the current position? We also seem to pass it twice in the method and the controller. Regards, Alan On Aug 19, 2011, at 1:10 PM, Julien Vermillard wrote: Ok I committed the modification, for passing a chain controller to the filter for delegating the call to next filter. First I did it only for read write chaining because the other events (created,open,closed,idle) are fine like they are. They don't need to block an event or send it multiple time to the following filter. Here the modified IoFilter, some controller interface are passed to read write events : http://svn.apache.org/repos/asf/mina/branches/3.0/core/src/main/java/org/apache/mina/api/IoFilter.java Here sample implementation of LoggingFilter : http://svn.apache.org/repos/asf/mina/branches/3.0/core/src/main/java/org/apache/mina/filter/logging/LoggingFilter.java Here the FilterChain implementation, still simple : http://svn.apache.org/repos/asf/mina/branches/3.0/core/src/main/java/org/apache/mina/filterchain/DefaultIoFilterChain.java Now I need to figure how to remove the current position argument of the filter call. Julien On Fri, Aug 19, 2011 at 2:46 PM, Julien Vermillard jvermill...@gmail.com wrote: I half implemented the controller idea, it's looking like working, I'll finish that during the weekend or next week and commit it for review. Julien On Fri, Aug 19, 2011 at 2:39 PM, Steve Ulrichsteve.ulr...@proemion.com wrote: Hi! Besides the points you mentioned, there are some other flaws: 1) How do you handle post-message-forwarding logic? 2) How do you handle filters that transparently push messages to the underlying filters (e.g. keep alive)? So the filters should decide about what to do, the chain about the where. regards Steve There could be specific (empty, single-result, multi-result) implementations that can be extended as needed. Julien Vermillard [mailto:jvermill...@gmail.com] wrote: Hi, I implemented some really simple chain for MINA 3 based on a list of chain processed by a filter chain. The implementation is very simple, sounded like a good idea : http://svn.apache.org/repos/asf/mina/branches/3.0/core/src/main/java/or g/apache/mina/filterchain/DefaultIoFilterChain.java But when i started implementing an HTTP codec I started to see the real design issues. The problem is when you need to produce more than one object during a filter processing, like when you find more than one PDU in a ByteBuffer. We could change IoFilter method : Object messageReceived(IoSession session, Object message); To : ListObject messageReceived(IoSession session, Object message); But starting to use collection here sound like an overkill and not really a nice idea if we want to keep the memory usage low enough. So I'm scraping the current implementation, I'm going to try something else suggested by Emmanuel : When a filter want to call the next filter, he ask it to the filter controller (aka the FilterChain). Let's see if it's going somewhere this time ;) Julien -- Regards, Cordialement, Emmanuel Lécharny www.iktek.com
Re: IoBuffers of bytebuffers
On Fri, Aug 19, 2011 at 3:19 PM, Emmanuel Lecharny elecha...@gmail.comwrote: On 8/19/11 2:16 PM, Alan D. Cabrera wrote: I'm wondering. Do you guys think it's a good idea? It seems to make things pretty complicated and adds another dimension to groking the behavior of your service. I'm not sure that it's necessary. Definitively a bad idea. What we need is an abstraction on top of an Array of ByteBuffer (the Java NIO class), which extends the size by adding new ByteBuffer on the fly. Exactly! +1 Perhaps it might be best to have a linked list of byte buffers internally in this MinaBuffer kind a thingy? The array must behave exactly as the ByteBuffer. +1 I wrote such a class 2 years ago, but I can't find the code. Will look again on my USB keys this week-end. -- Best Regards, -- Alex
Re: IoBuffers of bytebuffers
On Sat, Aug 20, 2011 at 2:46 AM, Emmanuel Lecharny elecha...@gmail.comwrote: On 8/19/11 2:26 PM, Alan D. Cabrera wrote: On Aug 19, 2011, at 5:19 AM, Emmanuel Lecharny wrote: On 8/19/11 2:16 PM, Alan D. Cabrera wrote: I'm wondering. Do you guys think it's a good idea? It seems to make things pretty complicated and adds another dimension to groking the behavior of your service. I'm not sure that it's necessary. Definitively a bad idea. What we need is an abstraction on top of an Array of ByteBuffer (the Java NIO class), which extends the size by adding new ByteBuffer on the fly. The array must behave exactly as the ByteBuffer. I wrote such a class 2 years ago, but I can't find the code. Will look again on my USB keys this week-end. So, what is the scenario that we're trying to support? I imagine appending headers to binary data would be one. In this case is an Array of ByteBuffers really needed? Why not just send down one ByteBuffer for the header and another for the body that was sent to you? Julien and Steve laready responded with clear examples where expandable BB are necessary. I do think this is a common case, assuming you can perfectly receive the data byte by byte, and you want to store those bytes in a place which does not need to be reallocated every time. The current IoBuffer, when full, is copied in a new loBuffer which size is doubled. If you are expecting big PDUs, you might allocate huge Iobuffer for nothing, wasting space. Plus you have to copy all the old IoBuffer into the new one. Not really a good thing. *if* we are using DirectBuffer, we even might want to use fixed size IoBuffer (say 1k), and store them in a queue of avilable free buffers for reuse, sparing the cost of allocating them (to be validated, I'm not sure that the cost of managing concurrent access to this queue does not overweight the cost of allocating a new direct buffer) For HeapBuffers, i'm quite sure that we should allocate new ByteBuffer with the exact size of the received data (if it's 10 bytes, then the new ByteBuffer will be 10 byte slong. If it's bigger, then fine). This additional ByteBuffer can be appended a the end of the ByteBuffer array, and the filter will be able to process the list when it has everything needed. Julien mentionned the Cumulative decoder, which can be used in may cases : - we are waiting for a \n (HTTP protocol) - we are waiting for a closing tag (XML) - we are dealing with LV (Length/value) PDU, and the received value is not fully received (we are expecting Length bytes) - we are dealing with fixed size PDU One more thing to consider : at some point, we may want to flush the ByteBuffer to disk, if we are waiting for a huge PDU, to avoid sucking all the memory : this is typically the case in LDAP when receiving a JpegPhot attribute, inside a PDU (we don't know that it's a JpegPhoto attribute until we have read and processed teh full PDU). With a proxy on top of the ByteBuffer, we can eventually hide the fact that the bytes are on disk. Although these PDU's are large you can store arbitrary data as well, who knows what kinds of data and what scenarios users will come up with. It would be very nice if MINA allowed some way behind the scenes to stream large PDU's to disk and transparently allow upstream consumers to read from disk as if it were coming directly from the wire. However this might be something added on top of the framework WDYT? -- Best Regards, -- Alex
Re : [MINA 3.0] filter chains
We shouldn't systematically copy the chain in the session as imho it's not so usual to dynamically add filters in one particular session (the first case that comes to my mind is dynamically protecting the session with ssl but it requires a new connection most of the time no ? maybe some dynamic compression or some negociation mechanism that will need a particular filter wrapping and unwrapping messages) A copy on write strategy would improve memory footprint. So when would we copy the chain ? when some change happens : add/remove filter a chain controller could associate to the session it's alternative chain or use the default one. So now what about threads coming into play ? using threadlocal seems a good idea Em So to resume - copy when necessary - custom chain should be searched in the following places : threadlocal - session - default chain of the controller ChainController.getChain() will hide this whole complexity providing flexibility and efficiency wdyt ? Cordialement, Regards, -Edouard De Oliveira- De : Emmanuel Lecharny elecha...@gmail.com À : dev@mina.apache.org Envoyé le : Samedi 20 Août 2011 9h29 Objet : Re: [MINA 3.0] filter chains On 8/20/11 8:49 AM, Julien Vermillard wrote: Hi, Because a filterchain can be shared across different sessions and threads, so you need to pass the local chain index because you can store it locally in the filter chain. Perhaps there is something smarter to do, because it's the dark point of this API. The filter chain is copied into the session (at least, it was what was done for MINA 2). Assuming that two different sessions might use two different chains, and assumng that the chain might be dynamically changed, it makes sense to do this copy. Now, if we can split the session (using an executor for instance), then the chain must be copied. It would make sense to store the chain into a ThreadLocal variable instead of storing it into a session object. Julien On Sat, Aug 20, 2011 at 12:41 AM, Alan D. Cabreral...@toolazydogs.com wrote: Why do we pass the current position? We also seem to pass it twice in the method and the controller. Regards, Alan On Aug 19, 2011, at 1:10 PM, Julien Vermillard wrote: Ok I committed the modification, for passing a chain controller to the filter for delegating the call to next filter. First I did it only for read write chaining because the other events (created,open,closed,idle) are fine like they are. They don't need to block an event or send it multiple time to the following filter. Here the modified IoFilter, some controller interface are passed to read write events : http://svn.apache.org/repos/asf/mina/branches/3.0/core/src/main/java/org/apache/mina/api/IoFilter.java Here sample implementation of LoggingFilter : http://svn.apache.org/repos/asf/mina/branches/3.0/core/src/main/java/org/apache/mina/filter/logging/LoggingFilter.java Here the FilterChain implementation, still simple : http://svn.apache.org/repos/asf/mina/branches/3.0/core/src/main/java/org/apache/mina/filterchain/DefaultIoFilterChain.java Now I need to figure how to remove the current position argument of the filter call. Julien On Fri, Aug 19, 2011 at 2:46 PM, Julien Vermillard jvermill...@gmail.com wrote: I half implemented the controller idea, it's looking like working, I'll finish that during the weekend or next week and commit it for review. Julien On Fri, Aug 19, 2011 at 2:39 PM, Steve Ulrichsteve.ulr...@proemion.com wrote: Hi! Besides the points you mentioned, there are some other flaws: 1) How do you handle post-message-forwarding logic? 2) How do you handle filters that transparently push messages to the underlying filters (e.g. keep alive)? So the filters should decide about what to do, the chain about the where. regards Steve There could be specific (empty, single-result, multi-result) implementations that can be extended as needed. Julien Vermillard [mailto:jvermill...@gmail.com] wrote: Hi, I implemented some really simple chain for MINA 3 based on a list of chain processed by a filter chain. The implementation is very simple, sounded like a good idea : http://svn.apache.org/repos/asf/mina/branches/3.0/core/src/main/java/or g/apache/mina/filterchain/DefaultIoFilterChain.java But when i started implementing an HTTP codec I started to see the real design issues. The problem is when you need to produce more than one object during a filter processing, like when you find more than one PDU in a ByteBuffer. We could change IoFilter method : Object messageReceived(IoSession session, Object message); To : ListObject messageReceived(IoSession session, Object message); But starting to use collection here sound like an overkill and not really a nice idea if we want to keep the memory usage low enough. So I'm scraping the current implementation, I'm going to try something else suggested by Emmanuel :
Re: IoBuffers of bytebuffers
On 8/20/11 12:59 PM, Alex Karasulu wrote: On Sat, Aug 20, 2011 at 2:46 AM, Emmanuel Lecharnyelecha...@gmail.comwrote: On 8/19/11 2:26 PM, Alan D. Cabrera wrote: On Aug 19, 2011, at 5:19 AM, Emmanuel Lecharny wrote: On 8/19/11 2:16 PM, Alan D. Cabrera wrote: I'm wondering. Do you guys think it's a good idea? It seems to make things pretty complicated and adds another dimension to groking the behavior of your service. I'm not sure that it's necessary. Definitively a bad idea. What we need is an abstraction on top of an Array of ByteBuffer (the Java NIO class), which extends the size by adding new ByteBuffer on the fly. The array must behave exactly as the ByteBuffer. I wrote such a class 2 years ago, but I can't find the code. Will look again on my USB keys this week-end. So, what is the scenario that we're trying to support? I imagine appending headers to binary data would be one. In this case is an Array of ByteBuffers really needed? Why not just send down one ByteBuffer for the header and another for the body that was sent to you? Julien and Steve laready responded with clear examples where expandable BB are necessary. I do think this is a common case, assuming you can perfectly receive the data byte by byte, and you want to store those bytes in a place which does not need to be reallocated every time. The current IoBuffer, when full, is copied in a new loBuffer which size is doubled. If you are expecting big PDUs, you might allocate huge Iobuffer for nothing, wasting space. Plus you have to copy all the old IoBuffer into the new one. Not really a good thing. *if* we are using DirectBuffer, we even might want to use fixed size IoBuffer (say 1k), and store them in a queue of avilable free buffers for reuse, sparing the cost of allocating them (to be validated, I'm not sure that the cost of managing concurrent access to this queue does not overweight the cost of allocating a new direct buffer) For HeapBuffers, i'm quite sure that we should allocate new ByteBuffer with the exact size of the received data (if it's 10 bytes, then the new ByteBuffer will be 10 byte slong. If it's bigger, then fine). This additional ByteBuffer can be appended a the end of the ByteBuffer array, and the filter will be able to process the list when it has everything needed. Julien mentionned the Cumulative decoder, which can be used in may cases : - we are waiting for a \n (HTTP protocol) - we are waiting for a closing tag (XML) - we are dealing with LV (Length/value) PDU, and the received value is not fully received (we are expecting Length bytes) - we are dealing with fixed size PDU One more thing to consider : at some point, we may want to flush the ByteBuffer to disk, if we are waiting for a huge PDU, to avoid sucking all the memory : this is typically the case in LDAP when receiving a JpegPhot attribute, inside a PDU (we don't know that it's a JpegPhoto attribute until we have read and processed teh full PDU). With a proxy on top of the ByteBuffer, we can eventually hide the fact that the bytes are on disk. Although these PDU's are large you can store arbitrary data as well, who knows what kinds of data and what scenarios users will come up with. It would be very nice if MINA allowed some way behind the scenes to stream large PDU's to disk and transparently allow upstream consumers to read from disk as if it were coming directly from the wire. However this might be something added on top of the framework WDYT? This is what I have in mind. It should be transparent to the user. -- Regards, Cordialement, Emmanuel Lécharny www.iktek.com
Re: IoBuffers of bytebuffers
On 8/20/11 12:50 PM, Alex Karasulu wrote: On Fri, Aug 19, 2011 at 3:19 PM, Emmanuel Lecharnyelecha...@gmail.comwrote: On 8/19/11 2:16 PM, Alan D. Cabrera wrote: I'm wondering. Do you guys think it's a good idea? It seems to make things pretty complicated and adds another dimension to groking the behavior of your service. I'm not sure that it's necessary. Definitively a bad idea. What we need is an abstraction on top of an Array of ByteBuffer (the Java NIO class), which extends the size by adding new ByteBuffer on the fly. Exactly! +1 Perhaps it might be best to have a linked list of byte buffers internally in this MinaBuffer kind a thingy? Yes. -- Regards, Cordialement, Emmanuel Lécharny www.iktek.com
Re: [MINA 3.0] filter chains
Yeah, I smell implementation leaking into API. I, as a filter in a chain, should not care what my index is in the grand scheme of things. Regards, Alan On Aug 19, 2011, at 11:49 PM, Julien Vermillard wrote: Hi, Because a filterchain can be shared across different sessions and threads, so you need to pass the local chain index because you can store it locally in the filter chain. Perhaps there is something smarter to do, because it's the dark point of this API. Julien On Sat, Aug 20, 2011 at 12:41 AM, Alan D. Cabrera l...@toolazydogs.com wrote: Why do we pass the current position? We also seem to pass it twice in the method and the controller. Regards, Alan On Aug 19, 2011, at 1:10 PM, Julien Vermillard wrote: Ok I committed the modification, for passing a chain controller to the filter for delegating the call to next filter. First I did it only for read write chaining because the other events (created,open,closed,idle) are fine like they are. They don't need to block an event or send it multiple time to the following filter. Here the modified IoFilter, some controller interface are passed to read write events : http://svn.apache.org/repos/asf/mina/branches/3.0/core/src/main/java/org/apache/mina/api/IoFilter.java Here sample implementation of LoggingFilter : http://svn.apache.org/repos/asf/mina/branches/3.0/core/src/main/java/org/apache/mina/filter/logging/LoggingFilter.java Here the FilterChain implementation, still simple : http://svn.apache.org/repos/asf/mina/branches/3.0/core/src/main/java/org/apache/mina/filterchain/DefaultIoFilterChain.java Now I need to figure how to remove the current position argument of the filter call. Julien On Fri, Aug 19, 2011 at 2:46 PM, Julien Vermillard jvermill...@gmail.com wrote: I half implemented the controller idea, it's looking like working, I'll finish that during the weekend or next week and commit it for review. Julien On Fri, Aug 19, 2011 at 2:39 PM, Steve Ulrich steve.ulr...@proemion.com wrote: Hi! Besides the points you mentioned, there are some other flaws: 1) How do you handle post-message-forwarding logic? 2) How do you handle filters that transparently push messages to the underlying filters (e.g. keep alive)? So the filters should decide about what to do, the chain about the where. regards Steve There could be specific (empty, single-result, multi-result) implementations that can be extended as needed. Julien Vermillard [mailto:jvermill...@gmail.com] wrote: Hi, I implemented some really simple chain for MINA 3 based on a list of chain processed by a filter chain. The implementation is very simple, sounded like a good idea : http://svn.apache.org/repos/asf/mina/branches/3.0/core/src/main/java/or g/apache/mina/filterchain/DefaultIoFilterChain.java But when i started implementing an HTTP codec I started to see the real design issues. The problem is when you need to produce more than one object during a filter processing, like when you find more than one PDU in a ByteBuffer. We could change IoFilter method : Object messageReceived(IoSession session, Object message); To : ListObject messageReceived(IoSession session, Object message); But starting to use collection here sound like an overkill and not really a nice idea if we want to keep the memory usage low enough. So I'm scraping the current implementation, I'm going to try something else suggested by Emmanuel : When a filter want to call the next filter, he ask it to the filter controller (aka the FilterChain). Let's see if it's going somewhere this time ;) Julien
Re: [MINA 3.0] filter chains
I guess one of the things that I'm truing to understand is why do we need that index? Why should that be exposed to the filter implementer? Regards, Alan On Aug 19, 2011, at 11:56 PM, Julien Vermillard wrote: Here another idea (not really sure it's a good one) http://nopaste.info/0c43874851.html It would be regrouping session created/open/close in one IoFilter method, with a SessionEvent are a parameter. I doesn't change much the inner logic, but perhaps it can make impelemention of IoFilter less verbose. WDYT ? Julien On Sat, Aug 20, 2011 at 8:49 AM, Julien Vermillard jvermill...@gmail.com wrote: Hi, Because a filterchain can be shared across different sessions and threads, so you need to pass the local chain index because you can store it locally in the filter chain. Perhaps there is something smarter to do, because it's the dark point of this API. Julien On Sat, Aug 20, 2011 at 12:41 AM, Alan D. Cabrera l...@toolazydogs.com wrote: Why do we pass the current position? We also seem to pass it twice in the method and the controller. Regards, Alan On Aug 19, 2011, at 1:10 PM, Julien Vermillard wrote: Ok I committed the modification, for passing a chain controller to the filter for delegating the call to next filter. First I did it only for read write chaining because the other events (created,open,closed,idle) are fine like they are. They don't need to block an event or send it multiple time to the following filter. Here the modified IoFilter, some controller interface are passed to read write events : http://svn.apache.org/repos/asf/mina/branches/3.0/core/src/main/java/org/apache/mina/api/IoFilter.java Here sample implementation of LoggingFilter : http://svn.apache.org/repos/asf/mina/branches/3.0/core/src/main/java/org/apache/mina/filter/logging/LoggingFilter.java Here the FilterChain implementation, still simple : http://svn.apache.org/repos/asf/mina/branches/3.0/core/src/main/java/org/apache/mina/filterchain/DefaultIoFilterChain.java Now I need to figure how to remove the current position argument of the filter call. Julien On Fri, Aug 19, 2011 at 2:46 PM, Julien Vermillard jvermill...@gmail.com wrote: I half implemented the controller idea, it's looking like working, I'll finish that during the weekend or next week and commit it for review. Julien On Fri, Aug 19, 2011 at 2:39 PM, Steve Ulrich steve.ulr...@proemion.com wrote: Hi! Besides the points you mentioned, there are some other flaws: 1) How do you handle post-message-forwarding logic? 2) How do you handle filters that transparently push messages to the underlying filters (e.g. keep alive)? So the filters should decide about what to do, the chain about the where. regards Steve There could be specific (empty, single-result, multi-result) implementations that can be extended as needed. Julien Vermillard [mailto:jvermill...@gmail.com] wrote: Hi, I implemented some really simple chain for MINA 3 based on a list of chain processed by a filter chain. The implementation is very simple, sounded like a good idea : http://svn.apache.org/repos/asf/mina/branches/3.0/core/src/main/java/or g/apache/mina/filterchain/DefaultIoFilterChain.java But when i started implementing an HTTP codec I started to see the real design issues. The problem is when you need to produce more than one object during a filter processing, like when you find more than one PDU in a ByteBuffer. We could change IoFilter method : Object messageReceived(IoSession session, Object message); To : ListObject messageReceived(IoSession session, Object message); But starting to use collection here sound like an overkill and not really a nice idea if we want to keep the memory usage low enough. So I'm scraping the current implementation, I'm going to try something else suggested by Emmanuel : When a filter want to call the next filter, he ask it to the filter controller (aka the FilterChain). Let's see if it's going somewhere this time ;) Julien
Re: [MINA 3.0] filter chains
On Aug 20, 2011, at 4:32 AM, Edouard De Oliveira wrote: We shouldn't systematically copy the chain in the session as imho it's not so usual to dynamically add filters in one particular session (the first case that comes to my mind is dynamically protecting the session with ssl but it requires a new connection most of the time no ? maybe some dynamic compression or some negociation mechanism that will need a particular filter wrapping and unwrapping messages) There's been a fair bit of discussion on this before, this being the need for dynamically modifying filter chains in a session that's already being used. It is my assertion that it is an anti-pattern that signals the need for a state machine. Getting a protocol right on both ends is very hard and dynamic chains make it even more difficult. APIs should promote good practices With that said, look how complicated the implementation gets below. Just my 2 cents. Regards, Alan A copy on write strategy would improve memory footprint. So when would we copy the chain ? when some change happens : add/remove filter a chain controller could associate to the session it's alternative chain or use the default one. So now what about threads coming into play ? using threadlocal seems a good idea Em So to resume - copy when necessary - custom chain should be searched in the following places : threadlocal - session - default chain of the controller ChainController.getChain() will hide this whole complexity providing flexibility and efficiency wdyt ? Cordialement, Regards, -Edouard De Oliveira- De : Emmanuel Lecharny elecha...@gmail.com À : dev@mina.apache.org Envoyé le : Samedi 20 Août 2011 9h29 Objet : Re: [MINA 3.0] filter chains On 8/20/11 8:49 AM, Julien Vermillard wrote: Hi, Because a filterchain can be shared across different sessions and threads, so you need to pass the local chain index because you can store it locally in the filter chain. Perhaps there is something smarter to do, because it's the dark point of this API. The filter chain is copied into the session (at least, it was what was done for MINA 2). Assuming that two different sessions might use two different chains, and assumng that the chain might be dynamically changed, it makes sense to do this copy. Now, if we can split the session (using an executor for instance), then the chain must be copied. It would make sense to store the chain into a ThreadLocal variable instead of storing it into a session object. Julien On Sat, Aug 20, 2011 at 12:41 AM, Alan D. Cabreral...@toolazydogs.com wrote: Why do we pass the current position? We also seem to pass it twice in the method and the controller. Regards, Alan On Aug 19, 2011, at 1:10 PM, Julien Vermillard wrote: Ok I committed the modification, for passing a chain controller to the filter for delegating the call to next filter. First I did it only for read write chaining because the other events (created,open,closed,idle) are fine like they are. They don't need to block an event or send it multiple time to the following filter. Here the modified IoFilter, some controller interface are passed to read write events : http://svn.apache.org/repos/asf/mina/branches/3.0/core/src/main/java/org/apache/mina/api/IoFilter.java Here sample implementation of LoggingFilter : http://svn.apache.org/repos/asf/mina/branches/3.0/core/src/main/java/org/apache/mina/filter/logging/LoggingFilter.java Here the FilterChain implementation, still simple : http://svn.apache.org/repos/asf/mina/branches/3.0/core/src/main/java/org/apache/mina/filterchain/DefaultIoFilterChain.java Now I need to figure how to remove the current position argument of the filter call. Julien On Fri, Aug 19, 2011 at 2:46 PM, Julien Vermillard jvermill...@gmail.com wrote: I half implemented the controller idea, it's looking like working, I'll finish that during the weekend or next week and commit it for review. Julien On Fri, Aug 19, 2011 at 2:39 PM, Steve Ulrichsteve.ulr...@proemion.com wrote: Hi! Besides the points you mentioned, there are some other flaws: 1) How do you handle post-message-forwarding logic? 2) How do you handle filters that transparently push messages to the underlying filters (e.g. keep alive)? So the filters should decide about what to do, the chain about the where. regards Steve There could be specific (empty, single-result, multi-result) implementations that can be extended as needed. Julien Vermillard [mailto:jvermill...@gmail.com] wrote: Hi, I implemented some really simple chain for MINA 3 based on a list of chain processed by a filter chain. The implementation is very simple, sounded like a good idea : http://svn.apache.org/repos/asf/mina/branches/3.0/core/src/main/java/or g/apache/mina/filterchain/DefaultIoFilterChain.java But when i started implementing an
Re: IoBuffers of bytebuffers
On Aug 20, 2011, at 3:59 AM, Alex Karasulu wrote: On Sat, Aug 20, 2011 at 2:46 AM, Emmanuel Lecharny elecha...@gmail.comwrote: On 8/19/11 2:26 PM, Alan D. Cabrera wrote: On Aug 19, 2011, at 5:19 AM, Emmanuel Lecharny wrote: On 8/19/11 2:16 PM, Alan D. Cabrera wrote: I'm wondering. Do you guys think it's a good idea? It seems to make things pretty complicated and adds another dimension to groking the behavior of your service. I'm not sure that it's necessary. Definitively a bad idea. What we need is an abstraction on top of an Array of ByteBuffer (the Java NIO class), which extends the size by adding new ByteBuffer on the fly. The array must behave exactly as the ByteBuffer. I wrote such a class 2 years ago, but I can't find the code. Will look again on my USB keys this week-end. So, what is the scenario that we're trying to support? I imagine appending headers to binary data would be one. In this case is an Array of ByteBuffers really needed? Why not just send down one ByteBuffer for the header and another for the body that was sent to you? Julien and Steve laready responded with clear examples where expandable BB are necessary. I do think this is a common case, assuming you can perfectly receive the data byte by byte, and you want to store those bytes in a place which does not need to be reallocated every time. The current IoBuffer, when full, is copied in a new loBuffer which size is doubled. If you are expecting big PDUs, you might allocate huge Iobuffer for nothing, wasting space. Plus you have to copy all the old IoBuffer into the new one. Not really a good thing. *if* we are using DirectBuffer, we even might want to use fixed size IoBuffer (say 1k), and store them in a queue of avilable free buffers for reuse, sparing the cost of allocating them (to be validated, I'm not sure that the cost of managing concurrent access to this queue does not overweight the cost of allocating a new direct buffer) For HeapBuffers, i'm quite sure that we should allocate new ByteBuffer with the exact size of the received data (if it's 10 bytes, then the new ByteBuffer will be 10 byte slong. If it's bigger, then fine). This additional ByteBuffer can be appended a the end of the ByteBuffer array, and the filter will be able to process the list when it has everything needed. Julien mentionned the Cumulative decoder, which can be used in may cases : - we are waiting for a \n (HTTP protocol) - we are waiting for a closing tag (XML) - we are dealing with LV (Length/value) PDU, and the received value is not fully received (we are expecting Length bytes) - we are dealing with fixed size PDU One more thing to consider : at some point, we may want to flush the ByteBuffer to disk, if we are waiting for a huge PDU, to avoid sucking all the memory : this is typically the case in LDAP when receiving a JpegPhot attribute, inside a PDU (we don't know that it's a JpegPhoto attribute until we have read and processed teh full PDU). With a proxy on top of the ByteBuffer, we can eventually hide the fact that the bytes are on disk. Although these PDU's are large you can store arbitrary data as well, who knows what kinds of data and what scenarios users will come up with. It would be very nice if MINA allowed some way behind the scenes to stream large PDU's to disk and transparently allow upstream consumers to read from disk as if it were coming directly from the wire. However this might be something added on top of the framework WDYT? This is my thinking as well. I see a need to batch together byte buffers to be more efficient when TCP_NO_DELAY has been set but the scenarios listed above can be implemented without a more complex byte buffer, i.e. the byte data should be translated to higher order data structures opportunistically. Regards, Alan
Re: [MINA 3.0] filter chains
On Aug 19, 2011, at 4:53 PM, Emmanuel Lecharny wrote: On 8/19/11 2:29 PM, Alan D. Cabrera wrote: On Aug 19, 2011, at 5:16 AM, Emmanuel Lecharny wrote: On 8/19/11 1:55 PM, Alan D. Cabrera wrote: On Aug 19, 2011, at 4:16 AM, Julien Vermillard wrote: Hi, I implemented some really simple chain for MINA 3 based on a list of chain processed by a filter chain. The implementation is very simple, sounded like a good idea : http://svn.apache.org/repos/asf/mina/branches/3.0/core/src/main/java/org/apache/mina/filterchain/DefaultIoFilterChain.java But when i started implementing an HTTP codec I started to see the real design issues. The problem is when you need to produce more than one object during a filter processing, like when you find more than one PDU in a ByteBuffer. We could change IoFilter method : Object messageReceived(IoSession session, Object message); To : ListObject messageReceived(IoSession session, Object message); But starting to use collection here sound like an overkill and not really a nice idea if we want to keep the memory usage low enough. So I'm scraping the current implementation, I'm going to try something else suggested by Emmanuel : When a filter want to call the next filter, he ask it to the filter controller (aka the FilterChain). Let's see if it's going somewhere this time ;) I ran into a different scenario that dictated the same result. For me, often times when a message is sent or received no message is passed on so my protocol had to always return a null. This always seemed awkward to me. Also, then that dictated that I had to check for nulls in the framework, also not so good. Take a look at my class org.apache.mina.link.UpState to see what I mean. We should not compare filters and pipes. Think about the whole mechanism as if it was a state machine, because it is. The only relation between two states are those dicatetd by the transition we allow. We don't transite from one state to another one randomly, so are the data passed from one state to another : they are well known by both the initial and the final states. I'm not sure that I'm following. What is this data that you speak of? Let me clarify my thought : I'm talking about any data transiting from one filter to another. If it's from the socket to the first filter, then it's a ByteBuffer. If it's between the decoder and the Handler, then it's a protocol message. Etc. We should then make no general presumption about the received and sent data. If one state does not require anything, fine : the caller will know it. OTOH, if we can have multiple data to send to the next state, then we just have to loop and call as many times the next step as necessary. I'm not sure that I'm following. Who is the caller and why does he need to know all this? Simply because the kind of data the receiver will process has to be a data it *can* handle. A decoder is supposed to receive some bytes and transform them to a message. But we can also imagine a 2 layers decoder, and then the second decoder will expect a message, not some bytes. One example : in LDAP, we received PDU, which contains TLVs (Type/length/value). Then we transform those TLVs into LDAP message. The first decoder will transform bytes to TLVs; The second decoder will transform the TLVs to LDAP messages (searchRequest, etc); Here, the second decoder won't be able to do anything if it receives some ByteBuffer, so the caller (the first decoder) *must* know what the second decoder will accept, otherwise you'll get some nasty exception. Is it clearer? Yes, filters in a chain must pass messages of a known type between themselves. There's the added dimension of endpoint cardinality, which this thread discusses, of zero, one, many that can occur on both ends of a link in the chain. PS : That's the problem when both of us are not english fluent... When I write a mail (a message), I'm losing some context, and you can't decode exactly what I had in mind when I wrote it. Sorry for that :/ No worries. I, as an American, struggle to keep my english fluent as well. ;) Regards, Alan
Re: Re : [MINA 3.0] filter chains
Hmm consensus about copy on write chain is already here no ? Wanna try to implements ? ;) On Sat, Aug 20, 2011 at 1:32 PM, Edouard De Oliveira doe_wan...@yahoo.fr wrote: We shouldn't systematically copy the chain in the session as imho it's not so usual to dynamically add filters in one particular session (the first case that comes to my mind is dynamically protecting the session with ssl but it requires a new connection most of the time no ? maybe some dynamic compression or some negociation mechanism that will need a particular filter wrapping and unwrapping messages) A copy on write strategy would improve memory footprint. So when would we copy the chain ? when some change happens : add/remove filter a chain controller could associate to the session it's alternative chain or use the default one. So now what about threads coming into play ? using threadlocal seems a good idea Em So to resume - copy when necessary - custom chain should be searched in the following places : threadlocal - session - default chain of the controller ChainController.getChain() will hide this whole complexity providing flexibility and efficiency wdyt ? Cordialement, Regards, -Edouard De Oliveira- De : Emmanuel Lecharny elecha...@gmail.com À : dev@mina.apache.org Envoyé le : Samedi 20 Août 2011 9h29 Objet : Re: [MINA 3.0] filter chains On 8/20/11 8:49 AM, Julien Vermillard wrote: Hi, Because a filterchain can be shared across different sessions and threads, so you need to pass the local chain index because you can store it locally in the filter chain. Perhaps there is something smarter to do, because it's the dark point of this API. The filter chain is copied into the session (at least, it was what was done for MINA 2). Assuming that two different sessions might use two different chains, and assumng that the chain might be dynamically changed, it makes sense to do this copy. Now, if we can split the session (using an executor for instance), then the chain must be copied. It would make sense to store the chain into a ThreadLocal variable instead of storing it into a session object. Julien On Sat, Aug 20, 2011 at 12:41 AM, Alan D. Cabreral...@toolazydogs.com wrote: Why do we pass the current position? We also seem to pass it twice in the method and the controller. Regards, Alan On Aug 19, 2011, at 1:10 PM, Julien Vermillard wrote: Ok I committed the modification, for passing a chain controller to the filter for delegating the call to next filter. First I did it only for read write chaining because the other events (created,open,closed,idle) are fine like they are. They don't need to block an event or send it multiple time to the following filter. Here the modified IoFilter, some controller interface are passed to read write events : http://svn.apache.org/repos/asf/mina/branches/3.0/core/src/main/java/org/apache/mina/api/IoFilter.java Here sample implementation of LoggingFilter : http://svn.apache.org/repos/asf/mina/branches/3.0/core/src/main/java/org/apache/mina/filter/logging/LoggingFilter.java Here the FilterChain implementation, still simple : http://svn.apache.org/repos/asf/mina/branches/3.0/core/src/main/java/org/apache/mina/filterchain/DefaultIoFilterChain.java Now I need to figure how to remove the current position argument of the filter call. Julien On Fri, Aug 19, 2011 at 2:46 PM, Julien Vermillard jvermill...@gmail.com wrote: I half implemented the controller idea, it's looking like working, I'll finish that during the weekend or next week and commit it for review. Julien On Fri, Aug 19, 2011 at 2:39 PM, Steve Ulrichsteve.ulr...@proemion.com wrote: Hi! Besides the points you mentioned, there are some other flaws: 1) How do you handle post-message-forwarding logic? 2) How do you handle filters that transparently push messages to the underlying filters (e.g. keep alive)? So the filters should decide about what to do, the chain about the where. regards Steve There could be specific (empty, single-result, multi-result) implementations that can be extended as needed. Julien Vermillard [mailto:jvermill...@gmail.com] wrote: Hi, I implemented some really simple chain for MINA 3 based on a list of chain processed by a filter chain. The implementation is very simple, sounded like a good idea : http://svn.apache.org/repos/asf/mina/branches/3.0/core/src/main/java/or g/apache/mina/filterchain/DefaultIoFilterChain.java But when i started implementing an HTTP codec I started to see the real design issues. The problem is when you need to produce more than one object during a filter processing, like when you find more than one PDU in a ByteBuffer. We could change IoFilter method : Object messageReceived(IoSession session, Object message); To : ListObject messageReceived(IoSession session, Object message); But starting to use collection here sound
Re: [MINA 3.0] filter chains
Consensus? Really? On Aug 20, 2011, at 10:43 AM, Julien Vermillard wrote: Hmm consensus about copy on write chain is already here no ? Wanna try to implements ? ;) On Sat, Aug 20, 2011 at 1:32 PM, Edouard De Oliveira doe_wan...@yahoo.fr wrote: We shouldn't systematically copy the chain in the session as imho it's not so usual to dynamically add filters in one particular session (the first case that comes to my mind is dynamically protecting the session with ssl but it requires a new connection most of the time no ? maybe some dynamic compression or some negociation mechanism that will need a particular filter wrapping and unwrapping messages) A copy on write strategy would improve memory footprint. So when would we copy the chain ? when some change happens : add/remove filter a chain controller could associate to the session it's alternative chain or use the default one. So now what about threads coming into play ? using threadlocal seems a good idea Em So to resume - copy when necessary - custom chain should be searched in the following places : threadlocal - session - default chain of the controller ChainController.getChain() will hide this whole complexity providing flexibility and efficiency wdyt ? Cordialement, Regards, -Edouard De Oliveira- De : Emmanuel Lecharny elecha...@gmail.com À : dev@mina.apache.org Envoyé le : Samedi 20 Août 2011 9h29 Objet : Re: [MINA 3.0] filter chains On 8/20/11 8:49 AM, Julien Vermillard wrote: Hi, Because a filterchain can be shared across different sessions and threads, so you need to pass the local chain index because you can store it locally in the filter chain. Perhaps there is something smarter to do, because it's the dark point of this API. The filter chain is copied into the session (at least, it was what was done for MINA 2). Assuming that two different sessions might use two different chains, and assumng that the chain might be dynamically changed, it makes sense to do this copy. Now, if we can split the session (using an executor for instance), then the chain must be copied. It would make sense to store the chain into a ThreadLocal variable instead of storing it into a session object. Julien On Sat, Aug 20, 2011 at 12:41 AM, Alan D. Cabreral...@toolazydogs.com wrote: Why do we pass the current position? We also seem to pass it twice in the method and the controller. Regards, Alan On Aug 19, 2011, at 1:10 PM, Julien Vermillard wrote: Ok I committed the modification, for passing a chain controller to the filter for delegating the call to next filter. First I did it only for read write chaining because the other events (created,open,closed,idle) are fine like they are. They don't need to block an event or send it multiple time to the following filter. Here the modified IoFilter, some controller interface are passed to read write events : http://svn.apache.org/repos/asf/mina/branches/3.0/core/src/main/java/org/apache/mina/api/IoFilter.java Here sample implementation of LoggingFilter : http://svn.apache.org/repos/asf/mina/branches/3.0/core/src/main/java/org/apache/mina/filter/logging/LoggingFilter.java Here the FilterChain implementation, still simple : http://svn.apache.org/repos/asf/mina/branches/3.0/core/src/main/java/org/apache/mina/filterchain/DefaultIoFilterChain.java Now I need to figure how to remove the current position argument of the filter call. Julien On Fri, Aug 19, 2011 at 2:46 PM, Julien Vermillard jvermill...@gmail.com wrote: I half implemented the controller idea, it's looking like working, I'll finish that during the weekend or next week and commit it for review. Julien On Fri, Aug 19, 2011 at 2:39 PM, Steve Ulrichsteve.ulr...@proemion.com wrote: Hi! Besides the points you mentioned, there are some other flaws: 1) How do you handle post-message-forwarding logic? 2) How do you handle filters that transparently push messages to the underlying filters (e.g. keep alive)? So the filters should decide about what to do, the chain about the where. regards Steve There could be specific (empty, single-result, multi-result) implementations that can be extended as needed. Julien Vermillard [mailto:jvermill...@gmail.com] wrote: Hi, I implemented some really simple chain for MINA 3 based on a list of chain processed by a filter chain. The implementation is very simple, sounded like a good idea : http://svn.apache.org/repos/asf/mina/branches/3.0/core/src/main/java/or g/apache/mina/filterchain/DefaultIoFilterChain.java But when i started implementing an HTTP codec I started to see the real design issues. The problem is when you need to produce more than one object during a filter processing, like when you find more than one PDU in a ByteBuffer. We could change IoFilter method : Object messageReceived(IoSession session, Object
Re: [MINA 3.0] filter chains
I was thinking about this page : https://cwiki.apache.org/confluence/display/MINA/MINA+3.0+design Look like you have comments about CoW chain, can you please elaborate ? Julien On Sat, Aug 20, 2011 at 10:29 PM, Alan D. Cabrera l...@toolazydogs.com wrote: Consensus? Really? On Aug 20, 2011, at 10:43 AM, Julien Vermillard wrote: Hmm consensus about copy on write chain is already here no ? Wanna try to implements ? ;) On Sat, Aug 20, 2011 at 1:32 PM, Edouard De Oliveira doe_wan...@yahoo.fr wrote: We shouldn't systematically copy the chain in the session as imho it's not so usual to dynamically add filters in one particular session (the first case that comes to my mind is dynamically protecting the session with ssl but it requires a new connection most of the time no ? maybe some dynamic compression or some negociation mechanism that will need a particular filter wrapping and unwrapping messages) A copy on write strategy would improve memory footprint. So when would we copy the chain ? when some change happens : add/remove filter a chain controller could associate to the session it's alternative chain or use the default one. So now what about threads coming into play ? using threadlocal seems a good idea Em So to resume - copy when necessary - custom chain should be searched in the following places : threadlocal - session - default chain of the controller ChainController.getChain() will hide this whole complexity providing flexibility and efficiency wdyt ? Cordialement, Regards, -Edouard De Oliveira- De : Emmanuel Lecharny elecha...@gmail.com À : dev@mina.apache.org Envoyé le : Samedi 20 Août 2011 9h29 Objet : Re: [MINA 3.0] filter chains On 8/20/11 8:49 AM, Julien Vermillard wrote: Hi, Because a filterchain can be shared across different sessions and threads, so you need to pass the local chain index because you can store it locally in the filter chain. Perhaps there is something smarter to do, because it's the dark point of this API. The filter chain is copied into the session (at least, it was what was done for MINA 2). Assuming that two different sessions might use two different chains, and assumng that the chain might be dynamically changed, it makes sense to do this copy. Now, if we can split the session (using an executor for instance), then the chain must be copied. It would make sense to store the chain into a ThreadLocal variable instead of storing it into a session object. Julien On Sat, Aug 20, 2011 at 12:41 AM, Alan D. Cabreral...@toolazydogs.com wrote: Why do we pass the current position? We also seem to pass it twice in the method and the controller. Regards, Alan On Aug 19, 2011, at 1:10 PM, Julien Vermillard wrote: Ok I committed the modification, for passing a chain controller to the filter for delegating the call to next filter. First I did it only for read write chaining because the other events (created,open,closed,idle) are fine like they are. They don't need to block an event or send it multiple time to the following filter. Here the modified IoFilter, some controller interface are passed to read write events : http://svn.apache.org/repos/asf/mina/branches/3.0/core/src/main/java/org/apache/mina/api/IoFilter.java Here sample implementation of LoggingFilter : http://svn.apache.org/repos/asf/mina/branches/3.0/core/src/main/java/org/apache/mina/filter/logging/LoggingFilter.java Here the FilterChain implementation, still simple : http://svn.apache.org/repos/asf/mina/branches/3.0/core/src/main/java/org/apache/mina/filterchain/DefaultIoFilterChain.java Now I need to figure how to remove the current position argument of the filter call. Julien On Fri, Aug 19, 2011 at 2:46 PM, Julien Vermillard jvermill...@gmail.com wrote: I half implemented the controller idea, it's looking like working, I'll finish that during the weekend or next week and commit it for review. Julien On Fri, Aug 19, 2011 at 2:39 PM, Steve Ulrichsteve.ulr...@proemion.com wrote: Hi! Besides the points you mentioned, there are some other flaws: 1) How do you handle post-message-forwarding logic? 2) How do you handle filters that transparently push messages to the underlying filters (e.g. keep alive)? So the filters should decide about what to do, the chain about the where. regards Steve There could be specific (empty, single-result, multi-result) implementations that can be extended as needed. Julien Vermillard [mailto:jvermill...@gmail.com] wrote: Hi, I implemented some really simple chain for MINA 3 based on a list of chain processed by a filter chain. The implementation is very simple, sounded like a good idea : http://svn.apache.org/repos/asf/mina/branches/3.0/core/src/main/java/or g/apache/mina/filterchain/DefaultIoFilterChain.java But when i started implementing an HTTP codec I started to see the real design issues. The problem is when
Re: [MINA 3.0] filter chains
Wow, I totally forgot about these pages. There's been a fair bit of discussion on this topic on the mailing list before, this being the need for dynamically modifying filter chains in a session that's already being used. It is my assertion that it is an anti-pattern that signals the need for a state machine. Getting a protocol right on both network endpoints is very hard and dynamic chains make it even more difficult and error prone. APIs should promote good practices. There are implementation issues as well. Look how complicated the implementation, sketched by Edouard, gets below. Normally I'm a let a thousand flowers bloom kind of guy. But, as you know, I've been a strong advocate of thinning Mina's bloated class library. I find it difficult justifying CoW chains in a library that people already find bewildering. Just my 2 cents. Let's get this finally resolved as this topic seems to pop up on a regular basis. Regards, Alan On Aug 20, 2011, at 2:10 PM, Julien Vermillard wrote: I was thinking about this page : https://cwiki.apache.org/confluence/display/MINA/MINA+3.0+design Look like you have comments about CoW chain, can you please elaborate ? Julien On Sat, Aug 20, 2011 at 10:29 PM, Alan D. Cabrera l...@toolazydogs.com wrote: Consensus? Really? On Aug 20, 2011, at 10:43 AM, Julien Vermillard wrote: Hmm consensus about copy on write chain is already here no ? Wanna try to implements ? ;) On Sat, Aug 20, 2011 at 1:32 PM, Edouard De Oliveira doe_wan...@yahoo.fr wrote: We shouldn't systematically copy the chain in the session as imho it's not so usual to dynamically add filters in one particular session (the first case that comes to my mind is dynamically protecting the session with ssl but it requires a new connection most of the time no ? maybe some dynamic compression or some negociation mechanism that will need a particular filter wrapping and unwrapping messages) A copy on write strategy would improve memory footprint. So when would we copy the chain ? when some change happens : add/remove filter a chain controller could associate to the session it's alternative chain or use the default one. So now what about threads coming into play ? using threadlocal seems a good idea Em So to resume - copy when necessary - custom chain should be searched in the following places : threadlocal - session - default chain of the controller ChainController.getChain() will hide this whole complexity providing flexibility and efficiency wdyt ? Cordialement, Regards, -Edouard De Oliveira- De : Emmanuel Lecharny elecha...@gmail.com À : dev@mina.apache.org Envoyé le : Samedi 20 Août 2011 9h29 Objet : Re: [MINA 3.0] filter chains On 8/20/11 8:49 AM, Julien Vermillard wrote: Hi, Because a filterchain can be shared across different sessions and threads, so you need to pass the local chain index because you can store it locally in the filter chain. Perhaps there is something smarter to do, because it's the dark point of this API. The filter chain is copied into the session (at least, it was what was done for MINA 2). Assuming that two different sessions might use two different chains, and assumng that the chain might be dynamically changed, it makes sense to do this copy. Now, if we can split the session (using an executor for instance), then the chain must be copied. It would make sense to store the chain into a ThreadLocal variable instead of storing it into a session object. Julien On Sat, Aug 20, 2011 at 12:41 AM, Alan D. Cabreral...@toolazydogs.com wrote: Why do we pass the current position? We also seem to pass it twice in the method and the controller. Regards, Alan On Aug 19, 2011, at 1:10 PM, Julien Vermillard wrote: Ok I committed the modification, for passing a chain controller to the filter for delegating the call to next filter. First I did it only for read write chaining because the other events (created,open,closed,idle) are fine like they are. They don't need to block an event or send it multiple time to the following filter. Here the modified IoFilter, some controller interface are passed to read write events : http://svn.apache.org/repos/asf/mina/branches/3.0/core/src/main/java/org/apache/mina/api/IoFilter.java Here sample implementation of LoggingFilter : http://svn.apache.org/repos/asf/mina/branches/3.0/core/src/main/java/org/apache/mina/filter/logging/LoggingFilter.java Here the FilterChain implementation, still simple : http://svn.apache.org/repos/asf/mina/branches/3.0/core/src/main/java/org/apache/mina/filterchain/DefaultIoFilterChain.java Now I need to figure how to remove the current position argument of the filter call. Julien On Fri, Aug 19, 2011 at 2:46 PM, Julien Vermillard jvermill...@gmail.com wrote: I half implemented the controller idea, it's looking like working, I'll finish that during
