Re: Deadlock when using SslFilter and ProxyFilter together
Le 10/02/16 22:11, Jon V. a écrit : > I would have thought the reason SSL Filter needs to be before Proxy is > because the SSL handshaking from the client side would get proxied away > from the SSL Filter. That too. But all in all, the handshake is just an exchange of data, as any other data beinge exchanged later on. It can also be split in pieces, the wame way...
Re: Deadlock when using SslFilter and ProxyFilter together
I would have thought the reason SSL Filter needs to be before Proxy is
because the SSL handshaking from the client side would get proxied away
from the SSL Filter.
On Wed, Feb 10, 2016 at 12:57 PM, Emmanuel Lécharny
wrote:
> Le 10/02/16 18:16, Norbert Irmer a écrit :
> >> Ok, but you can insert teh SslFilter *before* the proxyFilter. Once you
> >> get the ProxyConnector instance, get the connector and do :
> >
> > Well, I am using a patched QuickFixJ version, in which the control flow
> is a bit different.
> >
> > Here is an excerpt from the code they are using:
> >
> >
> > CompositeIoFilterChainBuilder ioFilterChainBuilder = new
> CompositeIoFilterChainBuilder(
> > userIoFilterChainBuilder);
> >
> > boolean useProxy = proxyType != null && proxyPort > 0 &&
> > socketAddresses[0] instanceof
> InetSocketAddress;
> >
> >
> > SslFilter sslFilter = null;
> > if (sslEnabled) {
> > boolean autoStart = !useProxy;
> > sslFilter = installSslFilter(ioFilterChainBuilder,
> autoStart);
> > }
> >
> >
> ioFilterChainBuilder.addLast(FIXProtocolCodecFactory.FILTER_NAME,
> > new ProtocolCodecFilter(new
> FIXProtocolCodecFactory()));
> >
> > if (useProxy) {
> > SocketConnector socketConnector = new
> NioSocketConnector();
> >
> socketConnector.setFilterChainBuilder(ioFilterChainBuilder);
> > ioConnector = ProtocolFactory.createIoProxyConnector(
> socketConnector,
> > (InetSocketAddress) socketAddresses[0], new
> InetSocketAddress(proxyHost,
> > proxyPort), proxyType, proxyVersion,
> proxyUser, proxyPassword,
> > proxyDomain, proxyWorkstation);
> > } else {
> > ioConnector =
> ProtocolFactory.createIoConnector(socketAddresses[0]);
> > ioConnector.setFilterChainBuilder(ioFilterChainBuilder);
> > }
> >
> > ioHandler = new InitiatorIoHandler(fixSession, sslFilter,
> networkingOptions, eventHandlingStrategy);
> > }
> >
> > private SslFilter installSslFilter(CompositeIoFilterChainBuilder
> ioFilterChainBuilder,
> >boolean autoStart)
> > throws GeneralSecurityException {
> > SslFilter sslFilter = new
> SslFilter(SSLContextFactory.getInstance(keyStoreName,
> > keyStorePassword.toCharArray()), autoStart);
> > if (enableProtocole != null)
> > sslFilter.setEnabledProtocols(enableProtocole);
> > if (cipherSuites != null)
> > sslFilter.setEnabledCipherSuites(cipherSuites);
> > sslFilter.setUseClientMode(true);
> > ioFilterChainBuilder.addLast(SSLSupport.FILTER_NAME,
> sslFilter);
> > return sslFilter;
> > }
> >
> >
> > Sure, with a lot of time, I could possibly rewrite the code, to swap the
> positions of the filters in the filterchain.
> >
> > But I see little sense in it, since the code is working. The only
> problem I have, is that I get this deadlock after a few minutes or so.
> How can you say the code is working when you get a deadlock ?
>
> As Jon said, having the ProcyFilter *before* the SSLFilter in the chain
> is not the right thing. The reason being that you may perfectly get
> fragmented TCP buffers, and the proxy might not be able to deal with
> de-fragmentation when it comes to encrypted data.
>
> I suggest that you patch teh code to first call the installSslFilter
> method after the proxy has been created, and in the
> installSslFiltermethod, use addFirst() instead of addLast().
>
> Otherwise, you can also check with the latest trunk revision I built
> today to see if it's any better (on http://people.apache.org/~elecharny)
>
>
Re: Deadlock when using SslFilter and ProxyFilter together
Le 10/02/16 18:16, Norbert Irmer a écrit :
>> Ok, but you can insert teh SslFilter *before* the proxyFilter. Once you
>> get the ProxyConnector instance, get the connector and do :
>
> Well, I am using a patched QuickFixJ version, in which the control flow is a
> bit different.
>
> Here is an excerpt from the code they are using:
>
>
> CompositeIoFilterChainBuilder ioFilterChainBuilder = new
> CompositeIoFilterChainBuilder(
> userIoFilterChainBuilder);
>
> boolean useProxy = proxyType != null && proxyPort > 0 &&
> socketAddresses[0] instanceof
> InetSocketAddress;
>
>
> SslFilter sslFilter = null;
> if (sslEnabled) {
> boolean autoStart = !useProxy;
> sslFilter = installSslFilter(ioFilterChainBuilder, autoStart);
> }
>
> ioFilterChainBuilder.addLast(FIXProtocolCodecFactory.FILTER_NAME,
> new ProtocolCodecFilter(new FIXProtocolCodecFactory()));
>
> if (useProxy) {
> SocketConnector socketConnector = new NioSocketConnector();
> socketConnector.setFilterChainBuilder(ioFilterChainBuilder);
> ioConnector = ProtocolFactory.createIoProxyConnector(
> socketConnector,
> (InetSocketAddress) socketAddresses[0], new
> InetSocketAddress(proxyHost,
> proxyPort), proxyType, proxyVersion,
> proxyUser, proxyPassword,
> proxyDomain, proxyWorkstation);
> } else {
> ioConnector =
> ProtocolFactory.createIoConnector(socketAddresses[0]);
> ioConnector.setFilterChainBuilder(ioFilterChainBuilder);
> }
>
> ioHandler = new InitiatorIoHandler(fixSession, sslFilter,
> networkingOptions, eventHandlingStrategy);
> }
>
> private SslFilter installSslFilter(CompositeIoFilterChainBuilder
> ioFilterChainBuilder,
>boolean autoStart)
> throws GeneralSecurityException {
> SslFilter sslFilter = new
> SslFilter(SSLContextFactory.getInstance(keyStoreName,
> keyStorePassword.toCharArray()), autoStart);
> if (enableProtocole != null)
> sslFilter.setEnabledProtocols(enableProtocole);
> if (cipherSuites != null)
> sslFilter.setEnabledCipherSuites(cipherSuites);
> sslFilter.setUseClientMode(true);
> ioFilterChainBuilder.addLast(SSLSupport.FILTER_NAME, sslFilter);
> return sslFilter;
> }
>
>
> Sure, with a lot of time, I could possibly rewrite the code, to swap the
> positions of the filters in the filterchain.
>
> But I see little sense in it, since the code is working. The only problem I
> have, is that I get this deadlock after a few minutes or so.
How can you say the code is working when you get a deadlock ?
As Jon said, having the ProcyFilter *before* the SSLFilter in the chain
is not the right thing. The reason being that you may perfectly get
fragmented TCP buffers, and the proxy might not be able to deal with
de-fragmentation when it comes to encrypted data.
I suggest that you patch teh code to first call the installSslFilter
method after the proxy has been created, and in the
installSslFiltermethod, use addFirst() instead of addLast().
Otherwise, you can also check with the latest trunk revision I built
today to see if it's any better (on http://people.apache.org/~elecharny)
Re: Deadlock when using SslFilter and ProxyFilter together
> Ok, but you can insert teh SslFilter *before* the proxyFilter. Once you
> get the ProxyConnector instance, get the connector and do :
Well, I am using a patched QuickFixJ version, in which the control flow is a
bit different.
Here is an excerpt from the code they are using:
CompositeIoFilterChainBuilder ioFilterChainBuilder = new
CompositeIoFilterChainBuilder(
userIoFilterChainBuilder);
boolean useProxy = proxyType != null && proxyPort > 0 &&
socketAddresses[0] instanceof InetSocketAddress;
SslFilter sslFilter = null;
if (sslEnabled) {
boolean autoStart = !useProxy;
sslFilter = installSslFilter(ioFilterChainBuilder, autoStart);
}
ioFilterChainBuilder.addLast(FIXProtocolCodecFactory.FILTER_NAME,
new ProtocolCodecFilter(new FIXProtocolCodecFactory()));
if (useProxy) {
SocketConnector socketConnector = new NioSocketConnector();
socketConnector.setFilterChainBuilder(ioFilterChainBuilder);
ioConnector = ProtocolFactory.createIoProxyConnector(
socketConnector,
(InetSocketAddress) socketAddresses[0], new
InetSocketAddress(proxyHost,
proxyPort), proxyType, proxyVersion, proxyUser,
proxyPassword,
proxyDomain, proxyWorkstation);
} else {
ioConnector =
ProtocolFactory.createIoConnector(socketAddresses[0]);
ioConnector.setFilterChainBuilder(ioFilterChainBuilder);
}
ioHandler = new InitiatorIoHandler(fixSession, sslFilter,
networkingOptions, eventHandlingStrategy);
}
private SslFilter installSslFilter(CompositeIoFilterChainBuilder
ioFilterChainBuilder,
boolean autoStart)
throws GeneralSecurityException {
SslFilter sslFilter = new
SslFilter(SSLContextFactory.getInstance(keyStoreName,
keyStorePassword.toCharArray()), autoStart);
if (enableProtocole != null)
sslFilter.setEnabledProtocols(enableProtocole);
if (cipherSuites != null)
sslFilter.setEnabledCipherSuites(cipherSuites);
sslFilter.setUseClientMode(true);
ioFilterChainBuilder.addLast(SSLSupport.FILTER_NAME, sslFilter);
return sslFilter;
}
Sure, with a lot of time, I could possibly rewrite the code, to swap the
positions of the filters in the filterchain.
But I see little sense in it, since the code is working. The only problem I
have, is that I get this deadlock after a few minutes or so.
At the moment, as a workaround, I simply disabled the "synchronized(handler) {
... }" in ProxyFilter.writeData(),
I think for my special case this will work.
I don't think it is possible, that the filterchains are traverted in the same
order for reading and writing.
Re: Deadlock when using SslFilter and ProxyFilter together
Le 10/02/16 16:37, Norbert Irmer a écrit :
>> seems like you have installed the proxy filter before the ssl filter in
>> your chain. Can you do the opposite ?
> I don't build the chain manually, I simply use a "ProxyConnector" (from
> package "org.apache.mina.proxy")
> to make a proxy connection (this also explains, why I had the
> SslFilter/autoStart problem in my earlier mail).
>
> "ProxyConnector" does the following in "setConnector":
>
>
> private void setConnector(final SocketConnector connector) {
> if (connector == null) {
> throw new IllegalArgumentException("connector cannot be null");
> }
>
> this.connector = connector;
> String className = ProxyFilter.class.getName();
>
> // Removes an old ProxyFilter instance from the chain
> if (connector.getFilterChain().contains(className)) {
> connector.getFilterChain().remove(className);
> }
>
> // Insert the ProxyFilter as the first filter in the filter chain
> builder
> connector.getFilterChain().addFirst(className, proxyFilter);
> }
Ok, but you can insert teh SslFilter *before* the proxyFilter. Once you
get the ProxyConnector instance, get the connector and do :
SslFilter sslFilter = new SslFilter(BogusSslContextFactory
.getInstance(true));
connector.getFilterChain().addFirst( "MySslFilter", sslFilter );
Re: Deadlock when using SslFilter and ProxyFilter together
> seems like you have installed the proxy filter before the ssl filter in
> your chain. Can you do the opposite ?
I don't build the chain manually, I simply use a "ProxyConnector" (from package
"org.apache.mina.proxy")
to make a proxy connection (this also explains, why I had the
SslFilter/autoStart problem in my earlier mail).
"ProxyConnector" does the following in "setConnector":
private void setConnector(final SocketConnector connector) {
if (connector == null) {
throw new IllegalArgumentException("connector cannot be null");
}
this.connector = connector;
String className = ProxyFilter.class.getName();
// Removes an old ProxyFilter instance from the chain
if (connector.getFilterChain().contains(className)) {
connector.getFilterChain().remove(className);
}
// Insert the ProxyFilter as the first filter in the filter chain
builder
connector.getFilterChain().addFirst(className, proxyFilter);
}
Re: Deadlock when using SslFilter and ProxyFilter together
Le 10/02/16 13:11, Norbert Irmer a écrit : > Hello, > > I did a "threaddump" with visualvm in the deadlock situation (not sure if > this is what you wanted, but I don't see > anything else), it shows the same deadlock situation (at the bottom of this > mail), as in my previous mail. Thanks ! seems like you have installed the proxy filter before the ssl filter in your chain. Can you do the opposite ?
Re: Deadlock when using SslFilter and ProxyFilter together
Hello, I did a "threaddump" with visualvm in the deadlock situation (not sure if this is what you wanted, but I don't see anything else), it shows the same deadlock situation (at the bottom of this mail), as in my previous mail. 2016-02-10 13:04:58 Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.60-b23 mixed mode): "Thread-7" #29 daemon prio=5 os_prio=0 tid=0x1ed49000 nid=0xfb0 runnable [0x1f66e000] java.lang.Thread.State: RUNNABLE at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) at java.net.SocketInputStream.read(SocketInputStream.java:170) at java.net.SocketInputStream.read(SocketInputStream.java:141) at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284) at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326) at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178) - locked <0x00076ee06830> (a java.io.InputStreamReader) at java.io.InputStreamReader.read(InputStreamReader.java:184) at java.io.BufferedReader.fill(BufferedReader.java:161) at java.io.BufferedReader.skip(BufferedReader.java:411) - locked <0x00076ee06830> (a java.io.InputStreamReader) at de.digitec.d3.pricing.streaming.impl.StreamingChannelImpl.readMessage(StreamingChannelImpl.java:331) at de.digitec.d3.pricing.streaming.impl.Session.run(Session.java:116) - locked <0x00076ee068c8> (a java.lang.Object) Locked ownable synchronizers: - None "SiDispatcher_ReadLoop" #28 prio=5 os_prio=0 tid=0x1ed48800 nid=0x17fc waiting on condition [0x221de000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00076f04c960> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at de.digitec.d3.pricing.streaming.impl.SessionManagerAbstractImpl.readMessage(SessionManagerAbstractImpl.java:49) at de.digitec.d3.pricing.streaming.impl.SiDispatcher.run(SiDispatcher.java:455) at java.lang.Thread.run(Thread.java:745) Locked ownable synchronizers: - None "Thread-6" #27 daemon prio=5 os_prio=0 tid=0x1ed47800 nid=0x1674 runnable [0x2202f000] java.lang.Thread.State: RUNNABLE at java.net.DualStackPlainSocketImpl.accept0(Native Method) at java.net.DualStackPlainSocketImpl.socketAccept(DualStackPlainSocketImpl.java:131) at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409) at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:199) - locked <0x00076ee102c8> (a java.net.SocksSocketImpl) at java.net.ServerSocket.implAccept(ServerSocket.java:545) at java.net.ServerSocket.accept(ServerSocket.java:513) at de.digitec.d3.pricing.streaming.impl.SocketAcceptor.accept(SocketAcceptor.java:152) at de.digitec.d3.pricing.streaming.impl.SocketAcceptor.run(SocketAcceptor.java:128) Locked ownable synchronizers: - None "SessionManagerStreamingImpl_Timer" #26 daemon prio=5 os_prio=0 tid=0x1ed47000 nid=0x1bbc in Object.wait() [0x21d9f000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.util.TimerThread.mainLoop(Timer.java:552) - locked <0x00076ee18180> (a java.util.TaskQueue) at java.util.TimerThread.run(Timer.java:505) Locked ownable synchronizers: - None "SiDispatcher_Timer" #25 prio=5 os_prio=0 tid=0x1ed46000 nid=0x12d8 in Object.wait() [0x2030f000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:502) at java.util.TimerThread.mainLoop(Timer.java:526) - locked <0x00076ee18350> (a java.util.TaskQueue) at java.util.TimerThread.run(Timer.java:505) Locked ownable synchronizers: - None "RMI TCP Connection(2)-172.16.125.86" #24 daemon prio=5 os_prio=0 tid=0x1ed45800 nid=0x1704 runnable [0x218ad000] java.lang.Thread.State: RUNNABLE at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) at java.net.SocketInputStream.read(SocketInputStream.java:170) at java.net.SocketInputStream.read(SocketInputStream.java:141) at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) at java.io.BufferedInputStream.read(BufferedInputStream.java:265) - locked <0x00076ee12618> (a java.io.BufferedInputStream) at java.io.FilterInput
Re: Deadlock when using SslFilter and ProxyFilter together
Le 10/02/16 10:08, Norbert Irmer a écrit : > I attached a stacktrace of the two threads, which are in a deadlock. > > Jon maybe right, when saying the write stack is threadsafe, the problem is, > that simultaneously > a read operation is going on, and both stacks use the same filters. The stakcs you provided are useless. What we really need here is the full stack dump. Also note that the filter is instanciated as many times as there are sessions, so the real problem is when a session is processed by two different threads. That might make sense in some use case, but we have to guarantee that the encryption and decryption of messages are done fully before dealing with the next message. This is the reason we have a lock in the SslHandler. That being said, there *is* something wrong and we need to figure out what it is and how to get it fixed.
Re: Deadlock when using SslFilter and ProxyFilter together
I attached a stacktrace of the two threads, which are in a deadlock.
Jon maybe right, when saying the write stack is threadsafe, the problem is,
that simultaneously
a read operation is going on, and both stacks use the same filters.
Thread "NioProcessor-2":
flushScheduledEvents():308, SslHandler {org.apache.mina.filter.ssl}
messageReceived():534, SslFilter {org.apache.mina.filter.ssl}
callNextMessageReceived():542, DefaultIoFilterChain
{org.apache.mina.core.filterchain}
access$1300():48, DefaultIoFilterChain {org.apache.mina.core.filterchain}
messageReceived():943, DefaultIoFilterChain$EntryImpl$1
{org.apache.mina.core.filterchain}
messageReceived():153, ProxyFilter {org.apache.mina.proxy.filter}
callNextMessageReceived():542, DefaultIoFilterChain
{org.apache.mina.core.filterchain}
access$1300():48, DefaultIoFilterChain {org.apache.mina.core.filterchain}
messageReceived():943, DefaultIoFilterChain$EntryImpl$1
{org.apache.mina.core.filterchain}
messageReceived():109, IoFilterAdapter {org.apache.mina.core.filterchain}
callNextMessageReceived():542, DefaultIoFilterChain
{org.apache.mina.core.filterchain}
fireMessageReceived():535, DefaultIoFilterChain
{org.apache.mina.core.filterchain}
read():684, AbstractPollingIoProcessor {org.apache.mina.core.polling}
process():638, AbstractPollingIoProcessor {org.apache.mina.core.polling}
process():627, AbstractPollingIoProcessor {org.apache.mina.core.polling}
access$600():68, AbstractPollingIoProcessor {org.apache.mina.core.polling}
run():1096, AbstractPollingIoProcessor$Processor {org.apache.mina.core.polling}
run():64, NamePreservingRunnable {org.apache.mina.util}
runWorker():1142, ThreadPoolExecutor {java.util.concurrent}
run():617, ThreadPoolExecutor$Worker {java.util.concurrent}
run():745, Thread {java.lang}
Thread "FixMessageProcessor(0)::ABFX":
writeData():209, ProxyFilter {org.apache.mina.proxy.filter}
filterWrite():192, ProxyFilter {org.apache.mina.proxy.filter}
callPreviousFilterWrite():625, DefaultIoFilterChain
{org.apache.mina.core.filterchain}
access$1500():48, DefaultIoFilterChain {org.apache.mina.core.filterchain}
filterWrite():953, DefaultIoFilterChain$EntryImpl$1
{org.apache.mina.core.filterchain}
flushScheduledEvents():316, SslHandler {org.apache.mina.filter.ssl}
filterWrite():653, SslFilter {org.apache.mina.filter.ssl}
callPreviousFilterWrite():625, DefaultIoFilterChain
{org.apache.mina.core.filterchain}
access$1500():48, DefaultIoFilterChain {org.apache.mina.core.filterchain}
filterWrite():953, DefaultIoFilterChain$EntryImpl$1
{org.apache.mina.core.filterchain}
filterWrite():321, ProtocolCodecFilter {org.apache.mina.filter.codec}
callPreviousFilterWrite():625, DefaultIoFilterChain
{org.apache.mina.core.filterchain}
access$1500():48, DefaultIoFilterChain {org.apache.mina.core.filterchain}
filterWrite():953, DefaultIoFilterChain$EntryImpl$1
{org.apache.mina.core.filterchain}
filterWrite():877, DefaultIoFilterChain$TailFilter
{org.apache.mina.core.filterchain}
callPreviousFilterWrite():625, DefaultIoFilterChain
{org.apache.mina.core.filterchain}
fireFilterWrite():618, DefaultIoFilterChain {org.apache.mina.core.filterchain}
write():530, AbstractIoSession {org.apache.mina.core.session}
write():474, AbstractIoSession {org.apache.mina.core.session}
send():51, IoSessionResponder {quickfix.mina}
send():2534, Session {quickfix}
sendRaw():2466, Session {quickfix}
send():2519, Session {quickfix}
sendToTarget():635, Session {quickfix}
sendMessageToTarget():529, FixApplication {de.digitec.d3.gateway.fix}
subscribe():52, FixProducerHandle {de.digitec.d3.gateway.fix}
subscribe():134, FixApplication {de.digitec.d3.gateway.fix}
processMessage():288, FixMessageProcessor {de.digitec.d3.gateway.fix}
run():245, FixMessageProcessor {de.digitec.d3.gateway.fix}
run():878, FixApplication$MessageProcessorThread {de.digitec.d3.gateway.fix}
DIGITEC GmbH
Johannes-Brahms-Platz 1
20355 Hamburg
Tel +49 40 23776-147
Fax +49 40 23776-200
www.DIGITEC.de
___
DIGITEC GmbH EDV-Beratung, Soft- und Hardware
Sitz Hamburg - HRB 288 18 Amtsgericht Hamburg
Geschäftsführer: Hans-Joachim Hänel, Hans-Jürgen Joost
Finanzamt Hamburg-Mitte, Steuer-Nr. 48/715/01586 - VAT-Nr. DE 118646977
WICHTIGER HINWEIS
Die Kommunikation mit DIGITEC GmbH per Email dient nur dem Zweck der schnellen
Information.
Rechtsverbindliche Absprachen kommen über dieses Medium nicht zustande !
IMPORTANT NOTICE
Communication with digitec GmbH via email is for quick information purposes
only.
This medium is not to be used for the exchange of legally binding statements !
