[jira] [Commented] (CASSANDRA-11749) CQLSH gets SSL exception following a COPY FROM
[
https://issues.apache.org/jira/browse/CASSANDRA-11749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15325275#comment-15325275
]
Stefania commented on CASSANDRA-11749:
--
Committed to 2.1 as 68319f7c3be232a58e68ca91206283076aa3dedb and merged upwards.
> CQLSH gets SSL exception following a COPY FROM
> --
>
> Key: CASSANDRA-11749
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11749
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
>Reporter: Stefania
>Assignee: Stefania
> Fix For: 2.1.15, 2.2.7, 3.8, 3.0.8
>
> Attachments: driver_debug.txt, stdout.txt.zip,
> stdout_single_process.txt.zip
>
>
> When running Cassandra and cqlsh with SSL, the following command occasionally
> results in the exception below:
> {code}
> cqlsh --ssl -f kv.cql
> {code}
> {code}
> ERROR [SharedPool-Worker-2] 2016-05-11 12:41:03,583 Message.java:538 -
> Unexpected exception during request; channel = [id: 0xeb75e05d,
> /127.0.0.1:51083 => /127.0.0.1:9042]
> io.netty.handler.codec.DecoderException: javax.net.ssl.SSLException: bad
> record MAC
> at
> io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:280)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:149)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.channel.epoll.EpollSocketChannel$EpollSocketUnsafe.epollInReady(EpollSocketChannel.java:722)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:326)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:264)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]
> Caused by: javax.net.ssl.SSLException: bad record MAC
> at sun.security.ssl.Alerts.getSSLException(Alerts.java:208)
> ~[na:1.8.0_91]
> at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1728)
> ~[na:1.8.0_91]
> at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:981)
> ~[na:1.8.0_91]
> at
> sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:907)
> ~[na:1.8.0_91]
> at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:781)
> ~[na:1.8.0_91]
> at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624) ~[na:1.8.0_91]
> at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:982)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:908)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:854)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:249)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> ... 10 common frames omitted
> Caused by: javax.crypto.BadPaddingException: bad record MAC
> at sun.security.ssl.InputRecord.decrypt(InputRecord.java:219)
> ~[na:1.8.0_91]
> at
> sun.security.ssl.EngineInputRecord.decrypt(EngineInputRecord.java:177)
> ~[na:1.8.0_91]
> at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:974)
> ~[na:1.8.0_91]
> ... 17 common frames omitted
> {code}
> where
> {code}
> cat kv.cql
> create keyspace if not exists cvs_copy_ks with replication = {'class':
> 'SimpleStrategy', 'replication_factor':1};
> create table if not exists cvs_copy_ks.kv (key int primary key, value text);
> truncate cvs_copy_ks.kv;
> copy cvs_copy_ks.kv (key, value) from 'kv.csv' with header='true';
> select * from cvs_copy_ks.kv;
> drop keyspace cvs_copy_ks;
> stefi@cuoricina:~/git/cstar/cassandra$ cat kv.c
> kv.cql kv.csv
> cat kv.csv
> key,value
> 1,'a'
> 2,'b'
> 3,'c'
> {code}
> The COPY FROM succeeds, however the follow
[jira] [Commented] (CASSANDRA-11749) CQLSH gets SSL exception following a COPY FROM
[
https://issues.apache.org/jira/browse/CASSANDRA-11749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15316239#comment-15316239
]
Stefania commented on CASSANDRA-11749:
--
Thanks for the review. You're correct regarding Windows dtests: I aborted them
before the {{PlainTextAuthProvider}} revert, and then forgot to restart them.
I've just started them now, I will commit if the results on Windows are also
good.
> CQLSH gets SSL exception following a COPY FROM
> --
>
> Key: CASSANDRA-11749
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11749
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
>Reporter: Stefania
>Assignee: Stefania
> Fix For: 2.1.x, 2.2.x, 3.0.x, 3.x
>
> Attachments: driver_debug.txt, stdout.txt.zip,
> stdout_single_process.txt.zip
>
>
> When running Cassandra and cqlsh with SSL, the following command occasionally
> results in the exception below:
> {code}
> cqlsh --ssl -f kv.cql
> {code}
> {code}
> ERROR [SharedPool-Worker-2] 2016-05-11 12:41:03,583 Message.java:538 -
> Unexpected exception during request; channel = [id: 0xeb75e05d,
> /127.0.0.1:51083 => /127.0.0.1:9042]
> io.netty.handler.codec.DecoderException: javax.net.ssl.SSLException: bad
> record MAC
> at
> io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:280)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:149)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.channel.epoll.EpollSocketChannel$EpollSocketUnsafe.epollInReady(EpollSocketChannel.java:722)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:326)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:264)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]
> Caused by: javax.net.ssl.SSLException: bad record MAC
> at sun.security.ssl.Alerts.getSSLException(Alerts.java:208)
> ~[na:1.8.0_91]
> at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1728)
> ~[na:1.8.0_91]
> at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:981)
> ~[na:1.8.0_91]
> at
> sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:907)
> ~[na:1.8.0_91]
> at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:781)
> ~[na:1.8.0_91]
> at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624) ~[na:1.8.0_91]
> at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:982)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:908)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:854)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:249)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> ... 10 common frames omitted
> Caused by: javax.crypto.BadPaddingException: bad record MAC
> at sun.security.ssl.InputRecord.decrypt(InputRecord.java:219)
> ~[na:1.8.0_91]
> at
> sun.security.ssl.EngineInputRecord.decrypt(EngineInputRecord.java:177)
> ~[na:1.8.0_91]
> at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:974)
> ~[na:1.8.0_91]
> ... 17 common frames omitted
> {code}
> where
> {code}
> cat kv.cql
> create keyspace if not exists cvs_copy_ks with replication = {'class':
> 'SimpleStrategy', 'replication_factor':1};
> create table if not exists cvs_copy_ks.kv (key int primary key, value text);
> truncate cvs_copy_ks.kv;
> copy cvs_copy_ks.kv (key, value) from 'kv.csv' with header='true';
> select * from cvs_copy_ks.kv;
> drop keyspace cvs_copy_ks;
> stefi@cuo
[jira] [Commented] (CASSANDRA-11749) CQLSH gets SSL exception following a COPY FROM
[
https://issues.apache.org/jira/browse/CASSANDRA-11749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15314722#comment-15314722
]
Tyler Hobbs commented on CASSANDRA-11749:
-
The patch looks good to me. However, it looks like the Windows dtest run needs
to be done again (I'm guessing it didn't get the {{PlainTextAuthProvider}}
revert). Assuming that test has no problems, +1 from me.
> CQLSH gets SSL exception following a COPY FROM
> --
>
> Key: CASSANDRA-11749
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11749
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
>Reporter: Stefania
>Assignee: Stefania
> Fix For: 2.1.x, 2.2.x, 3.0.x, 3.x
>
> Attachments: driver_debug.txt, stdout.txt.zip,
> stdout_single_process.txt.zip
>
>
> When running Cassandra and cqlsh with SSL, the following command occasionally
> results in the exception below:
> {code}
> cqlsh --ssl -f kv.cql
> {code}
> {code}
> ERROR [SharedPool-Worker-2] 2016-05-11 12:41:03,583 Message.java:538 -
> Unexpected exception during request; channel = [id: 0xeb75e05d,
> /127.0.0.1:51083 => /127.0.0.1:9042]
> io.netty.handler.codec.DecoderException: javax.net.ssl.SSLException: bad
> record MAC
> at
> io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:280)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:149)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.channel.epoll.EpollSocketChannel$EpollSocketUnsafe.epollInReady(EpollSocketChannel.java:722)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:326)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:264)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]
> Caused by: javax.net.ssl.SSLException: bad record MAC
> at sun.security.ssl.Alerts.getSSLException(Alerts.java:208)
> ~[na:1.8.0_91]
> at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1728)
> ~[na:1.8.0_91]
> at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:981)
> ~[na:1.8.0_91]
> at
> sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:907)
> ~[na:1.8.0_91]
> at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:781)
> ~[na:1.8.0_91]
> at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624) ~[na:1.8.0_91]
> at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:982)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:908)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:854)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:249)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> ... 10 common frames omitted
> Caused by: javax.crypto.BadPaddingException: bad record MAC
> at sun.security.ssl.InputRecord.decrypt(InputRecord.java:219)
> ~[na:1.8.0_91]
> at
> sun.security.ssl.EngineInputRecord.decrypt(EngineInputRecord.java:177)
> ~[na:1.8.0_91]
> at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:974)
> ~[na:1.8.0_91]
> ... 17 common frames omitted
> {code}
> where
> {code}
> cat kv.cql
> create keyspace if not exists cvs_copy_ks with replication = {'class':
> 'SimpleStrategy', 'replication_factor':1};
> create table if not exists cvs_copy_ks.kv (key int primary key, value text);
> truncate cvs_copy_ks.kv;
> copy cvs_copy_ks.kv (key, value) from 'kv.csv' with header='true';
> select * from cvs_copy_ks.kv;
> drop keyspace cvs_copy_ks;
> stefi@cuoricina:~/git/cstar/cas
[jira] [Commented] (CASSANDRA-11749) CQLSH gets SSL exception following a COPY FROM
[
https://issues.apache.org/jira/browse/CASSANDRA-11749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15310349#comment-15310349
]
Stefania commented on CASSANDRA-11749:
--
I've reverted the changes for {{PlainTextAuthProvider}} since they were not
necessary and were causing problems. I've restarted the tests for 2.1, if they
are OK I'll restart the other branches as well.
> CQLSH gets SSL exception following a COPY FROM
> --
>
> Key: CASSANDRA-11749
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11749
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
>Reporter: Stefania
>Assignee: Stefania
> Fix For: 2.1.x, 2.2.x, 3.0.x, 3.x
>
> Attachments: driver_debug.txt, stdout.txt.zip,
> stdout_single_process.txt.zip
>
>
> When running Cassandra and cqlsh with SSL, the following command occasionally
> results in the exception below:
> {code}
> cqlsh --ssl -f kv.cql
> {code}
> {code}
> ERROR [SharedPool-Worker-2] 2016-05-11 12:41:03,583 Message.java:538 -
> Unexpected exception during request; channel = [id: 0xeb75e05d,
> /127.0.0.1:51083 => /127.0.0.1:9042]
> io.netty.handler.codec.DecoderException: javax.net.ssl.SSLException: bad
> record MAC
> at
> io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:280)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:149)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.channel.epoll.EpollSocketChannel$EpollSocketUnsafe.epollInReady(EpollSocketChannel.java:722)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:326)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:264)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]
> Caused by: javax.net.ssl.SSLException: bad record MAC
> at sun.security.ssl.Alerts.getSSLException(Alerts.java:208)
> ~[na:1.8.0_91]
> at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1728)
> ~[na:1.8.0_91]
> at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:981)
> ~[na:1.8.0_91]
> at
> sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:907)
> ~[na:1.8.0_91]
> at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:781)
> ~[na:1.8.0_91]
> at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624) ~[na:1.8.0_91]
> at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:982)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:908)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:854)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:249)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> ... 10 common frames omitted
> Caused by: javax.crypto.BadPaddingException: bad record MAC
> at sun.security.ssl.InputRecord.decrypt(InputRecord.java:219)
> ~[na:1.8.0_91]
> at
> sun.security.ssl.EngineInputRecord.decrypt(EngineInputRecord.java:177)
> ~[na:1.8.0_91]
> at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:974)
> ~[na:1.8.0_91]
> ... 17 common frames omitted
> {code}
> where
> {code}
> cat kv.cql
> create keyspace if not exists cvs_copy_ks with replication = {'class':
> 'SimpleStrategy', 'replication_factor':1};
> create table if not exists cvs_copy_ks.kv (key int primary key, value text);
> truncate cvs_copy_ks.kv;
> copy cvs_copy_ks.kv (key, value) from 'kv.csv' with header='true';
> select * from cvs_copy_ks.kv;
> drop keyspace cvs_copy_ks;
> stefi@cuoricina:~/git/cstar/cassandra$ cat kv.c
[jira] [Commented] (CASSANDRA-11749) CQLSH gets SSL exception following a COPY FROM
[
https://issues.apache.org/jira/browse/CASSANDRA-11749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15309486#comment-15309486
]
Stefania commented on CASSANDRA-11749:
--
Thanks for looking into this [~aholmber]. You're correct: when I experienced
races I was definitely shutting down the cluster after creating the new session.
Here is the full patch, the 2.1 patch applies to 2.2 as well, and the 3.0 patch
up-merges without conflicts:
||2.1||2.2||3.0||3.7||trunk||
|[patch|https://github.com/stef1927/cassandra/commits/11749-cqlsh-2.1]|[patch|https://github.com/stef1927/cassandra/commits/11749-cqlsh-2.2]|[patch|https://github.com/stef1927/cassandra/commits/11749-cqlsh-3.0]|[patch|https://github.com/stef1927/cassandra/commits/11749-cqlsh-3.7]|[patch|https://github.com/stef1927/cassandra/commits/11749-cqlsh]|
|[dtest|http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-11749-cqlsh-2.1-cqlsh-tests/]|[dtest|http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-11749-cqlsh-2.2-cqlsh-tests/]|[dtest|http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-11749-cqlsh-3.0-cqlsh-tests/]|[dtest|http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-11749-cqlsh-3.7-cqlsh-tests/]|[dtest|http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-11749-cqlsh-cqlsh-tests/]|
I've also launched the dtests for 2.2 on Windows
[here|http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-11749-cqlsh-2.2-windows-dtest_win32/].
[~pauloricardomg] or [~thobbs] would you mind reviewing this small patch to
shut down the parent cluster after forking child processes? The parent cluster
is shutdown also in the feeding process even though we strictly don't need to
do this for this process, since it does not connect to Cassandra. However, it
is cleaner to shut it down. Also, we now create a new {{PlainTextAuthProvider}}
rather than sharing the one of the parent. On Windows, to avoid pickle-ability
problems, we don't pass the parent connection to the child processes, we don't
need to shutdown the parent connections on Windows because processes are
spawned and not forked.
> CQLSH gets SSL exception following a COPY FROM
> --
>
> Key: CASSANDRA-11749
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11749
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
>Reporter: Stefania
>Assignee: Stefania
> Fix For: 2.1.x
>
> Attachments: driver_debug.txt, stdout.txt.zip,
> stdout_single_process.txt.zip
>
>
> When running Cassandra and cqlsh with SSL, the following command occasionally
> results in the exception below:
> {code}
> cqlsh --ssl -f kv.cql
> {code}
> {code}
> ERROR [SharedPool-Worker-2] 2016-05-11 12:41:03,583 Message.java:538 -
> Unexpected exception during request; channel = [id: 0xeb75e05d,
> /127.0.0.1:51083 => /127.0.0.1:9042]
> io.netty.handler.codec.DecoderException: javax.net.ssl.SSLException: bad
> record MAC
> at
> io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:280)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:149)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.channel.epoll.EpollSocketChannel$EpollSocketUnsafe.epollInReady(EpollSocketChannel.java:722)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:326)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:264)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]
> Caused by: javax.net.ssl.SSLException: bad record MAC
> at sun.security.ssl.Alerts.getSSLException(Alerts.java:208)
> ~[na:1.8.0_91]
> at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1728)
> ~[na:1.8.0_91]
> at sun
[jira] [Commented] (CASSANDRA-11749) CQLSH gets SSL exception following a COPY FROM
[
https://issues.apache.org/jira/browse/CASSANDRA-11749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15308147#comment-15308147
]
Adam Holmberg commented on CASSANDRA-11749:
---
The driver should handle this better. There is fork detection, but it doesn't
go far enough when examined in the context of this issue. I created
[PYTHON-577|https://datastax-oss.atlassian.net/browse/PYTHON-577] to address
this. In the mean time, your approach of shutting down the parent cluster in
the child process should remove the problem without a race, provided the
shutdown is *before the child Cluster/Session is created*. This effectively
cleans up inherited sockets before the new child event loop is started.
> CQLSH gets SSL exception following a COPY FROM
> --
>
> Key: CASSANDRA-11749
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11749
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
>Reporter: Stefania
>Assignee: Stefania
> Fix For: 2.1.x
>
> Attachments: driver_debug.txt, stdout.txt.zip,
> stdout_single_process.txt.zip
>
>
> When running Cassandra and cqlsh with SSL, the following command occasionally
> results in the exception below:
> {code}
> cqlsh --ssl -f kv.cql
> {code}
> {code}
> ERROR [SharedPool-Worker-2] 2016-05-11 12:41:03,583 Message.java:538 -
> Unexpected exception during request; channel = [id: 0xeb75e05d,
> /127.0.0.1:51083 => /127.0.0.1:9042]
> io.netty.handler.codec.DecoderException: javax.net.ssl.SSLException: bad
> record MAC
> at
> io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:280)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:149)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.channel.epoll.EpollSocketChannel$EpollSocketUnsafe.epollInReady(EpollSocketChannel.java:722)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:326)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:264)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]
> Caused by: javax.net.ssl.SSLException: bad record MAC
> at sun.security.ssl.Alerts.getSSLException(Alerts.java:208)
> ~[na:1.8.0_91]
> at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1728)
> ~[na:1.8.0_91]
> at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:981)
> ~[na:1.8.0_91]
> at
> sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:907)
> ~[na:1.8.0_91]
> at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:781)
> ~[na:1.8.0_91]
> at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624) ~[na:1.8.0_91]
> at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:982)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:908)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:854)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:249)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> ... 10 common frames omitted
> Caused by: javax.crypto.BadPaddingException: bad record MAC
> at sun.security.ssl.InputRecord.decrypt(InputRecord.java:219)
> ~[na:1.8.0_91]
> at
> sun.security.ssl.EngineInputRecord.decrypt(EngineInputRecord.java:177)
> ~[na:1.8.0_91]
> at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:974)
> ~[na:1.8.0_91]
> ... 17 common frames omitted
> {code}
> where
> {code}
> cat kv.cql
> create keyspace if not exists cvs_copy_ks with replication = {'class':
> 'SimpleStrategy
[jira] [Commented] (CASSANDRA-11749) CQLSH gets SSL exception following a COPY FROM
[
https://issues.apache.org/jira/browse/CASSANDRA-11749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15303688#comment-15303688
]
Stefania commented on CASSANDRA-11749:
--
Thanks for the suggestions [~jjordan]:
1. This is only available with Python 3.4+ unfortunately (doc
[here|https://docs.python.org/3/library/multiprocessing.html#multiprocessing.set_start_method])
and also forking is much faster than spawing.
2. This seems a bit too complicated and wasteful if I understood it correctly,
unless we somehow use a fork server, which again is only available in Python
3.4, we need to know in advance how many child processes we need, which could
change depending on the copy command options.
3. This is the easiest and safest solution. I thought about it as well but it
seems wrong to me that we need to shut down the main cqlsh connection only to
reopen it later on, that's why I wanted [~aholmber]'s opinion. Maybe there is a
way to stop the control connection only, or maybe the child control connection
is trying to write to the socket as well, and this can be avoided by detecting
the fork. It does not show in the logs but maybe there aren't any logs printed
at all and hence I wanted confirmation first.
> CQLSH gets SSL exception following a COPY FROM
> --
>
> Key: CASSANDRA-11749
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11749
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
>Reporter: Stefania
>Assignee: Stefania
> Fix For: 2.1.x
>
> Attachments: driver_debug.txt, stdout.txt.zip,
> stdout_single_process.txt.zip
>
>
> When running Cassandra and cqlsh with SSL, the following command occasionally
> results in the exception below:
> {code}
> cqlsh --ssl -f kv.cql
> {code}
> {code}
> ERROR [SharedPool-Worker-2] 2016-05-11 12:41:03,583 Message.java:538 -
> Unexpected exception during request; channel = [id: 0xeb75e05d,
> /127.0.0.1:51083 => /127.0.0.1:9042]
> io.netty.handler.codec.DecoderException: javax.net.ssl.SSLException: bad
> record MAC
> at
> io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:280)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:149)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.channel.epoll.EpollSocketChannel$EpollSocketUnsafe.epollInReady(EpollSocketChannel.java:722)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:326)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:264)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]
> Caused by: javax.net.ssl.SSLException: bad record MAC
> at sun.security.ssl.Alerts.getSSLException(Alerts.java:208)
> ~[na:1.8.0_91]
> at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1728)
> ~[na:1.8.0_91]
> at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:981)
> ~[na:1.8.0_91]
> at
> sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:907)
> ~[na:1.8.0_91]
> at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:781)
> ~[na:1.8.0_91]
> at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624) ~[na:1.8.0_91]
> at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:982)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:908)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:854)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:249)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> ... 10 common frames omitted
> Cau
[jira] [Commented] (CASSANDRA-11749) CQLSH gets SSL exception following a COPY FROM
[
https://issues.apache.org/jira/browse/CASSANDRA-11749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15303660#comment-15303660
]
Jeremiah Jordan commented on CASSANDRA-11749:
-
[~Stefania] I can think of a couple ways.
1. Don't fork to do copy, make a new process
2. Fork right at startup before opening the connection
3. Shutdown the cluster object before forking to start new ones in both parent
and child.
Of the 3 I would probably think just doing shutdown, fork, connect is easiest.
> CQLSH gets SSL exception following a COPY FROM
> --
>
> Key: CASSANDRA-11749
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11749
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
>Reporter: Stefania
>Assignee: Stefania
> Fix For: 2.1.x
>
> Attachments: driver_debug.txt, stdout.txt.zip,
> stdout_single_process.txt.zip
>
>
> When running Cassandra and cqlsh with SSL, the following command occasionally
> results in the exception below:
> {code}
> cqlsh --ssl -f kv.cql
> {code}
> {code}
> ERROR [SharedPool-Worker-2] 2016-05-11 12:41:03,583 Message.java:538 -
> Unexpected exception during request; channel = [id: 0xeb75e05d,
> /127.0.0.1:51083 => /127.0.0.1:9042]
> io.netty.handler.codec.DecoderException: javax.net.ssl.SSLException: bad
> record MAC
> at
> io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:280)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:149)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.channel.epoll.EpollSocketChannel$EpollSocketUnsafe.epollInReady(EpollSocketChannel.java:722)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:326)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:264)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]
> Caused by: javax.net.ssl.SSLException: bad record MAC
> at sun.security.ssl.Alerts.getSSLException(Alerts.java:208)
> ~[na:1.8.0_91]
> at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1728)
> ~[na:1.8.0_91]
> at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:981)
> ~[na:1.8.0_91]
> at
> sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:907)
> ~[na:1.8.0_91]
> at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:781)
> ~[na:1.8.0_91]
> at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624) ~[na:1.8.0_91]
> at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:982)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:908)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:854)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:249)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> ... 10 common frames omitted
> Caused by: javax.crypto.BadPaddingException: bad record MAC
> at sun.security.ssl.InputRecord.decrypt(InputRecord.java:219)
> ~[na:1.8.0_91]
> at
> sun.security.ssl.EngineInputRecord.decrypt(EngineInputRecord.java:177)
> ~[na:1.8.0_91]
> at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:974)
> ~[na:1.8.0_91]
> ... 17 common frames omitted
> {code}
> where
> {code}
> cat kv.cql
> create keyspace if not exists cvs_copy_ks with replication = {'class':
> 'SimpleStrategy', 'replication_factor':1};
> create table if not exists cvs_copy_ks.kv (key int primary key, value text);
> truncate cvs_copy_ks.kv;
> copy cvs_copy_ks.kv (key, value) from 'kv.csv' with header='true';
> selec
[jira] [Commented] (CASSANDRA-11749) CQLSH gets SSL exception following a COPY FROM
[
https://issues.apache.org/jira/browse/CASSANDRA-11749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15303588#comment-15303588
]
Stefania commented on CASSANDRA-11749:
--
No worries, thanks for taking the time to look into this! :)
> CQLSH gets SSL exception following a COPY FROM
> --
>
> Key: CASSANDRA-11749
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11749
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
>Reporter: Stefania
>Assignee: Stefania
> Fix For: 2.1.x
>
> Attachments: driver_debug.txt, stdout.txt.zip,
> stdout_single_process.txt.zip
>
>
> When running Cassandra and cqlsh with SSL, the following command occasionally
> results in the exception below:
> {code}
> cqlsh --ssl -f kv.cql
> {code}
> {code}
> ERROR [SharedPool-Worker-2] 2016-05-11 12:41:03,583 Message.java:538 -
> Unexpected exception during request; channel = [id: 0xeb75e05d,
> /127.0.0.1:51083 => /127.0.0.1:9042]
> io.netty.handler.codec.DecoderException: javax.net.ssl.SSLException: bad
> record MAC
> at
> io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:280)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:149)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.channel.epoll.EpollSocketChannel$EpollSocketUnsafe.epollInReady(EpollSocketChannel.java:722)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:326)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:264)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]
> Caused by: javax.net.ssl.SSLException: bad record MAC
> at sun.security.ssl.Alerts.getSSLException(Alerts.java:208)
> ~[na:1.8.0_91]
> at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1728)
> ~[na:1.8.0_91]
> at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:981)
> ~[na:1.8.0_91]
> at
> sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:907)
> ~[na:1.8.0_91]
> at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:781)
> ~[na:1.8.0_91]
> at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624) ~[na:1.8.0_91]
> at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:982)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:908)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:854)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:249)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> ... 10 common frames omitted
> Caused by: javax.crypto.BadPaddingException: bad record MAC
> at sun.security.ssl.InputRecord.decrypt(InputRecord.java:219)
> ~[na:1.8.0_91]
> at
> sun.security.ssl.EngineInputRecord.decrypt(EngineInputRecord.java:177)
> ~[na:1.8.0_91]
> at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:974)
> ~[na:1.8.0_91]
> ... 17 common frames omitted
> {code}
> where
> {code}
> cat kv.cql
> create keyspace if not exists cvs_copy_ks with replication = {'class':
> 'SimpleStrategy', 'replication_factor':1};
> create table if not exists cvs_copy_ks.kv (key int primary key, value text);
> truncate cvs_copy_ks.kv;
> copy cvs_copy_ks.kv (key, value) from 'kv.csv' with header='true';
> select * from cvs_copy_ks.kv;
> drop keyspace cvs_copy_ks;
> stefi@cuoricina:~/git/cstar/cassandra$ cat kv.c
> kv.cql kv.csv
> cat kv.csv
> key,value
> 1,'a'
> 2,'b'
> 3,'c'
> {code}
> The COPY FROM succeeds, however the following select does not.
> The easiest way
[jira] [Commented] (CASSANDRA-11749) CQLSH gets SSL exception following a COPY FROM
[
https://issues.apache.org/jira/browse/CASSANDRA-11749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15303499#comment-15303499
]
Norman Maurer commented on CASSANDRA-11749:
---
[~Stefania] no problems... Sorry that it took me so long :(
> CQLSH gets SSL exception following a COPY FROM
> --
>
> Key: CASSANDRA-11749
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11749
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
>Reporter: Stefania
>Assignee: Stefania
> Fix For: 2.1.x
>
> Attachments: driver_debug.txt, stdout.txt.zip,
> stdout_single_process.txt.zip
>
>
> When running Cassandra and cqlsh with SSL, the following command occasionally
> results in the exception below:
> {code}
> cqlsh --ssl -f kv.cql
> {code}
> {code}
> ERROR [SharedPool-Worker-2] 2016-05-11 12:41:03,583 Message.java:538 -
> Unexpected exception during request; channel = [id: 0xeb75e05d,
> /127.0.0.1:51083 => /127.0.0.1:9042]
> io.netty.handler.codec.DecoderException: javax.net.ssl.SSLException: bad
> record MAC
> at
> io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:280)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:149)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.channel.epoll.EpollSocketChannel$EpollSocketUnsafe.epollInReady(EpollSocketChannel.java:722)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:326)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:264)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]
> Caused by: javax.net.ssl.SSLException: bad record MAC
> at sun.security.ssl.Alerts.getSSLException(Alerts.java:208)
> ~[na:1.8.0_91]
> at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1728)
> ~[na:1.8.0_91]
> at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:981)
> ~[na:1.8.0_91]
> at
> sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:907)
> ~[na:1.8.0_91]
> at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:781)
> ~[na:1.8.0_91]
> at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624) ~[na:1.8.0_91]
> at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:982)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:908)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:854)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:249)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> ... 10 common frames omitted
> Caused by: javax.crypto.BadPaddingException: bad record MAC
> at sun.security.ssl.InputRecord.decrypt(InputRecord.java:219)
> ~[na:1.8.0_91]
> at
> sun.security.ssl.EngineInputRecord.decrypt(EngineInputRecord.java:177)
> ~[na:1.8.0_91]
> at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:974)
> ~[na:1.8.0_91]
> ... 17 common frames omitted
> {code}
> where
> {code}
> cat kv.cql
> create keyspace if not exists cvs_copy_ks with replication = {'class':
> 'SimpleStrategy', 'replication_factor':1};
> create table if not exists cvs_copy_ks.kv (key int primary key, value text);
> truncate cvs_copy_ks.kv;
> copy cvs_copy_ks.kv (key, value) from 'kv.csv' with header='true';
> select * from cvs_copy_ks.kv;
> drop keyspace cvs_copy_ks;
> stefi@cuoricina:~/git/cstar/cassandra$ cat kv.c
> kv.cql kv.csv
> cat kv.csv
> key,value
> 1,'a'
> 2,'b'
> 3,'c'
> {code}
> The COPY FROM succeeds, however the following select does not.
> The eas
[jira] [Commented] (CASSANDRA-11749) CQLSH gets SSL exception following a COPY FROM
[
https://issues.apache.org/jira/browse/CASSANDRA-11749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15303485#comment-15303485
]
Stefania commented on CASSANDRA-11749:
--
Thanks for the pointer [~norman]: it's not a race amongst multiple threads but
something similar. There is an out-of-band connection that the clients create
so that the server can push notifications for things like schema changes and so
forth. The forked copy processes are inheriting the socket handle from the
parent cqlsh process. Releasing these socket handles in the forked processes
fixes it _provided it is done as soon as possible_.
[~aholmber]: I've fixed by calling {{cluster.shutdown()}} as soon as the child
processes start running but, I think we could still have a small window if a
notification is received before the sockets have been closed. I'm not sure I
understood the driver debug messages entirely, I've attached them in this file
here [^driver_debug.txt]. We can see {{\[control connection\] Waiting for
schema agreement}} logged by the parent process and then the bad record MAC
received by both parent and child process. Is there anything else that could be
done to avoid this race totally?
The fix in cqlsh copyutil is
[here|https://github.com/stef1927/cassandra/commit/3fb2e0c286f5f4ae42e1e877d7635604b1896a09].
> CQLSH gets SSL exception following a COPY FROM
> --
>
> Key: CASSANDRA-11749
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11749
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
>Reporter: Stefania
>Assignee: Stefania
> Fix For: 2.1.x
>
> Attachments: driver_debug.txt, stdout.txt.zip,
> stdout_single_process.txt.zip
>
>
> When running Cassandra and cqlsh with SSL, the following command occasionally
> results in the exception below:
> {code}
> cqlsh --ssl -f kv.cql
> {code}
> {code}
> ERROR [SharedPool-Worker-2] 2016-05-11 12:41:03,583 Message.java:538 -
> Unexpected exception during request; channel = [id: 0xeb75e05d,
> /127.0.0.1:51083 => /127.0.0.1:9042]
> io.netty.handler.codec.DecoderException: javax.net.ssl.SSLException: bad
> record MAC
> at
> io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:280)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:149)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.channel.epoll.EpollSocketChannel$EpollSocketUnsafe.epollInReady(EpollSocketChannel.java:722)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:326)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:264)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]
> Caused by: javax.net.ssl.SSLException: bad record MAC
> at sun.security.ssl.Alerts.getSSLException(Alerts.java:208)
> ~[na:1.8.0_91]
> at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1728)
> ~[na:1.8.0_91]
> at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:981)
> ~[na:1.8.0_91]
> at
> sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:907)
> ~[na:1.8.0_91]
> at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:781)
> ~[na:1.8.0_91]
> at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624) ~[na:1.8.0_91]
> at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:982)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:908)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:854)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.ja
[jira] [Commented] (CASSANDRA-11749) CQLSH gets SSL exception following a COPY FROM
[
https://issues.apache.org/jira/browse/CASSANDRA-11749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15302311#comment-15302311
]
Norman Maurer commented on CASSANDRA-11749:
---
[~jjordan] Only very briefly but my suspection so far is that it is actually a
race in cqlsh where two multiple threads write to the same connection
concurrently. Like one writes and the second starts to write as well before the
first is complete. Could this be possible ? This would also explain why a sleep
may "workaround" this.
> CQLSH gets SSL exception following a COPY FROM
> --
>
> Key: CASSANDRA-11749
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11749
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
>Reporter: Stefania
>Assignee: Stefania
> Fix For: 2.1.x
>
> Attachments: stdout.txt.zip, stdout_single_process.txt.zip
>
>
> When running Cassandra and cqlsh with SSL, the following command occasionally
> results in the exception below:
> {code}
> cqlsh --ssl -f kv.cql
> {code}
> {code}
> ERROR [SharedPool-Worker-2] 2016-05-11 12:41:03,583 Message.java:538 -
> Unexpected exception during request; channel = [id: 0xeb75e05d,
> /127.0.0.1:51083 => /127.0.0.1:9042]
> io.netty.handler.codec.DecoderException: javax.net.ssl.SSLException: bad
> record MAC
> at
> io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:280)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:149)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.channel.epoll.EpollSocketChannel$EpollSocketUnsafe.epollInReady(EpollSocketChannel.java:722)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:326)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:264)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]
> Caused by: javax.net.ssl.SSLException: bad record MAC
> at sun.security.ssl.Alerts.getSSLException(Alerts.java:208)
> ~[na:1.8.0_91]
> at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1728)
> ~[na:1.8.0_91]
> at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:981)
> ~[na:1.8.0_91]
> at
> sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:907)
> ~[na:1.8.0_91]
> at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:781)
> ~[na:1.8.0_91]
> at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624) ~[na:1.8.0_91]
> at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:982)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:908)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:854)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:249)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> ... 10 common frames omitted
> Caused by: javax.crypto.BadPaddingException: bad record MAC
> at sun.security.ssl.InputRecord.decrypt(InputRecord.java:219)
> ~[na:1.8.0_91]
> at
> sun.security.ssl.EngineInputRecord.decrypt(EngineInputRecord.java:177)
> ~[na:1.8.0_91]
> at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:974)
> ~[na:1.8.0_91]
> ... 17 common frames omitted
> {code}
> where
> {code}
> cat kv.cql
> create keyspace if not exists cvs_copy_ks with replication = {'class':
> 'SimpleStrategy', 'replication_factor':1};
> create table if not exists cvs_copy_ks.kv (key int primary key, value text);
> truncate cvs_copy_ks.kv;
> copy cvs_copy_ks.kv (key, value) from 'kv.csv' with header='true';
> select
[jira] [Commented] (CASSANDRA-11749) CQLSH gets SSL exception following a COPY FROM
[
https://issues.apache.org/jira/browse/CASSANDRA-11749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15302207#comment-15302207
]
Jeremiah Jordan commented on CASSANDRA-11749:
-
[~norman] have you had a chance to look into this at all?
[~Stefania] maybe we should put the hack in place for now?
> CQLSH gets SSL exception following a COPY FROM
> --
>
> Key: CASSANDRA-11749
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11749
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
>Reporter: Stefania
>Assignee: Stefania
> Fix For: 2.1.x
>
> Attachments: stdout.txt.zip, stdout_single_process.txt.zip
>
>
> When running Cassandra and cqlsh with SSL, the following command occasionally
> results in the exception below:
> {code}
> cqlsh --ssl -f kv.cql
> {code}
> {code}
> ERROR [SharedPool-Worker-2] 2016-05-11 12:41:03,583 Message.java:538 -
> Unexpected exception during request; channel = [id: 0xeb75e05d,
> /127.0.0.1:51083 => /127.0.0.1:9042]
> io.netty.handler.codec.DecoderException: javax.net.ssl.SSLException: bad
> record MAC
> at
> io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:280)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:149)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.channel.epoll.EpollSocketChannel$EpollSocketUnsafe.epollInReady(EpollSocketChannel.java:722)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:326)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:264)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]
> Caused by: javax.net.ssl.SSLException: bad record MAC
> at sun.security.ssl.Alerts.getSSLException(Alerts.java:208)
> ~[na:1.8.0_91]
> at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1728)
> ~[na:1.8.0_91]
> at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:981)
> ~[na:1.8.0_91]
> at
> sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:907)
> ~[na:1.8.0_91]
> at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:781)
> ~[na:1.8.0_91]
> at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624) ~[na:1.8.0_91]
> at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:982)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:908)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:854)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:249)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> ... 10 common frames omitted
> Caused by: javax.crypto.BadPaddingException: bad record MAC
> at sun.security.ssl.InputRecord.decrypt(InputRecord.java:219)
> ~[na:1.8.0_91]
> at
> sun.security.ssl.EngineInputRecord.decrypt(EngineInputRecord.java:177)
> ~[na:1.8.0_91]
> at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:974)
> ~[na:1.8.0_91]
> ... 17 common frames omitted
> {code}
> where
> {code}
> cat kv.cql
> create keyspace if not exists cvs_copy_ks with replication = {'class':
> 'SimpleStrategy', 'replication_factor':1};
> create table if not exists cvs_copy_ks.kv (key int primary key, value text);
> truncate cvs_copy_ks.kv;
> copy cvs_copy_ks.kv (key, value) from 'kv.csv' with header='true';
> select * from cvs_copy_ks.kv;
> drop keyspace cvs_copy_ks;
> stefi@cuoricina:~/git/cstar/cassandra$ cat kv.c
> kv.cql kv.csv
> cat kv.csv
> key,value
> 1,'a'
> 2,'b'
> 3,'c'
> {code}
> The COPY FROM succeeds, however
[jira] [Commented] (CASSANDRA-11749) CQLSH gets SSL exception following a COPY FROM
[
https://issues.apache.org/jira/browse/CASSANDRA-11749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15284053#comment-15284053
]
Stefania commented on CASSANDRA-11749:
--
Thank you Norman.
You can use [this
branch|https://github.com/apache/cassandra/compare/trunk...stef1927:11749-cqlsh-2.1].
It's configured to run Cassandra with SSL, it contains the test files and it
links to Netty 4.0.36 (rather than 4.0.23). It's otherwise identical to
cassandra-2.1 HEAD (the workaround mentioned above has been commented out).
Here are the instructions:
* Dependencies: Java 8 JDK, Python 2.7, ant 1.9+
* Get the branch: {{git clone http://github.com/stef1927/cassandra.git --branch
11749-cqlsh-2.1 --single-branch}}
* Set the {{CASSANDRA_DIR}} environment to the location of the branch and add
{{CASSANDRA_DIR/bin}} to the PATH
* Build: {{ant build}}
* If you need an IntelliJ project: {{ant generate-idea-files}} or Eclipse:
{{ant generate-eclipse-files}}
* Generate the certificates by following [these
instructions|http://docs.datastax.com/en/cassandra/2.1/cassandra/security/secureSSLCertificates_t.html].
These are the certificates you should end up with:
{code}
keystore.node0
node0.cer
node0.cer.pem
node0.key.pem
node0.p12
truststore.node0
{code}
* Edit {{$CASSANDRA_DIR/cqlshrc}} and {{$CASSANDRA_DIR/conf/cassandra.yaml}} to
point to your certificates. The easiest is to search for my absolute path
{{/home/stefi}} and change all occurrences. There are 3 occurrences in
{{cqlshrc}} and 2 in {{cassandra.yaml}}.
* Download the
[JCE|http://www.oracle.com/technetwork/java/javase/downloads/jce8-download-2133166.html],
unzip and copy the 2 jars to {{$JAVA_HOME/jre/lib/security/}}
* Set any additional JVM properties via the {{JVM_OPTS}} environment variable,
for example: {{export JVM_OPTS=-Djavax.net.debug=ssl}}
* Launch cassandra in the foreground: {{cassandra -f}}. Stop with CTRL-C.
* If you need to run in IntelliJ you can you the Cassandra run config.
* The log file containing the exception is {{CASSANDRA_DIR/logs/system.log}}
* Run the test with {{cqlsh --debug --ssl --cqlshrc=./conf/cqlshrc -f kv.cql}}
This is a sample output when it works:
{code}
Using CQL driver:
Using connect timeout: 5 seconds
Reading options from the command line: {'header': 'true', 'numprocesses': '1'}
Using options: '{'header': 'true', 'numprocesses': '1'}'
Using 1 child processes
Starting copy of cvs_copy_ks.kv with columns ['key', 'value'].
Closing queues...; Rate: 12 rows/s; Avg. rate: 12 rows/s
Processed: 3 rows; Rate: 6 rows/s; Avg. rate: 8 rows/s
3 rows imported from 1 files in 0.358 seconds (0 skipped).
key | value
-+---
1 | 'a'
2 | 'b'
3 | 'c'
(3 rows)
{code}
This is a sample output when it fails, plus the exception will be visible in
logs/system.log:
{code}
stefi@cuoricina:~/git/cstar/cassandra$ cqlsh --debug --ssl
--cqlshrc=./conf/cqlshrc -f kv.cql
Using CQL driver:
Using connect timeout: 5 seconds
Reading options from the command line: {'header': 'true', 'numprocesses': '1'}
Using options: '{'header': 'true', 'numprocesses': '1'}'
Using 1 child processes
Starting copy of cvs_copy_ks.kv with columns ['key', 'value'].
Closing queues...; Rate: 9 rows/s; Avg. rate: 9 rows/s
Processed: 3 rows; Rate: 4 rows/s; Avg. rate: 7 rows/s
3 rows imported from 1 files in 0.449 seconds (0 skipped).
kv.cql:6:NoHostAvailable: ('Unable to complete the operation against any
hosts', {})
kv.cql:7:NoHostAvailable: ('Unable to complete the operation against any
hosts', {})
{code}
You should be able to reproduce this fairly easily since the workaround has
been commented out. I typically run it 5 or 6 times before reproducing it. To
give you some context on the test, {{copy cvs_copy_ks.kv (key, value) from
'kv.csv' with header='true' and numprocesses=1;}} will spawn a Python child
process to import kv.csv into Cassandra. This command works but the two
following commands fail with {{NoHostAvailable}}, which indicate that the
server did not respond to cqlsh, plus we see the exception in the logs. You
also find a file called {{loop.sh}} if you want to run the test several times.
I hope I haven't forgotten any steps, if you run into trouble do not hesitate
to let me know. The instructions on generating certificates have a couple of
typos, unfortunately I did not save the exact commands I've used. You are
probably familiar with those commands but if not let me know and I'll recreate
the certificates and give you the exact commands.
> CQLSH gets SSL exception following a COPY FROM
> --
>
> Key: CASSANDRA-11749
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11749
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
>Reporter: Stefania
>
[jira] [Commented] (CASSANDRA-11749) CQLSH gets SSL exception following a COPY FROM
[
https://issues.apache.org/jira/browse/CASSANDRA-11749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15282697#comment-15282697
]
Norman Maurer commented on CASSANDRA-11749:
---
[~Stefania] if you can give me "step-by-step" way to reproduce this in my
laptop I will have a look and try to figure out what is wrong
> CQLSH gets SSL exception following a COPY FROM
> --
>
> Key: CASSANDRA-11749
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11749
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
>Reporter: Stefania
>Assignee: Stefania
> Fix For: 2.1.x
>
> Attachments: stdout.txt.zip, stdout_single_process.txt.zip
>
>
> When running Cassandra and cqlsh with SSL, the following command occasionally
> results in the exception below:
> {code}
> cqlsh --ssl -f kv.cql
> {code}
> {code}
> ERROR [SharedPool-Worker-2] 2016-05-11 12:41:03,583 Message.java:538 -
> Unexpected exception during request; channel = [id: 0xeb75e05d,
> /127.0.0.1:51083 => /127.0.0.1:9042]
> io.netty.handler.codec.DecoderException: javax.net.ssl.SSLException: bad
> record MAC
> at
> io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:280)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:149)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.channel.epoll.EpollSocketChannel$EpollSocketUnsafe.epollInReady(EpollSocketChannel.java:722)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:326)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:264)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]
> Caused by: javax.net.ssl.SSLException: bad record MAC
> at sun.security.ssl.Alerts.getSSLException(Alerts.java:208)
> ~[na:1.8.0_91]
> at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1728)
> ~[na:1.8.0_91]
> at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:981)
> ~[na:1.8.0_91]
> at
> sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:907)
> ~[na:1.8.0_91]
> at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:781)
> ~[na:1.8.0_91]
> at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624) ~[na:1.8.0_91]
> at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:982)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:908)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:854)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:249)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> ... 10 common frames omitted
> Caused by: javax.crypto.BadPaddingException: bad record MAC
> at sun.security.ssl.InputRecord.decrypt(InputRecord.java:219)
> ~[na:1.8.0_91]
> at
> sun.security.ssl.EngineInputRecord.decrypt(EngineInputRecord.java:177)
> ~[na:1.8.0_91]
> at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:974)
> ~[na:1.8.0_91]
> ... 17 common frames omitted
> {code}
> where
> {code}
> cat kv.cql
> create keyspace if not exists cvs_copy_ks with replication = {'class':
> 'SimpleStrategy', 'replication_factor':1};
> create table if not exists cvs_copy_ks.kv (key int primary key, value text);
> truncate cvs_copy_ks.kv;
> copy cvs_copy_ks.kv (key, value) from 'kv.csv' with header='true';
> select * from cvs_copy_ks.kv;
> drop keyspace cvs_copy_ks;
> stefi@cuoricina:~/git/cstar/cassandra$ cat kv.c
> kv.cql kv.csv
> cat kv.csv
> key,value
> 1,'a'
> 2,'b'
> 3,'c'
> {code}
> The COPY FROM suc
[jira] [Commented] (CASSANDRA-11749) CQLSH gets SSL exception following a COPY FROM
[
https://issues.apache.org/jira/browse/CASSANDRA-11749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15281206#comment-15281206
]
Stefania commented on CASSANDRA-11749:
--
Thank you for the information Sam.
The Android netty issue is similar but, if there is a problem with Netty, it
must be for an entirely different reason. So far I've reproduced this exception
with Netty 4.0.23 and 4.0.36 using JDK 1.8.0_91 and various versions of
Python/OpenSSL (Python 2.7.6 with openssl 1.0.1f-1ubuntu2.19, which is the
default on Ubuntu Trusty, Python 2.7.10 custom built with openssl 1.0.2h or
1.0.2g). I know that Datastax have also reproduced it with JDK 1.8.0_40 and
netty 4.0.34. I've also verified that the exception exists with SSLv3 as well,
so it is not a TLS bug.
Adding a pause of 1 second between the end of the COPY FROM and the next SELECT
statement makes the exception disappear.
Reducing the pause makes the exception gradually easier to reproduce. Without
any pause the exception occurs once every 10 times approximately. Therefore I
suspect a concurrency bug server side, possibly in Netty. I think the closing
of the SSL connections from the COPY FROM Python processes somehow overlap with
the main session from the cqlsh SELECT. A single COPY FROM process is
sufficient to reproduce the exception, and in fact makes it easier to debug
server side (attached [^stdout_single_process.txt.zip]).
If [~norman] has any suggestions I can test them or I can provide further debug
information by adding debug statements to the Netty SSLHandler.
If we cannot understand what the problem is, as a workaround we can introduce a
pause after a COPY FROM when using SSL.
> CQLSH gets SSL exception following a COPY FROM
> --
>
> Key: CASSANDRA-11749
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11749
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
>Reporter: Stefania
>Assignee: Stefania
> Fix For: 2.1.x
>
> Attachments: stdout.txt.zip, stdout_single_process.txt.zip
>
>
> When running Cassandra and cqlsh with SSL, the following command occasionally
> results in the exception below:
> {code}
> cqlsh --ssl -f kv.cql
> {code}
> {code}
> ERROR [SharedPool-Worker-2] 2016-05-11 12:41:03,583 Message.java:538 -
> Unexpected exception during request; channel = [id: 0xeb75e05d,
> /127.0.0.1:51083 => /127.0.0.1:9042]
> io.netty.handler.codec.DecoderException: javax.net.ssl.SSLException: bad
> record MAC
> at
> io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:280)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:149)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.channel.epoll.EpollSocketChannel$EpollSocketUnsafe.epollInReady(EpollSocketChannel.java:722)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:326)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:264)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]
> Caused by: javax.net.ssl.SSLException: bad record MAC
> at sun.security.ssl.Alerts.getSSLException(Alerts.java:208)
> ~[na:1.8.0_91]
> at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1728)
> ~[na:1.8.0_91]
> at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:981)
> ~[na:1.8.0_91]
> at
> sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:907)
> ~[na:1.8.0_91]
> at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:781)
> ~[na:1.8.0_91]
> at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624) ~[na:1.8.0_91]
> at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:982)
> ~[netty-all-4.0.23.Final.jar
[jira] [Commented] (CASSANDRA-11749) CQLSH gets SSL exception following a COPY FROM
[
https://issues.apache.org/jira/browse/CASSANDRA-11749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15280220#comment-15280220
]
Sam Tunnicliffe commented on CASSANDRA-11749:
-
I don't I'm afraid & as far as I am aware, {{bad_record_mac}} is basically a
catch-all for any error with the encryption handshake. I did find a [netty
issue|https://github.com/netty/netty/issues/4116] presenting similar symptoms,
but it was only affecting Android clients. Perhaps [~norman] has more of an
idea?
One other thing, in {{stdout.txt}} there do seem to be quite a few warnings
prior to that too which suggest the client is shutting down the connection
prematurely, but I don't know if that's at all relevant:
{code}
epollEventLoopGroup-2-4, fatal error: 80: Inbound closed before receiving
peer's close_notify: possible truncation attack?
javax.net.ssl.SSLException: Inbound closed before receiving peer's
close_notify: possible truncation attack?
{code}
> CQLSH gets SSL exception following a COPY FROM
> --
>
> Key: CASSANDRA-11749
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11749
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
>Reporter: Stefania
>Assignee: Stefania
> Fix For: 2.1.x
>
> Attachments: stdout.txt.zip
>
>
> When running Cassandra and cqlsh with SSL, the following command occasionally
> results in the exception below:
> {code}
> cqlsh --ssl -f kv.cql
> {code}
> {code}
> ERROR [SharedPool-Worker-2] 2016-05-11 12:41:03,583 Message.java:538 -
> Unexpected exception during request; channel = [id: 0xeb75e05d,
> /127.0.0.1:51083 => /127.0.0.1:9042]
> io.netty.handler.codec.DecoderException: javax.net.ssl.SSLException: bad
> record MAC
> at
> io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:280)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:149)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.channel.epoll.EpollSocketChannel$EpollSocketUnsafe.epollInReady(EpollSocketChannel.java:722)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:326)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:264)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]
> Caused by: javax.net.ssl.SSLException: bad record MAC
> at sun.security.ssl.Alerts.getSSLException(Alerts.java:208)
> ~[na:1.8.0_91]
> at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1728)
> ~[na:1.8.0_91]
> at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:981)
> ~[na:1.8.0_91]
> at
> sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:907)
> ~[na:1.8.0_91]
> at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:781)
> ~[na:1.8.0_91]
> at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624) ~[na:1.8.0_91]
> at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:982)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:908)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:854)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:249)
> ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
> ... 10 common frames omitted
> Caused by: javax.crypto.BadPaddingException: bad record MAC
> at sun.security.ssl.InputRecord.decrypt(InputRecord.java:219)
> ~[na:1.8.0_91]
> at
> sun.security.ssl.EngineInputRecord.decrypt(EngineInputRecord.java:177)
> ~[na:1.8.0_91]
> at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineIm
