[jira] [Commented] (CASSANDRA-13964) Tracing interferes with digest requests when using RandomPartitioner
[
https://issues.apache.org/jira/browse/CASSANDRA-13964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16209343#comment-16209343
]
Jason Brown commented on CASSANDRA-13964:
-
I'm +1 on the patch and dtest. I ran the dtest w/o the patch can confirmed the
failure, then with and confirmed success.
The only minor nit I have is on the dtest where you changed the {{prepare}}
function. Maybe I'm missing something, but couldn't
{code}
if not random_partitioner:
cluster.set_partitioner("org.apache.cassandra.dht.Murmur3Partitioner")
else:
if random_partitioner:
cluster.set_partitioner("org.apache.cassandra.dht.RandomPartitioner")
else:
cluster.set_partitioner("org.apache.cassandra.dht.Murmur3Partitioner")
{code}
be simplified as
{code}
if random_partitioner:
cluster.set_partitioner("org.apache.cassandra.dht.RandomPartitioner")
else:
cluster.set_partitioner("org.apache.cassandra.dht.Murmur3Partitioner")
{code}
wrt branches to commit on, for sure we need this on 3.0 and 3.11. We also need
it on trunk, but should wait to see what happens with CASSANDRA-13291?
> Tracing interferes with digest requests when using RandomPartitioner
>
>
> Key: CASSANDRA-13964
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13964
> Project: Cassandra
> Issue Type: Bug
> Components: Local Write-Read Paths, Observability
>Reporter: Sam Tunnicliffe
>Assignee: Sam Tunnicliffe
>
> A {{ThreadLocal}} is used to generate the MD5 digest when a
> replica serves a read command and the {{isDigestQuery}} flag is set. The same
> threadlocal is also used by {{RandomPartitioner}} to decorate partition keys.
> So in a cluster with RP, if tracing is enabled the data digest is corrupted
> by the partitioner making tokens for the tracing mutations. This causes a
> digest mismatch on the coordinator, triggering a full data read on every read
> where CL > 1 (or speculative execution/read repair kick in).
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-13964) Tracing interferes with digest requests when using RandomPartitioner
[
https://issues.apache.org/jira/browse/CASSANDRA-13964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Brown updated CASSANDRA-13964:
Reproduced In: 3.11.1, 3.0.15, 4.0 (was: 3.0.15, 3.11.1, 4.0)
Reviewer: Jason Brown
> Tracing interferes with digest requests when using RandomPartitioner
>
>
> Key: CASSANDRA-13964
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13964
> Project: Cassandra
> Issue Type: Bug
> Components: Local Write-Read Paths, Observability
>Reporter: Sam Tunnicliffe
>Assignee: Sam Tunnicliffe
>
> A {{ThreadLocal}} is used to generate the MD5 digest when a
> replica serves a read command and the {{isDigestQuery}} flag is set. The same
> threadlocal is also used by {{RandomPartitioner}} to decorate partition keys.
> So in a cluster with RP, if tracing is enabled the data digest is corrupted
> by the partitioner making tokens for the tracing mutations. This causes a
> digest mismatch on the coordinator, triggering a full data read on every read
> where CL > 1 (or speculative execution/read repair kick in).
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-13902) document memtable_flush_period_in_ms
[ https://issues.apache.org/jira/browse/CASSANDRA-13902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Brown updated CASSANDRA-13902: Status: Ready to Commit (was: Patch Available) > document memtable_flush_period_in_ms > > > Key: CASSANDRA-13902 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13902 > Project: Cassandra > Issue Type: Improvement > Components: Documentation and Website >Reporter: Jon Haddad >Assignee: Jon Haddad > Labels: lhf > Fix For: 4.x > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13291) Replace usages of MessageDigest with Guava's Hasher
[
https://issues.apache.org/jira/browse/CASSANDRA-13291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16209259#comment-16209259
]
Jason Brown commented on CASSANDRA-13291:
-
[~beobal] Hmm, you are correct, and I can repro easily. With a two node ccm
cluster, I got this error in node1:
{noformat}
org.apache.cassandra.exceptions.ConfigurationException: Token must be >= 0 and
<= 2**127
at
org.apache.cassandra.dht.RandomPartitioner$2.validate(RandomPartitioner.java:160)
~[main/:na]
at
org.apache.cassandra.config.DatabaseDescriptor.applyInitialTokens(DatabaseDescriptor.java:927)
~[main/:na]
at
org.apache.cassandra.config.DatabaseDescriptor.applyAll(DatabaseDescriptor.java:317)
~[main/:na]
at
org.apache.cassandra.config.DatabaseDescriptor.daemonInitialization(DatabaseDescriptor.java:141)
~[main/:na]
at
org.apache.cassandra.service.CassandraDaemon.applyConfig(CassandraDaemon.java:602)
[main/:na]
at
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:537)
[main/:na]
at
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:646)
[main/:na]
{noformat}
And this error in node2, which got a bit further in the boot sequence:
{noformat}
ERROR [main] 2017-10-18 05:37:18,547 CassandraDaemon.java:663 - Exception
encountered during startup
org.apache.cassandra.exceptions.InvalidRequestException: Key may not be empty
at
org.apache.cassandra.cql3.QueryProcessor.validateKey(QueryProcessor.java:186)
~[main/:na]
at
org.apache.cassandra.cql3.statements.ModificationStatement.buildPartitionKeyNames(ModificationStatement.java:310)
~[main/:na]
at
org.apache.cassandra.cql3.statements.ModificationStatement.addUpdates(ModificationStatement.java:656)
~[main/:na]
at
org.apache.cassandra.cql3.statements.ModificationStatement.getMutations(ModificationStatement.java:644)
~[main/:na]
at
org.apache.cassandra.cql3.statements.ModificationStatement.executeInternalWithoutCondition(ModificationStatement.java:595)
~[main/:na]
at
org.apache.cassandra.cql3.statements.ModificationStatement.executeInternal(ModificationStatement.java:590)
~[main/:na]
at
org.apache.cassandra.cql3.QueryProcessor.executeInternal(QueryProcessor.java:296)
~[main/:na]
at
org.apache.cassandra.db.SystemKeyspace.updateSchemaVersion(SystemKeyspace.java:648)
~[main/:na]
at org.apache.cassandra.schema.Schema.updateVersion(Schema.java:533)
~[main/:na]
at
org.apache.cassandra.schema.Schema.updateVersionAndAnnounce(Schema.java:541)
~[main/:na]
at
org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:787)
~[main/:na]
at
org.apache.cassandra.service.StorageService.initServer(StorageService.java:622)
~[main/:na]
at
org.apache.cassandra.service.StorageService.initServer(StorageService.java:569)
~[main/:na]
at
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:364)
[main/:na]
at
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:557)
[main/:na]
at
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:646)
[main/:na]
{noformat}
> Replace usages of MessageDigest with Guava's Hasher
> ---
>
> Key: CASSANDRA-13291
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13291
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
>Reporter: Michael Kjellman
>Assignee: Michael Kjellman
> Attachments: CASSANDRA-13291-trunk.diff
>
>
> During my profiling of C* I frequently see lots of aggregate time across
> threads being spent inside the MD5 MessageDigest implementation. Given that
> there are tons of modern alternative hashing functions better than MD5
> available -- both in terms of providing better collision resistance and
> actual computational speed -- I wanted to switch out our usage of MD5 for
> alternatives (like adler128 or murmur3_128) and test for performance
> improvements.
> Unfortunately, I found given the fact we use MessageDigest everywhere --
> switching out the hashing function to something like adler128 or murmur3_128
> (for example) -- which don't ship with the JDK -- wasn't straight forward.
> The goal of this ticket is to propose switching out usages of MessageDigest
> directly in favor of Hasher from Guava. This means going forward we can
> change a single line of code to switch the hashing algorithm being used
> (assuming there is an implementation in Guava).
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional c
[jira] [Commented] (CASSANDRA-13915) Create a Docker container to build the docs
[
https://issues.apache.org/jira/browse/CASSANDRA-13915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16209247#comment-16209247
]
Jason Brown commented on CASSANDRA-13915:
-
[~rustyrazorblade] I thought you were going to investigate some stuff, as
[~j.casares] said:
bq. As far as Mac vs Linux differences, I'll leave that to Jon Haddad to debug
and send me anything he finds, if you wouldn't mind? If it's a permissions
issue, send me to the file you're having issues with and I'll make necessary
Docker changes since I'm on OSX right now. Other than permissions issues, I've
never seen a difference between OSX and Linux containers and the container is
currently running as the root user, so there shouldn't be any issues that I can
think of.
wrt the {{sudo ant realclean}}, it's kind of a PITA, but not insurmountable.
[~mshuler] Will those linux file permissions (that I mention) be a problem for
the build system(s)?
> Create a Docker container to build the docs
> ---
>
> Key: CASSANDRA-13915
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13915
> Project: Cassandra
> Issue Type: Improvement
>Reporter: Joaquin Casares
>Assignee: Joaquin Casares
>
> As requested by [~rustyrazorblade], I will be adding a Docker container to
> build the docs without any prereqs (other than Docker).
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13902) document memtable_flush_period_in_ms
[ https://issues.apache.org/jira/browse/CASSANDRA-13902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16209233#comment-16209233 ] Jason Brown commented on CASSANDRA-13902: - +1 > document memtable_flush_period_in_ms > > > Key: CASSANDRA-13902 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13902 > Project: Cassandra > Issue Type: Improvement > Components: Documentation and Website >Reporter: Jon Haddad >Assignee: Jon Haddad > Labels: lhf > Fix For: 4.x > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-13902) document memtable_flush_period_in_ms
[ https://issues.apache.org/jira/browse/CASSANDRA-13902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Brown updated CASSANDRA-13902: Reviewer: Jason Brown > document memtable_flush_period_in_ms > > > Key: CASSANDRA-13902 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13902 > Project: Cassandra > Issue Type: Improvement > Components: Documentation and Website >Reporter: Jon Haddad >Assignee: Jon Haddad > Labels: lhf > Fix For: 4.x > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13291) Replace usages of MessageDigest with Guava's Hasher
[ https://issues.apache.org/jira/browse/CASSANDRA-13291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16208858#comment-16208858 ] Jason Brown commented on CASSANDRA-13291: - bq. the reason we got here is because of the fact that the ultimate goal here (to switch the hashing function we're using) ... ok, sgtm Beyond that, I'm +1. If the tests all look legit, I'll commit tomorrow. > Replace usages of MessageDigest with Guava's Hasher > --- > > Key: CASSANDRA-13291 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13291 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Michael Kjellman >Assignee: Michael Kjellman > Attachments: CASSANDRA-13291-trunk.diff > > > During my profiling of C* I frequently see lots of aggregate time across > threads being spent inside the MD5 MessageDigest implementation. Given that > there are tons of modern alternative hashing functions better than MD5 > available -- both in terms of providing better collision resistance and > actual computational speed -- I wanted to switch out our usage of MD5 for > alternatives (like adler128 or murmur3_128) and test for performance > improvements. > Unfortunately, I found given the fact we use MessageDigest everywhere -- > switching out the hashing function to something like adler128 or murmur3_128 > (for example) -- which don't ship with the JDK -- wasn't straight forward. > The goal of this ticket is to propose switching out usages of MessageDigest > directly in favor of Hasher from Guava. This means going forward we can > change a single line of code to switch the hashing algorithm being used > (assuming there is an implementation in Guava). -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13291) Replace usages of MessageDigest with Guava's Hasher
[
https://issues.apache.org/jira/browse/CASSANDRA-13291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16208846#comment-16208846
]
Jason Brown commented on CASSANDRA-13291:
-
[~beobal] Thanks for doing the hard part of this review!
bq. if we want to go back to just using the old thread local here or not
I think the {{Hasher}} instance will live so briefly (just the scope of the
{{#hashToBigInteger()}} function) it's probably not worth mucking with the
{{ThreadLocal}} anymore.
trivial/petty nits:
- In about 2-3 places in the non-test code, you call
{{HashingUtils.CURRENT_HASH_FUNCTION#newHasher()}}. Part of me wants to hide
away the {{CURRENT_HASH_FUNCTION}} and just have a function
{{HashingUtils#newHasher()}}, which knows to use the {{CURRENT_HASH_FUNCTION}}.
wdyt? (not a blocker for +1'ing, btw)
- in {{RandomPartitioner}}, {{#hashToBigInteger()}} is called four times from
within the same class, and only from this class. Can we a) make the function
private, and b) instead of {{RandomPartitioner#hashToBigInteger()}} can we just
call {{#hashToBigInteger()}}?
- I'm 99% sure {{Validator.CountingHasher#putObject()}} is not being called,
and it's just there to fill out the interface. Can you add a quick note that
we're not expecting it to be be used, and thus not counting the bytes?
- Once again on the "petty, but let's make it correct" front,
{{Validator.CountingHasher#putUnencodedChars()}} and
{{Validator.CountingHasher#putString()}} both call {{CharSequence#length()}} to
get the value to add to the {{count}}. According to the javadoc for
{{CharSequence#length()}} reads:
{code}
Returns the length of this character sequence. The length is the number of
16-bit chars in the sequence.
{code}
At a minimum we should multiply the output of {{CharSequence#length()}} by 2 to
get the number of 8-bit bytes to correctly increment {{count}}; we should also
add a small comment, as well.
- {{GuidGenerator}} - remove import of {{RandomParitioner}}
I think we're almost there!
> Replace usages of MessageDigest with Guava's Hasher
> ---
>
> Key: CASSANDRA-13291
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13291
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
>Reporter: Michael Kjellman
>Assignee: Michael Kjellman
> Attachments: CASSANDRA-13291-trunk.diff
>
>
> During my profiling of C* I frequently see lots of aggregate time across
> threads being spent inside the MD5 MessageDigest implementation. Given that
> there are tons of modern alternative hashing functions better than MD5
> available -- both in terms of providing better collision resistance and
> actual computational speed -- I wanted to switch out our usage of MD5 for
> alternatives (like adler128 or murmur3_128) and test for performance
> improvements.
> Unfortunately, I found given the fact we use MessageDigest everywhere --
> switching out the hashing function to something like adler128 or murmur3_128
> (for example) -- which don't ship with the JDK -- wasn't straight forward.
> The goal of this ticket is to propose switching out usages of MessageDigest
> directly in favor of Hasher from Guava. This means going forward we can
> change a single line of code to switch the hashing algorithm being used
> (assuming there is an implementation in Guava).
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13291) Replace usages of MessageDigest with Guava's Hasher
[ https://issues.apache.org/jira/browse/CASSANDRA-13291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16208811#comment-16208811 ] Jason Brown commented on CASSANDRA-13291: - Pulled [~mkjellman]'s patch into a fresh branch off of trunk, and pushed here for running the tests: ||13291|| |[branch|https://github.com/jasobrown/cassandra/tree/13291]| |[dtest|https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-dtest/378/]| |[utests|https://circleci.com/gh/jasobrown/cassandra/tree/13291]| > Replace usages of MessageDigest with Guava's Hasher > --- > > Key: CASSANDRA-13291 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13291 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Michael Kjellman >Assignee: Michael Kjellman > Attachments: CASSANDRA-13291-trunk.diff > > > During my profiling of C* I frequently see lots of aggregate time across > threads being spent inside the MD5 MessageDigest implementation. Given that > there are tons of modern alternative hashing functions better than MD5 > available -- both in terms of providing better collision resistance and > actual computational speed -- I wanted to switch out our usage of MD5 for > alternatives (like adler128 or murmur3_128) and test for performance > improvements. > Unfortunately, I found given the fact we use MessageDigest everywhere -- > switching out the hashing function to something like adler128 or murmur3_128 > (for example) -- which don't ship with the JDK -- wasn't straight forward. > The goal of this ticket is to propose switching out usages of MessageDigest > directly in favor of Hasher from Guava. This means going forward we can > change a single line of code to switch the hashing algorithm being used > (assuming there is an implementation in Guava). -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-13291) Replace usages of MessageDigest with Guava's Hasher
[ https://issues.apache.org/jira/browse/CASSANDRA-13291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16208811#comment-16208811 ] Jason Brown edited comment on CASSANDRA-13291 at 10/18/17 4:57 AM: --- Pulled [~mkjellman]'s V2 patch into a fresh branch off of trunk, and pushed here for running the tests: ||13291|| |[branch|https://github.com/jasobrown/cassandra/tree/13291]| |[dtest|https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-dtest/378/]| |[utests|https://circleci.com/gh/jasobrown/cassandra/tree/13291]| was (Author: jasobrown): Pulled [~mkjellman]'s patch into a fresh branch off of trunk, and pushed here for running the tests: ||13291|| |[branch|https://github.com/jasobrown/cassandra/tree/13291]| |[dtest|https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-dtest/378/]| |[utests|https://circleci.com/gh/jasobrown/cassandra/tree/13291]| > Replace usages of MessageDigest with Guava's Hasher > --- > > Key: CASSANDRA-13291 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13291 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Michael Kjellman >Assignee: Michael Kjellman > Attachments: CASSANDRA-13291-trunk.diff > > > During my profiling of C* I frequently see lots of aggregate time across > threads being spent inside the MD5 MessageDigest implementation. Given that > there are tons of modern alternative hashing functions better than MD5 > available -- both in terms of providing better collision resistance and > actual computational speed -- I wanted to switch out our usage of MD5 for > alternatives (like adler128 or murmur3_128) and test for performance > improvements. > Unfortunately, I found given the fact we use MessageDigest everywhere -- > switching out the hashing function to something like adler128 or murmur3_128 > (for example) -- which don't ship with the JDK -- wasn't straight forward. > The goal of this ticket is to propose switching out usages of MessageDigest > directly in favor of Hasher from Guava. This means going forward we can > change a single line of code to switch the hashing algorithm being used > (assuming there is an implementation in Guava). -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-13291) Replace usages of MessageDigest with Guava's Hasher
[ https://issues.apache.org/jira/browse/CASSANDRA-13291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Brown updated CASSANDRA-13291: Reviewer: Jason Brown (was: Robert Stupp) > Replace usages of MessageDigest with Guava's Hasher > --- > > Key: CASSANDRA-13291 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13291 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Michael Kjellman >Assignee: Michael Kjellman > Attachments: CASSANDRA-13291-trunk.diff > > > During my profiling of C* I frequently see lots of aggregate time across > threads being spent inside the MD5 MessageDigest implementation. Given that > there are tons of modern alternative hashing functions better than MD5 > available -- both in terms of providing better collision resistance and > actual computational speed -- I wanted to switch out our usage of MD5 for > alternatives (like adler128 or murmur3_128) and test for performance > improvements. > Unfortunately, I found given the fact we use MessageDigest everywhere -- > switching out the hashing function to something like adler128 or murmur3_128 > (for example) -- which don't ship with the JDK -- wasn't straight forward. > The goal of this ticket is to propose switching out usages of MessageDigest > directly in favor of Hasher from Guava. This means going forward we can > change a single line of code to switch the hashing algorithm being used > (assuming there is an implementation in Guava). -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-13949) java.lang.ArrayIndexOutOfBoundsException while executing query
[
https://issues.apache.org/jira/browse/CASSANDRA-13949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Brown updated CASSANDRA-13949:
Resolution: Fixed
Fix Version/s: (was: 3.11.x)
(was: 4.x)
4.0
3.11.2
Status: Resolved (was: Patch Available)
committed as sha {{bbda20155ae0f3443cbb5fee0659234e81b2e914}}.
Thanks!
> java.lang.ArrayIndexOutOfBoundsException while executing query
> --
>
> Key: CASSANDRA-13949
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13949
> Project: Cassandra
> Issue Type: Bug
> Components: CQL
> Environment: Setup of 3 servers y using docker image
> [https://github.com/docker-library/cassandra/blob/ca3c9df03cab318d34377bba0610c741253b0466/3.11/Dockerfile]
>Reporter: Luis E Rodriguez Pupo
>Assignee: Jason Brown
> Fix For: 3.11.2, 4.0
>
> Attachments: 13949.png, insert.cql, query.cql, schema.cql
>
>
> While executing a query on a table contaninig a field with a (escaped) json,
> the following exception occurs:
> java.lang.ArrayIndexOutOfBoundsException: null
> at
> org.codehaus.jackson.io.JsonStringEncoder.quoteAsString(JsonStringEncoder.java:141)
> ~[jackson-core-asl-1.9.2.jar:1.9.2]
> at org.apache.cassandra.cql3.Json.quoteAsJsonString(Json.java:45)
> ~[apache-cassandra-3.11.0.jar:3.11.0]
> at
> org.apache.cassandra.db.marshal.UTF8Type.toJSONString(UTF8Type.java:66)
> ~[apache-cassandra-3.11.0.jar:3.11.0]
> at
> org.apache.cassandra.cql3.selection.Selection.rowToJson(Selection.java:291)
> ~[apache-cassandra-3.11.0.jar:3.11.0]
> at
> org.apache.cassandra.cql3.selection.Selection$ResultSetBuilder.getOutputRow(Selection.java:431)
> ~[apache-cassandra-3.11.0.jar:3.11.0]
> at
> org.apache.cassandra.cql3.selection.Selection$ResultSetBuilder.build(Selection.java:417)
> ~[apache-cassandra-3.11.0.jar:3.11.0]
> at
> org.apache.cassandra.cql3.statements.SelectStatement.process(SelectStatement.java:763)
> ~[apache-cassandra-3.11.0.jar:3.11.0]
> at
> org.apache.cassandra.cql3.statements.SelectStatement.processResults(SelectStatement.java:400)
> ~[apache-cassandra-3.11.0.jar:3.11.0]
> at
> org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:378)
> ~[apache-cassandra-3.11.0.jar:3.11.0]
> at
> org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:251)
> ~[apache-cassandra-3.11.0.jar:3.11.0]
> at
> org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:79)
> ~[apache-cassandra-3.11.0.jar:3.11.0]
> at
> org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:217)
> ~[apache-cassandra-3.11.0.jar:3.11.0]
> at
> org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:248)
> ~[apache-cassandra-3.11.0.jar:3.11.0]
> at
> org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:233)
> ~[apache-cassandra-3.11.0.jar:3.11.0]
> at
> org.apache.cassandra.transport.messages.QueryMessage.execute(QueryMessage.java:116)
> ~[apache-cassandra-3.11.0.jar:3.11.0]
> at
> org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:517)
> [apache-cassandra-3.11.0.jar:3.11.0]
> at
> org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:410)
> [apache-cassandra-3.11.0.jar:3.11.0]
> at
> io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
> [netty-all-4.0.44.Final.jar:4.0.44.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:357)
> [netty-all-4.0.44.Final.jar:4.0.44.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:35)
> [netty-all-4.0.44.Final.jar:4.0.44.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:348)
> [netty-all-4.0.44.Final.jar:4.0.44.Final]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [na:1.8.0_131]
> at
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:162)
> [apache-cassandra-3.11.0.jar:3.11.0]
> at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:109)
> [apache-cassandra-3.11.0.jar:3.11.0]
> at java.lang.Thread.run(Thread.java:748) [na:1.8.0_131]
> Find attached the schema of the table, the insertion query with the data
> provoking the failure, and the failing query.
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
[jira] [Commented] (CASSANDRA-13949) java.lang.ArrayIndexOutOfBoundsException while executing query
[ https://issues.apache.org/jira/browse/CASSANDRA-13949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16207616#comment-16207616 ] Jason Brown commented on CASSANDRA-13949: - I ran the [dtests again|https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-dtest/374] for trunk, and the results are still a bitt messy. I compared with the trunk dtests on apache jenkins and cassci, and they are also having problems on the paging_tests and write_failure_tests. Thus I think this patch is probably safe to apply. Committing shortly. > java.lang.ArrayIndexOutOfBoundsException while executing query > -- > > Key: CASSANDRA-13949 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13949 > Project: Cassandra > Issue Type: Bug > Components: CQL > Environment: Setup of 3 servers y using docker image > [https://github.com/docker-library/cassandra/blob/ca3c9df03cab318d34377bba0610c741253b0466/3.11/Dockerfile] >Reporter: Luis E Rodriguez Pupo >Assignee: Jason Brown > Fix For: 3.11.x, 4.x > > Attachments: 13949.png, insert.cql, query.cql, schema.cql > > > While executing a query on a table contaninig a field with a (escaped) json, > the following exception occurs: > java.lang.ArrayIndexOutOfBoundsException: null > at > org.codehaus.jackson.io.JsonStringEncoder.quoteAsString(JsonStringEncoder.java:141) > ~[jackson-core-asl-1.9.2.jar:1.9.2] > at org.apache.cassandra.cql3.Json.quoteAsJsonString(Json.java:45) > ~[apache-cassandra-3.11.0.jar:3.11.0] > at > org.apache.cassandra.db.marshal.UTF8Type.toJSONString(UTF8Type.java:66) > ~[apache-cassandra-3.11.0.jar:3.11.0] > at > org.apache.cassandra.cql3.selection.Selection.rowToJson(Selection.java:291) > ~[apache-cassandra-3.11.0.jar:3.11.0] > at > org.apache.cassandra.cql3.selection.Selection$ResultSetBuilder.getOutputRow(Selection.java:431) > ~[apache-cassandra-3.11.0.jar:3.11.0] > at > org.apache.cassandra.cql3.selection.Selection$ResultSetBuilder.build(Selection.java:417) > ~[apache-cassandra-3.11.0.jar:3.11.0] > at > org.apache.cassandra.cql3.statements.SelectStatement.process(SelectStatement.java:763) > ~[apache-cassandra-3.11.0.jar:3.11.0] > at > org.apache.cassandra.cql3.statements.SelectStatement.processResults(SelectStatement.java:400) > ~[apache-cassandra-3.11.0.jar:3.11.0] > at > org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:378) > ~[apache-cassandra-3.11.0.jar:3.11.0] > at > org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:251) > ~[apache-cassandra-3.11.0.jar:3.11.0] > at > org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:79) > ~[apache-cassandra-3.11.0.jar:3.11.0] > at > org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:217) > ~[apache-cassandra-3.11.0.jar:3.11.0] > at > org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:248) > ~[apache-cassandra-3.11.0.jar:3.11.0] > at > org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:233) > ~[apache-cassandra-3.11.0.jar:3.11.0] > at > org.apache.cassandra.transport.messages.QueryMessage.execute(QueryMessage.java:116) > ~[apache-cassandra-3.11.0.jar:3.11.0] > at > org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:517) > [apache-cassandra-3.11.0.jar:3.11.0] > at > org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:410) > [apache-cassandra-3.11.0.jar:3.11.0] > at > io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) > [netty-all-4.0.44.Final.jar:4.0.44.Final] > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:357) > [netty-all-4.0.44.Final.jar:4.0.44.Final] > at > io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:35) > [netty-all-4.0.44.Final.jar:4.0.44.Final] > at > io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:348) > [netty-all-4.0.44.Final.jar:4.0.44.Final] > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > [na:1.8.0_131] > at > org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:162) > [apache-cassandra-3.11.0.jar:3.11.0] > at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:109) > [apache-cassandra-3.11.0.jar:3.11.0] > at java.lang.Thread.run(Thread.java:748) [na:1.8.0_131] > Find attached the schema
[jira] [Commented] (CASSANDRA-13260) Add UDT support to Cassandra stress
[ https://issues.apache.org/jira/browse/CASSANDRA-13260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16205785#comment-16205785 ] Jason Brown commented on CASSANDRA-13260: - [~kwern] Please do, and I've assigned to you. thanks! > Add UDT support to Cassandra stress > --- > > Key: CASSANDRA-13260 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13260 > Project: Cassandra > Issue Type: Improvement > Components: Tools >Reporter: Jeremy Hanna >Assignee: Kevin Wern > Labels: lhf, stress > > Splitting out UDT support in cassandra stress from CASSANDRA-9556. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Assigned] (CASSANDRA-13260) Add UDT support to Cassandra stress
[ https://issues.apache.org/jira/browse/CASSANDRA-13260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Brown reassigned CASSANDRA-13260: --- Assignee: Kevin Wern (was: Jason Brown) > Add UDT support to Cassandra stress > --- > > Key: CASSANDRA-13260 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13260 > Project: Cassandra > Issue Type: Improvement > Components: Tools >Reporter: Jeremy Hanna >Assignee: Kevin Wern > Labels: lhf, stress > > Splitting out UDT support in cassandra stress from CASSANDRA-9556. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Assigned] (CASSANDRA-13260) Add UDT support to Cassandra stress
[ https://issues.apache.org/jira/browse/CASSANDRA-13260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Brown reassigned CASSANDRA-13260: --- Assignee: Jason Brown > Add UDT support to Cassandra stress > --- > > Key: CASSANDRA-13260 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13260 > Project: Cassandra > Issue Type: Improvement > Components: Tools >Reporter: Jeremy Hanna >Assignee: Jason Brown > Labels: lhf, stress > > Splitting out UDT support in cassandra stress from CASSANDRA-9556. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13955) NullPointerException when using CqlBulkOutputFormat
[
https://issues.apache.org/jira/browse/CASSANDRA-13955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16204625#comment-16204625
]
Jason Brown commented on CASSANDRA-13955:
-
I'm completely unfamiliar with the hadoop-related part of the code base, but as
for resolving the NPE with {{DatabaseDescriptor#conf}}, can you call this
somewhere in your code before initializing {{CqlBulkOutputFormat}}:
{code}
DatabaseDescriptor.toolInitialization()
{code}
> NullPointerException when using CqlBulkOutputFormat
> ---
>
> Key: CASSANDRA-13955
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13955
> Project: Cassandra
> Issue Type: Bug
> Environment: casandra-all:3.11.0
>Reporter: Anish
>
> I am able to insert data of reducers by using CqlOutputFormat. Due to
> performance reasons (I have large amount of inserts), I want to switch over
> to CqlBulkOutputFormat. As per the documentation, everything remains same
> except changing the format for reducers. But I get null pointer exception on
> line below of CqlBulkRecordWriter
> {code}
> DatabaseDescriptor.setStreamThroughputOutboundMegabitsPerSec(Integer.parseInt(conf.get(STREAM_THROTTLE_MBITS,
> "0")));
> {code}
> This is because "conf" is null in DatabaseDescriptor. I don't see any calls
> where this would get initialized by reducer's CqlBulkOutputFormat
> Unfortunately, I could not find any documentation or samples of using
> CqlBulkOutputFormat.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13949) java.lang.ArrayIndexOutOfBoundsException while executing query
[ https://issues.apache.org/jira/browse/CASSANDRA-13949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16204196#comment-16204196 ] Jason Brown commented on CASSANDRA-13949: - I've created a simple patch which just updates the jackson jars: ||3.11||trunk|| |[branch|https://github.com/jasobrown/cassandra/tree/13949-3.11]|[branch|https://github.com/jasobrown/cassandra/tree/13949-trunk]| |[dtest|https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-dtest/370/]|[dtest|https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-dtest/371/]| |[utests|https://circleci.com/gh/jasobrown/cassandra/tree/13949-3.11]|[utests|https://circleci.com/gh/jasobrown/cassandra/tree/13949-trunk]| > java.lang.ArrayIndexOutOfBoundsException while executing query > -- > > Key: CASSANDRA-13949 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13949 > Project: Cassandra > Issue Type: Bug > Components: CQL > Environment: Setup of 3 servers y using docker image > [https://github.com/docker-library/cassandra/blob/ca3c9df03cab318d34377bba0610c741253b0466/3.11/Dockerfile] >Reporter: Luis E Rodriguez Pupo >Assignee: Jason Brown > Fix For: 3.11.x > > Attachments: 13949.png, insert.cql, query.cql, schema.cql > > > While executing a query on a table contaninig a field with a (escaped) json, > the following exception occurs: > java.lang.ArrayIndexOutOfBoundsException: null > at > org.codehaus.jackson.io.JsonStringEncoder.quoteAsString(JsonStringEncoder.java:141) > ~[jackson-core-asl-1.9.2.jar:1.9.2] > at org.apache.cassandra.cql3.Json.quoteAsJsonString(Json.java:45) > ~[apache-cassandra-3.11.0.jar:3.11.0] > at > org.apache.cassandra.db.marshal.UTF8Type.toJSONString(UTF8Type.java:66) > ~[apache-cassandra-3.11.0.jar:3.11.0] > at > org.apache.cassandra.cql3.selection.Selection.rowToJson(Selection.java:291) > ~[apache-cassandra-3.11.0.jar:3.11.0] > at > org.apache.cassandra.cql3.selection.Selection$ResultSetBuilder.getOutputRow(Selection.java:431) > ~[apache-cassandra-3.11.0.jar:3.11.0] > at > org.apache.cassandra.cql3.selection.Selection$ResultSetBuilder.build(Selection.java:417) > ~[apache-cassandra-3.11.0.jar:3.11.0] > at > org.apache.cassandra.cql3.statements.SelectStatement.process(SelectStatement.java:763) > ~[apache-cassandra-3.11.0.jar:3.11.0] > at > org.apache.cassandra.cql3.statements.SelectStatement.processResults(SelectStatement.java:400) > ~[apache-cassandra-3.11.0.jar:3.11.0] > at > org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:378) > ~[apache-cassandra-3.11.0.jar:3.11.0] > at > org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:251) > ~[apache-cassandra-3.11.0.jar:3.11.0] > at > org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:79) > ~[apache-cassandra-3.11.0.jar:3.11.0] > at > org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:217) > ~[apache-cassandra-3.11.0.jar:3.11.0] > at > org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:248) > ~[apache-cassandra-3.11.0.jar:3.11.0] > at > org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:233) > ~[apache-cassandra-3.11.0.jar:3.11.0] > at > org.apache.cassandra.transport.messages.QueryMessage.execute(QueryMessage.java:116) > ~[apache-cassandra-3.11.0.jar:3.11.0] > at > org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:517) > [apache-cassandra-3.11.0.jar:3.11.0] > at > org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:410) > [apache-cassandra-3.11.0.jar:3.11.0] > at > io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) > [netty-all-4.0.44.Final.jar:4.0.44.Final] > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:357) > [netty-all-4.0.44.Final.jar:4.0.44.Final] > at > io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:35) > [netty-all-4.0.44.Final.jar:4.0.44.Final] > at > io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:348) > [netty-all-4.0.44.Final.jar:4.0.44.Final] > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > [na:1.8.0_131] > at > org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:162) > [apache-cassandra-3.11.0.jar:3.11.0] > at org.apache.cassandra.concurrent.SEPWork
[jira] [Assigned] (CASSANDRA-13949) java.lang.ArrayIndexOutOfBoundsException while executing query
[ https://issues.apache.org/jira/browse/CASSANDRA-13949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Brown reassigned CASSANDRA-13949: --- Assignee: Jason Brown > java.lang.ArrayIndexOutOfBoundsException while executing query > -- > > Key: CASSANDRA-13949 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13949 > Project: Cassandra > Issue Type: Bug > Components: CQL > Environment: Setup of 3 servers y using docker image > [https://github.com/docker-library/cassandra/blob/ca3c9df03cab318d34377bba0610c741253b0466/3.11/Dockerfile] >Reporter: Luis E Rodriguez Pupo >Assignee: Jason Brown > Fix For: 3.11.x > > Attachments: 13949.png, insert.cql, query.cql, schema.cql > > > While executing a query on a table contaninig a field with a (escaped) json, > the following exception occurs: > java.lang.ArrayIndexOutOfBoundsException: null > at > org.codehaus.jackson.io.JsonStringEncoder.quoteAsString(JsonStringEncoder.java:141) > ~[jackson-core-asl-1.9.2.jar:1.9.2] > at org.apache.cassandra.cql3.Json.quoteAsJsonString(Json.java:45) > ~[apache-cassandra-3.11.0.jar:3.11.0] > at > org.apache.cassandra.db.marshal.UTF8Type.toJSONString(UTF8Type.java:66) > ~[apache-cassandra-3.11.0.jar:3.11.0] > at > org.apache.cassandra.cql3.selection.Selection.rowToJson(Selection.java:291) > ~[apache-cassandra-3.11.0.jar:3.11.0] > at > org.apache.cassandra.cql3.selection.Selection$ResultSetBuilder.getOutputRow(Selection.java:431) > ~[apache-cassandra-3.11.0.jar:3.11.0] > at > org.apache.cassandra.cql3.selection.Selection$ResultSetBuilder.build(Selection.java:417) > ~[apache-cassandra-3.11.0.jar:3.11.0] > at > org.apache.cassandra.cql3.statements.SelectStatement.process(SelectStatement.java:763) > ~[apache-cassandra-3.11.0.jar:3.11.0] > at > org.apache.cassandra.cql3.statements.SelectStatement.processResults(SelectStatement.java:400) > ~[apache-cassandra-3.11.0.jar:3.11.0] > at > org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:378) > ~[apache-cassandra-3.11.0.jar:3.11.0] > at > org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:251) > ~[apache-cassandra-3.11.0.jar:3.11.0] > at > org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:79) > ~[apache-cassandra-3.11.0.jar:3.11.0] > at > org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:217) > ~[apache-cassandra-3.11.0.jar:3.11.0] > at > org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:248) > ~[apache-cassandra-3.11.0.jar:3.11.0] > at > org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:233) > ~[apache-cassandra-3.11.0.jar:3.11.0] > at > org.apache.cassandra.transport.messages.QueryMessage.execute(QueryMessage.java:116) > ~[apache-cassandra-3.11.0.jar:3.11.0] > at > org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:517) > [apache-cassandra-3.11.0.jar:3.11.0] > at > org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:410) > [apache-cassandra-3.11.0.jar:3.11.0] > at > io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) > [netty-all-4.0.44.Final.jar:4.0.44.Final] > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:357) > [netty-all-4.0.44.Final.jar:4.0.44.Final] > at > io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:35) > [netty-all-4.0.44.Final.jar:4.0.44.Final] > at > io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:348) > [netty-all-4.0.44.Final.jar:4.0.44.Final] > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > [na:1.8.0_131] > at > org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:162) > [apache-cassandra-3.11.0.jar:3.11.0] > at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:109) > [apache-cassandra-3.11.0.jar:3.11.0] > at java.lang.Thread.run(Thread.java:748) [na:1.8.0_131] > Find attached the schema of the table, the insertion query with the data > provoking the failure, and the failing query. > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13949) java.lang.ArrayIndexOutOfBoundsException while executing query
[ https://issues.apache.org/jira/browse/CASSANDRA-13949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16204138#comment-16204138 ] Jason Brown commented on CASSANDRA-13949: - bq. still old 1.9.13 According to maven central, [1.9.13 is the most current version|http://search.maven.org/#search%7Cga%7C1%7Corg.codehaus.jackson] of jackson. bq. It happens if you run again the query requesting the json I did run it a bunch of times, but if the updated jackson is working for you, let's just move ahead on that. Patch coming shortly. > java.lang.ArrayIndexOutOfBoundsException while executing query > -- > > Key: CASSANDRA-13949 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13949 > Project: Cassandra > Issue Type: Bug > Components: CQL > Environment: Setup of 3 servers y using docker image > [https://github.com/docker-library/cassandra/blob/ca3c9df03cab318d34377bba0610c741253b0466/3.11/Dockerfile] >Reporter: Luis E Rodriguez Pupo > Fix For: 3.11.x > > Attachments: 13949.png, insert.cql, query.cql, schema.cql > > > While executing a query on a table contaninig a field with a (escaped) json, > the following exception occurs: > java.lang.ArrayIndexOutOfBoundsException: null > at > org.codehaus.jackson.io.JsonStringEncoder.quoteAsString(JsonStringEncoder.java:141) > ~[jackson-core-asl-1.9.2.jar:1.9.2] > at org.apache.cassandra.cql3.Json.quoteAsJsonString(Json.java:45) > ~[apache-cassandra-3.11.0.jar:3.11.0] > at > org.apache.cassandra.db.marshal.UTF8Type.toJSONString(UTF8Type.java:66) > ~[apache-cassandra-3.11.0.jar:3.11.0] > at > org.apache.cassandra.cql3.selection.Selection.rowToJson(Selection.java:291) > ~[apache-cassandra-3.11.0.jar:3.11.0] > at > org.apache.cassandra.cql3.selection.Selection$ResultSetBuilder.getOutputRow(Selection.java:431) > ~[apache-cassandra-3.11.0.jar:3.11.0] > at > org.apache.cassandra.cql3.selection.Selection$ResultSetBuilder.build(Selection.java:417) > ~[apache-cassandra-3.11.0.jar:3.11.0] > at > org.apache.cassandra.cql3.statements.SelectStatement.process(SelectStatement.java:763) > ~[apache-cassandra-3.11.0.jar:3.11.0] > at > org.apache.cassandra.cql3.statements.SelectStatement.processResults(SelectStatement.java:400) > ~[apache-cassandra-3.11.0.jar:3.11.0] > at > org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:378) > ~[apache-cassandra-3.11.0.jar:3.11.0] > at > org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:251) > ~[apache-cassandra-3.11.0.jar:3.11.0] > at > org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:79) > ~[apache-cassandra-3.11.0.jar:3.11.0] > at > org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:217) > ~[apache-cassandra-3.11.0.jar:3.11.0] > at > org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:248) > ~[apache-cassandra-3.11.0.jar:3.11.0] > at > org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:233) > ~[apache-cassandra-3.11.0.jar:3.11.0] > at > org.apache.cassandra.transport.messages.QueryMessage.execute(QueryMessage.java:116) > ~[apache-cassandra-3.11.0.jar:3.11.0] > at > org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:517) > [apache-cassandra-3.11.0.jar:3.11.0] > at > org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:410) > [apache-cassandra-3.11.0.jar:3.11.0] > at > io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) > [netty-all-4.0.44.Final.jar:4.0.44.Final] > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:357) > [netty-all-4.0.44.Final.jar:4.0.44.Final] > at > io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:35) > [netty-all-4.0.44.Final.jar:4.0.44.Final] > at > io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:348) > [netty-all-4.0.44.Final.jar:4.0.44.Final] > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > [na:1.8.0_131] > at > org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:162) > [apache-cassandra-3.11.0.jar:3.11.0] > at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:109) > [apache-cassandra-3.11.0.jar:3.11.0] > at java.lang.Thread.run(Thread.java:748) [na:1.8.0_131] > Find attached the schema of the table, the insertion query with the data > provo
[jira] [Commented] (CASSANDRA-7839) Support standard EC2 naming conventions in Ec2Snitch
[
https://issues.apache.org/jira/browse/CASSANDRA-7839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16201520#comment-16201520
]
Jason Brown commented on CASSANDRA-7839:
I've rebased [~ramsperger]'s patch on trunk and running tests here:
||7839||
|[branch|https://github.com/jasobrown/cassandra/tree/7839]|
|[dtest|https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-dtest/366/]|
|[utests|https://circleci.com/gh/jasobrown/cassandra/tree/7839]|
I cleaned up some constants from the original patch, but added a NEWS entry and
the cross-schema naming check, as per [~pauloricardomg].
bq. should we add a bootstrap check (similar to checkForEndpointCollision)
failing to boostrap a node if it's using the new scheme and nodes are detected
in gossip using the legacy scheme
FWIW, this would prevent an operator from adding a new region in the standard
style to an existing cluster (they would always need to set {{legacy}}). I am
totally ok with that and it's not worth the extra effort to allow it. I'm just
noting this point here for posterity.
Further, we should check at all node startups, not just bootstrap, because if
the value in the {{conf/cassandra-rackdc.properties}} somehow gets flipped to
the other value, it's gonna mess up the cluster metadata big time as peers will
think the node is in a different DC/rack now and incorrectly adjust their view
of the cluster.
I want to add some unit tests for
{{Ec2Snitch@#hasConflictingDatacenterOrRack}}, and I'd like some feedback on
the implementation, as well. [~pauloricardomg], if you don't mind taking a
look, since you've done so in the past for this ticket, I'd appreciate it.
> Support standard EC2 naming conventions in Ec2Snitch
>
>
> Key: CASSANDRA-7839
> URL: https://issues.apache.org/jira/browse/CASSANDRA-7839
> Project: Cassandra
> Issue Type: Improvement
>Reporter: Gregory Ramsperger
>Assignee: Gregory Ramsperger
> Labels: docs-impacting
> Attachments: CASSANDRA-7839-aws-naming-conventions.patch
>
>
> The EC2 snitches use datacenter and rack naming conventions inconsistent with
> those presented in Amazon EC2 APIs as region and availability zone. A
> discussion of this is found in CASSANDRA-4026. This has not been changed for
> valid backwards compatibility reasons. Using SnitchProperties, it is possible
> to switch between the legacy naming and the full, AWS-style naming.
> Proposal:
> * introduce a property (ec2_naming_scheme) to switch naming schemes.
> * default to current/legacy naming scheme
> * add support for a new scheme ("standard") which is consistent AWS
> conventions
> ** data centers will be the region name, including the number
> ** racks will be the availability zone name, including the region name
> Examples:
> * * legacy* : datacenter is the part of the availability zone name preceding
> the last "\-" when the zone ends in \-1 and includes the number if not \-1.
> Rack is the portion of the availability zone name following the last "\-".
> ** us-west-1a => dc: us-west, rack: 1a
> ** us-west-2b => dc: us-west-2, rack: 2b;
> * *standard* : datacenter is the part of the availability zone name preceding
> zone letter. rack is the entire availability zone name.
> ** us-west-1a => dc: us-west-1, rack: us-west-1a
> ** us-west-2b => dc: us-west-2, rack: us-west-2b;
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-7839) Support standard EC2 naming conventions in Ec2Snitch
[
https://issues.apache.org/jira/browse/CASSANDRA-7839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16201424#comment-16201424
]
Jason Brown commented on CASSANDRA-7839:
[~danielbankhead] us-west-2 should be fine, as long as you can confirm the
correct outputs. Is it possible for you to try both legacy and standard
variants?
> Support standard EC2 naming conventions in Ec2Snitch
>
>
> Key: CASSANDRA-7839
> URL: https://issues.apache.org/jira/browse/CASSANDRA-7839
> Project: Cassandra
> Issue Type: Improvement
>Reporter: Gregory Ramsperger
>Assignee: Gregory Ramsperger
> Labels: docs-impacting
> Attachments: CASSANDRA-7839-aws-naming-conventions.patch
>
>
> The EC2 snitches use datacenter and rack naming conventions inconsistent with
> those presented in Amazon EC2 APIs as region and availability zone. A
> discussion of this is found in CASSANDRA-4026. This has not been changed for
> valid backwards compatibility reasons. Using SnitchProperties, it is possible
> to switch between the legacy naming and the full, AWS-style naming.
> Proposal:
> * introduce a property (ec2_naming_scheme) to switch naming schemes.
> * default to current/legacy naming scheme
> * add support for a new scheme ("standard") which is consistent AWS
> conventions
> ** data centers will be the region name, including the number
> ** racks will be the availability zone name, including the region name
> Examples:
> * * legacy* : datacenter is the part of the availability zone name preceding
> the last "\-" when the zone ends in \-1 and includes the number if not \-1.
> Rack is the portion of the availability zone name following the last "\-".
> ** us-west-1a => dc: us-west, rack: 1a
> ** us-west-2b => dc: us-west-2, rack: 2b;
> * *standard* : datacenter is the part of the availability zone name preceding
> zone letter. rack is the entire availability zone name.
> ** us-west-1a => dc: us-west-1, rack: us-west-1a
> ** us-west-2b => dc: us-west-2, rack: us-west-2b;
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-7839) Support standard EC2 naming conventions in Ec2Snitch
[
https://issues.apache.org/jira/browse/CASSANDRA-7839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16201308#comment-16201308
]
Jason Brown edited comment on CASSANDRA-7839 at 10/12/17 1:51 AM:
--
I agree this would be a nice thing to get in for 4.0. I can rebase and kick off
a round of tests. [~danielbankhead] Would you be able to give this patch a
whirl (assuming you run in AWS)? I've also reached out to Netflix to see if
they can try it out, as well.
was (Author: jasobrown):
I agree this would be a nice thing to get in for 4.0. I can rebase and kick off
a round of tests. [~danielbankhead] Would you be able to give this patch a
whorl (assuming you run in AWS)? I've also reached out to Netflix to see if
they can try it out, as well.
> Support standard EC2 naming conventions in Ec2Snitch
>
>
> Key: CASSANDRA-7839
> URL: https://issues.apache.org/jira/browse/CASSANDRA-7839
> Project: Cassandra
> Issue Type: Improvement
>Reporter: Gregory Ramsperger
>Assignee: Gregory Ramsperger
> Labels: docs-impacting
> Attachments: CASSANDRA-7839-aws-naming-conventions.patch
>
>
> The EC2 snitches use datacenter and rack naming conventions inconsistent with
> those presented in Amazon EC2 APIs as region and availability zone. A
> discussion of this is found in CASSANDRA-4026. This has not been changed for
> valid backwards compatibility reasons. Using SnitchProperties, it is possible
> to switch between the legacy naming and the full, AWS-style naming.
> Proposal:
> * introduce a property (ec2_naming_scheme) to switch naming schemes.
> * default to current/legacy naming scheme
> * add support for a new scheme ("standard") which is consistent AWS
> conventions
> ** data centers will be the region name, including the number
> ** racks will be the availability zone name, including the region name
> Examples:
> * * legacy* : datacenter is the part of the availability zone name preceding
> the last "\-" when the zone ends in \-1 and includes the number if not \-1.
> Rack is the portion of the availability zone name following the last "\-".
> ** us-west-1a => dc: us-west, rack: 1a
> ** us-west-2b => dc: us-west-2, rack: 2b;
> * *standard* : datacenter is the part of the availability zone name preceding
> zone letter. rack is the entire availability zone name.
> ** us-west-1a => dc: us-west-1, rack: us-west-1a
> ** us-west-2b => dc: us-west-2, rack: us-west-2b;
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-7839) Support standard EC2 naming conventions in Ec2Snitch
[
https://issues.apache.org/jira/browse/CASSANDRA-7839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16201308#comment-16201308
]
Jason Brown commented on CASSANDRA-7839:
I agree this would be a nice thing to get in for 4.0. I can rebase and kick off
a round of tests. [~danielbankhead] Would you be able to give this patch a
whorl (assuming you run in AWS)? I've also reached out to Netflix to see if
they can try it out, as well.
> Support standard EC2 naming conventions in Ec2Snitch
>
>
> Key: CASSANDRA-7839
> URL: https://issues.apache.org/jira/browse/CASSANDRA-7839
> Project: Cassandra
> Issue Type: Improvement
>Reporter: Gregory Ramsperger
>Assignee: Gregory Ramsperger
> Labels: docs-impacting
> Attachments: CASSANDRA-7839-aws-naming-conventions.patch
>
>
> The EC2 snitches use datacenter and rack naming conventions inconsistent with
> those presented in Amazon EC2 APIs as region and availability zone. A
> discussion of this is found in CASSANDRA-4026. This has not been changed for
> valid backwards compatibility reasons. Using SnitchProperties, it is possible
> to switch between the legacy naming and the full, AWS-style naming.
> Proposal:
> * introduce a property (ec2_naming_scheme) to switch naming schemes.
> * default to current/legacy naming scheme
> * add support for a new scheme ("standard") which is consistent AWS
> conventions
> ** data centers will be the region name, including the number
> ** racks will be the availability zone name, including the region name
> Examples:
> * * legacy* : datacenter is the part of the availability zone name preceding
> the last "\-" when the zone ends in \-1 and includes the number if not \-1.
> Rack is the portion of the availability zone name following the last "\-".
> ** us-west-1a => dc: us-west, rack: 1a
> ** us-west-2b => dc: us-west-2, rack: 2b;
> * *standard* : datacenter is the part of the availability zone name preceding
> zone letter. rack is the entire availability zone name.
> ** us-west-1a => dc: us-west-1, rack: us-west-1a
> ** us-west-2b => dc: us-west-2, rack: us-west-2b;
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-13949) java.lang.ArrayIndexOutOfBoundsException while executing query
[
https://issues.apache.org/jira/browse/CASSANDRA-13949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16200242#comment-16200242
]
Jason Brown edited comment on CASSANDRA-13949 at 10/11/17 1:08 PM:
---
I tried to reproduce using your attached scripts (very handy, thanks!) on a
local running instance of casandra, but was unable to reproduce the
{{NullPointerException}}. I did however, get a very curious output in the
{{value}} field: at four point in the large json blob, there are a few Kb of
printed null characters {{\x00}}. See the attached image. I hexdump'ed your
insert.cql and it looked legit (no stray characters).
I then checked our json libs, and they are quite old - jackson*-1.9.2,
committed in 2011! The latest jackson jars are version 1.9.13 on maven central
(from 2013). I naively replaced the jars in lib/, and I didn't get the null
character output anymore.
[~lrodriguez2002cu] Can you try simply replacing the jars and see if the NPE
stops occurring? If it does, then I'll create a more formal patch, run tests,
and so on.
was (Author: jasobrown):
I tried to reproduce using your attached scripts (very handy, thanks!) on a
local running instance of casandra, but was unable to reproduce the
{{NullPointerException}}. I did however, get a very curious output in the
{{value}} field: at four point in the large json blob, there are a few Kb of
printed null characters {{\x00}}. See the attached image. I hexdump'ed your
insert.cql and it looked legit (no stray characters).
I then checked our json libs, and they are quite old - jackson*-1.9.2,
committed in 2011! The latest jackson jars are version 1.9.13 on maven central
(from 2013). I naively replaced the jars in lib/, and I didn't get the null
character output anymore.
Can you try simply replacing the jars and see if the NPE stops occurring? If it
does, then I'll create a more formal patch, run tests, and so on.
> java.lang.ArrayIndexOutOfBoundsException while executing query
> --
>
> Key: CASSANDRA-13949
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13949
> Project: Cassandra
> Issue Type: Bug
> Components: CQL
> Environment: Setup of 3 servers y using docker image
> [https://github.com/docker-library/cassandra/blob/ca3c9df03cab318d34377bba0610c741253b0466/3.11/Dockerfile]
>Reporter: Luis E Rodriguez Pupo
> Fix For: 3.11.x
>
> Attachments: 13949.png, insert.cql, query.cql, schema.cql
>
>
> While executing a query on a table contaninig a field with a (escaped) json,
> the following exception occurs:
> java.lang.ArrayIndexOutOfBoundsException: null
> at
> org.codehaus.jackson.io.JsonStringEncoder.quoteAsString(JsonStringEncoder.java:141)
> ~[jackson-core-asl-1.9.2.jar:1.9.2]
> at org.apache.cassandra.cql3.Json.quoteAsJsonString(Json.java:45)
> ~[apache-cassandra-3.11.0.jar:3.11.0]
> at
> org.apache.cassandra.db.marshal.UTF8Type.toJSONString(UTF8Type.java:66)
> ~[apache-cassandra-3.11.0.jar:3.11.0]
> at
> org.apache.cassandra.cql3.selection.Selection.rowToJson(Selection.java:291)
> ~[apache-cassandra-3.11.0.jar:3.11.0]
> at
> org.apache.cassandra.cql3.selection.Selection$ResultSetBuilder.getOutputRow(Selection.java:431)
> ~[apache-cassandra-3.11.0.jar:3.11.0]
> at
> org.apache.cassandra.cql3.selection.Selection$ResultSetBuilder.build(Selection.java:417)
> ~[apache-cassandra-3.11.0.jar:3.11.0]
> at
> org.apache.cassandra.cql3.statements.SelectStatement.process(SelectStatement.java:763)
> ~[apache-cassandra-3.11.0.jar:3.11.0]
> at
> org.apache.cassandra.cql3.statements.SelectStatement.processResults(SelectStatement.java:400)
> ~[apache-cassandra-3.11.0.jar:3.11.0]
> at
> org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:378)
> ~[apache-cassandra-3.11.0.jar:3.11.0]
> at
> org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:251)
> ~[apache-cassandra-3.11.0.jar:3.11.0]
> at
> org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:79)
> ~[apache-cassandra-3.11.0.jar:3.11.0]
> at
> org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:217)
> ~[apache-cassandra-3.11.0.jar:3.11.0]
> at
> org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:248)
> ~[apache-cassandra-3.11.0.jar:3.11.0]
> at
> org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:233)
> ~[apache-cassandra-3.11.0.jar:3.11.0]
> at
> org.apache.cassandra.transport.messages.QueryMessage.execute(QueryMessage.java:116)
> ~[apache-cassandra-3.11.0.jar:3.11.0]
> at
> org.apache.cassandra.transport.Message$Dis
[jira] [Updated] (CASSANDRA-13949) java.lang.ArrayIndexOutOfBoundsException while executing query
[
https://issues.apache.org/jira/browse/CASSANDRA-13949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Brown updated CASSANDRA-13949:
Attachment: 13949.png
I tried to reproduce using your attached scripts (very handy, thanks!) on a
local running instance of casandra, but was unable to reproduce the
{{NullPointerException}}. I did however, get a very curious output in the
{{value}} field: at four point in the large json blob, there are a few Kb of
printed null characters {{\x00}}. See the attached image. I hexdump'ed your
insert.cql and it looked legit (no stray characters).
I then checked our json libs, and they are quite old - jackson*-1.9.2,
committed in 2011! The latest jackson jars are version 1.9.13 on maven central
(from 2013). I naively replaced the jars in lib/, and I didn't get the null
character output anymore.
Can you try simply replacing the jars and see if the NPE stops occurring? If it
does, then I'll create a more formal patch, run tests, and so on.
> java.lang.ArrayIndexOutOfBoundsException while executing query
> --
>
> Key: CASSANDRA-13949
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13949
> Project: Cassandra
> Issue Type: Bug
> Components: CQL
> Environment: Setup of 3 servers y using docker image
> [https://github.com/docker-library/cassandra/blob/ca3c9df03cab318d34377bba0610c741253b0466/3.11/Dockerfile]
>Reporter: Luis E Rodriguez Pupo
> Fix For: 3.11.x
>
> Attachments: 13949.png, insert.cql, query.cql, schema.cql
>
>
> While executing a query on a table contaninig a field with a (escaped) json,
> the following exception occurs:
> java.lang.ArrayIndexOutOfBoundsException: null
> at
> org.codehaus.jackson.io.JsonStringEncoder.quoteAsString(JsonStringEncoder.java:141)
> ~[jackson-core-asl-1.9.2.jar:1.9.2]
> at org.apache.cassandra.cql3.Json.quoteAsJsonString(Json.java:45)
> ~[apache-cassandra-3.11.0.jar:3.11.0]
> at
> org.apache.cassandra.db.marshal.UTF8Type.toJSONString(UTF8Type.java:66)
> ~[apache-cassandra-3.11.0.jar:3.11.0]
> at
> org.apache.cassandra.cql3.selection.Selection.rowToJson(Selection.java:291)
> ~[apache-cassandra-3.11.0.jar:3.11.0]
> at
> org.apache.cassandra.cql3.selection.Selection$ResultSetBuilder.getOutputRow(Selection.java:431)
> ~[apache-cassandra-3.11.0.jar:3.11.0]
> at
> org.apache.cassandra.cql3.selection.Selection$ResultSetBuilder.build(Selection.java:417)
> ~[apache-cassandra-3.11.0.jar:3.11.0]
> at
> org.apache.cassandra.cql3.statements.SelectStatement.process(SelectStatement.java:763)
> ~[apache-cassandra-3.11.0.jar:3.11.0]
> at
> org.apache.cassandra.cql3.statements.SelectStatement.processResults(SelectStatement.java:400)
> ~[apache-cassandra-3.11.0.jar:3.11.0]
> at
> org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:378)
> ~[apache-cassandra-3.11.0.jar:3.11.0]
> at
> org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:251)
> ~[apache-cassandra-3.11.0.jar:3.11.0]
> at
> org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:79)
> ~[apache-cassandra-3.11.0.jar:3.11.0]
> at
> org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:217)
> ~[apache-cassandra-3.11.0.jar:3.11.0]
> at
> org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:248)
> ~[apache-cassandra-3.11.0.jar:3.11.0]
> at
> org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:233)
> ~[apache-cassandra-3.11.0.jar:3.11.0]
> at
> org.apache.cassandra.transport.messages.QueryMessage.execute(QueryMessage.java:116)
> ~[apache-cassandra-3.11.0.jar:3.11.0]
> at
> org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:517)
> [apache-cassandra-3.11.0.jar:3.11.0]
> at
> org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:410)
> [apache-cassandra-3.11.0.jar:3.11.0]
> at
> io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
> [netty-all-4.0.44.Final.jar:4.0.44.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:357)
> [netty-all-4.0.44.Final.jar:4.0.44.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:35)
> [netty-all-4.0.44.Final.jar:4.0.44.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:348)
> [netty-all-4.0.44.Final.jar:4.0.44.Final]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [na:1.8.0_131]
>
[jira] [Commented] (CASSANDRA-13938) Default repair is broken, crashes other nodes participating in repair (in trunk)
[
https://issues.apache.org/jira/browse/CASSANDRA-13938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16193127#comment-16193127
]
Jason Brown commented on CASSANDRA-13938:
-
[~zznate] thanks for detailed steps to create the error. I am able to
reproduce, and am digging in now.
> Default repair is broken, crashes other nodes participating in repair (in
> trunk)
>
>
> Key: CASSANDRA-13938
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13938
> Project: Cassandra
> Issue Type: Bug
> Components: Repair
>Reporter: Nate McCall
>Assignee: Jason Brown
>Priority: Critical
>
> Running through a simple scenario to test some of the new repair features, I
> was not able to make a repair command work. Further, the exception seemed to
> trigger a nasty failure state that basically shuts down the netty connections
> for messaging *and* CQL on the nodes transferring back data to the node being
> repaired. The following steps reproduce this issue consistently.
> Cassandra stress profile (probably not necessary, but this one provides a
> really simple schema and consistent data shape):
> {noformat}
> keyspace: standard_long
> keyspace_definition: |
> CREATE KEYSPACE standard_long WITH replication = {'class':'SimpleStrategy',
> 'replication_factor':3};
> table: test_data
> table_definition: |
> CREATE TABLE test_data (
> key text,
> ts bigint,
> val text,
> PRIMARY KEY (key, ts)
> ) WITH COMPACT STORAGE AND
> CLUSTERING ORDER BY (ts DESC) AND
> bloom_filter_fp_chance=0.01 AND
> caching={'keys':'ALL', 'rows_per_partition':'NONE'} AND
> comment='' AND
> dclocal_read_repair_chance=0.00 AND
> gc_grace_seconds=864000 AND
> read_repair_chance=0.00 AND
> compaction={'class': 'SizeTieredCompactionStrategy'} AND
> compression={'sstable_compression': 'LZ4Compressor'};
> columnspec:
> - name: key
> population: uniform(1..5000) # 50 million records available
> - name: ts
> cluster: gaussian(1..50) # Up to 50 inserts per record
> - name: val
> population: gaussian(128..1024) # varrying size of value data
> insert:
> partitions: fixed(1) # only one insert per batch for individual partitions
> select: fixed(1)/1 # each insert comes in one at a time
> batchtype: UNLOGGED
> queries:
> single:
> cql: select * from test_data where key = ? and ts = ? limit 1;
> series:
> cql: select key,ts,val from test_data where key = ? limit 10;
> {noformat}
> The commands to build and run:
> {noformat}
> ccm create 4_0_test -v git:trunk -n 3 -s
> ccm stress user profile=./histo-test-schema.yml
> ops\(insert=20,single=1,series=1\) duration=15s -rate threads=4
> # flush the memtable just to get everything on disk
> ccm node1 nodetool flush
> ccm node2 nodetool flush
> ccm node3 nodetool flush
> # disable hints for nodes 2 and 3
> ccm node2 nodetool disablehandoff
> ccm node3 nodetool disablehandoff
> # stop node1
> ccm node1 stop
> ccm stress user profile=./histo-test-schema.yml
> ops\(insert=20,single=1,series=1\) duration=45s -rate threads=4
> # wait 10 seconds
> ccm node1 start
> # Note that we are local to ccm's nodetool install 'cause repair preview is
> not reported yet
> node1/bin/nodetool repair --preview
> node1/bin/nodetool repair standard_long test_data
> {noformat}
> The error outputs from the last repair command follow. First, this is stdout
> from node1:
> {noformat}
> $ node1/bin/nodetool repair standard_long test_data
> objc[47876]: Class JavaLaunchHelper is implemented in both
> /Library/Java/JavaVirtualMachines/jdk1.8.0_101.jdk/Contents/Home/bin/java
> (0x10274d4c0) and
> /Library/Java/JavaVirtualMachines/jdk1.8.0_101.jdk/Contents/Home/jre/lib/libinstrument.dylib
> (0x1047b64e0). One of the two will be used. Which one is undefined.
> [2017-10-05 14:31:52,425] Starting repair command #4
> (7e1a9150-a98e-11e7-ad86-cbd2801b8de2), repairing keyspace standard_long with
> repair options (parallelism: parallel, primary range: false, incremental:
> true, job threads: 1, ColumnFamilies: [test_data], dataCenters: [], hosts:
> [], previewKind: NONE, # of ranges: 3, pull repair: false, force repair:
> false)
> [2017-10-05 14:32:07,045] Repair session 7e2e8e80-a98e-11e7-ad86-cbd2801b8de2
> for range [(3074457345618258602,-9223372036854775808],
> (-9223372036854775808,-3074457345618258603],
> (-3074457345618258603,3074457345618258602]] failed with error Stream failed
> [2017-10-05 14:32:07,048] null
> [2017-10-05 14:32:07,050] Repair command #4 finished in 14 seconds
> error: Repair job has failed with the error message: [2017-10-05
> 14:32:07,048] null
> -- StackTrace --
> java.lang.RuntimeException: Repair job has failed with the error message:
> [2017-10-05 14:32:07,048]
[jira] [Assigned] (CASSANDRA-13938) Default repair is broken, crashes other nodes participating in repair (in trunk)
[
https://issues.apache.org/jira/browse/CASSANDRA-13938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Brown reassigned CASSANDRA-13938:
---
Assignee: Jason Brown
> Default repair is broken, crashes other nodes participating in repair (in
> trunk)
>
>
> Key: CASSANDRA-13938
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13938
> Project: Cassandra
> Issue Type: Bug
> Components: Repair
>Reporter: Nate McCall
>Assignee: Jason Brown
>Priority: Critical
>
> Running through a simple scenario to test some of the new repair features, I
> was not able to make a repair command work. Further, the exception seemed to
> trigger a nasty failure state that basically shuts down the netty connections
> for messaging *and* CQL on the nodes transferring back data to the node being
> repaired. The following steps reproduce this issue consistently.
> Cassandra stress profile (probably not necessary, but this one provides a
> really simple schema and consistent data shape):
> {noformat}
> keyspace: standard_long
> keyspace_definition: |
> CREATE KEYSPACE standard_long WITH replication = {'class':'SimpleStrategy',
> 'replication_factor':3};
> table: test_data
> table_definition: |
> CREATE TABLE test_data (
> key text,
> ts bigint,
> val text,
> PRIMARY KEY (key, ts)
> ) WITH COMPACT STORAGE AND
> CLUSTERING ORDER BY (ts DESC) AND
> bloom_filter_fp_chance=0.01 AND
> caching={'keys':'ALL', 'rows_per_partition':'NONE'} AND
> comment='' AND
> dclocal_read_repair_chance=0.00 AND
> gc_grace_seconds=864000 AND
> read_repair_chance=0.00 AND
> compaction={'class': 'SizeTieredCompactionStrategy'} AND
> compression={'sstable_compression': 'LZ4Compressor'};
> columnspec:
> - name: key
> population: uniform(1..5000) # 50 million records available
> - name: ts
> cluster: gaussian(1..50) # Up to 50 inserts per record
> - name: val
> population: gaussian(128..1024) # varrying size of value data
> insert:
> partitions: fixed(1) # only one insert per batch for individual partitions
> select: fixed(1)/1 # each insert comes in one at a time
> batchtype: UNLOGGED
> queries:
> single:
> cql: select * from test_data where key = ? and ts = ? limit 1;
> series:
> cql: select key,ts,val from test_data where key = ? limit 10;
> {noformat}
> The commands to build and run:
> {noformat}
> ccm create 4_0_test -v git:trunk -n 3 -s
> ccm stress user profile=./histo-test-schema.yml
> ops\(insert=20,single=1,series=1\) duration=15s -rate threads=4
> # flush the memtable just to get everything on disk
> ccm node1 nodetool flush
> ccm node2 nodetool flush
> ccm node3 nodetool flush
> # disable hints for nodes 2 and 3
> ccm node2 nodetool disablehandoff
> ccm node3 nodetool disablehandoff
> # stop node1
> ccm node1 stop
> ccm stress user profile=./histo-test-schema.yml
> ops\(insert=20,single=1,series=1\) duration=45s -rate threads=4
> # wait 10 seconds
> ccm node1 start
> # Note that we are local to ccm's nodetool install 'cause repair preview is
> not reported yet
> node1/bin/nodetool repair --preview
> node1/bin/nodetool repair standard_long test_data
> {noformat}
> The error outputs from the last repair command follow. First, this is stdout
> from node1:
> {noformat}
> $ node1/bin/nodetool repair standard_long test_data
> objc[47876]: Class JavaLaunchHelper is implemented in both
> /Library/Java/JavaVirtualMachines/jdk1.8.0_101.jdk/Contents/Home/bin/java
> (0x10274d4c0) and
> /Library/Java/JavaVirtualMachines/jdk1.8.0_101.jdk/Contents/Home/jre/lib/libinstrument.dylib
> (0x1047b64e0). One of the two will be used. Which one is undefined.
> [2017-10-05 14:31:52,425] Starting repair command #4
> (7e1a9150-a98e-11e7-ad86-cbd2801b8de2), repairing keyspace standard_long with
> repair options (parallelism: parallel, primary range: false, incremental:
> true, job threads: 1, ColumnFamilies: [test_data], dataCenters: [], hosts:
> [], previewKind: NONE, # of ranges: 3, pull repair: false, force repair:
> false)
> [2017-10-05 14:32:07,045] Repair session 7e2e8e80-a98e-11e7-ad86-cbd2801b8de2
> for range [(3074457345618258602,-9223372036854775808],
> (-9223372036854775808,-3074457345618258603],
> (-3074457345618258603,3074457345618258602]] failed with error Stream failed
> [2017-10-05 14:32:07,048] null
> [2017-10-05 14:32:07,050] Repair command #4 finished in 14 seconds
> error: Repair job has failed with the error message: [2017-10-05
> 14:32:07,048] null
> -- StackTrace --
> java.lang.RuntimeException: Repair job has failed with the error message:
> [2017-10-05 14:32:07,048] null
> at org.apache.cassandra.tools.RepairRunner.progress(RepairRunner.java:122)
> at
> org.apache.cassandra.utils.progr
[jira] [Resolved] (CASSANDRA-13800) dtest failure: native_transport_ssl_test.NativeTransportSSL.connect_to_ssl_test
[
https://issues.apache.org/jira/browse/CASSANDRA-13800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Brown resolved CASSANDRA-13800.
-
Resolution: Cannot Reproduce
I've poked around the cassci and apache jenkins runs for the last month or so,
and I'm not seeing any failures related to {{native_transport_ssl_test}}. Also
running this dtest heavily in a loop on my laptop doesn't yield any problems
either.
Closing for now, but if errors start happening again, I'm happy to jump in and
investigate.
> dtest failure:
> native_transport_ssl_test.NativeTransportSSL.connect_to_ssl_test
>
>
> Key: CASSANDRA-13800
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13800
> Project: Cassandra
> Issue Type: Bug
>Reporter: Marcus Eriksson
>Assignee: Jason Brown
> Labels: dtest
>
> http://cassci.datastax.com/job/trunk_dtest/1623/testReport/native_transport_ssl_test/NativeTransportSSL/connect_to_ssl_test
> http://cassci.datastax.com/job/trunk_dtest/1619/testReport/native_transport_ssl_test/NativeTransportSSL/connect_to_ssl_test
> http://cassci.datastax.com/job/trunk_dtest/1612/testReport/native_transport_ssl_test/NativeTransportSSL/connect_to_ssl_test
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-13915) Create a Docker container to build the docs
[ https://issues.apache.org/jira/browse/CASSANDRA-13915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16191652#comment-16191652 ] Jason Brown edited comment on CASSANDRA-13915 at 10/4/17 5:42 PM: -- bq. Otherwise the backup message all the time works for me Let's keep things simple and print the 'backup' message. Anyone in the know on macOS can do other, clever things. was (Author: jasobrown): bq. Otherwise the backup message all the time works for me Let's keep things simple and print the 'backup' message. Anyone in the know on Mac can do other, clever things. > Create a Docker container to build the docs > --- > > Key: CASSANDRA-13915 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13915 > Project: Cassandra > Issue Type: Improvement >Reporter: Joaquin Casares >Assignee: Joaquin Casares > > As requested by [~rustyrazorblade], I will be adding a Docker container to > build the docs without any prereqs (other than Docker). -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13915) Create a Docker container to build the docs
[ https://issues.apache.org/jira/browse/CASSANDRA-13915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16191652#comment-16191652 ] Jason Brown commented on CASSANDRA-13915: - bq. Otherwise the backup message all the time works for me Let's keep things simple and print the 'backup' message. Anyone in the know on Mac can do other, clever things. > Create a Docker container to build the docs > --- > > Key: CASSANDRA-13915 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13915 > Project: Cassandra > Issue Type: Improvement >Reporter: Joaquin Casares >Assignee: Joaquin Casares > > As requested by [~rustyrazorblade], I will be adding a Docker container to > build the docs without any prereqs (other than Docker). -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-13915) Create a Docker container to build the docs
[ https://issues.apache.org/jira/browse/CASSANDRA-13915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Brown updated CASSANDRA-13915: Reviewer: Jason Brown > Create a Docker container to build the docs > --- > > Key: CASSANDRA-13915 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13915 > Project: Cassandra > Issue Type: Improvement >Reporter: Joaquin Casares >Assignee: Joaquin Casares > > As requested by [~rustyrazorblade], I will be adding a Docker container to > build the docs without any prereqs (other than Docker). -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13915) Create a Docker container to build the docs
[ https://issues.apache.org/jira/browse/CASSANDRA-13915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16191647#comment-16191647 ] Jason Brown commented on CASSANDRA-13915: - Also, another mac vs linux difference: the navigation on the left works on a mac, completely broken on linux. Actually, all hyperlinks are broken on linux. Can you take a look? > Create a Docker container to build the docs > --- > > Key: CASSANDRA-13915 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13915 > Project: Cassandra > Issue Type: Improvement >Reporter: Joaquin Casares >Assignee: Joaquin Casares > > As requested by [~rustyrazorblade], I will be adding a Docker container to > build the docs without any prereqs (other than Docker). -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13915) Create a Docker container to build the docs
[ https://issues.apache.org/jira/browse/CASSANDRA-13915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16191643#comment-16191643 ] Jason Brown commented on CASSANDRA-13915: - Also, fwiw, the design/look of the output html files is different that what we have currently [on the apache.org site|https://cassandra.apache.org/doc/latest/]. Was this is intentional? Is this a default? Does it matter? I don't care too much either way, just want to make sure this is a known thing. > Create a Docker container to build the docs > --- > > Key: CASSANDRA-13915 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13915 > Project: Cassandra > Issue Type: Improvement >Reporter: Joaquin Casares >Assignee: Joaquin Casares > > As requested by [~rustyrazorblade], I will be adding a Docker container to > build the docs without any prereqs (other than Docker). -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13915) Create a Docker container to build the docs
[
https://issues.apache.org/jira/browse/CASSANDRA-13915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16191621#comment-16191621
]
Jason Brown commented on CASSANDRA-13915:
-
bq. Dockerfile and docker-compose.yml are the default names for working with
Docker-related projects
sgtm. as stated above, I am a container noob.
bq. you two fine with me adding the doc/build directory to the ant realclean?
totally legit: {{realclean}} *should* clean all build artifacts.
When I build the docs, there's the message at the end:
{code}
The locally built documentation can be found here:
open build/html/index.html
{code}
However, when I do this on ubuntu 17.04, I get the following error:
{code}
$ open build/html/index.html
Couldn't get a file descriptor referring to the console
{code}
A naive internet search reveals that {{open}} [isn't quite what we
want|https://ubuntuforums.org/showthread.php?t=2124061}. How did you test this?
Next error: after I build the docs a few times, on the same linux install, I
run {{ant realclean}} again, which produces this error:
{code}
BUILD FAILED
/opt/dev/cassandra/build.xml:196: Unable to delete directory
/opt/dev/cassandra/build/test/lib
{code}
A quick look shows everything in {{/opt/dev/cassandra/build/}} is now owned by
root, and I need to {{sudo rm -rf build}} to clean it up. Which, of course,
leads to:
{code}
BUILD FAILED
/opt/dev/cassandra/build.xml:207: Unable to delete file
/opt/dev/cassandra/doc/build/doctrees/configuration/index.doctree
{code}
Once again, {{sudo rm -rf}}. Can you change the privileges of the artifacts
that get generated, both in ./build and ./doc/build?
UPDATE: I tried out all these commands on a Mac (High Sierra), and after
installing Docker, everything was much smoother. No permissions issues for
{{ant realclean}}, and {{open}} worked correctly. At a minimum, the
{{realclean}} problem needs to be fixed as I suspect we'll have it part of thew
automation for building the online docs hosted on cassandra.apache.org. For
{{open}}, either remove the whole statement or add a note saying "mac only" or
something. [~rustyrazorblade] thoughts?
> Create a Docker container to build the docs
> ---
>
> Key: CASSANDRA-13915
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13915
> Project: Cassandra
> Issue Type: Improvement
>Reporter: Joaquin Casares
>Assignee: Joaquin Casares
>
> As requested by [~rustyrazorblade], I will be adding a Docker container to
> build the docs without any prereqs (other than Docker).
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-13915) Create a Docker container to build the docs
[
https://issues.apache.org/jira/browse/CASSANDRA-13915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16191621#comment-16191621
]
Jason Brown edited comment on CASSANDRA-13915 at 10/4/17 5:28 PM:
--
bq. Dockerfile and docker-compose.yml are the default names for working with
Docker-related projects
sgtm. as stated above, I am a container noob.
bq. you two fine with me adding the doc/build directory to the ant realclean?
totally legit: {{realclean}} *should* clean all build artifacts.
When I build the docs, there's the message at the end:
{code}
The locally built documentation can be found here:
open build/html/index.html
{code}
However, when I do this on ubuntu 17.04, I get the following error:
{code}
$ open build/html/index.html
Couldn't get a file descriptor referring to the console
{code}
A naive internet search reveals that {{open}} [isn't quite what we
want|https://ubuntuforums.org/showthread.php?t=2124061}. How did you test this?
Next error: after I build the docs a few times, on the same linux install, I
run {{ant realclean}} again, which produces this error:
{code}
BUILD FAILED
/opt/dev/cassandra/build.xml:196: Unable to delete directory
/opt/dev/cassandra/build/test/lib
{code}
A quick look shows everything in {{/opt/dev/cassandra/build/}} is now owned by
root, and I need to {{sudo rm -rf build}} to clean it up. Which, of course,
leads to:
{code}
BUILD FAILED
/opt/dev/cassandra/build.xml:207: Unable to delete file
/opt/dev/cassandra/doc/build/doctrees/configuration/index.doctree
{code}
Once again, {{sudo rm -rf}}. Can you change the privileges of the artifacts
that get generated, both in ./build and ./doc/build?
UPDATE: I tried out all these commands on a Mac (High Sierra), and after
installing Docker, everything was much smoother. No permissions issues for
{{ant realclean}}, and {{open}} worked correctly. At a minimum, the
{{realclean}} problem needs to be fixed as I suspect we'll have it part of the
automation for building the online docs hosted on cassandra.apache.org. For
{{open}}, either remove the whole statement or add a note saying "mac only" or
something. [~rustyrazorblade] thoughts?
was (Author: jasobrown):
bq. Dockerfile and docker-compose.yml are the default names for working with
Docker-related projects
sgtm. as stated above, I am a container noob.
bq. you two fine with me adding the doc/build directory to the ant realclean?
totally legit: {{realclean}} *should* clean all build artifacts.
When I build the docs, there's the message at the end:
{code}
The locally built documentation can be found here:
open build/html/index.html
{code}
However, when I do this on ubuntu 17.04, I get the following error:
{code}
$ open build/html/index.html
Couldn't get a file descriptor referring to the console
{code}
A naive internet search reveals that {{open}} [isn't quite what we
want|https://ubuntuforums.org/showthread.php?t=2124061}. How did you test this?
Next error: after I build the docs a few times, on the same linux install, I
run {{ant realclean}} again, which produces this error:
{code}
BUILD FAILED
/opt/dev/cassandra/build.xml:196: Unable to delete directory
/opt/dev/cassandra/build/test/lib
{code}
A quick look shows everything in {{/opt/dev/cassandra/build/}} is now owned by
root, and I need to {{sudo rm -rf build}} to clean it up. Which, of course,
leads to:
{code}
BUILD FAILED
/opt/dev/cassandra/build.xml:207: Unable to delete file
/opt/dev/cassandra/doc/build/doctrees/configuration/index.doctree
{code}
Once again, {{sudo rm -rf}}. Can you change the privileges of the artifacts
that get generated, both in ./build and ./doc/build?
UPDATE: I tried out all these commands on a Mac (High Sierra), and after
installing Docker, everything was much smoother. No permissions issues for
{{ant realclean}}, and {{open}} worked correctly. At a minimum, the
{{realclean}} problem needs to be fixed as I suspect we'll have it part of thew
automation for building the online docs hosted on cassandra.apache.org. For
{{open}}, either remove the whole statement or add a note saying "mac only" or
something. [~rustyrazorblade] thoughts?
> Create a Docker container to build the docs
> ---
>
> Key: CASSANDRA-13915
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13915
> Project: Cassandra
> Issue Type: Improvement
>Reporter: Joaquin Casares
>Assignee: Joaquin Casares
>
> As requested by [~rustyrazorblade], I will be adding a Docker container to
> build the docs without any prereqs (other than Docker).
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additi
[jira] [Commented] (CASSANDRA-10404) Node to Node encryption transitional mode
[
https://issues.apache.org/jira/browse/CASSANDRA-10404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16191008#comment-16191008
]
Jason Brown commented on CASSANDRA-10404:
-
Missed this update, for some stupid reason
bq. 1) Assuming we have a 3.x cluster already running with ssl enabled and now
start to bump the first node to 4.0. If we connect to storage_port by default
in 4.0, won't the upgraded node fail to start with a "Unable to gossip with any
seeds" error?
As I explained in the previous comment, this is the trickiest part of this
patch. The upgraded node, after it bounces, must have at least one 3.0 node
connect to it, and have a gossip session, for the upgraded node to a) learn
about the version of the node that contacted via the handshake protocol, and b)
learn about the version of other nodes in the the cluster via that round of
gossip. Having that first round of gossip is critical for getting the versions
of peers, which we check on each {{OutboundMessagingService#tryConnect}}
attempt.
This does not guarantee that the upgraded node will learn about the 3.0 status
of nodes in it's seed list before the shadow round fails, although I think the
possiblity of shadow round failure is rather low as un-upgraded nodes will be
trying to connect to the upgraded node (as it will be seen as DOWN by the
peers, and they will be attempting to connect via gossip). The possibility of
shadow round failure is non-zero, however, and I'm open to ideas on the cluster
upgrade scenario.
bq. 2) Do we want to add an option to disable the ssl_storage_port? E.g. by
setting it to the same value as storage_port?
Maybe we can add another property under the {{server_encryption_options}},
something like {{enable_legacy_ssl_storage_port}}. That would also clean up
{{MessagingService#listen}} a little bit. wdyt?
bq. 3) doc/source/operating/security.rst: needs to be updated
Yup, I'll be happy do so when we have solid idea of where the code is going.
bq. 4) cassandra.yaml: comments for storage_port and ssl_storage_port not
accurate anymore, as both can use encryption now. We also should clearly
describe the port as legacy port only used during upgrades. There should be a
link to security.rst for further details.
Ahh, good catch. Let's figure out the code and yaml props first, then I'll come
back to properly wording the yaml comments.
bq. 5) Some of the native transport and internode netty code has become
redundant, e.g. Server.OptionalSecureInitializer and the new
OptionalSslHandler. It's probably not in scope of this ticket, but should maybe
addressed in another ticket at some point.
I totally agree about the redundancy, and I would like to make that a follow up
ticket.
bq. 6) Use of server_encryption in NettyFactory.OutboundInitializer could use
some comments, especially on why we don't have to check all remaining options
such as internode_encryption (already checked in MessagingService)
I've added a comment to {{NettyFactory.OutboundInitializeer#initChannel()}}
about where the TLS determination happens, but admittedly I'm not a fan of
burying the pointer to that so far down the code path. Not sure what's a better
solution ...
> Node to Node encryption transitional mode
> -
>
> Key: CASSANDRA-10404
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10404
> Project: Cassandra
> Issue Type: New Feature
>Reporter: Tom Lewis
>Assignee: Jason Brown
> Fix For: 4.x
>
>
> Create a transitional mode for encryption that allows encrypted and
> unencrypted traffic node-to-node during a change over to encryption from
> unencrypted. This alleviates downtime during the switch.
> This is similar to CASSANDRA-10559 which is intended for client-to-node
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13899) Fix buffer length comparison when decompressing in netty-based streaming
[
https://issues.apache.org/jira/browse/CASSANDRA-13899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16190894#comment-16190894
]
Jason Brown commented on CASSANDRA-13899:
-
updated the ticket title to more accurately describe the problem
> Fix buffer length comparison when decompressing in netty-based streaming
>
>
> Key: CASSANDRA-13899
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13899
> Project: Cassandra
> Issue Type: Bug
>Reporter: Paulo Motta
>Assignee: Jason Brown
> Fix For: 4.0
>
> Attachments: largepartition.yaml
>
>
> Streaming a single partition with ~100K rows fails with the following
> exception:
> {noformat}
> ERROR [Stream-Deserializer-/127.0.0.1:35149-a92e5e12] 2017-09-21 04:03:41,237
> StreamSession.java:617 - [Stream #c2e5b640-9eab-11e7-99c0-e9864ca8da8e]
> Streaming error occurred on session with peer 127.0.0.1
> org.apache.cassandra.streaming.StreamReceiveException:
> java.lang.RuntimeException: Last written key
> DecoratedKey(-1000328290821038380) >= current key
> DecoratedKey(-1055007227842125139) writing into
> /home/paulo/.ccm/test/node2/data0/stresscql/typestest-482ac7b09e8d11e787cf85d073c
> 8e037/na-1-big-Data.db
> at
> org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:63)
> ~[main/:na]
> at
> org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:41)
> ~[main/:na]
> at
> org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:55)
> ~[main/:na]
> at
> org.apache.cassandra.streaming.async.StreamingInboundHandler$StreamDeserializingTask.run(StreamingInboundHandler.java:178)
> ~[main/:na]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_121]
> {noformat}
> Reproduction steps:
> * Create CCM cluster with 2 nodes
> * Start only first node, disable hinted handoff
> * Run stress with the attached yaml: {{tools/bin/cassandra-stress "user
> profile=largepartition.yaml n=10K ops(insert=1) no-warmup -node whitelist
> 127.0.0.1 -mode native cql3 compression=lz4 -rate threads=4 -insert
> visits=FIXED(100K) revisit=FIXED(100K)"}}
> * Start second node, run repair on {{stresscql}} table - the exception above
> will be thrown.
> I investigated briefly and haven't found anything suspicious. This seems to
> be related to CASSANDRA-12229 as I tested the steps above in a branch without
> that and the repair completed successfully. I haven't tested with a smaller
> number of rows per partition to see at which point it starts to be a problem.
> We should probably add a regression dtest to stream large partitions to catch
> similar problems in the future.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-13899) Fix buffer length comparison when decompressing in netty-based streaming
[
https://issues.apache.org/jira/browse/CASSANDRA-13899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Brown updated CASSANDRA-13899:
Summary: Fix buffer length comparison when decompressing in netty-based
streaming (was: Streaming of compressed partition fails )
> Fix buffer length comparison when decompressing in netty-based streaming
>
>
> Key: CASSANDRA-13899
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13899
> Project: Cassandra
> Issue Type: Bug
>Reporter: Paulo Motta
>Assignee: Jason Brown
> Fix For: 4.0
>
> Attachments: largepartition.yaml
>
>
> Streaming a single partition with ~100K rows fails with the following
> exception:
> {noformat}
> ERROR [Stream-Deserializer-/127.0.0.1:35149-a92e5e12] 2017-09-21 04:03:41,237
> StreamSession.java:617 - [Stream #c2e5b640-9eab-11e7-99c0-e9864ca8da8e]
> Streaming error occurred on session with peer 127.0.0.1
> org.apache.cassandra.streaming.StreamReceiveException:
> java.lang.RuntimeException: Last written key
> DecoratedKey(-1000328290821038380) >= current key
> DecoratedKey(-1055007227842125139) writing into
> /home/paulo/.ccm/test/node2/data0/stresscql/typestest-482ac7b09e8d11e787cf85d073c
> 8e037/na-1-big-Data.db
> at
> org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:63)
> ~[main/:na]
> at
> org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:41)
> ~[main/:na]
> at
> org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:55)
> ~[main/:na]
> at
> org.apache.cassandra.streaming.async.StreamingInboundHandler$StreamDeserializingTask.run(StreamingInboundHandler.java:178)
> ~[main/:na]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_121]
> {noformat}
> Reproduction steps:
> * Create CCM cluster with 2 nodes
> * Start only first node, disable hinted handoff
> * Run stress with the attached yaml: {{tools/bin/cassandra-stress "user
> profile=largepartition.yaml n=10K ops(insert=1) no-warmup -node whitelist
> 127.0.0.1 -mode native cql3 compression=lz4 -rate threads=4 -insert
> visits=FIXED(100K) revisit=FIXED(100K)"}}
> * Start second node, run repair on {{stresscql}} table - the exception above
> will be thrown.
> I investigated briefly and haven't found anything suspicious. This seems to
> be related to CASSANDRA-12229 as I tested the steps above in a branch without
> that and the repair completed successfully. I haven't tested with a smaller
> number of rows per partition to see at which point it starts to be a problem.
> We should probably add a regression dtest to stream large partitions to catch
> similar problems in the future.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-13899) Streaming of compressed partition fails
[
https://issues.apache.org/jira/browse/CASSANDRA-13899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Brown updated CASSANDRA-13899:
Resolution: Fixed
Status: Resolved (was: Ready to Commit)
Holy crap - I didn't know you were vacationing. Sorry about that, but thanks
soo much for the review!!
committed as sha {{d080a73723d9aa402507c1ae04eef92ff6d44948}} to the cassandra
repo, and as sha {{6ea3964d18b54b4e23b6e7ebf63ca42080e8404b}} to the dtests
repo.
> Streaming of compressed partition fails
>
>
> Key: CASSANDRA-13899
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13899
> Project: Cassandra
> Issue Type: Bug
>Reporter: Paulo Motta
>Assignee: Jason Brown
> Fix For: 4.0
>
> Attachments: largepartition.yaml
>
>
> Streaming a single partition with ~100K rows fails with the following
> exception:
> {noformat}
> ERROR [Stream-Deserializer-/127.0.0.1:35149-a92e5e12] 2017-09-21 04:03:41,237
> StreamSession.java:617 - [Stream #c2e5b640-9eab-11e7-99c0-e9864ca8da8e]
> Streaming error occurred on session with peer 127.0.0.1
> org.apache.cassandra.streaming.StreamReceiveException:
> java.lang.RuntimeException: Last written key
> DecoratedKey(-1000328290821038380) >= current key
> DecoratedKey(-1055007227842125139) writing into
> /home/paulo/.ccm/test/node2/data0/stresscql/typestest-482ac7b09e8d11e787cf85d073c
> 8e037/na-1-big-Data.db
> at
> org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:63)
> ~[main/:na]
> at
> org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:41)
> ~[main/:na]
> at
> org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:55)
> ~[main/:na]
> at
> org.apache.cassandra.streaming.async.StreamingInboundHandler$StreamDeserializingTask.run(StreamingInboundHandler.java:178)
> ~[main/:na]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_121]
> {noformat}
> Reproduction steps:
> * Create CCM cluster with 2 nodes
> * Start only first node, disable hinted handoff
> * Run stress with the attached yaml: {{tools/bin/cassandra-stress "user
> profile=largepartition.yaml n=10K ops(insert=1) no-warmup -node whitelist
> 127.0.0.1 -mode native cql3 compression=lz4 -rate threads=4 -insert
> visits=FIXED(100K) revisit=FIXED(100K)"}}
> * Start second node, run repair on {{stresscql}} table - the exception above
> will be thrown.
> I investigated briefly and haven't found anything suspicious. This seems to
> be related to CASSANDRA-12229 as I tested the steps above in a branch without
> that and the repair completed successfully. I haven't tested with a smaller
> number of rows per partition to see at which point it starts to be a problem.
> We should probably add a regression dtest to stream large partitions to catch
> similar problems in the future.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-13915) Create a Docker container to build the docs
[
https://issues.apache.org/jira/browse/CASSANDRA-13915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16190692#comment-16190692
]
Jason Brown edited comment on CASSANDRA-13915 at 10/4/17 2:07 AM:
--
[~rustyrazorblade] I have way too many scars of *not* running {{ant realclean}}
that I instinctively always run it. However, for the purposes of working on the
docs, I agree that quicker turn around time is probably more important than
cleaning everything, everytime.
[~j.casares] you add the docker file as {{/doc/Dockerfile}}. Is that possibly
confusing for anyone who thinks that this will allow them to build a docker
container to actually run cassandra? At a minimum, add a quick comment at the
top of the file in case someone skips everything else in our distro, looking
for a docker file. Maybe rename the file if you think it's appropriate. Same
goes for the {{/doc/docker-compose.yml}}, as well.
was (Author: jasobrown):
[~rustyrazorblade] I have way too many scars of *not* running {{ant realclean}}
that I instinctively always run it. However, for the purposes of working on the
docs, I agree that quicker turn around time is probably more important than
cleaning everything, everytime.
[~j.casares] you add the docker file as {{/doc/Dockerfile}}. Is that possibly
confusing for anyone who thinks that this will allow them to build a docker
container to actually run cassandra? At a minimum, add a quick comment at the
top of the file in case someone skips everything else in our distro, looking
for a docker file. Same goes for the {{/doc/docker-compose.yml}}, as well.
> Create a Docker container to build the docs
> ---
>
> Key: CASSANDRA-13915
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13915
> Project: Cassandra
> Issue Type: Improvement
>Reporter: Joaquin Casares
>Assignee: Joaquin Casares
>
> As requested by [~rustyrazorblade], I will be adding a Docker container to
> build the docs without any prereqs (other than Docker).
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13915) Create a Docker container to build the docs
[
https://issues.apache.org/jira/browse/CASSANDRA-13915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16190692#comment-16190692
]
Jason Brown commented on CASSANDRA-13915:
-
[~rustyrazorblade] I have way too many scars of *not* running {{ant realclean}}
that I instinctively always run it. However, for the purposes of working on the
docs, I agree that quicker turn around time is probably more important than
cleaning everything, everytime.
[~j.casares] you add the docker file as {{/doc/Dockerfile}}. Is that possibly
confusing for anyone who thinks that this will allow them to build a docker
container to actually run cassandra? At a minimum, add a quick comment at the
top of the file in case someone skips everything else in our distro, looking
for a docker file. Same goes for the {{/doc/docker-compose.yml}}, as well.
> Create a Docker container to build the docs
> ---
>
> Key: CASSANDRA-13915
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13915
> Project: Cassandra
> Issue Type: Improvement
>Reporter: Joaquin Casares
>Assignee: Joaquin Casares
>
> As requested by [~rustyrazorblade], I will be adding a Docker container to
> build the docs without any prereqs (other than Docker).
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13899) Streaming of compressed partition fails
[
https://issues.apache.org/jira/browse/CASSANDRA-13899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16190187#comment-16190187
]
Jason Brown commented on CASSANDRA-13899:
-
[~pauloricardomg] do you think you can finish up the review on this soonish?
> Streaming of compressed partition fails
>
>
> Key: CASSANDRA-13899
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13899
> Project: Cassandra
> Issue Type: Bug
>Reporter: Paulo Motta
>Assignee: Jason Brown
> Fix For: 4.0
>
> Attachments: largepartition.yaml
>
>
> Streaming a single partition with ~100K rows fails with the following
> exception:
> {noformat}
> ERROR [Stream-Deserializer-/127.0.0.1:35149-a92e5e12] 2017-09-21 04:03:41,237
> StreamSession.java:617 - [Stream #c2e5b640-9eab-11e7-99c0-e9864ca8da8e]
> Streaming error occurred on session with peer 127.0.0.1
> org.apache.cassandra.streaming.StreamReceiveException:
> java.lang.RuntimeException: Last written key
> DecoratedKey(-1000328290821038380) >= current key
> DecoratedKey(-1055007227842125139) writing into
> /home/paulo/.ccm/test/node2/data0/stresscql/typestest-482ac7b09e8d11e787cf85d073c
> 8e037/na-1-big-Data.db
> at
> org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:63)
> ~[main/:na]
> at
> org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:41)
> ~[main/:na]
> at
> org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:55)
> ~[main/:na]
> at
> org.apache.cassandra.streaming.async.StreamingInboundHandler$StreamDeserializingTask.run(StreamingInboundHandler.java:178)
> ~[main/:na]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_121]
> {noformat}
> Reproduction steps:
> * Create CCM cluster with 2 nodes
> * Start only first node, disable hinted handoff
> * Run stress with the attached yaml: {{tools/bin/cassandra-stress "user
> profile=largepartition.yaml n=10K ops(insert=1) no-warmup -node whitelist
> 127.0.0.1 -mode native cql3 compression=lz4 -rate threads=4 -insert
> visits=FIXED(100K) revisit=FIXED(100K)"}}
> * Start second node, run repair on {{stresscql}} table - the exception above
> will be thrown.
> I investigated briefly and haven't found anything suspicious. This seems to
> be related to CASSANDRA-12229 as I tested the steps above in a branch without
> that and the repair completed successfully. I haven't tested with a smaller
> number of rows per partition to see at which point it starts to be a problem.
> We should probably add a regression dtest to stream large partitions to catch
> similar problems in the future.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-13906) Properly close StreamCompressionInputStream to release any ByteBuf
[
https://issues.apache.org/jira/browse/CASSANDRA-13906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Brown updated CASSANDRA-13906:
Resolution: Fixed
Fix Version/s: 4.0
Status: Resolved (was: Patch Available)
> Properly close StreamCompressionInputStream to release any ByteBuf
> --
>
> Key: CASSANDRA-13906
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13906
> Project: Cassandra
> Issue Type: Bug
>Reporter: Jason Brown
>Assignee: Jason Brown
> Fix For: 4.0
>
>
> When running dtests for trunk (4.x) that perform some streaming, sometimes a
> {{ByteBuf}} is not released properly, and we get this error in the logs
> (causing the dtest to fail):
> {code}
> ERROR [MessagingService-NettyOutbound-Thread-4-2] 2017-09-26 13:42:37,940
> Slf4JLogger.java:176 - LEAK: ByteBuf.release() was not called before it's
> garbage-collected. Enable advanced leak reporting to find out where the leak
> occurred. To enable advanced leak reporting, specify the JVM option
> '-Dio.netty.leakDetection.level=advanced' or call
> ResourceLeakDetector.setLevel() See
> http://netty.io/wiki/reference-counted-objects.html for more information.
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13906) Properly close StreamCompressionInputStream to release any ByteBuf
[
https://issues.apache.org/jira/browse/CASSANDRA-13906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16190184#comment-16190184
]
Jason Brown commented on CASSANDRA-13906:
-
committed as sha {{982ab93a2f8a0f5c56af9378f65d3e9e43b9}}
> Properly close StreamCompressionInputStream to release any ByteBuf
> --
>
> Key: CASSANDRA-13906
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13906
> Project: Cassandra
> Issue Type: Bug
>Reporter: Jason Brown
>Assignee: Jason Brown
>
> When running dtests for trunk (4.x) that perform some streaming, sometimes a
> {{ByteBuf}} is not released properly, and we get this error in the logs
> (causing the dtest to fail):
> {code}
> ERROR [MessagingService-NettyOutbound-Thread-4-2] 2017-09-26 13:42:37,940
> Slf4JLogger.java:176 - LEAK: ByteBuf.release() was not called before it's
> garbage-collected. Enable advanced leak reporting to find out where the leak
> occurred. To enable advanced leak reporting, specify the JVM option
> '-Dio.netty.leakDetection.level=advanced' or call
> ResourceLeakDetector.setLevel() See
> http://netty.io/wiki/reference-counted-objects.html for more information.
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13910) Consider deprecating (then removing) read_repair_chance/dclocal_read_repair_chance
[
https://issues.apache.org/jira/browse/CASSANDRA-13910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16190165#comment-16190165
]
Jason Brown commented on CASSANDRA-13910:
-
bq. I did sent an email to the user list a few days ago here. No feedback on
that threads just yet but happy to leave it at least 1-2 more weeks before
making any move.
Excellent, thank you. I think you can wait up to a week, and if no one is
screaming, please proceed full speed ahead.
> Consider deprecating (then removing)
> read_repair_chance/dclocal_read_repair_chance
> --
>
> Key: CASSANDRA-13910
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13910
> Project: Cassandra
> Issue Type: Improvement
>Reporter: Sylvain Lebresne
>Priority: Minor
> Labels: CommunityFeedbackRequested
>
> First, let me clarify so this is not misunderstood that I'm not *at all*
> suggesting to remove the read-repair mechanism of detecting and repairing
> inconsistencies between read responses: that mechanism is imo fine and
> useful. But the {{read_repair_chance}} and {{dclocal_read_repair_chance}}
> have never been about _enabling_ that mechanism, they are about querying all
> replicas (even when this is not required by the consistency level) for the
> sole purpose of maybe read-repairing some of the replica that wouldn't have
> been queried otherwise. Which btw, bring me to reason 1 for considering their
> removal: their naming/behavior is super confusing. Over the years, I've seen
> countless users (and not only newbies) misunderstanding what those options
> do, and as a consequence misunderstand when read-repair itself was happening.
> But my 2nd reason for suggesting this is that I suspect
> {{read_repair_chance}}/{{dclocal_read_repair_chance}} are, especially
> nowadays, more harmful than anything else when enabled. When those option
> kick in, what you trade-off is additional resources consumption (all nodes
> have to execute the read) for a _fairly remote chance_ of having some
> inconsistencies repaired on _some_ replica _a bit faster_ than they would
> otherwise be. To justify that last part, let's recall that:
> # most inconsistencies are actually fixed by hints in practice; and in the
> case where a node stay dead for a long time so that hints ends up timing-out,
> you really should repair the node when it comes back (if not simply
> re-bootstrapping it). Read-repair probably don't fix _that_ much stuff in
> the first place.
> # again, read-repair do happen without those options kicking in. If you do
> reads at {{QUORUM}}, inconsistencies will eventually get read-repaired all
> the same. Just a tiny bit less quickly.
> # I suspect almost everyone use a low "chance" for those options at best
> (because the extra resources consumption is real), so at the end of the day,
> it's up to chance how much faster this fixes inconsistencies.
> Overall, I'm having a hard time imagining real cases where that trade-off
> really make sense. Don't get me wrong, those options had their places a long
> time ago when hints weren't working all that well, but I think they bring
> more confusion than benefits now.
> And I think it's sane to reconsider stuffs every once in a while, and to
> clean up anything that may not make all that much sense anymore, which I
> think is the case here.
> Tl;dr, I feel the benefits brought by those options are very slim at best and
> well overshadowed by the confusion they bring, and not worth maintaining the
> code that supports them (which, to be fair, isn't huge, but getting rid of
> {{ReadCallback.AsyncRepairRunner}} wouldn't hurt for instance).
> Lastly, if the consensus here ends up being that they can have their use in
> weird case and that we fill supporting those cases is worth confusing
> everyone else and maintaining that code, I would still suggest disabling them
> totally by default.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13906) Properly close StreamCompressionInputStream to release any ByteBuf
[
https://issues.apache.org/jira/browse/CASSANDRA-13906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16190153#comment-16190153
]
Jason Brown commented on CASSANDRA-13906:
-
[~aweisberg] and I discussed offline, and I'll revert the overly cautious (and
perhaps incorrect) change around that refCnt.
> Properly close StreamCompressionInputStream to release any ByteBuf
> --
>
> Key: CASSANDRA-13906
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13906
> Project: Cassandra
> Issue Type: Bug
>Reporter: Jason Brown
>Assignee: Jason Brown
>
> When running dtests for trunk (4.x) that perform some streaming, sometimes a
> {{ByteBuf}} is not released properly, and we get this error in the logs
> (causing the dtest to fail):
> {code}
> ERROR [MessagingService-NettyOutbound-Thread-4-2] 2017-09-26 13:42:37,940
> Slf4JLogger.java:176 - LEAK: ByteBuf.release() was not called before it's
> garbage-collected. Enable advanced leak reporting to find out where the leak
> occurred. To enable advanced leak reporting, specify the JVM option
> '-Dio.netty.leakDetection.level=advanced' or call
> ResourceLeakDetector.setLevel() See
> http://netty.io/wiki/reference-counted-objects.html for more information.
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-13321) Add a checksum component for the sstable metadata (-Statistics.db) file
[ https://issues.apache.org/jira/browse/CASSANDRA-13321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Brown updated CASSANDRA-13321: Status: Ready to Commit (was: Patch Available) > Add a checksum component for the sstable metadata (-Statistics.db) file > --- > > Key: CASSANDRA-13321 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13321 > Project: Cassandra > Issue Type: Improvement >Reporter: Marcus Eriksson >Assignee: Marcus Eriksson > Fix For: 4.x > > > Since we keep important information in the sstable metadata file now, we > should add a checksum component for it. One danger being if a bit gets > flipped in repairedAt we could consider the sstable repaired when it is not. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13321) Add a checksum component for the sstable metadata (-Statistics.db) file
[
https://issues.apache.org/jira/browse/CASSANDRA-13321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16189881#comment-16189881
]
Jason Brown commented on CASSANDRA-13321:
-
Coming back to this after a long time I agree with the decision to simplify the
solution for adding checksums, and the most recent branch satisfies that.
A few small comments:
- on the serialize path, call {{DataOutputBuffer@getData()}} instead of
{{DataOutputBuffer#toByteArray}} as the latter allocates a new buffer and
copies, whereas the former just hands over it's backing byte array from the
{{ByteBuffer}}.
- {{Hashing.md5()}} - we *could* choose to swap to some other, more lighter
weight algo from guava'a {{Hasher}}, but as this code path is called very
infrequently it's probably not worth bikeshedding
- on the deserialize path, you build up the {{lengths}} map in the first
{{for}} loop. Then in the second {{for}} loop, you determine the {{size}} to
read from the {{in}} stream. Admittedly, it took me some staring at that {{if}}
to figure out what exactly it was doing. While correct, it might be friendlier
for code reading if we add the length for the {{lastType}} to the map after the
first {{for}} loop completes - then you won't need the {{if}} branching in the
second loop.
Beyond these nits, I'm +1. Nice work simplifying this patch to the minimal work
required.
> Add a checksum component for the sstable metadata (-Statistics.db) file
> ---
>
> Key: CASSANDRA-13321
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13321
> Project: Cassandra
> Issue Type: Improvement
>Reporter: Marcus Eriksson
>Assignee: Marcus Eriksson
> Fix For: 4.x
>
>
> Since we keep important information in the sstable metadata file now, we
> should add a checksum component for it. One danger being if a bit gets
> flipped in repairedAt we could consider the sstable repaired when it is not.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-13922) nodetool verify should also verify sstable metadata
[ https://issues.apache.org/jira/browse/CASSANDRA-13922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Brown updated CASSANDRA-13922: Reviewer: Jason Brown > nodetool verify should also verify sstable metadata > --- > > Key: CASSANDRA-13922 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13922 > Project: Cassandra > Issue Type: Improvement >Reporter: Marcus Eriksson >Assignee: Marcus Eriksson > Fix For: 3.0.x, 3.11.x, 4.x > > > nodetool verify should also try to deserialize the sstable metadata (and once > CASSANDRA-13321 makes it in, verify the checksums) -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-13922) nodetool verify should also verify sstable metadata
[ https://issues.apache.org/jira/browse/CASSANDRA-13922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Brown updated CASSANDRA-13922: Status: Ready to Commit (was: Patch Available) > nodetool verify should also verify sstable metadata > --- > > Key: CASSANDRA-13922 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13922 > Project: Cassandra > Issue Type: Improvement >Reporter: Marcus Eriksson >Assignee: Marcus Eriksson > Fix For: 3.0.x, 3.11.x, 4.x > > > nodetool verify should also try to deserialize the sstable metadata (and once > CASSANDRA-13321 makes it in, verify the checksums) -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13922) nodetool verify should also verify sstable metadata
[
https://issues.apache.org/jira/browse/CASSANDRA-13922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16189875#comment-16189875
]
Jason Brown commented on CASSANDRA-13922:
-
On the whole this patch looks fine. I have some petty nits are ones beyond what
you changed in this patch (just general code clean up), so feel free to ignore
or change on commit:
- remove dead imports like {{FBUtilities}} (in 3.0 branch)
- remove dead field {{badRows}} - not sure what it's original use was.
- add a comment to li. 208 {{UnfilteredRowIterator iterator}} that the variable
is intentionally unused.
Also, and this was there from before, if {{markAndThrow}} fails on
{{mutateRepairedAt}}, we'll percolate that error, rather than the
{{CorruptSSTableException}} that it is coded to do. If you feel it's warranted,
maybe add a try-catch block around the {{mutateRepairedAt}}, log that error if
one occurs, and still throw the original {{CorruptSSTableException}}.
Either way, I'm +1.
Thanks for adding a test in {{VerifyTest}} - saves me from concating a one-off
test to verify this patch ;)
> nodetool verify should also verify sstable metadata
> ---
>
> Key: CASSANDRA-13922
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13922
> Project: Cassandra
> Issue Type: Improvement
>Reporter: Marcus Eriksson
>Assignee: Marcus Eriksson
> Fix For: 3.0.x, 3.11.x, 4.x
>
>
> nodetool verify should also try to deserialize the sstable metadata (and once
> CASSANDRA-13321 makes it in, verify the checksums)
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13906) Properly close StreamCompressionInputStream to release any ByteBuf
[
https://issues.apache.org/jira/browse/CASSANDRA-13906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16186600#comment-16186600
]
Jason Brown commented on CASSANDRA-13906:
-
bq. Isn't that kind of abusing the idiom of reference counting by not counting?
That is true to a degree, but I'm never sure if any code would, in some broken
way, become executed twice and fail on the refCnt decrement and mask some other
problem. I could set the buffer to null after a simple call to {{release()}}.
wdyt?
> Properly close StreamCompressionInputStream to release any ByteBuf
> --
>
> Key: CASSANDRA-13906
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13906
> Project: Cassandra
> Issue Type: Bug
>Reporter: Jason Brown
>Assignee: Jason Brown
>
> When running dtests for trunk (4.x) that perform some streaming, sometimes a
> {{ByteBuf}} is not released properly, and we get this error in the logs
> (causing the dtest to fail):
> {code}
> ERROR [MessagingService-NettyOutbound-Thread-4-2] 2017-09-26 13:42:37,940
> Slf4JLogger.java:176 - LEAK: ByteBuf.release() was not called before it's
> garbage-collected. Enable advanced leak reporting to find out where the leak
> occurred. To enable advanced leak reporting, specify the JVM option
> '-Dio.netty.leakDetection.level=advanced' or call
> ResourceLeakDetector.setLevel() See
> http://netty.io/wiki/reference-counted-objects.html for more information.
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-13915) Create a Docker container to build the docs
[
https://issues.apache.org/jira/browse/CASSANDRA-13915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16186557#comment-16186557
]
Jason Brown edited comment on CASSANDRA-13915 at 9/29/17 10:14 PM:
---
Ok, so I applied this patch to trunk. I had docker installed, but I needed to
install docker-compose. I'm using ubuntu 17.04 and just the
docker/docker-compase packages it provides via apt-get: docker version =
"Docker version 1.12.6, build 78d1802", docker-compose version =
"docker-compose version 1.8.0, build unknown". I get the following error when
using the instructions in the patch:
{code}
ERROR: Version in "./docker-compose.yml" is unsupported. You might be seeing
this error because you're using the wrong Compose file version. Either specify
a version of "2" (or "2.0") and place your service definitions under the
`services` key, or omit the `version` key and place your service definitions at
the root of the file to use version 1.
For more on the Compose file format versions, see
https://docs.docker.com/compose/compose-file/
{code}
Note: I know nothing about docker, so this might be a noob problem.
UPDATE: this was a total noob problem. Removed the apt packages and used the
instructions on the docker website to install
was (Author: jasobrown):
Ok, so I applied this patch to trunk. I had docker installed, but I needed to
install docker-compose. I'm using ubuntu 17.04 and just the
docker/docker-compase packages it provides via apt-get: docker version =
"Docker version 1.12.6, build 78d1802", docker-compose version =
"docker-compose version 1.8.0, build unknown". I get the following error when
using the instructions in the patch:
{code}
ERROR: Version in "./docker-compose.yml" is unsupported. You might be seeing
this error because you're using the wrong Compose file version. Either specify
a version of "2" (or "2.0") and place your service definitions under the
`services` key, or omit the `version` key and place your service definitions at
the root of the file to use version 1.
For more on the Compose file format versions, see
https://docs.docker.com/compose/compose-file/
{code}
Note: I know nothing about docker, so this might be a noob problem.
> Create a Docker container to build the docs
> ---
>
> Key: CASSANDRA-13915
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13915
> Project: Cassandra
> Issue Type: Improvement
>Reporter: Joaquin Casares
>Assignee: Joaquin Casares
>
> As requested by [~rustyrazorblade], I will be adding a Docker container to
> build the docs without any prereqs (other than Docker).
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-13915) Create a Docker container to build the docs
[
https://issues.apache.org/jira/browse/CASSANDRA-13915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16186557#comment-16186557
]
Jason Brown edited comment on CASSANDRA-13915 at 9/29/17 10:00 PM:
---
Ok, so I applied this patch to trunk. I had docker installed, but I needed to
install docker-compose. I'm using ubuntu 17.04 and just the
docker/docker-compase packages it provides via apt-get: docker version =
"Docker version 1.12.6, build 78d1802", docker-compose version =
"docker-compose version 1.8.0, build unknown". I get the following error when
using the instructions in the patch:
{code}
ERROR: Version in "./docker-compose.yml" is unsupported. You might be seeing
this error because you're using the wrong Compose file version. Either specify
a version of "2" (or "2.0") and place your service definitions under the
`services` key, or omit the `version` key and place your service definitions at
the root of the file to use version 1.
For more on the Compose file format versions, see
https://docs.docker.com/compose/compose-file/
{code}
Note: I know nothing about docker, so this might be a noob problem.
was (Author: jasobrown):
Ok, so I applied this patch to trunk. I had docker installed, but I needed to
install docker-compose. I'm using ubuntu 17.04 and just the
docker/docker-compase packages it provides via apt-get: docker version =
"Docker version 1.12.6, build 78d1802", docker-compose version =
"docker-compose version 1.8.0, build unknown". I get the following error when
using the instructions in the patch:
{notformat}
ERROR: Version in "./docker-compose.yml" is unsupported. You might be seeing
this error because you're using the wrong Compose file version. Either specify
a version of "2" (or "2.0") and place your service definitions under the
`services` key, or omit the `version` key and place your service definitions at
the root of the file to use version 1.
For more on the Compose file format versions, see
https://docs.docker.com/compose/compose-file/
{noformat}
Note: I know nothing about docker, so this might be a noob problem.
> Create a Docker container to build the docs
> ---
>
> Key: CASSANDRA-13915
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13915
> Project: Cassandra
> Issue Type: Improvement
>Reporter: Joaquin Casares
>Assignee: Joaquin Casares
>
> As requested by [~rustyrazorblade], I will be adding a Docker container to
> build the docs without any prereqs (other than Docker).
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13915) Create a Docker container to build the docs
[
https://issues.apache.org/jira/browse/CASSANDRA-13915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16186557#comment-16186557
]
Jason Brown commented on CASSANDRA-13915:
-
Ok, so I applied this patch to trunk. I had docker installed, but I needed to
install docker-compose. I'm using ubuntu 17.04 and just the
docker/docker-compase packages it provides via apt-get: docker version =
"Docker version 1.12.6, build 78d1802", docker-compose version =
"docker-compose version 1.8.0, build unknown". I get the following error when
using the instructions in the patch:
{notformat}
ERROR: Version in "./docker-compose.yml" is unsupported. You might be seeing
this error because you're using the wrong Compose file version. Either specify
a version of "2" (or "2.0") and place your service definitions under the
`services` key, or omit the `version` key and place your service definitions at
the root of the file to use version 1.
For more on the Compose file format versions, see
https://docs.docker.com/compose/compose-file/
{noformat}
Note: I know nothing about docker, so this might be a noob problem.
> Create a Docker container to build the docs
> ---
>
> Key: CASSANDRA-13915
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13915
> Project: Cassandra
> Issue Type: Improvement
>Reporter: Joaquin Casares
>Assignee: Joaquin Casares
>
> As requested by [~rustyrazorblade], I will be adding a Docker container to
> build the docs without any prereqs (other than Docker).
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13915) Create a Docker container to build the docs
[ https://issues.apache.org/jira/browse/CASSANDRA-13915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16186442#comment-16186442 ] Jason Brown commented on CASSANDRA-13915: - [~rustyrazorblade] [~mshuler] I love this. Thanks for the details! I can review the ticket, as well, since I've started poking into it anyway. > Create a Docker container to build the docs > --- > > Key: CASSANDRA-13915 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13915 > Project: Cassandra > Issue Type: Improvement >Reporter: Joaquin Casares >Assignee: Joaquin Casares > > As requested by [~rustyrazorblade], I will be adding a Docker container to > build the docs without any prereqs (other than Docker). -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-12966) Gossip thread slows down when using batch commit log
[
https://issues.apache.org/jira/browse/CASSANDRA-12966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16186341#comment-16186341
]
Jason Brown commented on CASSANDRA-12966:
-
[~jjirsa] fixed
> Gossip thread slows down when using batch commit log
>
>
> Key: CASSANDRA-12966
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12966
> Project: Cassandra
> Issue Type: Improvement
>Reporter: Jason Brown
>Assignee: Jason Brown
>Priority: Minor
> Fix For: 3.0.15, 3.11.1, 4.0
>
>
> When using batch commit log mode, the Gossip thread slows down when peers
> after a node bounces. This is because we perform a bunch of updates to the
> peers table via {{SystemKeyspace.updatePeerInfo}}, which is a synchronized
> method. How quickly each one of those individual updates takes depends on how
> busy the system is at the time wrt write traffic. If the system is largely
> quiescent, each update will be relatively quick (just waiting for the fsync).
> If the system is getting a lot of writes, and depending on the
> commitlog_sync_batch_window_in_ms, each of the Gossip thread's updates can
> get stuck in the backlog, which causes the Gossip thread to stop processing.
> We have observed in large clusters that a rolling restart causes triggers and
> exacerbates this behavior.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-12966) Gossip thread slows down when using batch commit log
[
https://issues.apache.org/jira/browse/CASSANDRA-12966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Brown updated CASSANDRA-12966:
Fix Version/s: 4.0
3.11.1
3.0.15
> Gossip thread slows down when using batch commit log
>
>
> Key: CASSANDRA-12966
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12966
> Project: Cassandra
> Issue Type: Improvement
>Reporter: Jason Brown
>Assignee: Jason Brown
>Priority: Minor
> Fix For: 3.0.15, 3.11.1, 4.0
>
>
> When using batch commit log mode, the Gossip thread slows down when peers
> after a node bounces. This is because we perform a bunch of updates to the
> peers table via {{SystemKeyspace.updatePeerInfo}}, which is a synchronized
> method. How quickly each one of those individual updates takes depends on how
> busy the system is at the time wrt write traffic. If the system is largely
> quiescent, each update will be relatively quick (just waiting for the fsync).
> If the system is getting a lot of writes, and depending on the
> commitlog_sync_batch_window_in_ms, each of the Gossip thread's updates can
> get stuck in the backlog, which causes the Gossip thread to stop processing.
> We have observed in large clusters that a rolling restart causes triggers and
> exacerbates this behavior.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13915) Create a Docker container to build the docs
[ https://issues.apache.org/jira/browse/CASSANDRA-13915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16186203#comment-16186203 ] Jason Brown commented on CASSANDRA-13915: - [~rustyrazorblade] & [~j.casares] stoopidly naive question: why do we need Docker to build our docs? > Create a Docker container to build the docs > --- > > Key: CASSANDRA-13915 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13915 > Project: Cassandra > Issue Type: Improvement >Reporter: Joaquin Casares >Assignee: Joaquin Casares > > As requested by [~rustyrazorblade], I will be adding a Docker container to > build the docs without any prereqs (other than Docker). -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13906) Properly close StreamCompressionInputStream to release any ByteBuf
[
https://issues.apache.org/jira/browse/CASSANDRA-13906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16186149#comment-16186149
]
Jason Brown commented on CASSANDRA-13906:
-
OK, I pushed up a fresh branch using try-with-resources for both
{{StreamReader}} and {{CompressedStreamReader}}. utests and dtests have been
been kicked off, as well.
> Properly close StreamCompressionInputStream to release any ByteBuf
> --
>
> Key: CASSANDRA-13906
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13906
> Project: Cassandra
> Issue Type: Bug
>Reporter: Jason Brown
>Assignee: Jason Brown
>
> When running dtests for trunk (4.x) that perform some streaming, sometimes a
> {{ByteBuf}} is not released properly, and we get this error in the logs
> (causing the dtest to fail):
> {code}
> ERROR [MessagingService-NettyOutbound-Thread-4-2] 2017-09-26 13:42:37,940
> Slf4JLogger.java:176 - LEAK: ByteBuf.release() was not called before it's
> garbage-collected. Enable advanced leak reporting to find out where the leak
> occurred. To enable advanced leak reporting, specify the JVM option
> '-Dio.netty.leakDetection.level=advanced' or call
> ResourceLeakDetector.setLevel() See
> http://netty.io/wiki/reference-counted-objects.html for more information.
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13906) Properly close StreamCompressionInputStream to release any ByteBuf
[
https://issues.apache.org/jira/browse/CASSANDRA-13906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16186066#comment-16186066
]
Jason Brown commented on CASSANDRA-13906:
-
bq. Wouldn't the most idiomatic way to do this be to have
{{TrackedDataInputPlus}} implement {{Closable}} and use try with resources
I thought about that when I was fixing this, but {{TrackedDataInputPlus}}
wraps a {{DataInput}}, which does not declare a {{#close()}} method. In a
{{TrackedDataInputPlus#close()}} I could check if the wrapped instance
implements {{Closeable}} and invoke it if it does. wdyt?
There are a couple of other uses of {{TrackedDataInputPlus}} ({{HintMessage}},
{{IndexedEntry}} called on load saved cache path), but they should not be
affected by {{TrackedDataInputPlus}} implementing {{Closeable}} as the are not
allocated via try-with-resources.
Note: if we choose to make this change, which is reasonable, I can also cleanup
{{CompressedStreamReader}} to also allocate {{TrackedDataInputPlus}} in a
try-with-resources - it has the same concerns as {{StreamReader}} that you
raised. Branch coming shortly ...
> Properly close StreamCompressionInputStream to release any ByteBuf
> --
>
> Key: CASSANDRA-13906
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13906
> Project: Cassandra
> Issue Type: Bug
>Reporter: Jason Brown
>Assignee: Jason Brown
>
> When running dtests for trunk (4.x) that perform some streaming, sometimes a
> {{ByteBuf}} is not released properly, and we get this error in the logs
> (causing the dtest to fail):
> {code}
> ERROR [MessagingService-NettyOutbound-Thread-4-2] 2017-09-26 13:42:37,940
> Slf4JLogger.java:176 - LEAK: ByteBuf.release() was not called before it's
> garbage-collected. Enable advanced leak reporting to find out where the leak
> occurred. To enable advanced leak reporting, specify the JVM option
> '-Dio.netty.leakDetection.level=advanced' or call
> ResourceLeakDetector.setLevel() See
> http://netty.io/wiki/reference-counted-objects.html for more information.
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-13905) Correctly close netty channels when a stream session ends
[
https://issues.apache.org/jira/browse/CASSANDRA-13905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Brown updated CASSANDRA-13905:
Resolution: Fixed
Fix Version/s: (was: 4.x)
4.0
Status: Resolved (was: Ready to Commit)
bq. Collectors are the don't be lazy step in streams
Yeah that's where I messed up originally :).
committed as sha {[ebefc96a8fe63aca5f324984f7f3147f10218643}}.
> Correctly close netty channels when a stream session ends
> -
>
> Key: CASSANDRA-13905
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13905
> Project: Cassandra
> Issue Type: Bug
> Components: Streaming and Messaging
>Reporter: Jason Brown
>Assignee: Jason Brown
> Fix For: 4.0
>
>
> Netty channels in stream sessions were not being closed correctly. TL;DR I
> was using a lambda that was not executing as it is lazily evaluated. This was
> causing a {{RejectedExecutionException}} at the end of some streaming-related
> dtests
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13789) Reduce memory copies and object creations when acting on ByteBufs
[
https://issues.apache.org/jira/browse/CASSANDRA-13789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16185791#comment-16185791
]
Jason Brown commented on CASSANDRA-13789:
-
reverted as sha {{e296ff0632a41d5bbd29b407a83d886c10787f51}}.
> Reduce memory copies and object creations when acting on ByteBufs
> --
>
> Key: CASSANDRA-13789
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13789
> Project: Cassandra
> Issue Type: Improvement
>Reporter: Norman Maurer
>Assignee: Norman Maurer
> Fix For: 4.0
>
> Attachments:
> 0001-CBUtil.sizeOfLongString-encodes-String-to-byte-to-ca.patch,
> 0001-Reduce-memory-copies-and-object-creations-when-actin.patch
>
>
> There are multiple "low-hanging-fruits" when it comes to reduce memory copies
> and object allocations when acting on ByteBufs.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13789) Reduce memory copies and object creations when acting on ByteBufs
[
https://issues.apache.org/jira/browse/CASSANDRA-13789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16185785#comment-16185785
]
Jason Brown commented on CASSANDRA-13789:
-
I like [~slebresne]'s suggestion, and we can pursue that on the netty project
itself, and upgrade if it happens. For the time being, though, I will revert
the {{CBUtil#writeValue()}} problem that [~Stefania] uncovered.
> Reduce memory copies and object creations when acting on ByteBufs
> --
>
> Key: CASSANDRA-13789
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13789
> Project: Cassandra
> Issue Type: Improvement
>Reporter: Norman Maurer
>Assignee: Norman Maurer
> Fix For: 4.0
>
> Attachments:
> 0001-CBUtil.sizeOfLongString-encodes-String-to-byte-to-ca.patch,
> 0001-Reduce-memory-copies-and-object-creations-when-actin.patch
>
>
> There are multiple "low-hanging-fruits" when it comes to reduce memory copies
> and object allocations when acting on ByteBufs.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-13741) Replace Cassandra's lz4-1.3.0.jar with lz4-java-1.4.0.jar
[
https://issues.apache.org/jira/browse/CASSANDRA-13741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Brown updated CASSANDRA-13741:
Resolution: Fixed
Fix Version/s: (was: 4.x)
4.0
Status: Resolved (was: Ready to Commit)
+1. Committed as sha {{55772088e0f6e227c45df935bd8283492a5f73f8}}. Thanks,
everybody.
> Replace Cassandra's lz4-1.3.0.jar with lz4-java-1.4.0.jar
> -
>
> Key: CASSANDRA-13741
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13741
> Project: Cassandra
> Issue Type: Improvement
> Components: Libraries
>Reporter: Amitkumar Ghatwal
>Assignee: Michael Kjellman
> Fix For: 4.0
>
>
> Hi All,
> The latest lz4-java library has been released
> (https://github.com/lz4/lz4-java/releases) and uploaded to maven central .
> Please replace in mainline the current version ( 1.3.0) with the latest one (
> 1.4.0) from here - http://repo1.maven.org/maven2/org/lz4/lz4-java/1.4.0/
> Adding : [~ReiOdaira].
> Regards,
> Amit
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-13741) Replace Cassandra's lz4-1.3.0.jar with lz4-java-1.4.0.jar
[ https://issues.apache.org/jira/browse/CASSANDRA-13741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Brown updated CASSANDRA-13741: Status: Ready to Commit (was: Patch Available) > Replace Cassandra's lz4-1.3.0.jar with lz4-java-1.4.0.jar > - > > Key: CASSANDRA-13741 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13741 > Project: Cassandra > Issue Type: Improvement > Components: Libraries >Reporter: Amitkumar Ghatwal >Assignee: Michael Kjellman > Fix For: 4.x > > > Hi All, > The latest lz4-java library has been released > (https://github.com/lz4/lz4-java/releases) and uploaded to maven central . > Please replace in mainline the current version ( 1.3.0) with the latest one ( > 1.4.0) from here - http://repo1.maven.org/maven2/org/lz4/lz4-java/1.4.0/ > Adding : [~ReiOdaira]. > Regards, > Amit -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Resolved] (CASSANDRA-13149) AssertionError prepending to a list
[
https://issues.apache.org/jira/browse/CASSANDRA-13149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Brown resolved CASSANDRA-13149.
-
Resolution: Fixed
Fix Version/s: 4.0
3.11.1
3.0.15
Committed as sha {{ab0adf9f9bc72074a02025bdecd9479f790d6463}}. Thanks!
> AssertionError prepending to a list
> ---
>
> Key: CASSANDRA-13149
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13149
> Project: Cassandra
> Issue Type: Bug
> Components: CQL
> Environment: 3.0.8
>Reporter: Steven Warren
>Assignee: Jason Brown
> Fix For: 3.0.15, 3.11.1, 4.0
>
>
> Prepending to a list produces the following AssertionError randomly. Changing
> the update to append (and sort in the client) works around the issue.
> {code}
> java.lang.AssertionError: null
> at
> org.apache.cassandra.cql3.Lists$PrecisionTime.getNext(Lists.java:275)
> ~[apache-cassandra-3.0.8.jar:3.0.8]
> at org.apache.cassandra.cql3.Lists$Prepender.execute(Lists.java:430)
> ~[apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.cql3.statements.UpdateStatement.addUpdateForKey(UpdateStatement.java:94)
> ~[apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.cql3.statements.ModificationStatement.addUpdates(ModificationStatement.java:682)
> ~[apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.cql3.statements.ModificationStatement.getMutations(ModificationStatement.java:613)
> ~[apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.cql3.statements.ModificationStatement.executeWithoutCondition(ModificationStatement.java:420)
> ~[apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.cql3.statements.ModificationStatement.execute(ModificationStatement.java:408)
> ~[apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:206)
> ~[apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:487)
> ~[apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:464)
> ~[apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.transport.messages.ExecuteMessage.execute(ExecuteMessage.java:130)
> ~[apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:507)
> [apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:401)
> [apache-cassandra-3.0.8.jar:3.0.8]
> at
> io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
> [netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
> [netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.access$700(AbstractChannelHandlerContext.java:32)
> [netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext$8.run(AbstractChannelHandlerContext.java:324)
> [netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [na:1.8.0_101]
> at
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164)
> [apache-cassandra-3.0.8.jar:3.0.8]
> at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105)
> [apache-cassandra-3.0.8.jar:3.0.8]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_101]
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13910) Consider deprecating (then removing) read_repair_chance/dclocal_read_repair_chance
[
https://issues.apache.org/jira/browse/CASSANDRA-13910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16184549#comment-16184549
]
Jason Brown commented on CASSANDRA-13910:
-
I like this idea. However, I think it would be good to send out a discussion
thread to dev@/user@ to get user input rather than just forcing this change on
them. While I find it unlikely, there might be some users who find (or think
they find) value from this feature.
> Consider deprecating (then removing)
> read_repair_chance/dclocal_read_repair_chance
> --
>
> Key: CASSANDRA-13910
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13910
> Project: Cassandra
> Issue Type: Improvement
>Reporter: Sylvain Lebresne
>Priority: Minor
>
> First, let me clarify so this is not misunderstood that I'm not *at all*
> suggesting to remove the read-repair mechanism of detecting and repairing
> inconsistencies between read responses: that mechanism is imo fine and
> useful. But the {{read_repair_chance}} and {{dclocal_read_repair_chance}}
> have never been about _enabling_ that mechanism, they are about querying all
> replicas (even when this is not required by the consistency level) for the
> sole purpose of maybe read-repairing some of the replica that wouldn't have
> been queried otherwise. Which btw, bring me to reason 1 for considering their
> removal: their naming/behavior is super confusing. Over the years, I've seen
> countless users (and not only newbies) misunderstanding what those options
> do, and as a consequence misunderstand when read-repair itself was happening.
> But my 2nd reason for suggesting this is that I suspect
> {{read_repair_chance}}/{{dclocal_read_repair_chance}} are, especially
> nowadays, more harmful than anything else when enabled. When those option
> kick in, what you trade-off is additional resources consumption (all nodes
> have to execute the read) for a _fairly remote chance_ of having some
> inconsistencies repaired on _some_ replica _a bit faster_ than they would
> otherwise be. To justify that last part, let's recall that:
> # most inconsistencies are actually fixed by hints in practice; and in the
> case where a node stay dead for a long time so that hints ends up timing-out,
> you really should repair the node when it comes back (if not simply
> re-bootstrapping it). Read-repair probably don't fix _that_ much stuff in
> the first place.
> # again, read-repair do happen without those options kicking in. If you do
> reads at {{QUORUM}}, inconsistencies will eventually get read-repaired all
> the same. Just a tiny bit less quickly.
> # I suspect almost everyone use a low "chance" for those options at best
> (because the extra resources consumption is real), so at the end of the day,
> it's up to chance how much faster this fixes inconsistencies.
> Overall, I'm having a hard time imagining real cases where that trade-off
> really make sense. Don't get me wrong, those options had their places a long
> time ago when hints weren't working all that well, but I think they bring
> more confusion than benefits now.
> And I think it's sane to reconsider stuffs every once in a while, and to
> clean up anything that may not make all that much sense anymore, which I
> think is the case here.
> Tl;dr, I feel the benefits brought by those options are very slim at best and
> well overshadowed by the confusion they bring, and not worth maintaining the
> code that supports them (which, to be fair, isn't huge, but getting rid of
> {{ReadCallback.AsyncRepairRunner}} wouldn't hurt for instance).
> Lastly, if the consensus here ends up being that they can have their use in
> weird case and that we fill supporting those cases is worth confusing
> everyone else and maintaining that code, I would still suggest disabling them
> totally by default.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13789) Reduce memory copies and object creations when acting on ByteBufs
[ https://issues.apache.org/jira/browse/CASSANDRA-13789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16184214#comment-16184214 ] Jason Brown commented on CASSANDRA-13789: - [~norman] Thanks for the update. I'll go ahead and revert that part of the patch today. > Reduce memory copies and object creations when acting on ByteBufs > -- > > Key: CASSANDRA-13789 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13789 > Project: Cassandra > Issue Type: Improvement >Reporter: Norman Maurer >Assignee: Norman Maurer > Fix For: 4.0 > > Attachments: > 0001-CBUtil.sizeOfLongString-encodes-String-to-byte-to-ca.patch, > 0001-Reduce-memory-copies-and-object-creations-when-actin.patch > > > There are multiple "low-hanging-fruits" when it comes to reduce memory copies > and object allocations when acting on ByteBufs. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13899) Streaming of compressed partition fails
[
https://issues.apache.org/jira/browse/CASSANDRA-13899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16184194#comment-16184194
]
Jason Brown commented on CASSANDRA-13899:
-
[~pauloricardomg] Good call on dropping the {{chunk_length_in_kb}} down to 1
for the dtest. I can trigger the error on trunk with it and a much lower insert
count in under 80 seconds total. I've turned it into a
[dtest|https://github.com/jasobrown/cassandra-dtest/tree/13899]. Please review
and let me know what you think.
> Streaming of compressed partition fails
>
>
> Key: CASSANDRA-13899
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13899
> Project: Cassandra
> Issue Type: Bug
>Reporter: Paulo Motta
>Assignee: Jason Brown
> Fix For: 4.0
>
> Attachments: largepartition.yaml
>
>
> Streaming a single partition with ~100K rows fails with the following
> exception:
> {noformat}
> ERROR [Stream-Deserializer-/127.0.0.1:35149-a92e5e12] 2017-09-21 04:03:41,237
> StreamSession.java:617 - [Stream #c2e5b640-9eab-11e7-99c0-e9864ca8da8e]
> Streaming error occurred on session with peer 127.0.0.1
> org.apache.cassandra.streaming.StreamReceiveException:
> java.lang.RuntimeException: Last written key
> DecoratedKey(-1000328290821038380) >= current key
> DecoratedKey(-1055007227842125139) writing into
> /home/paulo/.ccm/test/node2/data0/stresscql/typestest-482ac7b09e8d11e787cf85d073c
> 8e037/na-1-big-Data.db
> at
> org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:63)
> ~[main/:na]
> at
> org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:41)
> ~[main/:na]
> at
> org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:55)
> ~[main/:na]
> at
> org.apache.cassandra.streaming.async.StreamingInboundHandler$StreamDeserializingTask.run(StreamingInboundHandler.java:178)
> ~[main/:na]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_121]
> {noformat}
> Reproduction steps:
> * Create CCM cluster with 2 nodes
> * Start only first node, disable hinted handoff
> * Run stress with the attached yaml: {{tools/bin/cassandra-stress "user
> profile=largepartition.yaml n=10K ops(insert=1) no-warmup -node whitelist
> 127.0.0.1 -mode native cql3 compression=lz4 -rate threads=4 -insert
> visits=FIXED(100K) revisit=FIXED(100K)"}}
> * Start second node, run repair on {{stresscql}} table - the exception above
> will be thrown.
> I investigated briefly and haven't found anything suspicious. This seems to
> be related to CASSANDRA-12229 as I tested the steps above in a branch without
> that and the repair completed successfully. I haven't tested with a smaller
> number of rows per partition to see at which point it starts to be a problem.
> We should probably add a regression dtest to stream large partitions to catch
> similar problems in the future.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-13291) Replace usages of MessageDigest with Guava's Hasher
[
https://issues.apache.org/jira/browse/CASSANDRA-13291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16184029#comment-16184029
]
Jason Brown edited comment on CASSANDRA-13291 at 9/28/17 12:21 PM:
---
I'm +1 on the code. [~mkjellman] wdyt about including the microbenchmark I put
together?
UPDATE: Actually, I did have a few comments/questions:
- it would be nice to have a unit test class for {{HashingUtils}}. mostly, i'm
thinking it would be nice to cover the different cases in {{#updateBytes}}:
empty buffer, on-heap, and off-heap (large and small).
- {{RandomPartitioner#hashToBigInteger}} - is this suppossed to explicitly use
{{MD5Digest#hash}} still? If so, can we add a comment that it's intentionally
doing so?
- for the {{Validator.CountingHasher#put*}} primitive functions, I think you
need to increment the {{count}} member field by the appropriate number of
bytes. I *think* the way the existing {{MessageDigest}} works is that for each
byte {{#updateEngine}} is invoked, so we would count it in the original.
- petty nits for for petty nits gods: clean up the added but unused imports on
{{MessagingService}} and {{SchemaConstants}}
was (Author: jasobrown):
I'm +1 on the code. [~mkjellman] wdyt about including the microbenchmark I put
together?
> Replace usages of MessageDigest with Guava's Hasher
> ---
>
> Key: CASSANDRA-13291
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13291
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
>Reporter: Michael Kjellman
>Assignee: Michael Kjellman
> Attachments: CASSANDRA-13291-trunk.diff
>
>
> During my profiling of C* I frequently see lots of aggregate time across
> threads being spent inside the MD5 MessageDigest implementation. Given that
> there are tons of modern alternative hashing functions better than MD5
> available -- both in terms of providing better collision resistance and
> actual computational speed -- I wanted to switch out our usage of MD5 for
> alternatives (like adler128 or murmur3_128) and test for performance
> improvements.
> Unfortunately, I found given the fact we use MessageDigest everywhere --
> switching out the hashing function to something like adler128 or murmur3_128
> (for example) -- which don't ship with the JDK -- wasn't straight forward.
> The goal of this ticket is to propose switching out usages of MessageDigest
> directly in favor of Hasher from Guava. This means going forward we can
> change a single line of code to switch the hashing algorithm being used
> (assuming there is an implementation in Guava).
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-13899) Streaming of compressed partition fails
[
https://issues.apache.org/jira/browse/CASSANDRA-13899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Brown updated CASSANDRA-13899:
Reviewer: Paulo Motta
> Streaming of compressed partition fails
>
>
> Key: CASSANDRA-13899
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13899
> Project: Cassandra
> Issue Type: Bug
>Reporter: Paulo Motta
>Assignee: Jason Brown
> Fix For: 4.0
>
> Attachments: largepartition.yaml
>
>
> Streaming a single partition with ~100K rows fails with the following
> exception:
> {noformat}
> ERROR [Stream-Deserializer-/127.0.0.1:35149-a92e5e12] 2017-09-21 04:03:41,237
> StreamSession.java:617 - [Stream #c2e5b640-9eab-11e7-99c0-e9864ca8da8e]
> Streaming error occurred on session with peer 127.0.0.1
> org.apache.cassandra.streaming.StreamReceiveException:
> java.lang.RuntimeException: Last written key
> DecoratedKey(-1000328290821038380) >= current key
> DecoratedKey(-1055007227842125139) writing into
> /home/paulo/.ccm/test/node2/data0/stresscql/typestest-482ac7b09e8d11e787cf85d073c
> 8e037/na-1-big-Data.db
> at
> org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:63)
> ~[main/:na]
> at
> org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:41)
> ~[main/:na]
> at
> org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:55)
> ~[main/:na]
> at
> org.apache.cassandra.streaming.async.StreamingInboundHandler$StreamDeserializingTask.run(StreamingInboundHandler.java:178)
> ~[main/:na]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_121]
> {noformat}
> Reproduction steps:
> * Create CCM cluster with 2 nodes
> * Start only first node, disable hinted handoff
> * Run stress with the attached yaml: {{tools/bin/cassandra-stress "user
> profile=largepartition.yaml n=10K ops(insert=1) no-warmup -node whitelist
> 127.0.0.1 -mode native cql3 compression=lz4 -rate threads=4 -insert
> visits=FIXED(100K) revisit=FIXED(100K)"}}
> * Start second node, run repair on {{stresscql}} table - the exception above
> will be thrown.
> I investigated briefly and haven't found anything suspicious. This seems to
> be related to CASSANDRA-12229 as I tested the steps above in a branch without
> that and the repair completed successfully. I haven't tested with a smaller
> number of rows per partition to see at which point it starts to be a problem.
> We should probably add a regression dtest to stream large partitions to catch
> similar problems in the future.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13291) Replace usages of MessageDigest with Guava's Hasher
[ https://issues.apache.org/jira/browse/CASSANDRA-13291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16184029#comment-16184029 ] Jason Brown commented on CASSANDRA-13291: - I'm +1 on the code. [~mkjellman] wdyt about including the microbenchmark I put together? > Replace usages of MessageDigest with Guava's Hasher > --- > > Key: CASSANDRA-13291 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13291 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Michael Kjellman >Assignee: Michael Kjellman > Attachments: CASSANDRA-13291-trunk.diff > > > During my profiling of C* I frequently see lots of aggregate time across > threads being spent inside the MD5 MessageDigest implementation. Given that > there are tons of modern alternative hashing functions better than MD5 > available -- both in terms of providing better collision resistance and > actual computational speed -- I wanted to switch out our usage of MD5 for > alternatives (like adler128 or murmur3_128) and test for performance > improvements. > Unfortunately, I found given the fact we use MessageDigest everywhere -- > switching out the hashing function to something like adler128 or murmur3_128 > (for example) -- which don't ship with the JDK -- wasn't straight forward. > The goal of this ticket is to propose switching out usages of MessageDigest > directly in favor of Hasher from Guava. This means going forward we can > change a single line of code to switch the hashing algorithm being used > (assuming there is an implementation in Guava). -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13291) Replace usages of MessageDigest with Guava's Hasher
[
https://issues.apache.org/jira/browse/CASSANDRA-13291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16183505#comment-16183505
]
Jason Brown commented on CASSANDRA-13291:
-
A [slightly
fairer|https://github.com/jasobrown/cassandra/commit/e57bc21903687dfed573ff427ad4eeededac41a9]
comparison, wherein I call {{MessageDigest#close()}} on a prototype instance
each time, instead of a fresh new instance via
{{MessageDigest.getInstance("MD5")}}.
Update results:
{noformat}
[java] Benchmark(bufferSize) Mode Cnt
Score Error Units
[java] HashingBench.benchHasherMD531 avgt5
340.186 ± 54.611 ns/op
[java] HashingBench.benchHasherMD5 131 avgt5
708.117 ± 42.826 ns/op
[java] HashingBench.benchHasherMD5 517 avgt5
1801.402 ± 47.358 ns/op
[java] HashingBench.benchHasherMD5 2041 avgt5
6294.723 ± 518.325 ns/op
[java] HashingBench.benchHasherMurmur3_12831 avgt5
286.312 ± 65.617 ns/op
[java] HashingBench.benchHasherMurmur3_128 131 avgt5
429.138 ± 36.589 ns/op
[java] HashingBench.benchHasherMurmur3_128 517 avgt5
908.452 ± 27.860 ns/op
[java] HashingBench.benchHasherMurmur3_128 2041 avgt5
2830.657 ± 225.470 ns/op
[java] HashingBench.benchMessageDigestMD5 31 avgt5
484.350 ± 474.141 ns/op
[java] HashingBench.benchMessageDigestMD5131 avgt5
1059.691 ± 53.677 ns/op
[java] HashingBench.benchMessageDigestMD5517 avgt5
2557.586 ± 319.597 ns/op
[java] HashingBench.benchMessageDigestMD5 2041 avgt5
8585.662 ± 135.474 ns/op
{noformat}
Either way, the guava hasher is faster.
In other news, The guava MD5 implementation uses {{MessageDigest}} under the
covers, so I think the hash results from the guava md5 and the
{{MessageDigest}} should be the same. [~mkjellman] can you confirm?
> Replace usages of MessageDigest with Guava's Hasher
> ---
>
> Key: CASSANDRA-13291
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13291
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
>Reporter: Michael Kjellman
>Assignee: Michael Kjellman
> Attachments: CASSANDRA-13291-trunk.diff
>
>
> During my profiling of C* I frequently see lots of aggregate time across
> threads being spent inside the MD5 MessageDigest implementation. Given that
> there are tons of modern alternative hashing functions better than MD5
> available -- both in terms of providing better collision resistance and
> actual computational speed -- I wanted to switch out our usage of MD5 for
> alternatives (like adler128 or murmur3_128) and test for performance
> improvements.
> Unfortunately, I found given the fact we use MessageDigest everywhere --
> switching out the hashing function to something like adler128 or murmur3_128
> (for example) -- which don't ship with the JDK -- wasn't straight forward.
> The goal of this ticket is to propose switching out usages of MessageDigest
> directly in favor of Hasher from Guava. This means going forward we can
> change a single line of code to switch the hashing algorithm being used
> (assuming there is an implementation in Guava).
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-13291) Replace usages of MessageDigest with Guava's Hasher
[
https://issues.apache.org/jira/browse/CASSANDRA-13291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16183400#comment-16183400
]
Jason Brown edited comment on CASSANDRA-13291 at 9/27/17 11:19 PM:
---
I knocked out a quick [JMH
bench|https://github.com/jasobrown/cassandra/commit/cd35b1a771a74c2bf1d3bf2c0916967e74821385]
to see what the difference between {{MessageDigest}} and {{Hasher}} would be.
I selected guava's MD5 and murmur3_128 hashers for comparison. Here's what I
found:
{noformat}
[java] Benchmark(bufferSize) Mode Cnt
Score Error Units
[java] HashingBench.benchHasherMD531 avgt5
336.613 ± 18.826 ns/op
[java] HashingBench.benchHasherMD5 131 avgt5
709.226 ± 19.489 ns/op
[java] HashingBench.benchHasherMD5 517 avgt5
1800.091 ± 37.748 ns/op
[java] HashingBench.benchHasherMD5 2041 avgt5
6275.607 ± 623.008 ns/op
[java] HashingBench.benchHasherMurmur3_12831 avgt5
260.859 ± 39.229 ns/op
[java] HashingBench.benchHasherMurmur3_128 131 avgt5
421.268 ± 68.287 ns/op
[java] HashingBench.benchHasherMurmur3_128 517 avgt5
861.577 ± 68.423 ns/op
[java] HashingBench.benchHasherMurmur3_128 2041 avgt5
2863.952 ± 314.205 ns/op
[java] HashingBench.benchMessageDigestMD5 31 avgt5
787.373 ± 69.869 ns/op
[java] HashingBench.benchMessageDigestMD5131 avgt5
1264.677 ± 117.790 ns/op
[java] HashingBench.benchMessageDigestMD5517 avgt5
2822.846 ± 178.416 ns/op
[java] HashingBench.benchMessageDigestMD5 2041 avgt5
9611.875 ± 1760.809 ns/op
{noformat}
Naively, I used byte arrays for four varying sizes, updated the hasher/digest,
and got the hashed result. I selected buffer sizes that are close to
powers-of-2, but intentionally not. It looks like the guava {{Hasher}} s do
indeed perform better than {{MessageDigest}}.
was (Author: jasobrown):
I knocked out a quick [JMH
bench|https://github.com/jasobrown/cassandra/commit/cd35b1a771a74c2bf1d3bf2c0916967e74821385]
to see what the difference between {{MessageDigest}} and {{Hasher}} would be.
I selected guava's MD5 and murmur3_128 hashers for comparison. Here's what I
found:
{noformat}
[java] Benchmark(bufferSize) Mode Cnt
Score Error Units
[java] HashingBench.benchHasherMD531 avgt5
336.613 ± 18.826 ns/op
[java] HashingBench.benchHasherMD5 131 avgt5
709.226 ± 19.489 ns/op
[java] HashingBench.benchHasherMD5 517 avgt5
1800.091 ± 37.748 ns/op
[java] HashingBench.benchHasherMD5 2041 avgt5
6275.607 ± 623.008 ns/op
[java] HashingBench.benchHasherMurmur3_12831 avgt5
260.859 ± 39.229 ns/op
[java] HashingBench.benchHasherMurmur3_128 131 avgt5
421.268 ± 68.287 ns/op
[java] HashingBench.benchHasherMurmur3_128 517 avgt5
861.577 ± 68.423 ns/op
[java] HashingBench.benchHasherMurmur3_128 2041 avgt5
2863.952 ± 314.205 ns/op
[java] HashingBench.benchMessageDigestMD5 31 avgt5
787.373 ± 69.869 ns/op
[java] HashingBench.benchMessageDigestMD5131 avgt5
1264.677 ± 117.790 ns/op
[java] HashingBench.benchMessageDigestMD5517 avgt5
2822.846 ± 178.416 ns/op
[java] HashingBench.benchMessageDigestMD5 2041 avgt5
9611.875 ± 1760.809 ns/op
{noformat}
Naively, I used byte arrays for four varying sizes, updated the hasher/digest,
and got the hashed result. I selected buffer sizes that are close to
powers-of-2, but intentionally not. It looks like the guava {{Hasher}}s do
indeed perform better than {{MessageDigest}}.
> Replace usages of MessageDigest with Guava's Hasher
> ---
>
> Key: CASSANDRA-13291
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13291
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
>Reporter: Michael Kjellman
>Assignee: Michael Kjellman
> Attachments: CASSANDRA-13291-trunk.diff
>
>
> During my profiling of C* I frequently see lots of aggregate time across
> threads being spent inside the MD5 MessageDigest implementation. Given that
> there are tons of modern alternative hashing functions better than MD5
> available -- both in terms of providing better collision resistance and
> actual computational speed -- I wanted to switch out our usage of MD5 for
[jira] [Commented] (CASSANDRA-13291) Replace usages of MessageDigest with Guava's Hasher
[
https://issues.apache.org/jira/browse/CASSANDRA-13291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16183400#comment-16183400
]
Jason Brown commented on CASSANDRA-13291:
-
I knocked out a quick [JMH
bench|https://github.com/jasobrown/cassandra/commit/cd35b1a771a74c2bf1d3bf2c0916967e74821385]
to see what the difference between {{MessageDigest}} and {{Hasher}} would be.
I selected guava's MD5 and murmur3_128 hashers for comparison. Here's what I
found:
{noformat}
[java] Benchmark(bufferSize) Mode Cnt
Score Error Units
[java] HashingBench.benchHasherMD531 avgt5
336.613 ± 18.826 ns/op
[java] HashingBench.benchHasherMD5 131 avgt5
709.226 ± 19.489 ns/op
[java] HashingBench.benchHasherMD5 517 avgt5
1800.091 ± 37.748 ns/op
[java] HashingBench.benchHasherMD5 2041 avgt5
6275.607 ± 623.008 ns/op
[java] HashingBench.benchHasherMurmur3_12831 avgt5
260.859 ± 39.229 ns/op
[java] HashingBench.benchHasherMurmur3_128 131 avgt5
421.268 ± 68.287 ns/op
[java] HashingBench.benchHasherMurmur3_128 517 avgt5
861.577 ± 68.423 ns/op
[java] HashingBench.benchHasherMurmur3_128 2041 avgt5
2863.952 ± 314.205 ns/op
[java] HashingBench.benchMessageDigestMD5 31 avgt5
787.373 ± 69.869 ns/op
[java] HashingBench.benchMessageDigestMD5131 avgt5
1264.677 ± 117.790 ns/op
[java] HashingBench.benchMessageDigestMD5517 avgt5
2822.846 ± 178.416 ns/op
[java] HashingBench.benchMessageDigestMD5 2041 avgt5
9611.875 ± 1760.809 ns/op
{noformat}
Naively, I used byte arrays for four varying sizes, updated the hasher/digest,
and got the hashed result. I selected buffer sizes that are close to
powers-of-2, but intentionally not. It looks like the guava {{Hasher}}s do
indeed perform better than {{MessageDigest}}.
> Replace usages of MessageDigest with Guava's Hasher
> ---
>
> Key: CASSANDRA-13291
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13291
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
>Reporter: Michael Kjellman
>Assignee: Michael Kjellman
> Attachments: CASSANDRA-13291-trunk.diff
>
>
> During my profiling of C* I frequently see lots of aggregate time across
> threads being spent inside the MD5 MessageDigest implementation. Given that
> there are tons of modern alternative hashing functions better than MD5
> available -- both in terms of providing better collision resistance and
> actual computational speed -- I wanted to switch out our usage of MD5 for
> alternatives (like adler128 or murmur3_128) and test for performance
> improvements.
> Unfortunately, I found given the fact we use MessageDigest everywhere --
> switching out the hashing function to something like adler128 or murmur3_128
> (for example) -- which don't ship with the JDK -- wasn't straight forward.
> The goal of this ticket is to propose switching out usages of MessageDigest
> directly in favor of Hasher from Guava. This means going forward we can
> change a single line of code to switch the hashing algorithm being used
> (assuming there is an implementation in Guava).
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13291) Replace usages of MessageDigest with Guava's Hasher
[ https://issues.apache.org/jira/browse/CASSANDRA-13291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16183191#comment-16183191 ] Jason Brown commented on CASSANDRA-13291: - fwiw, I've kicked off utests and dtests here: ||13291|| |[branch|https://github.com/jasobrown/cassandra/tree/13291]| |[dtest|https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-dtest/340/]| |[utests|https://circleci.com/gh/jasobrown/cassandra/tree/13291]| Note: I just took [~mkjellman]'s patch and applied to a fresh branch of trunk in my repo. Review forthcoming ... > Replace usages of MessageDigest with Guava's Hasher > --- > > Key: CASSANDRA-13291 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13291 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Michael Kjellman >Assignee: Michael Kjellman > Attachments: CASSANDRA-13291-trunk.diff > > > During my profiling of C* I frequently see lots of aggregate time across > threads being spent inside the MD5 MessageDigest implementation. Given that > there are tons of modern alternative hashing functions better than MD5 > available -- both in terms of providing better collision resistance and > actual computational speed -- I wanted to switch out our usage of MD5 for > alternatives (like adler128 or murmur3_128) and test for performance > improvements. > Unfortunately, I found given the fact we use MessageDigest everywhere -- > switching out the hashing function to something like adler128 or murmur3_128 > (for example) -- which don't ship with the JDK -- wasn't straight forward. > The goal of this ticket is to propose switching out usages of MessageDigest > directly in favor of Hasher from Guava. This means going forward we can > change a single line of code to switch the hashing algorithm being used > (assuming there is an implementation in Guava). -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13789) Reduce memory copies and object creations when acting on ByteBufs
[ https://issues.apache.org/jira/browse/CASSANDRA-13789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16182610#comment-16182610 ] Jason Brown commented on CASSANDRA-13789: - [~Stefania] you are totally right. In fact, I even had to deal with debugging that same problem once when working on CASSANDRA-12229, and it was nasty to debug (not sure how i missed that in review :-/ ). Let's give [~norman] a day or two to respond with any comments, else, I'll go ahead and revert that part. > Reduce memory copies and object creations when acting on ByteBufs > -- > > Key: CASSANDRA-13789 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13789 > Project: Cassandra > Issue Type: Improvement >Reporter: Norman Maurer >Assignee: Norman Maurer > Fix For: 4.0 > > Attachments: > 0001-CBUtil.sizeOfLongString-encodes-String-to-byte-to-ca.patch, > 0001-Reduce-memory-copies-and-object-creations-when-actin.patch > > > There are multiple "low-hanging-fruits" when it comes to reduce memory copies > and object allocations when acting on ByteBufs. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13899) Streaming of very large partition fails
[
https://issues.apache.org/jira/browse/CASSANDRA-13899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16181837#comment-16181837
]
Jason Brown commented on CASSANDRA-13899:
-
tbh, I don't think it's large partitions that necessarily triggered this bug;
streaming just transfers the bytes of sstable Data file naively. I think it was
something in the dataset and the way it compresses that exposed the bug. Thus,
I'm not sure how to proceed with a dtest, especially one based on
[~pauloricardomg]'s submitted patch (as highly useful as it is!).
> Streaming of very large partition fails
>
>
> Key: CASSANDRA-13899
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13899
> Project: Cassandra
> Issue Type: Bug
>Reporter: Paulo Motta
>Assignee: Jason Brown
> Fix For: 4.0
>
> Attachments: largepartition.yaml
>
>
> Streaming a single partition with ~100K rows fails with the following
> exception:
> {noformat}
> ERROR [Stream-Deserializer-/127.0.0.1:35149-a92e5e12] 2017-09-21 04:03:41,237
> StreamSession.java:617 - [Stream #c2e5b640-9eab-11e7-99c0-e9864ca8da8e]
> Streaming error occurred on session with peer 127.0.0.1
> org.apache.cassandra.streaming.StreamReceiveException:
> java.lang.RuntimeException: Last written key
> DecoratedKey(-1000328290821038380) >= current key
> DecoratedKey(-1055007227842125139) writing into
> /home/paulo/.ccm/test/node2/data0/stresscql/typestest-482ac7b09e8d11e787cf85d073c
> 8e037/na-1-big-Data.db
> at
> org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:63)
> ~[main/:na]
> at
> org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:41)
> ~[main/:na]
> at
> org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:55)
> ~[main/:na]
> at
> org.apache.cassandra.streaming.async.StreamingInboundHandler$StreamDeserializingTask.run(StreamingInboundHandler.java:178)
> ~[main/:na]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_121]
> {noformat}
> Reproduction steps:
> * Create CCM cluster with 2 nodes
> * Start only first node, disable hinted handoff
> * Run stress with the attached yaml: {{tools/bin/cassandra-stress "user
> profile=largepartition.yaml n=10K ops(insert=1) no-warmup -node whitelist
> 127.0.0.1 -mode native cql3 compression=lz4 -rate threads=4 -insert
> visits=FIXED(100K) revisit=FIXED(100K)"}}
> * Start second node, run repair on {{stresscql}} table - the exception above
> will be thrown.
> I investigated briefly and haven't found anything suspicious. This seems to
> be related to CASSANDRA-12229 as I tested the steps above in a branch without
> that and the repair completed successfully. I haven't tested with a smaller
> number of rows per partition to see at which point it starts to be a problem.
> We should probably add a regression dtest to stream large partitions to catch
> similar problems in the future.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-13905) Correctly close netty channels when a stream session ends
[
https://issues.apache.org/jira/browse/CASSANDRA-13905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Brown updated CASSANDRA-13905:
Status: Patch Available (was: Open)
> Correctly close netty channels when a stream session ends
> -
>
> Key: CASSANDRA-13905
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13905
> Project: Cassandra
> Issue Type: Bug
> Components: Streaming and Messaging
>Reporter: Jason Brown
>Assignee: Jason Brown
> Fix For: 4.x
>
>
> Netty channels in stream sessions were not being closed correctly. TL;DR I
> was using a lambda that was not executing as it is lazily evaluated. This was
> causing a {{RejectedExecutionException}} at the end of some streaming-related
> dtests
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-13899) Streaming of very large partition fails
[
https://issues.apache.org/jira/browse/CASSANDRA-13899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Brown updated CASSANDRA-13899:
Status: Patch Available (was: Open)
> Streaming of very large partition fails
>
>
> Key: CASSANDRA-13899
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13899
> Project: Cassandra
> Issue Type: Bug
>Reporter: Paulo Motta
>Assignee: Jason Brown
> Fix For: 4.0
>
> Attachments: largepartition.yaml
>
>
> Streaming a single partition with ~100K rows fails with the following
> exception:
> {noformat}
> ERROR [Stream-Deserializer-/127.0.0.1:35149-a92e5e12] 2017-09-21 04:03:41,237
> StreamSession.java:617 - [Stream #c2e5b640-9eab-11e7-99c0-e9864ca8da8e]
> Streaming error occurred on session with peer 127.0.0.1
> org.apache.cassandra.streaming.StreamReceiveException:
> java.lang.RuntimeException: Last written key
> DecoratedKey(-1000328290821038380) >= current key
> DecoratedKey(-1055007227842125139) writing into
> /home/paulo/.ccm/test/node2/data0/stresscql/typestest-482ac7b09e8d11e787cf85d073c
> 8e037/na-1-big-Data.db
> at
> org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:63)
> ~[main/:na]
> at
> org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:41)
> ~[main/:na]
> at
> org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:55)
> ~[main/:na]
> at
> org.apache.cassandra.streaming.async.StreamingInboundHandler$StreamDeserializingTask.run(StreamingInboundHandler.java:178)
> ~[main/:na]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_121]
> {noformat}
> Reproduction steps:
> * Create CCM cluster with 2 nodes
> * Start only first node, disable hinted handoff
> * Run stress with the attached yaml: {{tools/bin/cassandra-stress "user
> profile=largepartition.yaml n=10K ops(insert=1) no-warmup -node whitelist
> 127.0.0.1 -mode native cql3 compression=lz4 -rate threads=4 -insert
> visits=FIXED(100K) revisit=FIXED(100K)"}}
> * Start second node, run repair on {{stresscql}} table - the exception above
> will be thrown.
> I investigated briefly and haven't found anything suspicious. This seems to
> be related to CASSANDRA-12229 as I tested the steps above in a branch without
> that and the repair completed successfully. I haven't tested with a smaller
> number of rows per partition to see at which point it starts to be a problem.
> We should probably add a regression dtest to stream large partitions to catch
> similar problems in the future.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-13906) Properly close StreamCompressionInputStream to release any ByteBuf
[
https://issues.apache.org/jira/browse/CASSANDRA-13906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Brown updated CASSANDRA-13906:
Status: Patch Available (was: Open)
> Properly close StreamCompressionInputStream to release any ByteBuf
> --
>
> Key: CASSANDRA-13906
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13906
> Project: Cassandra
> Issue Type: Bug
>Reporter: Jason Brown
>Assignee: Jason Brown
>
> When running dtests for trunk (4.x) that perform some streaming, sometimes a
> {{ByteBuf}} is not released properly, and we get this error in the logs
> (causing the dtest to fail):
> {code}
> ERROR [MessagingService-NettyOutbound-Thread-4-2] 2017-09-26 13:42:37,940
> Slf4JLogger.java:176 - LEAK: ByteBuf.release() was not called before it's
> garbage-collected. Enable advanced leak reporting to find out where the leak
> occurred. To enable advanced leak reporting, specify the JVM option
> '-Dio.netty.leakDetection.level=advanced' or call
> ResourceLeakDetector.setLevel() See
> http://netty.io/wiki/reference-counted-objects.html for more information.
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13906) Properly close StreamCompressionInputStream to release any ByteBuf
[
https://issues.apache.org/jira/browse/CASSANDRA-13906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16181752#comment-16181752
]
Jason Brown commented on CASSANDRA-13906:
-
The cause of the leaked {{ByteBuf}} was due to a wrapping class not being
closed, and thus never had the chance to call {{release()}} on the {{ByteBuf}}.
||trunk||
|[branch|https://github.com/jasobrown/cassandra/tree/netty-leak]|
|[dtest|https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-dtest/338/]|
|[utests|https://circleci.com/gh/jasobrown/cassandra/tree/netty-leak]|
I ran one of the failing dtests
(repair_tests/incremental_repair_test.py:TestIncRepair.multiple_repair_test) on
my laptop, and without the patch could reproduce within 2-3 runs. With the
patch, I ran it fifty time and could not reproduce.
[~aweisberg] Would you mind reviewing?
> Properly close StreamCompressionInputStream to release any ByteBuf
> --
>
> Key: CASSANDRA-13906
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13906
> Project: Cassandra
> Issue Type: Bug
>Reporter: Jason Brown
>Assignee: Jason Brown
>
> When running dtests for trunk (4.x) that perform some streaming, sometimes a
> {{ByteBuf}} is not released properly, and we get this error in the logs
> (causing the dtest to fail):
> {code}
> ERROR [MessagingService-NettyOutbound-Thread-4-2] 2017-09-26 13:42:37,940
> Slf4JLogger.java:176 - LEAK: ByteBuf.release() was not called before it's
> garbage-collected. Enable advanced leak reporting to find out where the leak
> occurred. To enable advanced leak reporting, specify the JVM option
> '-Dio.netty.leakDetection.level=advanced' or call
> ResourceLeakDetector.setLevel() See
> http://netty.io/wiki/reference-counted-objects.html for more information.
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-13906) Properly close StreamCompressionInputStream to release any ByteBuf
Jason Brown created CASSANDRA-13906:
---
Summary: Properly close StreamCompressionInputStream to release
any ByteBuf
Key: CASSANDRA-13906
URL: https://issues.apache.org/jira/browse/CASSANDRA-13906
Project: Cassandra
Issue Type: Bug
Reporter: Jason Brown
Assignee: Jason Brown
When running dtests for trunk (4.x) that perform some streaming, sometimes a
{{ByteBuf}} is not released properly, and we get this error in the logs
(causing the dtest to fail):
{code}
ERROR [MessagingService-NettyOutbound-Thread-4-2] 2017-09-26 13:42:37,940
Slf4JLogger.java:176 - LEAK: ByteBuf.release() was not called before it's
garbage-collected. Enable advanced leak reporting to find out where the leak
occurred. To enable advanced leak reporting, specify the JVM option
'-Dio.netty.leakDetection.level=advanced' or call
ResourceLeakDetector.setLevel() See
http://netty.io/wiki/reference-counted-objects.html for more information.
{code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13899) Streaming of very large partition fails
[
https://issues.apache.org/jira/browse/CASSANDRA-13899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16181621#comment-16181621
]
Jason Brown commented on CASSANDRA-13899:
-
This turned out to be a simple, stupid bug - i used the wrong length to compare
against when translating from CASSANDRA-10520.
Patch here:
||13899||
|[branch|https://github.com/jasobrown/cassandra/tree/13899]|
|[dtest|https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-dtest/337/]|
|[utests|https://circleci.com/gh/jasobrown/cassandra/tree/13899]|
[~pauloricardomg] Do you mind reviewing?
> Streaming of very large partition fails
>
>
> Key: CASSANDRA-13899
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13899
> Project: Cassandra
> Issue Type: Bug
>Reporter: Paulo Motta
>Assignee: Jason Brown
> Fix For: 4.0
>
> Attachments: largepartition.yaml
>
>
> Streaming a single partition with ~100K rows fails with the following
> exception:
> {noformat}
> ERROR [Stream-Deserializer-/127.0.0.1:35149-a92e5e12] 2017-09-21 04:03:41,237
> StreamSession.java:617 - [Stream #c2e5b640-9eab-11e7-99c0-e9864ca8da8e]
> Streaming error occurred on session with peer 127.0.0.1
> org.apache.cassandra.streaming.StreamReceiveException:
> java.lang.RuntimeException: Last written key
> DecoratedKey(-1000328290821038380) >= current key
> DecoratedKey(-1055007227842125139) writing into
> /home/paulo/.ccm/test/node2/data0/stresscql/typestest-482ac7b09e8d11e787cf85d073c
> 8e037/na-1-big-Data.db
> at
> org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:63)
> ~[main/:na]
> at
> org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:41)
> ~[main/:na]
> at
> org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:55)
> ~[main/:na]
> at
> org.apache.cassandra.streaming.async.StreamingInboundHandler$StreamDeserializingTask.run(StreamingInboundHandler.java:178)
> ~[main/:na]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_121]
> {noformat}
> Reproduction steps:
> * Create CCM cluster with 2 nodes
> * Start only first node, disable hinted handoff
> * Run stress with the attached yaml: {{tools/bin/cassandra-stress "user
> profile=largepartition.yaml n=10K ops(insert=1) no-warmup -node whitelist
> 127.0.0.1 -mode native cql3 compression=lz4 -rate threads=4 -insert
> visits=FIXED(100K) revisit=FIXED(100K)"}}
> * Start second node, run repair on {{stresscql}} table - the exception above
> will be thrown.
> I investigated briefly and haven't found anything suspicious. This seems to
> be related to CASSANDRA-12229 as I tested the steps above in a branch without
> that and the repair completed successfully. I haven't tested with a smaller
> number of rows per partition to see at which point it starts to be a problem.
> We should probably add a regression dtest to stream large partitions to catch
> similar problems in the future.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13905) Correctly close netty channels when a stream session ends
[
https://issues.apache.org/jira/browse/CASSANDRA-13905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16181406#comment-16181406
]
Jason Brown commented on CASSANDRA-13905:
-
Minor patch here:
||trunk||
|[branch|https://github.com/jasobrown/cassandra/tree/netty-RejectedExecutionException]|
|[dtest|https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-dtest/336/]|
|[utests|https://circleci.com/gh/jasobrown/cassandra/tree/netty-RejectedExecutionException]|
utest errors are unrelated, and number of dtest failures are lower (and none
caused by {{RejectedExecutionEception}})
[~aweisberg] or [~pauloricardomg] : Would one of you mind reviewing?
> Correctly close netty channels when a stream session ends
> -
>
> Key: CASSANDRA-13905
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13905
> Project: Cassandra
> Issue Type: Bug
> Components: Streaming and Messaging
>Reporter: Jason Brown
>Assignee: Jason Brown
> Fix For: 4.x
>
>
> Netty channels in stream sessions were not being closed correctly. TL;DR I
> was using a lambda that was not executing as it is lazily evaluated. This was
> causing a {{RejectedExecutionException}} at the end of some streaming-related
> dtests
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-13905) Correctly close netty channels when a stream session ends
Jason Brown created CASSANDRA-13905:
---
Summary: Correctly close netty channels when a stream session ends
Key: CASSANDRA-13905
URL: https://issues.apache.org/jira/browse/CASSANDRA-13905
Project: Cassandra
Issue Type: Bug
Components: Streaming and Messaging
Reporter: Jason Brown
Assignee: Jason Brown
Fix For: 4.x
Netty channels in stream sessions were not being closed correctly. TL;DR I was
using a lambda that was not executing as it is lazily evaluated. This was
causing a {{RejectedExecutionException}} at the end of some streaming-related
dtests
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13649) Uncaught exceptions in Netty pipeline
[
https://issues.apache.org/jira/browse/CASSANDRA-13649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16180599#comment-16180599
]
Jason Brown commented on CASSANDRA-13649:
-
[~ajithshetty28] Please look at the "Fix Version" field above: this was patched
in 3.0.15
> Uncaught exceptions in Netty pipeline
> -
>
> Key: CASSANDRA-13649
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13649
> Project: Cassandra
> Issue Type: Bug
> Components: Streaming and Messaging, Testing
>Reporter: Stefan Podkowinski
>Assignee: Norman Maurer
> Labels: patch
> Fix For: 2.2.11, 3.0.15, 3.11.1, 4.0
>
> Attachments:
> 0001-CASSANDRA-13649-Ensure-all-exceptions-are-correctly-.patch,
> test_stdout.txt
>
>
> I've noticed some netty related errors in trunk in [some of the dtest
> results|https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-dtest/106/#showFailuresLink].
> Just want to make sure that we don't have to change anything related to the
> exception handling in our pipeline and that this isn't a netty issue.
> Actually if this causes flakiness but is otherwise harmless, we should do
> something about it, even if it's just on the dtest side.
> {noformat}
> WARN [epollEventLoopGroup-2-9] 2017-06-28 17:23:49,699 Slf4JLogger.java:151
> - An exceptionCaught() event was fired, and it reached at the tail of the
> pipeline. It usually means the last handler in the pipeline did not handle
> the exception.
> io.netty.channel.unix.Errors$NativeIoException: syscall:read(...)() failed:
> Connection reset by peer
> at io.netty.channel.unix.FileDescriptor.readAddress(...)(Unknown
> Source) ~[netty-all-4.0.44.Final.jar:4.0.44.Final]
> {noformat}
> And again in another test:
> {noformat}
> WARN [epollEventLoopGroup-2-8] 2017-06-29 02:27:31,300 Slf4JLogger.java:151
> - An exceptionCaught() event was fired, and it reached at the tail of the
> pipeline. It usually means the last handler in the pipeline did not handle
> the exception.
> io.netty.channel.unix.Errors$NativeIoException: syscall:read(...)() failed:
> Connection reset by peer
> at io.netty.channel.unix.FileDescriptor.readAddress(...)(Unknown
> Source) ~[netty-all-4.0.44.Final.jar:4.0.44.Final]
> {noformat}
> Edit:
> The {{io.netty.channel.unix.Errors$NativeIoException: syscall:read(...)()
> failed}} error also causes tests to fail for 3.0 and 3.11.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13899) Streaming of very large partition fails
[
https://issues.apache.org/jira/browse/CASSANDRA-13899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16180041#comment-16180041
]
Jason Brown commented on CASSANDRA-13899:
-
[~pauloricardomg] Thanks for the scripts and instructions - I was able to
reproduce. Will have a patch hopefully in a day or two
> Streaming of very large partition fails
>
>
> Key: CASSANDRA-13899
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13899
> Project: Cassandra
> Issue Type: Bug
>Reporter: Paulo Motta
>Assignee: Jason Brown
> Fix For: 4.0
>
> Attachments: largepartition.yaml
>
>
> Streaming a single partition with ~100K rows fails with the following
> exception:
> {noformat}
> ERROR [Stream-Deserializer-/127.0.0.1:35149-a92e5e12] 2017-09-21 04:03:41,237
> StreamSession.java:617 - [Stream #c2e5b640-9eab-11e7-99c0-e9864ca8da8e]
> Streaming error occurred on session with peer 127.0.0.1
> org.apache.cassandra.streaming.StreamReceiveException:
> java.lang.RuntimeException: Last written key
> DecoratedKey(-1000328290821038380) >= current key
> DecoratedKey(-1055007227842125139) writing into
> /home/paulo/.ccm/test/node2/data0/stresscql/typestest-482ac7b09e8d11e787cf85d073c
> 8e037/na-1-big-Data.db
> at
> org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:63)
> ~[main/:na]
> at
> org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:41)
> ~[main/:na]
> at
> org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:55)
> ~[main/:na]
> at
> org.apache.cassandra.streaming.async.StreamingInboundHandler$StreamDeserializingTask.run(StreamingInboundHandler.java:178)
> ~[main/:na]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_121]
> {noformat}
> Reproduction steps:
> * Create CCM cluster with 2 nodes
> * Start only first node, disable hinted handoff
> * Run stress with the attached yaml: {{tools/bin/cassandra-stress "user
> profile=largepartition.yaml n=10K ops(insert=1) no-warmup -node whitelist
> 127.0.0.1 -mode native cql3 compression=lz4 -rate threads=4 -insert
> visits=FIXED(100K) revisit=FIXED(100K)"}}
> * Start second node, run repair on {{stresscql}} table - the exception above
> will be thrown.
> I investigated briefly and haven't found anything suspicious. This seems to
> be related to CASSANDRA-12229 as I tested the steps above in a branch without
> that and the repair completed successfully. I haven't tested with a smaller
> number of rows per partition to see at which point it starts to be a problem.
> We should probably add a regression dtest to stream large partitions to catch
> similar problems in the future.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13149) AssertionError prepending to a list
[
https://issues.apache.org/jira/browse/CASSANDRA-13149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16179416#comment-16179416
]
Jason Brown commented on CASSANDRA-13149:
-
[~beobal] yeah, I like your change - correct, and simpler! I've added that to
all three branches, and they are queued up for testing on circleci.
> AssertionError prepending to a list
> ---
>
> Key: CASSANDRA-13149
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13149
> Project: Cassandra
> Issue Type: Bug
> Components: CQL
> Environment: 3.0.8
>Reporter: Steven Warren
>Assignee: Jason Brown
>
> Prepending to a list produces the following AssertionError randomly. Changing
> the update to append (and sort in the client) works around the issue.
> {code}
> java.lang.AssertionError: null
> at
> org.apache.cassandra.cql3.Lists$PrecisionTime.getNext(Lists.java:275)
> ~[apache-cassandra-3.0.8.jar:3.0.8]
> at org.apache.cassandra.cql3.Lists$Prepender.execute(Lists.java:430)
> ~[apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.cql3.statements.UpdateStatement.addUpdateForKey(UpdateStatement.java:94)
> ~[apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.cql3.statements.ModificationStatement.addUpdates(ModificationStatement.java:682)
> ~[apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.cql3.statements.ModificationStatement.getMutations(ModificationStatement.java:613)
> ~[apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.cql3.statements.ModificationStatement.executeWithoutCondition(ModificationStatement.java:420)
> ~[apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.cql3.statements.ModificationStatement.execute(ModificationStatement.java:408)
> ~[apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:206)
> ~[apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:487)
> ~[apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:464)
> ~[apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.transport.messages.ExecuteMessage.execute(ExecuteMessage.java:130)
> ~[apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:507)
> [apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:401)
> [apache-cassandra-3.0.8.jar:3.0.8]
> at
> io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
> [netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
> [netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.access$700(AbstractChannelHandlerContext.java:32)
> [netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext$8.run(AbstractChannelHandlerContext.java:324)
> [netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [na:1.8.0_101]
> at
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164)
> [apache-cassandra-3.0.8.jar:3.0.8]
> at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105)
> [apache-cassandra-3.0.8.jar:3.0.8]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_101]
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Assigned] (CASSANDRA-13899) Streaming of very large partition fails
[
https://issues.apache.org/jira/browse/CASSANDRA-13899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Brown reassigned CASSANDRA-13899:
---
Assignee: Jason Brown
> Streaming of very large partition fails
>
>
> Key: CASSANDRA-13899
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13899
> Project: Cassandra
> Issue Type: Bug
>Reporter: Paulo Motta
>Assignee: Jason Brown
> Fix For: 4.0
>
> Attachments: largepartition.yaml
>
>
> Streaming a single partition with ~100K rows fails with the following
> exception:
> {noformat}
> ERROR [Stream-Deserializer-/127.0.0.1:35149-a92e5e12] 2017-09-21 04:03:41,237
> StreamSession.java:617 - [Stream #c2e5b640-9eab-11e7-99c0-e9864ca8da8e]
> Streaming error occurred on session with peer 127.0.0.1
> org.apache.cassandra.streaming.StreamReceiveException:
> java.lang.RuntimeException: Last written key
> DecoratedKey(-1000328290821038380) >= current key
> DecoratedKey(-1055007227842125139) writing into
> /home/paulo/.ccm/test/node2/data0/stresscql/typestest-482ac7b09e8d11e787cf85d073c
> 8e037/na-1-big-Data.db
> at
> org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:63)
> ~[main/:na]
> at
> org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:41)
> ~[main/:na]
> at
> org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:55)
> ~[main/:na]
> at
> org.apache.cassandra.streaming.async.StreamingInboundHandler$StreamDeserializingTask.run(StreamingInboundHandler.java:178)
> ~[main/:na]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_121]
> {noformat}
> Reproduction steps:
> * Create CCM cluster with 2 nodes
> * Start only first node, disable hinted handoff
> * Run stress with the attached yaml: {{tools/bin/cassandra-stress "user
> profile=largepartition.yaml n=10K ops(insert=1) no-warmup -node whitelist
> 127.0.0.1 -mode native cql3 compression=lz4 -rate threads=4 -insert
> visits=FIXED(100K) revisit=FIXED(100K)"}}
> * Start second node, run repair on {{stresscql}} table - the exception above
> will be thrown.
> I investigated briefly and haven't found anything suspicious. This seems to
> be related to CASSANDRA-12229 as I tested the steps above in a branch without
> that and the repair completed successfully. I haven't tested with a smaller
> number of rows per partition to see at which point it starts to be a problem.
> We should probably add a regression dtest to stream large partitions to catch
> similar problems in the future.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13149) AssertionError prepending to a list
[
https://issues.apache.org/jira/browse/CASSANDRA-13149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16173977#comment-16173977
]
Jason Brown commented on CASSANDRA-13149:
-
Pushed a few updates to the branches.
bq. handle the case where the list of elements to prepend > {{MAX_NANOS}}
Done, even though a client would be rather bananas to prepare 10,000 elements
to the beginning of list! Also, added test for that.
bq. nothing currently stops us overflowing the range of the PT instance except
the construction of the for loop
I tried turning the "range" represented by the instance returned by
{{PrecisionTime#getNext()}} into something like an interator, as wel basically
want to iterate/walk the value represented by the range. It quickly started to
feel like it was over-engineered. While there is this "tied-at-the-hip"
relationship between {{Prepender}} and {{PrecisionTime}}, I'm not sure that
additional guards won't feel similiarly out of place. wdyt?
Tests and other nits have been addressed,as well.
> AssertionError prepending to a list
> ---
>
> Key: CASSANDRA-13149
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13149
> Project: Cassandra
> Issue Type: Bug
> Components: CQL
> Environment: 3.0.8
>Reporter: Steven Warren
>Assignee: Jason Brown
>
> Prepending to a list produces the following AssertionError randomly. Changing
> the update to append (and sort in the client) works around the issue.
> {code}
> java.lang.AssertionError: null
> at
> org.apache.cassandra.cql3.Lists$PrecisionTime.getNext(Lists.java:275)
> ~[apache-cassandra-3.0.8.jar:3.0.8]
> at org.apache.cassandra.cql3.Lists$Prepender.execute(Lists.java:430)
> ~[apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.cql3.statements.UpdateStatement.addUpdateForKey(UpdateStatement.java:94)
> ~[apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.cql3.statements.ModificationStatement.addUpdates(ModificationStatement.java:682)
> ~[apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.cql3.statements.ModificationStatement.getMutations(ModificationStatement.java:613)
> ~[apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.cql3.statements.ModificationStatement.executeWithoutCondition(ModificationStatement.java:420)
> ~[apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.cql3.statements.ModificationStatement.execute(ModificationStatement.java:408)
> ~[apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:206)
> ~[apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:487)
> ~[apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:464)
> ~[apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.transport.messages.ExecuteMessage.execute(ExecuteMessage.java:130)
> ~[apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:507)
> [apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:401)
> [apache-cassandra-3.0.8.jar:3.0.8]
> at
> io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
> [netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
> [netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.access$700(AbstractChannelHandlerContext.java:32)
> [netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext$8.run(AbstractChannelHandlerContext.java:324)
> [netty-all-4.0.23.Final.jar:4.0.23.Final]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [na:1.8.0_101]
> at
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164)
> [apache-cassandra-3.0.8.jar:3.0.8]
> at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105)
> [apache-cassandra-3.0.8.jar:3.0.8]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_101]
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13404) Hostname verification for client-to-node encryption
[ https://issues.apache.org/jira/browse/CASSANDRA-13404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16173535#comment-16173535 ] Jason Brown commented on CASSANDRA-13404: - [~jjirsa] I am fine with that idea. > Hostname verification for client-to-node encryption > --- > > Key: CASSANDRA-13404 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13404 > Project: Cassandra > Issue Type: New Feature >Reporter: Jan Karlsson >Assignee: Jan Karlsson > Fix For: 4.x > > Attachments: 13404-trunk.txt > > > Similarily to CASSANDRA-9220, Cassandra should support hostname verification > for client-node connections. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-13844) sstableloader doesn't support non default storage_port and ssl_storage_port
[
https://issues.apache.org/jira/browse/CASSANDRA-13844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Brown updated CASSANDRA-13844:
Resolution: Fixed
Reproduced In: 2.2.10, 2.1.18 (was: 2.1.18, 2.2.10)
Status: Resolved (was: Patch Available)
I took the liberty of committing to unblock the next round of releases. commit
sha is {{428eaa3e37cab7227c81fdf124d29dfc1db4257c}}
Thanks!
> sstableloader doesn't support non default storage_port and ssl_storage_port
> ---
>
> Key: CASSANDRA-13844
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13844
> Project: Cassandra
> Issue Type: Improvement
> Components: Tools
>Reporter: Eduard Tudenhoefner
>Assignee: Eduard Tudenhoefner
> Fix For: 2.1.19, 2.2.11
>
>
> Currently *storage_port* and *ssl_storage_port* are hardcoded to the
> defaults. The problem was already fixed in CASSANDRA-13518 for C* 3.0+, so
> this here is just backporting it to C* 2.1/2.2.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13404) Hostname verification for client-to-node encryption
[
https://issues.apache.org/jira/browse/CASSANDRA-13404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16172394#comment-16172394
]
Jason Brown commented on CASSANDRA-13404:
-
In client-side endpoint verification, the client knows the server's hostname,
retrieves the server's cert, and compares the hostname it already has with
what's in the cert. In other words, it uses a piece of information it already
had as well as a piece of information from the server.
In the case of server-side enpoint verification, what you are proposing is to
use the client IP address from the incoming socket as the hostname and then a
cert that will also be provided by the client. Thus, both pieces of information
that the server uses to verify are coming from the client. The certificate is
cryptographically signed, but the IP address is not. This means that the IP
address could potentially be spoofed and will not satisfy the original
requirement of "preventing certificates from being copied and used elsewhere".
bq. Hostname verification on the server side would improve security in that no
one is able to copy a certificate from a client node to another location, and
then use it to manipulate data in Cassandra.
By the time an attacker can copy a cert, can't they also spoof an IP address,
as well?
I'll be honest, I'm unconfortable with the patch - taking the incoming IP
address and passing that directly into the {{SslHandler}} just seems wrong.
More importantly, I don't think this solution really solves your problem of
ensuring that only legit clients can connect as IP addresses can be spoofed.
I'm -1 unless [~spo...@gmail.com] or anybody else has a convincing argument.
/cc [~smitchelus] of the core netty team, as he and i discussed this issue at
length.
> Hostname verification for client-to-node encryption
> ---
>
> Key: CASSANDRA-13404
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13404
> Project: Cassandra
> Issue Type: New Feature
>Reporter: Jan Karlsson
>Assignee: Jan Karlsson
> Fix For: 4.x
>
> Attachments: 13404-trunk.txt
>
>
> Similarily to CASSANDRA-9220, Cassandra should support hostname verification
> for client-node connections.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-13631) add header parameter size to internode messaging protocol
[
https://issues.apache.org/jira/browse/CASSANDRA-13631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16171906#comment-16171906
]
Jason Brown edited comment on CASSANDRA-13631 at 9/19/17 3:32 PM:
--
Here is a patch that repurposes the 4-byte parameters count to the total count
of bytes in the parameters.
||13631||
|[branch|https://github.com/jasobrown/cassandra/tree/13631]|
|[dtest|https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-dtest/330/]|
|[utests|https://circleci.com/gh/jasobrown/cassandra/tree/13631]|
was (Author: jasobrown):
||13631||
|[branch|https://github.com/jasobrown/cassandra/tree/13631]|
|[dtest|https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-dtest/330/]|
|[utests|https://circleci.com/gh/jasobrown/cassandra/tree/13631]|
> add header parameter size to internode messaging protocol
> -
>
> Key: CASSANDRA-13631
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13631
> Project: Cassandra
> Issue Type: Task
> Components: Streaming and Messaging
>Reporter: Jason Brown
>Assignee: Jason Brown
>Priority: Minor
> Fix For: 4.0
>
>
> as netty is not a streaming/blocking protocol and works best when buffer
> sizes are known, I had to do a bunch of contortions in {{MessageInHandler}}
> (as part of CASSANDRA-8457) to safely read the message header parameters. If
> we add a header parameters size field to the internode messaging protocol,
> the header parsing code would be dramatically simpler (note: we'll still need
> the existing parsing to support the cluster upgrade use case). An alternative
> to adding a new field is to hijack the existing header parameter count field;
> that field is an {{int}}, so field width is not an issue.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-13631) add header parameter size to internode messaging protocol
[
https://issues.apache.org/jira/browse/CASSANDRA-13631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Brown updated CASSANDRA-13631:
Fix Version/s: 4.0
Status: Patch Available (was: Open)
||13631||
|[branch|https://github.com/jasobrown/cassandra/tree/13631]|
|[dtest|https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-dtest/330/]|
|[utests|https://circleci.com/gh/jasobrown/cassandra/tree/13631]|
> add header parameter size to internode messaging protocol
> -
>
> Key: CASSANDRA-13631
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13631
> Project: Cassandra
> Issue Type: Task
> Components: Streaming and Messaging
>Reporter: Jason Brown
>Assignee: Jason Brown
>Priority: Minor
> Fix For: 4.0
>
>
> as netty is not a streaming/blocking protocol and works best when buffer
> sizes are known, I had to do a bunch of contortions in {{MessageInHandler}}
> (as part of CASSANDRA-8457) to safely read the message header parameters. If
> we add a header parameters size field to the internode messaging protocol,
> the header parsing code would be dramatically simpler (note: we'll still need
> the existing parsing to support the cluster upgrade use case). An alternative
> to adding a new field is to hijack the existing header parameter count field;
> that field is an {{int}}, so field width is not an issue.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-13669) Error when starting cassandra: Unable to make UUID from 'aa' (SASI index)
[
https://issues.apache.org/jira/browse/CASSANDRA-13669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Brown updated CASSANDRA-13669:
Labels: sasi (was: )
> Error when starting cassandra: Unable to make UUID from 'aa' (SASI index)
> -
>
> Key: CASSANDRA-13669
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13669
> Project: Cassandra
> Issue Type: Bug
> Components: sasi
> Environment: Tested on:
> * macOS Sierra 10.12.5
> * Ubuntu 14.04.5 LTS
>Reporter: Lukasz Biedrycki
>Priority: Critical
> Labels: sasi
> Fix For: 3.11.x
>
>
> Recently I experienced a problem that prevents me to restart cassandra.
> I narrowed it down to SASI Index when added on uuid field.
> Steps to reproduce:
> 1. start cassandra (./bin/cassandra -f)
> 2. create keyspace, table, index and add data:
> {noformat}
> CREATE KEYSPACE testkeyspace
> WITH replication = {'class': 'SimpleStrategy', 'replication_factor': '1'}
>AND durable_writes = true;
> use testkeyspace ;
> CREATE TABLE testtable (
>col1 uuid,
>col2 uuid,
>ts timeuuid,
>col3 uuid,
>PRIMARY KEY((col1, col2), ts) ) with clustering order by (ts desc);
> CREATE CUSTOM INDEX col3_testtable_idx ON testtable(col3)
> USING 'org.apache.cassandra.index.sasi.SASIIndex'
> WITH OPTIONS = {'analyzer_class':
> 'org.apache.cassandra.index.sasi.analyzer.StandardAnalyzer', 'mode':
> 'PREFIX'};
> INSERT INTO testtable(col1, col2, ts, col3)
> VALUES(898e0014-6161-11e7-b9b7-238ea83bd70b,
>898e0014-6161-11e7-b9b7-238ea83bd70b,
>now(), 898e0014-6161-11e7-b9b7-238ea83bd70b);
> {noformat}
> 3. restart cassandra
> It crashes with an error (sorry it's huge):
> {noformat}
> DEBUG 09:09:20 Writing Memtable-testtable@1005362073(0.075KiB serialized
> bytes, 1 ops, 0%/0% of on/off-heap limit), flushed range =
> (min(-9223372036854775808), max(9223372036854775807)]
> ERROR 09:09:20 Exception in thread
> Thread[PerDiskMemtableFlushWriter_0:1,5,main]
> org.apache.cassandra.serializers.MarshalException: Unable to make UUID from
> 'aa'
> at
> org.apache.cassandra.db.marshal.UUIDType.fromString(UUIDType.java:118)
> ~[apache-cassandra-3.9.jar:3.9]
> at
> org.apache.cassandra.index.sasi.analyzer.StandardAnalyzer.hasNext(StandardAnalyzer.java:168)
> ~[apache-cassandra-3.9.jar:3.9]
> at
> org.apache.cassandra.index.sasi.disk.PerSSTableIndexWriter$Index.add(PerSSTableIndexWriter.java:208)
> ~[apache-cassandra-3.9.jar:3.9]
> at
> org.apache.cassandra.index.sasi.disk.PerSSTableIndexWriter.lambda$nextUnfilteredCluster$0(PerSSTableIndexWriter.java:132)
> ~[apache-cassandra-3.9.jar:3.9]
> at java.util.Collections$SingletonSet.forEach(Collections.java:4767)
> ~[na:1.8.0_131]
> at
> org.apache.cassandra.index.sasi.disk.PerSSTableIndexWriter.nextUnfilteredCluster(PerSSTableIndexWriter.java:119)
> ~[apache-cassandra-3.9.jar:3.9]
> at
> org.apache.cassandra.db.ColumnIndex.lambda$add$1(ColumnIndex.java:233)
> ~[apache-cassandra-3.9.jar:3.9]
> at java.lang.Iterable.forEach(Iterable.java:75) ~[na:1.8.0_131]
> at org.apache.cassandra.db.ColumnIndex.add(ColumnIndex.java:233)
> ~[apache-cassandra-3.9.jar:3.9]
> at
> org.apache.cassandra.db.ColumnIndex.buildRowIndex(ColumnIndex.java:107)
> ~[apache-cassandra-3.9.jar:3.9]
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.append(BigTableWriter.java:169)
> ~[apache-cassandra-3.9.jar:3.9]
> at
> org.apache.cassandra.io.sstable.SimpleSSTableMultiWriter.append(SimpleSSTableMultiWriter.java:48)
> ~[apache-cassandra-3.9.jar:3.9]
> at
> org.apache.cassandra.db.Memtable$FlushRunnable.writeSortedContents(Memtable.java:458)
> ~[apache-cassandra-3.9.jar:3.9]
> at
> org.apache.cassandra.db.Memtable$FlushRunnable.call(Memtable.java:493)
> ~[apache-cassandra-3.9.jar:3.9]
> at
> org.apache.cassandra.db.Memtable$FlushRunnable.call(Memtable.java:380)
> ~[apache-cassandra-3.9.jar:3.9]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ~[na:1.8.0_131]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> ~[na:1.8.0_131]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_131]
> at java.lang.Thread.run(Thread.java:748) [na:1.8.0_131]
> Exception (java.lang.RuntimeException) encountered during startup:
> java.util.concurrent.ExecutionException: java.lang.RuntimeException:
> java.util.concurrent.ExecutionException:
> org.apache.cassandra.serializers.MarshalException: Unable to make UUID from
> 'aa'
> java.lang.RuntimeException: java.util.concurrent.ExecutionException:
> java.lang.RuntimeException: java.util.concur
