[
https://issues.apache.org/jira/browse/CASSANDRA-21111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18052874#comment-18052874
]
Stefan Miklosovic commented on CASSANDRA-21111:
-----------------------------------------------
What is quite worrisome is that in {{getNextId}} where we loop we are using
static {{id}} from AbstractSSTableSimpleWriter. Static - so whatever new class
we instantiate, it will have that {{id}} shared across all instances. So
whatever table that writer deals with, it will see that "last id" from
completely unrelated tables.
Table A - writer gets instantiated, id = 1
Table A - getNextId gets called, lists SSTables, there is 5 sstables in
{{existingIds}}, "lastId" is 1, then it concatenates both streams, gets max
which is 5, increases that by 1 to 6, that is set to STATIC {{id}}.
TABLE B - writer gets instantiaed, id = 6.
"Good news" is that this seems to happen in SSTableSimpleWriter and
SSTableSimpleUnsortedWriter.DiskWriter. That is instantiated in
{{CQLSSTableWriter}}'s builder
TABLE B - getNextId gets called, lists SSTables in B, there is 20 sstables in
{{exsistingIds}}, "lastId" is 6, then it concatenates both streams, gets max
which is 20, increases by 1 to 21, that is set to STATIC {{id}}.
Now, when A gets to write SSTables again, it will take STATIC {{id}}, that is
21, then it looks into SSTables it has, which is 1 to 6, concatenates it to two
streams, picks max, 21, and adds one to it, which is 22.
So we just "jumped" from 6 to 22 just like that ...
Now imagine we are doing this accross a lot of tables, it can add up pretty
quickly. For every new SSTable we create like that, its generation id will be
+1 basically ...
The "good news" is that this is used in CQLSSTableWriter's Builder in build()
where SSTableSimpleWriter or SSTableSimpleUnsortedWriter is instantiated (which
both extends SSTableSimpleWriter). This is used in bulk loader or in stress
tests. So if somebody is bulkloading SSTables or doing stress tests I think
they can hit this after some extended period. Other usages of that code path
are only in tests.
> Cassandra sstable generation ID's are a signed int which might overflow
> -----------------------------------------------------------------------
>
> Key: CASSANDRA-21111
> URL: https://issues.apache.org/jira/browse/CASSANDRA-21111
> Project: Apache Cassandra
> Issue Type: Bug
> Components: Local/SSTable
> Reporter: Stefan Miklosovic
> Assignee: Stefan Miklosovic
> Priority: Normal
> Fix For: 4.0.x, 4.1.x, 5.0.x, 5.x
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> We need to triage this but this seems to be legit:
> We have hit an issue with a cluster that where sstable generation ID’s for a
> peer has incremented so high that it has overflowed, which has introduced a
> 2nd dash into the file name. For example:
> nb--2146021117-big-data.db
> This caused an issue with our solution which is aimed to consolidate
> partition statistics in the data set. The current version of the solution
> does not recognise the possibility of sstable generation ID’s being a
> negative value - which causes it to fail.
> Workaround:
> Rename all sstables in the table to “reset” the generation ID.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
