[jira] [Commented] (CASSANDRA-9666) Provide an alternative to DTCS
[
https://issues.apache.org/jira/browse/CASSANDRA-9666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14615060#comment-14615060
]
Jeff Jirsa commented on CASSANDRA-9666:
---
{quote}
We should not provide 2 different compaction strategies for time series data,
instead we should try to fix the issues you list in DTCS
{quote}
I'm willing to consider that premise, however, doing so will require the
addition of a number of tunable variables, turning DTCS into a complex,
unwieldy beast. We both agree that "DTCS has proven quite difficult to use",
and adding complexity to an already complex situation seems like a dangerous
path. It's already very, very difficult to tune, and the difficulty is tied to
the windowing and tiering algorithms - adding MORE knobs to those same
algorithms seems unlikely to make it easier to operate in production.
{quote}
We should do STCS within all windows, CASSANDRA-9644
{quote}
When the window is above max_threshold, potentially. Right now TWCS sorts older
windows by filesize, which has the effect of creating a pseudo-STCS behavior to
make sure to get total number of sstables in the later windows down to 1 as
quickly as possible. Doing a pass with STCS isn't necessarily guaranteed to
knock us below max_threshold (unless, perhaps, we call it with some non-default
thresholds to make it more aggressive). We can look at whether or not STCS is
actually productive, but right now I'm faking STCS in TWCS past windows.
{quote}
This will really hurt read performance if we do this, you will end up hitting
all sstables on disk for every read. The real fix would be to either flush to
several sstables (split by timestamps), or, doing a first special compaction
that writes several sstables into the correct windows.
{quote}
I wasnt about to submit a change to the flushing code, I'm not nearly familiar
enough to consider doing that. I wouldn't mind seeing it, but it's probably
above my head.
However, I expected to write a splitting major for TWCS to split into the
correct window. I will write it in advance if you think it addresses a critical
problem that fundamentally breaks TWCS.
I also believe the same problem also exists in DTCS: any foreground read repair
has the exact same characteristics of throwing off sstable timestamps and
potentially causing reads to hit multiple sstables. In DTCS, it's magnified by
the fact that it ALSO throws off sstable compaction after the fact.
{quote}
unsure what you mean here, could you elaborate?
{quote}
Basically, unlike DTCS, if an old sstable DOES END UP beyond the window we
expect it (max_sstable_age_days), we still compact it. This isn't limited to
repair - it can also happen due to boostrap and decommission, important
operations that real clusters do regularly, even those holding time series
data. Resulting sstables will be compacted regardless of whether or not STCS
would chose it as a candidate, by sorting all files in that window smallest to
largest (again, windows based on max timestamp), and choosing up to
max_threshold. DTCS uses timestamps to sort, which then has a very likely
chance of compacting biggest files together first, leaving small files to live
far longer than they should, and - in many cases - causing the repeated
recompaction of the original large sstables.
{quote}
Could you elaborate here? How do you avoid old data getting into the windows?
{quote}
You don't - we accept that we'll get data into old windows, and make sure we
compact it rather than punting with max_sstable_age_days. Instead, we just make
sure we always use max timestamp for placing files into a window, and
consistent epoch-based windows, to avoid surprises caused by having old data
into the windows. Conversely, DTCS pretends like you can prevent this from
happening, when it's not actually possible in a real world cluster, and then
gets confused wildly confused.
{quote}
Unsure what the issue with DTCS is here (re: streaming operations)?
{quote}
1) Streaming operations beyond max_sstable_age_days are awful in DTCS for
reasons that should be fairly obvious. Not just repair, all streaming
(bootstrap/decom/bulk load).
2) Raising max_sstable_age_days may not actually be viable for most people
because the tiering then joins adjacent windows beyond what they expect/desire
(not only does it cause a ton of IO for new compactions, in many cases it will
run servers out of disk trying to get to the expected state)
{quote}
No, but as mentioned above, this will confuse users in other ways, by making
reads very slow
{quote}
Again, I believe in most real clusters, this is also true with DTCS.
> Provide an alternative to DTCS
> --
>
> Key: CASSANDRA-9666
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9666
> Project: Cassandra
> Issue Type: Improvement
>
[jira] [Commented] (CASSANDRA-9644) DTCS configuration proposals for handling consequences of repairs
[
https://issues.apache.org/jira/browse/CASSANDRA-9644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14615062#comment-14615062
]
Jeff Jirsa commented on CASSANDRA-9644:
---
{quote}
Should also note that using incremental repair makes sure that we don't stream
in a bunch of data in the old windows
{quote}
No such guarantee exists, you'll still stream in a bunch of data into old
windows during bootstrap and decom.
> DTCS configuration proposals for handling consequences of repairs
> -
>
> Key: CASSANDRA-9644
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9644
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
>Reporter: Antti Nissinen
> Fix For: 3.x, 2.1.x
>
> Attachments: node0_20150621_1646_time_graph.txt,
> node0_20150621_2320_time_graph.txt, node0_20150623_1526_time_graph.txt,
> node1_20150621_1646_time_graph.txt, node1_20150621_2320_time_graph.txt,
> node1_20150623_1526_time_graph.txt, node2_20150621_1646_time_graph.txt,
> node2_20150621_2320_time_graph.txt, node2_20150623_1526_time_graph.txt,
> nodetool status infos.txt, sstable_compaction_trace.txt,
> sstable_compaction_trace_snipped.txt, sstable_counts.jpg
>
>
> This is a document bringing up some issues when DTCS is used to compact time
> series data in a three node cluster. The DTCS is currently configured with a
> few parameters that are making the configuration fairly simple, but might
> cause problems in certain special cases like recovering from the flood of
> small SSTables due to repair operation. We are suggesting some ideas that
> might be a starting point for further discussions. Following sections are
> containing:
> - Description of the cassandra setup
> - Feeding process of the data
> - Failure testing
> - Issues caused by the repair operations for the DTCS
> - Proposal for the DTCS configuration parameters
> Attachments are included to support the discussion and there is a separate
> section giving explanation for those.
> Cassandra setup and data model
> - Cluster is composed from three nodes running Cassandra 2.1.2. Replication
> factor is two and read and write consistency levels are ONE.
> - Data is time series data. Data is saved so that one row contains a certain
> time span of data for a given metric ( 20 days in this case). The row key
> contains information about the start time of the time span and metrix name.
> Column name gives the offset from the beginning of time span. Column time
> stamp is set to correspond time stamp when adding together the timestamp from
> the row key and the offset (the actual time stamp of data point). Data model
> is analog to KairosDB implementation.
> - Average sampling rate is 10 seconds varying significantly from metric to
> metric.
> - 100 000 metrics are fed to the Cassandra.
> - max_sstable_age_days is set to 5 days (objective is to keep SStable files
> in manageable size, around 50 GB)
> - TTL is not in use in the test.
> Procedure for the failure test.
> - Data is first dumped to Cassandra for 11 days and the data dumping is
> stopped so that DTCS will have a change to finish all compactions. Data is
> dumped with "fake timestamps" so that column time stamp is set when data is
> written to Cassandra.
> - One of the nodes is taken down and new data is dumped on top of the earlier
> data covering couple of hours worth of data (faked time stamps).
> - Dumping is stopped and the node is kept down for few hours.
> - Node is taken up and the "nodetool repair" is applied on the node that was
> down.
> Consequences
> - Repair operation will lead to massive amount of new SStables far back in
> the history. New SStables are covering similar time spans than the files that
> were created by DTCS before the shutdown of one of the nodes.
> - To be able to compact the small files the max_sstable_age_days should be
> increased to allow compaction to handle the files. However, the in a
> practical case the time window will increase so large that generated files
> will be huge that is not desirable. The compaction also combines together one
> very large file with a bunch of small files in several phases that is not
> effective. Generating really large files may also lead to out of disc space
> problems.
> - See the list of time graphs later in the document.
> Improvement proposals for the DTCS configuration
> Below is a list of desired properties for the configuration. Current
> parameters are mentioned if available.
> - Initial window size (currently:base_time_seconds)
> - The amount of similar size windows for the bucketing (currently:
> min_threshold)
> - The multiplier for the window size when increased (currently:
> min_threshold). This we would like to be independent from the min_threshold
> p
[jira] [Comment Edited] (CASSANDRA-9644) DTCS configuration proposals for handling consequences of repairs
[
https://issues.apache.org/jira/browse/CASSANDRA-9644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14615062#comment-14615062
]
Jeff Jirsa edited comment on CASSANDRA-9644 at 7/6/15 2:10 PM:
---
{quote}
Should also note that using incremental repair makes sure that we don't stream
in a bunch of data in the old windows
{quote}
While that will handle the repair case, the same problem exists in other
streaming operations - you'll still stream in a bunch of data into old windows
during bootstrap and decom.
was (Author: jjirsa):
{quote}
Should also note that using incremental repair makes sure that we don't stream
in a bunch of data in the old windows
{quote}
No such guarantee exists, you'll still stream in a bunch of data into old
windows during bootstrap and decom.
> DTCS configuration proposals for handling consequences of repairs
> -
>
> Key: CASSANDRA-9644
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9644
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
>Reporter: Antti Nissinen
> Fix For: 3.x, 2.1.x
>
> Attachments: node0_20150621_1646_time_graph.txt,
> node0_20150621_2320_time_graph.txt, node0_20150623_1526_time_graph.txt,
> node1_20150621_1646_time_graph.txt, node1_20150621_2320_time_graph.txt,
> node1_20150623_1526_time_graph.txt, node2_20150621_1646_time_graph.txt,
> node2_20150621_2320_time_graph.txt, node2_20150623_1526_time_graph.txt,
> nodetool status infos.txt, sstable_compaction_trace.txt,
> sstable_compaction_trace_snipped.txt, sstable_counts.jpg
>
>
> This is a document bringing up some issues when DTCS is used to compact time
> series data in a three node cluster. The DTCS is currently configured with a
> few parameters that are making the configuration fairly simple, but might
> cause problems in certain special cases like recovering from the flood of
> small SSTables due to repair operation. We are suggesting some ideas that
> might be a starting point for further discussions. Following sections are
> containing:
> - Description of the cassandra setup
> - Feeding process of the data
> - Failure testing
> - Issues caused by the repair operations for the DTCS
> - Proposal for the DTCS configuration parameters
> Attachments are included to support the discussion and there is a separate
> section giving explanation for those.
> Cassandra setup and data model
> - Cluster is composed from three nodes running Cassandra 2.1.2. Replication
> factor is two and read and write consistency levels are ONE.
> - Data is time series data. Data is saved so that one row contains a certain
> time span of data for a given metric ( 20 days in this case). The row key
> contains information about the start time of the time span and metrix name.
> Column name gives the offset from the beginning of time span. Column time
> stamp is set to correspond time stamp when adding together the timestamp from
> the row key and the offset (the actual time stamp of data point). Data model
> is analog to KairosDB implementation.
> - Average sampling rate is 10 seconds varying significantly from metric to
> metric.
> - 100 000 metrics are fed to the Cassandra.
> - max_sstable_age_days is set to 5 days (objective is to keep SStable files
> in manageable size, around 50 GB)
> - TTL is not in use in the test.
> Procedure for the failure test.
> - Data is first dumped to Cassandra for 11 days and the data dumping is
> stopped so that DTCS will have a change to finish all compactions. Data is
> dumped with "fake timestamps" so that column time stamp is set when data is
> written to Cassandra.
> - One of the nodes is taken down and new data is dumped on top of the earlier
> data covering couple of hours worth of data (faked time stamps).
> - Dumping is stopped and the node is kept down for few hours.
> - Node is taken up and the "nodetool repair" is applied on the node that was
> down.
> Consequences
> - Repair operation will lead to massive amount of new SStables far back in
> the history. New SStables are covering similar time spans than the files that
> were created by DTCS before the shutdown of one of the nodes.
> - To be able to compact the small files the max_sstable_age_days should be
> increased to allow compaction to handle the files. However, the in a
> practical case the time window will increase so large that generated files
> will be huge that is not desirable. The compaction also combines together one
> very large file with a bunch of small files in several phases that is not
> effective. Generating really large files may also lead to out of disc space
> problems.
> - See the list of time graphs later in the document.
> Improvement proposals for the DTCS configuration
> Below is a
[jira] [Commented] (CASSANDRA-9130) reduct default dtcs max_sstable_age
[ https://issues.apache.org/jira/browse/CASSANDRA-9130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14615085#comment-14615085 ] Jeff Jirsa commented on CASSANDRA-9130: --- I know this sounds obvious, but I'm repeating it for the record, because it's relevant: You will still introduce fragmentation during bootstrap, bulk load, and decom, potentially thousands or tens of thousands of sstables per node for decom. Lowing max_sstable_age_days below the table TTL is not necessarily safe, with or without CASSANDRA-9056 > reduct default dtcs max_sstable_age > --- > > Key: CASSANDRA-9130 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9130 > Project: Cassandra > Issue Type: Improvement >Reporter: Jonathan Ellis >Assignee: Marcus Eriksson >Priority: Minor > Fix For: 3.x, 2.1.x, 2.0.x > > > Now that CASSANDRA-9056 is fixed it should be safe to reduce the default age > and increase performance correspondingly. [~jshook] suggests that two weeks > may be appropriate, or we could make it dynamic based on gcgs (since that's > the window past which we should expect repair to not introduce fragmentation > anymore). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9666) Provide an alternative to DTCS
[
https://issues.apache.org/jira/browse/CASSANDRA-9666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14615167#comment-14615167
]
Jeff Jirsa commented on CASSANDRA-9666:
---
{quote}
I think we would come a long way if we did these things to DTCS;
- make it possible to put a cap on the size of the windows
{quote}
Let me think this through -
The second and third points are a given - I think they make very good sense.
The first point -
Capping the size of a window mitigates some of the pain - it basically renders
sstable_max_age_days less meaningful (everyone would have it set to gcgs if
they don't plan on changing topology, or the table's ttl so that
bootstrap/decom works as intended if they do - smart operators who know that
occasionally bootstraps happen mean they probably just set it to ttl, which
means it's disabled).
Instead, you'd have some new parameter that says "I want the largest window no
larger than X" (seconds? days? megabytes?), below which you'd have a tiering
system that does weird things (by our experience, and I imagine by yours, too,
as you note DTCS gets confused with invalid timestamps in CASSANDRA-8340), and
above which is essentially a fixed window.
Don't we end up with something that looks very much like TWCS, with a difficult
to tune, confusion prone tiering below some cutoff parameter?
> Provide an alternative to DTCS
> --
>
> Key: CASSANDRA-9666
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9666
> Project: Cassandra
> Issue Type: Improvement
>Reporter: Jeff Jirsa
>Assignee: Jeff Jirsa
> Fix For: 2.1.x, 2.2.x
>
>
> DTCS is great for time series data, but it comes with caveats that make it
> difficult to use in production (typical operator behaviors such as bootstrap,
> removenode, and repair have MAJOR caveats as they relate to
> max_sstable_age_days, and hints/read repair break the selection algorithm).
> I'm proposing an alternative, TimeWindowCompactionStrategy, that sacrifices
> the tiered nature of DTCS in order to address some of DTCS' operational
> shortcomings. I believe it is necessary to propose an alternative rather than
> simply adjusting DTCS, because it fundamentally removes the tiered nature in
> order to remove the parameter max_sstable_age_days - the result is very very
> different, even if it is heavily inspired by DTCS.
> Specifically, rather than creating a number of windows of ever increasing
> sizes, this strategy allows an operator to choose the window size, compact
> with STCS within the first window of that size, and aggressive compact down
> to a single sstable once that window is no longer current. The window size is
> a combination of unit (minutes, hours, days) and size (1, etc), such that an
> operator can expect all data using a block of that size to be compacted
> together (that is, if your unit is hours, and size is 6, you will create
> roughly 4 sstables per day, each one containing roughly 6 hours of data).
> The result addresses a number of the problems with
> DateTieredCompactionStrategy:
> - At the present time, DTCS’s first window is compacted using an unusual
> selection criteria, which prefers files with earlier timestamps, but ignores
> sizes. In TimeWindowCompactionStrategy, the first window data will be
> compacted with the well tested, fast, reliable STCS. All STCS options can be
> passed to TimeWindowCompactionStrategy to configure the first window’s
> compaction behavior.
> - HintedHandoff may put old data in new sstables, but it will have little
> impact other than slightly reduced efficiency (sstables will cover a wider
> range, but the old timestamps will not impact sstable selection criteria
> during compaction)
> - ReadRepair may put old data in new sstables, but it will have little impact
> other than slightly reduced efficiency (sstables will cover a wider range,
> but the old timestamps will not impact sstable selection criteria during
> compaction)
> - Small, old sstables resulting from streams of any kind will be swiftly and
> aggressively compacted with the other sstables matching their similar
> maxTimestamp, without causing sstables in neighboring windows to grow in size.
> - The configuration options are explicit and straightforward - the tuning
> parameters leave little room for error. The window is set in common, easily
> understandable terms such as “12 hours”, “1 Day”, “30 days”. The
> minute/hour/day options are granular enough for users keeping data for hours,
> and users keeping data for years.
> - There is no explicitly configurable max sstable age, though sstables will
> naturally stop compacting once new data is written in that window.
> - Streaming operations can create sstables with old timestamps, and they'll
> naturally be joined together wi
[jira] [Updated] (CASSANDRA-9645) Make DateTieredCompactionStrategy easier to use
[ https://issues.apache.org/jira/browse/CASSANDRA-9645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Jirsa updated CASSANDRA-9645: -- Labels: dtcs (was: ) > Make DateTieredCompactionStrategy easier to use > --- > > Key: CASSANDRA-9645 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9645 > Project: Cassandra > Issue Type: Improvement >Reporter: Marcus Eriksson > Labels: dtcs > > It has proven to be quite difficult to use DTCS properly, we need to make it > easier and safer. Things to do could be (but there is surely more things) : > * put warnings in logs if read repair is used > * better debug logging which explains why certain sstables are selected etc > * Auto-tune settings, this could be quite difficult, but we should atleast do > CASSANDRA-9130 to default max_sstable_age to gcgs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-9644) DTCS configuration proposals for handling consequences of repairs
[ https://issues.apache.org/jira/browse/CASSANDRA-9644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Jirsa updated CASSANDRA-9644: -- Labels: dtcs (was: ) > DTCS configuration proposals for handling consequences of repairs > - > > Key: CASSANDRA-9644 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9644 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Antti Nissinen > Labels: dtcs > Fix For: 3.x, 2.1.x > > Attachments: node0_20150621_1646_time_graph.txt, > node0_20150621_2320_time_graph.txt, node0_20150623_1526_time_graph.txt, > node1_20150621_1646_time_graph.txt, node1_20150621_2320_time_graph.txt, > node1_20150623_1526_time_graph.txt, node2_20150621_1646_time_graph.txt, > node2_20150621_2320_time_graph.txt, node2_20150623_1526_time_graph.txt, > nodetool status infos.txt, sstable_compaction_trace.txt, > sstable_compaction_trace_snipped.txt, sstable_counts.jpg > > > This is a document bringing up some issues when DTCS is used to compact time > series data in a three node cluster. The DTCS is currently configured with a > few parameters that are making the configuration fairly simple, but might > cause problems in certain special cases like recovering from the flood of > small SSTables due to repair operation. We are suggesting some ideas that > might be a starting point for further discussions. Following sections are > containing: > - Description of the cassandra setup > - Feeding process of the data > - Failure testing > - Issues caused by the repair operations for the DTCS > - Proposal for the DTCS configuration parameters > Attachments are included to support the discussion and there is a separate > section giving explanation for those. > Cassandra setup and data model > - Cluster is composed from three nodes running Cassandra 2.1.2. Replication > factor is two and read and write consistency levels are ONE. > - Data is time series data. Data is saved so that one row contains a certain > time span of data for a given metric ( 20 days in this case). The row key > contains information about the start time of the time span and metrix name. > Column name gives the offset from the beginning of time span. Column time > stamp is set to correspond time stamp when adding together the timestamp from > the row key and the offset (the actual time stamp of data point). Data model > is analog to KairosDB implementation. > - Average sampling rate is 10 seconds varying significantly from metric to > metric. > - 100 000 metrics are fed to the Cassandra. > - max_sstable_age_days is set to 5 days (objective is to keep SStable files > in manageable size, around 50 GB) > - TTL is not in use in the test. > Procedure for the failure test. > - Data is first dumped to Cassandra for 11 days and the data dumping is > stopped so that DTCS will have a change to finish all compactions. Data is > dumped with "fake timestamps" so that column time stamp is set when data is > written to Cassandra. > - One of the nodes is taken down and new data is dumped on top of the earlier > data covering couple of hours worth of data (faked time stamps). > - Dumping is stopped and the node is kept down for few hours. > - Node is taken up and the "nodetool repair" is applied on the node that was > down. > Consequences > - Repair operation will lead to massive amount of new SStables far back in > the history. New SStables are covering similar time spans than the files that > were created by DTCS before the shutdown of one of the nodes. > - To be able to compact the small files the max_sstable_age_days should be > increased to allow compaction to handle the files. However, the in a > practical case the time window will increase so large that generated files > will be huge that is not desirable. The compaction also combines together one > very large file with a bunch of small files in several phases that is not > effective. Generating really large files may also lead to out of disc space > problems. > - See the list of time graphs later in the document. > Improvement proposals for the DTCS configuration > Below is a list of desired properties for the configuration. Current > parameters are mentioned if available. > - Initial window size (currently:base_time_seconds) > - The amount of similar size windows for the bucketing (currently: > min_threshold) > - The multiplier for the window size when increased (currently: > min_threshold). This we would like to be independent from the min_threshold > parameter so that you could actually control the rate how fast the window > size is increased. > - Maximum length of the time window inside which the files are assigned for a > certain bucket (not currently defined). This means that expansion of time
[jira] [Comment Edited] (CASSANDRA-9130) reduct default dtcs max_sstable_age
[
https://issues.apache.org/jira/browse/CASSANDRA-9130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14615585#comment-14615585
]
Jeff Jirsa edited comment on CASSANDRA-9130 at 7/6/15 8:27 PM:
---
This is also discussed in CASSANDRA-9644, but quoting [~krummas] from
CASSANDRA-9666 (which, full disclosure, is my ticket about how I'd like to
replace DTCS with an alternative that doesn't need max_sstable_age_days at all):
{quote}
- make it possible to put a cap on the size of the windows
{quote}
The problem is that windows grow larger as they get older (increasing the cost
of re-compaction). If max_sstable_age_days is set low, and fragmentation
happens - and it will happen, sooner or later, to any real production cluster -
the alternatives are either raise max_sstable_age_days to allow compaction, or
eat the performance/memory hit from having tens of thousands of sstables (in
2.1.5/6, this was a near-instant OOM/crash). When you raise
max_sstable_age_days, you'll PROBABLY run your server out of disk, because it's
going to pick the biggest sstables and recompact them in parallel (and likely
using a sub-optimal selection strategy - CASSANDRA-9597 - fixable, but as of
July 2015, still a problem), more than doubling your usage as if you were doing
an STCS major. Yea, we all know that the recommendation in STCS days was to
stay below 50% disk capacity, but reaching that limit is about the time you'll
start streaming, and that's when all hell will break loose with DTCS as it
exists.
was (Author: jjirsa):
This is also discussed in CASSANDRA-9644, but quoting [~krummas] from
CASSANDRA-9666 (which, full disclosure, is my ticket about how I'd like to
replace DTCS with an alternative that doesn't need max_sstable_age_days at all):
{quote}
- make it possible to put a cap on the size of the windows
{quote}
The problem is that windows grow larger as they get older (increasing the cost
of re-compaction). If max_sstable_age_days is set low, and fragmentation
happens - and it will happen, sooner or later, to any real production cluster -
the alternatives are either raise max_sstable_age_days to allow compaction, or
eat the performance/memory hit from having tens of thousands of sstables (in
2.1.5/6, this was a near-instant OOM/crash). When you raise
max_sstable_age_days, you'll PROBABLY run your server out of disk, because it's
going to pick the biggest sstables and recompact them in parallel, doubling
your usage as if you were doing an STCS major. Yea, we all know that the
recommendation in STCS days was to stay below 50% disk capacity, but reaching
that limit is about the time you'll start streaming, and that's when all hell
will break loose with DTCS as it exists.
> reduct default dtcs max_sstable_age
> ---
>
> Key: CASSANDRA-9130
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9130
> Project: Cassandra
> Issue Type: Improvement
>Reporter: Jonathan Ellis
>Assignee: Marcus Eriksson
>Priority: Minor
> Fix For: 3.x, 2.1.x, 2.0.x
>
>
> Now that CASSANDRA-9056 is fixed it should be safe to reduce the default age
> and increase performance correspondingly. [~jshook] suggests that two weeks
> may be appropriate, or we could make it dynamic based on gcgs (since that's
> the window past which we should expect repair to not introduce fragmentation
> anymore).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9130) reduct default dtcs max_sstable_age
[
https://issues.apache.org/jira/browse/CASSANDRA-9130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14615585#comment-14615585
]
Jeff Jirsa commented on CASSANDRA-9130:
---
This is also discussed in CASSANDRA-9644, but quoting [~krummas] from
CASSANDRA-9666 (which, full disclosure, is my ticket about how I'd like to
replace DTCS with an alternative that doesn't need max_sstable_age_days at all):
{quote}
- make it possible to put a cap on the size of the windows
{quote}
The problem is that windows grow larger as they get older (increasing the cost
of re-compaction). If max_sstable_age_days is set low, and fragmentation
happens - and it will happen, sooner or later, to any real production cluster -
the alternatives are either raise max_sstable_age_days to allow compaction, or
eat the performance/memory hit from having tens of thousands of sstables (in
2.1.5/6, this was a near-instant OOM/crash). When you raise
max_sstable_age_days, you'll PROBABLY run your server out of disk, because it's
going to pick the biggest sstables and recompact them in parallel, doubling
your usage as if you were doing an STCS major. Yea, we all know that the
recommendation in STCS days was to stay below 50% disk capacity, but reaching
that limit is about the time you'll start streaming, and that's when all hell
will break loose with DTCS as it exists.
> reduct default dtcs max_sstable_age
> ---
>
> Key: CASSANDRA-9130
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9130
> Project: Cassandra
> Issue Type: Improvement
>Reporter: Jonathan Ellis
>Assignee: Marcus Eriksson
>Priority: Minor
> Fix For: 3.x, 2.1.x, 2.0.x
>
>
> Now that CASSANDRA-9056 is fixed it should be safe to reduce the default age
> and increase performance correspondingly. [~jshook] suggests that two weeks
> may be appropriate, or we could make it dynamic based on gcgs (since that's
> the window past which we should expect repair to not introduce fragmentation
> anymore).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-9130) reduct default dtcs max_sstable_age
[
https://issues.apache.org/jira/browse/CASSANDRA-9130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14615585#comment-14615585
]
Jeff Jirsa edited comment on CASSANDRA-9130 at 7/6/15 8:40 PM:
---
This is also discussed in CASSANDRA-9644, but quoting [~krummas] from
CASSANDRA-9666 (which, full disclosure, is my ticket about how I'd like to
replace DTCS with an alternative that doesn't need max_sstable_age_days at all):
{quote}
- make it possible to put a cap on the size of the windows
{quote}
The problem is that windows grow larger as they get older (increasing the cost
of re-compaction). If max_sstable_age_days is set low, and fragmentation
happens - and it will happen, sooner or later, to any real production cluster -
the alternatives are either raise max_sstable_age_days to allow compaction, or
eat the performance/memory hit from having tens of thousands of sstables (in
2.1.5/6, this was a near-instant OOM/crash).
When you raise max_sstable_age_days, you'll not only have to re-compact in the
smaller files, but you'll also re-compact neighboring files in the larger
windows which were shielded by max_sstable_age_days, causing 1-min_threshold
larger sstables to result, which probably wasn't the intention of the operator
who had configured the lower max_sstable_age_days
was (Author: jjirsa):
This is also discussed in CASSANDRA-9644, but quoting [~krummas] from
CASSANDRA-9666 (which, full disclosure, is my ticket about how I'd like to
replace DTCS with an alternative that doesn't need max_sstable_age_days at all):
{quote}
- make it possible to put a cap on the size of the windows
{quote}
The problem is that windows grow larger as they get older (increasing the cost
of re-compaction). If max_sstable_age_days is set low, and fragmentation
happens - and it will happen, sooner or later, to any real production cluster -
the alternatives are either raise max_sstable_age_days to allow compaction, or
eat the performance/memory hit from having tens of thousands of sstables (in
2.1.5/6, this was a near-instant OOM/crash). When you raise
max_sstable_age_days, you'll PROBABLY run your server out of disk, because it's
going to pick the biggest sstables and recompact them in parallel (and likely
using a sub-optimal selection strategy - CASSANDRA-9597 - fixable, but as of
July 2015, still a problem), more than doubling your usage as if you were doing
an STCS major. Yea, we all know that the recommendation in STCS days was to
stay below 50% disk capacity, but reaching that limit is about the time you'll
start streaming, and that's when all hell will break loose with DTCS as it
exists.
> reduct default dtcs max_sstable_age
> ---
>
> Key: CASSANDRA-9130
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9130
> Project: Cassandra
> Issue Type: Improvement
>Reporter: Jonathan Ellis
>Assignee: Marcus Eriksson
>Priority: Minor
> Fix For: 3.x, 2.1.x, 2.0.x
>
>
> Now that CASSANDRA-9056 is fixed it should be safe to reduce the default age
> and increase performance correspondingly. [~jshook] suggests that two weeks
> may be appropriate, or we could make it dynamic based on gcgs (since that's
> the window past which we should expect repair to not introduce fragmentation
> anymore).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9742) Nodetool verify
[
https://issues.apache.org/jira/browse/CASSANDRA-9742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14617790#comment-14617790
]
Jeff Jirsa commented on CASSANDRA-9742:
---
Operator perspective, fwiw: I already have repair schedules. I already know
what needs to be repaired and what doesn't. What I didn't have, previously, was
a way to validate the files on disk actually matched what I believed they
matched, short of running scrub.
`verify` was very literally `read only scrub` - when I wrote 5791, I followed
the scrub code path very closely, because that was the use case I was worried
about when I wrote it (the concern was bit level corruption due to failing
HDD/RAID controller - scrub would do the job, but it's a heavy hammer hitting a
tiny nail). The notion of "verify this node has all the data" was already
covered by repair, so I never even considered having `verify` do that.
Why not just have incremental repair validate checksums for all sstables - the
verifier will {{mutateRepairedAt(sstable.descriptor,
ActiveRepairService.UNREPAIRED_SSTABLE)}} on checksum failure which then allows
incremental repair to re-repair that data?
> Nodetool verify
> ---
>
> Key: CASSANDRA-9742
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9742
> Project: Cassandra
> Issue Type: New Feature
> Components: Tools
>Reporter: Jonathan Ellis
> Fix For: 3.x
>
>
> We introduced incremental repair in 2.1 but it is difficult to make that the
> default without unpleasant surprises for incautious users.
> Additionally, while we now store sstable checksums, we leave verification to
> the user.
> I propose introducing a new command, {{nodetool verify}}, that would address
> both of these.
> Default operation would be to do an incremental repair, plus validate
> checksums on *all* sstables (not just unrepaired ones). We could also have
> --local mode (checksums only) and --full (classic repair).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-9742) Nodetool verify
[
https://issues.apache.org/jira/browse/CASSANDRA-9742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14617790#comment-14617790
]
Jeff Jirsa edited comment on CASSANDRA-9742 at 7/8/15 1:17 AM:
---
Operator perspective, fwiw: I already have repair schedules. I already know
what needs to be repaired and what doesn't. What I didn't have, previously, was
a way to validate the files on disk actually matched what I believed they
matched, short of running scrub.
`verify` was very literally `read only scrub` - when I wrote 5791, I followed
the scrub code path very closely, because that was the use case I was worried
about when I wrote it (the concern was bit level corruption due to failing
HDD/RAID controller - scrub would do the job, but it's a heavy hammer hitting a
tiny nail). The notion of "verify this node has all the data" was already
covered by repair, so I never even considered having `verify` do that.
Why not just (add a flag to) enable incremental repair validate checksums for
all sstables - the verifier will {{mutateRepairedAt(sstable.descriptor,
ActiveRepairService.UNREPAIRED_SSTABLE)}} on checksum failure which then allows
incremental repair to re-repair that data?
was (Author: jjirsa):
Operator perspective, fwiw: I already have repair schedules. I already know
what needs to be repaired and what doesn't. What I didn't have, previously, was
a way to validate the files on disk actually matched what I believed they
matched, short of running scrub.
`verify` was very literally `read only scrub` - when I wrote 5791, I followed
the scrub code path very closely, because that was the use case I was worried
about when I wrote it (the concern was bit level corruption due to failing
HDD/RAID controller - scrub would do the job, but it's a heavy hammer hitting a
tiny nail). The notion of "verify this node has all the data" was already
covered by repair, so I never even considered having `verify` do that.
Why not just have incremental repair validate checksums for all sstables - the
verifier will {{mutateRepairedAt(sstable.descriptor,
ActiveRepairService.UNREPAIRED_SSTABLE)}} on checksum failure which then allows
incremental repair to re-repair that data?
> Nodetool verify
> ---
>
> Key: CASSANDRA-9742
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9742
> Project: Cassandra
> Issue Type: New Feature
> Components: Tools
>Reporter: Jonathan Ellis
> Fix For: 3.x
>
>
> We introduced incremental repair in 2.1 but it is difficult to make that the
> default without unpleasant surprises for incautious users.
> Additionally, while we now store sstable checksums, we leave verification to
> the user.
> I propose introducing a new command, {{nodetool verify}}, that would address
> both of these.
> Default operation would be to do an incremental repair, plus validate
> checksums on *all* sstables (not just unrepaired ones). We could also have
> --local mode (checksums only) and --full (classic repair).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-9742) Nodetool verify
[
https://issues.apache.org/jira/browse/CASSANDRA-9742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14617790#comment-14617790
]
Jeff Jirsa edited comment on CASSANDRA-9742 at 7/8/15 1:38 AM:
---
Operator perspective, fwiw: I already have repair schedules. I already know
what needs to be repaired and what doesn't. What I didn't have, previously, was
a way to validate the files on disk actually matched what I believed they
matched, short of running scrub.
`verify` was very literally `read only scrub` - when I wrote 5791, I followed
the scrub code path very closely (as [~kohlisankalp] suggested in that ticket),
because that was the use case I was worried about when I wrote it (my concern
was bit level corruption due to failing HDD/RAID controller - scrub would do
the job, but it's a heavy hammer hitting a tiny nail). The notion of "verify
this node has all the data" was already covered by repair, so I never even
considered having `verify` do that.
Why not just (add a flag to) enable incremental repair validate checksums for
all sstables - the verifier will {{mutateRepairedAt(sstable.descriptor,
ActiveRepairService.UNREPAIRED_SSTABLE)}} on checksum failure which then allows
incremental repair to re-repair that data?
was (Author: jjirsa):
Operator perspective, fwiw: I already have repair schedules. I already know
what needs to be repaired and what doesn't. What I didn't have, previously, was
a way to validate the files on disk actually matched what I believed they
matched, short of running scrub.
`verify` was very literally `read only scrub` - when I wrote 5791, I followed
the scrub code path very closely, because that was the use case I was worried
about when I wrote it (the concern was bit level corruption due to failing
HDD/RAID controller - scrub would do the job, but it's a heavy hammer hitting a
tiny nail). The notion of "verify this node has all the data" was already
covered by repair, so I never even considered having `verify` do that.
Why not just (add a flag to) enable incremental repair validate checksums for
all sstables - the verifier will {{mutateRepairedAt(sstable.descriptor,
ActiveRepairService.UNREPAIRED_SSTABLE)}} on checksum failure which then allows
incremental repair to re-repair that data?
> Nodetool verify
> ---
>
> Key: CASSANDRA-9742
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9742
> Project: Cassandra
> Issue Type: New Feature
> Components: Tools
>Reporter: Jonathan Ellis
> Fix For: 3.x
>
>
> We introduced incremental repair in 2.1 but it is difficult to make that the
> default without unpleasant surprises for incautious users.
> Additionally, while we now store sstable checksums, we leave verification to
> the user.
> I propose introducing a new command, {{nodetool verify}}, that would address
> both of these.
> Default operation would be to do an incremental repair, plus validate
> checksums on *all* sstables (not just unrepaired ones). We could also have
> --local mode (checksums only) and --full (classic repair).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9661) Endless compaction to a tiny, tombstoned SStable
[
https://issues.apache.org/jira/browse/CASSANDRA-9661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14621818#comment-14621818
]
Jeff Jirsa commented on CASSANDRA-9661:
---
[~noel2004] - can you confirm this was on 2.1.5?
> Endless compaction to a tiny, tombstoned SStable
>
>
> Key: CASSANDRA-9661
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9661
> Project: Cassandra
> Issue Type: Bug
> Components: Core
>Reporter: WeiFan
>Assignee: Yuki Morishita
> Labels: compaction, dtcs
>
> We deployed a 3-nodes cluster (with 2.1.5) which worked under stable write
> requests ( about 2k wps) to a CF with DTCS, a default TTL as 43200s and
> gc_grace as 21600s. The CF contained inserted only, complete time series
> data. We found cassandra will occasionally keep writing logs like this:
> INFO [CompactionExecutor:30551] 2015-06-26 18:10:06,195
> CompactionTask.java:270 - Compacted 1 sstables to
> [/home/cassandra/workdata/data/sen_vaas_test/nodestatus-f96c7c50155811e589f69752ac9b06c7/sen_vaas_test-nodestatus-ka-2516270,].
> 449 bytes to 449 (~100% of original) in 12ms = 0.035683MB/s. 4 total
> partitions merged to 4. Partition merge counts were {1:4, }
> INFO [CompactionExecutor:30551] 2015-06-26 18:10:06,241
> CompactionTask.java:140 - Compacting
> [SSTableReader(path='/home/cassandra/workdata/data/sen_vaas_test/nodestatus-f96c7c50155811e589f69752ac9b06c7/sen_vaas_test-nodestatus-ka-2516270-Data.db')]
> INFO [CompactionExecutor:30551] 2015-06-26 18:10:06,253
> CompactionTask.java:270 - Compacted 1 sstables to
> [/home/cassandra/workdata/data/sen_vaas_test/nodestatus-f96c7c50155811e589f69752ac9b06c7/sen_vaas_test-nodestatus-ka-2516271,].
> 449 bytes to 449 (~100% of original) in 12ms = 0.035683MB/s. 4 total
> partitions merged to 4. Partition merge counts were {1:4, }
> It seems that cassandra kept doing compacting to a single SStable, serveral
> times per second, and lasted for many hours. Tons of logs were thrown and one
> CPU core exhausted during this time. The endless compacting finally end when
> another compaction started with a group of SStables (including previous one).
> All of our 3 nodes have been hit by this problem, but occurred in different
> time.
> We could not figure out how the problematic SStable come up because the log
> has wrapped around.
> We have dumped the records in the SStable and found it has the oldest data in
> our CF (again, our data was time series), and all of the record in this
> SStable have bben expired for more than 18 hours (12 hrs TTL + 6 hrs gc) so
> they should be dropped. However, c* do nothing to this SStable but compacting
> it again and again, until more SStable were out-dated enough to be considered
> for compacting together with this one by DTCS.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9774) fix sstableverify dtest
[
https://issues.apache.org/jira/browse/CASSANDRA-9774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14623251#comment-14623251
]
Jeff Jirsa commented on CASSANDRA-9774:
---
Is this related to CASSANDRA-9686 ?
> fix sstableverify dtest
> ---
>
> Key: CASSANDRA-9774
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9774
> Project: Cassandra
> Issue Type: Sub-task
>Reporter: Jim Witschey
>Assignee: Jim Witschey
>Priority: Blocker
> Fix For: 3.0.x
>
>
> One of our dtests for {{sstableverify}}
> ({{offline_tools_test.py:TestOfflineTools.sstableverify_test}}) is failing
> hard on trunk ([cassci
> history|http://cassci.datastax.com/view/trunk/job/trunk_dtest/lastCompletedBuild/testReport/offline_tools_test/TestOfflineTools/sstableverify_test/history/])
> The way the test works is by deleting an SSTable, then running
> {{sstableverify}} on its table. In earlier versions, it successfully detects
> this problem and outputs that it "was not released before the reference was
> garbage collected". The test no longer finds this string in the output;
> looking through the output of the test, it doesn't look like it reports any
> problems at all.
> EDIT: After digging into the C* source a bit, I may have misattributed the
> problem to {{sstableverify}}; this could be a more general memory management
> problem, as the error text expected in the dtest is emitted by part of the
> {{Ref}} implementation:
> https://github.com/apache/cassandra/blob/075ff5000ced24b42f3b540815cae471bee4049d/src/java/org/apache/cassandra/utils/concurrent/Ref.java#L187
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9686) FSReadError and LEAK DETECTED after upgrading
[
https://issues.apache.org/jira/browse/CASSANDRA-9686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14623252#comment-14623252
]
Jeff Jirsa commented on CASSANDRA-9686:
---
Is this related to CASSANDRA-9774 ?
> FSReadError and LEAK DETECTED after upgrading
> -
>
> Key: CASSANDRA-9686
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9686
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: Windows-7-32 bit, 3.2GB RAM, Java 1.7.0_55
>Reporter: Andreas Schnitzerling
>Assignee: Stefania
> Fix For: 2.1.x, 2.2.x
>
> Attachments: cassandra.bat, cassandra.yaml,
> compactions_in_progress.zip, sstable_activity.zip, system.log
>
>
> After upgrading one of 15 nodes from 2.1.7 to 2.2.0-rc1 I get FSReadError and
> LEAK DETECTED on start. Deleting the listed files, the failure goes away.
> {code:title=system.log}
> ERROR [SSTableBatchOpen:1] 2015-06-29 14:38:34,554
> DebuggableThreadPoolExecutor.java:242 - Error in ThreadPoolExecutor
> org.apache.cassandra.io.FSReadError: java.io.IOException: Compressed file
> with 0 chunks encountered: java.io.DataInputStream@1c42271
> at
> org.apache.cassandra.io.compress.CompressionMetadata.readChunkOffsets(CompressionMetadata.java:178)
> ~[apache-cassandra-2.2.0-rc1.jar:2.2.0-rc1]
> at
> org.apache.cassandra.io.compress.CompressionMetadata.(CompressionMetadata.java:117)
> ~[apache-cassandra-2.2.0-rc1.jar:2.2.0-rc1]
> at
> org.apache.cassandra.io.compress.CompressionMetadata.create(CompressionMetadata.java:86)
> ~[apache-cassandra-2.2.0-rc1.jar:2.2.0-rc1]
> at
> org.apache.cassandra.io.util.CompressedSegmentedFile$Builder.metadata(CompressedSegmentedFile.java:142)
> ~[apache-cassandra-2.2.0-rc1.jar:2.2.0-rc1]
> at
> org.apache.cassandra.io.util.CompressedPoolingSegmentedFile$Builder.complete(CompressedPoolingSegmentedFile.java:101)
> ~[apache-cassandra-2.2.0-rc1.jar:2.2.0-rc1]
> at
> org.apache.cassandra.io.util.SegmentedFile$Builder.complete(SegmentedFile.java:178)
> ~[apache-cassandra-2.2.0-rc1.jar:2.2.0-rc1]
> at
> org.apache.cassandra.io.sstable.format.SSTableReader.load(SSTableReader.java:681)
> ~[apache-cassandra-2.2.0-rc1.jar:2.2.0-rc1]
> at
> org.apache.cassandra.io.sstable.format.SSTableReader.load(SSTableReader.java:644)
> ~[apache-cassandra-2.2.0-rc1.jar:2.2.0-rc1]
> at
> org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:443)
> ~[apache-cassandra-2.2.0-rc1.jar:2.2.0-rc1]
> at
> org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:350)
> ~[apache-cassandra-2.2.0-rc1.jar:2.2.0-rc1]
> at
> org.apache.cassandra.io.sstable.format.SSTableReader$4.run(SSTableReader.java:480)
> ~[apache-cassandra-2.2.0-rc1.jar:2.2.0-rc1]
> at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
> ~[na:1.7.0_55]
> at java.util.concurrent.FutureTask.run(Unknown Source) ~[na:1.7.0_55]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> [na:1.7.0_55]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> [na:1.7.0_55]
> at java.lang.Thread.run(Unknown Source) [na:1.7.0_55]
> Caused by: java.io.IOException: Compressed file with 0 chunks encountered:
> java.io.DataInputStream@1c42271
> at
> org.apache.cassandra.io.compress.CompressionMetadata.readChunkOffsets(CompressionMetadata.java:174)
> ~[apache-cassandra-2.2.0-rc1.jar:2.2.0-rc1]
> ... 15 common frames omitted
> ERROR [Reference-Reaper:1] 2015-06-29 14:38:34,734 Ref.java:189 - LEAK
> DETECTED: a reference
> (org.apache.cassandra.utils.concurrent.Ref$State@3e547f) to class
> org.apache.cassandra.io.sstable.format.SSTableReader$InstanceTidier@1926439:D:\Programme\Cassandra\data\data\system\compactions_in_progress\system-compactions_in_progress-ka-6866
> was not released before the reference was garbage collected
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-8748) Backport memory leak fix from CASSANDRA-8707 to 2.0
[ https://issues.apache.org/jira/browse/CASSANDRA-8748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Jirsa updated CASSANDRA-8748: -- Fix Version/s: 2.0.13 > Backport memory leak fix from CASSANDRA-8707 to 2.0 > --- > > Key: CASSANDRA-8748 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8748 > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Jeremy Hanna >Assignee: Benedict > Fix For: 2.0.13 > > Attachments: 8748.2.txt, 8748.txt > > > There are multiple elements in CASSANDRA-8707 but the memory leak is common > to Cassandra 2.0. This ticket is to fix the memory leak specifically for 2.0. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-9666) Provide an alternative to DTCS
[ https://issues.apache.org/jira/browse/CASSANDRA-9666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Jirsa updated CASSANDRA-9666: -- Description: DTCS is great for time series data, but it comes with caveats that make it difficult to use in production (typical operator behaviors such as bootstrap, removenode, and repair have MAJOR caveats as they relate to max_sstable_age_days, and hints/read repair break the selection algorithm). I'm proposing an alternative, TimeWindowCompactionStrategy, that sacrifices the tiered nature of DTCS in order to address some of DTCS' operational shortcomings. I believe it is necessary to propose an alternative rather than simply adjusting DTCS, because it fundamentally removes the tiered nature in order to remove the parameter max_sstable_age_days - the result is very very different, even if it is heavily inspired by DTCS. Specifically, rather than creating a number of windows of ever increasing sizes, this strategy allows an operator to choose the window size, compact with STCS within the first window of that size, and aggressive compact down to a single sstable once that window is no longer current. The window size is a combination of unit (minutes, hours, days) and size (1, etc), such that an operator can expect all data using a block of that size to be compacted together (that is, if your unit is hours, and size is 6, you will create roughly 4 sstables per day, each one containing roughly 6 hours of data). The result addresses a number of the problems with DateTieredCompactionStrategy: - At the present time, DTCS’s first window is compacted using an unusual selection criteria, which prefers files with earlier timestamps, but ignores sizes. In TimeWindowCompactionStrategy, the first window data will be compacted with the well tested, fast, reliable STCS. All STCS options can be passed to TimeWindowCompactionStrategy to configure the first window’s compaction behavior. - HintedHandoff may put old data in new sstables, but it will have little impact other than slightly reduced efficiency (sstables will cover a wider range, but the old timestamps will not impact sstable selection criteria during compaction) - ReadRepair may put old data in new sstables, but it will have little impact other than slightly reduced efficiency (sstables will cover a wider range, but the old timestamps will not impact sstable selection criteria during compaction) - Small, old sstables resulting from streams of any kind will be swiftly and aggressively compacted with the other sstables matching their similar maxTimestamp, without causing sstables in neighboring windows to grow in size. - The configuration options are explicit and straightforward - the tuning parameters leave little room for error. The window is set in common, easily understandable terms such as “12 hours”, “1 Day”, “30 days”. The minute/hour/day options are granular enough for users keeping data for hours, and users keeping data for years. - There is no explicitly configurable max sstable age, though sstables will naturally stop compacting once new data is written in that window. - Streaming operations can create sstables with old timestamps, and they'll naturally be joined together with sstables in the same time bucket. This is true for bootstrap/repair/sstableloader/removenode. - It remains true that if old data and new data is written into the memtable at the same time, the resulting sstables will be treated as if they were new sstables, however, that no longer negatively impacts the compaction strategy’s selection criteria for older windows. Patch provided for (Rebased July 18, with bug fixes for estimated pending compactions and potential starvation if more than min_threshold tables existed in current window but STCS did not consider them viable candidates): - 2.1: https://github.com/jeffjirsa/cassandra/commits/twcs-2.1 - 2.2: https://github.com/jeffjirsa/cassandra/commits/twcs-2.2 - trunk (post-8099): https://github.com/jeffjirsa/cassandra/commits/twcs was: DTCS is great for time series data, but it comes with caveats that make it difficult to use in production (typical operator behaviors such as bootstrap, removenode, and repair have MAJOR caveats as they relate to max_sstable_age_days, and hints/read repair break the selection algorithm). I'm proposing an alternative, TimeWindowCompactionStrategy, that sacrifices the tiered nature of DTCS in order to address some of DTCS' operational shortcomings. I believe it is necessary to propose an alternative rather than simply adjusting DTCS, because it fundamentally removes the tiered nature in order to remove the parameter max_sstable_age_days - the result is very very different, even if it is heavily inspired by DTCS. Specifically, rather than creating a number of windows of ever increasing sizes, this strategy allows an operator t
[jira] [Commented] (CASSANDRA-9774) fix sstableverify dtest
[
https://issues.apache.org/jira/browse/CASSANDRA-9774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14637476#comment-14637476
]
Jeff Jirsa commented on CASSANDRA-9774:
---
I wrote it. I'll take a look, but I can't see the cassci history for some
reason (get 404 at
http://cassci.datastax.com/view/trunk/job/trunk_dtest/lastCompletedBuild/testReport/offline_tools_test/TestOfflineTools/sstableverify_test/history/
)
I haven't looked at that code in a while (certainly not since 8099 went in),
but I don't recall it ever deleting sstables - however, it does intentionally
corrupt sstables, and rewrite checksums to test different failure modes, so
it's possible it's leaking a reference.
I'll look at it when I can, but if someone else beats me to it, I won't mind.
> fix sstableverify dtest
> ---
>
> Key: CASSANDRA-9774
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9774
> Project: Cassandra
> Issue Type: Sub-task
>Reporter: Jim Witschey
>Priority: Blocker
> Fix For: 3.0.x
>
>
> One of our dtests for {{sstableverify}}
> ({{offline_tools_test.py:TestOfflineTools.sstableverify_test}}) is failing
> hard on trunk ([cassci
> history|http://cassci.datastax.com/view/trunk/job/trunk_dtest/lastCompletedBuild/testReport/offline_tools_test/TestOfflineTools/sstableverify_test/history/])
> The way the test works is by deleting an SSTable, then running
> {{sstableverify}} on its table. In earlier versions, it successfully detects
> this problem and outputs that it "was not released before the reference was
> garbage collected". The test no longer finds this string in the output;
> looking through the output of the test, it doesn't look like it reports any
> problems at all.
> EDIT: After digging into the C* source a bit, I may have misattributed the
> problem to {{sstableverify}}; this could be a more general memory management
> problem, as the error text expected in the dtest is emitted by part of the
> {{Ref}} implementation:
> https://github.com/apache/cassandra/blob/075ff5000ced24b42f3b540815cae471bee4049d/src/java/org/apache/cassandra/utils/concurrent/Ref.java#L187
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-5791) A nodetool command to validate all sstables in a node
[ https://issues.apache.org/jira/browse/CASSANDRA-5791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14355775#comment-14355775 ] Jeff Jirsa commented on CASSANDRA-5791: --- New Diff at https://github.com/apache/cassandra/compare/trunk...jeffjirsa:cassandra-5791.diff > A nodetool command to validate all sstables in a node > - > > Key: CASSANDRA-5791 > URL: https://issues.apache.org/jira/browse/CASSANDRA-5791 > Project: Cassandra > Issue Type: New Feature > Components: Core >Reporter: sankalp kohli >Assignee: Jeff Jirsa >Priority: Minor > Attachments: cassandra-5791-patch-3.diff, cassandra-5791.patch-2 > > > CUrrently there is no nodetool command to validate all sstables on disk. The > only way to do this is to run a repair and see if it succeeds. But we cannot > repair the system keyspace. > Also we can run upgrade sstables but that re writes all the sstables. > This command should check the hash of all sstables and return whether all > data is readable all not. This should NOT care about consistency. > The compressed sstables do not have hash so not sure how it will work there. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-5791) A nodetool command to validate all sstables in a node
[
https://issues.apache.org/jira/browse/CASSANDRA-5791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14370077#comment-14370077
]
Jeff Jirsa edited comment on CASSANDRA-5791 at 3/19/15 8:43 PM:
Cause of tests failing is that checksums are incorrect for compressed sstables
again.
{noformat}
# cat
/Users/jeff/.ccm/snapshot/node1/data/test2/metrics-aded07e0ce7711e4897c85b755fc16c4/la-1-big-Digest.adler32
822598308
# java AdlerCheckSum
/Users/jeff/.ccm/snapshot/node1/data/test2/metrics-aded07e0ce7711e4897c85b755fc16c4/la-1-big-Data.db
864477438
{noformat}
The checksums should have been corrected by CASSANDRA-8778 so I'll figure out
where the regression happened tonight after business hours PST.
was (Author: jjirsa):
Cause of tests failing is that checksums are incorrect for compressed sstables
again.
{noformat}
# cat
/Users/jeff/.ccm/snapshot/node1/data/test2/metrics-aded07e0ce7711e4897c85b755fc16c4/la-1-big-Digest.adler32
822598308
# java AdlerCheckSum
/Users/jeff/.ccm/snapshot/node1/data/test2/metrics-aded07e0ce7711e4897c85b755fc16c4/la-1-big-Data.db
864477438
{/oformat}
The checksums should have been corrected by CASSANDRA-8778 so I'll figure out
where the regression happened tonight after business hours PST.
> A nodetool command to validate all sstables in a node
> -
>
> Key: CASSANDRA-5791
> URL: https://issues.apache.org/jira/browse/CASSANDRA-5791
> Project: Cassandra
> Issue Type: New Feature
> Components: Core
>Reporter: sankalp kohli
>Assignee: Jeff Jirsa
>Priority: Minor
> Fix For: 3.0
>
> Attachments: cassandra-5791-patch-3.diff, cassandra-5791.patch-2
>
>
> CUrrently there is no nodetool command to validate all sstables on disk. The
> only way to do this is to run a repair and see if it succeeds. But we cannot
> repair the system keyspace.
> Also we can run upgrade sstables but that re writes all the sstables.
> This command should check the hash of all sstables and return whether all
> data is readable all not. This should NOT care about consistency.
> The compressed sstables do not have hash so not sure how it will work there.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-5791) A nodetool command to validate all sstables in a node
[
https://issues.apache.org/jira/browse/CASSANDRA-5791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14370077#comment-14370077
]
Jeff Jirsa commented on CASSANDRA-5791:
---
Cause of tests failing is that checksums are incorrect for compressed sstables
again.
{noformat}
# cat
/Users/jeff/.ccm/snapshot/node1/data/test2/metrics-aded07e0ce7711e4897c85b755fc16c4/la-1-big-Digest.adler32
822598308
# java AdlerCheckSum
/Users/jeff/.ccm/snapshot/node1/data/test2/metrics-aded07e0ce7711e4897c85b755fc16c4/la-1-big-Data.db
864477438
{/oformat}
The checksums should have been corrected by CASSANDRA-8778 so I'll figure out
where the regression happened tonight after business hours PST.
> A nodetool command to validate all sstables in a node
> -
>
> Key: CASSANDRA-5791
> URL: https://issues.apache.org/jira/browse/CASSANDRA-5791
> Project: Cassandra
> Issue Type: New Feature
> Components: Core
>Reporter: sankalp kohli
>Assignee: Jeff Jirsa
>Priority: Minor
> Fix For: 3.0
>
> Attachments: cassandra-5791-patch-3.diff, cassandra-5791.patch-2
>
>
> CUrrently there is no nodetool command to validate all sstables on disk. The
> only way to do this is to run a repair and see if it succeeds. But we cannot
> repair the system keyspace.
> Also we can run upgrade sstables but that re writes all the sstables.
> This command should check the hash of all sstables and return whether all
> data is readable all not. This should NOT care about consistency.
> The compressed sstables do not have hash so not sure how it will work there.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-5791) A nodetool command to validate all sstables in a node
[
https://issues.apache.org/jira/browse/CASSANDRA-5791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14370665#comment-14370665
]
Jeff Jirsa commented on CASSANDRA-5791:
---
So the same bug corrected in CASSANDRA-8778 was re-introduced by CASSANDRA-8709
, as it was developed in parallel and was likely merged/reviewed without the
benefit of knowing about #8778.
I've done the following:
1) Rebased to trunk as of 20150319
2) Removed o.a.c.io.DataIntegrityMetadata#append
3) Corrected o.a.c.io.DataIntegrityMetadata#appendDirect
4) Brought over [~benedict]'s PureJavaCRC32 's fix from above (which was
correct - 7bef6f93aea3a6897b53e909688f5948c018ccdf)
Commit:
https://github.com/jeffjirsa/cassandra/commit/79642ea4f56a33f249e807abdd562f89d20f6c36
Diff at
https://github.com/apache/cassandra/compare/trunk...jeffjirsa:cassandra-5791.diff
, I'll also attach here as cassandra-5791-20150319.diff
Passing:
{noformat}
[junit] - ---
[junit] Testsuite: org.apache.cassandra.db.VerifyTest
[junit] Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
2.803 sec
[junit]
{noformat}
[~jbellis] and [~benedict] - if you want unit tests for DataIntegrityMetadata,
Jira it, assign me, and I'll write them. I'd have done it tonight but I can't
convince myself that they're not redundant with the (included) verifier unit
tests which will test the checksums anyway.
> A nodetool command to validate all sstables in a node
> -
>
> Key: CASSANDRA-5791
> URL: https://issues.apache.org/jira/browse/CASSANDRA-5791
> Project: Cassandra
> Issue Type: New Feature
> Components: Core
>Reporter: sankalp kohli
>Assignee: Jeff Jirsa
>Priority: Minor
> Fix For: 3.0
>
> Attachments: cassandra-5791-20150319.diff,
> cassandra-5791-patch-3.diff, cassandra-5791.patch-2
>
>
> CUrrently there is no nodetool command to validate all sstables on disk. The
> only way to do this is to run a repair and see if it succeeds. But we cannot
> repair the system keyspace.
> Also we can run upgrade sstables but that re writes all the sstables.
> This command should check the hash of all sstables and return whether all
> data is readable all not. This should NOT care about consistency.
> The compressed sstables do not have hash so not sure how it will work there.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-5791) A nodetool command to validate all sstables in a node
[ https://issues.apache.org/jira/browse/CASSANDRA-5791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Jirsa updated CASSANDRA-5791: -- Attachment: cassandra-5791-20150319.diff > A nodetool command to validate all sstables in a node > - > > Key: CASSANDRA-5791 > URL: https://issues.apache.org/jira/browse/CASSANDRA-5791 > Project: Cassandra > Issue Type: New Feature > Components: Core >Reporter: sankalp kohli >Assignee: Jeff Jirsa >Priority: Minor > Fix For: 3.0 > > Attachments: cassandra-5791-20150319.diff, > cassandra-5791-patch-3.diff, cassandra-5791.patch-2 > > > CUrrently there is no nodetool command to validate all sstables on disk. The > only way to do this is to run a repair and see if it succeeds. But we cannot > repair the system keyspace. > Also we can run upgrade sstables but that re writes all the sstables. > This command should check the hash of all sstables and return whether all > data is readable all not. This should NOT care about consistency. > The compressed sstables do not have hash so not sure how it will work there. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-5791) A nodetool command to validate all sstables in a node
[
https://issues.apache.org/jira/browse/CASSANDRA-5791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14370665#comment-14370665
]
Jeff Jirsa edited comment on CASSANDRA-5791 at 3/20/15 4:54 AM:
So the same bug corrected in CASSANDRA-8778 was re-introduced by CASSANDRA-8709
.
I've done the following:
1) Rebased to trunk as of 20150319
2) Removed o.a.c.io.DataIntegrityMetadata#append
3) Corrected o.a.c.io.DataIntegrityMetadata#appendDirect
4) Brought over [~benedict]'s PureJavaCRC32 's fix from above (which was
correct - 7bef6f93aea3a6897b53e909688f5948c018ccdf)
Commit:
https://github.com/jeffjirsa/cassandra/commit/79642ea4f56a33f249e807abdd562f89d20f6c36
Diff at
https://github.com/apache/cassandra/compare/trunk...jeffjirsa:cassandra-5791.diff
, I'll also attach here as cassandra-5791-20150319.diff
Passing:
{noformat}
[junit] - ---
[junit] Testsuite: org.apache.cassandra.db.VerifyTest
[junit] Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
2.803 sec
[junit]
{noformat}
[~jbellis] and [~benedict] - if you want unit tests for DataIntegrityMetadata,
Jira it, assign me, and I'll write them. I'd have done it tonight but I can't
convince myself that they're not redundant with the (included) verifier unit
tests which will test the checksums anyway.
was (Author: jjirsa):
So the same bug corrected in CASSANDRA-8778 was re-introduced by CASSANDRA-8709
, as it was developed in parallel and was likely merged/reviewed without the
benefit of knowing about #8778.
I've done the following:
1) Rebased to trunk as of 20150319
2) Removed o.a.c.io.DataIntegrityMetadata#append
3) Corrected o.a.c.io.DataIntegrityMetadata#appendDirect
4) Brought over [~benedict]'s PureJavaCRC32 's fix from above (which was
correct - 7bef6f93aea3a6897b53e909688f5948c018ccdf)
Commit:
https://github.com/jeffjirsa/cassandra/commit/79642ea4f56a33f249e807abdd562f89d20f6c36
Diff at
https://github.com/apache/cassandra/compare/trunk...jeffjirsa:cassandra-5791.diff
, I'll also attach here as cassandra-5791-20150319.diff
Passing:
{noformat}
[junit] - ---
[junit] Testsuite: org.apache.cassandra.db.VerifyTest
[junit] Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
2.803 sec
[junit]
{noformat}
[~jbellis] and [~benedict] - if you want unit tests for DataIntegrityMetadata,
Jira it, assign me, and I'll write them. I'd have done it tonight but I can't
convince myself that they're not redundant with the (included) verifier unit
tests which will test the checksums anyway.
> A nodetool command to validate all sstables in a node
> -
>
> Key: CASSANDRA-5791
> URL: https://issues.apache.org/jira/browse/CASSANDRA-5791
> Project: Cassandra
> Issue Type: New Feature
> Components: Core
>Reporter: sankalp kohli
>Assignee: Jeff Jirsa
>Priority: Minor
> Fix For: 3.0
>
> Attachments: cassandra-5791-20150319.diff,
> cassandra-5791-patch-3.diff, cassandra-5791.patch-2
>
>
> CUrrently there is no nodetool command to validate all sstables on disk. The
> only way to do this is to run a repair and see if it succeeds. But we cannot
> repair the system keyspace.
> Also we can run upgrade sstables but that re writes all the sstables.
> This command should check the hash of all sstables and return whether all
> data is readable all not. This should NOT care about consistency.
> The compressed sstables do not have hash so not sure how it will work there.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-5791) A nodetool command to validate all sstables in a node
[
https://issues.apache.org/jira/browse/CASSANDRA-5791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14370665#comment-14370665
]
Jeff Jirsa edited comment on CASSANDRA-5791 at 3/26/15 3:25 AM:
So the same bug corrected in CASSANDRA-8778 was re-introduced by CASSANDRA-8709
.
I've done the following:
1) Rebased to trunk as of 20150325
2) Removed o.a.c.io.DataIntegrityMetadata#append
3) Corrected o.a.c.io.DataIntegrityMetadata#appendDirect
4) Brought over [~benedict]'s PureJavaCRC32 's fix from above (which was
correct - 7bef6f93aea3a6897b53e909688f5948c018ccdf)
Commit:
https://github.com/jeffjirsa/cassandra/commit/daaa88878c023fbcd94d4aa7d02696a675a118dd
Diff at
https://github.com/apache/cassandra/compare/trunk...jeffjirsa:cassandra-5791.diff
,
Passing:
{noformat}
[junit] - ---
[junit] Testsuite: org.apache.cassandra.db.VerifyTest
[junit] Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
2.803 sec
[junit]
{noformat}
[~jbellis] and [~benedict] - if you want unit tests for DataIntegrityMetadata,
Jira it, assign me, and I'll write them. I'd have done it tonight but I can't
convince myself that they're not redundant with the (included) verifier unit
tests which will test the checksums anyway.
(Will continue rebasing every few days as required - latest rebase 20150325,
pull latest from github rather than using attached files)
was (Author: jjirsa):
So the same bug corrected in CASSANDRA-8778 was re-introduced by CASSANDRA-8709
.
I've done the following:
1) Rebased to trunk as of 20150319
2) Removed o.a.c.io.DataIntegrityMetadata#append
3) Corrected o.a.c.io.DataIntegrityMetadata#appendDirect
4) Brought over [~benedict]'s PureJavaCRC32 's fix from above (which was
correct - 7bef6f93aea3a6897b53e909688f5948c018ccdf)
Commit:
https://github.com/jeffjirsa/cassandra/commit/79642ea4f56a33f249e807abdd562f89d20f6c36
Diff at
https://github.com/apache/cassandra/compare/trunk...jeffjirsa:cassandra-5791.diff
, I'll also attach here as cassandra-5791-20150319.diff
Passing:
{noformat}
[junit] - ---
[junit] Testsuite: org.apache.cassandra.db.VerifyTest
[junit] Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
2.803 sec
[junit]
{noformat}
[~jbellis] and [~benedict] - if you want unit tests for DataIntegrityMetadata,
Jira it, assign me, and I'll write them. I'd have done it tonight but I can't
convince myself that they're not redundant with the (included) verifier unit
tests which will test the checksums anyway.
> A nodetool command to validate all sstables in a node
> -
>
> Key: CASSANDRA-5791
> URL: https://issues.apache.org/jira/browse/CASSANDRA-5791
> Project: Cassandra
> Issue Type: New Feature
> Components: Core
>Reporter: sankalp kohli
>Assignee: Jeff Jirsa
>Priority: Minor
> Fix For: 3.0
>
> Attachments: cassandra-5791-20150319.diff,
> cassandra-5791-patch-3.diff, cassandra-5791.patch-2
>
>
> CUrrently there is no nodetool command to validate all sstables on disk. The
> only way to do this is to run a repair and see if it succeeds. But we cannot
> repair the system keyspace.
> Also we can run upgrade sstables but that re writes all the sstables.
> This command should check the hash of all sstables and return whether all
> data is readable all not. This should NOT care about consistency.
> The compressed sstables do not have hash so not sure how it will work there.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9048) Delimited File Bulk Loader
[ https://issues.apache.org/jira/browse/CASSANDRA-9048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14382602#comment-14382602 ] Jeff Jirsa commented on CASSANDRA-9048: --- 2 cents: Agree with [~carlyeks]'s comment: seems useful, but don't see why it would go into the tree. > Delimited File Bulk Loader > -- > > Key: CASSANDRA-9048 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9048 > Project: Cassandra > Issue Type: Improvement > Components: Tools >Reporter: Brian Hess > Fix For: 3.0 > > Attachments: CASSANDRA-9048.patch > > > There is a strong need for bulk loading data from delimited files into > Cassandra. Starting with delimited files means that the data is not > currently in the SSTable format, and therefore cannot immediately leverage > Cassandra's bulk loading tool, sstableloader, directly. > A tool supporting delimited files much closer matches the format of the data > more often than the SSTable format itself, and a tool that loads from > delimited files is very useful. > In order for this bulk loader to be more generally useful to customers, it > should handle a number of options at a minimum: > - support specifying the input file or to read the data from stdin (so other > command-line programs can pipe into the loader) > - supply the CQL schema for the input data > - support all data types other than collections (collections is a stretch > goal/need) > - an option to specify the delimiter > - an option to specify comma as the decimal delimiter (for international use > casese) > - an option to specify how NULL values are specified in the file (e.g., the > empty string or the string NULL) > - an option to specify how BOOLEAN values are specified in the file (e.g., > TRUE/FALSE or 0/1) > - an option to specify the Date and Time format > - an option to skip some number of rows at the beginning of the file > - an option to only read in some number of rows from the file > - an option to indicate how many parse errors to tolerate > - an option to specify a file that will contain all the lines that did not > parse correctly (up to the maximum number of parse errors) > - an option to specify the CQL port to connect to (with 9042 as the default). > Additional options would be useful, but this set of options/features is a > start. > A word on COPY. COPY comes via CQLSH which requires the client to be the > same version as the server (e.g., 2.0 CQLSH does not work with 2.1 Cassandra, > etc). This tool should be able to connect to any version of Cassandra > (within reason). For example, it should be able to handle 2.0.x and 2.1.x. > Moreover, CQLSH's COPY command does not support a number of the options > above. Lastly, the performance of COPY in 2.0.x is not high enough to be > considered a bulk ingest tool. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (CASSANDRA-9076) Consistency: make default max_hint_window_in_ms actually 3 hours
Jeff Jirsa created CASSANDRA-9076: - Summary: Consistency: make default max_hint_window_in_ms actually 3 hours Key: CASSANDRA-9076 URL: https://issues.apache.org/jira/browse/CASSANDRA-9076 Project: Cassandra Issue Type: Improvement Reporter: Jeff Jirsa Assignee: Jeff Jirsa Priority: Trivial Attachments: max_hint_window.diff CASSANDRA-4632 changed max_hint_window_in_ms from 1 hour to 3 hours, however, it only changed the default in cassandra.yaml - it left the actual default in Config.java at 1 hour. This issue is to update Config.java for consistency, in case users comment out the actual entry in cassandra.yaml and revert to the "real" default of 1 hour. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-9076) Consistency: make default max_hint_window_in_ms actually 3 hours
[ https://issues.apache.org/jira/browse/CASSANDRA-9076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Jirsa updated CASSANDRA-9076: -- Description: >From IRC/#cassandra: CASSANDRA-4632 changed max_hint_window_in_ms from 1 hour to 3 hours, however, it only changed the default in cassandra.yaml - it left the actual default in Config.java at 1 hour. This issue is to update Config.java for consistency, in case users comment out the actual entry in cassandra.yaml and revert to the "real" default of 1 hour. was: CASSANDRA-4632 changed max_hint_window_in_ms from 1 hour to 3 hours, however, it only changed the default in cassandra.yaml - it left the actual default in Config.java at 1 hour. This issue is to update Config.java for consistency, in case users comment out the actual entry in cassandra.yaml and revert to the "real" default of 1 hour. > Consistency: make default max_hint_window_in_ms actually 3 hours > > > Key: CASSANDRA-9076 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9076 > Project: Cassandra > Issue Type: Improvement >Reporter: Jeff Jirsa >Assignee: Jeff Jirsa >Priority: Trivial > Attachments: max_hint_window.diff > > > From IRC/#cassandra: > CASSANDRA-4632 changed max_hint_window_in_ms from 1 hour to 3 hours, however, > it only changed the default in cassandra.yaml - it left the actual default in > Config.java at 1 hour. > This issue is to update Config.java for consistency, in case users comment > out the actual entry in cassandra.yaml and revert to the "real" default of 1 > hour. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9077) NPE
[
https://issues.apache.org/jira/browse/CASSANDRA-9077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14387992#comment-14387992
]
Jeff Jirsa commented on CASSANDRA-9077:
---
Reproduced in trunk, I'll see if I can track it down.
> NPE
> ---
>
> Key: CASSANDRA-9077
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9077
> Project: Cassandra
> Issue Type: Bug
> Components: Core
>Reporter: dan jatnieks
>Priority: Minor
> Attachments: 9077-stacktrace.log
>
>
> I am seeing an NPE on the latest 2.1 branch with this sequence of deletes
> from a list - first delete the entire list, then attempt to delete one
> element.
> I expected to see {{List index 0 out of bound, list has size 0}} but instead
> got an NPE.
> {noformat}
> ./bin/cqlsh
> Connected to Test Cluster at 127.0.0.1:9042.
> [cqlsh 5.0.1 | Cassandra 2.1.3-SNAPSHOT | CQL spec 3.2.0 | Native protocol v3]
> Use HELP for help.
> cqlsh> use frozen_collections ;
> cqlsh:frozen_collections> DROP TABLE IF EXISTS t;
> cqlsh:frozen_collections> CREATE TABLE t (id text PRIMARY KEY, l list,
> s set);
> cqlsh:frozen_collections> INSERT INTO t (id, l, s) VALUES ('user', ['1'],
> {'1'});
> cqlsh:frozen_collections>
> cqlsh:frozen_collections> DELETE l FROM t WHERE id ='user';
> cqlsh:frozen_collections> //INSERT INTO t (id, l) VALUES ('user', ['1']);
> cqlsh:frozen_collections> DELETE l[0] FROM t WHERE id = 'user';
> ServerError: message="java.lang.NullPointerException">
> cqlsh:frozen_collections>
> cqlsh:frozen_collections> DELETE s FROM t WHERE id ='user';
> cqlsh:frozen_collections> DELETE s['1'] FROM t WHERE id = 'user';
> {noformat}
> It appears the {{DELETE emails...}} directly followed by {{DELETE
> emails[0]...}} is the offending sequence. Either one alone works fine, as
> does adding an intervening insert/update.
> The same sequence performed on a Set rather than List works (as shown above).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9077) NPE
[
https://issues.apache.org/jira/browse/CASSANDRA-9077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14388054#comment-14388054
]
Jeff Jirsa commented on CASSANDRA-9077:
---
The underlying problem is that
o.a.c.db.composites.AbstractCellNameType.CQL3RowOfSparse getMultiCellColumn()
can return null (if collection==null), which happens if/when you delete the
list.
A very similar NPE would be thrown if you tried to update to set a list element
by index once the list was deleted:
{noformat}
cqlsh:test2> update t set l[0] = '1' where id='user';
ServerError:
{noformat}
{noformat}
ERROR [SharedPool-Worker-1] 2015-03-30 22:22:32,682 QueryMessage.java:132 -
Unexpected error during query
java.lang.NullPointerException: null
at
org.apache.cassandra.cql3.Lists$SetterByIndex.execute(Lists.java:345)
~[main/:na]
at
org.apache.cassandra.cql3.statements.UpdateStatement.addUpdateForKey(UpdateStatement.java:111)
~[main/:na]
at
org.apache.cassandra.cql3.statements.UpdateStatement.addUpdateForKey(UpdateStatement.java:58)
~[main/:na]
at
org.apache.cassandra.cql3.statements.ModificationStatement.getMutations(ModificationStatement.java:643)
~[main/:na]
at
org.apache.cassandra.cql3.statements.ModificationStatement.executeWithoutCondition(ModificationStatement.java:475)
~[main/:na]
at
org.apache.cassandra.cql3.statements.ModificationStatement.execute(ModificationStatement.java:461)
~[main/:na]
at
org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:234)
~[main/:na]
at
org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:261)
~[main/:na]
at
org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:246)
~[main/:na]
at
org.apache.cassandra.transport.messages.QueryMessage.execute(QueryMessage.java:119)
~[main/:na]
at
org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:475)
~[main/:na]
at
org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:371)
~[main/:na]
at
io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
~[netty-all-4.0.23.Final.jar:4.0.23.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
~[netty-all-4.0.23.Final.jar:4.0.23.Final]
at
io.netty.channel.AbstractChannelHandlerContext.access$700(AbstractChannelHandlerContext.java:32)
~[netty-all-4.0.23.Final.jar:4.0.23.Final]
at
io.netty.channel.AbstractChannelHandlerContext$8.run(AbstractChannelHandlerContext.java:324)
~[netty-all-4.0.23.Final.jar:4.0.23.Final]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
~[na:1.7.0_25]
at
org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164)
~[main/:na]
at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105)
~[main/:na]
at java.lang.Thread.run(Thread.java:724) ~[na:1.7.0_25]
ERROR [SharedPool-Worker-1] 2015-03-30 22:22:32,683 ErrorMessage.java:329 -
Unexpected exception during request
java.lang.NullPointerException: null
at
org.apache.cassandra.cql3.Lists$SetterByIndex.execute(Lists.java:345)
~[main/:na]
at
org.apache.cassandra.cql3.statements.UpdateStatement.addUpdateForKey(UpdateStatement.java:111)
~[main/:na]
at
org.apache.cassandra.cql3.statements.UpdateStatement.addUpdateForKey(UpdateStatement.java:58)
~[main/:na]
at
org.apache.cassandra.cql3.statements.ModificationStatement.getMutations(ModificationStatement.java:643)
~[main/:na]
at
org.apache.cassandra.cql3.statements.ModificationStatement.executeWithoutCondition(ModificationStatement.java:475)
~[main/:na]
at
org.apache.cassandra.cql3.statements.ModificationStatement.execute(ModificationStatement.java:461)
~[main/:na]
at
org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:234)
~[main/:na]
at
org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:261)
~[main/:na]
at
org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:246)
~[main/:na]
at
org.apache.cassandra.transport.messages.QueryMessage.execute(QueryMessage.java:119)
~[main/:na]
at
org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:475)
~[main/:na]
at
org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:371)
~[main/:na]
at
io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
~[netty-all-4.0.23.Final.jar:4.0.23.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
~[netty-all-4.0.23.Final.jar:4
[jira] [Comment Edited] (CASSANDRA-9077) NPE
[
https://issues.apache.org/jira/browse/CASSANDRA-9077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14388054#comment-14388054
]
Jeff Jirsa edited comment on CASSANDRA-9077 at 3/31/15 5:47 AM:
The underlying problem is that
o.a.c.db.composites.AbstractCellNameType.CQL3RowOfSparse getMultiCellColumn()
can return null (if collection==null), which happens if/when you delete the
list.
A very similar NPE would be thrown if you tried to update to set a list element
by index once the list was deleted:
{noformat}
cqlsh:test2> update t set l[0] = '1' where id='user';
ServerError:
{noformat}
{noformat}
ERROR [SharedPool-Worker-1] 2015-03-30 22:22:32,682 QueryMessage.java:132 -
Unexpected error during query
java.lang.NullPointerException: null
at
org.apache.cassandra.cql3.Lists$SetterByIndex.execute(Lists.java:345)
~[main/:na]
at
org.apache.cassandra.cql3.statements.UpdateStatement.addUpdateForKey(UpdateStatement.java:111)
~[main/:na]
at
org.apache.cassandra.cql3.statements.UpdateStatement.addUpdateForKey(UpdateStatement.java:58)
~[main/:na]
at
org.apache.cassandra.cql3.statements.ModificationStatement.getMutations(ModificationStatement.java:643)
~[main/:na]
at
org.apache.cassandra.cql3.statements.ModificationStatement.executeWithoutCondition(ModificationStatement.java:475)
~[main/:na]
at
org.apache.cassandra.cql3.statements.ModificationStatement.execute(ModificationStatement.java:461)
~[main/:na]
at
org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:234)
~[main/:na]
at
org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:261)
~[main/:na]
at
org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:246)
~[main/:na]
at
org.apache.cassandra.transport.messages.QueryMessage.execute(QueryMessage.java:119)
~[main/:na]
at
org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:475)
~[main/:na]
at
org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:371)
~[main/:na]
at
io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
~[netty-all-4.0.23.Final.jar:4.0.23.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
~[netty-all-4.0.23.Final.jar:4.0.23.Final]
at
io.netty.channel.AbstractChannelHandlerContext.access$700(AbstractChannelHandlerContext.java:32)
~[netty-all-4.0.23.Final.jar:4.0.23.Final]
at
io.netty.channel.AbstractChannelHandlerContext$8.run(AbstractChannelHandlerContext.java:324)
~[netty-all-4.0.23.Final.jar:4.0.23.Final]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
~[na:1.7.0_25]
at
org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164)
~[main/:na]
at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105)
~[main/:na]
at java.lang.Thread.run(Thread.java:724) ~[na:1.7.0_25]
ERROR [SharedPool-Worker-1] 2015-03-30 22:22:32,683 ErrorMessage.java:329 -
Unexpected exception during request
java.lang.NullPointerException: null
at
org.apache.cassandra.cql3.Lists$SetterByIndex.execute(Lists.java:345)
~[main/:na]
at
org.apache.cassandra.cql3.statements.UpdateStatement.addUpdateForKey(UpdateStatement.java:111)
~[main/:na]
at
org.apache.cassandra.cql3.statements.UpdateStatement.addUpdateForKey(UpdateStatement.java:58)
~[main/:na]
at
org.apache.cassandra.cql3.statements.ModificationStatement.getMutations(ModificationStatement.java:643)
~[main/:na]
at
org.apache.cassandra.cql3.statements.ModificationStatement.executeWithoutCondition(ModificationStatement.java:475)
~[main/:na]
at
org.apache.cassandra.cql3.statements.ModificationStatement.execute(ModificationStatement.java:461)
~[main/:na]
at
org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:234)
~[main/:na]
at
org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:261)
~[main/:na]
at
org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:246)
~[main/:na]
at
org.apache.cassandra.transport.messages.QueryMessage.execute(QueryMessage.java:119)
~[main/:na]
at
org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:475)
~[main/:na]
at
org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:371)
~[main/:na]
at
io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
~[netty-all-4.0.23.Final.jar:4.0.23.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandler
[jira] [Assigned] (CASSANDRA-9077) NPE
[
https://issues.apache.org/jira/browse/CASSANDRA-9077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jeff Jirsa reassigned CASSANDRA-9077:
-
Assignee: Jeff Jirsa
> NPE
> ---
>
> Key: CASSANDRA-9077
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9077
> Project: Cassandra
> Issue Type: Bug
> Components: Core
>Reporter: dan jatnieks
>Assignee: Jeff Jirsa
>Priority: Minor
> Attachments: 9077-stacktrace.log, cassandra-9077-option1.txt,
> cassandra-9077-option2.txt
>
>
> I am seeing an NPE on the latest 2.1 branch with this sequence of deletes
> from a list - first delete the entire list, then attempt to delete one
> element.
> I expected to see {{List index 0 out of bound, list has size 0}} but instead
> got an NPE.
> {noformat}
> ./bin/cqlsh
> Connected to Test Cluster at 127.0.0.1:9042.
> [cqlsh 5.0.1 | Cassandra 2.1.3-SNAPSHOT | CQL spec 3.2.0 | Native protocol v3]
> Use HELP for help.
> cqlsh> use frozen_collections ;
> cqlsh:frozen_collections> DROP TABLE IF EXISTS t;
> cqlsh:frozen_collections> CREATE TABLE t (id text PRIMARY KEY, l list,
> s set);
> cqlsh:frozen_collections> INSERT INTO t (id, l, s) VALUES ('user', ['1'],
> {'1'});
> cqlsh:frozen_collections>
> cqlsh:frozen_collections> DELETE l FROM t WHERE id ='user';
> cqlsh:frozen_collections> //INSERT INTO t (id, l) VALUES ('user', ['1']);
> cqlsh:frozen_collections> DELETE l[0] FROM t WHERE id = 'user';
> ServerError: message="java.lang.NullPointerException">
> cqlsh:frozen_collections>
> cqlsh:frozen_collections> DELETE s FROM t WHERE id ='user';
> cqlsh:frozen_collections> DELETE s['1'] FROM t WHERE id = 'user';
> {noformat}
> It appears the {{DELETE emails...}} directly followed by {{DELETE
> emails[0]...}} is the offending sequence. Either one alone works fine, as
> does adding an intervening insert/update.
> The same sequence performed on a Set rather than List works (as shown above).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-9077) NPE
[
https://issues.apache.org/jira/browse/CASSANDRA-9077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jeff Jirsa updated CASSANDRA-9077:
--
Attachment: cassandra-9077-option1.txt
> NPE
> ---
>
> Key: CASSANDRA-9077
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9077
> Project: Cassandra
> Issue Type: Bug
> Components: Core
>Reporter: dan jatnieks
>Priority: Minor
> Attachments: 9077-stacktrace.log, cassandra-9077-option1.txt,
> cassandra-9077-option2.txt
>
>
> I am seeing an NPE on the latest 2.1 branch with this sequence of deletes
> from a list - first delete the entire list, then attempt to delete one
> element.
> I expected to see {{List index 0 out of bound, list has size 0}} but instead
> got an NPE.
> {noformat}
> ./bin/cqlsh
> Connected to Test Cluster at 127.0.0.1:9042.
> [cqlsh 5.0.1 | Cassandra 2.1.3-SNAPSHOT | CQL spec 3.2.0 | Native protocol v3]
> Use HELP for help.
> cqlsh> use frozen_collections ;
> cqlsh:frozen_collections> DROP TABLE IF EXISTS t;
> cqlsh:frozen_collections> CREATE TABLE t (id text PRIMARY KEY, l list,
> s set);
> cqlsh:frozen_collections> INSERT INTO t (id, l, s) VALUES ('user', ['1'],
> {'1'});
> cqlsh:frozen_collections>
> cqlsh:frozen_collections> DELETE l FROM t WHERE id ='user';
> cqlsh:frozen_collections> //INSERT INTO t (id, l) VALUES ('user', ['1']);
> cqlsh:frozen_collections> DELETE l[0] FROM t WHERE id = 'user';
> ServerError: message="java.lang.NullPointerException">
> cqlsh:frozen_collections>
> cqlsh:frozen_collections> DELETE s FROM t WHERE id ='user';
> cqlsh:frozen_collections> DELETE s['1'] FROM t WHERE id = 'user';
> {noformat}
> It appears the {{DELETE emails...}} directly followed by {{DELETE
> emails[0]...}} is the offending sequence. Either one alone works fine, as
> does adding an intervening insert/update.
> The same sequence performed on a Set rather than List works (as shown above).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-9077) NPE
[
https://issues.apache.org/jira/browse/CASSANDRA-9077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jeff Jirsa updated CASSANDRA-9077:
--
Attachment: cassandra-9077-option2.txt
> NPE
> ---
>
> Key: CASSANDRA-9077
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9077
> Project: Cassandra
> Issue Type: Bug
> Components: Core
>Reporter: dan jatnieks
>Priority: Minor
> Attachments: 9077-stacktrace.log, cassandra-9077-option1.txt,
> cassandra-9077-option2.txt
>
>
> I am seeing an NPE on the latest 2.1 branch with this sequence of deletes
> from a list - first delete the entire list, then attempt to delete one
> element.
> I expected to see {{List index 0 out of bound, list has size 0}} but instead
> got an NPE.
> {noformat}
> ./bin/cqlsh
> Connected to Test Cluster at 127.0.0.1:9042.
> [cqlsh 5.0.1 | Cassandra 2.1.3-SNAPSHOT | CQL spec 3.2.0 | Native protocol v3]
> Use HELP for help.
> cqlsh> use frozen_collections ;
> cqlsh:frozen_collections> DROP TABLE IF EXISTS t;
> cqlsh:frozen_collections> CREATE TABLE t (id text PRIMARY KEY, l list,
> s set);
> cqlsh:frozen_collections> INSERT INTO t (id, l, s) VALUES ('user', ['1'],
> {'1'});
> cqlsh:frozen_collections>
> cqlsh:frozen_collections> DELETE l FROM t WHERE id ='user';
> cqlsh:frozen_collections> //INSERT INTO t (id, l) VALUES ('user', ['1']);
> cqlsh:frozen_collections> DELETE l[0] FROM t WHERE id = 'user';
> ServerError: message="java.lang.NullPointerException">
> cqlsh:frozen_collections>
> cqlsh:frozen_collections> DELETE s FROM t WHERE id ='user';
> cqlsh:frozen_collections> DELETE s['1'] FROM t WHERE id = 'user';
> {noformat}
> It appears the {{DELETE emails...}} directly followed by {{DELETE
> emails[0]...}} is the offending sequence. Either one alone works fine, as
> does adding an intervening insert/update.
> The same sequence performed on a Set rather than List works (as shown above).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Issue Comment Deleted] (CASSANDRA-9077) NPE
[
https://issues.apache.org/jira/browse/CASSANDRA-9077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jeff Jirsa updated CASSANDRA-9077:
--
Comment: was deleted
(was: Reproduced in trunk, I'll see if I can track it down.
)
> NPE
> ---
>
> Key: CASSANDRA-9077
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9077
> Project: Cassandra
> Issue Type: Bug
> Components: Core
>Reporter: dan jatnieks
>Assignee: Jeff Jirsa
>Priority: Minor
> Attachments: 9077-stacktrace.log, cassandra-9077-option1.txt,
> cassandra-9077-option2.txt
>
>
> I am seeing an NPE on the latest 2.1 branch with this sequence of deletes
> from a list - first delete the entire list, then attempt to delete one
> element.
> I expected to see {{List index 0 out of bound, list has size 0}} but instead
> got an NPE.
> {noformat}
> ./bin/cqlsh
> Connected to Test Cluster at 127.0.0.1:9042.
> [cqlsh 5.0.1 | Cassandra 2.1.3-SNAPSHOT | CQL spec 3.2.0 | Native protocol v3]
> Use HELP for help.
> cqlsh> use frozen_collections ;
> cqlsh:frozen_collections> DROP TABLE IF EXISTS t;
> cqlsh:frozen_collections> CREATE TABLE t (id text PRIMARY KEY, l list,
> s set);
> cqlsh:frozen_collections> INSERT INTO t (id, l, s) VALUES ('user', ['1'],
> {'1'});
> cqlsh:frozen_collections>
> cqlsh:frozen_collections> DELETE l FROM t WHERE id ='user';
> cqlsh:frozen_collections> //INSERT INTO t (id, l) VALUES ('user', ['1']);
> cqlsh:frozen_collections> DELETE l[0] FROM t WHERE id = 'user';
> ServerError: message="java.lang.NullPointerException">
> cqlsh:frozen_collections>
> cqlsh:frozen_collections> DELETE s FROM t WHERE id ='user';
> cqlsh:frozen_collections> DELETE s['1'] FROM t WHERE id = 'user';
> {noformat}
> It appears the {{DELETE emails...}} directly followed by {{DELETE
> emails[0]...}} is the offending sequence. Either one alone works fine, as
> does adding an intervening insert/update.
> The same sequence performed on a Set rather than List works (as shown above).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-9077) NPE
[
https://issues.apache.org/jira/browse/CASSANDRA-9077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jeff Jirsa updated CASSANDRA-9077:
--
Attachment: cassandra-9077-option2.txt
> NPE
> ---
>
> Key: CASSANDRA-9077
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9077
> Project: Cassandra
> Issue Type: Bug
> Components: Core
>Reporter: dan jatnieks
>Assignee: Jeff Jirsa
>Priority: Minor
> Attachments: 9077-stacktrace.log, cassandra-9077-option1.txt,
> cassandra-9077-option2.txt
>
>
> I am seeing an NPE on the latest 2.1 branch with this sequence of deletes
> from a list - first delete the entire list, then attempt to delete one
> element.
> I expected to see {{List index 0 out of bound, list has size 0}} but instead
> got an NPE.
> {noformat}
> ./bin/cqlsh
> Connected to Test Cluster at 127.0.0.1:9042.
> [cqlsh 5.0.1 | Cassandra 2.1.3-SNAPSHOT | CQL spec 3.2.0 | Native protocol v3]
> Use HELP for help.
> cqlsh> use frozen_collections ;
> cqlsh:frozen_collections> DROP TABLE IF EXISTS t;
> cqlsh:frozen_collections> CREATE TABLE t (id text PRIMARY KEY, l list,
> s set);
> cqlsh:frozen_collections> INSERT INTO t (id, l, s) VALUES ('user', ['1'],
> {'1'});
> cqlsh:frozen_collections>
> cqlsh:frozen_collections> DELETE l FROM t WHERE id ='user';
> cqlsh:frozen_collections> //INSERT INTO t (id, l) VALUES ('user', ['1']);
> cqlsh:frozen_collections> DELETE l[0] FROM t WHERE id = 'user';
> ServerError: message="java.lang.NullPointerException">
> cqlsh:frozen_collections>
> cqlsh:frozen_collections> DELETE s FROM t WHERE id ='user';
> cqlsh:frozen_collections> DELETE s['1'] FROM t WHERE id = 'user';
> {noformat}
> It appears the {{DELETE emails...}} directly followed by {{DELETE
> emails[0]...}} is the offending sequence. Either one alone works fine, as
> does adding an intervening insert/update.
> The same sequence performed on a Set rather than List works (as shown above).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-9077) NPE
[
https://issues.apache.org/jira/browse/CASSANDRA-9077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jeff Jirsa updated CASSANDRA-9077:
--
Attachment: (was: cassandra-9077-option2.txt)
> NPE
> ---
>
> Key: CASSANDRA-9077
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9077
> Project: Cassandra
> Issue Type: Bug
> Components: Core
>Reporter: dan jatnieks
>Assignee: Jeff Jirsa
>Priority: Minor
> Attachments: 9077-stacktrace.log, cassandra-9077-option1.txt,
> cassandra-9077-option2.txt
>
>
> I am seeing an NPE on the latest 2.1 branch with this sequence of deletes
> from a list - first delete the entire list, then attempt to delete one
> element.
> I expected to see {{List index 0 out of bound, list has size 0}} but instead
> got an NPE.
> {noformat}
> ./bin/cqlsh
> Connected to Test Cluster at 127.0.0.1:9042.
> [cqlsh 5.0.1 | Cassandra 2.1.3-SNAPSHOT | CQL spec 3.2.0 | Native protocol v3]
> Use HELP for help.
> cqlsh> use frozen_collections ;
> cqlsh:frozen_collections> DROP TABLE IF EXISTS t;
> cqlsh:frozen_collections> CREATE TABLE t (id text PRIMARY KEY, l list,
> s set);
> cqlsh:frozen_collections> INSERT INTO t (id, l, s) VALUES ('user', ['1'],
> {'1'});
> cqlsh:frozen_collections>
> cqlsh:frozen_collections> DELETE l FROM t WHERE id ='user';
> cqlsh:frozen_collections> //INSERT INTO t (id, l) VALUES ('user', ['1']);
> cqlsh:frozen_collections> DELETE l[0] FROM t WHERE id = 'user';
> ServerError: message="java.lang.NullPointerException">
> cqlsh:frozen_collections>
> cqlsh:frozen_collections> DELETE s FROM t WHERE id ='user';
> cqlsh:frozen_collections> DELETE s['1'] FROM t WHERE id = 'user';
> {noformat}
> It appears the {{DELETE emails...}} directly followed by {{DELETE
> emails[0]...}} is the offending sequence. Either one alone works fine, as
> does adding an intervening insert/update.
> The same sequence performed on a Set rather than List works (as shown above).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9076) Consistency: make default max_hint_window_in_ms actually 3 hours
[ https://issues.apache.org/jira/browse/CASSANDRA-9076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14388679#comment-14388679 ] Jeff Jirsa commented on CASSANDRA-9076: --- The change itself is trivial and should be safe. At least 3.0 and 2.1. The issue was mentioned when it bit a 2.0 user. I know 2.0 is fairly late in the game at this point, but this is also a pretty non-invasive change - I'd personally like to see it in 2.0, but I'll live either way, since there's an easy cassandra.yaml workaround for getting 3hrs instead of 1hr. Up to Aleksey. > Consistency: make default max_hint_window_in_ms actually 3 hours > > > Key: CASSANDRA-9076 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9076 > Project: Cassandra > Issue Type: Improvement >Reporter: Jeff Jirsa >Assignee: Jeff Jirsa >Priority: Trivial > Fix For: 3.0, 2.1.4 > > Attachments: max_hint_window.diff > > > From IRC/#cassandra: > CASSANDRA-4632 changed max_hint_window_in_ms from 1 hour to 3 hours, however, > it only changed the default in cassandra.yaml - it left the actual default in > Config.java at 1 hour. > This issue is to update Config.java for consistency, in case users comment > out the actual entry in cassandra.yaml and revert to the "real" default of 1 > hour. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-9077) Deleting an element from a List which is null throws a NPE
[
https://issues.apache.org/jira/browse/CASSANDRA-9077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jeff Jirsa updated CASSANDRA-9077:
--
Attachment: cassandra-9077-option1-v2.txt
option1-v2 addresses all 3 requests:
- Changed wording to be consistent
- Handles NPE in Discarder (removing element from null list)
- Adds unit tests for all 3 cases (Discarder, DiscarderByIndex, SetterByIndex)
> Deleting an element from a List which is null throws a NPE
> --
>
> Key: CASSANDRA-9077
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9077
> Project: Cassandra
> Issue Type: Bug
> Components: Core
>Reporter: dan jatnieks
>Assignee: Jeff Jirsa
>Priority: Minor
> Attachments: 9077-stacktrace.log, cassandra-9077-option1-v2.txt,
> cassandra-9077-option1.txt, cassandra-9077-option2.txt
>
>
> I am seeing an NPE on the latest 2.1 branch with this sequence of deletes
> from a list - first delete the entire list, then attempt to delete one
> element.
> I expected to see {{List index 0 out of bound, list has size 0}} but instead
> got an NPE.
> {noformat}
> ./bin/cqlsh
> Connected to Test Cluster at 127.0.0.1:9042.
> [cqlsh 5.0.1 | Cassandra 2.1.3-SNAPSHOT | CQL spec 3.2.0 | Native protocol v3]
> Use HELP for help.
> cqlsh> use frozen_collections ;
> cqlsh:frozen_collections> DROP TABLE IF EXISTS t;
> cqlsh:frozen_collections> CREATE TABLE t (id text PRIMARY KEY, l list,
> s set);
> cqlsh:frozen_collections> INSERT INTO t (id, l, s) VALUES ('user', ['1'],
> {'1'});
> cqlsh:frozen_collections>
> cqlsh:frozen_collections> DELETE l FROM t WHERE id ='user';
> cqlsh:frozen_collections> //INSERT INTO t (id, l) VALUES ('user', ['1']);
> cqlsh:frozen_collections> DELETE l[0] FROM t WHERE id = 'user';
> ServerError: message="java.lang.NullPointerException">
> cqlsh:frozen_collections>
> cqlsh:frozen_collections> DELETE s FROM t WHERE id ='user';
> cqlsh:frozen_collections> DELETE s['1'] FROM t WHERE id = 'user';
> {noformat}
> It appears the {{DELETE emails...}} directly followed by {{DELETE
> emails[0]...}} is the offending sequence. Either one alone works fine, as
> does adding an intervening insert/update.
> The same sequence performed on a Set rather than List works (as shown above).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-9077) Deleting an element from a List which is null throws a NPE
[
https://issues.apache.org/jira/browse/CASSANDRA-9077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14389255#comment-14389255
]
Jeff Jirsa edited comment on CASSANDRA-9077 at 3/31/15 8:36 PM:
option1-v2 addresses all 3 requests:
- Changed wording to be consistent
- Handles NPE in Discarder (removing element from null list)
- Adds unit tests for all 3 cases (Discarder, DiscarderByIndex, SetterByIndex)
Diff was based on trunk, applies cleanly to 2.1 as well.
was (Author: jjirsa):
option1-v2 addresses all 3 requests:
- Changed wording to be consistent
- Handles NPE in Discarder (removing element from null list)
- Adds unit tests for all 3 cases (Discarder, DiscarderByIndex, SetterByIndex)
> Deleting an element from a List which is null throws a NPE
> --
>
> Key: CASSANDRA-9077
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9077
> Project: Cassandra
> Issue Type: Bug
> Components: Core
>Reporter: dan jatnieks
>Assignee: Jeff Jirsa
>Priority: Minor
> Attachments: 9077-stacktrace.log, cassandra-9077-option1-v2.txt,
> cassandra-9077-option1.txt, cassandra-9077-option2.txt
>
>
> I am seeing an NPE on the latest 2.1 branch with this sequence of deletes
> from a list - first delete the entire list, then attempt to delete one
> element.
> I expected to see {{List index 0 out of bound, list has size 0}} but instead
> got an NPE.
> {noformat}
> ./bin/cqlsh
> Connected to Test Cluster at 127.0.0.1:9042.
> [cqlsh 5.0.1 | Cassandra 2.1.3-SNAPSHOT | CQL spec 3.2.0 | Native protocol v3]
> Use HELP for help.
> cqlsh> use frozen_collections ;
> cqlsh:frozen_collections> DROP TABLE IF EXISTS t;
> cqlsh:frozen_collections> CREATE TABLE t (id text PRIMARY KEY, l list,
> s set);
> cqlsh:frozen_collections> INSERT INTO t (id, l, s) VALUES ('user', ['1'],
> {'1'});
> cqlsh:frozen_collections>
> cqlsh:frozen_collections> DELETE l FROM t WHERE id ='user';
> cqlsh:frozen_collections> //INSERT INTO t (id, l) VALUES ('user', ['1']);
> cqlsh:frozen_collections> DELETE l[0] FROM t WHERE id = 'user';
> ServerError: message="java.lang.NullPointerException">
> cqlsh:frozen_collections>
> cqlsh:frozen_collections> DELETE s FROM t WHERE id ='user';
> cqlsh:frozen_collections> DELETE s['1'] FROM t WHERE id = 'user';
> {noformat}
> It appears the {{DELETE emails...}} directly followed by {{DELETE
> emails[0]...}} is the offending sequence. Either one alone works fine, as
> does adding an intervening insert/update.
> The same sequence performed on a Set rather than List works (as shown above).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9077) Deleting an element from a List which is null throws a NPE
[
https://issues.apache.org/jira/browse/CASSANDRA-9077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14391002#comment-14391002
]
Jeff Jirsa commented on CASSANDRA-9077:
---
Much better.
> Deleting an element from a List which is null throws a NPE
> --
>
> Key: CASSANDRA-9077
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9077
> Project: Cassandra
> Issue Type: Bug
> Components: Core
>Reporter: dan jatnieks
>Assignee: Jeff Jirsa
>Priority: Minor
> Fix For: 2.1.5
>
> Attachments: 9077-stacktrace.log, CASSANDRA-9077-v3.txt,
> cassandra-9077-option1-v2.txt, cassandra-9077-option1.txt,
> cassandra-9077-option2.txt
>
>
> I am seeing an NPE on the latest 2.1 branch with this sequence of deletes
> from a list - first delete the entire list, then attempt to delete one
> element.
> I expected to see {{List index 0 out of bound, list has size 0}} but instead
> got an NPE.
> {noformat}
> ./bin/cqlsh
> Connected to Test Cluster at 127.0.0.1:9042.
> [cqlsh 5.0.1 | Cassandra 2.1.3-SNAPSHOT | CQL spec 3.2.0 | Native protocol v3]
> Use HELP for help.
> cqlsh> use frozen_collections ;
> cqlsh:frozen_collections> DROP TABLE IF EXISTS t;
> cqlsh:frozen_collections> CREATE TABLE t (id text PRIMARY KEY, l list,
> s set);
> cqlsh:frozen_collections> INSERT INTO t (id, l, s) VALUES ('user', ['1'],
> {'1'});
> cqlsh:frozen_collections>
> cqlsh:frozen_collections> DELETE l FROM t WHERE id ='user';
> cqlsh:frozen_collections> //INSERT INTO t (id, l) VALUES ('user', ['1']);
> cqlsh:frozen_collections> DELETE l[0] FROM t WHERE id = 'user';
> ServerError: message="java.lang.NullPointerException">
> cqlsh:frozen_collections>
> cqlsh:frozen_collections> DELETE s FROM t WHERE id ='user';
> cqlsh:frozen_collections> DELETE s['1'] FROM t WHERE id = 'user';
> {noformat}
> It appears the {{DELETE emails...}} directly followed by {{DELETE
> emails[0]...}} is the offending sequence. Either one alone works fine, as
> does adding an intervening insert/update.
> The same sequence performed on a Set rather than List works (as shown above).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-5791) A nodetool command to validate all sstables in a node
[ https://issues.apache.org/jira/browse/CASSANDRA-5791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14391712#comment-14391712 ] Jeff Jirsa commented on CASSANDRA-5791: --- Sylvain asked the same thing in IRC this morning - the intent was to mimic the scrub read path, which would allow the SSTableIdentityIterator to discover compression checksum exceptions while decompressing, though - as Sylvain pointed out to me - it's probably insufficient for large partitions. I'll need investigate to determine whether or not it will actually throw on blocks beyond the start of the partition. > A nodetool command to validate all sstables in a node > - > > Key: CASSANDRA-5791 > URL: https://issues.apache.org/jira/browse/CASSANDRA-5791 > Project: Cassandra > Issue Type: New Feature > Components: Core >Reporter: sankalp kohli >Assignee: Jeff Jirsa >Priority: Minor > Fix For: 3.0 > > Attachments: cassandra-5791-20150319.diff, > cassandra-5791-patch-3.diff, cassandra-5791.patch-2 > > > CUrrently there is no nodetool command to validate all sstables on disk. The > only way to do this is to run a repair and see if it succeeds. But we cannot > repair the system keyspace. > Also we can run upgrade sstables but that re writes all the sstables. > This command should check the hash of all sstables and return whether all > data is readable all not. This should NOT care about consistency. > The compressed sstables do not have hash so not sure how it will work there. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (CASSANDRA-6412) Custom creation and merge functions for user-defined column types
[ https://issues.apache.org/jira/browse/CASSANDRA-6412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Jirsa reassigned CASSANDRA-6412: - Assignee: Jeff Jirsa > Custom creation and merge functions for user-defined column types > - > > Key: CASSANDRA-6412 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6412 > Project: Cassandra > Issue Type: New Feature > Components: Core >Reporter: Nicolas Favre-Felix >Assignee: Jeff Jirsa > > This is a proposal for a new feature, mapping custom types to Cassandra > columns. > These types would provide a creation function and a merge function, to be > implemented in Java by the user. > This feature relates to the concept of CRDTs; the proposal is to replicate > "operations" on these types during write, to apply these operations > internally during merge (Column.reconcile), and to also merge their values on > read. > The following operations are made possible without reading back any data: > * MIN or MAX(value) for a column > * First value for a column > * Count Distinct > * HyperLogLog > * Count-Min > And any composition of these too, e.g. a Candlestick type includes first, > last, min, and max. > The merge operations exposed by these types need to be commutative; this is > the case for many functions used in analytics. > This feature is incomplete without some integration with CASSANDRA-4775 > (Counters 2.0) which provides a Read-Modify-Write implementation for > distributed counters. Integrating custom creation and merge functions with > new counters would let users implement complex CRDTs in Cassandra, including: > * Averages & related (sum of squares, standard deviation) > * Graphs > * Sets > * Custom registers (even with vector clocks) > I have a working prototype with implementations for min, max, and Candlestick > at https://github.com/acunu/cassandra/tree/crdts - I'd appreciate any > feedback on the design and interfaces. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-6412) Custom creation and merge functions for user-defined column types
[ https://issues.apache.org/jira/browse/CASSANDRA-6412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Jirsa updated CASSANDRA-6412: -- Assignee: (was: Jeff Jirsa) > Custom creation and merge functions for user-defined column types > - > > Key: CASSANDRA-6412 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6412 > Project: Cassandra > Issue Type: New Feature > Components: Core >Reporter: Nicolas Favre-Felix > > This is a proposal for a new feature, mapping custom types to Cassandra > columns. > These types would provide a creation function and a merge function, to be > implemented in Java by the user. > This feature relates to the concept of CRDTs; the proposal is to replicate > "operations" on these types during write, to apply these operations > internally during merge (Column.reconcile), and to also merge their values on > read. > The following operations are made possible without reading back any data: > * MIN or MAX(value) for a column > * First value for a column > * Count Distinct > * HyperLogLog > * Count-Min > And any composition of these too, e.g. a Candlestick type includes first, > last, min, and max. > The merge operations exposed by these types need to be commutative; this is > the case for many functions used in analytics. > This feature is incomplete without some integration with CASSANDRA-4775 > (Counters 2.0) which provides a Read-Modify-Write implementation for > distributed counters. Integrating custom creation and merge functions with > new counters would let users implement complex CRDTs in Cassandra, including: > * Averages & related (sum of squares, standard deviation) > * Graphs > * Sets > * Custom registers (even with vector clocks) > I have a working prototype with implementations for min, max, and Candlestick > at https://github.com/acunu/cassandra/tree/crdts - I'd appreciate any > feedback on the design and interfaces. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (CASSANDRA-9120) OutOfMemoryError when read auto-saved cache (probably broken)
[
https://issues.apache.org/jira/browse/CASSANDRA-9120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jeff Jirsa reassigned CASSANDRA-9120:
-
Assignee: Jeff Jirsa
> OutOfMemoryError when read auto-saved cache (probably broken)
> -
>
> Key: CASSANDRA-9120
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9120
> Project: Cassandra
> Issue Type: Bug
> Environment: Linux
>Reporter: Vladimir
>Assignee: Jeff Jirsa
> Fix For: 2.0.15
>
>
> Found during tests on a 100 nodes cluster. After restart I found that one
> node constantly crashes with OutOfMemory Exception. I guess that auto-saved
> cache was corrupted and Cassandra can't recognize it. I see that similar
> issues was already fixed (when negative size of some structure was read).
> Does auto-saved cache have checksum? it'd help to reject corrupted cache at
> the very beginning.
> As far as I can see current code still have that problem. Stack trace is:
> {code}
> INFO [main] 2015-03-28 01:04:13,503 AutoSavingCache.java (line 114) reading
> saved cache
> /storage/core/loginsight/cidata/cassandra/saved_caches/system-sstable_activity-KeyCache-b.db
> ERROR [main] 2015-03-28 01:04:14,718 CassandraDaemon.java (line 513)
> Exception encountered during startup
> java.lang.OutOfMemoryError: Java heap space
> at java.util.ArrayList.(Unknown Source)
> at
> org.apache.cassandra.db.RowIndexEntry$Serializer.deserialize(RowIndexEntry.java:120)
> at
> org.apache.cassandra.service.CacheService$KeyCacheSerializer.deserialize(CacheService.java:365)
> at
> org.apache.cassandra.cache.AutoSavingCache.loadSaved(AutoSavingCache.java:119)
> at
> org.apache.cassandra.db.ColumnFamilyStore.(ColumnFamilyStore.java:262)
> at
> org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:421)
> at
> org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:392)
> at org.apache.cassandra.db.Keyspace.initCf(Keyspace.java:315)
> at org.apache.cassandra.db.Keyspace.(Keyspace.java:272)
> at org.apache.cassandra.db.Keyspace.open(Keyspace.java:114)
> at org.apache.cassandra.db.Keyspace.open(Keyspace.java:92)
> at
> org.apache.cassandra.db.SystemKeyspace.checkHealth(SystemKeyspace.java:536)
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:261)
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:496)
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:585)
> {code}
> I looked at source code of Cassandra and see:
> http://grepcode.com/file/repo1.maven.org/maven2/org.apache.cassandra/cassandra-all/2.0.10/org/apache/cassandra/db/RowIndexEntry.java
> 119 int entries = in.readInt();
> 120 List columnsIndex = new
> ArrayList(entries);
> It seems that value entries is invalid (negative) and it tries too allocate
> an array with huge initial capacity and hits OOM. I have deleted saved_cache
> directory and was able to start node correctly. We should expect that it may
> happen in real world. Cassandra should be able to skip incorrect cached data
> and run.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9120) OutOfMemoryError when read auto-saved cache (probably broken)
[
https://issues.apache.org/jira/browse/CASSANDRA-9120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14396082#comment-14396082
]
Jeff Jirsa commented on CASSANDRA-9120:
---
If it were negative, I suspect you'd be getting IllegalArgumentException for a
negative ArrayList. It's more likely very, very large positive int, which is
allocating a very large array.
There are two potential limits to the number of keys to deserialize:
1) key_cache_keys_to_save, which defaults to Integer.MAX_VALUE, which will
probably still overflow heap on most systems, and
2) the amount of memory available
Given that key_cache_keys_to_save defaults to a very high number, I think the
most sensible approach is to estimate the memory used by the array, and skip
loading if it's more than the amount of free RAM
Suggested patch here:
https://github.com/apache/cassandra/compare/cassandra-2.0...jeffjirsa:cassandra-2.0-9120.diff
Glancing at the code, this probably also impacts
[2.1|https://github.com/apache/cassandra/blob/cassandra-2.1/src/java/org/apache/cassandra/db/RowIndexEntry.java#L125-L129]
and
[trunk|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/RowIndexEntry.java#L137-L142].
> OutOfMemoryError when read auto-saved cache (probably broken)
> -
>
> Key: CASSANDRA-9120
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9120
> Project: Cassandra
> Issue Type: Bug
> Environment: Linux
>Reporter: Vladimir
>Assignee: Jeff Jirsa
> Fix For: 2.0.15
>
>
> Found during tests on a 100 nodes cluster. After restart I found that one
> node constantly crashes with OutOfMemory Exception. I guess that auto-saved
> cache was corrupted and Cassandra can't recognize it. I see that similar
> issues was already fixed (when negative size of some structure was read).
> Does auto-saved cache have checksum? it'd help to reject corrupted cache at
> the very beginning.
> As far as I can see current code still have that problem. Stack trace is:
> {code}
> INFO [main] 2015-03-28 01:04:13,503 AutoSavingCache.java (line 114) reading
> saved cache
> /storage/core/loginsight/cidata/cassandra/saved_caches/system-sstable_activity-KeyCache-b.db
> ERROR [main] 2015-03-28 01:04:14,718 CassandraDaemon.java (line 513)
> Exception encountered during startup
> java.lang.OutOfMemoryError: Java heap space
> at java.util.ArrayList.(Unknown Source)
> at
> org.apache.cassandra.db.RowIndexEntry$Serializer.deserialize(RowIndexEntry.java:120)
> at
> org.apache.cassandra.service.CacheService$KeyCacheSerializer.deserialize(CacheService.java:365)
> at
> org.apache.cassandra.cache.AutoSavingCache.loadSaved(AutoSavingCache.java:119)
> at
> org.apache.cassandra.db.ColumnFamilyStore.(ColumnFamilyStore.java:262)
> at
> org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:421)
> at
> org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:392)
> at org.apache.cassandra.db.Keyspace.initCf(Keyspace.java:315)
> at org.apache.cassandra.db.Keyspace.(Keyspace.java:272)
> at org.apache.cassandra.db.Keyspace.open(Keyspace.java:114)
> at org.apache.cassandra.db.Keyspace.open(Keyspace.java:92)
> at
> org.apache.cassandra.db.SystemKeyspace.checkHealth(SystemKeyspace.java:536)
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:261)
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:496)
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:585)
> {code}
> I looked at source code of Cassandra and see:
> http://grepcode.com/file/repo1.maven.org/maven2/org.apache.cassandra/cassandra-all/2.0.10/org/apache/cassandra/db/RowIndexEntry.java
> 119 int entries = in.readInt();
> 120 List columnsIndex = new
> ArrayList(entries);
> It seems that value entries is invalid (negative) and it tries too allocate
> an array with huge initial capacity and hits OOM. I have deleted saved_cache
> directory and was able to start node correctly. We should expect that it may
> happen in real world. Cassandra should be able to skip incorrect cached data
> and run.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9120) OutOfMemoryError when read auto-saved cache (probably broken)
[
https://issues.apache.org/jira/browse/CASSANDRA-9120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14396375#comment-14396375
]
Jeff Jirsa commented on CASSANDRA-9120:
---
[~dbrosius] I don't disagree; I thought about cutting off at freeMemory instead
of maxMemory, but really was just looking for a common sense upper bound above
which we can assume the cache file is invalid and not even try to load it.
I'd agree that for trunk, the right approach would be to load as many of the
valid entries as possible with the memory available, but in 2.0, a safer "if
cache file seems invalid because the number of entries in it wouldn't fit into
all of the memory, punt" may be appropriately conservative for the existing
stable users?
> OutOfMemoryError when read auto-saved cache (probably broken)
> -
>
> Key: CASSANDRA-9120
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9120
> Project: Cassandra
> Issue Type: Bug
> Environment: Linux
>Reporter: Vladimir
>Assignee: Jeff Jirsa
> Fix For: 3.0, 2.0.15, 2.1.5
>
>
> Found during tests on a 100 nodes cluster. After restart I found that one
> node constantly crashes with OutOfMemory Exception. I guess that auto-saved
> cache was corrupted and Cassandra can't recognize it. I see that similar
> issues was already fixed (when negative size of some structure was read).
> Does auto-saved cache have checksum? it'd help to reject corrupted cache at
> the very beginning.
> As far as I can see current code still have that problem. Stack trace is:
> {code}
> INFO [main] 2015-03-28 01:04:13,503 AutoSavingCache.java (line 114) reading
> saved cache
> /storage/core/loginsight/cidata/cassandra/saved_caches/system-sstable_activity-KeyCache-b.db
> ERROR [main] 2015-03-28 01:04:14,718 CassandraDaemon.java (line 513)
> Exception encountered during startup
> java.lang.OutOfMemoryError: Java heap space
> at java.util.ArrayList.(Unknown Source)
> at
> org.apache.cassandra.db.RowIndexEntry$Serializer.deserialize(RowIndexEntry.java:120)
> at
> org.apache.cassandra.service.CacheService$KeyCacheSerializer.deserialize(CacheService.java:365)
> at
> org.apache.cassandra.cache.AutoSavingCache.loadSaved(AutoSavingCache.java:119)
> at
> org.apache.cassandra.db.ColumnFamilyStore.(ColumnFamilyStore.java:262)
> at
> org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:421)
> at
> org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:392)
> at org.apache.cassandra.db.Keyspace.initCf(Keyspace.java:315)
> at org.apache.cassandra.db.Keyspace.(Keyspace.java:272)
> at org.apache.cassandra.db.Keyspace.open(Keyspace.java:114)
> at org.apache.cassandra.db.Keyspace.open(Keyspace.java:92)
> at
> org.apache.cassandra.db.SystemKeyspace.checkHealth(SystemKeyspace.java:536)
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:261)
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:496)
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:585)
> {code}
> I looked at source code of Cassandra and see:
> http://grepcode.com/file/repo1.maven.org/maven2/org.apache.cassandra/cassandra-all/2.0.10/org/apache/cassandra/db/RowIndexEntry.java
> 119 int entries = in.readInt();
> 120 List columnsIndex = new
> ArrayList(entries);
> It seems that value entries is invalid (negative) and it tries too allocate
> an array with huge initial capacity and hits OOM. I have deleted saved_cache
> directory and was able to start node correctly. We should expect that it may
> happen in real world. Cassandra should be able to skip incorrect cached data
> and run.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-6412) Custom creation and merge functions for user-defined column types
[
https://issues.apache.org/jira/browse/CASSANDRA-6412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14484355#comment-14484355
]
Jeff Jirsa commented on CASSANDRA-6412:
---
I'm playing with this, just to understand it conceptually, using CASSANDRA-8099
as a base.
{noformat}
cqlsh> create keyspace test2 WITH replication = {'class': 'SimpleStrategy',
'replication_factor': 2}; use test2;
cqlsh:test2> create table table_with_resolvers ( id text, last text with
resolver 'org.apache.cassandra.db.resolvers.TimestampResolver', first text with
resolver 'org.apache.cassandra.db.resolvers.ReverseTimestampResolver', PRIMARY
KEY(id));
cqlsh:test2> select column_name, column_resolver, type, validator from
system.schema_columns where keyspace_name='test2' and
columnfamily_name='table_with_resolvers';
column_name | column_resolver|
type | validator
-++---+--
first | org.apache.cassandra.db.resolvers.ReverseTimestampResolver |
regular | org.apache.cassandra.db.marshal.UTF8Type
id |org.apache.cassandra.db.resolvers.TimestampResolver |
partition_key | org.apache.cassandra.db.marshal.UTF8Type
last |org.apache.cassandra.db.resolvers.TimestampResolver |
regular | org.apache.cassandra.db.marshal.UTF8Type
cqlsh:test2> insert into table_with_resolvers (id, first, last ) values ('1',
'1', '1');
cqlsh:test2> insert into table_with_resolvers (id, first, last ) values ('1',
'2', '2');
cqlsh:test2> insert into table_with_resolvers (id, first, last ) values ('1',
'3', '3');
cqlsh:test2> select * from table_with_resolvers ;
id | first | last
+---+--
1 | 1 |3
(1 rows)
{noformat}
My diff/patch isn't fit for sharing at this time but as I'm going through, I
had some questions:
1) Given that user types are frozen, does it make sense to allow a resolver per
field in user types, assuming that eventually user types will become un-frozen?
2) My initial pass disallows custom resolvers on counters and collections -
does anyone have any strong opinion on whether or not user defined merge
functions should be allowed for collections?
3) Given that deletes are not commutative, I'm strongly considering making it
so that built-in resolvers (min, max, first-write-wins, and default
last-write-wins) simply always allow tombstones with a higher timestamp to take
priority over anything else with a lower tombstone (that is,
last-write-always-wins with tombstones). That works around SOME of the corner
issues involving deletes - given that these are regular cells and have valid
timestamps, does that not address some of the concern?
> Custom creation and merge functions for user-defined column types
> -
>
> Key: CASSANDRA-6412
> URL: https://issues.apache.org/jira/browse/CASSANDRA-6412
> Project: Cassandra
> Issue Type: New Feature
> Components: Core
>Reporter: Nicolas Favre-Felix
>
> This is a proposal for a new feature, mapping custom types to Cassandra
> columns.
> These types would provide a creation function and a merge function, to be
> implemented in Java by the user.
> This feature relates to the concept of CRDTs; the proposal is to replicate
> "operations" on these types during write, to apply these operations
> internally during merge (Column.reconcile), and to also merge their values on
> read.
> The following operations are made possible without reading back any data:
> * MIN or MAX(value) for a column
> * First value for a column
> * Count Distinct
> * HyperLogLog
> * Count-Min
> And any composition of these too, e.g. a Candlestick type includes first,
> last, min, and max.
> The merge operations exposed by these types need to be commutative; this is
> the case for many functions used in analytics.
> This feature is incomplete without some integration with CASSANDRA-4775
> (Counters 2.0) which provides a Read-Modify-Write implementation for
> distributed counters. Integrating custom creation and merge functions with
> new counters would let users implement complex CRDTs in Cassandra, including:
> * Averages & related (sum of squares, standard deviation)
> * Graphs
> * Sets
> * Custom registers (even with vector clocks)
> I have a working prototype with implementations for min, max, and Candlestick
> at https://github.com/acunu/cassandra/tree/crdts - I'd appreciate any
> feedback on the design and interfaces.
--
This message was sent by Atlassian JI
[jira] [Comment Edited] (CASSANDRA-6412) Custom creation and merge functions for user-defined column types
[
https://issues.apache.org/jira/browse/CASSANDRA-6412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14484355#comment-14484355
]
Jeff Jirsa edited comment on CASSANDRA-6412 at 4/8/15 2:32 AM:
---
I'm playing with this, just to understand it conceptually, using CASSANDRA-8099
as a base.
{noformat}
cqlsh> create keyspace test2 WITH replication = {'class': 'SimpleStrategy',
'replication_factor': 2}; use test2;
cqlsh:test2> select column_name, column_resolver from system.schema_columns
where keyspace_name='test2' and columnfamily_name='table_with_resolvers';
column_name | column_resolver
-+
first | org.apache.cassandra.db.resolvers.ReverseTimestampResolver
high | org.apache.cassandra.db.resolvers.MaxValueResolver
id |org.apache.cassandra.db.resolvers.TimestampResolver
last |org.apache.cassandra.db.resolvers.TimestampResolver
low | org.apache.cassandra.db.resolvers.MinValueResolver
(5 rows)
cqlsh:test2> create table table_with_resolvers ( id text, low int with resolver
'org.apache.cassandra.db.resolvers.MinValueResolver', high int with resolver
'org.apache.cassandra.db.resolvers.MaxValueResolver', last int with resolver
'org.apache.cassandra.db.resolvers.TimestampResolver', first int with resolver
'org.apache.cassandra.db.resolvers.ReverseTimestampResolver', PRIMARY KEY(id));
cqlsh:test2> insert into table_with_resolvers (id, low, high, first, last )
values ('1', 1, 1, 1, 1);
cqlsh:test2> insert into table_with_resolvers (id, low, high, first, last )
values ('1', 2, 2, 2, 2);
cqlsh:test2> insert into table_with_resolvers (id, low, high, first, last )
values ('1', 3, 3, 3, 3);
cqlsh:test2> insert into table_with_resolvers (id, low, high, first, last )
values ('1', 5, 5, 5, 5);
cqlsh:test2> insert into table_with_resolvers (id, low, high, first, last )
values ('1', 4, 4, 4, 4);
cqlsh:test2> select * from table_with_resolvers;
id | first | high | last | low
+---+--+--+-
1 | 1 |5 |4 | 1
(1 rows)
{noformat}
My diff/patch isn't fit for sharing at this time but as I'm going through, I
had some questions:
1) Given that user types are frozen, does it make sense to allow a resolver per
field in user types, assuming that eventually user types will become un-frozen?
2) My initial pass disallows custom resolvers on counters and collections -
does anyone have any strong opinion on whether or not user defined merge
functions should be allowed for collections?
3) Given that deletes are not commutative, I'm strongly considering making it
so that built-in resolvers (min, max, first-write-wins, and default
last-write-wins) simply always allow tombstones with a higher timestamp to take
priority over anything else with a lower tombstone (that is,
last-write-always-wins with tombstones). That works around SOME of the corner
issues involving deletes - given that these are regular cells and have valid
timestamps, does that not address some of the concern?
was (Author: jjirsa):
I'm playing with this, just to understand it conceptually, using CASSANDRA-8099
as a base.
{noformat}
cqlsh> create keyspace test2 WITH replication = {'class': 'SimpleStrategy',
'replication_factor': 2}; use test2;
cqlsh:test2> create table table_with_resolvers ( id text, last text with
resolver 'org.apache.cassandra.db.resolvers.TimestampResolver', first text with
resolver 'org.apache.cassandra.db.resolvers.ReverseTimestampResolver', PRIMARY
KEY(id));
cqlsh:test2> select column_name, column_resolver, type, validator from
system.schema_columns where keyspace_name='test2' and
columnfamily_name='table_with_resolvers';
column_name | column_resolver|
type | validator
-++---+--
first | org.apache.cassandra.db.resolvers.ReverseTimestampResolver |
regular | org.apache.cassandra.db.marshal.UTF8Type
id |org.apache.cassandra.db.resolvers.TimestampResolver |
partition_key | org.apache.cassandra.db.marshal.UTF8Type
last |org.apache.cassandra.db.resolvers.TimestampResolver |
regular | org.apache.cassandra.db.marshal.UTF8Type
cqlsh:test2> insert into table_with_resolvers (id, first, last ) values ('1',
'1', '1');
cqlsh:test2> insert into table_with_resolvers (id, first, last ) values ('1',
'2', '2');
cqlsh:test2> inse
[jira] [Comment Edited] (CASSANDRA-6412) Custom creation and merge functions for user-defined column types
[
https://issues.apache.org/jira/browse/CASSANDRA-6412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14484355#comment-14484355
]
Jeff Jirsa edited comment on CASSANDRA-6412 at 4/8/15 3:03 AM:
---
I'm playing with this, just to understand it conceptually, using CASSANDRA-8099
as a base.
{noformat}
cqlsh> create keyspace test2 WITH replication = {'class': 'SimpleStrategy',
'replication_factor': 2}; use test2;
cqlsh:test2> select column_name, column_resolver from system.schema_columns
where keyspace_name='test2' and columnfamily_name='table_with_resolvers';
column_name | column_resolver
-+
first | org.apache.cassandra.db.resolvers.ReverseTimestampResolver
high | org.apache.cassandra.db.resolvers.MaxValueResolver
id |org.apache.cassandra.db.resolvers.TimestampResolver
last |org.apache.cassandra.db.resolvers.TimestampResolver
low | org.apache.cassandra.db.resolvers.MinValueResolver
(5 rows)
cqlsh:test2> create table table_with_resolvers ( id text, low int with resolver
'org.apache.cassandra.db.resolvers.MinValueResolver', high int with resolver
'org.apache.cassandra.db.resolvers.MaxValueResolver', last int with resolver
'org.apache.cassandra.db.resolvers.TimestampResolver', first int with resolver
'org.apache.cassandra.db.resolvers.ReverseTimestampResolver', PRIMARY KEY(id));
cqlsh:test2> insert into table_with_resolvers (id, low, high, first, last )
values ('1', 1, 1, 1, 1);
cqlsh:test2> insert into table_with_resolvers (id, low, high, first, last )
values ('1', 2, 2, 2, 2);
cqlsh:test2> insert into table_with_resolvers (id, low, high, first, last )
values ('1', 3, 3, 3, 3);
cqlsh:test2> insert into table_with_resolvers (id, low, high, first, last )
values ('1', 5, 5, 5, 5);
cqlsh:test2> insert into table_with_resolvers (id, low, high, first, last )
values ('1', 4, 4, 4, 4);
cqlsh:test2> select * from table_with_resolvers;
id | first | high | last | low
+---+--+--+-
1 | 1 |5 |4 | 1
(1 rows)
{noformat}
My diff/patch isn't fit for sharing at this time but as I'm going through, I
had some questions:
1) Given that user types are frozen, does it make sense to allow a resolver per
field in user types, assuming that eventually user types will become un-frozen?
2) My initial pass disallows custom resolvers on counters and collections -
does anyone have any strong opinion on whether or not user defined merge
functions should be allowed for collections?
3) Still battling through deletion/tombstone reconciliation. Still making sure
I fully understand all of the problem cases.
was (Author: jjirsa):
I'm playing with this, just to understand it conceptually, using CASSANDRA-8099
as a base.
{noformat}
cqlsh> create keyspace test2 WITH replication = {'class': 'SimpleStrategy',
'replication_factor': 2}; use test2;
cqlsh:test2> select column_name, column_resolver from system.schema_columns
where keyspace_name='test2' and columnfamily_name='table_with_resolvers';
column_name | column_resolver
-+
first | org.apache.cassandra.db.resolvers.ReverseTimestampResolver
high | org.apache.cassandra.db.resolvers.MaxValueResolver
id |org.apache.cassandra.db.resolvers.TimestampResolver
last |org.apache.cassandra.db.resolvers.TimestampResolver
low | org.apache.cassandra.db.resolvers.MinValueResolver
(5 rows)
cqlsh:test2> create table table_with_resolvers ( id text, low int with resolver
'org.apache.cassandra.db.resolvers.MinValueResolver', high int with resolver
'org.apache.cassandra.db.resolvers.MaxValueResolver', last int with resolver
'org.apache.cassandra.db.resolvers.TimestampResolver', first int with resolver
'org.apache.cassandra.db.resolvers.ReverseTimestampResolver', PRIMARY KEY(id));
cqlsh:test2> insert into table_with_resolvers (id, low, high, first, last )
values ('1', 1, 1, 1, 1);
cqlsh:test2> insert into table_with_resolvers (id, low, high, first, last )
values ('1', 2, 2, 2, 2);
cqlsh:test2> insert into table_with_resolvers (id, low, high, first, last )
values ('1', 3, 3, 3, 3);
cqlsh:test2> insert into table_with_resolvers (id, low, high, first, last )
values ('1', 5, 5, 5, 5);
cqlsh:test2> insert into table_with_resolvers (id, low, high, first, last )
values ('1', 4, 4, 4, 4);
cqlsh:test2> select * from table_with_resolvers;
id | first | high | last | low
+---+--+--+-
1 | 1 |5 |4 | 1
(1 rows
[jira] [Updated] (CASSANDRA-9120) OutOfMemoryError when read auto-saved cache (probably broken)
[
https://issues.apache.org/jira/browse/CASSANDRA-9120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jeff Jirsa updated CASSANDRA-9120:
--
Assignee: (was: Jeff Jirsa)
> OutOfMemoryError when read auto-saved cache (probably broken)
> -
>
> Key: CASSANDRA-9120
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9120
> Project: Cassandra
> Issue Type: Bug
> Environment: Linux
>Reporter: Vladimir
> Fix For: 3.0, 2.0.15, 2.1.5
>
>
> Found during tests on a 100 nodes cluster. After restart I found that one
> node constantly crashes with OutOfMemory Exception. I guess that auto-saved
> cache was corrupted and Cassandra can't recognize it. I see that similar
> issues was already fixed (when negative size of some structure was read).
> Does auto-saved cache have checksum? it'd help to reject corrupted cache at
> the very beginning.
> As far as I can see current code still have that problem. Stack trace is:
> {code}
> INFO [main] 2015-03-28 01:04:13,503 AutoSavingCache.java (line 114) reading
> saved cache
> /storage/core/loginsight/cidata/cassandra/saved_caches/system-sstable_activity-KeyCache-b.db
> ERROR [main] 2015-03-28 01:04:14,718 CassandraDaemon.java (line 513)
> Exception encountered during startup
> java.lang.OutOfMemoryError: Java heap space
> at java.util.ArrayList.(Unknown Source)
> at
> org.apache.cassandra.db.RowIndexEntry$Serializer.deserialize(RowIndexEntry.java:120)
> at
> org.apache.cassandra.service.CacheService$KeyCacheSerializer.deserialize(CacheService.java:365)
> at
> org.apache.cassandra.cache.AutoSavingCache.loadSaved(AutoSavingCache.java:119)
> at
> org.apache.cassandra.db.ColumnFamilyStore.(ColumnFamilyStore.java:262)
> at
> org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:421)
> at
> org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:392)
> at org.apache.cassandra.db.Keyspace.initCf(Keyspace.java:315)
> at org.apache.cassandra.db.Keyspace.(Keyspace.java:272)
> at org.apache.cassandra.db.Keyspace.open(Keyspace.java:114)
> at org.apache.cassandra.db.Keyspace.open(Keyspace.java:92)
> at
> org.apache.cassandra.db.SystemKeyspace.checkHealth(SystemKeyspace.java:536)
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:261)
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:496)
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:585)
> {code}
> I looked at source code of Cassandra and see:
> http://grepcode.com/file/repo1.maven.org/maven2/org.apache.cassandra/cassandra-all/2.0.10/org/apache/cassandra/db/RowIndexEntry.java
> 119 int entries = in.readInt();
> 120 List columnsIndex = new
> ArrayList(entries);
> It seems that value entries is invalid (negative) and it tries too allocate
> an array with huge initial capacity and hits OOM. I have deleted saved_cache
> directory and was able to start node correctly. We should expect that it may
> happen in real world. Cassandra should be able to skip incorrect cached data
> and run.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-9148) Issue when modifying UDT
[ https://issues.apache.org/jira/browse/CASSANDRA-9148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Jirsa updated CASSANDRA-9148: -- Attachment: 9148-2.1.txt Apologies for [~blerer], didn't notice he was already assigned. Perhaps he can review. Adding patch. Should apply to both 2.1 and trunk. Also on github against trunk: https://github.com/jeffjirsa/cassandra/commit/2853aa4e01dd91f15b47a829bb53c499c729c5d8.diff Or against 2.1 (identical): https://github.com/jeffjirsa/cassandra/commit/3375ac6fc1dc15f414e8d594f854dee2676711fd.diff Includes unit test. The assert in db/composites/CellNames.java is to throw an AssertionError rather than an NPE if there's another code path that somehow hits this same issue. > Issue when modifying UDT > > > Key: CASSANDRA-9148 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9148 > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Oskar Kjellin >Assignee: Benjamin Lerer > Fix For: 2.1.5 > > Attachments: 9148-2.1.txt > > > I'm trying out the user defined types but ran into some issues when adding a > column to an existing type. > Unfortunately I had to scrap the entire cluster so I cannot access it any > more. > After creating the UDT i adde two tables using it. 1 was just using > frozen. The other was using both frozen frozen map. > Then I realized I needed to add a new field to the user type. Then when I > tried to put to any of the two tables (setting all fields to the UDT in the > datastax java driver) I got this error message that I could not find anywhere > else but in the cassandra code: > com.datastax.driver.core.exceptions.InvalidQueryException: Invalid remaining > data after end of UDT value > I had to scrap my keyspace in order to be able to use it again. Could not > even drop one of the tables. > I know that they are frozen so we cannot modify the value of individual > fields once they are written but we must be able to modify the schema right? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-9148) Issue when modifying UDT
[ https://issues.apache.org/jira/browse/CASSANDRA-9148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Jirsa updated CASSANDRA-9148: -- Attachment: (was: 9148-2.1.txt) > Issue when modifying UDT > > > Key: CASSANDRA-9148 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9148 > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Oskar Kjellin >Assignee: Jeff Jirsa > Fix For: 2.1.5 > > Attachments: 9148.txt > > > I'm trying out the user defined types but ran into some issues when adding a > column to an existing type. > Unfortunately I had to scrap the entire cluster so I cannot access it any > more. > After creating the UDT i adde two tables using it. 1 was just using > frozen. The other was using both frozen frozen map. > Then I realized I needed to add a new field to the user type. Then when I > tried to put to any of the two tables (setting all fields to the UDT in the > datastax java driver) I got this error message that I could not find anywhere > else but in the cassandra code: > com.datastax.driver.core.exceptions.InvalidQueryException: Invalid remaining > data after end of UDT value > I had to scrap my keyspace in order to be able to use it again. Could not > even drop one of the tables. > I know that they are frozen so we cannot modify the value of individual > fields once they are written but we must be able to modify the schema right? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-9148) Issue when modifying UDT
[ https://issues.apache.org/jira/browse/CASSANDRA-9148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Jirsa updated CASSANDRA-9148: -- Attachment: 9148.txt > Issue when modifying UDT > > > Key: CASSANDRA-9148 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9148 > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Oskar Kjellin >Assignee: Jeff Jirsa > Fix For: 2.1.5 > > Attachments: 9148.txt > > > I'm trying out the user defined types but ran into some issues when adding a > column to an existing type. > Unfortunately I had to scrap the entire cluster so I cannot access it any > more. > After creating the UDT i adde two tables using it. 1 was just using > frozen. The other was using both frozen frozen map. > Then I realized I needed to add a new field to the user type. Then when I > tried to put to any of the two tables (setting all fields to the UDT in the > datastax java driver) I got this error message that I could not find anywhere > else but in the cassandra code: > com.datastax.driver.core.exceptions.InvalidQueryException: Invalid remaining > data after end of UDT value > I had to scrap my keyspace in order to be able to use it again. Could not > even drop one of the tables. > I know that they are frozen so we cannot modify the value of individual > fields once they are written but we must be able to modify the schema right? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-9148) Issue when modifying UDT
[ https://issues.apache.org/jira/browse/CASSANDRA-9148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14491978#comment-14491978 ] Jeff Jirsa edited comment on CASSANDRA-9148 at 4/13/15 11:01 PM: - Apologies for [~blerer], didn't notice he was already assigned. Perhaps he can review. https://github.com/jeffjirsa/cassandra/commit/3802b59b5df33d546832095df010287f3cebe0f5.diff Includes unit test. The assert in db/composites/CellNames.java is to throw an AssertionError rather than an NPE if there's another code path that somehow hits this same issue. was (Author: jjirsa): Apologies for [~blerer], didn't notice he was already assigned. Perhaps he can review. Adding patch. Should apply to both 2.1 and trunk. Also on github against trunk: https://github.com/jeffjirsa/cassandra/commit/2853aa4e01dd91f15b47a829bb53c499c729c5d8.diff Or against 2.1 (identical): https://github.com/jeffjirsa/cassandra/commit/3375ac6fc1dc15f414e8d594f854dee2676711fd.diff Includes unit test. The assert in db/composites/CellNames.java is to throw an AssertionError rather than an NPE if there's another code path that somehow hits this same issue. > Issue when modifying UDT > > > Key: CASSANDRA-9148 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9148 > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Oskar Kjellin >Assignee: Jeff Jirsa > Fix For: 2.1.5 > > Attachments: 9148.txt > > > I'm trying out the user defined types but ran into some issues when adding a > column to an existing type. > Unfortunately I had to scrap the entire cluster so I cannot access it any > more. > After creating the UDT i adde two tables using it. 1 was just using > frozen. The other was using both frozen frozen map. > Then I realized I needed to add a new field to the user type. Then when I > tried to put to any of the two tables (setting all fields to the UDT in the > datastax java driver) I got this error message that I could not find anywhere > else but in the cassandra code: > com.datastax.driver.core.exceptions.InvalidQueryException: Invalid remaining > data after end of UDT value > I had to scrap my keyspace in order to be able to use it again. Could not > even drop one of the tables. > I know that they are frozen so we cannot modify the value of individual > fields once they are written but we must be able to modify the schema right? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-9148) Issue when modifying UDT
[
https://issues.apache.org/jira/browse/CASSANDRA-9148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jeff Jirsa updated CASSANDRA-9148:
--
Attachment: 9148-v2.txt
Thanks [~blerer].
Uploaded 9148-v2.txt - also available at
https://github.com/jeffjirsa/cassandra/commit/cbfccbe7a3f56db47d560afb0bc8882950d2a5f1
Rebased to trunk as of 20150415
Unit tests updated as suggested (I was originally keeping it minimal and
matching the naming of existing tests, but let's get better tests written now
instead of waiting on the 3.0 triage). New patch includes set/list tests.
No longer causing the double migrations when running this quick test script:
{noformat}
create keyspace test2 WITH replication = {'class': 'SimpleStrategy',
'replication_factor': 1};
use test2;
CREATE TYPE test1 (foo text);
CREATE TYPE test2 (foo text, bar text);
CREATE TABLE testtable (id text, map1 frozen>, primary key
(id));
INSERT INTO testtable (id, map1) VALUES ('foo', {'foo':{foo:'bar'}});
CREATE TABLE testtable2 (id text, map2 frozen>, primary
key(id));
ALTER TYPE test1 add bar text;
{noformat}
The ALTER TYPE statement now results in a the proper number of announced
migrations, rather than announcing migrations for any table that had any UDT,
and that announced migration has the properly updated comparator for the
updated UDT.
{noformat}
INFO [SharedPool-Worker-2] 2015-04-15 22:25:01,882 MigrationManager.java:
Update table 'test2/testtable' From
org.apache.cassandra.config.CFMetaData@1aee5414[cfId=ee34a6a0-e3f8-11e4-af98-95e9d2c49838,ksName=test2,cfName=testtable,cfType=Standard,comparator=org.apache.cassandra.db.marshal.CompositeType(org.apache.cassandra.db.marshal.UTF8Type),comment=,readRepairChance=0.0,dcLocalReadRepairChance=0.1,gcGraceSeconds=864000,defaultValidator=org.apache.cassandra.db.marshal.BytesType,keyValidator=org.apache.cassandra.db.marshal.UTF8Type,minCompactionThreshold=4,maxCompactionThreshold=32,columnMetadata=[ColumnDefinition{name=id,
type=org.apache.cassandra.db.marshal.UTF8Type, kind=PARTITION_KEY,
componentIndex=null, indexName=null, indexType=null},
ColumnDefinition{name=map1,
type=org.apache.cassandra.db.marshal.FrozenType(org.apache.cassandra.db.marshal.MapType(org.apache.cassandra.db.marshal.UTF8Type,org.apache.cassandra.db.marshal.UserType(test2,7465737431,666f6f:org.apache.cassandra.db.marshal.UTF8Type))),
kind=REGULAR, componentIndex=0, indexName=null,
indexType=null}],compactionStrategyClass=class
org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy,compactionStrategyOptions={},compressionParameters={sstable_compression=org.apache.cassandra.io.compress.LZ4Compressor},bloomFilterFpChance=0.01,memtableFlushPeriod=0,caching={"keys":"ALL",
"rows_per_partition":"NONE"},defaultTimeToLive=0,minIndexInterval=128,maxIndexInterval=2048,speculativeRetry=99.0PERCENTILE,droppedColumns={},triggers=[],isDense=false]
To
org.apache.cassandra.config.CFMetaData@3e779c17[cfId=ee34a6a0-e3f8-11e4-af98-95e9d2c49838,ksName=test2,cfName=testtable,cfType=Standard,comparator=org.apache.cassandra.db.marshal.CompositeType(org.apache.cassandra.db.marshal.UTF8Type),comment=,readRepairChance=0.0,dcLocalReadRepairChance=0.1,gcGraceSeconds=864000,defaultValidator=org.apache.cassandra.db.marshal.BytesType,keyValidator=org.apache.cassandra.db.marshal.UTF8Type,minCompactionThreshold=4,maxCompactionThreshold=32,columnMetadata=[ColumnDefinition{name=id,
type=org.apache.cassandra.db.marshal.UTF8Type, kind=PARTITION_KEY,
componentIndex=null, indexName=null, indexType=null},
ColumnDefinition{name=map1,
type=org.apache.cassandra.db.marshal.FrozenType(org.apache.cassandra.db.marshal.MapType(org.apache.cassandra.db.marshal.UTF8Type,org.apache.cassandra.db.marshal.UserType(test2,7465737431,666f6f:org.apache.cassandra.db.marshal.UTF8Type,626172:org.apache.cassandra.db.marshal.UTF8Type))),
kind=REGULAR, componentIndex=0, indexName=null,
indexType=null}],compactionStrategyClass=class
org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy,compactionStrategyOptions={},compressionParameters={sstable_compression=org.apache.cassandra.io.compress.LZ4Compressor},bloomFilterFpChance=0.01,memtableFlushPeriod=0,caching={"keys":"ALL",
"rows_per_partition":"NONE"},defaultTimeToLive=0,minIndexInterval=128,maxIndexInterval=2048,speculativeRetry=99.0PERCENTILE,droppedColumns={},triggers=[],isDense=false]
{noformat}
> Issue when modifying UDT
>
>
> Key: CASSANDRA-9148
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9148
> Project: Cassandra
> Issue Type: Bug
> Components: Core
>Reporter: Oskar Kjellin
>Assignee: Jeff Jirsa
> Fix For: 2.1.5
>
> Attachments: 9148-v2.txt, 9148.txt
>
>
> I'm trying out the user defined types but ran into some issues when adding a
> column to an existing type.
> Unfortunately I h
[jira] [Comment Edited] (CASSANDRA-9148) Issue when modifying UDT
[
https://issues.apache.org/jira/browse/CASSANDRA-9148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14497593#comment-14497593
]
Jeff Jirsa edited comment on CASSANDRA-9148 at 4/16/15 5:52 AM:
Thanks [~blerer].
Uploaded 9148-v2.txt - also available at
https://github.com/jeffjirsa/cassandra/commit/cbfccbe7a3f56db47d560afb0bc8882950d2a5f1
Rebased to trunk as of 20150415
Unit tests updated as suggested (I was originally keeping it minimal and
matching the naming of existing tests, but let's get better tests written now
instead of waiting on the 3.0 triage). New patch includes set/list tests.
No longer causing the double migrations when running this quick test script:
{noformat}
create keyspace test2 WITH replication = {'class': 'SimpleStrategy',
'replication_factor': 1};
use test2;
CREATE TYPE test1 (foo text);
CREATE TYPE test2 (foo text, bar text);
CREATE TABLE testtable (id text, map1 frozen>, primary key
(id));
INSERT INTO testtable (id, map1) VALUES ('foo', {'foo':{foo:'bar'}});
CREATE TABLE testtable2 (id text, map2 frozen>, primary
key(id));
ALTER TYPE test1 add bar text;
{noformat}
The ALTER TYPE statement now results in a the proper number of announced
migrations, rather than announcing migrations for any table that had any UDT,
and that announced migration has the properly updated comparator for the
updated UDT.
{noformat}
INFO [SharedPool-Worker-2] 2015-04-15 22:25:01,882 MigrationManager.java:
Update table 'test2/testtable' From
org.apache.cassandra.config.CFMetaData@1aee5414[cfId=ee34a6a0-e3f8-11e4-af98-95e9d2c49838,
ksName=test2,
cfName=testtable,
cfType=Standard,
comparator=org.apache.cassandra.db.marshal.CompositeType(org.apache.cassandra.db.marshal.UTF8Type),comment=,readRepairChance=0.0,dcLocalReadRepairChance=0.1,gcGraceSeconds=864000,
defaultValidator=org.apache.cassandra.db.marshal.BytesType,keyValidator=org.apache.cassandra.db.marshal.UTF8Type,minCompactionThreshold=4,maxCompactionThreshold=32,columnMetadata=[ColumnDefinition{name=id,
type=org.apache.cassandra.db.marshal.UTF8Type, kind=PARTITION_KEY,
componentIndex=null, indexName=null, indexType=null},
ColumnDefinition{name=map1,
type=org.apache.cassandra.db.marshal.FrozenType(org.apache.cassandra.db.marshal.MapType(org.apache.cassandra.db.marshal.UTF8Type,org.apache.cassandra.db.marshal.UserType(test2,7465737431,666f6f:org.apache.cassandra.db.marshal.UTF8Type))),
kind=REGULAR, componentIndex=0, indexName=null, indexType=null}],
compactionStrategyClass=class
org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy,compactionStrategyOptions={},
compressionParameters={sstable_compression=org.apache.cassandra.io.compress.LZ4Compressor},bloomFilterFpChance=0.01,memtableFlushPeriod=0,
caching={"keys":"ALL",
"rows_per_partition":"NONE"},defaultTimeToLive=0,minIndexInterval=128,maxIndexInterval=2048,speculativeRetry=99.0PERCENTILE,droppedColumns={},triggers=[],isDense=false]
To
org.apache.cassandra.config.CFMetaData@3e779c17[cfId=ee34a6a0-e3f8-11e4-af98-95e9d2c49838,
ksName=test2,
cfName=testtable,
cfType=Standard,
comparator=org.apache.cassandra.db.marshal.CompositeType(org.apache.cassandra.db.marshal.UTF8Type),
comment=,readRepairChance=0.0,dcLocalReadRepairChance=0.1,gcGraceSeconds=864000,
defaultValidator=org.apache.cassandra.db.marshal.BytesType,keyValidator=org.apache.cassandra.db.marshal.UTF8Type,minCompactionThreshold=4,maxCompactionThreshold=32,
columnMetadata=[ColumnDefinition{name=id,
type=org.apache.cassandra.db.marshal.UTF8Type, kind=PARTITION_KEY,
componentIndex=null, indexName=null, indexType=null},
ColumnDefinition{name=map1,
type=org.apache.cassandra.db.marshal.FrozenType(org.apache.cassandra.db.marshal.MapType(org.apache.cassandra.db.marshal.UTF8Type,org.apache.cassandra.db.marshal.UserType(test2,7465737431,666f6f:org.apache.cassandra.db.marshal.UTF8Type,626172:org.apache.cassandra.db.marshal.UTF8Type))),
kind=REGULAR, componentIndex=0, indexName=null,
indexType=null}],compactionStrategyClass=class
org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy,compactionStrategyOptions={},compressionParameters={sstable_compression=org.apache.cassandra.io.compress.LZ4Compressor},bloomFilterFpChance=0.01,memtableFlushPeriod=0,caching={"keys":"ALL",
"rows_per_partition":"NONE"},defaultTimeToLive=0,minIndexInterval=128,maxIndexInterval=2048,speculativeRetry=99.0PERCENTILE,droppedColumns={},triggers=[],isDense=false]
{noformat}
was (Author: jjirsa):
Thanks [~blerer].
Uploaded 9148-v2.txt - also available at
https://github.com/jeffjirsa/cassandra/commit/cbfccbe7a3f56db47d560afb0bc8882950d2a5f1
Rebased to trunk as of 20150415
Unit tests updated as suggested (I was originally keeping it minimal and
matching the naming of existing tests, but let's get better tests written now
instead of waiting on the 3.0 triage). New patch includes set/list tests.
No longer
[jira] [Updated] (CASSANDRA-8798) don't throw TombstoneOverwhelmingException during bootstrap
[
https://issues.apache.org/jira/browse/CASSANDRA-8798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jeff Jirsa updated CASSANDRA-8798:
--
Attachment: 8798.txt
I'm still writing a test case to reliably reproduce this on trunk, but
conceptually, I believe a patch like the one attached SHOULD address this
issue. Conceptually, it's extending
SliceQueryFilter.respectTombstoneThresholds() to properly ignore tombstone
thresholds in streaming related cases where the interrupting that process is
undesirable. To do this, we loop in a StorageService instance to see if it's
either bootstrapping or rebuilding, and a StreamManager instance to see if it's
repairing (if the map of receivingStreams is non-empty).
On a conceptual level, is anyone else able to confirm they agree with the basic
approach in theory?
As far as I can tell, there are no tests in trunk that contain the words
'TombstoneOverwhelmingException', so that may also be something to note for
CASSANDRA-9012
> don't throw TombstoneOverwhelmingException during bootstrap
> ---
>
> Key: CASSANDRA-8798
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8798
> Project: Cassandra
> Issue Type: Bug
> Components: Core
>Reporter: mck
> Fix For: 2.0.15
>
> Attachments: 8798.txt
>
>
> During bootstrap honouring tombstone_failure_threshold seems
> counter-productive as the node is not serving requests so not protecting
> anything.
> Instead what happens is bootstrap fails, and a cluster that obviously needs
> an extra node isn't getting it...
> **History**
> When adding a new node bootstrap process looks complete in that streaming is
> finished, compactions finished, and all disk and cpu activity is calm.
> But the node is still stuck in "joining" status.
> The last stage in the bootstrapping process is the rebuilding of secondary
> indexes. grepping the logs confirmed it failed during this stage.
> {code}grep SecondaryIndexManager cassandra/logs/*{code}
> To see what secondary index rebuilding was initiated
> {code}
> grep "index build of " cassandra/logs/* | awk -F" for data in " '{print $1}'
> INFO 13:18:11,252 Submitting index build of addresses.unobfuscatedIndex
> INFO 13:18:11,352 Submitting index build of Inbox.FINNBOXID_INDEX
> INFO 23:03:54,758 Submitting index build of [events.collected_tbIndex,
> events.real_tbIndex]
> {code}
> To get an idea of successful secondary index rebuilding
> {code}grep "Index build of "cassandra/logs/*
> INFO 13:18:11,263 Index build of addresses.unobfuscatedIndex complete
> INFO 13:18:11,355 Index build of Inbox.FINNBOXID_INDEX complete
> {code}
> Looking closer at {{[events.collected_tbIndex, events.real_tbIndex]}} showed
> the following stacktrace
> {code}
> ERROR [StreamReceiveTask:121] 2015-02-12 05:54:47,768 CassandraDaemon.java
> (line 199) Exception in thread Thread[StreamReceiveTask:121,5,main]
> java.lang.RuntimeException: java.util.concurrent.ExecutionException:
> java.lang.RuntimeException:
> org.apache.cassandra.db.filter.TombstoneOverwhelmingException
> at
> org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:413)
> at
> org.apache.cassandra.db.index.SecondaryIndexManager.maybeBuildSecondaryIndexes(SecondaryIndexManager.java:142)
> at
> org.apache.cassandra.streaming.StreamReceiveTask$OnCompletionRunnable.run(StreamReceiveTask.java:130)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.util.concurrent.ExecutionException:
> java.lang.RuntimeException:
> org.apache.cassandra.db.filter.TombstoneOverwhelmingException
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:188)
> at
> org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:409)
> ... 7 more
> Caused by: java.lang.RuntimeException:
> org.apache.cassandra.db.filter.TombstoneOverwhelmingException
> at
> org.apache.cassandra.service.pager.QueryPagers$1.next(QueryPagers.java:160)
> at
> org.apache.cassandra.service.pager.QueryPagers$1.next(QueryPagers.java:143)
> at org.apache.cassandra.db.Keyspace.indexRow(Keyspace.java:406)
> at
> org.apache.cassandra.db.index.SecondaryIndexBuilder.build(SecondaryIndexBuilder.java:62)
> at
> org.apache.cassandra.db.compaction.CompactionManager$9.run(CompactionManager.java:834)
> ... 5 more
> Caused by: org.apach
[jira] [Commented] (CASSANDRA-9181) Improve index versus secondary index selection
[ https://issues.apache.org/jira/browse/CASSANDRA-9181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14500842#comment-14500842 ] Jeff Jirsa commented on CASSANDRA-9181: --- For what it's worth, I've tested this on 2.0.14 and 3.0 (trunk), and it appears to be working as intended (if partition key is included, it's hitting only one node; if no partition key is included, it's hitting all nodes in the cluster). Your reproduced-in is 2.0.7 - have you seen this in more recent versions? Attaching logs for review. On 2.0.14: https://gist.github.com/jeffjirsa/5c0f63395269a85cdcb2 On trunk: https://gist.github.com/jeffjirsa/87e2b95113e3366bc00b The data generator for schema/etc is top of https://gist.github.com/jeffjirsa/87e2b95113e3366bc00b > Improve index versus secondary index selection > -- > > Key: CASSANDRA-9181 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9181 > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Jeremy Hanna > Labels: 2i > Fix For: 3.0 > > > There is a special case for secondary indexes if you always supply the > partition key. For example, if you have a family with ID "a456" which has 6 > family members and I have a secondary index on first name. Currently, if I > do a query like this "select * from families where id = 'a456' and firstname > = 'alowishus';" you can see from a query trace, that it will first scan the > entire cluster based on the firstname, then look for the key within that. > If it's not terribly invasive, I think this would be a valid use case to > narrow down the results by key first. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-9181) Improve index versus secondary index selection
[ https://issues.apache.org/jira/browse/CASSANDRA-9181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14500842#comment-14500842 ] Jeff Jirsa edited comment on CASSANDRA-9181 at 4/17/15 11:04 PM: - For what it's worth, I've tested this on 2.0.14 and 3.0 (trunk) using ccm (RF=1, N=3), and it appears to be working as intended (if partition key is included, it's hitting only one node; if no partition key is included, it's hitting all nodes in the cluster). Your reproduced-in is 2.0.7 - have you seen this in more recent versions? Attaching logs for review. On 2.0.14: https://gist.github.com/jeffjirsa/5c0f63395269a85cdcb2 On trunk: https://gist.github.com/jeffjirsa/87e2b95113e3366bc00b The data generator for schema/etc is top of https://gist.github.com/jeffjirsa/87e2b95113e3366bc00b was (Author: jjirsa): For what it's worth, I've tested this on 2.0.14 and 3.0 (trunk), and it appears to be working as intended (if partition key is included, it's hitting only one node; if no partition key is included, it's hitting all nodes in the cluster). Your reproduced-in is 2.0.7 - have you seen this in more recent versions? Attaching logs for review. On 2.0.14: https://gist.github.com/jeffjirsa/5c0f63395269a85cdcb2 On trunk: https://gist.github.com/jeffjirsa/87e2b95113e3366bc00b The data generator for schema/etc is top of https://gist.github.com/jeffjirsa/87e2b95113e3366bc00b > Improve index versus secondary index selection > -- > > Key: CASSANDRA-9181 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9181 > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Jeremy Hanna > Labels: 2i > Fix For: 3.0 > > > There is a special case for secondary indexes if you always supply the > partition key. For example, if you have a family with ID "a456" which has 6 > family members and I have a secondary index on first name. Currently, if I > do a query like this "select * from families where id = 'a456' and firstname > = 'alowishus';" you can see from a query trace, that it will first scan the > entire cluster based on the firstname, then look for the key within that. > If it's not terribly invasive, I think this would be a valid use case to > narrow down the results by key first. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9198) Deleting from an empty list produces an error
[
https://issues.apache.org/jira/browse/CASSANDRA-9198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14501694#comment-14501694
]
Jeff Jirsa commented on CASSANDRA-9198:
---
We brushed on this in CASSANDRA-9077
The fact that the list can become null just by removing elements from it seems
non-intuitive to me (not to say it's wrong, just that I don't think most
developers would expect it) - it seems like it should become empty, not null.
I would expect `DELETE FROM foo WHERE k=1` to result in a null list, but simply
removing an element from a list should result in an empty list.
> Deleting from an empty list produces an error
> -
>
> Key: CASSANDRA-9198
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9198
> Project: Cassandra
> Issue Type: Bug
> Components: API
>Reporter: Olivier Michallat
>Assignee: Benjamin Lerer
>Priority: Minor
> Fix For: 3.0
>
>
> While deleting an element from a list that does not contain it is a no-op,
> deleting it from an empty list causes an error.
> This edge case is a bit inconsistent, because it makes list deletion non
> idempotent:
> {code}
> cqlsh:test> create table foo (k int primary key, v list);
> cqlsh:test> insert into foo(k,v) values (1, [1,2]);
> cqlsh:test> update foo set v = v - [1] where k = 1;
> cqlsh:test> update foo set v = v - [1] where k = 1;
> cqlsh:test> update foo set v = v - [2] where k = 1;
> cqlsh:test> update foo set v = v - [2] where k = 1;
> InvalidRequest: code=2200 [Invalid query] message="Attempted to delete an
> element from a list which is null"
> {code}
> With speculative retries coming to the drivers, idempotency becomes more
> important because it determines which query we might retry or not. So it
> would be better if deleting from an empty list succeeded.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-9198) Deleting from an empty list produces an error
[
https://issues.apache.org/jira/browse/CASSANDRA-9198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jeff Jirsa updated CASSANDRA-9198:
--
Attachment: 9198.txt
Attaching patch that was previously option #2 on CASSANDRA-9077. Addresses this
without modification:
{noformat}
cqlsh> create keyspace IF NOT EXISTS test WITH replication = {'class':
'SimpleStrategy', 'replication_factor': 1};
cqlsh> USE test;
cqlsh:test> create table foo (k int primary key, v list);
cqlsh:test> insert into foo(k,v) values (1, [1,2]);
cqlsh:test> update foo set v = v - [1] where k = 1;
cqlsh:test> update foo set v = v - [1] where k = 1;
cqlsh:test> update foo set v = v - [2] where k = 1;
cqlsh:test> update foo set v = v - [2] where k = 1;
cqlsh:test>
cqlsh:test> select * from foo;
k | v
---+--
1 | null
(1 rows)
{noformat}
> Deleting from an empty list produces an error
> -
>
> Key: CASSANDRA-9198
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9198
> Project: Cassandra
> Issue Type: Bug
> Components: API
>Reporter: Olivier Michallat
>Assignee: Benjamin Lerer
>Priority: Minor
> Fix For: 3.0
>
> Attachments: 9198.txt
>
>
> While deleting an element from a list that does not contain it is a no-op,
> deleting it from an empty list causes an error.
> This edge case is a bit inconsistent, because it makes list deletion non
> idempotent:
> {code}
> cqlsh:test> create table foo (k int primary key, v list);
> cqlsh:test> insert into foo(k,v) values (1, [1,2]);
> cqlsh:test> update foo set v = v - [1] where k = 1;
> cqlsh:test> update foo set v = v - [1] where k = 1;
> cqlsh:test> update foo set v = v - [2] where k = 1;
> cqlsh:test> update foo set v = v - [2] where k = 1;
> InvalidRequest: code=2200 [Invalid query] message="Attempted to delete an
> element from a list which is null"
> {code}
> With speculative retries coming to the drivers, idempotency becomes more
> important because it determines which query we might retry or not. So it
> would be better if deleting from an empty list succeeded.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-9198) Deleting from an empty list produces an error
[
https://issues.apache.org/jira/browse/CASSANDRA-9198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14503146#comment-14503146
]
Jeff Jirsa edited comment on CASSANDRA-9198 at 4/20/15 4:37 PM:
Attaching patch that was previously option #2 on CASSANDRA-9077. Addresses this
without modification (though I'll also attach an update shortly that ALSO
addresses the now-failing unit tests that expected the older behavior)
{noformat}
cqlsh> create keyspace IF NOT EXISTS test WITH replication = {'class':
'SimpleStrategy', 'replication_factor': 1};
cqlsh> USE test;
cqlsh:test> create table foo (k int primary key, v list);
cqlsh:test> insert into foo(k,v) values (1, [1,2]);
cqlsh:test> update foo set v = v - [1] where k = 1;
cqlsh:test> update foo set v = v - [1] where k = 1;
cqlsh:test> update foo set v = v - [2] where k = 1;
cqlsh:test> update foo set v = v - [2] where k = 1;
cqlsh:test>
cqlsh:test> select * from foo;
k | v
---+--
1 | null
(1 rows)
{noformat}
was (Author: jjirsa):
Attaching patch that was previously option #2 on CASSANDRA-9077. Addresses this
without modification:
{noformat}
cqlsh> create keyspace IF NOT EXISTS test WITH replication = {'class':
'SimpleStrategy', 'replication_factor': 1};
cqlsh> USE test;
cqlsh:test> create table foo (k int primary key, v list);
cqlsh:test> insert into foo(k,v) values (1, [1,2]);
cqlsh:test> update foo set v = v - [1] where k = 1;
cqlsh:test> update foo set v = v - [1] where k = 1;
cqlsh:test> update foo set v = v - [2] where k = 1;
cqlsh:test> update foo set v = v - [2] where k = 1;
cqlsh:test>
cqlsh:test> select * from foo;
k | v
---+--
1 | null
(1 rows)
{noformat}
> Deleting from an empty list produces an error
> -
>
> Key: CASSANDRA-9198
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9198
> Project: Cassandra
> Issue Type: Bug
> Components: API
>Reporter: Olivier Michallat
>Assignee: Benjamin Lerer
>Priority: Minor
> Fix For: 3.0
>
> Attachments: 9198.txt
>
>
> While deleting an element from a list that does not contain it is a no-op,
> deleting it from an empty list causes an error.
> This edge case is a bit inconsistent, because it makes list deletion non
> idempotent:
> {code}
> cqlsh:test> create table foo (k int primary key, v list);
> cqlsh:test> insert into foo(k,v) values (1, [1,2]);
> cqlsh:test> update foo set v = v - [1] where k = 1;
> cqlsh:test> update foo set v = v - [1] where k = 1;
> cqlsh:test> update foo set v = v - [2] where k = 1;
> cqlsh:test> update foo set v = v - [2] where k = 1;
> InvalidRequest: code=2200 [Invalid query] message="Attempted to delete an
> element from a list which is null"
> {code}
> With speculative retries coming to the drivers, idempotency becomes more
> important because it determines which query we might retry or not. So it
> would be better if deleting from an empty list succeeded.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-9198) Deleting from an empty list produces an error
[
https://issues.apache.org/jira/browse/CASSANDRA-9198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jeff Jirsa updated CASSANDRA-9198:
--
Attachment: (was: 9198.txt)
> Deleting from an empty list produces an error
> -
>
> Key: CASSANDRA-9198
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9198
> Project: Cassandra
> Issue Type: Bug
> Components: API
>Reporter: Olivier Michallat
>Assignee: Benjamin Lerer
>Priority: Minor
> Fix For: 3.0
>
>
> While deleting an element from a list that does not contain it is a no-op,
> deleting it from an empty list causes an error.
> This edge case is a bit inconsistent, because it makes list deletion non
> idempotent:
> {code}
> cqlsh:test> create table foo (k int primary key, v list);
> cqlsh:test> insert into foo(k,v) values (1, [1,2]);
> cqlsh:test> update foo set v = v - [1] where k = 1;
> cqlsh:test> update foo set v = v - [1] where k = 1;
> cqlsh:test> update foo set v = v - [2] where k = 1;
> cqlsh:test> update foo set v = v - [2] where k = 1;
> InvalidRequest: code=2200 [Invalid query] message="Attempted to delete an
> element from a list which is null"
> {code}
> With speculative retries coming to the drivers, idempotency becomes more
> important because it determines which query we might retry or not. So it
> would be better if deleting from an empty list succeeded.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-9198) Deleting from an empty list produces an error
[
https://issues.apache.org/jira/browse/CASSANDRA-9198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jeff Jirsa updated CASSANDRA-9198:
--
Attachment: 9198.txt
Updated patch, allows idempotent deletes and corrects unit tests.
> Deleting from an empty list produces an error
> -
>
> Key: CASSANDRA-9198
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9198
> Project: Cassandra
> Issue Type: Bug
> Components: API
>Reporter: Olivier Michallat
>Assignee: Benjamin Lerer
>Priority: Minor
> Fix For: 3.0
>
> Attachments: 9198.txt
>
>
> While deleting an element from a list that does not contain it is a no-op,
> deleting it from an empty list causes an error.
> This edge case is a bit inconsistent, because it makes list deletion non
> idempotent:
> {code}
> cqlsh:test> create table foo (k int primary key, v list);
> cqlsh:test> insert into foo(k,v) values (1, [1,2]);
> cqlsh:test> update foo set v = v - [1] where k = 1;
> cqlsh:test> update foo set v = v - [1] where k = 1;
> cqlsh:test> update foo set v = v - [2] where k = 1;
> cqlsh:test> update foo set v = v - [2] where k = 1;
> InvalidRequest: code=2200 [Invalid query] message="Attempted to delete an
> element from a list which is null"
> {code}
> With speculative retries coming to the drivers, idempotency becomes more
> important because it determines which query we might retry or not. So it
> would be better if deleting from an empty list succeeded.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-6477) Global indexes
[ https://issues.apache.org/jira/browse/CASSANDRA-6477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14503881#comment-14503881 ] Jeff Jirsa commented on CASSANDRA-6477: --- As an end user, MV + matching SELECT described above looks appealing, and would match what I would expect from hearing the name and being familiar with VIEWs in other databases. > Global indexes > -- > > Key: CASSANDRA-6477 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6477 > Project: Cassandra > Issue Type: New Feature > Components: API, Core >Reporter: Jonathan Ellis >Assignee: Carl Yeksigian > Labels: cql > Fix For: 3.0 > > > Local indexes are suitable for low-cardinality data, where spreading the > index across the cluster is a Good Thing. However, for high-cardinality > data, local indexes require querying most nodes in the cluster even if only a > handful of rows is returned. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (CASSANDRA-9221) No test coverage for TombstoneOverwhelmingException
Jeff Jirsa created CASSANDRA-9221: - Summary: No test coverage for TombstoneOverwhelmingException Key: CASSANDRA-9221 URL: https://issues.apache.org/jira/browse/CASSANDRA-9221 Project: Cassandra Issue Type: Test Reporter: Jeff Jirsa Priority: Minor While investigating CASSANDRA-8798 discovered there were no unit tests for TombstoneOverwhelmingException. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8798) don't throw TombstoneOverwhelmingException during bootstrap
[
https://issues.apache.org/jira/browse/CASSANDRA-8798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14505550#comment-14505550
]
Jeff Jirsa commented on CASSANDRA-8798:
---
[~aweisberg] - Yes, respectTombstoneThresholds() would also effectively disable
the threshold during other streaming cases (such as rebuild and repair), even
outside of bootstrapping. The thought process was that if you added another DC
to the cluster, best practices (such as
http://docs.datastax.com/en/cassandra/2.0/cassandra/operations/ops_add_dc_to_cluster_t.html
) will cause you to bootstrap and then, once all new nodes in the new DC were
online, you'd issue a rebuild. That rebuild could fail due to
TombstoneOverwhelmingException in the same way (when building indexes for data
streamed over).
It is a scary patch, I agree, which is why I wanted someone else to agree with
the approach in theory before I wrote a full test. I'm open to other
suggestions, though.
> don't throw TombstoneOverwhelmingException during bootstrap
> ---
>
> Key: CASSANDRA-8798
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8798
> Project: Cassandra
> Issue Type: Bug
> Components: Core
>Reporter: mck
>Assignee: Jeff Jirsa
> Fix For: 2.0.15
>
> Attachments: 8798.txt
>
>
> During bootstrap honouring tombstone_failure_threshold seems
> counter-productive as the node is not serving requests so not protecting
> anything.
> Instead what happens is bootstrap fails, and a cluster that obviously needs
> an extra node isn't getting it...
> **History**
> When adding a new node bootstrap process looks complete in that streaming is
> finished, compactions finished, and all disk and cpu activity is calm.
> But the node is still stuck in "joining" status.
> The last stage in the bootstrapping process is the rebuilding of secondary
> indexes. grepping the logs confirmed it failed during this stage.
> {code}grep SecondaryIndexManager cassandra/logs/*{code}
> To see what secondary index rebuilding was initiated
> {code}
> grep "index build of " cassandra/logs/* | awk -F" for data in " '{print $1}'
> INFO 13:18:11,252 Submitting index build of addresses.unobfuscatedIndex
> INFO 13:18:11,352 Submitting index build of Inbox.FINNBOXID_INDEX
> INFO 23:03:54,758 Submitting index build of [events.collected_tbIndex,
> events.real_tbIndex]
> {code}
> To get an idea of successful secondary index rebuilding
> {code}grep "Index build of "cassandra/logs/*
> INFO 13:18:11,263 Index build of addresses.unobfuscatedIndex complete
> INFO 13:18:11,355 Index build of Inbox.FINNBOXID_INDEX complete
> {code}
> Looking closer at {{[events.collected_tbIndex, events.real_tbIndex]}} showed
> the following stacktrace
> {code}
> ERROR [StreamReceiveTask:121] 2015-02-12 05:54:47,768 CassandraDaemon.java
> (line 199) Exception in thread Thread[StreamReceiveTask:121,5,main]
> java.lang.RuntimeException: java.util.concurrent.ExecutionException:
> java.lang.RuntimeException:
> org.apache.cassandra.db.filter.TombstoneOverwhelmingException
> at
> org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:413)
> at
> org.apache.cassandra.db.index.SecondaryIndexManager.maybeBuildSecondaryIndexes(SecondaryIndexManager.java:142)
> at
> org.apache.cassandra.streaming.StreamReceiveTask$OnCompletionRunnable.run(StreamReceiveTask.java:130)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.util.concurrent.ExecutionException:
> java.lang.RuntimeException:
> org.apache.cassandra.db.filter.TombstoneOverwhelmingException
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:188)
> at
> org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:409)
> ... 7 more
> Caused by: java.lang.RuntimeException:
> org.apache.cassandra.db.filter.TombstoneOverwhelmingException
> at
> org.apache.cassandra.service.pager.QueryPagers$1.next(QueryPagers.java:160)
> at
> org.apache.cassandra.service.pager.QueryPagers$1.next(QueryPagers.java:143)
> at org.apache.cassandra.db.Keyspace.indexRow(Keyspace.java:406)
> at
> org.apache.cassandra.db.index.SecondaryIndexBuilder.build(SecondaryIndexBuilder.java:62)
> at
> org.apache.cassandra.db.compaction.CompactionManager$9.run(CompactionManager.java:834)
> ... 5 mo
[jira] [Updated] (CASSANDRA-9198) Deleting from an empty list produces an error
[
https://issues.apache.org/jira/browse/CASSANDRA-9198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jeff Jirsa updated CASSANDRA-9198:
--
Attachment: 9198-v2-trunk.txt
9198-v2-2.1.txt
> Deleting from an empty list produces an error
> -
>
> Key: CASSANDRA-9198
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9198
> Project: Cassandra
> Issue Type: Bug
> Components: API
>Reporter: Olivier Michallat
>Assignee: Jeff Jirsa
>Priority: Minor
> Fix For: 2.1.5
>
> Attachments: 9198-v2-2.1.txt, 9198-v2-trunk.txt, 9198.txt
>
>
> While deleting an element from a list that does not contain it is a no-op,
> deleting it from an empty list causes an error.
> This edge case is a bit inconsistent, because it makes list deletion non
> idempotent:
> {code}
> cqlsh:test> create table foo (k int primary key, v list);
> cqlsh:test> insert into foo(k,v) values (1, [1,2]);
> cqlsh:test> update foo set v = v - [1] where k = 1;
> cqlsh:test> update foo set v = v - [1] where k = 1;
> cqlsh:test> update foo set v = v - [2] where k = 1;
> cqlsh:test> update foo set v = v - [2] where k = 1;
> InvalidRequest: code=2200 [Invalid query] message="Attempted to delete an
> element from a list which is null"
> {code}
> With speculative retries coming to the drivers, idempotency becomes more
> important because it determines which query we might retry or not. So it
> would be better if deleting from an empty list succeeded.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9198) Deleting from an empty list produces an error
[
https://issues.apache.org/jira/browse/CASSANDRA-9198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14505622#comment-14505622
]
Jeff Jirsa commented on CASSANDRA-9198:
---
[~blerer] - sincere apologies, I originally meant to just link to the patch
from 9077, then figured "why not at least make tests pass" for your
convenience. Didn't mean to step on toes.
Attaching two diffs, one for trunk and a second for 2.1.
I adjusted the set/delete logic to give the same messaging as in 9077, and
cleaned up the ugly if/else return in favor of a ternary.
{noformat}
cqlsh> create keyspace IF NOT EXISTS test WITH replication = {'class':
'SimpleStrategy', 'replication_factor': 1};
cqlsh> USE test;
cqlsh:test> create table foo (k int primary key, v list);
cqlsh:test> insert into foo(k,v) values (1, [1,2]);
cqlsh:test> update foo set v = v - [1] where k = 1;
cqlsh:test> update foo set v = v - [1] where k = 1;
cqlsh:test> update foo set v = v - [2] where k = 1;
cqlsh:test> update foo set v = v - [2] where k = 1;
cqlsh:test>
cqlsh:test> update foo set v[0] = 2 where k=1;
InvalidRequest: code=2200 [Invalid query] message="Attempted to set an element
on a list which is null"
cqlsh:test> delete v[0] from foo where k=1;
InvalidRequest: code=2200 [Invalid query] message="Attempted to delete an
element from a list which is null"
{noformat}
> Deleting from an empty list produces an error
> -
>
> Key: CASSANDRA-9198
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9198
> Project: Cassandra
> Issue Type: Bug
> Components: API
>Reporter: Olivier Michallat
>Assignee: Jeff Jirsa
>Priority: Minor
> Fix For: 2.1.5
>
> Attachments: 9198-v2-2.1.txt, 9198-v2-trunk.txt, 9198.txt
>
>
> While deleting an element from a list that does not contain it is a no-op,
> deleting it from an empty list causes an error.
> This edge case is a bit inconsistent, because it makes list deletion non
> idempotent:
> {code}
> cqlsh:test> create table foo (k int primary key, v list);
> cqlsh:test> insert into foo(k,v) values (1, [1,2]);
> cqlsh:test> update foo set v = v - [1] where k = 1;
> cqlsh:test> update foo set v = v - [1] where k = 1;
> cqlsh:test> update foo set v = v - [2] where k = 1;
> cqlsh:test> update foo set v = v - [2] where k = 1;
> InvalidRequest: code=2200 [Invalid query] message="Attempted to delete an
> element from a list which is null"
> {code}
> With speculative retries coming to the drivers, idempotency becomes more
> important because it determines which query we might retry or not. So it
> would be better if deleting from an empty list succeeded.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-12975) Exception (java.lang.RuntimeException) encountered during startup: org.codehaus.jackson.JsonParseException:
[
https://issues.apache.org/jira/browse/CASSANDRA-12975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jeff Jirsa updated CASSANDRA-12975:
---
Description:
We upgrade our thrift tables from v1.2.4 --> 1.2.9 --> 2.0.9 --> v2.1.13
without any problems, but when upgrade it to v3.9 something wrong.
any help?
{code}
[root@bj-dev-infra-001 cassandra]# apache-cassandra-3.9/bin/cassandra -R -f
CompilerOracle: dontinline
org/apache/cassandra/db/Columns$Serializer.deserializeLargeSubset
(Lorg/apache/cassandra/io/util/DataInputPlus;Lorg/apache/cassandra/db/Columns;I)Lorg/apache/cassandra/db/Columns;
CompilerOracle: dontinline
org/apache/cassandra/db/Columns$Serializer.serializeLargeSubset
(Ljava/util/Collection;ILorg/apache/cassandra/db/Columns;ILorg/apache/cassandra/io/util/DataOutputPlus;)V
CompilerOracle: dontinline
org/apache/cassandra/db/Columns$Serializer.serializeLargeSubsetSize
(Ljava/util/Collection;ILorg/apache/cassandra/db/Columns;I)I
CompilerOracle: dontinline
org/apache/cassandra/db/transform/BaseIterator.tryGetMoreContents ()Z
CompilerOracle: dontinline
org/apache/cassandra/db/transform/StoppingTransformation.stop ()V
CompilerOracle: dontinline
org/apache/cassandra/db/transform/StoppingTransformation.stopInPartition ()V
CompilerOracle: dontinline
org/apache/cassandra/io/util/BufferedDataOutputStreamPlus.doFlush (I)V
CompilerOracle: dontinline
org/apache/cassandra/io/util/BufferedDataOutputStreamPlus.writeExcessSlow ()V
CompilerOracle: dontinline
org/apache/cassandra/io/util/BufferedDataOutputStreamPlus.writeSlow (JI)V
CompilerOracle: dontinline
org/apache/cassandra/io/util/RebufferingInputStream.readPrimitiveSlowly (I)J
CompilerOracle: inline
org/apache/cassandra/db/rows/UnfilteredSerializer.serializeRowBody
(Lorg/apache/cassandra/db/rows/Row;ILorg/apache/cassandra/db/SerializationHeader;Lorg/apache/cassandra/io/util/DataOutputPlus;)V
CompilerOracle: inline org/apache/cassandra/io/util/Memory.checkBounds (JJ)V
CompilerOracle: inline org/apache/cassandra/io/util/SafeMemory.checkBounds (JJ)V
CompilerOracle: inline
org/apache/cassandra/utils/AsymmetricOrdering.selectBoundary
(Lorg/apache/cassandra/utils/AsymmetricOrdering/Op;II)I
CompilerOracle: inline
org/apache/cassandra/utils/AsymmetricOrdering.strictnessOfLessThan
(Lorg/apache/cassandra/utils/AsymmetricOrdering/Op;)I
CompilerOracle: inline org/apache/cassandra/utils/BloomFilter.indexes
(Lorg/apache/cassandra/utils/IFilter/FilterKey;)[J
CompilerOracle: inline org/apache/cassandra/utils/BloomFilter.setIndexes
(JJIJ[J)V
CompilerOracle: inline org/apache/cassandra/utils/ByteBufferUtil.compare
(Ljava/nio/ByteBuffer;[B)I
CompilerOracle: inline org/apache/cassandra/utils/ByteBufferUtil.compare
([BLjava/nio/ByteBuffer;)I
CompilerOracle: inline
org/apache/cassandra/utils/ByteBufferUtil.compareUnsigned
(Ljava/nio/ByteBuffer;Ljava/nio/ByteBuffer;)I
CompilerOracle: inline
org/apache/cassandra/utils/FastByteOperations$UnsafeOperations.compareTo
(Ljava/lang/Object;JILjava/lang/Object;JI)I
CompilerOracle: inline
org/apache/cassandra/utils/FastByteOperations$UnsafeOperations.compareTo
(Ljava/lang/Object;JILjava/nio/ByteBuffer;)I
CompilerOracle: inline
org/apache/cassandra/utils/FastByteOperations$UnsafeOperations.compareTo
(Ljava/nio/ByteBuffer;Ljava/nio/ByteBuffer;)I
CompilerOracle: inline org/apache/cassandra/utils/vint/VIntCoding.encodeVInt
(JI)[B
INFO 06:05:28 Configuration location:
file:/usr/local/cassandra/apache-cassandra-3.9/conf/cassandra.yaml
INFO 06:05:28 Node configuration:[allocate_tokens_for_keyspace=null;
authenticator=AllowAllAuthenticator; authorizer=AllowAllAuthorizer;
auto_bootstrap=true; auto_snapshot=true; batch_size_fail_threshold_in_kb=50;
batch_size_warn_threshold_in_kb=5; batchlog_replay_throttle_in_kb=1024;
broadcast_address=null; broadcast_rpc_address=null;
buffer_pool_use_heap_if_exhausted=true; cas_contention_timeout_in_ms=1000;
cdc_enabled=false; cdc_free_space_check_interval_ms=250;
cdc_raw_directory=/usr/local/cassandra/data/cdc_raw;
cdc_total_space_in_mb=null; client_encryption_options=;
cluster_name=TestCluster; column_index_cache_size_in_kb=2;
column_index_size_in_kb=64; commit_failure_policy=stop;
commitlog_compression=null;
commitlog_directory=/usr/local/cassandra/data/commitlog;
commitlog_max_compression_buffers_in_pool=3; commitlog_periodic_queue_size=-1;
commitlog_segment_size_in_mb=32; commitlog_sync=periodic;
commitlog_sync_batch_window_in_ms=null; commitlog_sync_period_in_ms=1;
commitlog_total_space_in_mb=null;
compaction_large_partition_warning_threshold_mb=100;
compaction_throughput_mb_per_sec=0; concurrent_compactors=20;
concurrent_counter_writes=32; concurrent_materialized_view_writes=32;
concurrent_reads=32; concurrent_replicates=null; concurrent_writes=32;
counter_cache_keys_to_save=2147483647; counter_cache_save_period=7200;
counter_cache_size_in_mb=null; counter_write_request_timeout_in
[jira] [Updated] (CASSANDRA-13000) slow query log analysis tool
[
https://issues.apache.org/jira/browse/CASSANDRA-13000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jeff Jirsa updated CASSANDRA-13000:
---
Component/s: Observability
> slow query log analysis tool
>
>
> Key: CASSANDRA-13000
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13000
> Project: Cassandra
> Issue Type: New Feature
> Components: Observability
>Reporter: Jon Haddad
>
> As a follow up to CASSANDRA-12403, it would be very helpful to have a tool to
> process the slow queries that are logged. In the MySQL world, there's a tool
> called mysqldumpslow, which processes a slow query log, abstracts the
> parameters to prepared statements, and shows the queries which are causing
> problems based on frequency. The {{mysqldumpslow}} utillity shows an
> aggregated count & time statistics spent on slow queries. For instance:
> {code}shell> mysqldumpslow
> Reading mysql slow query log from
> /usr/local/mysql/data/mysqld51-apple-slow.log
> Count: 1 Time=4.32s (4s) Lock=0.00s (0s) Rows=0.0 (0), root[root]@localhost
> insert into t2 select * from t1
> Count: 3 Time=2.53s (7s) Lock=0.00s (0s) Rows=0.0 (0), root[root]@localhost
> insert into t2 select * from t1 limit N
> Count: 3 Time=2.13s (6s) Lock=0.00s (0s) Rows=0.0 (0), root[root]@localhost
> insert into t1 select * from t1{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-13005) Cassandra TWCS is not removing fully expired tables
[ https://issues.apache.org/jira/browse/CASSANDRA-13005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15726936#comment-15726936 ] Jeff Jirsa commented on CASSANDRA-13005: I'm unsure how you're suddenly missing components. However, a lot of factors go into what can/cant be safely deleted. One of the most common reasons TWCS doesn't drop a fully expired table is overlapping files. There exists a tool - sstableexpiredblockers - meant to help diagnose such a problem. Are you able to run it on that table? > Cassandra TWCS is not removing fully expired tables > --- > > Key: CASSANDRA-13005 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13005 > Project: Cassandra > Issue Type: Bug > Components: Compaction > Environment: Cassandra 3.0.9 > Java HotSpot(TM) 64-Bit Server VM version 25.112-b15 (Java version > 1.8.0_112-b15) > Linux 3.16 >Reporter: Christian Esken > Attachments: sstablemetadata-empty-type-that-is-3GB.txt > > > I have a table where all columns are stored with TTL of maximum 4 hours. > Usually TWCS compaction properly removes expired data via tombstone > compaction and also removes fully expired tables. The number of SSTables is > nearly constant since weeks. Good. > The problem: Suddenly TWCS does not remove old SSTables any longer. They are > being recreated frequently (judging form the file creation timestamp), but > the number of tables is growing. Analysis and actions take so far: > - sstablemetadata shows strange data, as if the table is completely empty. > - sstabledump throws an Exception when running it on such a SSTable > - Even triggering a manual major compaction will not remove the old > SSTable's. To be more precise: They are recreated with new id and timestamp > (not sure whether they are identical as I cannot inspect content due to the > sstabledump crash) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-13005) Cassandra TWCS is not removing fully expired tables
[ https://issues.apache.org/jira/browse/CASSANDRA-13005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Jirsa updated CASSANDRA-13005: --- Labels: twcs (was: ) > Cassandra TWCS is not removing fully expired tables > --- > > Key: CASSANDRA-13005 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13005 > Project: Cassandra > Issue Type: Bug > Components: Compaction > Environment: Cassandra 3.0.9 > Java HotSpot(TM) 64-Bit Server VM version 25.112-b15 (Java version > 1.8.0_112-b15) > Linux 3.16 >Reporter: Christian Esken > Labels: twcs > Attachments: sstablemetadata-empty-type-that-is-3GB.txt > > > I have a table where all columns are stored with TTL of maximum 4 hours. > Usually TWCS compaction properly removes expired data via tombstone > compaction and also removes fully expired tables. The number of SSTables is > nearly constant since weeks. Good. > The problem: Suddenly TWCS does not remove old SSTables any longer. They are > being recreated frequently (judging form the file creation timestamp), but > the number of tables is growing. Analysis and actions take so far: > - sstablemetadata shows strange data, as if the table is completely empty. > - sstabledump throws an Exception when running it on such a SSTable > - Even triggering a manual major compaction will not remove the old > SSTable's. To be more precise: They are recreated with new id and timestamp > (not sure whether they are identical as I cannot inspect content due to the > sstabledump crash) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9779) Append-only optimization
[
https://issues.apache.org/jira/browse/CASSANDRA-9779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15729356#comment-15729356
]
Jeff Jirsa commented on CASSANDRA-9779:
---
Old stale ticket, but fwiw re: [~slebresne] 's comment on dtcs (/twcs) -
This ticket would enable modifying
{{CompactionController.getFullyExpiredSSTables()}} to be more closely aligned
with what people expect it to do, eliminating the need for
{{sstableexpiredblockers}} and the like. Performance wise, its probably a minor
gain, but operationally it's probably a big net gain in terms of least-surprise.
> Append-only optimization
>
>
> Key: CASSANDRA-9779
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9779
> Project: Cassandra
> Issue Type: New Feature
> Components: CQL
>Reporter: Jonathan Ellis
> Fix For: 3.x
>
>
> Many common workloads are append-only: that is, they insert new rows but do
> not update existing ones. However, Cassandra has no way to infer this and so
> it must treat all tables as if they may experience updates in the future.
> If we added syntax to tell Cassandra about this ({{WITH INSERTS ONLY}} for
> instance) then we could do a number of optimizations:
> - Compaction would only need to worry about defragmenting partitions, not
> rows. We could default to DTCS or similar.
> - CollationController could stop scanning sstables as soon as it finds a
> matching row
> - Most importantly, materialized views wouldn't need to worry about deleting
> prior values, which would eliminate the majority of the MV overhead
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (CASSANDRA-13018) Exceptions encountered calling getSeeds() breaks messaging service
Jeff Jirsa created CASSANDRA-13018: -- Summary: Exceptions encountered calling getSeeds() breaks messaging service Key: CASSANDRA-13018 URL: https://issues.apache.org/jira/browse/CASSANDRA-13018 Project: Cassandra Issue Type: Bug Components: Streaming and Messaging Reporter: Jeff Jirsa OutboundTcpConnection.connect() calls getSeeds(). If getSeeds() throws an exception (for example, DD/Config invalid yaml error), messaging thread(s) break(s). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (CASSANDRA-13018) Exceptions encountered calling getSeeds() breaks messaging service
[ https://issues.apache.org/jira/browse/CASSANDRA-13018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Jirsa reassigned CASSANDRA-13018: -- Assignee: Jeff Jirsa > Exceptions encountered calling getSeeds() breaks messaging service > -- > > Key: CASSANDRA-13018 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13018 > Project: Cassandra > Issue Type: Bug > Components: Streaming and Messaging >Reporter: Jeff Jirsa >Assignee: Jeff Jirsa > > OutboundTcpConnection.connect() calls getSeeds(). If getSeeds() throws an > exception (for example, DD/Config invalid yaml error), messaging thread(s) > break(s). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-13018) Exceptions encountered calling getSeeds() breaks messaging service
[ https://issues.apache.org/jira/browse/CASSANDRA-13018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Jirsa updated CASSANDRA-13018: --- Assignee: (was: Jeff Jirsa) > Exceptions encountered calling getSeeds() breaks messaging service > -- > > Key: CASSANDRA-13018 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13018 > Project: Cassandra > Issue Type: Bug > Components: Streaming and Messaging >Reporter: Jeff Jirsa > > OutboundTcpConnection.connect() calls getSeeds(). If getSeeds() throws an > exception (for example, DD/Config invalid yaml error), messaging thread(s) > break(s). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-13018) Exceptions encountered calling getSeeds() breaks messaging service
[ https://issues.apache.org/jira/browse/CASSANDRA-13018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Jirsa updated CASSANDRA-13018: --- Labels: lhf (was: ) > Exceptions encountered calling getSeeds() breaks messaging service > -- > > Key: CASSANDRA-13018 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13018 > Project: Cassandra > Issue Type: Bug > Components: Streaming and Messaging >Reporter: Jeff Jirsa > Labels: lhf > > OutboundTcpConnection.connect() calls getSeeds(). If getSeeds() throws an > exception (for example, DD/Config invalid yaml error), messaging thread(s) > break(s). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (CASSANDRA-13018) Exceptions encountered calling getSeeds() breaks messaging service
[ https://issues.apache.org/jira/browse/CASSANDRA-13018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Jirsa reassigned CASSANDRA-13018: -- Assignee: Jeff Jirsa > Exceptions encountered calling getSeeds() breaks messaging service > -- > > Key: CASSANDRA-13018 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13018 > Project: Cassandra > Issue Type: Bug > Components: Streaming and Messaging >Reporter: Jeff Jirsa >Assignee: Jeff Jirsa > Labels: lhf > > OutboundTcpConnection.connect() calls getSeeds(). If getSeeds() throws an > exception (for example, DD/Config invalid yaml error), messaging thread(s) > break(s). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-12676) Message coalescing regression
[ https://issues.apache.org/jira/browse/CASSANDRA-12676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Jirsa updated CASSANDRA-12676: --- Assignee: Jeff Jirsa Reviewer: T Jake Luciani Status: Patch Available (was: Open) > Message coalescing regression > - > > Key: CASSANDRA-12676 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12676 > Project: Cassandra > Issue Type: Bug >Reporter: T Jake Luciani >Assignee: Jeff Jirsa > Labels: docs-impacting > Fix For: 4.0, 3.x > > Attachments: 12676.diff, result.html > > > The default in 2.2+ was to enable TIMEHORIZON message coalescing. After > reports of performance regressions after upgrading from 2.1 to 2.2/3.0 we > have discovered the issue to be this default. > We need to re-test our assumptions on this feature but in the meantime we > should default back to disabled. > Here is a performance run [with and without message > coalescing|http://cstar.datastax.com/graph?command=one_job&stats=9a26b5f2-7f48-11e6-92e7-0256e416528f&metric=op_rate&operation=2_user&smoothing=1&show_aggregates=true&xmin=0&xmax=508.86&ymin=0&ymax=91223] > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-12676) Message coalescing regression
[ https://issues.apache.org/jira/browse/CASSANDRA-12676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Jirsa updated CASSANDRA-12676: --- Attachment: 12676.diff [~tjake] - ok with this for a NEWS.txt entry? Targeting 3.12 (so skipping 3.11 branch), which would of course include 4.0. > Message coalescing regression > - > > Key: CASSANDRA-12676 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12676 > Project: Cassandra > Issue Type: Bug >Reporter: T Jake Luciani > Labels: docs-impacting > Fix For: 4.0, 3.x > > Attachments: 12676.diff, result.html > > > The default in 2.2+ was to enable TIMEHORIZON message coalescing. After > reports of performance regressions after upgrading from 2.1 to 2.2/3.0 we > have discovered the issue to be this default. > We need to re-test our assumptions on this feature but in the meantime we > should default back to disabled. > Here is a performance run [with and without message > coalescing|http://cstar.datastax.com/graph?command=one_job&stats=9a26b5f2-7f48-11e6-92e7-0256e416528f&metric=op_rate&operation=2_user&smoothing=1&show_aggregates=true&xmin=0&xmax=508.86&ymin=0&ymax=91223] > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-13018) Exceptions encountered calling getSeeds() breaks messaging service
[ https://issues.apache.org/jira/browse/CASSANDRA-13018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Jirsa updated CASSANDRA-13018: --- Attachment: 0001-Better-handle-config-errors-during-outbound-connecti.patch > Exceptions encountered calling getSeeds() breaks messaging service > -- > > Key: CASSANDRA-13018 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13018 > Project: Cassandra > Issue Type: Bug > Components: Streaming and Messaging >Reporter: Jeff Jirsa >Assignee: Jeff Jirsa > Labels: lhf > Attachments: > 0001-Better-handle-config-errors-during-outbound-connecti.patch > > > OutboundTcpConnection.connect() calls getSeeds(). If getSeeds() throws an > exception (for example, DD/Config invalid yaml error), messaging thread(s) > break(s). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-13079) Repair doesn't work after several replication factor changes
[
https://issues.apache.org/jira/browse/CASSANDRA-13079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15793609#comment-15793609
]
Jeff Jirsa commented on CASSANDRA-13079:
{quote}And this rise a question - shouldn't replication factor change also
reset repair state for this keyspace?{quote}
If replication factor is increased, it seems like it should, in fact, reset the
repair state for that keyspace. The fact that we don't is probably a bug.
{quote}
I think it would be a good idea for this type of scenario to change the repair
state during replication altering, but I'm not sure if that's always the case.
{quote}
The principle of least astonishment applies here - a user running repair should
expect all data to be repaired, and a user who adds a new DC and then runs
repair will see a lot of data streamed. That's not something that SHOULD
surprise a user. They can work around it if they choose. The fact that
incremental (default) repair doesn't do anything if you change from rf=1 to
rf=2 is more surprising and dangerous than extra streaming, so I imagine we
should consider that more important.
> Repair doesn't work after several replication factor changes
>
>
> Key: CASSANDRA-13079
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13079
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: Debian
>Reporter: Vladimir Yudovin
>Priority: Critical
>
> Scenario:
> Start two nodes cluster.
> Create keyspace with rep.factor *one*:
> CREATE KEYSPACE rep WITH replication = {'class': 'SimpleStrategy',
> 'replication_factor': 1};
> CREATE TABLE rep.data (str text PRIMARY KEY );
> INSERT INTO rep.data (str) VALUES ( 'qwerty');
> Run *nodetool flush* on all nodes. On one of them table files are created.
> Change replication factor to *two*:
> ALTER KEYSPACE rep WITH replication = {'class': 'SimpleStrategy',
> 'replication_factor': 2};
> Run repair, then *nodetool flush* on all nodes. On all nodes table files are
> created.
> Change replication factor to *one*:
> ALTER KEYSPACE rep WITH replication = {'class': 'SimpleStrategy',
> 'replication_factor': 1};
> Then *nodetool cleanup*, only on initial node remained data files.
> Change replication factor to *two* again:
> ALTER KEYSPACE rep WITH replication = {'class': 'SimpleStrategy',
> 'replication_factor': 2};
> Run repair, then *nodetool flush* on all nodes. No data files on second node
> (though expected, as after first repair/flush).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-13060) NPE in StorageService.java while bootstrapping
[
https://issues.apache.org/jira/browse/CASSANDRA-13060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15793636#comment-15793636
]
Jeff Jirsa commented on CASSANDRA-13060:
If the thought is that this is caused by CASSANDRA-12653 , it seems like a
reasonable extra guard in addition to your work on 12653, right
[~spo...@gmail.com] ?
If it's not, then it seems like understanding how we get in this state is worth
investigating?
> NPE in StorageService.java while bootstrapping
> --
>
> Key: CASSANDRA-13060
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13060
> Project: Cassandra
> Issue Type: Bug
>Reporter: Jay Zhuang
>Assignee: Jay Zhuang
>Priority: Minor
> Labels: lhf
> Fix For: 3.0.x
>
> Attachments: 13060-3.0.txt
>
>
> Lots of NPE happens when bootstrapping new node:
> {code}
> WARN [SharedPool-Worker-1] 2016-12-19 23:09:09,034
> AbstractLocalAwareExecutorService.java:169 - Uncaught exception on thread
> Thread[SharedPool-Worker-1,5,main]: {}
> java.lang.NullPointerException: null
> at
> org.apache.cassandra.service.StorageService.isRpcReady(StorageService.java:1829)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.service.StorageService.notifyUp(StorageService.java:1787)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.service.StorageService.onAlive(StorageService.java:2424)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at org.apache.cassandra.gms.Gossiper.realMarkAlive(Gossiper.java:999)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at org.apache.cassandra.gms.Gossiper$3.response(Gossiper.java:979)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:53)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:67)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[na:1.8.0_111]
> at
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164)
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:136)
> [apache-cassandra-3.0.10.jar:3.0.10]
> at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105)
> [apache-cassandra-3.0.10.jar:3.0.10]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-13038) 33% of compaction time spent in StreamingHistogram.update()
[
https://issues.apache.org/jira/browse/CASSANDRA-13038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15797119#comment-15797119
]
Jeff Jirsa commented on CASSANDRA-13038:
Rounding up to the next hour may be suitable for special cases (including your
use case), and you may be willing to tolerate that until the streaming
histogram is more efficient, but is not reasonable for the general population.
> 33% of compaction time spent in StreamingHistogram.update()
> ---
>
> Key: CASSANDRA-13038
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13038
> Project: Cassandra
> Issue Type: Bug
> Components: Compaction
>Reporter: Corentin Chary
>Assignee: Corentin Chary
> Attachments: compaction-speedup.patch,
> compaction-streaminghistrogram.png, profiler-snapshot.nps
>
>
> With the following table, that contains a *lot* of cells:
> {code}
> CREATE TABLE biggraphite.datapoints_11520p_60s (
> metric uuid,
> time_start_ms bigint,
> offset smallint,
> count int,
> value double,
> PRIMARY KEY ((metric, time_start_ms), offset)
> ) WITH CLUSTERING ORDER BY (offset DESC);
> AND compaction = {'class':
> 'org.apache.cassandra.db.compaction.TimeWindowCompactionStrategy',
> 'compaction_window_size': '6', 'compaction_window_unit': 'HOURS',
> 'max_threshold': '32', 'min_threshold': '6'}
> Keyspace : biggraphite
> Read Count: 1822
> Read Latency: 1.8870054884742042 ms.
> Write Count: 2212271647
> Write Latency: 0.027705127678653473 ms.
> Pending Flushes: 0
> Table: datapoints_11520p_60s
> SSTable count: 47
> Space used (live): 300417555945
> Space used (total): 303147395017
> Space used by snapshots (total): 0
> Off heap memory used (total): 207453042
> SSTable Compression Ratio: 0.4955200053039823
> Number of keys (estimate): 16343723
> Memtable cell count: 220576
> Memtable data size: 17115128
> Memtable off heap memory used: 0
> Memtable switch count: 2872
> Local read count: 0
> Local read latency: NaN ms
> Local write count: 1103167888
> Local write latency: 0.025 ms
> Pending flushes: 0
> Percent repaired: 0.0
> Bloom filter false positives: 0
> Bloom filter false ratio: 0.0
> Bloom filter space used: 105118296
> Bloom filter off heap memory used: 106547192
> Index summary off heap memory used: 27730962
> Compression metadata off heap memory used: 73174888
> Compacted partition minimum bytes: 61
> Compacted partition maximum bytes: 51012
> Compacted partition mean bytes: 7899
> Average live cells per slice (last five minutes): NaN
> Maximum live cells per slice (last five minutes): 0
> Average tombstones per slice (last five minutes): NaN
> Maximum tombstones per slice (last five minutes): 0
> Dropped Mutations: 0
> {code}
> It looks like a good chunk of the compaction time is lost in
> StreamingHistogram.update() (which is used to store the estimated tombstone
> drop times).
> This could be caused by a huge number of different deletion times which would
> makes the bin huge but it this histogram should be capped to 100 keys. It's
> more likely caused by the huge number of cells.
> A simple solutions could be to only take into accounts part of the cells, the
> fact the this table has a TWCS also gives us an additional hint that sampling
> deletion times would be fine.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-13038) 33% of compaction time spent in StreamingHistogram.update()
[
https://issues.apache.org/jira/browse/CASSANDRA-13038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15798631#comment-15798631
]
Jeff Jirsa commented on CASSANDRA-13038:
[~benedict] offered one suggestion - we can make {{StreamingHistogram}} far
more efficient.
As written, it creates 100 buckets, and each time it sees a tombstone drop time
outside of one of those buckets, it merges two adjacent buckets to create a new
bucket.
You're correct in noting that compacting on high throughput, TTL-only systems
will probably see that bucket merging happen on nearly every cell once we get
past 100 cells - so you need an algorithmic change that avoids the constant
merges. For example, [rather than merging every time we overflow the max
size|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/utils/StreamingHistogram.java#L98-L99],
allow {{StreamingHistogram}} to grow to more than {{maxBinSize}} and only
reduce/shrink/merge it at a later time (like right before we use/serialize it,
perhaps with an additional upper threshold, to avoid it from growing unbounded
prior to the final trim).
> 33% of compaction time spent in StreamingHistogram.update()
> ---
>
> Key: CASSANDRA-13038
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13038
> Project: Cassandra
> Issue Type: Bug
> Components: Compaction
>Reporter: Corentin Chary
>Assignee: Corentin Chary
> Attachments: compaction-speedup.patch,
> compaction-streaminghistrogram.png, profiler-snapshot.nps
>
>
> With the following table, that contains a *lot* of cells:
> {code}
> CREATE TABLE biggraphite.datapoints_11520p_60s (
> metric uuid,
> time_start_ms bigint,
> offset smallint,
> count int,
> value double,
> PRIMARY KEY ((metric, time_start_ms), offset)
> ) WITH CLUSTERING ORDER BY (offset DESC);
> AND compaction = {'class':
> 'org.apache.cassandra.db.compaction.TimeWindowCompactionStrategy',
> 'compaction_window_size': '6', 'compaction_window_unit': 'HOURS',
> 'max_threshold': '32', 'min_threshold': '6'}
> Keyspace : biggraphite
> Read Count: 1822
> Read Latency: 1.8870054884742042 ms.
> Write Count: 2212271647
> Write Latency: 0.027705127678653473 ms.
> Pending Flushes: 0
> Table: datapoints_11520p_60s
> SSTable count: 47
> Space used (live): 300417555945
> Space used (total): 303147395017
> Space used by snapshots (total): 0
> Off heap memory used (total): 207453042
> SSTable Compression Ratio: 0.4955200053039823
> Number of keys (estimate): 16343723
> Memtable cell count: 220576
> Memtable data size: 17115128
> Memtable off heap memory used: 0
> Memtable switch count: 2872
> Local read count: 0
> Local read latency: NaN ms
> Local write count: 1103167888
> Local write latency: 0.025 ms
> Pending flushes: 0
> Percent repaired: 0.0
> Bloom filter false positives: 0
> Bloom filter false ratio: 0.0
> Bloom filter space used: 105118296
> Bloom filter off heap memory used: 106547192
> Index summary off heap memory used: 27730962
> Compression metadata off heap memory used: 73174888
> Compacted partition minimum bytes: 61
> Compacted partition maximum bytes: 51012
> Compacted partition mean bytes: 7899
> Average live cells per slice (last five minutes): NaN
> Maximum live cells per slice (last five minutes): 0
> Average tombstones per slice (last five minutes): NaN
> Maximum tombstones per slice (last five minutes): 0
> Dropped Mutations: 0
> {code}
> It looks like a good chunk of the compaction time is lost in
> StreamingHistogram.update() (which is used to store the estimated tombstone
> drop times).
> This could be caused by a huge number of different deletion times which would
> makes the bin huge but it this histogram should be capped to 100 keys. It's
> more likely caused by the huge number of cells.
> A simple solutions could be to only take into accounts part of the cells, the
> fact the this table has a TWCS also gives us an additional hint that sampling
> deletion times would be fine.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-13115) Read repair is not blocking repair to finish in foreground repair
[
https://issues.apache.org/jira/browse/CASSANDRA-13115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jeff Jirsa updated CASSANDRA-13115:
---
Description:
The code trying to wait(block) for repair result to come back in 3.X is below:
{code}
public void close()
{
try
{
FBUtilities.waitOnFutures(repairResults,
DatabaseDescriptor.getWriteRpcTimeout());
}
catch (TimeoutException ex)
{
// We got all responses, but timed out while repairing
int blockFor = consistency.blockFor(keyspace);
if (Tracing.isTracing())
Tracing.trace("Timed out while read-repairing after
receiving all {} data and digest responses", blockFor);
else
logger.debug("Timeout while read-repairing after receiving
all {} data and digest responses", blockFor);
throw new ReadTimeoutException(consistency, blockFor-1,
blockFor, true);
}
}
{code}
in DataResolver class, but this close method is never called and it's also not
auto close(RepairMergeListener is not extending from
AutoCloseable/CloseableIterator) which means we never wait for repair to finish
before returning final result.
The steps to reproduce:
1. create some keyspace/table with RF = 2
2. start 2 nodes using ccm
3. stop node2
4. disable node1 hinted hand off
5. write some data to node1 with consistency level one
6. start node2
7. query some data from node1
This should trigger read repair. I put some log in above close method, and can
not see log print put.
So this bug will basically violate "monotonic quorum reads " guarantee.
was:
The code trying to wait(block) for repair result to come back in 3.X is below:
public void close()
{
try
{
FBUtilities.waitOnFutures(repairResults,
DatabaseDescriptor.getWriteRpcTimeout());
}
catch (TimeoutException ex)
{
// We got all responses, but timed out while repairing
int blockFor = consistency.blockFor(keyspace);
if (Tracing.isTracing())
Tracing.trace("Timed out while read-repairing after
receiving all {} data and digest responses", blockFor);
else
logger.debug("Timeout while read-repairing after receiving
all {} data and digest responses", blockFor);
throw new ReadTimeoutException(consistency, blockFor-1,
blockFor, true);
}
}
in DataResolver class, but this close method is never called and it's also not
auto close(RepairMergeListener is not extending from
AutoCloseable/CloseableIterator) which means we never wait for repair to finish
before returning final result.
The steps to reproduce:
1. create some keyspace/table with RF = 2
2. start 2 nodes using ccm
3. stop node2
4. disable node1 hinted hand off
5. write some data to node1 with consistency level one
6. start node2
7. query some data from node1
This should trigger read repair. I put some log in above close method, and can
not see log print put.
So this bug will basically violate "monotonic quorum reads " guarantee.
> Read repair is not blocking repair to finish in foreground repair
> -
>
> Key: CASSANDRA-13115
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13115
> Project: Cassandra
> Issue Type: Bug
> Environment: ccm on OSX
>Reporter: Xiaolong Jiang
>
> The code trying to wait(block) for repair result to come back in 3.X is below:
> {code}
> public void close()
> {
> try
> {
> FBUtilities.waitOnFutures(repairResults,
> DatabaseDescriptor.getWriteRpcTimeout());
> }
> catch (TimeoutException ex)
> {
> // We got all responses, but timed out while repairing
> int blockFor = consistency.blockFor(keyspace);
> if (Tracing.isTracing())
> Tracing.trace("Timed out while read-repairing after
> receiving all {} data and digest responses", blockFor);
> else
> logger.debug("Timeout while read-repairing after
> receiving all {} data and digest responses", blockFor);
> throw new ReadTimeoutException(consistency, blockFor-1,
> blockFor, true);
> }
> }
> {code}
> in DataResolver class, but this close method is never called and it's also
> not auto close(RepairMergeListener is not extending from
> AutoCloseable/CloseableIterator) which means we never wait for repair to
> finish before returning final result.
> The steps to reproduce:
> 1. create some keyspace/table with RF = 2
> 2. start 2 nodes
[jira] [Assigned] (CASSANDRA-11983) Migration task failed to complete
[ https://issues.apache.org/jira/browse/CASSANDRA-11983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Jirsa reassigned CASSANDRA-11983: -- Assignee: Jeff Jirsa > Migration task failed to complete > - > > Key: CASSANDRA-11983 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11983 > Project: Cassandra > Issue Type: Bug > Components: Lifecycle > Environment: Docker / Kubernetes running > Linux cassandra-21 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt25-1 (2016-03-06) > x86_64 GNU/Linux > openjdk version "1.8.0_91" > OpenJDK Runtime Environment (build 1.8.0_91-8u91-b14-1~bpo8+1-b14) > OpenJDK 64-Bit Server VM (build 25.91-b14, mixed mode) > Cassnadra 3.5 installed from > deb-src http://www.apache.org/dist/cassandra/debian 35x main >Reporter: Chris Love >Assignee: Jeff Jirsa > Attachments: cass.log > > > When nodes are boostrapping I am getting mulitple errors: "Migration task > failed to complete", from MigrationManager.java > The errors increase as more nodes are added to the ring, as I am creating a > ring of 1k nodes. > Cassandra yaml i here > https://github.com/k8s-for-greeks/gpmr/blob/3d50ff91a139b9c4a7a26eda0fb4dcf9a008fbed/pet-race-devops/docker/cassandra-debian/files/cassandra.yaml -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-11983) Migration task failed to complete
[ https://issues.apache.org/jira/browse/CASSANDRA-11983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Jirsa updated CASSANDRA-11983: --- Fix Version/s: 3.x 3.0.x > Migration task failed to complete > - > > Key: CASSANDRA-11983 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11983 > Project: Cassandra > Issue Type: Bug > Components: Lifecycle > Environment: Docker / Kubernetes running > Linux cassandra-21 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt25-1 (2016-03-06) > x86_64 GNU/Linux > openjdk version "1.8.0_91" > OpenJDK Runtime Environment (build 1.8.0_91-8u91-b14-1~bpo8+1-b14) > OpenJDK 64-Bit Server VM (build 25.91-b14, mixed mode) > Cassnadra 3.5 installed from > deb-src http://www.apache.org/dist/cassandra/debian 35x main >Reporter: Chris Love >Assignee: Jeff Jirsa > Fix For: 3.0.x, 3.x > > Attachments: cass.log > > > When nodes are boostrapping I am getting mulitple errors: "Migration task > failed to complete", from MigrationManager.java > The errors increase as more nodes are added to the ring, as I am creating a > ring of 1k nodes. > Cassandra yaml i here > https://github.com/k8s-for-greeks/gpmr/blob/3d50ff91a139b9c4a7a26eda0fb4dcf9a008fbed/pet-race-devops/docker/cassandra-debian/files/cassandra.yaml -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-13126) native transport protocol corruption when using SSL
[
https://issues.apache.org/jira/browse/CASSANDRA-13126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15826727#comment-15826727
]
Jeff Jirsa commented on CASSANDRA-13126:
Do you have the full stack trace for the {{DecoderException}} ?
> native transport protocol corruption when using SSL
> ---
>
> Key: CASSANDRA-13126
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13126
> Project: Cassandra
> Issue Type: Bug
> Components: Core
>Reporter: Tom van der Woerdt
>Priority: Critical
>
> This is a series of conditions that can result in client connections becoming
> unusable.
> 1) Cassandra GC must be well-tuned, to have short GC pauses every minute or so
> 2) *client* SSL must be enabled and transmitting a significant amount of data
> 3) Cassandra must run with the default library versions
> 4) disableexplicitgc must be set (this is the default in the current
> cassandra-env.sh)
> This ticket relates to CASSANDRA-13114 which is a possible workaround (but
> not a fix) for the SSL requirement to trigger this bug.
> * Netty allocates nio.ByteBuffers for every outgoing SSL message.
> * ByteBuffers consist of two parts, the jvm object and the off-heap object.
> The jvm object is small and goes with regular GC cycles, the off-heap object
> gets freed only when the small jvm object is freed. To avoid exploding the
> native memory use, the jvm defaults to limiting its allocation to the max
> heap size. Allocating beyond that limit triggers a System.gc(), a retry, and
> potentially an exception.
> * System.gc is a no-op under disableexplicitgc
> * This means ByteBuffers are likely to throw an exception when too many
> objects are being allocated
> * The netty version shipped in Cassandra is broken when using SSL (see
> CASSANDRA-13114) and causes significantly too many bytebuffers to be
> allocated.
> This gets more complicated though.
> When /some/ clients use SSL, and others don't, the clients not using SSL can
> still be affected by this bug, as bytebuffer starvation caused by ssl will
> leak to other users.
> ByteBuffers are used very early on in the native protocol as well. Before
> even being able to decode the network protocol, this error can be thrown :
> {noformat}
> io.netty.handler.codec.DecoderException: java.lang.OutOfMemoryError: Direct
> buffer memory
> {noformat}
> Note that this comes back with stream_id 0, so clients end up waiting for the
> client timeout before the query is considered failed and retried.
> A few frames later on the same connection, this appears:
> {noformat}
> Provided frame does not appear to be Snappy compressed
> {noformat}
> And after that everything errors out with:
> {noformat}
> Invalid or unsupported protocol version (54); the lowest supported version is
> 3 and the greatest is 4
> {noformat}
> So this bug ultimately affects the binary protocol and the connection becomes
> useless if not downright dangerous.
> I think there are several things that need to be done here.
> * CASSANDRA-13114 should be fixed (easy, and probably needs to land in 3.0.11
> anyway)
> * Connections should be closed after a DecoderException
> * DisableExplicitGC should be removed from the default JVM arguments
> Any of these three would limit the impact to clients.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9143) Improving consistency of repairAt field across replicas
[
https://issues.apache.org/jira/browse/CASSANDRA-9143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15829017#comment-15829017
]
Jeff Jirsa commented on CASSANDRA-9143:
---
I know [~krummas] has review (and he's certainly the most qualified and most
likely to give the best feedback here), but a I took a few notes as I read
through the patches:
- ARS {{registerParentRepairSession}} assert is inverted
-
https://github.com/bdeggleston/cassandra/commit/8e7eb081625b1749716f60bcb109ade8c84d8558#diff-93e6fa14f908d0ce3c24d56fbf484ba3R88
- double-checked locking needs volatile
- Comment in CompactionStrategyManager about 2 strategies per data dir is now
misleading, if not incorrect (also have pendingRepairs, which may be multiple
other strategies?):
https://github.com/bdeggleston/cassandra/blob/9143-trunk/src/java/org/apache/cassandra/db/compaction/CompactionStrategyManager.java#L59
- {{ConsistentSession.AbstractBuilder}} doesn't need to be public (especially
with other {{AbstractBuilder}}s in the codebase)
- {{LocalSessions.start()}} loads rows creating builders that should always
work, but we've seen in the past (like CASSANDRA-12700) that we shouldn't rely
on all of those values to be correct - maybe you can try to explicitly handle
invalid rows being returned? If incomplete row/session, maybe that's failed by
definition?
- If you remove the enum definition for anticompaction request (
https://github.com/bdeggleston/cassandra/commit/76ee1a667818c5c72aa513c4a75777b1400cb69d#diff-9a5c76380064186d8f89003e1bab73bfL46
), and we're in a mixed cluster for some reason, and get that verb on the
wire, we'll throw - perhaps instead we should probably keep that verb, but
log+ignore it if received.
Less Importantly:
-
https://github.com/bdeggleston/cassandra/commit/8e7eb081625b1749716f60bcb109ade8c84d8558#diff-93e6fa14f908d0ce3c24d56fbf484ba3R303
- {{needsCleanup()}} could be renamed to avoid potentially confusing
{{CompactionManager.needsCleanup()}}, especially since {{PendingRepairManager}}
is very very much like {{CompactionManager}} (seems as if you may have started
this in
https://github.com/bdeggleston/cassandra/commit/ade7fe3373a1b44da02caaefc3180503d298e92b
)
-
https://github.com/bdeggleston/cassandra/blob/9143-trunk/src/java/org/apache/cassandra/repair/consistent/LocalSessions.java#L605
- log from address as well, otherwise the log message is much less useful?
-
https://github.com/bdeggleston/cassandra/blob/9143-trunk/src/java/org/apache/cassandra/repair/consistent/LocalSessions.java#L619
- could pass address for better logging here as well (and pretty much all of
these, {{handlePrepareMessage}}, {{handleFinalizeCommitMessage}}, etc)
> Improving consistency of repairAt field across replicas
>
>
> Key: CASSANDRA-9143
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9143
> Project: Cassandra
> Issue Type: Improvement
>Reporter: sankalp kohli
>Assignee: Blake Eggleston
>
> We currently send an anticompaction request to all replicas. During this, a
> node will split stables and mark the appropriate ones repaired.
> The problem is that this could fail on some replicas due to many reasons
> leading to problems in the next repair.
> This is what I am suggesting to improve it.
> 1) Send anticompaction request to all replicas. This can be done at session
> level.
> 2) During anticompaction, stables are split but not marked repaired.
> 3) When we get positive ack from all replicas, coordinator will send another
> message called markRepaired.
> 4) On getting this message, replicas will mark the appropriate stables as
> repaired.
> This will reduce the window of failure. We can also think of "hinting"
> markRepaired message if required.
> Also the stables which are streaming can be marked as repaired like it is
> done now.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11163) Summaries are needlessly rebuilt when the BF FP ratio is changed
[
https://issues.apache.org/jira/browse/CASSANDRA-11163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15832752#comment-15832752
]
Jeff Jirsa commented on CASSANDRA-11163:
It persists until all sstables are rewritten - the bloom filter is recalculated
on startup but not persisted to disk until compaction runs. You can force the
bloom filter to be rewritten without waiting dircompaction or this bug to be
fixed by running "nodetool upgradesstables -a" to rewrite all of the sstables
(note that this can be IO intensive and impact your latencies)
> Summaries are needlessly rebuilt when the BF FP ratio is changed
>
>
> Key: CASSANDRA-11163
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11163
> Project: Cassandra
> Issue Type: Bug
> Components: Core
>Reporter: Brandon Williams
> Fix For: 3.x
>
>
> This is from trunk, but I also saw this happen on 2.0:
> Before:
> {noformat}
> root@bw-1:/srv/cassandra# ls -ltr
> /var/lib/cassandra/data/keyspace1/standard1-071efdc0d11811e590c3413ee28a6c90/
> total 221460
> drwxr-xr-x 2 root root 4096 Feb 11 23:34 backups
> -rw-r--r-- 1 root root80 Feb 11 23:50 ma-6-big-TOC.txt
> -rw-r--r-- 1 root root 26518 Feb 11 23:50 ma-6-big-Summary.db
> -rw-r--r-- 1 root root 10264 Feb 11 23:50 ma-6-big-Statistics.db
> -rw-r--r-- 1 root root 2607705 Feb 11 23:50 ma-6-big-Index.db
> -rw-r--r-- 1 root root192440 Feb 11 23:50 ma-6-big-Filter.db
> -rw-r--r-- 1 root root10 Feb 11 23:50 ma-6-big-Digest.crc32
> -rw-r--r-- 1 root root 35212125 Feb 11 23:50 ma-6-big-Data.db
> -rw-r--r-- 1 root root 2156 Feb 11 23:50 ma-6-big-CRC.db
> -rw-r--r-- 1 root root80 Feb 11 23:50 ma-7-big-TOC.txt
> -rw-r--r-- 1 root root 26518 Feb 11 23:50 ma-7-big-Summary.db
> -rw-r--r-- 1 root root 10264 Feb 11 23:50 ma-7-big-Statistics.db
> -rw-r--r-- 1 root root 2607614 Feb 11 23:50 ma-7-big-Index.db
> -rw-r--r-- 1 root root192432 Feb 11 23:50 ma-7-big-Filter.db
> -rw-r--r-- 1 root root 9 Feb 11 23:50 ma-7-big-Digest.crc32
> -rw-r--r-- 1 root root 35190400 Feb 11 23:50 ma-7-big-Data.db
> -rw-r--r-- 1 root root 2152 Feb 11 23:50 ma-7-big-CRC.db
> -rw-r--r-- 1 root root80 Feb 11 23:50 ma-5-big-TOC.txt
> -rw-r--r-- 1 root root104178 Feb 11 23:50 ma-5-big-Summary.db
> -rw-r--r-- 1 root root 10264 Feb 11 23:50 ma-5-big-Statistics.db
> -rw-r--r-- 1 root root 10289077 Feb 11 23:50 ma-5-big-Index.db
> -rw-r--r-- 1 root root757384 Feb 11 23:50 ma-5-big-Filter.db
> -rw-r--r-- 1 root root 9 Feb 11 23:50 ma-5-big-Digest.crc32
> -rw-r--r-- 1 root root 139201355 Feb 11 23:50 ma-5-big-Data.db
> -rw-r--r-- 1 root root 8508 Feb 11 23:50 ma-5-big-CRC.db
> root@bw-1:/srv/cassandra# md5sum
> /var/lib/cassandra/data/keyspace1/standard1-071efdc0d11811e590c3413ee28a6c90/ma-5-big-Summary.db
> 5fca154fc790f7cfa37e8ad6d1c7552c
> {noformat}
> BF ratio changed, node restarted:
> {noformat}
> root@bw-1:/srv/cassandra# ls -ltr
> /var/lib/cassandra/data/keyspace1/standard1-071efdc0d11811e590c3413ee28a6c90/
> total 242168
> drwxr-xr-x 2 root root 4096 Feb 11 23:34 backups
> -rw-r--r-- 1 root root80 Feb 11 23:50 ma-6-big-TOC.txt
> -rw-r--r-- 1 root root 10264 Feb 11 23:50 ma-6-big-Statistics.db
> -rw-r--r-- 1 root root 2607705 Feb 11 23:50 ma-6-big-Index.db
> -rw-r--r-- 1 root root192440 Feb 11 23:50 ma-6-big-Filter.db
> -rw-r--r-- 1 root root10 Feb 11 23:50 ma-6-big-Digest.crc32
> -rw-r--r-- 1 root root 35212125 Feb 11 23:50 ma-6-big-Data.db
> -rw-r--r-- 1 root root 2156 Feb 11 23:50 ma-6-big-CRC.db
> -rw-r--r-- 1 root root80 Feb 11 23:50 ma-7-big-TOC.txt
> -rw-r--r-- 1 root root 10264 Feb 11 23:50 ma-7-big-Statistics.db
> -rw-r--r-- 1 root root 2607614 Feb 11 23:50 ma-7-big-Index.db
> -rw-r--r-- 1 root root192432 Feb 11 23:50 ma-7-big-Filter.db
> -rw-r--r-- 1 root root 9 Feb 11 23:50 ma-7-big-Digest.crc32
> -rw-r--r-- 1 root root 35190400 Feb 11 23:50 ma-7-big-Data.db
> -rw-r--r-- 1 root root 2152 Feb 11 23:50 ma-7-big-CRC.db
> -rw-r--r-- 1 root root80 Feb 11 23:50 ma-5-big-TOC.txt
> -rw-r--r-- 1 root root 10264 Feb 11 23:50 ma-5-big-Statistics.db
> -rw-r--r-- 1 root root 10289077 Feb 11 23:50 ma-5-big-Index.db
> -rw-r--r-- 1 root root757384 Feb 11 23:50 ma-5-big-Filter.db
> -rw-r--r-- 1 root root 9 Feb 11 23:50 ma-5-big-Digest.crc32
> -rw-r--r-- 1 root root 139201355 Feb 11 23:50 ma-5-big-Data.db
> -rw-r--r-- 1 root root 8508 Feb 11 23:50 ma-5-big-CRC.db
> -rw-r--r-- 1 root root80 Feb 12 00:03 ma-8-big-TOC.txt
> -rw-r--r-- 1 root root 14902 Feb 12 00:03 ma-8-big-Summary.db
> -rw-r--r-- 1 root root 10264 Feb 12 00:03 ma-8-big-Statistics.db
> -rw-r--r-- 1 root root 1458631 Feb 1
[jira] [Updated] (CASSANDRA-13149) AssertionError prepending to a list
[
https://issues.apache.org/jira/browse/CASSANDRA-13149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jeff Jirsa updated CASSANDRA-13149:
---
Description:
Prepending to a list produces the following AssertionError randomly. Changing
the update to append (and sort in the client) works around the issue.
{code}
java.lang.AssertionError: null
at
org.apache.cassandra.cql3.Lists$PrecisionTime.getNext(Lists.java:275)
~[apache-cassandra-3.0.8.jar:3.0.8]
at org.apache.cassandra.cql3.Lists$Prepender.execute(Lists.java:430)
~[apache-cassandra-3.0.8.jar:3.0.8]
at
org.apache.cassandra.cql3.statements.UpdateStatement.addUpdateForKey(UpdateStatement.java:94)
~[apache-cassandra-3.0.8.jar:3.0.8]
at
org.apache.cassandra.cql3.statements.ModificationStatement.addUpdates(ModificationStatement.java:682)
~[apache-cassandra-3.0.8.jar:3.0.8]
at
org.apache.cassandra.cql3.statements.ModificationStatement.getMutations(ModificationStatement.java:613)
~[apache-cassandra-3.0.8.jar:3.0.8]
at
org.apache.cassandra.cql3.statements.ModificationStatement.executeWithoutCondition(ModificationStatement.java:420)
~[apache-cassandra-3.0.8.jar:3.0.8]
at
org.apache.cassandra.cql3.statements.ModificationStatement.execute(ModificationStatement.java:408)
~[apache-cassandra-3.0.8.jar:3.0.8]
at
org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:206)
~[apache-cassandra-3.0.8.jar:3.0.8]
at
org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:487)
~[apache-cassandra-3.0.8.jar:3.0.8]
at
org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:464)
~[apache-cassandra-3.0.8.jar:3.0.8]
at
org.apache.cassandra.transport.messages.ExecuteMessage.execute(ExecuteMessage.java:130)
~[apache-cassandra-3.0.8.jar:3.0.8]
at
org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:507)
[apache-cassandra-3.0.8.jar:3.0.8]
at
org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:401)
[apache-cassandra-3.0.8.jar:3.0.8]
at
io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
[netty-all-4.0.23.Final.jar:4.0.23.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
[netty-all-4.0.23.Final.jar:4.0.23.Final]
at
io.netty.channel.AbstractChannelHandlerContext.access$700(AbstractChannelHandlerContext.java:32)
[netty-all-4.0.23.Final.jar:4.0.23.Final]
at
io.netty.channel.AbstractChannelHandlerContext$8.run(AbstractChannelHandlerContext.java:324)
[netty-all-4.0.23.Final.jar:4.0.23.Final]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[na:1.8.0_101]
at
org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164)
[apache-cassandra-3.0.8.jar:3.0.8]
at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105)
[apache-cassandra-3.0.8.jar:3.0.8]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_101]
{code}
was:
Prepending to a list produces the following AssertionError randomly. Changing
the update to append (and sort in the client) works around the issue.
java.lang.AssertionError: null
at
org.apache.cassandra.cql3.Lists$PrecisionTime.getNext(Lists.java:275)
~[apache-cassandra-3.0.8.jar:3.0.8]
at org.apache.cassandra.cql3.Lists$Prepender.execute(Lists.java:430)
~[apache-cassandra-3.0.8.jar:3.0.8]
at
org.apache.cassandra.cql3.statements.UpdateStatement.addUpdateForKey(UpdateStatement.java:94)
~[apache-cassandra-3.0.8.jar:3.0.8]
at
org.apache.cassandra.cql3.statements.ModificationStatement.addUpdates(ModificationStatement.java:682)
~[apache-cassandra-3.0.8.jar:3.0.8]
at
org.apache.cassandra.cql3.statements.ModificationStatement.getMutations(ModificationStatement.java:613)
~[apache-cassandra-3.0.8.jar:3.0.8]
at
org.apache.cassandra.cql3.statements.ModificationStatement.executeWithoutCondition(ModificationStatement.java:420)
~[apache-cassandra-3.0.8.jar:3.0.8]
at
org.apache.cassandra.cql3.statements.ModificationStatement.execute(ModificationStatement.java:408)
~[apache-cassandra-3.0.8.jar:3.0.8]
at
org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:206)
~[apache-cassandra-3.0.8.jar:3.0.8]
at
org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:487)
~[apache-cassandra-3.0.8.jar:3.0.8]
at
org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:464)
~[apache-cassandra-3.0.8.jar:3.0.8]
at
org.apache.cassandra.transport.messages.ExecuteMessage.execute(ExecuteMessage.java:130)
~[apache-cassandra-3.0.8.jar:3.0.8]
at
org.apache.cassa
[jira] [Commented] (CASSANDRA-13004) Corruption while adding a column to a table
[
https://issues.apache.org/jira/browse/CASSANDRA-13004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15838688#comment-15838688
]
Jeff Jirsa commented on CASSANDRA-13004:
[~stanislav] or [~jhg] - can you confirm what CL you used to read and write
data? Do you happen to know if your data was corrupt on more than one node?
> Corruption while adding a column to a table
> ---
>
> Key: CASSANDRA-13004
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13004
> Project: Cassandra
> Issue Type: Bug
>Reporter: Stanislav Vishnevskiy
>
> We had the following schema in production.
> {code:none}
> CREATE TYPE IF NOT EXISTS discord_channels.channel_recipient (
> nick text
> );
> CREATE TYPE IF NOT EXISTS discord_channels.channel_permission_overwrite (
> id bigint,
> type int,
> allow_ int,
> deny int
> );
> CREATE TABLE IF NOT EXISTS discord_channels.channels (
> id bigint,
> guild_id bigint,
> type tinyint,
> name text,
> topic text,
> position int,
> owner_id bigint,
> icon_hash text,
> recipients map>,
> permission_overwrites map>,
> bitrate int,
> user_limit int,
> last_pin_timestamp timestamp,
> last_message_id bigint,
> PRIMARY KEY (id)
> );
> {code}
> And then we executed the following alter.
> {code:none}
> ALTER TABLE discord_channels.channels ADD application_id bigint;
> {code}
> And one row (that we can tell) got corrupted at the same time and could no
> longer be read from the Python driver.
> {code:none}
> [E 161206 01:56:58 geventreactor:141] Error decoding response from Cassandra.
> ver(4); flags(); stream(27); op(8); offset(9); len(887); buffer:
> '\x84\x00\x00\x1b\x08\x00\x00\x03w\x00\x00\x00\x02\x00\x00\x00\x01\x00\x00\x00\x0f\x00\x10discord_channels\x00\x08channels\x00\x02id\x00\x02\x00\x0eapplication_id\x00\x02\x00\x07bitrate\x00\t\x00\x08guild_id\x00\x02\x00\ticon_hash\x00\r\x00\x0flast_message_id\x00\x02\x00\x12last_pin_timestamp\x00\x0b\x00\x04name\x00\r\x00\x08owner_id\x00\x02\x00\x15permission_overwrites\x00!\x00\x02\x000\x00\x10discord_channels\x00\x1cchannel_permission_overwrite\x00\x04\x00\x02id\x00\x02\x00\x04type\x00\t\x00\x06allow_\x00\t\x00\x04deny\x00\t\x00\x08position\x00\t\x00\nrecipients\x00!\x00\x02\x000\x00\x10discord_channels\x00\x11channel_recipient\x00\x01\x00\x04nick\x00\r\x00\x05topic\x00\r\x00\x04type\x00\x14\x00\nuser_limit\x00\t\x00\x00\x00\x01\x00\x00\x00\x08\x03\x8a\x19\x8e\xf8\x82\x00\x01\xff\xff\xff\xff\x00\x00\x00\x04\x00\x00\xfa\x00\x00\x00\x00\x08\x00\x00\xfa\x00\x00\xf8G\xc5\x00\x00\x00\x00\x00\x00\x00\x08\x03\x8b\xc0\xb5nB\x00\x02\x00\x00\x00\x08G\xc5\xffI\x98\xc4\xb4(\x00\x00\x00\x03\x8b\xc0\xa8\xff\xff\xff\xff\x00\x00\x01<\x00\x00\x00\x06\x00\x00\x00\x08\x03\x81L\xea\xfc\x82\x00\n\x00\x00\x00$\x00\x00\x00\x08\x03\x81L\xea\xfc\x82\x00\n\x00\x00\x00\x04\x00\x00\x00\x01\x00\x00\x00\x04\x00\x00\x08\x00\x00\x00\x00\x04\x00\x00\x00\x00\x00\x00\x00\x08\x03\x8a\x1e\xe6\x8b\x80\x00\n\x00\x00\x00$\x00\x00\x00\x08\x03\x8a\x1e\xe6\x8b\x80\x00\n\x00\x00\x00\x04\x00\x00\x00\x00\x00\x00\x00\x040\x07\xf8Q\x00\x00\x00\x04\x00\x00\x00\x00\x00\x00\x00\x08\x03\x8a\x1f\x1b{\x82\x00\x00\x00\x00\x00$\x00\x00\x00\x08\x03\x8a\x1f\x1b{\x82\x00\x00\x00\x00\x00\x04\x00\x00\x00\x00\x00\x00\x00\x04\x00\x07\xf8Q\x00\x00\x00\x04\x10\x00\x00\x00\x00\x00\x00\x08\x03\x8a\x1fH6\x82\x00\x01\x00\x00\x00$\x00\x00\x00\x08\x03\x8a\x1fH6\x82\x00\x01\x00\x00\x00\x04\x00\x00\x00\x00\x00\x00\x00\x04\x00\x05\xe8A\x00\x00\x00\x04\x10\x02\x00\x00\x00\x00\x00\x08\x03\x8a+=\xca\xc0\x00\n\x00\x00\x00$\x00\x00\x00\x08\x03\x8a+=\xca\xc0\x00\n\x00\x00\x00\x04\x00\x00\x00\x00\x00\x00\x00\x04\x00\x00\x08\x00\x00\x00\x00\x04\x00\x00\x00\x00\x00\x00\x00\x08\x03\x8a\x8f\x979\x80\x00\n\x00\x00\x00$\x00\x00\x00\x08\x03\x8a\x8f\x979\x80\x00\n\x00\x00\x00\x04\x00\x00\x00\x00\x00\x00\x00\x04\x00\x00\x00\x00\x00\x00\x00\x04\x00
>
> \x08\x01\x00\x00\x00\x04\xc4\xb4(\x00\xff\xff\xff\xff\x00\x00\x00O[f\x80Q\x07general\x05\xf8G\xc5\xffI\x98\xc4\xb4(\x00\xf8O[f\x80Q\x00\x00\x00\x02\x04\xf8O[f\x80Q\x00\xf8G\xc5\xffI\x98\x01\x00\x00\xf8O[f\x80Q\x00\x00\x00\x00\xf8G\xc5\xffI\x97\xc4\xb4(\x06\x00\xf8O\x7fe\x1fm\x08\x03\x00\x00\x00\x01\x00\x00\x00\x00\x04\x00\x00\x00\x00'
> {code}
> And then in cqlsh when trying to read the row we got this.
> {code:none}
> /usr/bin/cqlsh.py:632: DateOverFlowWarning: Some timestamps are larger than
> Python datetime can represent. Timestamps are displayed in milliseconds from
> epoch.
> Traceback (most recent call last):
> File "/usr/bin/cqlsh.py", line 1301, in perform_simple_statement
> result = future.result()
> File
> "/usr/share/cassandra/lib/cassandra-driver-internal-only-3.5.0.post0-d8d0456.zip/cassandra-driver-3.5.0.post0-d8d0456/cassandra/cluster.py",
> line 3650, in result
> raise self._fi
[jira] [Updated] (CASSANDRA-9530) SSTable corruption can trigger OOM
[
https://issues.apache.org/jira/browse/CASSANDRA-9530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jeff Jirsa updated CASSANDRA-9530:
--
Description:
If a sstable is corrupted so that the length of a given value is bogus, we'll
still happily try to allocate a buffer of that bogus size to read the value,
which can easily lead to an OOM.
We should probably protect against this. In practice, a given value can be so
big since it's limited by the protocol frame size in the first place. Maybe we
could add a max_value_size_in_mb setting and we'd considered a sstable
corrupted if it was containing a value bigger than that.
I'll note that this ticket would be a good occasion to improve
{{BlacklistingCompactionsTest}}. Typically, it currently generate empty values
which makes it pretty much impossible to get the problem described here. And as
described in CASSANDRA-9478, it also doesn't test properly for thing like early
opening of compaction results. We could try to randomize as much of the
parameters of this test as possible to make it more likely to catch any type of
corruption that could happen.
was:
If a sstable is corrupted so that the length of a given is bogus, we'll still
happily try to allocate a buffer of that bogus size to read the value, which
can easily lead to an OOM.
We should probably protect against this. In practice, a given value can be so
big since it's limited by the protocol frame size in the first place. Maybe we
could add a max_value_size_in_mb setting and we'd considered a sstable
corrupted if it was containing a value bigger than that.
I'll note that this ticket would be a good occasion to improve
{{BlacklistingCompactionsTest}}. Typically, it currently generate empty values
which makes it pretty much impossible to get the problem described here. And as
described in CASSANDRA-9478, it also doesn't test properly for thing like early
opening of compaction results. We could try to randomize as much of the
parameters of this test as possible to make it more likely to catch any type of
corruption that could happen.
> SSTable corruption can trigger OOM
> --
>
> Key: CASSANDRA-9530
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9530
> Project: Cassandra
> Issue Type: Bug
> Components: Local Write-Read Paths
>Reporter: Sylvain Lebresne
>Assignee: Alex Petrov
> Fix For: 3.0.7, 3.7, 3.8
>
>
> If a sstable is corrupted so that the length of a given value is bogus, we'll
> still happily try to allocate a buffer of that bogus size to read the value,
> which can easily lead to an OOM.
> We should probably protect against this. In practice, a given value can be so
> big since it's limited by the protocol frame size in the first place. Maybe
> we could add a max_value_size_in_mb setting and we'd considered a sstable
> corrupted if it was containing a value bigger than that.
> I'll note that this ticket would be a good occasion to improve
> {{BlacklistingCompactionsTest}}. Typically, it currently generate empty
> values which makes it pretty much impossible to get the problem described
> here. And as described in CASSANDRA-9478, it also doesn't test properly for
> thing like early opening of compaction results. We could try to randomize as
> much of the parameters of this test as possible to make it more likely to
> catch any type of corruption that could happen.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-12694) PAXOS Update Corrupted empty row exception
[
https://issues.apache.org/jira/browse/CASSANDRA-12694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jeff Jirsa updated CASSANDRA-12694:
---
Reproduced In: 3.7, 3.0.10 (was: 3.7)
> PAXOS Update Corrupted empty row exception
> --
>
> Key: CASSANDRA-12694
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12694
> Project: Cassandra
> Issue Type: Bug
> Components: Local Write-Read Paths
> Environment: 3 node cluster using RF=3 running on cassandra 3.7
>Reporter: Cameron Zemek
>Assignee: Alex Petrov
> Fix For: 3.0.11, 3.10
>
>
> {noformat}
> cqlsh> create table test.test (test_id TEXT, last_updated TIMESTAMP,
> message_id TEXT, PRIMARY KEY(test_id));
> update test.test set last_updated = 1474494363669 where test_id = 'test1' if
> message_id = null;
> {noformat}
> Then nodetool flush on the all 3 nodes.
> {noformat}
> cqlsh> update test.test set last_updated = 1474494363669 where test_id =
> 'test1' if message_id = null;
> ServerError:
> {noformat}
> From cassandra log
> {noformat}
> ERROR [SharedPool-Worker-1] 2016-09-23 12:09:13,179 Message.java:611 -
> Unexpected exception during request; channel = [id: 0x7a22599e,
> L:/127.0.0.1:9042 - R:/127.0.0.1:58297]
> java.io.IOError: java.io.IOException: Corrupt empty row found in unfiltered
> partition
> at
> org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer$1.computeNext(UnfilteredRowIteratorSerializer.java:224)
> ~[main/:na]
> at
> org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer$1.computeNext(UnfilteredRowIteratorSerializer.java:212)
> ~[main/:na]
> at
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
> ~[main/:na]
> at
> org.apache.cassandra.db.rows.UnfilteredRowIterators.digest(UnfilteredRowIterators.java:125)
> ~[main/:na]
> at
> org.apache.cassandra.db.partitions.UnfilteredPartitionIterators.digest(UnfilteredPartitionIterators.java:249)
> ~[main/:na]
> at
> org.apache.cassandra.db.ReadResponse.makeDigest(ReadResponse.java:87)
> ~[main/:na]
> at
> org.apache.cassandra.db.ReadResponse$DataResponse.digest(ReadResponse.java:192)
> ~[main/:na]
> at
> org.apache.cassandra.service.DigestResolver.resolve(DigestResolver.java:80)
> ~[main/:na]
> at
> org.apache.cassandra.service.ReadCallback.get(ReadCallback.java:139)
> ~[main/:na]
> at
> org.apache.cassandra.service.AbstractReadExecutor.get(AbstractReadExecutor.java:145)
> ~[main/:na]
> at
> org.apache.cassandra.service.StorageProxy$SinglePartitionReadLifecycle.awaitResultsAndRetryOnDigestMismatch(StorageProxy.java:1714)
> ~[main/:na]
> at
> org.apache.cassandra.service.StorageProxy.fetchRows(StorageProxy.java:1663)
> ~[main/:na]
> at
> org.apache.cassandra.service.StorageProxy.readRegular(StorageProxy.java:1604)
> ~[main/:na]
> at
> org.apache.cassandra.service.StorageProxy.read(StorageProxy.java:1523)
> ~[main/:na]
> at
> org.apache.cassandra.service.StorageProxy.readOne(StorageProxy.java:1497)
> ~[main/:na]
> at
> org.apache.cassandra.service.StorageProxy.readOne(StorageProxy.java:1491)
> ~[main/:na]
> at
> org.apache.cassandra.service.StorageProxy.cas(StorageProxy.java:249)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.statements.ModificationStatement.executeWithCondition(ModificationStatement.java:441)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.statements.ModificationStatement.execute(ModificationStatement.java:416)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:208)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:239)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:224)
> ~[main/:na]
> at
> org.apache.cassandra.transport.messages.QueryMessage.execute(QueryMessage.java:115)
> ~[main/:na]
> at
> org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:507)
> [main/:na]
> at
> org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:401)
> [main/:na]
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12867) Batch with multiple conditional updates for the same partition causes AssertionError
[
https://issues.apache.org/jira/browse/CASSANDRA-12867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15839008#comment-15839008
]
Jeff Jirsa commented on CASSANDRA-12867:
[~KurtG] - Just to clarify - the description says that this is reproduced AND
fixed in 3.0.10.
The commit is a week before 3.0.10 was cut, so this was a regression from 3.0.9
that was fixed before 3.0.10 actually shipped, correct?
> Batch with multiple conditional updates for the same partition causes
> AssertionError
>
>
> Key: CASSANDRA-12867
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12867
> Project: Cassandra
> Issue Type: Bug
> Components: CQL
>Reporter: Kurt Greaves
>Assignee: Sylvain Lebresne
>Priority: Critical
> Fix For: 3.0.10, 3.10
>
> Attachments: 12867-3.0.patch
>
>
> Reproduced in 3.0.10 and 3.10. Used to work in 3.0.9 and earlier. Bug was
> introduced in CASSANDRA-12060.
> The following causes an AssertionError:
> {code}
> CREATE KEYSPACE test WITH replication = { 'class' : 'SimpleStrategy',
> 'replication_factor' : 1 };
> create table test.test (id int PRIMARY KEY, val text);
> BEGIN BATCH INSERT INTO test.test (id, val) VALUES (999, 'aaa') IF NOT
> EXISTS; INSERT INTO test.test (id, val) VALUES (999, 'ccc') IF NOT EXISTS;
> APPLY BATCH ;
> {code}
> Stack trace is as follows:
> {code}
> ERROR [Native-Transport-Requests-2] 2016-10-31 04:16:44,231 Message.java:622
> - Unexpected exception during request; channel = [id: 0x176e1c04,
> L:/127.0.0.1:9042 - R:/127.0.0.1:59743]
> java.lang.AssertionError: null
> at
> org.apache.cassandra.cql3.statements.CQL3CasRequest.setConditionsForRow(CQL3CasRequest.java:138)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.statements.CQL3CasRequest.addExistsCondition(CQL3CasRequest.java:104)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.statements.CQL3CasRequest.addNotExist(CQL3CasRequest.java:84)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.IfNotExistsCondition.addConditionsTo(IfNotExistsCondition.java:28)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.statements.ModificationStatement.addConditions(ModificationStatement.java:482)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.statements.BatchStatement.makeCasRequest(BatchStatement.java:434)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.statements.BatchStatement.executeWithConditions(BatchStatement.java:379)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.statements.BatchStatement.execute(BatchStatement.java:358)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.statements.BatchStatement.execute(BatchStatement.java:346)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.statements.BatchStatement.execute(BatchStatement.java:341)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:218)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:249)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:234)
> ~[main/:na]
> at
> org.apache.cassandra.transport.messages.QueryMessage.execute(QueryMessage.java:115)
> ~[main/:na]
> at
> org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:516)
> [main/:na]
> at
> org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:409)
> [main/:na]
> at
> io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
> [netty-all-4.0.39.Final.jar:4.0.39.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:366)
> [netty-all-4.0.39.Final.jar:4.0.39.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:35)
> [netty-all-4.0.39.Final.jar:4.0.39.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:357)
> [netty-all-4.0.39.Final.jar:4.0.39.Final]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [na:1.8.0_102]
> at
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:162)
> [main/:na]
> at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:109)
> [main/:na]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_102]
> {code}
> The problem is that previous will receive a value after the first statement
> in the batch is evaluated in BatchStatement.makeCasRequest. I can't see any
> reason why we have this assertion, it seems to me
[jira] [Commented] (CASSANDRA-12694) PAXOS Update Corrupted empty row exception
[
https://issues.apache.org/jira/browse/CASSANDRA-12694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15839091#comment-15839091
]
Jeff Jirsa commented on CASSANDRA-12694:
FYI, this commit (
https://github.com/apache/cassandra/commit/d9b06e8af41c42244f76058641aeecda53a9bf75
) didn't have the CHANGES.txt entry and used an incorrect JIRA number.
> PAXOS Update Corrupted empty row exception
> --
>
> Key: CASSANDRA-12694
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12694
> Project: Cassandra
> Issue Type: Bug
> Components: Local Write-Read Paths
> Environment: 3 node cluster using RF=3 running on cassandra 3.7
>Reporter: Cameron Zemek
>Assignee: Alex Petrov
> Fix For: 3.0.11, 3.10
>
>
> {noformat}
> cqlsh> create table test.test (test_id TEXT, last_updated TIMESTAMP,
> message_id TEXT, PRIMARY KEY(test_id));
> update test.test set last_updated = 1474494363669 where test_id = 'test1' if
> message_id = null;
> {noformat}
> Then nodetool flush on the all 3 nodes.
> {noformat}
> cqlsh> update test.test set last_updated = 1474494363669 where test_id =
> 'test1' if message_id = null;
> ServerError:
> {noformat}
> From cassandra log
> {noformat}
> ERROR [SharedPool-Worker-1] 2016-09-23 12:09:13,179 Message.java:611 -
> Unexpected exception during request; channel = [id: 0x7a22599e,
> L:/127.0.0.1:9042 - R:/127.0.0.1:58297]
> java.io.IOError: java.io.IOException: Corrupt empty row found in unfiltered
> partition
> at
> org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer$1.computeNext(UnfilteredRowIteratorSerializer.java:224)
> ~[main/:na]
> at
> org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer$1.computeNext(UnfilteredRowIteratorSerializer.java:212)
> ~[main/:na]
> at
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
> ~[main/:na]
> at
> org.apache.cassandra.db.rows.UnfilteredRowIterators.digest(UnfilteredRowIterators.java:125)
> ~[main/:na]
> at
> org.apache.cassandra.db.partitions.UnfilteredPartitionIterators.digest(UnfilteredPartitionIterators.java:249)
> ~[main/:na]
> at
> org.apache.cassandra.db.ReadResponse.makeDigest(ReadResponse.java:87)
> ~[main/:na]
> at
> org.apache.cassandra.db.ReadResponse$DataResponse.digest(ReadResponse.java:192)
> ~[main/:na]
> at
> org.apache.cassandra.service.DigestResolver.resolve(DigestResolver.java:80)
> ~[main/:na]
> at
> org.apache.cassandra.service.ReadCallback.get(ReadCallback.java:139)
> ~[main/:na]
> at
> org.apache.cassandra.service.AbstractReadExecutor.get(AbstractReadExecutor.java:145)
> ~[main/:na]
> at
> org.apache.cassandra.service.StorageProxy$SinglePartitionReadLifecycle.awaitResultsAndRetryOnDigestMismatch(StorageProxy.java:1714)
> ~[main/:na]
> at
> org.apache.cassandra.service.StorageProxy.fetchRows(StorageProxy.java:1663)
> ~[main/:na]
> at
> org.apache.cassandra.service.StorageProxy.readRegular(StorageProxy.java:1604)
> ~[main/:na]
> at
> org.apache.cassandra.service.StorageProxy.read(StorageProxy.java:1523)
> ~[main/:na]
> at
> org.apache.cassandra.service.StorageProxy.readOne(StorageProxy.java:1497)
> ~[main/:na]
> at
> org.apache.cassandra.service.StorageProxy.readOne(StorageProxy.java:1491)
> ~[main/:na]
> at
> org.apache.cassandra.service.StorageProxy.cas(StorageProxy.java:249)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.statements.ModificationStatement.executeWithCondition(ModificationStatement.java:441)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.statements.ModificationStatement.execute(ModificationStatement.java:416)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:208)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:239)
> ~[main/:na]
> at
> org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:224)
> ~[main/:na]
> at
> org.apache.cassandra.transport.messages.QueryMessage.execute(QueryMessage.java:115)
> ~[main/:na]
> at
> org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:507)
> [main/:na]
> at
> org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:401)
> [main/:na]
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
