[jira] [Updated] (CASSANDRA-10801) Unexplained inconsistent data with Cassandra 2.1
[
https://issues.apache.org/jira/browse/CASSANDRA-10801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Imri Zvik updated CASSANDRA-10801:
--
Description:
We are experiencing weird behavior which we cannot explain.
We have a CF, with RF=3, and we are writing and reading data to it with
consistency level of ONE.
For some reason, we see inconsistent results when querying for data.
Even for rows that were written a day ago, we're seeing inconsistent results (1
replca has the data, the two others don't).
Now, I would expect to see timeouts/dropped mutations, but all relevant
counters are not advancing, and I would also expect hints to fix this
inconsistency within minutes, but yet it doesn't.
{code}
cqlsh:testing> SELECT WRITETIME(last_update),site_id, tree_id, individual_id,
last_update FROM testcf WHERE site_id = 229673621 AND tree_id = 9 AND
individual_id = 9032483;
writetime(last_update) | site_id | tree_id | individual_id | last_update
+---+-+---+-
1448988343028000 | 229673621 | 9 | 9032483 | 1380912397
(1 rows)
cqlsh:testing> SELECT WRITETIME(last_update),site_id, tree_id, individual_id,
last_update FROM testcf WHERE site_id = 229673621 AND tree_id = 9 AND
individual_id = 9032483;
site_id | tree_id | individual_id | last_update
---+-+---+-
(0 rows)
cqlsh:testing> SELECT dateof(now()) FROM system.local ;
dateof(now())
--
2015-12-02 14:48:44+
(1 rows)
{code}
We are running with Cassandra 2.1.11 with Oracle Java 1.8.0_65-b17
was:
We are using weird behavior which we cannot explain.
We have a CF, with RF=3, and we are writing and reading data to it with
consistency level of ONE.
For some reason, we see inconsistent results when querying for data.
Even for rows that were written a day ago, we're seeing inconsistent results (1
replca has the data, the two others don't).
Now, I would expect to see timeouts/dropped mutations, but all relevant
counters are not advancing, and I would also expect hints to fix this
inconsistency within minutes, but yet it doesn't.
{code}
cqlsh:testing> SELECT WRITETIME(last_update),site_id, tree_id, individual_id,
last_update FROM testcf WHERE site_id = 229673621 AND tree_id = 9 AND
individual_id = 9032483;
writetime(last_update) | site_id | tree_id | individual_id | last_update
+---+-+---+-
1448988343028000 | 229673621 | 9 | 9032483 | 1380912397
(1 rows)
cqlsh:testing> SELECT WRITETIME(last_update),site_id, tree_id, individual_id,
last_update FROM testcf WHERE site_id = 229673621 AND tree_id = 9 AND
individual_id = 9032483;
site_id | tree_id | individual_id | last_update
---+-+---+-
(0 rows)
cqlsh:testing> SELECT dateof(now()) FROM system.local ;
dateof(now())
--
2015-12-02 14:48:44+
(1 rows)
{code}
We are running with Cassandra 2.1.11 with Oracle Java 1.8.0_65-b17
> Unexplained inconsistent data with Cassandra 2.1
>
>
> Key: CASSANDRA-10801
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10801
> Project: Cassandra
> Issue Type: Bug
>Reporter: Imri Zvik
>
> We are experiencing weird behavior which we cannot explain.
> We have a CF, with RF=3, and we are writing and reading data to it with
> consistency level of ONE.
> For some reason, we see inconsistent results when querying for data.
> Even for rows that were written a day ago, we're seeing inconsistent results
> (1 replca has the data, the two others don't).
> Now, I would expect to see timeouts/dropped mutations, but all relevant
> counters are not advancing, and I would also expect hints to fix this
> inconsistency within minutes, but yet it doesn't.
> {code}
> cqlsh:testing> SELECT WRITETIME(last_update),site_id, tree_id, individual_id,
> last_update FROM testcf WHERE site_id = 229673621 AND tree_id = 9 AND
> individual_id = 9032483;
> writetime(last_update) | site_id | tree_id | individual_id | last_update
> +---+-+---+-
>1448988343028000 | 229673621 | 9 | 9032483 | 1380912397
> (1 rows)
> cqlsh:testing> SELECT WRITETIME(last_update),site_id, tree_id, individual_id,
> last_update FROM testcf WHERE site_id = 229673621 AND tree_id = 9 AND
> individual_id = 9032483;
> site_id | tree_id | individual_id | last_update
> ---+-+---+-
> (0 rows)
> cqlsh:testing> SELECT dateof(now()) FROM system.local ;
> dateof(now())
> --
> 2015-12-02 14:48:44+
> (1 rows)
> {code}
> We are running with
[jira] [Commented] (CASSANDRA-10801) Unexplained inconsistent data with Cassandra 2.1
[
https://issues.apache.org/jira/browse/CASSANDRA-10801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15037983#comment-15037983
]
Imri Zvik commented on CASSANDRA-10801:
---
Please correct me if I am wrong, but to my understanding, even if I am writing
with ONE, and my RF is 3, and at the time of the write 2 out of 3 replicas did
not get the write, hints should be written and replayed within 10 minutes.
So, assuming this row was written hours before the SELECT, all 3 replicas
should already be up to date.
> Unexplained inconsistent data with Cassandra 2.1
>
>
> Key: CASSANDRA-10801
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10801
> Project: Cassandra
> Issue Type: Bug
>Reporter: Imri Zvik
> Fix For: 2.1.x
>
>
> We are experiencing weird behavior which we cannot explain.
> We have a CF, with RF=3, and we are writing and reading data to it with
> consistency level of ONE.
> For some reason, we see inconsistent results when querying for data.
> Even for rows that were written a day ago, we're seeing inconsistent results
> (1 replca has the data, the two others don't).
> Now, I would expect to see timeouts/dropped mutations, but all relevant
> counters are not advancing, and I would also expect hints to fix this
> inconsistency within minutes, but yet it doesn't.
> {code}
> cqlsh:testing> SELECT WRITETIME(last_update),site_id, tree_id, individual_id,
> last_update FROM testcf WHERE site_id = 229673621 AND tree_id = 9 AND
> individual_id = 9032483;
> writetime(last_update) | site_id | tree_id | individual_id | last_update
> +---+-+---+-
>1448988343028000 | 229673621 | 9 | 9032483 | 1380912397
> (1 rows)
> cqlsh:testing> SELECT WRITETIME(last_update),site_id, tree_id, individual_id,
> last_update FROM testcf WHERE site_id = 229673621 AND tree_id = 9 AND
> individual_id = 9032483;
> site_id | tree_id | individual_id | last_update
> ---+-+---+-
> (0 rows)
> cqlsh:testing> SELECT dateof(now()) FROM system.local ;
> dateof(now())
> --
> 2015-12-02 14:48:44+
> (1 rows)
> {code}
> We are running with Cassandra 2.1.11 with Oracle Java 1.8.0_65-b17
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-10801) Unexplained inconsistent data with Cassandra 2.1
[
https://issues.apache.org/jira/browse/CASSANDRA-10801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15038001#comment-15038001
]
Imri Zvik commented on CASSANDRA-10801:
---
This is not what the docs at
http://docs.datastax.com/en/cassandra/2.1/cassandra/dml/dml_about_hh_c.html
says.
{quote}
During a write operation, when hinted handoff is enabled and consistency can be
met, the coordinator stores a hint about dead replicas in the local
system.hints table under either of these conditions:
A replica node for the row is known to be down ahead of time.
A replica node does not respond to the write request.
{quote}
According to that, it will also write a hint upon a timeout.
> Unexplained inconsistent data with Cassandra 2.1
>
>
> Key: CASSANDRA-10801
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10801
> Project: Cassandra
> Issue Type: Bug
>Reporter: Imri Zvik
>
> We are experiencing weird behavior which we cannot explain.
> We have a CF, with RF=3, and we are writing and reading data to it with
> consistency level of ONE.
> For some reason, we see inconsistent results when querying for data.
> Even for rows that were written a day ago, we're seeing inconsistent results
> (1 replca has the data, the two others don't).
> Now, I would expect to see timeouts/dropped mutations, but all relevant
> counters are not advancing, and I would also expect hints to fix this
> inconsistency within minutes, but yet it doesn't.
> {code}
> cqlsh:testing> SELECT WRITETIME(last_update),site_id, tree_id, individual_id,
> last_update FROM testcf WHERE site_id = 229673621 AND tree_id = 9 AND
> individual_id = 9032483;
> writetime(last_update) | site_id | tree_id | individual_id | last_update
> +---+-+---+-
>1448988343028000 | 229673621 | 9 | 9032483 | 1380912397
> (1 rows)
> cqlsh:testing> SELECT WRITETIME(last_update),site_id, tree_id, individual_id,
> last_update FROM testcf WHERE site_id = 229673621 AND tree_id = 9 AND
> individual_id = 9032483;
> site_id | tree_id | individual_id | last_update
> ---+-+---+-
> (0 rows)
> cqlsh:testing> SELECT dateof(now()) FROM system.local ;
> dateof(now())
> --
> 2015-12-02 14:48:44+
> (1 rows)
> {code}
> We are running with Cassandra 2.1.11 with Oracle Java 1.8.0_65-b17
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (CASSANDRA-10801) Unexplained inconsistent data with Cassandra 2.1
Imri Zvik created CASSANDRA-10801:
-
Summary: Unexplained inconsistent data with Cassandra 2.1
Key: CASSANDRA-10801
URL: https://issues.apache.org/jira/browse/CASSANDRA-10801
Project: Cassandra
Issue Type: Bug
Reporter: Imri Zvik
We are using weird behavior which we cannot explain.
We have a CF, with RF=3, and we are writing and reading data to it with
consistency level of ONE.
For some reason, we see inconsistent results when querying for data.
Even for rows that were written a day ago, we're seeing inconsistent results (1
replca has the data, the two others don't).
Now, I would expect to see timeouts/dropped mutations, but all relevant
counters are not advancing, and I would also expect hints to fix this
inconsistency within minutes, but yet it doesn't.
{code}
cqlsh:testing> SELECT WRITETIME(last_update),site_id, tree_id, individual_id,
last_update FROM testcf WHERE site_id = 229673621 AND tree_id = 9 AND
individual_id = 9032483;
writetime(last_update) | site_id | tree_id | individual_id | last_update
+---+-+---+-
1448988343028000 | 229673621 | 9 | 9032483 | 1380912397
(1 rows)
cqlsh:testing> SELECT WRITETIME(last_update),site_id, tree_id, individual_id,
last_update FROM testcf WHERE site_id = 229673621 AND tree_id = 9 AND
individual_id = 9032483;
site_id | tree_id | individual_id | last_update
---+-+---+-
(0 rows)
cqlsh:testing> SELECT dateof(now()) FROM system.local ;
dateof(now())
--
2015-12-02 14:48:44+
(1 rows)
{code}
We are running with Cassandra 2.1.11 with Oracle Java 1.8.0_65-b17
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8696) nodetool repair on cassandra 2.1.2 keyspaces return java.lang.RuntimeException: Could not create snapshot
[
https://issues.apache.org/jira/browse/CASSANDRA-8696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14734391#comment-14734391
]
Imri Zvik commented on CASSANDRA-8696:
--
We are also seeing this on 2.0.13
> nodetool repair on cassandra 2.1.2 keyspaces return
> java.lang.RuntimeException: Could not create snapshot
> -
>
> Key: CASSANDRA-8696
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8696
> Project: Cassandra
> Issue Type: Bug
>Reporter: Jeff Liu
>Assignee: Yuki Morishita
> Fix For: 2.1.x
>
>
> When trying to run nodetool repair -pr on cassandra node ( 2.1.2), cassandra
> throw java exceptions: cannot create snapshot.
> the error log from system.log:
> {noformat}
> INFO [STREAM-IN-/10.97.9.110] 2015-01-28 02:07:28,815
> StreamResultFuture.java:166 - [Stream #692c1450-a692-11e4-9973-070e938df227
> ID#0] Prepare completed. Receiving 2 files(221187 bytes), sending 5
> files(632105 bytes)
> INFO [STREAM-IN-/10.97.9.110] 2015-01-28 02:07:29,046
> StreamResultFuture.java:180 - [Stream #692c1450-a692-11e4-9973-070e938df227]
> Session with /10.97.9.110 is complete
> INFO [STREAM-IN-/10.97.9.110] 2015-01-28 02:07:29,046
> StreamResultFuture.java:212 - [Stream #692c1450-a692-11e4-9973-070e938df227]
> All sessions completed
> INFO [STREAM-IN-/10.97.9.110] 2015-01-28 02:07:29,047
> StreamingRepairTask.java:96 - [repair #685e3d00-a692-11e4-9973-070e938df227]
> streaming task succeed, returning response to /10.98.194.68
> INFO [RepairJobTask:1] 2015-01-28 02:07:29,065 StreamResultFuture.java:86 -
> [Stream #692c6270-a692-11e4-9973-070e938df227] Executing streaming plan for
> Repair
> INFO [StreamConnectionEstablisher:4] 2015-01-28 02:07:29,065
> StreamSession.java:213 - [Stream #692c6270-a692-11e4-9973-070e938df227]
> Starting streaming to /10.66.187.201
> INFO [StreamConnectionEstablisher:4] 2015-01-28 02:07:29,070
> StreamCoordinator.java:209 - [Stream #692c6270-a692-11e4-9973-070e938df227,
> ID#0] Beginning stream session with /10.66.187.201
> INFO [STREAM-IN-/10.66.187.201] 2015-01-28 02:07:29,465
> StreamResultFuture.java:166 - [Stream #692c6270-a692-11e4-9973-070e938df227
> ID#0] Prepare completed. Receiving 5 files(627994 bytes), sending 5
> files(632105 bytes)
> INFO [StreamReceiveTask:22] 2015-01-28 02:07:31,971
> StreamResultFuture.java:180 - [Stream #692c6270-a692-11e4-9973-070e938df227]
> Session with /10.66.187.201 is complete
> INFO [StreamReceiveTask:22] 2015-01-28 02:07:31,972
> StreamResultFuture.java:212 - [Stream #692c6270-a692-11e4-9973-070e938df227]
> All sessions completed
> INFO [StreamReceiveTask:22] 2015-01-28 02:07:31,972
> StreamingRepairTask.java:96 - [repair #685e3d00-a692-11e4-9973-070e938df227]
> streaming task succeed, returning response to /10.98.194.68
> ERROR [RepairJobTask:1] 2015-01-28 02:07:39,444 RepairJob.java:127 - Error
> occurred during snapshot phase
> java.lang.RuntimeException: Could not create snapshot at /10.97.9.110
> at
> org.apache.cassandra.repair.SnapshotTask$SnapshotCallback.onFailure(SnapshotTask.java:77)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.net.MessagingService$5$1.run(MessagingService.java:347)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> ~[na:1.7.0_45]
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> ~[na:1.7.0_45]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> [na:1.7.0_45]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> [na:1.7.0_45]
> at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]
> INFO [AntiEntropySessions:6] 2015-01-28 02:07:39,445 RepairSession.java:260
> - [repair #6f85e740-a692-11e4-9973-070e938df227] new session: will sync
> /10.98.194.68, /10.66.187.201, /10.226.218.135 on range
> (12817179804668051873746972069086
> 2638799,12863540308359254031520865977436165] for events.[bigint0text,
> bigint0boolean, bigint0int, dataset_catalog, column_categories,
> bigint0double, bigint0bigint]
> ERROR [AntiEntropySessions:5] 2015-01-28 02:07:39,445 RepairSession.java:303
> - [repair #685e3d00-a692-11e4-9973-070e938df227] session completed with the
> following error
> java.io.IOException: Failed during snapshot creation.
> at
> org.apache.cassandra.repair.RepairSession.failedSnapshot(RepairSession.java:344)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.repair.RepairJob$2.onFailure(RepairJob.java:128)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at com.google.common.util.concurrent.Futures$4.run(Futures.java:1172)
>
[jira] [Created] (CASSANDRA-9953) Snapshot file handlers are not released after snapshot deleted
Imri Zvik created CASSANDRA-9953:
Summary: Snapshot file handlers are not released after snapshot
deleted
Key: CASSANDRA-9953
URL: https://issues.apache.org/jira/browse/CASSANDRA-9953
Project: Cassandra
Issue Type: Bug
Components: Core
Reporter: Imri Zvik
We are seeing a lot of opened file descriptors to deleted snapshots:
{code}
java128657 cassandra DELREG 253,2 569272514
/var/lib/cassandra/data/accounts/account_store_data/snapshots/feb5f790-316e-11e5-aec0-472b0d6e3fd4/accounts-account_store_data-jb-264593-Index.db
java128657 cassandra DELREG 253,2 1610616657
/var/lib/cassandra/data/accounts/account_store_counters/snapshots/03aa4710-316f-11e5-aec0-472b0d6e3fd4/accounts-account_store_counters-jb-635527-Index.db
java128657 cassandra DELREG 253,2 1610613856
/var/lib/cassandra/data/accounts/account_store_counters/snapshots/43c17170-316f-11e5-aec0-472b0d6e3fd4/accounts-account_store_counters-jb-635675-Index.db
java128657 cassandra DELREG 253,2 1610613052
/var/lib/cassandra/data/accounts/account_store_counters/snapshots/18e001a0-3170-11e5-aec0-472b0d6e3fd4/accounts-account_store_counters-jb-636200-Index.db
[root@cassandra002 ~]# lsof -np 128657 |grep -c DEL
56682
{code}
They are probably created by the routine repair process, but they are never
cleared (restarting the Cassandra process clears them, of course).
There are no errors or fatals in the system.log.
We are using Datastax community edition 2.0.13, installed from RPMs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-9953) Snapshot file handlers are not released after snapshot deleted
[
https://issues.apache.org/jira/browse/CASSANDRA-9953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Imri Zvik updated CASSANDRA-9953:
-
Description:
We are seeing a lot of opened file descriptors to deleted snapshots (deleted
using nodetool clearsnapshot):
{code}
java128657 cassandra DELREG 253,2 569272514
/var/lib/cassandra/data/accounts/account_store_data/snapshots/feb5f790-316e-11e5-aec0-472b0d6e3fd4/accounts-account_store_data-jb-264593-Index.db
java128657 cassandra DELREG 253,2 1610616657
/var/lib/cassandra/data/accounts/account_store_counters/snapshots/03aa4710-316f-11e5-aec0-472b0d6e3fd4/accounts-account_store_counters-jb-635527-Index.db
java128657 cassandra DELREG 253,2 1610613856
/var/lib/cassandra/data/accounts/account_store_counters/snapshots/43c17170-316f-11e5-aec0-472b0d6e3fd4/accounts-account_store_counters-jb-635675-Index.db
java128657 cassandra DELREG 253,2 1610613052
/var/lib/cassandra/data/accounts/account_store_counters/snapshots/18e001a0-3170-11e5-aec0-472b0d6e3fd4/accounts-account_store_counters-jb-636200-Index.db
[root@cassandra002 ~]# lsof -np 128657 |grep -c DEL
56682
{code}
They are probably created by the routine repair process, but they are never
cleared (restarting the Cassandra process clears them, of course).
We are seeing these also after all repair processes finished, and no repair
process is running in the cluster.
There are no errors or fatals in the system.log.
We are using Datastax community edition 2.0.13, installed from RPMs.
was:
We are seeing a lot of opened file descriptors to deleted snapshots:
{code}
java128657 cassandra DELREG 253,2 569272514
/var/lib/cassandra/data/accounts/account_store_data/snapshots/feb5f790-316e-11e5-aec0-472b0d6e3fd4/accounts-account_store_data-jb-264593-Index.db
java128657 cassandra DELREG 253,2 1610616657
/var/lib/cassandra/data/accounts/account_store_counters/snapshots/03aa4710-316f-11e5-aec0-472b0d6e3fd4/accounts-account_store_counters-jb-635527-Index.db
java128657 cassandra DELREG 253,2 1610613856
/var/lib/cassandra/data/accounts/account_store_counters/snapshots/43c17170-316f-11e5-aec0-472b0d6e3fd4/accounts-account_store_counters-jb-635675-Index.db
java128657 cassandra DELREG 253,2 1610613052
/var/lib/cassandra/data/accounts/account_store_counters/snapshots/18e001a0-3170-11e5-aec0-472b0d6e3fd4/accounts-account_store_counters-jb-636200-Index.db
[root@cassandra002 ~]# lsof -np 128657 |grep -c DEL
56682
{code}
They are probably created by the routine repair process, but they are never
cleared (restarting the Cassandra process clears them, of course).
There are no errors or fatals in the system.log.
We are using Datastax community edition 2.0.13, installed from RPMs.
Snapshot file handlers are not released after snapshot deleted
--
Key: CASSANDRA-9953
URL: https://issues.apache.org/jira/browse/CASSANDRA-9953
Project: Cassandra
Issue Type: Bug
Components: Core
Reporter: Imri Zvik
We are seeing a lot of opened file descriptors to deleted snapshots (deleted
using nodetool clearsnapshot):
{code}
java128657 cassandra DELREG 253,2
569272514
/var/lib/cassandra/data/accounts/account_store_data/snapshots/feb5f790-316e-11e5-aec0-472b0d6e3fd4/accounts-account_store_data-jb-264593-Index.db
java128657 cassandra DELREG 253,2
1610616657
/var/lib/cassandra/data/accounts/account_store_counters/snapshots/03aa4710-316f-11e5-aec0-472b0d6e3fd4/accounts-account_store_counters-jb-635527-Index.db
java128657 cassandra DELREG 253,2
1610613856
/var/lib/cassandra/data/accounts/account_store_counters/snapshots/43c17170-316f-11e5-aec0-472b0d6e3fd4/accounts-account_store_counters-jb-635675-Index.db
java128657 cassandra DELREG 253,2
1610613052
/var/lib/cassandra/data/accounts/account_store_counters/snapshots/18e001a0-3170-11e5-aec0-472b0d6e3fd4/accounts-account_store_counters-jb-636200-Index.db
[root@cassandra002 ~]# lsof -np 128657 |grep -c DEL
56682
{code}
They are probably created by the routine repair process, but they are never
cleared (restarting the Cassandra process clears them, of course).
We are seeing these also after all repair processes finished, and no repair
process is running in the cluster.
There are no errors or fatals in the system.log.
We are using Datastax community edition 2.0.13, installed from RPMs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8211) Overlapping sstables in L1+
[
https://issues.apache.org/jira/browse/CASSANDRA-8211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14301143#comment-14301143
]
Imri Zvik commented on CASSANDRA-8211:
--
[~krummas] The same goes to what I'm seeing in CASSANDRA-8210? Also un-related?
Overlapping sstables in L1+
---
Key: CASSANDRA-8211
URL: https://issues.apache.org/jira/browse/CASSANDRA-8211
Project: Cassandra
Issue Type: Bug
Reporter: Marcus Eriksson
Assignee: Marcus Eriksson
Fix For: 2.0.12, 2.1.3
Attachments: 0001-Avoid-overlaps-in-L1-v2.patch,
0001-Avoid-overlaps-in-L1.patch
Seems we have a bug that can create overlapping sstables in L1:
{code}
WARN [main] 2014-10-28 04:09:42,295 LeveledManifest.java (line 164) At level
2, SSTableReader(path='sstable') [DecoratedKey(2838397575996053472, 00
10066059b210066059b210400100),
DecoratedKey(5516674013223138308,
001000ff2d161000ff2d160
00010400100)] overlaps
SSTableReader(path='sstable') [DecoratedKey(2839992722300822584,
0010
00229ad21000229ad210400100),
DecoratedKey(5532836928694021724,
0010034b05a610034b05a6100
000400100)]. This could be caused by a bug in
Cassandra 1.1.0 .. 1.1.3 or due to the fact that you have dropped sstables
from another node into the data directory. Sending back to L0. If
you didn't drop in sstables, and have not yet run scrub, you should do so
since you may also have rows out-of-order within an sstable
{code}
Which might manifest itself during compaction with this exception:
{code}
ERROR [CompactionExecutor:3152] 2014-10-28 00:24:06,134 CassandraDaemon.java
(line 199) Exception in thread Thread[CompactionExecutor:3152,1,main]
java.lang.RuntimeException: Last written key
DecoratedKey(5516674013223138308,
001000ff2d161000ff2d1610400100)
= current key DecoratedKey(2839992722300822584,
001000229ad21000229ad210400100)
writing into sstable
{code}
since we use LeveledScanner when compacting (the backing sstable scanner
might go beyond the start of the next sstable scanner)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8210) java.lang.AssertionError: Memory was freed exception in CompactionExecutor
[
https://issues.apache.org/jira/browse/CASSANDRA-8210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14301151#comment-14301151
]
Imri Zvik commented on CASSANDRA-8210:
--
Sure:
INFO [CompactionExecutor:513] 2015-02-01 09:54:21,803 CompactionManager.java
(line 563) Cleaning up
SSTableReader(path='/var/lib/cassandra/data/accounts/account_store_data/accounts-account_store_data-jb-6-Data.db')
ERROR [CompactionExecutor:513] 2015-02-01 09:54:21,815 CassandraDaemon.java
(line 199) Exception in thread Thread[CompactionExecutor:513,1,RMI Runtime]
java.lang.AssertionError: Memory was freed
at org.apache.cassandra.io.util.Memory.checkPosition(Memory.java:259)
at org.apache.cassandra.io.util.Memory.getInt(Memory.java:211)
at
org.apache.cassandra.io.sstable.IndexSummary.getIndex(IndexSummary.java:79)
at
org.apache.cassandra.io.sstable.IndexSummary.getKey(IndexSummary.java:84)
at
org.apache.cassandra.io.sstable.IndexSummary.binarySearch(IndexSummary.java:58)
at
org.apache.cassandra.io.sstable.SSTableReader.getIndexScanPosition(SSTableReader.java:602)
at
org.apache.cassandra.io.sstable.SSTableReader.getPosition(SSTableReader.java:947)
at
org.apache.cassandra.io.sstable.SSTableReader.getPosition(SSTableReader.java:910)
at
org.apache.cassandra.io.sstable.SSTableReader.getPositionsForRanges(SSTableReader.java:819)
at
org.apache.cassandra.db.ColumnFamilyStore.getExpectedCompactedFileSize(ColumnFamilyStore.java:1088)
at
org.apache.cassandra.db.compaction.CompactionManager.doCleanupCompaction(CompactionManager.java:564)
at
org.apache.cassandra.db.compaction.CompactionManager.access$400(CompactionManager.java:63)
at
org.apache.cassandra.db.compaction.CompactionManager$5.perform(CompactionManager.java:281)
at
org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:225)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
java.lang.AssertionError: Memory was freed exception in CompactionExecutor
Key: CASSANDRA-8210
URL: https://issues.apache.org/jira/browse/CASSANDRA-8210
Project: Cassandra
Issue Type: Bug
Components: Core
Environment: DSE 4.5.2, Cassandra 2.0.10, OEL 6.5, kernel
3.8.13-44.el6uek.x86_64, 128Gb of RAM, swap disabled, JRE 1.7.0_67-b01
Reporter: Nikolai Grigoriev
Priority: Minor
Attachments: cassandra-env.sh, cassandra.yaml, occurence
frequency.png, system.log.gz
I have just got this problem on multiple nodes. Cassandra 2.0.10 (DSE 4.5.2).
After looking through the history I have found that it was actually happening
on all nodes since the start of large compaction process (I've loaded tons of
data in the system and then turned off all load to let it compact the data).
{code}
ERROR [CompactionExecutor:1196] 2014-10-28 17:14:50,124 CassandraDaemon.java
(line 199) Exception in thread Thread[CompactionExecutor:1196,1,main]
java.lang.AssertionError: Memory was freed
at org.apache.cassandra.io.util.Memory.checkPosition(Memory.java:259)
at org.apache.cassandra.io.util.Memory.getInt(Memory.java:211)
at
org.apache.cassandra.io.sstable.IndexSummary.getIndex(IndexSummary.java:79)
at
org.apache.cassandra.io.sstable.IndexSummary.getKey(IndexSummary.java:84)
at
org.apache.cassandra.io.sstable.IndexSummary.binarySearch(IndexSummary.java:58)
at
org.apache.cassandra.io.sstable.SSTableReader.getSampleIndexesForRanges(SSTableReader.java:692)
at
org.apache.cassandra.io.sstable.SSTableReader.estimatedKeysForRanges(SSTableReader.java:663)
at
org.apache.cassandra.db.compaction.AbstractCompactionStrategy.worthDroppingTombstones(AbstractCompactionStrategy.java:328)
at
org.apache.cassandra.db.compaction.LeveledCompactionStrategy.findDroppableSSTable(LeveledCompactionStrategy.java:354)
at
org.apache.cassandra.db.compaction.LeveledCompactionStrategy.getMaximalTask(LeveledCompactionStrategy.java:125)
at
org.apache.cassandra.db.compaction.LeveledCompactionStrategy.getNextBackgroundTask(LeveledCompactionStrategy.java:113)
at
org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:192)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
[jira] [Updated] (CASSANDRA-8716) java.util.concurrent.ExecutionException: java.lang.AssertionError: Memory was freed when running cleanup
[ https://issues.apache.org/jira/browse/CASSANDRA-8716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imri Zvik updated CASSANDRA-8716: - Description: Error occurred during cleanup java.util.concurrent.ExecutionException: java.lang.AssertionError: Memory was freed at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:188) at org.apache.cassandra.db.compaction.CompactionManager.performAllSSTableOperation(CompactionManager.java:234) at org.apache.cassandra.db.compaction.CompactionManager.performCleanup(CompactionManager.java:272) at org.apache.cassandra.db.ColumnFamilyStore.forceCleanup(ColumnFamilyStore.java:1115) at org.apache.cassandra.service.StorageService.forceKeyspaceCleanup(StorageService.java:2177) at sun.reflect.GeneratedMethodAccessor29.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:75) at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:279) at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112) at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:46) at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237) at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138) at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:252) at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819) at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801) at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1487) at javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:97) at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1328) at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1420) at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:848) at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322) at sun.rmi.transport.Transport$1.run(Transport.java:177) at sun.rmi.transport.Transport$1.run(Transport.java:174) at java.security.AccessController.doPrivileged(Native Method) at sun.rmi.transport.Transport.serviceCall(Transport.java:173) at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:556) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:811) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:670) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.AssertionError: Memory was freed at org.apache.cassandra.io.util.Memory.checkPosition(Memory.java:259) at org.apache.cassandra.io.util.Memory.getInt(Memory.java:211) at org.apache.cassandra.io.sstable.IndexSummary.getIndex(IndexSummary.java:79) at org.apache.cassandra.io.sstable.IndexSummary.getKey(IndexSummary.java:84) at org.apache.cassandra.io.sstable.IndexSummary.binarySearch(IndexSummary.java:58) at org.apache.cassandra.io.sstable.SSTableReader.getIndexScanPosition(SSTableReader.java:602) at org.apache.cassandra.io.sstable.SSTableReader.getPosition(SSTableReader.java:947) at org.apache.cassandra.io.sstable.SSTableReader.getPosition(SSTableReader.java:910) at org.apache.cassandra.io.sstable.SSTableReader.getPositionsForRanges(SSTableReader.java:819) at org.apache.cassandra.db.ColumnFamilyStore.getExpectedCompactedFileSize(ColumnFamilyStore.java:1088) at org.apache.cassandra.db.compaction.CompactionManager.doCleanupCompaction(CompactionManager.java:564) at org.apache.cassandra.db.compaction.CompactionManager.access$400(CompactionManager.java:63) at
[jira] [Created] (CASSANDRA-8716) java.util.concurrent.ExecutionException: java.lang.AssertionError: Memory was freed when running cleanup
Imri Zvik created CASSANDRA-8716: Summary: java.util.concurrent.ExecutionException: java.lang.AssertionError: Memory was freed when running cleanup Key: CASSANDRA-8716 URL: https://issues.apache.org/jira/browse/CASSANDRA-8716 Project: Cassandra Issue Type: Bug Components: Core Environment: Centos 6.6, Cassandra 2.0.12, Oracle JDK 1.7.0_67 Reporter: Imri Zvik Priority: Minor Attachments: system.log.gz The error continue to happen even after scrub. If I retry a few time, it may run successfully. I am attaching the server log. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-8210) java.lang.AssertionError: Memory was freed exception in CompactionExecutor
[ https://issues.apache.org/jira/browse/CASSANDRA-8210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14301151#comment-14301151 ] Imri Zvik edited comment on CASSANDRA-8210 at 2/2/15 10:59 AM: --- Sure: INFO [CompactionExecutor:513] 2015-02-01 09:54:21,803 CompactionManager.java (line 563) Cleaning up SSTableReader(path='/var/lib/cassandra/data/accounts/account_store_data/accounts-account_store_data-jb-6-Data.db') ERROR [CompactionExecutor:513] 2015-02-01 09:54:21,815 CassandraDaemon.java (line 199) Exception in thread Thread[CompactionExecutor:513,1,RMI Runtime] java.lang.AssertionError: Memory was freed at org.apache.cassandra.io.util.Memory.checkPosition(Memory.java:259) at org.apache.cassandra.io.util.Memory.getInt(Memory.java:211) at org.apache.cassandra.io.sstable.IndexSummary.getIndex(IndexSummary.java:79) at org.apache.cassandra.io.sstable.IndexSummary.getKey(IndexSummary.java:84) at org.apache.cassandra.io.sstable.IndexSummary.binarySearch(IndexSummary.java:58) at org.apache.cassandra.io.sstable.SSTableReader.getIndexScanPosition(SSTableReader.java:602) at org.apache.cassandra.io.sstable.SSTableReader.getPosition(SSTableReader.java:947) at org.apache.cassandra.io.sstable.SSTableReader.getPosition(SSTableReader.java:910) at org.apache.cassandra.io.sstable.SSTableReader.getPositionsForRanges(SSTableReader.java:819) at org.apache.cassandra.db.ColumnFamilyStore.getExpectedCompactedFileSize(ColumnFamilyStore.java:1088) at org.apache.cassandra.db.compaction.CompactionManager.doCleanupCompaction(CompactionManager.java:564) at org.apache.cassandra.db.compaction.CompactionManager.access$400(CompactionManager.java:63) at org.apache.cassandra.db.compaction.CompactionManager$5.perform(CompactionManager.java:281) at org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:225) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) (By the way, I've seen this on all of our nodes, not just one node) was (Author: imriz): Sure: INFO [CompactionExecutor:513] 2015-02-01 09:54:21,803 CompactionManager.java (line 563) Cleaning up SSTableReader(path='/var/lib/cassandra/data/accounts/account_store_data/accounts-account_store_data-jb-6-Data.db') ERROR [CompactionExecutor:513] 2015-02-01 09:54:21,815 CassandraDaemon.java (line 199) Exception in thread Thread[CompactionExecutor:513,1,RMI Runtime] java.lang.AssertionError: Memory was freed at org.apache.cassandra.io.util.Memory.checkPosition(Memory.java:259) at org.apache.cassandra.io.util.Memory.getInt(Memory.java:211) at org.apache.cassandra.io.sstable.IndexSummary.getIndex(IndexSummary.java:79) at org.apache.cassandra.io.sstable.IndexSummary.getKey(IndexSummary.java:84) at org.apache.cassandra.io.sstable.IndexSummary.binarySearch(IndexSummary.java:58) at org.apache.cassandra.io.sstable.SSTableReader.getIndexScanPosition(SSTableReader.java:602) at org.apache.cassandra.io.sstable.SSTableReader.getPosition(SSTableReader.java:947) at org.apache.cassandra.io.sstable.SSTableReader.getPosition(SSTableReader.java:910) at org.apache.cassandra.io.sstable.SSTableReader.getPositionsForRanges(SSTableReader.java:819) at org.apache.cassandra.db.ColumnFamilyStore.getExpectedCompactedFileSize(ColumnFamilyStore.java:1088) at org.apache.cassandra.db.compaction.CompactionManager.doCleanupCompaction(CompactionManager.java:564) at org.apache.cassandra.db.compaction.CompactionManager.access$400(CompactionManager.java:63) at org.apache.cassandra.db.compaction.CompactionManager$5.perform(CompactionManager.java:281) at org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:225) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) java.lang.AssertionError: Memory was freed exception in CompactionExecutor Key: CASSANDRA-8210 URL: https://issues.apache.org/jira/browse/CASSANDRA-8210 Project: Cassandra Issue Type: Bug Components: Core Environment: DSE 4.5.2, Cassandra 2.0.10, OEL 6.5, kernel 3.8.13-44.el6uek.x86_64, 128Gb of RAM, swap
[jira] [Commented] (CASSANDRA-8211) Overlapping sstables in L1+
[
https://issues.apache.org/jira/browse/CASSANDRA-8211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14301133#comment-14301133
]
Imri Zvik commented on CASSANDRA-8211:
--
Hi Marcus,
Thanks. I'll run the scrub and re verify.
Overlapping sstables in L1+
---
Key: CASSANDRA-8211
URL: https://issues.apache.org/jira/browse/CASSANDRA-8211
Project: Cassandra
Issue Type: Bug
Reporter: Marcus Eriksson
Assignee: Marcus Eriksson
Fix For: 2.0.12, 2.1.3
Attachments: 0001-Avoid-overlaps-in-L1-v2.patch,
0001-Avoid-overlaps-in-L1.patch
Seems we have a bug that can create overlapping sstables in L1:
{code}
WARN [main] 2014-10-28 04:09:42,295 LeveledManifest.java (line 164) At level
2, SSTableReader(path='sstable') [DecoratedKey(2838397575996053472, 00
10066059b210066059b210400100),
DecoratedKey(5516674013223138308,
001000ff2d161000ff2d160
00010400100)] overlaps
SSTableReader(path='sstable') [DecoratedKey(2839992722300822584,
0010
00229ad21000229ad210400100),
DecoratedKey(5532836928694021724,
0010034b05a610034b05a6100
000400100)]. This could be caused by a bug in
Cassandra 1.1.0 .. 1.1.3 or due to the fact that you have dropped sstables
from another node into the data directory. Sending back to L0. If
you didn't drop in sstables, and have not yet run scrub, you should do so
since you may also have rows out-of-order within an sstable
{code}
Which might manifest itself during compaction with this exception:
{code}
ERROR [CompactionExecutor:3152] 2014-10-28 00:24:06,134 CassandraDaemon.java
(line 199) Exception in thread Thread[CompactionExecutor:3152,1,main]
java.lang.RuntimeException: Last written key
DecoratedKey(5516674013223138308,
001000ff2d161000ff2d1610400100)
= current key DecoratedKey(2839992722300822584,
001000229ad21000229ad210400100)
writing into sstable
{code}
since we use LeveledScanner when compacting (the backing sstable scanner
might go beyond the start of the next sstable scanner)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8210) java.lang.AssertionError: Memory was freed exception in CompactionExecutor
[
https://issues.apache.org/jira/browse/CASSANDRA-8210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14301159#comment-14301159
]
Imri Zvik commented on CASSANDRA-8210:
--
[~krummas] Sure - I am currently running scrub on all the nodes (although I
already did that yesterday), just to make sure.
I will try to reproduce it after the scrub is over, and open a new ticket if
the issue will persist.
java.lang.AssertionError: Memory was freed exception in CompactionExecutor
Key: CASSANDRA-8210
URL: https://issues.apache.org/jira/browse/CASSANDRA-8210
Project: Cassandra
Issue Type: Bug
Components: Core
Environment: DSE 4.5.2, Cassandra 2.0.10, OEL 6.5, kernel
3.8.13-44.el6uek.x86_64, 128Gb of RAM, swap disabled, JRE 1.7.0_67-b01
Reporter: Nikolai Grigoriev
Priority: Minor
Attachments: cassandra-env.sh, cassandra.yaml, occurence
frequency.png, system.log.gz
I have just got this problem on multiple nodes. Cassandra 2.0.10 (DSE 4.5.2).
After looking through the history I have found that it was actually happening
on all nodes since the start of large compaction process (I've loaded tons of
data in the system and then turned off all load to let it compact the data).
{code}
ERROR [CompactionExecutor:1196] 2014-10-28 17:14:50,124 CassandraDaemon.java
(line 199) Exception in thread Thread[CompactionExecutor:1196,1,main]
java.lang.AssertionError: Memory was freed
at org.apache.cassandra.io.util.Memory.checkPosition(Memory.java:259)
at org.apache.cassandra.io.util.Memory.getInt(Memory.java:211)
at
org.apache.cassandra.io.sstable.IndexSummary.getIndex(IndexSummary.java:79)
at
org.apache.cassandra.io.sstable.IndexSummary.getKey(IndexSummary.java:84)
at
org.apache.cassandra.io.sstable.IndexSummary.binarySearch(IndexSummary.java:58)
at
org.apache.cassandra.io.sstable.SSTableReader.getSampleIndexesForRanges(SSTableReader.java:692)
at
org.apache.cassandra.io.sstable.SSTableReader.estimatedKeysForRanges(SSTableReader.java:663)
at
org.apache.cassandra.db.compaction.AbstractCompactionStrategy.worthDroppingTombstones(AbstractCompactionStrategy.java:328)
at
org.apache.cassandra.db.compaction.LeveledCompactionStrategy.findDroppableSSTable(LeveledCompactionStrategy.java:354)
at
org.apache.cassandra.db.compaction.LeveledCompactionStrategy.getMaximalTask(LeveledCompactionStrategy.java:125)
at
org.apache.cassandra.db.compaction.LeveledCompactionStrategy.getNextBackgroundTask(LeveledCompactionStrategy.java:113)
at
org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:192)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8548) Nodetool Cleanup - java.lang.AssertionError
[
https://issues.apache.org/jira/browse/CASSANDRA-8548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14300223#comment-14300223
]
Imri Zvik commented on CASSANDRA-8548:
--
I'm still getting this when running cleanup in 2.0.12, even after the
restart-scrub-cleanup procedure.
Nodetool Cleanup - java.lang.AssertionError
---
Key: CASSANDRA-8548
URL: https://issues.apache.org/jira/browse/CASSANDRA-8548
Project: Cassandra
Issue Type: Bug
Reporter: Sebastian Estevez
Assignee: Marcus Eriksson
Fix For: 2.0.12
Attachments: 0001-make-sure-we-unmark-compacting.patch
Needed to free up some space on a node but getting the dump below when
running nodetool cleanup.
Tried turning on debug to try to obtain additional details in the logs but
nothing gets added to the logs when running cleanup. Added:
log4j.logger.org.apache.cassandra.db=DEBUG
in log4j-server.properties
See the stack trace below:
root@cassandra-019:~# nodetool cleanup
{code}Error occurred during cleanup
java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:188)
at
org.apache.cassandra.db.compaction.CompactionManager.performAllSSTableOperation(CompactionManager.java:228)
at
org.apache.cassandra.db.compaction.CompactionManager.performCleanup(CompactionManager.java:266)
at
org.apache.cassandra.db.ColumnFamilyStore.forceCleanup(ColumnFamilyStore.java:1112)
at
org.apache.cassandra.service.StorageService.forceKeyspaceCleanup(StorageService.java:2162)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:75)
at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:279)
at
com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112)
at
com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:46)
at
com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237)
at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:252)
at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819)
at
com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801)
at
javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1487)
at
javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:97)
at
javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1328)
at
javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1420)
at
javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:848)
at sun.reflect.GeneratedMethodAccessor64.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
at sun.rmi.transport.Transport$1.run(Transport.java:177)
at sun.rmi.transport.Transport$1.run(Transport.java:174)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
at
sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:556)
at
sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:811)
at
sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:670)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: java.lang.IllegalArgumentException
at java.nio.Buffer.limit(Buffer.java:267)
at
[jira] [Comment Edited] (CASSANDRA-8548) Nodetool Cleanup - java.lang.AssertionError
[
https://issues.apache.org/jira/browse/CASSANDRA-8548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14300223#comment-14300223
]
Imri Zvik edited comment on CASSANDRA-8548 at 2/1/15 6:43 PM:
--
I'm still getting this when running cleanup in 2.0.12, even after the
restart-scrub-cleanup procedure.
*EDIT*
Actually, when I examine the stack traces, in my case they are different. I
will open a new issue regarding my instance
was (Author: imriz):
I'm still getting this when running cleanup in 2.0.12, even after the
restart-scrub-cleanup procedure.
Nodetool Cleanup - java.lang.AssertionError
---
Key: CASSANDRA-8548
URL: https://issues.apache.org/jira/browse/CASSANDRA-8548
Project: Cassandra
Issue Type: Bug
Reporter: Sebastian Estevez
Assignee: Marcus Eriksson
Fix For: 2.0.12
Attachments: 0001-make-sure-we-unmark-compacting.patch
Needed to free up some space on a node but getting the dump below when
running nodetool cleanup.
Tried turning on debug to try to obtain additional details in the logs but
nothing gets added to the logs when running cleanup. Added:
log4j.logger.org.apache.cassandra.db=DEBUG
in log4j-server.properties
See the stack trace below:
root@cassandra-019:~# nodetool cleanup
{code}Error occurred during cleanup
java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:188)
at
org.apache.cassandra.db.compaction.CompactionManager.performAllSSTableOperation(CompactionManager.java:228)
at
org.apache.cassandra.db.compaction.CompactionManager.performCleanup(CompactionManager.java:266)
at
org.apache.cassandra.db.ColumnFamilyStore.forceCleanup(ColumnFamilyStore.java:1112)
at
org.apache.cassandra.service.StorageService.forceKeyspaceCleanup(StorageService.java:2162)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:75)
at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:279)
at
com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112)
at
com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:46)
at
com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237)
at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:252)
at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819)
at
com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801)
at
javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1487)
at
javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:97)
at
javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1328)
at
javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1420)
at
javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:848)
at sun.reflect.GeneratedMethodAccessor64.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
at sun.rmi.transport.Transport$1.run(Transport.java:177)
at sun.rmi.transport.Transport$1.run(Transport.java:174)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
at
sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:556)
at
sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:811)
at
sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:670)
at
[jira] [Commented] (CASSANDRA-8210) java.lang.AssertionError: Memory was freed exception in CompactionExecutor
[ https://issues.apache.org/jira/browse/CASSANDRA-8210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14300409#comment-14300409 ] Imri Zvik commented on CASSANDRA-8210: -- I am getting a similar error when I run cleanup (using 2.0.12): [root@cassandra005 ~]# nodetool cleanup Error occurred during cleanup java.util.concurrent.ExecutionException: java.lang.AssertionError: Memory was freed at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:188) at org.apache.cassandra.db.compaction.CompactionManager.performAllSSTableOperation(CompactionManager.java:234) at org.apache.cassandra.db.compaction.CompactionManager.performCleanup(CompactionManager.java:272) at org.apache.cassandra.db.ColumnFamilyStore.forceCleanup(ColumnFamilyStore.java:1115) at org.apache.cassandra.service.StorageService.forceKeyspaceCleanup(StorageService.java:2177) at sun.reflect.GeneratedMethodAccessor70.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:75) at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:279) at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112) at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:46) at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237) at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138) at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:252) at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819) at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801) at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1487) at javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:97) at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1328) at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1420) at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:848) at sun.reflect.GeneratedMethodAccessor34.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322) at sun.rmi.transport.Transport$1.run(Transport.java:177) at sun.rmi.transport.Transport$1.run(Transport.java:174) at java.security.AccessController.doPrivileged(Native Method) at sun.rmi.transport.Transport.serviceCall(Transport.java:173) at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:556) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:811) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:670) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.AssertionError: Memory was freed at org.apache.cassandra.io.util.Memory.checkPosition(Memory.java:259) at org.apache.cassandra.io.util.Memory.getInt(Memory.java:211) at org.apache.cassandra.io.sstable.IndexSummary.getIndex(IndexSummary.java:79) at org.apache.cassandra.io.sstable.IndexSummary.getKey(IndexSummary.java:84) at org.apache.cassandra.io.sstable.IndexSummary.binarySearch(IndexSummary.java:58) at org.apache.cassandra.io.sstable.SSTableReader.getIndexScanPosition(SSTableReader.java:602) at org.apache.cassandra.io.sstable.SSTableReader.getPosition(SSTableReader.java:947) at org.apache.cassandra.io.sstable.SSTableReader.getPosition(SSTableReader.java:910) at org.apache.cassandra.io.sstable.SSTableReader.getPositionsForRanges(SSTableReader.java:819) at org.apache.cassandra.db.ColumnFamilyStore.getExpectedCompactedFileSize(ColumnFamilyStore.java:1088) at org.apache.cassandra.db.compaction.CompactionManager.doCleanupCompaction(CompactionManager.java:564) at
[jira] [Commented] (CASSANDRA-8211) Overlapping sstables in L1+
[
https://issues.apache.org/jira/browse/CASSANDRA-8211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14300933#comment-14300933
]
Imri Zvik commented on CASSANDRA-8211:
--
I am still seeing this/related symptoms (CASSANDRA-8191), after upgrading to
2.0.12:
ERROR [CompactionExecutor:1116] 2015-02-01 20:47:46,448 CassandraDaemon.java
(line 199) Exception in thread Thread[CompactionExecutor:1116,1,main]
java.lang.RuntimeException: Last written key DecoratedKey(-9223372036854775808,
) = current key DecoratedKey(-9223372036854775808, ) writing into
/var/lib/cassandra/data/system/compaction_history/system-compaction_history-tmp-jb-12-Data.db
at
org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:143)
at
org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:166)
at
org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:167)
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
at
org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:60)
at
org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
at
org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:198)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
I am also experiencing similar symptoms to those described in CASSANDRA-8210.
Overlapping sstables in L1+
---
Key: CASSANDRA-8211
URL: https://issues.apache.org/jira/browse/CASSANDRA-8211
Project: Cassandra
Issue Type: Bug
Reporter: Marcus Eriksson
Assignee: Marcus Eriksson
Fix For: 2.0.12, 2.1.3
Attachments: 0001-Avoid-overlaps-in-L1-v2.patch,
0001-Avoid-overlaps-in-L1.patch
Seems we have a bug that can create overlapping sstables in L1:
{code}
WARN [main] 2014-10-28 04:09:42,295 LeveledManifest.java (line 164) At level
2, SSTableReader(path='sstable') [DecoratedKey(2838397575996053472, 00
10066059b210066059b210400100),
DecoratedKey(5516674013223138308,
001000ff2d161000ff2d160
00010400100)] overlaps
SSTableReader(path='sstable') [DecoratedKey(2839992722300822584,
0010
00229ad21000229ad210400100),
DecoratedKey(5532836928694021724,
0010034b05a610034b05a6100
000400100)]. This could be caused by a bug in
Cassandra 1.1.0 .. 1.1.3 or due to the fact that you have dropped sstables
from another node into the data directory. Sending back to L0. If
you didn't drop in sstables, and have not yet run scrub, you should do so
since you may also have rows out-of-order within an sstable
{code}
Which might manifest itself during compaction with this exception:
{code}
ERROR [CompactionExecutor:3152] 2014-10-28 00:24:06,134 CassandraDaemon.java
(line 199) Exception in thread Thread[CompactionExecutor:3152,1,main]
java.lang.RuntimeException: Last written key
DecoratedKey(5516674013223138308,
001000ff2d161000ff2d1610400100)
= current key DecoratedKey(2839992722300822584,
001000229ad21000229ad210400100)
writing into sstable
{code}
since we use LeveledScanner when compacting (the backing sstable scanner
might go beyond the start of the next sstable scanner)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
