[jira] [Commented] (CASSANDRA-7103) Very poor performance with simple setup
[
https://issues.apache.org/jira/browse/CASSANDRA-7103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13984424#comment-13984424
]
Martin Bligh commented on CASSANDRA-7103:
-
FWIW, in case anyone trips across this later:
changing PRIMARY KEY (time_order, time_start) to PRIMARY KEY ((time_order,
time_start)) fixed the insert side of it - ie using both for the storage row
index. And this: http://www.datastax.com/dev/blog/whats-new-in-cql-3-0
explained it in more detail, and now I can at least see what they were trying
to say in the wiki ;-) Stops me from being able to use < and > operators, but
that's a different problem ...
Thanks for the explanation.
> Very poor performance with simple setup
> ---
>
> Key: CASSANDRA-7103
> URL: https://issues.apache.org/jira/browse/CASSANDRA-7103
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: Fedora 19 (also happens on Ubuntu), Cassandra 2.0.7. dsc
> standard install
>Reporter: Martin Bligh
>
> Single node (this is just development, 32GB 20 core server), single disk
> array.
> Create the following table:
> {code}
> CREATE TABLE reut (
> time_order bigint,
> time_start bigint,
> ack_us map,
> gc_strategy map,
> gc_strategy_symbol map,
> gc_symbol map,
> ge_strategy map,
> ge_strategy_symbol map,
> ge_symbol map,
> go_strategy map,
> go_strategy_symbol map,
> go_symbol map,
> message_type map,
> PRIMARY KEY (time_order, time_start)
> ) WITH
> bloom_filter_fp_chance=0.01 AND
> caching='KEYS_ONLY' AND
> comment='' AND
> dclocal_read_repair_chance=0.00 AND
> gc_grace_seconds=864000 AND
> index_interval=128 AND
> read_repair_chance=0.10 AND
> replicate_on_write='true' AND
> populate_io_cache_on_flush='false' AND
> default_time_to_live=0 AND
> speculative_retry='99.0PERCENTILE' AND
> memtable_flush_period_in_ms=0 AND
> compaction={'class': 'SizeTieredCompactionStrategy'} AND
> compression={};
> {code}
> Now I just insert data into it (using python driver, async insert, prepared
> insert statement). Each row only fills out one of the gc_*, go_*, or ge_*
> columns, and there's something like 20-100 entries per map column,
> occasionally 1000, but it's nothing huge.
> First run 685 inserts in 1.004860 seconds (681.687053 Operations/s).
> OK, not great, but that's fine.
> Now throw 50,000 rows at it.
> Now run the first run again, and it takes 53s to do the same insert of 685
> rows - I'm getting about 10 rows per second.
> It's not IO bound - "iostat 1" shows quiescent for 9 seconds, then ~640KB
> write, then sleeps again - seems like the fflush sync.
> Run "nodetool flush" and performance goes back to as before
> Not sure why this gets so slow - I think it just builds huge commit logs and
> memtables, but never writes out to the data/ directory with sstables because
> I only have one table? That doesn't seem like a good situation.
> Worse ... if you let the python driver just throw stuff at it async (I think
> this allows up to 128 request if I understand the underlying protocol, then
> it gets so slow that a single write takes over 10s and times out). Seems to
> be some sort of synchronization problem in Java ... if I limit the concurrent
> async requests to the left column below, I get the number of seconds elapsed
> on the right:
> 1: 103 seconds
> 2: 63 seconds
> 8: 53 seconds
> 16: 53 seconds
> 32: 66 seconds
> 64: so slow it explodes in timeouts on write (over 10s each).
> I guess there's some thundering herd type locking issue in whatever Java
> primitive you are using to lock concurrent access to a single table. I know
> some of the Java concurrent.* stuff has this issue. So for the other tests
> above, I was limiting async writes to 16 pending.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (CASSANDRA-7103) Very poor performance with simple setup
[
https://issues.apache.org/jira/browse/CASSANDRA-7103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13984223#comment-13984223
]
Martin Bligh commented on CASSANDRA-7103:
-
OK, that explains some stuff ... will take it to user list as you suggest. I
was under the impression that the row key was the whole of the primary key, not
just the partition key, as described here, for instance:
http://wiki.apache.org/cassandra/DataModel
"The first component of a table's primary key is the partition key; within a
partition, rows are clustered by the remaining columns of the PK."
In which case, that description seems very misleading
> Very poor performance with simple setup
> ---
>
> Key: CASSANDRA-7103
> URL: https://issues.apache.org/jira/browse/CASSANDRA-7103
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: Fedora 19 (also happens on Ubuntu), Cassandra 2.0.7. dsc
> standard install
>Reporter: Martin Bligh
>
> Single node (this is just development, 32GB 20 core server), single disk
> array.
> Create the following table:
> {code}
> CREATE TABLE reut (
> time_order bigint,
> time_start bigint,
> ack_us map,
> gc_strategy map,
> gc_strategy_symbol map,
> gc_symbol map,
> ge_strategy map,
> ge_strategy_symbol map,
> ge_symbol map,
> go_strategy map,
> go_strategy_symbol map,
> go_symbol map,
> message_type map,
> PRIMARY KEY (time_order, time_start)
> ) WITH
> bloom_filter_fp_chance=0.01 AND
> caching='KEYS_ONLY' AND
> comment='' AND
> dclocal_read_repair_chance=0.00 AND
> gc_grace_seconds=864000 AND
> index_interval=128 AND
> read_repair_chance=0.10 AND
> replicate_on_write='true' AND
> populate_io_cache_on_flush='false' AND
> default_time_to_live=0 AND
> speculative_retry='99.0PERCENTILE' AND
> memtable_flush_period_in_ms=0 AND
> compaction={'class': 'SizeTieredCompactionStrategy'} AND
> compression={};
> {code}
> Now I just insert data into it (using python driver, async insert, prepared
> insert statement). Each row only fills out one of the gc_*, go_*, or ge_*
> columns, and there's something like 20-100 entries per map column,
> occasionally 1000, but it's nothing huge.
> First run 685 inserts in 1.004860 seconds (681.687053 Operations/s).
> OK, not great, but that's fine.
> Now throw 50,000 rows at it.
> Now run the first run again, and it takes 53s to do the same insert of 685
> rows - I'm getting about 10 rows per second.
> It's not IO bound - "iostat 1" shows quiescent for 9 seconds, then ~640KB
> write, then sleeps again - seems like the fflush sync.
> Run "nodetool flush" and performance goes back to as before
> Not sure why this gets so slow - I think it just builds huge commit logs and
> memtables, but never writes out to the data/ directory with sstables because
> I only have one table? That doesn't seem like a good situation.
> Worse ... if you let the python driver just throw stuff at it async (I think
> this allows up to 128 request if I understand the underlying protocol, then
> it gets so slow that a single write takes over 10s and times out). Seems to
> be some sort of synchronization problem in Java ... if I limit the concurrent
> async requests to the left column below, I get the number of seconds elapsed
> on the right:
> 1: 103 seconds
> 2: 63 seconds
> 8: 53 seconds
> 16: 53 seconds
> 32: 66 seconds
> 64: so slow it explodes in timeouts on write (over 10s each).
> I guess there's some thundering herd type locking issue in whatever Java
> primitive you are using to lock concurrent access to a single table. I know
> some of the Java concurrent.* stuff has this issue. So for the other tests
> above, I was limiting async writes to 16 pending.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Reopened] (CASSANDRA-7103) Very poor performance with simple setup
[
https://issues.apache.org/jira/browse/CASSANDRA-7103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Martin Bligh reopened CASSANDRA-7103:
-
We're not understanding each other. I'm not updating the same row, I'm just
inserting new ones, so 5417 doesn't seem to be relevant.
There are two fundamental and very serious performance problems in here:
1. There's a 70x performance degradation over time as I insert a few thousand
rows, that's fixed by running "nodetool flush".
2. Scaling of parallel row inserts (just new inserts, not updates) on a single
table per node is terrible. 10s per insert with 64 writers ?
If you want me to break this out into two separate bugs, thats fine, I probably
should have done that to start with.
> Very poor performance with simple setup
> ---
>
> Key: CASSANDRA-7103
> URL: https://issues.apache.org/jira/browse/CASSANDRA-7103
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: Fedora 19 (also happens on Ubuntu), Cassandra 2.0.7. dsc
> standard install
>Reporter: Martin Bligh
>
> Single node (this is just development, 32GB 20 core server), single disk
> array.
> Create the following table:
> {code}
> CREATE TABLE reut (
> time_order bigint,
> time_start bigint,
> ack_us map,
> gc_strategy map,
> gc_strategy_symbol map,
> gc_symbol map,
> ge_strategy map,
> ge_strategy_symbol map,
> ge_symbol map,
> go_strategy map,
> go_strategy_symbol map,
> go_symbol map,
> message_type map,
> PRIMARY KEY (time_order, time_start)
> ) WITH
> bloom_filter_fp_chance=0.01 AND
> caching='KEYS_ONLY' AND
> comment='' AND
> dclocal_read_repair_chance=0.00 AND
> gc_grace_seconds=864000 AND
> index_interval=128 AND
> read_repair_chance=0.10 AND
> replicate_on_write='true' AND
> populate_io_cache_on_flush='false' AND
> default_time_to_live=0 AND
> speculative_retry='99.0PERCENTILE' AND
> memtable_flush_period_in_ms=0 AND
> compaction={'class': 'SizeTieredCompactionStrategy'} AND
> compression={};
> {code}
> Now I just insert data into it (using python driver, async insert, prepared
> insert statement). Each row only fills out one of the gc_*, go_*, or ge_*
> columns, and there's something like 20-100 entries per map column,
> occasionally 1000, but it's nothing huge.
> First run 685 inserts in 1.004860 seconds (681.687053 Operations/s).
> OK, not great, but that's fine.
> Now throw 50,000 rows at it.
> Now run the first run again, and it takes 53s to do the same insert of 685
> rows - I'm getting about 10 rows per second.
> It's not IO bound - "iostat 1" shows quiescent for 9 seconds, then ~640KB
> write, then sleeps again - seems like the fflush sync.
> Run "nodetool flush" and performance goes back to as before
> Not sure why this gets so slow - I think it just builds huge commit logs and
> memtables, but never writes out to the data/ directory with sstables because
> I only have one table? That doesn't seem like a good situation.
> Worse ... if you let the python driver just throw stuff at it async (I think
> this allows up to 128 request if I understand the underlying protocol, then
> it gets so slow that a single write takes over 10s and times out). Seems to
> be some sort of synchronization problem in Java ... if I limit the concurrent
> async requests to the left column below, I get the number of seconds elapsed
> on the right:
> 1: 103 seconds
> 2: 63 seconds
> 8: 53 seconds
> 16: 53 seconds
> 32: 66 seconds
> 64: so slow it explodes in timeouts on write (over 10s each).
> I guess there's some thundering herd type locking issue in whatever Java
> primitive you are using to lock concurrent access to a single table. I know
> some of the Java concurrent.* stuff has this issue. So for the other tests
> above, I was limiting async writes to 16 pending.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (CASSANDRA-7103) Very poor performance with simple setup
[
https://issues.apache.org/jira/browse/CASSANDRA-7103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13983567#comment-13983567
]
Martin Bligh commented on CASSANDRA-7103:
-
BTW, I'm aware that this is
(a) a single node
(b) half my time_order entries are zero (which I don't think matters as it's a
single node anyway), so my partition key doesn't have much variance
(c) disk is not performant (but we're not even trying to write to it from
iostat, so I don't think this matters).
(d) I'm writing to one table
(e) I'm using a single writer.
So I'm creating a "hotspot" of some form. But really,
1. I think it should be able to handle more than 700 writes a second to one
table.
2. It shouldn't degrade to about 10 writes per second.
Sure, I could throw masses of hardware at it, and make it scale a bit better,
but ... unless it can perform better than this on a single table, single node,
I don't see how it'd perform in any reasonable fashion on a larger cluster.
> Very poor performance with simple setup
> ---
>
> Key: CASSANDRA-7103
> URL: https://issues.apache.org/jira/browse/CASSANDRA-7103
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: Fedora 19 (also happens on Ubuntu), Cassandra 2.0.7. dsc
> standard install
>Reporter: Martin Bligh
>
> Single node (this is just development, 32GB 20 core server), single disk
> array.
> Create the following table:
> CREATE TABLE reut (
> time_order bigint,
> time_start bigint,
> ack_us map,
> gc_strategy map,
> gc_strategy_symbol map,
> gc_symbol map,
> ge_strategy map,
> ge_strategy_symbol map,
> ge_symbol map,
> go_strategy map,
> go_strategy_symbol map,
> go_symbol map,
> message_type map,
> PRIMARY KEY (time_order, time_start)
> ) WITH
> bloom_filter_fp_chance=0.01 AND
> caching='KEYS_ONLY' AND
> comment='' AND
> dclocal_read_repair_chance=0.00 AND
> gc_grace_seconds=864000 AND
> index_interval=128 AND
> read_repair_chance=0.10 AND
> replicate_on_write='true' AND
> populate_io_cache_on_flush='false' AND
> default_time_to_live=0 AND
> speculative_retry='99.0PERCENTILE' AND
> memtable_flush_period_in_ms=0 AND
> compaction={'class': 'SizeTieredCompactionStrategy'} AND
> compression={};
> Now I just insert data into it (using python driver, async insert, prepared
> insert statement). Each row only fills out one of the gc_*, go_*, or ge_*
> columns, and there's something like 20-100 entries per map column,
> occasionally 1000, but it's nothing huge.
> First run 685 inserts in 1.004860 seconds (681.687053 Operations/s).
> OK, not great, but that's fine.
> Now throw 50,000 rows at it.
> Now run the first run again, and it takes 53s to do the same insert of 685
> rows - I'm getting about 10 rows per second.
> It's not IO bound - "iostat 1" shows quiescent for 9 seconds, then ~640KB
> write, then sleeps again - seems like the fflush sync.
> Run "nodetool flush" and performance goes back to as before
> Not sure why this gets so slow - I think it just builds huge commit logs and
> memtables, but never writes out to the data/ directory with sstables because
> I only have one table? That doesn't seem like a good situation.
> Worse ... if you let the python driver just throw stuff at it async (I think
> this allows up to 128 request if I understand the underlying protocol, then
> it gets so slow that a single write takes over 10s and times out). Seems to
> be some sort of synchronization problem in Java ... if I limit the concurrent
> async requests to the left column below, I get the number of seconds elapsed
> on the right:
> 1: 103 seconds
> 2: 63 seconds
> 8: 53 seconds
> 16: 53 seconds
> 32: 66 seconds
> 64: so slow it explodes in timeouts on write (over 10s each).
> I guess there's some thundering herd type locking issue in whatever Java
> primitive you are using to lock concurrent access to a single table. I know
> some of the Java concurrent.* stuff has this issue. So for the other tests
> above, I was limiting async writes to 16 pending.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (CASSANDRA-7103) Very poor performance with simple setup
Martin Bligh created CASSANDRA-7103:
---
Summary: Very poor performance with simple setup
Key: CASSANDRA-7103
URL: https://issues.apache.org/jira/browse/CASSANDRA-7103
Project: Cassandra
Issue Type: Bug
Components: Core
Environment: Fedora 19 (also happens on Ubuntu), Cassandra 2.0.7. dsc
standard install
Reporter: Martin Bligh
Single node (this is just development, 32GB 20 core server), single disk array.
Create the following table:
CREATE TABLE reut (
time_order bigint,
time_start bigint,
ack_us map,
gc_strategy map,
gc_strategy_symbol map,
gc_symbol map,
ge_strategy map,
ge_strategy_symbol map,
ge_symbol map,
go_strategy map,
go_strategy_symbol map,
go_symbol map,
message_type map,
PRIMARY KEY (time_order, time_start)
) WITH
bloom_filter_fp_chance=0.01 AND
caching='KEYS_ONLY' AND
comment='' AND
dclocal_read_repair_chance=0.00 AND
gc_grace_seconds=864000 AND
index_interval=128 AND
read_repair_chance=0.10 AND
replicate_on_write='true' AND
populate_io_cache_on_flush='false' AND
default_time_to_live=0 AND
speculative_retry='99.0PERCENTILE' AND
memtable_flush_period_in_ms=0 AND
compaction={'class': 'SizeTieredCompactionStrategy'} AND
compression={};
Now I just insert data into it (using python driver, async insert, prepared
insert statement). Each row only fills out one of the gc_*, go_*, or ge_*
columns, and there's something like 20-100 entries per map column, occasionally
1000, but it's nothing huge.
First run 685 inserts in 1.004860 seconds (681.687053 Operations/s).
OK, not great, but that's fine.
Now throw 50,000 rows at it.
Now run the first run again, and it takes 53s to do the same insert of 685 rows
- I'm getting about 10 rows per second.
It's not IO bound - "iostat 1" shows quiescent for 9 seconds, then ~640KB
write, then sleeps again - seems like the fflush sync.
Run "nodetool flush" and performance goes back to as before
Not sure why this gets so slow - I think it just builds huge commit logs and
memtables, but never writes out to the data/ directory with sstables because I
only have one table? That doesn't seem like a good situation.
Worse ... if you let the python driver just throw stuff at it async (I think
this allows up to 128 request if I understand the underlying protocol, then it
gets so slow that a single write takes over 10s and times out). Seems to be
some sort of synchronization problem in Java ... if I limit the concurrent
async requests to the left column below, I get the number of seconds elapsed on
the right:
1: 103 seconds
2: 63 seconds
8: 53 seconds
16: 53 seconds
32: 66 seconds
64: so slow it explodes in timeouts on write (over 10s each).
I guess there's some thundering herd type locking issue in whatever Java
primitive you are using to lock concurrent access to a single table. I know
some of the Java concurrent.* stuff has this issue. So for the other tests
above, I was limiting async writes to 16 pending.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (CASSANDRA-6525) Cannot select data which using "WHERE"
[ https://issues.apache.org/jira/browse/CASSANDRA-6525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13966868#comment-13966868 ] Martin Bligh commented on CASSANDRA-6525: - (copied from 6981) I thought it was interesting how far apart these two numbers were: "java.io.IOError: java.io.IOException: mmap segment underflow; remaining is 20402577 but 1879048192 requested" And that the requested number is vaguely close to 2^^31 - did something do a negative number and wrap a 32 bit signed here? To be fair, it's not that close to 2^^31, but still way off what was expected? > Cannot select data which using "WHERE" > -- > > Key: CASSANDRA-6525 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6525 > Project: Cassandra > Issue Type: Bug > Environment: Linux RHEL5 > RAM: 1GB > Cassandra 2.0.3 > CQL spec 3.1.1 > Thrift protocol 19.38.0 >Reporter: Silence Chow >Assignee: Michael Shuler > > I am developing a system on my single machine using VMware Player with 1GB > Ram and 1Gb HHD. When I select all data, I didn't have any problems. But when > I using "WHERE" and it has just below 10 records. I have got this error in > system log: > ERROR [ReadStage:41] 2013-12-25 18:52:11,913 CassandraDaemon.java (line 187) > Exception in thread Thread[ReadStage:41,5,main] > java.io.IOError: java.io.EOFException > at org.apache.cassandra.db.Column$1.computeNext(Column.java:79) > at org.apache.cassandra.db.Column$1.computeNext(Column.java:64) > at > com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) > at > com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) > at > org.apache.cassandra.db.columniterator.SimpleSliceReader.computeNext(SimpleSliceReader.java:88) > at > org.apache.cassandra.db.columniterator.SimpleSliceReader.computeNext(SimpleSliceReader.java:37) > at > com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) > at > com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) > at > org.apache.cassandra.db.columniterator.SSTableSliceIterator.hasNext(SSTableSliceIterator.java:82) > at > org.apache.cassandra.db.filter.QueryFilter$2.getNext(QueryFilter.java:157) > at > org.apache.cassandra.db.filter.QueryFilter$2.hasNext(QueryFilter.java:140) > at > org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:144) > at > org.apache.cassandra.utils.MergeIterator$ManyToOne.(MergeIterator.java:87) > at org.apache.cassandra.utils.MergeIterator.get(MergeIterator.java:46) > at > org.apache.cassandra.db.filter.QueryFilter.collateColumns(QueryFilter.java:120) > at > org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:80) > at > org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:72) > at > org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:297) > at > org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:53) > at > org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1487) > at > org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1306) > at org.apache.cassandra.db.Keyspace.getRow(Keyspace.java:332) > at > org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:65) > at > org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:1401) > at > org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1936) > at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) > at java.lang.Thread.run(Unknown Source) > Caused by: java.io.EOFException > at java.io.RandomAccessFile.readFully(Unknown Source) > at java.io.RandomAccessFile.readFully(Unknown Source) > at > org.apache.cassandra.io.util.RandomAccessReader.readBytes(RandomAccessReader.java:348) > at > org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:392) > at > org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:371) > at > org.apache.cassandra.db.OnDiskAtom$Serializer.deserializeFromSSTable(OnDiskAtom.java:74) > at org.apache.cassandra.db.Column$1.computeNext(Column.java:75) > ... 27 more > E.g. > SELECT * FROM table; > Its fine. > SELECT * FROM table WHERE field = 'N'; > field is the partition key. > Its said "Request did not complete within rpc_timeout.
[jira] [Comment Edited] (CASSANDRA-6981) java.io.EOFException from Cassandra when doing select
[
https://issues.apache.org/jira/browse/CASSANDRA-6981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13965328#comment-13965328
]
Martin Bligh edited comment on CASSANDRA-6981 at 4/10/14 1:23 PM:
--
So this is a little tricky, because it's proprietary data and I've changed
things around a bit since then. Basically what I was doing was on a desktop
machine with 32GB of RAM and just one disk (regular HDD, not SSD), created
about 16 tables, all the same, each with about 5 text fields and 5 binary
fields. Most of those fields had a secondary index. Then insert into all the
tables in parallel.
I'm aware this isn't a great design scheme, but it certainly shouldn't fall
over like this
PS. I've never seen this again since disabling mmap access, but am fairly sure
that's great for performance.
was (Author: mbligh):
So this is a little tricky, because it's proprietary data and I've changed
things around a bit since then. Basically what I was doing was on a desktop
machine with 32GB of RAM and just one disk (regular HDD, not SSD), created
about 16 tables, all the same, each with about 5 text fields and 5 binary
fields. Most of those fields had a secondary index. Then insert into all the
tables in parallel.
I'm aware this isn't a great design scheme, but it certainly shouldn't fall
over like this
> java.io.EOFException from Cassandra when doing select
> -
>
> Key: CASSANDRA-6981
> URL: https://issues.apache.org/jira/browse/CASSANDRA-6981
> Project: Cassandra
> Issue Type: Bug
> Environment: Cassandra 2.06, Oracle Java version "1.7.0_51", Linux
> Mint 16
>Reporter: Martin Bligh
>
> Cassandra 2.06, Oracle Java version "1.7.0_51", Linux Mint 16
> I have a cassandra keyspace with about 12 tables that are all the same.
> If I load 100,000 rows or so into a couple of those tables in Cassandra, it
> works fine.
> If I load a larger dataset, after a while one of the tables won't do lookups
> any more (not always the same one).
> {noformat}
> SELECT recv_time,symbol from table6 where mid='S-AUR01-20140324A-1221';
> {noformat}
> results in "Request did not complete within rpc_timeout."
> where "mid" is the primary key (varchar). If I look at the logs, it has an
> EOFException ... presumably it's running out of some resource (it's
> definitely not out of disk space)
> Sometimes it does this on secondary indexes too: dropping and rebuilding the
> index will fix it for a while. When it's broken, it seems like only one
> particular lookup key causes timeouts (and the EOFException every time) -
> other lookups work fine. I presume the index is corrupt somehow.
> {noformat}
> ERROR [ReadStage:110] 2014-04-03 12:39:47,018 CassandraDaemon.java (line 196)
> Exception in thread Thread[ReadStage:110,5,main]
> java.io.IOError: java.io.EOFException
> at org.apache.cassandra.db.Column$1.computeNext(Column.java:79)
> at org.apache.cassandra.db.Column$1.computeNext(Column.java:64)
> at
> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
> at
> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
> at
> org.apache.cassandra.db.columniterator.SimpleSliceReader.computeNext(SimpleSliceReader.java:88)
> at
> org.apache.cassandra.db.columniterator.SimpleSliceReader.computeNext(SimpleSliceReader.java:37)
> at
> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
> at
> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
> at
> org.apache.cassandra.db.columniterator.SSTableSliceIterator.hasNext(SSTableSliceIterator.java:82)
> at
> org.apache.cassandra.db.filter.QueryFilter$2.getNext(QueryFilter.java:157)
> at
> org.apache.cassandra.db.filter.QueryFilter$2.hasNext(QueryFilter.java:140)
> at
> org.apache.cassandra.utils.MergeIterator$OneToOne.computeNext(MergeIterator.java:200)
> at
> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
> at
> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
> at
> org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(SliceQueryFilter.java:185)
> at
> org.apache.cassandra.db.filter.QueryFilter.collateColumns(QueryFilter.java:122)
> at
> org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:80)
> at
> org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:72)
> at
> org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:297)
> at
> org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:53)
> at
> org.apache.cassandra.db.ColumnFamilySto
[jira] [Commented] (CASSANDRA-6981) java.io.EOFException from Cassandra when doing select
[
https://issues.apache.org/jira/browse/CASSANDRA-6981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13965328#comment-13965328
]
Martin Bligh commented on CASSANDRA-6981:
-
So this is a little tricky, because it's proprietary data and I've changed
things around a bit since then. Basically what I was doing was on a desktop
machine with 32GB of RAM and just one disk (regular HDD, not SSD), created
about 16 tables, all the same, each with about 5 text fields and 5 binary
fields. Most of those fields had a secondary index. Then insert into all the
tables in parallel.
I'm aware this isn't a great design scheme, but it certainly shouldn't fall
over like this
> java.io.EOFException from Cassandra when doing select
> -
>
> Key: CASSANDRA-6981
> URL: https://issues.apache.org/jira/browse/CASSANDRA-6981
> Project: Cassandra
> Issue Type: Bug
> Environment: Cassandra 2.06, Oracle Java version "1.7.0_51", Linux
> Mint 16
>Reporter: Martin Bligh
>
> Cassandra 2.06, Oracle Java version "1.7.0_51", Linux Mint 16
> I have a cassandra keyspace with about 12 tables that are all the same.
> If I load 100,000 rows or so into a couple of those tables in Cassandra, it
> works fine.
> If I load a larger dataset, after a while one of the tables won't do lookups
> any more (not always the same one).
> {noformat}
> SELECT recv_time,symbol from table6 where mid='S-AUR01-20140324A-1221';
> {noformat}
> results in "Request did not complete within rpc_timeout."
> where "mid" is the primary key (varchar). If I look at the logs, it has an
> EOFException ... presumably it's running out of some resource (it's
> definitely not out of disk space)
> Sometimes it does this on secondary indexes too: dropping and rebuilding the
> index will fix it for a while. When it's broken, it seems like only one
> particular lookup key causes timeouts (and the EOFException every time) -
> other lookups work fine. I presume the index is corrupt somehow.
> {noformat}
> ERROR [ReadStage:110] 2014-04-03 12:39:47,018 CassandraDaemon.java (line 196)
> Exception in thread Thread[ReadStage:110,5,main]
> java.io.IOError: java.io.EOFException
> at org.apache.cassandra.db.Column$1.computeNext(Column.java:79)
> at org.apache.cassandra.db.Column$1.computeNext(Column.java:64)
> at
> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
> at
> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
> at
> org.apache.cassandra.db.columniterator.SimpleSliceReader.computeNext(SimpleSliceReader.java:88)
> at
> org.apache.cassandra.db.columniterator.SimpleSliceReader.computeNext(SimpleSliceReader.java:37)
> at
> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
> at
> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
> at
> org.apache.cassandra.db.columniterator.SSTableSliceIterator.hasNext(SSTableSliceIterator.java:82)
> at
> org.apache.cassandra.db.filter.QueryFilter$2.getNext(QueryFilter.java:157)
> at
> org.apache.cassandra.db.filter.QueryFilter$2.hasNext(QueryFilter.java:140)
> at
> org.apache.cassandra.utils.MergeIterator$OneToOne.computeNext(MergeIterator.java:200)
> at
> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
> at
> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
> at
> org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(SliceQueryFilter.java:185)
> at
> org.apache.cassandra.db.filter.QueryFilter.collateColumns(QueryFilter.java:122)
> at
> org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:80)
> at
> org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:72)
> at
> org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:297)
> at
> org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:53)
> at
> org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1551)
> at
> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1380)
> at org.apache.cassandra.db.Keyspace.getRow(Keyspace.java:327)
> at
> org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:65)
> at
> org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:1341)
> at
> org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1896)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.
[jira] [Commented] (CASSANDRA-6981) java.io.EOFException from Cassandra when doing select
[ https://issues.apache.org/jira/browse/CASSANDRA-6981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13959355#comment-13959355 ] Martin Bligh commented on CASSANDRA-6981: - Seems to be a lot more stable with disk_access_mode: standard ie, I could consistently make it crash before, but haven't been able to since that change (approx 10x the runtime it took before to crash it) > java.io.EOFException from Cassandra when doing select > - > > Key: CASSANDRA-6981 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6981 > Project: Cassandra > Issue Type: Bug > Environment: Cassandra 2.06, Oracle Java version "1.7.0_51", Linux > Mint 16 >Reporter: Martin Bligh > > Cassandra 2.06, Oracle Java version "1.7.0_51", Linux Mint 16 > I have a cassandra keyspace with about 12 tables that are all the same. > If I load 100,000 rows or so into a couple of those tables in Cassandra, it > works fine. > If I load a larger dataset, after a while one of the tables won't do lookups > any more (not always the same one). > SELECT recv_time,symbol from table6 where mid='S-AUR01-20140324A-1221'; > results in "Request did not complete within rpc_timeout." > where "mid" is the primary key (varchar). If I look at the logs, it has an > EOFException ... presumably it's running out of some resource (it's > definitely not out of disk space) > Sometimes it does this on secondary indexes too: dropping and rebuilding the > index will fix it for a while. When it's broken, it seems like only one > particular lookup key causes timeouts (and the EOFException every time) - > other lookups work fine. I presume the index is corrupt somehow. > ERROR [ReadStage:110] 2014-04-03 12:39:47,018 CassandraDaemon.java (line 196) > Exception in thread Thread[ReadStage:110,5,main] > java.io.IOError: java.io.EOFException > at org.apache.cassandra.db.Column$1.computeNext(Column.java:79) > at org.apache.cassandra.db.Column$1.computeNext(Column.java:64) > at > com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) > at > com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) > at > org.apache.cassandra.db.columniterator.SimpleSliceReader.computeNext(SimpleSliceReader.java:88) > at > org.apache.cassandra.db.columniterator.SimpleSliceReader.computeNext(SimpleSliceReader.java:37) > at > com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) > at > com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) > at > org.apache.cassandra.db.columniterator.SSTableSliceIterator.hasNext(SSTableSliceIterator.java:82) > at > org.apache.cassandra.db.filter.QueryFilter$2.getNext(QueryFilter.java:157) > at > org.apache.cassandra.db.filter.QueryFilter$2.hasNext(QueryFilter.java:140) > at > org.apache.cassandra.utils.MergeIterator$OneToOne.computeNext(MergeIterator.java:200) > at > com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) > at > com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) > at > org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(SliceQueryFilter.java:185) > at > org.apache.cassandra.db.filter.QueryFilter.collateColumns(QueryFilter.java:122) > at > org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:80) > at > org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:72) > at > org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:297) > at > org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:53) > at > org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1551) > at > org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1380) > at org.apache.cassandra.db.Keyspace.getRow(Keyspace.java:327) > at > org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:65) > at > org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:1341) > at > org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1896) > at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) > at java.lang.Thread.run(Unknown Source) > Caused by: java.io.EOFException > at java.io.RandomAccessFile.readFully(Unknown Source) > at java.io.RandomAccessFile.readFully(Unknown Source) > at > org.apache.cassandra.io.util.RandomAccessReader.readBytes(RandomAccessReader.java:348) > at org.apache.cassandra.utils.ByteBuffer
[jira] [Commented] (CASSANDRA-6981) java.io.EOFException from Cassandra when doing select
[ https://issues.apache.org/jira/browse/CASSANDRA-6981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13959119#comment-13959119 ] Martin Bligh commented on CASSANDRA-6981: - Turning off compression changes the error to this: java.io.IOError: java.io.IOException: mmap segment underflow; remaining is 20402577 but 1879048192 requested > java.io.EOFException from Cassandra when doing select > - > > Key: CASSANDRA-6981 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6981 > Project: Cassandra > Issue Type: Bug > Environment: Cassandra 2.06, Oracle Java version "1.7.0_51", Linux > Mint 16 >Reporter: Martin Bligh > > Cassandra 2.06, Oracle Java version "1.7.0_51", Linux Mint 16 > I have a cassandra keyspace with about 12 tables that are all the same. > If I load 100,000 rows or so into a couple of those tables in Cassandra, it > works fine. > If I load a larger dataset, after a while one of the tables won't do lookups > any more (not always the same one). > SELECT recv_time,symbol from table6 where mid='S-AUR01-20140324A-1221'; > results in "Request did not complete within rpc_timeout." > where "mid" is the primary key (varchar). If I look at the logs, it has an > EOFException ... presumably it's running out of some resource (it's > definitely not out of disk space) > Sometimes it does this on secondary indexes too: dropping and rebuilding the > index will fix it for a while. When it's broken, it seems like only one > particular lookup key causes timeouts (and the EOFException every time) - > other lookups work fine. I presume the index is corrupt somehow. > ERROR [ReadStage:110] 2014-04-03 12:39:47,018 CassandraDaemon.java (line 196) > Exception in thread Thread[ReadStage:110,5,main] > java.io.IOError: java.io.EOFException > at org.apache.cassandra.db.Column$1.computeNext(Column.java:79) > at org.apache.cassandra.db.Column$1.computeNext(Column.java:64) > at > com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) > at > com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) > at > org.apache.cassandra.db.columniterator.SimpleSliceReader.computeNext(SimpleSliceReader.java:88) > at > org.apache.cassandra.db.columniterator.SimpleSliceReader.computeNext(SimpleSliceReader.java:37) > at > com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) > at > com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) > at > org.apache.cassandra.db.columniterator.SSTableSliceIterator.hasNext(SSTableSliceIterator.java:82) > at > org.apache.cassandra.db.filter.QueryFilter$2.getNext(QueryFilter.java:157) > at > org.apache.cassandra.db.filter.QueryFilter$2.hasNext(QueryFilter.java:140) > at > org.apache.cassandra.utils.MergeIterator$OneToOne.computeNext(MergeIterator.java:200) > at > com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) > at > com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) > at > org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(SliceQueryFilter.java:185) > at > org.apache.cassandra.db.filter.QueryFilter.collateColumns(QueryFilter.java:122) > at > org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:80) > at > org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:72) > at > org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:297) > at > org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:53) > at > org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1551) > at > org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1380) > at org.apache.cassandra.db.Keyspace.getRow(Keyspace.java:327) > at > org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:65) > at > org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:1341) > at > org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1896) > at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) > at java.lang.Thread.run(Unknown Source) > Caused by: java.io.EOFException > at java.io.RandomAccessFile.readFully(Unknown Source) > at java.io.RandomAccessFile.readFully(Unknown Source) > at > org.apache.cassandra.io.util.RandomAccessReader.readBytes(RandomAccessReader.java:348) > at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:392) > at
[jira] [Updated] (CASSANDRA-6981) java.io.EOFException from Cassandra when doing select
[ https://issues.apache.org/jira/browse/CASSANDRA-6981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Martin Bligh updated CASSANDRA-6981: Description: Cassandra 2.06, Oracle Java version "1.7.0_51", Linux Mint 16 I have a cassandra keyspace with about 12 tables that are all the same. If I load 100,000 rows or so into a couple of those tables in Cassandra, it works fine. If I load a larger dataset, after a while one of the tables won't do lookups any more (not always the same one). SELECT recv_time,symbol from table6 where mid='S-AUR01-20140324A-1221'; results in "Request did not complete within rpc_timeout." where "mid" is the primary key (varchar). If I look at the logs, it has an EOFException ... presumably it's running out of some resource (it's definitely not out of disk space) Sometimes it does this on secondary indexes too: dropping and rebuilding the index will fix it for a while. When it's broken, it seems like only one particular lookup key causes timeouts (and the EOFException every time) - other lookups work fine. I presume the index is corrupt somehow. ERROR [ReadStage:110] 2014-04-03 12:39:47,018 CassandraDaemon.java (line 196) Exception in thread Thread[ReadStage:110,5,main] java.io.IOError: java.io.EOFException at org.apache.cassandra.db.Column$1.computeNext(Column.java:79) at org.apache.cassandra.db.Column$1.computeNext(Column.java:64) at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) at org.apache.cassandra.db.columniterator.SimpleSliceReader.computeNext(SimpleSliceReader.java:88) at org.apache.cassandra.db.columniterator.SimpleSliceReader.computeNext(SimpleSliceReader.java:37) at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) at org.apache.cassandra.db.columniterator.SSTableSliceIterator.hasNext(SSTableSliceIterator.java:82) at org.apache.cassandra.db.filter.QueryFilter$2.getNext(QueryFilter.java:157) at org.apache.cassandra.db.filter.QueryFilter$2.hasNext(QueryFilter.java:140) at org.apache.cassandra.utils.MergeIterator$OneToOne.computeNext(MergeIterator.java:200) at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) at org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(SliceQueryFilter.java:185) at org.apache.cassandra.db.filter.QueryFilter.collateColumns(QueryFilter.java:122) at org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:80) at org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:72) at org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:297) at org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:53) at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1551) at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1380) at org.apache.cassandra.db.Keyspace.getRow(Keyspace.java:327) at org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:65) at org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:1341) at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1896) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Caused by: java.io.EOFException at java.io.RandomAccessFile.readFully(Unknown Source) at java.io.RandomAccessFile.readFully(Unknown Source) at org.apache.cassandra.io.util.RandomAccessReader.readBytes(RandomAccessReader.java:348) at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:392) at org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:355) at org.apache.cassandra.db.ColumnSerializer.deserializeColumnBody(ColumnSerializer.java:110) at org.apache.cassandra.db.OnDiskAtom$Serializer.deserializeFromSSTable(OnDiskAtom.java:85) at org.apache.cassandra.db.Column$1.computeNext(Column.java:75) ... 28 more was: Cassandra 2.06, Oracle Java version "1.7.0_51", Linux Mint 16 I have a cassandra keyspace with about 12 tables that are all the same. If I load 100,000 rows or so into a couple of those tables in Cassandra, it works fine. If I load a larger dataset, after a while one of the tables won't do lookups any more (not always the same one). SELECT recv_time,symbol from table6 where mid='S-AUR
[jira] [Created] (CASSANDRA-6981) java.io.EOFException from Cassandra when doing select
Martin Bligh created CASSANDRA-6981: --- Summary: java.io.EOFException from Cassandra when doing select Key: CASSANDRA-6981 URL: https://issues.apache.org/jira/browse/CASSANDRA-6981 Project: Cassandra Issue Type: Bug Environment: Cassandra 2.06, Oracle Java version "1.7.0_51", Linux Mint 16 Reporter: Martin Bligh Cassandra 2.06, Oracle Java version "1.7.0_51", Linux Mint 16 I have a cassandra keyspace with about 12 tables that are all the same. If I load 100,000 rows or so into a couple of those tables in Cassandra, it works fine. If I load a larger dataset, after a while one of the tables won't do lookups any more (not always the same one). SELECT recv_time,symbol from table6 where mid='S-AUR01-20140324A-1221'; results in "Request did not complete within rpc_timeout." where "mid" is the primary key (varchar). If I look at the logs, it has an EOFException ... presumably it's running out of some resource (it's definitely not out of disk space) ERROR [ReadStage:110] 2014-04-03 12:39:47,018 CassandraDaemon.java (line 196) Exception in thread Thread[ReadStage:110,5,main] java.io.IOError: java.io.EOFException at org.apache.cassandra.db.Column$1.computeNext(Column.java:79) at org.apache.cassandra.db.Column$1.computeNext(Column.java:64) at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) at org.apache.cassandra.db.columniterator.SimpleSliceReader.computeNext(SimpleSliceReader.java:88) at org.apache.cassandra.db.columniterator.SimpleSliceReader.computeNext(SimpleSliceReader.java:37) at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) at org.apache.cassandra.db.columniterator.SSTableSliceIterator.hasNext(SSTableSliceIterator.java:82) at org.apache.cassandra.db.filter.QueryFilter$2.getNext(QueryFilter.java:157) at org.apache.cassandra.db.filter.QueryFilter$2.hasNext(QueryFilter.java:140) at org.apache.cassandra.utils.MergeIterator$OneToOne.computeNext(MergeIterator.java:200) at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) at org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(SliceQueryFilter.java:185) at org.apache.cassandra.db.filter.QueryFilter.collateColumns(QueryFilter.java:122) at org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:80) at org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:72) at org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:297) at org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:53) at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1551) at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1380) at org.apache.cassandra.db.Keyspace.getRow(Keyspace.java:327) at org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:65) at org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:1341) at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1896) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Caused by: java.io.EOFException at java.io.RandomAccessFile.readFully(Unknown Source) at java.io.RandomAccessFile.readFully(Unknown Source) at org.apache.cassandra.io.util.RandomAccessReader.readBytes(RandomAccessReader.java:348) at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:392) at org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:355) at org.apache.cassandra.db.ColumnSerializer.deserializeColumnBody(ColumnSerializer.java:110) at org.apache.cassandra.db.OnDiskAtom$Serializer.deserializeFromSSTable(OnDiskAtom.java:85) at org.apache.cassandra.db.Column$1.computeNext(Column.java:75) ... 28 more -- This message was sent by Atlassian JIRA (v6.2#6252)
