So from code review, the error message you get from the log is coming from
the CASSANDRA-11990:
https://github.com/ifesdjeen/cassandra/commit/dc4ae57f452e19adbe5a6a2c85f8a4b5a24d4103#diff-eae81aa3b81f9b1e07b109c446447a50R357
Now, it's just the consequence of the problem (throwing an assertion
error), we have to dig further to understand why we fall into this situation
On Sat, Nov 5, 2016 at 5:15 AM, Jonathan Haddad <j...@jonhaddad.com> wrote:
> Can you file a Jira for this? Would be good to make sure 3.10 doesn't get
> released with this bug.
> On Fri, Nov 4, 2016 at 6:11 PM Voytek Jarnot <voytek.jar...@gmail.com>
> wrote:
>
>> Thought I'd follow-up to myself, in case anyone else comes across this
>> problem. I found a reasonably easy test case to reproduce the problem:
>>
>> This works in 3.9, but doesn't work in 3.10-snapshot:
>>
>> CREATE KEYSPACE vjtest WITH replication = {'class': 'SimpleStrategy',
>> 'replication_factor': '1'};
>> use vjtest ;
>> create table tester(id1 text, id2 text, id3 text, val1 text, primary
>> key((id1, id2), id3));
>> create custom index tester_idx_val1 on tester(val1) using
>> 'org.apache.cassandra.index.sasi.SASIIndex';
>> insert into tester(id1,id2,id3, val1) values ('1-1','1-2','1-3','asdf');
>> insert into tester(id1,id2,id3, val1) values ('1-1','1-2','2-3','asdf');
>> insert into tester(id1,id2,id3, val1) values ('1-1','1-2','3-3','asdf');
>> insert into tester(id1,id2,id3, val1) values ('1-1','1-2','4-3','asdf');
>> insert into tester(id1,id2,id3, val1) values ('1-1','1-2','5-3','asdf');
>> insert into tester(id1,id2,id3, val1) values ('1-1','1-2','6-3','asdf');
>> insert into tester(id1,id2,id3, val1) values ('1-1','1-2','7-3','asdf');
>> insert into tester(id1,id2,id3, val1) values ('1-1','1-2','8-3','asdf');
>> insert into tester(id1,id2,id3, val1) values ('1-1','1-2','9-3','asdf');
>>
>> That's it - when Cassandra tries to flush all hell breaks loose (well,
>> maaybe not, but an unhandled error gets logged). Also, the index doesn't
>> actually work subsequently.
>>
>> On Fri, Nov 4, 2016 at 3:58 PM, Voytek Jarnot <voytek.jar...@gmail.com>
>> wrote:
>>
>> Wondering if anyone has encountered the same...
>>
>> Full story and stacktraces below, short version is that creating a SASI
>> index fails for me when running a 3.10-SNAPSHOT build. One caveat: creating
>> the index on an empty table doesn't fail; however, soon after I start
>> pumping data into the table similar problems occur.
>>
>> I created CASSANDRA-12877 for this, but am beginning to suspect it might
>> be related to CASSANDRA-11990. The thing that's throwing me is that I
>> can't seem to duplicate this with a simple test table.
>>
>> Background:
>>
>> Ended up building/loading a 3.10-SNAPSHOT to try to get past
>> CASSANDRA-11670, CASSANDRA-12223, and CASSANDRA-12689.
>>
>> 1) built/installed 3.10-SNAPSHOT from git branch cassandra-3.X
>> 2) created keyspace (SimpleStrategy, RF 1)
>> 3) created table: (simplified version below, many more valX columns
>> present)
>>
>> CREATE TABLE test_table (
>> id1 text,
>> id2 text,
>> id3 date,
>> id4 timestamp,
>> id5 text,
>> val1 text,
>> val2 text,
>> val3 text,
>> task_id text,
>> val4 text,
>> val5 text,
>> PRIMARY KEY ((id1, id2), id3, id4, id5)
>> ) WITH CLUSTERING ORDER BY (id3 DESC, id4 DESC, id5 ASC)
>>
>> 4) created materialized view:
>>
>> CREATE MATERIALIZED VIEW test_table_by_task_id AS
>> SELECT *
>> FROM test_table
>> WHERE id1 IS NOT NULL AND id2 IS NOT NULL AND id3 IS NOT NULL AND id4
>> IS NOT NULL AND id5 IS NOT NULL AND task_id IS NOT NULL
>> PRIMARY KEY (task_id, id3, id4, id1, id2, id5)
>> WITH CLUSTERING ORDER BY (id3 DESC, id4 DESC, id1 ASC, id2 ASC, id5
>> ASC)
>>
>> 5) inserted 27 million "rows" (i.e., unique values for id5)
>> 6) create index attempt
>>
>> create custom index idx_test_table_val5 on test_table(val5) using
>> 'org.apache.cassandra.index.sasi.SASIIndex';
>>
>> 7) no error in cqlsh, but system.log shows many of the following:
>>
>> INFO [SASI-General:1] 2016-11-04 13:46:47,578
>> PerSSTableIndexWriter.java:277 - Flushed index segment
>> /mydir/cassandra/apache-cassandra-3.10-SNAPSHOT/data/
>> data/mykeyspace/test_table-133dd090a2b411e6b1bf6df2a1af06
>> f0/mc-149-big-SI_idx_test_table_val5.db_0, took 869 ms.
>> ERROR [SASI-General:1] 2016-11-04 13:46:47,584 CassandraDaemon.java:229 -
>> Exception in thread Thread[SASI-General:1,5,main]
>> java.lang.AssertionError: cannot have more than 8 overflow collisions per
>> leaf, but had: 12
>> at org.apache.cassandra.index.sasi.disk.
>> AbstractTokenTreeBuilder$Leaf.createOverflowEntry(
>> AbstractTokenTreeBuilder.java:357) ~[apache-cassandra-3.10-
>> SNAPSHOT.jar:3.10-SNAPSHOT]
>> at org.apache.cassandra.index.sasi.disk.
>> AbstractTokenTreeBuilder$Leaf.createEntry(AbstractTokenTreeBuilder.java:346)
>> ~[apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
>> at org.apache.cassandra.index.sasi.disk.DynamicTokenTreeBuilder$
>> DynamicLeaf.serializeData(DynamicTokenTreeBuilder.java:180)
>> ~[apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
>> at org.apache.cassandra.index.sasi.disk.
>> AbstractTokenTreeBuilder$Leaf.serialize(AbstractTokenTreeBuilder.java:306)
>> ~[apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
>> at org.apache.cassandra.index.sasi.disk.AbstractTokenTreeBuilder.
>> write(AbstractTokenTreeBuilder.java:90) ~[apache-cassandra-3.10-
>> SNAPSHOT.jar:3.10-SNAPSHOT]
>> at org.apache.cassandra.index.sasi.disk.OnDiskIndexBuilder$
>> MutableDataBlock.flushAndClear(OnDiskIndexBuilder.java:629)
>> ~[apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
>> at org.apache.cassandra.index.sasi.disk.OnDiskIndexBuilder$
>> MutableLevel.flush(OnDiskIndexBuilder.java:446) ~[apache-cassandra-3.10-
>> SNAPSHOT.jar:3.10-SNAPSHOT]
>> at org.apache.cassandra.index.sasi.disk.OnDiskIndexBuilder$
>> MutableLevel.add(OnDiskIndexBuilder.java:433) ~[apache-cassandra-3.10-
>> SNAPSHOT.jar:3.10-SNAPSHOT]
>> at org.apache.cassandra.index.sasi.disk.OnDiskIndexBuilder.
>> addTerm(OnDiskIndexBuilder.java:207) ~[apache-cassandra-3.10-
>> SNAPSHOT.jar:3.10-SNAPSHOT]
>> at org.apache.cassandra.index.sasi.disk.OnDiskIndexBuilder.
>> finish(OnDiskIndexBuilder.java:293) ~[apache-cassandra-3.10-
>> SNAPSHOT.jar:3.10-SNAPSHOT]
>> at org.apache.cassandra.index.sasi.disk.OnDiskIndexBuilder.
>> finish(OnDiskIndexBuilder.java:258) ~[apache-cassandra-3.10-
>> SNAPSHOT.jar:3.10-SNAPSHOT]
>> at org.apache.cassandra.index.sasi.disk.OnDiskIndexBuilder.
>> finish(OnDiskIndexBuilder.java:241) ~[apache-cassandra-3.10-
>> SNAPSHOT.jar:3.10-SNAPSHOT]
>> at org.apache.cassandra.index.sasi.disk.PerSSTableIndexWriter$Index.
>> lambda$scheduleSegmentFlush$0(PerSSTableIndexWriter.java:267)
>> ~[apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>> ~[na:1.8.0_101]
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> ~[na:1.8.0_101]
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> [na:1.8.0_101]
>> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_101]
>>
>> As well as some of these:
>>
>> ERROR [CompactionExecutor:3] 2016-11-04 13:49:13,142 DataTracker.java:168
>> - Can't open index file at /mydir/cassandra/apache-
>> cassandra-3.10-SNAPSHOT/data/data/mykeyspace/test_table-
>> 133dd090a2b411e6b1bf6df2a1af06f0/mc-300-big-SI_idx_test_table_val5.db,
>> skipping.
>> java.lang.IllegalArgumentException: position: 3472329188772431788,
>> limit: 8180
>> at org.apache.cassandra.index.sasi.utils.MappedBuffer.
>> position(MappedBuffer.java:106) ~[apache-cassandra-3.10-
>> SNAPSHOT.jar:3.10-SNAPSHOT]
>> at
>> org.apache.cassandra.index.sasi.disk.OnDiskIndex.<init>(OnDiskIndex.java:147)
>> ~[apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
>> at
>> org.apache.cassandra.index.sasi.SSTableIndex.<init>(SSTableIndex.java:62)
>> ~[apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
>> at org.apache.cassandra.index.sasi.conf.DataTracker.
>> getIndexes(DataTracker.java:150) [apache-cassandra-3.10-
>> SNAPSHOT.jar:3.10-SNAPSHOT]
>> at
>> org.apache.cassandra.index.sasi.conf.DataTracker.update(DataTracker.java:69)
>> [apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
>> at
>> org.apache.cassandra.index.sasi.conf.ColumnIndex.update(ColumnIndex.java:147)
>> [apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
>> at
>> org.apache.cassandra.index.sasi.SASIIndexBuilder.completeSSTable(SASIIndexBuilder.java:156)
>> [apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
>> at
>> org.apache.cassandra.index.sasi.SASIIndexBuilder.build(SASIIndexBuilder.java:125)
>> [apache-cassandra-3.10-SNAPSHOT.jar:3.10-SNAPSHOT]
>> at org.apache.cassandra.db.compaction.CompactionManager$
>> 14.run(CompactionManager.java:1583) [apache-cassandra-3.10-
>> SNAPSHOT.jar:3.10-SNAPSHOT]
>> at
>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>> [na:1.8.0_101]
>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>> [na:1.8.0_101]
>> at
>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>> [na:1.8.0_101]
>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>> [na:1.8.0_101]
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> [na:1.8.0_101]
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> [na:1.8.0_101]
>> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_101]
>>
>>
>>
