Yes, sometimes it is create table and sometime it is create index. It doesn't happen all the time, but feel like if multiple tests trying to do schema change(create or drop), Cassandra has a long delay on the schema change statements.
I also just read about "auto_snapshot", and I turn it off but still no luck. On Mon, Apr 20, 2015 at 6:42 AM, Jim Witschey <jim.witsc...@datastax.com> wrote: > Jimmy, > > What's the exact command that produced this trace? Are you saying that > the 16-second wait in your trace what times out in your CREATE TABLE > statements? > > Jim Witschey > > Software Engineer in Test | jim.witsc...@datastax.com > > On Sun, Apr 19, 2015 at 7:13 PM, Jimmy Lin <y2klyf+w...@gmail.com> wrote: > > hi, > > we have some unit tests that run parallel that will create tmp keyspace, > and > > tables and then drop them after tests are done. > > > > From time to time, our create table statement run into "All hosts(s) for > > query failed... Timeout during read" (from datastax driver) error. > > > > We later turn on tracing, and record something in the following. > > See below between "===" , Native_Transport-Request thread and > MigrationStage > > thread, there was like 16 seconds doing something. > > > > Any idea what that 16 seconds Cassandra was doing? We can work around > that > > but increasing our datastax driver timeout value, but wondering if there > is > > actually better way to solve this? > > > > thanks > > > > > > > > ---------------- tracing ---------- > > > > > > 5872bf70-e6e2-11e4-823d-93572f3db015 | > 58730d97-e6e2-11e4-823d-93572f3db015 > > | > > Key cache hit for sstable 95588 | 127.0.0.1 | 1592 | > > Native-Transport-Requests:102 > > 5872bf70-e6e2-11e4-823d-93572f3db015 | > 58730d98-e6e2-11e4-823d-93572f3db015 > > | > Seeking > > to partition beginning in data file | 127.0.0.1 | 1593 | > > Native-Transport-Requests:102 > > 5872bf70-e6e2-11e4-823d-93572f3db015 | > 58730d99-e6e2-11e4-823d-93572f3db015 > > | > Merging > > data from memtables and 3 sstables | 127.0.0.1 | 1595 | > > Native-Transport-Requests:102 > > > > ===================== > > 5872bf70-e6e2-11e4-823d-93572f3db015 | > 58730d9a-e6e2-11e4-823d-93572f3db015 > > | > > Read 3 live and 0 tombstoned cells | 127.0.0.1 | 1610 | > > Native-Transport-Requests:102 > > 5872bf70-e6e2-11e4-823d-93572f3db015 | > 62364a40-e6e2-11e4-823d-93572f3db015 > > | Executing seq scan across 1 sstables for > > (min(-9223372036854775808), min(-9223372036854775808)] | 127.0.0.1 | > > 16381594 | MigrationStage:1 > > ===================== > > > > 5872bf70-e6e2-11e4-823d-93572f3db015 | > 62364a41-e6e2-11e4-823d-93572f3db015 > > | > Seeking > > to partition beginning in data file | 127.0.0.1 | 16381782 | > > MigrationStage:1 > > 5872bf70-e6e2-11e4-823d-93572f3db015 | > 62364a42-e6e2-11e4-823d-93572f3db015 > > | > > Read 0 live and 0 tombstoned cells | 127.0.0.1 | 16381787 | > > MigrationStage:1 > > 5872bf70-e6e2-11e4-823d-93572f3db015 | > 62364a43-e6e2-11e4-823d-93572f3db015 > > | > Seeking > > to partition beginning in data file | 127.0.0.1 | 16381789 | > > MigrationStage:1 > > 5872bf70-e6e2-11e4-823d-93572f3db015 | > 62364a44-e6e2-11e4-823d-93572f3db015 > > | > > Read 0 live and 0 tombstoned cells | 127.0.0.1 | 16381791 | > > MigrationStage:1 > > 5872bf70-e6e2-11e4-823d-93572f3db015 | > 62364a45-e6e2-11e4-823d-93572f3db015 > > | > Seeking > > to partition beginning in data file | 127.0.0.1 | 16381792 | > > MigrationStage:1 > > 5872bf70-e6e2-11e4-823d-93572f3db015 | > 62364a46-e6e2-11e4-823d-93572f3db015 > > | > > Read 0 live and 0 tombstoned cells | 127.0.0.1 | 16381794 | > > MigrationStage:1 > > . > > . > > . > > >
