hi, there were only a few (4 of them across 4 minutes with around 200ms), so shouldn't be the reason
The system log has tons of INFO [MigrationStage:1] 2015-04-20 11:03:21,880 ColumnFamilyStore.java (line 633) Enqueuing flush of Memtable-schema_keyspaces@2079381036(138/1215 serialized/live bytes, 3 ops) INFO [MigrationStage:1] 2015-04-20 11:03:21,900 ColumnFamilyStore.java (line 633) Enqueuing flush of Memtable-schema_columnfamilies@1283263314(1036/3946 serialized/live bytes, 24 ops) INFO [MigrationStage:1] 2015-04-20 11:03:21,921 ColumnFamilyStore.java (line 633) Enqueuing flush of Memtable-schema_columns But that could be just normal given that our unit tests are doing lot of droping keyspace and creating keyspace/tables. I read the MigrationStage thread pool is default to one, so wondering if that could be a reason it may be doing something that block others? On Mon, Apr 20, 2015 at 2:40 PM, Sebastian Estevez < sebastian.este...@datastax.com> wrote: > Can you grep for GCInspector in your system.log? Maybe you have long GC > pauses. > > All the best, > > > [image: datastax_logo.png] <http://www.datastax.com/> > > Sebastián Estévez > > Solutions Architect | 954 905 8615 | sebastian.este...@datastax.com > > [image: linkedin.png] <https://www.linkedin.com/company/datastax> [image: > facebook.png] <https://www.facebook.com/datastax> [image: twitter.png] > <https://twitter.com/datastax> [image: g+.png] > <https://plus.google.com/+Datastax/about> > <http://feeds.feedburner.com/datastax> > > <http://cassandrasummit-datastax.com/> > > DataStax is the fastest, most scalable distributed database technology, > delivering Apache Cassandra to the world’s most innovative enterprises. > Datastax is built to be agile, always-on, and predictably scalable to any > size. With more than 500 customers in 45 countries, DataStax is the > database technology and transactional backbone of choice for the worlds > most innovative companies such as Netflix, Adobe, Intuit, and eBay. > > On Mon, Apr 20, 2015 at 12:19 PM, Jimmy Lin <y2klyf+w...@gmail.com> wrote: > >> Yes, sometimes it is create table and sometime it is create index. >> It doesn't happen all the time, but feel like if multiple tests trying to >> do schema change(create or drop), Cassandra has a long delay on the schema >> change statements. >> >> I also just read about "auto_snapshot", and I turn it off but still no >> luck. >> >> >> >> On Mon, Apr 20, 2015 at 6:42 AM, Jim Witschey <jim.witsc...@datastax.com> >> wrote: >> >>> Jimmy, >>> >>> What's the exact command that produced this trace? Are you saying that >>> the 16-second wait in your trace what times out in your CREATE TABLE >>> statements? >>> >>> Jim Witschey >>> >>> Software Engineer in Test | jim.witsc...@datastax.com >>> >>> On Sun, Apr 19, 2015 at 7:13 PM, Jimmy Lin <y2klyf+w...@gmail.com> >>> wrote: >>> > hi, >>> > we have some unit tests that run parallel that will create tmp >>> keyspace, and >>> > tables and then drop them after tests are done. >>> > >>> > From time to time, our create table statement run into "All hosts(s) >>> for >>> > query failed... Timeout during read" (from datastax driver) error. >>> > >>> > We later turn on tracing, and record something in the following. >>> > See below between "===" , Native_Transport-Request thread and >>> MigrationStage >>> > thread, there was like 16 seconds doing something. >>> > >>> > Any idea what that 16 seconds Cassandra was doing? We can work around >>> that >>> > but increasing our datastax driver timeout value, but wondering if >>> there is >>> > actually better way to solve this? >>> > >>> > thanks >>> > >>> > >>> > >>> > ---------------- tracing ---------- >>> > >>> > >>> > 5872bf70-e6e2-11e4-823d-93572f3db015 | >>> 58730d97-e6e2-11e4-823d-93572f3db015 >>> > | >>> > Key cache hit for sstable 95588 | 127.0.0.1 | 1592 | >>> > Native-Transport-Requests:102 >>> > 5872bf70-e6e2-11e4-823d-93572f3db015 | >>> 58730d98-e6e2-11e4-823d-93572f3db015 >>> > | >>> Seeking >>> > to partition beginning in data file | 127.0.0.1 | 1593 | >>> > Native-Transport-Requests:102 >>> > 5872bf70-e6e2-11e4-823d-93572f3db015 | >>> 58730d99-e6e2-11e4-823d-93572f3db015 >>> > | >>> Merging >>> > data from memtables and 3 sstables | 127.0.0.1 | 1595 | >>> > Native-Transport-Requests:102 >>> > >>> > ===================== >>> > 5872bf70-e6e2-11e4-823d-93572f3db015 | >>> 58730d9a-e6e2-11e4-823d-93572f3db015 >>> > | >>> > Read 3 live and 0 tombstoned cells | 127.0.0.1 | 1610 | >>> > Native-Transport-Requests:102 >>> > 5872bf70-e6e2-11e4-823d-93572f3db015 | >>> 62364a40-e6e2-11e4-823d-93572f3db015 >>> > | Executing seq scan across 1 sstables for >>> > (min(-9223372036854775808), min(-9223372036854775808)] | 127.0.0.1 | >>> > 16381594 | MigrationStage:1 >>> > ===================== >>> > >>> > 5872bf70-e6e2-11e4-823d-93572f3db015 | >>> 62364a41-e6e2-11e4-823d-93572f3db015 >>> > | >>> Seeking >>> > to partition beginning in data file | 127.0.0.1 | 16381782 | >>> > MigrationStage:1 >>> > 5872bf70-e6e2-11e4-823d-93572f3db015 | >>> 62364a42-e6e2-11e4-823d-93572f3db015 >>> > | >>> > Read 0 live and 0 tombstoned cells | 127.0.0.1 | 16381787 | >>> > MigrationStage:1 >>> > 5872bf70-e6e2-11e4-823d-93572f3db015 | >>> 62364a43-e6e2-11e4-823d-93572f3db015 >>> > | >>> Seeking >>> > to partition beginning in data file | 127.0.0.1 | 16381789 | >>> > MigrationStage:1 >>> > 5872bf70-e6e2-11e4-823d-93572f3db015 | >>> 62364a44-e6e2-11e4-823d-93572f3db015 >>> > | >>> > Read 0 live and 0 tombstoned cells | 127.0.0.1 | 16381791 | >>> > MigrationStage:1 >>> > 5872bf70-e6e2-11e4-823d-93572f3db015 | >>> 62364a45-e6e2-11e4-823d-93572f3db015 >>> > | >>> Seeking >>> > to partition beginning in data file | 127.0.0.1 | 16381792 | >>> > MigrationStage:1 >>> > 5872bf70-e6e2-11e4-823d-93572f3db015 | >>> 62364a46-e6e2-11e4-823d-93572f3db015 >>> > | >>> > Read 0 live and 0 tombstoned cells | 127.0.0.1 | 16381794 | >>> > MigrationStage:1 >>> > . >>> > . >>> > . >>> > >>> >> >> >
