Hi What do you mean by "broken time synchronization" is it a know problem with virtualized os ?
On Wed, Feb 17, 2010 at 7:34 PM, Wojciech Kaczmarek <[email protected]>wrote: > Just guessing: > are you sure that these virtual machines don't suffer from a broken time > synchronization? > > > On Wed, Feb 17, 2010 at 12:26, Richard Grossman <[email protected]>wrote: > >> Hi >> >> I've configured 4 Virtual Machines CentOS 4GB memory. Each run cassandra >> 0.5 release. >> All is ok util I begin to get error like on the client side : >> UnavailableException() >> at >> org.apache.cassandra.service.Cassandra$batch_insert_result.read(Cassandra.java:10892) >> at >> org.apache.cassandra.service.Cassandra$Client.recv_batch_insert(Cassandra.java:616) >> at >> org.apache.cassandra.service.Cassandra$Client.batch_insert(Cassandra.java:591) >> at >> tv.bee.hiveplus.crud.CassandraThread.insertChannelShow(CassandraThread.java:229) >> at tv.bee.hiveplus.crud.CassandraThread.call(CassandraThread.java:59) >> at tv.bee.hiveplus.crud.CassandraThread.call(CassandraThread.java:1) >> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) >> at java.util.concurrent.FutureTask.run(FutureTask.java:166) >> at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) >> at java.lang.Thread.run(Thread.java:636) >> >> On the server more interresting : >> INFO [Timer-1] 2010-02-17 11:17:58,268 Gossiper.java (line 194) >> InetAddress /38.96.191.42 is now dead. >> INFO [GMFD:1] 2010-02-17 11:17:58,680 Gossiper.java (line 543) >> InetAddress /38.96.191.42 is now UP >> INFO [FLUSH-WRITER-POOL:1] 2010-02-17 11:18:06,604 Memtable.java (line >> 209) Completed flushing >> /root/cassandraDB/data/Keyspace1/channelShow-14-Data.db >> INFO [COMPACTION-POOL:1] 2010-02-17 11:18:06,604 ColumnFamilyStore.java >> (line 875) Compacting >> [org.apache.cassandra.io.SSTableReader(path='/root/cassandraDB/data/Keyspace1/channelShow-11-Data.db'),org.apache.cassandra.io.SSTableReader(path='/root/cassandraDB/data/Keyspace1/channelShow-12-Data.db'),org.apache.cassandra.io.SSTableReader(path='/root/cassandraDB/data/Keyspace1/channelShow-13-Data.db'),org.apache.cassandra.io.SSTableReader(path='/root/cassandraDB/data/Keyspace1/channelShow-14-Data.db')] >> INFO [COMPACTION-POOL:1] 2010-02-17 11:19:41,231 ColumnFamilyStore.java >> (line 943) Compacted to >> /root/cassandraDB/data/Keyspace1/channelShow-15-Data.db. 80405396/80405396 >> bytes for 110384 keys. Time: 94627ms. >> INFO [Timer-1] 2010-02-17 11:20:15,047 Gossiper.java (line 194) >> InetAddress /38.96.191.40 is now dead. >> WARN [MESSAGING-SERVICE-POOL:2] 2010-02-17 11:21:50,307 >> TcpConnection.java (line 484) Problem reading from socket connected to : >> java.nio.channels.SocketChannel[connected local=/38.96.191.41:7000remote=/ >> 38.96.191.39:50133] >> WARN [MESSAGING-SERVICE-POOL:2] 2010-02-17 11:21:50,307 >> TcpConnection.java (line 485) Exception was generated at : 02/17/2010 >> 11:21:50 on thread MESSAGING-SERVICE-POOL:2 >> Reached an EOL or something bizzare occured. Reading from: >> /38.96.191.39BufferSizeRemaining: 16 >> java.io.IOException: Reached an EOL or something bizzare occured. Reading >> from: /38.96.191.39 BufferSizeRemaining: 16 >> at org.apache.cassandra.net.io.StartState.doRead(StartState.java:44) >> at >> org.apache.cassandra.net.io.ProtocolState.read(ProtocolState.java:39) >> at org.apache.cassandra.net.io.TcpReader.read(TcpReader.java:95) >> at >> org.apache.cassandra.net.TcpConnection$ReadWorkItem.run(TcpConnection.java:445) >> at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) >> at java.lang.Thread.run(Thread.java:636) >> >> As you can see it's like after some time the communication between node >> just gone the node is declared dead but after some millis the node is up >> again between this short time all the insert just gone. >> It occurs after some time meaning the data is loaded into the memtable. >> >> If someone can help me to understand what going on. Is the machine itself >> overloaded. It's a single machine with 4 virtual machine sharing the same >> disk is it the cause ? >> >> Thanks for any help >> Richard >> > >
