Xen has had issues with it. If you're using it, google for "xen independent_wallclock", "xen ntp" and see if the issues described fit your configuration.
On Thu, Feb 18, 2010 at 09:03, Richard Grossman <[email protected]> wrote: > Hi > > What do you mean by "broken time synchronization" is it a know problem with > virtualized os ? > > > > On Wed, Feb 17, 2010 at 7:34 PM, Wojciech Kaczmarek <[email protected] > > wrote: > >> Just guessing: >> are you sure that these virtual machines don't suffer from a broken time >> synchronization? >> >> >> On Wed, Feb 17, 2010 at 12:26, Richard Grossman <[email protected]>wrote: >> >>> Hi >>> >>> I've configured 4 Virtual Machines CentOS 4GB memory. Each run cassandra >>> 0.5 release. >>> All is ok util I begin to get error like on the client side : >>> UnavailableException() >>> at >>> org.apache.cassandra.service.Cassandra$batch_insert_result.read(Cassandra.java:10892) >>> at >>> org.apache.cassandra.service.Cassandra$Client.recv_batch_insert(Cassandra.java:616) >>> at >>> org.apache.cassandra.service.Cassandra$Client.batch_insert(Cassandra.java:591) >>> at >>> tv.bee.hiveplus.crud.CassandraThread.insertChannelShow(CassandraThread.java:229) >>> at tv.bee.hiveplus.crud.CassandraThread.call(CassandraThread.java:59) >>> at tv.bee.hiveplus.crud.CassandraThread.call(CassandraThread.java:1) >>> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) >>> at java.util.concurrent.FutureTask.run(FutureTask.java:166) >>> at >>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) >>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) >>> at java.lang.Thread.run(Thread.java:636) >>> >>> On the server more interresting : >>> INFO [Timer-1] 2010-02-17 11:17:58,268 Gossiper.java (line 194) >>> InetAddress /38.96.191.42 is now dead. >>> INFO [GMFD:1] 2010-02-17 11:17:58,680 Gossiper.java (line 543) >>> InetAddress /38.96.191.42 is now UP >>> INFO [FLUSH-WRITER-POOL:1] 2010-02-17 11:18:06,604 Memtable.java (line >>> 209) Completed flushing >>> /root/cassandraDB/data/Keyspace1/channelShow-14-Data.db >>> INFO [COMPACTION-POOL:1] 2010-02-17 11:18:06,604 ColumnFamilyStore.java >>> (line 875) Compacting >>> [org.apache.cassandra.io.SSTableReader(path='/root/cassandraDB/data/Keyspace1/channelShow-11-Data.db'),org.apache.cassandra.io.SSTableReader(path='/root/cassandraDB/data/Keyspace1/channelShow-12-Data.db'),org.apache.cassandra.io.SSTableReader(path='/root/cassandraDB/data/Keyspace1/channelShow-13-Data.db'),org.apache.cassandra.io.SSTableReader(path='/root/cassandraDB/data/Keyspace1/channelShow-14-Data.db')] >>> INFO [COMPACTION-POOL:1] 2010-02-17 11:19:41,231 ColumnFamilyStore.java >>> (line 943) Compacted to >>> /root/cassandraDB/data/Keyspace1/channelShow-15-Data.db. 80405396/80405396 >>> bytes for 110384 keys. Time: 94627ms. >>> INFO [Timer-1] 2010-02-17 11:20:15,047 Gossiper.java (line 194) >>> InetAddress /38.96.191.40 is now dead. >>> WARN [MESSAGING-SERVICE-POOL:2] 2010-02-17 11:21:50,307 >>> TcpConnection.java (line 484) Problem reading from socket connected to : >>> java.nio.channels.SocketChannel[connected local=/38.96.191.41:7000remote=/ >>> 38.96.191.39:50133] >>> WARN [MESSAGING-SERVICE-POOL:2] 2010-02-17 11:21:50,307 >>> TcpConnection.java (line 485) Exception was generated at : 02/17/2010 >>> 11:21:50 on thread MESSAGING-SERVICE-POOL:2 >>> Reached an EOL or something bizzare occured. Reading from: >>> /38.96.191.39BufferSizeRemaining: 16 >>> java.io.IOException: Reached an EOL or something bizzare occured. Reading >>> from: /38.96.191.39 BufferSizeRemaining: 16 >>> at org.apache.cassandra.net.io.StartState.doRead(StartState.java:44) >>> at >>> org.apache.cassandra.net.io.ProtocolState.read(ProtocolState.java:39) >>> at org.apache.cassandra.net.io.TcpReader.read(TcpReader.java:95) >>> at >>> org.apache.cassandra.net.TcpConnection$ReadWorkItem.run(TcpConnection.java:445) >>> at >>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) >>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) >>> at java.lang.Thread.run(Thread.java:636) >>> >>> As you can see it's like after some time the communication between node >>> just gone the node is declared dead but after some millis the node is up >>> again between this short time all the insert just gone. >>> It occurs after some time meaning the data is loaded into the memtable. >>> >>> If someone can help me to understand what going on. Is the machine itself >>> overloaded. It's a single machine with 4 virtual machine sharing the same >>> disk is it the cause ? >>> >>> Thanks for any help >>> Richard >>> >> >> >
