Huge multi-data center latencies

[Hobin Yoon]

I am experiencing huge latencies with a multi-data center Cassandra
cluster. With consistency level ONE, I expected almost the same latency
with the single data center setup. What could possibly affect the latency
in multi-data center setup?

multi-DC setup
min max avg (ms)
1 969 164.554264

single-DC setup
min max avg (ms)
1 51 2.371786

I am using the Datastax Java client library (
https://github.com/datastax/java-driver).

This is the keyspace description.

~/work/cassandra/bin$ ./cqlsh `hostname`
Connected to cdbp at mdc-s70:9160.
[cqlsh 3.1.7 | Cassandra 1.2.9-SNAPSHOT | CQL spec 3.0.0 | Thrift protocol
19.36.0]
Use HELP for help.
cqlsh> desc keyspace pbdp;

CREATE KEYSPACE pbdp WITH replication = {
  'class': 'NetworkTopologyStrategy',
  'DC2': '1',
  'DC3': '1',
  'DC0': '1',
  'DC1': '1'
};

USE pbdp;

CREATE TABLE tweet (
  tid bigint PRIMARY KEY,
  created_at_rt bigint,
  created_at_st text,
  lati float,
  longi float,
  real_coord boolean,
  sn text,
  text_ text
) WITH
  bloom_filter_fp_chance=0.010000 AND
  caching='KEYS_ONLY' AND
  comment='' AND
  dclocal_read_repair_chance=0.000000 AND
  gc_grace_seconds=864000 AND
  read_repair_chance=0.100000 AND
  replicate_on_write='true' AND
  populate_io_cache_on_flush='false' AND
  compaction={'class': 'SizeTieredCompactionStrategy'} AND
  compression={'sstable_compression': 'SnappyCompressor'};

Thanks,
Hobin