[jira] [Updated] (CASSANDRA-8819) LOCAL_QUORUM writes returns wrong message
[
https://issues.apache.org/jira/browse/CASSANDRA-8819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alan Boudreault updated CASSANDRA-8819:
---
Labels: qa-resolved (was: )
LOCAL_QUORUM writes returns wrong message
-
Key: CASSANDRA-8819
URL: https://issues.apache.org/jira/browse/CASSANDRA-8819
Project: Cassandra
Issue Type: Bug
Components: Core
Environment: CentOS 6.6
Reporter: Wei Zhu
Assignee: Sylvain Lebresne
Labels: qa-resolved
Fix For: 2.0.13
Attachments: 8819-2.0.patch
We have two DC3, each with 7 nodes.
Here is the keyspace setup:
create keyspace test
with placement_strategy = 'NetworkTopologyStrategy'
and strategy_options = {DC2 : 3, DC1 : 3}
and durable_writes = true;
We brought down two nodes in DC2 for maintenance. We only write to DC1 using
local_quroum (using datastax JavaClient)
But we see this errors in the log:
Cassandra timeout during write query at consistency LOCAL_QUORUM (4 replica
were required but only 3 acknowledged the write
why does it say 4 replica were required? and Why would it give error back to
client since local_quorum should succeed.
Here are the output from nodetool status
Note: Ownership information does not include topology; for complete
information, specify a keyspace
Datacenter: DC2
===
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID
Rack
UN 10.2.0.1 10.92 GB 256 7.9% RAC206
UN 10.2.0.2 6.17 GB256 8.0% RAC106
UN 10.2.0.3 6.63 GB256 7.3% RAC107
DL 10.2.0.4 1.54 GB256 7.7% RAC107
UN 10.2.0.5 6.02 GB256 6.6% RAC106
UJ 10.2.0.6 3.68 GB256 ? RAC205
UN 10.2.0.7 7.22 GB256 7.7% RAC205
Datacenter: DC1
===
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID
Rack
UN 10.1.0.1 6.04 GB256 8.6% RAC10
UN 10.1.0.2 7.55 GB256 7.4% RAC8
UN 10.1.0.3 5.83 GB256 7.0% RAC9
UN 10.1.0.47.34 GB256 7.9% RAC6
UN 10.1.0.5 7.57 GB256 8.0% RAC7
UN 10.1.0.6 5.31 GB256 7.3% RAC10
UN 10.1.0.7 5.47 GB256 8.6% RAC9
I did a cql trace on the query and here is the trace, and it does say
Write timeout; received 3 of 4 required replies | 17:27:52,831 | 10.1.0.1
|2002873
at the end. I guess that is where the client gets the error from. But the
rows was inserted to Cassandra correctly. And I traced read with local_quorum
and it behaves correctly and the reads don't go to DC2. The problem is only
with writes on local_quorum.
{code}
Tracing session: 5a789fb0-b70d-11e4-8fca-99bff9c19890
activity
| timestamp
| source | source_elapsed
-+--+-+
execute_cql3_query | 17:27:50,828
| 10.1.0.1 | 0
Parsing insert into test (user_id, created, event_data, event_id)values (
123456789 , 9eab8950-b70c-11e4-8fca-99bff9c19891, 'test', '16'); |
17:27:50,828 | 10.1.0.1 | 39
Preparing statement | 17:27:50,828
| 10.1.0.1 |135
Message received from /10.1.0.1 | 17:27:50,829 |
10.1.0.5 | 25
Sending message to /10.1.0.5 | 17:27:50,829 |
10.1.0.1 |421
Executing single-partition query on users | 17:27:50,829
| 10.1.0.5 |177
[jira] [Updated] (CASSANDRA-8819) LOCAL_QUORUM writes returns wrong message
[
https://issues.apache.org/jira/browse/CASSANDRA-8819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Philip Thompson updated CASSANDRA-8819:
---
Tester: Alan Boudreault (was: Philip Thompson)
LOCAL_QUORUM writes returns wrong message
-
Key: CASSANDRA-8819
URL: https://issues.apache.org/jira/browse/CASSANDRA-8819
Project: Cassandra
Issue Type: Bug
Components: Core
Environment: CentOS 6.6
Reporter: Wei Zhu
Assignee: Sylvain Lebresne
Fix For: 2.0.13
Attachments: 8819-2.0.patch
We have two DC3, each with 7 nodes.
Here is the keyspace setup:
create keyspace test
with placement_strategy = 'NetworkTopologyStrategy'
and strategy_options = {DC2 : 3, DC1 : 3}
and durable_writes = true;
We brought down two nodes in DC2 for maintenance. We only write to DC1 using
local_quroum (using datastax JavaClient)
But we see this errors in the log:
Cassandra timeout during write query at consistency LOCAL_QUORUM (4 replica
were required but only 3 acknowledged the write
why does it say 4 replica were required? and Why would it give error back to
client since local_quorum should succeed.
Here are the output from nodetool status
Note: Ownership information does not include topology; for complete
information, specify a keyspace
Datacenter: DC2
===
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID
Rack
UN 10.2.0.1 10.92 GB 256 7.9% RAC206
UN 10.2.0.2 6.17 GB256 8.0% RAC106
UN 10.2.0.3 6.63 GB256 7.3% RAC107
DL 10.2.0.4 1.54 GB256 7.7% RAC107
UN 10.2.0.5 6.02 GB256 6.6% RAC106
UJ 10.2.0.6 3.68 GB256 ? RAC205
UN 10.2.0.7 7.22 GB256 7.7% RAC205
Datacenter: DC1
===
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID
Rack
UN 10.1.0.1 6.04 GB256 8.6% RAC10
UN 10.1.0.2 7.55 GB256 7.4% RAC8
UN 10.1.0.3 5.83 GB256 7.0% RAC9
UN 10.1.0.47.34 GB256 7.9% RAC6
UN 10.1.0.5 7.57 GB256 8.0% RAC7
UN 10.1.0.6 5.31 GB256 7.3% RAC10
UN 10.1.0.7 5.47 GB256 8.6% RAC9
I did a cql trace on the query and here is the trace, and it does say
Write timeout; received 3 of 4 required replies | 17:27:52,831 | 10.1.0.1
|2002873
at the end. I guess that is where the client gets the error from. But the
rows was inserted to Cassandra correctly. And I traced read with local_quorum
and it behaves correctly and the reads don't go to DC2. The problem is only
with writes on local_quorum.
{code}
Tracing session: 5a789fb0-b70d-11e4-8fca-99bff9c19890
activity
| timestamp
| source | source_elapsed
-+--+-+
execute_cql3_query | 17:27:50,828
| 10.1.0.1 | 0
Parsing insert into test (user_id, created, event_data, event_id)values (
123456789 , 9eab8950-b70c-11e4-8fca-99bff9c19891, 'test', '16'); |
17:27:50,828 | 10.1.0.1 | 39
Preparing statement | 17:27:50,828
| 10.1.0.1 |135
Message received from /10.1.0.1 | 17:27:50,829 |
10.1.0.5 | 25
Sending message to /10.1.0.5 | 17:27:50,829 |
10.1.0.1 |421
Executing single-partition query on users | 17:27:50,829
| 10.1.0.5 |177
[jira] [Updated] (CASSANDRA-8819) LOCAL_QUORUM writes returns wrong message
[
https://issues.apache.org/jira/browse/CASSANDRA-8819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sylvain Lebresne updated CASSANDRA-8819:
Attachment: 8819-2.0.patch
Alright then, attaching relatively simple patch to filter non-local pending
endpoints for {{LOCAL_QUORUM}}. It should be possible to write a dtests for
this, but that's probably a bit involved. The simpler setup is probably with 2
DC, RF=1 in each, queries in DC1 and one joining node in DC2 (but you'll still
want to query multiple keys to be sure you the joining node is pending for the
queried token, and of course you need to make sure the queries are done while
the node is bootstraping. [~philipthompson], want to give a shot at such dtest
when you have some time?
LOCAL_QUORUM writes returns wrong message
-
Key: CASSANDRA-8819
URL: https://issues.apache.org/jira/browse/CASSANDRA-8819
Project: Cassandra
Issue Type: Bug
Components: Core
Environment: CentOS 6.6
Reporter: Wei Zhu
Assignee: Sylvain Lebresne
Fix For: 2.0.13
Attachments: 8819-2.0.patch
We have two DC3, each with 7 nodes.
Here is the keyspace setup:
create keyspace test
with placement_strategy = 'NetworkTopologyStrategy'
and strategy_options = {DC2 : 3, DC1 : 3}
and durable_writes = true;
We brought down two nodes in DC2 for maintenance. We only write to DC1 using
local_quroum (using datastax JavaClient)
But we see this errors in the log:
Cassandra timeout during write query at consistency LOCAL_QUORUM (4 replica
were required but only 3 acknowledged the write
why does it say 4 replica were required? and Why would it give error back to
client since local_quorum should succeed.
Here are the output from nodetool status
Note: Ownership information does not include topology; for complete
information, specify a keyspace
Datacenter: DC2
===
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID
Rack
UN 10.2.0.1 10.92 GB 256 7.9% RAC206
UN 10.2.0.2 6.17 GB256 8.0% RAC106
UN 10.2.0.3 6.63 GB256 7.3% RAC107
DL 10.2.0.4 1.54 GB256 7.7% RAC107
UN 10.2.0.5 6.02 GB256 6.6% RAC106
UJ 10.2.0.6 3.68 GB256 ? RAC205
UN 10.2.0.7 7.22 GB256 7.7% RAC205
Datacenter: DC1
===
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID
Rack
UN 10.1.0.1 6.04 GB256 8.6% RAC10
UN 10.1.0.2 7.55 GB256 7.4% RAC8
UN 10.1.0.3 5.83 GB256 7.0% RAC9
UN 10.1.0.47.34 GB256 7.9% RAC6
UN 10.1.0.5 7.57 GB256 8.0% RAC7
UN 10.1.0.6 5.31 GB256 7.3% RAC10
UN 10.1.0.7 5.47 GB256 8.6% RAC9
I did a cql trace on the query and here is the trace, and it does say
Write timeout; received 3 of 4 required replies | 17:27:52,831 | 10.1.0.1
|2002873
at the end. I guess that is where the client gets the error from. But the
rows was inserted to Cassandra correctly. And I traced read with local_quorum
and it behaves correctly and the reads don't go to DC2. The problem is only
with writes on local_quorum.
{code}
Tracing session: 5a789fb0-b70d-11e4-8fca-99bff9c19890
activity
| timestamp
| source | source_elapsed
-+--+-+
execute_cql3_query | 17:27:50,828
| 10.1.0.1 | 0
Parsing insert into test (user_id, created, event_data, event_id)values (
123456789 , 9eab8950-b70c-11e4-8fca-99bff9c19891, 'test', '16'); |
17:27:50,828 | 10.1.0.1 | 39
Preparing statement | 17:27:50,828
| 10.1.0.1 |135
[jira] [Updated] (CASSANDRA-8819) LOCAL_QUORUM writes returns wrong message
[
https://issues.apache.org/jira/browse/CASSANDRA-8819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Philip Thompson updated CASSANDRA-8819:
---
Description:
We have two DC3, each with 7 nodes.
Here is the keyspace setup:
create keyspace test
with placement_strategy = 'NetworkTopologyStrategy'
and strategy_options = {DC2 : 3, DC1 : 3}
and durable_writes = true;
We brought down two nodes in DC2 for maintenance. We only write to DC1 using
local_quroum (using datastax JavaClient)
But we see this errors in the log:
Cassandra timeout during write query at consistency LOCAL_QUORUM (4 replica
were required but only 3 acknowledged the write
why does it say 4 replica were required? and Why would it give error back to
client since local_quorum should succeed.
Here are the output from nodetool status
Note: Ownership information does not include topology; for complete
information, specify a keyspace
Datacenter: DC2
===
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID
Rack
UN 10.2.0.1 10.92 GB 256 7.9% RAC206
UN 10.2.0.2 6.17 GB256 8.0% RAC106
UN 10.2.0.3 6.63 GB256 7.3% RAC107
DL 10.2.0.4 1.54 GB256 7.7% RAC107
UN 10.2.0.5 6.02 GB256 6.6% RAC106
UJ 10.2.0.6 3.68 GB256 ? RAC205
UN 10.2.0.7 7.22 GB256 7.7% RAC205
Datacenter: DC1
===
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID
Rack
UN 10.1.0.1 6.04 GB256 8.6% RAC10
UN 10.1.0.2 7.55 GB256 7.4% RAC8
UN 10.1.0.3 5.83 GB256 7.0% RAC9
UN 10.1.0.47.34 GB256 7.9% RAC6
UN 10.1.0.5 7.57 GB256 8.0% RAC7
UN 10.1.0.6 5.31 GB256 7.3% RAC10
UN 10.1.0.7 5.47 GB256 8.6% RAC9
I did a cql trace on the query and here is the trace, and it does say
Write timeout; received 3 of 4 required replies | 17:27:52,831 | 10.1.0.1 |
2002873
at the end. I guess that is where the client gets the error from. But the rows
was inserted to Cassandra correctly. And I traced read with local_quorum and it
behaves correctly and the reads don't go to DC2. The problem is only with
writes on local_quorum.
{code}
Tracing session: 5a789fb0-b70d-11e4-8fca-99bff9c19890
activity
| timestamp|
source | source_elapsed
-+--+-+
execute_cql3_query | 17:27:50,828 |
10.1.0.1 | 0
Parsing insert into test (user_id, created, event_data, event_id)values (
123456789 , 9eab8950-b70c-11e4-8fca-99bff9c19891, 'test', '16'); | 17:27:50,828
| 10.1.0.1 | 39
Preparing statement | 17:27:50,828 |
10.1.0.1 |135
Message received from /10.1.0.1 | 17:27:50,829 |
10.1.0.5 | 25
Sending message to /10.1.0.5 | 17:27:50,829 |
10.1.0.1 |421
Executing single-partition query on users | 17:27:50,829 |
10.1.0.5 |177
Acquiring sstable references | 17:27:50,829 |
10.1.0.5 |191
Merging memtable tombstones | 17:27:50,830 |
10.1.0.5 |208
Message received from /10.1.0.5 | 17:27:50,830 |
10.1.0.1 | 1461
[jira] [Updated] (CASSANDRA-8819) LOCAL_QUORUM writes returns wrong message
[
https://issues.apache.org/jira/browse/CASSANDRA-8819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Philip Thompson updated CASSANDRA-8819:
---
Reviewer: (was: Philip Thompson)
Reproduced In: 2.0.8
Tester: Philip Thompson
Fix Version/s: (was: 2.0.8)
2.0.13
Assignee: Tyler Hobbs
Any ideas what this could be Tyler? We discussed it on IRC today. It doesn't
appear to be CASSANDRA-7947 and Wei says they aren't using LWT. All nodes agree
on the schema.
LOCAL_QUORUM writes returns wrong message
-
Key: CASSANDRA-8819
URL: https://issues.apache.org/jira/browse/CASSANDRA-8819
Project: Cassandra
Issue Type: Bug
Components: Core
Environment: CentOS 6.6
Reporter: Wei Zhu
Assignee: Tyler Hobbs
Fix For: 2.0.13
We have two DC3, each with 7 nodes.
Here is the keyspace setup:
create keyspace test
with placement_strategy = 'NetworkTopologyStrategy'
and strategy_options = {DC2 : 3, DC1 : 3}
and durable_writes = true;
We brought down two nodes in DC2 for maintenance. We only write to DC1 using
local_quroum (using datastax JavaClient)
But we see this errors in the log:
Cassandra timeout during write query at consistency LOCAL_QUORUM (4 replica
were required but only 3 acknowledged the write
why does it say 4 replica were required? and Why would it give error back to
client since local_quorum should succeed.
Here are the output from nodetool status
Note: Ownership information does not include topology; for complete
information, specify a keyspace
Datacenter: DC2
===
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID
Rack
UN 10.2.0.1 10.92 GB 256 7.9% RAC206
UN 10.2.0.2 6.17 GB256 8.0% RAC106
UN 10.2.0.3 6.63 GB256 7.3% RAC107
DL 10.2.0.4 1.54 GB256 7.7% RAC107
UN 10.2.0.5 6.02 GB256 6.6% RAC106
UJ 10.2.0.6 3.68 GB256 ? RAC205
UN 10.2.0.7 7.22 GB256 7.7% RAC205
Datacenter: DC1
===
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID
Rack
UN 10.1.0.1 6.04 GB256 8.6% RAC10
UN 10.1.0.2 7.55 GB256 7.4% RAC8
UN 10.1.0.3 5.83 GB256 7.0% RAC9
UN 10.1.0.47.34 GB256 7.9% RAC6
UN 10.1.0.5 7.57 GB256 8.0% RAC7
UN 10.1.0.6 5.31 GB256 7.3% RAC10
UN 10.1.0.7 5.47 GB256 8.6% RAC9
I did a cql trace on the query and here is the trace, and it does say
Write timeout; received 3 of 4 required replies | 17:27:52,831 | 10.1.0.1
|2002873
at the end. I guess that is where the client gets the error from. But the
rows was inserted to Cassandra correctly. And I traced read with local_quorum
and it behaves correctly and the reads don't go to DC2. The problem is only
with writes on local_quorum.
Tracing session: 5a789fb0-b70d-11e4-8fca-99bff9c19890
activity
| timestamp
| source | source_elapsed
-+--+-+
execute_cql3_query | 17:27:50,828
| 10.1.0.1 | 0
Parsing insert into test (user_id, created, event_data, event_id)values (
123456789 , 9eab8950-b70c-11e4-8fca-99bff9c19891, 'test', '16'); |
17:27:50,828 | 10.1.0.1 | 39
Preparing statement | 17:27:50,828
| 10.1.0.1 |135
Message received from /10.1.0.1 | 17:27:50,829 |
10.1.0.5 | 25
Sending message to /10.1.0.5 | 17:27:50,829 |
[jira] [Updated] (CASSANDRA-8819) LOCAL_QUORUM writes returns wrong message
[
https://issues.apache.org/jira/browse/CASSANDRA-8819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei Zhu updated CASSANDRA-8819:
---
Reviewer: Philip Thompson
LOCAL_QUORUM writes returns wrong message
-
Key: CASSANDRA-8819
URL: https://issues.apache.org/jira/browse/CASSANDRA-8819
Project: Cassandra
Issue Type: Bug
Components: Core
Environment: CentOS 6.6
Reporter: Wei Zhu
Fix For: 2.0.8
We have two DC3, each with 7 nodes.
Here is the keyspace setup:
create keyspace test
with placement_strategy = 'NetworkTopologyStrategy'
and strategy_options = {DC2 : 3, DC1 : 3}
and durable_writes = true;
We brought down two nodes in DC2 for maintenance. We only write to DC1 using
local_quroum (using datastax JavaClient)
But we see this errors in the log:
Cassandra timeout during write query at consistency LOCAL_QUORUM (4 replica
were required but only 3 acknowledged the write
why does it say 4 replica were required? and Why would it give error back to
client since local_quorum should succeed.
Here are the output from nodetool status
Note: Ownership information does not include topology; for complete
information, specify a keyspace
Datacenter: DC2
===
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID
Rack
UN 10.2.0.1 10.92 GB 256 7.9% RAC206
UN 10.2.0.2 6.17 GB256 8.0% RAC106
UN 10.2.0.3 6.63 GB256 7.3% RAC107
DL 10.2.0.4 1.54 GB256 7.7% RAC107
UN 10.2.0.5 6.02 GB256 6.6% RAC106
UJ 10.2.0.6 3.68 GB256 ? RAC205
UN 10.2.0.7 7.22 GB256 7.7% RAC205
Datacenter: DC1
===
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID
Rack
UN 10.1.0.1 6.04 GB256 8.6% RAC10
UN 10.1.0.2 7.55 GB256 7.4% RAC8
UN 10.1.0.3 5.83 GB256 7.0% RAC9
UN 10.1.0.47.34 GB256 7.9% RAC6
UN 10.1.0.5 7.57 GB256 8.0% RAC7
UN 10.1.0.6 5.31 GB256 7.3% RAC10
UN 10.1.0.7 5.47 GB256 8.6% RAC9
I did a cql trace on the query and here is the trace, and it does say
Write timeout; received 3 of 4 required replies | 17:27:52,831 | 10.1.0.1
|2002873
at the end. I guess that is where the client gets the error from.
Tracing session: 5a789fb0-b70d-11e4-8fca-99bff9c19890
activity
| timestamp
| source | source_elapsed
-+--+-+
execute_cql3_query | 17:27:50,828
| 10.1.0.1 | 0
Parsing insert into test (user_id, created, event_data, event_id)values (
123456789 , 9eab8950-b70c-11e4-8fca-99bff9c19891, 'test', '16'); |
17:27:50,828 | 10.1.0.1 | 39
Preparing statement | 17:27:50,828
| 10.1.0.1 |135
Message received from /10.1.0.1 | 17:27:50,829 |
10.1.0.5 | 25
Sending message to /10.1.0.5 | 17:27:50,829 |
10.1.0.1 |421
Executing single-partition query on users | 17:27:50,829
| 10.1.0.5 |177
Acquiring sstable references | 17:27:50,829
| 10.1.0.5 |191
Merging memtable tombstones | 17:27:50,830
| 10.1.0.5 |
[jira] [Updated] (CASSANDRA-8819) LOCAL_QUORUM writes returns wrong message
[
https://issues.apache.org/jira/browse/CASSANDRA-8819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei Zhu updated CASSANDRA-8819:
---
Description:
We have two DC3, each with 7 nodes.
Here is the keyspace setup:
create keyspace test
with placement_strategy = 'NetworkTopologyStrategy'
and strategy_options = {DC2 : 3, DC1 : 3}
and durable_writes = true;
We brought down two nodes in DC2 for maintenance. We only write to DC1 using
local_quroum (using datastax JavaClient)
But we see this errors in the log:
Cassandra timeout during write query at consistency LOCAL_QUORUM (4 replica
were required but only 3 acknowledged the write
why does it say 4 replica were required? and Why would it give error back to
client since local_quorum should succeed.
Here are the output from nodetool status
Note: Ownership information does not include topology; for complete
information, specify a keyspace
Datacenter: DC2
===
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID
Rack
UN 10.2.0.1 10.92 GB 256 7.9% RAC206
UN 10.2.0.2 6.17 GB256 8.0% RAC106
UN 10.2.0.3 6.63 GB256 7.3% RAC107
DL 10.2.0.4 1.54 GB256 7.7% RAC107
UN 10.2.0.5 6.02 GB256 6.6% RAC106
UJ 10.2.0.6 3.68 GB256 ? RAC205
UN 10.2.0.7 7.22 GB256 7.7% RAC205
Datacenter: DC1
===
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID
Rack
UN 10.1.0.1 6.04 GB256 8.6% RAC10
UN 10.1.0.2 7.55 GB256 7.4% RAC8
UN 10.1.0.3 5.83 GB256 7.0% RAC9
UN 10.1.0.47.34 GB256 7.9% RAC6
UN 10.1.0.5 7.57 GB256 8.0% RAC7
UN 10.1.0.6 5.31 GB256 7.3% RAC10
UN 10.1.0.7 5.47 GB256 8.6% RAC9
I did a cql trace on the query and here is the trace, and it does say
Write timeout; received 3 of 4 required replies | 17:27:52,831 | 10.1.0.1 |
2002873
at the end. I guess that is where the client gets the error from. But the rows
was inserted to Cassandra correctly. And I traced read with local_quorum and it
behaves correctly and the reads don't go to DC2. The problem is only with
writes on local_quorum.
Tracing session: 5a789fb0-b70d-11e4-8fca-99bff9c19890
activity
| timestamp|
source | source_elapsed
-+--+-+
execute_cql3_query | 17:27:50,828 |
10.1.0.1 | 0
Parsing insert into test (user_id, created, event_data, event_id)values (
123456789 , 9eab8950-b70c-11e4-8fca-99bff9c19891, 'test', '16'); | 17:27:50,828
| 10.1.0.1 | 39
Preparing statement | 17:27:50,828 |
10.1.0.1 |135
Message received from /10.1.0.1 | 17:27:50,829 |
10.1.0.5 | 25
Sending message to /10.1.0.5 | 17:27:50,829 |
10.1.0.1 |421
Executing single-partition query on users | 17:27:50,829 |
10.1.0.5 |177
Acquiring sstable references | 17:27:50,829 |
10.1.0.5 |191
Merging memtable tombstones | 17:27:50,830 |
10.1.0.5 |208
Message received from /10.1.0.5 | 17:27:50,830 |
10.1.0.1 | 1461
