[
https://issues.apache.org/jira/browse/CASSANDRA-2081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12989689#comment-12989689
]
Thibaut commented on CASSANDRA-2081:
------------------------------------
Debug Output:
========================================
192.168.0.1
--------------------
2011-02-02 12:54:29,117 DEBUG [ScheduledTasks:1] StorageLoadBalancer.java (line
349) Disseminating load info ...
2011-02-02 12:54:35,016 DEBUG [ReadStage:8] RangeSliceVerbHandler.java (line
55) Sending RangeSliceReply{rows=} to 567@/192.168.0.7
2011-02-02 12:54:45,015 DEBUG [ReadStage:9] RangeSliceVerbHandler.java (line
55) Sending RangeSliceReply{rows=} to 600@/192.168.0.7
2011-02-02 12:54:49,679 DEBUG [pool-1-thread-5] ClientState.java (line 91)
logged out: #<User allow_all groups=[]>
========================================
192.168.0.2
--------------------
2011-02-02 12:54:30,147 DEBUG [ScheduledTasks:1] StorageLoadBalancer.java (line
349) Disseminating load info ...
2011-02-02 12:54:34,789 DEBUG [MutationStage:5] RowMutationVerbHandler.java
(line 52) Applying RowMutation(keyspace='table_lists',
key='32383663623561363162363164306231333162326264656232303730303866625f7777772e7768617469736d79697076362e6e65745f686f7374726571756573746c6973745f393232333337303734303230373130313432315f36346334613666322d333737392d343331642d623334382d663636383533316233656334',
modifications=[ColumnFamily(table_lists
[7461626c655f6c69737473:false:116@1296647674390000!2419199,]),
ColumnFamily(table_lists_meta
[6e6578745f72657175657374:false:8@1296647674391000!2419199,])])
2011-02-02 12:54:34,790 DEBUG [MutationStage:5] Table.java (line 397) applying
mutation of row
32383663623561363162363164306231333162326264656232303730303866625f7777772e7768617469736d79697076362e6e65745f686f7374726571756573746c6973745f393232333337303734303230373130313432315f36346334613666322d333737392d343331642d623334382d663636383533316233656334
2011-02-02 12:54:34,792 DEBUG [MutationStage:5] RowMutationVerbHandler.java
(line 81) RowMutation(keyspace='table_lists',
key='32383663623561363162363164306231333162326264656232303730303866625f7777772e7768617469736d79697076362e6e65745f686f7374726571756573746c6973745f393232333337303734303230373130313432315f36346334613666322d333737392d343331642d623334382d663636383533316233656334',
modifications=[ColumnFamily(table_lists
[7461626c655f6c69737473:false:116@1296647674390000!2419199,]),
ColumnFamily(table_lists_meta
[6e6578745f72657175657374:false:8@1296647674391000!2419199,])]) applied.
Sending response to 566@/192.168.0.7
2011-02-02 12:54:34,953 DEBUG [ReadStage:8] RangeSliceVerbHandler.java (line
55) Sending RangeSliceReply{rows=} to 567@/192.168.0.7
2011-02-02 12:54:44,965 DEBUG [ReadStage:9] RangeSliceVerbHandler.java (line
55) Sending RangeSliceReply{rows=} to 600@/192.168.0.7
2011-02-02 12:54:45,057 DEBUG [ScheduledTasks:1] GCInspector.java (line 135) GC
for ParNew: 14 ms, 13086296 reclaimed leaving 2087311488 used; max is 4856348672
2011-02-02 12:54:49,613 DEBUG [pool-1-thread-11] ClientState.java (line 91)
logged out: #<User allow_all groups=[]>
========================================
192.168.0.3
--------------------
2011-02-02 12:54:30,920 DEBUG [ScheduledTasks:1] StorageLoadBalancer.java (line
349) Disseminating load info ...
2011-02-02 12:54:34,368 DEBUG [MutationStage:8] RowMutationVerbHandler.java
(line 52) Applying RowMutation(keyspace='table_lists',
key='36306463373934666139396136366665303539636130373063333366323066615f2e7768617469736d79697076362e6e65745f686f7374726571756573746c6973745f393232333337303734303230373130313636345f65626263303033372d356639612d343066382d383833382d356436336433616233366165',
modifications=[ColumnFamily(table_lists
[7461626c655f6c69737473:false:116@1296647674314000!2419199,]),
ColumnFamily(table_lists_meta
[6e6578745f72657175657374:false:8@1296647674322000!2419199,])])
2011-02-02 12:54:34,369 DEBUG [MutationStage:8] Table.java (line 397) applying
mutation of row
36306463373934666139396136366665303539636130373063333366323066615f2e7768617469736d79697076362e6e65745f686f7374726571756573746c6973745f393232333337303734303230373130313636345f65626263303033372d356639612d343066382d383833382d356436336433616233366165
2011-02-02 12:54:34,371 DEBUG [MutationStage:8] RowMutationVerbHandler.java
(line 81) RowMutation(keyspace='table_lists',
key='36306463373934666139396136366665303539636130373063333366323066615f2e7768617469736d79697076362e6e65745f686f7374726571756573746c6973745f393232333337303734303230373130313636345f65626263303033372d356639612d343066382d383833382d356436336433616233366165',
modifications=[ColumnFamily(table_lists
[7461626c655f6c69737473:false:116@1296647674314000!2419199,]),
ColumnFamily(table_lists_meta
[6e6578745f72657175657374:false:8@1296647674322000!2419199,])]) applied.
Sending response to 562@/192.168.0.7
2011-02-02 12:54:34,381 DEBUG [MutationStage:9] RowMutationVerbHandler.java
(line 52) Applying RowMutation(keyspace='table_lists',
key='32383663623561363162363164306231333162326264656232303730303866625f7777772e7768617469736d79697076362e6e65745f686f7374726571756573746c6973745f393232333337303734303230373130313432315f36346334613666322d333737392d343331642d623334382d663636383533316233656334',
modifications=[ColumnFamily(table_lists
[7461626c655f6c69737473:false:116@1296647674390000!2419199,]),
ColumnFamily(table_lists_meta
[6e6578745f72657175657374:false:8@1296647674391000!2419199,])])
2011-02-02 12:54:34,382 DEBUG [MutationStage:9] Table.java (line 397) applying
mutation of row
32383663623561363162363164306231333162326264656232303730303866625f7777772e7768617469736d79697076362e6e65745f686f7374726571756573746c6973745f393232333337303734303230373130313432315f36346334613666322d333737392d343331642d623334382d663636383533316233656334
2011-02-02 12:54:34,383 DEBUG [MutationStage:9] RowMutationVerbHandler.java
(line 81) RowMutation(keyspace='table_lists',
key='32383663623561363162363164306231333162326264656232303730303866625f7777772e7768617469736d79697076362e6e65745f686f7374726571756573746c6973745f393232333337303734303230373130313432315f36346334613666322d333737392d343331642d623334382d663636383533316233656334',
modifications=[ColumnFamily(table_lists
[7461626c655f6c69737473:false:116@1296647674390000!2419199,]),
ColumnFamily(table_lists_meta
[6e6578745f72657175657374:false:8@1296647674391000!2419199,])]) applied.
Sending response to 564@/192.168.0.7
2011-02-02 12:54:34,539 DEBUG [ReadStage:8] RangeSliceVerbHandler.java (line
55) Sending RangeSliceReply{rows=} to 567@/192.168.0.7
2011-02-02 12:54:44,540 DEBUG [ReadStage:9] RangeSliceVerbHandler.java (line
55) Sending RangeSliceReply{rows=} to 600@/192.168.0.7
2011-02-02 12:54:49,202 DEBUG [pool-1-thread-13] ClientState.java (line 91)
logged out: #<User allow_all groups=[]>
========================================
192.168.0.7
--------------------
2011-02-02 12:54:34,059 DEBUG [ScheduledTasks:1] StorageLoadBalancer.java (line
349) Disseminating load info ...
2011-02-02 12:54:34,356 DEBUG [pool-1-thread-1] CassandraServer.java (line 355)
batch_mutate
2011-02-02 12:54:34,375 DEBUG [pool-1-thread-1] StorageProxy.java (line 154)
insert writing key
36306463373934666139396136366665303539636130373063333366323066615f2e7768617469736d79697076362e6e65745f686f7374726571756573746c6973745f393232333337303734303230373130313636345f65626263303033372d356639612d343066382d383833382d356436336433616233366165
to 561@/192.168.0.5
2011-02-02 12:54:34,375 DEBUG [pool-1-thread-1] StorageProxy.java (line 154)
insert writing key
36306463373934666139396136366665303539636130373063333366323066615f2e7768617469736d79697076362e6e65745f686f7374726571756573746c6973745f393232333337303734303230373130313636345f65626263303033372d356639612d343066382d383833382d356436336433616233366165
to 562@/192.168.0.3
2011-02-02 12:54:34,381 DEBUG [pool-1-thread-1] StorageProxy.java (line 154)
insert writing key
36306463373934666139396136366665303539636130373063333366323066615f2e7768617469736d79697076362e6e65745f686f7374726571756573746c6973745f393232333337303734303230373130313636345f65626263303033372d356639612d343066382d383833382d356436336433616233366165
to 563@/94.242.198.13
2011-02-02 12:54:34,384 DEBUG [RequestResponseStage:3] ResponseVerbHandler.java
(line 48) Processing response on a callback from 561@/192.168.0.5
2011-02-02 12:54:34,386 DEBUG [RequestResponseStage:5] ResponseVerbHandler.java
(line 48) Processing response on a callback from 563@/192.168.0.4
2011-02-02 12:54:34,386 DEBUG [RequestResponseStage:4] ResponseVerbHandler.java
(line 48) Processing response on a callback from 562@/192.168.0.3
2011-02-02 12:54:34,391 DEBUG [pool-1-thread-2] CassandraServer.java (line 355)
batch_mutate
2011-02-02 12:54:34,393 DEBUG [pool-1-thread-2] StorageProxy.java (line 154)
insert writing key
32383663623561363162363164306231333162326264656232303730303866625f7777772e7768617469736d79697076362e6e65745f686f7374726571756573746c6973745f393232333337303734303230373130313432315f36346334613666322d333737392d343331642d623334382d663636383533316233656334
to 564@/192.168.0.3
2011-02-02 12:54:34,394 DEBUG [pool-1-thread-2] StorageProxy.java (line 154)
insert writing key
32383663623561363162363164306231333162326264656232303730303866625f7777772e7768617469736d79697076362e6e65745f686f7374726571756573746c6973745f393232333337303734303230373130313432315f36346334613666322d333737392d343331642d623334382d663636383533316233656334
to 565@/192.168.0.4
2011-02-02 12:54:34,394 DEBUG [pool-1-thread-2] StorageProxy.java (line 154)
insert writing key
32383663623561363162363164306231333162326264656232303730303866625f7777772e7768617469736d79697076362e6e65745f686f7374726571756573746c6973745f393232333337303734303230373130313432315f36346334613666322d333737392d343331642d623334382d663636383533316233656334
to 566@/192.168.0.2
2011-02-02 12:54:34,398 DEBUG [RequestResponseStage:6] ResponseVerbHandler.java
(line 48) Processing response on a callback from 564@/192.168.0.3
2011-02-02 12:54:34,398 DEBUG [RequestResponseStage:7] ResponseVerbHandler.java
(line 48) Processing response on a callback from 565@/192.168.0.4
2011-02-02 12:54:34,399 DEBUG [RequestResponseStage:8] ResponseVerbHandler.java
(line 48) Processing response on a callback from 566@/192.168.0.2
2011-02-02 12:54:34,531 DEBUG [pool-1-thread-3] CassandraServer.java (line 445)
range_slice
2011-02-02 12:54:34,533 DEBUG [pool-1-thread-3] StorageProxy.java (line 514)
RangeSliceCommand{keyspace='table_usersources',
column_family='table_usersources_meta', super_column=null,
predicate=SlicePredicate(column_names:[java.nio.HeapByteBuffer[pos=76 lim=88
cap=65536]]), range=[,], max_keys=250}
2011-02-02 12:54:34,534 DEBUG [pool-1-thread-3] StorageProxy.java (line 705)
restricted ranges for query [,] are [[,24], (24,49], (49,6d], (6d,92], (92,b6],
(b6,db], (db,ffffffffffffffff], (ffffffffffffffff,]]
2011-02-02 12:54:34,536 DEBUG [pool-1-thread-3] ReadCallback.java (line 58)
ReadCallback blocking for 2 responses
2011-02-02 12:54:34,536 DEBUG [pool-1-thread-3] StorageProxy.java (line 562)
reading RangeSliceCommand{keyspace='table_usersources',
column_family='table_usersources_meta', super_column=null,
predicate=SlicePredicate(column_names:[java.nio.HeapByteBuffer[pos=76 lim=88
cap=65536]]), range=[,24], max_keys=250} from 567@/192.168.0.1
2011-02-02 12:54:34,537 DEBUG [pool-1-thread-3] StorageProxy.java (line 562)
reading RangeSliceCommand{keyspace='table_usersources',
column_family='table_usersources_meta', super_column=null,
predicate=SlicePredicate(column_names:[java.nio.HeapByteBuffer[pos=76 lim=88
cap=65536]]), range=[,24], max_keys=250} from 567@/192.168.0.2
2011-02-02 12:54:34,537 DEBUG [pool-1-thread-3] StorageProxy.java (line 562)
reading RangeSliceCommand{keyspace='table_usersources',
column_family='table_usersources_meta', super_column=null,
predicate=SlicePredicate(column_names:[java.nio.HeapByteBuffer[pos=76 lim=88
cap=65536]]), range=[,24], max_keys=250} from 567@/192.168.0.3
2011-02-02 12:54:34,554 DEBUG [RequestResponseStage:1] ResponseVerbHandler.java
(line 48) Processing response on a callback from 567@/192.168.0.3
2011-02-02 12:54:44,537 DEBUG [pool-1-thread-3] CassandraServer.java (line 483)
... timed out
2011-02-02 12:54:44,548 DEBUG [pool-1-thread-3] ClientState.java (line 91)
logged out: #<User allow_all groups=[]>
2011-02-02 12:54:44,549 DEBUG [pool-1-thread-4] CassandraServer.java (line 445)
range_slice
2011-02-02 12:54:44,550 DEBUG [pool-1-thread-4] StorageProxy.java (line 514)
RangeSliceCommand{keyspace='table_usersources',
column_family='table_usersources_meta', super_column=null,
predicate=SlicePredicate(column_names:[java.nio.HeapByteBuffer[pos=76 lim=88
cap=65536]]), range=[,], max_keys=250}
2011-02-02 12:54:44,550 DEBUG [pool-1-thread-4] StorageProxy.java (line 705)
restricted ranges for query [,] are [[,24], (24,49], (49,6d], (6d,92], (92,b6],
(b6,db], (db,ffffffffffffffff], (ffffffffffffffff,]]
2011-02-02 12:54:44,550 DEBUG [pool-1-thread-4] ReadCallback.java (line 58)
ReadCallback blocking for 2 responses
2011-02-02 12:54:44,552 DEBUG [pool-1-thread-4] StorageProxy.java (line 562)
reading RangeSliceCommand{keyspace='table_usersources',
column_family='table_usersources_meta', super_column=null,
predicate=SlicePredicate(column_names:[java.nio.HeapByteBuffer[pos=76 lim=88
cap=65536]]), range=[,24], max_keys=250} from 600@/192.168.0.1
2011-02-02 12:54:44,552 DEBUG [pool-1-thread-4] StorageProxy.java (line 562)
reading RangeSliceCommand{keyspace='table_usersources',
column_family='table_usersources_meta', super_column=null,
predicate=SlicePredicate(column_names:[java.nio.HeapByteBuffer[pos=76 lim=88
cap=65536]]), range=[,24], max_keys=250} from 600@/192.168.0.2
2011-02-02 12:54:44,552 DEBUG [pool-1-thread-4] StorageProxy.java (line 562)
reading RangeSliceCommand{keyspace='table_usersources',
column_family='table_usersources_meta', super_column=null,
predicate=SlicePredicate(column_names:[java.nio.HeapByteBuffer[pos=76 lim=88
cap=65536]]), range=[,24], max_keys=250} from 600@/192.168.0.3
2011-02-02 12:54:44,554 DEBUG [RequestResponseStage:5] ResponseVerbHandler.java
(line 48) Processing response on a callback from 600@/192.168.0.1
2011-02-02 12:54:54,550 DEBUG [pool-1-thread-4] CassandraServer.java (line 483)
... timed out
2011-02-02 12:54:54,552 DEBUG [pool-1-thread-4] ClientState.java (line 91)
logged out: #<User allow_all groups=[]>
> Consistency QUORUM does not work anymore (hector:Could not fullfill request
> on this host)
> -----------------------------------------------------------------------------------------
>
> Key: CASSANDRA-2081
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2081
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: linux, hector + cassandra
> Reporter: Thibaut
> Priority: Blocker
> Fix For: 0.7.1
>
>
> I'm using apache-cassandra-2011-01-28_20-06-01.jar and hector 7.0.25.
> Using consistency level Quorum won't work anymore (tested it on read).
> Consisteny level ONE still works though
> I have tried this with one dead node in my cluster.
> If I restart cassandra with an older svn revision
> (apache-cassandra-2011-01-28_20-06-01.jar), I can access the cluster with
> consistency level QUORUM again, while still using
> apache-cassandra-2011-01-28_20-06-01.jar and hector 7.0.25 in my application.
> 11/01/31 19:54:38 ERROR connection.CassandraHostRetryService: Downed
> intr1n18(192.168.0.18):9160 host still appears to be down: Unable to open
> transport to intr1n18(192.168.0.18):9160 , java.net.NoRouteToHostException:
> No route to host
> 11/01/31 19:54:38 INFO connection.CassandraHostRetryService: Downed Host
> retry status false with host: intr1n18(192.168.0.18):9160
> 11/01/31 19:54:45 ERROR connection.HConnectionManager: Could not fullfill
> request on this host CassandraClient<intr1n11:9160-483>
> intr1n11 is marked as up however and I can also access the node through the
> cassandra cli.
> 192.168.0.1 Up Normal 8.02 GB 5.00% 0cc
> 192.168.0.2 Up Normal 7.96 GB 5.00% 199
> 192.168.0.3 Up Normal 8.24 GB 5.00% 266
> 192.168.0.4 Up Normal 4.94 GB 5.00% 333
> 192.168.0.5 Up Normal 5.02 GB 5.00% 400
> 192.168.0.6 Up Normal 5 GB 5.00% 4cc
> 192.168.0.7 Up Normal 5.1 GB 5.00% 599
> 192.168.0.8 Up Normal 5.07 GB 5.00% 666
> 192.168.0.9 Up Normal 4.78 GB 5.00% 733
> 192.168.0.10 Up Normal 4.34 GB 5.00% 7ff
> 192.168.0.11 Up Normal 5.01 GB 5.00% 8cc
> 192.168.0.12 Up Normal 5.31 GB 5.00% 999
> 192.168.0.13 Up Normal 5.56 GB 5.00% a66
> 192.168.0.14 Up Normal 5.82 GB 5.00% b33
> 192.168.0.15 Up Normal 5.57 GB 5.00% c00
> 192.168.0.16 Up Normal 5.03 GB 5.00% ccc
> 192.168.0.17 Up Normal 4.77 GB 5.00% d99
> 192.168.0.18 Down Normal ? 5.00% e66
> 192.168.0.19 Up Normal 4.78 GB 5.00% f33
> 192.168.0.20 Up Normal 4.83 GB 5.00% ffffffffffffffff
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
