[jira] [Commented] (CASSANDRA-3935) Hints not delivered possibly because of pagination issue
[ https://issues.apache.org/jira/browse/CASSANDRA-3935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13563164#comment-13563164 ] Jonathan Ellis commented on CASSANDRA-3935: --- See CASSANDRA-5068 > Hints not delivered possibly because of pagination issue > > > Key: CASSANDRA-3935 > URL: https://issues.apache.org/jira/browse/CASSANDRA-3935 > Project: Cassandra > Issue Type: Bug >Affects Versions: 1.1.0 >Reporter: B. Todd Burruss > > I'm testing hinted handoff in 1.1 beta1 and cannot seem to get a hint > delivered. 3 node cluster, RF = 3, writing with CL = ONE. killed a host > then did the write using the CLI on another node. I can see hint waiting > using CLI and I see the log messages at the end of this email. this suggests > the hints exist bu are not being delivered (and I'll see the log messages > over and over.) > I did see tracing with debugger and see that in > HintedHandoffManager.deliverHintsToEndpointInternal, this line will remove > the hint because of the Integer.MAX_VALUE > ColumnFamily hintsPage = > ColumnFamilyStore.removeDeleted(hintStore.getColumnFamily(filter), > Integer.MAX_VALUE); > I'm not sure I quite understand while MAX is used when the same "remove" is > done in getColumnFamily(filter). regardless if it is useful or not, it > prevents the hints from delivery. > any thoughts? > [default@unknown] use system; > Authenticated to keyspace: system > [default@system] list hintscolumnfamily; > Using default limit of 100 > --- > RowKey: 00 > => (super_column=493ecfa05c1411e10da23097c7ff, > (column=6b6579, value=6b35, timestamp=132999580, ttl=86400) > (column=6d75746174696f6e, > value=000662746f64646200026b35000103e80103e87fff80010002633504b96d055fd13c68696e746564207772697465, > timestamp=132999579, ttl=86400) > (column=7461626c65, value=62746f646462, timestamp=132999580, > ttl=86400) > (column=76657273696f6e, value=0004, timestamp=132999580, > ttl=86400)) > 1 Row Returned. > Elapsed time: 58 msec(s). > INFO [HintedHandoff:1] 2012-02-20 14:44:53,811 HintedHandOffManager.java > (line 296) Started hinted handoff for token: 0 with IP: /192.168.56.1 > INFO [HintedHandoff:1] 2012-02-20 14:44:53,815 HintedHandOffManager.java > (line 373) Finished hinted handoff of 0 rows to endpoint /192.168.56.1 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3935) Hints not delivered possibly because of pagination issue
[ https://issues.apache.org/jira/browse/CASSANDRA-3935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13423891#comment-13423891 ] Will Pierce commented on CASSANDRA-3935: i saw a similar thing with cassandra 1.1.2 on a test cluster of 5 nodes when doing some testing of writes with 1 node down. the hinted handoffs (of 0 rows) only stopped after i did a manual compact. for a while i was seeing data show up in memtables on the resurrected node, long after it had already recovered. > Hints not delivered possibly because of pagination issue > > > Key: CASSANDRA-3935 > URL: https://issues.apache.org/jira/browse/CASSANDRA-3935 > Project: Cassandra > Issue Type: Bug >Affects Versions: 1.1.0 >Reporter: B. Todd Burruss > > I'm testing hinted handoff in 1.1 beta1 and cannot seem to get a hint > delivered. 3 node cluster, RF = 3, writing with CL = ONE. killed a host > then did the write using the CLI on another node. I can see hint waiting > using CLI and I see the log messages at the end of this email. this suggests > the hints exist bu are not being delivered (and I'll see the log messages > over and over.) > I did see tracing with debugger and see that in > HintedHandoffManager.deliverHintsToEndpointInternal, this line will remove > the hint because of the Integer.MAX_VALUE > ColumnFamily hintsPage = > ColumnFamilyStore.removeDeleted(hintStore.getColumnFamily(filter), > Integer.MAX_VALUE); > I'm not sure I quite understand while MAX is used when the same "remove" is > done in getColumnFamily(filter). regardless if it is useful or not, it > prevents the hints from delivery. > any thoughts? > [default@unknown] use system; > Authenticated to keyspace: system > [default@system] list hintscolumnfamily; > Using default limit of 100 > --- > RowKey: 00 > => (super_column=493ecfa05c1411e10da23097c7ff, > (column=6b6579, value=6b35, timestamp=132999580, ttl=86400) > (column=6d75746174696f6e, > value=000662746f64646200026b35000103e80103e87fff80010002633504b96d055fd13c68696e746564207772697465, > timestamp=132999579, ttl=86400) > (column=7461626c65, value=62746f646462, timestamp=132999580, > ttl=86400) > (column=76657273696f6e, value=0004, timestamp=132999580, > ttl=86400)) > 1 Row Returned. > Elapsed time: 58 msec(s). > INFO [HintedHandoff:1] 2012-02-20 14:44:53,811 HintedHandOffManager.java > (line 296) Started hinted handoff for token: 0 with IP: /192.168.56.1 > INFO [HintedHandoff:1] 2012-02-20 14:44:53,815 HintedHandOffManager.java > (line 373) Finished hinted handoff of 0 rows to endpoint /192.168.56.1 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3935) Hints not delivered possibly because of pagination issue
[ https://issues.apache.org/jira/browse/CASSANDRA-3935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13215674#comment-13215674 ] Brandon Williams commented on CASSANDRA-3935: - bq. after that periodically logs that "Finished hinted handoff of 0 rows to endpoint ... " This happens when there is one sstable for hints, since it won't currently compact it away and tombstones remain inside it. > Hints not delivered possibly because of pagination issue > > > Key: CASSANDRA-3935 > URL: https://issues.apache.org/jira/browse/CASSANDRA-3935 > Project: Cassandra > Issue Type: Bug >Affects Versions: 1.1.0 >Reporter: B. Todd Burruss > > I'm testing hinted handoff in 1.1 beta1 and cannot seem to get a hint > delivered. 3 node cluster, RF = 3, writing with CL = ONE. killed a host > then did the write using the CLI on another node. I can see hint waiting > using CLI and I see the log messages at the end of this email. this suggests > the hints exist bu are not being delivered (and I'll see the log messages > over and over.) > I did see tracing with debugger and see that in > HintedHandoffManager.deliverHintsToEndpointInternal, this line will remove > the hint because of the Integer.MAX_VALUE > ColumnFamily hintsPage = > ColumnFamilyStore.removeDeleted(hintStore.getColumnFamily(filter), > Integer.MAX_VALUE); > I'm not sure I quite understand while MAX is used when the same "remove" is > done in getColumnFamily(filter). regardless if it is useful or not, it > prevents the hints from delivery. > any thoughts? > [default@unknown] use system; > Authenticated to keyspace: system > [default@system] list hintscolumnfamily; > Using default limit of 100 > --- > RowKey: 00 > => (super_column=493ecfa05c1411e10da23097c7ff, > (column=6b6579, value=6b35, timestamp=132999580, ttl=86400) > (column=6d75746174696f6e, > value=000662746f64646200026b35000103e80103e87fff80010002633504b96d055fd13c68696e746564207772697465, > timestamp=132999579, ttl=86400) > (column=7461626c65, value=62746f646462, timestamp=132999580, > ttl=86400) > (column=76657273696f6e, value=0004, timestamp=132999580, > ttl=86400)) > 1 Row Returned. > Elapsed time: 58 msec(s). > INFO [HintedHandoff:1] 2012-02-20 14:44:53,811 HintedHandOffManager.java > (line 296) Started hinted handoff for token: 0 with IP: /192.168.56.1 > INFO [HintedHandoff:1] 2012-02-20 14:44:53,815 HintedHandOffManager.java > (line 373) Finished hinted handoff of 0 rows to endpoint /192.168.56.1 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3935) Hints not delivered possibly because of pagination issue
[ https://issues.apache.org/jira/browse/CASSANDRA-3935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13215331#comment-13215331 ] Manoj Kanta Mainali commented on CASSANDRA-3935: The above message is also logged in the 1.0.7 version, even when there is no hints left in the hints column family. I was able to reproduce it with the following steps, 1. Create a cluster with 2 nodes 2. Create keyspace with rf=2 and also create a column family 3. Shut down one node and insert some rows, say 100 to the column family, with CL=1 4. Restart the down node Node 1 will log that "Finished hinted handoff of 100 rows to endpoint ... " and after that periodically logs that "Finished hinted handoff of 0 rows to endpoint ... " > Hints not delivered possibly because of pagination issue > > > Key: CASSANDRA-3935 > URL: https://issues.apache.org/jira/browse/CASSANDRA-3935 > Project: Cassandra > Issue Type: Bug >Affects Versions: 1.1.0 >Reporter: B. Todd Burruss > > I'm testing hinted handoff in 1.1 beta1 and cannot seem to get a hint > delivered. 3 node cluster, RF = 3, writing with CL = ONE. killed a host > then did the write using the CLI on another node. I can see hint waiting > using CLI and I see the log messages at the end of this email. this suggests > the hints exist bu are not being delivered (and I'll see the log messages > over and over.) > I did see tracing with debugger and see that in > HintedHandoffManager.deliverHintsToEndpointInternal, this line will remove > the hint because of the Integer.MAX_VALUE > ColumnFamily hintsPage = > ColumnFamilyStore.removeDeleted(hintStore.getColumnFamily(filter), > Integer.MAX_VALUE); > I'm not sure I quite understand while MAX is used when the same "remove" is > done in getColumnFamily(filter). regardless if it is useful or not, it > prevents the hints from delivery. > any thoughts? > [default@unknown] use system; > Authenticated to keyspace: system > [default@system] list hintscolumnfamily; > Using default limit of 100 > --- > RowKey: 00 > => (super_column=493ecfa05c1411e10da23097c7ff, > (column=6b6579, value=6b35, timestamp=132999580, ttl=86400) > (column=6d75746174696f6e, > value=000662746f64646200026b35000103e80103e87fff80010002633504b96d055fd13c68696e746564207772697465, > timestamp=132999579, ttl=86400) > (column=7461626c65, value=62746f646462, timestamp=132999580, > ttl=86400) > (column=76657273696f6e, value=0004, timestamp=132999580, > ttl=86400)) > 1 Row Returned. > Elapsed time: 58 msec(s). > INFO [HintedHandoff:1] 2012-02-20 14:44:53,811 HintedHandOffManager.java > (line 296) Started hinted handoff for token: 0 with IP: /192.168.56.1 > INFO [HintedHandoff:1] 2012-02-20 14:44:53,815 HintedHandOffManager.java > (line 373) Finished hinted handoff of 0 rows to endpoint /192.168.56.1 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
