Re: Range of 'Bootstrap'?
>From the log you pasted on IRC, it's clear that the existing node is
telling the loadbalancing one to use "Bootstrap" as a token but I
can't see any reason for that to be happening.
Attached is a patch to add some more debug logs to this process; if
you can reproduce with that (after editing log4j level to debug) that
would be very useful.
On Mon, Mar 22, 2010 at 12:51 PM, Benjamin Black wrote:
> Could've misread it, it was late. Regardless, seems this should never happen.
>
> On Mon, Mar 22, 2010 at 6:19 AM, Gary Dusbabek wrote:
>> On Mon, Mar 22, 2010 at 01:58, Benjamin Black wrote:
>>> Looking at db/SystemTable.java I see the use of Bootstrap as a token
>>> during bootstrap, but it seems to be for the system table, not other
>>> keyspaces. Is it used more generally than that or is this a bug?
>>>
>>
>> I only see 'Bootstrap' being used as a row key in SystemTable. Which
>> lines of code are you referring to?
>>
>> Gary
>>
>
Index: src/java/org/apache/cassandra/dht/BootStrapper.java
===
--- src/java/org/apache/cassandra/dht/BootStrapper.java (revision 925698)
+++ src/java/org/apache/cassandra/dht/BootStrapper.java (working copy)
@@ -168,6 +168,7 @@
private static Token getBootstrapTokenFrom(InetAddress maxEndpoint)
{
Message message = new Message(FBUtilities.getLocalAddress(), "",
StorageService.Verb.BOOTSTRAP_TOKEN, ArrayUtils.EMPTY_BYTE_ARRAY);
+logger.debug("requesting bootstrap token with " + message);
BootstrapTokenCallback btc = new BootstrapTokenCallback();
MessagingService.instance.sendRR(message, maxEndpoint, btc);
return btc.getToken();
@@ -216,6 +217,7 @@
{
throw new AssertionError();
}
+logger.debug("replying to token request with " + tokenString + "
as " + response);
MessagingService.instance.sendOneWay(response, message.getFrom());
}
}
Index: src/java/org/apache/cassandra/net/Message.java
===
--- src/java/org/apache/cassandra/net/Message.java (revision 925698)
+++ src/java/org/apache/cassandra/net/Message.java (working copy)
@@ -26,6 +26,7 @@
import org.apache.cassandra.concurrent.StageManager;
import org.apache.cassandra.io.ICompactSerializer;
import org.apache.cassandra.service.StorageService;
+import org.apache.cassandra.utils.FBUtilities;
public class Message
{
@@ -116,7 +117,8 @@
.append("TYPE:" + getMessageType())
.append(separator)
.append("VERB:" + getVerb())
- .append(separator);
+ .append(separator)
+.append(FBUtilities.bytesToHex(body_));
return sbuf.toString();
}
}
Index: src/java/org/apache/cassandra/service/StorageService.java
===
--- src/java/org/apache/cassandra/service/StorageService.java (revision
925698)
+++ src/java/org/apache/cassandra/service/StorageService.java (working copy)
@@ -1268,6 +1268,7 @@
}
}
FBUtilities.sortSampledKeys(keys, range);
+logger_.debug("sampled keys are " + StringUtils.join(keys, ","));
if (keys.size() < 3)
return partitioner_.getRandomToken();
Re: Range of 'Bootstrap'?
Could've misread it, it was late. Regardless, seems this should never happen. On Mon, Mar 22, 2010 at 6:19 AM, Gary Dusbabek wrote: > On Mon, Mar 22, 2010 at 01:58, Benjamin Black wrote: >> Looking at db/SystemTable.java I see the use of Bootstrap as a token >> during bootstrap, but it seems to be for the system table, not other >> keyspaces. Is it used more generally than that or is this a bug? >> > > I only see 'Bootstrap' being used as a row key in SystemTable. Which > lines of code are you referring to? > > Gary >
Re: Range of 'Bootstrap'?
On Mon, Mar 22, 2010 at 01:58, Benjamin Black wrote: > Looking at db/SystemTable.java I see the use of Bootstrap as a token > during bootstrap, but it seems to be for the system table, not other > keyspaces. Is it used more generally than that or is this a bug? > I only see 'Bootstrap' being used as a row key in SystemTable. Which lines of code are you referring to? Gary
Re: Range of 'Bootstrap'?
Definitely a bug. On Mon, Mar 22, 2010 at 1:58 AM, Benjamin Black wrote: > Looking at db/SystemTable.java I see the use of Bootstrap as a token > during bootstrap, but it seems to be for the system table, not other > keyspaces. Is it used more generally than that or is this a bug? > > On Sun, Mar 21, 2010 at 11:29 PM, Benjamin Black wrote: >> As part of my continuous abuse of a small cluster for Chef cookbook >> development, I've run across a strange issue I'm hoping someone can >> explain. The following is output after upgrading from beta2 to beta3 >> and running nodetool rebalance on .140.224: >> >> Address Status Load Range >> Ring >> HS3ltoTahEIsK65e >> 10.177.144.176Up 5.41 KB B61cmDQkiI92SoCu >> |<--| >> 10.177.140.224Up 2.33 KB Bootstrap >> | | >> 10.177.138.136Up 6.13 KB HS3ltoTahEIsK65e >> |-->| >> >> From the logs: >> >> INFO [STREAM-STAGE:1] 2010-03-22 03:16:28,586 StreamOut.java (line >> 66) Sending a stream initiate message to /10.177.138.136 ... >> INFO [RMI TCP Connection(6)-173.203.205.232] 2010-03-22 03:16:30,596 >> StorageLoadBalancer.java (line 365) Sleeping 9 ms to wait for load >> information... >> INFO [RMI TCP Connection(6)-173.203.205.232] 2010-03-22 03:18:00,616 >> BootStrapper.java (line 104) New token will be Bootstrap to assume >> load from /10.177.138.136 >> INFO [RMI TCP Connection(6)-173.203.205.232] 2010-03-22 03:18:00,616 >> StorageService.java (line 1430) re-bootstrapping to new token >> Bootstrap >> INFO [RMI TCP Connection(6)-173.203.205.232] 2010-03-22 03:18:00,616 >> StorageService.java (line 376) Joining: sleeping 3 for pending >> range setup >> INFO [RMI TCP Connection(6)-173.203.205.232] 2010-03-22 03:18:30,626 >> StorageService.java (line 376) Bootstrapping >> INFO [MESSAGE-DESERIALIZER-POOL:1] 2010-03-22 03:18:30,656 >> StorageService.java (line 180) Bootstrap/move completed! Now serving >> reads. >> >> >> Although it is possible the range 'Bootstrap' was randomly generated, >> it seems more likely it has somehow been assigned explicitly. Can >> someone illuminate what is going on here? >> >> >> b >> >
Re: Range of 'Bootstrap'?
Looking at db/SystemTable.java I see the use of Bootstrap as a token during bootstrap, but it seems to be for the system table, not other keyspaces. Is it used more generally than that or is this a bug? On Sun, Mar 21, 2010 at 11:29 PM, Benjamin Black wrote: > As part of my continuous abuse of a small cluster for Chef cookbook > development, I've run across a strange issue I'm hoping someone can > explain. The following is output after upgrading from beta2 to beta3 > and running nodetool rebalance on .140.224: > > Address Status Load Range > Ring > HS3ltoTahEIsK65e > 10.177.144.176Up 5.41 KB B61cmDQkiI92SoCu > |<--| > 10.177.140.224Up 2.33 KB Bootstrap > | | > 10.177.138.136Up 6.13 KB HS3ltoTahEIsK65e > |-->| > > From the logs: > > INFO [STREAM-STAGE:1] 2010-03-22 03:16:28,586 StreamOut.java (line > 66) Sending a stream initiate message to /10.177.138.136 ... > INFO [RMI TCP Connection(6)-173.203.205.232] 2010-03-22 03:16:30,596 > StorageLoadBalancer.java (line 365) Sleeping 9 ms to wait for load > information... > INFO [RMI TCP Connection(6)-173.203.205.232] 2010-03-22 03:18:00,616 > BootStrapper.java (line 104) New token will be Bootstrap to assume > load from /10.177.138.136 > INFO [RMI TCP Connection(6)-173.203.205.232] 2010-03-22 03:18:00,616 > StorageService.java (line 1430) re-bootstrapping to new token > Bootstrap > INFO [RMI TCP Connection(6)-173.203.205.232] 2010-03-22 03:18:00,616 > StorageService.java (line 376) Joining: sleeping 3 for pending > range setup > INFO [RMI TCP Connection(6)-173.203.205.232] 2010-03-22 03:18:30,626 > StorageService.java (line 376) Bootstrapping > INFO [MESSAGE-DESERIALIZER-POOL:1] 2010-03-22 03:18:30,656 > StorageService.java (line 180) Bootstrap/move completed! Now serving > reads. > > > Although it is possible the range 'Bootstrap' was randomly generated, > it seems more likely it has somehow been assigned explicitly. Can > someone illuminate what is going on here? > > > b >
Range of 'Bootstrap'?
As part of my continuous abuse of a small cluster for Chef cookbook development, I've run across a strange issue I'm hoping someone can explain. The following is output after upgrading from beta2 to beta3 and running nodetool rebalance on .140.224: Address Status Load Range Ring HS3ltoTahEIsK65e 10.177.144.176Up 5.41 KB B61cmDQkiI92SoCu |<--| 10.177.140.224Up 2.33 KB Bootstrap | | 10.177.138.136Up 6.13 KB HS3ltoTahEIsK65e |-->| >From the logs: INFO [STREAM-STAGE:1] 2010-03-22 03:16:28,586 StreamOut.java (line 66) Sending a stream initiate message to /10.177.138.136 ... INFO [RMI TCP Connection(6)-173.203.205.232] 2010-03-22 03:16:30,596 StorageLoadBalancer.java (line 365) Sleeping 9 ms to wait for load information... INFO [RMI TCP Connection(6)-173.203.205.232] 2010-03-22 03:18:00,616 BootStrapper.java (line 104) New token will be Bootstrap to assume load from /10.177.138.136 INFO [RMI TCP Connection(6)-173.203.205.232] 2010-03-22 03:18:00,616 StorageService.java (line 1430) re-bootstrapping to new token Bootstrap INFO [RMI TCP Connection(6)-173.203.205.232] 2010-03-22 03:18:00,616 StorageService.java (line 376) Joining: sleeping 3 for pending range setup INFO [RMI TCP Connection(6)-173.203.205.232] 2010-03-22 03:18:30,626 StorageService.java (line 376) Bootstrapping INFO [MESSAGE-DESERIALIZER-POOL:1] 2010-03-22 03:18:30,656 StorageService.java (line 180) Bootstrap/move completed! Now serving reads. Although it is possible the range 'Bootstrap' was randomly generated, it seems more likely it has somehow been assigned explicitly. Can someone illuminate what is going on here? b
