> [2013-12-08 11:04:02,047] Repair session ff16c510-5ff7-11e3-97c0-5973cc397f8f
> for range (1246984843639507027,1266616572749926276] failed with error
> org.apache.cassandra.exceptions.RepairException: [repair
> #ff16c510-5ff7-11e3-97c0-5973cc397f8f on keyspace_name/col_family1,
> (1246984843639507027,1266616572749926276]] Validation failed in /10.x.x.48
the 10.x.x.48 node sent a tree response (merkle tree) to this node that did not
contain the tree. This node then killed the repair session.
Look for log messages on 10.x.x.48 that correlate with the repair session ID
above. They may look like
logger.error("Failed creating a merkle tree for " + desc + ", " + initiator + "
(see log for details)”);
or
logger.info(String.format("[repair #%s] Sending completed merkle tree to %s for
%s/%s", desc.sessionId, initiator, desc.keyspace, desc.columnFamily));
Hope that helps.
-----------------
Aaron Morton
New Zealand
@aaronmorton
Co-Founder & Principal Consultant
Apache Cassandra Consulting
http://www.thelastpickle.com
On 10/12/2013, at 12:57 pm, Laing, Michael <michael.la...@nytimes.com> wrote:
> My experience is that you must upgrade to 2.0.3 ASAP to fix this.
>
> Michael
>
>
> On Mon, Dec 9, 2013 at 6:39 PM, David Laube <d...@stormpath.com> wrote:
> Hi All,
>
> We are running Cassandra 2.0.2 and have recently stumbled upon an issue with
> nodetool repair. Upon running nodetool repair on each of the 5 nodes in the
> ring (one at a time) we observe the following exceptions returned to standard
> out;
>
>
> [2013-12-08 11:04:02,047] Repair session ff16c510-5ff7-11e3-97c0-5973cc397f8f
> for range (1246984843639507027,1266616572749926276] failed with error
> org.apache.cassandra.exceptions.RepairException: [repair
> #ff16c510-5ff7-11e3-97c0-5973cc397f8f on keyspace_name/col_family1,
> (1246984843639507027,1266616572749926276]] Validation failed in /10.x.x.48
> [2013-12-08 11:04:02,063] Repair session 284c8b40-5ff8-11e3-97c0-5973cc397f8f
> for range (-109256956528331396,-89316884701275697] failed with error
> org.apache.cassandra.exceptions.RepairException: [repair
> #284c8b40-5ff8-11e3-97c0-5973cc397f8f on keyspace_name/col_family2,
> (-109256956528331396,-89316884701275697]] Validation failed in /10.x.x.103
> [2013-12-08 11:04:02,070] Repair session 399e7160-5ff8-11e3-97c0-5973cc397f8f
> for range (8901153810410866970,8915879751739915956] failed with error
> org.apache.cassandra.exceptions.RepairException: [repair
> #399e7160-5ff8-11e3-97c0-5973cc397f8f on keyspace_name/col_family1,
> (8901153810410866970,8915879751739915956]] Validation failed in /10.x.x.103
> [2013-12-08 11:04:02,072] Repair session 3ea73340-5ff8-11e3-97c0-5973cc397f8f
> for range (1149084504576970235,1190026362216198862] failed with error
> org.apache.cassandra.exceptions.RepairException: [repair
> #3ea73340-5ff8-11e3-97c0-5973cc397f8f on keyspace_name/col_family1,
> (1149084504576970235,1190026362216198862]] Validation failed in /10.x.x.103
> [2013-12-08 11:04:02,091] Repair session 6f0da460-5ff8-11e3-97c0-5973cc397f8f
> for range (-5407189524618266750,-5389231566389960750] failed with error
> org.apache.cassandra.exceptions.RepairException: [repair
> #6f0da460-5ff8-11e3-97c0-5973cc397f8f on keyspace_name/col_family1,
> (-5407189524618266750,-5389231566389960750]] Validation failed in /10.x.x.103
> [2013-12-09 23:16:36,962] Repair session 7efc2740-6127-11e3-97c0-5973cc397f8f
> for range (1246984843639507027,1266616572749926276] failed with error
> org.apache.cassandra.exceptions.RepairException: [repair
> #7efc2740-6127-11e3-97c0-5973cc397f8f on keyspace_name/col_family1,
> (1246984843639507027,1266616572749926276]] Validation failed in /10.x.x.48
> [2013-12-09 23:16:36,986] Repair session a8c44260-6127-11e3-97c0-5973cc397f8f
> for range (-109256956528331396,-89316884701275697] failed with error
> org.apache.cassandra.exceptions.RepairException: [repair
> #a8c44260-6127-11e3-97c0-5973cc397f8f on keyspace_name/col_family2,
> (-109256956528331396,-89316884701275697]] Validation failed in /10.x.x.210
>
> The /var/log/cassandra/system.log shows similar info as above with no real
> explanation as to the root cause behind the exception(s). There also does
> not appear to be any additional info in /var/log/cassandra/cassandra.log. We
> have tried restoring a recent snapshot of the keyespace in question to a
> separate staging ring and the repair runs successfully and without exception
> there. This is even after we tried insert/delete on the keyspace in the
> separate staging ring. Has anyone seen this behavior before and what can we
> do to resolve this? Any assistance would be greatly appreciated.
>
> Best regards,
> -Dave
>
