[ https://issues.apache.org/jira/browse/CASSANDRA-14470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16505205#comment-16505205 ]
Harry Hough commented on CASSANDRA-14470: ----------------------------------------- CPU of all nodes immediately spikes to 100% so this sounds like it is this same issue. > Repair validation failed/unable to create merkle tree > ----------------------------------------------------- > > Key: CASSANDRA-14470 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14470 > Project: Cassandra > Issue Type: Bug > Reporter: Harry Hough > Priority: Major > > I had trouble repairing with a full repair across all nodes and keyspaces so > I swapped to doing table by table. This table will not repair even after > scrub/restart of all nodes. I am using command: > {code:java} > nodetool repair -full -seq keyspace table > {code} > {code:java} > [2018-05-25 19:26:36,525] Repair session 0198ee50-6050-11e8-a3b7-9d0793eab507 > for range [(165598500763544933,166800441975877433], > (-5455068259072262254,-5445777107512274819], > (-4614366950466274594,-4609359222424798148], > (3417371506258365094,3421921915575816226], > (5221788898381458942,5222846663270250559], > (3421921915575816226,3429175540277204991], > (3276484330153091115,3282213186258578546], > (-3306169730424140596,-3303439264231406101], > (5228704360821395206,5242415853745535023], > (5808045095951939338,5808562658315740708], > (-3303439264231406101,-3302592736123212969]] finished (progress: 1%) > [2018-05-25 19:27:23,848] Repair session 0180f980-6050-11e8-a3b7-9d0793eab507 > for range [(-8495158945319933291,-8482949618583319581], > (1803296697741516342,1805330812863783941], > (8633191319643427141,8637771071728131257], > (2214097236323810344,2218253238829661319], > (8637771071728131257,8639627594735133685], > (2195525904029414718,2214097236323810344], > (-8500127431270773970,-8495158945319933291], > (7151693083782264341,7152162989417914407], > (-8482949618583319581,-8481973749935314249]] finished (progress: 1%) > [2018-05-25 19:30:32,590] Repair session 01ac9d62-6050-11e8-a3b7-9d0793eab507 > for range [(7887346492105510731,7893062759268864220], > (-153277717939330979,-151986584968539220], > (-6351665356961460262,-6336288442758847669], > (7881942012672602731,7887346492105510731], > (-5884528383037906783,-5878097817437987368], > (6054625594262089428,6060773114960761336], > (-6354401100436622515,-6351665356961460262], > (3358411934943460772,3363367777663817876], > (6255644242745576360,6278718135193665575], > (-6321106762570843270,-6316788220143151823], > (1754319239259058661,1759314644652031521], > (7893062759268864220,7894890594190784729], > (-8012293411840276426,-8011781808288431224]] failed with error [repair > #01ac9d62-6050-11e8-a3b7-9d0793eab507 on keyspace/table, > [(7887346492105510731,7893062759268864220], > (-153277717939330979,-151986584968539220], > (-6351665356961460262,-6336288442758847669], > (7881942012672602731,7887346492105510731], > (-5884528383037906783,-5878097817437987368], > (6054625594262089428,6060773114960761336], > (-6354401100436622515,-6351665356961460262], > (3358411934943460772,3363367777663817876], > (6255644242745576360,6278718135193665575], > (-6321106762570843270,-6316788220143151823], > (1754319239259058661,1759314644652031521], > (7893062759268864220,7894890594190784729], > (-8012293411840276426,-8011781808288431224]]] Validation failed in > /192.168.8.64 (progress: 1%) > [2018-05-25 19:30:38,744] Repair session 01ab16c1-6050-11e8-a3b7-9d0793eab507 > for range [(4474598255414218354,4477186372547790770], > (-8368931070988054567,-8367389908801757978], > (4445104759712094068,4445123832517144036], > (6749641233379918040,6749879473217708908], > (717627050679001698,729408043324000761], > (8984622403893999385,8990662643404904110], > (4457612694557846994,4474598255414218354], > (5589049422573545528,5593079877787783784], > (3609693317839644945,3613727999875360405], > (8499016262183246473,8504603366117127178], > (-5421277973540712245,-5417725796037372830], > (5586405751301680690,5589049422573545528], > (-2611069890590917549,-2603911539353128123], > (2424772330724108233,2427564448454334730], > (3172651438220766183,3175226710613527829], > (4445123832517144036,4457612694557846994], > (-6827531712183440570,-6800863837312326365], > (5593079877787783784,5596020904874304252], > (716705770783505310,717627050679001698], > (115377252345874298,119626359210683992], > (239394377432130766,240250561347730054]] failed with error [repair > #01ab16c1-6050-11e8-a3b7-9d0793eab507 on keyspace/table, > [(4474598255414218354,4477186372547790770], > (-8368931070988054567,-8367389908801757978], > (4445104759712094068,4445123832517144036], > (6749641233379918040,6749879473217708908], > (717627050679001698,729408043324000761], > (8984622403893999385,8990662643404904110], > (4457612694557846994,4474598255414218354], > (5589049422573545528,5593079877787783784], > (3609693317839644945,3613727999875360405], > (8499016262183246473,8504603366117127178], > (-5421277973540712245,-5417725796037372830], > (5586405751301680690,5589049422573545528], > (-2611069890590917549,-2603911539353128123], > (2424772330724108233,2427564448454334730], > (3172651438220766183,3175226710613527829], > (4445123832517144036,4457612694557846994], > (-6827531712183440570,-6800863837312326365], > (5593079877787783784,5596020904874304252], > (716705770783505310,717627050679001698], > (115377252345874298,119626359210683992], > (239394377432130766,240250561347730054]]] Validation failed in > /192.168.8.63 (progress: 1%) > [2018-05-25 19:31:49,787] Repair session 01a4ae20-6050-11e8-a3b7-9d0793eab507 > for range [(-2541759376733803975,-2534654569942446346], > (5879245607426320709,5880012885546321040], > (-6369551868880447648,-6359409984081717656], > (-6599114937188060013,-6597469275333616279], > (-5074096572632539578,-5067488659471711472], > (-6379754598016153113,-6369551868880447648], > (2064405355459946002,2071996664850745669], > (-2534654569942446346,-2517719430302560572], > (7881309182913674059,7881942012672602731], > (-2544088936726049385,-2541759376733803975], > (2279496339605311864,2281121064700207175], > (7872992433920056063,7881309182913674059], > (2062114659748646544,2064405355459946002], > (-2150878401005443227,-2148033787477253835], > (-1741268532521628862,-1723492194304925672], > (-2148033787477253835,-2148008030576152684], > (2274175180327961853,2279496339605311864]] failed with error [repair > #01a4ae20-6050-11e8-a3b7-9d0793eab507 on keyspace/table, > [(-2541759376733803975,-2534654569942446346], > (5879245607426320709,5880012885546321040], > (-6369551868880447648,-6359409984081717656], > (-6599114937188060013,-6597469275333616279], > (-5074096572632539578,-5067488659471711472], > (-6379754598016153113,-6369551868880447648], > (2064405355459946002,2071996664850745669], > (-2534654569942446346,-2517719430302560572], > (7881309182913674059,7881942012672602731], > (-2544088936726049385,-2541759376733803975], > (2279496339605311864,2281121064700207175], > (7872992433920056063,7881309182913674059], > (2062114659748646544,2064405355459946002], > (-2150878401005443227,-2148033787477253835], > (-1741268532521628862,-1723492194304925672], > (-2148033787477253835,-2148008030576152684], > (2274175180327961853,2279496339605311864]]] Validation failed in > /192.168.8.64 (progress: 1%) > [2018-05-25 19:31:49,845] Repair session 01c26f52-6050-11e8-a3b7-9d0793eab507 > for range [(-6336288442758847669,-6327494039552357362], > (-6596499651894591521,-6570651311582753946], > (-6597469275333616279,-6596499651894591521], > (2057770067222008303,2062114659748646544], > (-5870054111151365631,-5835304364517776345], > (-3812151910311844467,-3802006636037441627], > (-2619800330042834297,-2615481117037091603], > (4808940926778034213,4810350864294758856], > (-7508256920307222829,-7506372018227268626], > (-7104590653728972577,-7104546570237712729], > (3158009800098518496,3172651438220766183], > (-2615481117037091603,-2611069890590917549], > (-5878097817437987368,-5870054111151365631], > (-2547658065527858190,-2544088936726049385], > (232652608016417486,239394377432130766], > (3154311195118940026,3158009800098518496]] failed with error [repair > #01c26f52-6050-11e8-a3b7-9d0793eab507 on keyspace/table, > [(-6336288442758847669,-6327494039552357362], > (-6596499651894591521,-6570651311582753946], > (-6597469275333616279,-6596499651894591521], > (2057770067222008303,2062114659748646544], > (-5870054111151365631,-5835304364517776345], > (-3812151910311844467,-3802006636037441627], > (-2619800330042834297,-2615481117037091603], > (4808940926778034213,4810350864294758856], > (-7508256920307222829,-7506372018227268626], > (-7104590653728972577,-7104546570237712729], > (3158009800098518496,3172651438220766183], > (-2615481117037091603,-2611069890590917549], > (-5878097817437987368,-5870054111151365631], > (-2547658065527858190,-2544088936726049385], > (232652608016417486,239394377432130766], > (3154311195118940026,3158009800098518496]]] Validation failed in > /192.168.10.63 (progress: 1%) > [2018-05-25 19:31:50,027] Repair session 01b3f061-6050-11e8-a3b7-9d0793eab507 > for range [(2424051311739332070,2424772330724108233], > (6848066208555197,10521229928033262], > (992385332284940308,1000066900542109637], > (4418797036920007266,4421783585221695744], > (-5417725796037372830,-5412149532100548404], > (178766242164281045,191217736969025363], > (-3802006636037441627,-3796416071827586080], > (5683533739750457455,5688298632819249302], > (3653327414143088744,3655860906328373441], > (3655860906328373441,3657219071532471378], > (5746716543928841040,5753897313199191356], > (-7506372018227268626,-7477180353912675682], > (1911795960615895165,1921474545637686707], > (4421783585221695744,4445104759712094068], > (-4428987737460108139,-4413904067417968038], > (5680321325075541449,5683533739750457455]] failed with error [repair > #01b3f061-6050-11e8-a3b7-9d0793eab507 on keyspace/table, > [(2424051311739332070,2424772330724108233], > (6848066208555197,10521229928033262], > (992385332284940308,1000066900542109637], > (4418797036920007266,4421783585221695744], > (-5417725796037372830,-5412149532100548404], > (178766242164281045,191217736969025363], > (-3802006636037441627,-3796416071827586080], > (5683533739750457455,5688298632819249302], > (3653327414143088744,3655860906328373441], > (3655860906328373441,3657219071532471378], > (5746716543928841040,5753897313199191356], > (-7506372018227268626,-7477180353912675682], > (1911795960615895165,1921474545637686707], > (4421783585221695744,4445104759712094068], > (-4428987737460108139,-4413904067417968038], > (5680321325075541449,5683533739750457455]]] Validation failed in > /192.168.10.63 (progress: 1%) > [2018-05-25 19:31:50,065] Repair session 01d226c2-6050-11e8-a3b7-9d0793eab507 > for range [(731483217573828589,749016052425471844], > (3349217091766639630,3355743728768043539], > (8297509817744988677,8299811671851037140], > (-1080064213437365415,-1067683134584617984], > (-8988387420898594746,-8988256206650322851], > (-1083473978088553649,-1080064213437365415], > (-7068314886788869981,-7062826172876507507], > (8299811671851037140,8306379796303668520], > (-8500393685425499630,-8500127431270773970], > (9077374236600850244,9080101637323836166], > (9080101637323836166,9095536755598180114], > (-2759657072078827823,-2750629632199441038], > (-7938459356954944009,-7933123149264580832], > (1759642905348136701,1772996641768793656], > (-2788441126655538224,-2774970527117004032], > (-7070810217579746608,-7068314886788869981], > (-7959560447639828128,-7938459356954944009], > (-7679921498492428955,-7664015662435807775]] failed with error [repair > #01d226c2-6050-11e8-a3b7-9d0793eab507 on keyspace/table, > [(731483217573828589,749016052425471844], > (3349217091766639630,3355743728768043539], > (8297509817744988677,8299811671851037140], > (-1080064213437365415,-1067683134584617984], > (-8988387420898594746,-8988256206650322851], > (-1083473978088553649,-1080064213437365415], > (-7068314886788869981,-7062826172876507507], > (8299811671851037140,8306379796303668520], > (-8500393685425499630,-8500127431270773970], > (9077374236600850244,9080101637323836166], > (9080101637323836166,9095536755598180114], > (-2759657072078827823,-2750629632199441038], > (-7938459356954944009,-7933123149264580832], > (1759642905348136701,1772996641768793656], > (-2788441126655538224,-2774970527117004032], > (-7070810217579746608,-7068314886788869981], > (-7959560447639828128,-7938459356954944009], > (-7679921498492428955,-7664015662435807775]]] Validation failed in > /192.168.8.63 (progress: 2%) > [2018-05-25 19:32:24,797] Repair session 01aff8c0-6050-11e8-a3b7-9d0793eab507 > for range [(119626359210683992,128454334208965433], > (6169854579148936152,6189260921105966960], > (8460580156771389602,8466680988634247357], > (10521229928033262,11278848941988721], > (6165215300562655515,6169854579148936152], > (191217736969025363,212964375650430729], > (-5297146550802223153,-5294434130239676253], > (6189260921105966960,6193074220809370652], > (-655425716305023073,-647730635946823030]] failed with error [repair > #01aff8c0-6050-11e8-a3b7-9d0793eab507 on keyspace/table, > [(119626359210683992,128454334208965433], > (6169854579148936152,6189260921105966960], > (8460580156771389602,8466680988634247357], > (10521229928033262,11278848941988721], > (6165215300562655515,6169854579148936152], > (191217736969025363,212964375650430729], > (-5297146550802223153,-5294434130239676253], > (6189260921105966960,6193074220809370652], > (-655425716305023073,-647730635946823030]]] Validation failed in > /192.168.10.63 (progress: 2%) > [2018-05-25 19:32:24,873] Repair session 0199d8b1-6050-11e8-a3b7-9d0793eab507 > for range [(2708724319719658573,2710986923384204956], > (6278718135193665575,6281813004301666161], > (-8025315476660819134,-8015410683496661099], > (2516704840921371424,2519633614752918103], > (2519633614752918103,2526922953145276348], > (8641102301927501454,8641256970223193109], > (8643632109719583963,8645181823655307237], > (-8015410683496661099,-8012293411840276426], > (1368548173174048881,1373330457443776421], > (5550121777767121,6848066208555197], > (8641256970223193109,8643632109719583963], > (-4201893423037098789,-4196287665648271477], > (2692054381245703566,2708724319719658573], > (-4208139091663389178,-4201893423037098789], > (6281813004301666161,6282606461503930756], > (-3470325001213070915,-3465759276556337455], > (-4196287665648271477,-4185162268982289501], > (-5006305410789315624,-5000646423000423501], > (2714363942918413158,2722577239100121227], > (5692402142504566885,5693342630493279303], > (2710986923384204956,2714363942918413158], > (5688298632819249302,5692402142504566885]] failed with error [repair > #0199d8b1-6050-11e8-a3b7-9d0793eab507 on keyspace/table, > [(2708724319719658573,2710986923384204956], > (6278718135193665575,6281813004301666161], > (-8025315476660819134,-8015410683496661099], > (2516704840921371424,2519633614752918103], > (2519633614752918103,2526922953145276348], > (8641102301927501454,8641256970223193109], > (8643632109719583963,8645181823655307237], > (-8015410683496661099,-8012293411840276426], > (1368548173174048881,1373330457443776421], > (5550121777767121,6848066208555197], > (8641256970223193109,8643632109719583963], > (-4201893423037098789,-4196287665648271477], > (2692054381245703566,2708724319719658573], > (-4208139091663389178,-4201893423037098789], > (6281813004301666161,6282606461503930756], > (-3470325001213070915,-3465759276556337455], > (-4196287665648271477,-4185162268982289501], > (-5006305410789315624,-5000646423000423501], > (2714363942918413158,2722577239100121227], > (5692402142504566885,5693342630493279303], > (2710986923384204956,2714363942918413158], > (5688298632819249302,5692402142504566885]]] Validation failed in > /192.168.8.65 (progress: 2%) > Exception occurred during clean-up. > java.lang.reflect.UndeclaredThrowableException > Cassandra has shutdown. > error: [2018-05-25 19:36:47,652] JMX connection closed. You should check > server log for repair status of keyspace keyspace(Subsequent keyspaces are > not going to be repaired). > -- StackTrace -- > May 25, 2018 7:36:47 PM ClientCommunicatorAdmin Checker-run > WARNING: Failed to check connection: java.net.SocketException: Connection > reset > java.io.IOException: [2018-05-25 19:36:47,652] JMX connection closed. You > should check server log for repair status of keyspace keyspace(Subsequent > keyspaces are not going to be repaired). > at > org.apache.cassandra.tools.RepairRunner.handleConnectionFailed(RepairRunner.java:98) > at > org.apache.cassandra.utils.progress.jmx.JMXNotificationProgressListener.handleNotification(JMXNotificationProgressListener.java:86) > at > javax.management.NotificationBroadcasterSupport.handleNotification(NotificationBroadcasterSupport.java:275) > at > javax.management.NotificationBroadcasterSupport$SendNotifJob.run(NotificationBroadcasterSupport.java:352) > at > javax.management.NotificationBroadcasterSupport$1.execute(NotificationBroadcasterSupport.java:337) > at > javax.management.NotificationBroadcasterSupport.sendNotification(NotificationBroadcasterSupport.java:248) > at > javax.management.remote.rmi.RMIConnector.sendNotification(RMIConnector.java:441) > at > javax.management.remote.rmi.RMIConnector.access$1200(RMIConnector.java:121) > at > javax.management.remote.rmi.RMIConnector$RMIClientCommunicatorAdmin.gotIOException(RMIConnector.java:1531) > at > com.sun.jmx.remote.internal.ClientCommunicatorAdmin$Checker.run(ClientCommunicatorAdmin.java:199) > at java.lang.Thread.run(Thread.java:748) > May 25, 2018 7:36:47 PM ClientCommunicatorAdmin Checker-run > WARNING: stopping > {code} > Here is the log on one of the nodes where validation fails. > {code:java} > INFO [AntiEntropyStage:1] 2018-05-25 19:23:10,548 Validator.java:281 - > [repair #01cf67a1-6050-11e8-a3b7-9d0793eab507] Sending completed merkle tree > to /192.168.10.65 for pr$ > INFO [AntiEntropyStage:1] 2018-05-25 19:26:17,161 Validator.java:281 - > [repair #01828020-6050-11e8-a3b7-9d0793eab507] Sending completed merkle tree > to /192.168.10.65 for pr$ > INFO [AntiEntropyStage:1] 2018-05-25 19:26:23,909 Validator.java:281 - > [repair #019dd051-6050-11e8-a3b7-9d0793eab507] Sending completed merkle tree > to /192.168.10.65 for pr$ > INFO [AntiEntropyStage:1] 2018-05-25 19:28:15,118 Validator.java:281 - > [repair #01c52e71-6050-11e8-a3b7-9d0793eab507] Sending completed merkle tree > to /192.168.10.65 for pr$ > INFO [GossipTasks:1] 2018-05-25 19:30:23,087 Gossiper.java:1034 - > InetAddress /192.168.10.65 is now DOWN > INFO [HANDSHAKE-/192.168.10.65] 2018-05-25 19:30:31,093 > OutboundTcpConnection.java:560 - Handshaking version with /192.168.10.65 > INFO [HANDSHAKE-/192.168.10.65] 2018-05-25 19:30:31,281 > OutboundTcpConnection.java:560 - Handshaking version with /192.168.10.65 > INFO [RequestResponseStage-4] 2018-05-25 19:30:31,320 Gossiper.java:1019 - > InetAddress /192.168.10.65 is now UP > INFO [RequestResponseStage-3] 2018-05-25 19:30:31,320 Gossiper.java:1019 - > InetAddress /192.168.10.65 is now UP > INFO [RequestResponseStage-2] 2018-05-25 19:30:31,320 Gossiper.java:1019 - > InetAddress /192.168.10.65 is now UP > INFO [RequestResponseStage-1] 2018-05-25 19:30:31,320 Gossiper.java:1019 - > InetAddress /192.168.10.65 is now UP > INFO [RequestResponseStage-5] 2018-05-25 19:30:31,320 Gossiper.java:1019 - > InetAddress /192.168.10.65 is now UP > INFO [AntiEntropyStage:1] 2018-05-25 19:30:49,172 Validator.java:281 - > [repair #01860291-6050-11e8-a3b7-9d0793eab507] Sending completed merkle tree > to /192.168.10.65 for pr$ > INFO [HANDSHAKE-/192.168.10.65] 2018-05-25 19:30:49,188 > OutboundTcpConnection.java:560 - Handshaking version with /192.168.10.65 > INFO [HANDSHAKE-/192.168.10.65] 2018-05-25 19:30:54,188 > OutboundTcpConnection.java:569 - Cannot handshake version with /192.168.10.65 > INFO [HANDSHAKE-/192.168.10.65] 2018-05-25 19:30:54,188 > OutboundTcpConnection.java:560 - Handshaking version with /192.168.10.65 > INFO [HANDSHAKE-/192.168.10.65] 2018-05-25 19:30:59,188 > OutboundTcpConnection.java:569 - Cannot handshake version with /192.168.10.65 > INFO [GossipTasks:1] 2018-05-25 19:31:03,247 Gossiper.java:1034 - > InetAddress /192.168.10.65 is now DOWN > INFO [HANDSHAKE-/192.168.10.65] 2018-05-25 19:31:10,250 > OutboundTcpConnection.java:560 - Handshaking version with /192.168.10.65 > INFO [HANDSHAKE-/192.168.10.65] 2018-05-25 19:31:12,237 > OutboundTcpConnection.java:560 - Handshaking version with /192.168.10.65 > INFO [RequestResponseStage-7] 2018-05-25 19:31:12,712 Gossiper.java:1019 - > InetAddress /192.168.10.65 is now UP > INFO [RequestResponseStage-9] 2018-05-25 19:31:12,712 Gossiper.java:1019 - > InetAddress /192.168.10.65 is now UP > INFO [RequestResponseStage-13] 2018-05-25 19:31:12,712 Gossiper.java:1019 - > InetAddress /192.168.10.65 is now UP > INFO [GossipTasks:1] 2018-05-25 19:31:37,252 Gossiper.java:1034 - > InetAddress /192.168.10.65 is now DOWN > INFO [HANDSHAKE-/192.168.10.65] 2018-05-25 19:31:45,254 > OutboundTcpConnection.java:560 - Handshaking version with /192.168.10.65 > INFO [HANDSHAKE-/192.168.10.65] 2018-05-25 19:31:48,759 > OutboundTcpConnection.java:560 - Handshaking version with /192.168.10.65 > ERROR [ValidationExecutor:7] 2018-05-25 19:31:49,021 Validator.java:268 - > Failed creating a merkle tree for [repair > #01c26f52-6050-11e8-a3b7-9d0793eab507 on keyspace/$ > ERROR [ValidationExecutor:7] 2018-05-25 19:31:49,022 CassandraDaemon.java:228 > - Exception in thread > Thread[ValidationExecutor:7,1,main]java.lang.RuntimeException: Parent repair > session with id = 0103da40-6050-11e8-a3b7-9d0793eab507 has failed. at > org.apache.cassandra.service.ActiveRepairService.getParentRepairSession(ActiveRepairService.java:412) > ~[apache-cassandra-3.11.2.jar:3.11.2] at > org.apache.cassandra.db.compaction.CompactionManager.getSSTablesToValidate(CompactionManager.java:1459) > ~[apache-cassandra-3.11.2.jar:3.11.2] at > org.apache.cassandra.db.compaction.CompactionManager.doValidationCompaction(CompactionManager.java:1366) > ~[apache-cassandra-3.11.2.jar:3.11.2] at > org.apache.cassandra.db.compaction.CompactionManager.access$600(CompactionManager.java:86) > ~[apache-cassandra-3.11.2.jar:3.11.2] at > org.apache.cassandra.db.compaction.CompactionManager$13.call(CompactionManager.java:955) > ~[apache-cassandra-3.11.2.jar:3.11.2] at > java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_171] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > ~[na:1.8.0_171] at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > [na:1.8.0_171] at > org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81) > [apache-cassandra-3.11.2.jar:3.11.2] at > java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_171 > ]INFO [RequestResponseStage-2] 2018-05-25 19:31:49,025 Gossiper.java:1019 - > InetAddress /192.168.10.65 is now UP > INFO [RequestResponseStage-3] 2018-05-25 19:31:49,025 Gossiper.java:1019 - > InetAddress /192.168.10.65 is now UP > INFO [RequestResponseStage-1] 2018-05-25 19:31:49,039 Gossiper.java:1019 - > InetAddress /192.168.10.65 is now UP > ERROR [ValidationExecutor:7] 2018-05-25 19:31:49,817 Validator.java:268 - > Failed creating a merkle tree for [repair > #01b3f061-6050-11e8-a3b7-9d0793eab507 on keyspace/$ > ERROR [ValidationExecutor:7] 2018-05-25 19:31:49,817 CassandraDaemon.java:228 > - Exception in thread > Thread[ValidationExecutor:7,1,main]java.lang.RuntimeException: Parent repair > session with id = 0103da40-6050-11e8-a3b7-9d0793eab507 has failed. at > org.apache.cassandra.service.ActiveRepairService.getParentRepairSession(ActiveRepairService.java:412) > ~[apache-cassandra-3.11.2.jar:3.11.2] at > org.apache.cassandra.db.compaction.CompactionManager.getSSTablesToValidate(CompactionManager.java:1459) > ~[apache-cassandra-3.11.2.jar:3.11.2] at > org.apache.cassandra.db.compaction.CompactionManager.doValidationCompaction(CompactionManager.java:1366) > ~[apache-cassandra-3.11.2.jar:3.11.2] at > org.apache.cassandra.db.compaction.CompactionManager.access$600(CompactionManager.java:86) > ~[apache-cassandra-3.11.2.jar:3.11.2] at > org.apache.cassandra.db.compaction.CompactionManager$13.call(CompactionManager.java:955) > ~[apache-cassandra-3.11.2.jar:3.11.2] at > java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_171] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > ~[na:1.8.0_171] at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > [na:1.8.0_171] at > org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81) > [apache-cassandra-3.11.2.jar:3.11.2] at > java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_171] > {code} > 192.168.10.65 is the node where I started the repair. It looks like this node > goes down before the merkle tree creation failure occurs? The debug log on > the repair node is full of the below and doesn't help me much. > {code:java} > DEBUG [RepairJobTask:11] 2018-05-25 19:25:49,646 MerkleTree.java:295 - (17) > Hashing sub-ranges [#<TreeRange (8732300281801533308,8732300339552037321] > depth=18>, #<TreeRange (8732300339552037321,8732300397302541334] depth=18>] > for #<TreeRange (8732300281801533308,8732300397302541334] depth=17> divided > by midpoint 8732300339552037321 > DEBUG [RepairJobTask:11] 2018-05-25 19:25:49,647 MerkleTree.java:311 - (17) > Inconsistent digest on left sub-range #<TreeRange > (8732300281801533308,8732300339552037321] depth=18>: [#<Leaf > [16cd9a47184232c7ed028a9d4546332d8a8c8ff83526f8f274997592eecc722d]>, #<Leaf > [97b8f2fd61c1130ed2cfd8c2db52afdea59b2cabb5f39aed967cf8f2539f08b8]>] > DEBUG [RepairJobTask:11] 2018-05-25 19:25:49,647 MerkleTree.java:333 - (17) > Inconsistent digest on right sub-range #<TreeRange > (8732300339552037321,8732300397302541334] depth=18>: [#<Leaf > [fcf6daa5b5124a1e099e7776475aff22f2befedd88dc7b3e4277b92fd3115833]>, #<Leaf > []>] > DEBUG [RepairJobTask:11] 2018-05-25 19:25:49,647 MerkleTree.java:346 - (17) > Fully inconsistent range [#<TreeRange > (8732300281801533308,8732300339552037321] depth=18>, #<TreeRange > (8732300339552037321,8732300397302541334] depth=18>] > DEBUG [RepairJobTask:11] 2018-05-25 19:25:49,647 MerkleTree.java:346 - (16) > Fully inconsistent range [#<TreeRange > (8732300166300525283,8732300281801533308] depth=17>, #<TreeRange > (8732300281801533308,8732300397302541334] depth=17>] > DEBUG [RepairJobTask:11] 2018-05-25 19:25:49,647 MerkleTree.java:346 - (15) > Fully inconsistent range [#<TreeRange > (8732299935298509232,8732300166300525283] depth=16>, #<TreeRange > (8732300166300525283,8732300397302541334] depth=16>] > DEBUG [RepairJobTask:11] 2018-05-25 19:25:49,647 MerkleTree.java:346 - (14) > Fully inconsistent range [#<TreeRange > (8732299473294477131,8732299935298509232] depth=15>, #<TreeRange > (8732299935298509232,8732300397302541334] depth=15>] > DEBUG [RepairJobTask:11] 2018-05-25 19:25:49,647 MerkleTree.java:346 - (13) > Fully inconsistent range [#<TreeRange > (8732298549286412929,8732299473294477131] depth=14>, #<TreeRange > (8732299473294477131,8732300397302541334] depth=14>] > {code} > Really at a loss of how to repair this table at this point. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org