keith-turner commented on issue #2667: URL: https://github.com/apache/accumulo/issues/2667#issuecomment-1271312449
I was able to reproduce this yesterday and the situation seemed similar to the one above. It seems like the same situation that was speculated as a cause in an earlier comment, a compaction of an empty tablet followed by a close. Saw the following in the tserver logs. ``` 2022-10-06T19:41:09,701 [tablet.location] DEBUG: Loading 3s<< on localhost:9997[1000325bae30004] 2022-10-06T19:41:09,728 [tablet.location] DEBUG: Loaded 3s<< on localhost:9997[1000325bae30004] 2022-10-06T19:41:09,830 [tablet.Tablet] DEBUG: Tablet 3s<< had no dir, creating hdfs://localhost:8020/accumulo/tables/3s/default_tablet 2022-10-06T19:41:09,837 [tablet.Tablet] ERROR: Closed tablet 3s<< lastCompactID is inconsistent with metadata : 2 != -1 2022-10-06T19:41:09,839 [tablet.Tablet] ERROR: Failed to do close consistency check for tablet 3s<< java.lang.RuntimeException: Closed tablet 3s<< lastCompactID is inconsistent with metadata : 2 != -1 java.lang.RuntimeException: Failed to do close consistency check for tablet 3s<< ``` Saw the following the manager logs ``` 2022-10-06T19:41:09,520 [security.SecurityOperation] INFO : Granted table permission READ for user root on the table 3s at the request of user !SYSTEM 2022-10-06T19:41:09,523 [security.SecurityOperation] INFO : Granted table permission WRITE for user root on the table 3s at the request of user !SYSTEM 2022-10-06T19:41:09,526 [security.SecurityOperation] INFO : Granted table permission BULK_IMPORT for user root on the table 3s at the request of user !SYSTEM 2022-10-06T19:41:09,528 [security.SecurityOperation] INFO : Granted table permission ALTER_TABLE for user root on the table 3s at the request of user !SYSTEM 2022-10-06T19:41:09,530 [security.SecurityOperation] INFO : Granted table permission GRANT for user root on the table 3s at the request of user !SYSTEM 2022-10-06T19:41:09,533 [security.SecurityOperation] INFO : Granted table permission DROP_TABLE for user root on the table 3s at the request of user !SYSTEM 2022-10-06T19:41:09,537 [security.SecurityOperation] INFO : Granted table permission GET_SUMMARIES for user root on the table 3s at the request of user !SYSTEM 2022-10-06T19:41:09,550 [tableOps.Utils] INFO : table 3s FATE[3d5c0cbb1211e4a9] locked for write operation: CREATE 2022-10-06T19:41:09,550 [tables.TableManager] DEBUG: Creating ZooKeeper entries for new table bulk_dev_uk_2030907_1665085269455 (ID: 3s) in namespace (ID: +default) 2022-10-06T19:41:09,560 [manager.EventCoordinator] INFO : Table state in zookeeper changed for 3s to NEW 2022-10-06T19:41:09,581 [balancer.TableLoadBalancer] INFO : Loaded class org.apache.accumulo.core.spi.balancer.SimpleLoadBalancer for table 3s 2022-10-06T19:41:09,603 [tables.TableManager] DEBUG: Transitioning state for table 3s from NEW to ONLINE 2022-10-06T19:41:09,604 [manager.EventCoordinator] INFO : Table state in zookeeper changed for 3s to ONLINE 2022-10-06T19:41:09,613 [tableOps.Utils] INFO : table 3s FATE[3d5c0cbb1211e4a9] unlocked for write 2022-10-06T19:41:09,680 [accumulo.audit] INFO : operation: permitted; user: root; client: 127.0.0.1:53720; action: compactTable; targetTable: 3s; targetNamespace: +default; 2022-10-06T19:41:09,685 [accumulo.audit] INFO : operation: permitted; user: root; client: 127.0.0.1:53712; action: compactTable; targetTable: 3s; targetNamespace: +default; 2022-10-06T19:41:09,700 [tablet.location] DEBUG: Assigned 3s<< to localhost:9997[1000325bae30004] 2022-10-06T19:41:09,730 [manager.EventCoordinator] INFO : tablet 3s<< was loaded on localhost:9997 2022-10-06T19:41:09,731 [tableOps.Utils] INFO : table 3s FATE[68a7eab3013b5aaa] locked for read operation: COMPACT 2022-10-06T19:41:09,751 [tableOps.Utils] INFO : table 3s FATE[71a8f902cf66da2a] locked for read operation: COMPACT 2022-10-06T19:41:11,854 [schema.LinkingIterator] DEBUG: First tablet seen provides evidence of earlier tablet in range, retrying 3s;r0e773 [3s; : [] 9223372036854775807 false,3s<%00; : [] 9223372036854775807 false) 2022-10-06T19:41:12,124 [manager.EventCoordinator] INFO : localhost:9997 reported split 3s;r0e773<, 3s<;r0e773 2022-10-06T19:41:13,897 [schema.LinkingIterator] DEBUG: First tablet seen provides evidence of earlier tablet in range, retrying 3s;r057b6 [3s; : [] 9223372036854775807 false,3s<%00; : [] 9223372036854775807 false) 2022-10-06T19:41:14,123 [manager.EventCoordinator] INFO : localhost:9997 reported split 3s;r057b6<, 3s;r0e773;r057b6 2022-10-06T19:41:14,905 [manager.EventCoordinator] INFO : localhost:9997 reported split 3s;r11bf7;r0e773, 3s<;r11bf7 2022-10-06T19:41:16,423 [manager.EventCoordinator] INFO : localhost:9997 reported split 3s;r0af73;r057b6, 3s;r0e773;r0af73 2022-10-06T19:41:16,902 [schema.LinkingIterator] DEBUG: Tablets end row and prev end row not equals 3s;r11bf7;r0e773 3s<;r1391e 2022-10-06T19:41:17,425 [manager.EventCoordinator] INFO : localhost:9997 reported split 3s;r1391e;r11bf7, 3s<;r1391e 2022-10-06T19:41:18,168 [accumulo.audit] INFO : operation: permitted; user: root; client: 127.0.0.1:53712; action: mergeTable; targetTable: 3s; targetNamespace: +default; 2022-10-06T19:41:18,168 [manager.Manager] DEBUG: Creating merge op: 3s from startRow: -inf to endRow: +inf 2022-10-06T19:41:18,237 [tableOps.Utils] INFO : table 3s FATE[57d4471744875c60] locked for write operation: MERGE 2022-10-06T19:41:18,242 [manager.EventCoordinator] INFO : Merge state of 3s<< set to STARTED 2022-10-06T19:41:18,382 [manager.Manager] DEBUG: mergeInfo overlaps: 3s;r057b6< true ``` I added some logging to the tserver that will help verify the hypothesis and kicked the test off again. Didn't see the problem again, but did see #3006. Also noticed some of the bulk import random walk test are stuck this morning i have not had a chance to track that down. Going to keep trying to reproduce the problem with improved logging. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
