[ https://issues.apache.org/jira/browse/DIR-345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
daicheng closed DIR-345. ------------------------ Resolution: Duplicate > Could not remove renamed recovery dir(nfs) when kudu restarts > ------------------------------------------------------------- > > Key: DIR-345 > URL: https://issues.apache.org/jira/browse/DIR-345 > Project: Directory > Issue Type: Bug > Reporter: daicheng > Assignee: Emmanuel Lécharny > Priority: Major > > Configured kudu directories to NFS on k8s , and insert some data to > kudu,after restart kudu, the kudu tserver fails to bootstrap with error like > : > {code:java} > IO error: Could not remove renamed recovery dir > /var/lib/kudu/tserver/wals/1bb9b2f91c3f48d7a97fb974112dedd6.recovery-1703662995452637: > > /var/lib/kudu/tserver/wals/1bb9b2f91c3f48d7a97fb974112dedd6.recovery-1703662995452637: > One or more errors occurred {code} > while the issue didn't comes when the directory on local disk. > here some error details: > {code:java} > Config source | Replicas | Current term | Config index | > Committed? > ---------------+------------------------+--------------+--------------+------------ > master | A* B | | | Yes > A | [config not available] | | | > B | [config not available] | | | > Tablet 1bb9b2f91c3f48d7a97fb974112dedd6 of table 'impala::test.test_kudu' is > unavailable: 2 replica(s) not RUNNING > 1bf087d776394884b2031385cd7e8b82 > (kudu-tserver-0.kudu-tservers.qilu-local.svc.cluster.local:7050): not running > State: FAILED > Data state: TABLET_DATA_READY > Last status: IO error: Could not remove renamed recovery dir > /var/lib/kudu/tserver/wals/1bb9b2f91c3f48d7a97fb974112dedd6.recovery-1703663028897150: > > /var/lib/kudu/tserver/wals/1bb9b2f91c3f48d7a97fb974112dedd6.recovery-1703663028897150: > One or more errors occurred > ea0e0a381c284877aa234228ed81a24f > (kudu-tserver-1.kudu-tservers.qilu-local.svc.cluster.local:7050): not running > [LEADER] > State: FAILED > Data state: TABLET_DATA_READY > Last status: IO error: Could not remove renamed recovery dir > /var/lib/kudu/tserver/wals/1bb9b2f91c3f48d7a97fb974112dedd6.recovery-1703662995452637: > > /var/lib/kudu/tserver/wals/1bb9b2f91c3f48d7a97fb974112dedd6.recovery-1703662995452637: > One or more errors occurred{code} > {code:java} > W1227 07:43:15.222187 74 env_posix.cc:2337] Could not delete directory: IO > error: > /var/lib/kudu/tserver/wals/3b734a27abc74768ad6cff599b66f0f1.recovery-1703662995205917: > Directory not empty (error 39)Wed, Dec 27 2023 3:43:15 pmW1227 > 07:43:15.222219 74 env_posix.cc:2063] Error running callback with file > /var/lib/kudu/tserver/wals/3b734a27abc74768ad6cff599b66f0f1.recovery-1703662995205917 > during walk: IO error: > /var/lib/kudu/tserver/wals/3b734a27abc74768ad6cff599b66f0f1.recovery-1703662995205917: > Directory not empty (error 39)Wed, Dec 27 2023 3:43:15 pmE1227 > 07:43:15.261075 74 ts_tablet_manager.cc:1378] T > 3b734a27abc74768ad6cff599b66f0f1 P ea0e0a381c284877aa234228ed81a24f: Tablet > failed to bootstrap: IO error: Could not remove renamed recovery dir > /var/lib/kudu/tserver/wals/3b734a27abc74768ad6cff599b66f0f1.recovery-1703662995205917: > > /var/lib/kudu/tserver/wals/3b734a27abc74768ad6cff599b66f0f1.recovery-1703662995205917: > One or more errors occurredWed, Dec 27 2023 3:43:15 pmI1227 07:43:15.261124 > 74 ts_tablet_manager.cc:1356] T 3b734a27abc74768ad6cff599b66f0f1 P > ea0e0a381c284877aa234228ed81a24f: Time spent bootstrapping tablet: real > 0.213s user 0.070s sys 0.035sWed, Dec 27 2023 3:43:15 pmI1227 07:43:15.261147 > 74 tablet_replica.cc:323] stopping tablet replicaWed, Dec 27 2023 3:43:15 > pmI1227 07:43:15.261160 74 raft_consensus.cc:2227] T > 3b734a27abc74768ad6cff599b66f0f1 P ea0e0a381c284877aa234228ed81a24f [term 1 > FOLLOWER]: Raft consensus shutting down.Wed, Dec 27 2023 3:43:15 pmI1227 > 07:43:15.261169 74 raft_consensus.cc:2256] T 3b734a27abc74768ad6cff599b66f0f1 > P ea0e0a381c284877aa234228ed81a24f [term 1 FOLLOWER]: Raft consensus is shut > down!Wed, Dec 27 2023 3:43:15 pmI1227 07:43:15.261204 74 > tablet_bootstrap.cc:492] T 1bb9b2f91c3f48d7a97fb974112dedd6 P > ea0e0a381c284877aa234228ed81a24f: Bootstrap starting.Wed, Dec 27 2023 3:43:15 > pmI1227 07:43:15.452575 74 tablet_bootstrap.cc:492] T > 1bb9b2f91c3f48d7a97fb974112dedd6 P ea0e0a381c284877aa234228ed81a24f: > Bootstrap replayed 1/1 log segments. Stats: ops{read=4406 overwritten=0 > applied=4406 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} > orphaned_commits=0. Pending: 0 replicatesWed, Dec 27 2023 3:43:15 pmW1227 > 07:43:15.469259 74 env_posix.cc:2337] Could not delete directory: IO error: > /var/lib/kudu/tserver/wals/1bb9b2f91c3f48d7a97fb974112dedd6.recovery-1703662995452637: > Directory not empty (error 39)Wed, Dec 27 2023 3:43:15 pmW1227 > 07:43:15.469303 74 env_posix.cc:2063] Error running callback with file > /var/lib/kudu/tserver/wals/1bb9b2f91c3f48d7a97fb974112dedd6.recovery-1703662995452637 > during walk: IO error: > /var/lib/kudu/tserver/wals/1bb9b2f91c3f48d7a97fb974112dedd6.recovery-1703662995452637: > Directory not empty (error 39)Wed, Dec 27 2023 3:43:15 pmE1227 > 07:43:15.504146 74 ts_tablet_manager.cc:1378] T > 1bb9b2f91c3f48d7a97fb974112dedd6 P ea0e0a381c284877aa234228ed81a24f: Tablet > failed to bootstrap: IO error: Could not remove renamed recovery dir > /var/lib/kudu/tserver/wals/1bb9b2f91c3f48d7a97fb974112dedd6.recovery-1703662995452637: > > /var/lib/kudu/tserver/wals/1bb9b2f91c3f48d7a97fb974112dedd6.recovery-1703662995452637: > One or more errors occurredWed, Dec 27 2023 3:43:15 pmI1227 07:43:15.504194 > 74 ts_tablet_manager.cc:1356] T 1bb9b2f91c3f48d7a97fb974112dedd6 P > ea0e0a381c284877aa234228ed81a24f: Time spent bootstrapping tablet: real > 0.243s user 0.062s sys 0.046sWed, Dec 27 2023 3:43:15 pmI1227 07:43:15.504212 > 74 tablet_replica.cc:323] stopping tablet replicaWed, Dec 27 2023 3:43:15 > pmI1227 07:43:15.504217 74 raft_consensus.cc:2227] T > 1bb9b2f91c3f48d7a97fb974112dedd6 P ea0e0a381c284877aa234228ed81a24f [term 1 > FOLLOWER]: Raft consensus shutting down.Wed, Dec 27 2023 3:43:15 pmI1227 > 07:43:15.504230 74 raft_consensus.cc:2256] T 1bb9b2f91c3f48d7a97fb974112dedd6 > P ea0e0a381c284877aa234228ed81a24f [term 1 FOLLOWER]: Raft consensus is shut > down!Wed, Dec 27 2023 3:43:15 pmI1227 07:43:15.504251 74 > tablet_bootstrap.cc:492] T d7eff00a19c44c728b4d46505c1ac5f2 P > ea0e0a381c284877aa234228ed81a24f: Bootstrap starting.Wed, Dec 27 2023 3:43:15 > pmI1227 07:43:15.669176 74 tablet_bootstrap.cc:492] T > d7eff00a19c44c728b4d46505c1ac5f2 P ea0e0a381c284877aa234228ed81a24f: > Bootstrap replayed 1/1 log segments. Stats: ops{read=4975 overwritten=0 > applied=4975 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} > orphaned_commits=0. Pending: 0 replicatesWed, Dec 27 2023 3:43:15 pmW1227 > 07:43:15.687026 74 env_posix.cc:2337] Could not delete directory: IO error: > /var/lib/kudu/tserver/wals/d7eff00a19c44c728b4d46505c1ac5f2.recovery-1703662995669230: > Directory not empty (error 39)Wed, Dec 27 2023 3:43:15 pmW1227 > 07:43:15.687069 74 env_posix.cc:2063] Error running callback with file > /var/lib/kudu/tserver/wals/d7eff00a19c44c728b4d46505c1ac5f2.recovery-1703662995669230 > during walk: IO error: > /var/lib/kudu/tserver/wals/d7eff00a19c44c728b4d46505c1ac5f2.recovery-1703662995669230: > Directory not empty (error 39)Wed, Dec 27 2023 3:43:15 pmE1227 > 07:43:15.722580 74 ts_tablet_manager.cc:1378] T > d7eff00a19c44c728b4d46505c1ac5f2 P ea0e0a381c284877aa234228ed81a24f: Tablet > failed to bootstrap: IO error: Could not remove renamed recovery dir > /var/lib/kudu/tserver/wals/d7eff00a19c44c728b4d46505c1ac5f2.recovery-1703662995669230: > > /var/lib/kudu/tserver/wals/d7eff00a19c44c728b4d46505c1ac5f2.recovery-1703662995669230: > One or more errors occurredWed, Dec 27 2023 3:43:15 pmI1227 07:43:15.722630 > 74 ts_tablet_manager.cc:1356] T d7eff00a19c44c728b4d46505c1ac5f2 P > ea0e0a381c284877aa234228ed81a24f: Time spent bootstrapping tablet: real > 0.218s user 0.073s sys 0.048sWed, Dec 27 2023 3:43:15 pmI1227 07:43:15.722642 > 74 tablet_replica.cc:323] stopping tablet replicaWed, Dec 27 2023 3:43:15 > pmI1227 07:43:15.722648 74 raft_consensus.cc:2227] T > d7eff00a19c44c728b4d46505c1ac5f2 P ea0e0a381c284877aa234228ed81a24f [term 2 > FOLLOWER]: Raft consensus shutting down.Wed, Dec 27 2023 3:43:15 pmI1227 > 07:43:15.722656 74 raft_consensus.cc:2256] T d7eff00a19c44c728b4d46505c1ac5f2 > P ea0e0a381c284877aa234228ed81a24f [term 2 FOLLOWER]: Raft consensus is shut > down! {code} > > > > -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@directory.apache.org For additional commands, e-mail: dev-h...@directory.apache.org