[jira] [Commented] (STORM-682) Supervisor local worker state corrupted and failing to start.
[ https://issues.apache.org/jira/browse/STORM-682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14329449#comment-14329449 ] ASF GitHub Bot commented on STORM-682: -- GitHub user Parth-Brahmbhatt opened a pull request: https://github.com/apache/storm/pull/437 STORM-682: supervisor should handle worker state corruption gracefully. You can merge this pull request into a Git repository by running: $ git pull https://github.com/Parth-Brahmbhatt/incubator-storm STORM-682 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/437.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #437 commit afd8f81ba2650423184be3fcf6e00dd7c558acbe Author: Parth Brahmbhatt Date: 2015-02-20T19:56:22Z STORM-682: supervisor should handle worker state corruption gracefully. > Supervisor local worker state corrupted and failing to start. > - > > Key: STORM-682 > URL: https://issues.apache.org/jira/browse/STORM-682 > Project: Apache Storm > Issue Type: Bug >Reporter: Parth Brahmbhatt >Assignee: Parth Brahmbhatt > > If supervisor's cleanup of a worker fails to delete some heartbeat files the > local state of the supervisors get corrupted.The only way to recover the > supervisor from this state is to delete the local state folder where > supervisor stores all worker information.This fix can get very cumbersome if > it happens on multiple worker nodes. > The root cause of the issue is the order in which worker heartbeat versioned > store files are created vs the deletion order of those files. LocalState.put > first creates a data file X and then marks a success by creating a file > X.version. During get it first checks for all *.version files , tries to > find the largest value of X and then issues a read against X. See the below > pseudo code > {code:java} > start_supervisor() { > workerIds = `ls local-state/workers` > for each workerId in workerIds > versions = `ls local-state/workers/workerId/heartbeats/*.version` > latest_version = max(versions) > read local-state/workers/workerId/heartbeats/latest_version [Note there > is no .version extension] > } > {code} > During cleanup it first tries to delete file X and then X.version. If X gets > deleted but X.version fails to delete the supervisor fails to start with > FileNotFoundException in the code above. > We propose to change the deletion order so the .version files get deleted > before the data file and catch any IOException when reading worker heartbeats > to avoid supervisor failure. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-682) Supervisor local worker state corrupted and failing to start.
[ https://issues.apache.org/jira/browse/STORM-682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14329997#comment-14329997 ] ASF GitHub Bot commented on STORM-682: -- Github user nathanmarz commented on the pull request: https://github.com/apache/storm/pull/437#issuecomment-75356387 -1 Please change the catch block to explicitly return nil. Don't depend on log-warn to do that for you. > Supervisor local worker state corrupted and failing to start. > - > > Key: STORM-682 > URL: https://issues.apache.org/jira/browse/STORM-682 > Project: Apache Storm > Issue Type: Bug >Reporter: Parth Brahmbhatt >Assignee: Parth Brahmbhatt > > If supervisor's cleanup of a worker fails to delete some heartbeat files the > local state of the supervisors get corrupted.The only way to recover the > supervisor from this state is to delete the local state folder where > supervisor stores all worker information.This fix can get very cumbersome if > it happens on multiple worker nodes. > The root cause of the issue is the order in which worker heartbeat versioned > store files are created vs the deletion order of those files. LocalState.put > first creates a data file X and then marks a success by creating a file > X.version. During get it first checks for all *.version files , tries to > find the largest value of X and then issues a read against X. See the below > pseudo code > {code:java} > start_supervisor() { > workerIds = `ls local-state/workers` > for each workerId in workerIds > versions = `ls local-state/workers/workerId/heartbeats/*.version` > latest_version = max(versions) > read local-state/workers/workerId/heartbeats/latest_version [Note there > is no .version extension] > } > {code} > During cleanup it first tries to delete file X and then X.version. If X gets > deleted but X.version fails to delete the supervisor fails to start with > FileNotFoundException in the code above. > We propose to change the deletion order so the .version files get deleted > before the data file and catch any IOException when reading worker heartbeats > to avoid supervisor failure. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-682) Supervisor local worker state corrupted and failing to start.
[ https://issues.apache.org/jira/browse/STORM-682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14333603#comment-14333603 ] ASF GitHub Bot commented on STORM-682: -- Github user Parth-Brahmbhatt commented on the pull request: https://github.com/apache/storm/pull/437#issuecomment-75597555 @nathanmarz done. > Supervisor local worker state corrupted and failing to start. > - > > Key: STORM-682 > URL: https://issues.apache.org/jira/browse/STORM-682 > Project: Apache Storm > Issue Type: Bug >Reporter: Parth Brahmbhatt >Assignee: Parth Brahmbhatt > > If supervisor's cleanup of a worker fails to delete some heartbeat files the > local state of the supervisors get corrupted.The only way to recover the > supervisor from this state is to delete the local state folder where > supervisor stores all worker information.This fix can get very cumbersome if > it happens on multiple worker nodes. > The root cause of the issue is the order in which worker heartbeat versioned > store files are created vs the deletion order of those files. LocalState.put > first creates a data file X and then marks a success by creating a file > X.version. During get it first checks for all *.version files , tries to > find the largest value of X and then issues a read against X. See the below > pseudo code > {code:java} > start_supervisor() { > workerIds = `ls local-state/workers` > for each workerId in workerIds > versions = `ls local-state/workers/workerId/heartbeats/*.version` > latest_version = max(versions) > read local-state/workers/workerId/heartbeats/latest_version [Note there > is no .version extension] > } > {code} > During cleanup it first tries to delete file X and then X.version. If X gets > deleted but X.version fails to delete the supervisor fails to start with > FileNotFoundException in the code above. > We propose to change the deletion order so the .version files get deleted > before the data file and catch any IOException when reading worker heartbeats > to avoid supervisor failure. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-682) Supervisor local worker state corrupted and failing to start.
[ https://issues.apache.org/jira/browse/STORM-682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14343386#comment-14343386 ] ASF GitHub Bot commented on STORM-682: -- Github user kishorvpatil commented on the pull request: https://github.com/apache/storm/pull/437#issuecomment-76751433 Looks good. +1 > Supervisor local worker state corrupted and failing to start. > - > > Key: STORM-682 > URL: https://issues.apache.org/jira/browse/STORM-682 > Project: Apache Storm > Issue Type: Bug >Reporter: Parth Brahmbhatt >Assignee: Parth Brahmbhatt > > If supervisor's cleanup of a worker fails to delete some heartbeat files the > local state of the supervisors get corrupted.The only way to recover the > supervisor from this state is to delete the local state folder where > supervisor stores all worker information.This fix can get very cumbersome if > it happens on multiple worker nodes. > The root cause of the issue is the order in which worker heartbeat versioned > store files are created vs the deletion order of those files. LocalState.put > first creates a data file X and then marks a success by creating a file > X.version. During get it first checks for all *.version files , tries to > find the largest value of X and then issues a read against X. See the below > pseudo code > {code:java} > start_supervisor() { > workerIds = `ls local-state/workers` > for each workerId in workerIds > versions = `ls local-state/workers/workerId/heartbeats/*.version` > latest_version = max(versions) > read local-state/workers/workerId/heartbeats/latest_version [Note there > is no .version extension] > } > {code} > During cleanup it first tries to delete file X and then X.version. If X gets > deleted but X.version fails to delete the supervisor fails to start with > FileNotFoundException in the code above. > We propose to change the deletion order so the .version files get deleted > before the data file and catch any IOException when reading worker heartbeats > to avoid supervisor failure. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-682) Supervisor local worker state corrupted and failing to start.
[ https://issues.apache.org/jira/browse/STORM-682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14344522#comment-14344522 ] ASF GitHub Bot commented on STORM-682: -- Github user harshach commented on the pull request: https://github.com/apache/storm/pull/437#issuecomment-76887513 +1. @nathanmarz can you review the latest patch. Thanks. > Supervisor local worker state corrupted and failing to start. > - > > Key: STORM-682 > URL: https://issues.apache.org/jira/browse/STORM-682 > Project: Apache Storm > Issue Type: Bug >Reporter: Parth Brahmbhatt >Assignee: Parth Brahmbhatt > > If supervisor's cleanup of a worker fails to delete some heartbeat files the > local state of the supervisors get corrupted.The only way to recover the > supervisor from this state is to delete the local state folder where > supervisor stores all worker information.This fix can get very cumbersome if > it happens on multiple worker nodes. > The root cause of the issue is the order in which worker heartbeat versioned > store files are created vs the deletion order of those files. LocalState.put > first creates a data file X and then marks a success by creating a file > X.version. During get it first checks for all *.version files , tries to > find the largest value of X and then issues a read against X. See the below > pseudo code > {code:java} > start_supervisor() { > workerIds = `ls local-state/workers` > for each workerId in workerIds > versions = `ls local-state/workers/workerId/heartbeats/*.version` > latest_version = max(versions) > read local-state/workers/workerId/heartbeats/latest_version [Note there > is no .version extension] > } > {code} > During cleanup it first tries to delete file X and then X.version. If X gets > deleted but X.version fails to delete the supervisor fails to start with > FileNotFoundException in the code above. > We propose to change the deletion order so the .version files get deleted > before the data file and catch any IOException when reading worker heartbeats > to avoid supervisor failure. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-682) Supervisor local worker state corrupted and failing to start.
[ https://issues.apache.org/jira/browse/STORM-682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14349012#comment-14349012 ] ASF GitHub Bot commented on STORM-682: -- Github user ptgoetz commented on the pull request: https://github.com/apache/storm/pull/437#issuecomment-77394741 +1 > Supervisor local worker state corrupted and failing to start. > - > > Key: STORM-682 > URL: https://issues.apache.org/jira/browse/STORM-682 > Project: Apache Storm > Issue Type: Bug >Reporter: Parth Brahmbhatt >Assignee: Parth Brahmbhatt > > If supervisor's cleanup of a worker fails to delete some heartbeat files the > local state of the supervisors get corrupted.The only way to recover the > supervisor from this state is to delete the local state folder where > supervisor stores all worker information.This fix can get very cumbersome if > it happens on multiple worker nodes. > The root cause of the issue is the order in which worker heartbeat versioned > store files are created vs the deletion order of those files. LocalState.put > first creates a data file X and then marks a success by creating a file > X.version. During get it first checks for all *.version files , tries to > find the largest value of X and then issues a read against X. See the below > pseudo code > {code:java} > start_supervisor() { > workerIds = `ls local-state/workers` > for each workerId in workerIds > versions = `ls local-state/workers/workerId/heartbeats/*.version` > latest_version = max(versions) > read local-state/workers/workerId/heartbeats/latest_version [Note there > is no .version extension] > } > {code} > During cleanup it first tries to delete file X and then X.version. If X gets > deleted but X.version fails to delete the supervisor fails to start with > FileNotFoundException in the code above. > We propose to change the deletion order so the .version files get deleted > before the data file and catch any IOException when reading worker heartbeats > to avoid supervisor failure. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-682) Supervisor local worker state corrupted and failing to start.
[ https://issues.apache.org/jira/browse/STORM-682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14349014#comment-14349014 ] ASF GitHub Bot commented on STORM-682: -- Github user ptgoetz commented on the pull request: https://github.com/apache/storm/pull/437#issuecomment-77394892 We may want to consider applying this to the 0.9.x branch as well. > Supervisor local worker state corrupted and failing to start. > - > > Key: STORM-682 > URL: https://issues.apache.org/jira/browse/STORM-682 > Project: Apache Storm > Issue Type: Bug >Reporter: Parth Brahmbhatt >Assignee: Parth Brahmbhatt > > If supervisor's cleanup of a worker fails to delete some heartbeat files the > local state of the supervisors get corrupted.The only way to recover the > supervisor from this state is to delete the local state folder where > supervisor stores all worker information.This fix can get very cumbersome if > it happens on multiple worker nodes. > The root cause of the issue is the order in which worker heartbeat versioned > store files are created vs the deletion order of those files. LocalState.put > first creates a data file X and then marks a success by creating a file > X.version. During get it first checks for all *.version files , tries to > find the largest value of X and then issues a read against X. See the below > pseudo code > {code:java} > start_supervisor() { > workerIds = `ls local-state/workers` > for each workerId in workerIds > versions = `ls local-state/workers/workerId/heartbeats/*.version` > latest_version = max(versions) > read local-state/workers/workerId/heartbeats/latest_version [Note there > is no .version extension] > } > {code} > During cleanup it first tries to delete file X and then X.version. If X gets > deleted but X.version fails to delete the supervisor fails to start with > FileNotFoundException in the code above. > We propose to change the deletion order so the .version files get deleted > before the data file and catch any IOException when reading worker heartbeats > to avoid supervisor failure. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-682) Supervisor local worker state corrupted and failing to start.
[ https://issues.apache.org/jira/browse/STORM-682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14352173#comment-14352173 ] ASF GitHub Bot commented on STORM-682: -- Github user harshach commented on the pull request: https://github.com/apache/storm/pull/437#issuecomment-77764874 @nathanmarz we need this PR for 0.9.4 release . Please take a look at the latest patch. > Supervisor local worker state corrupted and failing to start. > - > > Key: STORM-682 > URL: https://issues.apache.org/jira/browse/STORM-682 > Project: Apache Storm > Issue Type: Bug >Reporter: Parth Brahmbhatt >Assignee: Parth Brahmbhatt > > If supervisor's cleanup of a worker fails to delete some heartbeat files the > local state of the supervisors get corrupted.The only way to recover the > supervisor from this state is to delete the local state folder where > supervisor stores all worker information.This fix can get very cumbersome if > it happens on multiple worker nodes. > The root cause of the issue is the order in which worker heartbeat versioned > store files are created vs the deletion order of those files. LocalState.put > first creates a data file X and then marks a success by creating a file > X.version. During get it first checks for all *.version files , tries to > find the largest value of X and then issues a read against X. See the below > pseudo code > {code:java} > start_supervisor() { > workerIds = `ls local-state/workers` > for each workerId in workerIds > versions = `ls local-state/workers/workerId/heartbeats/*.version` > latest_version = max(versions) > read local-state/workers/workerId/heartbeats/latest_version [Note there > is no .version extension] > } > {code} > During cleanup it first tries to delete file X and then X.version. If X gets > deleted but X.version fails to delete the supervisor fails to start with > FileNotFoundException in the code above. > We propose to change the deletion order so the .version files get deleted > before the data file and catch any IOException when reading worker heartbeats > to avoid supervisor failure. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-682) Supervisor local worker state corrupted and failing to start.
[ https://issues.apache.org/jira/browse/STORM-682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14353369#comment-14353369 ] ASF GitHub Bot commented on STORM-682: -- Github user Parth-Brahmbhatt commented on the pull request: https://github.com/apache/storm/pull/437#issuecomment-77916212 @ptgoetz I will create new jira for 0.9.x branch and apply the fix. @nathanmarz given this is blocking the release and you have a -1 vote I would really appreciate if you could take some time to review this again. > Supervisor local worker state corrupted and failing to start. > - > > Key: STORM-682 > URL: https://issues.apache.org/jira/browse/STORM-682 > Project: Apache Storm > Issue Type: Bug >Reporter: Parth Brahmbhatt >Assignee: Parth Brahmbhatt > > If supervisor's cleanup of a worker fails to delete some heartbeat files the > local state of the supervisors get corrupted.The only way to recover the > supervisor from this state is to delete the local state folder where > supervisor stores all worker information.This fix can get very cumbersome if > it happens on multiple worker nodes. > The root cause of the issue is the order in which worker heartbeat versioned > store files are created vs the deletion order of those files. LocalState.put > first creates a data file X and then marks a success by creating a file > X.version. During get it first checks for all *.version files , tries to > find the largest value of X and then issues a read against X. See the below > pseudo code > {code:java} > start_supervisor() { > workerIds = `ls local-state/workers` > for each workerId in workerIds > versions = `ls local-state/workers/workerId/heartbeats/*.version` > latest_version = max(versions) > read local-state/workers/workerId/heartbeats/latest_version [Note there > is no .version extension] > } > {code} > During cleanup it first tries to delete file X and then X.version. If X gets > deleted but X.version fails to delete the supervisor fails to start with > FileNotFoundException in the code above. > We propose to change the deletion order so the .version files get deleted > before the data file and catch any IOException when reading worker heartbeats > to avoid supervisor failure. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-682) Supervisor local worker state corrupted and failing to start.
[ https://issues.apache.org/jira/browse/STORM-682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14353401#comment-14353401 ] ASF GitHub Bot commented on STORM-682: -- Github user nathanmarz commented on the pull request: https://github.com/apache/storm/pull/437#issuecomment-77919851 +1 > Supervisor local worker state corrupted and failing to start. > - > > Key: STORM-682 > URL: https://issues.apache.org/jira/browse/STORM-682 > Project: Apache Storm > Issue Type: Bug >Reporter: Parth Brahmbhatt >Assignee: Parth Brahmbhatt > > If supervisor's cleanup of a worker fails to delete some heartbeat files the > local state of the supervisors get corrupted.The only way to recover the > supervisor from this state is to delete the local state folder where > supervisor stores all worker information.This fix can get very cumbersome if > it happens on multiple worker nodes. > The root cause of the issue is the order in which worker heartbeat versioned > store files are created vs the deletion order of those files. LocalState.put > first creates a data file X and then marks a success by creating a file > X.version. During get it first checks for all *.version files , tries to > find the largest value of X and then issues a read against X. See the below > pseudo code > {code:java} > start_supervisor() { > workerIds = `ls local-state/workers` > for each workerId in workerIds > versions = `ls local-state/workers/workerId/heartbeats/*.version` > latest_version = max(versions) > read local-state/workers/workerId/heartbeats/latest_version [Note there > is no .version extension] > } > {code} > During cleanup it first tries to delete file X and then X.version. If X gets > deleted but X.version fails to delete the supervisor fails to start with > FileNotFoundException in the code above. > We propose to change the deletion order so the .version files get deleted > before the data file and catch any IOException when reading worker heartbeats > to avoid supervisor failure. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-682) Supervisor local worker state corrupted and failing to start.
[ https://issues.apache.org/jira/browse/STORM-682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14359579#comment-14359579 ] ASF GitHub Bot commented on STORM-682: -- Github user asfgit closed the pull request at: https://github.com/apache/storm/pull/437 > Supervisor local worker state corrupted and failing to start. > - > > Key: STORM-682 > URL: https://issues.apache.org/jira/browse/STORM-682 > Project: Apache Storm > Issue Type: Bug >Reporter: Parth Brahmbhatt >Assignee: Parth Brahmbhatt > > If supervisor's cleanup of a worker fails to delete some heartbeat files the > local state of the supervisors get corrupted.The only way to recover the > supervisor from this state is to delete the local state folder where > supervisor stores all worker information.This fix can get very cumbersome if > it happens on multiple worker nodes. > The root cause of the issue is the order in which worker heartbeat versioned > store files are created vs the deletion order of those files. LocalState.put > first creates a data file X and then marks a success by creating a file > X.version. During get it first checks for all *.version files , tries to > find the largest value of X and then issues a read against X. See the below > pseudo code > {code:java} > start_supervisor() { > workerIds = `ls local-state/workers` > for each workerId in workerIds > versions = `ls local-state/workers/workerId/heartbeats/*.version` > latest_version = max(versions) > read local-state/workers/workerId/heartbeats/latest_version [Note there > is no .version extension] > } > {code} > During cleanup it first tries to delete file X and then X.version. If X gets > deleted but X.version fails to delete the supervisor fails to start with > FileNotFoundException in the code above. > We propose to change the deletion order so the .version files get deleted > before the data file and catch any IOException when reading worker heartbeats > to avoid supervisor failure. -- This message was sent by Atlassian JIRA (v6.3.4#6332)