[jira] [Updated] (HADOOP-15679) ShutdownHookManager shutdown time needs to be configurable & extended
[ https://issues.apache.org/jira/browse/HADOOP-15679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-15679: Fix Version/s: (was: 2.8.5) 2.8.6 > ShutdownHookManager shutdown time needs to be configurable & extended > - > > Key: HADOOP-15679 > URL: https://issues.apache.org/jira/browse/HADOOP-15679 > Project: Hadoop Common > Issue Type: Bug > Components: util >Affects Versions: 2.8.0, 3.0.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Fix For: 2.9.2, 3.0.4, 3.1.2, 2.8.6 > > Attachments: HADOOP-15679-001.patch, HADOOP-15679-002.patch, > HADOOP-15679-002.patch, HADOOP-15679-003.patch, > HADOOP-15679-branch-2-001.patch, HADOOP-15679-branch-2-001.patch, > HADOOP-15679-branch-2-003.patch, HADOOP-15679-branch-2-003.patch, > HADOOP-15679-branch-2-004.patch, HADOOP-15679-branch-2-004.patch, > HADOOP-15679-branch-2.8-005.patch, HADOOP-15679-branch-2.8-005.patch > > > HADOOP-12950 added a timeout on shutdowns to avoid problems with hanging > shutdowns. But the timeout is too short for applications where a large flush > of data is needed on shutdown. > A key example of this is Spark apps which save their history to object > stores, where the file close() call triggers an upload of the final local > cached block of data (could be 32+MB), and then execute the final mutipart > commit. > Proposed > # make the default sleep time 30s, not 10s > # make it configurable with a time duration property (with minimum time of > 1s.?) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15679) ShutdownHookManager shutdown time needs to be configurable & extended
[ https://issues.apache.org/jira/browse/HADOOP-15679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-15679: Resolution: Fixed Fix Version/s: 3.1.2 3.0.4 2.8.5 2.9.2 Target Version/s: 3.0.3, 2.10.0, 2.9.2, 2.8.5 (was: 2.10.0, 2.9.2, 3.0.3, 2.8.5) Status: Resolved (was: Patch Available) fixed across everything where the initial change went in. > ShutdownHookManager shutdown time needs to be configurable & extended > - > > Key: HADOOP-15679 > URL: https://issues.apache.org/jira/browse/HADOOP-15679 > Project: Hadoop Common > Issue Type: Bug > Components: util >Affects Versions: 2.8.0, 3.0.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Fix For: 2.9.2, 2.8.5, 3.0.4, 3.1.2 > > Attachments: HADOOP-15679-001.patch, HADOOP-15679-002.patch, > HADOOP-15679-002.patch, HADOOP-15679-003.patch, > HADOOP-15679-branch-2-001.patch, HADOOP-15679-branch-2-001.patch, > HADOOP-15679-branch-2-003.patch, HADOOP-15679-branch-2-003.patch, > HADOOP-15679-branch-2-004.patch, HADOOP-15679-branch-2-004.patch, > HADOOP-15679-branch-2.8-005.patch, HADOOP-15679-branch-2.8-005.patch > > > HADOOP-12950 added a timeout on shutdowns to avoid problems with hanging > shutdowns. But the timeout is too short for applications where a large flush > of data is needed on shutdown. > A key example of this is Spark apps which save their history to object > stores, where the file close() call triggers an upload of the final local > cached block of data (could be 32+MB), and then execute the final mutipart > commit. > Proposed > # make the default sleep time 30s, not 10s > # make it configurable with a time duration property (with minimum time of > 1s.?) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15679) ShutdownHookManager shutdown time needs to be configurable & extended
[ https://issues.apache.org/jira/browse/HADOOP-15679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-15679: Target Version/s: 3.0.3, 2.10.0, 2.9.2, 2.8.5 (was: 2.10.0, 2.9.2, 3.0.3, 2.8.5) Status: Patch Available (was: Open) > ShutdownHookManager shutdown time needs to be configurable & extended > - > > Key: HADOOP-15679 > URL: https://issues.apache.org/jira/browse/HADOOP-15679 > Project: Hadoop Common > Issue Type: Bug > Components: util >Affects Versions: 3.0.0, 2.8.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-15679-001.patch, HADOOP-15679-002.patch, > HADOOP-15679-002.patch, HADOOP-15679-003.patch, > HADOOP-15679-branch-2-001.patch, HADOOP-15679-branch-2-001.patch, > HADOOP-15679-branch-2-003.patch, HADOOP-15679-branch-2-003.patch, > HADOOP-15679-branch-2-004.patch, HADOOP-15679-branch-2-004.patch, > HADOOP-15679-branch-2.8-005.patch, HADOOP-15679-branch-2.8-005.patch > > > HADOOP-12950 added a timeout on shutdowns to avoid problems with hanging > shutdowns. But the timeout is too short for applications where a large flush > of data is needed on shutdown. > A key example of this is Spark apps which save their history to object > stores, where the file close() call triggers an upload of the final local > cached block of data (could be 32+MB), and then execute the final mutipart > commit. > Proposed > # make the default sleep time 30s, not 10s > # make it configurable with a time duration property (with minimum time of > 1s.?) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15679) ShutdownHookManager shutdown time needs to be configurable & extended
[ https://issues.apache.org/jira/browse/HADOOP-15679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-15679: Target Version/s: 3.0.3, 2.10.0, 2.9.2, 2.8.5 (was: 2.10.0, 2.9.2, 3.0.3, 2.8.5) Status: Open (was: Patch Available) > ShutdownHookManager shutdown time needs to be configurable & extended > - > > Key: HADOOP-15679 > URL: https://issues.apache.org/jira/browse/HADOOP-15679 > Project: Hadoop Common > Issue Type: Bug > Components: util >Affects Versions: 3.0.0, 2.8.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-15679-001.patch, HADOOP-15679-002.patch, > HADOOP-15679-002.patch, HADOOP-15679-003.patch, > HADOOP-15679-branch-2-001.patch, HADOOP-15679-branch-2-001.patch, > HADOOP-15679-branch-2-003.patch, HADOOP-15679-branch-2-003.patch, > HADOOP-15679-branch-2-004.patch, HADOOP-15679-branch-2-004.patch, > HADOOP-15679-branch-2.8-005.patch, HADOOP-15679-branch-2.8-005.patch > > > HADOOP-12950 added a timeout on shutdowns to avoid problems with hanging > shutdowns. But the timeout is too short for applications where a large flush > of data is needed on shutdown. > A key example of this is Spark apps which save their history to object > stores, where the file close() call triggers an upload of the final local > cached block of data (could be 32+MB), and then execute the final mutipart > commit. > Proposed > # make the default sleep time 30s, not 10s > # make it configurable with a time duration property (with minimum time of > 1s.?) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15679) ShutdownHookManager shutdown time needs to be configurable & extended
[ https://issues.apache.org/jira/browse/HADOOP-15679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-15679: Attachment: (was: HADOOP-15679-branch-2.8-005.patch) > ShutdownHookManager shutdown time needs to be configurable & extended > - > > Key: HADOOP-15679 > URL: https://issues.apache.org/jira/browse/HADOOP-15679 > Project: Hadoop Common > Issue Type: Bug > Components: util >Affects Versions: 2.8.0, 3.0.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-15679-001.patch, HADOOP-15679-002.patch, > HADOOP-15679-002.patch, HADOOP-15679-003.patch, > HADOOP-15679-branch-2-001.patch, HADOOP-15679-branch-2-001.patch, > HADOOP-15679-branch-2-003.patch, HADOOP-15679-branch-2-003.patch, > HADOOP-15679-branch-2-004.patch, HADOOP-15679-branch-2-004.patch, > HADOOP-15679-branch-2.8-005.patch, HADOOP-15679-branch-2.8-005.patch > > > HADOOP-12950 added a timeout on shutdowns to avoid problems with hanging > shutdowns. But the timeout is too short for applications where a large flush > of data is needed on shutdown. > A key example of this is Spark apps which save their history to object > stores, where the file close() call triggers an upload of the final local > cached block of data (could be 32+MB), and then execute the final mutipart > commit. > Proposed > # make the default sleep time 30s, not 10s > # make it configurable with a time duration property (with minimum time of > 1s.?) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15679) ShutdownHookManager shutdown time needs to be configurable & extended
[ https://issues.apache.org/jira/browse/HADOOP-15679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-15679: Attachment: HADOOP-15679-branch-2.8-005.patch > ShutdownHookManager shutdown time needs to be configurable & extended > - > > Key: HADOOP-15679 > URL: https://issues.apache.org/jira/browse/HADOOP-15679 > Project: Hadoop Common > Issue Type: Bug > Components: util >Affects Versions: 2.8.0, 3.0.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-15679-001.patch, HADOOP-15679-002.patch, > HADOOP-15679-002.patch, HADOOP-15679-003.patch, > HADOOP-15679-branch-2-001.patch, HADOOP-15679-branch-2-001.patch, > HADOOP-15679-branch-2-003.patch, HADOOP-15679-branch-2-003.patch, > HADOOP-15679-branch-2-004.patch, HADOOP-15679-branch-2-004.patch, > HADOOP-15679-branch-2.8-005.patch, HADOOP-15679-branch-2.8-005.patch > > > HADOOP-12950 added a timeout on shutdowns to avoid problems with hanging > shutdowns. But the timeout is too short for applications where a large flush > of data is needed on shutdown. > A key example of this is Spark apps which save their history to object > stores, where the file close() call triggers an upload of the final local > cached block of data (could be 32+MB), and then execute the final mutipart > commit. > Proposed > # make the default sleep time 30s, not 10s > # make it configurable with a time duration property (with minimum time of > 1s.?) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15679) ShutdownHookManager shutdown time needs to be configurable & extended
[ https://issues.apache.org/jira/browse/HADOOP-15679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-15679: Target Version/s: 3.0.3, 2.10.0, 2.9.2, 2.8.5 (was: 2.10.0, 2.9.2, 3.0.3, 2.8.5) Status: Patch Available (was: Open) > ShutdownHookManager shutdown time needs to be configurable & extended > - > > Key: HADOOP-15679 > URL: https://issues.apache.org/jira/browse/HADOOP-15679 > Project: Hadoop Common > Issue Type: Bug > Components: util >Affects Versions: 3.0.0, 2.8.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-15679-001.patch, HADOOP-15679-002.patch, > HADOOP-15679-002.patch, HADOOP-15679-003.patch, > HADOOP-15679-branch-2-001.patch, HADOOP-15679-branch-2-001.patch, > HADOOP-15679-branch-2-003.patch, HADOOP-15679-branch-2-003.patch, > HADOOP-15679-branch-2-004.patch, HADOOP-15679-branch-2-004.patch, > HADOOP-15679-branch-2.8-005.patch, HADOOP-15679-branch-2.8-005.patch > > > HADOOP-12950 added a timeout on shutdowns to avoid problems with hanging > shutdowns. But the timeout is too short for applications where a large flush > of data is needed on shutdown. > A key example of this is Spark apps which save their history to object > stores, where the file close() call triggers an upload of the final local > cached block of data (could be 32+MB), and then execute the final mutipart > commit. > Proposed > # make the default sleep time 30s, not 10s > # make it configurable with a time duration property (with minimum time of > 1s.?) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15679) ShutdownHookManager shutdown time needs to be configurable & extended
[ https://issues.apache.org/jira/browse/HADOOP-15679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-15679: Attachment: HADOOP-15679-branch-2.8-005.patch > ShutdownHookManager shutdown time needs to be configurable & extended > - > > Key: HADOOP-15679 > URL: https://issues.apache.org/jira/browse/HADOOP-15679 > Project: Hadoop Common > Issue Type: Bug > Components: util >Affects Versions: 2.8.0, 3.0.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-15679-001.patch, HADOOP-15679-002.patch, > HADOOP-15679-002.patch, HADOOP-15679-003.patch, > HADOOP-15679-branch-2-001.patch, HADOOP-15679-branch-2-001.patch, > HADOOP-15679-branch-2-003.patch, HADOOP-15679-branch-2-003.patch, > HADOOP-15679-branch-2-004.patch, HADOOP-15679-branch-2-004.patch, > HADOOP-15679-branch-2.8-005.patch, HADOOP-15679-branch-2.8-005.patch > > > HADOOP-12950 added a timeout on shutdowns to avoid problems with hanging > shutdowns. But the timeout is too short for applications where a large flush > of data is needed on shutdown. > A key example of this is Spark apps which save their history to object > stores, where the file close() call triggers an upload of the final local > cached block of data (could be 32+MB), and then execute the final mutipart > commit. > Proposed > # make the default sleep time 30s, not 10s > # make it configurable with a time duration property (with minimum time of > 1s.?) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15679) ShutdownHookManager shutdown time needs to be configurable & extended
[ https://issues.apache.org/jira/browse/HADOOP-15679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-15679: Target Version/s: 3.0.3, 2.10.0, 2.9.2, 2.8.5 (was: 2.10.0, 2.9.2, 3.0.3, 2.8.5) Status: Open (was: Patch Available) > ShutdownHookManager shutdown time needs to be configurable & extended > - > > Key: HADOOP-15679 > URL: https://issues.apache.org/jira/browse/HADOOP-15679 > Project: Hadoop Common > Issue Type: Bug > Components: util >Affects Versions: 3.0.0, 2.8.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-15679-001.patch, HADOOP-15679-002.patch, > HADOOP-15679-002.patch, HADOOP-15679-003.patch, > HADOOP-15679-branch-2-001.patch, HADOOP-15679-branch-2-001.patch, > HADOOP-15679-branch-2-003.patch, HADOOP-15679-branch-2-003.patch, > HADOOP-15679-branch-2-004.patch, HADOOP-15679-branch-2-004.patch, > HADOOP-15679-branch-2.8-005.patch > > > HADOOP-12950 added a timeout on shutdowns to avoid problems with hanging > shutdowns. But the timeout is too short for applications where a large flush > of data is needed on shutdown. > A key example of this is Spark apps which save their history to object > stores, where the file close() call triggers an upload of the final local > cached block of data (could be 32+MB), and then execute the final mutipart > commit. > Proposed > # make the default sleep time 30s, not 10s > # make it configurable with a time duration property (with minimum time of > 1s.?) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15679) ShutdownHookManager shutdown time needs to be configurable & extended
[ https://issues.apache.org/jira/browse/HADOOP-15679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-15679: Target Version/s: 3.0.3, 2.10.0, 2.9.2, 2.8.5 (was: 2.10.0, 2.9.2, 3.0.3, 2.8.5) Status: Patch Available (was: Open) patch HADOOP-15679-branch-2.8-005.patch this is the branch-2.8 one, which moves the logging in the shutdown hook code over to SLF4J from commons logging, keeping the diff between the hook and branch-2.9+ down to a minimum > ShutdownHookManager shutdown time needs to be configurable & extended > - > > Key: HADOOP-15679 > URL: https://issues.apache.org/jira/browse/HADOOP-15679 > Project: Hadoop Common > Issue Type: Bug > Components: util >Affects Versions: 3.0.0, 2.8.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-15679-001.patch, HADOOP-15679-002.patch, > HADOOP-15679-002.patch, HADOOP-15679-003.patch, > HADOOP-15679-branch-2-001.patch, HADOOP-15679-branch-2-001.patch, > HADOOP-15679-branch-2-003.patch, HADOOP-15679-branch-2-003.patch, > HADOOP-15679-branch-2-004.patch, HADOOP-15679-branch-2-004.patch, > HADOOP-15679-branch-2.8-005.patch > > > HADOOP-12950 added a timeout on shutdowns to avoid problems with hanging > shutdowns. But the timeout is too short for applications where a large flush > of data is needed on shutdown. > A key example of this is Spark apps which save their history to object > stores, where the file close() call triggers an upload of the final local > cached block of data (could be 32+MB), and then execute the final mutipart > commit. > Proposed > # make the default sleep time 30s, not 10s > # make it configurable with a time duration property (with minimum time of > 1s.?) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15679) ShutdownHookManager shutdown time needs to be configurable & extended
[ https://issues.apache.org/jira/browse/HADOOP-15679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-15679: Attachment: HADOOP-15679-branch-2.8-005.patch > ShutdownHookManager shutdown time needs to be configurable & extended > - > > Key: HADOOP-15679 > URL: https://issues.apache.org/jira/browse/HADOOP-15679 > Project: Hadoop Common > Issue Type: Bug > Components: util >Affects Versions: 2.8.0, 3.0.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-15679-001.patch, HADOOP-15679-002.patch, > HADOOP-15679-002.patch, HADOOP-15679-003.patch, > HADOOP-15679-branch-2-001.patch, HADOOP-15679-branch-2-001.patch, > HADOOP-15679-branch-2-003.patch, HADOOP-15679-branch-2-003.patch, > HADOOP-15679-branch-2-004.patch, HADOOP-15679-branch-2-004.patch, > HADOOP-15679-branch-2.8-005.patch > > > HADOOP-12950 added a timeout on shutdowns to avoid problems with hanging > shutdowns. But the timeout is too short for applications where a large flush > of data is needed on shutdown. > A key example of this is Spark apps which save their history to object > stores, where the file close() call triggers an upload of the final local > cached block of data (could be 32+MB), and then execute the final mutipart > commit. > Proposed > # make the default sleep time 30s, not 10s > # make it configurable with a time duration property (with minimum time of > 1s.?) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15679) ShutdownHookManager shutdown time needs to be configurable & extended
[ https://issues.apache.org/jira/browse/HADOOP-15679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-15679: Target Version/s: 3.0.3, 2.10.0, 2.9.2, 2.8.5 (was: 2.10.0, 2.9.2, 3.0.3, 2.8.5) Status: Open (was: Patch Available) > ShutdownHookManager shutdown time needs to be configurable & extended > - > > Key: HADOOP-15679 > URL: https://issues.apache.org/jira/browse/HADOOP-15679 > Project: Hadoop Common > Issue Type: Bug > Components: util >Affects Versions: 3.0.0, 2.8.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-15679-001.patch, HADOOP-15679-002.patch, > HADOOP-15679-002.patch, HADOOP-15679-003.patch, > HADOOP-15679-branch-2-001.patch, HADOOP-15679-branch-2-001.patch, > HADOOP-15679-branch-2-003.patch, HADOOP-15679-branch-2-003.patch, > HADOOP-15679-branch-2-004.patch, HADOOP-15679-branch-2-004.patch, > HADOOP-15679-branch-2.8-005.patch > > > HADOOP-12950 added a timeout on shutdowns to avoid problems with hanging > shutdowns. But the timeout is too short for applications where a large flush > of data is needed on shutdown. > A key example of this is Spark apps which save their history to object > stores, where the file close() call triggers an upload of the final local > cached block of data (could be 32+MB), and then execute the final mutipart > commit. > Proposed > # make the default sleep time 30s, not 10s > # make it configurable with a time duration property (with minimum time of > 1s.?) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15679) ShutdownHookManager shutdown time needs to be configurable & extended
[ https://issues.apache.org/jira/browse/HADOOP-15679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-15679: Target Version/s: 3.0.3, 2.10.0, 2.9.2, 2.8.5 (was: 2.10.0, 2.9.2, 3.0.3, 2.8.5) Status: Patch Available (was: Open) resubmitting patch 004 to get it through yetus again, will commit if i am happy > ShutdownHookManager shutdown time needs to be configurable & extended > - > > Key: HADOOP-15679 > URL: https://issues.apache.org/jira/browse/HADOOP-15679 > Project: Hadoop Common > Issue Type: Bug > Components: util >Affects Versions: 3.0.0, 2.8.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-15679-001.patch, HADOOP-15679-002.patch, > HADOOP-15679-002.patch, HADOOP-15679-003.patch, > HADOOP-15679-branch-2-001.patch, HADOOP-15679-branch-2-001.patch, > HADOOP-15679-branch-2-003.patch, HADOOP-15679-branch-2-003.patch, > HADOOP-15679-branch-2-004.patch, HADOOP-15679-branch-2-004.patch > > > HADOOP-12950 added a timeout on shutdowns to avoid problems with hanging > shutdowns. But the timeout is too short for applications where a large flush > of data is needed on shutdown. > A key example of this is Spark apps which save their history to object > stores, where the file close() call triggers an upload of the final local > cached block of data (could be 32+MB), and then execute the final mutipart > commit. > Proposed > # make the default sleep time 30s, not 10s > # make it configurable with a time duration property (with minimum time of > 1s.?) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15679) ShutdownHookManager shutdown time needs to be configurable & extended
[ https://issues.apache.org/jira/browse/HADOOP-15679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-15679: Attachment: HADOOP-15679-branch-2-004.patch > ShutdownHookManager shutdown time needs to be configurable & extended > - > > Key: HADOOP-15679 > URL: https://issues.apache.org/jira/browse/HADOOP-15679 > Project: Hadoop Common > Issue Type: Bug > Components: util >Affects Versions: 2.8.0, 3.0.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-15679-001.patch, HADOOP-15679-002.patch, > HADOOP-15679-002.patch, HADOOP-15679-003.patch, > HADOOP-15679-branch-2-001.patch, HADOOP-15679-branch-2-001.patch, > HADOOP-15679-branch-2-003.patch, HADOOP-15679-branch-2-003.patch, > HADOOP-15679-branch-2-004.patch, HADOOP-15679-branch-2-004.patch > > > HADOOP-12950 added a timeout on shutdowns to avoid problems with hanging > shutdowns. But the timeout is too short for applications where a large flush > of data is needed on shutdown. > A key example of this is Spark apps which save their history to object > stores, where the file close() call triggers an upload of the final local > cached block of data (could be 32+MB), and then execute the final mutipart > commit. > Proposed > # make the default sleep time 30s, not 10s > # make it configurable with a time duration property (with minimum time of > 1s.?) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15679) ShutdownHookManager shutdown time needs to be configurable & extended
[ https://issues.apache.org/jira/browse/HADOOP-15679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-15679: Target Version/s: 3.0.3, 2.10.0, 2.9.2, 2.8.5 (was: 2.10.0, 2.9.2, 3.0.3, 2.8.5) Status: Open (was: Patch Available) > ShutdownHookManager shutdown time needs to be configurable & extended > - > > Key: HADOOP-15679 > URL: https://issues.apache.org/jira/browse/HADOOP-15679 > Project: Hadoop Common > Issue Type: Bug > Components: util >Affects Versions: 3.0.0, 2.8.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-15679-001.patch, HADOOP-15679-002.patch, > HADOOP-15679-002.patch, HADOOP-15679-003.patch, > HADOOP-15679-branch-2-001.patch, HADOOP-15679-branch-2-001.patch, > HADOOP-15679-branch-2-003.patch, HADOOP-15679-branch-2-003.patch, > HADOOP-15679-branch-2-004.patch, HADOOP-15679-branch-2-004.patch > > > HADOOP-12950 added a timeout on shutdowns to avoid problems with hanging > shutdowns. But the timeout is too short for applications where a large flush > of data is needed on shutdown. > A key example of this is Spark apps which save their history to object > stores, where the file close() call triggers an upload of the final local > cached block of data (could be 32+MB), and then execute the final mutipart > commit. > Proposed > # make the default sleep time 30s, not 10s > # make it configurable with a time duration property (with minimum time of > 1s.?) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15679) ShutdownHookManager shutdown time needs to be configurable & extended
[ https://issues.apache.org/jira/browse/HADOOP-15679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-15679: Attachment: HADOOP-15679-branch-2-004.patch > ShutdownHookManager shutdown time needs to be configurable & extended > - > > Key: HADOOP-15679 > URL: https://issues.apache.org/jira/browse/HADOOP-15679 > Project: Hadoop Common > Issue Type: Bug > Components: util >Affects Versions: 2.8.0, 3.0.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-15679-001.patch, HADOOP-15679-002.patch, > HADOOP-15679-002.patch, HADOOP-15679-003.patch, > HADOOP-15679-branch-2-001.patch, HADOOP-15679-branch-2-001.patch, > HADOOP-15679-branch-2-003.patch, HADOOP-15679-branch-2-003.patch, > HADOOP-15679-branch-2-004.patch > > > HADOOP-12950 added a timeout on shutdowns to avoid problems with hanging > shutdowns. But the timeout is too short for applications where a large flush > of data is needed on shutdown. > A key example of this is Spark apps which save their history to object > stores, where the file close() call triggers an upload of the final local > cached block of data (could be 32+MB), and then execute the final mutipart > commit. > Proposed > # make the default sleep time 30s, not 10s > # make it configurable with a time duration property (with minimum time of > 1s.?) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15679) ShutdownHookManager shutdown time needs to be configurable & extended
[ https://issues.apache.org/jira/browse/HADOOP-15679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-15679: Target Version/s: 3.0.3, 2.10.0, 2.9.2, 2.8.5 (was: 2.10.0, 2.9.2, 3.0.3, 2.8.5) Status: Patch Available (was: Open) > ShutdownHookManager shutdown time needs to be configurable & extended > - > > Key: HADOOP-15679 > URL: https://issues.apache.org/jira/browse/HADOOP-15679 > Project: Hadoop Common > Issue Type: Bug > Components: util >Affects Versions: 3.0.0, 2.8.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-15679-001.patch, HADOOP-15679-002.patch, > HADOOP-15679-002.patch, HADOOP-15679-003.patch, > HADOOP-15679-branch-2-001.patch, HADOOP-15679-branch-2-001.patch, > HADOOP-15679-branch-2-003.patch, HADOOP-15679-branch-2-003.patch, > HADOOP-15679-branch-2-004.patch > > > HADOOP-12950 added a timeout on shutdowns to avoid problems with hanging > shutdowns. But the timeout is too short for applications where a large flush > of data is needed on shutdown. > A key example of this is Spark apps which save their history to object > stores, where the file close() call triggers an upload of the final local > cached block of data (could be 32+MB), and then execute the final mutipart > commit. > Proposed > # make the default sleep time 30s, not 10s > # make it configurable with a time duration property (with minimum time of > 1s.?) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15679) ShutdownHookManager shutdown time needs to be configurable & extended
[ https://issues.apache.org/jira/browse/HADOOP-15679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-15679: Target Version/s: 3.0.3, 2.10.0, 2.9.2, 2.8.5 (was: 2.10.0, 2.9.2, 3.0.3, 2.8.5) Status: Open (was: Patch Available) > ShutdownHookManager shutdown time needs to be configurable & extended > - > > Key: HADOOP-15679 > URL: https://issues.apache.org/jira/browse/HADOOP-15679 > Project: Hadoop Common > Issue Type: Bug > Components: util >Affects Versions: 3.0.0, 2.8.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-15679-001.patch, HADOOP-15679-002.patch, > HADOOP-15679-002.patch, HADOOP-15679-003.patch, > HADOOP-15679-branch-2-001.patch, HADOOP-15679-branch-2-001.patch, > HADOOP-15679-branch-2-003.patch, HADOOP-15679-branch-2-003.patch, > HADOOP-15679-branch-2-004.patch > > > HADOOP-12950 added a timeout on shutdowns to avoid problems with hanging > shutdowns. But the timeout is too short for applications where a large flush > of data is needed on shutdown. > A key example of this is Spark apps which save their history to object > stores, where the file close() call triggers an upload of the final local > cached block of data (could be 32+MB), and then execute the final mutipart > commit. > Proposed > # make the default sleep time 30s, not 10s > # make it configurable with a time duration property (with minimum time of > 1s.?) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15679) ShutdownHookManager shutdown time needs to be configurable & extended
[ https://issues.apache.org/jira/browse/HADOOP-15679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-15679: Attachment: HADOOP-15679-branch-2-003.patch > ShutdownHookManager shutdown time needs to be configurable & extended > - > > Key: HADOOP-15679 > URL: https://issues.apache.org/jira/browse/HADOOP-15679 > Project: Hadoop Common > Issue Type: Bug > Components: util >Affects Versions: 2.8.0, 3.0.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-15679-001.patch, HADOOP-15679-002.patch, > HADOOP-15679-002.patch, HADOOP-15679-003.patch, > HADOOP-15679-branch-2-001.patch, HADOOP-15679-branch-2-001.patch, > HADOOP-15679-branch-2-003.patch, HADOOP-15679-branch-2-003.patch > > > HADOOP-12950 added a timeout on shutdowns to avoid problems with hanging > shutdowns. But the timeout is too short for applications where a large flush > of data is needed on shutdown. > A key example of this is Spark apps which save their history to object > stores, where the file close() call triggers an upload of the final local > cached block of data (could be 32+MB), and then execute the final mutipart > commit. > Proposed > # make the default sleep time 30s, not 10s > # make it configurable with a time duration property (with minimum time of > 1s.?) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15679) ShutdownHookManager shutdown time needs to be configurable & extended
[ https://issues.apache.org/jira/browse/HADOOP-15679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-15679: Target Version/s: 3.0.3, 2.10.0, 2.9.2, 2.8.5 (was: 2.10.0, 2.9.2, 3.0.3, 2.8.5) Status: Patch Available (was: Open) > ShutdownHookManager shutdown time needs to be configurable & extended > - > > Key: HADOOP-15679 > URL: https://issues.apache.org/jira/browse/HADOOP-15679 > Project: Hadoop Common > Issue Type: Bug > Components: util >Affects Versions: 3.0.0, 2.8.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-15679-001.patch, HADOOP-15679-002.patch, > HADOOP-15679-002.patch, HADOOP-15679-003.patch, > HADOOP-15679-branch-2-001.patch, HADOOP-15679-branch-2-001.patch, > HADOOP-15679-branch-2-003.patch, HADOOP-15679-branch-2-003.patch > > > HADOOP-12950 added a timeout on shutdowns to avoid problems with hanging > shutdowns. But the timeout is too short for applications where a large flush > of data is needed on shutdown. > A key example of this is Spark apps which save their history to object > stores, where the file close() call triggers an upload of the final local > cached block of data (could be 32+MB), and then execute the final mutipart > commit. > Proposed > # make the default sleep time 30s, not 10s > # make it configurable with a time duration property (with minimum time of > 1s.?) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15679) ShutdownHookManager shutdown time needs to be configurable & extended
[ https://issues.apache.org/jira/browse/HADOOP-15679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-15679: Target Version/s: 3.0.3, 2.10.0, 2.9.2, 2.8.5 (was: 2.10.0, 2.9.2, 3.0.3, 2.8.5) Status: Open (was: Patch Available) > ShutdownHookManager shutdown time needs to be configurable & extended > - > > Key: HADOOP-15679 > URL: https://issues.apache.org/jira/browse/HADOOP-15679 > Project: Hadoop Common > Issue Type: Bug > Components: util >Affects Versions: 3.0.0, 2.8.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-15679-001.patch, HADOOP-15679-002.patch, > HADOOP-15679-002.patch, HADOOP-15679-003.patch, > HADOOP-15679-branch-2-001.patch, HADOOP-15679-branch-2-001.patch, > HADOOP-15679-branch-2-003.patch, HADOOP-15679-branch-2-003.patch > > > HADOOP-12950 added a timeout on shutdowns to avoid problems with hanging > shutdowns. But the timeout is too short for applications where a large flush > of data is needed on shutdown. > A key example of this is Spark apps which save their history to object > stores, where the file close() call triggers an upload of the final local > cached block of data (could be 32+MB), and then execute the final mutipart > commit. > Proposed > # make the default sleep time 30s, not 10s > # make it configurable with a time duration property (with minimum time of > 1s.?) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15679) ShutdownHookManager shutdown time needs to be configurable & extended
[ https://issues.apache.org/jira/browse/HADOOP-15679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-15679: Attachment: HADOOP-15679-branch-2-003.patch > ShutdownHookManager shutdown time needs to be configurable & extended > - > > Key: HADOOP-15679 > URL: https://issues.apache.org/jira/browse/HADOOP-15679 > Project: Hadoop Common > Issue Type: Bug > Components: util >Affects Versions: 2.8.0, 3.0.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-15679-001.patch, HADOOP-15679-002.patch, > HADOOP-15679-002.patch, HADOOP-15679-003.patch, > HADOOP-15679-branch-2-001.patch, HADOOP-15679-branch-2-001.patch, > HADOOP-15679-branch-2-003.patch > > > HADOOP-12950 added a timeout on shutdowns to avoid problems with hanging > shutdowns. But the timeout is too short for applications where a large flush > of data is needed on shutdown. > A key example of this is Spark apps which save their history to object > stores, where the file close() call triggers an upload of the final local > cached block of data (could be 32+MB), and then execute the final mutipart > commit. > Proposed > # make the default sleep time 30s, not 10s > # make it configurable with a time duration property (with minimum time of > 1s.?) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15679) ShutdownHookManager shutdown time needs to be configurable & extended
[ https://issues.apache.org/jira/browse/HADOOP-15679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-15679: Target Version/s: 3.0.3, 2.10.0, 2.9.2, 2.8.5 (was: 2.10.0, 2.9.2, 3.0.3, 2.8.5) Status: Open (was: Patch Available) > ShutdownHookManager shutdown time needs to be configurable & extended > - > > Key: HADOOP-15679 > URL: https://issues.apache.org/jira/browse/HADOOP-15679 > Project: Hadoop Common > Issue Type: Bug > Components: util >Affects Versions: 3.0.0, 2.8.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-15679-001.patch, HADOOP-15679-002.patch, > HADOOP-15679-002.patch, HADOOP-15679-003.patch, > HADOOP-15679-branch-2-001.patch, HADOOP-15679-branch-2-001.patch, > HADOOP-15679-branch-2-003.patch > > > HADOOP-12950 added a timeout on shutdowns to avoid problems with hanging > shutdowns. But the timeout is too short for applications where a large flush > of data is needed on shutdown. > A key example of this is Spark apps which save their history to object > stores, where the file close() call triggers an upload of the final local > cached block of data (could be 32+MB), and then execute the final mutipart > commit. > Proposed > # make the default sleep time 30s, not 10s > # make it configurable with a time duration property (with minimum time of > 1s.?) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15679) ShutdownHookManager shutdown time needs to be configurable & extended
[ https://issues.apache.org/jira/browse/HADOOP-15679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-15679: Target Version/s: 3.0.3, 2.10.0, 2.9.2, 2.8.5 (was: 2.10.0, 2.9.2, 3.0.3, 2.8.5) Status: Patch Available (was: Open) > ShutdownHookManager shutdown time needs to be configurable & extended > - > > Key: HADOOP-15679 > URL: https://issues.apache.org/jira/browse/HADOOP-15679 > Project: Hadoop Common > Issue Type: Bug > Components: util >Affects Versions: 3.0.0, 2.8.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-15679-001.patch, HADOOP-15679-002.patch, > HADOOP-15679-002.patch, HADOOP-15679-003.patch, > HADOOP-15679-branch-2-001.patch, HADOOP-15679-branch-2-001.patch, > HADOOP-15679-branch-2-003.patch > > > HADOOP-12950 added a timeout on shutdowns to avoid problems with hanging > shutdowns. But the timeout is too short for applications where a large flush > of data is needed on shutdown. > A key example of this is Spark apps which save their history to object > stores, where the file close() call triggers an upload of the final local > cached block of data (could be 32+MB), and then execute the final mutipart > commit. > Proposed > # make the default sleep time 30s, not 10s > # make it configurable with a time duration property (with minimum time of > 1s.?) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15679) ShutdownHookManager shutdown time needs to be configurable & extended
[ https://issues.apache.org/jira/browse/HADOOP-15679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-15679: Target Version/s: 3.0.3, 2.10.0, 2.9.2, 2.8.5 (was: 2.10.0, 2.9.2, 3.0.3, 2.8.5) Status: Patch Available (was: Open) resubmit > ShutdownHookManager shutdown time needs to be configurable & extended > - > > Key: HADOOP-15679 > URL: https://issues.apache.org/jira/browse/HADOOP-15679 > Project: Hadoop Common > Issue Type: Bug > Components: util >Affects Versions: 3.0.0, 2.8.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-15679-001.patch, HADOOP-15679-002.patch, > HADOOP-15679-002.patch, HADOOP-15679-003.patch, > HADOOP-15679-branch-2-001.patch, HADOOP-15679-branch-2-001.patch > > > HADOOP-12950 added a timeout on shutdowns to avoid problems with hanging > shutdowns. But the timeout is too short for applications where a large flush > of data is needed on shutdown. > A key example of this is Spark apps which save their history to object > stores, where the file close() call triggers an upload of the final local > cached block of data (could be 32+MB), and then execute the final mutipart > commit. > Proposed > # make the default sleep time 30s, not 10s > # make it configurable with a time duration property (with minimum time of > 1s.?) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15679) ShutdownHookManager shutdown time needs to be configurable & extended
[ https://issues.apache.org/jira/browse/HADOOP-15679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-15679: Target Version/s: 3.0.3, 2.10.0, 2.9.2, 2.8.5 (was: 2.10.0, 2.9.2, 3.0.3, 2.8.5) Status: Open (was: Patch Available) > ShutdownHookManager shutdown time needs to be configurable & extended > - > > Key: HADOOP-15679 > URL: https://issues.apache.org/jira/browse/HADOOP-15679 > Project: Hadoop Common > Issue Type: Bug > Components: util >Affects Versions: 3.0.0, 2.8.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-15679-001.patch, HADOOP-15679-002.patch, > HADOOP-15679-002.patch, HADOOP-15679-003.patch, > HADOOP-15679-branch-2-001.patch, HADOOP-15679-branch-2-001.patch > > > HADOOP-12950 added a timeout on shutdowns to avoid problems with hanging > shutdowns. But the timeout is too short for applications where a large flush > of data is needed on shutdown. > A key example of this is Spark apps which save their history to object > stores, where the file close() call triggers an upload of the final local > cached block of data (could be 32+MB), and then execute the final mutipart > commit. > Proposed > # make the default sleep time 30s, not 10s > # make it configurable with a time duration property (with minimum time of > 1s.?) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15679) ShutdownHookManager shutdown time needs to be configurable & extended
[ https://issues.apache.org/jira/browse/HADOOP-15679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-15679: Attachment: HADOOP-15679-branch-2-001.patch > ShutdownHookManager shutdown time needs to be configurable & extended > - > > Key: HADOOP-15679 > URL: https://issues.apache.org/jira/browse/HADOOP-15679 > Project: Hadoop Common > Issue Type: Bug > Components: util >Affects Versions: 2.8.0, 3.0.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-15679-001.patch, HADOOP-15679-002.patch, > HADOOP-15679-002.patch, HADOOP-15679-003.patch, > HADOOP-15679-branch-2-001.patch, HADOOP-15679-branch-2-001.patch > > > HADOOP-12950 added a timeout on shutdowns to avoid problems with hanging > shutdowns. But the timeout is too short for applications where a large flush > of data is needed on shutdown. > A key example of this is Spark apps which save their history to object > stores, where the file close() call triggers an upload of the final local > cached block of data (could be 32+MB), and then execute the final mutipart > commit. > Proposed > # make the default sleep time 30s, not 10s > # make it configurable with a time duration property (with minimum time of > 1s.?) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15679) ShutdownHookManager shutdown time needs to be configurable & extended
[ https://issues.apache.org/jira/browse/HADOOP-15679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-15679: Target Version/s: 3.0.3, 2.10.0, 2.9.2, 2.8.5 (was: 2.10.0, 3.0.3) Status: Open (was: Patch Available) > ShutdownHookManager shutdown time needs to be configurable & extended > - > > Key: HADOOP-15679 > URL: https://issues.apache.org/jira/browse/HADOOP-15679 > Project: Hadoop Common > Issue Type: Bug > Components: util >Affects Versions: 3.0.0, 2.8.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-15679-001.patch, HADOOP-15679-002.patch, > HADOOP-15679-002.patch, HADOOP-15679-003.patch, > HADOOP-15679-branch-2-001.patch > > > HADOOP-12950 added a timeout on shutdowns to avoid problems with hanging > shutdowns. But the timeout is too short for applications where a large flush > of data is needed on shutdown. > A key example of this is Spark apps which save their history to object > stores, where the file close() call triggers an upload of the final local > cached block of data (could be 32+MB), and then execute the final mutipart > commit. > Proposed > # make the default sleep time 30s, not 10s > # make it configurable with a time duration property (with minimum time of > 1s.?) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15679) ShutdownHookManager shutdown time needs to be configurable & extended
[ https://issues.apache.org/jira/browse/HADOOP-15679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-15679: Target Version/s: 3.0.3, 2.10.0, 2.9.2, 2.8.5 (was: 2.10.0, 2.9.2, 3.0.3, 2.8.5) Status: Patch Available (was: Open) branch-2 patch; main difference is HadoopExecutors class doesn't exist. As I backport further, the move to commons-logging will force more change > ShutdownHookManager shutdown time needs to be configurable & extended > - > > Key: HADOOP-15679 > URL: https://issues.apache.org/jira/browse/HADOOP-15679 > Project: Hadoop Common > Issue Type: Bug > Components: util >Affects Versions: 3.0.0, 2.8.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-15679-001.patch, HADOOP-15679-002.patch, > HADOOP-15679-002.patch, HADOOP-15679-003.patch, > HADOOP-15679-branch-2-001.patch > > > HADOOP-12950 added a timeout on shutdowns to avoid problems with hanging > shutdowns. But the timeout is too short for applications where a large flush > of data is needed on shutdown. > A key example of this is Spark apps which save their history to object > stores, where the file close() call triggers an upload of the final local > cached block of data (could be 32+MB), and then execute the final mutipart > commit. > Proposed > # make the default sleep time 30s, not 10s > # make it configurable with a time duration property (with minimum time of > 1s.?) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15679) ShutdownHookManager shutdown time needs to be configurable & extended
[ https://issues.apache.org/jira/browse/HADOOP-15679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-15679: Attachment: HADOOP-15679-branch-2-001.patch > ShutdownHookManager shutdown time needs to be configurable & extended > - > > Key: HADOOP-15679 > URL: https://issues.apache.org/jira/browse/HADOOP-15679 > Project: Hadoop Common > Issue Type: Bug > Components: util >Affects Versions: 2.8.0, 3.0.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-15679-001.patch, HADOOP-15679-002.patch, > HADOOP-15679-002.patch, HADOOP-15679-003.patch, > HADOOP-15679-branch-2-001.patch > > > HADOOP-12950 added a timeout on shutdowns to avoid problems with hanging > shutdowns. But the timeout is too short for applications where a large flush > of data is needed on shutdown. > A key example of this is Spark apps which save their history to object > stores, where the file close() call triggers an upload of the final local > cached block of data (could be 32+MB), and then execute the final mutipart > commit. > Proposed > # make the default sleep time 30s, not 10s > # make it configurable with a time duration property (with minimum time of > 1s.?) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15679) ShutdownHookManager shutdown time needs to be configurable & extended
[ https://issues.apache.org/jira/browse/HADOOP-15679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-15679: Target Version/s: 3.0.3, 2.10.0 (was: 2.10.0, 3.1.2) > ShutdownHookManager shutdown time needs to be configurable & extended > - > > Key: HADOOP-15679 > URL: https://issues.apache.org/jira/browse/HADOOP-15679 > Project: Hadoop Common > Issue Type: Bug > Components: util >Affects Versions: 2.8.0, 3.0.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-15679-001.patch, HADOOP-15679-002.patch, > HADOOP-15679-002.patch, HADOOP-15679-003.patch > > > HADOOP-12950 added a timeout on shutdowns to avoid problems with hanging > shutdowns. But the timeout is too short for applications where a large flush > of data is needed on shutdown. > A key example of this is Spark apps which save their history to object > stores, where the file close() call triggers an upload of the final local > cached block of data (could be 32+MB), and then execute the final mutipart > commit. > Proposed > # make the default sleep time 30s, not 10s > # make it configurable with a time duration property (with minimum time of > 1s.?) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15679) ShutdownHookManager shutdown time needs to be configurable & extended
[ https://issues.apache.org/jira/browse/HADOOP-15679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-15679: Status: Patch Available (was: Open) patch 003; address comments and checkstyle > ShutdownHookManager shutdown time needs to be configurable & extended > - > > Key: HADOOP-15679 > URL: https://issues.apache.org/jira/browse/HADOOP-15679 > Project: Hadoop Common > Issue Type: Bug > Components: util >Affects Versions: 3.0.0, 2.8.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-15679-001.patch, HADOOP-15679-002.patch, > HADOOP-15679-002.patch, HADOOP-15679-003.patch > > > HADOOP-12950 added a timeout on shutdowns to avoid problems with hanging > shutdowns. But the timeout is too short for applications where a large flush > of data is needed on shutdown. > A key example of this is Spark apps which save their history to object > stores, where the file close() call triggers an upload of the final local > cached block of data (could be 32+MB), and then execute the final mutipart > commit. > Proposed > # make the default sleep time 30s, not 10s > # make it configurable with a time duration property (with minimum time of > 1s.?) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15679) ShutdownHookManager shutdown time needs to be configurable & extended
[ https://issues.apache.org/jira/browse/HADOOP-15679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-15679: Attachment: HADOOP-15679-003.patch > ShutdownHookManager shutdown time needs to be configurable & extended > - > > Key: HADOOP-15679 > URL: https://issues.apache.org/jira/browse/HADOOP-15679 > Project: Hadoop Common > Issue Type: Bug > Components: util >Affects Versions: 2.8.0, 3.0.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-15679-001.patch, HADOOP-15679-002.patch, > HADOOP-15679-002.patch, HADOOP-15679-003.patch > > > HADOOP-12950 added a timeout on shutdowns to avoid problems with hanging > shutdowns. But the timeout is too short for applications where a large flush > of data is needed on shutdown. > A key example of this is Spark apps which save their history to object > stores, where the file close() call triggers an upload of the final local > cached block of data (could be 32+MB), and then execute the final mutipart > commit. > Proposed > # make the default sleep time 30s, not 10s > # make it configurable with a time duration property (with minimum time of > 1s.?) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15679) ShutdownHookManager shutdown time needs to be configurable & extended
[ https://issues.apache.org/jira/browse/HADOOP-15679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-15679: Attachment: (was: HADOOP-15679-003.patch) > ShutdownHookManager shutdown time needs to be configurable & extended > - > > Key: HADOOP-15679 > URL: https://issues.apache.org/jira/browse/HADOOP-15679 > Project: Hadoop Common > Issue Type: Bug > Components: util >Affects Versions: 2.8.0, 3.0.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-15679-001.patch, HADOOP-15679-002.patch, > HADOOP-15679-002.patch > > > HADOOP-12950 added a timeout on shutdowns to avoid problems with hanging > shutdowns. But the timeout is too short for applications where a large flush > of data is needed on shutdown. > A key example of this is Spark apps which save their history to object > stores, where the file close() call triggers an upload of the final local > cached block of data (could be 32+MB), and then execute the final mutipart > commit. > Proposed > # make the default sleep time 30s, not 10s > # make it configurable with a time duration property (with minimum time of > 1s.?) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15679) ShutdownHookManager shutdown time needs to be configurable & extended
[ https://issues.apache.org/jira/browse/HADOOP-15679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-15679: Attachment: HADOOP-15679-003.patch > ShutdownHookManager shutdown time needs to be configurable & extended > - > > Key: HADOOP-15679 > URL: https://issues.apache.org/jira/browse/HADOOP-15679 > Project: Hadoop Common > Issue Type: Bug > Components: util >Affects Versions: 2.8.0, 3.0.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-15679-001.patch, HADOOP-15679-002.patch, > HADOOP-15679-002.patch, HADOOP-15679-003.patch > > > HADOOP-12950 added a timeout on shutdowns to avoid problems with hanging > shutdowns. But the timeout is too short for applications where a large flush > of data is needed on shutdown. > A key example of this is Spark apps which save their history to object > stores, where the file close() call triggers an upload of the final local > cached block of data (could be 32+MB), and then execute the final mutipart > commit. > Proposed > # make the default sleep time 30s, not 10s > # make it configurable with a time duration property (with minimum time of > 1s.?) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15679) ShutdownHookManager shutdown time needs to be configurable & extended
[ https://issues.apache.org/jira/browse/HADOOP-15679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-15679: Status: Open (was: Patch Available) > ShutdownHookManager shutdown time needs to be configurable & extended > - > > Key: HADOOP-15679 > URL: https://issues.apache.org/jira/browse/HADOOP-15679 > Project: Hadoop Common > Issue Type: Bug > Components: util >Affects Versions: 3.0.0, 2.8.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-15679-001.patch, HADOOP-15679-002.patch, > HADOOP-15679-002.patch, HADOOP-15679-003.patch > > > HADOOP-12950 added a timeout on shutdowns to avoid problems with hanging > shutdowns. But the timeout is too short for applications where a large flush > of data is needed on shutdown. > A key example of this is Spark apps which save their history to object > stores, where the file close() call triggers an upload of the final local > cached block of data (could be 32+MB), and then execute the final mutipart > commit. > Proposed > # make the default sleep time 30s, not 10s > # make it configurable with a time duration property (with minimum time of > 1s.?) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15679) ShutdownHookManager shutdown time needs to be configurable & extended
[ https://issues.apache.org/jira/browse/HADOOP-15679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-15679: Status: Patch Available (was: Open) > ShutdownHookManager shutdown time needs to be configurable & extended > - > > Key: HADOOP-15679 > URL: https://issues.apache.org/jira/browse/HADOOP-15679 > Project: Hadoop Common > Issue Type: Bug > Components: util >Affects Versions: 3.0.0, 2.8.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-15679-001.patch, HADOOP-15679-002.patch, > HADOOP-15679-002.patch > > > HADOOP-12950 added a timeout on shutdowns to avoid problems with hanging > shutdowns. But the timeout is too short for applications where a large flush > of data is needed on shutdown. > A key example of this is Spark apps which save their history to object > stores, where the file close() call triggers an upload of the final local > cached block of data (could be 32+MB), and then execute the final mutipart > commit. > Proposed > # make the default sleep time 30s, not 10s > # make it configurable with a time duration property (with minimum time of > 1s.?) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15679) ShutdownHookManager shutdown time needs to be configurable & extended
[ https://issues.apache.org/jira/browse/HADOOP-15679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-15679: Status: Open (was: Patch Available) > ShutdownHookManager shutdown time needs to be configurable & extended > - > > Key: HADOOP-15679 > URL: https://issues.apache.org/jira/browse/HADOOP-15679 > Project: Hadoop Common > Issue Type: Bug > Components: util >Affects Versions: 3.0.0, 2.8.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-15679-001.patch, HADOOP-15679-002.patch, > HADOOP-15679-002.patch > > > HADOOP-12950 added a timeout on shutdowns to avoid problems with hanging > shutdowns. But the timeout is too short for applications where a large flush > of data is needed on shutdown. > A key example of this is Spark apps which save their history to object > stores, where the file close() call triggers an upload of the final local > cached block of data (could be 32+MB), and then execute the final mutipart > commit. > Proposed > # make the default sleep time 30s, not 10s > # make it configurable with a time duration property (with minimum time of > 1s.?) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15679) ShutdownHookManager shutdown time needs to be configurable & extended
[ https://issues.apache.org/jira/browse/HADOOP-15679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-15679: Attachment: HADOOP-15679-002.patch > ShutdownHookManager shutdown time needs to be configurable & extended > - > > Key: HADOOP-15679 > URL: https://issues.apache.org/jira/browse/HADOOP-15679 > Project: Hadoop Common > Issue Type: Bug > Components: util >Affects Versions: 2.8.0, 3.0.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-15679-001.patch, HADOOP-15679-002.patch, > HADOOP-15679-002.patch > > > HADOOP-12950 added a timeout on shutdowns to avoid problems with hanging > shutdowns. But the timeout is too short for applications where a large flush > of data is needed on shutdown. > A key example of this is Spark apps which save their history to object > stores, where the file close() call triggers an upload of the final local > cached block of data (could be 32+MB), and then execute the final mutipart > commit. > Proposed > # make the default sleep time 30s, not 10s > # make it configurable with a time duration property (with minimum time of > 1s.?) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15679) ShutdownHookManager shutdown time needs to be configurable & extended
[ https://issues.apache.org/jira/browse/HADOOP-15679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-15679: Status: Patch Available (was: Open) > ShutdownHookManager shutdown time needs to be configurable & extended > - > > Key: HADOOP-15679 > URL: https://issues.apache.org/jira/browse/HADOOP-15679 > Project: Hadoop Common > Issue Type: Bug > Components: util >Affects Versions: 3.0.0, 2.8.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-15679-001.patch, HADOOP-15679-002.patch > > > HADOOP-12950 added a timeout on shutdowns to avoid problems with hanging > shutdowns. But the timeout is too short for applications where a large flush > of data is needed on shutdown. > A key example of this is Spark apps which save their history to object > stores, where the file close() call triggers an upload of the final local > cached block of data (could be 32+MB), and then execute the final mutipart > commit. > Proposed > # make the default sleep time 30s, not 10s > # make it configurable with a time duration property (with minimum time of > 1s.?) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15679) ShutdownHookManager shutdown time needs to be configurable & extended
[ https://issues.apache.org/jira/browse/HADOOP-15679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-15679: Target Version/s: 2.10.0, 3.1.2 (was: 3.1.2) > ShutdownHookManager shutdown time needs to be configurable & extended > - > > Key: HADOOP-15679 > URL: https://issues.apache.org/jira/browse/HADOOP-15679 > Project: Hadoop Common > Issue Type: Bug > Components: util >Affects Versions: 2.8.0, 3.0.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-15679-001.patch, HADOOP-15679-002.patch > > > HADOOP-12950 added a timeout on shutdowns to avoid problems with hanging > shutdowns. But the timeout is too short for applications where a large flush > of data is needed on shutdown. > A key example of this is Spark apps which save their history to object > stores, where the file close() call triggers an upload of the final local > cached block of data (could be 32+MB), and then execute the final mutipart > commit. > Proposed > # make the default sleep time 30s, not 10s > # make it configurable with a time duration property (with minimum time of > 1s.?) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15679) ShutdownHookManager shutdown time needs to be configurable & extended
[ https://issues.apache.org/jira/browse/HADOOP-15679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-15679: Status: Open (was: Patch Available) > ShutdownHookManager shutdown time needs to be configurable & extended > - > > Key: HADOOP-15679 > URL: https://issues.apache.org/jira/browse/HADOOP-15679 > Project: Hadoop Common > Issue Type: Bug > Components: util >Affects Versions: 3.0.0, 2.8.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-15679-001.patch, HADOOP-15679-002.patch > > > HADOOP-12950 added a timeout on shutdowns to avoid problems with hanging > shutdowns. But the timeout is too short for applications where a large flush > of data is needed on shutdown. > A key example of this is Spark apps which save their history to object > stores, where the file close() call triggers an upload of the final local > cached block of data (could be 32+MB), and then execute the final mutipart > commit. > Proposed > # make the default sleep time 30s, not 10s > # make it configurable with a time duration property (with minimum time of > 1s.?) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15679) ShutdownHookManager shutdown time needs to be configurable & extended
[ https://issues.apache.org/jira/browse/HADOOP-15679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-15679: Attachment: HADOOP-15679-002.patch > ShutdownHookManager shutdown time needs to be configurable & extended > - > > Key: HADOOP-15679 > URL: https://issues.apache.org/jira/browse/HADOOP-15679 > Project: Hadoop Common > Issue Type: Bug > Components: util >Affects Versions: 2.8.0, 3.0.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-15679-001.patch, HADOOP-15679-002.patch > > > HADOOP-12950 added a timeout on shutdowns to avoid problems with hanging > shutdowns. But the timeout is too short for applications where a large flush > of data is needed on shutdown. > A key example of this is Spark apps which save their history to object > stores, where the file close() call triggers an upload of the final local > cached block of data (could be 32+MB), and then execute the final mutipart > commit. > Proposed > # make the default sleep time 30s, not 10s > # make it configurable with a time duration property (with minimum time of > 1s.?) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15679) ShutdownHookManager shutdown time needs to be configurable & extended
[ https://issues.apache.org/jira/browse/HADOOP-15679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-15679: Status: Patch Available (was: Open) > ShutdownHookManager shutdown time needs to be configurable & extended > - > > Key: HADOOP-15679 > URL: https://issues.apache.org/jira/browse/HADOOP-15679 > Project: Hadoop Common > Issue Type: Bug > Components: util >Affects Versions: 3.0.0, 2.8.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-15679-001.patch > > > HADOOP-12950 added a timeout on shutdowns to avoid problems with hanging > shutdowns. But the timeout is too short for applications where a large flush > of data is needed on shutdown. > A key example of this is Spark apps which save their history to object > stores, where the file close() call triggers an upload of the final local > cached block of data (could be 32+MB), and then execute the final mutipart > commit. > Proposed > # make the default sleep time 30s, not 10s > # make it configurable with a time duration property (with minimum time of > 1s.?) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15679) ShutdownHookManager shutdown time needs to be configurable & extended
[ https://issues.apache.org/jira/browse/HADOOP-15679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-15679: Target Version/s: 3.1.2 > ShutdownHookManager shutdown time needs to be configurable & extended > - > > Key: HADOOP-15679 > URL: https://issues.apache.org/jira/browse/HADOOP-15679 > Project: Hadoop Common > Issue Type: Bug > Components: util >Affects Versions: 2.8.0, 3.0.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-15679-001.patch > > > HADOOP-12950 added a timeout on shutdowns to avoid problems with hanging > shutdowns. But the timeout is too short for applications where a large flush > of data is needed on shutdown. > A key example of this is Spark apps which save their history to object > stores, where the file close() call triggers an upload of the final local > cached block of data (could be 32+MB), and then execute the final mutipart > commit. > Proposed > # make the default sleep time 30s, not 10s > # make it configurable with a time duration property (with minimum time of > 1s.?) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15679) ShutdownHookManager shutdown time needs to be configurable & extended
[ https://issues.apache.org/jira/browse/HADOOP-15679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-15679: Attachment: HADOOP-15679-001.patch > ShutdownHookManager shutdown time needs to be configurable & extended > - > > Key: HADOOP-15679 > URL: https://issues.apache.org/jira/browse/HADOOP-15679 > Project: Hadoop Common > Issue Type: Bug > Components: util >Affects Versions: 2.8.0, 3.0.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-15679-001.patch > > > HADOOP-12950 added a timeout on shutdowns to avoid problems with hanging > shutdowns. But the timeout is too short for applications where a large flush > of data is needed on shutdown. > A key example of this is Spark apps which save their history to object > stores, where the file close() call triggers an upload of the final local > cached block of data (could be 32+MB), and then execute the final mutipart > commit. > Proposed > # make the default sleep time 30s, not 10s > # make it configurable with a time duration property (with minimum time of > 1s.?) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
