Hi,
did you create a Jira issue for this? (I'm just getting up to speed after
vacation so sorry if you already did this, I didn't yet read the Jira mail.)
Cheers,
Aljoscah
On Fri, 6 Jan 2017 at 19:08 Stephan Ewen <se...@apache.org> wrote:
> I think you are right, enabling checkpointing should not override the
> cluster settings per se.
>
> This is probably an unwanted artifact of the was that configuration
> currently works: Setting explicitly set in the program trump the
> cluster-defaults (in the config). Since activating checkpointing sets a
> strategy in the ExecutionConfig (program), it overrides the cluster default.
>
> It is definitely not intended in that case. For that specific case, it
> makes to simply leave the restart strategy "undefined" and use the "fixed
> delay" one at runtime if none other is specified.
>
> Stephan
>
>
>
>
> On Fri, Jan 6, 2017 at 6:44 PM, Shannon Carey <sca...@expedia.com> wrote:
>
> I think I figured it out: the problem is due to Flink's behavior when a
> job has checkpointing enabled.
>
> When the job graph is created, if checkpointing is enabled but a restart
> strategy hasn't been programmatically configured, Flink changes the job
> graph's execution config to use the fixed delay restart strategy. That gets
> serialized with the job graph. Then, when the JobManager deserializes the
> execution config, it sees that there's a restart strategy configured for
> the job and uses that instead of using the restart strategy that's
> configured on the cluster.
>
> Clearly, the documentation definitely needs to be adjusted. Maybe I can
> add some changes to https://github.com/apache/flink/pull/3059
>
> However, should we also consider some implementation changes? Is it
> intentional that enabling checkpoint overrides the restart strategy set on
> the cluster, and that the only way to control the restart strategy on a
> checkpointed job is to set it programmatically? If not, then would it be
> reasonable to only set fixed-delay restart strategy if checkpointing is
> enabled AND the cluster doesn't explicitly configure it? Flink would no
> longer be use the execution config to control the strategy, but would
> instead do it in the JobManager#submitJob().
>
> -Shannon
>
> From: Shannon Carey <sca...@expedia.com>
> Date: Thursday, January 5, 2017 at 1:50 PM
> To: "user@flink.apache.org" <user@flink.apache.org>
> Subject: failure-rate restart strategy not working?
>
> I recently updated my cluster with the following config:
>
> restart-strategy: failure-rate
> restart-strategy.failure-rate.max-failures-per-interval: 3
> restart-strategy.failure-rate.failure-rate-interval: 5 min
> restart-strategy.failure-rate.delay: 10 s
>
> I see the settings inside the JobManager web UI, as expected. I am not
> setting the restart-strategy programmatically, but the job does have
> checkpointing enabled.
>
> However, if I launch a job that (intentionally) fails every 10 seconds by
> throwing a RuntimeException, it continues to restart beyond the limit of 3
> failures.
>
> Does anyone know why this might be happening? Any ideas of things I could
> check?
>
> Thanks!
> Shannon
>
>
>
