[jira] [Commented] (FLINK-13848) Support “scheduleAtFixedRate/scheduleAtFixedDelay” in RpcEndpoint#MainThreadExecutor
[
https://issues.apache.org/jira/browse/FLINK-13848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17336435#comment-17336435
]
Flink Jira Bot commented on FLINK-13848:
This issue was labeled "stale-major" 7 ago and has not received any updates so
it is being deprioritized. If this ticket is actually Major, please raise the
priority and ask a committer to assign you the issue or revive the public
discussion.
> Support “scheduleAtFixedRate/scheduleAtFixedDelay” in
> RpcEndpoint#MainThreadExecutor
>
>
> Key: FLINK-13848
> URL: https://issues.apache.org/jira/browse/FLINK-13848
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Coordination
>Reporter: Biao Liu
>Priority: Major
> Labels: pull-request-available, stale-major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Currently the methods “scheduleAtFixedRate/scheduleAtFixedDelay" of
> {{RpcEndpoint#MainThreadExecutor}} are not implemented. Because there was no
> requirement on them before.
> Now we are planning to implement these methods to support periodic checkpoint
> triggering.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-13848) Support “scheduleAtFixedRate/scheduleAtFixedDelay” in RpcEndpoint#MainThreadExecutor
[
https://issues.apache.org/jira/browse/FLINK-13848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17328326#comment-17328326
]
Flink Jira Bot commented on FLINK-13848:
This major issue is unassigned and itself and all of its Sub-Tasks have not
been updated for 30 days. So, it has been labeled "stale-major". If this ticket
is indeed "major", please either assign yourself or give an update. Afterwards,
please remove the label. In 7 days the issue will be deprioritized.
> Support “scheduleAtFixedRate/scheduleAtFixedDelay” in
> RpcEndpoint#MainThreadExecutor
>
>
> Key: FLINK-13848
> URL: https://issues.apache.org/jira/browse/FLINK-13848
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Coordination
>Reporter: Biao Liu
>Priority: Major
> Labels: pull-request-available, stale-major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Currently the methods “scheduleAtFixedRate/scheduleAtFixedDelay" of
> {{RpcEndpoint#MainThreadExecutor}} are not implemented. Because there was no
> requirement on them before.
> Now we are planning to implement these methods to support periodic checkpoint
> triggering.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-13848) Support “scheduleAtFixedRate/scheduleAtFixedDelay” in RpcEndpoint#MainThreadExecutor
[
https://issues.apache.org/jira/browse/FLINK-13848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17064761#comment-17064761
]
Biao Liu commented on FLINK-13848:
--
Just an update, we have a work-around solution instead which keeps the timer
for periodic trigger of {{CheckpointCoordinator}} in FLINK-14971. So this
ticket is unnecessary for FLINK-13698. Maybe someday we would come back to this
issue if this feature is needed.
> Support “scheduleAtFixedRate/scheduleAtFixedDelay” in
> RpcEndpoint#MainThreadExecutor
>
>
> Key: FLINK-13848
> URL: https://issues.apache.org/jira/browse/FLINK-13848
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Coordination
>Reporter: Biao Liu
>Assignee: Biao Liu
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Currently the methods “scheduleAtFixedRate/scheduleAtFixedDelay" of
> {{RpcEndpoint#MainThreadExecutor}} are not implemented. Because there was no
> requirement on them before.
> Now we are planning to implement these methods to support periodic checkpoint
> triggering.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-13848) Support “scheduleAtFixedRate/scheduleAtFixedDelay” in RpcEndpoint#MainThreadExecutor
[
https://issues.apache.org/jira/browse/FLINK-13848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16993362#comment-16993362
]
Till Rohrmann commented on FLINK-13848:
---
Great, thanks for letting me know [~SleePy].
> Support “scheduleAtFixedRate/scheduleAtFixedDelay” in
> RpcEndpoint#MainThreadExecutor
>
>
> Key: FLINK-13848
> URL: https://issues.apache.org/jira/browse/FLINK-13848
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Coordination
>Reporter: Biao Liu
>Assignee: Biao Liu
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.11.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Currently the methods “scheduleAtFixedRate/scheduleAtFixedDelay" of
> {{RpcEndpoint#MainThreadExecutor}} are not implemented. Because there was no
> requirement on them before.
> Now we are planning to implement these methods to support periodic checkpoint
> triggering.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-13848) Support “scheduleAtFixedRate/scheduleAtFixedDelay” in RpcEndpoint#MainThreadExecutor
[
https://issues.apache.org/jira/browse/FLINK-13848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16992662#comment-16992662
]
Biao Liu commented on FLINK-13848:
--
[~trohrmann], this is an alternative implementation. [~pnowojski] and me
haven't reached agreement yet. It's probably needed I guess. We will get back
to this discussion after FLINK-13905.
BTW, since FLINK-15132 is a false alarm. This ticket and its parent ticket are
not blocker anymore.
> Support “scheduleAtFixedRate/scheduleAtFixedDelay” in
> RpcEndpoint#MainThreadExecutor
>
>
> Key: FLINK-13848
> URL: https://issues.apache.org/jira/browse/FLINK-13848
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Coordination
>Reporter: Biao Liu
>Assignee: Biao Liu
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.11.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Currently the methods “scheduleAtFixedRate/scheduleAtFixedDelay" of
> {{RpcEndpoint#MainThreadExecutor}} are not implemented. Because there was no
> requirement on them before.
> Now we are planning to implement these methods to support periodic checkpoint
> triggering.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-13848) Support “scheduleAtFixedRate/scheduleAtFixedDelay” in RpcEndpoint#MainThreadExecutor
[
https://issues.apache.org/jira/browse/FLINK-13848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16992592#comment-16992592
]
Till Rohrmann commented on FLINK-13848:
---
Not sure whether this issue is strictly needed for the proper fix of
FLINK-13698. [~SleePy] please clarify.
> Support “scheduleAtFixedRate/scheduleAtFixedDelay” in
> RpcEndpoint#MainThreadExecutor
>
>
> Key: FLINK-13848
> URL: https://issues.apache.org/jira/browse/FLINK-13848
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Coordination
>Reporter: Biao Liu
>Assignee: Biao Liu
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.10.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Currently the methods “scheduleAtFixedRate/scheduleAtFixedDelay" of
> {{RpcEndpoint#MainThreadExecutor}} are not implemented. Because there was no
> requirement on them before.
> Now we are planning to implement these methods to support periodic checkpoint
> triggering.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-13848) Support “scheduleAtFixedRate/scheduleAtFixedDelay” in RpcEndpoint#MainThreadExecutor
[
https://issues.apache.org/jira/browse/FLINK-13848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16961732#comment-16961732
]
Biao Liu commented on FLINK-13848:
--
{quote}> Anyway it's no harm to enrich the MainThreadExecutor.
Kind of there is - having to maintain unused feature.{quote}
Strictly speaking, it's true :)
The good news is that the periodic scheduling methods is already introduced in
`MainThreadExecutor`. It's just not implemented yet. So it's kind of I'm
implementing something already in a plan.
> Support “scheduleAtFixedRate/scheduleAtFixedDelay” in
> RpcEndpoint#MainThreadExecutor
>
>
> Key: FLINK-13848
> URL: https://issues.apache.org/jira/browse/FLINK-13848
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Coordination
>Reporter: Biao Liu
>Assignee: Biao Liu
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.10.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Currently the methods “scheduleAtFixedRate/scheduleAtFixedDelay" of
> {{RpcEndpoint#MainThreadExecutor}} are not implemented. Because there was no
> requirement on them before.
> Now we are planning to implement these methods to support periodic checkpoint
> triggering.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-13848) Support “scheduleAtFixedRate/scheduleAtFixedDelay” in RpcEndpoint#MainThreadExecutor
[
https://issues.apache.org/jira/browse/FLINK-13848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16961152#comment-16961152
]
Biao Liu commented on FLINK-13848:
--
[~pnowojski],
{quote}Do I understand it correctly, that in the current code base this is not
an issue, because all of the A, B and C are executed synchronously from one
method?
{quote}
Yes, you are right, it's not an issue based on current implementation. But
it'll be an issue if we make the {{triggerCheckpoint}} asynchronously.
{quote}have you thought how to resolve the issue from you previous comment
(posted on 30/Aug/19 04:52)? With triggering next "A" only after the previous
"C" completes?
{quote}
That's a good question. I would like to talk about it, although I believe it's
an orthogonal issue. It indeed annoys me and makes the implementation more
complicated.
First of all, we have to guarantee the workflow is "A -> B -> C -> A ...",
because the checkpoint id sent to task should be monotonically increasing.
There are two kinds of triggering logic, the periodic checkpoint and savepoint.
We could avoid competition between periodic checkpoints by triggering
checkpoint manually or failing the later one. But we can't avoid competition
between savepoint and checkpoint anyway. The savepoint could be triggered
anytime and it can't be failed.
In brief, my solution is introducing a queue of trigger request. If the prior
trigger request is not finished, the later one (including checkpoint and
savepoint) will be kept in this queue. The queue will be checked when the prior
trigger request is finished. It would be included in
https://issues.apache.org/jira/browse/FLINK-13905.
> Support “scheduleAtFixedRate/scheduleAtFixedDelay” in
> RpcEndpoint#MainThreadExecutor
>
>
> Key: FLINK-13848
> URL: https://issues.apache.org/jira/browse/FLINK-13848
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Coordination
>Reporter: Biao Liu
>Assignee: Biao Liu
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.10.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Currently the methods “scheduleAtFixedRate/scheduleAtFixedDelay" of
> {{RpcEndpoint#MainThreadExecutor}} are not implemented. Because there was no
> requirement on them before.
> Now we are planning to implement these methods to support periodic checkpoint
> triggering.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-13848) Support “scheduleAtFixedRate/scheduleAtFixedDelay” in RpcEndpoint#MainThreadExecutor
[
https://issues.apache.org/jira/browse/FLINK-13848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16959721#comment-16959721
]
Piotr Nowojski commented on FLINK-13848:
Have you thought how to resolve the issue from you previous comment (posted on
30/Aug/19 04:52)? With triggering next "A" only after the previous "C"
completes?
Do I understand it correctly, that in the current code base this is not an
issue, because all of the A, B and C are executed synchronously from one
method?
> Anyway it's no harm to enrich the {{MainThreadExecutor}}.
Kind of there is - having to maintain unused feature.
> Support “scheduleAtFixedRate/scheduleAtFixedDelay” in
> RpcEndpoint#MainThreadExecutor
>
>
> Key: FLINK-13848
> URL: https://issues.apache.org/jira/browse/FLINK-13848
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Coordination
>Reporter: Biao Liu
>Assignee: Biao Liu
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.10.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Currently the methods “scheduleAtFixedRate/scheduleAtFixedDelay" of
> {{RpcEndpoint#MainThreadExecutor}} are not implemented. Because there was no
> requirement on them before.
> Now we are planning to implement these methods to support periodic checkpoint
> triggering.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-13848) Support “scheduleAtFixedRate/scheduleAtFixedDelay” in RpcEndpoint#MainThreadExecutor
[
https://issues.apache.org/jira/browse/FLINK-13848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16956009#comment-16956009
]
Biao Liu commented on FLINK-13848:
--
I would like to activate this ticket.
There is a discussion that whether it's a good way to schedule trigger manually
of {{CheckpointCoordinator}} (in other PR). We probably still need periodic
scheduling in main thread.
Anyway it's no harm to enrich the {{MainThreadExecutor}}.
> Support “scheduleAtFixedRate/scheduleAtFixedDelay” in
> RpcEndpoint#MainThreadExecutor
>
>
> Key: FLINK-13848
> URL: https://issues.apache.org/jira/browse/FLINK-13848
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Coordination
>Reporter: Biao Liu
>Assignee: Biao Liu
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.10.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Currently the methods “scheduleAtFixedRate/scheduleAtFixedDelay" of
> {{RpcEndpoint#MainThreadExecutor}} are not implemented. Because there was no
> requirement on them before.
> Now we are planning to implement these methods to support periodic checkpoint
> triggering.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-13848) Support “scheduleAtFixedRate/scheduleAtFixedDelay” in RpcEndpoint#MainThreadExecutor
[
https://issues.apache.org/jira/browse/FLINK-13848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16919552#comment-16919552
]
Till Rohrmann commented on FLINK-13848:
---
Yes this makes sense [~SleePy]. Thanks a lot for the explanation.
> Support “scheduleAtFixedRate/scheduleAtFixedDelay” in
> RpcEndpoint#MainThreadExecutor
>
>
> Key: FLINK-13848
> URL: https://issues.apache.org/jira/browse/FLINK-13848
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Coordination
>Reporter: Biao Liu
>Priority: Major
> Fix For: 1.10.0
>
>
> Currently the methods “scheduleAtFixedRate/scheduleAtFixedDelay" of
> {{RpcEndpoint#MainThreadExecutor}} are not implemented. Because there was no
> requirement on them before.
> Now we are planning to implement these methods to support periodic checkpoint
> triggering.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (FLINK-13848) Support “scheduleAtFixedRate/scheduleAtFixedDelay” in RpcEndpoint#MainThreadExecutor
[
https://issues.apache.org/jira/browse/FLINK-13848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16919138#comment-16919138
]
Biao Liu commented on FLINK-13848:
--
[~till.rohrmann]
Sorry I didn't describe it clearly.
In this refactoring, the {{triggerCheckpoint}} would be separated into several
stages. For example A -> B -> C. A is "start triggering", running in main
thread. B is doing some heavy initializations, running in IO threads. C is
sending the trigger message to tasks, running in main thread. The workflow
should be like:
# First A is triggered and executed in main thread. A launches the B into IO
threads, and returns immediately.
# B is executed in IO thread. When it's finished, a callback of triggering C
executes (maybe through {{CompletableFuture#thenRunAsync}} with main thread
executor).
# C is executed in main thread. After then the next A is scheduled after a
delay.
It's A -> B -> C -> A ... There would be no competition. That's what we really
want.
But for the {{scheduleAtFixedDelay}} way, what we scheduled here is just A. The
next A would be scheduled when the prior A is finished.
It's A -> B -> C
\
-> A -> B -> C
So I think we should abandon this ticket. Or convert it to a normal issue not a
subtask of this refactoring. What do you think?
> Support “scheduleAtFixedRate/scheduleAtFixedDelay” in
> RpcEndpoint#MainThreadExecutor
>
>
> Key: FLINK-13848
> URL: https://issues.apache.org/jira/browse/FLINK-13848
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Coordination
>Reporter: Biao Liu
>Priority: Major
> Fix For: 1.10.0
>
>
> Currently the methods “scheduleAtFixedRate/scheduleAtFixedDelay" of
> {{RpcEndpoint#MainThreadExecutor}} are not implemented. Because there was no
> requirement on them before.
> Now we are planning to implement these methods to support periodic checkpoint
> triggering.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (FLINK-13848) Support “scheduleAtFixedRate/scheduleAtFixedDelay” in RpcEndpoint#MainThreadExecutor
[
https://issues.apache.org/jira/browse/FLINK-13848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16918749#comment-16918749
]
Till Rohrmann commented on FLINK-13848:
---
I'm not sure whether I fully understand why the implementation of
{{scheduleAtFixedDelay}} won't work [~SleePy]. But I'm also not opposed to
schedule the next trigger task from within the previous one.
> Support “scheduleAtFixedRate/scheduleAtFixedDelay” in
> RpcEndpoint#MainThreadExecutor
>
>
> Key: FLINK-13848
> URL: https://issues.apache.org/jira/browse/FLINK-13848
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Coordination
>Reporter: Biao Liu
>Priority: Major
> Fix For: 1.10.0
>
>
> Currently the methods “scheduleAtFixedRate/scheduleAtFixedDelay" of
> {{RpcEndpoint#MainThreadExecutor}} are not implemented. Because there was no
> requirement on them before.
> Now we are planning to implement these methods to support periodic checkpoint
> triggering.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (FLINK-13848) Support “scheduleAtFixedRate/scheduleAtFixedDelay” in RpcEndpoint#MainThreadExecutor
[
https://issues.apache.org/jira/browse/FLINK-13848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16918535#comment-16918535
]
Biao Liu commented on FLINK-13848:
--
Hi [~till.rohrmann],
I just found this issue could not satisfy our requirement. In the discussion
on design doc, we want to avoid competition between different triggers. Our
target is making triggers processed one by one. {{"scheduleAtFixedDelay"}}
seems to be the way we want. The {{"scheduleAtFixedDelay"}} schedules
{{runnable}} one after the other. But here the {{runnable}} of triggering is
asynchronous. The {{runnable}} is finished does not mean the triggering is also
finished.
I think we have to go back to the original plan, doing the triggering manually
(triggering the next one when the future of prior trigger is done), not through
{{MainThreadExecutor#scheduleAtFixedDelay}}.
> Support “scheduleAtFixedRate/scheduleAtFixedDelay” in
> RpcEndpoint#MainThreadExecutor
>
>
> Key: FLINK-13848
> URL: https://issues.apache.org/jira/browse/FLINK-13848
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Coordination
>Reporter: Biao Liu
>Priority: Major
> Fix For: 1.10.0
>
>
> Currently the methods “scheduleAtFixedRate/scheduleAtFixedDelay" of
> {{RpcEndpoint#MainThreadExecutor}} are not implemented. Because there was no
> requirement on them before.
> Now we are planning to implement these methods to support periodic checkpoint
> triggering.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (FLINK-13848) Support “scheduleAtFixedRate/scheduleAtFixedDelay” in RpcEndpoint#MainThreadExecutor
[
https://issues.apache.org/jira/browse/FLINK-13848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16916617#comment-16916617
]
Xiaogang Shi commented on FLINK-13848:
--
[~SleePy] Sorry that I misunderstand your second question.
You are right that we can use delayed scheduling of {{MainThreadExecutor}} to
achieve periodically scheduling. But it requires two messages sent to the actor
for each triggering. It may affect the throughput when the rpc endpoint is
flooded.
> Support “scheduleAtFixedRate/scheduleAtFixedDelay” in
> RpcEndpoint#MainThreadExecutor
>
>
> Key: FLINK-13848
> URL: https://issues.apache.org/jira/browse/FLINK-13848
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Coordination
>Reporter: Biao Liu
>Priority: Major
> Fix For: 1.10.0
>
>
> Currently the methods “scheduleAtFixedRate/scheduleAtFixedDelay" of
> {{RpcEndpoint#MainThreadExecutor}} are not implemented. Because there was no
> requirement on them before.
> Now we are planning to implement these methods to support periodic checkpoint
> triggering.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (FLINK-13848) Support “scheduleAtFixedRate/scheduleAtFixedDelay” in RpcEndpoint#MainThreadExecutor
[
https://issues.apache.org/jira/browse/FLINK-13848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16916614#comment-16916614
]
Biao Liu commented on FLINK-13848:
--
Thanks for feedback. [~xiaogang.shi]
1. We could get back to this topic after doing some testings myself. Thanks for
explanation.
2. What I mean is that it seems that we could support periodic scheduling
without involving any other {{ScheduledExecutorService}}. Just using delayed
scheduling of {{MainThreadExecutor}}. Am I misunderstand your description of
"use a non-main-threaded ScheduledExecutorService together with runAsync"?
> Support “scheduleAtFixedRate/scheduleAtFixedDelay” in
> RpcEndpoint#MainThreadExecutor
>
>
> Key: FLINK-13848
> URL: https://issues.apache.org/jira/browse/FLINK-13848
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Coordination
>Reporter: Biao Liu
>Priority: Major
> Fix For: 1.10.0
>
>
> Currently the methods “scheduleAtFixedRate/scheduleAtFixedDelay" of
> {{RpcEndpoint#MainThreadExecutor}} are not implemented. Because there was no
> requirement on them before.
> Now we are planning to implement these methods to support periodic checkpoint
> triggering.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (FLINK-13848) Support “scheduleAtFixedRate/scheduleAtFixedDelay” in RpcEndpoint#MainThreadExecutor
[
https://issues.apache.org/jira/browse/FLINK-13848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16916607#comment-16916607
]
Xiaogang Shi commented on FLINK-13848:
--
1. Yes. If the task message is already sent to the actor's mailbox, it's very
difficult to prohibit it from being executed by the actor. Using a boolean flag
which can be accessed by returned ScheduledFuture may be a solution, but that
makes the cancellation very complicated.
2. Most rpc endpoints are already equipped with {{ScheduledExecutorService}}
which can be used to achieve periodical triggering. If you take a look at the
implementation, {{ScheduledExecutorService}} implements periodically scheduling
by scheduling next triggering after the completion of current triggering.
> Support “scheduleAtFixedRate/scheduleAtFixedDelay” in
> RpcEndpoint#MainThreadExecutor
>
>
> Key: FLINK-13848
> URL: https://issues.apache.org/jira/browse/FLINK-13848
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Coordination
>Reporter: Biao Liu
>Priority: Major
> Fix For: 1.10.0
>
>
> Currently the methods “scheduleAtFixedRate/scheduleAtFixedDelay" of
> {{RpcEndpoint#MainThreadExecutor}} are not implemented. Because there was no
> requirement on them before.
> Now we are planning to implement these methods to support periodic checkpoint
> triggering.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (FLINK-13848) Support “scheduleAtFixedRate/scheduleAtFixedDelay” in RpcEndpoint#MainThreadExecutor
[
https://issues.apache.org/jira/browse/FLINK-13848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16916429#comment-16916429
]
Biao Liu commented on FLINK-13848:
--
Hi [~xiaogang.shi],
Thanks a lot for sharing valuable experience!
>From my understanding, the first choice is via Akka dispatcher/scheduler. We
>could also support period scheduling manually, however IMO reinventing the
>wheel is not the best option.
I'm very interested in the issues you have met. I have a couple of questions on
these issues.
1. "we can only cancel the scheduling of task messages in Akka, but fail to
cancel those task messages already sent".
I didn't get this. Do you mean you could not cancel the message which is
already scheduled?
2. "we decided to use a non-main-threaded ScheduledExecutorService together
with runAsync to achieve periodic triggering in main thread"
Is it necessary that involving another {{ScheduledExecutorService}}? Could it
be satisfied by scheduling next trigger when processing the current triggering?
> Support “scheduleAtFixedRate/scheduleAtFixedDelay” in
> RpcEndpoint#MainThreadExecutor
>
>
> Key: FLINK-13848
> URL: https://issues.apache.org/jira/browse/FLINK-13848
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Coordination
>Reporter: Biao Liu
>Priority: Major
> Fix For: 1.10.0
>
>
> Currently the methods “scheduleAtFixedRate/scheduleAtFixedDelay" of
> {{RpcEndpoint#MainThreadExecutor}} are not implemented. Because there was no
> requirement on them before.
> Now we are planning to implement these methods to support periodic checkpoint
> triggering.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (FLINK-13848) Support “scheduleAtFixedRate/scheduleAtFixedDelay” in RpcEndpoint#MainThreadExecutor
[
https://issues.apache.org/jira/browse/FLINK-13848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16915689#comment-16915689
]
Xiaogang Shi commented on FLINK-13848:
--
Hi [~SleePy], what's your detailed plan on implementing these methods?
Once we attempted to implement these methods in our private branches. I am very
happy to share some experience here.
Our implementation is similar to that in java's {{ScheduledThreadPoolExecutor}}
except that new tasks are submitted via Akka's dispatcher. One problem we
encountered is that we can only cancel the scheduling of task messages in Akka,
but fail to cancel those task messages already sent. Finally, some tasks are
still be executed after the {{ScheduledFuture}} is cancelled, leading to some
weird concurrent behaviors.
Given that few benefits are brought by Akka's dispatcher, we decided to use a
non-main-threaded {{ScheduledExecutorService}} together with {{runAsync}} to
achieve periodic triggering in main thread. The new implementation suffers from
the same problem, but is much simpler.
> Support “scheduleAtFixedRate/scheduleAtFixedDelay” in
> RpcEndpoint#MainThreadExecutor
>
>
> Key: FLINK-13848
> URL: https://issues.apache.org/jira/browse/FLINK-13848
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Coordination
>Reporter: Biao Liu
>Priority: Major
> Fix For: 1.10.0
>
>
> Currently the methods “scheduleAtFixedRate/scheduleAtFixedDelay" of
> {{RpcEndpoint#MainThreadExecutor}} are not implemented. Because there was no
> requirement on them before.
> Now we are planning to implement these methods to support periodic checkpoint
> triggering.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
