[jira] [Commented] (FLINK-19698) Add close() method and onCheckpointComplete() to the Source.
[
https://issues.apache.org/jira/browse/FLINK-19698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17227416#comment-17227416
]
Xintong Song commented on FLINK-19698:
--
[~sewen],
Do you mean this issue on 1.11 is blocked on the FLIP-27 backportings?
> Add close() method and onCheckpointComplete() to the Source.
>
>
> Key: FLINK-19698
> URL: https://issues.apache.org/jira/browse/FLINK-19698
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / Common
>Affects Versions: 1.11.2
>Reporter: Jiangjie Qin
>Assignee: Jiangjie Qin
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.12.0, 1.11.3
>
>
> Right now there are some caveats to the new Source API. From the
> implementation of some connectors. We would like to make the following
> improvements to the current Source API.
> # Add the following method to the {{SplitReader}} API.
> {{public void close() throws Exception；}}
> This method allows the SplitReader implementations to be closed properly when
> the split fetcher exits.
> # Add the following method to the {{SourceReader}} API.
> {{public void checkpointComplete(long checkpointId);}}
> This method allows the {{SourceReader}} to take some cleanup / reporting
> actions when a checkpoint has been successfully taken.
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-19698) Add close() method and onCheckpointComplete() to the Source.
[
https://issues.apache.org/jira/browse/FLINK-19698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17227290#comment-17227290
]
Stephan Ewen commented on FLINK-19698:
--
I think only the close() part should be merged to 1.11
The source changes would best be picked in order, otherwise we'll have a lot of
merge conflicts.
> Add close() method and onCheckpointComplete() to the Source.
>
>
> Key: FLINK-19698
> URL: https://issues.apache.org/jira/browse/FLINK-19698
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / Common
>Affects Versions: 1.11.2
>Reporter: Jiangjie Qin
>Assignee: Jiangjie Qin
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.12.0, 1.11.3
>
>
> Right now there are some caveats to the new Source API. From the
> implementation of some connectors. We would like to make the following
> improvements to the current Source API.
> # Add the following method to the {{SplitReader}} API.
> {{public void close() throws Exception；}}
> This method allows the SplitReader implementations to be closed properly when
> the split fetcher exits.
> # Add the following method to the {{SourceReader}} API.
> {{public void checkpointComplete(long checkpointId);}}
> This method allows the {{SourceReader}} to take some cleanup / reporting
> actions when a checkpoint has been successfully taken.
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-19698) Add close() method and onCheckpointComplete() to the Source.
[
https://issues.apache.org/jira/browse/FLINK-19698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17227250#comment-17227250
]
Xintong Song commented on FLINK-19698:
--
Hi [~becket_qin],
I find the fix for this issue has been merged into the master branch, but not
yet into the release-1.11 branch. Anything blocks it from being merged to 1.11?
> Add close() method and onCheckpointComplete() to the Source.
>
>
> Key: FLINK-19698
> URL: https://issues.apache.org/jira/browse/FLINK-19698
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / Common
>Affects Versions: 1.11.2
>Reporter: Jiangjie Qin
>Assignee: Jiangjie Qin
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.12.0, 1.11.3
>
>
> Right now there are some caveats to the new Source API. From the
> implementation of some connectors. We would like to make the following
> improvements to the current Source API.
> # Add the following method to the {{SplitReader}} API.
> {{public void close() throws Exception；}}
> This method allows the SplitReader implementations to be closed properly when
> the split fetcher exits.
> # Add the following method to the {{SourceReader}} API.
> {{public void checkpointComplete(long checkpointId);}}
> This method allows the {{SourceReader}} to take some cleanup / reporting
> actions when a checkpoint has been successfully taken.
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-19698) Add close() method and onCheckpointComplete() to the Source.
[
https://issues.apache.org/jira/browse/FLINK-19698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17226772#comment-17226772
]
Jiangjie Qin commented on FLINK-19698:
--
Merged to master:
7ea3a6040dafd15733162447c0e3fe71009c3ed3
> Add close() method and onCheckpointComplete() to the Source.
>
>
> Key: FLINK-19698
> URL: https://issues.apache.org/jira/browse/FLINK-19698
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / Common
>Affects Versions: 1.11.2
>Reporter: Jiangjie Qin
>Assignee: Jiangjie Qin
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.12.0, 1.11.3
>
>
> Right now there are some caveats to the new Source API. From the
> implementation of some connectors. We would like to make the following
> improvements to the current Source API.
> # Add the following method to the {{SplitReader}} API.
> {{public void close() throws Exception；}}
> This method allows the SplitReader implementations to be closed properly when
> the split fetcher exits.
> # Add the following method to the {{SourceReader}} API.
> {{public void checkpointComplete(long checkpointId);}}
> This method allows the {{SourceReader}} to take some cleanup / reporting
> actions when a checkpoint has been successfully taken.
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-19698) Add close() method and onCheckpointComplete() to the Source.
[
https://issues.apache.org/jira/browse/FLINK-19698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17223432#comment-17223432
]
Xintong Song commented on FLINK-19698:
--
[~stevenz3wu],
I labeled this ticket as blocker for 1.11.3, as you have mentioned on the
mailing list.
> Add close() method and onCheckpointComplete() to the Source.
>
>
> Key: FLINK-19698
> URL: https://issues.apache.org/jira/browse/FLINK-19698
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / Common
>Affects Versions: 1.11.2
>Reporter: Jiangjie Qin
>Assignee: Jiangjie Qin
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.11.3
>
>
> Right now there are some caveats to the new Source API. From the
> implementation of some connectors. We would like to make the following
> improvements to the current Source API.
> # Add the following method to the {{SplitReader}} API.
> {{public void close() throws Exception；}}
> This method allows the SplitReader implementations to be closed properly when
> the split fetcher exits.
> # Add the following method to the {{SourceReader}} API.
> {{public void checkpointComplete(long checkpointId);}}
> This method allows the {{SourceReader}} to take some cleanup / reporting
> actions when a checkpoint has been successfully taken.
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-19698) Add close() method and onCheckpointComplete() to the Source.
[
https://issues.apache.org/jira/browse/FLINK-19698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17219664#comment-17219664
]
Stephan Ewen commented on FLINK-19698:
--
Fair question.
So far, there was no notion of checkpoints, just split assignment, which
generalized work assignment and checkpoint restoring.
The explicit checkpoint hook breaks this. In batch processing, it would never
be called, as I understand it.
One might argue that it should be called at the end of batch processing.
Similar in how the new sink API unifies "committing semantics" between "commit
on complete checkpoint" (streaming) and "commit on job end" (batch).
I think in practice, this "commit on checkpoint" is going to be used in a
critical way (needed for correct behavior) by sources that cannot be unified
(Rabbit MQ). And for sources that can be unified (Kafka) it will be used mainly
for "visibility purposes", like committing offsets to a consumer group such
that external tools can monitor the lag.
Given that, I'd be fin with ignoring this in batch execution for now.
> Add close() method and onCheckpointComplete() to the Source.
>
>
> Key: FLINK-19698
> URL: https://issues.apache.org/jira/browse/FLINK-19698
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / Common
>Affects Versions: 1.11.2
>Reporter: Jiangjie Qin
>Assignee: Jiangjie Qin
>Priority: Major
>
> Right now there are some caveats to the new Source API. From the
> implementation of some connectors. We would like to make the following
> improvements to the current Source API.
> # Add the following method to the {{SplitReader}} API.
> {{public void close() throws Exception；}}
> This method allows the SplitReader implementations to be closed properly when
> the split fetcher exits.
> # Add the following method to the {{SourceReader}} API.
> {{public void checkpointComplete(long checkpointId);}}
> This method allows the {{SourceReader}} to take some cleanup / reporting
> actions when a checkpoint has been successfully taken.
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-19698) Add close() method and onCheckpointComplete() to the Source.
[
https://issues.apache.org/jira/browse/FLINK-19698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17217111#comment-17217111
]
Steven Zhen Wu commented on FLINK-19698:
Since FLIP-27 is the unified source interface for both streaming and batch
modes, what does it the checkpoint method mean for batch mode?
> Add close() method and onCheckpointComplete() to the Source.
>
>
> Key: FLINK-19698
> URL: https://issues.apache.org/jira/browse/FLINK-19698
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / Common
>Affects Versions: 1.11.2
>Reporter: Jiangjie Qin
>Assignee: Jiangjie Qin
>Priority: Major
>
> Right now there are some caveats to the new Source API. From the
> implementation of some connectors. We would like to make the following
> improvements to the current Source API.
> # Add the following method to the {{SplitReader}} API.
> {{public void close() throws Exception；}}
> This method allows the SplitReader implementations to be closed properly when
> the split fetcher exits.
> # Add the following method to the {{SourceReader}} API.
> {{public void checkpointComplete(long checkpointId);}}
> This method allows the {{SourceReader}} to take some cleanup / reporting
> actions when a checkpoint has been successfully taken.
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-19698) Add close() method and onCheckpointComplete() to the Source.
[
https://issues.apache.org/jira/browse/FLINK-19698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17217031#comment-17217031
]
Stephan Ewen commented on FLINK-19698:
--
+1 to these two changes, they are good and important.
> Add close() method and onCheckpointComplete() to the Source.
>
>
> Key: FLINK-19698
> URL: https://issues.apache.org/jira/browse/FLINK-19698
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / Common
>Affects Versions: 1.11.2
>Reporter: Jiangjie Qin
>Assignee: Jiangjie Qin
>Priority: Major
>
> Right now there are some caveats to the new Source API. From the
> implementation of some connectors. We would like to make the following
> improvements to the current Source API.
> # Add the following method to the {{SplitReader}} API.
> {{public void close() throws Exception；}}
> This method allows the SplitReader implementations to be closed properly when
> the split fetcher exits.
> # Add the following method to the {{SourceReader}} API.
> {{public void checkpointComplete(long checkpointId);}}
> This method allows the {{SourceReader}} to take some cleanup / reporting
> actions when a checkpoint has been successfully taken.
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-19698) Add close() method and onCheckpointComplete() to the Source.
[
https://issues.apache.org/jira/browse/FLINK-19698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17216393#comment-17216393
]
Jiangjie Qin commented on FLINK-19698:
--
[~sewen] What do you think about these two changes? I'll reply to the FLIP-27
voting thread to let the folks who have voted to know these API change to the
Source API and update the FLIP-27 WIKI page once we reach consensus in this
ticket.
> Add close() method and onCheckpointComplete() to the Source.
>
>
> Key: FLINK-19698
> URL: https://issues.apache.org/jira/browse/FLINK-19698
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / Common
>Affects Versions: 1.11.2
>Reporter: Jiangjie Qin
>Assignee: Jiangjie Qin
>Priority: Major
>
> Right now there are some caveats to the new Source API. From the
> implementation of some connectors. We would like to make the following
> improvements to the current Source API.
> # Add the following method to the {{SplitReader}} API.
> {{public void close() throws Exception；}}
> This method allows the SplitReader implementations to be closed properly when
> the split fetcher exits.
> # Add the following method to the {{SourceReader}} API.
> {{public void checkpointComplete(long checkpointId);}}
> This method allows the {{SourceReader}} to take some cleanup / reporting
> actions when a checkpoint has been successfully taken.
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
