Re: create savepoint on bounded source in streaming mode

[Dawid Wysakowicz]

Hi Shawn,

You could also take a look at the hybrid source[1]

Best,

Dawid

[1]https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/datastream/hybridsource/

On 26/01/2022 08:39, Guowei Ma wrote:

Hi Shawn

Or you could see the state process api[1], which might be helpful.

Thanks for your sharing but I have another little question:

Currently Flink can not trigger the sp at the end of the input. An alternative way might be that you need to develop a customized source, which triggers a savepoint when it notices that all the input split has been handled. I think you need to process all the historical events to rebuild the correct state. So there might be no gain even if you periodically create a savepoint. So why did you need to "rebuild" the state periodically? Do I miss something? [1] https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/libs/state_processor_api/

Best,
Guowei


On Wed, Jan 26, 2022 at 2:17 PM Shawn Du <shawn...@neulion.com.cn> wrote:

         our application is stateful. processing live events depends
    on the state. but for kinds of reason, we need rebuild the state.
    it will be very costly to replay all data.
         our historical events data are stored in s3. so we want to
    create states/savepoints periodically so that we can rebuild the
    state from a point. we call this as a bootstrap process.
         any ideas?

         Thanks.

        ------------------------------------------------------------------
        Sender:Guowei Ma <guowei....@gmail.com>
        Sent At:2022 Jan. 26 (Wed.) 14:04
        Recipient:Shawn Du <shawn...@neulion.com.cn>
        Cc:user <user@flink.apache.org>
        Subject:Re: create savepoint on bounded source in streaming mode

        Hi, Shawn
        I think Flink does not support this mechanism yet.
        Would you like to share the scenario in which you need this
        savepoint at the end of the bounded input?
        Best,
        Guowei


        On Wed, Jan 26, 2022 at 1:50 PM Shawn Du
        <shawn...@neulion.com.cn> wrote:
        Hi experts,

        assume I have several files and I want replay these files in
        order in streaming mode and create a savepoint when files play
        at the end. it is possible?
        I wrote a simple test app, and job are finished when source is
        at the end. I have no chance to creat a savepoint. please help.

        Thanks
        Shawn

OpenPGP_signature
Description: OpenPGP digital signature