Re: Flink TCP custom source - secured server socket

[Jan Lukavský]

Hi,

in plain TCP socket, there is no 'state' you can return to when 
restoring from checkpoint. All you can do is reopening the socket. You 
would have to ensure fault-tolerance by some (custom) higher-level 
protocol you implement, including persistent storage, replication, etc. 
You will soon see you are actually implementing a commit-log like Apache 
Kafka, so it might be better to use Kafka directly. If your clients need 
to send data over plain TCP socket, you can receive it over TCP (with a 
custom TCP server application), store to Kafka and use Kafka as a data 
source for your Flink application.

Best,

 Jan

On 7/1/23 05:20, Kamal Mittal wrote:

Hello,

Thanks for your suggestion but please confirm below.

Is it the case that TCP socket source job can’t be restored from last 
checkpoint?

Rgds,

Kamal

*From:*Jan Lukavský <je...@seznam.cz>
*Sent:* 29 June 2023 06:18 PM
*To:* user@flink.apache.org
*Subject:* Re: Flink TCP custom source - secured server socket

> ... a state backward in (processing) time ...
(of course not processing, I meant to say event time)

On 6/29/23 14:45, Jan Lukavský wrote:

    Hi Kamal,

    you probably have several options:

     a) bundle your private key and certificate into your Flink
    application's jar (not recommended, your service's private key
    will have to be not exactly "private")
     b) create a service which will provide certificate for your
    service during runtime (e.g. ACME based or similar)

    I have a different note, though. Flink (or for that matters any
    streaming engine, I'm more focused on Apache Beam) heavily relies
    on the ability of sources to restore a state backward in
    (processing) time. That is definitely not the case of a plain TCP
    socket. It is likely you will experience data-loss issues with
    this solution (regardless of security). This might be okay for
    you, I just felt it would be good to stress this out.

    Best,

     Jan

    On 6/29/23 12:53, Kamal Mittal via user wrote:

        Hello Community,

        I have created TCP stream custom source and reading data from
        TCP stream source.

        But this TCP connection needs to be secured i.e. SSL based,
        query is how to configure/provide certificates via Flink for
        Client-Server secured TCP connection?

        Rgds,

        Kamal