Re: Kinesis Connector and Savepoint/Checkpoint restore.

2019-11-21 Thread Ravi Bhushan Ratnakar
han the actual number of shards which results >>>>> into >>>>> scheduling those shards with earliest stream position. >>>>> I am suspecting that somehow the checkpoint is missing state for some >>>>> of the shards. But if this is the case then

Re: Kinesis Connector and Savepoint/Checkpoint restore.

2019-11-21 Thread Congxian Qiu
this is the case then that checkpoint should had >>>> failed. >>>> >>>> Any further information to resolve this issue would be highly >>>> appreciated... >>>> >>>> Regards, >>>> Ravi >>>> >>>>

Re: Kinesis Connector and Savepoint/Checkpoint restore.

2019-10-16 Thread Steven Nelson
would be highly >>> appreciated... >>> >>> Regards, >>> Ravi >>> >>> On Wed, Oct 16, 2019 at 5:57 AM Yun Tang wrote: >>> >>>> Hi Steven >>>> >>>> If you restore savepoint/checkpoint successfully, I think

Re: Kinesis Connector and Savepoint/Checkpoint restore.

2019-10-16 Thread Ravi Bhushan Ratnakar
gt; to the shard wasn't discovered in the previous run, therefore it would be >>> consumed from the beginning. Please refer to the implementation here: [1] >>> >>> [1] >>> https://github.com/apache/flink/blob/2c411686d23f456cdc502abf1c6b97a61070a17d/flink-co

Re: Kinesis Connector and Savepoint/Checkpoint restore.

2019-10-16 Thread Steven Nelson
from the beginning. Please refer to the implementation here: [1] >> >> [1] >> https://github.com/apache/flink/blob/2c411686d23f456cdc502abf1c6b97a61070a17d/flink-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/FlinkKinesisConsumer.java#L307

Re: Kinesis Connector and Savepoint/Checkpoint restore.

2019-10-15 Thread Ravi Bhushan Ratnakar
gt; Best > Yun Tang > -- > *From:* Steven Nelson > *Sent:* Wednesday, October 16, 2019 4:31 > *To:* user > *Subject:* Kinesis Connector and Savepoint/Checkpoint restore. > > Hello, we currently use Flink 1.9.0 with Kinesis to process data. > > We

Re: Kinesis Connector and Savepoint/Checkpoint restore.

2019-10-15 Thread Yun Tang
/blob/2c411686d23f456cdc502abf1c6b97a61070a17d/flink-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/FlinkKinesisConsumer.java#L307 Best Yun Tang From: Steven Nelson Sent: Wednesday, October 16, 2019 4:31 To: user Subject: Kinesis Connecto

Kinesis Connector and Savepoint/Checkpoint restore.

2019-10-15 Thread Steven Nelson
Hello, we currently use Flink 1.9.0 with Kinesis to process data. We have extended data retention on the Kinesis stream, which gives us 7 days of data. We have found that when a savepoint/checkpoint is restored that it appears to be restarting the Kinesis Consumer from the start of the stream. Th