Re: Production Readiness of File Source

at 3:57 PM Dawid Wysakowicz wrote: > Hi, > > As for the issue of production readiness of the File Source(and other > components) I'd recommend having a look at the PR, which is close to being > merged where we express our opinion how we see certain components: > https://github.co

Re: Production Readiness of File Source

Hi, As for the issue of production readiness of the File Source(and other components) I'd recommend having a look at the PR, which is close to being merged where we express our opinion how we see certain components: https://github.com/apache/flink-web/pull/426 I am also cc'ing Stephan who wrote

Production Readiness of File Source

Hi, I am intending to use the File source for a production use case. I have a few use cases that are currently not supported like deleting a file once it's processed.  So I was wondering if we can use this in production or write my own implementation? Is there any recommendations around this?

Re: Production readiness

Hi Aitozi, resuming a job with a higher parallelism than the initially defined max parallelism (in your case 128) is not possible. For this one would need to rewrite the savepoint information (basically rehash the keys) as Andrey said. Cheers, Till On Thu, Feb 14, 2019 at 3:50 AM aitozi wrote:

Re: Production readiness

Hi, Andrey I have another question that if i do not set the maximum parallelism first(which be set to 128 by default), and then rescale to a parallelism bigger than 128. In this scenario，will the state lost? Thanks, Aitozi -- Sent from:

Re: Production readiness

, 2019 at 4:58 PM Avi Levi wrote: > Hi > Looking at the production readiness > <https://ci.apache.org/projects/flink/flink-docs-stable/ops/production_ready.html#set-maximum-parallelism-for-operators-explicitly> > checklist - is there any rule of thumb to determine the maximum paral

Production readiness

Hi Looking at the production readiness <https://ci.apache.org/projects/flink/flink-docs-stable/ops/production_ready.html#set-maximum-parallelism-for-operators-explicitly> checklist - is there any rule of thumb to determine the maximum parallelism ? we have a stateful pipeline with high thro

Re: Production readiness of Flink Job Stop Service

Hi Chirag, Actually, if you want to use "stop/resume/pause" pattern, you can trigger a savepoint before stopping job, then you can resume your job with specify this savepoint. The savepoint does not bind with cancel command, although you can only invoke "cancel with savepoint" now. Now you

Re: Production readiness of Flink Job Stop Service

Thanks a lot Fabian and Vino. Is there anyway I can do that without stop? Although the plan should be to use a Two Phase commit sink so that there are no duplicates, I am looking for stopping the sources before the savepoint is triggered. Thanks, Chirag Sent from Yahoo Mail on Android On

Re: Production readiness of Flink Job Stop Service

Hi Chirag, Stop with savepoint is not mentioned in the 1.5.0 release notes [1]. Since its a frequently requested feature, I'm pretty sure that it would have been mentioned if it was added. Best, Fabian [1] http://flink.apache.org/news/2018/05/25/release-1.5.0.html 2018-07-19 8:39 GMT+02:00

Re: Production readiness of Flink Job Stop Service

Hi Chirag, Did you read the latest stable Flink documentation about Savepoint[1] and Cancel with savepoint[2] and Upgrade application[3]? [1]: https://ci.apache.org/projects/flink/flink-docs-release-1.5/ops/state/savepoints.html#resuming-from-savepoints [2]:

Production readiness of Flink Job Stop Service

Hi, I am planning to use the Stop Service for stopping/resuming/pausing my Flink Job. My intention is to stop sources before we take the savepoint i.e. stop with savepoint.  I know that since Flink 1.4.2, Stop is not stable/not production ready.  With Flink 1.5 can it be used for stopping jobs?