Re: How many task managers can a cluster reasonably handle?

Hi Antonio， According to experience in our production, Flink totally can handle 150 TaskManagers per cluster. Actually, we have encountered much larger jobs with thousands that each single job demands thousands of TaskManagers. However, as the job scale increases, it gets harder to achieve good st

Re: How many task managers to launch for a job?

Hi black, If you are running Flink on Yarn or Mesos, Flink will automatically allocate resource and launch new TaskManagers as needed. If you are using Flink standalone mode, then the easiest way is to enable slot sharing and set all the vertices into the same group (which is by default). In that

Re: Count Window Trigger that only fires once

Hi Frank, Seems you want a custom trigger, maybe the doc[1] can help. [1]  https://ci.apache.org/projects/flink/flink-docs-release-1.8/dev/stream/operators/windows.html#triggers Best Congxian On May 23, 2019, 23:38 +0800, Frank Wilson , wrote: > Hi, > > Is there a way to make the count window tr

Upgrading from 1.4 to 1.8, losing Kafka consumer state

Howdy, We're in the process of upgrading to 1.8. When restoring state to the new cluster (using a savepoint) we are seeing our Kafka consumers restart from the earliest offset. We're not receiving any other indication that our state was not accepted as part of the deploy, e.g. we are not allowing

How many task managers can a cluster reasonably handle?

Hello Flink users, How many task managers one can expect a Flink cluster to be able to reasonably handle? I want to move a pretty big cluster from a setup on AWS EMR to one based on Kubernetes. I was wondering whether it makes sense to break up the beefy task managers the cluster had in something

Re: [SURVEY] Usage of flink-ml and [DISCUSS] Delete flink-ml

+1 for the deletion. Also I think it also might be a good idea to update the roadmap for the plan of removal/development since we've reached the consensus on FLIP-39. Thanks, Rong On Wed, May 22, 2019 at 8:26 AM Shaoxuan Wang wrote: > Hi Chesnay, > Yes, you are right. There is not any active

How many task managers to launch for a job?

Hi, I am redesigning the scheduler of the JobManager to place tasks of a job across TaskManagers accroding to a scheduling policy. I am reading the Flip-6 proposal and found that the common case is "one TaskManager launchs one slot", and "one Flink cluster serves one job". But I did not find how

Count Window Trigger that only fires once

Hi, Is there a way to make the count window trigger fire only once? I would like a session window to only emit the first element it receives immediately rather than waiting until the watermark passes the end of the window. Thanks, Frank

Re: Connectors (specifically Kinesis Connector)

Hi Steven, I assume you are referring to the problem that we don't publish the Kinesis connector artifacts to Maven, due to the licensing issue with KCL? I didn't manage to find any JIRAs that were addressing this issue directly, but the most related one would be this: https://issues.apache.org/ji

Re: Flink not giving full reason as to why job submission failed

Please open a new JIRA. FLINK-11902 modified the REST API to no longer hide the exception, but the WebUI isn't handling the error response properly (it only reads and displays part of it). On 20/05/2019 16:24, Wouter Zorgdrager wrote: Hi Harshith, This is indeed an issue not resolved in 1.8.