Re: Failed to transfer file from TaskExecutor : Vanilla Flink Cluster

I tried setting that option but did not work. 2020-02-07 19:28:45,999 INFO org.apache.flink.runtime.resourcemanager.StandaloneResourceManager - Registering TaskManager with ResourceID 32fb9e7dcc9d41917bce38a2d5bb0093 (akka.tcp://flink@ip-1:34718/user/taskmanager_0) at ResourceManager 2020-02-07

Re: Flink 1.10 on MapR secure cluster with high availability

Hi Chesnay, I managed to re-compile with MapR zookeeper and can confirm that it works with HA as well. Still I find it strange that HA uses shadow version of zookeeper instead of version from classpath how it is done for hadoop. Thanks, Maxim. On Wed, Feb 5, 2020 at 3:43 PM Chesnay Schepler

Flink HA for Job Cluster

Hi, In [1] where we can find setup for Stand Alone an YARN clusters to achieve Job Manager's HA. Is Standalone Cluster High Availability with a zookeeper the same approach for Docker's Job Cluster approach with Kubernetes? [1]

SSL configuration - default behaviour

Hi, In documentation [1] we can read that All internal connections are SSL authenticated and encrypted. The connections use mutual authentication, meaning both server and client side of each connection need to present the certificate to each other. The certificate acts effectively as a shared

Re: Running a Beam Pipeline on GCP Dataproc Flink Cluster

+u...@beam.apache.org On Fri, Feb 7, 2020 at 12:54 AM Xander Song wrote: > I am attempting to run a Beam pipeline on a GCP Dataproc Flink cluster. I > have followed the instructions at this repo > > to > create

Re: Performance issue with RegistryAvroSerializationSchema

Steve, thanks a lot for looking into this closer! Let's discuss the resolution of the issue in the ticket Dawid has created: https://issues.apache.org/jira/browse/FLINK-15941 Best, Robert On Thu, Feb 6, 2020 at 6:59 PM Steve Whelan wrote: > Robert, > > You are correct that it is using a

Backfilling From Kafka Topics With Varied Ingestion Rates

Hi Flink Users, We have a scenario where we're reading from multiple kafka topics using a single kafka consumer. Each topic has a very different ingestion rate, like CheckoutTopic has 500 rec/sec, PageViewTopic has 10,000 rec/sec. We are performing ordering of these events across topics using a

Re: Rescaling a running topology

Ooooh more fun... If I rescale down a job, the job's config at jobs/{jobid}/config does not reflect the new parallelism (there may not even be any way to detect such a parallelism change)... but more critically the job is now unstoppable and seems to end up stuck in the CANCELLING state for some

Re: Rescaling a running topology

And now the job is stuck in a suspended state and I seem to have no way to get it out of that state again! On Fri, 7 Feb 2020 at 11:50, Stephen Connolly < stephen.alan.conno...@gmail.com> wrote: > The plot thickens... I was able to rescale down... just not back up > again!!! > >

Re: Rescaling a running topology

The plot thickens... I was able to rescale down... just not back up again!!! root@flink-jobmanager-64bcfdf799-jhs7p:/opt/flink# bin/flink list -m localhost:8081 Waiting for response... -- Running/Restarting Jobs --- 07.02.2020 11:26:33 :

Rescaling a running topology

So I am looking at the Flink Management REST API... and, as I see it, there are two paths to rescale a running topology: 1. Stop the topology with a savepoint and then start it up with the new savepoint; or 2. Use the /jobs/:jobid/rescaling

Re: Task-manager kubernetes pods take a long time to terminate

Hi guys, It looks suspicious that the TM pod termination is potentially delayed by the reconnect to a killed JM. I created an issue to investigate this: https://issues.apache.org/jira/browse/FLINK-15946 Let's continue the discussion there. Best, Andrey On Wed, Feb 5, 2020 at 11:49 AM Yang Wang