Why assignTimestampsAndWatermarks parallelism same as map，it will not fired？

Hi all, I use the same parallelism between map and assignTimestampsAndWatermarks , and it not fired, I saw the extractTimestamp and generateWatermark all is fine, but watermark is always not change and keep as min long value. And then I changed parallelism and different with map, and windows

Class loading issues when using Remote Execution Environment

I am trying to get gradoop_demo (a gradoop based graph visualization app) working on Flink with *Remote* Execution Environment. This app, which is based on Gradoop, submits a job to the *preconfigured* execution environment, collects the results and

Testing Metrics

Hey guys and gals, Just wondering: Does anyone have an idea how to test if metrics are being generated? I have an integration test and I just added a processor to count elements in late arrivals (the general idea is to capture those, count and get an average, so we can adjust the allowedLateness)

elasticsearch5 , java.lang.NoClassDefFoundError on mesos

Hi , Im having some weird issue when running some stream job to ELK . The job i starting fine but after few hours im getting this exception and the TM/JB is crashed . this is the config for the elesticserch sink , may by 1 sec flush can cause the deadlock ??

Trigger state clear

Hi I have some issue possibly memory issue that causing the task manager to crash . full code : https://gist.github.com/miko-code/6d7010505c3cb95be122364b29057237 I defined fire_and_purge on element and also evictor so state should be very small ... Any suggestion how figure this issue ?

Re: Flink State monitoring

Anything to add? Is there a Jira ticket for this yet? On Fri, Apr 20, 2018 at 1:03 PM, Stefan Richter wrote: > If estimates are good enough, we should be able to expose something. Would > still like to double check the guarantees to see if the estimates of >

Flink + HDInsight Cluster Deployment

Hello everyone! My task is to install Flink on an HDInsight cluster in Azure. More specifically, I have installed a Kafka cluster (with preconfigured Kafka) as I would like to also combine Flink and Kafka. Unfortunately, Azure does not provide preconfigured cluster for Flink. So I have to

Re: Use gradle with flink

You can certainly setup and build Flink applications with Gradle. However the bad news is, the Flink project does not provide a pre-configured Gradle project/configuration yet. The good news is, the community is working on that [1] and there's already a PR [2] (opened 19 hours ago). Btw. besides

Re: Unsure how to further debug - operator threads stuck on java.lang.Thread.State: WAITING

Hi James, it is unlikely that your issue is the same as the one Miguel is having. His one https://issues.apache.org/jira/browse/FLINK-9242 is probably the same as https://issues.apache.org/jira/browse/FLINK-9144 and happens only in batch programs spilling data in Flink 1.5 and 1.6 versions before

Re: Use gradle with flink

Currently only maven build is supported. Original message From: Georgi Stoyanov Date: 4/24/18 2:17 AM (GMT-08:00) To: user@flink.apache.org Subject: Use gradle with flink Hi guys, I’m wondering is it possible to setup my java flink application with

Re: KafkaJsonTableSource purpose

Hi Sebastien, I think you can do that with Flink's Table API / SQL and the KafkaJsonTableSource. Note that in Flink 1.4.2, the KafkaJsonTableSource does not support flat JSON yet. You'd also need a table-valued UDFs for the parsing of the message and joining the result with the original row.

Re: data enrichment with SQL use case

Hi Alex, An operator that has to join two input streams obviously requires two inputs. In case of an enrichment join, the operator should first read the meta-data stream and build up a data structure as state against which the other input is joined. If the meta data is (infrequently) updated,

Re: Checkpointing barriers

Hi Alex, That's correct. The n refers to the n-th checkpoint. The checkpoint ID is important, because operators need to align the barriers to ensure that they consumed all inputs up to the point, where the barriers were injected into the stream. Each operator checkpoints its own state. For

Use gradle with flink

Hi guys, I’m wondering is it possible to setup my java flink application with gradle? I’m confused cause everywhere win the stackoverflow/documentation is used maven only :/ Kind Regards, Georgi Stoyanov

Install Flink on Microsoft Azure HDInsight

Hello everyone! My task is to install Flink on an HDInsight cluster in Azure. More specifically, I have installed a Kafka cluster (with preconfigured Kafka) as I would like to also combine Flink and Kafka. Unfortunately, Azure does not provide preconfigured cluster for Flink. So I have to

Re: Checkpointing barriers

ok, I got it. Barrier-n is an indicator or n-th checkpoint. My first impression was that barriers are carrying offset information, but it was wrong. Thanks for unblocking ;-) Alex

Re: How to run flip-6 on mesos

Its 1.4.2 ... Any approximate date for 1.5 release ? Thanks allot for your help . On Tue, Apr 24, 2018 at 10:39 AM, Gary Yao wrote: > Hi Miki, > > The stacktrace you posted looks familiar [1]. We have fixed the issue in > Flink > 1.5. What is the Flink version you are

Re: How to run flip-6 on mesos

Hi Miki, The stacktrace you posted looks familiar [1]. We have fixed the issue in Flink 1.5. What is the Flink version you are using? FLIP-6 before Flink 1.5 is very experimental, and I doubt that it is in a usable state. Since 1.5 is not out yet, you can either compile the release branch

Re: Testing on Flink 1.5

Hi Amit, web.timeout should only affect RPC calls originating from the REST API. In FLIP-6, the submission of the job graph happens via HTTP. The value under akka.ask.timeout is still used as the default timeout for RPC calls [1][2]. Since you also had custom heartbeats settings, you should

Re: How to run flip-6 on mesos

Hi Miki, IIRC the port on which the Web UI is listening is not allocated dynamically when deploying on Mesos, and should be 8081 by default (you can override the default by setting rest.port in flink-conf.yaml). If you can find out the hostname/IP of the JobManager, you can submit as usual via

Re: How to run flip-6 on mesos

NO :) ... I usually using the web UI . Can you refer me to some example how to submit a job ? Using REST ? to which port ? thanks, miki On Tue, Apr 24, 2018 at 9:42 AM, Gary Yao wrote: > Hi Miki, > > Did you try to submit a job? With the introduction of FLIP-6,

Re: akka.remote.OversizedPayloadException

Hi Alex, Can you try add the following two lines to your flink-conf.yaml? akka.framesize: 314572800b akka.client.timeout: 10min AFAIK it is not needed to use Java system properties here. Best, Gary On Mon, Apr 23, 2018 at 8:48 PM, Alex Soto wrote: > Hello, > > I

Re: How to run flip-6 on mesos

Hi Miki, Did you try to submit a job? With the introduction of FLIP-6, resources are allocated dynamically. Best, Gary On Tue, Apr 24, 2018 at 8:31 AM, miki haiat wrote: > > HI, > Im trying to tun flip-6 on mesos but its not clear to me what is the > correct way to do it .

How to run flip-6 on mesos

HI, Im trying to tun flip-6 on mesos but its not clear to me what is the correct way to do it . I run the session script and i can see that new framework has been created in mesos but the task manager hasn't been created running taskmanager-flip6.sh throw null pointer ... what is the correct