Re: Beam Application run on cluster setup (Kafka+Flink)

Hi, I mean same code works fine in flink local setup. I can able to see "Received Message from testkafka Topic : " on console when kafka receive some message (Kafka Producer is in other machine and sending some message frequently to testkafka topic). *Submitted the Beam

Timer coalescing necessary?

Hi, We are having a streaming job where we use timers to implement key timeout for stateful functions. Should we implement coalescing logic to reduce the number of timer trigger, or it is not necessary with Flink? Best regards, Kien

Re: Writing an Integration test for flink-metrics

Team wants an integration test, I'm not sure what unit test you had in mind. Actually feel that I've been trying to avoid the reporter method but that would be more end to end. The documentation for metrics and Scala are missing with the exception of Gauge:

Re: Task Manager was lost/killed due to full GC

On further investigation, seems to me the I/O exception I posted previously is not the cause of the TM being lost. it's the after effect of the TM being shut down and the channel being closed after a record is emitted but before it's processed. Previously, the logs didn't throw up this error and

Re: Task Manager was lost/killed due to full GC

Hi Stephan, Apologies, I hit send too soon on the last email. So, while trying to debug this, I ran it multiple times on different instance types(to increase RAM available) and while digging into the logs, I found this to be the error in the task manager logs: / java.lang.RuntimeException:

Re: Task Manager was lost/killed due to full GC

Hi Stephan, Thanks for your response! Task manager lost/killed has been a recurring problem I've had with Flink for the last few months, as I try to scale to larger and larger amounts of data. I would be very grateful for some help figuring out how I can avoid this. The program is set up

Re: ArrayIndexOutOfBoundExceptions while processing valve output watermark and while applying ReduceFunction in reducing state

I think I finally found the problem, there was also already another bug report for this: https://issues.apache.org/jira/browse/FLINK-7484 > On 12. Oct 2017, at 18:22, Federico D'Ambrosio > wrote: > > Hi Aljoscha, > > yes, just like you're guessing, without

Re: Writing to an HDFS file from a Flink stream job

Thanks for all your directions. BucketingSink worked. Isuru On Thu, Oct 12, 2017 at 9:05 AM, Piotr Nowojski wrote: > I think the issue might be that writeAsText (TextOutputFormat) doesn’t > flush the data anywhere (only on close, which in streaming doesn’t happen). >

Re: ArrayIndexOutOfBoundExceptions while processing valve output watermark and while applying ReduceFunction in reducing state

Hi Aljoscha, yes, just like you're guessing, without asynchronous checkpoints, there has been no crash so far. Regards, Federico 2017-10-12 18:08 GMT+02:00 Aljoscha Krettek : > Hi Federico, > > I'm guessing the job is still working without asynchronous watermarks? I'm >

Re: ArrayIndexOutOfBoundExceptions while processing valve output watermark and while applying ReduceFunction in reducing state

Hi Federico, I'm guessing the job is still working without asynchronous watermarks? I'm very eager to figure out what is actually going wrong with asynchronous checkpoints. Best, Aljoscha > On 2. Oct 2017, at 11:57, Federico D'Ambrosio > wrote: > > As a

Re: Best way to setup different log files for distinct jobs

what if you have 'dedicated' task managers for each job? so if you have 2 TMs each with 1 task slot and two jobs with -p1 then each job will go to the respective TM, I think? hence - each job in its own (TM's) log I'm new to Flink, hope it make sense > Оригинално писмо

Running flink on YARN

Hello, I'm running flink on AWS EMR and I would like to know how I can pass a custom log4j properties file. I changed the log4j.properties file in flink conf directory but it doesn't seem like the changes are reflected. Thanks. I'm using the below command to start my flink job. > flink run -m

Re: Empty state restore seems to be broken for Kafka source (1.3.2)

Still not nice, though, and it took a while to finalise discovery for 1.4. ;-) If you need that now you might be able to back port the 1.4 consumer to 1.3. > On 12. Oct 2017, at 17:05, Gyula Fóra wrote: > > Ok, thanks for the clarification. :) > > Gyula > > > On Thu, Oct

Re: Empty state restore seems to be broken for Kafka source (1.3.2)

Ok, thanks for the clarification. :) Gyula On Thu, Oct 12, 2017, 17:04 Aljoscha Krettek wrote: > It might be old but it's not forgotten, the issue I created is actually > marked as a blocker so we won't forget it when releasing 1.3.3 and 1.4.0. > > The issue in Kafka is

Re: Empty state restore seems to be broken for Kafka source (1.3.2)

It might be old but it's not forgotten, the issue I created is actually marked as a blocker so we won't forget it when releasing 1.3.3 and 1.4.0. The issue in Kafka is about new topics/partitions not being discovered or something else? That would be the expected behaviour in Flink < 1.4.0.

Re: Submitting a job via command line

Have you tried this http://mail-archives.apache.org/mod_mbox/flink-user/201705.mbox/%3ccagr9p8bxhljseexwzvxlk+drotyp1yxjy4n4_qgerdzxz8u...@mail.gmail.com%3E

Re: Monitoring job w/LocalStreamEnvironment

Hi, Have you read the following doc? https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/stream/testing.html There are some hints regarding testing your application. Especially take a look at the

Re: Beam Application run on cluster setup (Kafka+Flink)

Hi, What do you mean by: > With standalone beam application kafka can receive the message, But in cluster setup it is not working. In your example you are reading the data from Kafka and printing them to console. There doesn’t seems to be anything that writes back to Kafka, so what do you

Re: Empty state restore seems to be broken for Kafka source (1.3.2)

Hey, I know it's old discussion but there also seems to be a problem with the logic in the kafka source alone regarding new topics added after a checkpoint. Maybe there is a ticket for this already and I just missed it. Cheers, Gyula Gyula Fóra ezt írta (időpont: 2017.

Monitoring job w/LocalStreamEnvironment

Hi all, With an iteration-based workflow, it’s helpful to be able to monitor the job counters and explicitly terminate when the test has completed. I didn’t see support for async job creation, though. So I extended LocalStreamEnvironment to add an executeAsync(), which returns the

Submitting a job via command line

Hello All, I got the following error while attempting to execute a job via command line: [root@flink01 bin]# ./flink run -c com.five9.stream.PrecomputeJob /vagrant/flink-precompute-1.0-SNAPSHOT.jar -Xmx2048m -Xms2048m Cluster configuration: Standalone cluster with JobManager at

Beam Application run on cluster setup (Kafka+Flink)

Below is my setup 1. Kafka zookeeper and server in one machine (192.168.1.116) and producer (192.168.1.100) and consumer (192.168.1.117) in another machine. --> This work fine no issue 2. Running standalone beam application with kafka consumer --> This work fine 3.

Re: Writing to an HDFS file from a Flink stream job

I think the issue might be that writeAsText (TextOutputFormat) doesn’t flush the data anywhere (only on close, which in streaming doesn’t happen). You would need to use custom output format, but as Aljoscha pointed out BucketingSink makes more sense for streaming applications. Piotrek > On 12

Re: Kafka 11 connector on Flink 1.3

Thanks. I want to use the exactly once Kafka producer for my use case. Hence the question. I will wait for the update on the timelines. Thanks again. Moiz On Oct 12, 2017 5:26 PM, "Piotr Nowojski" wrote: > Hi, > > Kafka 0.11 connector depends on some API changes for

Re: Writing to an HDFS file from a Flink stream job

Hi, Maybe this is an access rights issue? Could you try to create and write to same file (same directory) in some other way (manually?), using the same user and the same machine as would Flink job do? Maybe there will be some hint in hdfs logs? Piotrek > On 12 Oct 2017, at 00:19, Isuru

Re: Writing an Integration test for flink-metrics

Well damn, i should've read the second part of the initial mail. I'm wondering though, could you not unit-test this behavior? On 12.10.2017 14:25, Chesnay Schepler wrote: You could also write a custom reporter that opens a socket or similar for communication purposes. You can then either

Re: Writing an Integration test for flink-metrics

You could also write a custom reporter that opens a socket or similar for communication purposes. You can then either query it for the metrics, or even just trigger the verification in the reporter, and fail with an error if the reporter returns an error. On 12.10.2017 14:02, Piotr Nowojski

Re: Writing an Integration test for flink-metrics

Hi, Doing as you proposed using JMXReporter (or custom reporter) should work. I think there is no easier way to do this at the moment. Piotrek > On 12 Oct 2017, at 04:58, Colin Williams > wrote: > > I have a RichMapFunction and I'd like to ensure Meter

Re: Kafka 11 connector on Flink 1.3

Hi, Kafka 0.11 connector depends on some API changes for Flink 1.4, so without rebasing the code and solving some small issues it is not possible to use it for 1.3.x. We are about to finalizing the timeframe for 1.4 release, it would be great if you could come back with this question after

Re: Exception in BucketingSink when cancelling Flink job

Hi Wangsan, We were struggling with this for many days as well. In the end we found a work around. Well work-around, this for sure qualifies as one of the ugliest hacks I have ever contemplated. Our work-around for Flink immediately interrupting the close, is to continue closing on another

Re: Implement bunch of transformations applied to same source stream in Apache Flink in parallel and combine result

Hi, What is the number of events per second that you wish to process? If it’s high enough (~ number of machines * number of cores) you should be just fine, instead of scaling with number of features, scale with number of events. If you have a single data source you still could randomly shuffle

Kafka 11 connector on Flink 1.3

Hi, I was just wondering if I could use the Kafka 11 connector on 1.3. If there are dependencies, then what would be the timeframe for 1.4 release Thanks Moiz

Re: PartitionNotFoundException when running in yarn-session.

Hey Niels, Flink currently restarts the complete job if you have a restart strategy configured: https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/restart_strategies.html. I agree that only restarting the required parts of the pipeline is an important optimization. Flink has not

Re: R/W traffic estimation between Flink and Zookeeper

Hi, Are you asking how to measure records/s or is it possible to achieve it? To measure it you can check numRecordsInPerSecond metric. As far if 1000 records/s is possible, it depends on many things like state backend used, state size, complexity of your application, size of the records,

Re: Write each group to its own file

Hi, There is no straightforward way to do that. First of all, the error you are getting is because you are trying to start new application ( env.fromElements(items) ) inside your reduce function. To do what you want, you have to hash partition the products based on category (instead of