Re: Jobmanager drops upon submitting a jar

a jar Hi Amir, What do you mean by “modernizing” the FlinkKafkaConsumer implementation? Could you explain a bit more? Thanks :) Chiming in just to check if there’s any issue we need to be aware of .. Cheers, Gordon On 14 April 2017 at 5:35:43 AM, amir bahmanyari (amirto...@yahoo.com.i

Why do I need this dependency?

Hi colleagues,I build my app (fat jar) with no errors with these dependencies:       org.apache.flink   flink-connector-kafka-0.8_2.10   1.2.0  org.apache.flink   flink-java  1.2.0  org.apache.flink   flink-streaming-java_2.10  1.2.0  org.apache.flink   flink-clients_2.10  

Re: Jobmanager drops upon submitting a jar

consumer object. Any suggestions on modernizing the FlinkKafkaConsumer implementation?Thanks+regards From: wenlong.lwl <wenlong88@gmail.com> To: dev@flink.apache.org; amir bahmanyari <amirto...@yahoo.com> Sent: Wednesday, April 12, 2017 8:01 PM Subject: Re: Jobmanage

Re: Jobmanager drops upon submitting a jar

) at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) From: wenlong.lwl <wenlong88@gmail.com> To: dev@flink.apache.org; amir bahmanyari <amirto...@yahoo.com> Sent: Wednesd

Jobmanager drops upon submitting a jar

Hi Colleagues,I have a simple test job when I submit it to the Flink cluster the JM seems to disconnect.Its a one node cluster implemented in a VirtualBox Centos 7 VM.Flink starts fine and everything else look fine. Following is stack trace.I appreciate a feedback.Cheers 17/04/12 15:53:04

Re: Flink limitations under Beam

Thanks Aljoscha,How sever of performance difference are we talking about here?  examples of "Iterations/feedback edges" and how they get applied?Thanks so much Aljoscha. From: Aljoscha Krettek <aljos...@apache.org> To: dev@flink.apache.org; amir bahmanyari <amirto..

Flink limitations under Beam

Hi colleagues, been a long time. New project.What feature(s)/capabilities of Flink would become unavailable/limited if the pipeline app is written in Beam sdk using FlinkRunner? Thanks+regards

Re: Machine Learning on Flink - Next steps

Great points to start:    - Online learning   - Offline learning with the streaming API Thanks + have a great weekend. From: Katherin Eri To: dev@flink.apache.org Sent: Friday, March 3, 2017 7:41 AM Subject: Re: Machine Learning on Flink - Next steps Thank

Re: Flink survey by data Artisans

From: Kostas Tzoumas To: "dev@flink.apache.org" ; u...@flink.apache.org Sent: Friday, November 18, 2016 7:28 AM Subject: Flink survey by data Artisans Hi everyone! The Apache Flink community has evolved quickly over the past 2+ years,

Re: [FLINK-3035] Redis as State Backend

scha...Cheers From: Ovidiu Cristian Marcu <ovidiu.cristian.ma...@huawei.com> To: "dev@flink.apache.org" <dev@flink.apache.org>; amir bahmanyari <amirto...@yahoo.com> Sent: Monday, November 7, 2016 3:56 AM Subject: RE: [FLINK-3035] Redis as State Backend Thanks, h

Re: [FLINK-3035] Redis as State Backend

FYI.I was using Redis as a state backend in my benchmarking Beam.It proved to be a bottleneck. Perhaps due to high frequency of updating state components.I replaced it with Java 8 ConcurrentHashmaps and it settled down tremendously.Amir- From: Ovidiu-Cristian MARCU

Re: Flink Metrics

Hi colleagues,Is there a link that described Flink Matrices & provides example on how to utilize it pls?I really appreciate it...Cheers From: Till Rohrmann To: u...@flink.apache.org Cc: dev@flink.apache.org Sent: Monday, October 17, 2016 12:52 AM Subject: Re:

Re: Use specific Task Manager for heavy computations

Looks like what I am currently doing, or at least close.No need to copy the big file on every node.Copy on one node. Read the data, and send it to a Kafka cluster using KafkaProducer() object.Use KafkaIO() (in case its a Beam app). Deploy to the node where the JM is running.It will be executed

Re: Get Flink ExecutionGraph Programmatically

My only 2 cents is that when I started to turn the mem pre-allocation param, to true & #slots & #buffersI started to get all kinds of Akka & Disassociated exceptions thrown by the JM regarding the TMs...So yes, since I am also not well aware of Akka internals...I went back to my previous

Re: Performance and Latency Chart for Flink

lots  exceptions.  So I am going incremental with that value. Have it set at 8192 (twice as much as before 4096).Thanks From: Greg Hogan <c...@greghogan.com> To: dev@flink.apache.org; amir bahmanyari <amirto...@yahoo.com> Sent: Monday, September 19, 2016 1:28 PM Subject: Re: Perf

Re: Performance and Latency Chart for Flink

ate roughly 300 MiBytes for network buffers." ?Thats roughly twice as much. Am i Missing something here?I still need your help to set the accurate number for my  - taskmanager.network.numberOfBuffers = 4096. Thanks for your response Greg.Amir- From: amir bahmanyari <amirto...@yahoo.

Re: Performance and Latency Chart for Flink

ory. Also, your number of buffers looks quite low and you may want to increase "taskmanager.network.numberOfBuffers". Your setting of 4096 is only 128 MiB. As this is a only benchmark are you able to post the code to github to solicit feedback? Greg On Sun, Sep 18, 2016 at 9:00 PM, amir ba

Re: Performance and Latency Chart for Flink

o under Beam APIs.Which was the initial intent anyways.If you have suggestions to make improvements in the above case, I am all ears & greatly appreciate it.Cheers,Amir- From: "Chawla,Sumit" <sumitkcha...@gmail.com> To: dev@flink.apache.org; amir bahmanyari <amirto...@yaho

Re: Performance and Latency Chart for Flink

Hi Fabian,FYI. This is report on other engines we did the same type of bench-marking.Also explains what Linear Road bench-marking is.Thanks for your help. http://www.slideshare.net/RedisLabs/walmart-ibm-revisit-the-linear-road-benchmark https://github.com/IBMStreams/benchmarks 

Re: Flink Cluster Load Distribution Question

Hi Robert,I forwarded my Qs to the users forum...No response ...Is there anyway I can get some answers to my questions pls?I really really appreciate your help...CheersAmir- From: Robert Metzger <rmetz...@apache.org> To: "dev@flink.apache.org" <dev@flink.apache.or

Flink Cluster Load Distribution Question

Hi Colleagues,Just joined this forum.I have done everything possible to get a 4 nodes Flink cluster to work peoperly & run a Beam app.It always generates system-output logs (*.out) in only one node. Its so slow for 4 nodes being there.Seems like the load is not distributed amongst all 4