Re: Memory Issue

2017-08-23 Thread Govindarajan Srinivasaraghavan
I have couple more questions regarding flink's jvm memory. In a streaming application what is managed memory used for? I read from a blog that all objects created inside the user function will go into unmanaged memory. Where does the managed key/ operator state state reside? Also when does the

custom writer fail to recover

2017-08-23 Thread Biswajit Das
Hi There , I'm using custom writer with hourly Rolling Bucket sink . I'm seeing two issue first one if write the same file on s3 all the files gets committed , however when I write the same on HDFS I see its remains on .pending state , could be related to second problem below Second issue : My

Re: Support for multiple HDFS

2017-08-23 Thread Vijay Srinivasaraghavan
Hi Ted, I believe HDFS-6584 is more of an HDFS feature supporting archive use case through some policy configurations. My ask is that I have two distinct HCFS File systems which are independent but the Flink job will decide which one to use for sink while the Flink infrastructure is by default

Apache Flink Reading CSV Files ,Transform and Writting Back to CSV using Paralliesm

2017-08-23 Thread Lokesh R
Hi Team, I am using the apache flink with java for below problem statement 1.where i will read a csv file with field delimeter character ; 2.transform the fields 3.write back the data back to csv my doubts are as below 1. if i need to read the csv file of size above 50 gb what would be the

Re: Support for multiple HDFS

2017-08-23 Thread Ted Yu
Would HDFS-6584 help with your use case ? On Wed, Aug 23, 2017 at 11:00 AM, Vijay Srinivasaraghavan < vijikar...@yahoo.com.invalid> wrote: > Hello, > Is it possible for a Flink cluster to use multiple HDFS repository (HDFS-1 > for managing Flink state backend, HDFS-2 for syncing results from

Support for multiple HDFS

2017-08-23 Thread Vijay Srinivasaraghavan
Hello, Is it possible for a Flink cluster to use multiple HDFS repository (HDFS-1 for managing Flink state backend, HDFS-2 for syncing results from user job)?  The scenario can be viewed in the context of running some jobs that are meant to push the results to an archive repository (cold

Re: akka timeout

2017-08-23 Thread Bowen Li
Hi Steven, Yes, GC is a big overhead, it may cause your CPU utilization to reach 100%, and every process stopped working. We ran into this a while too. How much memory did you assign to TaskManager? How much the your CPU utilization when your taskmanager is considered 'killed'? Bowen

Re: akka timeout

2017-08-23 Thread Steven Wu
Till, Once our job was restarted for some reason (e.g. taskmangaer container got killed), it can stuck in continuous restart loop for hours. Right now, I suspect it is caused by GC pause during restart, our job has very high memory allocation in steady state. High GC pause then caused akka

Re: Flink parquet read.write performance

2017-08-23 Thread Stephan Ewen
Hi! The sink is merely a union of the result of the co-group and the one data source. Can't you just make to distinct pipelines out of that? One with co-group -> data sink pipeline and one with the source->sink pipeline? They could even be part of the same job... Best, Stephan On Wed, Aug 23,

Re: Flink parquet read.write performance

2017-08-23 Thread Aljoscha Krettek
Hi, The reason is that there are two (or more) different Threads doing the reading. As an illustration, consider first this case: DataSet input = ... input.map(new MapA()).map(new MapB()) Here, MapB is technically "wrapped" by MapA and when MapA emits data this is directly going the the map()

Re: PageRank - 4x slower then Spark?!

2017-08-23 Thread Timo Walther
You could enable object reuse [0] if you application allows that. Also adjusting the managed memory size [1] can help. Are you using Flink's graph library Gelly? [0] https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/batch/index.html#object-reuse-enabled [1]

Re: Stand alone blob server

2017-08-23 Thread Nico Kruber
Hi Anugrah, you can track the progress at the accompanying jira issue: https://issues.apache.org/jira/browse/FLINK-6916 Currently, roughly half of the tasks are done with a few remaining in PR review stage. Note that the actual implementation differs a bit from what was proposed in FLIP 19

RE: Flink parquet read.write performance

2017-08-23 Thread Newport, Billy
Thanks Aljoscha for the prompt response. Can you explain the technical reason for the single predecessor rule? This makes what we are trying to do much more expensive. Really what we’re doing is reading a parquet file, doing several maps/filters on the records and writing to the parquet. There

Re: PageRank - 4x slower then Spark?!

2017-08-23 Thread Kaepke, Marc
Does someone has a current performance test based on PageRank or an idea why Flink lost the comparison? > Am 18.08.2017 um 19:51 schrieb Kaepke, Marc : > > Hi everyone, > > I compared Flink and Spark by using PageRank. I guessed Flink will beat Spark > or have the

Stand alone blob server

2017-08-23 Thread anugrah nayar
Hey, I was reading through the flink documentation and found that there were some plans around stand alone blob server. I found the details here https://cwiki.apache.org/confluence/display/FLINK/FLIP-19%3A+Improved+BLOB+storage+architecture. Just wanted to know whether this was done. regards,

Re: Unexpected behaviour of a periodic trigger.

2017-08-23 Thread Tomasz Dobrzycki
Hi Tony, Thank you for this thorough explanation. Helps a lot! Kind Regards, Tomasz On 23 August 2017 at 11:30, Tony Wei wrote: > Hi Tomasz, > > Actually, window is not a real operator shared by your operators created by > reduce() and apply() function. > Flink

Re: Unexpected behaviour of a periodic trigger.

2017-08-23 Thread Tony Wei
Hi Tomasz, Actually, window is not a real operator shared by your operators created by reduce() and apply() function. Flink implemented WindowOperator by binding window(), trigger() and evictor() as well with the WindowFunction. It is more like the prior operator sent elements to two following

Re: Unexpected behaviour of a periodic trigger.

2017-08-23 Thread Tomasz Dobrzycki
Hi Tony, Won't that increase the amount of processing Flink has to do? It would have to window twice, right? Thanks, Tomasz On 23 August 2017 at 11:02, Tony Wei wrote: > Hi Tomasz, > > In my opinion, I would move .window() function down to these two DataStream. >

Re: Unexpected behaviour of a periodic trigger.

2017-08-23 Thread Tony Wei
Hi Tomasz, In my opinion, I would move .window() function down to these two DataStream. (rawEvent.window().reduce().map(), and so does metrics) It makes sure that they won't share the same constructor. Regards, Tony Wei 2017-08-23 17:51 GMT+08:00 Tomasz Dobrzycki :

Re: Unexpected behaviour of a periodic trigger.

2017-08-23 Thread 魏偉哲
Hi Tomasz, I think this is because .window() is a lazy operator. It just creates a WindowedStream class but not create a corresponding operator. The operator will be created after you called .reduce() and .apply(). rawEvents and metrics actually shared the same object to create their own

Re: Flink HA with Kubernetes, without Zookeeper

2017-08-23 Thread Ufuk Celebi
Thanks James for sharing your experience. I find it very interesting :-) On Tue, Aug 22, 2017 at 9:50 PM, Hao Sun wrote: > Great suggestions, the etcd operator is very interesting, thanks James. > > > On Tue, Aug 22, 2017, 12:42 James Bucher wrote: >> >>

Re: Question about windowing

2017-08-23 Thread 魏偉哲
Hi Jerry, You can learn about Flink's windowing mechanics in this blog ( https://flink.apache.org/news/2015/12/04/Introducing-windows.html). To my understanging, window() defines how Flink use WindowAssigner to insert an element to the right windows, trigger() defines when to fire a window and

Re: Job submission timeout

2017-08-23 Thread Vishnu Viswanath
Never mind, it was a silly mistake, I used "=" instead of ":" while setting akka.ask.timeout. Now it works fine! On Tue, Aug 22, 2017 at 5:10 PM, Vishnu Viswanath < vishnu.viswanat...@gmail.com> wrote: > Hi, > > After I submit the job the client timeout after 10 seconds( Guess Job > manager is