Re: Slow Flink program

2018-03-01 Thread Supun Kamburugamuve
eData) is very > expensive to serialize and for RocksDB, you get a back and forth > serialization (off-heap => on-heap, compute, on-heap => off-heap) > > On Thu, Mar 1, 2018 at 4:32 PM, Supun Kamburugamuve > wrote: > >> Yes, the program runs fine, I can see it on the

Re: Slow Flink program

2018-03-01 Thread Supun Kamburugamuve
Environment? > > 2018-03-01 15:55 GMT+01:00 Supun Kamburugamuve : > >> Thanks Piotrek, >> >> I did it this way on purpose to see how Flink performs. With 128000 >> messages it takes an un-reasonable amount of time for Flink to complete the >> operation. With

Re: Slow Flink program

2018-03-01 Thread Supun Kamburugamuve
ords are gigantic? I can not tell, since you > didn’t provide full code. > - you are mapping the data to new Tuple2(0, s); > and then keying by the first field, which is always 0. Probably all of the > records are ending up on one single machine > > Piotrek > > On 28 Feb 20

Slow Flink program

2018-02-28 Thread Supun Kamburugamuve
Hi, I'm trying to run a simple benchmark on Flink streaming reduce. It seems it is very slow. Could you let me know if I'm doing something wrong. Here is the program. I'm running this on 32 nodes with 20 tasks in each node. So the parallelism is at 640. public class StreamingReduce { int size;

Re: Nested iterations

2016-09-01 Thread Supun Kamburugamuve
e-flink-user-mailing-list-archive.2336050. > n4.nabble.com/Iterative-queries-on-Flink-td3786.html > > > > > 2016-09-01 18:31 GMT+02:00 Supun Kamburugamuve : > > Thanks Gabor. I was thinking about starting separate jobs. > > > > Is there any plans to support

Re: Nested iterations

2016-09-01 Thread Supun Kamburugamuve
of this is that you might end up with > a too large Flink job, which might also hurt performance. > > Best, > Gábor > > > > > > > 2016-09-01 18:09 GMT+02:00 Supun Kamburugamuve : > > Hi, > > > > Does Flink support nested iterations? We are tr

Nested iterations

2016-09-01 Thread Supun Kamburugamuve
Hi, Does Flink support nested iterations? We are trying to develop a complex machine learning algorithm which has 3 iterations nested. Best, Supun..