Re: KeyBy/Rebalance overhead?

2019-12-09 Thread Komal Mariam
/flink-docs-release-1.9/concepts/runtime.html#tasks-and-operator-chains > > Komal Mariam 于2019年12月9日周一 上午9:11写道: > >> Anyone? >> >> On Fri, 6 Dec 2019 at 19:07, Komal Mariam wrote: >> >>> Hello everyone, >>> >>> I want to get some insi

Re: KeyBy/Rebalance overhead?

2019-12-08 Thread Komal Mariam
Anyone? On Fri, 6 Dec 2019 at 19:07, Komal Mariam wrote: > Hello everyone, > > I want to get some insights on the KeyBy (and Rebalance) operations as > according to my understanding they partition our tasks over the defined > parallelism and thus should make our pipeline f

KeyBy/Rebalance overhead?

2019-12-06 Thread Komal Mariam
Hello everyone, I want to get some insights on the KeyBy (and Rebalance) operations as according to my understanding they partition our tasks over the defined parallelism and thus should make our pipeline faster. I am reading a topic which contains 170,000,000 pre-stored records with 11 Kafka

Re: How to submit two jobs sequentially and view their outputs in .out file?

2019-11-24 Thread Komal Mariam
file? > > Best, > Vino > > Komal Mariam 于2019年11月22日周五 下午6:59写道: > >> Dear all, >> >> Thank you for your help regarding my previous queries. Unfortunately, I'm >> stuck with another one and will really appreciate your input. >> >> I can't seem t

How to submit two jobs sequentially and view their outputs in .out file?

2019-11-22 Thread Komal Mariam
Dear all, Thank you for your help regarding my previous queries. Unfortunately, I'm stuck with another one and will really appreciate your input. I can't seem to produce any outputs in "flink-taskexecutor-0.out" from my second job after submitting the first one in my 3-node-flink standalone

Re: ctx.timestamp() returning null when using Processing Time

2019-11-06 Thread Komal Mariam
ocessingTime()`. > > > > [1] > https://github.com/apache/flink/blob/9b43f13a50848382fbd634081b82509f464e62ca/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/functions/co/BaseBroadcastProcessFunction.java#L50 > > Best > > Yun Tang > > > > *Fro

Re: How to write type information for a Java Set and List inside a Tuple?

2019-11-06 Thread Komal Mariam
>> Consider snippet 2, now our type inference in TypeInformation.of can not >> infer the nested information. (It not get the information: List) >> >> On Fri, Nov 1, 2019 at 11:34 AM Komal Mariam >> wrote: >> >>> Hi all, >>> >>> I'm trying to cre

ctx.timestamp() returning null when using Processing Time

2019-11-06 Thread Komal Mariam
Dear all, I want to clear some of my variables in KeyedBroadcastProcessFunction after a certain time. I implemented the onTimer() function but even though I am using ProcessingTime like so: env.setStreamTimeCharacteristic(TimeCharacteristic.ProcessingTime), I am getting null when ctx.timestamp()

Flink 1.9 measuring time taken by each operator in DataStream API

2019-10-24 Thread Komal Mariam
Hello, I have a few questions regarding flink’s dashboard and monitoring tools. I have a fixed number of records that I process through the datastreaming API on my standalone cluster and want to know how long it takes to process them. My questions are: 1)How can I see the time taken in

Re: Passing parameters to filter function (in DataStreams)

2019-10-09 Thread Komal Mariam
c boolean filter(Integer temperature) { > return temperature > threshold; } > }); > > final int threshold = computeThreshold();temperatureStream.filter(temperature > -> temperature > threshold); > > > On 08/10/2019 12:46, Komal Mariam wrote: > >

Passing parameters to filter function (in DataStreams)

2019-10-08 Thread Komal Mariam
Hi everyone, Suppose I have to compute a filter condition Integer threshold = compute threshold(); If I: temperatureStream.filter(new FilterFunction() { @Override public boolean filter(Integer temperature) throws Exception { Integer threshold = compute threshold(); return temperature >

Re: Finding the Maximum Value Received so far in a Stream

2019-10-06 Thread Komal Mariam
Thank you for your help all. I understand now and made the changes. Since I needed return the entire object that contained the max value of X, I used reduce instead of max.

Re: Finding the Maximum Value Received so far in a Stream

2019-10-03 Thread Komal Mariam
your reduce function: > You execute it by fish_id if I see it correctly. This will create one > result by fish_id . I propose to map first all fish coordinates under a > single key and then reduce by this single key. > > Am 03.10.2019 um 08:26 schrieb Komal Mariam : > >  >

Finding the Maximum Value Received so far in a Stream

2019-10-03 Thread Komal Mariam
Hello all, I'm trying to do a fairly simple task that is to find the maximum value (Double) received so far in a stream. This is what I implemented: POJO class: public class Fish{ public Integer fish_id; public Point coordinate; //position public Fish() {}; public Fish(fish_id,double

Re: Problem starting taskexecutor daemons in 3 node cluster

2019-09-16 Thread Komal Mariam
could you check that every node can reach the other nodes? It looks a > little bit as if the TaskManager cannot talk to the JobManager running on > 150.82.218.218:6123. > > Cheers, > Till > > On Thu, Sep 12, 2019 at 9:30 AM Komal Mariam > wrote: > >> I managed

Re: Problem starting taskexecutor daemons in 3 node cluster

2019-09-12 Thread Komal Mariam
org.apache.flink.runtime.net.ConnectionUtils - Failed to connect from address '/fe80:0:0:0:1e10:83f4:a33a:a208%enp5s0f1': Network is unreachable (connect failed) "flink-komal-taskexecutor-0-salman-hpc.log" 157L, 29954C I'd appreciate help regarding the issue. Best Regards, Komal On Wed, 11 Sep 2019 at 14

Problem starting taskexecutor daemons in 3 node cluster

2019-09-10 Thread Komal Mariam
I'm trying to set up a 3 node Flink cluster (version 1.9) on the following machines: Node 1 (Master) : 4 GB (3.8 GB) Core2 Duo 2.80GHz, Ubuntu 16.04 LTS Node 2 (Slave) : 16 GB, Core i7-3.40GHz, Ubuntu 16.04 LTS Node 3 (Slave) : 16 GB, Core i7-3,40GHz, Ubuntu 16.04 LTS I have followed the