Re: SideOutput doesn't receive anything if filter is applied after the process function

Could someone with knowledge of the right terms create this in JIRA, please? I guess I could also create it if needed.. On Mon, Jan 15, 2018 at 3:15 PM, Chesnay Schepler wrote: > yes, i meant that process() returns the special operator. This would > definitely deserve a JIRA

How to get automatic fail over working in Flink

This question has been asked on StackOverflow: https://stackoverflow.com/questions/48262080/how-to-get-automatic-fail-over-working-in-flink I am using Apache Flink 1.4 on a cluster of 3 machines, out of which one is the JobManager and the other 2 host TaskManagers. I start flink in cluster mode

MapState Initializion

Any response to this? List State is created using checkpointedState = context.getOperatorStateStore.getListState (checkPointDescriptor) But there is no similar APIs for MapState Boris Lublinsky FDP Architect boris.lublin...@lightbend.com https://www.lightbend.com/ > Begin forwarded message: >

Re: Flink CEP exception during RocksDB update

Hi Varun, This can be related to this issue: https://issues.apache.org/jira/browse/FLINK-8226 which is currently fixed on the master. Could you please try the current master to see if the error persists? Thanks, Kostas > On Jan 15, 2018, at

Flink CEP exception during RocksDB update

> Hello Flink community, > > I have encountered following exception while testing 1.4.0 release. This > error is occurring intermittently and my CEP job keeps restarting after this > exception. I am running the job with Event time semantics and checkpoints > enabled. > > >

Re: Trigger not firing when using BoundedOutOfOrdernessTimestampExtractor

Hi Fabian, I want to extract timestamps from my event. However, the events stream can be sparse at times (e.g. 2 days without any events). What's the best strategy to create watermarks if I want real-time processing of the events which enter the stream? Jayant Ameta On Thu, Jan 11, 2018 at 4:53

Re: Aggregation using event timestamp than clock window

Hi Rohan, In your example, are you saying that after 5:40 you will not receive any events at all which could advance the watermark? I am asking because if you are receiving events for other keys/ids from your KafkaSource after 5:40, the watermark will still be advanced and fire the tumbling

Re: SideOutput doesn't receive anything if filter is applied after the process function

yes, i meant that process() returns the special operator. This would definitely deserve a JIRA issue. On 15.01.2018 14:09, Juho Autio wrote: Thanks for the explanation. Did you meant that process() would return a SingleOutputWithSideOutputOperator? Any way, that should be enough to avoid the

Re: SideOutput doesn't receive anything if filter is applied after the process function

Thanks for the explanation. Did you meant that process() would return a SingleOutputWithSideOutputOperator? Any way, that should be enough to avoid the problem that I hit (and it also seems like the best & only way). Maybe the name should be something more generic though, like

Re: How to Join a dimension table in flink sql

Hi XiangWei, Unfortunately, it's not possible for the Flink SQL API to join a stream with a common dataset now. As a workaround, if the table is just a tiny one, you can achieve a inner/left outer join with the user defined table functions

Low throughput when trying to send data with Sockets

Hello, I am trying to separate the logic of my application by generating and processing data in different physical engines. I have created my custom socket source class: class SocketSourceFunction extends SourceFunction[Event2]{ @volatile private var isRunning:Boolean = true;

Re: What's the meaning of "Registered `TaskManager` at akka://flink/deadLetters " ?

Hi, Could you post full job manager and task manager logs from startup until the first signs of the problem? Thanks, Piotrek > On 15 Jan 2018, at 11:21, Reza Samee wrote: > > Thanks for response; > And sorry the passed time. > > The JobManager & TaskManager logged

Re: SideOutput doesn't receive anything if filter is applied after the process function

It would mean that getSideOutput() would return a SingleOutputWithSideOutputOperator which extends SingleOutputOperator offering getSideOutput(). Other transformations would still return a SingleOutputOperator. With this the following code wouldn't compile. stream .process(...)

Re: What's the meaning of "Registered `TaskManager` at akka://flink/deadLetters " ?

Thanks for response; And sorry the passed time. The JobManager & TaskManager logged ports are open! Is this log OK? 2018-01-15 13:40:03,455 INFO org.apache.flink.runtime.webmonitor.JobManagerRetriever - New leader reachable under akka.tcp://flink@172.16.20.18:6123/user/jobmanager:null.

How to Join a dimension table in flink sql

Hi all, Is it possible to join records read from a kafka stream with one(or more) dimension tables which are saved as mysql table using flink streaming sql.

Re: SideOutput doesn't receive anything if filter is applied after the process function

> sideoutput might deserve a seperate class which inherit form singleoutput. It might prevent lot of confusions Thanks, but how could that be done? Do you mean that if one calls .process(), then the stream would change to another class which would only allow calls like .getMainOutput() or

Scaling Flink

Hi All, Has anyone tried scaling out flink cluster on EMR based on CPU usage/ kafka lag/ back pressure monitoring? If so can you provide some insights on how it could be achieved and sample scripts if possible. Thanks a lot. Thanks, Navneeth

Re: Aggregation using event timestamp than clock window

No. My question is slightly different. say i get report from 5.10-5.40. the device went offline and never comes back. i will not get any report after 5.40. So 5-6 window never gets closed as we will not get any report after 5.40. in this case 5.00-5.40 data is still in flink memory that will