回复:Flink Yarn session

2018-01-28 Thread 邓俊华
hi, Please check the logs in yours flink  stream app to see if there is some issues connecting yarn resourcemanager.I also encounted the issue when run flink stream in yarn with HA and we have discussed the issue in the preceding  months,please check the issue mail url in the below.http://mail-

Re: Aggregation using event timestamp than clock window

2018-01-28 Thread Rohan Thimmappa
Hi Gary, Thanks.I do have some of the events coming in after one pauses and i am able to see watermarked being advanced event being triggered. Rohan On Mon, Jan 15, 2018 at 5:40 AM, Gary Yao wrote: > Hi Rohan, > > In your example, are you saying that after 5:40 you will not receive any > ev

How to make savepoints more robust in the face of refactorings ?

2018-01-28 Thread jelmer
Changing the class operators are nested in can break compatibility with existing savepoints. The following piece of code demonstrates this https://gist.github.com/jelmerk/e55abeb0876539975cd32ad0ced8141a If I change Operators in this file to Operators2 i will not be able to recover from a savepo

End-to-end exactly once from kafka source to S3 sink

2018-01-28 Thread chris snow
I’m working with a kafka environment where I’m limited to 100 partitions @ 1GB log.retention.bytes each. I’m looking to implement exactly once processing from this kafka source to a S3 sink. If I have understood correctly, Flink will only commit the kafka offsets when the data has been saved to S

RE: S3 for state backend in Flink 1.4.0

2018-01-28 Thread Marchant, Hayden
I see that we can still use the other implementation, but were hoping that we'd benefit from the bug fix done in Flink 1.4.0 around 'repeated' load of configuration. I'll check with the old implementation and see if it still works. We also have seen discussions on a more native protocol that i