Flink isn't logging once we use RollingFileAppender

2017-08-09 Thread Hussein Baghdadi
Hello, We changed our log4j.properties file of our Flink cluster to: log4j.appender.file=org.apache.log4j.RollingFileAppender log4j.appender.file.File=/opt/flink/log/flink.log log4j.appender.file.MaxFileSize=100MB log4j.appender.file.MaxBackupIndex=2 log4j.appender.file.layout=org.apache.log4j.Pa

A Link Sink that writes to OAuth2.0 protected resource

2017-03-03 Thread Hussein Baghdadi
Hello, In our Sink, we are dealing with a system that uses OAuth 2.0. So the in the open() method of the Sink we are getting the token and then we initialise the client that we can use in order to write from Flink to that API. Is there a better approach to handle that? open() is a lifecycle m

How to write a Sink that needs to deal with conditions in thread-safe way?

2017-03-02 Thread Hussein Baghdadi
Hello, I have some basic questions regarding Sinks in Flink. 1) To implement our own Sink, which class to implement: RichSinkFunction, RichOutputFormat, etc .. 2) We are trying to write batches in our Sink. For that, in overrided invoke() , we are calling a synchronised function: // events =