Re: Netty channel closed at AKKA gated status

2019-04-19 Thread Wenrui Meng
Looked at a few same instances. The lost task manager was indeed not active anymore since there is no log for that task manager printed after the connection issue timestamp. I guess somehow that task manager died silently without exception or termination relevant information logged. I double checke

Re: Create Dynamic data type

2019-04-19 Thread Rong Rong
Hi Soheil, If I understand correctly, when you said "according to the number of rows", you were trying to dynamically determine the RowType based on how long one row is, correct? In this case, I am not sure this is considered supported in JDBCInputFormat at this moment and it would be hard to supp

RichAsyncFunction Timer Service

2019-04-19 Thread Mikhail Pryakhin
Hello, Flink community! It happens that I need to access a timer service in a RichAsyncFunction implementation. I know it's normally accomplished via StreamingRuntimeContext instance available in a RichFunction, but unfortunately, RichAsyncFunction extending RichFunction overrides “setRuntimeCo

Create Dynamic data type

2019-04-19 Thread Soheil Pourbafrani
Hi, Using JDBCInputFormat I want to read data from database but the problem is the table columns are dynamic according to the number of rows. In the schema the first column is of type int and in the rest of the column the first half is String and the second half is double. So I need a way to creat

Re: assignTimestampsAndWatermarks not work after KeyedStream.process

2019-04-19 Thread an0
Hi, First of all, thank you for the `shuffle()` tip. It works. However, I still don't understand why it doesn't work without calling `shuffle()`. Why would not all BoundedOutOfOrdernessTimestampExtractors receive trips? All the trips has keys and timestamps. As I said in my reply to Paul, I se

Re: PatternFlatSelectAdapter - Serialization issue after 1.8 upgrade

2019-04-19 Thread Oytun Tez
Forgot to answer one of your points: the parent class compiles well without this CEP selector (with timeout signature)... --- Oytun Tez *M O T A W O R D* The World's Fastest Human Translation Platform. oy...@motaword.com — www.motaword.com On Fri, Apr 19, 2019 at 9:40 AM Oytun Tez wrote: > He

Re: PatternFlatSelectAdapter - Serialization issue after 1.8 upgrade

2019-04-19 Thread Oytun Tez
Hey JingsongLee! Here are some findings... - flatSelect *without timeout* works normally: patternStream.flatSelect(PatternFlatSelectFunction), this compiles well. - Converted the both timeout and select selectors to an *inner class* (not static), yielded the same results, doesn't comp

TM occasionally hang in deploying state in Flink 1.5

2019-04-19 Thread qi luo
Hi all, We use Flink 1.5 batch and start thousands of jobs per day. Occasionally we observed some stuck jobs, due to some TM hang in “DEPLOYING” state. On checking TM log, it shows that it stuck in downloading jars in BlobClient: ... INFO org.apache.flink.runtime.taskexecutor.TaskExecuto

RE: kafka partitions, data locality

2019-04-19 Thread Smirnov Sergey Vladimirovich (39833)
Hi Ken, It’s a bad story for us: even for a small window we have a dozens of thousands events per job with 10x in peaks or even more. And the number of jobs was known to be high. So instead of N operations (our producer/consumer mechanism) with shuffle/resorting (current flink realization) it w

Re: [Discuss] Add JobListener (hook) in flink job lifecycle

2019-04-19 Thread Jeff Zhang
>>> The ExecutionEnvironment is usually used by the user who writes the code and this person (I assume) would not be really interested in these callbacks. Usually ExecutionEnvironment is used by the user who write the code, but it doesn't needs to be created and configured by this person. e.g. in

Re: PatternFlatSelectAdapter - Serialization issue after 1.8 upgrade

2019-04-19 Thread JingsongLee
Hi @Oytun Tez It Looks like your PatternFlatSelectFunction is not serializable. Because you use anonymous inner class, Check the class to which getPending belongs, maybe that class is not serializable? Or you may be advised not to use internal classes, but to use a static internal class. Best, J