[Discuss] Add JobListener (hook) in flink job lifecycle

Hi All, I created FLINK-12214 for adding JobListener (hook) in flink job lifecycle. Since this is a new public api for flink, so I'd like to discuss it more widely in community to get more feedback. The background and motivation is that I am

flink program in a spring bean can not consume from kafka

hi, i met a strange issue, the same code running in a java class can consume kafka , but when i change the java class to a spring bean(annotation is @service) , the program can not consume kafka amymore. does anyone met the similar problems or how can i debug this problems? thanks a lot

Re: assignTimestampsAndWatermarks not work after KeyedStream.process

Hi, Could you check the watermark of the window operator? One possible situation would be some of the keys are not getting enough inputs, so their watermarks remain below the window end time and hold the window operator watermark back. IMO, it’s a good practice to assign watermark earlier in

Re: Service discovery on YARN - find out which port was dynamically assigned to the JobManager Web Interface

As far as I know, the port will be set to random binding. Yarn actually have the ability to translate the proxy link to the right node/port. If your goal is trying to avoid going through the YARN rest proxy, this could be a problem: There's chances that the host/port will get changed by YARN

Re: java.io.IOException: NSS is already initialized

I think I found the root cause https://bugs.alpinelinux.org/issues/10126 I have to re-install nss after apk update/upgrade Hao Sun On Sun, Nov 11, 2018 at 10:50 AM Ufuk Celebi wrote: > Hey Hao, > > 1) Regarding Hadoop S3: are you using the repackaged Hadoop S3 > dependency from the /opt

BucketingSink compressed text exactlyonce

Hi guys Any good ideas to achieve exactly once BucketingSink for text file？truncating compressed binary file will corrupt the gzip file which means I need to -text that gzip and redirect to a text file then compressed it agan and finally upload to hdfs. Its really inefficient. Any other

assignTimestampsAndWatermarks not work after KeyedStream.process

`assignTimestampsAndWatermarks` before `keyBy` works: ```java DataStream trips = env.addSource(consumer).assignTimestampsAndWatermarks(new BoundedOutOfOrdernessTimestampExtractor(Time.days(1)) { @Override public long extractTimestamp(Trip trip) {

Service discovery on YARN - find out which port was dynamically assigned to the JobManager Web Interface

Hello, I want to be able to register a flink cluster into a service discovery system (Consul in our case). This flink cluster is scheduled on YARN. Is there a way to know which port was assigned to the rest interface ? Via the rest API /jobmanager/config, I see a key

Re: What is the best way to handle data skew processing in Data Stream applications?

I guess I could implement a solution which is not static and extends the OneInputStreamOperator Flink operator. https://github.com/felipegutierrez/explore-flink/blob/master/src/main/java/org/sense/flink/examples/stream/MqttSensorDataSkewedCombinerByKeySkewedDAG.java#L84 Best, Felipe *--* *--

Re: 回复：Is it possible to handle late data when using table API?

Hi Hequn Thanks for the details. I will give it a try. Med venlig hilsen / Best regards Lasse Nedergaard > Den 17. apr. 2019 kl. 04.09 skrev Hequn Cheng : > > Hi Lasse, > > > some devices can deliver data days back in time and I would like to have > > the results as fast as possible. > >

Re: How would I use OneInputStreamOperator to deal with data skew?

There is no reason for it, the operator and function doesn't rely on the logic which AbstractUdfStreamOperator supplied. Best, Kurt On Wed, Apr 17, 2019 at 4:35 PM Felipe Gutierrez < felipe.o.gutier...@gmail.com> wrote: > Thanks for the tip! I guess now it is working as it should be >

Re: How would I use OneInputStreamOperator to deal with data skew?

I mean no particular reason. Best, Kurt On Wed, Apr 17, 2019 at 7:44 PM Kurt Young wrote: > There is no reason for it, the operator and function doesn't rely on the > logic which AbstractUdfStreamOperator supplied. > > Best, > Kurt > > > On Wed, Apr 17, 2019 at 4:35 PM Felipe Gutierrez < >

Re: Scalaj vs akka as http client for Asyncio Flink

Check the logs what Akka is logging and verify that the port you try to bind to is free. Cheers, Till On Wed, Apr 17, 2019 at 12:50 PM Andy Hoang wrote: > Hi Till, > > Sorry to bother you again, so I manage to build and work with akka http > client in my local > After deploy to yarn node, the

Re: Scalaj vs akka as http client for Asyncio Flink

Hi Till, Sorry to bother you again, so I manage to build and work with akka http client in my local After deploy to yarn node, the actorsystem cant be connected. ``` PPLogger.getActivityLogger.info("### 1") implicit val system = ActorSystem("my-system")

Fast restart of a job with a large state

Hi Flinkers, Operating different flink jobs I've discovered that job restarts with a pretty large state (in my case this is up to 100GB+) take quite a lot of time. For example, to restart a job (e.g. to update it) the savepoint is created, and in case of savepoints all the state seems to be

kafka partitions, data locality

Hello, We planning to use apache flink as a core component of our new streaming system for internal processes (finance, banking business) based on apache kafka. So we starting some research with apache flink and one of the question, arises during that work, is how flink handle with data

Re: How would I use OneInputStreamOperator to deal with data skew?

Thanks for the tip! I guess now it is working as it should be . Just one last question. Why did you decide to use "AbstractStreamOperator" instead