Re: Reset Kafka Consumer using Flink Consumer 10 API

Thanks for Reply Robert . How do I specify start position of consumer for FlinkKafkaConsumer010? Because methods e.g. setStartFromSpecificOffsets sepecified in documentation

Re: How can I cancel a Flink job safely without a special stop message in the stream?

Hi, We use kafka because: - it is a high throughput message queue   - we want to have about 2GB/s in/write and 7GB/s out/read perforamance (400B/msg) - it is popular (well this is kind of important...) - the input has a start and an end, but we want to process new data as soon as

Issues in recovering state from last crash using custom sink

Hi all, I am working on a flink archiver application. In a gist this application tries to reads a bunch of schematized messages from kafka and archives them to s3. Due to the nature of the naming of the files, I had to go towards a custom sink implementation. As of the current progress the

Re: Flink Elastic Sink AWS ES

Hi Arpit, The response fromm _nodes doesn’t contain an ip address in my case. Is this something that you experienced? > curl -XGET 'http:///_nodes' Thanks, > On 27 Aug 2017, at 14:32, ant burton wrote: > > Thanks! I'll check later this evening. > > On Sun, 27 Aug

Re: Even out the number of generated windows

Hi Robert, Thank you for the suggestion, I'll try that. On a second thought, I can actually reduce the amount of generated output so there aren't that many records being sent to Kinesis. What I want to do is to use Flink's state to keep track of the last computation result of a

Re: Flink-HTM integration

Sure. Firstly I followed the steps showed here to build the project: flink-htm github In my project I want to perform anomaly detection of values in a stream. I have a Kafka broker as source: /DataStream

Re: Flink-HTM integration

Can you give us some detail on how flink-HTM is integrated in your project ? Cheers On Sun, Aug 27, 2017 at 6:28 AM, AndreaKinn wrote: > I think this is not a good idea, I don't know if it's a bug or a my fault. > I believe I integrated correctly flink-HTM in my project

Re: Flink Elastic Sink AWS ES

Hi, We have recently moved to AWS ES service, I am using the following code: https://github.com/awslabs/flink-stream-processing-refarch/blob/master/flink-taxi-stream-processor/src/main/java/com/amazonaws/flink/refarch/utils/ElasticsearchJestSink.java (Note that this is not the inbuilt Flink

Re: Flink-HTM integration

I think this is not a good idea, I don't know if it's a bug or a my fault. I believe I integrated correctly flink-HTM in my project (but I'm not sure, Flink-HTM is still an embryonal phase I think) and simply I can't see any output after the HTM elaboration, even there are no errors on console.

Re: Even out the number of generated windows

Hi Bowen, I don't know what kind of relationship your company has to AWS, maybe they are willing to look into the issue from their side. To throttle a stream, I would recommend just doing a map operation that is calling "Thread.sleep()" every n records. On Sat, Aug 26, 2017 at 4:11 AM, Bowen

Re: The implementation of the RichSinkFunction is not serializable.

Hi, could you elaborate, please? Marking conf, connection and table as transient wouldn't help because of the presence of the HTableDescriptor reference? 2017-08-27 12:44 GMT+02:00 Jörn Franke : > It looks like that in your case everything should be serializable. An >

The implementation of the RichSinkFunction is not serializable.

Hi, I'm trying to write on HBase using writeOutputFormat using a custom HBase format inspired from this example in flink-hbase (mind you, I'm

Re: Flink AggregateFunction Help

Fixed. Issue with Timestamp. That is why the windowing function wasnt working properly. -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Flink-AggregateFunction-Help-tp15160p15167.html Sent from the Apache Flink User Mailing List archive.

Streaming job gets slower and slower

We have a flink job with 7 subtasks. See graph below. This is on flink 1.2. [cid:image001.png@01D31EC7.FB4B5620] Here each source task consumes from a kafka topic. Data rate is low around 70-80 messages per sec. What we are noticing is after running for 2 hours or so the source tasks starts

Re: Flink Elastic Sink AWS ES

We also had same setup where ES cluster was behind a proxy server for which port 80 was used which redirected it to ES cluster 9200 port. For using Flink we got the actual ip address of the ES nodes and put that in ips below. transportAddresses.add(new