Re: Fink: KafkaProducer Data Loss

2017-07-31 Thread Tzu-Li (Gordon) Tai
ling-list-archive.2336050.n4.nabble.com/file/n14522/eventFilter.zip> -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Fink-KafkaProducer-Data-Loss-tp11413p14522.html Sent from the Apache Flink User Mailing List archive. mailing

Re: Fink: KafkaProducer Data Loss

2017-07-28 Thread ninad
or so. Thanks for all your help. eventFilter.zip <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/file/n14522/eventFilter.zip> -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Fink-KafkaProducer-Data-Loss-t

Re: Fink: KafkaProducer Data Loss

2017-07-13 Thread Piotr Nowojski
link 1.3.2 and 1.4 releases. >> >> Piotrek >> >>> On Jul 12, 2017, at 7:32 PM, ninad <nni...@gmail.com >>> <mailto:nni...@gmail.com>> wrote: >>> >>> Hey guys, any update on this? If needed I can attach our code. >>> >>>

Re: Fink: KafkaProducer Data Loss

2017-07-13 Thread Tzu-Li (Gordon) Tai
-- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Fink-KafkaProducer-Data-Loss-tp11413p14224.html Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.

Re: Fink: KafkaProducer Data Loss

2017-07-13 Thread Piotr Nowojski
message in context: > http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Fink-KafkaProducer-Data-Loss-tp11413p14224.html > Sent from the Apache Flink User Mailing List archive. mailing list archive at > Nabble.com.

Re: Fink: KafkaProducer Data Loss

2017-07-12 Thread ninad
Hey guys, any update on this? If needed I can attach our code. -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Fink-KafkaProducer-Data-Loss-tp11413p14224.html Sent from the Apache Flink User Mailing List archive. mailing list archive

Re: Fink: KafkaProducer Data Loss

2017-06-16 Thread ninad
ge in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Fink-KafkaProducer-Data-Loss-tp11413p13805.html Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.

Re: Fink: KafkaProducer Data Loss

2017-06-15 Thread Aljoscha Krettek
few messages were not replayed, and the kafka sink didn't receive all the >> messages. >> >> Attaching the files here. >> >> jmCloudera583.log >> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/file/n13597/jmCloudera583.log> >&

Re: Fink: KafkaProducer Data Loss

2017-06-11 Thread ninad
nk-user-mailing-list-archive.2336050. > n4.nabble.com/file/n13597/tmOneCloudera583.log> > tmTwoCloudera583.log > <http://apache-flink-user-mailing-list-archive.2336050. > n4.nabble.com/file/n13597/tmTwoCloudera583.log> > > BTW, I po

Re: Fink: KafkaProducer Data Loss

2017-06-11 Thread Tzu-Li (Gordon) Tai
that. Thanks for your patience guys. -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Fink-KafkaProducer-Data-Loss-tp11413p13597.html Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.

Re: Fink: KafkaProducer Data Loss

2017-06-08 Thread ninad
. Hope you guys saw that. Thanks for your patience guys. -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Fink-KafkaProducer-Data-Loss-tp11413p13597.html Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.

Re: Fink: KafkaProducer Data Loss

2017-06-06 Thread ninad
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Fink-KafkaProducer-Data-Loss-tp11413p13549.html Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.

Re: Fink: KafkaProducer Data Loss

2017-06-06 Thread ninad
Not yet. Planning to do that today. -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Fink-KafkaProducer-Data-Loss-tp11413p13544.html Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.

Re: Fink: KafkaProducer Data Loss

2017-06-06 Thread Tzu-Li (Gordon) Tai
13527/tmTwo.log> -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Fink-KafkaProducer-Data-Loss-tp11413p13527.html Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.

Re: Fink: KafkaProducer Data Loss

2017-06-06 Thread ninad
wo.log <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/file/n13527/tmTwo.log> -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Fink-KafkaProducer-Data-Loss-tp11413p13527.html Sent from the Apache Flink User Ma

Re: Fink: KafkaProducer Data Loss

2017-06-06 Thread Till Rohrmann
inad ([hidden email] >> <http:///user/SendEmail.jtp?type=node=13479=0>) wrote: >> >> I tested this with the standalone cluster, and I don't see this problem. >> So, >> the problem could be that we haven't built Flink against cloudera Hadoop? >> I >> will test it

Re: Fink: KafkaProducer Data Loss

2017-06-04 Thread ninad
sted this with the standalone cluster, and I don't see this problem. > So, > the problem could be that we haven't built Flink against cloudera Hadoop? > I > will test it out. > > > > -- > View this message in context: http://apache-flink-user- > mailing-list-archive.2

Re: Fink: KafkaProducer Data Loss

2017-06-04 Thread Tzu-Li (Gordon) Tai
? I will test it out. -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Fink-KafkaProducer-Data-Loss-tp11413p13477.html Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.

Re: Fink: KafkaProducer Data Loss

2017-06-04 Thread ninad
I tested this with the standalone cluster, and I don't see this problem. So, the problem could be that we haven't built Flink against cloudera Hadoop? I will test it out. -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Fink-KafkaProducer

Re: Fink: KafkaProducer Data Loss

2017-06-02 Thread ninad
aLoss.log> tmTwoNoDataLoss.log <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/file/n13467/tmTwoNoDataLoss.log> -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Fink-KafkaProducer-Data-Loss-tp11413p13467.html Sent fro

Re: Fink: KafkaProducer Data Loss

2017-06-02 Thread ninad
Thanks Till. The log files I have attached are the complete logs. They are DEBUG level. There are three files: jobManger.log, tmOne.log and tmTwo.log. -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Fink-KafkaProducer-Data-Loss-tp11413p13463

Re: Fink: KafkaProducer Data Loss

2017-06-02 Thread Till Rohrmann
e7f2caa0b75.* > > Now, if I bring up all Kafka brokers, I am expecting that the messages > which > didn't make it to the Kafka sink should be sent. > > But that's not happening. > > All these logs are present in the files that I attached. > > I am going to try this on

Re: Fink: KafkaProducer Data Loss

2017-06-02 Thread ninad
uld be sent. But that's not happening. All these logs are present in the files that I attached. I am going to try this on the standalone cluster today. -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Fink-KafkaProducer-Data-Loss-tp11413p13

Re: Fink: KafkaProducer Data Loss

2017-06-02 Thread Tzu-Li (Gordon) Tai
er=LEADER_NOT_AVAILABLE} -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Fink-KafkaProducer-Data-Loss-tp11413p13443.html Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.

Re: Fink: KafkaProducer Data Loss

2017-06-01 Thread Tzu-Li (Gordon) Tai
gt; at > org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducerBase.snapshotState(FlinkKafkaProducerBase.java:335) > > > > > > > > -- > View this message in context: > http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Fink-KafkaProducer-Data-Loss-tp11413p13412.html > > Sent from the Apache Flink User Mailing List archive. mailing list archive at > Nabble.com.

Re: Fink: KafkaProducer Data Loss

2017-06-01 Thread Kostas Kloudas
e > Caused by: java.lang.Exception: Failed to send data to Kafka: Batch Expired > at > org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducerBase.checkErroneous(FlinkKafkaProducerBase.java:366) > at > org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducerBase.snapshotStat

Re: Fink: KafkaProducer Data Loss

2017-02-03 Thread ninad
Thanks, Gordon and Till. -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Fink-KafkaProducer-Data-Loss-tp11413p11431.html Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.

Re: Fink: KafkaProducer Data Loss

2017-02-02 Thread Tzu-Li (Gordon) Tai
ka settings. So, is our understanding correct? Or is there a way we can avoid this data loss by modifying some Flink settings? Thanks. -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Fink-KafkaProducer-Data-Loss-tp11413.html Sent from the A

Re: Fink: KafkaProducer Data Loss

2017-02-02 Thread Till Rohrmann
g the possible side effects of changing the above > kafka settings. > > So, is our understanding correct? Or is there a way we can avoid this data > loss by modifying some Flink settings? > > Thanks. > > > > -- > View this message in context: http://apache-flink-user- > mailing-list-archive.2336050.n4.nabble.com/Fink-KafkaProducer-Data-Loss- > tp11413.html > Sent from the Apache Flink User Mailing List archive. mailing list archive > at Nabble.com. >

Fink: KafkaProducer Data Loss

2017-02-02 Thread ninad
an avoid this data loss by modifying some Flink settings? Thanks. -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Fink-KafkaProducer-Data-Loss-tp11413.html Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.